From 2975ceddcef744a1c4aa5cd92431e7431c4342ab Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:01:54 +0200 Subject: [PATCH 01/81] cas: classify transient GC round failures and stream emulated blob publication Two defects surfaced by the `content_addressed_garbage_collection_log` scenario cards for issue #2233. Any `S3_ERROR` timeout during a GC round was recorded as an indistinguishable `Failed` outcome with a free-text error, and a failed round zeroed out the real counters and cleared `i_am_leader`, suppressing the heartbeat and provoking leadership ping-pong on a flaky backend. Transient error codes (`S3_ERROR`, `NETWORK_ERROR`, `ABORTED`, timeouts, `MEMORY_LIMIT_EXCEEDED`) now produce an `Aborted` outcome while keeping leadership, and `system.cas_gc_log` gains an `error_code` column alongside the `Aborted` outcome. Separately, the emulated blob-publication path materialized the whole blob body in memory (about 1 GiB for a 512 MiB blob) under a global mutex; it now streams the body instead. Related: https://github.com/Altinity/ClickHouse/issues/2233 Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../en/operations/system-tables/cas_gc_log.md | 5 +- .../Backend/CasObjectStorageBackend.cpp | 76 +++---- .../Backend/CasObjectStorageBackend.h | 6 +- .../ContentAddressedMetadataStorage.cpp | 4 + .../ContentAddressed/Gc/CasGc.cpp | 7 +- .../ContentAddressed/Gc/CasGc.h | 9 +- .../ContentAddressed/Gc/CasGcScheduler.cpp | 82 +++++-- .../ContentAddressed/Gc/CasGcScheduler.h | 15 +- src/Disks/tests/gtest_cas_gc_log.cpp | 211 +++++++++++++++++- .../ContentAddressedGarbageCollectionLog.cpp | 8 +- .../ContentAddressedGarbageCollectionLog.h | 6 +- 11 files changed, 363 insertions(+), 66 deletions(-) diff --git a/docs/en/operations/system-tables/cas_gc_log.md b/docs/en/operations/system-tables/cas_gc_log.md index 5fd04b4fb11d..eb422cddc4d1 100644 --- a/docs/en/operations/system-tables/cas_gc_log.md +++ b/docs/en/operations/system-tables/cas_gc_log.md @@ -38,7 +38,7 @@ specified (it is enabled by default in the shipped `config.xml`). - `gc_id` ([String](/sql-reference/data-types/string)) — The GC scheduler instance id (which mounter ran the round). - `trigger` ([Enum8](/sql-reference/data-types/enum)) — `Scheduled` (background tick) or `Manual` (`SYSTEM` command). - `round` ([UInt64](/sql-reference/data-types/int-uint)) — The GC round number (`0` on a `Start` row). -- `outcome` ([Enum8](/sql-reference/data-types/enum)) — `Unknown` (on a `Start` row), `Success` (led, folded, and completed), `NotALeader` (another replica holds the GC lease), `Deferred` (led but took the skip-unchanged fast path — no fold ran, because no changed shard reached the fold threshold and no graduation was due), or `Error` (the round threw). +- `outcome` ([Enum8](/sql-reference/data-types/enum)) — `Unknown` (on a `Start` row), `Success` (led, folded, and completed), `NotALeader` (another replica holds the GC lease), `Deferred` (led but took the skip-unchanged fast path — no fold ran, because no changed shard reached the fold threshold and no graduation was due), `Aborted` (the round threw a transient error — backend unavailability, a lost lease, a concurrent leader; the next scheduled round retries it), or `Error` (the round threw a non-transient error — investigate). - `candidates_marked` ([UInt64](/sql-reference/data-types/int-uint)) — Objects retired (marked) this round. - `objects_deleted` ([UInt64](/sql-reference/data-types/int-uint)) — Objects physically deleted this round. - `objects_absent` ([UInt64](/sql-reference/data-types/int-uint)) — Retire candidates found already absent. @@ -51,7 +51,8 @@ specified (it is enabled by default in the shipped `config.xml`). - `fence_outs` ([UInt64](/sql-reference/data-types/int-uint)) — Expired mounts fenced out by this round's heartbeat floor. - `anomalies` ([UInt64](/sql-reference/data-types/int-uint)) — Fold clamps surfaced (and survived) this round. A steady non-zero value warrants a look at the round log details. - `duration_ms` ([UInt64](/sql-reference/data-types/int-uint)) — The round wall-clock duration (on a `Finish` row). -- `error` ([String](/sql-reference/data-types/string)) — The exception text when `outcome = 'Error'`. +- `error` ([String](/sql-reference/data-types/string)) — The exception text when `outcome = 'Aborted'` or `'Error'`. +- `error_code` ([Int32](/sql-reference/data-types/int-uint)) — The exception code when `outcome = 'Aborted'` or `'Error'`; `0` otherwise. Key monitoring on this column rather than on the `error` text. On an `Aborted` or `Error` row the counters still report everything the round completed before it threw, and `round != 0` on such a row means the round's closing compare-and-swap committed and the failure hit only post-commit cleanup. - `ProfileEvents` ([Map(LowCardinality(String), UInt64)](/sql-reference/data-types/map)) — On a `Start`/`Finish` row, the per-round `ProfileEvents` delta (the `CAS*` counters and S3/disk events for this round). On a `Phase` row, **that phase's** delta, so `GROUP BY phase` over `ProfileEvents['S3ListObjects']` attributes the round's `LIST` budget to the phase that spent it. - `round_id` ([String](/sql-reference/data-types/string)) — The correlator for every row of one round attempt: its `Start`, each of its `Phase` rows, and its `Finish`. Minted per attempt, so unlike `round` it exists even for a round that never committed and for a round that never led. Group by this column to reconstruct one round. - `phase` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — The GC phase this row describes; empty on `Start`/`Finish`. See [Per-phase rows](#per-phase-rows) for the phase list. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index b7b04fdb401b..d27e01b412b0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -252,13 +252,6 @@ PutResult ObjectStorageBackend::nativeConditionalPut(const String & key, const S namespace { -/// Keep the emulated backend's publication memory bound to one materialized body at a time. -std::mutex & emulatedBlobPublicationMutex() -{ - static std::mutex mutex; - return mutex; -} - } /// True when an exception from `IObjectStorage::readObject` means "the object is simply not there". @@ -485,7 +478,7 @@ Token ObjectStorageBackend::emuWrite(const String & key, const String & bytes, c return emuMintToken(key, metadata ? metadata->etag : String{}, /*just_wrote=*/true); } -void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const String & bytes) +void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const String & envelope, ReadBuffer & payload, uint64_t payload_size) { if (object_storage->getType() != ObjectStorageType::Local) throw Exception( @@ -497,14 +490,30 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St const String root = object_storage->getCommonKeyPrefix(); const String destination_path = resolvePathRelativelyToBase(destination_object, root); const String temporary_path = resolvePathRelativelyToBase(temporary_object, root); - const auto existing_token_state = emu_token_state.find(key); + /// The body is STREAMED into the temporary file -- envelope, then a bounded copy of the payload -- + /// never materialized in memory. (An earlier revision accumulated envelope+payload in one String, + /// whose growth doubling made the peak allocation up to 2x the payload, and serialized every + /// publication behind a dedicated mutex just to bound that peak to one body at a time; streaming + /// removes both.) The destination stays untouched until the byte count has been validated: a short + /// or long source aborts on the temporary file, which is then removed. try { auto out = object_storage->writeObject(StoredObject(temporary_object), WriteMode::Rewrite); - out->write(bytes.data(), bytes.size()); + out->write(envelope.data(), envelope.size()); + const auto copy_result = blob_publication_detail::copyBlobPayloadBounded(payload, *out, payload_size); + if (!copy_result.exact(payload_size)) + { + out->cancel(); + throw Exception( + ErrorCodes::CORRUPTED_DATA, + "ObjectStorageBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", + copy_result.has_excess ? "more than " : "", + copy_result.copied, + key, + payload_size); + } out->finalize(); - std::filesystem::rename(temporary_path, destination_path); } catch (...) { @@ -517,7 +526,21 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St /// existing disambiguator is sufficient: if the next observation sees the same ETag, it returns /// a token distinct from the old incarnation; if the ETag changed, emuMintToken resets the state /// to that new ETag. With no existing state, this backend has issued no same-process stale token - /// that needs fencing. The post-rename increment cannot allocate or throw. + /// that needs fencing. The post-rename increment cannot allocate or throw. `emu_mutex` spans the + /// rename and the bump so a concurrent emulated observation never sees the new incarnation with + /// the old disambiguator. + std::lock_guard lock(emu_mutex); + const auto existing_token_state = emu_token_state.find(key); + try + { + std::filesystem::rename(temporary_path, destination_path); + } + catch (...) + { + std::error_code cleanup_error; + std::filesystem::remove(temporary_path, cleanup_error); + throw; + } if (existing_token_state != emu_token_state.end()) ++existing_token_state->second.second; } @@ -872,32 +895,9 @@ void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) if (mode != Mode::Native) { - /// The emulated adapter's writes are whole-body operations. Serialize materialization so - /// concurrent publications retain the existing one-body peak-memory bound. - std::lock_guard publish_lock(emulatedBlobPublicationMutex()); - - String body = streaming->fresh_envelope; - blob_publication_detail::BlobPayloadCopyResult copy_result; - { - WriteBufferFromString out(body, AppendModeTag{}); - copy_result = blob_publication_detail::copyBlobPayloadBounded(*payload, out, streaming->payload_size); - if (copy_result.exact(streaming->payload_size)) - out.finalize(); - else - out.cancel(); - } - - if (!copy_result.exact(streaming->payload_size)) - throw Exception( - ErrorCodes::CORRUPTED_DATA, - "ObjectStorageBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", - copy_result.has_excess ? "more than " : "", - copy_result.copied, - request.destination_key, - streaming->payload_size); - - std::lock_guard lock(emu_mutex); - emuPublishBlobAtomically(request.destination_key, body); + /// Streams straight into the temporary file and renames -- see emuPublishBlobAtomically. + emuPublishBlobAtomically( + request.destination_key, streaming->fresh_envelope, *payload, streaming->payload_size); return; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index bd369d4e3603..7344c8fbede4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -264,7 +264,11 @@ class ObjectStorageBackend final : public Backend /// Write a complete blob body to a sibling temporary local object, then atomically replace `key` /// and advance any existing same-ETag disambiguator. A failure before the rename leaves the old /// destination and its token state untouched and cleans the temporary. - void emuPublishBlobAtomically(const String & key, const String & bytes); + /// Streams `envelope` + exactly `payload_size` bytes of `payload` into a temporary sibling of + /// `key`, then renames it into place -- nothing is visible at the destination until the byte count + /// has been validated, and the rename keeps publication atomic. Takes `emu_mutex` itself (for the + /// rename + token-state bump only); the caller must NOT hold it. + void emuPublishBlobAtomically(const String & key, const String & envelope, ReadBuffer & payload, uint64_t payload_size); /// Return the current emulated token for a key we just read/HEAD'd, reflecting its on-disk etag — /// does NOT advance the same-etag disambiguator (that only applies to a just-completed write). Token emuObserveToken(const String & key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index c9cf2b166389..447fe908e45e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -547,6 +547,9 @@ Cas::GcRoundLogger ContentAddressedMetadataStorage::makeGcRoundLogger() const case Cas::GcRoundLogRecord::Outcome::Deferred: e.outcome = ContentAddressedGarbageCollectionLogElement::DEFERRED; break; + case Cas::GcRoundLogRecord::Outcome::Aborted: + e.outcome = ContentAddressedGarbageCollectionLogElement::ABORTED; + break; } e.round = r.round; e.candidates_marked = r.candidates_marked; @@ -562,6 +565,7 @@ Cas::GcRoundLogger ContentAddressedMetadataStorage::makeGcRoundLogger() const e.anomalies = r.anomalies; e.duration_ms = r.duration_ms; e.error = r.error; + e.error_code = r.error_code; e.profile_events = r.profile_events; e.round_id = r.round_id; e.phase = r.phase; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 0a0e7cf2725c..358f619d5b34 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -348,9 +348,12 @@ void Gc::runNamespaceJanitorPage( t.metric("leaked", janitor_result.leaked); } -RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool allow_steal, UniversePolicy policy) +RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool allow_steal, UniversePolicy policy, + RoundReport * progress) { - RoundReport report; + RoundReport local_report; + RoundReport & report = progress ? *progress : local_report; + report = RoundReport{}; GcState state; Token state_token; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index 2754fe433c19..93209b70a3ff 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -429,8 +429,15 @@ class Gc /// `policy` is the destructive gate's universe seam — see `UniversePolicy`. Production passes /// nothing; a test whose subject is the suppressed gate passes `StageA_Suppressed` here, which is /// the only way to reach that posture. + /// `progress` (optional) is the caller's window into a round that THROWS: the round accumulates its + /// report directly in `*progress` (reset at entry) as each phase completes, so on an exception the + /// caller still sees everything the round durably did before it died -- `round` is stamped only + /// after the round's single `gc/state` CAS commits, so `progress->round != 0` on a failed round + /// proves the round committed and died in the post-CAS tail. On the success path `*progress` equals + /// the returned report. RoundReport runRegularRound(std::function on_lease_acquired = {}, bool allow_steal = true, - UniversePolicy policy = UniversePolicy::kDefault); + UniversePolicy policy = UniversePolicy::kDefault, + RoundReport * progress = nullptr); /// Advisory heartbeat: bump /gc/hb to {gc_id, hb_seq+1}. Best-effort (a lost CAS is /// harmless — the next pulse retries). Touches NO Gc instance state. Static by design. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp index f265444576a8..0ae8351de5ba 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp @@ -12,9 +12,35 @@ #include #include +namespace DB::ErrorCodes +{ + extern const int S3_ERROR; + extern const int NETWORK_ERROR; + extern const int ABORTED; + extern const int TIMEOUT_EXCEEDED; + extern const int SOCKET_TIMEOUT; + extern const int MEMORY_LIMIT_EXCEEDED; +} + namespace DB::Cas { +bool isTransientGcRoundError(int code) +{ + /// Codes that name a condition which clears without intervention: the backend refused or timed out + /// (`S3_ERROR`, `NETWORK_ERROR`, `TIMEOUT_EXCEEDED`, `SOCKET_TIMEOUT`), another actor legitimately + /// moved shared state (`ABORTED` -- the round CAS's own "another leader advanced it"), or memory + /// pressure hit a manual round running on a budgeted query thread (`MEMORY_LIMIT_EXCEEDED`). + /// Everything else -- notably `LOGICAL_ERROR`, `CORRUPTED_DATA`, `BAD_ARGUMENTS` -- stays + /// non-transient BY OMISSION: an unrecognised code must read as a real failure, never as noise. + return code == ErrorCodes::S3_ERROR + || code == ErrorCodes::NETWORK_ERROR + || code == ErrorCodes::ABORTED + || code == ErrorCodes::TIMEOUT_EXCEEDED + || code == ErrorCodes::SOCKET_TIMEOUT + || code == ErrorCodes::MEMORY_LIMIT_EXCEEDED; +} + namespace { /// Non-zero events of a per-round snapshot, keyed by event name. The snapshot is already a @@ -192,9 +218,29 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe Rec fin = start; fin.event_type = Rec::EventType::Finish; + /// Lives OUTSIDE the try and is filled progressively by the round (the `progress` out-parameter of + /// `runRegularRound`), so the Finish row of a THROWING round still carries everything the round + /// durably did before it died -- `round != 0` on such a row proves the round's `gc/state` CAS + /// committed and the failure hit the post-CAS tail. + Cas::RoundReport rep; + const auto fill_counters = [&fin](const Cas::RoundReport & r) + { + fin.round = r.round; + fin.candidates_marked = r.candidates; + fin.objects_deleted = r.deleted; + fin.objects_absent = r.absent; + fin.objects_replaced = r.replaced; + fin.objects_spared = r.spared; + fin.manifests_deleted = r.manifests_deleted; + fin.entries_condemned = r.condemned; + fin.entries_graduated = r.graduated; + fin.entries_redeleted = r.redeleted; + fin.fence_outs = r.fence_outs; + fin.anomalies = r.anomalies.size(); + }; try { - const Cas::RoundReport rep = round_gc.runRegularRound(std::move(on_lease_acquired), allow_steal); + (void)round_gc.runRegularRound(std::move(on_lease_acquired), allow_steal, Cas::UniversePolicy::kDefault, &rep); if (rep.acquired_lease) { /// Keep health state per scheduler. Process-global gauges cannot distinguish multiple @@ -210,18 +256,7 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe fin.outcome = !rep.acquired_lease ? Rec::Outcome::NotALeader : rep.deferred ? Rec::Outcome::Deferred : Rec::Outcome::Success; - fin.round = rep.round; - fin.candidates_marked = rep.candidates; - fin.objects_deleted = rep.deleted; - fin.objects_absent = rep.absent; - fin.objects_replaced = rep.replaced; - fin.objects_spared = rep.spared; - fin.manifests_deleted = rep.manifests_deleted; - fin.entries_condemned = rep.condemned; - fin.entries_graduated = rep.graduated; - fin.entries_redeleted = rep.redeleted; - fin.fence_outs = rep.fence_outs; - fin.anomalies = rep.anomalies.size(); + fill_counters(rep); fin.duration_ms = std::chrono::duration_cast( std::chrono::steady_clock::now() - t0).count(); fin.profile_events = collect_profile_events(); @@ -230,8 +265,10 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe } catch (...) { - fin.outcome = Rec::Outcome::Failed; + fin.error_code = getCurrentExceptionCode(); + fin.outcome = isTransientGcRoundError(fin.error_code) ? Rec::Outcome::Aborted : Rec::Outcome::Failed; fin.error = getCurrentExceptionMessage(false); + fill_counters(rep); fin.duration_ms = std::chrono::duration_cast( std::chrono::steady_clock::now() - t0).count(); fin.profile_events = collect_profile_events(); @@ -342,8 +379,21 @@ void CasGcScheduler::loop() catch (...) { /// Idempotent round - the next tick retries; failures must never kill the pacing thread. - /// runRoundLogged already emitted the Aborted Finish row before rethrowing. - i_am_leader.store(false, std::memory_order_relaxed); + /// runRoundLogged already emitted the classified (Aborted/Failed) Finish row before rethrowing. + /// + /// Leadership is dropped only on a NON-transient failure. Dropping it on every failure + /// silenced the advisory heartbeat for a whole interval (`heartbeatLoop` gates its pulses on + /// `i_am_leader`), and when the failure was itself a backend outage the durable lease + /// `(owner, seq)` was frozen too -- together exactly the two-of-two dead-leader signature + /// `acquireOrRenewLease` steals on. A live leader blocked on a flaky store was then deposed, + /// and every handover forces the successor into a full fold: more single-attempt conditional + /// writes against the same flaky backend, a self-reinforcing loop. Keeping the flag keeps the + /// pulses; the lease protocol stays authoritative -- a mounter that really died stops pulsing + /// with or without this flag. A non-transient failure still clears it: a logic-broken leader + /// must stay depositable, and with the flag held its heartbeat would keep beating and no + /// follower could ever steal a lease whose holder cannot complete a round. + if (!isTransientGcRoundError(getCurrentExceptionCode())) + i_am_leader.store(false, std::memory_order_relaxed); tryLogCurrentException(log, "CA GC round failed (will retry next tick)"); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h index 2cce48399b66..a01186507074 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h @@ -27,7 +27,11 @@ struct GcRoundLogRecord /// -- no fold, no pre-CAS deletes, no `gc/state` CAS. Distinct from `Success` so a reader of /// `system.cas_gc_log` (or this scheduler's own log line) can tell a round /// that genuinely folded and found nothing apart from one that never folded at all. - enum class Outcome { Unknown, Success, NotALeader, Failed, Deferred }; + /// `Aborted`: the round threw an exception whose code names a transient condition (backend + /// unavailability, a lost lease, a concurrent leader) -- the next scheduled round retries it and + /// nothing durable is wrong. `Failed` is reserved for everything else (a logic error, corrupted + /// data, an unclassified code): fail-closed, an unrecognised failure reads as real. + enum class Outcome { Unknown, Success, NotALeader, Failed, Deferred, Aborted }; enum class Trigger { Scheduled, Manual }; EventType event_type = EventType::Start; @@ -51,6 +55,9 @@ struct GcRoundLogRecord UInt64 anomalies = 0; /// fold clamps surfaced (never wedging) this round UInt64 duration_ms = 0; String error; + /// `getCurrentExceptionCode()` of the failure on an `Aborted`/`Failed` Finish row; 0 otherwise. + /// The structured twin of `error`: oracles and operators key on this, never on message wording. + Int32 error_code = 0; /// On a `Start`/`Finish` row: the whole round's delta. On a `Phase` row: THAT PHASE's delta. std::map profile_events; @@ -75,6 +82,12 @@ struct GcRoundLogRecord using GcRoundLogger = std::function; +/// True when an exception code names a condition that clears by itself -- the backend was unreachable +/// or slow, or another actor legitimately moved shared state -- so the next scheduled round is the +/// retry. False for everything else, deliberately including any code not on the list: an unrecognised +/// failure must read as a real one. +bool isTransientGcRoundError(int code); + /// Paces regular content-addressed garbage-collection rounds for one pool. The scheduler does not /// implement the GC protocol: `Cas::Gc` owns lease acquisition, work deduplication, and the /// split-brain-safe round operations, so schedulers on different mounters may run independently. diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index 5dbd654343d9..e48ad0e2ff1f 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -8,6 +8,8 @@ #include +#include +#include #include #include @@ -26,6 +28,7 @@ namespace DB::ErrorCodes { extern const int BAD_ARGUMENTS; + extern const int NETWORK_ERROR; } using namespace DB::Cas; @@ -289,7 +292,10 @@ TEST(CASGCLog, AbortedFinishOnThrowingRound) ASSERT_EQ(round_rows.size(), 2u) << "a throwing round still emits a Start and a (Aborted) Finish"; EXPECT_EQ(round_rows[0].event_type, Rec::EventType::Start); EXPECT_EQ(round_rows[1].event_type, Rec::EventType::Finish); - EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Failed); + EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Failed) + << "BAD_ARGUMENTS is not on the transient list, so the row must read as a real failure"; + EXPECT_EQ(round_rows[1].error_code, DB::ErrorCodes::BAD_ARGUMENTS) + << "the Finish row must carry the structured exception code, not only the message text"; EXPECT_FALSE(round_rows[1].error.empty()) << "a failed Finish must carry the exception text"; EXPECT_EQ(round_rows[1].disk_name, "ca"); EXPECT_FALSE(round_rows[1].gc_id.empty()); @@ -299,6 +305,209 @@ TEST(CASGCLog, AbortedFinishOnThrowingRound) << "every row of a FAILED round must still correlate through round_id"; } +/// A round that dies with a TRANSIENT code -- the backend was unreachable, timed out, or another +/// actor moved shared state -- must be classified `Aborted`, not `Failed`: the next scheduled round +/// is the retry and nothing durable is wrong. The classifier keys on the exception CODE +/// (`isTransientGcRoundError`), never on message wording. +class NetworkThrowingBackend : public InMemoryBackend +{ +public: + ListPage list(const String & prefix, const String & cursor, size_t limit) override + { + if (arm) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected backend outage"); + return InMemoryBackend::list(prefix, cursor, limit); + } + std::atomic arm{false}; +}; + +TEST(CASGCLog, TransientThrowIsClassifiedAborted) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + + std::vector rows; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) { rows.push_back(r); }); + + backend->arm.store(true); + EXPECT_THROW(sched.runOneRoundNow(Rec::Trigger::Manual), DB::Exception); + + const std::vector round_rows = roundRowsOnly(rows); + ASSERT_EQ(round_rows.size(), 2u); + EXPECT_EQ(round_rows[1].event_type, Rec::EventType::Finish); + EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Aborted) + << "NETWORK_ERROR names a transient condition; the row must not read as a GC defect"; + EXPECT_EQ(round_rows[1].error_code, DB::ErrorCodes::NETWORK_ERROR); + EXPECT_FALSE(round_rows[1].error.empty()); +} + +/// The classifier itself, pinned direct: the transient list is exact and everything else fails closed. +TEST(CASGCLog, TransientErrorClassifierFailsClosed) +{ + EXPECT_TRUE(DB::Cas::isTransientGcRoundError(DB::ErrorCodes::NETWORK_ERROR)); + EXPECT_FALSE(DB::Cas::isTransientGcRoundError(DB::ErrorCodes::BAD_ARGUMENTS)); + EXPECT_FALSE(DB::Cas::isTransientGcRoundError(0)); + EXPECT_FALSE(DB::Cas::isTransientGcRoundError(-1)); +} + +/// A backend that lets the round's FIRST `gc/state` CAS (the lease acquire/renew) through and throws +/// a transient error on the SECOND (the round-closing commit). The round therefore does all of its +/// pre-CAS work -- including condemning the dropped part -- and dies at `round_commit`. +class StateCommitThrowingBackend : public InMemoryBackend +{ +public: + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override + { + if (arm && key.ends_with("gc/state") && ++state_puts_since_arm >= 2) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected outage on the round-closing CAS"); + return InMemoryBackend::casPut(key, bytes, expected, meta); + } + std::atomic arm{false}; + std::atomic state_puts_since_arm{0}; +}; + +/// The Finish row of a THROWING round must still carry the counters of everything the round did +/// before it died. Before this existed, the exception path emitted a row with `round = 0` and every +/// counter zero, so a round that condemned entries and then lost its commit CAS was +/// indistinguishable from a round that never got past the lease. +TEST(CASGCLog, AbortedFinishCarriesProgressiveCounters) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, + PoolConfig{.pool_prefix = "p", .server_root_id = "test", .gc_fold_max_defer_rounds = 0}); + const RootNamespace ns{"srv1/tbl"}; + + publishPart(store, ns.string(), "all_0_0_0", "hello-progressive-counters"); + store->dropRef(ns, "all_0_0_0"); + store->renewWatermarkOnce(); + + std::vector rows; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) { rows.push_back(r); }); + + backend->arm.store(true); + EXPECT_THROW(sched.runOneRoundNow(Rec::Trigger::Manual), DB::Exception); + + const std::vector round_rows = roundRowsOnly(rows); + ASSERT_EQ(round_rows.size(), 2u); + const Rec & fin = round_rows[1]; + EXPECT_EQ(fin.outcome, Rec::Outcome::Aborted); + EXPECT_EQ(fin.error_code, DB::ErrorCodes::NETWORK_ERROR); + EXPECT_EQ(fin.round, 0u) << "the commit CAS never landed, so the round number must stay unstamped"; + EXPECT_GT(fin.candidates_marked + fin.entries_condemned + fin.entries_graduated + + fin.entries_redeleted + fin.objects_deleted + fin.fence_outs, 0u) + << "the pre-CAS work the round performed must survive into its failure row"; +} + +/// The pacing loop drops leadership only on a NON-transient round failure. A transient failure +/// (backend outage class) keeps `i_am_leader` set, so the advisory heartbeat keeps pulsing and a +/// live leader blocked on a flaky store is not deposed -- dropping the flag on every failure was +/// half of the dead-leader signature (`!incumbent_renewed && !hb_alive`) and produced leadership +/// ping-pong under backend fault windows. A non-transient failure must still clear the flag: a +/// logic-broken leader has to stay depositable. +class ModalThrowingBackend : public InMemoryBackend +{ +public: + enum Mode : int { Off = 0, Transient = 1, Logic = 2 }; + ListPage list(const String & prefix, const String & cursor, size_t limit) override + { + const int m = mode.load(); + if (m == Transient) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected backend outage"); + if (m == Logic) + throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected logic failure"); + return InMemoryBackend::list(prefix, cursor, limit); + } + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override + { + if (key.ends_with("gc/hb")) + ++hb_puts; + return InMemoryBackend::casPut(key, bytes, expected, meta); + } + std::atomic mode{Off}; + std::atomic hb_puts{0}; +}; + +TEST(CASGCScheduler, TransientRoundFailureKeepsLeadershipAndHeartbeat) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + + std::mutex rows_mutex; + std::condition_variable rows_cv; + std::vector finishes; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) + { + if (r.event_type != Rec::EventType::Finish) + return; + std::lock_guard g(rows_mutex); + finishes.push_back(r); + rows_cv.notify_all(); + }); + + const auto wait_for_finish = [&](size_t count) -> Rec + { + std::unique_lock lock(rows_mutex); + const bool ok = rows_cv.wait_for(lock, std::chrono::seconds(30), [&] { return finishes.size() >= count; }); + EXPECT_TRUE(ok) << "timed out waiting for Finish row #" << count; + return finishes.at(count - 1); + }; + /// Bounded poll for an ASYNC flag change. The loop stores `i_am_leader` after `runRoundLogged` + /// returns (after the Finish row was emitted), so the row alone is not a happens-before for the + /// flag -- poll to the expected value instead of asserting a racy instantaneous read. + const auto poll_leader = [&](bool expected) -> bool + { + for (int i = 0; i < 3000; ++i) + { + if (sched.gcHealth().is_leader == expected) + return true; + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + return sched.gcHealth().is_leader == expected; + }; + + sched.start(); + sched.requestRoundSoon(); + const Rec first = wait_for_finish(1); + EXPECT_TRUE(first.outcome == Rec::Outcome::Success || first.outcome == Rec::Outcome::Deferred) + << "outcome=" << static_cast(first.outcome); + EXPECT_TRUE(poll_leader(true)) << "a successful round must establish leadership"; + + backend->mode.store(ModalThrowingBackend::Transient); + sched.requestRoundSoon(); + const Rec aborted = wait_for_finish(2); + EXPECT_EQ(aborted.outcome, Rec::Outcome::Aborted); + /// Leadership kept => the advisory heartbeat keeps pulsing. Waiting for a NEW pulse after the + /// failed round is the happens-after proof that the flag survived; with the flag dropped the + /// heartbeat loop skips every pulse until the next successful round, and this wait times out. + const uint64_t hb_before = backend->hb_puts.load(); + bool pulsed = false; + for (int i = 0; i < 3000 && !pulsed; ++i) + { + pulsed = backend->hb_puts.load() > hb_before; + if (!pulsed) + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + EXPECT_TRUE(pulsed) << "a transient round failure must not silence the advisory heartbeat"; + EXPECT_TRUE(sched.gcHealth().is_leader) << "a transient round failure must not drop leadership"; + + backend->mode.store(ModalThrowingBackend::Logic); + sched.requestRoundSoon(); + const Rec failed = wait_for_finish(3); + EXPECT_EQ(failed.outcome, Rec::Outcome::Failed); + EXPECT_TRUE(poll_leader(false)) << "a non-transient round failure must still surrender leadership"; + + backend->mode.store(ModalThrowingBackend::Off); + sched.stop(); +} + /// Every row of one round -- its Start, each of its Phase rows, and its Finish -- carries the SAME /// non-empty `round_id`, and two rounds carry DIFFERENT ones. That is the property the column exists /// for: `round` is 0 on Start, is only known after the round's single `gc/state` CAS, and is absent on a diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp index fde269c4c999..2438cab2801f 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp @@ -21,7 +21,7 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri auto outcome_enum = std::make_shared(DataTypeEnum8::Values{ {"Unknown", static_cast(UNKNOWN)}, {"Success", static_cast(SUCCESS)}, {"NotALeader", static_cast(NOT_A_LEADER)}, {"Error", static_cast(FAILED)}, - {"Deferred", static_cast(DEFERRED)}}); + {"Deferred", static_cast(DEFERRED)}, {"Aborted", static_cast(ABORTED)}}); auto trigger_enum = std::make_shared(DataTypeEnum8::Values{ {"Scheduled", static_cast(SCHEDULED)}, {"Manual", static_cast(MANUAL)}}); auto lc_string = std::make_shared(std::make_shared()); @@ -38,7 +38,7 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"gc_id", std::make_shared(), "GC scheduler instance id (which mounter)."}, {"trigger", trigger_enum, "Scheduled (background tick) or Manual (SYSTEM command)."}, {"round", std::make_shared(), "GC round number (0 on Start)."}, - {"outcome", outcome_enum, "Unknown (Start) / Success (led, folded, and completed) / NotALeader (another replica holds the GC lease) / Deferred (led but took the skip-unchanged fast path -- no fold ran) / Error (the round threw)."}, + {"outcome", outcome_enum, "Unknown (Start) / Success (led, folded, and completed) / NotALeader (another replica holds the GC lease) / Deferred (led but took the skip-unchanged fast path -- no fold ran) / Aborted (the round threw a transient error -- backend unavailability, a lost lease, a concurrent leader -- and the next scheduled round retries) / Error (the round threw a non-transient error)."}, {"candidates_marked", std::make_shared(), "Objects retired (marked) this round."}, {"objects_deleted", std::make_shared(), "Objects physically deleted this round."}, {"objects_absent", std::make_shared(), "Retire candidates found already absent."}, @@ -51,7 +51,8 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"fence_outs", std::make_shared(), "Expired mounts fenced out by this round's heartbeat floor."}, {"anomalies", std::make_shared(), "Fold clamps surfaced (and survived) this round; steady >0 warrants a look at the round log details."}, {"duration_ms", std::make_shared(), "Round wall-clock duration (Finish)."}, - {"error", std::make_shared(), "Exception text when outcome = Error."}, + {"error", std::make_shared(), "Exception text when outcome = Aborted or Error."}, + {"error_code", std::make_shared(), "Exception code when outcome = Aborted or Error; 0 otherwise. The structured twin of `error`: key monitoring on this column, not on message text."}, {"ProfileEvents", std::make_shared(lc_string, std::make_shared()), "On a Start/Finish row: the per-round ProfileEvents delta (the Cas* counters and S3 events for this round). On a Phase row: THAT PHASE's delta, so `GROUP BY phase` over `ProfileEvents['S3ListObjects']` attributes the round's LIST budget to the phase that spent it. Empty on the `meta_pool_wait` row by construction — that phase's work runs on other threads (read its `phase_metrics` instead)."}, {"round_id", std::make_shared(), @@ -92,6 +93,7 @@ void ContentAddressedGarbageCollectionLogElement::appendToBlock(MutableColumns & columns[i++]->insert(anomalies); columns[i++]->insert(duration_ms); columns[i++]->insert(error); + columns[i++]->insert(error_code); { Map map; map.reserve(profile_events.size()); diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.h b/src/Interpreters/ContentAddressedGarbageCollectionLog.h index 9cbdbd3525f6..b7ffdc734c75 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.h +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.h @@ -15,7 +15,10 @@ struct ContentAddressedGarbageCollectionLogElement /// `DEFERRED`: the round acquired the GC lease and took the skip-unchanged fast path -- no fold, no /// pre-CAS deletes, no `gc/state` CAS. Kept distinct from `SUCCESS` so a query against this table can /// tell a round that genuinely folded and found nothing apart from one that never folded at all. - enum Outcome : int8_t { UNKNOWN = 1, SUCCESS = 2, NOT_A_LEADER = 3, FAILED = 4, DEFERRED = 5 }; + /// `ABORTED`: the round threw an exception whose code names a transient condition (backend + /// unavailability, a lost lease, a concurrent leader); the next scheduled round retries it. + /// `FAILED` is everything else -- fail-closed, an unclassified error reads as real. + enum Outcome : int8_t { UNKNOWN = 1, SUCCESS = 2, NOT_A_LEADER = 3, FAILED = 4, DEFERRED = 5, ABORTED = 6 }; enum Trigger : int8_t { SCHEDULED = 1, MANUAL = 2 }; time_t event_time = 0; @@ -42,6 +45,7 @@ struct ContentAddressedGarbageCollectionLogElement UInt64 anomalies = 0; /// fold clamps surfaced this round UInt64 duration_ms = 0; String error; + Int32 error_code = 0; /// exception code on an Aborted/Error FINISH; 0 otherwise std::map profile_events; /// per-round delta (FINISH); per-phase delta (PHASE) String round_id; /// correlator for every row of one round attempt From bf77615fe0a2ae1cb1bc89ce25cdb82dab9c9f85 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 26 Aug 2026 16:30:23 +0200 Subject: [PATCH 02/81] Classify a relink-confirm refusal as NO_REPLICA_HAS_PART, not NETWORK_ERROR A fetch-by-relink that loses the offer-to-confirm race -- the source's ref moved (a merge, a mutation, an outdated-part drop) between the offer and the confirm -- is a designed, fail-closed outcome: the receiver abandons the relink and the replication queue retries, re-selecting the source and the covering part. It was thrown as `NETWORK_ERROR`, which misdescribes it three ways: - both queue executors (`processQueueEntry`, `ReplicatedMergeTreeQueue`-driven `ReplicatedMergeMutateTaskBase`) treat `NETWORK_ERROR` as an unclassified failure, so every refusal printed an Error-level log line with a full stack trace; issue #2219 records a multi-hour false triage chasing a network fault that was never there (up to 53% of relink proofs refuse under small-part load); - stateless `part_log` hygiene checks tolerate the fetch-transient class under the code upstream fetches use for it, `NO_REPLICA_HAS_PART` (e.g. `02265_column_ttl` whitelists exactly that code), so a refusal landing in `part_log` as `NETWORK_ERROR` fails them -- this is what broke `02265_column_ttl` in the CAS lanes on PR #2159 (13/14 reruns under `prefer_fetch_merged_part_size_threshold=1`); - the label suggests retrying the transport, while the one recovery that is unsound here is a byte re-request to the same source. Both relink retry-later throw sites (taxonomy row 3, the confirm refusal, and row 5b, the unresolved promote) now throw `NO_REPLICA_HAS_PART`. The queue behavior is unchanged -- the exception is stored on the entry, backed off, and re-executed -- but both executors demote it to INFO with no stack trace. Unlike `ABORTED` (the other demoted code), it keeps `need_to_save_exception`, so a refusal storm stays visible in `system.replication_queue`; `ABORTED`'s save-nothing shape is the known pathology where a refusal loop runs invisibly with no backoff accounting. `test_confirm_refuses_when_source_dropped_in_window` now pins the classification: the refusal must not appear at Error level, must appear at Information level, and must reach `part_log` only as `NO_REPLICA_HAS_PART`. No message text changed; no generic queue code changed. Closes: https://github.com/Altinity/ClickHouse/issues/2219 Related: https://github.com/Altinity/ClickHouse/pull/2159 Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- src/Storages/MergeTree/DataPartsExchange.cpp | 25 +++++++++++----- .../test_cas_replicated_relink/test.py | 30 ++++++++++++++++++- 2 files changed, 46 insertions(+), 9 deletions(-) diff --git a/src/Storages/MergeTree/DataPartsExchange.cpp b/src/Storages/MergeTree/DataPartsExchange.cpp index 00f27191db7b..7a8ae2e54c42 100644 --- a/src/Storages/MergeTree/DataPartsExchange.cpp +++ b/src/Storages/MergeTree/DataPartsExchange.cpp @@ -72,7 +72,7 @@ namespace ErrorCodes extern const int CHECKSUM_DOESNT_MATCH; extern const int INSECURE_PATH; extern const int LOGICAL_ERROR; - extern const int NETWORK_ERROR; + extern const int NO_REPLICA_HAS_PART; extern const int S3_ERROR; extern const int ZERO_COPY_REPLICATION_ERROR; } @@ -1314,8 +1314,14 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// 3. THE CONFIRM DID NOT PROVE THE SOURCE: an `unproven` answer, an absent answer cookie, a transport /// failure, a timeout. All one outcome, deliberately (`CasConfirmAnswer`: only `yes` authorizes). /// `+1`: durable, then released by `abort`. Action: THROW a locally generated retry-later -/// `NETWORK_ERROR` naming the source and the part -- never `nullptr`, because a byte re-request goes -/// back to the very source whose state is in doubt. +/// `NO_REPLICA_HAS_PART` naming the source and the part -- never `nullptr`, because a byte +/// re-request goes back to the very source whose state is in doubt. That code, deliberately: +/// both queue executors (`processQueueEntry`, `ReplicatedMergeMutateTaskBase::executeStep`) +/// demote it to INFO with no stack trace -- a refusal is the designed outcome of racing a source +/// whose ref moved on, not a network fault (issue #2219 records a multi-hour false triage chasing +/// that label) -- yet, unlike `ABORTED`, it still records the exception on the queue entry, so a +/// refusal storm stays visible in `system.replication_queue`. It is also the one fetch-transient +/// code the stateless corpus already tolerates in `part_log` checks (e.g. `02265_column_ttl`). /// Lose a part? No -- the queue stores the exception, backs off, and re-executes the entry, which /// recomputes the source and the covering-part discovery. The fetch is postponed, not dropped. /// Double-promote? No -- `abort` appends the exact precommit removal and no committed ref exists. @@ -1339,7 +1345,8 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// `+1`: still owed -- the handle attempts its abandon, which is REJECTED by the state machine if /// the promote in fact landed (a promoted binding is no longer a precommit), so no committed ref is /// ever undone here. -/// Action: THROW the retry-later `NETWORK_ERROR`, as row 3 -- returning `nullptr` is the one thing +/// Action: THROW the retry-later `NO_REPLICA_HAS_PART`, as row 3 -- returning `nullptr` is the +/// one thing /// that must not happen, because a byte fetch would publish the part a SECOND time over a relink /// that may already be committed. /// Lose a part? No -- retry-later, as row 3. Double-promote? No -- nothing is published on this exit. @@ -1544,9 +1551,11 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( { /// Taxonomy row 3. Locally generated on purpose — nothing here is the source's error to report — /// and thrown rather than returned, because the one recovery that is NOT sound after this is a - /// byte re-request to the same source. `NETWORK_ERROR` puts it in the retry-later class, so the - /// queue stores it, backs off, and re-selects on re-execution. - throw Exception(ErrorCodes::NETWORK_ERROR, + /// byte re-request to the same source. `NO_REPLICA_HAS_PART` puts it in the retry-later class + /// (the queue stores it, backs off, and re-selects on re-execution) and both queue executors + /// demote it to INFO without a stack trace -- see the taxonomy, row 3, for why this refusal is + /// an ordinary outcome rather than a fault. + throw Exception(ErrorCodes::NO_REPLICA_HAS_PART, "Source {} did not prove it still holds the manifest it offered for part {} by relink; " "the relink is abandoned and the fetch will be retried later", fetch_uri.getHost(), part_name); @@ -1565,7 +1574,7 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( /// second time over a relink that may already be committed. Thrown in the retry-later class /// instead, exactly as an unproven confirm is (row 3) -- the queue stores it, backs off, and /// re-executes, by which time the ref lane has resolved the ambiguity one way or the other. - throw Exception(ErrorCodes::NETWORK_ERROR, + throw Exception(ErrorCodes::NO_REPLICA_HAS_PART, "Relink of part {} from {} could not be resolved: the promotion may or may not have " "committed, so the bytes must NOT be fetched; the fetch will be retried later", part_name, fetch_uri.getHost()); diff --git a/tests/integration/test_cas_replicated_relink/test.py b/tests/integration/test_cas_replicated_relink/test.py index 83d8d239ba92..234cbbfbe7a4 100644 --- a/tests/integration/test_cas_replicated_relink/test.py +++ b/tests/integration/test_cas_replicated_relink/test.py @@ -769,7 +769,10 @@ def test_confirm_refuses_when_source_dropped_in_window(): """Task 16 step 1 — the race the confirm exists to lose safely. Taxonomy row 3: the source cannot prove it still holds the offered manifest, so the receiver aborts - its durable `+1` and throws a retry-later `NETWORK_ERROR` INSTEAD of falling back to bytes. The two + its durable `+1` and throws a retry-later `NO_REPLICA_HAS_PART` INSTEAD of falling back to bytes. + That code is part of the contract (issue #2219): both queue executors demote it to INFO with no + stack trace, it stays recorded on the queue entry, and it is the one fetch-transient code the + stateless corpus already tolerates in `part_log` checks. The two assertions that matter are (a) the queue recovers by re-selecting — here, onto the covering part — and (b) NO byte re-request ever went to the source whose state was in doubt. (b) is the entire reason row 3 throws where rows 2 and 5 return `nullptr`. @@ -810,6 +813,31 @@ def test_confirm_refuses_when_source_dropped_in_window(): assert not log_lines(node2, relink_finished_pattern(table, part)) assert any_state_part_count(node2, table, part) == 0 + # (c) the refusal's CLASSIFICATION -- the contract pinned after issue #2219. The refusal must reach + # the operator as the tolerated fetch-transient `NO_REPLICA_HAS_PART` (both queue executors + # demote it to INFO, no stack trace; every stateless `part_log` hygiene check that whitelists + # that code -- e.g. `02265_column_ttl` -- stays green), never as an Error-level `NETWORK_ERROR` + # with a stack trace, which reads as a network fault and once cost a multi-hour false triage. + refusal_error_pattern = r".*did not prove it still holds the manifest" + refusal_info_pattern = r".*did not prove it still holds the manifest" + assert not log_lines(node2, refusal_error_pattern), ( + "the relink refusal is a designed outcome and must not be logged at Error level" + ) + assert log_lines(node2, refusal_info_pattern), ( + "the demoted refusal must still be visible at Information level -- silence would be worse than " + "the old noise" + ) + node2.query("SYSTEM FLUSH LOGS part_log") + stray_codes = node2.query( + "SELECT DISTINCT errorCodeToName(error) FROM system.part_log " + "WHERE table = '{}' AND error != 0 AND errorCodeToName(error) != 'NO_REPLICA_HAS_PART'".format( + table + ) + ).split() + assert stray_codes == [], ( + "a relink refusal must reach part_log only as NO_REPLICA_HAS_PART, got: {}".format(stray_codes) + ) + drop_everywhere(table) From b9140d458ec7380d95ad8d321f7b1f927bc358ab Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:02:32 +0200 Subject: [PATCH 03/81] =?UTF-8?q?cas:=20wire-keys=20phase=201=20=E2=80=94?= =?UTF-8?q?=20move=20every=20codec=20onto=20WireKey=20carriers=20(no=20key?= =?UTF-8?q?s=20renamed=20yet)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Behavior-preserving groundwork for an atomic rename of the CAS wire-format JSON keys from abstract letters (`t`, `k`, `s`, ...) to semantic names (`kind`, `outcome`, `state`, ...), landed as its own phase so the rename itself is a single reviewable diff. Adds `WireKey` and per-encoding field write helpers, and `EnumWireTable` — a table pairing each enum value with its wire word, proven complete against the enum by a set-equality coverage check with a failing witness for every member. `kMinBlobHeaderLen` gets one compile-time owner instead of several hand-kept constants. `TokenType`, `ObjectKind`, and `BlobHashAlgo` move onto `EnumWireTable`, and the blob-meta, pool-meta, GC state/heartbeat/ maintenance, server-root, blob-envelope, ref-log/ref-ckpt/ref-snapshot/ ref-catalog, run, fold-seal, and gc-outcomes codecs are all migrated onto the carriers — every one of them still writing its existing wire spelling. `RunMarker` becomes a typed enum, and the format test battery is closed out with a set-equality check over the codec registry. No wire-format bytes change in this phase; the follow-up phase (next commit) performs the actual key cut. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../Formats/CasBlobEnvelopeFormat.cpp | 92 ++++----- .../Formats/CasBlobEnvelopeFormat.h | 3 + .../Formats/CasBlobMetaFormat.cpp | 47 +++-- .../Formats/CasBlobMetaFormat.h | 2 + .../Formats/CasEnvelopeLimits.h | 14 ++ .../Formats/CasFoldSealFormat.cpp | 176 ++++++++++-------- .../Formats/CasFoldSealFormat.h | 4 +- .../ContentAddressed/Formats/CasFormat.cpp | 14 ++ .../ContentAddressed/Formats/CasFormat.h | 1 + .../Formats/CasGcMaintenanceStateFormat.cpp | 10 +- .../Formats/CasGcOutcomesFormat.cpp | 70 +++---- .../Formats/CasGcOutcomesFormat.h | 3 + .../Formats/CasGcStateFormat.cpp | 72 ++++--- .../Formats/CasPartManifestFormat.cpp | 104 +++++------ .../Formats/CasPartManifestFormat.h | 3 + .../Formats/CasPoolMetaFormat.cpp | 51 ++--- .../Formats/CasRecordStreamFormat.cpp | 89 +++++---- .../Formats/CasRecordStreamFormat.h | 41 +++- .../Formats/CasRefCatalogFormat.cpp | 85 +++++---- .../Formats/CasRefCkptFormat.cpp | 38 ++-- .../Formats/CasRefLogFormat.cpp | 176 +++++++++--------- .../Formats/CasRefLogFormat.h | 4 + .../Formats/CasRefSnapshotFormat.cpp | 89 ++++----- .../Formats/CasRefWireVocab.cpp | 42 +++-- .../Formats/CasRefWireVocab.h | 6 +- .../Formats/CasServerRootFormats.cpp | 92 +++++---- .../Formats/CasTextFormat.cpp | 3 +- .../ContentAddressed/Formats/CasTextFormat.h | 71 +++++-- .../ContentAddressed/Formats/CasWireVocab.cpp | 78 ++++---- .../ContentAddressed/Formats/CasWireVocab.h | 135 +++++++++++++- .../ContentAddressed/Gc/CasBlobInDegree.cpp | 68 +++---- .../ContentAddressed/Gc/CasBlobInDegree.h | 10 +- .../ContentAddressed/Gc/CasGc.cpp | 32 ++-- .../Primitives/CasBlobDigest.cpp | 14 +- .../Primitives/CasBlobDigest.h | 11 +- .../Primitives/CasEnumWireTable.h | 73 ++++++++ .../Primitives/CasEnumWireTableAsserts.h | 37 ++++ .../ContentAddressed/Tools/CasFsck.cpp | 8 +- .../ContentAddressed/Tools/CasInspect.cpp | 122 ++---------- src/Disks/tests/cas_format_test_battery.h | 18 ++ src/Disks/tests/cas_test_helpers.h | 14 +- .../tests/gtest_cas_blob_envelope_format.cpp | 2 + src/Disks/tests/gtest_cas_blob_indegree.cpp | 76 ++++++-- .../tests/gtest_cas_blob_meta_format.cpp | 29 ++- src/Disks/tests/gtest_cas_encoding_pins.cpp | 21 ++- src/Disks/tests/gtest_cas_enum_wire_table.cpp | 131 +++++++++++++ src/Disks/tests/gtest_cas_event_log.cpp | 2 +- .../tests/gtest_cas_fold_seal_format.cpp | 22 ++- src/Disks/tests/gtest_cas_format_battery.cpp | 25 +++ src/Disks/tests/gtest_cas_gc_attempt.cpp | 2 +- src/Disks/tests/gtest_cas_gc_fold.cpp | 4 +- src/Disks/tests/gtest_cas_gc_leak.cpp | 2 +- .../gtest_cas_gc_maintenance_state_format.cpp | 12 ++ .../tests/gtest_cas_gc_outcomes_format.cpp | 38 ++++ src/Disks/tests/gtest_cas_gc_rebuild.cpp | 2 +- src/Disks/tests/gtest_cas_gc_resume.cpp | 2 +- src/Disks/tests/gtest_cas_gc_round.cpp | 16 +- src/Disks/tests/gtest_cas_gc_shard_plan.cpp | 2 +- src/Disks/tests/gtest_cas_gc_state_format.cpp | 4 + src/Disks/tests/gtest_cas_inspect.cpp | 98 +++++++++- src/Disks/tests/gtest_cas_json_writer.cpp | 30 ++- src/Disks/tests/gtest_cas_observability.cpp | 50 ++++- .../tests/gtest_cas_orphan_nomination.cpp | 4 +- .../tests/gtest_cas_part_manifest_format.cpp | 2 + src/Disks/tests/gtest_cas_pluggable_hash.cpp | 4 +- .../gtest_cas_rebuild_condemn_nothing.cpp | 4 +- .../tests/gtest_cas_record_stream_format.cpp | 44 ++++- src/Disks/tests/gtest_cas_ref_catalog.cpp | 7 +- src/Disks/tests/gtest_cas_ref_ckpt.cpp | 16 ++ .../tests/gtest_cas_ref_epoch_seal_format.cpp | 2 + src/Disks/tests/gtest_cas_ref_log_format.cpp | 49 +++++ .../tests/gtest_cas_ref_snapshot_format.cpp | 2 + .../tests/gtest_cas_server_root_format.cpp | 8 + src/Disks/tests/gtest_cas_text_format.cpp | 10 + .../tests/gtest_cas_truncate_reclaim.cpp | 2 +- src/Disks/tests/gtest_cas_wire_vocab.cpp | 143 +++++++++++++- 76 files changed, 1997 insertions(+), 892 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h create mode 100644 src/Disks/tests/gtest_cas_enum_wire_table.cpp diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index 65176572e896..a4933ff2ccb8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include @@ -20,30 +21,29 @@ namespace constexpr std::string_view kBlobType = "cas_blob"; -std::string_view opToWord(ProvenanceOp op) +namespace EnvelopeWire { - switch (op) - { - case ProvenanceOp::Other: return "other"; - case ProvenanceOp::Insert: return "insert"; - case ProvenanceOp::Merge: return "merge"; - case ProvenanceOp::Mutation: return "mutation"; - case ProvenanceOp::Attach: return "attach"; - case ProvenanceOp::Repack: return "repack"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown ProvenanceOp {}", static_cast(op)); + constexpr WireKey type{"type"}; + constexpr WireKey version{"v"}; + constexpr WireKey tag{"tag"}; + constexpr WireKey build{"bld"}; + constexpr WireKey time_ms{"ts"}; + constexpr WireKey creator{"by"}; + constexpr WireKey op{"op"}; + constexpr WireKey chver{"ch"}; + constexpr WireKey ref{"ref"}; } -ProvenanceOp opFromWord(std::string_view w) -{ - if (w == "other") return ProvenanceOp::Other; - if (w == "insert") return ProvenanceOp::Insert; - if (w == "merge") return ProvenanceOp::Merge; - if (w == "mutation") return ProvenanceOp::Mutation; - if (w == "attach") return ProvenanceOp::Attach; - if (w == "repack") return ProvenanceOp::Repack; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: unknown op '{}'", w); -} +constexpr EnumWireTable kProvenanceOpWords{{{ + {ProvenanceOp::Other, "other"}, + {ProvenanceOp::Insert, "insert"}, + {ProvenanceOp::Merge, "merge"}, + {ProvenanceOp::Mutation, "mutation"}, + {ProvenanceOp::Attach, "attach"}, + {ProvenanceOp::Repack, "repack"}, +}}}; + +static_assert(casEnumTableCoversEnum()); /// The escaped byte-length of one raw ref char under the frozen envelope alphabet (see writeEnvelopeRefField). size_t escapedLen(char c) @@ -96,6 +96,11 @@ void writeEnvelopeRefField(String & json, size_t budget, std::string_view raw_re } +std::string_view provenanceOpToWireWord(ProvenanceOp op) +{ + return kProvenanceOpWords.toWord(op, "CAS blob envelope"); +} + String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { if (header.kind != ObjectKind::Blob) @@ -108,16 +113,16 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { CasJsonWriter buf(256); bool first = true; - writeKey(buf, "type", first); writeStringValue(buf, kBlobType); - writeKey(buf, "v", first); writeIntText(currentCompatibilityVersion(), buf); - writeKey(buf, "tag", first); writeHex128Value(buf, header.incarnation_tag); - writeKey(buf, "bld", first); writeHex128Value(buf, header.build_id); + writeKey(buf, EnvelopeWire::type, first); writeStringValue(buf, kBlobType); + writeKey(buf, EnvelopeWire::version, first); writeIntText(currentCompatibilityVersion(), buf); + writeKey(buf, EnvelopeWire::tag, first); writeHex128Value(buf, header.incarnation_tag); + writeKey(buf, EnvelopeWire::build, first); writeHex128Value(buf, header.build_id); if (header.provenance) { - writeKey(buf, "ts", first); writeIntText(header.provenance->created_at_ms, buf); - writeKey(buf, "by", first); writeHex128Value(buf, header.provenance->creator_server_id); - writeKey(buf, "op", first); writeStringValue(buf, opToWord(header.provenance->op)); - writeKey(buf, "ch", first); writeIntText(header.provenance->ch_version, buf); + writeKey(buf, EnvelopeWire::time_ms, first); writeIntText(header.provenance->created_at_ms, buf); + writeKey(buf, EnvelopeWire::creator, first); writeHex128Value(buf, header.provenance->creator_server_id); + writeKey(buf, EnvelopeWire::op, first); writeStringValue(buf, provenanceOpToWireWord(header.provenance->op)); + writeKey(buf, EnvelopeWire::chver, first); writeIntText(header.provenance->ch_version, buf); } /// Test-only critical extension: an unknown `!`-key BEFORE `ref`. if (header.emit_unknown_critical_key) @@ -132,15 +137,18 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) /// (byte blob_header_len-1 is reserved for '\n'; the pad zone fills the gap with spaces). if (header.intended_ref) { - static constexpr std::string_view ref_key = ",\"ref\":"; + /// 4 = the `,"` before and `":` after the key text — the `,"ref":` framing minus the key itself. + constexpr size_t ref_key_size = 4 + EnvelopeWire::ref.text.size(); /// +3 = opening quote + closing quote + closing brace. - const size_t fixed = json.size() + ref_key.size() + 3; + const size_t fixed = json.size() + ref_key_size + 3; if (blob_header_len < 1 || fixed > static_cast(blob_header_len) - 1) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS blob envelope: non-ref fields ({} bytes) do not fit blob_header_len {} before the ref", fixed, blob_header_len); const size_t budget = (static_cast(blob_header_len) - 1) - fixed; - json += ref_key; + json += ",\""; + json += EnvelopeWire::ref.text; + json += "\":"; writeEnvelopeRefField(json, budget, *header.intended_ref); } json += '}'; @@ -173,7 +181,7 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje String key; while (r.nextKey(key)) { - if (key == "type") + if (key == EnvelopeWire::type) { const String t = r.readString(); if (t != kBlobType) @@ -181,37 +189,37 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje "CAS blob envelope: object is a '{}', not a '{}'", t, kBlobType); saw_type = true; } - else if (key == "v") + else if (key == EnvelopeWire::version) { h.compatibility_version = r.readU32Number(); checkCompatibility(h.compatibility_version, "blob envelope"); saw_v = true; } - else if (key == "tag") + else if (key == EnvelopeWire::tag) h.incarnation_tag = r.readHex128(); - else if (key == "bld") + else if (key == EnvelopeWire::build) h.build_id = r.readHex128(); - else if (key == "ts") + else if (key == EnvelopeWire::time_ms) { prov.created_at_ms = r.readU64Number(); have_prov = true; } - else if (key == "by") + else if (key == EnvelopeWire::creator) { prov.creator_server_id = r.readHex128(); have_prov = true; } - else if (key == "op") + else if (key == EnvelopeWire::op) { - prov.op = opFromWord(r.readString()); + prov.op = kProvenanceOpWords.fromWord(r.readString(), "CAS blob envelope"); have_prov = true; } - else if (key == "ch") + else if (key == EnvelopeWire::chver) { prov.ch_version = static_cast(r.readU64Number()); have_prov = true; } - else if (key == "ref") + else if (key == EnvelopeWire::ref) h.intended_ref = r.readString(); else r.skipUnknown(key); /// `!`-key -> UNKNOWN_FORMAT_VERSION; unknown plain key -> skipped (tolerant) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 19250fe69ddd..68c473a70c3d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -32,6 +32,9 @@ enum class ProvenanceOp : uint8_t Repack = 5, }; +/// Returns the persisted wire word for a validated provenance operation. +std::string_view provenanceOpToWireWord(ProvenanceOp op); + /// Optional diagnostic metadata recorded with an envelope. The fields identify when and where the /// incarnation was created, the ClickHouse build that wrote it, and the operation that produced it; /// none of them participates in object identity or a protocol decision. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp index b62fd3b82424..ab98b8858693 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include @@ -17,26 +18,25 @@ namespace DB::Cas namespace { -std::string_view metaStateToWord(MetaState s) +namespace BlobMetaWire { - switch (s) - { - case MetaState::Clean: return "clean"; - case MetaState::Condemned: return "condemned"; - } - // The enum is persisted as a closed vocabulary. Do not silently invent a spelling for a value - // added without a corresponding format decision: that would make the writer emit data older - // readers cannot classify. - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: unknown MetaState {}", static_cast(s)); + constexpr WireKey state{"st"}; + constexpr WireKey condemn_round{"cr"}; + constexpr WireKey size{"sz"}; } -MetaState metaStateFromWord(std::string_view w) -{ - if (w == "clean") return MetaState::Clean; - if (w == "condemned") return MetaState::Condemned; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: unknown state '{}'", w); +constexpr EnumWireTable kMetaStateWords{{{ + {MetaState::Clean, "clean"}, + {MetaState::Condemned, "condemned"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + } +std::string_view metaStateToWireWord(MetaState state) +{ + return kMetaStateWords.toWord(state, "CAS blob meta"); } String encodeBlobMeta(const BlobMeta & meta) @@ -46,12 +46,9 @@ String encodeBlobMeta(const BlobMeta & meta) // `version` is represented by the header line. The JSON body contains only fields that describe // the current marker and its accounting data. bool first = true; - writeKey(out, "st", first); - writeStringValue(out, metaStateToWord(meta.state)); - writeKey(out, "cr", first); - writeU64StringValue(out, meta.condemn_round); - writeKey(out, "sz", first); - writeU64StringValue(out, meta.size); + writeWordField(out, BlobMetaWire::state, metaStateToWireWord(meta.state), first); + writeU64StringField(out, BlobMetaWire::condemn_round, meta.condemn_round, first); + writeU64StringField(out, BlobMetaWire::size, meta.size, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -72,14 +69,14 @@ BlobMeta decodeBlobMeta(std::string_view bytes) String key; while (r.nextKey(key)) { - if (key == "st") + if (key == BlobMetaWire::state) { - m.state = metaStateFromWord(r.readString()); + m.state = kMetaStateWords.fromWord(r.readString(), "CAS blob meta"); saw_state = true; } - else if (key == "cr") + else if (key == BlobMetaWire::condemn_round) m.condemn_round = r.readU64String(); - else if (key == "sz") + else if (key == BlobMetaWire::size) m.size = r.readU64String(); else r.skipUnknown(key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h index 6694fbafeb33..5290e7c831db 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h @@ -19,6 +19,8 @@ enum class MetaState : uint8_t /// so a writer may republish it by replacing the body and updating this marker. }; +std::string_view metaStateToWireWord(MetaState state); + /// The durable per-hash meta record. Its text representation consists of a format header followed by /// one JSON object with the state word, the GC condemnation round, and the raw body size. `size` is /// retained for introspection, fsck, and GC accounting; reads of the blob never consult the meta. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h new file mode 100644 index 000000000000..df8cee44bc47 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h @@ -0,0 +1,14 @@ +#pragma once + +#include + +namespace DB::Cas +{ + +/// The pool-wide floor for `blob_header_len`. One compile-time owner, read by BOTH +/// `validatePoolBlobHeaderLen` (pool creation / decode) and the blob-envelope codec, so the +/// mandatory-descriptor worst-case proof and the enforced floor can never guard different numbers. +/// The derivation of the floor lives in `CasPoolMetaFormat.cpp` next to the worst-case table. +inline constexpr uint64_t kMinBlobHeaderLen = 240; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index b4bba2bff08f..2324cc2c037a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -20,32 +21,52 @@ namespace ErrorCodes namespace DB::Cas { -std::string_view holdReasonToWord(HoldReason r) +namespace { - switch (r) - { - case HoldReason::GapBelowWitness: return "gap_below_witness"; - case HoldReason::UnconsumedSealCrossing: return "unconsumed_seal_crossing"; - case HoldReason::WitnessDisappeared: return "witness_disappeared"; - case HoldReason::BodyUndecodable: return "body_undecodable"; - case HoldReason::ManifestBodyMissing: return "manifest_body_missing"; - case HoldReason::CheckpointUndecodable: return "checkpoint_undecodable"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown hold reason {}", static_cast(r)); -} -namespace +namespace FoldSealWire { + constexpr WireKey generation{"g"}; + constexpr WireKey parent_generation{"pg"}; + constexpr WireKey kind{"k"}; + constexpr WireKey run_key{"key"}; + constexpr WireKey checksum{"ck"}; + constexpr WireKey shard{"shard"}; + constexpr WireKey key_generation{"gen"}; + constexpr WireKey life{"life"}; + constexpr WireKey classification{"cls"}; + constexpr WireKey fold_epoch{"lfe"}; + constexpr WireKey fold_seq{"lfs"}; + constexpr WireKey hold_reason{"hr"}; + constexpr WireKey hold_epoch{"hpe"}; + constexpr WireKey hold_seq{"hps"}; + constexpr WireKey retries{"hrc"}; + constexpr WireKey retry_round{"hnr"}; + constexpr WireKey remove_epoch{"rte"}; + constexpr WireKey remove_seq{"rts"}; + constexpr WireKey condemned_total{"ct"}; + constexpr WireKey pending_total{"pt"}; + constexpr WireKey oldest_round{"ocr"}; +} + +constexpr std::string_view kRefLifeTag = "rfl"; +constexpr std::string_view kBlobRunTag = "btr"; +constexpr std::string_view kCondemnedTag = "cnd"; + +constexpr EnumWireTable kHoldReasonWords{{{ + {HoldReason::GapBelowWitness, "gap_below_witness"}, + {HoldReason::UnconsumedSealCrossing, "unconsumed_seal_crossing"}, + {HoldReason::WitnessDisappeared, "witness_disappeared"}, + {HoldReason::BodyUndecodable, "body_undecodable"}, + {HoldReason::ManifestBodyMissing, "manifest_body_missing"}, + {HoldReason::CheckpointUndecodable, "checkpoint_undecodable"}, +}}}; + +static_assert(casEnumTableCoversEnum()); HoldReason holdReasonFromWord(std::string_view w) { - if (w == "gap_below_witness") return HoldReason::GapBelowWitness; - if (w == "unconsumed_seal_crossing") return HoldReason::UnconsumedSealCrossing; - if (w == "witness_disappeared") return HoldReason::WitnessDisappeared; - if (w == "body_undecodable") return HoldReason::BodyUndecodable; - if (w == "manifest_body_missing") return HoldReason::ManifestBodyMissing; - if (w == "checkpoint_undecodable") return HoldReason::CheckpointUndecodable; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown hold reason '{}'", w); + return kHoldReasonWords.fromWord(w, "CAS fold seal hold reason"); } /// The classification set is CLOSED. Every consumer of a coverage row branches on exact values — the @@ -88,11 +109,11 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { bool first = true; - writeKey(out, "k", first); writeStringValue(out, kind); - writeKey(out, "key", first); writeStringValue(out, r.key); - writeKey(out, "ck", first); writeHex128Value(out, r.checksum); - writeKey(out, "shard", first); writeIntText(r.shard, out); - writeKey(out, "gen", first); writeU64StringValue(out, r.generation); + writeStringField(out, FoldSealWire::kind, kind, first); + writeStringField(out, FoldSealWire::run_key, r.key, first); + writeHex128Field(out, FoldSealWire::checksum, r.checksum, first); + writeNumberField(out, FoldSealWire::shard, r.shard, first); + writeU64StringField(out, FoldSealWire::key_generation, r.key_generation, first); closeObject(out, first); } @@ -106,7 +127,7 @@ void validateFoldSealStructure( std::vector run_seen(gc_shards, false); for (const RunRef & run : seal.blob_target_runs) { - if (run.key.empty() || run.generation == 0) + if (run.key.empty() || run.key_generation == 0) throw Exception(error_code, "CAS fold seal {}: blob-target run requires a nonempty key and nonzero physical generation", source); @@ -121,10 +142,10 @@ void validateFoldSealStructure( run_seen[run.shard] = true; const auto parsed = layout.parseBlobTargetRunKey(run.key); - if (!parsed || parsed->generation != run.generation || parsed->shard != run.shard || parsed->seq != 0) + if (!parsed || parsed->generation != run.key_generation || parsed->shard != run.shard || parsed->seq != 0) throw Exception(error_code, "CAS fold seal {}: blob-target run key '{}' is not canonical for generation {}, shard {}, sequence 0", - source, run.key, run.generation, run.shard); + source, run.key, run.key_generation, run.shard); } if (seal.condemned_summary.size() != gc_shards) @@ -154,6 +175,11 @@ void validateFoldSealStructure( } +std::string_view holdReasonToWord(HoldReason r) +{ + return kHoldReasonWords.toWord(r, "CAS fold seal hold reason"); +} + FoldSealCaps foldSealCaps() { const FormatTraits & t = traitsFor(FormatId::FoldSeal); @@ -205,8 +231,8 @@ String encodeFoldSeal(const CasFoldSeal & seal) /// meta line { bool first = true; - writeKey(out, "g", first); writeU64StringValue(out, seal.generation); - writeKey(out, "pg", first); writeU64StringValue(out, seal.parent_generation); + writeU64StringField(out, FoldSealWire::generation, seal.generation, first); + writeU64StringField(out, FoldSealWire::parent_generation, seal.parent_generation, first); closeObject(out, first); closeLine("meta"); } @@ -263,25 +289,23 @@ String encodeFoldSeal(const CasFoldSeal & seal) life_state.cleanup_evidence->remove_txn_id.ref_sequence); bool first = true; - writeKey(out, "k", first); writeStringValue(out, "rfl"); - writeKey(out, "life", first); writeHex128Value(out, life_id); - writeKey(out, "cls", first); writeIntText(static_cast(cov.classification), out); - writeKey(out, "lfe", first); writeU64StringValue(out, cov.last_folded_ref_id.writer_epoch); - writeKey(out, "lfs", first); writeU64StringValue(out, cov.last_folded_ref_id.ref_sequence); + writeStringField(out, FoldSealWire::kind, kRefLifeTag, first); + writeHex128Field(out, FoldSealWire::life, life_id, first); + writeNumberField(out, FoldSealWire::classification, static_cast(cov.classification), first); + writeU64StringField(out, FoldSealWire::fold_epoch, cov.last_folded_ref_id.writer_epoch, first); + writeU64StringField(out, FoldSealWire::fold_seq, cov.last_folded_ref_id.ref_sequence, first); if (cov.hold) { - writeKey(out, "hr", first); writeStringValue(out, holdReasonToWord(cov.hold->reason)); - writeKey(out, "hpe", first); writeU64StringValue(out, cov.hold->offending_position.writer_epoch); - writeKey(out, "hps", first); writeU64StringValue(out, cov.hold->offending_position.ref_sequence); - writeKey(out, "hrc", first); writeIntText(cov.hold->retry_count, out); - writeKey(out, "hnr", first); writeU64StringValue(out, cov.hold->next_retry_round); + writeStringField(out, FoldSealWire::hold_reason, holdReasonToWord(cov.hold->reason), first); + writeU64StringField(out, FoldSealWire::hold_epoch, cov.hold->offending_position.writer_epoch, first); + writeU64StringField(out, FoldSealWire::hold_seq, cov.hold->offending_position.ref_sequence, first); + writeNumberField(out, FoldSealWire::retries, cov.hold->retry_count, first); + writeU64StringField(out, FoldSealWire::retry_round, cov.hold->next_retry_round, first); } if (life_state.cleanup_evidence) { - writeKey(out, "rte", first); - writeU64StringValue(out, life_state.cleanup_evidence->remove_txn_id.writer_epoch); - writeKey(out, "rts", first); - writeU64StringValue(out, life_state.cleanup_evidence->remove_txn_id.ref_sequence); + writeU64StringField(out, FoldSealWire::remove_epoch, life_state.cleanup_evidence->remove_txn_id.writer_epoch, first); + writeU64StringField(out, FoldSealWire::remove_seq, life_state.cleanup_evidence->remove_txn_id.ref_sequence, first); } closeObject(out, first); closeLine("rfl"); @@ -293,7 +317,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) std::sort(runs.begin(), runs.end(), [](const RunRef & a, const RunRef & b) { return a.key < b.key; }); for (const RunRef & r : runs) { - writeRun(out, "btr", r); + writeRun(out, kBlobRunTag, r); closeLine("btr"); } } @@ -303,11 +327,11 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const auto & [shard, s] : seal.condemned_summary) { bool first = true; - writeKey(out, "k", first); writeStringValue(out, "cnd"); - writeKey(out, "shard", first); writeIntText(shard, out); - writeKey(out, "ct", first); writeIntText(s.condemned_total, out); - writeKey(out, "pt", first); writeIntText(s.pending_total, out); - writeKey(out, "ocr", first); writeU64StringValue(out, s.oldest_nonpending_condemn_round); + writeStringField(out, FoldSealWire::kind, kCondemnedTag, first); + writeNumberField(out, FoldSealWire::shard, shard, first); + writeNumberField(out, FoldSealWire::condemned_total, s.condemned_total, first); + writeNumberField(out, FoldSealWire::pending_total, s.pending_total, first); + writeU64StringField(out, FoldSealWire::oldest_round, s.oldest_nonpending_condemn_round, first); closeObject(out, first); closeLine("cnd"); ++n; @@ -338,8 +362,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect String key; while (r.nextKey(key)) { - if (key == "g") seal.generation = r.readU64String(); - else if (key == "pg") seal.parent_generation = r.readU64String(); + if (key == FoldSealWire::generation) seal.generation = r.readU64String(); + else if (key == FoldSealWire::parent_generation) seal.parent_generation = r.readU64String(); else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } } @@ -370,11 +394,11 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect seal.generation, *expected_generation); return seal; } - if (key != "k") + if (key != FoldSealWire::kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"k\""); const String kind = r.readString(); - if (kind == "rfl") + if (kind == kRefLifeTag) { std::optional life_id; RefCoverage cov; @@ -395,17 +419,17 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional remove_txn_sequence; while (r.nextKey(key)) { - if (key == "life") life_id = r.readHex128(); - else if (key == "cls") classification = r.readU64Number(); - else if (key == "lfe") cov.last_folded_ref_id.writer_epoch = r.readU64String(); - else if (key == "lfs") cov.last_folded_ref_id.ref_sequence = r.readU64String(); - else if (key == "hr") hold_reason = holdReasonFromWord(r.readString()); - else if (key == "hpe") hold_epoch = r.readU64String(); - else if (key == "hps") hold_sequence = r.readU64String(); - else if (key == "hrc") hold_retry_count = r.readU32Number(); - else if (key == "hnr") hold_next_retry_round = r.readU64String(); - else if (key == "rte") remove_txn_epoch = r.readU64String(); - else if (key == "rts") remove_txn_sequence = r.readU64String(); + if (key == FoldSealWire::life) life_id = r.readHex128(); + else if (key == FoldSealWire::classification) classification = r.readU64Number(); + else if (key == FoldSealWire::fold_epoch) cov.last_folded_ref_id.writer_epoch = r.readU64String(); + else if (key == FoldSealWire::fold_seq) cov.last_folded_ref_id.ref_sequence = r.readU64String(); + else if (key == FoldSealWire::hold_reason) hold_reason = holdReasonFromWord(r.readString()); + else if (key == FoldSealWire::hold_epoch) hold_epoch = r.readU64String(); + else if (key == FoldSealWire::hold_seq) hold_sequence = r.readU64String(); + else if (key == FoldSealWire::retries) hold_retry_count = r.readU32Number(); + else if (key == FoldSealWire::retry_round) hold_next_retry_round = r.readU64String(); + else if (key == FoldSealWire::remove_epoch) remove_txn_epoch = r.readU64String(); + else if (key == FoldSealWire::remove_seq) remove_txn_sequence = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown rfl key '{}'", key); } @@ -478,7 +502,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect "CAS fold seal: a second ref-life record for '{}' -- a life id appears at most once", life_hex); } - else if (kind == "btr") + else if (kind == kBlobRunTag) { std::optional run_key; std::optional checksum; @@ -486,19 +510,19 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional generation; while (r.nextKey(key)) { - if (key == "key") run_key = r.readString(); - else if (key == "ck") checksum = r.readHex128(); - else if (key == "shard") shard = r.readU64Number(); - else if (key == "gen") generation = r.readU64String(); + if (key == FoldSealWire::run_key) run_key = r.readString(); + else if (key == FoldSealWire::checksum) checksum = r.readHex128(); + else if (key == FoldSealWire::shard) shard = r.readU64Number(); + else if (key == FoldSealWire::key_generation) generation = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown run key '{}'", key); } if (!run_key || !checksum || !shard || !generation) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: btr requires key, ck, shard, and gen"); seal.blob_target_runs.push_back(RunRef{ - .key = std::move(*run_key), .checksum = *checksum, .shard = *shard, .generation = *generation}); + .key = std::move(*run_key), .checksum = *checksum, .shard = *shard, .key_generation = *generation}); } - else if (kind == "cnd") + else if (kind == kCondemnedTag) { std::optional shard; std::optional condemned_total; @@ -506,10 +530,10 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect std::optional oldest_nonpending_condemn_round; while (r.nextKey(key)) { - if (key == "shard") shard = r.readU64Number(); - else if (key == "ct") condemned_total = r.readU64Number(); - else if (key == "pt") pending_total = r.readU64Number(); - else if (key == "ocr") oldest_nonpending_condemn_round = r.readU64String(); + if (key == FoldSealWire::shard) shard = r.readU64Number(); + else if (key == FoldSealWire::condemned_total) condemned_total = r.readU64Number(); + else if (key == FoldSealWire::pending_total) pending_total = r.readU64Number(); + else if (key == FoldSealWire::oldest_round) oldest_nonpending_condemn_round = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown cnd key '{}'", key); } if (!shard || !condemned_total || !pending_total || !oldest_nonpending_condemn_round) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h index f6d00a6e3b50..ea4200e29eea 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h @@ -27,7 +27,7 @@ struct RunRef String key; UInt128 checksum{}; uint64_t shard = 0; /// gc-shard this run belongs to (REQUIRED for blob_target_runs) - uint64_t generation = 0; /// generation whose key namespace physically holds the object (for retention) + uint64_t key_generation = 0; /// generation whose key namespace physically holds the object (for retention) bool operator==(const RunRef &) const = default; }; @@ -147,7 +147,7 @@ struct RefLifeFoldState /// must not be interpreted as zero. struct CondemnedSummary { - uint64_t condemned_total = 0; /// count of `kCondemned` rows in this shard's sealed run + uint64_t condemned_total = 0; /// count of `RunMarker::Condemned` rows in this shard's sealed run uint64_t pending_total = 0; /// how many of those are `delete_pending` (a graduation is due) uint64_t oldest_nonpending_condemn_round = UINT64_MAX; /// min condemn_round over non-pending; UINT64_MAX = none bool operator==(const CondemnedSummary &) const = default; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp index 362306691473..dc9665aff837 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp @@ -1,6 +1,8 @@ #include #include +#include + namespace DB { namespace ErrorCodes @@ -197,6 +199,18 @@ const FormatTraits * traitsForType(std::string_view type) return nullptr; } +std::span allRegisteredFormatIds() +{ + static const auto ids = [] + { + std::array out{}; + for (size_t i = 0; i < std::size(TRAITS); ++i) + out[i] = TRAITS[i].id; + return out; + }(); + return ids; +} + std::string_view storedSuffix(FormatId id) { return traitsFor(id).compression == CompressionPolicy::Always ? ".zst" : ""; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h index 1acc2b3925de..1c98e4f283f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h @@ -202,6 +202,7 @@ const FormatTraits & traitsFor(FormatId id); /// Looks up a header-line `type` string. Returns nullptr for an unregistered type; it does not throw /// because callers use this result to classify the input before decoding it. const FormatTraits * traitsForType(std::string_view type); +std::span allRegisteredFormatIds(); /// Returns the storage-key suffix for `id`: `.zst` for `Always`, and an empty suffix otherwise. /// Key builders use this policy directly so a point lookup never has to inspect the object body or /// try multiple keys. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp index c5dda3286ad6..bc0720bf6f27 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp @@ -13,6 +13,11 @@ namespace DB::ErrorCodes namespace DB::Cas { +namespace GcMaintenanceWire +{ + constexpr WireKey janitor_cursor{"cur"}; +} + String encodeGcMaintenanceState(const GcMaintenanceState & state) { if (state.janitor_cursor.size() > kMaxGcMaintenanceCursorBytes) @@ -22,8 +27,7 @@ String encodeGcMaintenanceState(const GcMaintenanceState & state) CasJsonWriter out; writeHeaderLine(out, FormatId::GcMaintenanceState); bool first = true; - writeKey(out, "cur", first); - writeStringValue(out, state.janitor_cursor); + writeStringField(out, GcMaintenanceWire::janitor_cursor, state.janitor_cursor, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -46,7 +50,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data) String key; while (reader.nextKey(key)) { - if (key == "cur") + if (key == GcMaintenanceWire::janitor_cursor) { result.janitor_cursor = reader.readString(); has_cursor = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index e69ea98a799e..9ee65ebafa78 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -18,29 +19,33 @@ namespace DB::Cas namespace { -std::string_view outcomeKindToWord(OutcomeKind o) +namespace GcOutcomesWire { - switch (o) - { - case OutcomeKind::Deleted: return "deleted"; - case OutcomeKind::Absent: return "absent"; - case OutcomeKind::Replaced: return "replaced"; - case OutcomeKind::Spared: return "spared"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: unknown OutcomeKind {}", static_cast(o)); + constexpr WireKey kind{"k"}; + constexpr WireKey outcome{"oc"}; } +constexpr EnumWireTable kOutcomeKindWords{{{ + {OutcomeKind::Deleted, "deleted"}, + {OutcomeKind::Absent, "absent"}, + {OutcomeKind::Replaced, "replaced"}, + {OutcomeKind::Spared, "spared"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + OutcomeKind outcomeKindFromWord(std::string_view w) { - if (w == "deleted") return OutcomeKind::Deleted; - if (w == "absent") return OutcomeKind::Absent; - if (w == "replaced") return OutcomeKind::Replaced; - if (w == "spared") return OutcomeKind::Spared; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: unknown outcome '{}'", w); + return kOutcomeKindWords.fromWord(w, "CAS outcome log outcome kind"); } } +std::string_view outcomeKindToWireWord(OutcomeKind outcome) +{ + return kOutcomeKindWords.toWord(outcome, "CAS outcome log outcome kind"); +} + String encodeOutcomeLog(const OutcomeLog & log) { CasJsonWriter out(256); @@ -48,12 +53,10 @@ String encodeOutcomeLog(const OutcomeLog & log) for (const OutcomeEntry & e : log.entries) { bool first = true; - writeKey(out, "k", first); - writeStringValue(out, objectKindToWord(e.kind)); + writeStringField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); writeBlobRefFields(out, first, e.ref); /// ha + h writeTokenFields(out, first, e.token); /// tt + tv - writeKey(out, "oc", first); - writeStringValue(out, outcomeKindToWord(e.outcome)); + writeStringField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); closeObject(out, first); writeChar('\n', out); } @@ -92,34 +95,21 @@ OutcomeLog decodeOutcomeLog(std::string_view data) } OutcomeEntry e; - String ha; - String hhex; - String tv; - bool have_ha = false; - bool have_h = false; - bool have_tt = false; - TokenType tt{}; + BlobRefFields blob_ref_fields; + TokenFields token_fields; do { - if (key == "k") e.kind = objectKindFromWord(r.readString(), "outcome log"); - else if (key == "ha") { ha = r.readString(); have_ha = true; } - else if (key == "h") { hhex = r.readString(); have_h = true; } - else if (key == "tt") { tt = tokenTypeFromWord(r.readString(), "outcome log"); have_tt = true; } - else if (key == "tv") tv = r.readString(); - else if (key == "oc") e.outcome = outcomeKindFromWord(r.readString()); + if (key == GcOutcomesWire::kind) e.kind = objectKindFromWord(r.readString(), "outcome log"); + else if (matchBlobRefFields(key, r, blob_ref_fields)) {} + else if (matchTokenFields(key, r, token_fields)) {} + else if (key == GcOutcomesWire::outcome) e.outcome = outcomeKindFromWord(r.readString()); else r.skipUnknown(key); } while (r.nextKey(key)); - if (!have_ha || !have_h || !have_tt) + if (!blob_ref_fields.algo_word || !blob_ref_fields.digest_hex || !token_fields.type_word) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: record missing ha/h/tt"); - const BlobHashAlgo algo = blobHashAlgoFromWord(ha, "outcome log"); - /// Validate the digest width before `fromHex`: a width mismatch must surface as the - /// CORRUPTED_DATA required for malformed serialized input, not fromHex's BAD_ARGUMENTS. - if (hhex.size() != blobHashLenFor(algo) * 2) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS outcome log: digest width {} does not match algo '{}'", hhex.size(), ha); - e.ref = BlobRef{algo, codecFor(algo).fromHex(hhex)}; - e.token = Token{tv, tt}; + e.ref = blob_ref_fields.build("outcome log"); + e.token = Token{token_fields.value.value_or(""), tokenTypeFromWord(*token_fields.type_word, "outcome log")}; if (!line_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: junk after record"); log.entries.push_back(std::move(e)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index 09a850ee66ff..474c3c136a0b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -27,6 +27,9 @@ enum class OutcomeKind : uint8_t Spared = 4, /// The merge found a positive in-degree, so the candidate was kept alive. }; +/// Canonical wire word for one `OutcomeKind`. +std::string_view outcomeKindToWireWord(OutcomeKind outcome); + /// One observation about a blob incarnation considered by GC. `token` identifies the exact /// incarnation that GC examined, while `ref` identifies the content address; retaining both lets /// replay and inspection distinguish an absent object from a replacement that won a race with GC. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp index 7012c6787f70..b70f016cbae0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp @@ -16,6 +16,24 @@ namespace ErrorCodes namespace DB::Cas { +namespace GcStateWire +{ + constexpr WireKey round{"rnd"}; + constexpr WireKey gc_shards{"gcs"}; + constexpr WireKey snap_generation{"sg"}; + constexpr WireKey snap_pruned_through{"spt"}; + constexpr WireKey snap_attempt{"sa"}; + constexpr WireKey manifest_sweep_cursor{"msc"}; + constexpr WireKey lease_owner{"lo"}; + constexpr WireKey lease_seq{"ls"}; +} + +namespace GcHeartbeatWire +{ + constexpr WireKey owner{"by"}; + constexpr WireKey hb_seq{"seq"}; +} + String encodeGcState(const GcState & state) { if (state.gc_shards < 1) @@ -23,14 +41,14 @@ String encodeGcState(const GcState & state) CasJsonWriter out(256); writeHeaderLine(out, FormatId::GcState); bool first = true; - writeKey(out, "rnd", first); writeU64StringValue(out, state.round); - writeKey(out, "gcs", first); writeIntText(state.gc_shards, out); - writeKey(out, "sg", first); writeU64StringValue(out, state.snap_generation); - writeKey(out, "spt", first); writeU64StringValue(out, state.snap_pruned_through); - writeKey(out, "sa", first); writeU64StringValue(out, state.snap_attempt); - writeKey(out, "msc", first); writeStringValue(out, state.manifest_sweep_cursor); - writeKey(out, "lo", first); writeHex128Value(out, state.lease.owner); - writeKey(out, "ls", first); writeU64StringValue(out, state.lease.seq); + writeU64StringField(out, GcStateWire::round, state.round, first); + writeNumberField(out, GcStateWire::gc_shards, state.gc_shards, first); + writeU64StringField(out, GcStateWire::snap_generation, state.snap_generation, first); + writeU64StringField(out, GcStateWire::snap_pruned_through, state.snap_pruned_through, first); + writeU64StringField(out, GcStateWire::snap_attempt, state.snap_attempt, first); + writeStringField(out, GcStateWire::manifest_sweep_cursor, state.manifest_sweep_cursor, first); + writeHex128Field(out, GcStateWire::lease_owner, state.lease.owner, first); + writeU64StringField(out, GcStateWire::lease_seq, state.lease.seq, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -49,15 +67,27 @@ GcState decodeGcState(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "rnd") state.round = r.readU64String(); - else if (key == "gcs") { state.gc_shards = r.readU64Number(); saw_gcs = true; } - else if (key == "sg") state.snap_generation = r.readU64String(); - else if (key == "spt") state.snap_pruned_through = r.readU64String(); - else if (key == "sa") state.snap_attempt = r.readU64String(); - else if (key == "msc") state.manifest_sweep_cursor = r.readString(); - else if (key == "lo") state.lease.owner = r.readHex128(); - else if (key == "ls") state.lease.seq = r.readU64String(); - else r.skipUnknown(key); + if (key == GcStateWire::round) + state.round = r.readU64String(); + else if (key == GcStateWire::gc_shards) + { + state.gc_shards = r.readU64Number(); + saw_gcs = true; + } + else if (key == GcStateWire::snap_generation) + state.snap_generation = r.readU64String(); + else if (key == GcStateWire::snap_pruned_through) + state.snap_pruned_through = r.readU64String(); + else if (key == GcStateWire::snap_attempt) + state.snap_attempt = r.readU64String(); + else if (key == GcStateWire::manifest_sweep_cursor) + state.manifest_sweep_cursor = r.readString(); + else if (key == GcStateWire::lease_owner) + state.lease.owner = r.readHex128(); + else if (key == GcStateWire::lease_seq) + state.lease.seq = r.readU64String(); + else + r.skipUnknown(key); } /// Fail closed on an absent gcs: the writer always emits it, so a missing key means a corrupt object. /// Do NOT silently keep the struct default (1) — that would hide corruption (no-fallback principle). @@ -75,8 +105,8 @@ String encodeGcHeartbeat(const GcHeartbeat & hb) CasJsonWriter out(256); writeHeaderLine(out, FormatId::GcHeartbeat); bool first = true; - writeKey(out, "by", first); writeHex128Value(out, hb.owner); - writeKey(out, "seq", first); writeU64StringValue(out, hb.hb_seq); + writeHex128Field(out, GcHeartbeatWire::owner, hb.owner, first); + writeU64StringField(out, GcHeartbeatWire::hb_seq, hb.hb_seq, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -96,12 +126,12 @@ GcHeartbeat decodeGcHeartbeat(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "by") + if (key == GcHeartbeatWire::owner) { hb.owner = r.readHex128(); saw_by = true; } - else if (key == "seq") + else if (key == GcHeartbeatWire::hb_seq) { hb.hb_seq = r.readU64String(); saw_seq = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index 5e7f9ff5ffbf..3e406ad01203 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -22,41 +23,37 @@ namespace DB::Cas namespace { -std::string_view placementToWord(EntryPlacement p) +namespace PartManifestWire { - switch (p) - { - case EntryPlacement::Inline: return "inline"; - case EntryPlacement::Blob: return "blob"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: unknown placement {}", static_cast(p)); + constexpr WireKey ns{"ns"}; + constexpr WireKey payload_digest{"pd"}; + constexpr WireKey path{"p"}; + constexpr WireKey place{"pm"}; + constexpr WireKey size{"sz"}; + constexpr WireKey inline_size{"il"}; } -EntryPlacement placementFromWord(std::string_view w) -{ - if (w == "inline") return EntryPlacement::Inline; - if (w == "blob") return EntryPlacement::Blob; - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: unknown placement '{}'", w); -} +constexpr EnumWireTable kEntryPlacementWords{{{ + {EntryPlacement::Inline, "inline"}, + {EntryPlacement::Blob, "blob"}, +}}}; + +static_assert(casEnumTableCoversEnum()); /// One entry-record line: {"p","pm", then either the Blob's "ha"/"h"/"sz" or the Inline's "il"}. void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) { bool first = true; - writeKey(out, "p", first); - writeStringValue(out, e.path); - writeKey(out, "pm", first); - writeStringValue(out, placementToWord(e.placement)); + writeStringField(out, PartManifestWire::path, e.path, first); + writeWordField(out, PartManifestWire::place, entryPlacementToWireWord(e.placement), first); if (e.placement == EntryPlacement::Blob) { writeBlobRefFields(out, first, e.ref); /// ha + h - writeKey(out, "sz", first); - writeIntText(e.blob_size, out); + writeNumberField(out, PartManifestWire::size, e.blob_size, first); } else { - writeKey(out, "il", first); - writeIntText(e.inline_bytes.size(), out); + writeNumberField(out, PartManifestWire::inline_size, e.inline_bytes.size(), first); } closeObject(out, first); writeChar('\n', out); @@ -80,6 +77,11 @@ String bannerFor(std::string_view path, uint64_t n) } +std::string_view entryPlacementToWireWord(EntryPlacement placement) +{ + return kEntryPlacementWords.toWord(placement, "PartManifest: EntryPlacement"); +} + String encodePartManifest(const PartManifest & m) { /// Canonical path order plus duplicate-path rejection makes the encoded record sequence @@ -101,11 +103,9 @@ String encodePartManifest(const PartManifest & m) /// namespace + payload digest. { bool first = true; - writeManifestRefFields(out, first, "", m.ref); - writeKey(out, "ns", first); - writeStringValue(out, m.root_namespace_id.string()); - writeKey(out, "pd", first); - writeHex128Value(out, m.payload_digest); + writeManifestRefFields(out, first, kBareManifestRefKeys, m.ref); + writeStringField(out, PartManifestWire::ns, m.root_namespace_id.string(), first); + writeHex128Field(out, PartManifestWire::payload_digest, m.payload_digest, first); closeObject(out, first); writeChar('\n', out); } @@ -144,28 +144,22 @@ PartManifest decodePartManifest(std::string_view data) const String meta = readLine(in, line_cap, "cas_part_manifest"); ReadBufferFromMemory mm(meta.data(), meta.size()); JsonObjectReader r(mm, KeyStrictness::Tolerant, "cas_part_manifest"); - std::optional me; - std::optional mb; - std::optional mo; + ManifestRefFields fields; std::optional ns; std::optional pd; String key; while (r.nextKey(key)) { - if (key == "me") me = r.readU64String(); - else if (key == "mb") mb = r.readU64String(); - else if (key == "mo") mo = r.readU64Number(); - else if (key == "ns") ns = r.readString(); - else if (key == "pd") pd = r.readHex128(); + if (matchManifestRefFields(key, r, kBareManifestRefKeys, fields)) {} + else if (key == PartManifestWire::ns) ns = r.readString(); + else if (key == PartManifestWire::payload_digest) pd = r.readHex128(); else r.skipUnknown(key); } - if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing me/mb/mo"); if (!ns) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing ns"); if (!pd) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing pd"); - m.ref = manifestRefFromFields(*me, *mb, *mo, "PartManifest", "descriptor"); + m.ref = fields.buildRef("PartManifest", "descriptor"); m.root_namespace_id = RootNamespace(*ns); m.payload_digest = *pd; if (!mm.eof()) @@ -176,6 +170,7 @@ PartManifest decodePartManifest(std::string_view data) /// the payload zone below can read exactly that many raw bytes back into `inline_bytes`. /// Index-aligned with `m.entries` (Blob entries push an unused 0 placeholder). std::vector inline_lens; + String blob_ref_what; /// reused across Blob entries so the error context does not allocate per row while (true) { const String line = readLine(in, line_cap, "cas_part_manifest"); @@ -198,7 +193,7 @@ PartManifest decodePartManifest(std::string_view data) break; } - if (key != "p") + if (key != PartManifestWire::path) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"p\""); ManifestEntry e; e.path = r.readString(); @@ -218,40 +213,31 @@ PartManifest decodePartManifest(std::string_view data) } std::optional pm; - std::optional ha; - std::optional h; + BlobRefFields blob_ref; std::optional sz; std::optional il; while (r.nextKey(key)) { - if (key == "pm") pm = r.readString(); - else if (key == "ha") ha = r.readString(); - else if (key == "h") h = r.readString(); - else if (key == "sz") sz = r.readU64Number(); - else if (key == "il") il = r.readU64Number(); + if (key == PartManifestWire::place) pm = r.readString(); + else if (matchBlobRefFields(key, r, blob_ref)) {} + else if (key == PartManifestWire::size) sz = r.readU64Number(); + else if (key == PartManifestWire::inline_size) il = r.readU64Number(); else r.skipUnknown(key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after record"); if (!pm) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing pm", e.path); - e.placement = placementFromWord(*pm); + e.placement = kEntryPlacementWords.fromWord(*pm, "PartManifest"); if (e.placement == EntryPlacement::Blob) { - if (!ha || !h || !sz) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing ha/h/sz", e.path); - const BlobHashAlgo algo = blobHashAlgoFromWord(*ha, "PartManifest entry"); - /// Validate the digest width before calling `fromHex`. A width mismatch otherwise - /// produces `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed - /// serialized input, allowing an invalid manifest to escape the decoder's fail-closed - /// error contract. - const uint64_t expected_hex_len = blobHashLenFor(algo) * 2; - if (h->size() != expected_hex_len) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "PartManifest: entry '{}' digest hex width {} does not match algo width {}", - e.path, h->size(), expected_hex_len); - e.ref = BlobRef{algo, codecFor(algo).fromHex(*h)}; + if (!sz) + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing sz", e.path); + blob_ref_what.assign("PartManifest entry '"); + blob_ref_what += e.path; + blob_ref_what += '\''; + e.ref = blob_ref.build(blob_ref_what); e.blob_size = *sz; inline_lens.push_back(0); /// unused for Blob; keeps inline_lens index-aligned with entries } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h index f2a416d15743..e02e1ce6a917 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h @@ -42,6 +42,9 @@ enum class EntryPlacement : uint8_t Blob = 2, /// bytes stored as a content-addressed blob at `blobKey` }; +/// Canonical wire word for one manifest entry placement. +std::string_view entryPlacementToWireWord(EntryPlacement placement); + /// One file entry inside a part manifest. `ref` is meaningful only for `Blob`; `inline_bytes` only /// for `Inline`. `blob_size` is the raw `Blob` byte count (0 for `Inline` — decode never fills it for /// an inline entry, since the wire format carries no redundant size for inline bytes). Use `size()` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp index 50e9843e9254..4e46131779cb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp @@ -1,3 +1,4 @@ +#include #include #include #include @@ -16,6 +17,15 @@ namespace ErrorCodes namespace DB::Cas { +namespace PoolMetaWire +{ + constexpr WireKey pool_id{"pid"}; + constexpr WireKey blob_header_len{"hln"}; + constexpr WireKey gc_shards{"gcs"}; + constexpr WireKey min_reader_generation{"mrg"}; + constexpr WireKey algos_used{"alg"}; +} + /// Minimum `blob_header_len` that provably fits the v3 `cas_blob` JSON envelope's mandatory (always- /// written) non-ref fields, computed at type maxima from `encodeEnvelopeHeader` (CasBlobEnvelopeFormat.cpp): /// {"type":"cas_blob" 18 @@ -33,7 +43,6 @@ namespace DB::Cas /// that used to mask this is gone). We floor at 240 (a multiple of 8 comfortably above 225, leaving /// >= 15 bytes for the diagnostic ref even at type maxima, and well under the 256 default) so a /// misconfigured pool fails at CREATION with BAD_ARGUMENTS, not at first write with LOGICAL_ERROR. -static constexpr uint64_t kMinBlobHeaderLen = 240; void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what) { @@ -52,14 +61,16 @@ void validatePoolAlgosUsed(const std::vector & algos_used, int error_co throw Exception(error_code, "CAS {}: algos_used must be non-empty", what); for (size_t i = 0; i < algos_used.size(); ++i) { - try - { - blobHashAlgoName(static_cast(algos_used[i])); - } - catch (const Exception &) - { + /// A direct membership scan, not `blobHashAlgoName`: that throws `LOGICAL_ERROR`, which + /// aborts at construction under a sanitizer/debug build before any catch can run, but a + /// persisted `algos_used` byte is exactly the unvalidated input this function must reject + /// cleanly instead. + bool known = false; + for (const auto & entry : kBlobHashAlgoWords.entries) + if (static_cast(entry.value) == algos_used[i]) + known = true; + if (!known) throw Exception(error_code, "CAS {}: algos_used contains an unknown algo {}", what, algos_used[i]); - } if (i > 0 && algos_used[i] <= algos_used[i - 1]) throw Exception(error_code, "CAS {}: algos_used must be strictly sorted with no duplicates, got {} at index {} not after {}", @@ -73,15 +84,11 @@ String encodePoolMeta(const PoolMeta & pm) writeHeaderLine(out, FormatId::PoolMeta); bool first = true; - writeKey(out, "pid", first); - writeHex128Value(out, pm.pool_id); - writeKey(out, "hln", first); - writeIntText(pm.blob_header_len, out); - writeKey(out, "gcs", first); - writeIntText(pm.gc_shards, out); - writeKey(out, "mrg", first); - writeIntText(pm.min_reader_generation, out); - writeKey(out, "alg", first); + writeHex128Field(out, PoolMetaWire::pool_id, pm.pool_id, first); + writeNumberField(out, PoolMetaWire::blob_header_len, pm.blob_header_len, first); + writeNumberField(out, PoolMetaWire::gc_shards, pm.gc_shards, first); + writeNumberField(out, PoolMetaWire::min_reader_generation, pm.min_reader_generation, first); + writeKey(out, PoolMetaWire::algos_used, first); { /// Comma-joined algo words (tiny list, <=3): "ch128" or "ch128,sha256". String joined; @@ -127,21 +134,21 @@ PoolMeta decodePoolMeta(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "pid") + if (key == PoolMetaWire::pool_id) { pm.pool_id = r.readHex128(); saw_pid = true; } - else if (key == "hln") + else if (key == PoolMetaWire::blob_header_len) pm.blob_header_len = r.readU64Number(); - else if (key == "gcs") + else if (key == PoolMetaWire::gc_shards) { pm.gc_shards = r.readU64Number(); saw_gc_shards = true; } - else if (key == "mrg") + else if (key == PoolMetaWire::min_reader_generation) pm.min_reader_generation = r.readU64Number(); - else if (key == "alg") + else if (key == PoolMetaWire::algos_used) { const String joined = r.readString(); size_t start = 0; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index b21458aaf6e2..985c1964e9e5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -19,6 +20,25 @@ namespace DB::Cas namespace { +namespace RunWire +{ + constexpr WireKey ref{"b"}; + constexpr WireKey src{"s"}; + constexpr WireKey mark{"m"}; + constexpr WireKey pending{"pend"}; + constexpr WireKey size{"sz"}; + constexpr WireKey condemn_round{"cr"}; + constexpr WireKey confirmed{"mc"}; +} + +constexpr EnumWireTable kRunMarkerWords{{{ + {RunMarker::Zero, "zero"}, + {RunMarker::Edge, "edge"}, + {RunMarker::Condemned, "condemned"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + UInt128 toWideChecksum(CityHash_v1_0_2::uint128 h) { /// Keep the high and low halves in the same order for the write-side helper and the streaming @@ -72,32 +92,20 @@ BlobRef parseB(std::string_view b) if (digest_hex.size() != static_cast(blobHashLenFor(algo)) * 2) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: digest hex width {} does not match algo width {}", digest_hex.size(), blobHashLenFor(algo) * 2); + for (const char c : digest_hex) + if (!isLowercaseHexChar(c)) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: non-lowercase-hex digest in record key"); BlobRef ref; ref.algo = algo; ref.digest = codecFor(algo).fromHex(String(digest_hex)); return ref; } -std::string_view markerToWord(char m) -{ - switch (m) - { - case kEdgeActive: return "edge"; - case kZeroMarker: return "zero"; - case kCondemned: return "condemned"; - default: - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: unknown row marker 0x{:02x}", static_cast(m)); - } } -char markerFromWord(std::string_view w) +std::string_view runMarkerToWireWord(RunMarker marker) { - if (w == "edge") return kEdgeActive; - if (w == "zero") return kZeroMarker; - if (w == "condemned") return kCondemned; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: unknown row marker '{}'", w); -} - + return kRunMarkerWords.toWord(marker, "CAS cas_run: RunMarker"); } void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) @@ -174,23 +182,16 @@ void SourceEdgeRunWriter::append(const SourceEdgeRecord & rec) scratch.clear(); bool first = true; - writeKey(scratch, "b", first); - writeStringValue(scratch, renderB(rec.ref)); - writeKey(scratch, "s", first); - writeHex128Value(scratch, rec.source_id); - writeKey(scratch, "m", first); - writeStringValue(scratch, markerToWord(rec.marker)); - if (rec.marker == kCondemned) + writeStringField(scratch, RunWire::ref, renderB(rec.ref), first); + writeHex128Field(scratch, RunWire::src, rec.source_id, first); + writeWordField(scratch, RunWire::mark, runMarkerToWireWord(rec.marker), first); + if (rec.marker == RunMarker::Condemned) { - writeKey(scratch, "pend", first); - writeBoolValue(scratch, rec.delete_pending); + writeBoolField(scratch, RunWire::pending, rec.delete_pending, first); writeTokenFields(scratch, first, rec.token); /// tt + tv - writeKey(scratch, "sz", first); - writeIntText(rec.size, scratch); - writeKey(scratch, "cr", first); - writeU64StringValue(scratch, rec.condemn_round); - writeKey(scratch, "mc", first); - writeBoolValue(scratch, rec.marker_confirmed); + writeNumberField(scratch, RunWire::size, rec.size, first); + writeU64StringField(scratch, RunWire::condemn_round, rec.condemn_round, first); + writeBoolField(scratch, RunWire::confirmed, rec.marker_confirmed, first); } closeObject(scratch, first); writeChar('\n', scratch); @@ -263,7 +264,7 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) SourceEdgeRecord out; String b; - String tv; + TokenFields token_fields; bool have_b = false; bool have_s = false; bool have_m = false; @@ -273,29 +274,27 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) bool have_sz = false; bool have_cr = false; bool have_mc = false; - TokenType tt{}; do { - if (key == "b") { b = r.readString(); have_b = true; } - else if (key == "s") { out.source_id = r.readHex128(); have_s = true; } - else if (key == "m") { out.marker = markerFromWord(r.readString()); have_m = true; } - else if (key == "pend") { out.delete_pending = r.readBool(); have_pend = true; } - else if (key == "tt") { tt = tokenTypeFromWord(r.readString(), "cas_run"); have_tt = true; } - else if (key == "tv") { tv = r.readString(); have_tv = true; } - else if (key == "sz") { out.size = r.readU64Number(); have_sz = true; } - else if (key == "cr") { out.condemn_round = r.readU64String(); have_cr = true; } - else if (key == "mc") { out.marker_confirmed = r.readBool(); have_mc = true; } + if (key == RunWire::ref) { b = r.readString(); have_b = true; } + else if (key == RunWire::src) { out.source_id = r.readHex128(); have_s = true; } + else if (key == RunWire::mark) { out.marker = kRunMarkerWords.fromWord(r.readString(), "CAS cas_run"); have_m = true; } + else if (key == RunWire::pending) { out.delete_pending = r.readBool(); have_pend = true; } + else if (matchTokenFields(key, r, token_fields)) { have_tt = token_fields.type_word.has_value(); have_tv = token_fields.value.has_value(); } + else if (key == RunWire::size) { out.size = r.readU64Number(); have_sz = true; } + else if (key == RunWire::condemn_round) { out.condemn_round = r.readU64String(); have_cr = true; } + else if (key == RunWire::confirmed) { out.marker_confirmed = r.readBool(); have_mc = true; } else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } while (r.nextKey(key)); if (!have_b || !have_s || !have_m) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing b/s/m"); out.ref = parseB(b); - if (out.marker == kCondemned) + if (out.marker == RunMarker::Condemned) { if (!have_pend || !have_tt || !have_tv || !have_sz || !have_cr || !have_mc) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pend/tt/tv/sz/cr/mc"); - out.token = Token{tv, tt}; + out.token = Token{*token_fields.value, tokenTypeFromWord(*token_fields.type_word, "cas_run")}; } else if (have_pend || have_tt || have_tv || have_sz || have_cr || have_mc) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: non-condemned record carries condemned fields"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index d5f9a4801caf..0d9c6dfdad4b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -11,6 +12,11 @@ #include #include +namespace DB::ErrorCodes +{ + extern const int CORRUPTED_DATA; +} + namespace DB::Cas { @@ -18,14 +24,30 @@ namespace DB::Cas /// format, shared by this codec and the GC fold that interprets the rows. /// /// Source-edge rows use `source_id == 0` as a sentinel key. A real active edge must never use that key; -/// both sentinel tags are restricted to it. `kZeroMarker` describes a zero transition for the current -/// generation and is dropped when the row is carried forward. `kCondemned` carries the condemned +/// both sentinel tags are restricted to it. `RunMarker::Zero` describes a zero transition for the current +/// generation and is dropped when the row is carried forward. `RunMarker::Condemned` carries the condemned /// incarnation at the sentinel key across generations until settlement; its payload contains the full /// deletion token and other condemned-row state. A condemned row subsumes the zero marker for that /// generation. -constexpr char kEdgeActive = 0x01; -constexpr char kZeroMarker = 0x00; -constexpr char kCondemned = 0x02; +enum class RunMarker : char +{ + Zero = 0x00, + Edge = 0x01, + Condemned = 0x02, +}; + +constexpr char runMarkerByte(RunMarker marker) +{ + return static_cast(marker); +} + +inline RunMarker runMarkerFromByte(char byte, std::string_view what) +{ + if (byte != runMarkerByte(RunMarker::Zero) && byte != runMarkerByte(RunMarker::Edge) + && byte != runMarkerByte(RunMarker::Condemned)) + throw Exception(ErrorCodes::CORRUPTED_DATA, "{}: unknown marker byte {}", what, static_cast(byte)); + return static_cast(byte); +} /// The `cas_run` codec represents the GC source-edge in-degree data plane as sorted NDJSON. This is /// the `RecordStream` family @@ -49,18 +71,18 @@ constexpr char kCondemned = 0x02; /// algo's width; `s` is the 32-hex source id. String-sorting records by (b, s) reproduces the current /// `(algorithm, digest, source_id)` byte order (lowercase hex preserves unsigned byte order and the /// algorithm byte is emitted first) — the invariant the fold's two-cursor merge depends on. The row-tag word -/// `m` maps to the `kEdgeActive`/`kZeroMarker`/`kCondemned` bytes; a `condemned` row additionally +/// `m` maps to the `RunMarker` bytes; a `condemned` row additionally /// carries the retired incarnation (`pend`/`tt`/`tv`/`sz`/`cr`) and the durable condemn-marker /// confirmation bit (`mc`). /// One decoded source-edge row. All fields are identifier-layer types so the codec stays backend-free. /// The condemned-only fields (`delete_pending`/`token`/`size`/`condemn_round`/`marker_confirmed`) are -/// meaningful only when `marker == kCondemned`. +/// meaningful only when `marker == RunMarker::Condemned`. struct SourceEdgeRecord { BlobRef ref{}; UInt128 source_id{}; - char marker = kEdgeActive; + RunMarker marker = RunMarker::Edge; bool delete_pending = false; Token token{}; uint64_t size = 0; @@ -71,6 +93,9 @@ struct SourceEdgeRecord /// The header-line `kind` word for the only live `cas_run` kind. inline constexpr std::string_view kSourceEdgeKindWord = "source_edge"; +/// Canonical wire word for one source-edge run marker. +std::string_view runMarkerToWireWord(RunMarker marker); + /// Write the typed header line `{"type":"cas_run","v":G_BUILD,"kind":""}\n` with a fixed key /// order for byte-determinism. The `kind` field distinguishes the record schema within the run /// family, so a reader can reject a valid run of the wrong kind before interpreting any records. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index c5b119ba44ca..66606ef5b632 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -20,30 +21,30 @@ namespace ErrorCodes namespace DB::Cas { -std::string_view nsStateToWord(NsState s) +namespace { - switch (s) - { - case NsState::Creating: return "creating"; - case NsState::Live: return "live"; - case NsState::Removing: return "removing"; - } - /// Every value reaching here came from a live `NsState` or from `nsStateFromWord`, which already - /// validated it on decode -- so this is a bug in THIS process, not corruption arriving from a - /// store. - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS ref catalog: unknown ns state {}", static_cast(s)); -} -NsState nsStateFromWord(std::string_view w) +namespace RefCatalogWire { - if (w == "creating") return NsState::Creating; - if (w == "live") return NsState::Live; - if (w == "removing") return NsState::Removing; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ns state '{}'", w); + constexpr WireKey kind{"k"}; + constexpr WireKey ns{"ns"}; + constexpr WireKey state{"st"}; + constexpr WireKey life{"inc"}; + constexpr WireKey remove_round{"rsr"}; + constexpr WireKey creator{"csr"}; + constexpr WireKey creator_epoch{"cwe"}; + constexpr WireKey creator_fence{"cfg"}; } -namespace -{ +constexpr std::string_view kEntryTag = "ent"; + +constexpr EnumWireTable kNsStateWords{{{ + {NsState::Creating, "creating"}, + {NsState::Live, "live"}, + {NsState::Removing, "removing"}, +}}}; + +static_assert(casEnumTableCoversEnum()); /// `creator` is required iff `state == Creating`, forbidden otherwise -- one predicate, used by both /// directions of the codec, so the writer's self-check and the reader's fail-close can never disagree. @@ -70,6 +71,16 @@ bool isCanonicalCatalogOrder(const std::vector & entries) } +std::string_view nsStateToWord(NsState s) +{ + return kNsStateWords.toWord(s, "CAS ref catalog"); +} + +NsState nsStateFromWord(std::string_view w) +{ + return kNsStateWords.fromWord(w, "CAS ref catalog ns state"); +} + String encodeRefCatalog(const RefCatalog & catalog) { const uint64_t line_cap = traitsFor(FormatId::RefCatalog).line_cap; @@ -136,19 +147,19 @@ String encodeRefCatalog(const RefCatalog & catalog) e.ns.string(), nsStateToWord(e.state), e.removal_started_round ? "carries" : "lacks"); bool first = true; - writeKey(out, "k", first); writeStringValue(out, "ent"); - writeKey(out, "ns", first); writeStringValue(out, e.ns.string()); - writeKey(out, "st", first); writeStringValue(out, nsStateToWord(e.state)); - writeKey(out, "inc", first); writeHex128Value(out, e.incarnation); + writeKey(out, RefCatalogWire::kind, first); writeStringValue(out, kEntryTag); + writeKey(out, RefCatalogWire::ns, first); writeStringValue(out, e.ns.string()); + writeKey(out, RefCatalogWire::state, first); writeStringValue(out, nsStateToWord(e.state)); + writeKey(out, RefCatalogWire::life, first); writeHex128Value(out, e.incarnation); if (e.removal_started_round) { - writeKey(out, "rsr", first); writeU64StringValue(out, *e.removal_started_round); + writeKey(out, RefCatalogWire::remove_round, first); writeU64StringValue(out, *e.removal_started_round); } if (e.creator) { - writeKey(out, "csr", first); writeStringValue(out, e.creator->server_root_id); - writeKey(out, "cwe", first); writeU64StringValue(out, e.creator->writer_epoch); - writeKey(out, "cfg", first); writeU64StringValue(out, e.creator->fence_generation); + writeKey(out, RefCatalogWire::creator, first); writeStringValue(out, e.creator->server_root_id); + writeKey(out, RefCatalogWire::creator_epoch, first); writeU64StringValue(out, e.creator->writer_epoch); + writeKey(out, RefCatalogWire::creator_fence, first); writeU64StringValue(out, e.creator->fence_generation); } closeObject(out, first); closeLine("ent"); @@ -190,10 +201,10 @@ RefCatalog decodeRefCatalog(std::string_view data) "CAS ref catalog: trailer count {} != {} records", n, seen); return catalog; } - if (key != "k") + if (key != RefCatalogWire::kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"k\""); const String kind = r.readString(); - if (kind != "ent") + if (kind != kEntryTag) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown record kind '{}'", kind); String ns_str; @@ -205,13 +216,13 @@ RefCatalog decodeRefCatalog(std::string_view data) std::optional removal_started_round; while (r.nextKey(key)) { - if (key == "ns") ns_str = r.readString(); - else if (key == "st") st_word = r.readString(); - else if (key == "inc") inc = r.readHex128(); - else if (key == "csr") csr = r.readString(); - else if (key == "cwe") cwe = r.readU64String(); - else if (key == "cfg") cfg = r.readU64String(); - else if (key == "rsr") removal_started_round = r.readU64String(); + if (key == RefCatalogWire::ns) ns_str = r.readString(); + else if (key == RefCatalogWire::state) st_word = r.readString(); + else if (key == RefCatalogWire::life) inc = r.readHex128(); + else if (key == RefCatalogWire::creator) csr = r.readString(); + else if (key == RefCatalogWire::creator_epoch) cwe = r.readU64String(); + else if (key == RefCatalogWire::creator_fence) cfg = r.readU64String(); + else if (key == RefCatalogWire::remove_round) removal_started_round = r.readU64String(); else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ent key '{}'", key); } if (!l.eof()) @@ -340,7 +351,7 @@ uint64_t widestBlobTargetRunReservationBytes(const Layout & layout, uint64_t gc_ .key = layout.blobTargetRunKey(max, max, gc_shards - 1, 0), .checksum = std::numeric_limits::max(), .shard = gc_shards - 1, - .generation = max}); + .key_generation = max}); return encodeFoldSeal(seal).size() - encodeFoldSeal(CasFoldSeal{}).size(); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index 6ff7fa5dda43..cc344f015ce2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -15,6 +15,22 @@ namespace ErrorCodes namespace DB::Cas { +namespace +{ + +namespace RefCkptWire +{ + constexpr WireKey life_epoch{"le"}; + constexpr WireKey committed_epoch{"cte"}; + constexpr WireKey committed_seq{"cts"}; + constexpr WireKey snapshot_epoch{"cse"}; + constexpr WireKey snapshot_seq{"css"}; + constexpr WireKey seal_epoch{"lse"}; + constexpr WireKey seal_seq{"lss"}; +} + +} + void checkRefCkptInvariants(const RefCkpt & ckpt, std::string_view what) { /// PRESENT means REAL. `life_epoch` may be absent (no writer of this object knew the namespace's @@ -90,15 +106,15 @@ String encodeRefCkpt(const RefCkpt & ckpt) /// three ref formats cannot disagree on the encoding. if (ckpt.life_epoch) { - writeKey(out, "le", first); + writeKey(out, RefCkptWire::life_epoch, first); writeU64StringValue(out, *ckpt.life_epoch); } if (ckpt.committed_through) - writeRefTxnIdFields(out, first, "cte", "cts", *ckpt.committed_through); + writeRefTxnIdFields(out, first, RefCkptWire::committed_epoch, RefCkptWire::committed_seq, *ckpt.committed_through); if (ckpt.checkpoint_snapshot_id) - writeRefTxnIdFields(out, first, "cse", "css", *ckpt.checkpoint_snapshot_id); + writeRefTxnIdFields(out, first, RefCkptWire::snapshot_epoch, RefCkptWire::snapshot_seq, *ckpt.checkpoint_snapshot_id); if (ckpt.last_epoch_seal) - writeRefTxnIdFields(out, first, "lse", "lss", *ckpt.last_epoch_seal); + writeRefTxnIdFields(out, first, RefCkptWire::seal_epoch, RefCkptWire::seal_seq, *ckpt.last_epoch_seal); closeObject(out, first); writeChar('\n', out); @@ -136,13 +152,13 @@ RefCkpt decodeRefCkpt(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "le") ckpt.life_epoch = r.readU64String(); - else if (key == "cte") cte = r.readU64String(); - else if (key == "cts") cts = r.readU64String(); - else if (key == "cse") cse = r.readU64String(); - else if (key == "css") css = r.readU64String(); - else if (key == "lse") lse = r.readU64String(); - else if (key == "lss") lss = r.readU64String(); + if (key == RefCkptWire::life_epoch) ckpt.life_epoch = r.readU64String(); + else if (key == RefCkptWire::committed_epoch) cte = r.readU64String(); + else if (key == RefCkptWire::committed_seq) cts = r.readU64String(); + else if (key == RefCkptWire::snapshot_epoch) cse = r.readU64String(); + else if (key == RefCkptWire::snapshot_seq) css = r.readU64String(); + else if (key == RefCkptWire::seal_epoch) lse = r.readU64String(); + else if (key == RefCkptWire::seal_seq) lss = r.readU64String(); else r.skipUnknown(key); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index be7ee5567575..c40fe9cfb7cc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -20,27 +21,31 @@ namespace DB::Cas namespace { -std::string_view opKindToWord(RefOpKind k) +namespace RefLogWire { - switch (k) - { - case RefOpKind::NamespaceBirth: return "namespace_birth"; - case RefOpKind::OwnerTransition: return "owner_transition"; - case RefOpKind::SetPublishedAt: return "set_published_at"; - case RefOpKind::RemoveNamespace: return "remove_namespace"; - case RefOpKind::EpochSeal: return "epoch_seal"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown op kind {}", static_cast(k)); + constexpr WireKey ns{"ns"}; + constexpr WireKey txn_epoch{"we"}; + constexpr WireKey txn_seq{"rs"}; + constexpr WireKey prev_epoch{"!pse"}; + constexpr WireKey prev_seq{"!pss"}; + constexpr WireKey op{"op"}; + constexpr WireKey ref{"rn"}; + constexpr WireKey published_ms{"ts"}; } +constexpr EnumWireTable kRefOpWords{{{ + {RefOpKind::NamespaceBirth, "namespace_birth"}, + {RefOpKind::OwnerTransition, "owner_transition"}, + {RefOpKind::SetPublishedAt, "set_published_at"}, + {RefOpKind::RemoveNamespace, "remove_namespace"}, + {RefOpKind::EpochSeal, "epoch_seal"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + RefOpKind opKindFromWord(std::string_view w) { - if (w == "namespace_birth") return RefOpKind::NamespaceBirth; - if (w == "owner_transition") return RefOpKind::OwnerTransition; - if (w == "set_published_at") return RefOpKind::SetPublishedAt; - if (w == "remove_namespace") return RefOpKind::RemoveNamespace; - if (w == "epoch_seal") return RefOpKind::EpochSeal; - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: unknown op kind '{}'", w); + return kRefOpWords.fromWord(w, "RefLogTxn"); } /// Byte budget over the encoded text. A removal-class transaction uses the larger complete-table @@ -69,22 +74,10 @@ void checkBudget(const std::vector & ops, size_t encoded_bytes) } } -void writeBindingFields(CasJsonWriter & out, bool & first, std::string_view prefix, const RefOwnerBinding & b) -{ - checkCanonicalRefName(b.ref_name, "RefLogTxn", "owner binding ref_name"); - checkManifestRef(b.manifest_ref, "RefLogTxn", "owner binding manifest_ref"); - out.key(prefix, "bk", first); - writeStringValue(out, refOwnerKindToWord(b.kind)); - out.key(prefix, "rn", first); - writeStringValue(out, b.ref_name); - writeManifestRefFields(out, first, prefix, b.manifest_ref); -} - void writeOp(CasJsonWriter & out, const RefOp & op) { bool first = true; - writeKey(out, "op", first); - writeStringValue(out, opKindToWord(op.kind)); + writeWordField(out, RefLogWire::op, refOpKindToWireWord(op.kind), first); switch (op.kind) { case RefOpKind::NamespaceBirth: @@ -93,57 +86,39 @@ void writeOp(CasJsonWriter & out, const RefOp & op) break; case RefOpKind::OwnerTransition: if (op.old_binding) - writeBindingFields(out, first, "o", *op.old_binding); + writeBindingFields(out, first, kOldBindingKeys, *op.old_binding); if (op.new_binding) - writeBindingFields(out, first, "n", *op.new_binding); + writeBindingFields(out, first, kNewBindingKeys, *op.new_binding); break; case RefOpKind::SetPublishedAt: checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); checkManifestRef(op.expected_manifest_ref, "RefLogTxn", "set_published_at manifest_ref"); - writeKey(out, "rn", first); - writeStringValue(out, op.ref_name); - writeManifestRefFields(out, first, "", op.expected_manifest_ref); - writeKey(out, "ts", first); - writeIntText(op.published_at_ms, out); + writeStringField(out, RefLogWire::ref, op.ref_name, first); + writeManifestRefFields(out, first, kBareManifestRefKeys, op.expected_manifest_ref); + writeNumberField(out, RefLogWire::published_ms, op.published_at_ms, first); break; } closeObject(out, first); writeChar('\n', out); } -/// Collector for a ManifestRef's three flat fields under an optional prefix. -struct ManifestFields -{ - std::optional me; - std::optional mb; - std::optional mo; - - bool any() const { return me || mb || mo; } - ManifestRef build(std::string_view what) const - { - if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} manifest_ref missing me/mb/mo", what); - return manifestRefFromFields(*me, *mb, *mo, "RefLogTxn", what); - } -}; - /// Collector for one binding (old/new) under a prefix. struct BindingFields { - std::optional bk; - std::optional rn; - ManifestFields mf; + std::optional kind; + std::optional ref; + ManifestRefFields manifest_fields; - bool any() const { return bk || rn || mf.any(); } + bool any() const { return kind || ref || manifest_fields.any(); } RefOwnerBinding build(std::string_view what) const { - if (!bk || !rn) + if (!kind || !ref) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing bk/rn", what); RefOwnerBinding b; - b.kind = refOwnerKindFromWord(*bk, "RefLogTxn owner binding"); - b.ref_name = *rn; + b.kind = refOwnerKindFromWord(*kind, "RefLogTxn owner binding"); + b.ref_name = *ref; checkCanonicalRefName(b.ref_name, "RefLogTxn", "owner binding ref_name"); - b.manifest_ref = mf.build(what); + b.manifest_ref = manifest_fields.buildRef("RefLogTxn", what); return b; } }; @@ -160,11 +135,10 @@ struct BindingFields void writeLogMeta(CasJsonWriter & out, const String & ns, const RefTxnId & txn_id, const std::optional & prev_epoch_seal) { bool first = true; - writeKey(out, "ns", first); - writeStringValue(out, ns); - writeRefTxnIdFields(out, first, "we", "rs", txn_id); + writeStringField(out, RefLogWire::ns, ns, first); + writeRefTxnIdFields(out, first, RefLogWire::txn_epoch, RefLogWire::txn_seq, txn_id); if (prev_epoch_seal) - writeRefTxnIdFields(out, first, "!pse", "!pss", *prev_epoch_seal); + writeRefTxnIdFields(out, first, RefLogWire::prev_epoch, RefLogWire::prev_seq, *prev_epoch_seal); closeObject(out, first); writeChar('\n', out); } @@ -176,7 +150,7 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) /// set_published_at fields std::optional sp_rn; - ManifestFields sp_mf; + ManifestRefFields sp_manifest_fields; std::optional sp_ts; /// owner_transition bindings BindingFields ob; @@ -185,21 +159,27 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) String key; while (r.nextKey(key)) { - if (key == "rn") sp_rn = r.readString(); - else if (key == "me") sp_mf.me = r.readU64String(); - else if (key == "mb") sp_mf.mb = r.readU64String(); - else if (key == "mo") sp_mf.mo = r.readU64Number(); - else if (key == "ts") sp_ts = r.readU64Number(); - else if (key == "obk") ob.bk = r.readString(); - else if (key == "orn") ob.rn = r.readString(); - else if (key == "ome") ob.mf.me = r.readU64String(); - else if (key == "omb") ob.mf.mb = r.readU64String(); - else if (key == "omo") ob.mf.mo = r.readU64Number(); - else if (key == "nbk") nb.bk = r.readString(); - else if (key == "nrn") nb.rn = r.readString(); - else if (key == "nme") nb.mf.me = r.readU64String(); - else if (key == "nmb") nb.mf.mb = r.readU64String(); - else if (key == "nmo") nb.mf.mo = r.readU64Number(); + if (key == RefLogWire::ref) + sp_rn = r.readString(); + else if (matchManifestRefFields(key, r, kBareManifestRefKeys, sp_manifest_fields)) + { + } + else if (key == RefLogWire::published_ms) + sp_ts = r.readU64Number(); + else if (key == kOldBindingKeys.kind) + ob.kind = r.readString(); + else if (key == kOldBindingKeys.ref) + ob.ref = r.readString(); + else if (matchManifestRefFields(key, r, kOldBindingKeys.manifest, ob.manifest_fields)) + { + } + else if (key == kNewBindingKeys.kind) + nb.kind = r.readString(); + else if (key == kNewBindingKeys.ref) + nb.ref = r.readString(); + else if (matchManifestRefFields(key, r, kNewBindingKeys.manifest, nb.manifest_fields)) + { + } else if (key == "pl") /// `"pl"` (payload) was removed from the op wire in stage-1 T12 (the `set_payload` op became /// `set_published_at`). The retired op WORD is already rejected by `opKindFromWord`, but this @@ -228,7 +208,7 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing rn/ts"); op.ref_name = *sp_rn; checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); - op.expected_manifest_ref = sp_mf.build("set_published_at manifest_ref"); + op.expected_manifest_ref = sp_manifest_fields.buildRef("RefLogTxn", "set_published_at"); op.published_at_ms = *sp_ts; break; } @@ -237,6 +217,11 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) } +std::string_view refOpKindToWireWord(RefOpKind kind) +{ + return kRefOpWords.toWord(kind, "RefLogTxn"); +} + bool refLogTxnIsEpochSeal(const RefLogTxn & txn) { return txn.ops.size() == 1 && txn.ops.front().kind == RefOpKind::EpochSeal; @@ -332,12 +317,27 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con String key; while (r.nextKey(key)) { - if (key == "ns") { txn.ns = r.readString(); saw_ns = true; } - else if (key == "we") { txn.txn_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { txn.txn_id.ref_sequence = r.readU64String(); saw_rs = true; } - else if (key == "!pse") pse = r.readU64String(); - else if (key == "!pss") pss = r.readU64String(); - else r.skipUnknown(key); + if (key == RefLogWire::ns) + { + txn.ns = r.readString(); + saw_ns = true; + } + else if (key == RefLogWire::txn_epoch) + { + txn.txn_id.writer_epoch = r.readU64String(); + saw_we = true; + } + else if (key == RefLogWire::txn_seq) + { + txn.txn_id.ref_sequence = r.readU64String(); + saw_rs = true; + } + else if (key == RefLogWire::prev_epoch) + pse = r.readU64String(); + else if (key == RefLogWire::prev_seq) + pss = r.readU64String(); + else + r.skipUnknown(key); } if (!saw_ns || !saw_we || !saw_rs) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing ns/we/rs"); @@ -385,7 +385,7 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con "RefLogTxn: trailer count {} != {} ops", n, txn.ops.size()); break; } - if (key != "op") + if (key != RefLogWire::op) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: record must start with \"op\""); const RefOpKind kind = opKindFromWord(r.readString()); txn.ops.push_back(readOpRecord(r, kind)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h index 34347fb97aaf..c61e11275e98 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h @@ -41,6 +41,10 @@ enum class RefOpKind : uint8_t EpochSeal = 5, }; +/// Convert a ref-log operation discriminator to its canonical wire word. Throws `LOGICAL_ERROR` if +/// `kind` is not represented by this format. +std::string_view refOpKindToWireWord(RefOpKind kind); + /// One operation inside a `RefLogTxn`. Only the fields documented next to `kind` are meaningful for /// that kind, and the codec never reads or writes the others. `OwnerTransition` optionally removes /// `old_binding` and/or installs `new_binding`; `SetPublishedAt` carries the expected manifest and the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index f31e9fed4ca2..8ed19c9a1ac3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -20,6 +20,20 @@ namespace DB::Cas namespace { +namespace RefSnapWire +{ + constexpr WireKey ns{"ns"}; + constexpr WireKey snapshot_epoch{"we"}; + constexpr WireKey snapshot_seq{"rs"}; + constexpr WireKey lifecycle{"lc"}; + constexpr WireKey kind{"k"}; + constexpr WireKey ref{"rn"}; + constexpr WireKey published_ms{"ts"}; +} + +constexpr std::string_view kCommittedTag = "c"; +constexpr std::string_view kPrecommitTag = "p"; + void checkCommittedSorted(const std::vector & rows) { for (size_t i = 1; i < rows.size(); ++i) @@ -59,13 +73,10 @@ void writeCommittedRow(CasJsonWriter & out, const RefCommittedRow & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "committed"); bool first = true; - writeKey(out, "k", first); - writeStringValue(out, "c"); - writeKey(out, "rn", first); - writeStringValue(out, row.ref_name); - writeManifestRefFields(out, first, "", row.manifest_ref); - writeKey(out, "ts", first); - writeIntText(row.published_at_ms, out); + writeWordField(out, RefSnapWire::kind, kCommittedTag, first); + writeStringField(out, RefSnapWire::ref, row.ref_name, first); + writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); + writeNumberField(out, RefSnapWire::published_ms, row.published_at_ms, first); closeObject(out, first); writeChar('\n', out); } @@ -79,11 +90,9 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "precommit"); bool first = true; - writeKey(out, "k", first); - writeStringValue(out, "p"); - writeKey(out, "rn", first); - writeStringValue(out, row.ref_name); - writeManifestRefFields(out, first, "", row.manifest_ref); + writeWordField(out, RefSnapWire::kind, kPrecommitTag, first); + writeStringField(out, RefSnapWire::ref, row.ref_name, first); + writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); closeObject(out, first); writeChar('\n', out); } @@ -95,33 +104,13 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) { bool first = true; - writeKey(out, "ns", first); - writeStringValue(out, snapshot.ns); - writeRefTxnIdFields(out, first, "we", "rs", snapshot.snapshot_id); - writeKey(out, "lc", first); - writeStringValue(out, "live"); + writeStringField(out, RefSnapWire::ns, snapshot.ns, first); + writeRefTxnIdFields(out, first, RefSnapWire::snapshot_epoch, RefSnapWire::snapshot_seq, snapshot.snapshot_id); + writeStringField(out, RefSnapWire::lifecycle, "live", first); closeObject(out, first); writeChar('\n', out); } -/// Collector for a ManifestRef's three flat fields (bare "me"/"mb"/"mo"). -struct ManifestFields -{ - std::optional me; - std::optional mb; - std::optional mo; - - /// Reconstruct a manifest reference after the tolerant reader has collected all three flat - /// fields. Missing fields are malformed input; `manifestRefFromFields` performs the remaining - /// range checks and reports the same corruption context as the row decoder. - ManifestRef build(std::string_view what) const - { - if (!me || !mb || !mo) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: {} manifest_ref missing me/mb/mo", what); - return manifestRefFromFields(*me, *mb, *mo, "RefTableSnapshot", what); - } -}; - } String encodeRefTableSnapshot(const RefTableSnapshot & snapshot) @@ -166,10 +155,10 @@ RefTableSnapshot decodeRefTableSnapshot( String key; while (r.nextKey(key)) { - if (key == "ns") { snapshot.ns = r.readString(); saw_ns = true; } - else if (key == "we") { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } - else if (key == "lc") + if (key == RefSnapWire::ns) { snapshot.ns = r.readString(); saw_ns = true; } + else if (key == RefSnapWire::snapshot_epoch) { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } + else if (key == RefSnapWire::snapshot_seq) { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } + else if (key == RefSnapWire::lifecycle) { const String lifecycle = r.readString(); if (lifecycle != "live") @@ -210,20 +199,20 @@ RefTableSnapshot decodeRefTableSnapshot( "RefTableSnapshot: trailer count {} != {} rows", n, snapshot.committed.size() + snapshot.precommits.size()); break; } - if (key != "k") + if (key != RefSnapWire::kind) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"k\""); const String k = r.readString(); std::optional rn; - ManifestFields mf; + ManifestRefFields mf; std::optional ts; while (r.nextKey(key)) { - if (key == "rn") rn = r.readString(); - else if (key == "me") mf.me = r.readU64String(); - else if (key == "mb") mf.mb = r.readU64String(); - else if (key == "mo") mf.mo = r.readU64Number(); - else if (key == "ts") ts = r.readU64Number(); + if (key == RefSnapWire::ref) rn = r.readString(); + else if (matchManifestRefFields(key, r, kBareManifestRefKeys, mf)) + { + } + else if (key == RefSnapWire::published_ms) ts = r.readU64Number(); else if (key == "pl") /// `"pl"` (payload) was removed from the row wire in stage-1 T12. It is a KNOWN-removed /// field, not a genuinely-unknown future one the tolerant reader may skip -- silently @@ -235,18 +224,18 @@ RefTableSnapshot decodeRefTableSnapshot( if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after record"); - if (k == "c") + if (k == kCommittedTag) { if (!rn || !ts) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing rn/ts"); RefCommittedRow row; row.ref_name = *rn; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); - row.manifest_ref = mf.build("committed"); + row.manifest_ref = mf.buildRef("RefTableSnapshot", "committed"); row.published_at_ms = *ts; snapshot.committed.push_back(std::move(row)); } - else if (k == "p") + else if (k == kPrecommitTag) { if (!rn) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing rn"); @@ -254,7 +243,7 @@ RefTableSnapshot decodeRefTableSnapshot( row.kind = RefOwnerKind::Precommit; row.ref_name = *rn; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); - row.manifest_ref = mf.build("precommit"); + row.manifest_ref = mf.buildRef("RefTableSnapshot", "precommit"); snapshot.precommits.push_back(std::move(row)); } else diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp index bf1a5445e4df..c184f99d4f4c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.cpp @@ -1,4 +1,6 @@ #include +#include +#include #include namespace DB @@ -12,21 +14,26 @@ namespace ErrorCodes namespace DB::Cas { +namespace +{ + +constexpr EnumWireTable kRefOwnerKindWords{{{ + {RefOwnerKind::Committed, "committed"}, + {RefOwnerKind::Precommit, "precommit"}, +}}}; + +static_assert(casEnumTableCoversEnum()); + +} + std::string_view refOwnerKindToWord(RefOwnerKind k) { - switch (k) - { - case RefOwnerKind::Committed: return "committed"; - case RefOwnerKind::Precommit: return "precommit"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref wire: unknown RefOwnerKind {}", static_cast(k)); + return kRefOwnerKindWords.toWord(k, "CAS ref wire: RefOwnerKind"); } RefOwnerKind refOwnerKindFromWord(std::string_view w, std::string_view what) { - if (w == "committed") return RefOwnerKind::Committed; - if (w == "precommit") return RefOwnerKind::Precommit; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown owner kind '{}'", what, w); + return kRefOwnerKindWords.fromWord(w, what); } void checkRefTxnIdNonzero(const RefTxnId & id, std::string_view format, std::string_view field) @@ -36,12 +43,19 @@ void checkRefTxnIdNonzero(const RefTxnId & id, std::string_view format, std::str "{}: {} fields must both be nonzero, got {}-{}", format, field, id.writer_epoch, id.ref_sequence); } -void writeRefTxnIdFields(CasJsonWriter & out, bool & first, std::string_view epoch_key, std::string_view seq_key, const RefTxnId & id) +void writeRefTxnIdFields(CasJsonWriter & out, bool & first, WireKey epoch_key, WireKey seq_key, const RefTxnId & id) +{ + writeU64StringField(out, epoch_key, id.writer_epoch, first); + writeU64StringField(out, seq_key, id.ref_sequence, first); +} + +void writeBindingFields(CasJsonWriter & out, bool & first, const BindingWireKeys & keys, const RefOwnerBinding & binding) { - writeKey(out, epoch_key, first); - writeU64StringValue(out, id.writer_epoch); - writeKey(out, seq_key, first); - writeU64StringValue(out, id.ref_sequence); + checkCanonicalRefName(binding.ref_name, "RefLogTxn", "owner binding ref_name"); + checkManifestRef(binding.manifest_ref, "RefLogTxn", "owner binding manifest_ref"); + writeWordField(out, keys.kind, refOwnerKindToWord(binding.kind), first); + writeStringField(out, keys.ref, binding.ref_name, first); + writeManifestRefFields(out, first, keys.manifest, binding.manifest_ref); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h index fdccef8f7fdb..e13fd116331e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h @@ -50,7 +50,11 @@ RefOwnerKind refOwnerKindFromWord(std::string_view w, std::string_view what); /// letting each format distinguish its primary id from any secondary id it embeds (for example, /// `cas_ref_log`'s `we`/`rs` versus its `prev_epoch_seal` pair) while sharing one writer so the /// formats can never disagree on the representation. -void writeRefTxnIdFields(CasJsonWriter & out, bool & first, std::string_view epoch_key, std::string_view seq_key, const RefTxnId & id); +void writeRefTxnIdFields(CasJsonWriter & out, bool & first, WireKey epoch_key, WireKey seq_key, const RefTxnId & id); + +/// Append one owner binding's flat fields named by `keys` to a ref-log `owner_transition` object. +/// The binding's ref name and manifest reference are validated before writing. +void writeBindingFields(CasJsonWriter & out, bool & first, const BindingWireKeys & keys, const RefOwnerBinding & binding); /// `RefTxnId`'s validity rule applied to ONE field of a decoded or about-to-be-encoded record: both /// components nonzero. `renderRefTxnId` refuses to build a key from anything else, so a half-zero id diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp index f523553271b4..68c26d47e0e6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp @@ -14,6 +14,31 @@ namespace ErrorCodes namespace DB::Cas { +namespace OwnerWire +{ + constexpr WireKey server_uuid{"su"}; + constexpr WireKey retired_at_ms{"rt"}; +} + +namespace ServerEpochWire +{ + constexpr WireKey next_writer_epoch{"nwe"}; +} + +namespace MountLeaseWire +{ + constexpr WireKey server_uuid{"su"}; + constexpr WireKey writer_epoch{"we"}; + constexpr WireKey hostname{"hn"}; + constexpr WireKey pid{"pid"}; + constexpr WireKey started_at_ms{"sat"}; + constexpr WireKey seq{"seq"}; + constexpr WireKey expires_at_ms{"eat"}; + constexpr WireKey min_active{"ma"}; + constexpr WireKey gc_fenced{"fen"}; + constexpr WireKey write_attempt_id{"write_attempt_id"}; +} + namespace { @@ -32,13 +57,9 @@ String encodeOwner(const OwnerObject & o) CasJsonWriter out(256); writeHeaderLine(out, FormatId::Owner); bool first = true; - writeKey(out, "su", first); - writeHex128Value(out, o.server_uuid); + writeHex128Field(out, OwnerWire::server_uuid, o.server_uuid, first); if (o.retired_at_ms) - { - writeKey(out, "rt", first); - writeIntText(*o.retired_at_ms, out); - } + writeNumberField(out, OwnerWire::retired_at_ms, *o.retired_at_ms, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -58,12 +79,12 @@ OwnerObject decodeOwner(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "su") + if (key == OwnerWire::server_uuid) { o.server_uuid = r.readHex128(); saw = true; } - else if (key == "rt") + else if (key == OwnerWire::retired_at_ms) rt = r.readU64Number(); else r.skipUnknown(key); @@ -81,8 +102,7 @@ String encodeServerEpoch(const ServerEpoch & e) CasJsonWriter out(256); writeHeaderLine(out, FormatId::ServerEpoch); bool first = true; - writeKey(out, "nwe", first); - writeU64StringValue(out, e.next_writer_epoch); + writeU64StringField(out, ServerEpochWire::next_writer_epoch, e.next_writer_epoch, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -101,7 +121,7 @@ ServerEpoch decodeServerEpoch(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "nwe") + if (key == ServerEpochWire::next_writer_epoch) { e.next_writer_epoch = r.readU64String(); saw = true; @@ -121,16 +141,16 @@ String encodeMountLease(const MountLease & m) CasJsonWriter out(256); writeHeaderLine(out, FormatId::MountLease); bool first = true; - writeKey(out, "su", first); writeHex128Value(out, m.server_uuid); - writeKey(out, "we", first); writeU64StringValue(out, m.writer_epoch); - writeKey(out, "hn", first); writeStringValue(out, m.hostname); - writeKey(out, "pid", first); writeIntText(m.pid, out); - writeKey(out, "sat", first); writeIntText(m.started_at_ms, out); - writeKey(out, "seq", first); writeU64StringValue(out, m.seq); - writeKey(out, "eat", first); writeIntText(m.expires_at_ms, out); - writeKey(out, "ma", first); writeU64StringValue(out, m.min_active); - writeKey(out, "fen", first); writeBoolValue(out, m.gc_fenced); - writeKey(out, "write_attempt_id", first); writeHex128Value(out, m.write_attempt_id); + writeHex128Field(out, MountLeaseWire::server_uuid, m.server_uuid, first); + writeU64StringField(out, MountLeaseWire::writer_epoch, m.writer_epoch, first); + writeStringField(out, MountLeaseWire::hostname, m.hostname, first); + writeNumberField(out, MountLeaseWire::pid, m.pid, first); + writeNumberField(out, MountLeaseWire::started_at_ms, m.started_at_ms, first); + writeU64StringField(out, MountLeaseWire::seq, m.seq, first); + writeNumberField(out, MountLeaseWire::expires_at_ms, m.expires_at_ms, first); + writeU64StringField(out, MountLeaseWire::min_active, m.min_active, first); + writeBoolField(out, MountLeaseWire::gc_fenced, m.gc_fenced, first); + writeHex128Field(out, MountLeaseWire::write_attempt_id, m.write_attempt_id, first); closeObject(out, first); writeChar('\n', out); return std::move(out).take(); @@ -151,29 +171,37 @@ MountLease decodeMountLease(std::string_view data) String key; while (r.nextKey(key)) { - if (key == "su") + if (key == MountLeaseWire::server_uuid) { m.server_uuid = r.readHex128(); saw_su = true; } - else if (key == "we") + else if (key == MountLeaseWire::writer_epoch) { m.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "hn") m.hostname = r.readString(); - else if (key == "pid") m.pid = r.readU64Number(); - else if (key == "sat") m.started_at_ms = r.readU64Number(); - else if (key == "seq") m.seq = r.readU64String(); - else if (key == "eat") m.expires_at_ms = r.readU64Number(); - else if (key == "ma") m.min_active = r.readU64String(); - else if (key == "fen") m.gc_fenced = r.readBool(); - else if (key == "write_attempt_id") + else if (key == MountLeaseWire::hostname) + m.hostname = r.readString(); + else if (key == MountLeaseWire::pid) + m.pid = r.readU64Number(); + else if (key == MountLeaseWire::started_at_ms) + m.started_at_ms = r.readU64Number(); + else if (key == MountLeaseWire::seq) + m.seq = r.readU64String(); + else if (key == MountLeaseWire::expires_at_ms) + m.expires_at_ms = r.readU64Number(); + else if (key == MountLeaseWire::min_active) + m.min_active = r.readU64String(); + else if (key == MountLeaseWire::gc_fenced) + m.gc_fenced = r.readBool(); + else if (key == MountLeaseWire::write_attempt_id) { m.write_attempt_id = r.readHex128(); saw_write_attempt_id = true; } - else r.skipUnknown(key); + else + r.skipUnknown(key); } if (!saw_su || !saw_we || !saw_write_attempt_id || m.write_attempt_id == UInt128{}) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing or zero identity field"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 9814d5d14811..8a53273956ee 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -192,8 +192,7 @@ UInt128 JsonObjectReader::readHex128() return guarded([&] { const String hex = readString(); - if (hex.size() != 32 - || std::any_of(hex.begin(), hex.end(), [](char c) { return unhex(c) == 0xff || (c >= 'A' && c <= 'F'); })) + if (hex.size() != 32 || std::any_of(hex.begin(), hex.end(), [](char c) { return !isLowercaseHexChar(c); })) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: expected 32 lowercase hex chars, got '{}'", what, hex); return unhexUInt(hex.data()); }); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 50edd33bd4ba..2ba6df5a1c5e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -59,18 +59,6 @@ class CasJsonWriter append("\":"); } - /// Same, for the prefixed key vocabulary ("o"/"n" + "me"/"mb"/"mo"/"bk"/"rn") — the - /// prefix and name are appended back to back, no composed temporary. - void key(std::string_view prefix, std::string_view name, bool & first) - { - appendChar(first ? '{' : ','); - first = false; - appendChar('"'); - append(prefix); - append(name); - append("\":"); - } - /// Quoted JSON string with full escaping (bulk-run scan). Defined in CasTextFormat.cpp. void stringValue(std::string_view s); @@ -154,6 +142,65 @@ inline void writeIntText(uint64_t v, CasJsonWriter & out) { out.u64Number(v); } void writeHeaderLine(CasJsonWriter & out, FormatId id); void writeTrailerLine(CasJsonWriter & out, uint64_t n); +/// A wire-key carrier. The explicit constructor keeps raw string literals out of writer call +/// sites: a codec passes its named constant, and an inline `WireKey{"..."}` is deliberately loud. +struct WireKey +{ + std::string_view text; + + explicit constexpr WireKey(std::string_view text_) : text(text_) {} + + friend constexpr bool operator==(std::string_view s, const WireKey & k) { return s == k.text; } +}; + +inline void writeKey(CasJsonWriter & out, WireKey key, bool & first) +{ + writeKey(out, key.text, first); +} + +inline void writeWordField(CasJsonWriter & out, WireKey key, std::string_view word, bool & first) +{ + writeKey(out, key, first); + writeStringValue(out, word); +} + +inline void writeStringField(CasJsonWriter & out, WireKey key, std::string_view value, bool & first) +{ + writeKey(out, key, first); + writeStringValue(out, value); +} + +inline void writeU64StringField(CasJsonWriter & out, WireKey key, uint64_t value, bool & first) +{ + writeKey(out, key, first); + writeU64StringValue(out, value); +} + +inline void writeNumberField(CasJsonWriter & out, WireKey key, uint64_t value, bool & first) +{ + writeKey(out, key, first); + out.u64Number(value); +} + +inline void writeHex128Field(CasJsonWriter & out, WireKey key, const UInt128 & value, bool & first) +{ + writeKey(out, key, first); + writeHex128Value(out, value); +} + +inline void writeBoolField(CasJsonWriter & out, WireKey key, bool value, bool & first) +{ + writeKey(out, key, first); + writeBoolValue(out, value); +} + +/// True iff `c` is one lowercase hexadecimal digit. Persisted CAS digests deliberately reject +/// uppercase spellings so each digest has one canonical textual representation. +constexpr bool isLowercaseHexChar(char c) +{ + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'); +} + /// Pull cursor over one canonical JSON object. /// /// The reader borrows the input buffer and records the object name for exception messages. It diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 31d44f260121..e44bd062c33d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -1,7 +1,10 @@ #include #include #include +#include #include +#include +#include namespace DB { @@ -14,73 +17,57 @@ namespace ErrorCodes namespace DB::Cas { +static_assert(casEnumTableCoversEnum()); +static_assert(casEnumTableCoversEnum()); + std::string_view tokenTypeToWord(TokenType t) { - switch (t) - { - case TokenType::ETag: return "etag"; - case TokenType::Generation: return "generation"; - case TokenType::Emulated: return "emulated"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS wire: unknown TokenType {}", static_cast(t)); + return kTokenTypeWords.toWord(t, "CAS wire: TokenType"); } TokenType tokenTypeFromWord(std::string_view w, std::string_view what) { - if (w == "etag") return TokenType::ETag; - if (w == "generation") return TokenType::Generation; - if (w == "emulated") return TokenType::Emulated; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown token type '{}'", what, w); + return kTokenTypeWords.fromWord(w, what); } BlobHashAlgo blobHashAlgoFromWord(std::string_view w, std::string_view what) { - if (w == "ch128") return BlobHashAlgo::CityHash128; - if (w == "xxh3") return BlobHashAlgo::XXH3_128; - if (w == "sha256") return BlobHashAlgo::Sha256; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown blob hash algo '{}'", what, w); + return kBlobHashAlgoWords.fromWord(w, what); } std::string_view objectKindToWord(ObjectKind k) { - switch (k) - { - case ObjectKind::Blob: return "blob"; - } - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS wire: unknown ObjectKind {}", static_cast(k)); + return kObjectKindWords.toWord(k, "CAS wire: ObjectKind"); } ObjectKind objectKindFromWord(std::string_view w, std::string_view what) { - if (w == "blob") return ObjectKind::Blob; - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown object kind '{}'", what, w); + return kObjectKindWords.fromWord(w, what); } void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t) { - writeKey(out, "tt", first); + writeKey(out, SharedWire::token_type, first); writeStringValue(out, tokenTypeToWord(t.type)); - writeKey(out, "tv", first); + writeKey(out, SharedWire::token, first); writeStringValue(out, t.value); } void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r) { - writeKey(out, "ha", first); + writeKey(out, SharedWire::algo, first); writeStringValue(out, blobHashAlgoName(r.algo)); - writeKey(out, "h", first); + writeKey(out, SharedWire::digest, first); writeStringValue(out, codecFor(r.algo).toHex(r.digest)); } -void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view prefix, const ManifestRef & r) +void writeManifestRefFields(CasJsonWriter & out, bool & first, const ManifestRefWireKeys & keys, const ManifestRef & r) { - /// Unlike the WriteBuffer overload, the two-part key() form appends the prefix and name back - /// to back with no composed String(prefix) + "..." temporary. - out.key(prefix, "me", first); + writeKey(out, keys.epoch, first); out.u64StringValue(r.writer_epoch); - out.key(prefix, "mb", first); + writeKey(out, keys.build, first); out.u64StringValue(r.build_sequence); - out.key(prefix, "mo", first); + writeKey(out, keys.ord, first); out.u64Number(r.manifest_ordinal); } @@ -100,4 +87,31 @@ ManifestRef manifestRefFromFields(uint64_t writer_epoch, uint64_t build_sequence return r; } +ManifestRef ManifestRefFields::buildRef(std::string_view what, std::string_view context) const +{ + if (!epoch || !build || !ord) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: {} manifest_ref missing epoch/build/ord", what, context); + return manifestRefFromFields(*epoch, *build, *ord, what, context); +} + +BlobRef BlobRefFields::build(std::string_view what) const +{ + if (!algo_word || !digest_hex) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: blob ref missing ha/h", what); + const BlobHashAlgo algo = blobHashAlgoFromWord(*algo_word, what); + /// Validate the digest width before calling `fromHex`. A width mismatch otherwise produces + /// `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed serialized input, + /// allowing an invalid record to escape the decoder's fail-closed error contract. + const uint64_t expected_hex_len = blobHashLenFor(algo) * 2; + if (digest_hex->size() != expected_hex_len) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS {}: digest hex width {} does not match algo width {}", what, digest_hex->size(), expected_hex_len); + /// Same fence, same reason as the width check above: a right-width but non-lowercase-hex digest + /// must also surface as CORRUPTED_DATA rather than `DigestCodec::fromHex`'s BAD_ARGUMENTS. Mirrors + /// `JsonObjectReader::readHex128`'s lowercase-hex predicate. + if (std::any_of(digest_hex->begin(), digest_hex->end(), [](char c) { return !isLowercaseHexChar(c); })) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: digest is not lowercase hex, got '{}'", what, *digest_hex); + return BlobRef{algo, codecFor(algo).fromHex(*digest_hex)}; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h index 8727e6e219b6..eea099e7711e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h @@ -1,9 +1,11 @@ #pragma once +#include #include #include #include #include #include +#include #include namespace DB::Cas @@ -15,6 +17,18 @@ namespace DB::Cas /// unrecognized value with `CORRUPTED_DATA`; silently choosing a default would turn malformed /// persisted data into a different valid-looking record. +/// The `TokenType` wire vocabulary; coverage is proven in `CasWireVocab.cpp`. +inline constexpr EnumWireTable kTokenTypeWords{{{ + {TokenType::ETag, "etag"}, + {TokenType::Generation, "generation"}, + {TokenType::Emulated, "emulated"}, +}}}; + +/// The `ObjectKind` wire vocabulary; coverage is proven in `CasWireVocab.cpp`. +inline constexpr EnumWireTable kObjectKindWords{{{ + {ObjectKind::Blob, "blob"}, +}}}; + /// Convert a token discriminator to its canonical wire word. Throws `CORRUPTED_DATA` if `t` is not /// one of the token types understood by this build. std::string_view tokenTypeToWord(TokenType t); @@ -44,12 +58,53 @@ void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t); /// lowercase digest are canonical, and the digest is rendered at the width required by `r.algo`. void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r); -/// Append the three flat `ManifestRef` fields `me`, `mb`, and `mo` to an in-progress JSON object. -/// `prefix` is prepended to each key, allowing the ref codecs to distinguish old and new owner -/// bindings (`ome`/`omb`/`omo` and `nme`/`nmb`/`nmo`) while part manifests and ordinary rows use an -/// empty prefix. The two unbounded `uint64_t` values are decimal JSON strings; the bounded ordinal -/// is a JSON number. All consumers use this exact spelling and representation. -void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view prefix, const ManifestRef & r); +/// The `ha`/`h` and `tt`/`tv` key spellings, named once so `writeBlobRefFields`/`writeTokenFields` +/// and the `match*Fields` collectors below can never drift apart on the literal. +namespace SharedWire +{ + inline constexpr WireKey algo{"ha"}; + inline constexpr WireKey digest{"h"}; + inline constexpr WireKey token_type{"tt"}; + inline constexpr WireKey token{"tv"}; +} + +/// One `ManifestRef`'s three flat key names. Every bundle spells the SAME wire representation +/// (two decimal-string `uint64_t`s and one JSON-number ordinal); only the key names vary per +/// binding role. Member names carry the semantic role the ref plays (`epoch`/`build`/`ord`); the +/// bundle constants below carry the CURRENT wire spelling for each role. +struct ManifestRefWireKeys +{ + WireKey epoch; + WireKey build; + WireKey ord; +}; + +/// The unprefixed `me`/`mb`/`mo` spelling used by part manifests, snapshot rows, and the +/// `set_published_at` ref-log op. +inline constexpr ManifestRefWireKeys kBareManifestRefKeys{WireKey{"me"}, WireKey{"mb"}, WireKey{"mo"}}; +/// The `ome`/`omb`/`omo` spelling for a ref-log owner_transition's OLD binding. +inline constexpr ManifestRefWireKeys kOldManifestRefKeys{WireKey{"ome"}, WireKey{"omb"}, WireKey{"omo"}}; +/// The `nme`/`nmb`/`nmo` spelling for a ref-log owner_transition's NEW binding. +inline constexpr ManifestRefWireKeys kNewManifestRefKeys{WireKey{"nme"}, WireKey{"nmb"}, WireKey{"nmo"}}; + +/// One owner binding's key names: the owner-kind word, the ref name, and its nested `ManifestRef` +/// bundle. Only the ref-log owner_transition op uses this bundle (old/new binding sides). +struct BindingWireKeys +{ + WireKey kind; + WireKey ref; + ManifestRefWireKeys manifest; +}; + +/// The `obk`/`orn`/`ome`/`omb`/`omo` spelling for the OLD binding side. +inline constexpr BindingWireKeys kOldBindingKeys{WireKey{"obk"}, WireKey{"orn"}, kOldManifestRefKeys}; +/// The `nbk`/`nrn`/`nme`/`nmb`/`nmo` spelling for the NEW binding side. +inline constexpr BindingWireKeys kNewBindingKeys{WireKey{"nbk"}, WireKey{"nrn"}, kNewManifestRefKeys}; + +/// Append the three flat `ManifestRef` fields named by `keys` to an in-progress JSON object. The +/// two unbounded `uint64_t` values are decimal JSON strings; the bounded ordinal is a JSON number. +/// All consumers use this exact representation; only the key spelling varies by `keys`. +void writeManifestRefFields(CasJsonWriter & out, bool & first, const ManifestRefWireKeys & keys, const ManifestRef & r); /// Construct a `ManifestRef` from decoded field values and validate the complete domain range: /// nonzero `writer_epoch` and `build_sequence`, and `manifest_ordinal` in @@ -59,4 +114,72 @@ void writeManifestRefFields(CasJsonWriter & out, bool & first, std::string_view ManifestRef manifestRefFromFields(uint64_t writer_epoch, uint64_t build_sequence, uint64_t manifest_ordinal, std::string_view caller, std::string_view what); +/// Collector for one `ManifestRef`'s three flat fields, filled in by repeated calls to +/// `matchManifestRefFields` as a tolerant reader walks an object's keys. `buildRef` checks that the +/// group is all-or-nothing complete, then delegates the completed group to `manifestRefFromFields`, +/// which performs the nonzero and range checks. +struct ManifestRefFields +{ + std::optional epoch; + std::optional build; + std::optional ord; + + bool any() const { return epoch || build || ord; } + + /// `what` names the codec (passed through to `manifestRefFromFields` as its `caller`); `context` + /// names the field being reconstructed (e.g. "descriptor", "committed"). Throws `CORRUPTED_DATA` + /// if the group is not all-or-nothing complete. + ManifestRef buildRef(std::string_view what, std::string_view context) const; +}; + +/// Collector for one `BlobRef`'s two flat fields (`ha`/`h`), filled in by `matchBlobRefFields`. +struct BlobRefFields +{ + std::optional algo_word; + std::optional digest_hex; + + /// Requires both fields, parses the algorithm word, and checks the digest hex width against the + /// algorithm's width BEFORE calling `fromHex` -- a width mismatch must surface as `CORRUPTED_DATA` + /// (malformed persisted input), not `DigestCodec::fromHex`'s `BAD_ARGUMENTS` (a caller-contract + /// violation). `what` identifies the field in the exception. + BlobRef build(std::string_view what) const; +}; + +/// Collector for one `Token`'s two flat fields (`tt`/`tv`), filled in by `matchTokenFields`. Phase 1 +/// deliberately has no `build`: callers keep their own local requiredness checks until the unified +/// both-required build is introduced. +struct TokenFields +{ + std::optional type_word; + std::optional value; +}; + +/// Each `match*Fields` helper tests `key` against the one or two field names it owns, consumes the +/// value on a match via `r`, and reports whether it recognized the key. None of them loop over an +/// object's keys or validate a completed group -- that is the caller's (tolerant-reader loop) and +/// the collector's `build`/`buildRef` job respectively. Defined inline: a decoder's per-key dispatch +/// is a hot path and must not gain a function-call boundary here. + +inline bool matchManifestRefFields(std::string_view key, JsonObjectReader & r, const ManifestRefWireKeys & keys, ManifestRefFields & fields) +{ + if (key == keys.epoch) { fields.epoch = r.readU64String(); return true; } + if (key == keys.build) { fields.build = r.readU64String(); return true; } + if (key == keys.ord) { fields.ord = r.readU64Number(); return true; } + return false; +} + +inline bool matchBlobRefFields(std::string_view key, JsonObjectReader & r, BlobRefFields & fields) +{ + if (key == SharedWire::algo) { fields.algo_word = r.readString(); return true; } + if (key == SharedWire::digest) { fields.digest_hex = r.readString(); return true; } + return false; +} + +inline bool matchTokenFields(std::string_view key, JsonObjectReader & r, TokenFields & fields) +{ + if (key == SharedWire::token_type) { fields.type_word = r.readString(); return true; } + if (key == SharedWire::token) { fields.value = r.readString(); return true; } + return false; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp index d018228750da..2700aec0291a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp @@ -40,11 +40,11 @@ const UInt128 kZeroSourceId{0}; /// Streams a shard's prior source-edge run at O(one block) resident memory: chains the run SEGMENTS the /// caller resolved from the parent seal (`blob_target_runs` filtered to one shard) and exposes a one-row /// lookahead for the fold merge. The prior run carries -/// BOTH surviving edges (`kEdgeActive`) AND the retired `kCondemned` sentinel rows at the zero source id, +/// BOTH surviving edges (`RunMarker::Edge`) AND the retired `RunMarker::Condemned` sentinel rows at the zero source id, /// so the cursor stops at edges AND at condemned rows (exposing the type via `rowType`), while zero-marker /// sentinels are dropped on carry (per-generation, never carried forward). Row/key invariants are enforced -/// while streaming: `kEdgeActive` never at `source_id = 0`; sentinel rows (`kZeroMarker` / -/// `kCondemned`) ONLY at `source_id = 0`; at most one sentinel per blob; an unknown value byte or an empty +/// while streaming: `RunMarker::Edge` never at `source_id = 0`; sentinel rows (`RunMarker::Zero` / +/// `RunMarker::Condemned`) ONLY at `source_id = 0`; at most one sentinel per blob; an unknown value byte or an empty /// payload is `CORRUPTED_DATA`. Resolution uses the exact object references supplied by the caller, so a run /// sealed for generation G that physically lives under an older generation's key is reached /// without key construction. An empty `segments` is the fresh-pool / empty baseline. The row stream is @@ -61,10 +61,10 @@ class PriorEdgeCursor bool valid() const { return has_current; } const String & key() const { return current_key; } - /// The value byte of the current row: `kEdgeActive` (a surviving edge) or `kCondemned` (a retired + /// The value byte of the current row: `RunMarker::Edge` (a surviving edge) or `RunMarker::Condemned` (a retired /// sentinel row). Zero markers are never surfaced (dropped on carry). - char rowType() const { return current_type; } - /// The decoded retired sentinel for the current row (only valid when `rowType() == kCondemned`). + RunMarker rowType() const { return current_type; } + /// The decoded retired sentinel for the current row (only valid when `rowType() == RunMarker::Condemned`). const CondemnedRow & condemnedRow() const { return current_condemned; } /// Advance to the next surviving edge OR retired sentinel, dropping zero markers, enforcing the @@ -87,40 +87,37 @@ class PriorEdgeCursor SourceEdgeKeyCodec::parse(k, bh, sid); if (p.empty()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: empty row payload"); - const char v = p[0]; + const RunMarker v = runMarkerFromByte(p[0], "CAS source-edge run"); const bool sentinel_key = (sid == kZeroSourceId); if (sentinel_key) { /// A sentinel key carries exactly one row per blob and never an edge. - if (v == kEdgeActive) + if (v == RunMarker::Edge) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: active edge at the reserved sentinel source_id 0"); - if (v != kZeroMarker && v != kCondemned) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS source-edge run: unknown sentinel row type 0x{:02x}", static_cast(v)); if (have_sentinel_blob && sentinel_blob == bh) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: duplicate sentinel row for one blob"); have_sentinel_blob = true; sentinel_blob = bh; - if (v == kZeroMarker) + if (v == RunMarker::Zero) continue; // A zero marker is per-generation and is dropped on carry. /// A retired sentinel: decode and surface it (settled at close-out, not an edge). current_condemned = decodeCondemnedRow(p); current_key = k; - current_type = kCondemned; + current_type = RunMarker::Condemned; has_current = true; return; } /// A non-sentinel key must carry a surviving edge and nothing else. - if (v != kEdgeActive) + if (v != RunMarker::Edge) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: sentinel row type 0x{:02x} at a non-sentinel key", static_cast(v)); current_key = k; - current_type = kEdgeActive; + current_type = RunMarker::Edge; has_current = true; return; } @@ -151,7 +148,7 @@ class PriorEdgeCursor size_t seg_idx = 0; std::optional reader; String current_key; - char current_type = kEdgeActive; + RunMarker current_type = RunMarker::Edge; CondemnedRow current_condemned; bool has_current = false; @@ -191,7 +188,7 @@ void assertValidSourceEdgeId(const UInt128 & source_id) String encodeCondemnedRow(const CondemnedRow & row) { String out; - out.push_back(kCondemned); + out.push_back(runMarkerByte(RunMarker::Condemned)); out.push_back(static_cast((row.delete_pending ? 1 : 0) | (row.marker_confirmed ? 2 : 0))); out.push_back(static_cast(row.token.type)); auto beU64 = [&](uint64_t v) { for (int i = 7; i >= 0; --i) out += static_cast((v >> (8 * i)) & 0xFF); }; @@ -209,7 +206,7 @@ CondemnedRow decodeCondemnedRow(std::string_view p) { /// [0]=0x02 [1]=flags [2]=token_type [3..10]=round [11..18]=size [19..20]=len [21..]=value constexpr size_t kFixed = 21; - if (p.size() < kFixed || p[0] != kCondemned) + if (p.size() < kFixed || runMarkerFromByte(p[0], "CAS condemned row") != RunMarker::Condemned) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS condemned row: malformed header"); CondemnedRow row; const uint8_t flags = static_cast(p[1]); @@ -248,19 +245,16 @@ bool SourceEdgeRunView::next(String & key, String & payload) key = SourceEdgeKeyCodec::key(rec.ref, rec.source_id); switch (rec.marker) { - case kEdgeActive: - case kZeroMarker: - payload = String(1, rec.marker); + case RunMarker::Edge: + case RunMarker::Zero: + payload = String(1, runMarkerByte(rec.marker)); break; - case kCondemned: + case RunMarker::Condemned: payload = encodeCondemnedRow(CondemnedRow{.delete_pending = rec.delete_pending, .token = rec.token, .size = rec.size, .condemn_round = rec.condemn_round, .marker_confirmed = rec.marker_confirmed}); break; - default: - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS source-edge run: unknown row marker 0x{:02x}", static_cast(rec.marker)); } return true; } @@ -394,7 +388,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); // sorted NDJSON; byte-deterministic for write-once adoption - // Streaming two-cursor merge over the prior run (surviving edges AND retired kCondemned + // Streaming two-cursor merge over the prior run (surviving edges AND retired RunMarker::Condemned // sentinel rows at the zero source id) and this round's edge deltas (by (blob_hash, source_id)). All // rows for one blob are adjacent in both inputs; the sentinel key (source_id 0) sorts first. We resolve // final presence per edge locally (idempotent: prior present + activate => present; any remove => @@ -572,23 +566,23 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, } } - /// Emit at most one sentinel row per blob: the `kCondemned` row when the + /// Emit at most one sentinel row per blob: the `RunMarker::Condemned` row when the /// blob is condemned/carried/graduated this pass (still_retired grew for it), else a per-generation - /// `kZeroMarker` when it transitioned to zero this pass but was not condemned (redelete-dropped or + /// `RunMarker::Zero` when it transitioned to zero this pass but was not condemned (redelete-dropped or /// absent-at-condemn). A blob with surviving edges (cur_edges > 0) emits neither — its edge rows /// were appended inline, and a condemned/zeroed blob has NO surviving edges, so appending the /// sentinel now (its key sorts first for the blob, and no edge rows precede it) keeps the run - /// sorted. `still_retired` therefore mirrors exactly the emitted `kCondemned` rows, in order. + /// sorted. `still_retired` therefore mirrors exactly the emitted `RunMarker::Condemned` rows, in order. if (rmr.still_retired.size() > retired_before) { const RetiredEntry & e = rmr.still_retired.back(); - writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = kCondemned, + writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = RunMarker::Condemned, .delete_pending = e.delete_pending, .token = e.token, .size = e.size, .condemn_round = e.condemn_round, .marker_confirmed = e.marker_confirmed}); } else if (cur_edges == 0 && cur_touched) - writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = kZeroMarker}); + writer.append(SourceEdgeRecord{.ref = cur_blob, .source_id = kZeroSourceId, .marker = RunMarker::Zero}); }; auto openBlobIfNeeded = [&](const BlobRef & b) { @@ -622,9 +616,9 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, openBlobIfNeeded(blob_ref); /// A retired sentinel row from the prior run: stash it for close-out settlement. It is not an edge - /// and NEVER a touch — a carried kCondemned row must not force a zero-marker or a peek_head HEAD + /// and NEVER a touch — a carried RunMarker::Condemned row must not force a zero-marker or a peek_head HEAD /// and never a touch. Deltas never key the zero source id, so no delta merges at this key. - if (from_prior && cursor.rowType() == kCondemned) + if (from_prior && cursor.rowType() == RunMarker::Condemned) { cur_condemned = cursor.condemnedRow(); cursor.advance(); @@ -673,7 +667,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, if (present) { - writer.append(SourceEdgeRecord{.ref = blob_ref, .source_id = source_id, .marker = kEdgeActive}); + writer.append(SourceEdgeRecord{.ref = blob_ref, .source_id = source_id, .marker = RunMarker::Edge}); ++cur_edges; } } @@ -689,7 +683,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, const String run_key = layout.blobTargetRunKey(new_generation, attempt, shard, 0); putDeterministicArtifact(backend, run_key, run_bytes); out_runs.push_back(RunRef{.key = run_key, .checksum = run_checksum, - .shard = shard, .generation = new_generation}); + .shard = shard, .key_generation = new_generation}); } std::vector zeroInDegree(Backend & backend, const std::vector & runs) @@ -700,12 +694,12 @@ std::vector zeroInDegree(Backend & backend, const std::vector redelete if d = 0 (the caller executes the exact-token @@ -342,9 +342,9 @@ struct GcRoundWorkBudget /// `confirm_condemned_marker` below): an unconfirmed /// entry is carried unchanged instead; /// d = 0 otherwise -> still_retired, carried byte-unchanged. -/// A carried `kCondemned` row is SETTLEMENT-ONLY: it never sets the blob's `cur_touched` bit, so a +/// A carried `RunMarker::Condemned` row is SETTLEMENT-ONLY: it never sets the blob's `cur_touched` bit, so a /// generation that only carries the row emits no zero-marker and pays no `peek_head` HEAD. The surviving -/// `still_retired` entries are re-emitted as `kCondemned` sentinel rows into the OUTPUT run (one sentinel +/// `still_retired` entries are re-emitted as `RunMarker::Condemned` sentinel rows into the OUTPUT run (one sentinel /// per blob, emitted before the blob's edges since the sentinel key sorts first), so the next generation /// reads them back — `still_retired` mirrors exactly those rows, in the same order. /// When the pass is clamped on any shard, landed-before-cut events may remain unfolded behind the clamp, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 358f619d5b34..f9c9ca6f72a5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -901,7 +901,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al } /// Retired-in-snapshot — there is NO separate retired-list object to publish anymore. The - /// round's surviving condemned entries were already sealed as `kCondemned` rows inside the fold's + /// round's surviving condemned entries were already sealed as `RunMarker::Condemned` rows inside the fold's /// `blob_target_runs` (durable before this CAS, via `putDeterministicArtifact`), and the per-shard /// `condemned_summary` the seal carries makes the next round's graduation/carry decisions zero-I/O. @@ -923,13 +923,13 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// current shard's run back at an older generation's key). Retention must never reclaim these. std::set referenced_generations; for (const RunRef & r : folded.fold_seal.blob_target_runs) - referenced_generations.insert(r.generation); + referenced_generations.insert(r.key_generation); /// ALSO protect every generation the PARENT (currently-adopted, pre-fold) seal references /// (`parent_seal_runs`, captured above): this prune runs BEFORE the round's own gc/state CAS below, so /// a losing leader must not destroy what the winning leader's already-adopted seal still points at — /// pre-CAS destructive actions may only rely on PREVIOUSLY PUBLISHED state (triage #5). for (const RunRef & r : parent_seal_runs) - referenced_generations.insert(r.generation); + referenced_generations.insert(r.key_generation); /// Retention floor uses THIS round's (post-fold) `generation`, so `gc_snapshot_generations_to_keep` /// keeps exactly that many generations back from the current one. If this round's `gc/state` CAS /// then LOSES, the prune reclaimed one generation deeper than the durably-adopted generation would @@ -980,7 +980,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al uint64_t objects_reclaimed = 0; std::set new_referenced_generations; for (const RunRef & r : folded.fold_seal.blob_target_runs) - new_referenced_generations.insert(r.generation); + new_referenced_generations.insert(r.key_generation); std::set handed_off; /// dedupe: multiple parent refs can share one generation /// GATED like every other destructive site, and it is also the FIRST destructive site of the @@ -1003,11 +1003,11 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al for (const RunRef & old_ref : handoff_candidates) { /// Only generations the wholesale prune already passed AND that no live ref still pins. - if (old_ref.generation > state.snap_pruned_through) + if (old_ref.key_generation > state.snap_pruned_through) continue; /// not yet pruned-through: the normal prune will reclaim it when it ages out - if (new_referenced_generations.contains(old_ref.generation)) + if (new_referenced_generations.contains(old_ref.key_generation)) continue; /// still referenced by a (possibly different-shard) live ref: keep it - if (!handed_off.insert(old_ref.generation).second) + if (!handed_off.insert(old_ref.key_generation).second) continue; /// already reclaimed this round via another shard's ref /// `bounded_remaining` draws from the hand-off's OWN reserve, never `UINT64_MAX` and never /// `pruneSupersededGenerations`' shared remainder: this hand-off is a ONE-SHOT event (see the @@ -1021,13 +1021,13 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al if (remaining == 0) break; const uint64_t reclaimed = deletePrefixWholesale( - backend, layout.gcGenPrefix(old_ref.generation), remaining); + backend, layout.gcGenPrefix(old_ref.key_generation), remaining); round_work_budget.handoff_prefix_wholesale_objects_used += reclaimed; objects_reclaimed += reclaimed; LOG_TRACE(logger, "CAS GC hand-off: generation {} moved out of the live seal below the retention cursor " "({} objects) — post-CAS wholesale reclaim (the prune had skipped it while referenced)", - old_ref.generation, reclaimed); + old_ref.key_generation, reclaimed); } t.metric("generations_reclaimed", handed_off.size()); t.metric("objects_reclaimed", objects_reclaimed); @@ -1665,7 +1665,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & result.fold_seal.ref_lives = walk_plan.successorFoldStates(); /// Retired-in-snapshot: the prior generation's condemned entries RIDE the source-edge run as - /// `kCondemned` sentinel rows, so the round no longer reads any separate retired-list object — + /// `RunMarker::Condemned` sentinel rows, so the round no longer reads any separate retired-list object — /// the parent seal's `blob_target_runs` ARE the retired input. The per-gc-shard `condemned_summary` /// the seal carries below is distilled from the `still_retired` rows each shard re-emits, making the /// next round's `graduationDue` / pure-carry decisions zero-I/O. @@ -2821,7 +2821,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// is deterministic (same refs for the same inputs), so seal determinism / crash-replay adoption hold. /// An empty delta with a NON-EMPTY retired list still runs the merge: settlement must happen every /// pass (carried/graduated/redeleted entries), and that pass reads the run to recompute in-degrees. - /// Distill one shard's `condemned_summary` entry from the `kCondemned` rows it re-emitted this pass + /// Distill one shard's `condemned_summary` entry from the `RunMarker::Condemned` rows it re-emitted this pass /// (`still_retired` mirrors those rows exactly). Folding shards call this; it makes the next /// round's `graduationDue` and pure-carry decisions read only the seal, never a run. auto summarize = [](const std::vector & still) -> CondemnedSummary @@ -3021,7 +3021,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & else { /// Either a real delta or a non-empty retired input: run the merge (empty deltas still settle - /// the kCondemned rows riding the parent run). The prior runs are the parent seal's shard-0 refs. + /// the RunMarker::Condemned rows riding the parent run). The prior runs are the parent seal's shard-0 refs. foldDeltasIntoGeneration(backend, layout, priorRunsFor(0), new_generation, attempt, /*shard*/0, std::move(deltas), result.fold_seal.blob_target_runs, @@ -3606,7 +3606,7 @@ bool Gc::graduationDue(const GcState & state, uint64_t current_round) { /// Retired-in-snapshot: the graduation signal is read from the adopted fold seal's per-shard /// `condemned_summary` — ZERO backend I/O beyond the single seal read. A summary distilled from this - /// generation's `kCondemned` rows says, per shard, how many entries are `delete_pending` (a graduation + /// generation's `RunMarker::Condemned` rows says, per shard, how many entries are `delete_pending` (a graduation /// is already published) and the oldest non-pending condemn round (one crosses the floor once /// `condemn_round < current_round`). if (state.snap_generation == 0) @@ -3980,7 +3980,7 @@ RebuildReport Gc::rebuildBaseline(bool force) std::vector attempt_of(gc_shards, 0); /// The fold is EDGE-ONLY here: a rebuild condemns nothing (spec §7, and the deletion below), so no /// condemn round is stamped and no head source is supplied. `current_round` 0 graduates nothing and - /// `condemn_round` 0 with an empty `head_blob` mints no `kCondemned` row -- this call is + /// `condemn_round` 0 with an empty `head_blob` mints no `RunMarker::Condemned` row -- this call is /// `foldDeltasIntoGeneration`'s pure edge form. auto flush_shard = [&](uint64_t shard) { @@ -4315,7 +4315,7 @@ std::vector Gc::previewDeletes() out.push_back(std::move(e)); } - /// Retired-in-snapshot: stream the SAME adopted seal runs and emit every `kCondemned` + /// Retired-in-snapshot: stream the SAME adopted seal runs and emit every `RunMarker::Condemned` /// sentinel row. The stored token IS the authority — NO HEAD here (a HEAD would defeat the point /// and cost I/O). `delete_pending` rows are deleted next fold; the rest await graduation. Preview /// stays WRITE-FREE (`openSourceEdgeRun` is a pure reader). Output is a superset of the above. @@ -4326,7 +4326,7 @@ std::vector Gc::previewDeletes() String payload; while (reader.next(key, payload)) { - if (payload.empty() || payload[0] != kCondemned) + if (payload.empty() || runMarkerFromByte(payload[0], "CAS source-edge run") != RunMarker::Condemned) continue; BlobRef ref; UInt128 source_id; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp index ae34e530bb1c..90a695349916 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.cpp @@ -1,20 +1,14 @@ #include +#include namespace DB::Cas { +static_assert(casEnumTableCoversEnum()); + std::string_view blobHashAlgoName(BlobHashAlgo algo) { - switch (algo) - { - case BlobHashAlgo::CityHash128: - return "ch128"; - case BlobHashAlgo::XXH3_128: - return "xxh3"; - case BlobHashAlgo::Sha256: - return "sha256"; - } - throw Exception(ErrorCodes::BAD_ARGUMENTS, "blobHashAlgoName: unknown BlobHashAlgo {}", static_cast(algo)); + return kBlobHashAlgoWords.toWord(algo, "blobHashAlgoName"); } uint64_t blobHashLenFor(BlobHashAlgo algo) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h index cc557fd753e1..54360403b8d7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -42,8 +43,16 @@ enum class BlobHashAlgo : uint8_t Sha256 = 3, }; +/// The `BlobHashAlgo` wire vocabulary (also the blob PATH SEGMENT, e.g. +/// `/blobs///`); coverage is proven in `CasBlobDigest.cpp`. +inline constexpr EnumWireTable kBlobHashAlgoWords{{{ + {BlobHashAlgo::CityHash128, "ch128"}, + {BlobHashAlgo::XXH3_128, "xxh3"}, + {BlobHashAlgo::Sha256, "sha256"}, +}}}; + /// The blob PATH SEGMENT for `algo`, e.g. `/blobs///`: `"ch128"` | `"xxh3"` | -/// `"sha256"`. Throws `BAD_ARGUMENTS` for an out-of-range enum value. +/// `"sha256"`. Throws `LOGICAL_ERROR` for an out-of-range enum value. std::string_view blobHashAlgoName(BlobHashAlgo algo); /// Returns the digest byte width for `algo`: 16 for `CityHash128` and `XXH3_128`, or 32 for diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h new file mode 100644 index 000000000000..31d76eee0bf8 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTable.h @@ -0,0 +1,73 @@ +#pragma once + +#include +#include + +#include +#include + +namespace DB::ErrorCodes +{ + extern const int CORRUPTED_DATA; + extern const int LOGICAL_ERROR; +} + +namespace DB::Cas +{ + +/// One persisted enum <-> wire-word vocabulary: the single carrier the encoder, the decoder, the +/// introspection renderer, and the tests all read. Every persisted enum is dense, so `toWord` is a +/// direct indexed lookup; `fromWord` is a linear pass over a handful of words. Coverage is proven +/// at each table's definition site by `casEnumTableCoversEnum` (CasEnumWireTableAsserts.h — .cpp +/// and tests only) together with the `denseAndOrdered`/`wordsUnique` predicates below. +template +struct EnumWireTable +{ + struct Entry + { + Enum value; + std::string_view word; + }; + + std::array entries; + + /// An empty table would make `denseAndOrdered`/`wordsUnique` vacuously true and `toWord`'s + /// index arithmetic read past the array — no wire vocabulary is empty, so reject at compile time. + static_assert(N > 0, "EnumWireTable must hold at least one entry"); + + constexpr bool denseAndOrdered() const + { + for (size_t i = 0; i < N; ++i) + if (static_cast(entries[i].value) != static_cast(entries[0].value) + i) + return false; + return true; + } + + constexpr bool wordsUnique() const + { + for (size_t i = 0; i < N; ++i) + for (size_t j = i + 1; j < N; ++j) + if (entries[i].word == entries[j].word) + return false; + return true; + } + + std::string_view toWord(Enum value, std::string_view what) const + { + const uint64_t index = static_cast(value) - static_cast(entries.front().value); + if (index >= entries.size() || entries[index].value != value) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "{}: value {} is outside the wire vocabulary", what, static_cast(value)); + return entries[index].word; + } + + Enum fromWord(std::string_view word, std::string_view what) const + { + for (const auto & entry : entries) + if (entry.word == word) + return entry.value; + throw Exception(ErrorCodes::CORRUPTED_DATA, "{}: unknown word '{}'", what, word); + } +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h new file mode 100644 index 000000000000..e59e13fcef6b --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h @@ -0,0 +1,37 @@ +#pragma once + +/// Compile-time coverage proof for EnumWireTable: SET EQUALITY with the enum's declared values. +/// Size-plus-uniqueness is not enough (an invalid casted value satisfies both while an enumerator +/// goes missing). This header pulls in magic_enum and therefore MUST be included only from .cpp +/// files and tests, never from another header. + +#include + +#include + +namespace DB::Cas +{ + +template +consteval bool casEnumTableCoversEnum() +{ + /// One assert per table carries all three obligations: a table author cannot forget density + /// or word uniqueness, because coverage subsumes them. + if (!Table.denseAndOrdered() || !Table.wordsUnique()) + return false; + constexpr auto declared = magic_enum::enum_values(); + if (declared.size() != Table.entries.size()) + return false; + for (size_t i = 0; i < declared.size(); ++i) + { + bool found = false; + for (const auto & entry : Table.entries) + if (entry.value == declared[i]) + found = true; + if (!found) + return false; + } + return true; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp index 7848ae591f27..33af5b0ec1e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp @@ -800,7 +800,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// The NON-SENTINEL source edges the snapshot still holds on each unreferenced blob, collected in /// `detail` mode only. `in_run_hashes` alone answers "does GC still see this blob at all"; the /// stale-edge cross-check below needs the edge IDENTITIES so it can ask whether their source - /// manifests still exist. Sentinel rows (`source_id == 0` — `kZeroMarker`/`kCondemned`) are not + /// manifests still exist. Sentinel rows (`source_id == 0` — `RunMarker::Zero`/`RunMarker::Condemned`) are not /// edges and are excluded. std::unordered_map, BlobRefHash> unref_edge_sources; bool have_gc_state = false; @@ -821,8 +821,8 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// The adopted fold seal names the snapshot runs; resolution is by ref, never by key /// construction. Every row whose hash is in our candidate set marks "known to GC" — /// edges still counted (drop unfolded), an explicit zero-marker mid-pipeline, or a - /// `kCondemned` sentinel row that carries the condemned state (retired-in-snapshot): - /// the `kCondemned` rows feed `retired_by_hash` (the `PendingGc` classification) in the + /// `RunMarker::Condemned` sentinel row that carries the condemned state (retired-in-snapshot): + /// the `RunMarker::Condemned` rows feed `retired_by_hash` (the `PendingGc` classification) in the /// SAME pass, replacing the removed `retired_refs`/`decodeRetiredSet` loop. /// /// These sets are keyed by the full `BlobRef`, not a narrowed digest. The run's own @@ -852,7 +852,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co in_run_hashes.insert(ref); if (detail && source_id != UInt128{0}) unref_edge_sources[ref].push_back(source_id); - if (!payload.empty() && payload[0] == kCondemned) + if (!payload.empty() && runMarkerFromByte(payload[0], "CAS source-edge run") == RunMarker::Condemned) { const CondemnedRow row = decodeCondemnedRow(payload); RetiredEntry e; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index 1428be282f30..4793eadd4e12 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -8,7 +9,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -126,20 +129,10 @@ String renderRefTxnIdObj(const RefTxnId & id) .str(); } -String refOwnerKindName(RefOwnerKind k) -{ - switch (k) - { - case RefOwnerKind::Committed: return "Committed"; - case RefOwnerKind::Precommit: return "Precommit"; - } - return "Unknown"; -} - String renderRefOwnerBinding(const RefOwnerBinding & b) { return JsonObj() - .add("kind", jsonEscape(refOwnerKindName(b.kind))) + .add("kind", jsonEscape(refOwnerKindToWord(b.kind))) .add("ref_name", jsonEscape(b.ref_name)) .add("manifest_ref", renderManifestRef(b.manifest_ref)) .str(); @@ -196,23 +189,10 @@ String renderRefCkpt(const RootNamespace & ns, const RefCkpt & c) .str(); } -String refOpKindName(RefOpKind k) -{ - switch (k) - { - case RefOpKind::NamespaceBirth: return "NamespaceBirth"; - case RefOpKind::OwnerTransition: return "OwnerTransition"; - case RefOpKind::SetPublishedAt: return "SetPublishedAt"; - case RefOpKind::RemoveNamespace: return "RemoveNamespace"; - case RefOpKind::EpochSeal: return "EpochSeal"; - } - return "Unknown"; -} - String renderRefOp(const RefOp & op) { return JsonObj() - .add("kind", jsonEscape(refOpKindName(op.kind))) + .add("kind", jsonEscape(refOpKindToWireWord(op.kind))) .add("old_binding", op.old_binding ? renderRefOwnerBinding(*op.old_binding) : "null") .add("new_binding", op.new_binding ? renderRefOwnerBinding(*op.new_binding) : "null") .add("ref_name", jsonEscape(op.ref_name)) @@ -237,16 +217,6 @@ String renderRefLogTxn(const RefLogTxn & t) .str(); } -String placementName(EntryPlacement p) -{ - switch (p) - { - case EntryPlacement::Inline: return "Inline"; - case EntryPlacement::Blob: return "Blob"; - } - return "Unknown"; -} - /// `inline_bytes` renders as its LENGTH only, not its content — an inline file's bytes are payload /// data, not part-manifest identity, and may be arbitrarily large / non-UTF8. String renderManifestEntry(const ManifestEntry & e) @@ -256,7 +226,7 @@ String renderManifestEntry(const ManifestEntry & e) /// digest widths, and each entry's own `ref.algo` determines its width. return JsonObj() .add("path", jsonEscape(e.path)) - .add("placement", jsonEscape(placementName(e.placement))) + .add("placement", jsonEscape(entryPlacementToWireWord(e.placement))) .add("blob", jsonEscape(blobIdOf(e.ref))) .add("blob_size", jsonUInt(e.blob_size)) .add("inline_bytes_size", jsonUInt(e.inline_bytes.size())) @@ -315,43 +285,23 @@ String renderGcState(const GcState & s) .str(); } -String tokenTypeName(TokenType t) -{ - switch (t) - { - case TokenType::ETag: return "ETag"; - case TokenType::Generation: return "Generation"; - case TokenType::Emulated: return "Emulated"; - } - return "Unknown"; -} - /// `Token::value` is an opaque backend-native string (e.g. an S3 ETag) — NOT a 128-bit hash — so it /// renders verbatim (escaped), not hex-converted; `type` names which backend family minted it. String renderToken(const Token & t) { return JsonObj() .add("value", jsonEscape(t.value)) - .add("type", jsonEscape(tokenTypeName(t.type))) + .add("type", jsonEscape(tokenTypeToWord(t.type))) .str(); } -String objectKindName(ObjectKind k) -{ - switch (k) - { - case ObjectKind::Blob: return "Blob"; - } - return "Unknown"; -} - String renderRunRef(const RunRef & r) { return JsonObj() .add("key", jsonEscape(r.key)) .add("checksum", jsonHex(r.checksum)) .add("shard", jsonUInt(r.shard)) - .add("generation", jsonUInt(r.generation)) + .add("generation", jsonUInt(r.key_generation)) .str(); } @@ -379,7 +329,7 @@ String renderFoldSeal(const CasFoldSeal & seal) for (const auto & r : seal.blob_target_runs) blob_target_runs.push_back(renderRunRef(r)); - /// A fold seal carries per-GC-shard totals for `kCondemned` rows in its source runs. Render the + /// A fold seal carries per-GC-shard totals for `RunMarker::Condemned` rows in its source runs. Render the /// summary from the seal itself; the older separate retired-reference object is no longer part /// of the current layout. JsonObj condemned_summary; @@ -399,40 +349,16 @@ String renderFoldSeal(const CasFoldSeal & seal) .str(); } -String provenanceOpName(ProvenanceOp op) -{ - switch (op) - { - case ProvenanceOp::Other: return "Other"; - case ProvenanceOp::Insert: return "Insert"; - case ProvenanceOp::Merge: return "Merge"; - case ProvenanceOp::Mutation: return "Mutation"; - case ProvenanceOp::Attach: return "Attach"; - case ProvenanceOp::Repack: return "Repack"; - } - return "Unknown"; -} - String renderProvenance(const Provenance & p) { return JsonObj() .add("created_at_ms", jsonUInt(p.created_at_ms)) .add("creator_server_id", jsonHex(p.creator_server_id)) .add("ch_version", jsonUInt(p.ch_version)) - .add("op", jsonEscape(provenanceOpName(p.op))) + .add("op", jsonEscape(provenanceOpToWireWord(p.op))) .str(); } -String metaStateName(MetaState s) -{ - switch (s) - { - case MetaState::Clean: return "clean"; - case MetaState::Condemned: return "condemned"; - } - return "unknown"; -} - /// The per-hash `.meta` descriptor is the blob body's sibling and records its freshness state /// (`Clean` or `Condemned`), not its payload. It is rendered separately from `renderEnvelopeHeader`: /// the body remains an enveloped object, while the descriptor has its own format. @@ -441,7 +367,7 @@ String renderBlobMeta(const BlobMeta & m) return JsonObj() .add("object", jsonEscape("blob_meta")) .add("version", jsonUInt(m.version)) - .add("state", jsonEscape(metaStateName(m.state))) + .add("state", jsonEscape(metaStateToWireWord(m.state))) .add("condemn_round", jsonUInt(m.condemn_round)) .add("size", jsonUInt(m.size)) .str(); @@ -450,7 +376,7 @@ String renderBlobMeta(const BlobMeta & m) String renderEnvelopeHeader(const EnvelopeHeader & h) { return JsonObj() - .add("kind", jsonEscape(objectKindName(h.kind))) + .add("kind", jsonEscape(objectKindToWord(h.kind))) /// The blob identity is carried by the object key, so the envelope keeps only the provenance /// fields needed for forensics (`ch` and `bld`) together with its compatibility version. .add("compatibility_version", jsonUInt(h.compatibility_version)) @@ -463,17 +389,11 @@ String renderEnvelopeHeader(const EnvelopeHeader & h) } /// The word vocabulary a row's marker byte renders as, matching the `cas_run` NDJSON's own `"m"` field -/// words (`CasRecordStreamFormat.cpp`'s private `markerToWord`) so cas-inspect speaks the same vocabulary +/// words (`runMarkerToWireWord`) so cas-inspect speaks the same vocabulary /// as the on-disk format rather than inventing a second one. -String sourceEdgeRowKindName(char marker) +String sourceEdgeRowKindName(RunMarker marker) { - switch (marker) - { - case kEdgeActive: return "edge"; - case kZeroMarker: return "zero"; - case kCondemned: return "condemned"; - default: return "unknown"; - } + return String(runMarkerToWireWord(marker)); } String renderCondemnedRow(const CondemnedRow & r) @@ -514,7 +434,7 @@ String renderBlobTargetRun(const ParsedBlobTargetRunKey & parsed, std::string_vi if (payload.empty()) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "cas-inspect: source-edge run row for blob {} has an empty payload", blobIdOf(ref)); - const char marker = payload[0]; + const RunMarker marker = runMarkerFromByte(payload[0], "cas-inspect: source-edge run row"); distinct_blobs.insert(ref); JsonObj row; @@ -527,20 +447,16 @@ String renderBlobTargetRun(const ParsedBlobTargetRunKey & parsed, std::string_vi switch (marker) { - case kEdgeActive: + case RunMarker::Edge: ++edge_count; break; - case kZeroMarker: + case RunMarker::Zero: ++zero_marker_count; break; - case kCondemned: + case RunMarker::Condemned: ++condemned_count; row.add("condemned", renderCondemnedRow(decodeCondemnedRow(payload))); // CORRUPTED_DATA on malformed (fail-closed) break; - default: - throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, - "cas-inspect: source-edge run row for blob {} has an unknown marker 0x{:02x}", - blobIdOf(ref), static_cast(marker)); } rows.push_back(row.str()); } diff --git a/src/Disks/tests/cas_format_test_battery.h b/src/Disks/tests/cas_format_test_battery.h index 173bfcc5c4df..361947968f3c 100644 --- a/src/Disks/tests/cas_format_test_battery.h +++ b/src/Disks/tests/cas_format_test_battery.h @@ -4,6 +4,7 @@ #include #include #include +#include namespace DB::ErrorCodes { @@ -53,6 +54,23 @@ void expectCode(int code, F && f, const String & context) } } +namespace DB::Cas::tests +{ +inline std::set & batteryCoveredIds() +{ + static std::set ids; + return ids; +} + +struct BatteryCoverageRegistrar +{ + explicit BatteryCoverageRegistrar(FormatId id) { batteryCoveredIds().insert(id); } +}; +} + +#define CAS_BATTERY_COVERS(format_id) \ + static const DB::Cas::tests::BatteryCoverageRegistrar battery_covers_##format_id{DB::Cas::FormatId::format_id} + inline void runFormatBattery(const FormatBatteryCase & c) { using namespace DB::Cas; diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index e7b5f6221c18..441ea58fc558 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -475,9 +475,9 @@ inline String encodeMinimalGcState(uint64_t round) /// Inject condemned bookkeeping + gc/state directly (bypassing a real GC round) so a test can seed the /// GC ledger's condemned state at an arbitrary round. Retired-in-snapshot: the condemned entries are -/// seeded the way a real round leaves them — as `kCondemned` sentinel rows inside an adopted fold seal's +/// seeded the way a real round leaves them — as `RunMarker::Condemned` sentinel rows inside an adopted fold seal's /// shard run (there is no separate retired-list object). A synthetic +edge/-edge pair nets each blob to -/// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `kCondemned` row. +/// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `RunMarker::Condemned` row. /// Also sets {round} on gc/state. Entries carry a `condemn_round` (default 0 → uses `round`); callers /// pass fresh (non-pending) condemns. An empty `entries` set just advances {round}. inline void injectRetire( @@ -621,7 +621,7 @@ inline bool runRoundsUntilAbsent( /// The CURRENT condemned entries for `shard`, read from the adopted fold seal's `blob_target_runs` /// (retired-in-snapshot T4): the round no longer writes a separate retired-list object — condemned -/// entries RIDE the source-edge run as `kCondemned` sentinel rows at the zero-sentinel key. This reads +/// entries RIDE the source-edge run as `RunMarker::Condemned` sentinel rows at the zero-sentinel key. This reads /// the seal at (snap_generation, snap_attempt), opens every run for `shard`, and reconstructs the /// `RetiredEntry` shape (hash from the run key, the rest from the decoded `CondemnedRow`). Empty when /// gc/state / the seal / the runs are absent. Used by ack-floor tests to assert pending/condemn state. @@ -649,7 +649,7 @@ inline std::vector currentRetiredSet( String p; while (r.next(k, p)) { - if (p.empty() || p[0] != DB::Cas::kCondemned) + if (p.empty() || DB::Cas::runMarkerFromByte(p[0], "CAS test source-edge run") != DB::Cas::RunMarker::Condemned) continue; DB::Cas::BlobRef ref; DB::UInt128 source_id{}; @@ -668,7 +668,7 @@ inline std::vector currentRetiredSet( return out; } -/// True iff ANY gc-shard's adopted-seal run still holds a `kCondemned` row — the ack-floor deletion +/// True iff ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row — the ack-floor deletion /// pipeline is in flight while this is true (retired-in-snapshot T4 replacement for the old /// "iterate gc/state.retired_refs" probe). `gc_shards` is read from gc/state when 0 is passed. inline bool anyCondemnedInSeal( @@ -861,7 +861,7 @@ inline std::vector runsForShard( } } -/// Stream the sealed in-degree run segments `runs` and count the active source edges (`kEdgeActive` +/// Stream the sealed in-degree run segments `runs` and count the active source edges (`RunMarker::Edge` /// rows) for `ref`. Test-side replacement for the deleted per-blob point query `inDegreeInGeneration` /// (codecs-v3 phase 5: a `cas_run` is a sequential NDJSON stream with no random access, so a blob's /// in-degree is recomputed by a full stream-and-count rather than a seek). A condemned / zero-marker @@ -877,7 +877,7 @@ inline int64_t inDegreeInRuns( String p; while (r.next(k, p)) { - if (p.empty() || p[0] != DB::Cas::kEdgeActive) + if (p.empty() || DB::Cas::runMarkerFromByte(p[0], "CAS test source-edge run") != DB::Cas::RunMarker::Edge) continue; DB::Cas::BlobRef row_ref; DB::UInt128 source_id{}; diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 1119a5be0a64..495ed57f0325 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -154,6 +154,8 @@ TEST(CASBlobEnvelopeFormat, RefEscaperAlphabetPinned) << "escaper alphabet drifted: '/' must be verbatim, quote/backslash escaped, control -> \\uXXXX"; } +CAS_BATTERY_COVERS(Blob); + TEST(CASFormatBattery, BlobEnvelope) { /// The golden is CONSTRUCTED from the hand-pinned json literal (same one FixedLengthAndPadZone diff --git a/src/Disks/tests/gtest_cas_blob_indegree.cpp b/src/Disks/tests/gtest_cas_blob_indegree.cpp index e47b7351f063..a340fde248ac 100644 --- a/src/Disks/tests/gtest_cas_blob_indegree.cpp +++ b/src/Disks/tests/gtest_cas_blob_indegree.cpp @@ -144,27 +144,27 @@ TEST(CASBlobInDegree, FoldDeltaDivergentBytesThrowsCorrupted) /// ==== two-cursor settlement merge (retired-in-snapshot T3, spec §2.1/§3) ==== /// -/// The retired input is no longer a separate `prior_retired` vector — the prior generation's `kCondemned` +/// The retired input is no longer a separate `prior_retired` vector — the prior generation's `RunMarker::Condemned` /// rows RIDE the source-edge run at the zero-sentinel key. These helpers build such a prior run directly /// (via the sorted-NDJSON `SourceEdgeRunWriter`, codecs-v3 phase 5) and decode a run for assertions. namespace { -/// A `kCondemned` sentinel record for `h` at the zero source_id, carrying the condemned incarnation. +/// A `RunMarker::Condemned` sentinel record for `h` at the zero source_id, carrying the condemned incarnation. SourceEdgeRecord condemnedRec(UInt128 h, const CondemnedRow & row) { return SourceEdgeRecord{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h)}, - .source_id = UInt128{0}, .marker = kCondemned, + .source_id = UInt128{0}, .marker = RunMarker::Condemned, .delete_pending = row.delete_pending, .token = row.token, .size = row.size, .condemn_round = row.condemn_round}; } -/// An active-edge record (`kEdgeActive`) for `h` at source `sid`. +/// An active-edge record (`RunMarker::Edge`) for `h` at source `sid`. SourceEdgeRecord edgeRec(UInt128 h, UInt128 sid) { return SourceEdgeRecord{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h)}, - .source_id = sid, .marker = kEdgeActive}; + .source_id = sid, .marker = RunMarker::Edge}; } /// head_blob / peek_head stub: present with a fixed token/size. @@ -189,7 +189,7 @@ CondemnedRow condemnedRowFor(uint64_t condemn_round, const String & tok = "t", .size = size, .condemn_round = condemn_round}; } -/// Build a source-edge run (`kSourceEdgeKeySchema128`) carrying the given `kCondemned` sentinel rows +/// Build a source-edge run (`kSourceEdgeKeySchema128`) carrying the given `RunMarker::Condemned` sentinel rows /// and surviving edges, write it under `blobTargetRunKey(gen, attempt, shard, 0)`, and return its /// `RunRef`. Rows are emitted in (blob_hash, source_id) order (sentinels at source_id 0 sort first /// per blob). @@ -224,7 +224,7 @@ RunRef writeSourceEdgeRun(InMemoryBackend & backend, const Layout & layout, const String bytes = out.str(); const String key = layout.blobTargetRunKey(gen, attempt, shard, 0); backend.putIfAbsent(key, bytes); - return RunRef{.key = key, .checksum = sourceEdgeRunChecksum(bytes), .shard = shard, .generation = gen}; + return RunRef{.key = key, .checksum = sourceEdgeRunChecksum(bytes), .shard = shard, .key_generation = gen}; } struct DecodedRun @@ -251,11 +251,11 @@ DecodedRun decodeRun(InMemoryBackend & backend, const RunRef & run) EXPECT_FALSE(p.empty()); if (p.empty()) continue; - if (p[0] == kCondemned) + if (runMarkerFromByte(p[0], "CAS test source-edge run") == RunMarker::Condemned) d.condemned.emplace_back(bh, decodeCondemnedRow(p)); - else if (p[0] == kZeroMarker) + else if (runMarkerFromByte(p[0], "CAS test source-edge run") == RunMarker::Zero) d.zero_markers.push_back(bh); - else if (p[0] == kEdgeActive) + else if (runMarkerFromByte(p[0], "CAS test source-edge run") == RunMarker::Edge) d.edges.emplace_back(bh, sid); else ADD_FAILURE() << "unknown run row type"; @@ -304,7 +304,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) InMemoryBackend backend; Layout layout{"pool"}; - /// Gen 1's run holds one unrelated surviving edge (b9) plus the carried kCondemned rows for A=b1 + /// Gen 1's run holds one unrelated surviving edge (b9) plus the carried RunMarker::Condemned rows for A=b1 /// (condemned round 2) and B=b2 (round 3); neither A nor B has any edge (in-degree 0 by definition). /// current_round = 3: strictly-below graduates, at-the-current-round stays. const RunRef gen1 = writeSourceEdgeRun(backend, layout, /*gen*/1, 0, 0, @@ -329,7 +329,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) EXPECT_TRUE(rmr.spared.empty()); EXPECT_TRUE(rmr.redelete.empty()); - /// still_retired mirrors exactly the kCondemned rows written into the output run, in order. + /// still_retired mirrors exactly the RunMarker::Condemned rows written into the output run, in order. const DecodedRun out = decodeRun(backend, runs2[0]); ASSERT_EQ(out.condemned.size(), 2u); EXPECT_EQ(out.condemned[0].first, b(1)); @@ -415,7 +415,7 @@ TEST(CASThreeCursorMerge, NewCandidateCondemned) EXPECT_TRUE(rmr.graduated.empty()); EXPECT_TRUE(rmr.spared.empty()); - /// The fresh condemn is emitted as a kCondemned row (not a zero marker) into the output run. + /// The fresh condemn is emitted as a RunMarker::Condemned row (not a zero marker) into the output run. const DecodedRun out = decodeRun(backend, runs2[0]); ASSERT_EQ(out.condemned.size(), 1u); EXPECT_EQ(out.condemned[0].first, b(3)); @@ -451,7 +451,7 @@ TEST(CASThreeCursorMerge, AbsentBlobNotCondemned) TEST(CASThreeCursorMerge, SnapshotEdgesUnperturbedByRetired) { - /// Retired-in-snapshot changes the byte-invariant: the retired machinery now WRITES kCondemned + /// Retired-in-snapshot changes the byte-invariant: the retired machinery now WRITES RunMarker::Condemned /// sentinel rows into the run, so a retired-engaged run is no longer byte-identical to a plain one. /// The preserved invariant (spec §2.1) is narrower: the retired machinery touches ONLY the sentinel /// namespace — the surviving EDGE rows are byte-identical to a plain fold of the same deltas. @@ -485,7 +485,7 @@ TEST(CASThreeCursorMerge, SnapshotEdgesUnperturbedByRetired) TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) { - /// Gen 1 condemns b (a real +edge/-edge net-to-zero with head_blob present) -> a kCondemned row. Gen 2 + /// Gen 1 condemns b (a real +edge/-edge net-to-zero with head_blob present) -> a RunMarker::Condemned row. Gen 2 /// has NO deltas at all: the carried row must (a) survive byte-identically, (b) emit no zero marker, /// (c) never call peek_head (a carried sentinel is not a touch). InMemoryBackend backend; @@ -502,7 +502,7 @@ TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) const DecodedRun g1 = decodeRun(backend, runs1[0]); ASSERT_EQ(g1.condemned.size(), 1u); EXPECT_EQ(g1.condemned[0].first, b(2)); - EXPECT_TRUE(g1.zero_markers.empty()); /// a condemned blob emits kCondemned, never a zero marker + EXPECT_TRUE(g1.zero_markers.empty()); /// a condemned blob emits RunMarker::Condemned, never a zero marker } /// Gen 2: empty deltas, current_round 1 (< 5 => b carries, does not graduate). peek_head must NOT fire. @@ -541,7 +541,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) out.finalize(); const String bytes = out.str(); const RunRef bad{.key = layout.blobTargetRunKey(1, 0, 0, 0), - .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .generation = 1}; + .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .key_generation = 1}; backend.putIfAbsent(bad.key, bytes); std::vector runs2; @@ -561,7 +561,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) out.finalize(); const String bytes = out.str(); const RunRef bad{.key = layout.blobTargetRunKey(1, 0, 0, 0), - .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .generation = 1}; + .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .key_generation = 1}; backend.putIfAbsent(bad.key, bytes); std::vector runs2; @@ -698,7 +698,7 @@ TEST(CASBlobInDegree, ZeroInDegreeStreamsBlockBounded) EXPECT_LE(backend.getCount(gen2_run_key), 2u); } -/// ==== kCondemned row codec + typed source-edge open (retired-in-snapshot T2, spec §2.1) ==== +/// ==== RunMarker::Condemned row codec + typed source-edge open (retired-in-snapshot T2, spec §2.1) ==== TEST(CASCondemnedRow, RoundTripAllTokenTypes) { @@ -711,11 +711,47 @@ TEST(CASCondemnedRow, RoundTripAllTokenTypes) row.size = 4096; row.condemn_round = 7; const auto bytes = DB::Cas::encodeCondemnedRow(row); - ASSERT_EQ(bytes[0], DB::Cas::kCondemned); + ASSERT_EQ(bytes[0], DB::Cas::runMarkerByte(DB::Cas::RunMarker::Condemned)); EXPECT_EQ(DB::Cas::decodeCondemnedRow(bytes), row); } } +TEST(CASCondemnedRow, UnknownMarkerByteFailsClosedWithCorruptedData) +{ + /// This pins the condemned-row decoder's own marker validation. + DB::Cas::CondemnedRow row; + row.token = DB::Cas::Token{.value = "t", .type = DB::Cas::TokenType::ETag}; + auto bytes = DB::Cas::encodeCondemnedRow(row); + bytes[0] = 0x03; + + try + { + static_cast(DB::Cas::decodeCondemnedRow(bytes)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + } +} + +TEST(CASRecordStream, RunMarkerByteContractFailsClosed) +{ + /// This helper is defense-in-depth; upstream word validation means no input path reaches it. + for (const auto marker : {DB::Cas::RunMarker::Zero, DB::Cas::RunMarker::Edge, DB::Cas::RunMarker::Condemned}) + EXPECT_EQ(DB::Cas::runMarkerFromByte(DB::Cas::runMarkerByte(marker), "CAS test"), marker); + + try + { + static_cast(DB::Cas::runMarkerFromByte(0x03, "CAS test")); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + } +} + TEST(CASCondemnedRow, UnknownFlagBitsFailClosed) { DB::Cas::CondemnedRow row; diff --git a/src/Disks/tests/gtest_cas_blob_meta_format.cpp b/src/Disks/tests/gtest_cas_blob_meta_format.cpp index bb055d2cef71..59d1f7205dc4 100644 --- a/src/Disks/tests/gtest_cas_blob_meta_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta_format.cpp @@ -6,6 +6,29 @@ using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } +namespace +{ +/// Same tiny inline copy as `gtest_cas_wire_vocab.cpp`'s `expectThrowsCode`: stays clear of +/// `Disks/tests/cas_test_helpers.h`'s `DB::Cas::tests::expectThrowsCode`, which would both drag +/// in the whole CAS backend/store machinery this file otherwise has no need for AND collide (same +/// namespace, same name and signature) if that header were ever included here too. +template +void expectThrowsCode(int expected_code, F && fn) +{ + try + { + fn(); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code); + } +} +} + +CAS_BATTERY_COVERS(BlobMeta); + TEST(CASFormatBattery, BlobMeta) { BlobMeta m; @@ -40,10 +63,10 @@ TEST(CASBlobMetaFormat, FailsClosedOnUnknownStateAndTruncation) /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes /// the header gate, which is the point — the BODY is what has to fail here. const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"st\":\"zombie\",\"cr\":\"0\",\"sz\":\"0\"}\n"; - EXPECT_THROW(decodeBlobMeta(bad_state), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(bad_state); }); /// Missing state key -> CORRUPTED_DATA. const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"cr\":\"0\",\"sz\":\"0\"}\n"; - EXPECT_THROW(decodeBlobMeta(no_state), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(no_state); }); /// Truncated (header only) -> CORRUPTED_DATA. - EXPECT_THROW(decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"); }); } diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index f4c3a913fda7..01f0b99a465a 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -91,9 +91,20 @@ TEST(CASEncodingPins, SourceEdgeRunLines) SourceEdgeRecord active; active.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(2))}; active.source_id = UInt128(5); - active.marker = kEdgeActive; + active.marker = RunMarker::Edge; writer.append(active); + SourceEdgeRecord condemned; + condemned.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(3))}; + condemned.source_id = UInt128(0); + condemned.marker = RunMarker::Condemned; + condemned.delete_pending = true; + condemned.token = Token{.value = "token", .type = TokenType::ETag}; + condemned.size = 9; + condemned.condemn_round = 7; + condemned.marker_confirmed = true; + writer.append(condemned); + writer.finish(); out.finalize(); @@ -103,8 +114,10 @@ TEST(CASEncodingPins, SourceEdgeRunLines) const String header = fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()); const String expected_record = "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; - const String trailer = "{\"n\":1}\n"; - /// There is exactly one record, so the whole buffer must be byte-identical to header + record + trailer. - const String expected_full = header + expected_record + trailer; + const String expected_condemned = + "{\"b\":\"0100000000000000000000000000000003\",\"s\":\"00000000000000000000000000000000\",\"m\":\"condemned\",\"pend\":true,\"tt\":\"etag\",\"tv\":\"token\",\"sz\":9,\"cr\":\"7\",\"mc\":true}\n"; + const String trailer = "{\"n\":2}\n"; + /// Both records must remain byte-identical to their canonical stored representation. + const String expected_full = header + expected_record + expected_condemned + trailer; EXPECT_EQ(text, expected_full) << text; } diff --git a/src/Disks/tests/gtest_cas_enum_wire_table.cpp b/src/Disks/tests/gtest_cas_enum_wire_table.cpp new file mode 100644 index 000000000000..90f59c866cb2 --- /dev/null +++ b/src/Disks/tests/gtest_cas_enum_wire_table.cpp @@ -0,0 +1,131 @@ +#include +#include +#include +#include +#include + +using namespace DB::Cas; + +namespace +{ + +enum class Fruit : uint8_t +{ + Apple = 0, + Pear = 1, + Plum = 2, +}; + +constexpr EnumWireTable fruits{{{ + {Fruit::Apple, "apple"}, + {Fruit::Pear, "pear"}, + {Fruit::Plum, "plum"}, +}}}; + +static_assert(fruits.denseAndOrdered()); +static_assert(fruits.wordsUnique()); +static_assert(casEnumTableCoversEnum()); + +/// A one-based dense enum exercises the index arithmetic from the first entry's value. +enum class Grade : uint8_t +{ + Low = 1, + Mid = 2, + High = 3, +}; + +constexpr EnumWireTable grades{{{ + {Grade::Low, "low"}, + {Grade::Mid, "mid"}, + {Grade::High, "high"}, +}}}; + +static_assert(grades.denseAndOrdered()); +static_assert(casEnumTableCoversEnum()); + +} + +TEST(CASEnumWireTable, RoundTripsEveryEntryBothWays) +{ + for (const auto & e : fruits.entries) + { + EXPECT_EQ(fruits.toWord(e.value, "fruits"), e.word); + EXPECT_EQ(fruits.fromWord(e.word, "fruits"), e.value); + } + for (const auto & e : grades.entries) + EXPECT_EQ(grades.fromWord(grades.toWord(e.value, "grades"), "grades"), e.value); +} + +TEST(CASEnumWireTable, FromWordFailsClosed) +{ + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { fruits.fromWord("banana", "fruits"); }); +} + +/// `LOGICAL_ERROR` aborts the process in debug/sanitizer builds (`handle_error_code`), so the +/// defensive toWord branch needs the death-test split this test directory already uses (see +/// `gtest_cas_gc_state_format.cpp`'s `RejectsZeroGcShardsOnEncode` pair) — a bare EXPECT_THROW +/// would SIGABRT the whole gate binary on those lanes. +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASEnumWireTableDeathTest, ToWordAbortsOnOutOfRangeValue) +{ + EXPECT_DEATH(fruits.toWord(static_cast(99), "fruits"), "outside the wire vocabulary"); +} +#else +TEST(CASEnumWireTable, ToWordThrowsLogicalErrorOnOutOfRangeValue) +{ + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, + [&] { fruits.toWord(static_cast(99), "fruits"); }); +} +#endif + +/// The compile-time proofs must also be exercised in the direction where they can fail — a +/// predicate rewritten to `return true;` must break this file. All three are constexpr, so the +/// negative cases are plain static_asserts over deliberately bad tables: +namespace bad_tables +{ + +enum class Sparse : uint8_t { A = 0, B = 2 }; +constexpr EnumWireTable sparse{{{{Sparse::A, "a"}, {Sparse::B, "b"}}}}; +static_assert(!sparse.denseAndOrdered()); +/// ...and through the folded coverage proof, so deleting its density disjunct breaks the file +/// (this table is set-equal and word-unique — only density rejects it): +static_assert(!casEnumTableCoversEnum()); + +constexpr EnumWireTable dup_words{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "apple"}, {Fruit::Plum, "plum"}}}}; +static_assert(!dup_words.wordsUnique()); +/// ...and through the folded proof (dense, right-sized, set-equal — only word uniqueness rejects +/// it), so deleting the uniqueness disjunct breaks the file: +static_assert(!casEnumTableCoversEnum()); + +constexpr EnumWireTable dup_value{{{ + {Fruit::Apple, "apple"}, {Fruit::Apple, "pear"}, {Fruit::Plum, "plum"}}}}; +static_assert(!casEnumTableCoversEnum()); + +constexpr EnumWireTable invalid_value{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}, {static_cast(99), "plum"}}}}; +static_assert(!casEnumTableCoversEnum()); + +/// The two cases above fail the folded density check before the set-equality core runs, so the +/// core needs its own failing witnesses — both dense and word-unique, so they reach it. +/// Reaches the size comparison: one enumerator short. +constexpr EnumWireTable missing_enumerator{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}}}}; +static_assert(!casEnumTableCoversEnum()); + +/// Reaches the declared-values scan: right size, dense from Pear, an out-of-enum value present +/// and `Apple` missing — the asserts header's own motivating scenario. +constexpr EnumWireTable enumerator_missing{{{ + {Fruit::Pear, "pear"}, {Fruit::Plum, "plum"}, {static_cast(3), "quince"}}}}; +static_assert(!casEnumTableCoversEnum()); + +/// Guards the size comparison itself: every declared value present PLUS one out-of-enum entry — +/// the only miscoverage the declared-values scan cannot see (an enumerator deleted from the enum +/// while its table row survived). +constexpr EnumWireTable extra_entry{{{ + {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}, {Fruit::Plum, "plum"}, + {static_cast(3), "quince"}}}}; +static_assert(!casEnumTableCoversEnum()); + +} diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index ae65ed27274b..808830e46a9f 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -554,7 +554,7 @@ String publishOneBlobPart(const PoolPtr & s, const String & ns, const String & r /// Whether the CURRENT retired list (any gc-shard) still holds an entry (ack-floor pipeline in flight). bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_fold_seal_format.cpp b/src/Disks/tests/gtest_cas_fold_seal_format.cpp index afe6331a4ed8..d5e2144093eb 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_format.cpp @@ -29,13 +29,15 @@ void eraseRequiredField(String & encoded, std::string_view field) } } +CAS_BATTERY_COVERS(FoldSeal); + TEST(CASFormatBattery, FoldSeal) { CasFoldSeal seal; seal.generation = 5; seal.parent_generation = 4; seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{7, 11}}; - seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(0x0f), .shard = 0, .generation = 5}); + seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(0x0f), .shard = 0, .key_generation = 5}); seal.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 4}; runFormatBattery({FormatId::FoldSeal, @@ -72,8 +74,8 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRejectsTwoBlobTargetRunsForOneShard) seal.generation = 7; seal.parent_generation = 6; seal.blob_target_runs = { - RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .generation = 7}, - RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .key_generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .key_generation = 7}, }; seal.condemned_summary[0] = CondemnedSummary{}; @@ -88,8 +90,8 @@ TEST(CASFoldSealFormatDeathTest, ProducerValidationRejectsMalformedSealBeforePut const Layout layout("p"); CasFoldSeal seal; seal.blob_target_runs = { - RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .generation = 7}, - RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .generation = 7}}; + RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .key_generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .key_generation = 7}}; seal.condemned_summary[0] = CondemnedSummary{}; EXPECT_DEATH({ validateFoldSealForWrite(seal, layout, 1); }, "duplicate blob-target shard"); } @@ -99,8 +101,8 @@ TEST(CASFoldSealFormat, ProducerValidationRejectsMalformedSealBeforePut) const Layout layout("p"); CasFoldSeal seal; seal.blob_target_runs = { - RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .generation = 7}, - RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .generation = 7}}; + RunRef{.key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, .key_generation = 7}, + RunRef{.key = layout.blobTargetRunKey(7, 2, 0, 0), .checksum = UInt128{2}, .shard = 0, .key_generation = 7}}; seal.condemned_summary[0] = CondemnedSummary{}; cas_battery_detail::expectCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { validateFoldSealForWrite(seal, layout, 1); }, "duplicate blob-target shard"); @@ -117,7 +119,7 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel .key = layout.blobTargetRunKey(7, 1, 0, 0), .checksum = UInt128{1}, .shard = 0, - .generation = 7}); + .key_generation = 7}); seal.condemned_summary[0] = CondemnedSummary{}; const String valid = encodeFoldSeal(seal); @@ -161,7 +163,7 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRejectsNoncanonicalRowsAndIncompleteS .key = layout.blobTargetRunKey(7, 1, 1, 0), .checksum = UInt128{1}, .shard = 1, - .generation = 7}); + .key_generation = 7}); seal.condemned_summary[0] = CondemnedSummary{}; cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, @@ -254,7 +256,7 @@ TEST(CASFoldSeal, FoldSealCondemnedSummaryRoundTrips) s.parent_generation = 8; s.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2}; s.blob_target_runs.push_back(RunRef{.key = "gc/gen/9/blob_target/0/0", .checksum = UInt128(0x77), - .shard = 0, .generation = 9}); + .shard = 0, .key_generation = 9}); s.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 5}; s.condemned_summary[1] = CondemnedSummary{}; /// explicit zero entry (totality over gc_shards) diff --git a/src/Disks/tests/gtest_cas_format_battery.cpp b/src/Disks/tests/gtest_cas_format_battery.cpp index f6ba9bcdbcd0..d6e9f01b3535 100644 --- a/src/Disks/tests/gtest_cas_format_battery.cpp +++ b/src/Disks/tests/gtest_cas_format_battery.cpp @@ -4,9 +4,28 @@ using namespace DB::Cas; +namespace +{ +template +void expectThrowsCode(int expected_code, F && fn) +{ + try + { + fn(); + FAIL() << "expected exception " << expected_code; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code) << e.message(); + } +} +} + /// The real cas_pool_meta case replaces the phase-1 toy proving instance. Every other control-plane /// format registers its own battery row in its own gtest_cas__format.cpp file (Tasks 3-6). +CAS_BATTERY_COVERS(PoolMeta); + TEST(CASFormatBattery, PoolMeta) { PoolMeta pm; @@ -21,3 +40,9 @@ TEST(CASFormatBattery, PoolMeta) .golden = currentFormatHeader("cas_pool_meta") + "{\"pid\":\"00112233445566778899aabbccddeeff\",\"hln\":256,\"gcs\":1,\"mrg\":3,\"alg\":\"ch128\"}\n"}); } + +TEST(CASPoolMeta, ValidateAlgosUsedRejectsUnknownByte) +{ + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [] { validatePoolAlgosUsed({7}, DB::ErrorCodes::CORRUPTED_DATA, "t"); }); +} diff --git a/src/Disks/tests/gtest_cas_gc_attempt.cpp b/src/Disks/tests/gtest_cas_gc_attempt.cpp index b919298ae24e..49e63031c6df 100644 --- a/src/Disks/tests/gtest_cas_gc_attempt.cpp +++ b/src/Disks/tests/gtest_cas_gc_attempt.cpp @@ -54,7 +54,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_fold.cpp b/src/Disks/tests/gtest_cas_gc_fold.cpp index c764aa0ac0b7..c3af2cbe4ef0 100644 --- a/src/Disks/tests/gtest_cas_gc_fold.cpp +++ b/src/Disks/tests/gtest_cas_gc_fold.cpp @@ -249,7 +249,7 @@ TEST(CASGCFold, EmptyDeltaShardCarriesParentRunRef) EXPECT_EQ(carried.key, parent_ref.key) << "carried ref points at the PARENT generation's run key"; EXPECT_EQ(carried.checksum, parent_ref.checksum); EXPECT_EQ(carried.shard, 0u); - EXPECT_EQ(carried.generation, st1.snap_generation) + EXPECT_EQ(carried.key_generation, st1.snap_generation) << "the carried ref names the generation whose key namespace physically holds the object"; } @@ -314,7 +314,7 @@ TEST(CASGCFold, PreviewResolvesCarriedRef) const auto seal2 = decodeFoldSeal( backend->get(store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes); ASSERT_EQ(seal2.blob_target_runs.size(), 1u); - ASSERT_EQ(seal2.blob_target_runs.front().generation, st1.snap_generation) + ASSERT_EQ(seal2.blob_target_runs.front().key_generation, st1.snap_generation) << "the current seal's ref physically lives at the parent generation (carried, not reconstructed)"; // The preview resolves the carried ref (a gen-1 physical key) and computes in-degree 1 => blob 1 is diff --git a/src/Disks/tests/gtest_cas_gc_leak.cpp b/src/Disks/tests/gtest_cas_gc_leak.cpp index a65b225b8f9a..36a77237e9dc 100644 --- a/src/Disks/tests/gtest_cas_gc_leak.cpp +++ b/src/Disks/tests/gtest_cas_gc_leak.cpp @@ -47,7 +47,7 @@ PoolPtr openTestPool(std::shared_ptr & out_backend) /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 158040be38c9..81a1dd46c2d3 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -1,4 +1,5 @@ #include "cas_test_helpers.h" +#include "cas_format_test_battery.h" #include #include #include @@ -24,6 +25,17 @@ class FailingMaintenanceReadBackend : public InMemoryBackend }; } +CAS_BATTERY_COVERS(GcMaintenanceState); + +TEST(CASFormatBattery, GcMaintenanceState) +{ + GcMaintenanceState state{.janitor_cursor = "cas/ns/a"}; + runFormatBattery({FormatId::GcMaintenanceState, + [&] { return sealObject(FormatId::GcMaintenanceState, encodeGcMaintenanceState(state)); }, + [](std::string_view s) { decodeGcMaintenanceState(std::string(openObject(FormatId::GcMaintenanceState, s))); }, + currentFormatHeader("cas_gc_maintenance_state") + "{\"cur\":\"cas/ns/a\"}\n"}); +} + TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) { EXPECT_EQ(static_cast(FormatId::GcMaintenanceState), 25); diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index e6530f131299..e03aa26dbaad 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -27,6 +27,8 @@ void expectThrowsCode(int expected_code, F && fn) } +CAS_BATTERY_COVERS(GcOutcomes); + TEST(CASFormatBattery, GcOutcomes) { OutcomeLog log; @@ -75,6 +77,42 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) EXPECT_EQ(encodeOutcomeLog(d), text); } +TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) +{ + OutcomeLog log; + log.entries.push_back({ObjectKind::Blob, + BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}, + Token{"e-1", TokenType::ETag}, OutcomeKind::Deleted}); + const String bytes = encodeOutcomeLog(log); + + const String token_value = R"(,"tv":"e-1")"; + const auto token_value_pos = bytes.find(token_value); + ASSERT_NE(token_value_pos, String::npos); + String missing_token_value = bytes; + missing_token_value.erase(token_value_pos, token_value.size()); + const OutcomeLog decoded = decodeOutcomeLog(missing_token_value); + ASSERT_EQ(decoded.entries.size(), 1u); + EXPECT_EQ(decoded.entries[0].token.value, ""); + + for (const String & field : {String(R"(,"ha":"ch128")"), String(R"(,"h":"00112233445566778899aabbccddeeff")"), String(R"(,"tt":"etag")")}) + { + const auto pos = bytes.find(field); + ASSERT_NE(pos, String::npos); + String incomplete = bytes; + incomplete.erase(pos, field.size()); + try + { + decodeOutcomeLog(incomplete); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), "CAS outcome log: record missing ha/h/tt"); + } + } +} + TEST(CASGCOutcomesFormat, GarbageAndUnknownWordsFailClosed) { EXPECT_THROW(decodeOutcomeLog(String("")), DB::Exception); diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index aa896dbf68df..1ad8f576809d 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -518,7 +518,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) const auto parsed = store->layout().parseBlobTargetRunKey(run.key); ASSERT_TRUE(parsed.has_value()); EXPECT_EQ(parsed->shard, run.shard); - EXPECT_EQ(parsed->generation, run.generation); + EXPECT_EQ(parsed->generation, run.key_generation); EXPECT_EQ(parsed->seq, 0u); } EXPECT_TRUE(run_seen[0]); diff --git a/src/Disks/tests/gtest_cas_gc_resume.cpp b/src/Disks/tests/gtest_cas_gc_resume.cpp index 6c707bdd60cd..55ff116b7109 100644 --- a/src/Disks/tests/gtest_cas_gc_resume.cpp +++ b/src/Disks/tests/gtest_cas_gc_resume.cpp @@ -28,7 +28,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// Whether the CURRENT retired list (any gc-shard) still holds an entry. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index b987cc39d8fb..8ea415a9f024 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -129,7 +129,7 @@ GcState readState(InMemoryBackend & b, const Pool & s) return decodeGcState(got->bytes); } -/// Whether ANY gc-shard's adopted-seal run still holds a `kCondemned` row (retired-in-snapshot T4: the +/// Whether ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row (retired-in-snapshot T4: the /// retired state rides the snapshot run, not a separate retired-list object) — the ack-floor deletion /// pipeline is still in flight while this is true. bool anyRetiredPending(InMemoryBackend & b, const Pool & s) @@ -542,7 +542,7 @@ TEST(CASGCRound, PublishDropReclaimsBlobAndManifestToFixpoint) /// retired-in-snapshot T4: after a round condemns one blob, the ADOPTED fold seal's per-shard /// condemned_summary reflects it (condemned_total == 1, pending_total == 0) — distilled zero-I/O from the -/// kCondemned rows the fold sealed into the snapshot run. +/// RunMarker::Condemned rows the fold sealed into the snapshot run. TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) { auto backend = std::make_shared(); @@ -582,7 +582,7 @@ TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) << "a non-pending condemned entry records its condemn round"; } -/// retired-in-snapshot T5: `previewDeletes` streams the adopted seal's `kCondemned` rows and reports each +/// retired-in-snapshot T5: `previewDeletes` streams the adopted seal's `RunMarker::Condemned` rows and reports each /// with the STORED condemn-time token — `awaiting_graduation` while newly condemned, then `delete_pending` /// once graduated, and NOTHING once the exact-token redelete has removed the blob. The preview performs no /// HEAD on the condemned rows (the token is durable in-run) and is WRITE-FREE throughout (spec §5 req 1). @@ -602,7 +602,7 @@ TEST(CASGCRound, PreviewReportsCondemnedRowsAndIsWriteFree) EXPECT_TRUE(gc.previewDeletes().empty()) << "a live-referenced blob is never previewed for deletion"; dropRefTransition(*backend, store->layout(), ns, "tbl", r); - runRegularRoundReclaiming(gc); /// condemning round: -1 => in-degree 0 => kCondemned row (not pending) + runRegularRoundReclaiming(gc); /// condemning round: -1 => in-degree 0 => RunMarker::Condemned row (not pending) /// Write-free contract: a full key->token snapshot must be identical across the previewDeletes call. const auto before = snapshotKeyTokens(*backend); @@ -689,7 +689,7 @@ TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) const auto parsed = store->layout().parseBlobTargetRunKey(run.key); ASSERT_TRUE(parsed.has_value()); EXPECT_EQ(parsed->shard, run.shard); - EXPECT_EQ(parsed->generation, run.generation); + EXPECT_EQ(parsed->generation, run.key_generation); EXPECT_EQ(parsed->seq, 0u); } EXPECT_TRUE(run_seen[0]); @@ -1468,7 +1468,7 @@ TEST(CASGCRetention, PruneRetainsLiveReferencedRun) backend->get(store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); ASSERT_EQ(seal1.blob_target_runs.size(), 1u); const String referenced_run_key = seal1.blob_target_runs.front().key; - ASSERT_EQ(seal1.blob_target_runs.front().generation, ref_gen); + ASSERT_EQ(seal1.blob_target_runs.front().key_generation, ref_gen); ASSERT_TRUE(backend->head(referenced_run_key).exists); /// Several idle rounds: no delta, no retired => pure ref-carry. Each round advances the generation @@ -1493,7 +1493,7 @@ TEST(CASGCRetention, PruneRetainsLiveReferencedRun) backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); ASSERT_EQ(seal_now.blob_target_runs.size(), 1u); EXPECT_EQ(seal_now.blob_target_runs.front().key, referenced_run_key); - EXPECT_EQ(seal_now.blob_target_runs.front().generation, ref_gen); + EXPECT_EQ(seal_now.blob_target_runs.front().key_generation, ref_gen); EXPECT_EQ(inDegreeOf(*backend, store->layout(), DB::UInt128(1)), 1) << "folding still resolves in-degree through the retained, carried parent ref"; @@ -1548,7 +1548,7 @@ TEST(CASGCRetention, HandOffDeletesSupersededRef) const auto seal_after = decodeFoldSeal( backend->get(store->layout().foldSealKey(st_after.snap_generation, st_after.snap_attempt))->bytes); for (const RunRef & rr : seal_after.blob_target_runs) - EXPECT_NE(rr.generation, old_gen) << "the live seal must have moved its ref off gen-1"; + EXPECT_NE(rr.key_generation, old_gen) << "the live seal must have moved its ref off gen-1"; /// ... and the post-CAS hand-off delete reclaimed gen-1's WHOLE prefix (not just the single run /// object): seal, attempt subtree, run — all gone. The ordinary prune would have leaked it because its diff --git a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp index c45cd3ff772d..c8bc92efffdf 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp @@ -564,7 +564,7 @@ TEST(CASGCShardRetireDrain, ReclaimsDroppableBlobOwnedByNonZeroShard) }; /// Whether ANY gc-shard still holds an in-flight condemned entry (the ack-floor deletion pipeline is /// in flight while this is true). Retired-in-snapshot (T4): reconstructed from the adopted fold seal's - /// kCondemned rows across all shards, not a separate retired list. + /// RunMarker::Condemned rows across all shards, not a separate retired list. auto anyRetiredPending = [&] { return anyCondemnedInSeal(*backend, layout); diff --git a/src/Disks/tests/gtest_cas_gc_state_format.cpp b/src/Disks/tests/gtest_cas_gc_state_format.cpp index aa661429f813..7e9452fdf8a7 100644 --- a/src/Disks/tests/gtest_cas_gc_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_state_format.cpp @@ -10,6 +10,8 @@ namespace DB::ErrorCodes extern const int LOGICAL_ERROR; } +CAS_BATTERY_COVERS(GcState); + TEST(CASFormatBattery, GcState) { GcState s; @@ -28,6 +30,8 @@ TEST(CASFormatBattery, GcState) "\"lo\":\"00000000000000000000000000000001\",\"ls\":\"12\"}\n"}); } +CAS_BATTERY_COVERS(GcHeartbeat); + TEST(CASFormatBattery, GcHeartbeat) { GcHeartbeat hb{UInt128(1), 1741}; diff --git a/src/Disks/tests/gtest_cas_inspect.cpp b/src/Disks/tests/gtest_cas_inspect.cpp index d807bec3f931..2ed70c293259 100644 --- a/src/Disks/tests/gtest_cas_inspect.cpp +++ b/src/Disks/tests/gtest_cas_inspect.cpp @@ -51,7 +51,7 @@ TEST(CASInspect, RendersSetPublishedAtOpWithNoPayloadSizeKey) const String bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); const String json = caInspectToJson(layout, key, bytes, DB::Cas::tests::fixture::fixtureLife(ns)); - EXPECT_NE(json.find(R"("kind":"SetPublishedAt")"), String::npos) << json; + EXPECT_NE(json.find(R"("kind":"set_published_at")"), String::npos) << json; EXPECT_EQ(json.find("payload"), String::npos) << json; } @@ -75,10 +75,99 @@ TEST(CASInspect, RendersEpochSealTxnWithPrevEpochSeal) const String bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); const String json = caInspectToJson(layout, key, bytes, DB::Cas::tests::fixture::fixtureLife(ns)); - EXPECT_NE(json.find(R"("kind":"EpochSeal")"), String::npos) << json; + EXPECT_NE(json.find(R"("kind":"epoch_seal")"), String::npos) << json; EXPECT_NE(json.find(R"("prev_epoch_seal":{"writer_epoch":2,"ref_sequence":9})"), String::npos) << json; } +/// The remaining two `RefOpKind` words this file's other tests do not exercise: a namespace's birth +/// record and its removal terminator. +TEST(CASInspect, RendersNamespaceBirthAndRemoveNamespaceOpKinds) +{ + const Layout layout("p"); + const RootNamespace ns{"srv1/db/tbl"}; + + RefLogTxn birth_txn; + birth_txn.ns = ns.string(); + birth_txn.txn_id = RefTxnId{1, 1}; + RefOp birth; + birth.kind = RefOpKind::NamespaceBirth; + birth_txn.ops.push_back(birth); + const String birth_key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), birth_txn.txn_id); + const String birth_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(birth_txn)); + const String birth_json = caInspectToJson( + layout, birth_key, birth_bytes, DB::Cas::tests::fixture::fixtureLife(ns)); + EXPECT_NE(birth_json.find(R"("kind":"namespace_birth")"), String::npos) << birth_json; + + RefLogTxn remove_txn; + remove_txn.ns = ns.string(); + remove_txn.txn_id = RefTxnId{1, 2}; + RefOp remove; + remove.kind = RefOpKind::RemoveNamespace; + remove_txn.ops.push_back(remove); + const String remove_key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), remove_txn.txn_id); + const String remove_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(remove_txn)); + const String remove_json = caInspectToJson( + layout, remove_key, remove_bytes, DB::Cas::tests::fixture::fixtureLife(ns)); + EXPECT_NE(remove_json.find(R"("kind":"remove_namespace")"), String::npos) << remove_json; +} + +/// `RefOwnerKind` renders as its full wire word (`committed`/`precommit`), not the enumerator spelling, +/// at both binding slots an `owner_transition` op carries. +TEST(CASInspect, RendersRefOwnerKindWireWords) +{ + const Layout layout("p"); + const RootNamespace ns{"srv1/db/tbl"}; + const RefTxnId id{1, 3}; + + RefLogTxn txn; + txn.ns = ns.string(); + txn.txn_id = id; + RefOp op; + op.kind = RefOpKind::OwnerTransition; + op.old_binding = RefOwnerBinding{RefOwnerKind::Committed, "all_1_1_0", manifestRef(1, 1, 1)}; + op.new_binding = RefOwnerBinding{RefOwnerKind::Precommit, "all_1_1_0", manifestRef(1, 1, 1)}; + txn.ops.push_back(op); + + const String key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id); + const String bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); + + const String json = caInspectToJson(layout, key, bytes, DB::Cas::tests::fixture::fixtureLife(ns)); + EXPECT_NE(json.find(R"("old_binding":{"kind":"committed")"), String::npos) << json; + EXPECT_NE(json.find(R"("new_binding":{"kind":"precommit")"), String::npos) << json; +} + +/// `TokenType` renders as its full wire word; the blob-target-run test below covers `emulated`, so +/// this pins the other two (`etag`/`generation`) via a second condemned-row-only run. +TEST(CASInspect, RendersTokenTypeWireWordsEtagAndGeneration) +{ + const Layout layout("p"); + + SourceEdgeRecord etag_rec; + etag_rec.ref = bh(1); + etag_rec.source_id = UInt128{0}; + etag_rec.marker = RunMarker::Condemned; + etag_rec.token = Token{.value = "v-etag", .type = TokenType::ETag}; + + SourceEdgeRecord gen_rec; + gen_rec.ref = bh(1); + gen_rec.source_id = UInt128{1}; + gen_rec.marker = RunMarker::Condemned; + gen_rec.token = Token{.value = "v-gen", .type = TokenType::Generation}; + + DB::WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(etag_rec); + writer.append(gen_rec); + writer.finish(); + out.finalize(); + const String bytes = out.str(); + + const String key = layout.blobTargetRunKey(/*generation*/3, /*attempt*/0, /*shard*/0, /*seq*/0); + const String json = caInspectToJson(layout, key, bytes); + EXPECT_NE(json.find(R"("type":"etag")"), String::npos) << json; + EXPECT_NE(json.find(R"("type":"generation")"), String::npos) << json; +} + TEST(CASInspect, RendersCommittedRowWithNoPayloadSizeKey) { const Layout layout("p"); @@ -117,7 +206,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) SourceEdgeRecord condemned_rec; condemned_rec.ref = bh(1); condemned_rec.source_id = UInt128{0}; - condemned_rec.marker = kCondemned; + condemned_rec.marker = RunMarker::Condemned; condemned_rec.delete_pending = true; condemned_rec.token = Token{.value = "etag-1", .type = TokenType::Emulated}; condemned_rec.size = 123; @@ -127,7 +216,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) SourceEdgeRecord edge_rec; edge_rec.ref = bh(2); edge_rec.source_id = UInt128(9); - edge_rec.marker = kEdgeActive; + edge_rec.marker = RunMarker::Edge; DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); @@ -147,6 +236,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) EXPECT_NE(json.find(R"("delete_pending":true)"), String::npos) << json; EXPECT_NE(json.find(R"("condemn_round":7)"), String::npos) << json; EXPECT_NE(json.find(R"("value":"etag-1")"), String::npos) << json; + EXPECT_NE(json.find(R"("type":"emulated")"), String::npos) << json; EXPECT_NE(json.find(R"("rows":2)"), String::npos) << json; EXPECT_NE(json.find(R"("distinct_blobs":2)"), String::npos) << json; EXPECT_NE(json.find(R"("edges":1)"), String::npos) << json; diff --git a/src/Disks/tests/gtest_cas_json_writer.cpp b/src/Disks/tests/gtest_cas_json_writer.cpp index f04b89255818..4eda28e717a3 100644 --- a/src/Disks/tests/gtest_cas_json_writer.cpp +++ b/src/Disks/tests/gtest_cas_json_writer.cpp @@ -21,7 +21,7 @@ TEST(CASJsonWriter, KeyValueSequenceMatchesCanonicalShape) w.u64Number(3); w.key("ok", first); w.boolValue(true); - w.key("o", "me", first); + w.key("ome", first); w.u64StringValue(1); w.closeObject(first); w.newline(); @@ -212,3 +212,31 @@ TEST(CASJsonWriterVocab, MatchesReferenceVocabulary) ref.finalize(); EXPECT_EQ(std::move(w).take(), ref.str()); } + +TEST(CASJsonWriter, WireKeyFieldHelpersMatchThePrimitivePairs) +{ + CasJsonWriter w; + bool first = true; + constexpr WireKey k_word{"st"}; + constexpr WireKey k_str{"hn"}; + constexpr WireKey k_u64s{"we"}; + constexpr WireKey k_num{"eat"}; + constexpr WireKey k_hex{"su"}; + constexpr WireKey k_bool{"fen"}; + writeWordField(w, k_word, "clean", first); + writeStringField(w, k_str, "host-1", first); + writeU64StringField(w, k_u64s, 7, first); + writeNumberField(w, k_num, 1752537630000, first); + writeHex128Field(w, k_hex, DB::UInt128{1}, first); + writeBoolField(w, k_bool, false, first); + w.closeObject(first); + w.newline(); + EXPECT_EQ(std::move(w).take(), + "{\"st\":\"clean\",\"hn\":\"host-1\",\"we\":\"7\",\"eat\":1752537630000," + "\"su\":\"00000000000000000000000000000001\",\"fen\":false}\n"); + + /// The reader-side comparison contract: a String key compares against the constant. + String key = "st"; + EXPECT_TRUE(key == k_word); + EXPECT_FALSE(key == k_str); +} diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index bf989f1e3206..d7430d321425 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -428,7 +428,7 @@ TEST(CASObservability, CaInspectDecodesRefLogToJson) const String json = caInspectToJson( layout, key, encodeRefLogTxn(txn), DB::Cas::tests::fixture::fixtureLife(ns)); EXPECT_NE(json.find("ref_log"), String::npos); - EXPECT_NE(json.find("OwnerTransition"), String::npos); + EXPECT_NE(json.find("owner_transition"), String::npos); EXPECT_NE(json.find("all_0_0_0"), String::npos); } @@ -440,11 +440,16 @@ TEST(CASObservability, CaInspectDecodesPartManifestToJson) PartManifest m; m.ref = ManifestRef{.writer_epoch = 1, .build_sequence = 2, .manifest_ordinal = 3}; m.root_namespace_id = ns; - ManifestEntry e; - e.path = "data.bin"; - e.placement = EntryPlacement::Inline; - e.inline_bytes = "hello"; - m.entries = {e}; + ManifestEntry inline_entry; + inline_entry.path = "data.bin"; + inline_entry.placement = EntryPlacement::Inline; + inline_entry.inline_bytes = "hello"; + ManifestEntry blob_entry; + blob_entry.path = "payload.bin"; + blob_entry.placement = EntryPlacement::Blob; + blob_entry.ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload.bin"))}; + blob_entry.blob_size = 5; + m.entries = {inline_entry, blob_entry}; m.payload_digest = computePayloadDigest(m); const ManifestId id{.root_namespace = ns, .ref = m.ref}; @@ -453,6 +458,9 @@ TEST(CASObservability, CaInspectDecodesPartManifestToJson) EXPECT_NE(json.find("\"root_namespace_id\""), String::npos); EXPECT_NE(json.find("data.bin"), String::npos); EXPECT_NE(json.find("\"manifest_ordinal\":3"), String::npos); + /// `EntryPlacement` renders as its full wire word (`inline`/`blob`), not the enumerator spelling. + EXPECT_NE(json.find(R"("placement":"inline")"), String::npos) << json; + EXPECT_NE(json.find(R"("placement":"blob")"), String::npos) << json; } TEST(CASObservability, CaInspectDecodesMountLeaseToJson) @@ -485,6 +493,36 @@ TEST(CASObservability, CaInspectDecodesGcStateToJson) EXPECT_NE(json.find("\"gc_shards\":4"), String::npos); } +/// `ObjectKind`/`ProvenanceOp` render as their full wire words, not the enumerator spelling. Loops +/// over every `ProvenanceOp` value so each one ends up pinned, not just whichever one a single case +/// would have picked. +TEST(CASObservability, CaInspectDecodesEnvelopeHeaderWithEveryProvenanceOpWord) +{ + Layout layout("p"); + const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of("envelope-inspect"))}; + const String key = layout.blobKey(ref); + + const std::vector> ops = { + {ProvenanceOp::Other, "other"}, + {ProvenanceOp::Insert, "insert"}, + {ProvenanceOp::Merge, "merge"}, + {ProvenanceOp::Mutation, "mutation"}, + {ProvenanceOp::Attach, "attach"}, + {ProvenanceOp::Repack, "repack"}, + }; + for (const auto & [op, word] : ops) + { + EnvelopeHeader h; + h.kind = ObjectKind::Blob; + h.provenance = Provenance{.op = op}; + const String bytes = encodeEnvelopeHeader(h, 256); + + const String json = caInspectToJson(layout, key, bytes); + EXPECT_NE(json.find(R"("kind":"blob")"), String::npos) << json; + EXPECT_NE(json.find("\"op\":\"" + word + "\""), String::npos) << json; + } +} + TEST(CASObservability, CaInspectUnknownKeyThrows) { Layout layout("p"); diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index d79f10bf73f7..43963c40e800 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -49,7 +49,7 @@ bool activeSourceExists(Backend & backend, const Layout & layout, const UInt128 String payload; while (view.next(key, payload)) { - if (payload.empty() || payload[0] != kEdgeActive) + if (payload.empty() || runMarkerFromByte(payload[0], "CAS test source-edge run") != RunMarker::Edge) continue; BlobRef ref; UInt128 row_source{}; @@ -74,7 +74,7 @@ size_t condemnedCount(Backend & backend, const Layout & layout) String key; String payload; while (view.next(key, payload)) - count += !payload.empty() && payload[0] == kCondemned; + count += !payload.empty() && runMarkerFromByte(payload[0], "CAS test source-edge run") == RunMarker::Condemned; view.verifyAgainst(run.checksum); } return count; diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index abb81c1928aa..ac14519e04c5 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -58,6 +58,8 @@ PartManifest sample() } +CAS_BATTERY_COVERS(PartManifest); + TEST(CASFormatBattery, PartManifest) { const PartManifest m = sample(); diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index dca4081c0041..c960c0547037 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -431,9 +431,9 @@ TEST(CASPluggableHash, Sha256BlobSeenByCondemnSweepAndFsckNotSilentlySkipped) ASSERT_NE(oit, frep.objects.end()) << "the sha256 blob must appear in fsck's detailed object list"; /// The fold above already condemned it into the GC snapshot, so fsck's GC-pipeline-view /// classification (not the generic Unaccounted bucket -- reachable only by width-correctly pairing - /// the fsck-side hash against the run's kCondemned row hash) must recognize it as known-to-GC. + /// the fsck-side hash against the run's RunMarker::Condemned row hash) must recognize it as known-to-GC. EXPECT_EQ(oit->cls, FsckClass::PendingGc) - << "THE CRUX: fsck must pair the sha256 blob against the GC snapshot's kCondemned row (a " + << "THE CRUX: fsck must pair the sha256 blob against the GC snapshot's RunMarker::Condemned row (a " "silent-leak regression in CasFsck.cpp's unref_hashes/in_run_hashes/retired_by_hash port " "leaves this as the generic Unaccounted bucket instead)"; } diff --git a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp index afe5ed26fb44..ee1b2a4d4d99 100644 --- a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp +++ b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp @@ -101,7 +101,7 @@ bool blobPresent(Backend & backend, const Layout & layout, const DB::UInt128 & h return backend.head(layout.blobKey(blobRefOf(hash))).exists; } -/// Whether ANY run the newest fold seal references carries a `kCondemned` row for `hash`. This is where +/// Whether ANY run the newest fold seal references carries a `RunMarker::Condemned` row for `hash`. This is where /// a rebuild used to put its zero-edge condemnations, so "nothing was condemned" is checked HERE rather /// than by watching for a deletion several rounds later. bool condemnedInSealedRuns(Backend & backend, const Layout & layout, const DB::UInt128 & hash) @@ -121,7 +121,7 @@ bool condemnedInSealedRuns(Backend & backend, const Layout & layout, const DB::U BlobRef ref; UInt128 sid; SourceEdgeKeyCodec::parse(k, ref, sid); - if (p.empty() || p[0] != kCondemned) + if (p.empty() || runMarkerFromByte(p[0], "CAS test source-edge run") != RunMarker::Condemned) continue; if (ref.digest.toU128() == hash) return true; diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 42e44585c357..2c2805f49310 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -1,4 +1,5 @@ #include +#include "cas_format_test_battery.h" #include #include #include @@ -26,17 +27,17 @@ BlobRef chRef(uint64_t n) SourceEdgeRecord edge(const BlobRef & ref, uint64_t source_id) { - return SourceEdgeRecord{.ref = ref, .source_id = UInt128(source_id), .marker = kEdgeActive}; + return SourceEdgeRecord{.ref = ref, .source_id = UInt128(source_id), .marker = RunMarker::Edge}; } SourceEdgeRecord zero(const BlobRef & ref) { - return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = kZeroMarker}; + return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Zero}; } SourceEdgeRecord condemned(const BlobRef & ref, const Token & token, uint64_t size, uint64_t round, bool pend) { - return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = kCondemned, + return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = pend, .token = token, .size = size, .condemn_round = round}; } @@ -66,6 +67,19 @@ std::vector decodeRun(const String & bytes) } +CAS_BATTERY_COVERS(RunFile); + +TEST(CASFormatBattery, RunFile) +{ + const std::vector records{edge(chRef(2), 5)}; + runFormatBattery({FormatId::RunFile, + [&] { return sealObject(FormatId::RunFile, encodeRun(records)); }, + [](std::string_view s) { decodeRun(std::string(openObject(FormatId::RunFile, s))); }, + fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()) + + "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n" + "{\"n\":1}\n"}); +} + TEST(CASRecordStream, EmptyRunRoundTripsAndChecksumMatches) { const String bytes = encodeRun({}); @@ -98,18 +112,18 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) EXPECT_EQ(back[0].ref, a); EXPECT_EQ(back[0].source_id, UInt128(10)); - EXPECT_EQ(back[0].marker, kEdgeActive); + EXPECT_EQ(back[0].marker, RunMarker::Edge); EXPECT_EQ(back[1].ref, b); EXPECT_EQ(back[1].source_id, UInt128(0)); - EXPECT_EQ(back[1].marker, kCondemned); + EXPECT_EQ(back[1].marker, RunMarker::Condemned); EXPECT_TRUE(back[1].delete_pending); EXPECT_EQ(back[1].token, (Token{"e-1", TokenType::ETag})); EXPECT_EQ(back[1].size, 4242u); EXPECT_EQ(back[1].condemn_round, 7u); EXPECT_EQ(back[2].ref, c); - EXPECT_EQ(back[2].marker, kZeroMarker); + EXPECT_EQ(back[2].marker, RunMarker::Zero); } TEST(CASRecordStream, WriterIsByteDeterministic) @@ -203,6 +217,24 @@ TEST(CASRecordStream, TrailerCountMismatchIsCorruptData) EXPECT_THROW(decodeRun(bytes), DB::Exception); } +TEST(CASRecordStream, UppercaseDigestInRecordKeyIsCorruptedData) +{ + String bytes = encodeRun({edge(chRef(10), 1)}); + const size_t digest = bytes.find("0000000000000000000000000000000a"); + ASSERT_NE(digest, String::npos); + bytes[digest + 31] = 'A'; + + try + { + static_cast(decodeRun(bytes)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + } +} + TEST(CASRecordStream, TruncationAtLineBoundaryFailsClosed) { const String bytes = encodeRun({edge(chRef(1), 10), edge(chRef(1), 20)}); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index 11a4c029f139..41d88fa57487 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -210,6 +210,8 @@ class ScopedCasGcLogCapture /// ---------- format-battery registration ---------- +CAS_BATTERY_COVERS(RefCatalog); + TEST(CASFormatBattery, RefCatalog) { RefCatalog c; @@ -261,6 +263,7 @@ TEST(CASRefCatalogFormat, RemovalStartedRoundIsRequiredExactlyForRemoving) const RefCatalog catalog{.entries = {removing}}; const String encoded = encodeRefCatalog(catalog); EXPECT_NE(encoded.find("\"rsr\":\"19\""), String::npos); + EXPECT_NE(encoded.find("\"st\":\"removing\""), String::npos); EXPECT_EQ(decodeRefCatalog(encoded), catalog); const String inc = "00000000000000000000000000000009"; @@ -566,7 +569,7 @@ TEST(CASRefCatalogFormat, NsStateToWordRaisesLogicalErrorOnImpossibleValue) #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASRefCatalogFormatDeathTest, NsStateToWordRaisesLogicalErrorOnImpossibleValueAborts) { - EXPECT_DEATH({ (void)nsStateToWord(static_cast(99)); }, "unknown ns state"); // NOLINT(clang-analyzer-optin.core.EnumCastOutOfRange): the whole point of this test is an impossible enum value + EXPECT_DEATH({ (void)nsStateToWord(static_cast(99)); }, "outside the wire vocabulary"); // NOLINT(clang-analyzer-optin.core.EnumCastOutOfRange): the whole point of this test is an impossible enum value } #endif @@ -703,7 +706,7 @@ TEST(CASRefCatalogAdmission, ReservationCoversActualWidestLegalRowsAcrossDecimal .key = layout.blobTargetRunKey(max, max, shard, 0), .checksum = std::numeric_limits::max(), .shard = shard, - .generation = max}); + .key_generation = max}); seal.condemned_summary.emplace(shard, CondemnedSummary{ .condemned_total = max, .pending_total = max, diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index 9b63fa4dfc92..b864587eff7d 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -1,6 +1,7 @@ #include #include "config.h" +#include "cas_format_test_battery.h" #include #include @@ -283,6 +284,21 @@ TEST(CASRefCheckpoint, CommittedThroughHasCanonicalExactWireEncoding) EXPECT_EQ(decodeRefCkpt(expected), ckpt); } +CAS_BATTERY_COVERS(RefCkpt); + +TEST(CASFormatBattery, RefCkpt) +{ + RefCkpt ckpt{.life_epoch = std::optional{7}, + .committed_through = RefTxnId{9, 11}, + .checkpoint_snapshot_id = RefTxnId{9, 10}, + .last_epoch_seal = RefTxnId{8, 12}}; + runFormatBattery({FormatId::RefCkpt, + [&] { return sealObject(FormatId::RefCkpt, encodeRefCkpt(ckpt)); }, + [](std::string_view s) { decodeRefCkpt(std::string(openObject(FormatId::RefCkpt, s))); }, + currentFormatHeader("cas_ref_ckpt") + + "{\"le\":\"7\",\"cte\":\"9\",\"cts\":\"11\",\"cse\":\"9\",\"css\":\"10\",\"lse\":\"8\",\"lss\":\"12\"}\n"}); +} + /// `last_epoch_seal` is chain evidence, not an arbitrary lower bound. It either names the frontier /// itself when that frontier is the terminal seal, or closes the immediately preceding numeric epoch. /// Accepting a gap or a later same-epoch frontier would manufacture a boundary that INV-2 never proved. diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index 5b6fa2070c46..cd9ce6c74291 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -471,6 +471,8 @@ TEST(CASRefEpochSealFormat, DecodeRejectsUnknownOpWordRegressionGuard) /// Shape-level failure-mode battery (truncation / v+1 gate / wrong type / leading garbage) /// =================================================================================== +static const DB::Cas::tests::BatteryCoverageRegistrar battery_covers_RefLog_seal{DB::Cas::FormatId::RefLog}; + TEST(CASRefEpochSealFormat, FormatBatteryEpochSeal) { RefLogTxn txn; diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index d5186e77aca1..ed6ed34ae599 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -285,6 +285,53 @@ TEST(CASRefCodec, RoundTripOwnerTransitionReplace) ASSERT_TRUE(decoded.ops[0].new_binding.has_value()); } +TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) +{ + RefLogTxn txn; + txn.ns = "ns"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = RefOpKind::OwnerTransition; + op.old_binding = RefOwnerBinding{RefOwnerKind::Precommit, "old", manifestRef(1, 1, 1)}; + op.new_binding = RefOwnerBinding{RefOwnerKind::Committed, "new", manifestRef(1, 1, 1)}; + txn.ops.push_back(op); + const String bytes = encodeRefLogTxn(txn); + + const String old_group = R"(,"obk":"precommit","orn":"old","ome":"1","omb":"1","omo":1)"; + const auto old_group_pos = bytes.find(old_group); + ASSERT_NE(old_group_pos, String::npos); + String old_absent = bytes; + old_absent.erase(old_group_pos, old_group.size()); + const RefLogTxn without_old = decodeRefLogTxn(old_absent, txn.ns, txn.txn_id); + ASSERT_EQ(without_old.ops.size(), 1u); + EXPECT_FALSE(without_old.ops[0].old_binding.has_value()); + EXPECT_TRUE(without_old.ops[0].new_binding.has_value()); + + const String old_ref = R"(,"orn":"old")"; + const auto old_ref_pos = bytes.find(old_ref); + ASSERT_NE(old_ref_pos, String::npos); + String incomplete_old = bytes; + incomplete_old.erase(old_ref_pos, old_ref.size()); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_old, txn.ns, txn.txn_id); }); + + const String new_group = R"(,"nbk":"committed","nrn":"new","nme":"1","nmb":"1","nmo":1)"; + const auto new_group_pos = bytes.find(new_group); + ASSERT_NE(new_group_pos, String::npos); + String new_absent = bytes; + new_absent.erase(new_group_pos, new_group.size()); + const RefLogTxn without_new = decodeRefLogTxn(new_absent, txn.ns, txn.txn_id); + ASSERT_EQ(without_new.ops.size(), 1u); + EXPECT_TRUE(without_new.ops[0].old_binding.has_value()); + EXPECT_FALSE(without_new.ops[0].new_binding.has_value()); + + const String new_ref = R"(,"nrn":"new")"; + const auto new_ref_pos = bytes.find(new_ref); + ASSERT_NE(new_ref_pos, String::npos); + String incomplete_new = bytes; + incomplete_new.erase(new_ref_pos, new_ref.size()); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_new, txn.ns, txn.txn_id); }); +} + TEST(CASRefCodec, RoundTripMultipleOpsInOneTransaction) { RefLogTxn txn; @@ -766,6 +813,8 @@ TEST(CASRefCodec, EncodeRejectsZeroManifestRefInSetPublishedAt) /// Shape-level failure-mode battery (truncation / v+1 gate / wrong type / leading garbage) /// =================================================================================== +CAS_BATTERY_COVERS(RefLog); + TEST(CASFormatBattery, RefLog) { RefLogTxn txn; diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 2293c0bc166c..2fea715879e2 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -419,6 +419,8 @@ TEST(CASRefSnapshotCodec, DecodeRejectsOversizedBufferDirectly) /// Shape-level failure-mode battery (truncation / v+1 gate / wrong type / leading garbage) /// =================================================================================== +CAS_BATTERY_COVERS(RefSnapshot); + TEST(CASFormatBattery, RefSnapshot) { const RefTableSnapshot s = makeLiveSnapshot(); diff --git a/src/Disks/tests/gtest_cas_server_root_format.cpp b/src/Disks/tests/gtest_cas_server_root_format.cpp index c2d1474dc29b..e302a4d81573 100644 --- a/src/Disks/tests/gtest_cas_server_root_format.cpp +++ b/src/Disks/tests/gtest_cas_server_root_format.cpp @@ -11,6 +11,8 @@ namespace DB::ErrorCodes extern const int CORRUPTED_DATA; } +CAS_BATTERY_COVERS(Owner); + TEST(CASFormatBattery, Owner) { OwnerObject o; @@ -31,11 +33,15 @@ TEST(CASOwnerFormat, RetiredAtRoundTrip) o.server_uuid = hexToU128("0123456789abcdeffedcba9876543210"); o.retired_at_ms = 1752537600000ULL; + EXPECT_EQ(encodeOwner(o), currentFormatHeader("cas_owner") + + "{\"su\":\"0123456789abcdeffedcba9876543210\",\"rt\":1752537600000}\n"); const OwnerObject back = decodeOwner(encodeOwner(o)); EXPECT_EQ(back.server_uuid, o.server_uuid); EXPECT_EQ(back.retired_at_ms, o.retired_at_ms); } +CAS_BATTERY_COVERS(ServerEpoch); + TEST(CASFormatBattery, ServerEpoch) { ServerEpoch e; @@ -46,6 +52,8 @@ TEST(CASFormatBattery, ServerEpoch) currentFormatHeader("cas_epoch") + "{\"nwe\":\"7\"}\n"}); } +CAS_BATTERY_COVERS(MountLease); + TEST(CASFormatBattery, MountLease) { MountLease m{hexToU128("0123456789abcdeffedcba9876543210"), 7, "host-1", 4242, diff --git a/src/Disks/tests/gtest_cas_text_format.cpp b/src/Disks/tests/gtest_cas_text_format.cpp index 4371afb3f9e8..625271435b24 100644 --- a/src/Disks/tests/gtest_cas_text_format.cpp +++ b/src/Disks/tests/gtest_cas_text_format.cpp @@ -1,4 +1,5 @@ #include +#include "cas_format_test_battery.h" #include #include #include @@ -34,6 +35,15 @@ void expectCode(int code, F && f) } } +TEST(CASFormatBattery, EveryRegisteredFormatIsBatteryCovered) +{ + std::set registered; + for (FormatId id : allRegisteredFormatIds()) + registered.insert(id); + EXPECT_EQ(registered, DB::Cas::tests::batteryCoveredIds()) + << "a registered codec is missing from the common battery (or vice versa)"; +} + /// ---- Task 2: FormatId entries for refsnaplog / blob meta / heartbeat ---- TEST(CASFormatIds, NewIdsExistWithFrozenValues) diff --git a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp index 0a76c338cdb5..522b7a738e68 100644 --- a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp +++ b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp @@ -75,7 +75,7 @@ ManifestId publishPart2( /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's kCondemned rows, not a + /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index efbe3da7a0ae..207ad24a295f 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -8,6 +9,43 @@ using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } +namespace +{ +/// Same tiny inline copy as `gtest_cas_part_manifest_format.cpp`'s `expectThrowsCode`: stays clear +/// of `Disks/tests/cas_test_helpers.h`'s `DB::Cas::tests::expectThrowsCode`, which would both drag +/// in the whole CAS backend/store machinery this file otherwise has no need for AND collide (same +/// namespace, same name and signature) if that header were ever included here too. +template +void expectThrowsCode(int expected_code, F && fn) +{ + try + { + fn(); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code); + } +} +} + +static_assert(DB::Cas::casEnumTableCoversEnum()); +static_assert(DB::Cas::casEnumTableCoversEnum()); +static_assert(DB::Cas::casEnumTableCoversEnum()); + +TEST(CASWireVocab, EnumTablesPinTheCurrentWords) +{ + using namespace DB::Cas; + EXPECT_EQ(kTokenTypeWords.toWord(TokenType::ETag, "t"), "etag"); + EXPECT_EQ(kTokenTypeWords.toWord(TokenType::Generation, "t"), "generation"); + EXPECT_EQ(kTokenTypeWords.toWord(TokenType::Emulated, "t"), "emulated"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::CityHash128, "t"), "ch128"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::XXH3_128, "t"), "xxh3"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::Sha256, "t"), "sha256"); + EXPECT_EQ(kObjectKindWords.toWord(ObjectKind::Blob, "t"), "blob"); +} + TEST(CASWireVocab, EnumWordsRoundTrip) { for (TokenType t : {TokenType::ETag, TokenType::Generation, TokenType::Emulated}) @@ -15,8 +53,8 @@ TEST(CASWireVocab, EnumWordsRoundTrip) for (BlobHashAlgo a : {BlobHashAlgo::CityHash128, BlobHashAlgo::XXH3_128, BlobHashAlgo::Sha256}) EXPECT_EQ(blobHashAlgoFromWord(blobHashAlgoName(a), "a"), a); EXPECT_EQ(objectKindFromWord(objectKindToWord(ObjectKind::Blob), "k"), ObjectKind::Blob); - EXPECT_THROW(tokenTypeFromWord("nope", "t"), DB::Exception); - EXPECT_THROW(blobHashAlgoFromWord("nope", "a"), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { tokenTypeFromWord("nope", "t"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { blobHashAlgoFromWord("nope", "a"); }); } TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) @@ -51,3 +89,104 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) const BlobRef back{blobHashAlgoFromWord(ha, "a"), codecFor(blobHashAlgoFromWord(ha, "a")).fromHex(h)}; EXPECT_EQ(back, ref); } + +TEST(CASWireVocab, ManifestRefBundleWritesTheOldPrefixedKeys) +{ + using namespace DB::Cas; + CasJsonWriter w; + bool first = true; + writeManifestRefFields(w, first, kOldManifestRefKeys, ManifestRef{1, 2, 3}); + w.closeObject(first); + EXPECT_EQ(std::move(w).take(), R"({"ome":"1","omb":"2","omo":3})"); +} + +TEST(CASWireVocab, MatchAndBuildRoundTripsABlobRef) +{ + using namespace DB::Cas; + const String rendered = R"({"ha":"ch128","h":"00112233445566778899aabbccddeeff"})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + BlobRefFields fields; + String key; + while (r.nextKey(key)) + { + if (matchBlobRefFields(key, r, fields)) + continue; + r.skipUnknown(key); + } + const BlobRef ref = fields.build("t"); + EXPECT_EQ(kBlobHashAlgoWords.toWord(ref.algo, "t"), "ch128"); +} + +TEST(CASWireVocab, BlobRefBuildFailsClosedOnHalfAGroupAndOnBadWidth) +{ + using namespace DB::Cas; + BlobRefFields only_algo; + only_algo.algo_word = "ch128"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { only_algo.build("t"); }); + + BlobRefFields short_digest; + short_digest.algo_word = "ch128"; + short_digest.digest_hex = "00112233445566778899aabbccddee"; /// 30 hex chars, needs 32 + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { short_digest.build("t"); }); +} + +TEST(CASWireVocab, BlobRefBuildFailsClosedOnRightWidthNonHexDigest) +{ + using namespace DB::Cas; + BlobRefFields bad_hex; + bad_hex.algo_word = "ch128"; + bad_hex.digest_hex = "gg112233445566778899aabbccddeeff"; /// 32 chars (right width), not hex + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { bad_hex.build("t"); }); +} + +TEST(CASWireVocab, MatchManifestRefFieldsAndBuildRefRoundTripInAnyKeyOrder) +{ + using namespace DB::Cas; + /// Fed out of writer order (mo, me, mb) to pin key-order independence. `me`/`mb` are quoted + /// decimal strings and `mo` is a bare number -- a swapped read primitive between the two shapes + /// would fail to parse this literal. + const String rendered = R"({"mo":3,"me":"7","mb":"9"})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + ManifestRefFields fields; + String key; + while (r.nextKey(key)) + { + if (matchManifestRefFields(key, r, kBareManifestRefKeys, fields)) + continue; + r.skipUnknown(key); + } + EXPECT_EQ(fields.buildRef("t", "ctx"), (ManifestRef{7, 9, 3})); +} + +TEST(CASWireVocab, ManifestRefFieldsBuildRefFailsClosedOnHalfAGroup) +{ + using namespace DB::Cas; + ManifestRefFields fields; + fields.epoch = 7; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { fields.buildRef("t", "ctx"); }); +} + +TEST(CASWireVocab, MatchTokenFieldsConsumesTtTvAndLeavesUnrelatedKeyUnmatched) +{ + using namespace DB::Cas; + const String rendered = R"({"tt":"etag","tv":"abc","zz":1})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + TokenFields fields; + String key; + bool saw_unmatched = false; + while (r.nextKey(key)) + { + if (matchTokenFields(key, r, fields)) + continue; + saw_unmatched = true; + r.skipUnknown(key); + } + ASSERT_TRUE(fields.type_word.has_value()); + EXPECT_EQ(*fields.type_word, "etag"); + ASSERT_TRUE(fields.value.has_value()); + EXPECT_EQ(*fields.value, "abc"); + EXPECT_TRUE(saw_unmatched); +} From 540890f654bce0d3233bf82f2ec25a5553c9a4a6 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:03:08 +0200 Subject: [PATCH 04/81] =?UTF-8?q?cas:=20wire-keys=20phase=202=20=E2=80=94?= =?UTF-8?q?=20cut=20the=20wire-format=20JSON=20keys=20to=20semantic=20name?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The actual key rename, on top of the phase-1 carrier infrastructure. The format generation history is first reset to a `{1, 1}` baseline, since CAS has no released, persisted data yet and pre-release generations exist only to prove the evolution machinery. Every CAS wire format switches its JSON keys from single-letter/abbreviated spellings to descriptive names in one pass: the shared `BlobRef`/`Token`/ `ManifestRef`/binding fields, `cas_blob_meta`, `cas_pool_meta` (`algos_used` becomes a JSON word array instead of a bitmask), GC state/heartbeat/ maintenance state, the server-root record (`MountLease::min_active` becomes `min_active_build_sequence`), `cas_ref_ckpt`, `cas_ref_log` (the seal link becomes `!prev_epoch`/`!prev_seq`), `cas_ref_snapshot`, `cas_part_manifest`, `cas_run`, `cas_gc_outcomes` (`kind`/`outcome`), the fold-seal record and its `CoverageClass` words, the blob descriptor (with its 239-byte worst case proved at compile time against the 240-byte floor), and `cas_ref_catalog`. Golden tests are re-pinned to the new bytes throughout. Token-group requiredness is unified through `TokenFields::build`: an outcome missing its token now fails closed instead of serializing a partial group. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../cas/architecture/manifests-and-refs.md | 2 +- .../cas/architecture/mounts-and-leases.md | 12 +- .../cas/architecture/storage-layout.md | 2 +- .../ContentAddressedTransaction.cpp | 6 +- .../Formats/CasBlobEnvelopeFormat.cpp | 74 ++++- .../Formats/CasBlobEnvelopeFormat.h | 18 +- .../Formats/CasBlobMetaFormat.cpp | 15 +- .../Formats/CasBlobMetaFormat.h | 5 + .../Formats/CasEnvelopeLimits.h | 4 +- .../Formats/CasFoldSealFormat.cpp | 179 ++++++----- .../Formats/CasFoldSealFormat.h | 72 +++-- .../ContentAddressed/Formats/CasFormat.cpp | 58 +--- .../ContentAddressed/Formats/CasFormat.h | 101 +----- .../Formats/CasGcMaintenanceStateFormat.cpp | 4 +- .../Formats/CasGcOutcomesFormat.cpp | 26 +- .../Formats/CasGcOutcomesFormat.h | 3 + .../Formats/CasGcStateFormat.cpp | 24 +- .../Formats/CasGcStateFormat.h | 4 +- .../ContentAddressed/Formats/CasLayout.cpp | 4 +- .../Formats/CasPartManifestFormat.cpp | 59 ++-- .../Formats/CasPartManifestFormat.h | 13 +- .../Formats/CasPoolMetaFormat.cpp | 99 +++--- .../Formats/CasPoolMetaFormat.h | 18 +- .../Formats/CasRecordStreamFormat.cpp | 69 ++-- .../Formats/CasRecordStreamFormat.h | 19 +- .../Formats/CasRefCatalogFormat.cpp | 30 +- .../Formats/CasRefCatalogFormat.h | 8 +- .../Formats/CasRefCkptFormat.cpp | 62 ++-- .../Formats/CasRefLogFormat.cpp | 119 ++++--- .../Formats/CasRefLogFormat.h | 25 +- .../Formats/CasRefSnapshotFormat.cpp | 96 +++--- .../Formats/CasRefSnapshotFormat.h | 4 +- .../Formats/CasRefWireVocab.h | 7 +- .../Formats/CasServerRootFormats.cpp | 41 ++- .../Formats/CasServerRootFormats.h | 16 +- .../Formats/CasTextFormat.cpp | 35 ++ .../ContentAddressed/Formats/CasTextFormat.h | 17 +- .../ContentAddressed/Formats/CasWireVocab.cpp | 9 +- .../ContentAddressed/Formats/CasWireVocab.h | 54 ++-- .../ContentAddressed/Formats/README.md | 77 +++-- .../ContentAddressed/Gc/CasGc.cpp | 24 +- .../Gc/CasOrphanManifestSweep.cpp | 22 +- .../Gc/CasOrphanManifestSweep.h | 4 +- .../ContentAddressed/Pool/CasPartWriteTxn.cpp | 4 +- .../ContentAddressed/Pool/CasPool.cpp | 55 +--- .../ContentAddressed/Pool/CasPool.h | 2 +- .../ContentAddressed/Pool/CasPoolMeta.cpp | 4 +- .../ContentAddressed/Pool/CasServerRoot.cpp | 28 +- .../ContentAddressed/Pool/CasServerRoot.h | 18 +- .../Primitives/CasBlobDigest.h | 5 +- .../Primitives/CasEnumWireTableAsserts.h | 3 +- .../Tools/CasDecommission.cpp | 4 +- .../ContentAddressed/Tools/CasInspect.cpp | 10 +- src/Disks/tests/cas_test_helpers.h | 12 +- .../tests/gtest_cas_blob_envelope_format.cpp | 129 +++++++- .../tests/gtest_cas_blob_meta_format.cpp | 29 +- src/Disks/tests/gtest_cas_decommission.cpp | 6 +- src/Disks/tests/gtest_cas_encoding_pins.cpp | 302 ++++++++++++++++-- src/Disks/tests/gtest_cas_enum_wire_table.cpp | 2 +- src/Disks/tests/gtest_cas_event_log.cpp | 2 +- src/Disks/tests/gtest_cas_fold_seal_codec.cpp | 2 +- .../tests/gtest_cas_fold_seal_format.cpp | 101 +++--- src/Disks/tests/gtest_cas_forget.cpp | 8 +- src/Disks/tests/gtest_cas_format.cpp | 120 +++---- src/Disks/tests/gtest_cas_format_battery.cpp | 18 +- src/Disks/tests/gtest_cas_fsck.cpp | 4 +- .../tests/gtest_cas_gc_arithmetic_intake.cpp | 36 +-- src/Disks/tests/gtest_cas_gc_attempt.cpp | 2 +- src/Disks/tests/gtest_cas_gc_bounded_walk.cpp | 8 +- src/Disks/tests/gtest_cas_gc_fold.cpp | 2 +- .../tests/gtest_cas_gc_frontier_gate.cpp | 10 +- src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 190 +++++------ src/Disks/tests/gtest_cas_gc_leak.cpp | 2 +- .../gtest_cas_gc_maintenance_state_format.cpp | 24 +- .../tests/gtest_cas_gc_outcomes_format.cpp | 63 ++-- src/Disks/tests/gtest_cas_gc_rebuild.cpp | 6 +- src/Disks/tests/gtest_cas_gc_resume.cpp | 2 +- src/Disks/tests/gtest_cas_gc_round.cpp | 12 +- src/Disks/tests/gtest_cas_gc_shard_plan.cpp | 2 +- src/Disks/tests/gtest_cas_gc_state_format.cpp | 38 +-- src/Disks/tests/gtest_cas_heartbeat.cpp | 20 +- src/Disks/tests/gtest_cas_inspect.cpp | 27 ++ src/Disks/tests/gtest_cas_json_writer.cpp | 32 +- src/Disks/tests/gtest_cas_mount.cpp | 38 +-- .../tests/gtest_cas_ns_file_incarnation.cpp | 54 ---- .../tests/gtest_cas_ns_file_read_contract.cpp | 2 +- .../tests/gtest_cas_orphan_manifest_sweep.cpp | 10 +- .../tests/gtest_cas_orphan_nomination.cpp | 2 +- .../tests/gtest_cas_part_manifest_format.cpp | 117 +++++-- src/Disks/tests/gtest_cas_part_write.cpp | 2 +- .../gtest_cas_part_write_root_dangle.cpp | 12 +- src/Disks/tests/gtest_cas_pluggable_hash.cpp | 29 +- src/Disks/tests/gtest_cas_pool.cpp | 13 +- .../gtest_cas_rebuild_condemn_nothing.cpp | 2 +- .../tests/gtest_cas_record_stream_format.cpp | 98 +++++- .../tests/gtest_cas_recovery_grounding.cpp | 12 +- src/Disks/tests/gtest_cas_ref_catalog.cpp | 127 +++++--- .../gtest_cas_ref_catalog_birth_wiring.cpp | 4 +- src/Disks/tests/gtest_cas_ref_ckpt.cpp | 64 ++-- src/Disks/tests/gtest_cas_ref_ckpt_join.cpp | 9 +- .../tests/gtest_cas_ref_contiguous_alloc.cpp | 108 ------- .../tests/gtest_cas_ref_epoch_seal_format.cpp | 30 +- src/Disks/tests/gtest_cas_ref_log_format.cpp | 65 +++- .../tests/gtest_cas_ref_read_contract.cpp | 2 +- .../tests/gtest_cas_ref_snapshot_format.cpp | 50 ++- .../tests/gtest_cas_server_root_format.cpp | 43 ++- .../tests/gtest_cas_shutdown_context.cpp | 2 +- .../gtest_cas_sweep_deletion_premise.cpp | 16 +- src/Disks/tests/gtest_cas_text_format.cpp | 46 ++- .../tests/gtest_cas_truncate_reclaim.cpp | 2 +- src/Disks/tests/gtest_cas_wire_vocab.cpp | 90 +++++- src/Disks/tests/gtest_cas_writer_duties.cpp | 2 +- .../StorageSystemContentAddressedMounts.cpp | 2 +- tests/integration/test_cas_gc_sharded/test.py | 16 +- .../test_cas_gcs/gcs_mocks/server.py | 2 +- tests/integration/test_cas_gcs/test.py | 4 +- .../05023_cas_dropns_leaked_namespace.sh | 26 +- 117 files changed, 2350 insertions(+), 1635 deletions(-) diff --git a/docs/en/antalya/cas/architecture/manifests-and-refs.md b/docs/en/antalya/cas/architecture/manifests-and-refs.md index df86d82e575d..ba476cffe88a 100644 --- a/docs/en/antalya/cas/architecture/manifests-and-refs.md +++ b/docs/en/antalya/cas/architecture/manifests-and-refs.md @@ -101,7 +101,7 @@ swept for that root. flowchart TD A["LIST one page of cas/manifests/
freeze candidates with exact GET"] --> B{"build-prefix eligible?
durable watermark fact only"} B -->|"epoch less than lease epoch"| ELIG["eligible, old-epoch debris"] - B -->|"same epoch, min_active clears build_seq"| ELIG + B -->|"same epoch, min_active_build_sequence clears build_seq"| ELIG B -->|"no lease, or epoch ahead, or build may be live"| SKIP["skip"] ELIG --> C["protection view: committed manifests
plus live precommits
plus manifests with an unfolded minus-one"] C -->|"key protected"| SKIP2["skip"] diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index d778756ce323..66e540a5464f 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -68,7 +68,7 @@ Two failure modes this closes: One object, `gc/server-roots//mount`, carries **both** the liveness lease and the build watermark — there is no separate watermark object. `MountLease` fields: `server_uuid`, `writer_epoch`, `write_attempt_id`, `hostname`, `pid`, `started_at_ms`, renewal `seq`, -`expires_at_ms`, `min_active` (the build-watermark floor), and `gc_fenced`. +`expires_at_ms`, `min_active_build_sequence` (the build-watermark floor), and `gc_fenced`. - **Logical renewal identity.** Each holder-originated body has a fresh nonzero `write_attempt_id`. One logical renewal fixes one immutable `(key, bytes, expected token, @@ -133,9 +133,9 @@ into "not found". Global build ordering is the **pair** `(writer_epoch, build_seq)` compared lexicographically — the exact comparison GC uses for eligibility. The durable authority for both is the mount object -itself: no mount means no deletion authority means nothing is swept. `min_active`, the oldest +itself: no mount means no deletion authority means nothing is swept. `min_active_build_sequence`, the oldest in-flight `build_seq`, rides in the same mount object as the watermark floor; `UINT64_MAX` in -`min_active` is the farewell/retired sentinel, not a real build. +`min_active_build_sequence` is the farewell/retired sentinel, not a real build. ## Mount claim outcomes {#claim-outcomes} @@ -153,7 +153,7 @@ a `MountClaimResult::Kind` together with a `MountPriorState` describing which ce | `MountPriorState` | Certificate that justified the reclaim | |---|---| | `None` | no reclaim needed (fresh claim or same-epoch refresh) | -| `Clean` | the predecessor's own graceful farewell (`min_active == UINT64_MAX`) | +| `Clean` | the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) | | `Fenced` | GC's own threshold-gated fence-out (`gc_fenced`) | | `UncleanObserved` | this claimant's own token-stability observation held for the full `TTL + drift` window | @@ -168,7 +168,7 @@ stateDiagram-v2 Absent --> Live: claimMount putIfAbsent, seq=1 Live --> Live: keeper beat, putOverwrite seq+1 Live --> Fenced: GC observes a stable token past threshold, gc_fenced=1, body preserved - Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active=MAX) + Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active_build_sequence=MAX) Fenced --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim Terminated --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim Live --> Live: same-uuid claim, proven-dead token via UncleanObserved @@ -218,7 +218,7 @@ processed before renewal resumes. **Clean unmount:** request stop and join both persistent workers, drain the ref lanes, and only if the drain *certified* quiescence call `MountLeaseKeeper::release` on an `Active` keeper to write the -terminal farewell (`expires_at_ms` already expired, `min_active = UINT64_MAX`). That sentinel is what +terminal farewell (`expires_at_ms` already expired, `min_active_build_sequence = UINT64_MAX`). That sentinel is what lets a successor reclaim instantly. A `RenewalTerminal` keeper, an unresolved ref write, or a sent renewal ambiguity writes no farewell — an unearned farewell would let a successor start mutating while a stale conditional request from the predecessor is still in flight. diff --git a/docs/en/antalya/cas/architecture/storage-layout.md b/docs/en/antalya/cas/architecture/storage-layout.md index e4d20725836b..b136552acf37 100644 --- a/docs/en/antalya/cas/architecture/storage-layout.md +++ b/docs/en/antalya/cas/architecture/storage-layout.md @@ -45,7 +45,7 @@ namespace's shape and never interprets its contents. | `gc/gen//attempt//outcomes//.zst` | GC outcome log | `cas_gc_outcomes` | GC | | `gc/server-roots//owner` | server-root owner singleton | `cas_owner` | mount | | `gc/server-roots//epoch` | server-root epoch singleton | `cas_epoch` | mount | -| `gc/server-roots//mount` | mount lease (incl. `min_active` watermark) | `cas_mount_lease` | mount | +| `gc/server-roots//mount` | mount lease (incl. `min_active_build_sequence` watermark) | `cas_mount_lease` | mount | | `roots/` | loose mountpoint object, verbatim | — (never interpreted) | upper layers | | `staging//…` | S3-native upload staging scratch | — | writer, own mount only | diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp index 1216906194b4..437c4d536250 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp @@ -113,7 +113,7 @@ ContentAddressedTransaction::~ContentAddressedTransaction() /// backstop for aborted/exception-unwound transactions whose publishStaging never ran. cleanupPendingTempFiles(); - /// An uncommitted transaction's uploads become min_active-spared debris: abandon every + /// An uncommitted transaction's uploads become min_active_build_sequence-spared debris: abandon every /// still-open PartWriteTxn so its build_seq is retired. This replaces the former pin machinery. if (committed) return; @@ -759,8 +759,8 @@ std::string ContentAddressedTransaction::buildS3StagingBlobHeader( header.kind = Cas::ObjectKind::Blob; header.incarnation_tag = (static_cast(thread_local_rng()) << 64) | thread_local_rng(); header.build_id = 0; /// not known at stream time; diagnostic-only (not read by GC/read paths) - /// ch = the real ClickHouse VERSION_INTEGER (diagnostic-only; consistent with `PartWriteTxn::buildHeader`). - /// The v3 envelope drops hash_algo/domain_id/writer_version, so forensics ride on ch + bld. + /// `chver` = the real ClickHouse VERSION_INTEGER (diagnostic-only; consistent with `PartWriteTxn::buildHeader`). + /// The envelope drops hash_algo/domain_id/writer_version, so forensics ride on `chver` + `build`. header.provenance = Cas::Provenance{ /*created_at_ms*/ 0, cfg.server_id, VERSION_INTEGER, Cas::ProvenanceOp::Other}; header.intended_ref = route.ns.string() + "/" + route.ref; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index a4933ff2ccb8..129714d6b9e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -1,8 +1,11 @@ #include +#include #include #include #include #include +#include +#include namespace DB { @@ -26,11 +29,11 @@ namespace EnvelopeWire constexpr WireKey type{"type"}; constexpr WireKey version{"v"}; constexpr WireKey tag{"tag"}; - constexpr WireKey build{"bld"}; - constexpr WireKey time_ms{"ts"}; - constexpr WireKey creator{"by"}; + constexpr WireKey build{"build"}; + constexpr WireKey time_ms{"time_ms"}; + constexpr WireKey creator{"creator"}; constexpr WireKey op{"op"}; - constexpr WireKey chver{"ch"}; + constexpr WireKey chver{"chver"}; constexpr WireKey ref{"ref"}; } @@ -45,6 +48,60 @@ constexpr EnumWireTable kProvenanceOpWords{{{ static_assert(casEnumTableCoversEnum()); +/// Byte cost of one JSON key as written by `CasJsonWriter::key`: the leading `{`/`,` separator (1) +/// plus the opening quote (1), the key text, and the closing quote and colon (2). +constexpr size_t keyCost(WireKey key) +{ + return 4 + key.text.size(); +} + +/// A quoted `hex128Value` is always exactly this wide -- 2 quote bytes plus 2 hex digits per +/// `UInt128` byte -- because `writeHexUIntLowercase` zero-pads; there is no smaller or larger case. +constexpr size_t kQuotedHex128Len = 2 + sizeof(UInt128) * 2; + +/// Maximum decimal digits an unquoted `writeIntText` value can produce for each integer width the +/// envelope persists, taken from the type itself rather than re-typed as a literal. +constexpr size_t kMaxU64DecimalLen = std::numeric_limits::digits10 + 1; +constexpr size_t kMaxU32DecimalLen = std::numeric_limits::digits10 + 1; + +/// The longest persisted `op` word, found by walking the table rather than hardcoding one -- the +/// worst case must track `kProvenanceOpWords` even if a future entry outgrows "mutation". +constexpr size_t maxProvenanceOpWordLen() +{ + size_t max_len = 0; + for (const auto & entry : kProvenanceOpWords.entries) + max_len = std::max(max_len, entry.word.size()); + return max_len; +} + +/// Mandatory (always-written whenever `provenance` is set) non-`ref` fields at type maxima, in the +/// exact field order `encodeEnvelopeHeader` writes them. `CasPoolMetaFormat.cpp` records why 240 was +/// chosen as the floor above this bound. +constexpr size_t kMandatoryNonRefWorstCase = + keyCost(EnvelopeWire::type) + 2 + kBlobType.size() + + keyCost(EnvelopeWire::version) + kMaxU32DecimalLen + + keyCost(EnvelopeWire::tag) + kQuotedHex128Len + + keyCost(EnvelopeWire::build) + kQuotedHex128Len + + keyCost(EnvelopeWire::time_ms) + kMaxU64DecimalLen + + keyCost(EnvelopeWire::creator) + kQuotedHex128Len + + keyCost(EnvelopeWire::op) + 2 + maxProvenanceOpWordLen() + + keyCost(EnvelopeWire::chver) + kMaxU32DecimalLen; + +/// The encoder always frames `ref`, even when empty: the key (`,"ref":`), the empty quotes, the +/// closing `}`, and the trailing '\n' reserved at byte `blob_header_len - 1`. +constexpr size_t kRefFramingAndTerminator = keyCost(EnvelopeWire::ref) + 2 + 1 + 1; + +/// The worst-case byte count `encodeEnvelopeHeader` can ever produce before the diagnostic `ref` +/// gets any budget at all. Proven, not merely documented: the static_assert below fails the BUILD if +/// a future key or type change ever closes the margin under `kMinBlobHeaderLen`. +constexpr size_t kMandatoryDescriptorWorstCase = kMandatoryNonRefWorstCase + kRefFramingAndTerminator; + +static_assert(kMandatoryDescriptorWorstCase <= kMinBlobHeaderLen - 1, + "the mandatory blob-envelope fields plus the empty-ref framing must fit under kMinBlobHeaderLen " + "(the trailing '\\n' is already counted above, so the spare byte is the diagnostic ref's floor " + "budget, not the newline); if a field grew, either shrink it back or " + "raise kMinBlobHeaderLen (CasEnvelopeLimits.h) to match"); + /// The escaped byte-length of one raw ref char under the frozen envelope alphabet (see writeEnvelopeRefField). size_t escapedLen(char c) { @@ -101,6 +158,11 @@ std::string_view provenanceOpToWireWord(ProvenanceOp op) return kProvenanceOpWords.toWord(op, "CAS blob envelope"); } +ProvenanceOp provenanceOpFromWireWord(std::string_view w) +{ + return kProvenanceOpWords.fromWord(w, "CAS blob envelope"); +} + String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { if (header.kind != ObjectKind::Blob) @@ -129,7 +191,7 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { writeKey(buf, "!x", first); writeStringValue(buf, "1"); } - json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":3,...,"ch":26006001 (no ref, no closing brace) + json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":1,...,"chver":26006001 (no ref, no closing brace) } /// Optional `ref`, truncated to the exact remaining budget. Layout after this block: @@ -211,7 +273,7 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje } else if (key == EnvelopeWire::op) { - prov.op = kProvenanceOpWords.fromWord(r.readString(), "CAS blob envelope"); + prov.op = provenanceOpFromWireWord(r.readString()); have_prov = true; } else if (key == EnvelopeWire::chver) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 68c473a70c3d..0aa26e22e182 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -35,6 +35,9 @@ enum class ProvenanceOp : uint8_t /// Returns the persisted wire word for a validated provenance operation. std::string_view provenanceOpToWireWord(ProvenanceOp op); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +ProvenanceOp provenanceOpFromWireWord(std::string_view w); + /// Optional diagnostic metadata recorded with an envelope. The fields identify when and where the /// incarnation was created, the ClickHouse build that wrote it, and the operation that produced it; /// none of them participates in object identity or a protocol decision. @@ -58,19 +61,20 @@ struct Provenance /// algorithm and digest are already present in the object key and manifest reference, `domain_id` had /// no validating consumer, and `header_hash` had no consumer once the CityHash64 check left the /// envelope. Writer forensics are represented -/// by `ch` and `bld`, so a separate `writer_version` is unnecessary. The `v` field is the sole format -/// compatibility gate; a reader rejects a version it does not understand before interpreting the body. +/// by `chver` and `build`, so a separate `writer_version` is unnecessary. The `v` field is the sole +/// format compatibility gate; a reader rejects a version it does not understand before interpreting +/// the body. struct EnvelopeHeader { ObjectKind kind = ObjectKind::Blob; /// Set by decode from the header `v`; encode stamps `currentCompatibilityVersion`. A reader /// fails closed (UNKNOWN_FORMAT_VERSION) when `v` exceeds what this build understands. uint32_t compatibility_version = 0; - UInt128 incarnation_tag{}; /// `tag` - UInt128 build_id{}; /// `bld` - std::optional provenance; /// `ts` / `by` / `op` / `ch` - std::optional intended_ref; /// `ref` (diagnostic; truncated on encode to fit the header) - uint32_t header_len = 0; /// filled by encode/decode = blob_header_len (payload offset) + UInt128 incarnation_tag{}; /// `tag` + UInt128 build_id{}; /// `build` + std::optional provenance; /// `time_ms` / `creator` / `op` / `chver` + std::optional intended_ref; /// `ref` (diagnostic; truncated on encode to fit the header) + uint32_t header_len = 0; /// filled by encode/decode = blob_header_len (payload offset) /// Test-only knob: emit an unknown `!`-critical key. Decoding the resulting header must fail /// closed with `UNKNOWN_FORMAT_VERSION`, exercising the compatibility rule for critical extensions. bool emit_unknown_critical_key = false; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp index ab98b8858693..2b40584e0199 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp @@ -20,9 +20,9 @@ namespace namespace BlobMetaWire { - constexpr WireKey state{"st"}; - constexpr WireKey condemn_round{"cr"}; - constexpr WireKey size{"sz"}; + constexpr WireKey state{"state"}; + constexpr WireKey condemn_round{"condemn_round"}; + constexpr WireKey size{"size"}; } constexpr EnumWireTable kMetaStateWords{{{ @@ -39,6 +39,11 @@ std::string_view metaStateToWireWord(MetaState state) return kMetaStateWords.toWord(state, "CAS blob meta"); } +MetaState metaStateFromWireWord(std::string_view w) +{ + return kMetaStateWords.fromWord(w, "CAS blob meta"); +} + String encodeBlobMeta(const BlobMeta & meta) { CasJsonWriter out(256); @@ -71,7 +76,7 @@ BlobMeta decodeBlobMeta(std::string_view bytes) { if (key == BlobMetaWire::state) { - m.state = kMetaStateWords.fromWord(r.readString(), "CAS blob meta"); + m.state = metaStateFromWireWord(r.readString()); saw_state = true; } else if (key == BlobMetaWire::condemn_round) @@ -82,7 +87,7 @@ BlobMeta decodeBlobMeta(std::string_view bytes) r.skipUnknown(key); } if (!saw_state) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing st"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: missing state"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob meta: trailing bytes"); return m; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h index 5290e7c831db..9176dc641203 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.h @@ -19,8 +19,13 @@ enum class MetaState : uint8_t /// so a writer may republish it by replacing the body and updating this marker. }; +/// Convert a meta-state discriminator to its canonical wire word. Throws `LOGICAL_ERROR` for an +/// out-of-range enum value. std::string_view metaStateToWireWord(MetaState state); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +MetaState metaStateFromWireWord(std::string_view w); + /// The durable per-hash meta record. Its text representation consists of a format header followed by /// one JSON object with the state word, the GC condemnation round, and the raw body size. `size` is /// retained for introspection, fsck, and GC accounting; reads of the blob never consult the meta. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h index df8cee44bc47..92e6b252ab1b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasEnvelopeLimits.h @@ -8,7 +8,9 @@ namespace DB::Cas /// The pool-wide floor for `blob_header_len`. One compile-time owner, read by BOTH /// `validatePoolBlobHeaderLen` (pool creation / decode) and the blob-envelope codec, so the /// mandatory-descriptor worst-case proof and the enforced floor can never guard different numbers. -/// The derivation of the floor lives in `CasPoolMetaFormat.cpp` next to the worst-case table. +/// The byte-for-byte worst-case derivation (`kMandatoryDescriptorWorstCase`) lives beside the +/// envelope key constants in `CasBlobEnvelopeFormat.cpp`; `CasPoolMetaFormat.cpp` records why 240 +/// (rather than the bare worst case) was chosen as the floor. inline constexpr uint64_t kMinBlobHeaderLen = 240; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index 2324cc2c037a..129c4b5c3493 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -26,32 +27,32 @@ namespace namespace FoldSealWire { - constexpr WireKey generation{"g"}; - constexpr WireKey parent_generation{"pg"}; - constexpr WireKey kind{"k"}; + constexpr WireKey generation{"generation"}; + constexpr WireKey parent_generation{"parent_generation"}; + constexpr WireKey kind{"kind"}; constexpr WireKey run_key{"key"}; - constexpr WireKey checksum{"ck"}; + constexpr WireKey checksum{"checksum"}; constexpr WireKey shard{"shard"}; - constexpr WireKey key_generation{"gen"}; + constexpr WireKey key_generation{"key_generation"}; constexpr WireKey life{"life"}; - constexpr WireKey classification{"cls"}; - constexpr WireKey fold_epoch{"lfe"}; - constexpr WireKey fold_seq{"lfs"}; - constexpr WireKey hold_reason{"hr"}; - constexpr WireKey hold_epoch{"hpe"}; - constexpr WireKey hold_seq{"hps"}; - constexpr WireKey retries{"hrc"}; - constexpr WireKey retry_round{"hnr"}; - constexpr WireKey remove_epoch{"rte"}; - constexpr WireKey remove_seq{"rts"}; - constexpr WireKey condemned_total{"ct"}; - constexpr WireKey pending_total{"pt"}; - constexpr WireKey oldest_round{"ocr"}; + constexpr WireKey classification{"class"}; + constexpr WireKey fold_epoch{"fold_epoch"}; + constexpr WireKey fold_seq{"fold_seq"}; + constexpr WireKey hold_reason{"hold_reason"}; + constexpr WireKey hold_epoch{"hold_epoch"}; + constexpr WireKey hold_seq{"hold_seq"}; + constexpr WireKey retries{"retries"}; + constexpr WireKey retry_round{"retry_round"}; + constexpr WireKey remove_epoch{"remove_epoch"}; + constexpr WireKey remove_seq{"remove_seq"}; + constexpr WireKey condemned_total{"condemned"}; + constexpr WireKey pending_total{"pending"}; + constexpr WireKey oldest_round{"oldest_round"}; } -constexpr std::string_view kRefLifeTag = "rfl"; -constexpr std::string_view kBlobRunTag = "btr"; -constexpr std::string_view kCondemnedTag = "cnd"; +constexpr std::string_view kRefLifeTag = "ref_life"; +constexpr std::string_view kBlobRunTag = "blob_run"; +constexpr std::string_view kCondemnedTag = "condemned"; constexpr EnumWireTable kHoldReasonWords{{{ {HoldReason::GapBelowWitness, "gap_below_witness"}, @@ -64,20 +65,14 @@ constexpr EnumWireTable kHoldReasonWords{{{ static_assert(casEnumTableCoversEnum()); -HoldReason holdReasonFromWord(std::string_view w) -{ - return kHoldReasonWords.fromWord(w, "CAS fold seal hold reason"); -} +constexpr EnumWireTable kCoverageClassWords{{{ + {CoverageClass::Absent, "absent"}, + {CoverageClass::Unchanged, "unchanged"}, + {CoverageClass::Folded, "folded"}, + {CoverageClass::Clamped, "clamped"}, +}}}; -/// The classification set is CLOSED. Every consumer of a coverage row branches on exact values — the -/// sweep's §6 deletion premise refuses a row by testing `== 4` and then `== 0` — so a value outside the -/// set is not an unknown variant to be tolerated forward: it is a row that passes every refusal written -/// in terms of the set and reaches the irreversible delete. One predicate, used by both directions, so -/// the writer's self-check and the reader's fail-close can never name different sets. -bool isKnownClassification(uint64_t classification) -{ - return classification == 0 || classification == 1 || classification == 2 || classification == 4; -} +static_assert(casEnumTableCoversEnum()); /// A hold names a position the fold must resolve, and both components of that id are nonzero (the /// canonical `RefTxnId` rule `renderRefTxnId` enforces for every id that becomes a key). A zero @@ -104,7 +99,7 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi what, key); } -/// Emit one run record (`k` = "btr") WITHOUT its line terminator; the caller closes (and measures) the +/// Emit one run record (`kind` = `blob_run`) WITHOUT its line terminator; the caller closes (and measures) the /// line, and sorts the vector by key first. void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { @@ -180,6 +175,36 @@ std::string_view holdReasonToWord(HoldReason r) return kHoldReasonWords.toWord(r, "CAS fold seal hold reason"); } +HoldReason holdReasonFromWord(std::string_view w) +{ + return kHoldReasonWords.fromWord(w, "CAS fold seal hold reason"); +} + +std::string_view coverageClassToWord(CoverageClass c) +{ + return kCoverageClassWords.toWord(c, "CAS fold seal classification"); +} + +CoverageClass coverageClassFromWord(std::string_view w) +{ + return kCoverageClassWords.fromWord(w, "CAS fold seal classification"); +} + +namespace +{ +/// A seal can carry thousands of `ref_life` rows, so a rejected word names the row that carries it -- +/// "unknown word" alone leaves an operator scanning the object by hand. +CoverageClass coverageClassInRow(std::string_view w, std::string_view life_hex) +{ + return kCoverageClassWords.fromWord(w, fmt::format("CAS fold seal: ref_life '{}' classification", life_hex)); +} + +HoldReason holdReasonInRow(std::string_view w, std::string_view life_hex) +{ + return kHoldReasonWords.fromWord(w, fmt::format("CAS fold seal: ref_life '{}' hold_reason", life_hex)); +} +} + FoldSealCaps foldSealCaps() { const FormatTraits & t = traitsFor(FormatId::FoldSeal); @@ -253,22 +278,19 @@ String encodeFoldSeal(const CasFoldSeal & seal) /// process, not corruption arriving from a store — and none of these shapes is repairable once /// durable, so none is ever written. /// - /// A classification outside the closed set first, because the two checks after it are stated in - /// terms of the set and a row they cannot classify makes their answers meaningless. - if (!isKnownClassification(cov.classification)) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS fold seal: coverage '{}' has classification {}, which is not one of the four the " - "fold grammar defines (0 absent, 1 unchanged, 2 folded, 4 clamped) — every consumer " - "branches on those exact values, so this row would pass refusals meant to stop it", - life_hex, cov.classification); - /// A classification-4 row whose hold was dropped is indistinguishable, once durable, from a - /// namespace that stopped for no reason — and a hold on any other classification claims a stop - /// that did not happen. - if ((cov.classification == 4) != cov.hold.has_value()) + /// A classification outside the four named values first, because the two checks after it are + /// stated in terms of those names and a row they cannot classify makes their answers meaningless. + /// `coverageClassToWord` IS that range check (it throws `LOGICAL_ERROR` for a value the table + /// does not index), so capturing its result here also gives the record its wire value below. + const std::string_view classification_word = coverageClassToWord(cov.classification); + /// A clamped row whose hold was dropped is indistinguishable, once durable, from a namespace that + /// stopped for no reason — and a hold on any other classification claims a stop that did not + /// happen. + if ((cov.classification == CoverageClass::Clamped) != cov.hold.has_value()) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS fold seal: coverage '{}' has classification {} and {} hold — the hold fields are " - "required for classification 4 and forbidden otherwise", - life_hex, cov.classification, cov.hold ? "a" : "no"); + "required for classification clamped and forbidden otherwise", + life_hex, classification_word, cov.hold ? "a" : "no"); /// A hold that names no position resolves itself on the next round (nothing sorts below /// `{0, 0}`) and cannot be rendered where the sweep reports why it retained a manifest. if (cov.hold && !isCanonicalHoldPosition(cov.hold->offending_position)) @@ -291,7 +313,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) bool first = true; writeStringField(out, FoldSealWire::kind, kRefLifeTag, first); writeHex128Field(out, FoldSealWire::life, life_id, first); - writeNumberField(out, FoldSealWire::classification, static_cast(cov.classification), first); + writeStringField(out, FoldSealWire::classification, classification_word, first); writeU64StringField(out, FoldSealWire::fold_epoch, cov.last_folded_ref_id.writer_epoch, first); writeU64StringField(out, FoldSealWire::fold_seq, cov.last_folded_ref_id.ref_sequence, first); if (cov.hold) @@ -308,7 +330,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) writeU64StringField(out, FoldSealWire::remove_seq, life_state.cleanup_evidence->remove_txn_id.ref_sequence, first); } closeObject(out, first); - closeLine("rfl"); + closeLine(kRefLifeTag); ++n; } @@ -318,7 +340,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const RunRef & r : runs) { writeRun(out, kBlobRunTag, r); - closeLine("btr"); + closeLine(kBlobRunTag); } } n += seal.blob_target_runs.size(); @@ -333,7 +355,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) writeNumberField(out, FoldSealWire::pending_total, s.pending_total, first); writeU64StringField(out, FoldSealWire::oldest_round, s.oldest_nonpending_condemn_round, first); closeObject(out, first); - closeLine("cnd"); + closeLine(kCondemnedTag); ++n; } @@ -395,21 +417,18 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect return seal; } if (key != FoldSealWire::kind) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"k\""); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: record must start with \"kind\""); const String kind = r.readString(); if (kind == kRefLifeTag) { std::optional life_id; RefCoverage cov; - /// Read WIDE and validated before it is narrowed to the persisted byte. `cls` is the field - /// every consumer branches on, and a plain `static_cast` maps 258 onto 2 ("all - /// records through the cursor were folded") and 256 onto 0 — a forged or damaged seal would - /// buy full coverage with an integer no reader ever sees. - std::optional classification; /// The hold fields are read individually so the grammar can be checked on WHICH of them /// arrived, not merely on how many. `JsonObjectReader` already rejects a duplicate key, so - /// a second `hr` can never quietly rewrite the reason. + /// a second `hold_reason` can never quietly rewrite the reason. + std::optional classification_word; + std::optional hold_reason_word; std::optional hold_reason; std::optional hold_epoch; std::optional hold_sequence; @@ -420,17 +439,20 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect while (r.nextKey(key)) { if (key == FoldSealWire::life) life_id = r.readHex128(); - else if (key == FoldSealWire::classification) classification = r.readU64Number(); + /// The two word-valued fields are collected as words and converted BELOW, once the row's + /// life id is known: a seal can carry thousands of rows, so an unknown word has to say + /// WHICH row carries it. + else if (key == FoldSealWire::classification) classification_word = r.readString(); else if (key == FoldSealWire::fold_epoch) cov.last_folded_ref_id.writer_epoch = r.readU64String(); else if (key == FoldSealWire::fold_seq) cov.last_folded_ref_id.ref_sequence = r.readU64String(); - else if (key == FoldSealWire::hold_reason) hold_reason = holdReasonFromWord(r.readString()); + else if (key == FoldSealWire::hold_reason) hold_reason_word = r.readString(); else if (key == FoldSealWire::hold_epoch) hold_epoch = r.readU64String(); else if (key == FoldSealWire::hold_seq) hold_sequence = r.readU64String(); else if (key == FoldSealWire::retries) hold_retry_count = r.readU32Number(); else if (key == FoldSealWire::retry_round) hold_next_retry_round = r.readU64String(); else if (key == FoldSealWire::remove_epoch) remove_txn_epoch = r.readU64String(); else if (key == FoldSealWire::remove_seq) remove_txn_sequence = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown rfl key '{}'", key); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown ref_life key '{}'", key); } if (!life_id || *life_id == 0) @@ -438,16 +460,13 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect "CAS fold seal: a ref-life row is missing a nonzero opaque life id"); const String life_hex = u128ToHex(*life_id); - /// `cls` is required, not defaulted: an absent one would read as 0 ("no round folded this - /// namespace"), which is a claim about a fold, not the absence of one. - if (!classification) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: rfl '{}' missing cls", life_hex); - if (!isKnownClassification(*classification)) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: coverage '{}' has classification {}, which is not one of the four " - "the fold grammar defines (0 absent, 1 unchanged, 2 folded, 4 clamped)", - life_hex, *classification); - cov.classification = static_cast(*classification); /// in range, so narrowing is exact + /// `class` is required, not defaulted: an absent one would read as `absent` ("no round + /// folded this namespace"), which is a claim about a fold, not the absence of one. + if (!classification_word) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: ref_life '{}' missing class", life_hex); + if (hold_reason_word) + hold_reason = holdReasonInRow(*hold_reason_word, life_hex); + cov.classification = coverageClassInRow(*classification_word, life_hex); /// The same strict grammar the encoder enforces, applied to bytes we did not write. A /// PARTIAL hold is corruption, never a hold with defaults: a hold whose offending position @@ -456,11 +475,11 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect || hold_retry_count || hold_next_retry_round; const bool every_hold_field = hold_reason && hold_epoch && hold_sequence && hold_retry_count && hold_next_retry_round; - if (cov.classification == 4) + if (cov.classification == CoverageClass::Clamped) { if (!every_hold_field) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: coverage '{}' is held (classification 4) but its hold is " + "CAS fold seal: coverage '{}' is held (classification clamped) but its hold is " "incomplete — reason, offending position, retry count and next retry round are " "all required", life_hex); /// PRESENT is not enough: the position must be one a fold can actually retry. `{0, 0}` @@ -480,8 +499,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect else if (any_hold_field) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: coverage '{}' carries hold fields at classification {} — they are " - "forbidden on anything but a held (classification 4) row", - life_hex, cov.classification); + "forbidden on anything but a held (classification clamped) row", + life_hex, coverageClassToWord(cov.classification)); const bool any_cleanup_field = remove_txn_epoch || remove_txn_sequence; const bool every_cleanup_field = remove_txn_epoch && remove_txn_sequence; @@ -518,7 +537,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect } if (!run_key || !checksum || !shard || !generation) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: btr requires key, ck, shard, and gen"); + "CAS fold seal: blob_run requires key, checksum, shard, and key_generation"); seal.blob_target_runs.push_back(RunRef{ .key = std::move(*run_key), .checksum = *checksum, .shard = *shard, .key_generation = *generation}); } @@ -534,11 +553,11 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect else if (key == FoldSealWire::condemned_total) condemned_total = r.readU64Number(); else if (key == FoldSealWire::pending_total) pending_total = r.readU64Number(); else if (key == FoldSealWire::oldest_round) oldest_nonpending_condemn_round = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown cnd key '{}'", key); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown condemned key '{}'", key); } if (!shard || !condemned_total || !pending_total || !oldest_nonpending_condemn_round) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS fold seal: cnd requires shard, ct, pt, and ocr"); + "CAS fold seal: condemned requires shard, condemned, pending, and oldest_round"); insertRecordOnce(seal.condemned_summary, *shard, CondemnedSummary{ .condemned_total = *condemned_total, .pending_total = *pending_total, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h index ea4200e29eea..02a043e9b2b2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.h @@ -36,10 +36,12 @@ struct RunRef /// correlating logs. Persisted as a word, so an unknown word is `CORRUPTED_DATA` rather than a silently /// reinterpreted integer. /// -/// THESE ARE WIRE VALUES, AND THEY ARE APPEND-ONLY. A durable seal written by one build is read by -/// another, so a renumbered value or a reused word makes an older seal describe a hold that is not the -/// one it recorded — and a hold's whole job is to say truthfully what stopped a namespace and where. -/// Add new reasons at the end; never renumber, never repurpose a retired word. +/// THE WORDS ARE THE WIRE, AND THE WORD VOCABULARY IS APPEND-ONLY. A durable seal written by one build +/// is read by another, so a reused or repurposed word makes an older seal describe a hold that is not +/// the one it recorded — and a hold's whole job is to say truthfully what stopped a namespace and +/// where. The enumerator NUMBERS never leave memory; they are constrained only by the wire table's +/// density-and-order proof, so inserting a value in the middle is a compile-time question, not a +/// durability one. Add new reasons freely; never reuse or repurpose a retired word. enum class HoldReason : uint8_t { GapBelowWitness = 1, /// 404 at the expected id with a durable witness above it, same epoch @@ -55,6 +57,34 @@ enum class HoldReason : uint8_t /// namespace — and a second rendering of these words elsewhere would be a second place for them to drift. std::string_view holdReasonToWord(HoldReason r); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. Paired with the renderer so a caller +/// that needs to prove the vocabulary round-trips does not have to reach for the table itself. +HoldReason holdReasonFromWord(std::string_view w); + +/// What the current round did for one life-keyed `CasFoldSeal::ref_lives` row. A BOUNDED enum: the type +/// itself is the closed set, so a producer cannot construct a fifth shape without an explicit cast, and +/// the decoder's wire-word lookup refuses anything else as `CORRUPTED_DATA` rather than silently +/// reinterpreting an integer. +/// +/// THE WORDS ARE THE WIRE, AND THE WORD VOCABULARY IS APPEND-ONLY, for the same reason `HoldReason`'s +/// is: a durable seal written by one build is read by another. The enumerator numbers never leave +/// memory. `Clamped` sits at 3, not the 4 a retired byte-valued wire used for it: nothing outside this +/// JSON ever persisted the raw byte, and a dense range is what makes the wire table's lookup a direct +/// index rather than a search. +enum class CoverageClass : uint8_t +{ + Absent = 0, /// no round has folded a ref cursor for this namespace + Unchanged = 1, /// folded, but nothing moved this round + Folded = 2, /// every record through the observed cursor was folded + Clamped = 3, /// folding stopped below the ref-log cursor; must be read again next round +}; + +/// The wire word one `CoverageClass` is persisted as; `fromWord` rejects anything else as +/// `CORRUPTED_DATA`. Exported for the same reason `holdReasonToWord` is: the classification is rendered +/// outside the codec too (`cas-inspect`, the sweep's retention messages). +std::string_view coverageClassToWord(CoverageClass c); +CoverageClass coverageClassFromWord(std::string_view w); + /// The durable hold on one namespace. It rides `RefCoverage` across rounds and across `REBUILD`, and /// clears ONLY by folding through `offending_position` and adopting the result in `gc/state` — never by /// observing another absent, because an absent is exactly the observation a lying store produces. @@ -87,21 +117,17 @@ struct RefHold bool operator==(const RefHold &) const = default; }; -/// Records what the current round did for one life-keyed `CasFoldSeal::ref_lives` row. -/// `classification` is a persisted byte: -/// 0 means absent, 1 means unchanged, 2 means all records through the observed cursor were folded, and 4 -/// means folding was clamped below the ref-log cursor. A clamped entry must be read again in the next -/// round, because an unfolded event may become foldable by then. +/// Records what the current round did for one life-keyed `CasFoldSeal::ref_lives` row. See +/// `CoverageClass` for what each value means. /// -/// THE SET {0, 1, 2, 4} IS CLOSED, and both codecs enforce it (decode `CORRUPTED_DATA`, encode -/// `LOGICAL_ERROR`). Every consumer branches on exact values — the sweep's §6 deletion premise refuses a -/// row by testing `== 4` and `== 0` — so an unrecognized byte is not a variant to tolerate: it passes -/// every refusal stated in terms of the set and reaches the delete. The decoder also validates BEFORE -/// narrowing to the byte, because a wide integer on the wire (258, say) truncates into the set and would -/// otherwise claim a coverage the fold never proved. +/// Every consumer branches on exact values — the sweep's §6 deletion premise refuses a row by testing +/// `== Clamped` and `== Absent` — so the CLOSED set matters beyond the codec, and it is now the type +/// itself: `CoverageClass` names exactly the four shapes, both codecs go through the shared wire table +/// (decode `CORRUPTED_DATA`, encode `LOGICAL_ERROR` on the one path that still reaches an out-of-range +/// value, an explicit cast), and an unrecognized wire word is refused before it ever reaches a consumer. struct RefCoverage { - uint8_t classification = 0; + CoverageClass classification = CoverageClass::Absent; /// The greatest `RefTxnId` whose owner changes have contributed their manifest-edge deltas. There is /// one ref-log stream per namespace life, so this cursor is stored in that life-keyed row. @@ -109,11 +135,11 @@ struct RefCoverage /// offending transaction so the complete transaction is retried rather than partially applied. RefTxnId last_folded_ref_id{}; - /// STRICT GRAMMAR: present if and only if `classification == 4`. Both directions enforce it — the - /// encoder refuses to write a classification-4 row without a hold (a clamp whose reason was lost is - /// indistinguishable from a clean cursor once it is durable) and refuses to write a hold on any - /// other classification (`LOGICAL_ERROR`); the decoder rejects both shapes as `CORRUPTED_DATA`. The - /// pairing lives in the type, not only in the codec, so no producer can construct the forbidden + /// STRICT GRAMMAR: present if and only if `classification == CoverageClass::Clamped`. Both directions + /// enforce it — the encoder refuses to write a clamped row without a hold (a clamp whose reason was + /// lost is indistinguishable from a clean cursor once it is durable) and refuses to write a hold on + /// any other classification (`LOGICAL_ERROR`); the decoder rejects both shapes as `CORRUPTED_DATA`. + /// The pairing lives in the type, not only in the codec, so no producer can construct the forbidden /// combination by forgetting a field. std::optional hold = std::nullopt; @@ -189,10 +215,10 @@ FoldSealCaps foldSealCaps(); void checkFoldSealObjectBytes(uint64_t encoded_bytes); /// Encodes a fold seal as a strict, raw text control object. The header and meta lines are followed by -/// tagged records in the fixed `rfl`/`btr`/`cnd` order and a record-count trailer. Map iteration and +/// tagged records in the fixed `ref_life`/`blob_run`/`condemned` order and a record-count trailer. Map iteration and /// run references are sorted so retries produce byte-identical output for write-once adoption. /// -/// Enforces the whole coverage grammar — the closed classification set, the classification-4 hold +/// Enforces the whole coverage grammar — the closed classification set, the clamped-classification hold /// pairing, and the hold's canonical offending position — and BOTH byte caps: every emitted line against /// `line_cap` — header, meta, records and trailer alike, with no exception — and the whole object /// against `object_cap`. Both PUT sites go through this function, so the gate cannot be bypassed by diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp index dc9665aff837..23b19ddc3508 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp @@ -18,57 +18,13 @@ namespace DB::Cas namespace { -/// Generation-1 baseline for every class. A future format change appends to that class's array and +/// Generation-1 baseline for every class. A future format change appends to that class's own array and /// bumps `G_BUILD`: additive changes use the previous reader floor, while breaking changes use the -/// new generation as the floor. Existing entries are immutable history. +/// new generation as the floor. Existing entries are immutable history. Every class currently shares +/// this baseline; a class that outgrows it gets its own named array again, the way the pre-reset +/// history once had. constexpr FormatChangePoint BASELINE[] = {{1, 1}}; -/// The two ref classes changed at generation 4 (INV-1, per-namespace contiguous ids) AND AGAIN at -/// generation 5 (Stage B's recreate-only "format bump B": the ref layer re-keyed under -/// `//`). Both changes are BREAKING even though not one byte of the encoding moved -/// either time -- a generation-3 stream's ids came from a pool-wide counter and legitimately skip, -/// which a generation-4 reader reports as corruption, and a generation-4 key names no incarnation at -/// all, which a generation-5 reader also reports as corruption (`Layout::parseRefObjectKey`). Each -/// floor is the change generation itself. -constexpr FormatChangePoint REF_STREAM[] = { - {1, 1}, - {kContiguousRefStreamsGeneration, kContiguousRefStreamsGeneration}, - {kNamespaceLifeKeyedGeneration, kNamespaceLifeKeyedGeneration}, - {kOpaqueNamespaceLifeLayoutGeneration, kOpaqueNamespaceLifeLayoutGeneration}, -}; - -/// `cas_ref_ckpt` is BORN at generation 4, so it has no generation-1 baseline to inherit: there is no -/// such thing as a generation-1 `_ckpt` object, and claiming one would say a generation-1 reader could -/// read it. Generation 5 re-keys it under `//` exactly like `REF_STREAM` above, for -/// the same reason and with the same floor. -constexpr FormatChangePoint REF_CKPT[] = { - {kContiguousRefStreamsGeneration, kContiguousRefStreamsGeneration}, - {kNamespaceLifeKeyedGeneration, kNamespaceLifeKeyedGeneration}, - {kOpaqueNamespaceLifeLayoutGeneration, kOpaqueNamespaceLifeLayoutGeneration}, - {kCommittedRefFrontierGeneration, kCommittedRefFrontierGeneration}, -}; - -/// `cas_ref_catalog` is BORN at generation 4, one generation BEFORE the bump that makes namespace -/// existence catalog-authoritative (Stage B's Task 4, "format bump B" -- `kNamespaceLifeKeyedGeneration`): -/// Task 2 introduced the catalog OBJECT while `G_BUILD` was still the value -/// `kContiguousRefStreamsGeneration` names, and Task 4 is the later change that actually wires -/// discovery to read it and bumps the floor. The catalog's own encoding is unaffected by that bump (it -/// reuses `kContiguousRefStreamsGeneration` as its birth generation, not a second constant named after -/// itself, for the same reason `REF_CKPT` originally did), so it carries no second change point here. -constexpr FormatChangePoint REF_CATALOG[] = {{kContiguousRefStreamsGeneration, kContiguousRefStreamsGeneration}}; -constexpr FormatChangePoint GC_MAINTENANCE_STATE[] = {{kUnifiedRefLifeFoldGeneration, kUnifiedRefLifeFoldGeneration}}; -constexpr FormatChangePoint POOL_META[] = { - {1, 1}, - {kPoolGcShardsGeneration, kPoolGcShardsGeneration}, - {kCommittedRefFrontierGeneration, kCommittedRefFrontierGeneration}, - {kMountWriteAttemptIdGeneration, kMountWriteAttemptIdGeneration}, -}; - -constexpr FormatChangePoint MOUNT_LEASE[] = { - {1, 1}, - {kMountWriteAttemptIdGeneration, kMountWriteAttemptIdGeneration}, -}; - } std::span changePoints(FormatId id) @@ -77,17 +33,11 @@ std::span changePoints(FormatId id) { case FormatId::RefLog: case FormatId::RefSnapshot: - return REF_STREAM; case FormatId::RefCkpt: - return REF_CKPT; case FormatId::RefCatalog: - return REF_CATALOG; case FormatId::GcMaintenanceState: - return GC_MAINTENANCE_STATE; case FormatId::PoolMeta: - return POOL_META; case FormatId::MountLease: - return MOUNT_LEASE; case FormatId::Blob: case FormatId::GcState: case FormatId::Roster: diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h index 1c98e4f283f6..7b287a01ae9d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h @@ -15,87 +15,10 @@ namespace DB::Cas /// compatibility_version <= G_BUILD. Bump this (and append a change-point in CasFormat.cpp) when a new /// format generation is introduced. /// -/// Generation 2 is the first generation that understands mixed-algorithm pools: the schema-3 -/// source-edge settlement key includes the algorithm prefix, so a generation-1 reader can open the -/// pool but cannot decode its GC state. Pool admission CAS-raises `min_reader_generation` to this -/// build's own floor (`G_BUILD`), and a persisted floor above `G_BUILD` fails closed. -/// -/// Generation 3 replaced mutable ref-shard objects with immutable `_log` and `_snap` objects. -/// -/// Generation 4 makes each namespace's ref-log ids per-namespace and CONTIGUOUS within a writer epoch -/// (INV-1). The bytes of a `_log`/`_snap` object did not change, but their MEANING did: a generation-3 -/// pool's ids were drawn from a pool-wide counter and are full of legitimate holes, which this build -/// reads as a truncated -- i.e. corrupt -- stream. The per-object forward gate cannot reject such a -/// pool (its version is not in the future), so pool-meta decoding applies -/// `kContiguousRefStreamsGeneration` as a backward floor. Pools below the floor must be recreated; -/// there is no migration path in the pre-release format. -/// -/// Generation 5 (Stage B's own recreate-only bump, the plan's "format bump B") re-keys the ref layer -/// under `//` (spec INV-3: the whole-pool namespace catalog mints the incarnation). -/// Again the bytes of `_log`/`_snap`/`_ckpt` objects did not change, but the KEY SHAPE they live under -/// did: a generation-4 key named a namespace directly (`cas/refs//_log/`), while this -/// generation's reader recognizes only the incarnation-qualified shape -/// (`cas/refs///_log/`) -- `Layout::parseRefObjectKey`/`parseRefCkptKey` already -/// refuse the un-incarnated shape with `CORRUPTED_DATA` (Stage B Tasks 1/1c landed that refusal ahead -/// of this bump, deliberately: the pre-release format carries zero persisted data and zero compat -/// obligation, so the key shapes and the bump that makes them the ONLY readable shape need not land in -/// the same commit). `kNamespaceLifeKeyedGeneration` is the backward floor for this change, applied the -/// same way `kContiguousRefStreamsGeneration` is. -/// -/// Generation 6 replaces that namespace-bearing grammar with opaque pool-wide life identifiers and -/// splits hot ref streams from point-read state: `cas/ns/stream//...` contains `_log`, `_snap` -/// while `cas/ns/state//...` contains `_ckpt` and `_files`. A generation-5 -/// pool must be recreated; no dual parser or copy-forward path exists. -/// -/// Generation 7 replaces the fold seal's independent namespace-keyed coverage and cleanup -/// collections with one opaque-life-keyed row and removes the retired terminal-marker object class. A generation-6 -/// pool must be recreated; there is no dual reader for the split grammar. -/// -/// Generation 8 persists the creation-time `gc_shards` authority in `_pool_meta`. Generation-7 pools -/// must be recreated because namespace admission can precede creation of `gc/state`; accepting a -/// metadata object without this field would leave different openers charging different seal bounds. -/// Generation 9 adds `_ckpt.committed_through`, the exact recovery frontier. Generation-8 pools -/// must be recreated: the absence of this field has the incompatible meaning that no transaction has -/// entered durable logical history. Generation 10 adds the required `write_attempt_id` to mount -/// leases. Generation-9 pools must be recreated because a missing attempt identity makes ambiguous -/// mount writes impossible to distinguish from a different body under the same writer incarnation. -constexpr uint32_t G_BUILD = 10; - -/// The pool-format generation at which ref-log ids became per-namespace and contiguous. Pool metadata -/// below this value cannot be opened, because its ref streams carry holes this build reports as -/// corruption; the backward-floor check is applied by `decodePoolMeta`. Named separately from `G_BUILD` -/// so a later generation that CAN still read a generation-4 pool does not silently move the floor with -/// it. -constexpr uint32_t kContiguousRefStreamsGeneration = 4; - -/// The pool-format generation at which the ref layer (and, per Stage B's Task 4b, namespace files) -/// became incarnation-scoped under `//`. Pool metadata below this value cannot be -/// opened: its ref-object keys carry no incarnation segment, which this build's parsers refuse as -/// corruption rather than read as a compatibility case (see the `G_BUILD` doc above). The backward- -/// floor check is applied by `decodePoolMeta`, exactly mirroring `kContiguousRefStreamsGeneration`; -/// named separately for the same reason that one is -- so a later generation that can still read a -/// generation-5 pool does not silently move this floor with it. Pools below the floor must be -/// recreated; there is no migration path in the pre-release format. -constexpr uint32_t kNamespaceLifeKeyedGeneration = 5; - -/// The recreate-only generation at which namespace text disappeared from physical life keys and hot -/// ref streams were separated from point-read namespace state. -constexpr uint32_t kOpaqueNamespaceLifeLayoutGeneration = 6; - -/// The recreate-only generation at which one unified ref-life row replaced the split coverage and -/// namespace-cleanup grammar. -constexpr uint32_t kUnifiedRefLifeFoldGeneration = 7; - -/// The recreate-only generation at which `_pool_meta` became the authority for `gc_shards`. -constexpr uint32_t kPoolGcShardsGeneration = 8; - -/// The recreate-only generation at which `_ckpt` gained its exact committed-transaction frontier. -constexpr uint32_t kCommittedRefFrontierGeneration = 9; - -/// The recreate-only generation at which mount leases gained their required durable holder-write -/// identity. The pool-level reader floor rejects every older pool before it can interpret a mount -/// body without this field. -constexpr uint32_t kMountWriteAttemptIdGeneration = 10; +/// The generation history was reset to this {1, 1} baseline: CAS is pre-release, carries no persisted +/// data, and so pays no compatibility cost for starting the count over. Every class's `changePoints` +/// begins at generation 1 until a future change appends a real entry. +constexpr uint32_t G_BUILD = 1; /// Stable identifiers for every self-describing persisted object class. The text registry uses the /// corresponding `type` string as the on-disk identity. Numeric values are part of the format history: @@ -151,20 +74,20 @@ void checkCompatibility(uint32_t compatibility_version, std::string_view what); /// One append-only entry in a class's format history. At `generation`, the class's ENCODING or the /// MEANING of what it encodes changed, and a reader must understand at least `min_reader` to read an /// object written at that generation. Additive changes retain the previous reader floor; breaking -/// changes set the floor to the change generation itself. Generation 4's ref-stream entry is the -/// worked example of the second kind: not one byte of `cas_ref_log` moved, but its ids became dense, -/// so an older stream is unreadable to this build and the floor is the change generation. +/// changes set the floor to the change generation itself — even when not one byte of the encoding +/// moves: a change that makes ids dense, for example, leaves the bytes readable but their MEANING +/// unreadable to an older build, so the floor is the change generation. struct FormatChangePoint { uint16_t generation; uint16_t min_reader; }; -/// Returns the append-only change-point history for `id`, oldest first. A class's history begins at -/// the generation it was BORN in, not at 1: the classes that existed from the start carry the frozen -/// `{1, 1}` baseline, while `RefCkpt` — introduced at generation 4 — begins at `{4, 4}`, because there -/// is no such thing as a generation-1 `_ckpt` and claiming one would say a generation-1 reader could -/// read it. Future changes append entries without editing old ones. +/// Returns the append-only change-point history for `id`, oldest first. After the pre-release +/// generation reset every class carries the shared `{1, 1}` baseline. A class born LATER than the +/// current baseline must begin its history at its birth generation, not at 1 — claiming an earlier +/// entry would say an older reader could read an object kind that did not yet exist. Future changes +/// append entries without editing old ones. std::span changePoints(FormatId id); /// The text-format registry has one row per decodable persisted object. Each row is the single source diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp index bc0720bf6f27..cc13021b020d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp @@ -15,7 +15,7 @@ namespace DB::Cas namespace GcMaintenanceWire { - constexpr WireKey janitor_cursor{"cur"}; + constexpr WireKey janitor_cursor{"janitor_cursor"}; } String encodeGcMaintenanceState(const GcMaintenanceState & state) @@ -59,7 +59,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data) reader.skipUnknown(key); } if (!has_cursor) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: missing cur"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: missing janitor_cursor"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc maintenance state: trailing bytes"); if (result.janitor_cursor.size() > kMaxGcMaintenanceCursorBytes) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index 9ee65ebafa78..1b19256d6d4b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -21,8 +21,8 @@ namespace namespace GcOutcomesWire { - constexpr WireKey kind{"k"}; - constexpr WireKey outcome{"oc"}; + constexpr WireKey kind{"kind"}; + constexpr WireKey outcome{"outcome"}; } constexpr EnumWireTable kOutcomeKindWords{{{ @@ -34,11 +34,6 @@ constexpr EnumWireTable kOutcomeKindWords{{{ static_assert(casEnumTableCoversEnum()); -OutcomeKind outcomeKindFromWord(std::string_view w) -{ - return kOutcomeKindWords.fromWord(w, "CAS outcome log outcome kind"); -} - } std::string_view outcomeKindToWireWord(OutcomeKind outcome) @@ -46,6 +41,11 @@ std::string_view outcomeKindToWireWord(OutcomeKind outcome) return kOutcomeKindWords.toWord(outcome, "CAS outcome log outcome kind"); } +OutcomeKind outcomeKindFromWireWord(std::string_view w) +{ + return kOutcomeKindWords.fromWord(w, "CAS outcome log outcome kind"); +} + String encodeOutcomeLog(const OutcomeLog & log) { CasJsonWriter out(256); @@ -54,8 +54,8 @@ String encodeOutcomeLog(const OutcomeLog & log) { bool first = true; writeStringField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); - writeBlobRefFields(out, first, e.ref); /// ha + h - writeTokenFields(out, first, e.token); /// tt + tv + writeBlobRefFields(out, first, e.ref); /// algo + digest + writeTokenFields(out, first, e.token); /// token_type + token writeStringField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); closeObject(out, first); writeChar('\n', out); @@ -78,7 +78,7 @@ OutcomeLog decodeOutcomeLog(std::string_view data) JsonObjectReader r(line_in, KeyStrictness::Tolerant, "outcome log"); String key; - /// The first key distinguishes a trailer ("n") from a record ("k"). + /// The first key distinguishes a trailer (`n`) from a record (`kind`). if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: empty line"); if (key == "n") @@ -102,14 +102,12 @@ OutcomeLog decodeOutcomeLog(std::string_view data) if (key == GcOutcomesWire::kind) e.kind = objectKindFromWord(r.readString(), "outcome log"); else if (matchBlobRefFields(key, r, blob_ref_fields)) {} else if (matchTokenFields(key, r, token_fields)) {} - else if (key == GcOutcomesWire::outcome) e.outcome = outcomeKindFromWord(r.readString()); + else if (key == GcOutcomesWire::outcome) e.outcome = outcomeKindFromWireWord(r.readString()); else r.skipUnknown(key); } while (r.nextKey(key)); - if (!blob_ref_fields.algo_word || !blob_ref_fields.digest_hex || !token_fields.type_word) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: record missing ha/h/tt"); e.ref = blob_ref_fields.build("outcome log"); - e.token = Token{token_fields.value.value_or(""), tokenTypeFromWord(*token_fields.type_word, "outcome log")}; + e.token = token_fields.build("outcome log"); if (!line_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS outcome log: junk after record"); log.entries.push_back(std::move(e)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index 474c3c136a0b..edefc816cff2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -30,6 +30,9 @@ enum class OutcomeKind : uint8_t /// Canonical wire word for one `OutcomeKind`. std::string_view outcomeKindToWireWord(OutcomeKind outcome); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +OutcomeKind outcomeKindFromWireWord(std::string_view w); + /// One observation about a blob incarnation considered by GC. `token` identifies the exact /// incarnation that GC examined, while `ref` identifies the content address; retaining both lets /// replay and inspection distinguish an absent object from a replacement that won a race with GC. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp index b70f016cbae0..5cc268a4d9c3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp @@ -18,20 +18,20 @@ namespace DB::Cas namespace GcStateWire { - constexpr WireKey round{"rnd"}; - constexpr WireKey gc_shards{"gcs"}; - constexpr WireKey snap_generation{"sg"}; - constexpr WireKey snap_pruned_through{"spt"}; - constexpr WireKey snap_attempt{"sa"}; - constexpr WireKey manifest_sweep_cursor{"msc"}; - constexpr WireKey lease_owner{"lo"}; - constexpr WireKey lease_seq{"ls"}; + constexpr WireKey round{"round"}; + constexpr WireKey gc_shards{"gc_shards"}; + constexpr WireKey snap_generation{"snap_generation"}; + constexpr WireKey snap_pruned_through{"snap_pruned_through"}; + constexpr WireKey snap_attempt{"snap_attempt"}; + constexpr WireKey manifest_sweep_cursor{"manifest_sweep_cursor"}; + constexpr WireKey lease_owner{"lease_owner"}; + constexpr WireKey lease_seq{"lease_seq"}; } namespace GcHeartbeatWire { - constexpr WireKey owner{"by"}; - constexpr WireKey hb_seq{"seq"}; + constexpr WireKey owner{"owner"}; + constexpr WireKey hb_seq{"hb_seq"}; } String encodeGcState(const GcState & state) @@ -89,10 +89,10 @@ GcState decodeGcState(std::string_view data) else r.skipUnknown(key); } - /// Fail closed on an absent gcs: the writer always emits it, so a missing key means a corrupt object. + /// Fail closed on an absent gc_shards: the writer always emits it, so a missing key means a corrupt object. /// Do NOT silently keep the struct default (1) — that would hide corruption (no-fallback principle). if (!saw_gcs) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gcs"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gc_shards"); if (state.gc_shards == 0) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: gc_shards must be >= 1"); if (!body_in.eof() || !in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h index 88bce088ed13..ca38e649b82c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.h @@ -45,7 +45,7 @@ String encodeGcState(const GcState & state); /// Decode a complete `cas_gc_state` text object. The header and size limits are checked before the /// body is parsed; unknown non-reserved fields are tolerated for forward evolution, but malformed -/// input, trailing bytes, a missing `gcs`, or a zero shard count raises `CORRUPTED_DATA` rather than +/// input, trailing bytes, a missing `gc_shards`, or a zero shard count raises `CORRUPTED_DATA` rather than /// falling back to a default state. GcState decodeGcState(std::string_view data); @@ -53,7 +53,7 @@ GcState decodeGcState(std::string_view data); /// independently of round progress, because its lease renewal counter can remain unchanged during a /// long fold. A follower that observes the heartbeat advance backs off from stealing the lease; this /// prevents mistaking an alive, mid-round leader for a stalled one. The value is persisted as the -/// versioned `cas_gc_hb` text object, whose body contains `by` and `seq` string values, replacing the +/// versioned `cas_gc_hb` text object, whose body contains `owner` and `hb_seq` string values, replacing the /// former unversioned 24-byte record. struct GcHeartbeat { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp index 5bd928ec01a3..9466a8af735e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp @@ -200,8 +200,8 @@ NamespaceLifePhysicalId Layout::namespaceLifePhysicalIdOf(std::string_view key, if (!incarnation) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "CasLayout: object '{}' names no life: '{}' is not 32 lower-case hex digits of a nonzero " - "life id. Generation-5 namespace-bearing pools are rejected by the pool-metadata format " - "gate before this generation-6 physical-key parser is reached", + "life id. Pools whose keys predate the opaque-life layout are rejected by the " + "pool-metadata format gate before this physical-key parser is reached", key, segment); return *incarnation; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index 3e406ad01203..e8ff121dfee7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -25,12 +25,11 @@ namespace namespace PartManifestWire { - constexpr WireKey ns{"ns"}; - constexpr WireKey payload_digest{"pd"}; - constexpr WireKey path{"p"}; - constexpr WireKey place{"pm"}; - constexpr WireKey size{"sz"}; - constexpr WireKey inline_size{"il"}; + constexpr WireKey ns{"root_namespace"}; + constexpr WireKey payload_digest{"payload_digest"}; + constexpr WireKey path{"path"}; + constexpr WireKey place{"place"}; + constexpr WireKey size{"size"}; } constexpr EnumWireTable kEntryPlacementWords{{{ @@ -40,7 +39,8 @@ constexpr EnumWireTable kEntryPlacementWords{{{ static_assert(casEnumTableCoversEnum()); -/// One entry-record line: {"p","pm", then either the Blob's "ha"/"h"/"sz" or the Inline's "il"}. +/// One entry-record line: `path`/`place`, followed by either `algo`/`digest`/`size` for a Blob or +/// `size` for Inline bytes. void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) { bool first = true; @@ -48,12 +48,12 @@ void writeEntryRecord(CasJsonWriter & out, const ManifestEntry & e) writeWordField(out, PartManifestWire::place, entryPlacementToWireWord(e.placement), first); if (e.placement == EntryPlacement::Blob) { - writeBlobRefFields(out, first, e.ref); /// ha + h + writeBlobRefFields(out, first, e.ref); /// algo + digest writeNumberField(out, PartManifestWire::size, e.blob_size, first); } else { - writeNumberField(out, PartManifestWire::inline_size, e.inline_bytes.size(), first); + writeNumberField(out, PartManifestWire::size, e.inline_bytes.size(), first); } closeObject(out, first); writeChar('\n', out); @@ -69,7 +69,7 @@ String bannerFor(std::string_view path, uint64_t n) CasJsonWriter w(path.size() + 32); w.append("==> "); w.stringValue(path); - w.append(" il="); + w.append(" size="); w.u64Number(n); w.append(" <=="); return std::move(w).take(); @@ -82,6 +82,11 @@ std::string_view entryPlacementToWireWord(EntryPlacement placement) return kEntryPlacementWords.toWord(placement, "PartManifest: EntryPlacement"); } +EntryPlacement entryPlacementFromWireWord(std::string_view w) +{ + return kEntryPlacementWords.fromWord(w, "PartManifest: EntryPlacement"); +} + String encodePartManifest(const PartManifest & m) { /// Canonical path order plus duplicate-path rejection makes the encoded record sequence @@ -99,7 +104,7 @@ String encodePartManifest(const PartManifest & m) CasJsonWriter out(256); writeHeaderLine(out, FormatId::PartManifest); - /// descriptor meta line: ManifestRef (me/mb/mo, shared rendering with refsnaplog) + root + /// descriptor meta line: ManifestRef (epoch/build/ord, shared rendering with refsnaplog) + root /// namespace + payload digest. { bool first = true; @@ -156,9 +161,9 @@ PartManifest decodePartManifest(std::string_view data) else r.skipUnknown(key); } if (!ns) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing ns"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing root_namespace"); if (!pd) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing pd"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing payload_digest"); m.ref = fields.buildRef("PartManifest", "descriptor"); m.root_namespace_id = RootNamespace(*ns); m.payload_digest = *pd; @@ -166,7 +171,7 @@ PartManifest decodePartManifest(std::string_view data) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after descriptor line"); } - /// entry record lines, until the trailer. Inline entries remember their declared `il` length so + /// entry record lines, until the trailer. Inline entries remember their declared `size` so /// the payload zone below can read exactly that many raw bytes back into `inline_bytes`. /// Index-aligned with `m.entries` (Blob entries push an unused 0 placeholder). std::vector inline_lens; @@ -194,7 +199,7 @@ PartManifest decodePartManifest(std::string_view data) } if (key != PartManifestWire::path) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"p\""); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"path\""); ManifestEntry e; e.path = r.readString(); @@ -214,38 +219,36 @@ PartManifest decodePartManifest(std::string_view data) std::optional pm; BlobRefFields blob_ref; - std::optional sz; - std::optional il; + std::optional size; while (r.nextKey(key)) { if (key == PartManifestWire::place) pm = r.readString(); else if (matchBlobRefFields(key, r, blob_ref)) {} - else if (key == PartManifestWire::size) sz = r.readU64Number(); - else if (key == PartManifestWire::inline_size) il = r.readU64Number(); + else if (key == PartManifestWire::size) size = r.readU64Number(); else r.skipUnknown(key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: junk after record"); if (!pm) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing pm", e.path); - e.placement = kEntryPlacementWords.fromWord(*pm, "PartManifest"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: entry '{}' missing place", e.path); + e.placement = entryPlacementFromWireWord(*pm); if (e.placement == EntryPlacement::Blob) { - if (!sz) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing sz", e.path); + if (!size) + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: blob entry '{}' missing size", e.path); blob_ref_what.assign("PartManifest entry '"); blob_ref_what += e.path; blob_ref_what += '\''; e.ref = blob_ref.build(blob_ref_what); - e.blob_size = *sz; + e.blob_size = *size; inline_lens.push_back(0); /// unused for Blob; keeps inline_lens index-aligned with entries } else { - if (!il) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: inline entry '{}' missing il", e.path); - inline_lens.push_back(*il); /// bytes filled from the payload zone below + if (!size) + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: inline entry '{}' missing size", e.path); + inline_lens.push_back(*size); /// bytes filled from the payload zone below } /// Canonical ascending-order and no-duplicate-path enforcement: compare only against the @@ -263,7 +266,7 @@ PartManifest decodePartManifest(std::string_view data) } /// payload zone: for each Inline entry, in the same order it appeared above, a banner line then - /// exactly `il` raw bytes then a terminating '\n'. + /// exactly `size` raw bytes then a terminating '\n'. for (size_t i = 0; i < m.entries.size(); ++i) { if (m.entries[i].placement != EntryPlacement::Inline) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h index e02e1ce6a917..a26597e1bebe 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.h @@ -15,14 +15,14 @@ namespace DB::Cas /// stable for the surrounding CAS protocol. /// /// header line {"type":"cas_part_manifest","v":N} -/// descriptor meta line {"me","mb","mo"} (the ManifestRef, shared rendering with -/// refsnaplog, `CasWireVocab.h`) + "ns" (root namespace) + "pd" +/// descriptor meta line {"epoch","build","ord"} (the ManifestRef, shared rendering with +/// refsnaplog, `CasWireVocab.h`) + `root_namespace` + `payload_digest` /// (payload digest, 32 lowercase hex) -/// one entry-record line each {"p":path,"pm":placement-word, then either the Blob's -/// {"ha","h","sz"} or the Inline's {"il"}}, in canonical path order +/// one entry-record line each {"path":path,"place":placement-word, then either the Blob's +/// {"algo","digest","size"} or the Inline's {"size"}}, in canonical path order /// trailer line {"n":entry-count} /// PAYLOAD ZONE (raw, follows the trailer): for each Inline entry, in path order, a -/// `head -v`-style banner line `==> "" il= <==\n`, then +/// `head -v`-style banner line `==> "" size= <==\n`, then /// exactly `n` raw bytes, then `\n`. The path uses the same writer as /// the entry-record line, so decode can rebuild the banner byte-wise. /// Blob entries carry no @@ -45,6 +45,9 @@ enum class EntryPlacement : uint8_t /// Canonical wire word for one manifest entry placement. std::string_view entryPlacementToWireWord(EntryPlacement placement); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +EntryPlacement entryPlacementFromWireWord(std::string_view w); + /// One file entry inside a part manifest. `ref` is meaningful only for `Blob`; `inline_bytes` only /// for `Inline`. `blob_size` is the raw `Blob` byte count (0 for `Inline` — decode never fills it for /// an inline entry, since the wire format carries no redundant size for inline bytes). Use `size()` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp index 4e46131779cb..80f9572c4547 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp @@ -4,6 +4,7 @@ #include #include #include +#include namespace DB { @@ -19,35 +20,26 @@ namespace DB::Cas namespace PoolMetaWire { - constexpr WireKey pool_id{"pid"}; - constexpr WireKey blob_header_len{"hln"}; - constexpr WireKey gc_shards{"gcs"}; - constexpr WireKey min_reader_generation{"mrg"}; - constexpr WireKey algos_used{"alg"}; + constexpr WireKey pool_id{"pool_id"}; + constexpr WireKey blob_header_len{"blob_header_len"}; + constexpr WireKey gc_shards{"gc_shards"}; + constexpr WireKey min_reader_generation{"min_reader_generation"}; + constexpr WireKey algos_used{"algos_used"}; } -/// Minimum `blob_header_len` that provably fits the v3 `cas_blob` JSON envelope's mandatory (always- -/// written) non-ref fields, computed at type maxima from `encodeEnvelopeHeader` (CasBlobEnvelopeFormat.cpp): -/// {"type":"cas_blob" 18 -/// ,"v": 5 + 10 (currentCompatibilityVersion) 15 -/// ,"tag":"<32 hex>" 7 + 34 41 -/// ,"bld":"<32 hex>" 7 + 34 41 -/// ,"ts": 6 + 20 (created_at_ms) 26 -/// ,"by":"<32 hex>" 7 + 34 41 -/// ,"op":"" 6 + 10 (longest op word "mutation") 16 -/// ,"ch": 6 + 10 (VERSION_INTEGER) 16 -/// non-ref JSON = 214 bytes -/// The encoder then always frames the ref: `,"ref":` (7) + `""` (2) + `}` (1), and reserves byte -/// blob_header_len-1 for '\n' (1) = 11 bytes. So the mandatory content needs 214 + 11 = 225 bytes; -/// below that, encodeEnvelopeHeader throws LOGICAL_ERROR on the FIRST blob write (the old drop-and-retry -/// that used to mask this is gone). We floor at 240 (a multiple of 8 comfortably above 225, leaving -/// >= 15 bytes for the diagnostic ref even at type maxima, and well under the 256 default) so a -/// misconfigured pool fails at CREATION with BAD_ARGUMENTS, not at first write with LOGICAL_ERROR. +/// Minimum `blob_header_len` that provably fits the `cas_blob` JSON envelope's mandatory-descriptor +/// worst case. The byte-for-byte derivation (`kMandatoryDescriptorWorstCase`, currently 239 bytes) lives +/// beside the envelope key constants in `CasBlobEnvelopeFormat.cpp`, next to the compile-time proof that +/// it fits under this floor; below that bound, `encodeEnvelopeHeader` throws `LOGICAL_ERROR` on the +/// FIRST blob write (the old drop-and-retry that used to mask this is gone). We floor at 240 (a +/// multiple of 8 comfortably above the worst case, leaving at least one byte for the diagnostic `ref` +/// even at type maxima, and well under the 256 default) so a misconfigured pool fails at CREATION with +/// `BAD_ARGUMENTS`, not at first write with `LOGICAL_ERROR`. void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what) { if (blob_header_len < kMinBlobHeaderLen) - throw Exception(error_code, "CAS {}: blob_header_len must be >= {} (v3 envelope minimum), got {}", + throw Exception(error_code, "CAS {}: blob_header_len must be >= {} (blob envelope minimum), got {}", what, kMinBlobHeaderLen, blob_header_len); if (blob_header_len % 8 != 0) throw Exception(error_code, "CAS {}: blob_header_len must be a multiple of 8, got {}", what, blob_header_len); @@ -62,9 +54,8 @@ void validatePoolAlgosUsed(const std::vector & algos_used, int error_co for (size_t i = 0; i < algos_used.size(); ++i) { /// A direct membership scan, not `blobHashAlgoName`: that throws `LOGICAL_ERROR`, which - /// aborts at construction under a sanitizer/debug build before any catch can run, but a - /// persisted `algos_used` byte is exactly the unvalidated input this function must reject - /// cleanly instead. + /// aborts at construction under a sanitizer/debug build before any catch can run, but + /// this function validates a raw byte vector, so it must reject cleanly rather than abort. bool known = false; for (const auto & entry : kBlobHashAlgoWords.entries) if (static_cast(entry.value) == algos_used[i]) @@ -80,6 +71,8 @@ void validatePoolAlgosUsed(const std::vector & algos_used, int error_co String encodePoolMeta(const PoolMeta & pm) { + validatePoolAlgosUsed(pm.algos_used, ErrorCodes::CORRUPTED_DATA, "pool meta"); + CasJsonWriter out(256); writeHeaderLine(out, FormatId::PoolMeta); @@ -88,18 +81,13 @@ String encodePoolMeta(const PoolMeta & pm) writeNumberField(out, PoolMetaWire::blob_header_len, pm.blob_header_len, first); writeNumberField(out, PoolMetaWire::gc_shards, pm.gc_shards, first); writeNumberField(out, PoolMetaWire::min_reader_generation, pm.min_reader_generation, first); - writeKey(out, PoolMetaWire::algos_used, first); - { - /// Comma-joined algo words (tiny list, <=3): "ch128" or "ch128,sha256". - String joined; - for (size_t i = 0; i < pm.algos_used.size(); ++i) - { - if (i != 0) - joined += ','; - joined += blobHashAlgoName(static_cast(pm.algos_used[i])); - } - writeStringValue(out, joined); - } + /// Sized by the whole algo vocabulary and safe to index by `algos_used`: the validation above + /// admits only known algo bytes in strictly increasing order, so the vector cannot be longer + /// than the table. Relaxing that check to non-strict ordering would overrun this array. + std::array algo_words; + for (size_t i = 0; i < pm.algos_used.size(); ++i) + algo_words[i] = kBlobHashAlgoWords.toWord(static_cast(pm.algos_used[i]), "CAS pool meta"); + writeWordArrayField(out, PoolMetaWire::algos_used, std::span{algo_words}.first(pm.algos_used.size()), first); closeObject(out, first); writeChar('\n', out); @@ -111,18 +99,14 @@ PoolMeta decodePoolMeta(std::string_view data) ReadBufferFromMemory in(data.data(), data.size()); const TextHeader header = expectHeaderLine(in, FormatId::PoolMeta); - /// An older pool predates a breaking ref-layer change this build cannot reconcile, so - /// reject it before reading the metadata body. Writers always emit the current generation, while - /// `expectHeaderLine` separately rejects a future generation that this build cannot understand. - /// Generation 10 is the latest recreate-only authority floor and rejects old pools before any - /// mount lease body lacking its durable write-attempt identity can be interpreted. - if (header.v < kMountWriteAttemptIdGeneration) + /// The format-generation baseline is 1; a header below it cannot have been written by any build + /// this codec understands. `expectHeaderLine` above already rejects the symmetric FUTURE case + /// (`v > G_BUILD`); reject the backward case here, before the metadata body is read. + if (header.v < 1) throw Exception(ErrorCodes::UNKNOWN_FORMAT_VERSION, - "CAS pool format {} predates generation-10 mount-attempt-identity floor; recreate the pool. " - "This build requires the durable mount write attempt identity " - "in the generation-10 format " - "(generation {}+), and CAS is pre-release: there is no in-place migration.", - header.v, kMountWriteAttemptIdGeneration); + "CAS pool format {} predates the format-generation baseline; recreate the pool " + "(CAS is pre-release, so there is no in-place migration)", + header.v); const String body = readLine(in, traitsFor(FormatId::PoolMeta).line_cap, "pool meta"); ReadBufferFromMemory body_in(body.data(), body.size()); @@ -150,27 +134,16 @@ PoolMeta decodePoolMeta(std::string_view data) pm.min_reader_generation = r.readU64Number(); else if (key == PoolMetaWire::algos_used) { - const String joined = r.readString(); - size_t start = 0; - while (start <= joined.size()) - { - const size_t comma = joined.find(',', start); - const String word = joined.substr(start, comma == String::npos ? String::npos : comma - start); - if (word.empty()) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: empty algo word in '{}'", joined); + for (const String & word : r.readStringArray()) pm.algos_used.push_back(static_cast(blobHashAlgoFromWord(word, "pool meta algo"))); - if (comma == String::npos) - break; - start = comma + 1; - } } else r.skipUnknown(key); } if (!saw_pid) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing pid"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing pool_id"); if (!saw_gc_shards || pm.gc_shards == 0) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing or zero gcs"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: missing or zero gc_shards"); if (!body_in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS pool meta: junk after body object"); if (!in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h index 2ca0894d2f01..e25e767fb5e3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h @@ -14,8 +14,9 @@ class Backend; class Layout; /// `_pool_meta` — the pool identity and the pool-wide constants that every reader and writer must -/// agree on. The v3 text representation is a header line followed by one JSON body object: -/// {"pid":"<32hex>","hln":,"mrg":,"alg":""}. +/// agree on. The text representation is a header line followed by one JSON body object: +/// {"pool_id":"<32hex>","blob_header_len":,"gc_shards":, +/// "min_reader_generation":,"algos_used":["",...]}. /// /// The persisted object is authoritative after creation. On reopen, `createOrValidate` uses its /// `blob_header_len` and reader-generation floor rather than replacing them with local configuration; @@ -67,7 +68,7 @@ struct PoolMeta /// Serializes valid pool metadata as the versioned `_pool_meta` text object. The output includes the /// format header, one JSON body line, and its terminating newline; it is suitable for a conditional -/// backend write and preserves the sorted algorithm set as comma-separated vocabulary words. +/// backend write and preserves the sorted algorithm set as a JSON array of vocabulary words. String encodePoolMeta(const PoolMeta &); /// Parses and validates a persisted `_pool_meta` object. Unknown JSON keys are tolerated for additive @@ -76,11 +77,12 @@ String encodePoolMeta(const PoolMeta &); /// corruption or compatibility error code. PoolMeta decodePoolMeta(std::string_view); -/// Checks the fixed blob-envelope size invariant. The length must be 8-byte aligned, at most 16 KiB, -/// and at least 240 bytes: v3's mandatory envelope fields, framing, and newline consume 225 bytes at -/// type maxima, while 240 leaves room for a diagnostic `ref`. The caller supplies the error code so -/// persisted violations can be reported as `CORRUPTED_DATA` and bad creation arguments as -/// `BAD_ARGUMENTS`. +/// Checks the fixed blob-envelope size invariant: 8-byte aligned, at most 16 KiB, and at least +/// `kMinBlobHeaderLen`. That floor and the worst case it must clear are derived once beside the +/// envelope encoder, which also proves the relation at compile time — no number is restated here, +/// because a second copy is exactly what a single owner exists to prevent. The caller supplies the +/// error code so persisted violations can be reported as `CORRUPTED_DATA` and bad creation arguments +/// as `BAD_ARGUMENTS`. void validatePoolBlobHeaderLen(uint64_t blob_header_len, int error_code, std::string_view what); /// Checks that every admitted hash algorithm is known, that the set is non-empty, and that its numeric diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index 985c1964e9e5..857e428538f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -22,13 +22,13 @@ namespace namespace RunWire { - constexpr WireKey ref{"b"}; - constexpr WireKey src{"s"}; - constexpr WireKey mark{"m"}; - constexpr WireKey pending{"pend"}; - constexpr WireKey size{"sz"}; - constexpr WireKey condemn_round{"cr"}; - constexpr WireKey confirmed{"mc"}; + constexpr WireKey ref{"ref"}; + constexpr WireKey src{"src"}; + constexpr WireKey mark{"mark"}; + constexpr WireKey pending{"pending"}; + constexpr WireKey size{"size"}; + constexpr WireKey condemn_round{"condemn_round"}; + constexpr WireKey confirmed{"confirmed"}; } constexpr EnumWireTable kRunMarkerWords{{{ @@ -66,8 +66,8 @@ BlobHashAlgo algoFromByte(uint8_t b, std::string_view what) } } -/// `b` = the algo byte as two lowercase hex chars, then the digest hex at the algo's width. The algo -/// byte leads so that string-sorting `b` reproduces the binary (algo, digest) byte order. +/// `ref` = the algo byte as two lowercase hex chars, then the digest hex at the algo's width. The +/// algo byte leads so that string-sorting `ref` reproduces the binary (algo, digest) byte order. String renderB(const BlobRef & ref) { static constexpr char H[] = "0123456789abcdef"; @@ -108,6 +108,11 @@ std::string_view runMarkerToWireWord(RunMarker marker) return kRunMarkerWords.toWord(marker, "CAS cas_run: RunMarker"); } +RunMarker runMarkerFromWireWord(std::string_view w) +{ + return kRunMarkerWords.fromWord(w, "CAS cas_run: RunMarker"); +} + void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) { const FormatTraits & t = traitsFor(FormatId::RunFile); @@ -188,7 +193,7 @@ void SourceEdgeRunWriter::append(const SourceEdgeRecord & rec) if (rec.marker == RunMarker::Condemned) { writeBoolField(scratch, RunWire::pending, rec.delete_pending, first); - writeTokenFields(scratch, first, rec.token); /// tt + tv + writeTokenFields(scratch, first, rec.token); /// token_type + token writeNumberField(scratch, RunWire::size, rec.size, first); writeU64StringField(scratch, RunWire::condemn_round, rec.condemn_round, first); writeBoolField(scratch, RunWire::confirmed, rec.marker_confirmed, first); @@ -265,38 +270,36 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) SourceEdgeRecord out; String b; TokenFields token_fields; - bool have_b = false; - bool have_s = false; - bool have_m = false; - bool have_pend = false; - bool have_tt = false; - bool have_tv = false; - bool have_sz = false; - bool have_cr = false; - bool have_mc = false; + bool have_ref = false; + bool have_src = false; + bool have_mark = false; + bool have_pending = false; + bool have_size = false; + bool have_condemn_round = false; + bool have_confirmed = false; do { - if (key == RunWire::ref) { b = r.readString(); have_b = true; } - else if (key == RunWire::src) { out.source_id = r.readHex128(); have_s = true; } - else if (key == RunWire::mark) { out.marker = kRunMarkerWords.fromWord(r.readString(), "CAS cas_run"); have_m = true; } - else if (key == RunWire::pending) { out.delete_pending = r.readBool(); have_pend = true; } - else if (matchTokenFields(key, r, token_fields)) { have_tt = token_fields.type_word.has_value(); have_tv = token_fields.value.has_value(); } - else if (key == RunWire::size) { out.size = r.readU64Number(); have_sz = true; } - else if (key == RunWire::condemn_round) { out.condemn_round = r.readU64String(); have_cr = true; } - else if (key == RunWire::confirmed) { out.marker_confirmed = r.readBool(); have_mc = true; } + if (key == RunWire::ref) { b = r.readString(); have_ref = true; } + else if (key == RunWire::src) { out.source_id = r.readHex128(); have_src = true; } + else if (key == RunWire::mark) { out.marker = runMarkerFromWireWord(r.readString()); have_mark = true; } + else if (key == RunWire::pending) { out.delete_pending = r.readBool(); have_pending = true; } + else if (matchTokenFields(key, r, token_fields)) {} + else if (key == RunWire::size) { out.size = r.readU64Number(); have_size = true; } + else if (key == RunWire::condemn_round) { out.condemn_round = r.readU64String(); have_condemn_round = true; } + else if (key == RunWire::confirmed) { out.marker_confirmed = r.readBool(); have_confirmed = true; } else r.skipUnknown(key); /// Strict => any unknown key is CORRUPTED_DATA } while (r.nextKey(key)); - if (!have_b || !have_s || !have_m) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing b/s/m"); + if (!have_ref || !have_src || !have_mark) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: record missing ref/src/mark"); out.ref = parseB(b); if (out.marker == RunMarker::Condemned) { - if (!have_pend || !have_tt || !have_tv || !have_sz || !have_cr || !have_mc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pend/tt/tv/sz/cr/mc"); - out.token = Token{*token_fields.value, tokenTypeFromWord(*token_fields.type_word, "cas_run")}; + if (!have_pending || !have_size || !have_condemn_round || !have_confirmed) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: condemned record missing pending/size/condemn_round/confirmed"); + out.token = token_fields.build("cas_run"); } - else if (have_pend || have_tt || have_tv || have_sz || have_cr || have_mc) + else if (have_pending || token_fields.type_word || token_fields.value || have_size || have_condemn_round || have_confirmed) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_run: non-condemned record carries condemned fields"); if (!line_in.eof()) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index 0d9c6dfdad4b..fcc216b121ee 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -62,18 +62,18 @@ inline RunMarker runMarkerFromByte(char byte, std::string_view what) /// leaking into the format implementation. /// /// File shape: -/// {"type":"cas_run","v":3,"kind":"source_edge"} header line (type + v + kind gate) -/// {"b":"01","s":"<32hex>","m":"edge"} an active-edge / zero-marker row -/// {"b":"01","s":"00000000000000000000000000000000","m":"condemned","pend":false,"tt":"etag","tv":"...","sz":123,"cr":"7","mc":false} +/// {"type":"cas_run","v":1,"kind":"source_edge"} header line (type + v + kind gate) +/// {"ref":"01","src":"<32hex>","mark":"edge"} an active-edge / zero-marker row +/// {"ref":"01","src":"00000000000000000000000000000000","mark":"condemned","pending":false,"token_type":"etag","token":"...","size":123,"condemn_round":"7","confirmed":false} /// {"n":184267} trailer: record count /// -/// The record key `b` is the algo BYTE as two lowercase hex chars followed by the digest hex at the -/// algo's width; `s` is the 32-hex source id. String-sorting records by (b, s) reproduces the current +/// The record key `ref` is the algo BYTE as two lowercase hex chars followed by the digest hex at the +/// algo's width; `src` is the 32-hex source id. String-sorting records by (`ref`, `src`) reproduces the current /// `(algorithm, digest, source_id)` byte order (lowercase hex preserves unsigned byte order and the /// algorithm byte is emitted first) — the invariant the fold's two-cursor merge depends on. The row-tag word -/// `m` maps to the `RunMarker` bytes; a `condemned` row additionally -/// carries the retired incarnation (`pend`/`tt`/`tv`/`sz`/`cr`) and the durable condemn-marker -/// confirmation bit (`mc`). +/// `mark` maps to the `RunMarker` bytes; a `condemned` row additionally +/// carries the retired incarnation (`pending`/`token_type`/`token`/`size`/`condemn_round`) and the durable condemn-marker +/// confirmation bit (`confirmed`). /// One decoded source-edge row. All fields are identifier-layer types so the codec stays backend-free. /// The condemned-only fields (`delete_pending`/`token`/`size`/`condemn_round`/`marker_confirmed`) are @@ -96,6 +96,9 @@ inline constexpr std::string_view kSourceEdgeKindWord = "source_edge"; /// Canonical wire word for one source-edge run marker. std::string_view runMarkerToWireWord(RunMarker marker); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +RunMarker runMarkerFromWireWord(std::string_view w); + /// Write the typed header line `{"type":"cas_run","v":G_BUILD,"kind":""}\n` with a fixed key /// order for byte-determinism. The `kind` field distinguishes the record schema within the run /// family, so a reader can reject a valid run of the wrong kind before interpreting any records. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index 66606ef5b632..f2cea307342c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -26,17 +26,17 @@ namespace namespace RefCatalogWire { - constexpr WireKey kind{"k"}; + constexpr WireKey kind{"kind"}; constexpr WireKey ns{"ns"}; - constexpr WireKey state{"st"}; - constexpr WireKey life{"inc"}; - constexpr WireKey remove_round{"rsr"}; - constexpr WireKey creator{"csr"}; - constexpr WireKey creator_epoch{"cwe"}; - constexpr WireKey creator_fence{"cfg"}; + constexpr WireKey state{"state"}; + constexpr WireKey life{"life"}; + constexpr WireKey remove_round{"remove_round"}; + constexpr WireKey creator{"creator"}; + constexpr WireKey creator_epoch{"creator_epoch"}; + constexpr WireKey creator_fence{"creator_fence"}; } -constexpr std::string_view kEntryTag = "ent"; +constexpr std::string_view kEntryTag = "entry"; constexpr EnumWireTable kNsStateWords{{{ {NsState::Creating, "creating"}, @@ -162,7 +162,7 @@ String encodeRefCatalog(const RefCatalog & catalog) writeKey(out, RefCatalogWire::creator_fence, first); writeU64StringValue(out, e.creator->fence_generation); } closeObject(out, first); - closeLine("ent"); + closeLine("entry"); } const size_t trailer_start = out.size(); @@ -202,7 +202,7 @@ RefCatalog decodeRefCatalog(std::string_view data) return catalog; } if (key != RefCatalogWire::kind) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"k\""); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: record must start with \"kind\""); const String kind = r.readString(); if (kind != kEntryTag) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown record kind '{}'", kind); @@ -223,13 +223,13 @@ RefCatalog decodeRefCatalog(std::string_view data) else if (key == RefCatalogWire::creator_epoch) cwe = r.readU64String(); else if (key == RefCatalogWire::creator_fence) cfg = r.readU64String(); else if (key == RefCatalogWire::remove_round) removal_started_round = r.readU64String(); - else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown ent key '{}'", key); + else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: unknown entry key '{}'", key); } if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: junk after record"); if (!st_word) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing st", ns_str); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing state", ns_str); const NsState state = nsStateFromWord(*st_word); /// throws CORRUPTED_DATA on an unknown word /// A missing "ns" key reads as the same empty string a present-but-empty one would, and both @@ -245,7 +245,7 @@ RefCatalog decodeRefCatalog(std::string_view data) ns_str, ns_str.size(), kMaxNamespaceBytes); if (!inc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing inc", ns_str); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: entry '{}' missing life", ns_str); if (*inc == 0) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: namespace '{}' has a zero incarnation -- 0 never names a life", ns_str); @@ -326,7 +326,7 @@ uint64_t worstCaseEntryFoldReservationBytes() /// coverage record plus terminal cleanup evidence, all numeric fields at maximum width. seal.ref_lives[std::numeric_limits::max()] = RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{kU64Max, kU64Max}, .hold = RefHold{.reason = HoldReason::UnconsumedSealCrossing, .offending_position = RefTxnId{kU64Max, kU64Max}, @@ -379,7 +379,7 @@ void checkFoldSealReservation( /// wrap to a remainder far smaller than the true reservation, which would answer "fits" for an /// `entry_count` that plainly does not. const uint64_t ref_lives = mulByteBudget(entry_count, worstCaseEntryFoldReservationBytes()); - /// `validateFoldSealStructure` permits at most one canonical seq-0 `btr` per shard, so charging + /// `validateFoldSealStructure` permits at most one canonical seq-0 `blob_run` per shard, so charging /// one widest row for every shard covers the full legal run domain without per-entry arithmetic. const uint64_t blob_target_runs = mulByteBudget( gc_shards, widestBlobTargetRunReservationBytes(layout, gc_shards)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h index ca1fbe5a6ddc..9515b00b425f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h @@ -16,7 +16,7 @@ class Layout; /// The byte bound every namespace name admitted into `ref_catalog` must satisfy (spec INV-3: /// "namespace names get a byte bound"). It keeps the catalog's operator-visible row and line grammar /// bounded, and both directions of the codec enforce it. Logical namespace bytes do NOT enter -/// predicate (2): fold-seal `rfl` rows are keyed only by the fixed-width opaque life id. +/// predicate (2): fold-seal `ref_life` rows are keyed only by the fixed-width opaque life id. constexpr size_t kMaxNamespaceBytes = 512; /// One namespace's catalog lifecycle state (spec INV-3, §3). `Creating` blocks publication and @@ -44,7 +44,7 @@ std::string_view nsStateToWord(NsState s); /// Inverse of `nsStateToWord`; throws `CORRUPTED_DATA` for anything but the three registered words. NsState nsStateFromWord(std::string_view w); -/// The fence identity of the mounted writer CREATING one namespace (spec §3): the server root plus +/// The fence identity of the mounted writer CREATING one namespace: the server root plus /// the writer epoch and admission fence generation captured at the moment `Creating` was minted. It /// is what a reconciler compares against `CasServerRoot`'s liveness/fence machinery before a stalled /// `Creating` entry may be CAS-reconciled away (INV-3: "stalled creators occupy entries until @@ -92,7 +92,7 @@ struct RefCatalog bool operator==(const RefCatalog &) const = default; }; -/// Encodes `catalog` as the canonical `cas_ref_catalog` text object: a header line, one "ent" record +/// Encodes `catalog` as the canonical `cas_ref_catalog` text object: a header line, one "entry" record /// per entry in canonical (ns-sorted) order, and a record-count trailer -- the same tagged-record /// container `encodeFoldSeal` uses. Enforces the FULL strict grammar on the way out: canonical order /// and no duplicate namespace, a non-empty namespace within the `kMaxNamespaceBytes` bound, nonzero @@ -144,7 +144,7 @@ uint64_t widestCondemnedSummaryReservationBytes(uint64_t gc_shards); /// PRE-PUT GATE, predicate (2) of INV-3's additive admission. Reserves the widest fixed frame, one /// widest ref-life row per candidate catalog entry, and one widest blob-target plus condemned-summary -/// row per authoritative GC shard. The `btr` multiplier follows the authoritative fold-seal grammar: +/// row per authoritative GC shard. The `blob_run` multiplier follows the authoritative fold-seal grammar: /// at most one canonical sequence-0 run is legal for each shard. Equality is accepted; refuses /// (`LIMIT_EXCEEDED`, naming `ns`) one entry over. Every multiplication and addition saturates, so an /// unreachable-in-practice count can never wrap into something that reads as "fits". diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index cc344f015ce2..dbc93315a99a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -20,13 +20,13 @@ namespace namespace RefCkptWire { - constexpr WireKey life_epoch{"le"}; - constexpr WireKey committed_epoch{"cte"}; - constexpr WireKey committed_seq{"cts"}; - constexpr WireKey snapshot_epoch{"cse"}; - constexpr WireKey snapshot_seq{"css"}; - constexpr WireKey seal_epoch{"lse"}; - constexpr WireKey seal_seq{"lss"}; + constexpr WireKey life_epoch{"life_epoch"}; + constexpr WireKey committed_epoch{"committed_epoch"}; + constexpr WireKey committed_seq{"committed_seq"}; + constexpr WireKey snapshot_epoch{"snapshot_epoch"}; + constexpr WireKey snapshot_seq{"snapshot_seq"}; + constexpr WireKey seal_epoch{"seal_epoch"}; + constexpr WireKey seal_seq{"seal_seq"}; } } @@ -143,22 +143,22 @@ RefCkpt decodeRefCkpt(std::string_view data) JsonObjectReader r(body_in, KeyStrictness::Strict, "cas_ref_ckpt"); RefCkpt ckpt; - std::optional cse; - std::optional css; - std::optional lse; - std::optional lss; - std::optional cte; - std::optional cts; + std::optional snapshot_epoch; + std::optional snapshot_seq; + std::optional seal_epoch; + std::optional seal_seq; + std::optional committed_epoch; + std::optional committed_seq; String key; while (r.nextKey(key)) { if (key == RefCkptWire::life_epoch) ckpt.life_epoch = r.readU64String(); - else if (key == RefCkptWire::committed_epoch) cte = r.readU64String(); - else if (key == RefCkptWire::committed_seq) cts = r.readU64String(); - else if (key == RefCkptWire::snapshot_epoch) cse = r.readU64String(); - else if (key == RefCkptWire::snapshot_seq) css = r.readU64String(); - else if (key == RefCkptWire::seal_epoch) lse = r.readU64String(); - else if (key == RefCkptWire::seal_seq) lss = r.readU64String(); + else if (key == RefCkptWire::committed_epoch) committed_epoch = r.readU64String(); + else if (key == RefCkptWire::committed_seq) committed_seq = r.readU64String(); + else if (key == RefCkptWire::snapshot_epoch) snapshot_epoch = r.readU64String(); + else if (key == RefCkptWire::snapshot_seq) snapshot_seq = r.readU64String(); + else if (key == RefCkptWire::seal_epoch) seal_epoch = r.readU64String(); + else if (key == RefCkptWire::seal_seq) seal_seq = r.readU64String(); else r.skipUnknown(key); } @@ -167,23 +167,23 @@ RefCkpt decodeRefCkpt(std::string_view data) /// deletable" today and as "recovery has no base" tomorrow -- both of which a reader would trust. /// Fail closed instead. (A missing whole field is a legitimate absence, not truncation: every field /// of this object is optional, so there is nothing to miss.) - if (cse || css) + if (snapshot_epoch || snapshot_seq) { - if (!cse || !css) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: checkpoint_snapshot_id needs both cse and css"); - ckpt.checkpoint_snapshot_id = RefTxnId{*cse, *css}; + if (!snapshot_epoch || !snapshot_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: checkpoint_snapshot_id needs both snapshot_epoch and snapshot_seq"); + ckpt.checkpoint_snapshot_id = RefTxnId{*snapshot_epoch, *snapshot_seq}; } - if (cte || cts) + if (committed_epoch || committed_seq) { - if (!cte || !cts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: committed_through needs both cte and cts"); - ckpt.committed_through = RefTxnId{*cte, *cts}; + if (!committed_epoch || !committed_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: committed_through needs both committed_epoch and committed_seq"); + ckpt.committed_through = RefTxnId{*committed_epoch, *committed_seq}; } - if (lse || lss) + if (seal_epoch || seal_seq) { - if (!lse || !lss) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: last_epoch_seal needs both lse and lss"); - ckpt.last_epoch_seal = RefTxnId{*lse, *lss}; + if (!seal_epoch || !seal_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: last_epoch_seal needs both seal_epoch and seal_seq"); + ckpt.last_epoch_seal = RefTxnId{*seal_epoch, *seal_seq}; } if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS cas_ref_ckpt: trailing bytes"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index c40fe9cfb7cc..4c6dc9fdf053 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -23,14 +23,14 @@ namespace namespace RefLogWire { - constexpr WireKey ns{"ns"}; - constexpr WireKey txn_epoch{"we"}; - constexpr WireKey txn_seq{"rs"}; - constexpr WireKey prev_epoch{"!pse"}; - constexpr WireKey prev_seq{"!pss"}; + constexpr WireKey ns{"namespace"}; + constexpr WireKey txn_epoch{"txn_epoch"}; + constexpr WireKey txn_seq{"txn_seq"}; + constexpr WireKey prev_epoch{"!prev_epoch"}; + constexpr WireKey prev_seq{"!prev_seq"}; constexpr WireKey op{"op"}; - constexpr WireKey ref{"rn"}; - constexpr WireKey published_ms{"ts"}; + constexpr WireKey ref{"ref"}; + constexpr WireKey published_ms{"published_ms"}; } constexpr EnumWireTable kRefOpWords{{{ @@ -43,11 +43,6 @@ constexpr EnumWireTable kRefOpWords{{{ static_assert(casEnumTableCoversEnum()); -RefOpKind opKindFromWord(std::string_view w) -{ - return kRefOpWords.fromWord(w, "RefLogTxn"); -} - /// Byte budget over the encoded text. A removal-class transaction uses the larger complete-table /// budget and has neither an op-count nor a per-op cap; normal transactions are bounded by /// `ref_txn_max_ops` and, per op, by `ref_op_max_bytes` (checked via `encodedOpSize`, one op at a @@ -113,7 +108,7 @@ struct BindingFields RefOwnerBinding build(std::string_view what) const { if (!kind || !ref) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing bk/rn", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: {} binding missing kind/ref", what); RefOwnerBinding b; b.kind = refOwnerKindFromWord(*kind, "RefLogTxn owner binding"); b.ref_name = *ref; @@ -123,10 +118,10 @@ struct BindingFields } }; -/// The log transaction's header-object meta line (ns + txn_id + the optional `prev_epoch_seal` +/// The log transaction's header-object meta line (`namespace` + txn_id + the optional `prev_epoch_seal` /// chain). Shared by `encodeRefLogTxn` and `removalFramingSize` so the two never disagree by a byte; /// `removalFramingSize` always passes `std::nullopt` -- a removal transaction is never a sequence-1 -/// epoch-transition record. Additive: the `"!pse"`/`"!pss"` pair is emitted only when +/// epoch-transition record. Additive: the `"!prev_epoch"`/`"!prev_seq"` pair is emitted only when /// `prev_epoch_seal` is set, so a body without it is byte-identical to the pre-EpochSeal wire shape. /// `!`-prefixed: `prev_epoch_seal` is INV-2 chain evidence, not cosmetic metadata -- a decoder that /// doesn't understand it must refuse the object rather than silently drop the chain link while @@ -149,9 +144,9 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) op.kind = kind; /// set_published_at fields - std::optional sp_rn; + std::optional sp_ref; ManifestRefFields sp_manifest_fields; - std::optional sp_ts; + std::optional sp_published_ms; /// owner_transition bindings BindingFields ob; BindingFields nb; @@ -160,12 +155,12 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) while (r.nextKey(key)) { if (key == RefLogWire::ref) - sp_rn = r.readString(); + sp_ref = r.readString(); else if (matchManifestRefFields(key, r, kBareManifestRefKeys, sp_manifest_fields)) { } else if (key == RefLogWire::published_ms) - sp_ts = r.readU64Number(); + sp_published_ms = r.readU64Number(); else if (key == kOldBindingKeys.kind) ob.kind = r.readString(); else if (key == kOldBindingKeys.ref) @@ -181,8 +176,8 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) { } else if (key == "pl") - /// `"pl"` (payload) was removed from the op wire in stage-1 T12 (the `set_payload` op became - /// `set_published_at`). The retired op WORD is already rejected by `opKindFromWord`, but this + /// `"pl"` (payload) was removed from the op wire when the `set_payload` op became + /// `set_published_at`. The retired op WORD is already rejected by `refOpKindFromWireWord`, but this /// generic reader reads field keys before switching on kind, so a `"pl"` field paired with a /// still-recognized op word would otherwise be `skipUnknown`'d. It is a KNOWN-removed field, /// not a genuinely-unknown one -- reject it explicitly rather than silently discard it. @@ -204,12 +199,12 @@ RefOp readOpRecord(JsonObjectReader & r, RefOpKind kind) op.new_binding = nb.build("new"); break; case RefOpKind::SetPublishedAt: - if (!sp_rn || !sp_ts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing rn/ts"); - op.ref_name = *sp_rn; + if (!sp_ref || !sp_published_ms) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: set_published_at missing ref/published_ms"); + op.ref_name = *sp_ref; checkCanonicalRefName(op.ref_name, "RefLogTxn", "set_published_at ref_name"); op.expected_manifest_ref = sp_manifest_fields.buildRef("RefLogTxn", "set_published_at"); - op.published_at_ms = *sp_ts; + op.published_at_ms = *sp_published_ms; break; } return op; @@ -222,6 +217,11 @@ std::string_view refOpKindToWireWord(RefOpKind kind) return kRefOpWords.toWord(kind, "RefLogTxn"); } +RefOpKind refOpKindFromWireWord(std::string_view w) +{ + return kRefOpWords.fromWord(w, "RefLogTxn"); +} + bool refLogTxnIsEpochSeal(const RefLogTxn & txn) { return txn.ops.size() == 1 && txn.ops.front().kind == RefOpKind::EpochSeal; @@ -310,10 +310,10 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con ReadBufferFromMemory m(line.data(), line.size()); JsonObjectReader r(m, KeyStrictness::Tolerant, "cas_ref_log"); bool saw_ns = false; - bool saw_we = false; - bool saw_rs = false; - std::optional pse; - std::optional pss; + bool saw_txn_epoch = false; + bool saw_txn_seq = false; + std::optional prev_epoch; + std::optional prev_seq; String key; while (r.nextKey(key)) { @@ -325,29 +325,29 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con else if (key == RefLogWire::txn_epoch) { txn.txn_id.writer_epoch = r.readU64String(); - saw_we = true; + saw_txn_epoch = true; } else if (key == RefLogWire::txn_seq) { txn.txn_id.ref_sequence = r.readU64String(); - saw_rs = true; + saw_txn_seq = true; } else if (key == RefLogWire::prev_epoch) - pse = r.readU64String(); + prev_epoch = r.readU64String(); else if (key == RefLogWire::prev_seq) - pss = r.readU64String(); + prev_seq = r.readU64String(); else r.skipUnknown(key); } - if (!saw_ns || !saw_we || !saw_rs) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing ns/we/rs"); - /// Both-or-neither: `nextKey` already rejects a repeated "!pse"/"!pss" (duplicate-key check), so + if (!saw_ns || !saw_txn_epoch || !saw_txn_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: meta line missing namespace/txn_epoch/txn_seq"); + /// Both-or-neither: `nextKey` already rejects a repeated "!prev_epoch"/"!prev_seq" (duplicate-key check), so /// this only guards against a body carrying exactly one of the pair. - if (pse || pss) + if (prev_epoch || prev_seq) { - if (!pse || !pss) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: prev_epoch_seal needs both !pse and !pss"); - txn.prev_epoch_seal = RefTxnId{*pse, *pss}; + if (!prev_epoch || !prev_seq) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: prev_epoch_seal needs both !prev_epoch and !prev_seq"); + txn.prev_epoch_seal = RefTxnId{*prev_epoch, *prev_seq}; } if (!m.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: junk after meta line"); @@ -387,7 +387,7 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con } if (key != RefLogWire::op) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: record must start with \"op\""); - const RefOpKind kind = opKindFromWord(r.readString()); + const RefOpKind kind = refOpKindFromWireWord(r.readString()); txn.ops.push_back(readOpRecord(r, kind)); if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: junk after op record"); @@ -440,4 +440,41 @@ size_t removalFramingSize(const String & ns, const RefTxnId & txn_id, uint64_t o return out.size(); } +std::optional peekRefLogMeta(const String & sealed_bytes) +{ + try + { + const String text = openObject(FormatId::RefLog, sealed_bytes); + ReadBufferFromMemory in(text.data(), text.size()); + const uint64_t line_cap = traitsFor(FormatId::RefLog).line_cap; + readLine(in, line_cap, "cas_ref_log"); /// header line -- skipped, the version is not judged here + const String meta = readLine(in, line_cap, "cas_ref_log"); + ReadBufferFromMemory m(meta.data(), meta.size()); + JsonObjectReader r(m, KeyStrictness::Tolerant, "cas_ref_log"); + RefLogMetaPeek peek; + bool saw_ns = false; + bool saw_epoch = false; + bool saw_seq = false; + String key; + while (r.nextKey(key)) + { + if (key == RefLogWire::ns) { peek.ns = r.readString(); saw_ns = true; } + else if (key == RefLogWire::txn_epoch) { peek.writer_epoch = r.readU64String(); saw_epoch = true; } + else if (key == RefLogWire::txn_seq) { peek.ref_sequence = r.readU64String(); saw_seq = true; } + else r.skipUnknown(key); + } + if (!saw_ns || !saw_epoch || !saw_seq) + return std::nullopt; + return peek; + } + catch (...) + { + /// Deliberately total: a diagnostic that throws while explaining an anomaly replaces the + /// anomaly's report with its own. A seal-linked txn reaches here too -- its `!`-prefixed chain + /// keys make the tolerant reader refuse the line -- and answering `nullopt` is correct: this + /// peek identifies a writer, it does not certify an object. + return std::nullopt; + } +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h index c61e11275e98..e4949cf130b8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.h @@ -13,7 +13,7 @@ namespace DB::Cas /// Text codec for `cas_ref_log`, the immutable object stored at `_log/`. Each object contains /// exactly one committed transaction: its namespace, transaction id, and the batch of `RefOp`s applied -/// by that commit. The body has a header, a meta line `{"ns","we","rs",["!pse","!pss"]}`, one JSON +/// by that commit. The body has a header, a meta line `{"namespace","txn_epoch","txn_seq",["!prev_epoch","!prev_seq"]}`, one JSON /// record per op, and a `{"n":count}` trailer. Records are emitted in the transaction's stored order /// and contain no codec-generated timestamps, so encoding the same value is byte-identical. This /// determinism is a property of the representation, not an adoption gate: ref commits use @@ -21,7 +21,7 @@ namespace DB::Cas /// returned text. /// /// `RefOpKind::EpochSeal` closes an epoch transition in-band (spec INV-2): a seal transaction contains -/// exactly that one op, and the meta line's optional `prev_epoch_seal` (wire fields `!pse`/`!pss`, +/// exactly that one op, and the meta line's optional `prev_epoch_seal` (wire fields `!prev_epoch`/`!prev_seq`, /// CRITICAL -- an unrecognized `!`-key fails closed with `UNKNOWN_FORMAT_VERSION` rather than being /// silently skipped, since dropping it would lose INV-2's chain evidence while still passing the /// structural grammar) chains to the transaction id of the seal that closed the PRECEDING epoch, and @@ -45,6 +45,9 @@ enum class RefOpKind : uint8_t /// `kind` is not represented by this format. std::string_view refOpKindToWireWord(RefOpKind kind); +/// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. +RefOpKind refOpKindFromWireWord(std::string_view w); + /// One operation inside a `RefLogTxn`. Only the fields documented next to `kind` are meaningful for /// that kind, and the codec never reads or writes the others. `OwnerTransition` optionally removes /// `old_binding` and/or installs `new_binding`; `SetPublishedAt` carries the expected manifest and the @@ -178,4 +181,22 @@ void validateEpochSealGrammarStructural(const RefLogTxn & txn); /// mint a sequence-1 transaction. Throws CORRUPTED_DATA on violation. void validateEpochSealGrammarContextual(const RefLogTxn & txn, uint64_t life_epoch); +/// The three identity fields of a `cas_ref_log` meta line, read WITHOUT trusting the object: this is +/// the anomaly diagnostic's view of an object found at a key it should not occupy, so the body is not +/// expected to match that key's identity. +struct RefLogMetaPeek +{ + String ns; + uint64_t writer_epoch = 0; + uint64_t ref_sequence = 0; +}; + +/// Best-effort identification of a sealed `cas_ref_log` object: opens it, skips the header line, and +/// reads the meta line's three identity fields. Never validates the header version, never reads past +/// the meta line, and answers `nullopt` for anything it cannot read -- truncation, garbage, a +/// different format, or a meta line missing one of the three. It lives HERE, beside the key +/// constants, because a caller that spelled those keys itself would silently stop matching the first +/// time they are renamed, and this reader has no output an ordinary test would miss. +std::optional peekRefLogMeta(const String & sealed_bytes); + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index 8ed19c9a1ac3..78caf9c5632c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -22,17 +22,16 @@ namespace namespace RefSnapWire { - constexpr WireKey ns{"ns"}; - constexpr WireKey snapshot_epoch{"we"}; - constexpr WireKey snapshot_seq{"rs"}; - constexpr WireKey lifecycle{"lc"}; - constexpr WireKey kind{"k"}; - constexpr WireKey ref{"rn"}; - constexpr WireKey published_ms{"ts"}; + constexpr WireKey ns{"namespace"}; + constexpr WireKey snapshot_epoch{"snapshot_epoch"}; + constexpr WireKey snapshot_seq{"snapshot_seq"}; + constexpr WireKey lifecycle{"lifecycle"}; + constexpr WireKey kind{"kind"}; + constexpr WireKey ref{"ref"}; + constexpr WireKey published_ms{"published_ms"}; } -constexpr std::string_view kCommittedTag = "c"; -constexpr std::string_view kPrecommitTag = "p"; +constexpr std::string_view kLiveLifecycleWord = "live"; void checkCommittedSorted(const std::vector & rows) { @@ -73,7 +72,7 @@ void writeCommittedRow(CasJsonWriter & out, const RefCommittedRow & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "committed"); bool first = true; - writeWordField(out, RefSnapWire::kind, kCommittedTag, first); + writeWordField(out, RefSnapWire::kind, refOwnerKindToWord(RefOwnerKind::Committed), first); writeStringField(out, RefSnapWire::ref, row.ref_name, first); writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); writeNumberField(out, RefSnapWire::published_ms, row.published_at_ms, first); @@ -90,15 +89,15 @@ void writePrecommitRow(CasJsonWriter & out, const RefOwnerBinding & row) checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); checkManifestRef(row.manifest_ref, "RefTableSnapshot", "precommit"); bool first = true; - writeWordField(out, RefSnapWire::kind, kPrecommitTag, first); + writeWordField(out, RefSnapWire::kind, refOwnerKindToWord(RefOwnerKind::Precommit), first); writeStringField(out, RefSnapWire::ref, row.ref_name, first); writeManifestRefFields(out, first, kBareManifestRefKeys, row.manifest_ref); closeObject(out, first); writeChar('\n', out); } -/// The snapshot's header-object meta line (`ns`, `snapshot_id`, and the required generation-8 -/// `lc:"live"` constant). Shared by +/// The snapshot's header-object meta line (`namespace`, `snapshot_id`, and the required +/// `lifecycle:"live"` constant). Shared by /// `encodeRefTableSnapshot` and `snapshotFramingSize` so the two never disagree by a /// byte. Assumes the caller has already validated the snapshot (or is measuring framing only). void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) @@ -106,7 +105,10 @@ void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) bool first = true; writeStringField(out, RefSnapWire::ns, snapshot.ns, first); writeRefTxnIdFields(out, first, RefSnapWire::snapshot_epoch, RefSnapWire::snapshot_seq, snapshot.snapshot_id); - writeStringField(out, RefSnapWire::lifecycle, "live", first); + /// A snapshot object exists only for a live namespace -- `RefLifecycle::Removed` has no snapshot + /// representation -- so the wire carries exactly one lifecycle word. The reader keeps the + /// fail-closed half: any other word, or none, is rejected there. + writeStringField(out, RefSnapWire::lifecycle, kLiveLifecycleWord, first); closeObject(out, first); writeChar('\n', out); } @@ -149,30 +151,30 @@ RefTableSnapshot decodeRefTableSnapshot( ReadBufferFromMemory meta_buf(line.data(), line.size()); JsonObjectReader r(meta_buf, KeyStrictness::Tolerant, "cas_ref_snap"); bool saw_ns = false; - bool saw_we = false; - bool saw_rs = false; - bool saw_lc = false; + bool saw_snapshot_epoch = false; + bool saw_snapshot_seq = false; + RefLifecycle lifecycle = RefLifecycle::Removed; String key; while (r.nextKey(key)) { if (key == RefSnapWire::ns) { snapshot.ns = r.readString(); saw_ns = true; } - else if (key == RefSnapWire::snapshot_epoch) { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == RefSnapWire::snapshot_seq) { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_rs = true; } + else if (key == RefSnapWire::snapshot_epoch) { snapshot.snapshot_id.writer_epoch = r.readU64String(); saw_snapshot_epoch = true; } + else if (key == RefSnapWire::snapshot_seq) { snapshot.snapshot_id.ref_sequence = r.readU64String(); saw_snapshot_seq = true; } else if (key == RefSnapWire::lifecycle) { - const String lifecycle = r.readString(); - if (lifecycle != "live") + const String lifecycle_word = r.readString(); + if (lifecycle_word != kLiveLifecycleWord) throw Exception(ErrorCodes::CORRUPTED_DATA, - "RefTableSnapshot: lifecycle must be exactly 'live', got '{}'", lifecycle); - saw_lc = true; + "RefTableSnapshot: lifecycle must be exactly '{}', got '{}'", kLiveLifecycleWord, lifecycle_word); + lifecycle = RefLifecycle::Live; } else if (key == "rte" || key == "rts") throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta carries retired terminal field '{}'", key); else r.skipUnknown(key); } - if (!saw_ns || !saw_we || !saw_rs || !saw_lc) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta line missing ns/we/rs/lc"); + if (!saw_ns || !saw_snapshot_epoch || !saw_snapshot_seq || lifecycle != RefLifecycle::Live) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: meta line missing namespace/snapshot_epoch/snapshot_seq/lifecycle"); if (!meta_buf.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after meta line"); } @@ -200,21 +202,21 @@ RefTableSnapshot decodeRefTableSnapshot( break; } if (key != RefSnapWire::kind) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"k\""); - const String k = r.readString(); + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: record must start with \"kind\""); + const RefOwnerKind kind = refOwnerKindFromWord(r.readString(), "RefTableSnapshot row kind"); - std::optional rn; + std::optional ref; ManifestRefFields mf; - std::optional ts; + std::optional published_ms; while (r.nextKey(key)) { - if (key == RefSnapWire::ref) rn = r.readString(); + if (key == RefSnapWire::ref) ref = r.readString(); else if (matchManifestRefFields(key, r, kBareManifestRefKeys, mf)) { } - else if (key == RefSnapWire::published_ms) ts = r.readU64Number(); + else if (key == RefSnapWire::published_ms) published_ms = r.readU64Number(); else if (key == "pl") - /// `"pl"` (payload) was removed from the row wire in stage-1 T12. It is a KNOWN-removed + /// `"pl"` (payload) was removed from the row wire. It is a KNOWN-removed /// field, not a genuinely-unknown future one the tolerant reader may skip -- silently /// discarding a persisted payload would lose data -- so reject it explicitly. throw Exception(ErrorCodes::CORRUPTED_DATA, @@ -224,30 +226,36 @@ RefTableSnapshot decodeRefTableSnapshot( if (!l.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: junk after record"); - if (k == kCommittedTag) + /// A `switch` rather than an if/else-if chain: the row kinds partition the enum, and a future + /// enumerator must not be able to arrive here, pass the word lookup, and then fall out of the + /// chain as a silently dropped row. With no default arm, adding one is a build error. + switch (kind) { - if (!rn || !ts) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing rn/ts"); + case RefOwnerKind::Committed: + { + if (!ref || !published_ms) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: committed row missing ref/published_ms"); RefCommittedRow row; - row.ref_name = *rn; + row.ref_name = *ref; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "committed ref_name"); row.manifest_ref = mf.buildRef("RefTableSnapshot", "committed"); - row.published_at_ms = *ts; + row.published_at_ms = *published_ms; snapshot.committed.push_back(std::move(row)); + break; } - else if (k == kPrecommitTag) + case RefOwnerKind::Precommit: { - if (!rn) - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing rn"); + if (!ref) + throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: precommit row missing ref"); RefOwnerBinding row; - row.kind = RefOwnerKind::Precommit; - row.ref_name = *rn; + row.kind = kind; + row.ref_name = *ref; checkCanonicalRefName(row.ref_name, "RefTableSnapshot", "precommit ref_name"); row.manifest_ref = mf.buildRef("RefTableSnapshot", "precommit"); snapshot.precommits.push_back(std::move(row)); + break; + } } - else - throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: unknown row kind '{}'", k); } /// The object key is supplied separately from the body. Check the binding before accepting any diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h index 07972a0d47ab..79573cf4767a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.h @@ -26,7 +26,7 @@ namespace DB::Cas /// published through the ordinary single-owner `putIfAbsentControlled` path, not a /// `putDeterministicArtifact` byte-adoption gate. -/// In-memory ref-table lifecycle. Only `Live` is serializable as a generation-8 snapshot; terminal +/// In-memory ref-table lifecycle. Only `Live` is serializable as a snapshot; terminal /// state lives in the removal log and fold evidence and has no snapshot DTO representation. enum class RefLifecycle : uint8_t { @@ -47,7 +47,7 @@ struct RefCommittedRow /// The complete state of one namespace's ref table in one canonical snapshot object. `precommits` /// reuses `RefOwnerBinding` from `CasRefWireVocab.h`; every entry's `kind` must be `Precommit`. -/// Generation 8 serializes only `Live` snapshots. Both row vectors must already be strictly sorted by +/// A snapshot serializes only `Live` namespaces. Both row vectors must already be strictly sorted by /// their documented keys, because the codec /// validates and emits the caller-provided order rather than sorting it. struct RefTableSnapshot diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h index e13fd116331e..0feacf7352d3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefWireVocab.h @@ -35,9 +35,8 @@ struct RefOwnerBinding bool operator==(const RefOwnerBinding &) const = default; }; -/// Convert an owner-kind discriminator to its canonical text word. Throws `CORRUPTED_DATA` for a -/// value not represented by this format; accepting an unknown value would produce an ambiguous wire -/// record. +/// Convert an owner-kind discriminator to its canonical text word. Throws `LOGICAL_ERROR` for an +/// out-of-range enum value. std::string_view refOwnerKindToWord(RefOwnerKind k); /// Parse a canonical owner-kind word. `what` identifies the containing field in the @@ -48,7 +47,7 @@ RefOwnerKind refOwnerKindFromWord(std::string_view w, std::string_view what); /// in-progress JSON object, both as decimal STRINGS -- the representation is width-independent, so no /// consumer has to care how large a `ref_sequence` can get. `epoch_key`/`seq_key` name the two fields, /// letting each format distinguish its primary id from any secondary id it embeds (for example, -/// `cas_ref_log`'s `we`/`rs` versus its `prev_epoch_seal` pair) while sharing one writer so the +/// `cas_ref_log`'s `txn_epoch`/`txn_seq` versus its `prev_epoch_seal` pair) while sharing one writer so the /// formats can never disagree on the representation. void writeRefTxnIdFields(CasJsonWriter & out, bool & first, WireKey epoch_key, WireKey seq_key, const RefTxnId & id); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp index 68c26d47e0e6..a01a62ed30f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.cpp @@ -16,26 +16,26 @@ namespace DB::Cas namespace OwnerWire { - constexpr WireKey server_uuid{"su"}; - constexpr WireKey retired_at_ms{"rt"}; + constexpr WireKey server_uuid{"server_uuid"}; + constexpr WireKey retired_at_ms{"retired_at_ms"}; } namespace ServerEpochWire { - constexpr WireKey next_writer_epoch{"nwe"}; + constexpr WireKey next_writer_epoch{"next_writer_epoch"}; } namespace MountLeaseWire { - constexpr WireKey server_uuid{"su"}; - constexpr WireKey writer_epoch{"we"}; - constexpr WireKey hostname{"hn"}; + constexpr WireKey server_uuid{"server_uuid"}; + constexpr WireKey writer_epoch{"writer_epoch"}; + constexpr WireKey hostname{"hostname"}; constexpr WireKey pid{"pid"}; - constexpr WireKey started_at_ms{"sat"}; + constexpr WireKey started_at_ms{"started_at_ms"}; constexpr WireKey seq{"seq"}; - constexpr WireKey expires_at_ms{"eat"}; - constexpr WireKey min_active{"ma"}; - constexpr WireKey gc_fenced{"fen"}; + constexpr WireKey expires_at_ms{"expires_at_ms"}; + constexpr WireKey min_active_build_sequence{"min_active_build_sequence"}; + constexpr WireKey gc_fenced{"gc_fenced"}; constexpr WireKey write_attempt_id{"write_attempt_id"}; } @@ -91,7 +91,7 @@ OwnerObject decodeOwner(std::string_view data) } o.retired_at_ms = rt; if (!saw) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: missing su"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: missing server_uuid"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS owner: trailing bytes"); return o; @@ -130,7 +130,7 @@ ServerEpoch decodeServerEpoch(std::string_view data) r.skipUnknown(key); } if (!saw) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: missing nwe"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: missing next_writer_epoch"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-epoch: trailing bytes"); return e; @@ -148,7 +148,7 @@ String encodeMountLease(const MountLease & m) writeNumberField(out, MountLeaseWire::started_at_ms, m.started_at_ms, first); writeU64StringField(out, MountLeaseWire::seq, m.seq, first); writeNumberField(out, MountLeaseWire::expires_at_ms, m.expires_at_ms, first); - writeU64StringField(out, MountLeaseWire::min_active, m.min_active, first); + writeU64StringField(out, MountLeaseWire::min_active_build_sequence, m.min_active_build_sequence, first); writeBoolField(out, MountLeaseWire::gc_fenced, m.gc_fenced, first); writeHex128Field(out, MountLeaseWire::write_attempt_id, m.write_attempt_id, first); closeObject(out, first); @@ -191,8 +191,8 @@ MountLease decodeMountLease(std::string_view data) m.seq = r.readU64String(); else if (key == MountLeaseWire::expires_at_ms) m.expires_at_ms = r.readU64Number(); - else if (key == MountLeaseWire::min_active) - m.min_active = r.readU64String(); + else if (key == MountLeaseWire::min_active_build_sequence) + m.min_active_build_sequence = r.readU64String(); else if (key == MountLeaseWire::gc_fenced) m.gc_fenced = r.readBool(); else if (key == MountLeaseWire::write_attempt_id) @@ -203,8 +203,15 @@ MountLease decodeMountLease(std::string_view data) else r.skipUnknown(key); } - if (!saw_su || !saw_we || !saw_write_attempt_id || m.write_attempt_id == UInt128{}) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing or zero identity field"); + /// Named one by one rather than as a single condition: these are three separate identities, and a + /// shared message cannot tell an operator which of them the object is missing -- nor let a test + /// prove that each is actually required. + if (!saw_su) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing server_uuid"); + if (!saw_we) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing writer_epoch"); + if (!saw_write_attempt_id || m.write_attempt_id == UInt128{}) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: missing or zero write_attempt_id"); if (!body_in.eof() || !in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS mount-lease: trailing bytes"); return m; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h index 0c73e8c5a6e6..8e0bd8118a1e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasServerRootFormats.h @@ -17,7 +17,7 @@ namespace DB::Cas /// The owner object permanently binds a configured `server_root_id` to one server UUID. The epoch /// object stores the next writer epoch and is CAS-bumped so an epoch is never reused after a /// restart or supersession. The mount object is the expirable liveness lease for one writer -/// incarnation; its `min_active` value carries the GC acknowledgement floor, while `gc_fenced` is a +/// incarnation; its `min_active_build_sequence` value carries the GC acknowledgement floor, while `gc_fenced` is a /// terminal fence-out marker for that incarnation. /// Permanent identity anchor for one configured server root. It is created with put-if-absent, never @@ -42,7 +42,7 @@ struct ServerEpoch /// The current liveness lease for one `(server_uuid, writer_epoch)` writer incarnation. The pool /// layer renews and replaces this object with CAS/overwrite operations, and GC may fence an expired -/// lease by setting `gc_fenced`; a fenced incarnation must not resume writing. `min_active` is the +/// lease by setting `gc_fenced`; a fenced incarnation must not resume writing. `min_active_build_sequence` is the /// merged GC acknowledgement floor, with `UINT64_MAX` marking a clean farewell (retired lease). struct MountLease { @@ -53,7 +53,7 @@ struct MountLease uint64_t started_at_ms = 0; uint64_t seq = 0; uint64_t expires_at_ms = 0; - uint64_t min_active = 0; /// UINT64_MAX = retired (farewell) + uint64_t min_active_build_sequence = 0; /// UINT64_MAX = retired (farewell) bool gc_fenced = false; /// GC fence-out of an expired lease; terminal /// One holder-originated body identity. Every physical retry of that one logical write reuses /// it; a GC fence copies the observed value while every successor holder body mints a new one. @@ -66,9 +66,9 @@ struct MountLease /// decisions belong to the caller that coordinates the server-root object. String encodeOwner(const OwnerObject & o); -/// Decode an owner anchor, requiring its `su` field, tolerating an absent optional `rt` retirement +/// Decode an owner anchor, requiring its `server_uuid` field, tolerating an absent optional `retired_at_ms` retirement /// timestamp, and rejecting bytes after the body line. Unknown JSON fields are skipped for -/// forward-compatible reads; malformed input, a missing `su`, and trailing data throw +/// forward-compatible reads; malformed input, a missing `server_uuid`, and trailing data throw /// `CORRUPTED_DATA`. OwnerObject decodeOwner(std::string_view data); @@ -77,13 +77,13 @@ OwnerObject decodeOwner(std::string_view data); /// codec. String encodeServerEpoch(const ServerEpoch & e); -/// Decode the epoch counter, requiring its `nwe` field and rejecting bytes after the body line. -/// Unknown JSON fields are skipped for forward-compatible reads; malformed input, a missing `nwe`, +/// Decode the epoch counter, requiring its `next_writer_epoch` field and rejecting bytes after the body line. +/// Unknown JSON fields are skipped for forward-compatible reads; malformed input, a missing `next_writer_epoch`, /// and trailing data throw `CORRUPTED_DATA`. ServerEpoch decodeServerEpoch(std::string_view data); /// Encode the complete mount-lease body as canonical text with the `cas_mount_lease` header and a -/// final newline. This preserves full-range `uint64_t` values such as `min_active` as decimal JSON +/// final newline. This preserves full-range `uint64_t` values such as `min_active_build_sequence` as decimal JSON /// strings and writes `gc_fenced` as a JSON boolean; lease renewal, fencing, and token checks remain /// in the caller. String encodeMountLease(const MountLease & m); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 8a53273956ee..8438ac786478 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -109,6 +109,20 @@ void CasJsonWriter::stringValue(std::string_view s) appendChar('"'); } +void CasJsonWriter::wordArray(std::span words) +{ + appendChar('['); + bool first = true; + for (const std::string_view word : words) + { + if (!first) + appendChar(','); + first = false; + stringValue(word); + } + appendChar(']'); +} + /// ---- read-side pull cursor ---- /// A canonical-text parse failure is CORRUPTED_DATA regardless of which ReadHelpers primitive @@ -187,6 +201,27 @@ String JsonObjectReader::readString() }); } +std::vector JsonObjectReader::readStringArray() +{ + return guarded([&] + { + std::vector words; + assertChar('[', in); + if (checkChar(']', in)) + return words; + + while (true) + { + String word; + readJSONString(word, in, jsonReadSettings()); + words.push_back(std::move(word)); + if (checkChar(']', in)) + return words; + assertChar(',', in); + } + }); +} + UInt128 JsonObjectReader::readHex128() { return guarded([&] diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 2ba6df5a1c5e..48f473c33d65 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -62,6 +63,9 @@ class CasJsonWriter /// Quoted JSON string with full escaping (bulk-run scan). Defined in CasTextFormat.cpp. void stringValue(std::string_view s); + /// JSON array of canonical word strings, emitted without intermediate storage. + void wordArray(std::span words); + void u64Number(uint64_t v) { char digits[24]; @@ -142,8 +146,9 @@ inline void writeIntText(uint64_t v, CasJsonWriter & out) { out.u64Number(v); } void writeHeaderLine(CasJsonWriter & out, FormatId id); void writeTrailerLine(CasJsonWriter & out, uint64_t n); -/// A wire-key carrier. The explicit constructor keeps raw string literals out of writer call -/// sites: a codec passes its named constant, and an inline `WireKey{"..."}` is deliberately loud. +/// A wire-key carrier for migrated writer call sites. Its explicit constructor makes a codec pass a +/// named constant, while an inline `WireKey{"..."}` is deliberately loud. `WireKey` borrows its +/// `string_view`; the referenced text must outlive the key, as with string literals and static constants. struct WireKey { std::string_view text; @@ -164,6 +169,12 @@ inline void writeWordField(CasJsonWriter & out, WireKey key, std::string_view wo writeStringValue(out, word); } +inline void writeWordArrayField(CasJsonWriter & out, WireKey key, std::span words, bool & first) +{ + writeKey(out, key, first); + out.wordArray(words); +} + inline void writeStringField(CasJsonWriter & out, WireKey key, std::string_view value, bool & first) { writeKey(out, key, first); @@ -220,6 +231,8 @@ class JsonObjectReader bool nextKey(String & key); /// Reads the value for the key returned by `nextKey` as a JSON string. String readString(); + /// Reads the value for the key returned by `nextKey` as an array of JSON strings. + std::vector readStringArray(); /// Reads a quoted 32-character lowercase hexadecimal string as a `UInt128`. UInt128 readHex128(); /// Reads a quoted decimal u64 string and rejects empty, trailing, or non-decimal text. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index e44bd062c33d..2b30d730aba5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -97,7 +97,7 @@ ManifestRef ManifestRefFields::buildRef(std::string_view what, std::string_view BlobRef BlobRefFields::build(std::string_view what) const { if (!algo_word || !digest_hex) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: blob ref missing ha/h", what); + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: blob ref missing algo/digest", what); const BlobHashAlgo algo = blobHashAlgoFromWord(*algo_word, what); /// Validate the digest width before calling `fromHex`. A width mismatch otherwise produces /// `BAD_ARGUMENTS` instead of the `CORRUPTED_DATA` required for malformed serialized input, @@ -114,4 +114,11 @@ BlobRef BlobRefFields::build(std::string_view what) const return BlobRef{algo, codecFor(algo).fromHex(*digest_hex)}; } +Token TokenFields::build(std::string_view what) const +{ + if (!type_word || !value) + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: token missing token_type/token", what); + return Token{*value, tokenTypeFromWord(*type_word, what)}; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h index eea099e7711e..5ca327462192 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h @@ -29,8 +29,8 @@ inline constexpr EnumWireTable kObjectKindWords{{{ {ObjectKind::Blob, "blob"}, }}}; -/// Convert a token discriminator to its canonical wire word. Throws `CORRUPTED_DATA` if `t` is not -/// one of the token types understood by this build. +/// Convert a token discriminator to its canonical wire word. Throws `LOGICAL_ERROR` for an +/// out-of-range enum value. std::string_view tokenTypeToWord(TokenType t); /// Parse a canonical token-type word. `what` identifies the containing codec or field in the @@ -42,30 +42,30 @@ TokenType tokenTypeFromWord(std::string_view w, std::string_view what); /// `CORRUPTED_DATA` exception. BlobHashAlgo blobHashAlgoFromWord(std::string_view w, std::string_view what); -/// Convert an envelope object-kind discriminator to its canonical wire word. Throws -/// `CORRUPTED_DATA` if `k` is not represented by this format. +/// Convert an envelope object-kind discriminator to its canonical wire word. Throws `LOGICAL_ERROR` +/// for an out-of-range enum value. std::string_view objectKindToWord(ObjectKind k); /// Parse a canonical envelope object-kind word. `what` identifies the containing codec or field in /// the `CORRUPTED_DATA` exception; unknown words are rejected rather than treated as a default kind. ObjectKind objectKindFromWord(std::string_view w, std::string_view what); -/// Append the sibling fields `tt` and `tv` to an in-progress JSON object. The caller owns `first`, +/// Append the sibling fields `token_type` and `token` to an in-progress JSON object. The caller owns `first`, /// which must describe the fields already written to that object; the token value is JSON-escaped. void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t); -/// Append the sibling fields `ha` and `h` to an in-progress JSON object. The algorithm word and +/// Append the sibling fields `algo` and `digest` to an in-progress JSON object. The algorithm word and /// lowercase digest are canonical, and the digest is rendered at the width required by `r.algo`. void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r); -/// The `ha`/`h` and `tt`/`tv` key spellings, named once so `writeBlobRefFields`/`writeTokenFields` +/// The `algo`/`digest` and `token_type`/`token` key spellings, named once so `writeBlobRefFields`/`writeTokenFields` /// and the `match*Fields` collectors below can never drift apart on the literal. namespace SharedWire { - inline constexpr WireKey algo{"ha"}; - inline constexpr WireKey digest{"h"}; - inline constexpr WireKey token_type{"tt"}; - inline constexpr WireKey token{"tv"}; + inline constexpr WireKey algo{"algo"}; + inline constexpr WireKey digest{"digest"}; + inline constexpr WireKey token_type{"token_type"}; + inline constexpr WireKey token{"token"}; } /// One `ManifestRef`'s three flat key names. Every bundle spells the SAME wire representation @@ -79,13 +79,13 @@ struct ManifestRefWireKeys WireKey ord; }; -/// The unprefixed `me`/`mb`/`mo` spelling used by part manifests, snapshot rows, and the +/// The unprefixed `epoch`/`build`/`ord` spelling used by part manifests, snapshot rows, and the /// `set_published_at` ref-log op. -inline constexpr ManifestRefWireKeys kBareManifestRefKeys{WireKey{"me"}, WireKey{"mb"}, WireKey{"mo"}}; -/// The `ome`/`omb`/`omo` spelling for a ref-log owner_transition's OLD binding. -inline constexpr ManifestRefWireKeys kOldManifestRefKeys{WireKey{"ome"}, WireKey{"omb"}, WireKey{"omo"}}; -/// The `nme`/`nmb`/`nmo` spelling for a ref-log owner_transition's NEW binding. -inline constexpr ManifestRefWireKeys kNewManifestRefKeys{WireKey{"nme"}, WireKey{"nmb"}, WireKey{"nmo"}}; +inline constexpr ManifestRefWireKeys kBareManifestRefKeys{WireKey{"epoch"}, WireKey{"build"}, WireKey{"ord"}}; +/// The `old_epoch`/`old_build`/`old_ord` spelling for a ref-log owner_transition's OLD binding. +inline constexpr ManifestRefWireKeys kOldManifestRefKeys{WireKey{"old_epoch"}, WireKey{"old_build"}, WireKey{"old_ord"}}; +/// The `new_epoch`/`new_build`/`new_ord` spelling for a ref-log owner_transition's NEW binding. +inline constexpr ManifestRefWireKeys kNewManifestRefKeys{WireKey{"new_epoch"}, WireKey{"new_build"}, WireKey{"new_ord"}}; /// One owner binding's key names: the owner-kind word, the ref name, and its nested `ManifestRef` /// bundle. Only the ref-log owner_transition op uses this bundle (old/new binding sides). @@ -96,10 +96,10 @@ struct BindingWireKeys ManifestRefWireKeys manifest; }; -/// The `obk`/`orn`/`ome`/`omb`/`omo` spelling for the OLD binding side. -inline constexpr BindingWireKeys kOldBindingKeys{WireKey{"obk"}, WireKey{"orn"}, kOldManifestRefKeys}; -/// The `nbk`/`nrn`/`nme`/`nmb`/`nmo` spelling for the NEW binding side. -inline constexpr BindingWireKeys kNewBindingKeys{WireKey{"nbk"}, WireKey{"nrn"}, kNewManifestRefKeys}; +/// The `old_kind`/`old_ref`/`old_epoch`/`old_build`/`old_ord` spelling for the OLD binding side. +inline constexpr BindingWireKeys kOldBindingKeys{WireKey{"old_kind"}, WireKey{"old_ref"}, kOldManifestRefKeys}; +/// The `new_kind`/`new_ref`/`new_epoch`/`new_build`/`new_ord` spelling for the NEW binding side. +inline constexpr BindingWireKeys kNewBindingKeys{WireKey{"new_kind"}, WireKey{"new_ref"}, kNewManifestRefKeys}; /// Append the three flat `ManifestRef` fields named by `keys` to an in-progress JSON object. The /// two unbounded `uint64_t` values are decimal JSON strings; the bounded ordinal is a JSON number. @@ -132,7 +132,7 @@ struct ManifestRefFields ManifestRef buildRef(std::string_view what, std::string_view context) const; }; -/// Collector for one `BlobRef`'s two flat fields (`ha`/`h`), filled in by `matchBlobRefFields`. +/// Collector for one `BlobRef`'s two flat fields (`algo`/`digest`), filled in by `matchBlobRefFields`. struct BlobRefFields { std::optional algo_word; @@ -145,20 +145,22 @@ struct BlobRefFields BlobRef build(std::string_view what) const; }; -/// Collector for one `Token`'s two flat fields (`tt`/`tv`), filled in by `matchTokenFields`. Phase 1 -/// deliberately has no `build`: callers keep their own local requiredness checks until the unified -/// both-required build is introduced. +/// Collector for one `Token`'s two flat fields (`token_type`/`token`), filled in by `matchTokenFields`. struct TokenFields { std::optional type_word; std::optional value; + + /// Requires both fields and parses the token type word. `what` identifies the enclosing codec + /// in `CORRUPTED_DATA` exceptions. + Token build(std::string_view what) const; }; /// Each `match*Fields` helper tests `key` against the one or two field names it owns, consumes the /// value on a match via `r`, and reports whether it recognized the key. None of them loop over an /// object's keys or validate a completed group -- that is the caller's (tolerant-reader loop) and /// the collector's `build`/`buildRef` job respectively. Defined inline: a decoder's per-key dispatch -/// is a hot path and must not gain a function-call boundary here. +/// is a hot path, so the helpers are header-defined for the per-key dispatch to inline them. inline bool matchManifestRefFields(std::string_view key, JsonObjectReader & r, const ManifestRefWireKeys & keys, ManifestRefFields & fields) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index 7c4c99aa36fd..ea7a73ab41bf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -18,30 +18,52 @@ trailer, followed by a banner-framed raw payload zone for inline file bytes. | Key (under the pool prefix) | Object | Codec | Writer | |---|---|---|---| -| `_pool_meta` | pool identity + floors | `CasPoolMetaFormat` | pool create/admit | -| `cas/ns/stream//_log/…​.zst` | ref transaction log | `CasRefLogFormat` (`.zst`) | writer commit path | -| `cas/ns/stream//_snap/…​.zst` | complete ref table | `CasRefSnapshotFormat` (`.zst`) | writer/GC fold | -| `cas/ns/state//_ckpt` | mutable life checkpoint | `CasRefCkptFormat` | writer/GC fold | +| `_pool_meta` | pool identity + floors (`pool_id`, `blob_header_len`, `gc_shards`, `min_reader_generation`, `algos_used` array) | `CasPoolMetaFormat` | pool create/admit | +| `cas/ns/stream//_log/…​.zst` | ref transaction log (`namespace`, `txn_epoch`/`txn_seq`, optional critical `!prev_epoch`/`!prev_seq`; `set_published_at` uses `ref`/`published_ms`) | `CasRefLogFormat` (`.zst`) | writer commit path | +| `cas/ns/stream//_snap/…​.zst` | complete live ref table (`namespace`, `snapshot_epoch`/`snapshot_seq`, `lifecycle:"live"`; `kind` is `committed`/`precommit`, with `ref` and committed-only `published_ms`) | `CasRefSnapshotFormat` (`.zst`) | writer/GC fold | +| `cas/ns/state//_ckpt` | mutable life checkpoint (`life_epoch`, `committed_epoch`/`committed_seq`, `snapshot_epoch`/`snapshot_seq`, `seal_epoch`/`seal_seq`) | `CasRefCkptFormat` | writer/GC fold | | `cas/ns/state//_files/…​` | namespace-owned raw files | — | upper layers | -| `cas/manifests//-/.zst` | part manifest | `CasPartManifestFormat` | part build | -| blob keys (`CasLayout::blobKey`) | blob envelope + payload | `CasBlobEnvelopeFormat` | uploads | -| blob-meta keys (`CasLayout::blobMetaKey`) | freshness sidecar | `CasBlobMetaFormat` | dedup/GC | -| `gc/state`, `gc/hb` | GC state / leader heartbeat | `CasGcStateFormat` | GC | -| `gc/maintenance_state` | leak-only namespace-janitor cursor | `CasGcMaintenanceStateFormat` | future janitor | -| `gc/gen//attempt//outcomes/…​.zst` | outcome log | `CasGcOutcomesFormat` (`.zst`) | GC | -| `gc/gen//attempt//fold_seal` | fold seal (deterministic) | `CasFoldSealFormat` | GC | -| `gc/gen//…​/runs` | GC source-edge record-stream runs | `CasRecordStreamFormat` | GC | -| `gc/server-roots//{owner,epoch,mount}` | server-root singletons | `CasServerRootFormats` | mount | +| `cas/ref_catalog` | namespace lifecycle catalog (`kind:"entry"`, `ns`, `state`, `life`, `remove_round`, `creator`, `creator_epoch`, `creator_fence`) | `CasRefCatalogFormat` | namespace admission/removal | +| `cas/manifests//-/.zst` | part manifest (`root_namespace`, `payload_digest`; entry `path`, `place`, `size`) | `CasPartManifestFormat` | part build | +| blob keys (`CasLayout::blobKey`) | blob envelope (`type`, `v`, `tag`, `build`, `time_ms`, `creator`, `op`, `chver`, `ref`) + payload | `CasBlobEnvelopeFormat` | uploads | +| blob-meta keys (`CasLayout::blobMetaKey`) | freshness sidecar (`state`, `condemn_round`, `size`) | `CasBlobMetaFormat` | dedup/GC | +| `gc/state`, `gc/hb` | GC state (`round`, `gc_shards`, `snap_generation`, `snap_pruned_through`, `snap_attempt`, `manifest_sweep_cursor`, `lease_owner`, `lease_seq`) / heartbeat (`owner`, `hb_seq`) | `CasGcStateFormat` | GC | +| `gc/maintenance_state` | leak-only namespace-janitor cursor (`janitor_cursor`) | `CasGcMaintenanceStateFormat` | future janitor | +| `gc/gen//attempt//outcomes/…​.zst` | outcome log (`kind`, `outcome`) | `CasGcOutcomesFormat` (`.zst`) | GC | +| `gc/gen//attempt//fold_seal` | fold seal (deterministic; `generation`/`parent_generation`, `kind`: `ref_life`/`blob_run`/`condemned`) | `CasFoldSealFormat` | GC | +| `gc/gen//…​/runs` | GC source-edge record-stream runs (`ref`, `src`, `mark`; condemned: `pending`, `size`, `condemn_round`, `confirmed`) | `CasRecordStreamFormat` | GC | +| `gc/server-roots//{owner,epoch,mount}` | server-root singletons (`server_uuid`, optional `retired_at_ms`; `next_writer_epoch`; `server_uuid`, `writer_epoch`, `hostname`, `pid`, `started_at_ms`, `seq`, `expires_at_ms`, `min_active_build_sequence`, `gc_fenced`, `write_attempt_id`) | `CasServerRootFormats` | mount | | `roots/…` | raw passthrough (verbatim) | — (never interpreted) | upper layers | ## Codec table Authoritative per-format traits (type string, family, strictness, compression policy, caps) live -in `CasFormat.cpp` (`TRAITS`), asserted complete by `gtest_cas_text_format.cpp`. Key naming: keys -2–5 chars; fixed-width `UInt128` identities = 32-char lowercase hex strings; blob digests = -algo-width hex (two chars per digest byte), rendered with their algo name (`sha256:ab12…`) wherever -a bare hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/lengths/ms-timestamps -= numbers; units documented here per object as codecs land. +in `CasFormat.cpp` (`TRAITS`), asserted complete by `gtest_cas_text_format.cpp`. + +Key naming follows a deliberate split between metadata written once per object and fields repeated +once per record, not a flat character-count budget: + +- metadata written once per object (`namespace`, `writer_epoch`, `blob_header_len`, …) uses + descriptive names; +- fields repeated once per record (`ref`, `mark`, `op`, `class`, `place`, …) use short, semantic + words whose meaning is clear in the record rather than the C++ member name verbatim; +- the fixed `cas_blob` descriptor uses its own separately budgeted compact vocabulary (`tag`, + `build`, `chver`, …), because it must fit before the pool-wide fixed payload offset; +- common framing stays `type`, `v`, and `n`; +- `!` stays the must-understand prefix for critical fields; +- C++ member names obey an asymmetric rule: a member may be fuller than its wire key, never more + cryptic than it. + +Exact full C++ member names everywhere were deliberately rejected — see +`docs/superpowers/specs/2026-08-28-cas-semantic-wire-keys-design.md` ("Rejected alternatives"). +Fixed-width `UInt128` identities render as 32-char lowercase hex strings; blob digests render as +algo-width hex (two chars per digest byte), with their algo name (`sha256:ab12…`) wherever a bare +hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/lengths/ms-timestamps = +numbers; units documented here per object as codecs land. + +`CasWireVocab.{h,cpp}` owns repeated value fields: `BlobRef` uses `algo`/`digest`, `Token` uses +the jointly required `token_type`/`token`, `ManifestRef` uses `epoch`/`build`/`ord`, and owner-transition bindings use +the corresponding `old_*` and `new_*` key bundles. ## Evolution rules (one screen) @@ -62,18 +84,9 @@ a bare hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/l enforcement. In practice the mismatch never arises: `Always` objects are read via a constructed `.zst`-suffixed key, so a raw body is not GETtable at that key. -## Generation 10 mount-attempt identity {#generation-10-mount-attempt-identity} - -Generation 10 is a breaking, recreate-only change for the unreleased CAS format: - -- `MountLease` adds the required full-word key `write_attempt_id`, encoded as a nonzero 32-character - lowercase `UInt128` hex value. It identifies one holder-originated logical write; all physical - retries reuse the exact body and ID. A GC fence preserves the observed ID, while reclaim and - successor bodies mint a new one. The decoder rejects a missing or zero value. -- `FormatId::MountLease` has a breaking generation-10 change point because its canonical body gained - that required field. -- `FormatId::PoolMeta` has the matching generation-10 change point and reader floor. `decodePoolMeta` - rejects a generation-9 pool before any old mount body can be interpreted without attempt identity. +## Generation history {#generation-history} -There is no generation-9 decoder, compatibility alias, or migration. CAS is pre-release; recreate a -generation-9 pool with a generation-10 writer. +The format's generation history was reset to a flat `{1, 1}` baseline (`G_BUILD == 1`): CAS is +pre-release, carries no persisted data, and pays no compatibility cost for starting the count over. +Every class's `changePoints` begins at generation 1; a future breaking change appends a real entry to +that class's own array and bumps `G_BUILD`, the same way it always has. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index f9c9ca6f72a5..46db0402ed7a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -218,7 +218,7 @@ RefPlan buildRefWalkPlan(RoundInput && round_input) ++plan.dropped_holds; continue; } - it->second.fold_state.coverage.classification = 4; + it->second.fold_state.coverage.classification = CoverageClass::Clamped; it->second.fold_state.coverage.hold = hold; } for (const auto & [life_id, checkpoint] : ref_scan.checkpoint_observations) @@ -1194,8 +1194,8 @@ bool Gc::foldManifestEdges(const ManifestId & id, int sign, std::vector this namespace's frontier this round (normal end); /// absent, a listed id above it => impossible under contiguity, so the store is lying or a /// durable record was lost: HOLD the namespace at - /// classification 4 with its cursor unmoved. + /// classification `Clamped` with its cursor unmoved. /// /// Epochs are crossed only over a consumed `EpochSeal` (INV-2): the seal folds as an applied table /// no-op (probe B2 `produced=false`) and the next epoch's start is `{E', 1}`, reached through the @@ -2065,7 +2065,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & cursor_it != parent_ref_lives.end() ? cursor_it->second.coverage.hold : std::nullopt; RefCoverage cov; - cov.classification = 0; + cov.classification = CoverageClass::Absent; bool table_changed = false; /// THE FRONTIER PROOF for this namespace, and there is exactly one thing that establishes it: /// the walk read the expected-next position by exact key, found it ABSENT, and no witness put @@ -2577,7 +2577,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & : (same_position ? UINT32_MAX : 0); effective->next_retry_round = current_round + 1; cov.hold = effective; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; ++intake_tables_held; /// A held namespace is unproven BY DEFINITION -- the hold names a position the walk could /// not resolve, so everything at or above it is unaccounted. Stated here rather than left to @@ -2587,7 +2587,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & unproven_reason = FoldResult::FrontierUnproven::Held; } else - cov.classification = table_changed ? 2 : 1; + cov.classification = table_changed ? CoverageClass::Folded : CoverageClass::Unchanged; result.fold_seal.ref_lives.at(target.life_id).coverage = cov; ++result.frontier_namespaces; @@ -2646,7 +2646,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & for (const WalkTarget & target : walk_targets) { RefCoverage cov; - cov.classification = 1; + cov.classification = CoverageClass::Unchanged; if (const auto pit = parent_ref_lives.find(target.life_id); pit != parent_ref_lives.end()) { cov.last_folded_ref_id = pit->second.coverage.last_folded_ref_id; @@ -2656,7 +2656,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & if (pit->second.coverage.hold) { cov.hold = pit->second.coverage.hold; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; } } RefLifeFoldState & ref_life_state = result.fold_seal.ref_lives.at(target.life_id); @@ -4057,7 +4057,7 @@ RebuildReport Gc::rebuildBaseline(bool force) const RefTableState & st = recovered.state; RefCoverage cov; - cov.classification = 2; /// Folded (full coverage) unless a bodiless precommit clamps + cov.classification = CoverageClass::Folded; /// unless a bodiless precommit clamps it below cov.last_folded_ref_id = st.getGreatestApplied(); /// Whether the hold on this row was minted BY THIS REBUILD (and so still owes a retry round) /// rather than carried from the prior seal. Tracked explicitly instead of by looking for a @@ -4103,7 +4103,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// still missing -- and clears once the namespace makes durable progress. /// RESIDUAL, named rather than hidden: progress unrelated to this precommit also clears /// it, and the precommit's edges stay missing until another rebuild. - cov.classification = 4; /// Clamped + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::ManifestBodyMissing, .offending_position = RefTxnId{cov.last_folded_ref_id.writer_epoch, cov.last_folded_ref_id.ref_sequence + 1}, @@ -4126,7 +4126,7 @@ RebuildReport Gc::rebuildBaseline(bool force) const auto pit = prior_seal->ref_lives.find(life.incarnation); if (pit != prior_seal->ref_lives.end() && pit->second.coverage.hold) { - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = pit->second.coverage.hold; minted_here = false; /// a carried hold rides VERBATIM; its retry fields are not ours } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp index 17ba6337d1e4..1a4cdf2e551c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp @@ -40,7 +40,7 @@ void onGcEnumerationPage() /// namespace is rooted by `server_root_id`, but that id is a clean relative path and can contain slashes. /// Try namespace prefixes from longest to shortest and accept the first durable mount body. Without a /// mount there is no deletion authority, so the caller must leave the prefix untouched. The mount's -/// `writer_epoch` and `min_active` are the single durable epoch/floor pair used for eligibility, including +/// `writer_epoch` and `min_active_build_sequence` are the single durable epoch/floor pair used for eligibility, including /// across process replacement and the retired sentinel. std::optional floorForNamespace(Pool & store, const RootNamespace & ns) { @@ -420,21 +420,21 @@ bool manifestDeletionPremise(const NamespaceFoldView & view, const ManifestKey & /// UNCERTAINTY, hold arm. A hold names the exact position the fold could not resolve, and everything /// at or above it is unaccounted -- including, for all this predicate can tell, the record that - /// grants or removes this very manifest. `classification == 4` is tested separately from the hold - /// even though the seal's strict grammar pairs them: the thing standing between a clamped namespace - /// and an irreversible delete must not be a codec invariant enforced somewhere else. + /// grants or removes this very manifest. `classification == Clamped` is tested separately from the + /// hold even though the seal's strict grammar pairs them: the thing standing between a clamped + /// namespace and an irreversible delete must not be a codec invariant enforced somewhere else. if (cov.hold) return retain(SweepRetainClass::Hold, "namespace held at " + renderRefTxnId(cov.hold->offending_position) + " (" + String{holdReasonToWord(cov.hold->reason)} + ", retried " + std::to_string(cov.hold->retry_count) + " round(s)): every record at or above " "that position is unaccounted for"); - if (cov.classification == 4) + if (cov.classification == CoverageClass::Clamped) return retain(SweepRetainClass::Hold, - "namespace coverage is classified clamped (4) with no hold recorded: whatever " + "namespace coverage is classified clamped with no hold recorded: whatever " "stopped the fold was not carried, so nothing above its cursor is accounted for"); - if (cov.classification == 0) + if (cov.classification == CoverageClass::Absent) return retain(SweepRetainClass::NoCoverage, - "namespace coverage is classified absent (0): no round folded it, so its cursor " + "namespace coverage is classified absent: no round folded it, so its cursor " "is not the result of any walk"); /// RULE 1 (spec §6). Grants do not cross epochs, so every `+1` that could name an epoch-`E` build @@ -478,7 +478,7 @@ bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & /// Eligibility comes only from the durable mount-lease floor. A missing floor means NOT eligible; /// do not replace that authority check with a frozen-sequence or judged-dead guess. Compare /// `writer_epoch` first, then `build_sequence`, so old-epoch - /// debris drains after a process restart even when its build_sequence is above the current min_active. + /// debris drains after a process restart even when its build_sequence is above the current min_active_build_sequence. const auto floor = floorForNamespace(store, ns); if (!floor) return false; @@ -488,9 +488,9 @@ bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & return true; if (prefix.writer_epoch > w.writer_epoch) return false; - if (w.min_active == std::numeric_limits::max()) + if (w.min_active_build_sequence == std::numeric_limits::max()) return true; /// farewell/retired sentinel: every seq is retired - return w.min_active > prefix.build_sequence; + return w.min_active_build_sequence > prefix.build_sequence; } uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h index d2f44b205cb7..e5ae9e3176a4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h @@ -39,7 +39,7 @@ struct ManifestKey enum class SweepRetainClass : uint8_t { None = 0, /// the premise admitted the deletion; no retention happened - NoCoverage, /// no sealed coverage row for the namespace (a classification-0 row counts here) + NoCoverage, /// no sealed coverage row for the namespace (a classification-`Absent` row counts here) Hold, /// the namespace is held, or is classified clamped UnconsumedSeal, /// rule (1): the cursor has not consumed the build epoch's closing seal TailRemoval, /// rule (2): an unconsumed tail record names this manifest as a removal target @@ -153,7 +153,7 @@ struct ManifestSweepResult /// manifest bodies written before `PrecommitAdd` and never named by any live owner, scoped to ONE /// namespace + ONE build prefix. Rules: /// - eligibility from the durable watermark fact only: the retired sentinel -/// (`min_active == UINT64_MAX`), or `min_active > build_sequence`, or a replaced incarnation — +/// (`min_active_build_sequence == UINT64_MAX`), or `min_active_build_sequence > build_sequence`, or a replaced incarnation — /// NEVER a frozen-seq / judged-dead heuristic alone (a missing watermark => not eligible); /// - the active `ManifestId` set comes from the namespace's committed + live-precommit owner view; /// - delete only bodies whose `ManifestId` is ABSENT from the active set, by exact token; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp index 92c502cde3d0..d5424b4786a6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp @@ -1083,10 +1083,10 @@ void PartWriteTxn::abandon() /// `Uncertain` tolerance above, no-ops) -- it never corrupts. alive = false; - /// No longer in-flight: retire the seq so the per-server active-build floor (`min_active`) can advance + /// No longer in-flight: retire the seq so the per-server active-build floor (`min_active_build_sequence`) can advance /// (idempotent). This runs AFTER the precommit removal above (mirrors `PartWriteTxn::promote`, which retires /// after its commit) so the build stays active until its precommit binding's removal is durable: - /// retiring first would advance `min_active` past a build whose precommit binding is still live in the + /// retiring first would advance `min_active_build_sequence` past a build whose precommit binding is still live in the /// ref log, letting a freshness-window consumer judge the manifest build-dead while an un-removed /// precommit still names it. Ordering removal-before-retire keeps that happens-before clean. store->retireBuildSeq(build_seq); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 1ea5c2d567cb..ed6a957cbde6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -1458,7 +1459,7 @@ void Pool::enqueueWriterCleanupDuty( } catch (...) { - /// The build deliberately remains active. Advancing `min_active` after losing the only cleanup + /// The build deliberately remains active. Advancing `min_active_build_sequence` after losing the only cleanup /// duty would make an uncertain owner grant look dead; pinning the floor until process exit is /// the safe failure direction, and successor recovery handles the durable remnant. writer_cleanup_queue_failed.store(true, std::memory_order_release); @@ -1600,56 +1601,6 @@ BlobLocation Pool::locate(const ManifestEntry & entry) const return manifest_reader.locate(entry); } -namespace -{ -/// a tolerant, read-only peek at the -/// `cas_ref_log` TEXT object (codecs-v3 phase 3) WITHOUT `decodeRefLogTxn`'s expected-value cross-check -/// -- the whole point of this diagnostic is that the body is NOT expected to match this key's identity. -/// It `openObject`s the stored `.zst`, skips the header line, and reads `ns`/`we`/`rs` off the meta -/// line (`we`/`rs` are decimal u64 strings). Never validates the header `v`, never reads past the meta -/// line (the ops are irrelevant to identifying the writer), and swallows any truncation/garbage: this -/// is a background diagnostic only, never a decode anything else depends on. -struct ForeignRefLogHeaderPeek -{ - String ns; - uint64_t writer_epoch = 0; - uint64_t ref_sequence = 0; -}; - -std::optional peekForeignRefLogHeader(const String & bytes) -{ - try - { - const String text = openObject(FormatId::RefLog, bytes); - ReadBufferFromMemory in(text.data(), text.size()); - const uint64_t line_cap = traitsFor(FormatId::RefLog).line_cap; - readLine(in, line_cap, "cas_ref_log"); /// header line -- skip - const String meta = readLine(in, line_cap, "cas_ref_log"); - ReadBufferFromMemory m(meta.data(), meta.size()); - JsonObjectReader r(m, KeyStrictness::Tolerant, "cas_ref_log"); - ForeignRefLogHeaderPeek peek; - bool saw_ns = false; - bool saw_we = false; - bool saw_rs = false; - String key; - while (r.nextKey(key)) - { - if (key == "ns") { peek.ns = r.readString(); saw_ns = true; } - else if (key == "we") { peek.writer_epoch = r.readU64String(); saw_we = true; } - else if (key == "rs") { peek.ref_sequence = r.readU64String(); saw_rs = true; } - else r.skipUnknown(key); - } - if (!saw_ns || !saw_we || !saw_rs) - return std::nullopt; - return peek; - } - catch (...) - { - return std::nullopt; - } -} -} - void Pool::reportImpossibleInterference(const String & key, const String & reason, const std::optional & offending_ns) { @@ -1693,7 +1644,7 @@ void Pool::reportImpossibleInterference(const String & key, const String & reaso "time the background diagnostic GET ran", key); return; } - if (const auto peek = peekForeignRefLogHeader(got->bytes)) + if (const auto peek = peekRefLogMeta(got->bytes)) LOG_ERROR(getLogger("CasPool"), "CAS anomaly diagnostics: offending object at '{}' ({} bytes) decodes as a ref-log " "header: namespace='{}', writer_epoch={}, ref_sequence={}", diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 69eba5061fd3..a56588dcb305 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -427,7 +427,7 @@ class Pool : public std::enable_shared_from_this uint64_t minActive(); /// Test/assertion accessor for the next-to-allocate build_seq under the lock. uint64_t peekNextBuildSeq(); - /// Renew the merged heartbeat once (bump seq, refresh min_active from the live callback, stamp a + /// Renew the merged heartbeat once (bump seq, refresh min_active_build_sequence from the live callback, stamp a /// fresh expires_at_ms). The build-watermark floor rides this beat. In production this is driven by /// the background renewer (background_watermark). void renewWatermarkOnce(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp index 388d2a110521..34aaeaa37308 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp @@ -113,8 +113,8 @@ PoolMeta PoolMeta::createOrValidate( validatePoolBlobHeaderLen(blob_header_len, ErrorCodes::BAD_ARGUMENTS, "pool meta"); if (gc_shards == 0) throw Exception(ErrorCodes::BAD_ARGUMENTS, "CAS pool meta: gc_shards must be >= 1"); - /// Defense against a garbage `static_cast` past the caller's own boundary: `blobHashAlgoName` - /// throws BAD_ARGUMENTS for anything `BlobHashAlgo` does not actually admit. + /// `blobHashAlgoName` rejects an out-of-range `BlobHashAlgo` with `LOGICAL_ERROR`: a programming + /// error that aborts debug and sanitizer builds, not an input-validation fence. blobHashAlgoName(blob_hash_algo); const String key = layout.poolMetaKey(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 93b962c6980b..65bced4938e6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -755,7 +755,7 @@ uint64_t allocateWriterEpoch( const MountLease surviving = decodeMountLease(*mount_probe.body); /// Deliberately weaker than claimMount's reclaim gate (this file, ~:370-380), /// which never trusts a bare wall-clock comparison alone (only gc_fenced / - /// the clean-farewell min_active==UINT64_MAX marker / a caller-proven-dead + /// the clean-farewell min_active_build_sequence==UINT64_MAX marker / a caller-proven-dead /// token justify a reclaim there, because clock skew can misjudge liveness). /// This is still safe: (a) the mint below is DISTINCT from the survivor's /// epoch by construction, so no same-(uuid, epoch) pair is ever representable @@ -965,7 +965,7 @@ MountClaimResult claimMount( /// every renewal fails the token guard forever, so it can never write again) — there is no /// liveness left to wait for. This is what makes self-remount (and a fast restart after a /// fence-out) instant instead of an observation wait. - /// - the clean marker (`min_active == UINT64_MAX`) → the predecessor's OWN graceful farewell + /// - the clean marker (`min_active_build_sequence == UINT64_MAX`) → the predecessor's OWN graceful farewell /// (`MountLeaseKeeper::terminate`) — no observation needed either. /// - `proven_dead_token` matches the token we just read → the CALLER (`claimMountAwaitingExpiry`) /// already watched this exact token hold stable for the full observation threshold on its own @@ -974,7 +974,7 @@ MountClaimResult claimMount( /// Anything else → `LiveDoubleStart` (do NOT write): a same-uuid, different-epoch, not fenced, not /// clean-marked, not (yet) proven-dead lease may simply be a live twin, and `expires_at_ms` alone /// can never distinguish that from a dead predecessor across two different clocks. - const bool clean_marker = existing.min_active == std::numeric_limits::max(); + const bool clean_marker = existing.min_active_build_sequence == std::numeric_limits::max(); const bool proven_dead = proven_dead_token && *proven_dead_token == got->token; if (existing.gc_fenced || clean_marker || proven_dead) { @@ -1176,7 +1176,7 @@ HeartbeatFloor computeHeartbeatFloor(Backend & b, const Layout & l, uint64_t now obs.erase(srid); /// terminal — no further observation needed break; } - if (m.min_active == std::numeric_limits::max()) + if (m.min_active_build_sequence == std::numeric_limits::max()) { ++floor.terminated; obs.erase(srid); /// terminal — no further observation needed @@ -1283,7 +1283,7 @@ std::vector probeNonTerminalMountSlots(Backend & b, const continue; } - if (m.gc_fenced || m.min_active == std::numeric_limits::max()) + if (m.gc_fenced || m.min_active_build_sequence == std::numeric_limits::max()) continue; /// terminal: fenced out by GC, or the holder's own graceful farewell. slots.push_back(NonTerminalMountSlot{srid, fmt::format( @@ -1333,7 +1333,7 @@ std::vector listMounts(Backend & backend, const Layout & layout, uint } if (info.lease.gc_fenced) info.state = "fenced"; - else if (info.lease.min_active == std::numeric_limits::max()) + else if (info.lease.min_active_build_sequence == std::numeric_limits::max()) info.state = "terminated"; else if (now_ms <= info.lease.expires_at_ms + skew_margin_ms) info.state = "live"; @@ -1366,7 +1366,7 @@ FenceCertificate classifyFenceCertificate(const MountLease & lease, uint64_t fen { if (lease.gc_fenced) return FenceCertificate::GcFenced; - if (lease.min_active == std::numeric_limits::max()) + if (lease.min_active_build_sequence == std::numeric_limits::max()) return FenceCertificate::CleanFarewell; if (lease.writer_epoch != fence_writer_epoch) return FenceCertificate::SupersededEpoch; @@ -1419,7 +1419,7 @@ bool isCreatorFenceTerminal(Backend & backend, const Layout & layout, const Stri MountLeaseKeeper::MountLeaseKeeper( BackendPtr backend_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, - std::function min_active_fn_, + std::function min_active_build_sequence_fn_, CasEventSink event_sink_, std::chrono::milliseconds lease_safety_margin_, std::function boot_ms_fn_) @@ -1430,7 +1430,7 @@ MountLeaseKeeper::MountLeaseKeeper( , writer_epoch(writer_epoch_) , ttl(ttl_) , now_ms_fn(std::move(now_ms_fn_)) - , min_active_fn(std::move(min_active_fn_)) + , min_active_build_sequence_fn(std::move(min_active_build_sequence_fn_)) , event_sink(std::move(event_sink_)) , lease_safety_margin(lease_safety_margin_) , boot_ms_fn(boot_ms_fn_ ? std::move(boot_ms_fn_) : defaultBootMs) @@ -1438,7 +1438,7 @@ MountLeaseKeeper::MountLeaseKeeper( } String MountLeaseKeeper::encodeBody( - uint64_t seq_, uint64_t wall_ms, uint64_t min_active, UInt128 write_attempt_id) const + uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const { const uint64_t ttl_ms = static_cast(ttl.count()); const uint64_t expires_at_ms = wall_ms > std::numeric_limits::max() - ttl_ms @@ -1452,7 +1452,7 @@ String MountLeaseKeeper::encodeBody( .started_at_ms = wall_ms, .seq = seq_, .expires_at_ms = expires_at_ms, - .min_active = min_active, + .min_active_build_sequence = min_active_build_sequence, .write_attempt_id = write_attempt_id, }); } @@ -1544,7 +1544,7 @@ uint64_t MountLeaseKeeper::start() const uint64_t wall_ms = now_ms_fn(); const uint64_t attempt_start_boot_ms = boot_ms_fn(); - const String body = encodeBody(/*seq_=*/1, wall_ms, min_active_fn(), newMountWriteAttemptId()); + const String body = encodeBody(/*seq_=*/1, wall_ms, min_active_build_sequence_fn(), newMountWriteAttemptId()); const Token token = claim(body); seq = 1; @@ -1691,7 +1691,7 @@ MountRenewResult MountLeaseKeeper::renew( const uint64_t attempt_start_boot_ms = boot_clock(); const uint64_t next_seq = seq + 1; const UInt128 write_attempt_id = newMountWriteAttemptId(); - const String body = encodeBody(next_seq, wall_ms, min_active_fn(), write_attempt_id); + const String body = encodeBody(next_seq, wall_ms, min_active_build_sequence_fn(), write_attempt_id); const Token expected = last_token; const uint64_t safety_ms = static_cast(lease_safety_margin.count()); @@ -1829,7 +1829,7 @@ void MountLeaseKeeper::terminate() .started_at_ms = wall_ms, .seq = seq + 1, .expires_at_ms = wall_ms, - .min_active = std::numeric_limits::max(), + .min_active_build_sequence = std::numeric_limits::max(), .write_attempt_id = newMountWriteAttemptId(), }); const PutResult result = backend->putOverwrite(key, body, last_token); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index 5f19862e2348..6108141415dc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -181,7 +181,7 @@ uint64_t allocateWriterEpoch(Backend & b, const Layout & l, const String & srid, enum class MountPriorState { None, - Clean, /// the predecessor's own graceful farewell (`min_active == UINT64_MAX`) + Clean, /// the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) Fenced, /// the GC leader's own (already threshold-gated) fence-out (`gc_fenced`) UncleanObserved, /// OUR observation watched the write-token hold stable for the full threshold }; @@ -201,7 +201,7 @@ enum class MountPriorState /// `claimMountAwaitingExpiry` below for how a plain "looks expired" reading is turned into one): /// - `gc_fenced` (the GC leader already, itself, threshold-gated this incarnation dead; a fence /// costs an epoch, so its keeper can never renew again) → reclaim, `prior = Fenced`; -/// - the clean marker (`min_active == UINT64_MAX`, the predecessor's own graceful farewell) → +/// - the clean marker (`min_active_build_sequence == UINT64_MAX`, the predecessor's own graceful farewell) → /// reclaim, `prior = Clean`; /// - `proven_dead_token` matches the CURRENTLY OBSERVED token (the caller itself watched this /// exact token hold stable for the full observation threshold) → reclaim, `prior = @@ -325,7 +325,7 @@ using MountObservationMap = std::map; /// dead mounts (liveness only — graduation itself paces on GC rounds, not on heartbeat acks). /// Classification per body: /// - `gc_fenced` already set → excluded (`already_fenced`); a fenced mount is terminal, no PUT; -/// - terminated (`min_active == UINT64_MAX`, the farewell sentinel stamped by +/// - terminated (`min_active_build_sequence == UINT64_MAX`, the farewell sentinel stamped by /// `MountLeaseKeeper::terminate`) → excluded (`terminated`). `expires_at_ms` alone cannot /// distinguish a graceful farewell from an unclean stop, so the sentinel — not the timestamps — is the /// terminated marker; @@ -382,7 +382,7 @@ struct NonTerminalMountSlot /// still entitled to this prefix? A slot counts as terminal on exactly the two clock-free certificates /// the mount protocol already recognises (`computeHeartbeatFloor`'s own classification): `gc_fenced` /// (the GC leader fenced that incarnation out, and a fence costs an epoch, so its keeper can never -/// renew again) and `min_active == UINT64_MAX` (the holder's own graceful farewell). Everything else is +/// renew again) and `min_active_build_sequence == UINT64_MAX` (the holder's own graceful farewell). Everything else is /// reported, INCLUDING a body this build cannot decode -- an unreadable lease of some other format /// generation is precisely the case that must block, not the one to wave through. /// @@ -398,7 +398,7 @@ std::vector probeNonTerminalMountSlots(Backend & b, const /// A read-only snapshot of one server's mount slot, for introspection (`system.cas_mounts`). /// state: `live` (lease within TTL+skew), `expired` (lease ran out; the next GC round's heartbeat floor -/// will fence it), `terminated` (clean farewell: `min_active == UINT64_MAX`), `fenced` (`gc_fenced`), +/// will fence it), `terminated` (clean farewell: `min_active_build_sequence == UINT64_MAX`), `fenced` (`gc_fenced`), /// `corrupt` (body failed to decode — surfaced as a row, never an exception). struct MountInfo { @@ -436,7 +436,7 @@ std::vector listMounts(Backend & backend, const Layout & layout, uint /// identical question at pool-prefix and GC-heartbeat granularity — /// - `gc_fenced` (the GC leader already fenced this incarnation; a fence costs an epoch, so its /// keeper can never renew again), -/// - the clean-farewell sentinel `min_active == UINT64_MAX`, +/// - the clean-farewell sentinel `min_active_build_sequence == UINT64_MAX`, /// PLUS one more certificate available here that neither of those needs: a DIFFERENT `writer_epoch` /// currently live at that slot proves `writer_epoch`'s specific incarnation is superseded regardless of /// its OWN certificate — `allocateWriterEpoch`/`claimMount` are why an epoch, once superseded, is never @@ -487,7 +487,7 @@ class MountLeaseKeeper MountLeaseKeeper( BackendPtr backend_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, - std::function min_active_fn_, + std::function min_active_build_sequence_fn_, CasEventSink event_sink_ = {}, std::chrono::milliseconds lease_safety_margin_ = std::chrono::milliseconds(2000), /// boot-domain clock for the on_renew_ok anchor; empty = real CLOCK_BOOTTIME. Injectable for @@ -504,7 +504,7 @@ class MountLeaseKeeper uint64_t lastCommittedAttemptStartBootMs() const { return last_committed_attempt_start_boot_ms; } private: - String encodeBody(uint64_t seq_, uint64_t wall_ms, uint64_t min_active, UInt128 write_attempt_id) const; + String encodeBody(uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const; Token claim(const String & body); [[noreturn]] void throwRenewConflict(const CasOverwriteDiagnostics & diagnostics) const; MountRenewResult terminalResult( @@ -521,7 +521,7 @@ class MountLeaseKeeper uint64_t writer_epoch; std::chrono::milliseconds ttl; std::function now_ms_fn; - std::function min_active_fn; + std::function min_active_build_sequence_fn; CasEventSink event_sink; std::chrono::milliseconds lease_safety_margin; /// boot-domain clock for the on_renew_ok anchor; empty = real CLOCK_BOOTTIME. Injectable for diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h index 54360403b8d7..70f50f4e6682 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasBlobDigest.h @@ -57,8 +57,9 @@ std::string_view blobHashAlgoName(BlobHashAlgo algo); /// Returns the digest byte width for `algo`: 16 for `CityHash128` and `XXH3_128`, or 32 for /// `Sha256`. This is also the width used by `Cas::codecFor(algo)`'s `DigestCodec`; callers must -/// derive it from the algorithm rather than from pool state. Throws `BAD_ARGUMENTS` for an -/// out-of-range enum value, preserving the fail-closed contract of `blobHashAlgoName`. +/// derive it from the algorithm rather than from pool state. The functions over `BlobHashAlgo` +/// intentionally use different defensive codes: this one throws `BAD_ARGUMENTS` for an out-of-range +/// enum value, while `blobHashAlgoName` throws `LOGICAL_ERROR`. uint64_t blobHashLenFor(BlobHashAlgo algo); /// Parses the per-disk `blob_hash` CONFIG value: `"cityhash128"` | `"xxh3-128"` | `"sha256"`. Throws diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h index e59e13fcef6b..f2cacfa32f28 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEnumWireTableAsserts.h @@ -3,7 +3,8 @@ /// Compile-time coverage proof for EnumWireTable: SET EQUALITY with the enum's declared values. /// Size-plus-uniqueness is not enough (an invalid casted value satisfies both while an enumerator /// goes missing). This header pulls in magic_enum and therefore MUST be included only from .cpp -/// files and tests, never from another header. +/// files and tests, never from another header. The proof assumes every enumerator is in +/// magic_enum's reflectable range (by default -128..127), because `enum_values` sees only that range. #include diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index 416f5c327663..44a86a0c312b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -304,7 +304,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, } /// Graceful close stamps an already-expired lease and the watermark farewell - /// (`min_active = UINT64_MAX`), making the slot `terminated` before its mutable control objects + /// (`min_active_build_sequence = UINT64_MAX`), making the slot `terminated` before its mutable control objects /// are removed and its owner anchor is tombstoned. admin.reset(); @@ -334,7 +334,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const MountLease mount_value = decodeMountLease(farewell_mount->bytes); captures_match = epoch_value.next_writer_epoch != 0 && mount_value.writer_epoch == epoch_value.next_writer_epoch - 1 - && mount_value.min_active == std::numeric_limits::max() + && mount_value.min_active_build_sequence == std::numeric_limits::max() && !mount_value.gc_fenced; if (!captures_match) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index 4793eadd4e12..dda1c5921f2d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -168,7 +168,7 @@ String renderRefTableSnapshot(const RefTableSnapshot & s) .str(); } -/// The namespace's checkpoint (spec INV-4). Every field is optional and each absence means something +/// The namespace's checkpoint. Every field is optional and each absence means something /// different an operator needs to see: no `life_epoch` means no writer that knew this namespace's /// genesis epoch has written here yet, no `committed_through` means the life has no committed /// transaction, no `checkpoint_snapshot_id` means recovery has no snapshot base, and no @@ -258,7 +258,7 @@ String renderMountLease(const MountLease & m) .add("started_at_ms", jsonUInt(m.started_at_ms)) .add("seq", jsonUInt(m.seq)) .add("expires_at_ms", jsonUInt(m.expires_at_ms)) - .add("min_active", jsonUInt(m.min_active)) + .add("min_active_build_sequence", jsonUInt(m.min_active_build_sequence)) .add("gc_fenced", jsonBool(m.gc_fenced)) .add("write_attempt_id", jsonHex(m.write_attempt_id)) .str(); @@ -308,7 +308,7 @@ String renderRunRef(const RunRef & r) String renderRefCoverage(const RefCoverage & c) { return JsonObj() - .add("classification", jsonUInt(c.classification)) + .add("classification", jsonEscape(coverageClassToWord(c.classification))) .add("last_folded_ref_id", renderRefTxnIdObj(c.last_folded_ref_id)) .str(); } @@ -378,7 +378,7 @@ String renderEnvelopeHeader(const EnvelopeHeader & h) return JsonObj() .add("kind", jsonEscape(objectKindToWord(h.kind))) /// The blob identity is carried by the object key, so the envelope keeps only the provenance - /// fields needed for forensics (`ch` and `bld`) together with its compatibility version. + /// fields needed for forensics (`chver` and `build`) together with its compatibility version. .add("compatibility_version", jsonUInt(h.compatibility_version)) .add("incarnation_tag", jsonHex(h.incarnation_tag)) .add("build_id", jsonHex(h.build_id)) @@ -388,7 +388,7 @@ String renderEnvelopeHeader(const EnvelopeHeader & h) .str(); } -/// The word vocabulary a row's marker byte renders as, matching the `cas_run` NDJSON's own `"m"` field +/// The word vocabulary a row's marker byte renders as, matching the `cas_run` NDJSON's own `mark` field /// words (`runMarkerToWireWord`) so cas-inspect speaks the same vocabulary /// as the on-disk format rather than inventing a second one. String sourceEdgeRowKindName(RunMarker marker) diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 441ea58fc558..df987e1325ac 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -940,8 +940,8 @@ inline UInt128 catalogLifeIdForTest( /// real round. This is the durable fact the sweep's §6 deletion premise reads /// (`CasOrphanManifestSweep.cpp`): `cursor` is the namespace's `last_folded_ref_id`, and a manifest of /// an epoch-`E` build is deletable only once that cursor sits in an epoch STRICTLY above `E`. -/// `hold`, when set, makes the row classification 4 — the strict grammar `encodeFoldSeal` enforces in -/// both directions, so a hold and a non-4 classification cannot be seeded together. +/// `hold`, when set, makes the row classification `Clamped` — the strict grammar `encodeFoldSeal` +/// enforces in both directions, so a hold and a non-clamped classification cannot be seeded together. /// /// SHARP EDGE, HANDLED HERE SO NO CALLER HAS TO KNOW IT: a fold seal must carry a `condemned_summary` /// entry for EVERY shard in `0..gc_shards-1`. A later real round adopts this object as its PARENT and @@ -988,7 +988,7 @@ inline void seedFoldCursorForTest( seal.generation = generation; DB::Cas::RefCoverage cov; - cov.classification = hold ? 4 : 2; + cov.classification = hold ? DB::Cas::CoverageClass::Clamped : DB::Cas::CoverageClass::Folded; cov.last_folded_ref_id = cursor; cov.hold = hold; seal.ref_lives[life.incarnation].coverage = cov; @@ -1050,15 +1050,15 @@ inline uint64_t foldCursorOf( /// Set a server root's durable floor (so orphan-sweep eligibility can be driven). After the ack-floor /// merge the floor rides the mount lease body (`mountKey`), so this seeds a MountLease carrying -/// `{writer_epoch, min_active}` — exactly what `prefixEligible` reads. +/// `{writer_epoch, min_active_build_sequence}` — exactly what `prefixEligible` reads. inline void setWatermarkMinActive( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const String & server_root_id, - uint64_t writer_epoch, uint64_t min_active) + uint64_t writer_epoch, uint64_t min_active_build_sequence) { DB::Cas::MountLease m; m.server_uuid = DB::UInt128(0); m.writer_epoch = writer_epoch; - m.min_active = min_active; + m.min_active_build_sequence = min_active_build_sequence; m.seq = 1; m.write_attempt_id = DB::UInt128{1}; const String key = layout.mountKey(server_root_id); diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 495ed57f0325..25cd9577fe76 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -1,7 +1,11 @@ #include "cas_format_test_battery.h" #include +#include #include +#include + +#include #include using namespace DB::Cas; @@ -22,6 +26,47 @@ EnvelopeHeader sampleHeader(const String & ref) } constexpr uint32_t L = 256; +/// The `op` word with the most bytes on the wire, found by walking the enum through the REAL +/// public encoder-facing lookup (never by hardcoding "mutation") so a future longer word is +/// automatically picked up by the boundary tests below. +ProvenanceOp longestProvenanceOp() +{ + ProvenanceOp best = ProvenanceOp::Other; + size_t best_len = 0; + for (const auto op : magic_enum::enum_values()) + { + const size_t len = provenanceOpToWireWord(op).size(); + if (len > best_len) + { + best_len = len; + best = op; + } + } + return best; +} + +/// A header whose numeric provenance fields sit at their type maxima (`created_at_ms` at the +/// `uint64_t` max, `ch_version` at the `uint32_t` max, `op` at its longest wire word), so the +/// non-`ref` JSON this produces is the largest `encodeEnvelopeHeader` can emit for real field +/// values. `v` is not settable this way -- `encodeEnvelopeHeader` always stamps +/// `currentCompatibilityVersion()` -- so this is the worst case reachable through the real encoder +/// today, not the type-level bound `kMandatoryDescriptorWorstCase` proves for a hypothetical future +/// `v` at its own `uint32_t` maximum. +EnvelopeHeader maxReachableHeader(const String & ref) +{ + EnvelopeHeader h; + h.kind = ObjectKind::Blob; + h.incarnation_tag = hexToU128("0102030405060708090a0b0c0d0e0f10"); + h.build_id = hexToU128("1112131415161718191a1b1c1d1e1f20"); + h.provenance = Provenance{ + std::numeric_limits::max(), + hexToU128("2122232425262728292a2b2c2d2e2f30"), + std::numeric_limits::max(), + longestProvenanceOp()}; + h.intended_ref = ref; + return h; +} + /// The envelope has a fixed physical length. At generation 9 there is no unsupported one-digit /// version, so replacing `9` with `10` must consume one byte from the space pad rather than silently /// turning the 256-byte fixture into a different wire shape. @@ -53,8 +98,8 @@ TEST(CASBlobEnvelopeFormat, FixedLengthAndPadZone) EXPECT_EQ(head[L - 1], '\n'); /// terminator at byte 255 const String json = fmt::format(R"({{"type":"cas_blob","v":{},)", currentCompatibilityVersion()) + "\"tag\":\"0102030405060708090a0b0c0d0e0f10\"," - "\"bld\":\"1112131415161718191a1b1c1d1e1f20\",\"ts\":1752537600123," - "\"by\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"ch\":26006001," + "\"build\":\"1112131415161718191a1b1c1d1e1f20\",\"time_ms\":1752537600123," + "\"creator\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"chver\":26006001," "\"ref\":\"t-abc/all_1_2_0\"}"; ASSERT_LT(json.size(), L); EXPECT_EQ(head.substr(0, json.size()), json); /// '/' UNescaped (local escaper) @@ -93,6 +138,82 @@ TEST(CASBlobEnvelopeFormat, RefTruncatedToExactBudget) EXPECT_EQ(c, 'a'); } +TEST(CASBlobEnvelopeFormat, MandatoryWorstCaseBoundary) +{ + /// `kMandatoryDescriptorWorstCase` (239, proven at compile time against the 240 floor) assumes + /// `v` at its OWN type maximum (10 digits), because `currentCompatibilityVersion()` could grow + /// with a future generation. Nothing can make a running build emit that many digits today -- + /// `encodeEnvelopeHeader` always stamps the CURRENT `currentCompatibilityVersion()`, one digit at + /// this generation -- so the worst case reachable through the real encoder right now is 9 bytes + /// smaller: a 10-byte `ref` budget at the floor, not 1. That 9-byte gap is exactly + /// `kMaxU32DecimalLen - digit count of the current compatibility version`, so a generation that + /// reaches two digits narrows it and this expectation must be re-derived then -- the literal below + /// is deliberate, since deriving it from the version width here would restate the formula the + /// compile-time bound already owns and prove nothing about the encoder. + EnvelopeHeader h_floor = maxReachableHeader(""); + const String head_floor = encodeEnvelopeHeader(h_floor, static_cast(kMinBlobHeaderLen)); + ASSERT_EQ(head_floor.size(), kMinBlobHeaderLen); + EXPECT_EQ(head_floor[kMinBlobHeaderLen - 1], '\n'); + EXPECT_EQ(payloadOffset(decodeEnvelopeHeader(head_floor, head_floor.size(), ObjectKind::Blob)), kMinBlobHeaderLen); + const size_t json_len_floor = head_floor.find_last_not_of(' ', kMinBlobHeaderLen - 2) + 1; + const size_t budget_floor = (kMinBlobHeaderLen - 1) - json_len_floor; + EXPECT_EQ(budget_floor, 10u) << "ref budget reachable through the real encoder at the floor"; + + /// The default 256-byte header is exactly 16 bytes above the floor, so the SAME max-reachable + /// content leaves exactly 16 more bytes of `ref` budget. + EnvelopeHeader h_default = maxReachableHeader(""); + const String head_default = encodeEnvelopeHeader(h_default, L); + ASSERT_EQ(head_default.size(), L); + EXPECT_EQ(head_default[L - 1], '\n'); + EXPECT_EQ(payloadOffset(decodeEnvelopeHeader(head_default, head_default.size(), ObjectKind::Blob)), L); + const size_t json_len_default = head_default.find_last_not_of(' ', L - 2) + 1; + const size_t budget_default = (L - 1) - json_len_default; + EXPECT_EQ(budget_default, budget_floor + (L - kMinBlobHeaderLen)) + << "ref budget reachable through the real encoder at the default header length"; +} + +TEST(CASBlobEnvelopeFormat, CriticalKeyDescriptorStillFitsAtDefaultLength) +{ + /// The test-only `!x` critical key is written BEFORE `ref`; even at max-reachable field values + /// the descriptor still fits the default 256-byte header and fails closed as + /// UNKNOWN_FORMAT_VERSION, never CORRUPTED_DATA or a LOGICAL_ERROR from encode itself. + EnvelopeHeader h = maxReachableHeader("r"); + h.emit_unknown_critical_key = true; + const String head = encodeEnvelopeHeader(h, L); + ASSERT_EQ(head.size(), L); + cas_battery_detail::expectCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, + [&] { decodeEnvelopeHeader(head, head.size(), ObjectKind::Blob); }, + "critical-key blob envelope at max-reachable field values"); +} + +/// Closed-set pin: the six `ProvenanceOp` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASBlobEnvelopeFormat, ClosedSetPinsProvenanceOpWords) +{ + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Other), "other"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Insert), "insert"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Merge), "merge"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Mutation), "mutation"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Attach), "attach"); + EXPECT_EQ(provenanceOpToWireWord(ProvenanceOp::Repack), "repack"); + for (const auto op : magic_enum::enum_values()) + EXPECT_EQ(provenanceOpFromWireWord(provenanceOpToWireWord(op)), op); +} + +TEST(CASBlobEnvelopeFormat, UnknownOpWordFailsClosed) +{ + /// `op` is written as a plain (non-critical) key, so an unrecognized word is a decode-time + /// vocabulary violation, not a missing-extension one: CORRUPTED_DATA, not UNKNOWN_FORMAT_VERSION. + EnvelopeHeader h = sampleHeader("r"); + String head = encodeEnvelopeHeader(h, L); + const size_t op_at = head.find("\"op\":\"merge\""); + ASSERT_NE(op_at, String::npos); + head.replace(op_at, String("\"op\":\"merge\"").size(), "\"op\":\"bogus\""); + cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { decodeEnvelopeHeader(head, head.size(), ObjectKind::Blob); }, "unknown op word"); +} + TEST(CASBlobEnvelopeFormat, PadZoneSmugglingFailsClosed) { EnvelopeHeader h = sampleHeader("r"); @@ -163,8 +284,8 @@ TEST(CASFormatBattery, BlobEnvelope) /// the encoder to itself and pin nothing. const String json = fmt::format(R"({{"type":"cas_blob","v":{},)", currentCompatibilityVersion()) + "\"tag\":\"0102030405060708090a0b0c0d0e0f10\"," - "\"bld\":\"1112131415161718191a1b1c1d1e1f20\",\"ts\":1752537600123," - "\"by\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"ch\":26006001," + "\"build\":\"1112131415161718191a1b1c1d1e1f20\",\"time_ms\":1752537600123," + "\"creator\":\"2122232425262728292a2b2c2d2e2f30\",\"op\":\"merge\",\"chver\":26006001," "\"ref\":\"t-abc/all_1_2_0\"}"; const String golden = json + String((L - 1) - json.size(), ' ') + '\n'; runFormatBattery(FormatBatteryCase{ diff --git a/src/Disks/tests/gtest_cas_blob_meta_format.cpp b/src/Disks/tests/gtest_cas_blob_meta_format.cpp index 59d1f7205dc4..581b56f1b070 100644 --- a/src/Disks/tests/gtest_cas_blob_meta_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta_format.cpp @@ -2,6 +2,8 @@ #include #include +#include + using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } @@ -39,8 +41,8 @@ TEST(CASFormatBattery, BlobMeta) .id = FormatId::BlobMeta, .encode = [&] { return sealObject(FormatId::BlobMeta, encodeBlobMeta(m)); }, .decode = [](std::string_view s) { decodeBlobMeta(std::string(openObject(FormatId::BlobMeta, s))); }, - .golden = "{\"type\":\"cas_blob_meta\",\"v\":10}\n" - "{\"st\":\"clean\",\"cr\":\"0\",\"sz\":\"12345\"}\n"}); + .golden = "{\"type\":\"cas_blob_meta\",\"v\":1}\n" + "{\"state\":\"clean\",\"condemn_round\":\"0\",\"size\":\"12345\"}\n"}); } TEST(CASBlobMetaFormat, CondemnedRoundTripAllFields) @@ -54,19 +56,30 @@ TEST(CASBlobMetaFormat, CondemnedRoundTripAllFields) EXPECT_EQ(back.condemn_round, 7u); EXPECT_EQ(back.size, 4096u); EXPECT_EQ(encodeBlobMeta(m), - "{\"type\":\"cas_blob_meta\",\"v\":10}\n{\"st\":\"condemned\",\"cr\":\"7\",\"sz\":\"4096\"}\n"); + "{\"type\":\"cas_blob_meta\",\"v\":1}\n{\"state\":\"condemned\",\"condemn_round\":\"7\",\"size\":\"4096\"}\n"); +} + +/// Closed-set pin: the two `MetaState` wire words, walked through `magic_enum::enum_values` so a +/// future state a `MetaState` construction can reach but no table entry names would fail this +/// exhaustive check rather than silently pass through unspecified. +TEST(CASBlobMetaFormat, ClosedSetPinsMetaStateWords) +{ + EXPECT_EQ(metaStateToWireWord(MetaState::Clean), "clean"); + EXPECT_EQ(metaStateToWireWord(MetaState::Condemned), "condemned"); + for (const auto state : magic_enum::enum_values()) + EXPECT_EQ(metaStateFromWireWord(metaStateToWireWord(state)), state); } TEST(CASBlobMetaFormat, FailsClosedOnUnknownStateAndTruncation) { /// Unknown state word -> CORRUPTED_DATA (mirrors the old `state > Condemned` reject). - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"st\":\"zombie\",\"cr\":\"0\",\"sz\":\"0\"}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String bad_state = "{\"type\":\"cas_blob_meta\",\"v\":1}\n{\"state\":\"zombie\",\"condemn_round\":\"0\",\"size\":\"0\"}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(bad_state); }); /// Missing state key -> CORRUPTED_DATA. - const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":3}\n{\"cr\":\"0\",\"sz\":\"0\"}\n"; + const String no_state = "{\"type\":\"cas_blob_meta\",\"v\":1}\n{\"condemn_round\":\"0\",\"size\":\"0\"}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeBlobMeta(no_state); }); /// Truncated (header only) -> CORRUPTED_DATA. - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":3}\n"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeBlobMeta("{\"type\":\"cas_blob_meta\",\"v\":1}\n"); }); } diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 620dea710a3f..4f7d37e2b4fa 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -166,7 +166,7 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend if (armed && key == mount_key && result.outcome == PutOutcome::Done) { const MountLease mount = decodeMountLease(bytes); - if (mount.min_active == std::numeric_limits::max()) + if (mount.min_active_build_sequence == std::numeric_limits::max()) farewell_seen = true; } return result; @@ -201,7 +201,7 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend ++mount_value.seq; ++mount_value.started_at_ms; mount_value.expires_at_ms = mount_value.started_at_ms + 30'000; - mount_value.min_active = 0; + mount_value.min_active_build_sequence = 0; mount_value.gc_fenced = false; successor_mount_bytes = encodeMountLease(mount_value); const PutResult mount_put = InMemoryBackend::putOverwrite( @@ -277,7 +277,7 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend .started_at_ms = 1'000, .seq = 1, .expires_at_ms = 31'000, - .min_active = 0, + .min_active_build_sequence = 0, }); const PutResult mount_put = InMemoryBackend::putIfAbsent(mount_key, successor_mount_bytes, {}); if (mount_put.outcome != PutOutcome::Done) diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index 01f0b99a465a..45a075e50d3d 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -2,6 +2,10 @@ #include #include #include +#include +#include +#include +#include #include #include #include @@ -9,11 +13,40 @@ using namespace DB; using namespace DB::Cas; -/// These literals pin the CANONICAL BYTES of the CAS text encoders as of the commit that -/// introduced this file. The CasJsonWriter migration (2026-07-20 spec) must keep every one of -/// them green UNMODIFIED: canonical text is byte-compared on retries and deterministic adoption, -/// and the incremental ref budget counters assume these exact sizes. Never edit an expected -/// string here to make a test pass — that means the encoder's bytes drifted, which is the bug. +namespace +{ +String lineAt(const String & text, size_t index) +{ + size_t begin = 0; + for (size_t i = 0; i < index; ++i) + begin = text.find('\n', begin) + 1; + const size_t end = text.find('\n', begin); + return text.substr(begin, end - begin + 1); +} + +void expectDelta(const String & old_bytes, const String & new_bytes, size_t expected) +{ + EXPECT_EQ(new_bytes.size() - old_bytes.size(), expected) << "old: " << old_bytes << "new: " << new_bytes; +} + +CasFoldSeal oneFoldSeal() +{ + CasFoldSeal seal; + seal.generation = 5; + seal.parent_generation = 4; + return seal; +} +} + +/// The `CASEncodingPins` literals below pin the CANONICAL BYTES of the CAS text encoders: canonical +/// text is byte-compared on retries and deterministic adoption, and the incremental ref budget +/// counters assume these exact sizes. Never edit one of those expected strings to make a test pass — +/// that means the encoder's bytes drifted, which is the bug. +/// +/// The `CASWireCutDeltas` literals are the opposite kind: each is a HISTORICAL pre-cut row, kept so +/// the cost of the semantic-key rename stays measurable against what it replaced. They are +/// deliberately not the current bytes and must never be refreshed toward them — a delta measured +/// against today's encoder on both sides would always be zero. TEST(CASEncodingPins, RefLogTxnAllOpKinds) { @@ -49,13 +82,13 @@ TEST(CASEncodingPins, RefLogTxnAllOpKinds) txn.ops.push_back(removal); const String expected = fmt::format("{{\"type\":\"cas_ref_log\",\"v\":{}}}\n", currentCompatibilityVersion()) + - "{\"ns\":\"roots/pin\",\"we\":\"7\",\"rs\":\"9\"}\n" + "{\"namespace\":\"roots/pin\",\"txn_epoch\":\"7\",\"txn_seq\":\"9\"}\n" "{\"op\":\"namespace_birth\"}\n" - "{\"op\":\"owner_transition\",\"obk\":\"precommit\",\"orn\":\"20260101_0_1_1_1\"," - "\"ome\":\"1\",\"omb\":\"2\",\"omo\":3,\"nbk\":\"committed\",\"nrn\":\"20260101_0_1_1_1\"," - "\"nme\":\"1\",\"nmb\":\"2\",\"nmo\":3}\n" - "{\"op\":\"set_published_at\",\"rn\":\"20260101_0_1_1_1\\\"c\\nd\\u0001e\\u2028f\"," - "\"me\":\"1\",\"mb\":\"2\",\"mo\":3,\"ts\":1234}\n" + "{\"op\":\"owner_transition\",\"old_kind\":\"precommit\",\"old_ref\":\"20260101_0_1_1_1\"," + "\"old_epoch\":\"1\",\"old_build\":\"2\",\"old_ord\":3,\"new_kind\":\"committed\",\"new_ref\":\"20260101_0_1_1_1\"," + "\"new_epoch\":\"1\",\"new_build\":\"2\",\"new_ord\":3}\n" + "{\"op\":\"set_published_at\",\"ref\":\"20260101_0_1_1_1\\\"c\\nd\\u0001e\\u2028f\"," + "\"epoch\":\"1\",\"build\":\"2\",\"ord\":3,\"published_ms\":1234}\n" "{\"op\":\"remove_namespace\"}\n" "{\"n\":4}\n"; EXPECT_EQ(encodeRefLogTxn(txn), expected); @@ -76,9 +109,9 @@ TEST(CASEncodingPins, RefSnapshotLive) snap.precommits.push_back(RefOwnerBinding{RefOwnerKind::Precommit, "20260102_0_2_2_2", ManifestRef{4, 5, 6}}); const String expected = fmt::format("{{\"type\":\"cas_ref_snap\",\"v\":{}}}\n", currentCompatibilityVersion()) + - "{\"ns\":\"roots/pin\",\"we\":\"7\",\"rs\":\"9\",\"lc\":\"live\"}\n" - "{\"k\":\"c\",\"rn\":\"20260101_0_1_1_1\",\"me\":\"1\",\"mb\":\"2\",\"mo\":3,\"ts\":5}\n" - "{\"k\":\"p\",\"rn\":\"20260102_0_2_2_2\",\"me\":\"4\",\"mb\":\"5\",\"mo\":6}\n" + "{\"namespace\":\"roots/pin\",\"snapshot_epoch\":\"7\",\"snapshot_seq\":\"9\",\"lifecycle\":\"live\"}\n" + "{\"kind\":\"committed\",\"ref\":\"20260101_0_1_1_1\",\"epoch\":\"1\",\"build\":\"2\",\"ord\":3,\"published_ms\":5}\n" + "{\"kind\":\"precommit\",\"ref\":\"20260102_0_2_2_2\",\"epoch\":\"4\",\"build\":\"5\",\"ord\":6}\n" "{\"n\":2}\n"; EXPECT_EQ(encodeRefTableSnapshot(snap), expected); } @@ -108,16 +141,249 @@ TEST(CASEncodingPins, SourceEdgeRunLines) writer.finish(); out.finalize(); - /// The exact "b" rendering (algo byte + digest hex) is pinned as a whole line; the point is - /// that Task 8's line-scratch rewrite must reproduce it byte-for-byte. + /// The exact `ref` rendering (algo byte + digest hex) is pinned as a whole line; the point is + /// that the line-scratch rendering must reproduce it byte-for-byte. const String text = out.str(); const String header = fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()); const String expected_record = - "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; + "{\"ref\":\"0100000000000000000000000000000002\",\"src\":\"00000000000000000000000000000005\",\"mark\":\"edge\"}\n"; const String expected_condemned = - "{\"b\":\"0100000000000000000000000000000003\",\"s\":\"00000000000000000000000000000000\",\"m\":\"condemned\",\"pend\":true,\"tt\":\"etag\",\"tv\":\"token\",\"sz\":9,\"cr\":\"7\",\"mc\":true}\n"; + "{\"ref\":\"0100000000000000000000000000000003\",\"src\":\"00000000000000000000000000000000\",\"mark\":\"condemned\",\"pending\":true,\"token_type\":\"etag\",\"token\":\"token\",\"size\":9,\"condemn_round\":\"7\",\"confirmed\":true}\n"; const String trailer = "{\"n\":2}\n"; /// Both records must remain byte-identical to their canonical stored representation. const String expected_full = header + expected_record + expected_condemned + trailer; EXPECT_EQ(text, expected_full) << text; } + +TEST(CASWireCutDeltas, ActiveCasRunRow) +{ + SourceEdgeRecord record{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(2))}, .source_id = UInt128(5), .marker = RunMarker::Edge}; + WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(record); + writer.finish(); + out.finalize(); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n"; + expectDelta(old_bytes, lineAt(out.str(), 1), 7); +} + +TEST(CASWireCutDeltas, CondemnedCasRunRow) +{ + SourceEdgeRecord record{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(3))}, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = true, .token = Token{"token", TokenType::ETag}, .size = 9, .condemn_round = 7, .marker_confirmed = true}; + WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(record); + writer.finish(); + out.finalize(); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"b\":\"0100000000000000000000000000000003\",\"s\":\"00000000000000000000000000000000\",\"m\":\"condemned\",\"pend\":true,\"tt\":\"etag\",\"tv\":\"token\",\"sz\":9,\"cr\":\"7\",\"mc\":true}\n"; + expectDelta(old_bytes, lineAt(out.str(), 1), 41); +} + +TEST(CASWireCutDeltas, BlobPartManifestEntry) +{ + PartManifest manifest; + manifest.ref = ManifestRef{1, 2, 3}; + manifest.root_namespace_id = RootNamespace{"root"}; + manifest.entries = {ManifestEntry{"a", EntryPlacement::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, 9, {}}}; + const String text = encodePartManifest(manifest); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"p\":\"a\",\"pm\":\"blob\",\"ha\":\"ch128\",\"h\":\"00000000000000000000000000000004\",\"sz\":9}\n"; + expectDelta(old_bytes, lineAt(text, 2), 15); +} + +TEST(CASWireCutDeltas, InlinePartManifestEntry) +{ + PartManifest manifest; + manifest.ref = ManifestRef{1, 2, 3}; + manifest.root_namespace_id = RootNamespace{"root"}; + manifest.entries = {ManifestEntry{"a", EntryPlacement::Inline, {}, 0, "x"}}; + const String text = encodePartManifest(manifest); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"p\":\"a\",\"pm\":\"inline\",\"il\":1}\n"; + expectDelta(old_bytes, lineAt(text, 2), 8); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_banner = "==> \"a\" il=1 <==\n"; + expectDelta(old_banner, lineAt(text, 4), 2); +} + +TEST(CASWireCutDeltas, GcOutcomesRow) +{ + OutcomeLog log{{OutcomeEntry{ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, Token{"t", TokenType::ETag}, OutcomeKind::Deleted}}}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00000000000000000000000000000004\",\"tt\":\"etag\",\"tv\":\"t\",\"oc\":\"deleted\"}\n"; + expectDelta(old_bytes, lineAt(encodeOutcomeLog(log), 1), 26); +} + +/// The ref-log's own op rows: the highest-cardinality record of the format and, for +/// `owner_transition`, the largest single-row cost of the whole cut -- both old-side groups and both +/// new-side groups are renamed at once. +TEST(CASWireCutDeltas, OwnerTransitionRefLogOpRow) +{ + RefLogTxn txn; + txn.ns = "root"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = RefOpKind::OwnerTransition; + op.old_binding = RefOwnerBinding{RefOwnerKind::Precommit, "r", ManifestRef{3, 4, 5}}; + op.new_binding = RefOwnerBinding{RefOwnerKind::Committed, "r", ManifestRef{3, 4, 5}}; + txn.ops.push_back(op); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"op\":\"owner_transition\",\"obk\":\"precommit\",\"orn\":\"r\",\"ome\":\"3\",\"omb\":\"4\",\"omo\":5," + "\"nbk\":\"committed\",\"nrn\":\"r\",\"nme\":\"3\",\"nmb\":\"4\",\"nmo\":5}\n"; + expectDelta(old_bytes, lineAt(encodeRefLogTxn(txn), 2), 50); +} + +TEST(CASWireCutDeltas, SetPublishedAtRefLogOpRow) +{ + RefLogTxn txn; + txn.ns = "root"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = RefOpKind::SetPublishedAt; + op.ref_name = "r"; + op.expected_manifest_ref = ManifestRef{3, 4, 5}; + op.published_at_ms = 6; + txn.ops.push_back(op); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"op\":\"set_published_at\",\"rn\":\"r\",\"me\":\"3\",\"mb\":\"4\",\"mo\":5,\"ts\":6}\n"; + expectDelta(old_bytes, lineAt(encodeRefLogTxn(txn), 2), 18); +} + +/// The body-less ops are the cut's only free rows: the record is the `op` key alone. This measures +/// that the ROW costs nothing extra, not that the word itself is unchanged -- it builds its old side +/// from the current word, so it cannot see a word rename. The words are pinned literally by the +/// closed-set tests; what this adds is that no framing crept in around them. +TEST(CASWireCutDeltas, BodylessRefLogOpRowsAreUnchanged) +{ + for (const RefOpKind kind : {RefOpKind::NamespaceBirth, RefOpKind::EpochSeal}) + { + RefLogTxn txn; + txn.ns = "root"; + txn.txn_id = RefTxnId{1, 1}; + RefOp op; + op.kind = kind; + txn.ops.push_back(op); + const String old_bytes = fmt::format("{{\"op\":\"{}\"}}\n", refOpKindToWireWord(kind)); + expectDelta(old_bytes, lineAt(encodeRefLogTxn(txn), 2), 0); + } +} + +TEST(CASWireCutDeltas, CommittedRefSnapshotRow) +{ + RefTableSnapshot snapshot; + snapshot.ns = "root"; + snapshot.snapshot_id = RefTxnId{1, 2}; + snapshot.committed.push_back(RefCommittedRow{"r", ManifestRef{3, 4, 5}, 6}); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"c\",\"rn\":\"r\",\"me\":\"3\",\"mb\":\"4\",\"mo\":5,\"ts\":6}\n"; + expectDelta(old_bytes, lineAt(encodeRefTableSnapshot(snapshot), 2), 29); +} + +TEST(CASWireCutDeltas, PrecommitRefSnapshotRow) +{ + RefTableSnapshot snapshot; + snapshot.ns = "root"; + snapshot.snapshot_id = RefTxnId{1, 2}; + snapshot.precommits.push_back(RefOwnerBinding{RefOwnerKind::Precommit, "r", ManifestRef{3, 4, 5}}); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"p\",\"rn\":\"r\",\"me\":\"3\",\"mb\":\"4\",\"mo\":5}\n"; + expectDelta(old_bytes, lineAt(encodeRefTableSnapshot(snapshot), 2), 19); +} + +TEST(CASWireCutDeltas, BaseRefCatalogRow) +{ + RefCatalog catalog{{CatalogEntry{.ns = RootNamespace{"root"}, .state = NsState::Live, .incarnation = UInt128(7)}}}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"ent\",\"ns\":\"root\",\"st\":\"live\",\"inc\":\"00000000000000000000000000000007\"}\n"; + expectDelta(old_bytes, lineAt(encodeRefCatalog(catalog), 1), 9); +} + +/// The base row's delta is 22 bytes of keys and tags plus the `class` word, which costs one byte more +/// than its length (quotes, less the single numeric digit it replaces). Each word is measured +/// separately: a range over all four would accept a key rename hiding inside the spread, and a single +/// fixture would pin only one point of it. `clamped` cannot be a base row at all -- the grammar +/// requires a hold on exactly those rows -- so it is measured whole and its base part recovered by +/// subtracting the hold segment the next test pins. +TEST(CASWireCutDeltas, BaseRefLifeFoldSealRow) +{ + const auto base_delta = [](CoverageClass classification, uint8_t old_wire_value) + { + CasFoldSeal seal = oneFoldSeal(); + seal.ref_lives[UInt128(1)].coverage + = RefCoverage{.classification = classification, .last_folded_ref_id = RefTxnId{7, 11}}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = fmt::format( + "{{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":{},\"lfe\":\"7\",\"lfs\":\"11\"}}\n", + old_wire_value); + return lineAt(encodeFoldSeal(seal), 2).size() - old_bytes.size(); + }; + + /// The pre-cut wire numbered these 0/1/2, not the current enum's values. + EXPECT_EQ(base_delta(CoverageClass::Absent, 0), 29u); /// 22 + "absent" + EXPECT_EQ(base_delta(CoverageClass::Unchanged, 1), 32u); /// 22 + "unchanged" + EXPECT_EQ(base_delta(CoverageClass::Folded, 2), 29u); /// 22 + "folded" +} + +/// The ADDITIONS a hold contributes, isolated from the row it rides on. The with/without trick used +/// for cleanup evidence is unavailable here: the grammar requires a hold on exactly the clamped rows, +/// so a clamped row WITHOUT one cannot be encoded at all. Instead both sides are cut down to the hold +/// segment itself -- from its first key to the closing brace -- so the tag, the `class` word and the +/// fold pair are outside the comparison by construction rather than by cancellation. +TEST(CASWireCutDeltas, HoldBearingRefLifeAdditions) +{ + CasFoldSeal seal = oneFoldSeal(); + seal.ref_lives[UInt128(1)].coverage = RefCoverage{.classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{7, 11}, .hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{12, 13}, .retry_count = 14, .next_retry_round = 15}}; + + /// This literal is the pre-cut baseline this delta is measured against. + const String old_row = "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":4,\"lfe\":\"7\",\"lfs\":\"11\",\"hr\":\"gap_below_witness\",\"hpe\":\"12\",\"hps\":\"13\",\"hrc\":14,\"hnr\":\"15\"}\n"; + const String new_row = lineAt(encodeFoldSeal(seal), 2); + + const auto hold_segment = [](const String & row, std::string_view first_hold_key) + { + const size_t from = row.find(first_hold_key); + const size_t to = row.rfind('}'); + EXPECT_NE(from, String::npos) << "row does not carry " << first_hold_key << ": " << row; + EXPECT_NE(to, String::npos); + return to > from ? to - from : 0; + }; + + EXPECT_EQ(hold_segment(new_row, ",\"hold_reason\"") - hold_segment(old_row, ",\"hr\""), 33u); +} + +/// The cleanup-evidence pair isolated the same way: with and without, on both sides, so only the +/// two added keys remain in the difference. +TEST(CASWireCutDeltas, CleanupEvidenceRefLifeAdditions) +{ + CasFoldSeal without_evidence = oneFoldSeal(); + without_evidence.ref_lives[UInt128(1)] = RefLifeFoldState{.coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{7, 11}}}; + CasFoldSeal with_evidence = oneFoldSeal(); + with_evidence.ref_lives[UInt128(1)] = RefLifeFoldState{.coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{7, 11}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{12, 13}}}; + + /// These literals are the pre-cut baselines these deltas are measured against. + const String old_without = "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\"}\n"; + const String old_with = "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\",\"rte\":\"12\",\"rts\":\"13\"}\n"; + + const size_t base_delta = lineAt(encodeFoldSeal(without_evidence), 2).size() - old_without.size(); + const size_t whole_delta = lineAt(encodeFoldSeal(with_evidence), 2).size() - old_with.size(); + EXPECT_EQ(whole_delta - base_delta, 16u); +} + +TEST(CASWireCutDeltas, BlobRunFoldSealRow) +{ + CasFoldSeal seal = oneFoldSeal(); + seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(15), .shard = 0, .key_generation = 5}); + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"btr\",\"key\":\"r0\",\"ck\":\"0000000000000000000000000000000f\",\"shard\":0,\"gen\":\"5\"}\n"; + expectDelta(old_bytes, lineAt(encodeFoldSeal(seal), 2), 25); +} + +TEST(CASWireCutDeltas, CondemnedFoldSealSummaryRow) +{ + CasFoldSeal seal = oneFoldSeal(); + seal.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 4}; + /// This literal is the pre-cut baseline this delta is measured against. + const String old_bytes = "{\"k\":\"cnd\",\"shard\":0,\"ct\":3,\"pt\":1,\"ocr\":\"4\"}\n"; + expectDelta(old_bytes, lineAt(encodeFoldSeal(seal), 2), 30); +} diff --git a/src/Disks/tests/gtest_cas_enum_wire_table.cpp b/src/Disks/tests/gtest_cas_enum_wire_table.cpp index 90f59c866cb2..7940c57944eb 100644 --- a/src/Disks/tests/gtest_cas_enum_wire_table.cpp +++ b/src/Disks/tests/gtest_cas_enum_wire_table.cpp @@ -107,7 +107,7 @@ constexpr EnumWireTable invalid_value{{{ {Fruit::Apple, "apple"}, {Fruit::Pear, "pear"}, {static_cast(99), "plum"}}}}; static_assert(!casEnumTableCoversEnum()); -/// The two cases above fail the folded density check before the set-equality core runs, so the +/// `dup_value` and `invalid_value` fail the folded density check before the set-equality core runs, so the /// core needs its own failing witnesses — both dense and word-unique, so they reach it. /// Reaches the size comparison: one enumerator short. constexpr EnumWireTable missing_enumerator{{{ diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index 808830e46a9f..82920acb177c 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -554,7 +554,7 @@ String publishOneBlobPart(const PoolPtr & s, const String & ns, const String & r /// Whether the CURRENT retired list (any gc-shard) still holds an entry (ack-floor pipeline in flight). bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_fold_seal_codec.cpp b/src/Disks/tests/gtest_cas_fold_seal_codec.cpp index feee48a70069..dabde6cf3b82 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_codec.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_codec.cpp @@ -23,7 +23,7 @@ TEST(CASFoldSealCodec, RefLifeCoverageRoundTripsLastFoldedRefId) seal.generation = 3; seal.parent_generation = 2; RefCoverage cov; - cov.classification = 1; + cov.classification = CoverageClass::Unchanged; cov.last_folded_ref_id = RefTxnId{4, 11}; constexpr UInt128 life_id{1}; seal.ref_lives[life_id].coverage = cov; diff --git a/src/Disks/tests/gtest_cas_fold_seal_format.cpp b/src/Disks/tests/gtest_cas_fold_seal_format.cpp index d5e2144093eb..ba8c6f131be9 100644 --- a/src/Disks/tests/gtest_cas_fold_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_fold_seal_format.cpp @@ -4,6 +4,8 @@ #include #include +#include + using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; extern const int LOGICAL_ERROR; } @@ -15,8 +17,8 @@ CasFoldSeal sampleFoldSeal() CasFoldSeal seal; seal.generation = 7; seal.parent_generation = 6; - seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{3, 4}}; - seal.ref_lives[UInt128{2}].coverage = RefCoverage{.classification = 1}; + seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{3, 4}}; + seal.ref_lives[UInt128{2}].coverage = RefCoverage{.classification = CoverageClass::Unchanged}; seal.blob_target_runs.push_back(RunRef{.key = "gc/gen/7/blob_target/0/0", .checksum = UInt128(0xABCDEF)}); return seal; } @@ -36,7 +38,7 @@ TEST(CASFormatBattery, FoldSeal) CasFoldSeal seal; seal.generation = 5; seal.parent_generation = 4; - seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{7, 11}}; + seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{7, 11}}; seal.blob_target_runs.push_back(RunRef{.key = "r0", .checksum = UInt128(0x0f), .shard = 0, .key_generation = 5}); seal.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, .oldest_nonpending_condemn_round = 4}; @@ -44,10 +46,10 @@ TEST(CASFormatBattery, FoldSeal) [&] { return sealObject(FormatId::FoldSeal, encodeFoldSeal(seal)); }, [](std::string_view s) { decodeFoldSeal(std::string(openObject(FormatId::FoldSeal, s))); }, currentFormatHeader("cas_fold_seal") + - "{\"g\":\"5\",\"pg\":\"4\"}\n" - "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000000001\",\"cls\":2,\"lfe\":\"7\",\"lfs\":\"11\"}\n" - "{\"k\":\"btr\",\"key\":\"r0\",\"ck\":\"0000000000000000000000000000000f\",\"shard\":0,\"gen\":\"5\"}\n" - "{\"k\":\"cnd\",\"shard\":0,\"ct\":3,\"pt\":1,\"ocr\":\"4\"}\n" + "{\"generation\":\"5\",\"parent_generation\":\"4\"}\n" + "{\"kind\":\"ref_life\",\"life\":\"00000000000000000000000000000001\",\"class\":\"folded\",\"fold_epoch\":\"7\",\"fold_seq\":\"11\"}\n" + "{\"kind\":\"blob_run\",\"key\":\"r0\",\"checksum\":\"0000000000000000000000000000000f\",\"shard\":0,\"key_generation\":\"5\"}\n" + "{\"kind\":\"condemned\",\"shard\":0,\"condemned\":3,\"pending\":1,\"oldest_round\":\"4\"}\n" "{\"n\":3}\n"}); } @@ -59,7 +61,7 @@ TEST(CASFoldSealFormat, RoundTripsAllFields) EXPECT_EQ(out.generation, in.generation); EXPECT_EQ(out.parent_generation, in.parent_generation); ASSERT_EQ(out.ref_lives.size(), in.ref_lives.size()); - EXPECT_EQ(out.ref_lives.at(UInt128{1}).coverage.classification, 2); + EXPECT_EQ(out.ref_lives.at(UInt128{1}).coverage.classification, CoverageClass::Folded); EXPECT_EQ(out.ref_lives.at(UInt128{1}).coverage.last_folded_ref_id, (RefTxnId{3, 4})); ASSERT_EQ(out.blob_target_runs.size(), 1u); EXPECT_EQ(out.blob_target_runs[0].key, "gc/gen/7/blob_target/0/0"); @@ -125,11 +127,11 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel for (const std::string_view field : { R"(,"key":"p/gc/gen/7/attempt/1/blob_target/0/0")", - R"(,"ck":"00000000000000000000000000000001")", - R"(,"gen":"7")", - ",\"ct\":0", - ",\"pt\":0", - R"(,"ocr":"18446744073709551615")"}) + R"(,"checksum":"00000000000000000000000000000001")", + R"(,"key_generation":"7")", + ",\"condemned\":0", + ",\"pending\":0", + R"(,"oldest_round":"18446744073709551615")"}) { String malformed = valid; eraseRequiredField(malformed, field); @@ -138,19 +140,19 @@ TEST(CASFoldSealFormat, AuthoritativeDecodeRequiresEveryBlobTargetAndSummaryFiel } /// `shard` occurs once on each row; remove each occurrence independently. - String missing_btr_shard = valid; - eraseRequiredField(missing_btr_shard, ",\"shard\":0"); + String missing_blob_run_shard = valid; + eraseRequiredField(missing_blob_run_shard, ",\"shard\":0"); cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(missing_btr_shard, layout, 1); }, "missing"); + [&] { decodeFoldSeal(missing_blob_run_shard, layout, 1); }, "missing"); - String missing_cnd_shard = valid; - const size_t first_shard = missing_cnd_shard.find(",\"shard\":0"); + String missing_condemned_shard = valid; + const size_t first_shard = missing_condemned_shard.find(",\"shard\":0"); ASSERT_NE(first_shard, String::npos); - const size_t second_shard = missing_cnd_shard.find(",\"shard\":0", first_shard + 1); + const size_t second_shard = missing_condemned_shard.find(",\"shard\":0", first_shard + 1); ASSERT_NE(second_shard, String::npos); - missing_cnd_shard.erase(second_shard, std::string_view(",\"shard\":0").size()); + missing_condemned_shard.erase(second_shard, std::string_view(",\"shard\":0").size()); cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(missing_cnd_shard, layout, 1); }, "missing"); + [&] { decodeFoldSeal(missing_condemned_shard, layout, 1); }, "missing"); } TEST(CASFoldSealFormat, AuthoritativeDecodeRejectsNoncanonicalRowsAndIncompleteSummaryDomain) @@ -241,7 +243,7 @@ TEST(CASFoldSeal, RejectsEmptyAndBadMagic) TEST(CASFoldSeal, CoverageRecordsEveryCatalogLife) { CasFoldSeal in = sampleFoldSeal(); - in.ref_lives[UInt128{3}].coverage = RefCoverage{.classification = 0}; + in.ref_lives[UInt128{3}].coverage = RefCoverage{.classification = CoverageClass::Absent}; const CasFoldSeal out = decodeFoldSeal(encodeFoldSeal(in)); EXPECT_TRUE(out.ref_lives.contains(UInt128{3})); EXPECT_EQ(out.ref_lives.size(), 3u); @@ -254,7 +256,7 @@ TEST(CASFoldSeal, FoldSealCondemnedSummaryRoundTrips) CasFoldSeal s; s.generation = 9; s.parent_generation = 8; - s.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = 2}; + s.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Folded}; s.blob_target_runs.push_back(RunRef{.key = "gc/gen/9/blob_target/0/0", .checksum = UInt128(0x77), .shard = 0, .key_generation = 9}); s.condemned_summary[0] = CondemnedSummary{.condemned_total = 3, .pending_total = 1, @@ -283,7 +285,7 @@ TEST(CASFoldSealFormat, UnifiedRefLifeRowRoundTripsCoverageHoldAndCleanupEvidenc const UInt128 life_id{0x1234}; seal.ref_lives.emplace(life_id, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{3, 4}, .hold = RefHold{ .reason = HoldReason::ManifestBodyMissing, @@ -293,36 +295,59 @@ TEST(CASFoldSealFormat, UnifiedRefLifeRowRoundTripsCoverageHoldAndCleanupEvidenc .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{9, 10}}}); const String expected = currentFormatHeader("cas_fold_seal") + - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"rfl\",\"life\":\"00000000000000000000000000001234\",\"cls\":4," - "\"lfe\":\"3\",\"lfs\":\"4\",\"hr\":\"manifest_body_missing\",\"hpe\":\"5\"," - "\"hps\":\"6\",\"hrc\":7,\"hnr\":\"8\",\"rte\":\"9\",\"rts\":\"10\"}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"ref_life\",\"life\":\"00000000000000000000000000001234\",\"class\":\"clamped\"," + "\"fold_epoch\":\"3\",\"fold_seq\":\"4\",\"hold_reason\":\"manifest_body_missing\",\"hold_epoch\":\"5\"," + "\"hold_seq\":\"6\",\"retries\":7,\"retry_round\":\"8\",\"remove_epoch\":\"9\",\"remove_seq\":\"10\"}\n" "{\"n\":1}\n"; EXPECT_EQ(encodeFoldSeal(seal), expected); EXPECT_EQ(decodeFoldSeal(expected), seal); } -/// Mutation caught: accepting the generation-6 split coverage collection would leave a second -/// namespace-keyed source of lifecycle work in a generation-7 process. +/// Closed-set pin: `CoverageClass` and `HoldReason` +/// each walked through `magic_enum::enum_values`, which is what proves the renderer and the parser +/// consult the SAME table: a table entry missing altogether is already a build error at the +/// coverage assert, but two delegates drifting onto different tables is not. +TEST(CASFoldSealFormat, ClosedSetPinsCoverageClassAndHoldReasonWords) +{ + EXPECT_EQ(coverageClassToWord(CoverageClass::Absent), "absent"); + EXPECT_EQ(coverageClassToWord(CoverageClass::Unchanged), "unchanged"); + EXPECT_EQ(coverageClassToWord(CoverageClass::Folded), "folded"); + EXPECT_EQ(coverageClassToWord(CoverageClass::Clamped), "clamped"); + for (const auto c : magic_enum::enum_values()) + EXPECT_EQ(coverageClassFromWord(coverageClassToWord(c)), c); + + EXPECT_EQ(holdReasonToWord(HoldReason::GapBelowWitness), "gap_below_witness"); + EXPECT_EQ(holdReasonToWord(HoldReason::UnconsumedSealCrossing), "unconsumed_seal_crossing"); + EXPECT_EQ(holdReasonToWord(HoldReason::WitnessDisappeared), "witness_disappeared"); + EXPECT_EQ(holdReasonToWord(HoldReason::BodyUndecodable), "body_undecodable"); + EXPECT_EQ(holdReasonToWord(HoldReason::ManifestBodyMissing), "manifest_body_missing"); + EXPECT_EQ(holdReasonToWord(HoldReason::CheckpointUndecodable), "checkpoint_undecodable"); + for (const auto r : magic_enum::enum_values()) + EXPECT_EQ(holdReasonFromWord(holdReasonToWord(r)), r); +} + +/// Mutation caught: accepting the retired split coverage-collection kind would revive a second +/// namespace-keyed source of lifecycle work alongside the unified per-life row. TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyCoverageRecord) { const String old = - "{\"type\":\"cas_fold_seal\",\"v\":7}\n" - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"cov\",\"key\":\"name/0\",\"cls\":2,\"lfe\":\"3\",\"lfs\":\"4\"}\n" + "{\"type\":\"cas_fold_seal\",\"v\":1}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"cov\",\"key\":\"name/0\",\"class\":\"folded\",\"fold_epoch\":\"3\",\"fold_seq\":\"4\"}\n" "{\"n\":1}\n"; cas_battery_detail::expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(old); }, "legacy coverage"); } -/// Mutation caught: accepting the generation-6 cleanup-item state would restore the independent -/// marker-driven `Pending`/`Completed` handshake. +/// Mutation caught: accepting the retired cleanup-item kind would restore the independent +/// marker-driven `Pending`/`Completed` handshake the unified row replaced. TEST(CASFoldSealFormat, UnifiedCodecRejectsLegacyNamespaceCleanupRecord) { const String old = - "{\"type\":\"cas_fold_seal\",\"v\":7}\n" - "{\"g\":\"8\",\"pg\":\"7\"}\n" - "{\"k\":\"nsc\",\"ns\":\"name\",\"rte\":\"3\",\"rts\":\"4\",\"st\":\"completed\"}\n" + "{\"type\":\"cas_fold_seal\",\"v\":1}\n" + "{\"generation\":\"8\",\"parent_generation\":\"7\"}\n" + "{\"kind\":\"nsc\",\"ns\":\"name\",\"remove_epoch\":\"3\",\"remove_seq\":\"4\",\"st\":\"completed\"}\n" "{\"n\":1}\n"; cas_battery_detail::expectCode( DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(old); }, "legacy namespace cleanup"); diff --git a/src/Disks/tests/gtest_cas_forget.cpp b/src/Disks/tests/gtest_cas_forget.cpp index 2d8a72fc1850..b1c19993e06c 100644 --- a/src/Disks/tests/gtest_cas_forget.cpp +++ b/src/Disks/tests/gtest_cas_forget.cpp @@ -306,7 +306,7 @@ TEST(CASForget, ForgetOnIdentityLostPoolVanishesForgotten) } /// (a'') The clean-farewell is EARNED, never unconditional: on a drained pool FORGET stamps the mount lease -/// with the terminated sentinel (`min_active == UINT64_MAX`) so a same-server restart reclaims immediately, +/// with the terminated sentinel (`min_active_build_sequence == UINT64_MAX`) so a same-server restart reclaims immediately, /// but with an UNSETTLED (wedged) ref lane it must NOT — the lease is left to expire by observation. TEST(CASForget, ForgetCleanFarewellGatedOnDrain) { @@ -318,14 +318,14 @@ TEST(CASForget, ForgetCleanFarewellGatedOnDrain) auto backend = std::make_shared(); auto store = DB::Cas::tests::openPoolForTest(backend); const String mount_key = store->layout().mountKey(kSrid); - ASSERT_NE(decodeMountLease(backend->get(mount_key)->bytes).min_active, kTerminated); /// baseline + ASSERT_NE(decodeMountLease(backend->get(mount_key)->bytes).min_active_build_sequence, kTerminated); /// baseline store->forgetDisk([] {}, kForgetReason); ASSERT_EQ(store->lifecycle(), PoolLifecycle::VanishedForgotten); const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()); - EXPECT_EQ(decodeMountLease(got->bytes).min_active, kTerminated) + EXPECT_EQ(decodeMountLease(got->bytes).min_active_build_sequence, kTerminated) << "a drained FORGET earns the clean-release farewell"; } @@ -344,7 +344,7 @@ TEST(CASForget, ForgetCleanFarewellGatedOnDrain) const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()) << "the lease object must still be present (expiry by observation)"; - EXPECT_NE(decodeMountLease(got->bytes).min_active, kTerminated) + EXPECT_NE(decodeMountLease(got->bytes).min_active_build_sequence, kTerminated) << "an unearned clean farewell must NOT be written when the ref lanes did not drain"; } } diff --git a/src/Disks/tests/gtest_cas_format.cpp b/src/Disks/tests/gtest_cas_format.cpp index 9aa93e99f76f..6c5ad84cc0cc 100644 --- a/src/Disks/tests/gtest_cas_format.cpp +++ b/src/Disks/tests/gtest_cas_format.cpp @@ -1,7 +1,8 @@ #include #include -#include #include +#include +#include namespace DB::ErrorCodes { @@ -11,93 +12,52 @@ namespace DB::ErrorCodes using namespace DB::Cas; -TEST(CASFormat, ChangePointsExistForEveryClass) +/// Closed-set pin: the registry's complete set of object `type` strings. `allRegisteredFormatIds` +/// is the registry's own enumeration accessor, so this walks the SAME set the codecs and the object +/// header gate see -- a registered class with no test coverage here is a registered class this test +/// cannot see either, which is the point: a 17th, 18th, ... entry the spec's closed set does not +/// name would show up as a set-size mismatch instead of passing unnoticed. +TEST(CASFormat, RegistryTypeStringsArePinnedClosedSet) { - /// Every class that existed from the start has a non-empty, gen-1 baseline. - for (auto id : {FormatId::Blob, - FormatId::GcState, - FormatId::PoolMeta, FormatId::Roster, - FormatId::GcOutcomes, - FormatId::PartManifest, FormatId::RunFile, - FormatId::FoldSeal}) - { - auto cps = changePoints(id); - ASSERT_FALSE(cps.empty()); - EXPECT_EQ(cps.front().generation, 1u); - EXPECT_EQ(cps.front().min_reader, 1u); - } -} - -/// A class BORN after generation 1 begins its history at its birth generation, not at 1. `RefCkpt` -/// (spec INV-4) was introduced at generation 4: there is no such thing as a generation-1 `_ckpt`, and a -/// `{1, 1}` baseline would assert that a generation-1 reader could read one. Its history then gained a -/// three later breaking entries: generation 5 re-keyed it under `//`, generation 6 -/// moved it to opaque life-owned state, and generation 9 added the exact committed frontier. Neither -/// change touches the gen-1 baseline. Pinned because the decision is -/// invisible otherwise — nothing consults `changePoints` at decode time yet, so a wrong entry here -/// would sit unnoticed until the day a per-class reader floor is wired and starts admitting objects it -/// should refuse. -TEST(CASFormat, ChangePointsOfAClassBornAfterGenerationOneStartAtItsBirth) -{ - const auto cps = changePoints(FormatId::RefCkpt); - ASSERT_EQ(cps.size(), 4u); - EXPECT_EQ(cps.front().generation, kContiguousRefStreamsGeneration); - EXPECT_EQ(cps.front().min_reader, kContiguousRefStreamsGeneration); - EXPECT_GT(cps.front().generation, 1u) << "the point of this test is that it is NOT the gen-1 baseline"; - EXPECT_EQ(cps[1].generation, kNamespaceLifeKeyedGeneration); - EXPECT_EQ(cps[1].min_reader, kNamespaceLifeKeyedGeneration); - EXPECT_EQ(cps[2].generation, kOpaqueNamespaceLifeLayoutGeneration); - EXPECT_EQ(cps[2].min_reader, kOpaqueNamespaceLifeLayoutGeneration); - EXPECT_EQ(cps.back().generation, kCommittedRefFrontierGeneration); - EXPECT_EQ(cps.back().min_reader, kCommittedRefFrontierGeneration); -} - -TEST(CASFormat, PoolMetaTracksTheRecreateOnlyRecoveryFrontierGeneration) -{ - const auto cps = changePoints(FormatId::PoolMeta); - ASSERT_EQ(cps.size(), 4u); - EXPECT_EQ(cps.back().generation, kMountWriteAttemptIdGeneration); - EXPECT_EQ(cps.back().min_reader, kMountWriteAttemptIdGeneration); -} - -TEST(CASFormat, MountAttemptIdentityIsARecreateOnlyGenerationTenChange) -{ - EXPECT_EQ(G_BUILD, 10u); - EXPECT_EQ(kMountWriteAttemptIdGeneration, 10u); + const std::set expected{ + "cas_blob", "cas_blob_meta", "cas_pool_meta", "cas_ref_log", "cas_ref_snap", + "cas_ref_ckpt", "cas_ref_catalog", "cas_gc_maintenance_state", "cas_part_manifest", + "cas_run", "cas_fold_seal", "cas_gc_state", "cas_gc_hb", "cas_gc_outcomes", + "cas_owner", "cas_epoch", "cas_mount_lease"}; + ASSERT_EQ(expected.size(), 17u); - const auto mount_points = changePoints(FormatId::MountLease); - ASSERT_EQ(mount_points.back().generation, kMountWriteAttemptIdGeneration); - EXPECT_EQ(mount_points.back().min_reader, kMountWriteAttemptIdGeneration); + std::set actual; + for (const auto id : allRegisteredFormatIds()) + actual.insert(traitsFor(id).type); + EXPECT_EQ(actual, expected); - const auto pool_points = changePoints(FormatId::PoolMeta); - ASSERT_EQ(pool_points.back().generation, kMountWriteAttemptIdGeneration); - EXPECT_EQ(pool_points.back().min_reader, kMountWriteAttemptIdGeneration); + for (const auto & type : expected) + { + const FormatTraits * t = traitsForType(type); + ASSERT_NE(t, nullptr) << type; + EXPECT_EQ(t->type, type); + } } -TEST(CASPoolMeta, GenerationNinePoolIsRejectedAtReaderFloor) +/// The generation history is reset to a flat `{1, 1}` baseline for every class: CAS is pre-release and +/// carries no persisted data, so there is no compatibility cost to starting the count over. Pinned +/// because the decision is invisible otherwise — nothing consults `changePoints` at decode time yet, so +/// a wrong entry here would sit unnoticed until the day a per-class reader floor is wired and starts +/// admitting objects it should refuse. +TEST(CASFormat, EveryClassResetToTheBaselineGeneration) { - PoolMeta meta; - meta.pool_id = UInt128{1}; - meta.blob_header_len = 256; - meta.gc_shards = 1; - meta.min_reader_generation = 10; - meta.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - String encoded = encodePoolMeta(meta); - const String current = "\"v\":10"; - const size_t version = encoded.find(current); - ASSERT_NE(version, String::npos); - encoded.replace(version, current.size(), "\"v\":9"); - - try - { - decodePoolMeta(encoded); - FAIL() << "expected UNKNOWN_FORMAT_VERSION"; - } - catch (const DB::Exception & e) + for (auto id : allRegisteredFormatIds()) { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find("generation-10 mount-attempt-identity"), String::npos); + const auto cps = changePoints(id); + ASSERT_EQ(cps.size(), 1u) << "FormatId " << static_cast(id); + EXPECT_EQ(cps.front().generation, 1u); + EXPECT_EQ(cps.front().min_reader, 1u); } + + const auto roster_cps = changePoints(FormatId::Roster); + ASSERT_EQ(roster_cps.size(), 1u); + EXPECT_EQ(roster_cps.front().generation, 1u); + EXPECT_EQ(roster_cps.front().min_reader, 1u); } TEST(CASFormat, CurrentVersionsAreGBuild) diff --git a/src/Disks/tests/gtest_cas_format_battery.cpp b/src/Disks/tests/gtest_cas_format_battery.cpp index d6e9f01b3535..21251204d8a5 100644 --- a/src/Disks/tests/gtest_cas_format_battery.cpp +++ b/src/Disks/tests/gtest_cas_format_battery.cpp @@ -31,14 +31,28 @@ TEST(CASFormatBattery, PoolMeta) PoolMeta pm; pm.pool_id = hexToU128("00112233445566778899aabbccddeeff"); pm.blob_header_len = 256; - pm.min_reader_generation = 3; + pm.min_reader_generation = 1; pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; runFormatBattery(FormatBatteryCase{ .id = FormatId::PoolMeta, .encode = [&] { return sealObject(FormatId::PoolMeta, encodePoolMeta(pm)); }, .decode = [](std::string_view s) { decodePoolMeta(std::string(openObject(FormatId::PoolMeta, s))); }, .golden = currentFormatHeader("cas_pool_meta") + - "{\"pid\":\"00112233445566778899aabbccddeeff\",\"hln\":256,\"gcs\":1,\"mrg\":3,\"alg\":\"ch128\"}\n"}); + "{\"pool_id\":\"00112233445566778899aabbccddeeff\",\"blob_header_len\":256,\"gc_shards\":1,\"min_reader_generation\":1,\"algos_used\":[\"ch128\"]}\n"}); +} + +TEST(CASPoolMeta, RejectsInvalidAlgoArrays) +{ + const auto decode = [](std::string_view algos_used) + { + return decodePoolMeta("{\"type\":\"cas_pool_meta\",\"v\":1}\n" + "{\"pool_id\":\"00112233445566778899aabbccddeeff\",\"blob_header_len\":256,\"gc_shards\":1,\"min_reader_generation\":1,\"algos_used\":" + String(algos_used) + "}\n"); + }; + + /// The first value is the field's PREVIOUS encoding -- a comma-joined string inside one JSON + /// value. It must fail closed rather than round-trip; the rest are malformed arrays. + for (const std::string_view bad : {"\"ch128,sha256\"", "[\"ch128\",1]", "[]", "[\"sha256\",\"ch128\"]", "[\"ch128\",\"ch128\"]", "[\"unknown\"]"}) + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decode(bad); }); } TEST(CASPoolMeta, ValidateAlgosUsedRejectsUnknownByte) diff --git a/src/Disks/tests/gtest_cas_fsck.cpp b/src/Disks/tests/gtest_cas_fsck.cpp index 71abb917eaa2..77c490656911 100644 --- a/src/Disks/tests/gtest_cas_fsck.cpp +++ b/src/Disks/tests/gtest_cas_fsck.cpp @@ -811,10 +811,10 @@ TEST(CASFsckAuthority, MissingBurnedEpochSealIsChainBroken) /// intermediate epoch is reported rather than treated as a sparse legal transition. String skipped_bytes = encodeRefLogTxn(RefLogTxn{ .ns = ns.string(), .txn_id = RefTxnId{7, 1}, .ops = {}, .prev_epoch_seal = RefTxnId{6, 1}}); - const String old_epoch_token = R"("!pse":"6")"; + const String old_epoch_token = R"("!prev_epoch":"6")"; const auto old_epoch = skipped_bytes.find(old_epoch_token); ASSERT_NE(old_epoch, String::npos); - skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!pse":"1")"); + skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!prev_epoch":"1")"); ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, skipped_bytes)).outcome, PutOutcome::Done); diff --git a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp index 870bcdc20295..ed4e76fac722 100644 --- a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp +++ b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp @@ -26,7 +26,7 @@ /// * absent at `expected`, no listed id above it => the namespace's frontier this round (normal end) /// * absent at `expected`, a listed id above it => IMPOSSIBLE under contiguity: the store is lying /// or a durable record was lost. Hold the namespace -/// (classification 4), cursor unmoved. +/// (classification `Clamped`), cursor unmoved. /// /// Epochs are crossed ONLY by consuming the `EpochSeal` that closes an epoch (INV-2). The seal folds as /// an applied no-op (probe B2: `produced=false`), and the next epoch's start is `{E', 1}` -- reached @@ -77,10 +77,10 @@ RefTxnId cursorOf(Backend & backend, const Layout & layout, const RootNamespace return cov ? cov->last_folded_ref_id : RefTxnId{}; } -uint8_t classificationOf(Backend & backend, const Layout & layout, const RootNamespace & ns) +CoverageClass classificationOf(Backend & backend, const Layout & layout, const RootNamespace & ns) { const auto cov = coverageOf(backend, layout, ns); - return cov ? cov->classification : 0; + return cov ? cov->classification : CoverageClass::Absent; } /// The `fold_ref_intake` phase metrics of the round `sched` runs -- the only place probe B1's two @@ -135,7 +135,7 @@ TEST(CASGCArithmeticIntake, HintOmittingMiddleRecordsFoldsThroughUnnoticed) ASSERT_GT(backend->holesServed(), 0u) << "the hint hole was never actually served"; EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 5})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2) << "a folded namespace is `changed`"; + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded) << "a folded namespace is `changed`"; for (uint64_t i = 1; i <= 5; ++i) EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(i)), 1) << "blob " << i << " lost its owner edge: its record was skipped because the hint omitted it"; @@ -166,13 +166,13 @@ TEST(CASGCArithmeticIntake, WalkEndsAtFrontierWithoutHold) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 3})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded); /// A second round over an unchanged namespace pays exactly one exact GET, finds the same frontier, /// and neither advances nor holds. ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 3})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 1) << "an unchanged namespace is `carried`"; + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Unchanged) << "an unchanged namespace is `carried`"; } /// ===================== EPOCHS ARE CROSSED ONLY BY CONSUMING A SEAL ===================== @@ -211,7 +211,7 @@ TEST(CASGCArithmeticIntake, SealCrossesEpochAndIsAppliedAsNoOp) ASSERT_GT(backend->holesServed(), 0u); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{2, 2})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded); for (uint64_t i = 1; i <= 4; ++i) EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(i)), 1) << "blob " << i; } @@ -291,19 +291,19 @@ TEST(CASGCArithmeticIntake, CursorRestingOnSealCrossesInALaterRound) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{2, 1})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Folded); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(2)), 1); } /// ===================== IMPOSSIBLE SHAPES HOLD THE NAMESPACE ===================== /// /// `{1,3}` is genuinely absent while `{1,4}` is present AND listed. Contiguity says that cannot happen, -/// so whatever sits behind the gap may be an acked `+1`: the namespace is held at classification 4 with -/// its cursor UNMOVED, rather than sealing past the gap. +/// so whatever sits behind the gap may be an acked `+1`: the namespace is held at classification +/// `Clamped` with its cursor UNMOVED, rather than sealing past the gap. /// /// Listing-driven intake folded `{1,4}` and sealed the cursor at it -- permanently, since a record below /// the cursor is never re-read. -TEST(CASGCArithmeticIntake, GapBelowWitnessHoldsNamespaceAtClassificationFour) +TEST(CASGCArithmeticIntake, GapBelowWitnessHoldsNamespaceAtClampedClassification) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); @@ -326,7 +326,7 @@ TEST(CASGCArithmeticIntake, GapBelowWitnessHoldsNamespaceAtClassificationFour) EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 2})) << "the cursor must not advance past a gap"; - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(4)), 0) << "the record above the gap was not folded"; } @@ -357,7 +357,7 @@ TEST(CASGCArithmeticIntake, UnconsumedSealCrossingHoldsNamespace) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 1})); - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); const auto coverage = coverageOf(*backend, layout, ns); ASSERT_TRUE(coverage && coverage->hold.has_value()); EXPECT_EQ(coverage->hold->reason, HoldReason::UnconsumedSealCrossing); @@ -395,7 +395,7 @@ TEST(CASGCArithmeticIntake, CrossingFromANonSealRecordIsRefusedEvenWhenTheChainM EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 2})) << "epoch 1 was never sealed, so the cursor may not leave it"; - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(1)), 1) << "epoch 1's records still fold"; EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(2)), 1); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(3)), 0) @@ -462,7 +462,7 @@ TEST(CASGCArithmeticIntake, EpochStartThatAnswersOnlyEveryOtherReadHoldsInsteadO EXPECT_EQ(cursorOf(*backend, layout, ns), (RefTxnId{1, 2})) << "the cursor stops on the seal it consumed and never enters the unstable epoch"; - EXPECT_EQ(classificationOf(*backend, layout, ns), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns), CoverageClass::Clamped); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(2)), 0); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(3)), 0) << "nothing above the unstable position may be folded either"; @@ -505,11 +505,11 @@ TEST(CASGCArithmeticIntake, CorruptBodyClampsOneNamespaceWhileAnotherFolds) ASSERT_TRUE(gc.runRegularRound().acquired_lease); EXPECT_EQ(cursorOf(*backend, layout, ns_a), (RefTxnId{1, 1})); - EXPECT_EQ(classificationOf(*backend, layout, ns_a), 4); + EXPECT_EQ(classificationOf(*backend, layout, ns_a), CoverageClass::Clamped); EXPECT_EQ(cursorOf(*backend, layout, ns_b), (RefTxnId{1, 3})) << "a sibling namespace's corrupt body must not stop this one"; - EXPECT_EQ(classificationOf(*backend, layout, ns_b), 2); + EXPECT_EQ(classificationOf(*backend, layout, ns_b), CoverageClass::Folded); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(11)), 1); EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(12)), 1); } @@ -582,7 +582,7 @@ TEST(CASGCArithmeticIntake, WhollyOmittedNamespaceFoldsThroughAuthoritativeCheck const auto hidden_cov = coverageOf(*backend, layout, ns); ASSERT_TRUE(hidden_cov.has_value()) << "the namespace is `Live` in the catalog, so it stays in the universe even fully hidden"; - EXPECT_EQ(hidden_cov->classification, 2) << "the checkpoint's frontier is folded by exact key"; + EXPECT_EQ(hidden_cov->classification, CoverageClass::Folded) << "the checkpoint's frontier is folded by exact key"; EXPECT_EQ(hidden_cov->last_folded_ref_id, (RefTxnId{1, 3})); /// The store stops lying: the already folded namespace reappears. diff --git a/src/Disks/tests/gtest_cas_gc_attempt.cpp b/src/Disks/tests/gtest_cas_gc_attempt.cpp index 49e63031c6df..0ae235777cbd 100644 --- a/src/Disks/tests/gtest_cas_gc_attempt.cpp +++ b/src/Disks/tests/gtest_cas_gc_attempt.cpp @@ -54,7 +54,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp index e9041dd7b223..384e9f29ea35 100644 --- a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp +++ b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp @@ -230,7 +230,7 @@ TEST(CASGCBoundedWalk, ARoundFoldsThroughItsRoundStartTailAndLeavesTheStragglers EXPECT_EQ(cov->last_folded_ref_id, (RefTxnId{1, planted})) << "the walk must fold through the round-start tail and no further -- it chased the writer"; EXPECT_FALSE(cov->hold.has_value()) << "reaching the committed frontier is not a hold"; - EXPECT_NE(cov->classification, 4) << "reaching the committed frontier is not a clamp"; + EXPECT_NE(cov->classification, CoverageClass::Clamped) << "reaching the committed frontier is not a clamp"; EXPECT_EQ(metric(intake, "tails_advanced"), 1u); EXPECT_EQ(metric(intake, "logs_applied"), planted) << "exactly the round-start backlog was folded"; @@ -505,7 +505,7 @@ TEST(CASGCBoundedWalk, AnAbsentManifestBodyStillHoldsWithoutAHead) ASSERT_TRUE(cov->hold.has_value()) << "an absent committed manifest body raises the fold barrier"; EXPECT_EQ(cov->hold->reason, HoldReason::ManifestBodyMissing); EXPECT_EQ(cov->hold->offending_position, (RefTxnId{1, 2})); - EXPECT_EQ(cov->classification, 4); + EXPECT_EQ(cov->classification, CoverageClass::Clamped); EXPECT_EQ(backend->headCount(layout.manifestKey(gone)), 0u) << "absence is decided by the GET, so the missing body costs no HEAD either"; } @@ -558,13 +558,13 @@ TEST(CASGCBoundedWalk, ANamespaceThatFoldedNothingKeepsItsSealedCursor) ASSERT_TRUE(after.has_value()) << "the coverage row was DROPPED -- the next round would re-fold this namespace from {0,0}"; /// The CURSOR and the HOLD are what the next round trusts, and both ride unchanged. - /// `classification` legitimately moves from 2 ("this round folded records") to 1 ("unchanged"), + /// `classification` legitimately moves from `Folded` ("this round folded records") to `Unchanged`, /// because that is what the round did — it is the one field that may differ, so it is the one field /// asserted loosely. EXPECT_EQ(after->last_folded_ref_id, before->last_folded_ref_id) << "a namespace that folded nothing must keep the cursor it had"; EXPECT_EQ(after->hold, before->hold); - EXPECT_NE(after->classification, 4) << "folding nothing is not a clamp"; + EXPECT_NE(after->classification, CoverageClass::Clamped) << "folding nothing is not a clamp"; EXPECT_EQ(metric(intake, "frontier_namespaces"), 2u) << "it stays in the round's universe, so its proof is still owed"; } diff --git a/src/Disks/tests/gtest_cas_gc_fold.cpp b/src/Disks/tests/gtest_cas_gc_fold.cpp index c3af2cbe4ef0..78435a6c2d78 100644 --- a/src/Disks/tests/gtest_cas_gc_fold.cpp +++ b/src/Disks/tests/gtest_cas_gc_fold.cpp @@ -464,7 +464,7 @@ TEST(CASGCFold, DeadPrecommitWithMissingBodyIsSkippedNotClampedForever) auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); /// The namespace's server-root prefix is "srv"; seed its watermark floor so build_sequence 5 is retired. const RootNamespace ns{"srv/tbl"}; - setWatermarkMinActive(*backend, store->layout(), "srv", /*writer_epoch*/1, /*min_active*/10); + setWatermarkMinActive(*backend, store->layout(), "srv", /*writer_epoch*/1, /*min_active_build_sequence*/10); /// A precommit naming a build (writer_epoch 1, build_sequence 5) whose body is never written. const ManifestRef dead = ManifestRef{.writer_epoch = 1, .build_sequence = 5, .manifest_ordinal = 1}; diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 13dc97ab826c..55f7247d87b3 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -321,7 +321,7 @@ CompletedRemovingFixture seedCompletedRemoving( CasFoldSeal parent; parent.generation = 1; parent.ref_lives.emplace(fixture.life_id, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); @@ -368,7 +368,7 @@ void seedCompletedRemovingBatch( parent.generation = 1; for (const CatalogEntry & entry : entries) parent.ref_lives.emplace(entry.incarnation, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); @@ -1480,7 +1480,7 @@ TEST(CASGCFrontierGate, TheOrphanManifestSweepAndItsCursorAreInertUnderSuppressi const ManifestRef r2{.writer_epoch = 5, .build_sequence = 0xCA02, .manifest_ordinal = 1}; writeManifestRaw(*backend, layout, ns, r1, {blobEntryFor("a", DB::UInt128(0xa1))}); writeManifestRaw(*backend, layout, ns, r2, {blobEntryFor("b", DB::UInt128(0xb2))}); - setWatermarkMinActive(*backend, layout, "test", r1.writer_epoch, /*min_active*/ 0xCA03); + setWatermarkMinActive(*backend, layout, "test", r1.writer_epoch, /*min_active_build_sequence*/ 0xCA03); /// The §6 deletion premise is a second precondition on the CONTROL arm below: a manifest of an /// epoch-`E` build is deletable only once the namespace's sealed fold cursor sits in an epoch /// strictly above `E`. Sealing that cursor here is what keeps this test about the GATE — without it @@ -2814,7 +2814,7 @@ TEST(CASGCFrontierGate, UnmatchedAdoptedParentLifeDoesNotSuppressAuthoritativeDe const UInt128 unmatched_life = hexToU128("fedcba98765432100123456789abcdef"); ASSERT_FALSE(parent.ref_lives.contains(unmatched_life)); parent.ref_lives.emplace(unmatched_life, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{9, 9}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{9, 9}}}); ASSERT_EQ( backend->putOverwrite(parent_seal_key, encodeFoldSeal(parent), parent_object->token).outcome, PutOutcome::Done); @@ -3137,7 +3137,7 @@ TEST(CASGCFrontierGate, DeferredRoundDrainsCompletedRemovingBeforeReturning) CasFoldSeal parent; parent.generation = 1; parent.ref_lives.emplace(life_id, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index 19a63bc6842a..9cc892165f14 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -24,9 +24,9 @@ /// DURABLE HOLDS (spec 2026-07-27 "ref chain complete cut" §5). /// /// A namespace whose ref-log walk meets an IMPOSSIBLE shape stops there, and that stop has to survive -/// the round. Before this task the stop was a single bit — `classification == 4` — and everything that -/// explained it (what went wrong, and exactly WHERE) lived in a log line and an in-memory anomaly, both -/// gone by the next round. That is not enough for three separate reasons: +/// the round. Before this task the stop was a single bit — `classification == CoverageClass::Clamped` — +/// and everything that explained it (what went wrong, and exactly WHERE) lived in a log line and an +/// in-memory anomaly, both gone by the next round. That is not enough for three separate reasons: /// /// * the next round could not RETRY the exact position, so a hold only survived while the round's /// hint happened to keep mentioning the namespace; @@ -36,10 +36,10 @@ /// baseline that looked proven when it was not. /// /// So the hold is now DURABLE and STRICTLY GRAMMARED: `{reason, offending_position, retry_count, -/// next_retry_round}` present if and only if `classification == 4`, rejected in both directions -/// otherwise. It rides the seal across rounds — including rounds whose hint omits the namespace -/// entirely — and across REBUILD, and it clears by exactly ONE event: the fold resolving the offending -/// position and that result being adopted in `gc/state`. +/// next_retry_round}` present if and only if `classification == CoverageClass::Clamped`, rejected in +/// both directions otherwise. It rides the seal across rounds — including rounds whose hint omits the +/// namespace entirely — and across REBUILD, and it clears by exactly ONE event: the fold resolving the +/// offending position and that result being adopted in `gc/state`. /// /// The carried hold is also a WITNESS, and a better one than the listing: it is durable proof that the /// walk once reached that position, so an absent below it is a gap rather than a frontier no matter @@ -177,8 +177,8 @@ RefHold holdOf(Backend & backend, const Layout & layout, const RootNamespace & n EXPECT_TRUE(cov.has_value()) << "no coverage row for " << ns.string(); if (!cov) return RefHold{}; - EXPECT_EQ(cov->classification, 4) << "a held namespace is classification 4"; - EXPECT_TRUE(cov->hold.has_value()) << "classification 4 without a hold is the forbidden shape"; + EXPECT_EQ(cov->classification, CoverageClass::Clamped) << "a held namespace is classification clamped"; + EXPECT_TRUE(cov->hold.has_value()) << "classification clamped without a hold is the forbidden shape"; return cov->hold ? *cov->hold : RefHold{}; } @@ -206,7 +206,7 @@ CasFoldSeal maximalHoldSeal(const String & map_key) seal.generation = std::numeric_limits::max(); seal.parent_generation = std::numeric_limits::max(); RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.last_folded_ref_id = RefTxnId{std::numeric_limits::max(), std::numeric_limits::max()}; cov.hold = RefHold{.reason = HoldReason::UnconsumedSealCrossing, /// the longest reason word @@ -233,7 +233,7 @@ CasFoldSeal cleanSeal(const String & map_key) seal.generation = 3; seal.parent_generation = 2; RefCoverage cov; - cov.classification = 2; + cov.classification = CoverageClass::Folded; cov.last_folded_ref_id = RefTxnId{4, 5}; fixtureCoverage(seal, map_key) = cov; return seal; @@ -245,7 +245,7 @@ CasFoldSeal heldSeal(const String & map_key) { CasFoldSeal seal = cleanSeal(map_key); RefCoverage & cov = fixtureCoverage(seal, map_key); - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{4, 6}, .retry_count = 7, .next_retry_round = 99}; return seal; @@ -271,20 +271,6 @@ String sealTextWith(const String & prototype, const std::vector & record return text + "{\"n\":" + std::to_string(records.size()) + "}\n"; } -/// Replace the coverage row's `cls` value with `raw`, VERBATIM. The point is to write integers no -/// `RefCoverage` can hold: the field is a byte in the struct, so a wide value exists only on the wire, -/// which is exactly where a reader has to catch it. `cls` is never the last field of a `cov` record, so -/// the value always ends at a comma. -String withRawClassification(const String & encoded, std::string_view raw) -{ - const size_t at = encoded.find("\"cls\":"); - EXPECT_NE(at, String::npos); - const size_t begin = at + strlen("\"cls\":"); - const size_t end = encoded.find(',', begin); - EXPECT_NE(end, String::npos); - return encoded.substr(0, begin) + String{raw} + encoded.substr(end); -} - /// Replace the FIRST occurrence of `field` with `replacement` (both are whole `"key":value` fragments), /// so a test states the exact wire shape it is feeding the decoder. String withField(const String & encoded, const String & field, const String & replacement) @@ -303,24 +289,27 @@ std::vector> illFormedSealsTheEncoderMustRe /// The pairing, both ways round. CasFoldSeal hold_on_folded = heldSeal("ns/0"); - fixtureCoverage(hold_on_folded, "ns/0").classification = 2; - out.emplace_back("a hold on a folded (2) row claims a stop that did not happen", hold_on_folded); + fixtureCoverage(hold_on_folded, "ns/0").classification = CoverageClass::Folded; + out.emplace_back("a hold on a folded row claims a stop that did not happen", hold_on_folded); CasFoldSeal clamped_without_hold = heldSeal("ns/0"); fixtureCoverage(clamped_without_hold, "ns/0").hold.reset(); - out.emplace_back("a clamped (4) row with no hold is indistinguishable from a clean cursor once " + out.emplace_back("a clamped row with no hold is indistinguishable from a clean cursor once " "durable", clamped_without_hold); - /// The closed set. 3 is the dangerous one: it passes the sweep's `== 4` and `== 0` refusals and - /// reaches the deletion premise, which is a refusal written in terms of the set. - CasFoldSeal classification_three = cleanSeal("ns/0"); - fixtureCoverage(classification_three, "ns/0").classification = 3; - out.emplace_back("classification 3 is not one of {0,1,2,4} and passes every refusal stated in terms " - "of them", classification_three); + /// The closed set is now the enum's declared values, so only an explicit cast reaches outside it. + /// 4 is the sharpest value to plant: it was the wire value for Clamped before this task's dense + /// renumbering, and under the new table it is simply out of range. + CasFoldSeal classification_retired_wire_value = cleanSeal("ns/0"); + fixtureCoverage(classification_retired_wire_value, "ns/0").classification + = static_cast(4); + out.emplace_back("classification 4 is outside the four values the wire table declares", + classification_retired_wire_value); CasFoldSeal classification_max = cleanSeal("ns/0"); - fixtureCoverage(classification_max, "ns/0").classification = 255; - out.emplace_back("classification 255 is not one of {0,1,2,4}", classification_max); + fixtureCoverage(classification_max, "ns/0").classification = static_cast(255); + out.emplace_back("classification 255 is outside the four values the wire table declares", + classification_max); /// The self-erasing hold, and its half-zero sibling. CasFoldSeal hold_at_zero = heldSeal("ns/0"); @@ -371,7 +360,7 @@ TEST(CASGCHoldGrammarBudget, SumsSaturateInsteadOfWrapping) EXPECT_FALSE(fitsObjectCap(kMax, 2, 256 * 1024 * 1024)); } -/// ===================== THE STRICT CLASSIFICATION-4 GRAMMAR ===================== +/// ===================== THE STRICT CLAMPED-CLASSIFICATION GRAMMAR ===================== TEST(CASGCHoldGrammar, EveryHoldReasonRoundTrips) { @@ -383,7 +372,7 @@ TEST(CASGCHoldGrammar, EveryHoldReasonRoundTrips) seal.generation = 3; seal.parent_generation = 2; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.last_folded_ref_id = RefTxnId{4, 5}; cov.hold = RefHold{.reason = reason, .offending_position = RefTxnId{4, 6}, .retry_count = 7, .next_retry_round = 99}; @@ -432,23 +421,20 @@ TEST(CASGCHoldGrammar, AHoldOnAnyOtherClassificationIsRefusedByTheDecoder) /// Bytes some other producer wrote. Built by demoting a legitimate held row's classification, so the /// hold fields are exactly the ones the encoder emits. - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, .retry_count = 0, .next_retry_round = 1}; fixtureCoverage(seal, "ns/0") = cov; - String text = encodeFoldSeal(seal); - const size_t at = text.find("\"cls\":4"); - ASSERT_NE(at, String::npos); - text[at + 6] = '2'; + const String text = withField(encodeFoldSeal(seal), R"("class":"clamped")", R"("class":"folded")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(text); }); } -TEST(CASGCHoldGrammar, ClassificationFourWithoutAHoldIsRefusedByTheDecoder) +TEST(CASGCHoldGrammar, ClampedWithoutAHoldIsRefusedByTheDecoder) { CasFoldSeal seal; seal.generation = 1; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.last_folded_ref_id = RefTxnId{1, 1}; /// Every single hold field is REQUIRED: dropping any one of them is corruption, not a default. @@ -456,8 +442,8 @@ TEST(CASGCHoldGrammar, ClassificationFourWithoutAHoldIsRefusedByTheDecoder) .retry_count = 3, .next_retry_round = 4}; fixtureCoverage(seal, "ns/0") = cov; const String whole = encodeFoldSeal(seal); - for (const String & field : {String(R"("hr":"body_undecodable")"), String(R"("hpe":"1")"), - String(R"("hps":"2")"), String(R"("hrc":3)"), String(R"("hnr":"4")")}) + for (const String & field : {String(R"("hold_reason":"body_undecodable")"), String(R"("hold_epoch":"1")"), + String(R"("hold_seq":"2")"), String(R"("retries":3)"), String(R"("retry_round":"4")")}) { SCOPED_TRACE("without " + field); const size_t at = whole.find(field); @@ -473,18 +459,18 @@ TEST(CASGCHoldGrammar, DuplicateHoldKeyIsCorruptedData) CasFoldSeal seal; seal.generation = 1; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, .retry_count = 0, .next_retry_round = 5}; fixtureCoverage(seal, "ns/0") = cov; const String whole = encodeFoldSeal(seal); - const String field = R"("hr":"gap_below_witness")"; + const String field = R"("hold_reason":"gap_below_witness")"; const size_t at = whole.find(field); ASSERT_NE(at, String::npos); /// The same key twice, with a DIFFERENT value: last-wins would silently rewrite the reason. String doubled = whole; - doubled.insert(at, R"("hr":"witness_disappeared",)"); + doubled.insert(at, R"("hold_reason":"witness_disappeared",)"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(doubled); }); } @@ -493,7 +479,7 @@ TEST(CASGCHoldGrammar, UnknownHoldReasonWordIsCorruptedData) CasFoldSeal seal; seal.generation = 1; RefCoverage cov; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, .retry_count = 0, .next_retry_round = 5}; fixtureCoverage(seal, "ns/0") = cov; @@ -510,48 +496,46 @@ TEST(CASGCHoldGrammar, UnknownHoldReasonWordIsCorruptedData) /// The three shapes below are one finding, and it is about what a fold seal is FOR. The hold is the only /// durable record that a namespace stopped and where; everything downstream reads the seal and nothing /// re-derives the stop. So a seal that decodes into "no hold here" is not a lossy read, it is a licence -/// to delete: the sweep's §6 refusals are stated as `classification == 4` / `== 0` / `hold.has_value()`, -/// and a row that slips past all three reaches an irreversible delete of a manifest the fold never -/// accounted for. Each shape gets past a DIFFERENT one of the decoder's checks, which is why they are -/// pinned separately rather than as one "malformed seal" case. - -/// (1) The classification the reader never sees. `cls` is narrowed to a byte, so an integer on the wire -/// is truncated first and validated (if at all) afterwards: 258 becomes 2, "everything through the -/// cursor was folded". The value has to be judged WIDE, before the narrowing, or the wire can buy -/// coverage that no fold ever performed. +/// to delete: the sweep's §6 refusals are stated as `classification == Clamped` / `== Absent` / +/// `hold.has_value()`, and a row that slips past all three reaches an irreversible delete of a manifest +/// the fold never accounted for. Each shape gets past a DIFFERENT one of the decoder's checks, which is +/// why they are pinned separately rather than as one "malformed seal" case. + +/// (1) The classification is a WORD, closed the same way `hold_reason` already is: +/// `coverageClassFromWord` refuses anything outside the four named values as `CORRUPTED_DATA` before a +/// `CoverageClass` is ever constructed, so there is no wide-integer narrowing attack left to catch here — +/// the wire carries no integer at all. TEST(CASGCHoldGrammar, AClassificationOutsideTheGrammarIsCorruptedData) { const String clean = encodeFoldSeal(cleanSeal("ns/0")); - ASSERT_EQ(fixtureCoverage(decodeFoldSeal(clean), "ns/0").classification, 2) + ASSERT_EQ(fixtureCoverage(decodeFoldSeal(clean), "ns/0").classification, CoverageClass::Folded) << "the unmodified row is the one every case below deviates from"; - /// In-range bytes that are simply not classifications. 3 is the one the sweep's refusals miss. - for (const std::string_view raw : {"3", "5", "6", "255"}) + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - SCOPED_TRACE(String{"cls="} + String{raw}); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withRawClassification(clean, raw)); }); - } + decodeFoldSeal(withField(clean, R"("class":"folded")", R"("class":"foldedx")")); + }); - /// Wide integers whose LOW BYTE lands inside the grammar: 258 -> 2 (fully folded), 256 -> 0 - /// (absent), 260 -> 4 (clamped). Each would decode as a row the fold never wrote. - for (const std::string_view raw : {"256", "258", "260", "18446744073709551615"}) + /// The bare number `4` is the classification's pre-cut wire representation — the old byte-valued + /// form. A retired spelling is legal here because this is a marked negative fixture proving the + /// decoder still refuses it now that `class` takes a word; the byte-delta pins hold the other such + /// fixtures, and both kinds are exempt from the vocabulary sweeps for the same reason. + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - SCOPED_TRACE(String{"cls="} + String{raw}); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withRawClassification(clean, raw)); }); - } + decodeFoldSeal(withField(clean, R"("class":"folded")", R"("class":4)")); + }); } -/// And the field itself is required: an absent `cls` reads as 0, which is not "nothing was said about -/// this namespace" but the positive claim "no round folded it". +/// And the field itself is required: an absent `class` reads as `absent`, which is not "nothing was +/// said about this namespace" but the positive claim "no round folded it". TEST(CASGCHoldGrammar, ACoverageRowWithoutAClassificationIsCorruptedData) { const String clean = encodeFoldSeal(cleanSeal("ns/0")); - const size_t at = clean.find("\"cls\":2,"); + const String field = R"("class":"folded",)"; + const size_t at = clean.find(field); ASSERT_NE(at, String::npos); String without = clean; - without.erase(at, strlen("\"cls\":2,")); + without.erase(at, field.size()); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeFoldSeal(without); }); } @@ -567,13 +551,13 @@ TEST(CASGCHoldGrammar, AHoldWhoseOffendingPositionHasAZeroComponentIsCorruptedDa expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - decodeFoldSeal(withField(withField(held, R"("hpe":"4")", R"("hpe":"0")"), - R"("hps":"6")", R"("hps":"0")")); + decodeFoldSeal(withField(withField(held, R"("hold_epoch":"4")", R"("hold_epoch":"0")"), + R"("hold_seq":"6")", R"("hold_seq":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(held, R"("hpe":"4")", R"("hpe":"0")")); }); + [&] { decodeFoldSeal(withField(held, R"("hold_epoch":"4")", R"("hold_epoch":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(held, R"("hps":"6")", R"("hps":"0")")); }); + [&] { decodeFoldSeal(withField(held, R"("hold_seq":"6")", R"("hold_seq":"0")")); }); } /// (3) The duplicate row. Two `cov` records for the same (namespace, shard) — held first, clean second — @@ -608,7 +592,7 @@ TEST(CASGCHoldGrammar, ASecondCoverageRowForTheSameKeyIsCorruptedData) EXPECT_EQ(seal.ref_lives.size(), 1u); } -/// The same one-record-per-key rule applies to `cnd`: a repeated row rewrites a shard's condemned +/// The same one-record-per-key rule applies to `condemned`: a repeated row rewrites a shard's condemned /// totals, which graduation paces on. TEST(CASGCHoldGrammar, ASecondCondemnedSummaryRecordIsCorruptedData) { @@ -617,7 +601,7 @@ TEST(CASGCHoldGrammar, ASecondCondemnedSummaryRecordIsCorruptedData) .oldest_nonpending_condemn_round = 3}; const String encoded = encodeFoldSeal(seal); - /// Lines 3..4 are `rfl`, `cnd` in the encoder's fixed order. + /// Lines 3..4 are `ref_life`, `condemned` in the encoder's fixed order. std::vector lines; for (size_t begin = headerAndMetaOf(encoded).size(); begin < encoded.size();) { @@ -626,14 +610,14 @@ TEST(CASGCHoldGrammar, ASecondCondemnedSummaryRecordIsCorruptedData) lines.push_back(encoded.substr(begin, end - begin)); begin = end + 1; } - ASSERT_EQ(lines.size(), 3u) << "rfl, cnd and the trailer"; + ASSERT_EQ(lines.size(), 3u) << "ref_life, condemned and the trailer"; const String ref_life_line = lines[0]; - const String cnd_line = lines[1]; + const String condemned_line = lines[1]; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(sealTextWith(encoded, {ref_life_line, cnd_line, cnd_line})); }); + [&] { decodeFoldSeal(sealTextWith(encoded, {ref_life_line, condemned_line, condemned_line})); }); /// The unduplicated assembly is the control. - const std::vector one_of_each{ref_life_line, cnd_line}; + const std::vector one_of_each{ref_life_line, condemned_line}; EXPECT_NO_THROW(decodeFoldSeal(sealTextWith(encoded, one_of_each))); } @@ -647,12 +631,12 @@ TEST(CASGCHoldGrammar, CleanupEvidenceWithAZeroRemovalIdIsCorruptedData) const String encoded = encodeFoldSeal(seal); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rte":"2")", R"("rte":"0")")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_epoch":"2")", R"("remove_epoch":"0")")); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rts":"3")", R"("rts":"0")")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_seq":"3")", R"("remove_seq":"0")")); }); /// Omitted entirely is the same thing: the fields default to zero. expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeFoldSeal(withField(encoded, R"("rte":"2",)", "")); }); + [&] { decodeFoldSeal(withField(encoded, R"("remove_epoch":"2",)", "")); }); } /// The OBJECT cap bounds the whole seal. Nothing on the fold-seal READ path enforces it (the seal @@ -1018,7 +1002,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointHoldsOnlyItsOwnNamespace) ASSERT_TRUE(good_cov.has_value()); EXPECT_FALSE(good_cov->hold.has_value()) << "the corrupt object belongs to the OTHER namespace"; EXPECT_EQ(good_cov->last_folded_ref_id, (RefTxnId{1, 2})); - EXPECT_EQ(good_cov->classification, 2); + EXPECT_EQ(good_cov->classification, CoverageClass::Folded); /// And nothing was destroyed for the held namespace: a hold shuts the round's destructive gate, so /// its ref objects — including the ones a cleanup range computed WITHOUT the unreadable checkpoint @@ -1091,7 +1075,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointWithNoWalkPositionRecordsAnAnomaly const auto cov = coverageOf(*backend, layout, phantom); ASSERT_TRUE(cov.has_value()); EXPECT_FALSE(cov->hold.has_value()) << "a hold here could only name a position no round ever read"; - EXPECT_EQ(cov->classification, 1) << "nothing was folded, so the row is `unchanged`"; + EXPECT_EQ(cov->classification, CoverageClass::Unchanged) << "nothing was folded, so the row is `unchanged`"; EXPECT_EQ(cov->last_folded_ref_id, (RefTxnId{})); /// Same isolation as the held arm: the pool keeps working. @@ -1199,7 +1183,7 @@ TEST(CASGCHoldGrammar, HoldClearsOnlyByFoldingThroughTheOffendingPosition) const auto cov = coverageOf(*backend, layout, ns); ASSERT_TRUE(cov.has_value()); EXPECT_FALSE(cov->hold.has_value()) << "folding through the offending position is what clears a hold"; - EXPECT_EQ(cov->classification, 2); + EXPECT_EQ(cov->classification, CoverageClass::Folded); EXPECT_EQ(cov->last_folded_ref_id, (RefTxnId{1, 4})) << "the walk resumed past the resolved gap"; EXPECT_EQ(inDegreeOf(*backend, layout, DB::UInt128(4)), 1) << "the record above the gap finally contributed its owner edge"; @@ -1261,10 +1245,10 @@ TEST(CASGCHoldGrammar, RebuildCarriesMatchingHoldAndDropsAbsentLife) mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); RefCoverage gone; - gone.classification = 4; + gone.classification = CoverageClass::Clamped; gone.last_folded_ref_id = RefTxnId{2, 2}; gone.hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{2, 3}, .retry_count = 1, .next_retry_round = 2}; @@ -1278,7 +1262,7 @@ TEST(CASGCHoldGrammar, RebuildCarriesMatchingHoldAndDropsAbsentLife) ASSERT_TRUE(rebuilt.has_value()); const auto rediscovered = rebuilt->ref_lives.find(life_id); ASSERT_NE(rediscovered, rebuilt->ref_lives.end()); - EXPECT_EQ(rediscovered->second.coverage.classification, 4); + EXPECT_EQ(rediscovered->second.coverage.classification, CoverageClass::Clamped); ASSERT_TRUE(rediscovered->second.coverage.hold.has_value()); EXPECT_EQ(*rediscovered->second.coverage.hold, plantedHold()); EXPECT_FALSE(rebuilt->ref_lives.contains(absent_life_id)); @@ -1320,7 +1304,7 @@ TEST(CASGCHoldGrammar, RebuildStepsDownPastACrashedNewestGenerationToTheSealBelo mutateSealAt(*backend, layout, older_generation, older_attempt, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); }); @@ -1398,7 +1382,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWithAnUndecodablePriorSeal) const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":4}\nthis is not a seal body\n", + backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n", backend->head(seal_key).token); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc.rebuildBaseline(/*force=*/true); }); @@ -1433,7 +1417,7 @@ TEST(CASGCHoldGrammar, RebuildWithLostStateStillCarriesHoldsFromTheNewestSeal) mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); }); @@ -1470,7 +1454,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenTheNewestSealIsUnreadableAndTheStateIsL const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":4}\nthis is not a seal body\n", + backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n", backend->head(seal_key).token); const HeadResult sh = backend->head(layout.gcStateKey()); ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); @@ -1537,7 +1521,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) { RefCoverage & cov = seal.ref_lives.at(life_id).coverage; - cov.classification = 4; + cov.classification = CoverageClass::Clamped; cov.hold = plantedHold(); }); diff --git a/src/Disks/tests/gtest_cas_gc_leak.cpp b/src/Disks/tests/gtest_cas_gc_leak.cpp index 36a77237e9dc..06983bc52aab 100644 --- a/src/Disks/tests/gtest_cas_gc_leak.cpp +++ b/src/Disks/tests/gtest_cas_gc_leak.cpp @@ -47,7 +47,7 @@ PoolPtr openTestPool(std::shared_ptr & out_backend) /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 81a1dd46c2d3..47715169db12 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -33,7 +33,7 @@ TEST(CASFormatBattery, GcMaintenanceState) runFormatBattery({FormatId::GcMaintenanceState, [&] { return sealObject(FormatId::GcMaintenanceState, encodeGcMaintenanceState(state)); }, [](std::string_view s) { decodeGcMaintenanceState(std::string(openObject(FormatId::GcMaintenanceState, s))); }, - currentFormatHeader("cas_gc_maintenance_state") + "{\"cur\":\"cas/ns/a\"}\n"}); + currentFormatHeader("cas_gc_maintenance_state") + "{\"janitor_cursor\":\"cas/ns/a\"}\n"}); } TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) @@ -41,8 +41,8 @@ TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) EXPECT_EQ(static_cast(FormatId::GcMaintenanceState), 25); const auto points = changePoints(FormatId::GcMaintenanceState); ASSERT_EQ(points.size(), 1u); - EXPECT_EQ(points[0].generation, 7); - EXPECT_EQ(points[0].min_reader, 7); + EXPECT_EQ(points[0].generation, 1); + EXPECT_EQ(points[0].min_reader, 1); const FormatTraits & traits = traitsFor(FormatId::GcMaintenanceState); EXPECT_EQ(traits.type, "cas_gc_maintenance_state"); EXPECT_EQ(traits.family, TextFamily::Control); @@ -60,7 +60,7 @@ TEST(CASGCMaintenanceStateFormat, RegistryLayoutAndCanonicalCodec) const GcMaintenanceState empty; EXPECT_EQ(encodeGcMaintenanceState(empty), fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"cur\":\"\"}}\n", currentCompatibilityVersion())); + "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion())); const GcMaintenanceState state{.janitor_cursor = R"(cas/ns/a/"quoted"\\next)"}; EXPECT_EQ(decodeGcMaintenanceState(encodeGcMaintenanceState(state)), state); } @@ -69,28 +69,28 @@ TEST(CASGCMaintenanceStateFormat, RejectsMalformedAndBoundsCursor) { const auto bad = [](std::string_view body) { - return "{\"type\":\"cas_gc_maintenance_state\",\"v\":7}\n" + String(body); + return "{\"type\":\"cas_gc_maintenance_state\",\"v\":1}\n" + String(body); }; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeGcMaintenanceState(bad("{}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\",\"cur\":\"b\"}\n")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\",\"janitor_cursor\":\"b\"}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\",\"extra\":1}\n")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\",\"extra\":1}\n")); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeGcMaintenanceState(bad("{\"cur\":\"a\"}\nx")); }); + [&] { (void)decodeGcMaintenanceState(bad("{\"janitor_cursor\":\"a\"}\nx")); }); const GcMaintenanceState at_limit{.janitor_cursor = String(kMaxGcMaintenanceCursorBytes, 'x')}; EXPECT_EQ(decodeGcMaintenanceState(encodeGcMaintenanceState(at_limit)), at_limit); const GcMaintenanceState over_limit{.janitor_cursor = String(kMaxGcMaintenanceCursorBytes + 1, 'x')}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LIMIT_EXCEEDED, [&] { (void)encodeGcMaintenanceState(over_limit); }); - const String raw = "{\"type\":\"cas_gc_maintenance_state\",\"v\":7}\n{\"cur\":\"" + over_limit.janitor_cursor + "\"}\n"; + const String raw = "{\"type\":\"cas_gc_maintenance_state\",\"v\":1}\n{\"janitor_cursor\":\"" + over_limit.janitor_cursor + "\"}\n"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeGcMaintenanceState(raw); }); - String oversized = R"({"type":"cas_gc_maintenance_state","v":7,"pad":")"; + String oversized = R"({"type":"cas_gc_maintenance_state","v":1,"pad":")"; oversized.append(448 * 1024, 'x'); - oversized += "\"}\n{\"cur\":\""; + oversized += "\"}\n{\"janitor_cursor\":\""; oversized.append(kMaxGcMaintenanceCursorBytes, 'y'); oversized += "\"}\n"; ASSERT_GT(oversized.size(), traitsFor(FormatId::GcMaintenanceState).object_cap); @@ -184,7 +184,7 @@ TEST(CASGCMaintenanceState, FutureVersionPropagatesInsteadOfResetting) const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); ASSERT_EQ(backend.putIfAbsent(key, fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"cur\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, + "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, PutOutcome::Done); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { (void)readGcMaintenanceState(backend, layout); }); diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index e03aa26dbaad..c9c20a7454cd 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -3,6 +3,8 @@ #include #include +#include + using namespace DB::Cas; namespace @@ -42,8 +44,8 @@ TEST(CASFormatBattery, GcOutcomes) [&] { return sealObject(FormatId::GcOutcomes, encodeOutcomeLog(log)); }, [](std::string_view d) { decodeOutcomeLog(std::string(openObject(FormatId::GcOutcomes, d))); }, currentFormatHeader("cas_gc_outcomes") + - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\"," - "\"tt\":\"etag\",\"tv\":\"e-1\",\"oc\":\"deleted\"}\n{\"n\":1}\n"}); + "{\"kind\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\"," + "\"token_type\":\"etag\",\"token\":\"e-1\",\"outcome\":\"deleted\"}\n{\"n\":1}\n"}); } TEST(CASGCOutcomesFormat, EmptyRoundTrips) @@ -77,7 +79,20 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) EXPECT_EQ(encodeOutcomeLog(d), text); } -TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) +/// Closed-set pin: the four `OutcomeKind` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASGCOutcomesFormat, ClosedSetPinsOutcomeKindWords) +{ + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Deleted), "deleted"); + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Absent), "absent"); + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Replaced), "replaced"); + EXPECT_EQ(outcomeKindToWireWord(OutcomeKind::Spared), "spared"); + for (const auto o : magic_enum::enum_values()) + EXPECT_EQ(outcomeKindFromWireWord(outcomeKindToWireWord(o)), o); +} + +TEST(CASGCOutcomesFormat, RecordRequiresCompleteBlobRefAndTokenGroups) { OutcomeLog log; log.entries.push_back({ObjectKind::Blob, @@ -85,16 +100,12 @@ TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) Token{"e-1", TokenType::ETag}, OutcomeKind::Deleted}); const String bytes = encodeOutcomeLog(log); - const String token_value = R"(,"tv":"e-1")"; - const auto token_value_pos = bytes.find(token_value); - ASSERT_NE(token_value_pos, String::npos); - String missing_token_value = bytes; - missing_token_value.erase(token_value_pos, token_value.size()); - const OutcomeLog decoded = decodeOutcomeLog(missing_token_value); - ASSERT_EQ(decoded.entries.size(), 1u); - EXPECT_EQ(decoded.entries[0].token.value, ""); - - for (const String & field : {String(R"(,"ha":"ch128")"), String(R"(,"h":"00112233445566778899aabbccddeeff")"), String(R"(,"tt":"etag")")}) + for (const auto & [field, expected_message] : { + std::pair{String(R"(,"algo":"ch128")"), "CAS outcome log: blob ref missing algo/digest"}, + std::pair{String(R"(,"digest":"00112233445566778899aabbccddeeff")"), "CAS outcome log: blob ref missing algo/digest"}, + std::pair{String(R"(,"token_type":"etag")"), "CAS outcome log: token missing token_type/token"}, + std::pair{String(R"(,"token":"e-1")"), "CAS outcome log: token missing token_type/token"}, + }) { const auto pos = bytes.find(field); ASSERT_NE(pos, String::npos); @@ -108,32 +119,32 @@ TEST(CASGCOutcomesFormat, RecordTokenValueIsOptionalButTokenIdentityIsRequired) catch (const DB::Exception & e) { EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); - EXPECT_EQ(e.message(), "CAS outcome log: record missing ha/h/tt"); + EXPECT_EQ(e.message(), expected_message); } } } TEST(CASGCOutcomesFormat, GarbageAndUnknownWordsFailClosed) { - EXPECT_THROW(decodeOutcomeLog(String("")), DB::Exception); - EXPECT_THROW(decodeOutcomeLog(String("not a cas object\n")), DB::Exception); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeOutcomeLog(String("")); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { decodeOutcomeLog(String("not a cas object\n")); }); /// A record with an unknown outcome word fails closed. - const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n" - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\"," - "\"tt\":\"etag\",\"tv\":\"x\",\"oc\":\"bogus\"}\n{\"n\":1}\n"; - EXPECT_THROW(decodeOutcomeLog(bad), DB::Exception); + const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":1}\n" + "{\"kind\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\"," + "\"token_type\":\"etag\",\"token\":\"x\",\"outcome\":\"bogus\"}\n{\"n\":1}\n"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(bad); }); /// A trailer count mismatch fails closed. - const String miscount = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n{\"n\":5}\n"; - EXPECT_THROW(decodeOutcomeLog(miscount), DB::Exception); + const String miscount = "{\"type\":\"cas_gc_outcomes\",\"v\":1}\n{\"n\":5}\n"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(miscount); }); } TEST(CASGCOutcomesFormat, DigestWidthMismatchFailsClosedWithCorruptedData) { - /// `ch128` (CityHash128) digests are 16 bytes = 32 hex chars; here the "h" field is truncated + /// `ch128` (CityHash128) digests are 16 bytes = 32 hex chars; here the `digest` field is truncated /// to 30 hex chars. Must surface as CORRUPTED_DATA (malformed serialized input), not /// `fromHex`'s BAD_ARGUMENTS. - const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":3}\n" - "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddee\"," - "\"tt\":\"etag\",\"tv\":\"x\",\"oc\":\"deleted\"}\n{\"n\":1}\n"; + const String bad = "{\"type\":\"cas_gc_outcomes\",\"v\":1}\n" + "{\"kind\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddee\"," + "\"token_type\":\"etag\",\"token\":\"x\",\"outcome\":\"deleted\"}\n{\"n\":1}\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeOutcomeLog(bad); }); } diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index 1ad8f576809d..1b6344af63fb 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -500,7 +500,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) EXPECT_EQ(rep.committed_refs, blobs.size()); /// Multiple rebuild flushes still converge to one authoritative row domain: no more than one - /// canonical seq-0 `btr` per shard and exactly one `cnd` per shard. These are the cardinalities the + /// canonical seq-0 `blob_run` per shard and exactly one `condemned` per shard. These are the cardinalities the /// catalog admission reservation over-covers independently of catalog-entry count. const GcState rebuilt_state = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); const CasFoldSeal rebuilt_seal = decodeFoldSeal( @@ -535,7 +535,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) } /// Trimmed-but-live (design delta 2): the precommit's journal evidence is gone (trim), the build -/// is NOT provably dead (a live build holds min_active down) — the unowned-alive sweep must +/// is NOT provably dead (a live build holds min_active_build_sequence down) — the unowned-alive sweep must /// over-protect the manifest's edges. TEST(CASGCRebuild, UnownedAliveManifestOverProtected) { @@ -543,7 +543,7 @@ TEST(CASGCRebuild, UnownedAliveManifestOverProtected) auto store = openPoolForTest(backend); const RootNamespace ns{"00/aa@cas@"}; - /// A LIVE build pins min_active at its build_seq, so higher build sequences are not provably dead. + /// A LIVE build pins min_active_build_sequence at its build_seq, so higher build sequences are not provably dead. auto live_build = store->beginPartWrite({}); store->renewWatermarkOnce(); diff --git a/src/Disks/tests/gtest_cas_gc_resume.cpp b/src/Disks/tests/gtest_cas_gc_resume.cpp index 55ff116b7109..0a8384e644e0 100644 --- a/src/Disks/tests/gtest_cas_gc_resume.cpp +++ b/src/Disks/tests/gtest_cas_gc_resume.cpp @@ -28,7 +28,7 @@ bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash /// Whether the CURRENT retired list (any gc-shard) still holds an entry. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index 8ea415a9f024..b54b5e71d648 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -631,7 +631,7 @@ TEST(CASGCRound, PreviewReportsCondemnedRowsAndIsWriteFree) /// A fully idle fold pure-carries every shard's authoritative rows verbatim. The parent is first made /// non-vacuous with one live blob in each of two shards; the forced no-delta successor must preserve -/// both `btr` rows and the total `cnd` domain byte-for-byte. +/// both `blob_run` rows and the total `condemned` domain byte-for-byte. TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) { auto backend = std::make_shared(); @@ -675,9 +675,9 @@ TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) EXPECT_TRUE(seal1.condemned_summary.contains(0) && seal1.condemned_summary.contains(1)); EXPECT_TRUE(seal2.condemned_summary.contains(0) && seal2.condemned_summary.contains(1)); - /// Capacity reserves one widest `btr` row per shard. Pin the production pure-carry seal to the + /// Capacity reserves one widest `blob_run` row per shard. Pin the production pure-carry seal to the /// authoritative grammar that makes that bound sufficient: at most one in-range canonical seq-0 - /// run per shard, beside exactly one `cnd` row for every shard. + /// run per shard, beside exactly one `condemned` row for every shard. bool run_seen[2] = {false, false}; ASSERT_EQ(seal1.blob_target_runs.size(), 2u); ASSERT_EQ(seal2.blob_target_runs.size(), 2u); @@ -1808,7 +1808,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) const ManifestRef r2 = ref(5, 0xCA02); writeManifestRaw(*backend, store->layout(), ns, r1, {blobEntryFor("a", DB::UInt128(1))}); writeManifestRaw(*backend, store->layout(), ns, r2, {blobEntryFor("b", DB::UInt128(2))}); - setWatermarkMinActive(*backend, store->layout(), "test", r1.writer_epoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), "test", r1.writer_epoch, /*min_active_build_sequence*/6); /// The §6 deletion premise is a second precondition on every sweep deletion: a manifest of an /// epoch-`E` build is deletable only once the namespace's sealed fold cursor sits in an epoch @@ -1823,7 +1823,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) /// injected cursor would prove only that the premise reads a number, not that the number can be /// produced. /// - /// The live publications use build sequences ABOVE the watermark's `min_active`, so the only + /// The live publications use build sequences ABOVE the watermark's `min_active_build_sequence`, so the only /// sweep-ELIGIBLE manifests in the namespace remain the two debris bodies -- the premise, not the /// watermark, is what this test varies. publishAt(*backend, store->layout(), ns, RefTxnId{1, 1}, "tbl", /*build_sequence=*/7, @@ -1900,7 +1900,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) foreign_config.server_root_id = "test"; auto invalid_store = openTestPoolWithConfig(foreign_backend, std::move(foreign_config)); const String foreign_mount_key = invalid_store->layout().mountKey("test"); - setWatermarkMinActive(*foreign_backend, invalid_store->layout(), "test", r1.writer_epoch, /*min_active*/6); + setWatermarkMinActive(*foreign_backend, invalid_store->layout(), "test", r1.writer_epoch, /*min_active_build_sequence*/6); const auto occupant_before = foreign_backend->get(foreign_mount_key); ASSERT_TRUE(occupant_before.has_value()); const uint64_t violations_before diff --git a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp index c8bc92efffdf..96f486781f5e 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp @@ -563,7 +563,7 @@ TEST(CASGCShardRetireDrain, ReclaimsDroppableBlobOwnedByNonZeroShard) return backend->head(layout.manifestKey(id)).exists; }; /// Whether ANY gc-shard still holds an in-flight condemned entry (the ack-floor deletion pipeline is - /// in flight while this is true). Retired-in-snapshot (T4): reconstructed from the adopted fold seal's + /// in flight while this is true). Condemned state is reconstructed from the adopted fold seal's /// RunMarker::Condemned rows across all shards, not a separate retired list. auto anyRetiredPending = [&] { diff --git a/src/Disks/tests/gtest_cas_gc_state_format.cpp b/src/Disks/tests/gtest_cas_gc_state_format.cpp index 7e9452fdf8a7..360063e76bdd 100644 --- a/src/Disks/tests/gtest_cas_gc_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_state_format.cpp @@ -26,8 +26,8 @@ TEST(CASFormatBattery, GcState) [&] { return sealObject(FormatId::GcState, encodeGcState(s)); }, [](std::string_view d) { decodeGcState(std::string(openObject(FormatId::GcState, d))); }, currentFormatHeader("cas_gc_state") + - "{\"rnd\":\"4\",\"gcs\":1,\"sg\":\"9\",\"spt\":\"7\",\"sa\":\"3\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000001\",\"ls\":\"12\"}\n"}); + "{\"round\":\"4\",\"gc_shards\":1,\"snap_generation\":\"9\",\"snap_pruned_through\":\"7\",\"snap_attempt\":\"3\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000001\",\"lease_seq\":\"12\"}\n"}); } CAS_BATTERY_COVERS(GcHeartbeat); @@ -39,7 +39,7 @@ TEST(CASFormatBattery, GcHeartbeat) [&] { return sealObject(FormatId::GcHeartbeat, encodeGcHeartbeat(hb)); }, [](std::string_view d) { decodeGcHeartbeat(std::string(openObject(FormatId::GcHeartbeat, d))); }, currentFormatHeader("cas_gc_hb") + - "{\"by\":\"00000000000000000000000000000001\",\"seq\":\"1741\"}\n"}); + "{\"owner\":\"00000000000000000000000000000001\",\"hb_seq\":\"1741\"}\n"}); } /// ---------- field round-trips (migrated from gtest_cas_gc_formats.cpp, re-pointed at the text codec) ---------- @@ -87,11 +87,11 @@ TEST(CASGCStateFormat, DefaultsRoundTrip) TEST(CASGCStateFormat, RejectsZeroGcShards) { - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String bad = "{\"type\":\"cas_gc_state\",\"v\":3}\n" - "{\"rnd\":\"0\",\"gcs\":0,\"sg\":\"0\",\"spt\":\"0\",\"sa\":\"0\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000000\",\"ls\":\"0\"}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String bad = "{\"type\":\"cas_gc_state\",\"v\":1}\n" + "{\"round\":\"0\",\"gc_shards\":0,\"snap_generation\":\"0\",\"snap_pruned_through\":\"0\",\"snap_attempt\":\"0\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000000\",\"lease_seq\":\"0\"}\n"; EXPECT_THROW(decodeGcState(bad), DB::Exception); } @@ -127,13 +127,13 @@ TEST(CASGCStateFormatDeathTest, RejectsZeroGcShardsOnEncodeAborts) TEST(CASGCStateFormat, RejectsAbsentGcShards) { - /// An absent gcs key must fail closed (the writer always emits it) rather than silently defaulting + /// An absent gc_shards key must fail closed (the writer always emits it) rather than silently defaulting /// to the struct's gc_shards = 1 — a missing shard count means a corrupt object, not "use the floor". - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String bad = "{\"type\":\"cas_gc_state\",\"v\":3}\n" - "{\"rnd\":\"0\",\"sg\":\"0\",\"spt\":\"0\",\"sa\":\"0\",\"msc\":\"\"," - "\"lo\":\"00000000000000000000000000000000\",\"ls\":\"0\"}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String bad = "{\"type\":\"cas_gc_state\",\"v\":1}\n" + "{\"round\":\"0\",\"snap_generation\":\"0\",\"snap_pruned_through\":\"0\",\"snap_attempt\":\"0\",\"manifest_sweep_cursor\":\"\"," + "\"lease_owner\":\"00000000000000000000000000000000\",\"lease_seq\":\"0\"}\n"; EXPECT_THROW(decodeGcState(bad), DB::Exception); } @@ -161,9 +161,9 @@ TEST(CASGCHeartbeatFormat, RoundTripAndBoundaries) TEST(CASGCHeartbeatFormat, RejectsMissingIdentityFields) { - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String header = "{\"type\":\"cas_gc_hb\",\"v\":3}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the BODY is what has + /// to fail here. + const String header = "{\"type\":\"cas_gc_hb\",\"v\":1}\n"; const auto expectCorrupted = [](const String & data) { @@ -178,6 +178,6 @@ TEST(CASGCHeartbeatFormat, RejectsMissingIdentityFields) } }; - expectCorrupted(header + "{\"seq\":\"1741\"}\n"); - expectCorrupted(header + "{\"by\":\"00000000000000000000000000000001\"}\n"); + expectCorrupted(header + "{\"hb_seq\":\"1741\"}\n"); + expectCorrupted(header + "{\"owner\":\"00000000000000000000000000000001\"}\n"); } diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index c32b03e23074..84d457b08158 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -26,7 +26,7 @@ using namespace DB::Cas; /// MountLeaseKeeper behavior: the per-server mount lease and the merged build-watermark floor ride the /// SAME slot, renewed by one beat. The keeper anchors durably before return, adopts a slot already /// written by `claimMount` (same uuid+epoch), re-reads the callback on each renew and bumps `seq`, -/// stamps the farewell sentinel (`min_active = UINT64_MAX`, `expires_at_ms <= now`) on `release`, and +/// stamps the farewell sentinel (`min_active_build_sequence = UINT64_MAX`, `expires_at_ms <= now`) on `release`, and /// returns typed terminal results on any foreign touch. namespace @@ -171,18 +171,18 @@ TEST(CASHeartbeat, AnchorCarriesFloor) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - uint64_t min_active_now = 5; + uint64_t min_active_build_sequence_now = 5; seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [&] { return min_active_now; }, {}, std::chrono::milliseconds(0)); + [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0)); keeper.start(); auto hr = backend->head(layout.mountKey(srid)); ASSERT_TRUE(hr.exists); auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); EXPECT_EQ(m.writer_epoch, 9u); - EXPECT_EQ(m.min_active, 5u); + EXPECT_EQ(m.min_active_build_sequence, 5u); EXPECT_EQ(m.seq, 1u); EXPECT_FALSE(m.gc_fenced); } @@ -194,20 +194,20 @@ TEST(CASHeartbeat, RenewRereadsCallbackAndBumpsSeq) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - uint64_t min_active_now = 5; + uint64_t min_active_build_sequence_now = 5; seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [&] { return min_active_now; }, {}, std::chrono::milliseconds(0)); + [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0)); keeper.start(); /// The dynamic field moves; the renewal re-reads it off the callback and bumps seq. now_ms = 1500; - min_active_now = 8; + min_active_build_sequence_now = 8; renewKeeperOrThrow(keeper); auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); - EXPECT_EQ(m.min_active, 8u); + EXPECT_EQ(m.min_active_build_sequence, 8u); EXPECT_EQ(m.seq, 2u); EXPECT_EQ(m.expires_at_ms, 1500u + 100u); } @@ -230,9 +230,9 @@ TEST(CASHeartbeat, StopStampsExpiredAndFarewellSentinel) auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); /// Terminal body stamps the lease already-expired (so a same-server reopen reclaims immediately) - /// AND folds the watermark farewell into it (min_active = UINT64_MAX). + /// AND folds the watermark farewell into it (min_active_build_sequence = UINT64_MAX). EXPECT_LE(m.expires_at_ms, now_ms); - EXPECT_EQ(m.min_active, std::numeric_limits::max()); + EXPECT_EQ(m.min_active_build_sequence, std::numeric_limits::max()); } /// Phase A (spec rev.4 2026-07-24): a confirmed renewal mismatch whose re-read shows OUR OWN diff --git a/src/Disks/tests/gtest_cas_inspect.cpp b/src/Disks/tests/gtest_cas_inspect.cpp index 2ed70c293259..c20466591ac4 100644 --- a/src/Disks/tests/gtest_cas_inspect.cpp +++ b/src/Disks/tests/gtest_cas_inspect.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include #include @@ -288,6 +289,32 @@ TEST(CASInspect, RendersRefCkptAbsencesAsExplicitNulls) EXPECT_NE(json.find(R"("last_epoch_seal":null)"), String::npos) << json; } +/// `CoverageClass` renders as its full wire word, not the enumerator's numeric value: `cas-inspect` is +/// exactly the tool an operator reaches for to read a fold seal directly, so a coverage row that still +/// printed a bare integer would send them back to this file's comment to decode it. +TEST(CASInspect, RendersCoverageClassificationWireWords) +{ + const Layout layout("p"); + CasFoldSeal seal; + seal.generation = 3; + seal.parent_generation = 2; + seal.ref_lives[UInt128{1}].coverage = RefCoverage{.classification = CoverageClass::Absent}; + seal.ref_lives[UInt128{2}].coverage = RefCoverage{.classification = CoverageClass::Unchanged}; + seal.ref_lives[UInt128{3}].coverage + = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}; + seal.ref_lives[UInt128{4}].coverage = RefCoverage{ + .classification = CoverageClass::Clamped, + .hold = RefHold{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{1, 2}, + .retry_count = 0, .next_retry_round = 1}}; + + const String key = layout.foldSealKey(/*generation*/3, /*attempt*/0); + const String json = caInspectToJson(layout, key, encodeFoldSeal(seal)); + EXPECT_NE(json.find(R"("classification":"absent")"), String::npos) << json; + EXPECT_NE(json.find(R"("classification":"unchanged")"), String::npos) << json; + EXPECT_NE(json.find(R"("classification":"folded")"), String::npos) << json; + EXPECT_NE(json.find(R"("classification":"clamped")"), String::npos) << json; +} + /// A listed physical id cannot supply a namespace. Inspect must receive the unique catalog join, and /// a different logical spelling at the same id is rejected by the decoded object's own namespace. TEST(CASInspect, RefObjectRequiresTheExactCatalogResolution) diff --git a/src/Disks/tests/gtest_cas_json_writer.cpp b/src/Disks/tests/gtest_cas_json_writer.cpp index 4eda28e717a3..ea12afa32bcb 100644 --- a/src/Disks/tests/gtest_cas_json_writer.cpp +++ b/src/Disks/tests/gtest_cas_json_writer.cpp @@ -15,17 +15,20 @@ TEST(CASJsonWriter, KeyValueSequenceMatchesCanonicalShape) { CasJsonWriter w; bool first = true; - w.key("we", first); + /// The names are shape labels, not format keys: this test is about the writer's primitives, and + /// borrowing a real wire spelling would put this file in every vocabulary sweep for no reason. + w.key("u64_string_field", first); w.u64StringValue(7); - w.key("mo", first); + w.key("number_field", first); w.u64Number(3); - w.key("ok", first); + w.key("bool_field", first); w.boolValue(true); - w.key("ome", first); + w.key("second_u64_string_field", first); w.u64StringValue(1); w.closeObject(first); w.newline(); - EXPECT_EQ(std::move(w).take(), "{\"we\":\"7\",\"mo\":3,\"ok\":true,\"ome\":\"1\"}\n"); + EXPECT_EQ(std::move(w).take(), + "{\"u64_string_field\":\"7\",\"number_field\":3,\"bool_field\":true,\"second_u64_string_field\":\"1\"}\n"); } TEST(CASJsonWriter, EmptyObjectAndClear) @@ -217,12 +220,12 @@ TEST(CASJsonWriter, WireKeyFieldHelpersMatchThePrimitivePairs) { CasJsonWriter w; bool first = true; - constexpr WireKey k_word{"st"}; - constexpr WireKey k_str{"hn"}; - constexpr WireKey k_u64s{"we"}; - constexpr WireKey k_num{"eat"}; - constexpr WireKey k_hex{"su"}; - constexpr WireKey k_bool{"fen"}; + constexpr WireKey k_word{"word_field"}; + constexpr WireKey k_str{"string_field"}; + constexpr WireKey k_u64s{"u64_string_field"}; + constexpr WireKey k_num{"number_field"}; + constexpr WireKey k_hex{"hex_field"}; + constexpr WireKey k_bool{"bool_field"}; writeWordField(w, k_word, "clean", first); writeStringField(w, k_str, "host-1", first); writeU64StringField(w, k_u64s, 7, first); @@ -232,11 +235,12 @@ TEST(CASJsonWriter, WireKeyFieldHelpersMatchThePrimitivePairs) w.closeObject(first); w.newline(); EXPECT_EQ(std::move(w).take(), - "{\"st\":\"clean\",\"hn\":\"host-1\",\"we\":\"7\",\"eat\":1752537630000," - "\"su\":\"00000000000000000000000000000001\",\"fen\":false}\n"); + "{\"word_field\":\"clean\",\"string_field\":\"host-1\",\"u64_string_field\":\"7\"," + "\"number_field\":1752537630000," + "\"hex_field\":\"00000000000000000000000000000001\",\"bool_field\":false}\n"); /// The reader-side comparison contract: a String key compares against the constant. - String key = "st"; + String key = "word_field"; EXPECT_TRUE(key == k_word); EXPECT_FALSE(key == k_str); } diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 55af475d2048..7a21b24985fd 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -1352,11 +1352,11 @@ TEST(CASMountLease, BodyCarriesFloorAndFence) m.started_at_ms = 1000; m.seq = 3; m.expires_at_ms = 2000; - m.min_active = 5; + m.min_active_build_sequence = 5; m.gc_fenced = true; m.write_attempt_id = UInt128{1}; const MountLease d = decodeMountLease(encodeMountLease(m)); - EXPECT_EQ(d.min_active, 5u); + EXPECT_EQ(d.min_active_build_sequence, 5u); EXPECT_TRUE(d.gc_fenced); EXPECT_EQ(d.writer_epoch, 7u); } @@ -1364,9 +1364,9 @@ TEST(CASMountLease, BodyCarriesFloorAndFence) TEST(CASMountLease, RetiredSentinelRoundTrips) { MountLease m; - m.min_active = std::numeric_limits::max(); + m.min_active_build_sequence = std::numeric_limits::max(); m.write_attempt_id = UInt128{1}; - EXPECT_EQ(decodeMountLease(encodeMountLease(m)).min_active, + EXPECT_EQ(decodeMountLease(encodeMountLease(m)).min_active_build_sequence, std::numeric_limits::max()); } @@ -1386,7 +1386,7 @@ constexpr uint64_t kStableThresholdMs = 10'000; /// `putIfAbsent`) — the same interface the keeper writes through. MountLease seedMount( Backend & b, const Layout & l, const String & srid, - uint64_t expires_at_ms, bool gc_fenced, uint64_t min_active, uint64_t seq = 1) + uint64_t expires_at_ms, bool gc_fenced, uint64_t min_active_build_sequence, uint64_t seq = 1) { MountLease m; m.server_uuid = UInt128(srid.back()); // distinct per srid; content is irrelevant to the gate @@ -1396,7 +1396,7 @@ MountLease seedMount( m.started_at_ms = kNowMs; m.seq = seq; m.expires_at_ms = expires_at_ms; - m.min_active = min_active; + m.min_active_build_sequence = min_active_build_sequence; m.gc_fenced = gc_fenced; m.write_attempt_id = UInt128{1}; b.putIfAbsent(l.mountKey(srid), encodeMountLease(m)); @@ -1425,7 +1425,7 @@ TEST(CASHeartbeatFloor, FirstSightNeverFencesEvenIfStampLooksExpired) /// A stamp that would have read as long-expired under the old skew-margin comparison — under /// rev.6 observation the stamp is never even consulted for the fence decision. - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; const HeartbeatFloor floor = computeHeartbeatFloor(*b, l, /*now_ms*/ kNowMs, /*mono_now_ms*/ 0, @@ -1441,7 +1441,7 @@ TEST(CASHeartbeatFloor, StableTokenPastThresholdIsFenced) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; const HeartbeatFloor floor_before = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); @@ -1465,7 +1465,7 @@ TEST(CASHeartbeatFloor, RenewalBetweenRoundsRestartsObservation) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); @@ -1494,8 +1494,8 @@ TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); - seedMount(*b, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(*b, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); @@ -1526,15 +1526,15 @@ TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) /// two live mounts — genuinely renewing between the two rounds below, so their observation never /// stabilizes. - seedMount(*b, l, "s1", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active*/ 0); - seedMount(*b, l, "s2", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(*b, l, "s2", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); /// dead — no renewal between the two rounds below — must be fenced-out by the second call. - seedMount(*b, l, "s3", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s3", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); /// already-fenced — excluded, body byte-identical after both calls (no PUT). - seedMount(*b, l, "s4", /*expires*/ kNowMs - 60'000, /*fenced*/ true, /*min_active*/ 0); - /// terminated (min_active == UINT64_MAX) with expired-looking timestamps — excluded, not fenced. + seedMount(*b, l, "s4", /*expires*/ kNowMs - 60'000, /*fenced*/ true, /*min_active_build_sequence*/ 0); + /// terminated (min_active_build_sequence == UINT64_MAX) with expired-looking timestamps — excluded, not fenced. seedMount(*b, l, "s5", /*expires*/ kNowMs - 60'000, /*fenced*/ false, - /*min_active*/ std::numeric_limits::max()); + /*min_active_build_sequence*/ std::numeric_limits::max()); MountObservationMap obs; @@ -1627,7 +1627,7 @@ TEST(CASHeartbeatFloor, FenceOutLosesTokenRaceReclassifiesLive) auto b = std::make_shared( l.mountKey("s1"), /*renewed_expires*/ kNowMs + 120'000); - seedMount(*b, l, "s1", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active*/ 0); + seedMount(*b, l, "s1", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; /// Round 1: first sight, observation starts — never reaches the fence-out path (the race @@ -1912,7 +1912,7 @@ TEST(CASFenceTerminal, CleanFarewellIsTerminal) auto got = b.get(l.mountKey("r")); ASSERT_TRUE(got.has_value()); MountLease retired = decodeMountLease(got->bytes); - retired.min_active = std::numeric_limits::max(); + retired.min_active_build_sequence = std::numeric_limits::max(); ASSERT_EQ(b.putOverwrite(l.mountKey("r"), encodeMountLease(retired), got->token).outcome, PutOutcome::Done); EXPECT_TRUE(isCreatorFenceTerminal(b, l, "r", 7)); diff --git a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp index 901b223cb90f..ba1b0925a20f 100644 --- a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp @@ -1,7 +1,6 @@ #include #include -#include #include #include #include @@ -9,11 +8,6 @@ #include "cas_test_helpers.h" #include -namespace DB::ErrorCodes -{ - extern const int UNKNOWN_FORMAT_VERSION; -} - /// Namespace files are keyed by an opaque LIFE, not by its name: `cas/ns/state//_files/` /// (Stage B Task 4b, directive design change 2). This file pins the three properties that re-key exists /// to produce, and the one it must NOT produce. @@ -229,51 +223,3 @@ TEST(CASNsFileIncarnation, RebirthDoesNotWaitForFilesToBeEmpty) EXPECT_EQ(row_it->second.cleanup_evidence->remove_txn_id, (RefTxnId{1, 1})); EXPECT_TRUE(backend->head(debris_key).exists) << "cleanup evidence does not gate on physical deletion"; } - -/// An old-format pool carrying unqualified `roots//_files/x` keys is REFUSED AT OPEN. It is not -/// read, not migrated, and not silently re-keyed: the file layer rides Task 4's format bump B, and the -/// pool-open floor is what makes "there is nothing to migrate" true rather than merely intended. -/// -/// Asserted at OPEN rather than at the parser on purpose: `Layout` has no unqualified key constructor -/// at all (a compile-time concept check in `gtest_cas_namespace_life_id.cpp` pins that, and -/// `parseNamespaceFileKey`'s refusal of a legacy key is pinned there too), so the only reachable -/// question left is whether a pool that CONTAINS such keys can be opened. It cannot. -TEST(CASNsFileIncarnation, LegacyUnqualifiedFileKeyIsRefusedAtOpen) -{ - auto backend = std::make_shared(); - const Layout layout("p"); - - /// A generation-5 `_pool_meta`: the current encoder's output with its header generation moved back - /// one, so every other byte is exactly what that generation really wrote. - PoolMeta meta; - meta.pool_id = hexToU128("0123456789abcdef0123456789abcdef"); - meta.blob_header_len = 256; - meta.min_reader_generation = kNamespaceLifeKeyedGeneration - 1; - meta.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - String encoded = encodePoolMeta(meta); - const String current_v = "\"v\":" + std::to_string(G_BUILD); - const String legacy_v = "\"v\":" + std::to_string(kNamespaceLifeKeyedGeneration); - const size_t at = encoded.find(current_v); - /// Guard the substitution itself: a silent no-op here would leave a CURRENT-generation pool and the - /// test would pass by opening a pool it believes it downgraded. - ASSERT_NE(at, String::npos) << "pool-meta header no longer spells its generation as " << current_v; - encoded.replace(at, current_v.size(), legacy_v); - ASSERT_NE(encoded.find(legacy_v), String::npos); - backend->putIfAbsent(layout.poolMetaKey(), encoded); - - /// The legacy artifact this task removes: a namespace file keyed by NAME ONLY, with no incarnation - /// segment. Written as raw bytes because no code path in the tree can produce this key any more. - backend->putIfAbsent("p/roots/" + kNsString + "/_files/" + kFile, "1\n"); - - try - { - openPoolForTest(backend); - FAIL() << "an old-format pool must fail closed at open, naming recreation"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find("recreate"), String::npos) - << "the refusal must tell the operator what to do; got: " << e.message(); - } -} diff --git a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp index 3166230f5ec8..77ff85d2b72d 100644 --- a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp @@ -104,7 +104,7 @@ void deleteCatalogLife( CasFoldSeal parent; parent.ref_lives.emplace(life1.incarnation, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); if (CasRefCatalog::deleteCompletedRemoving( backend, layout, *it, parent, 1, diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index 45f82c1a9e33..0a19b2418a00 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -166,7 +166,7 @@ TEST(CASOrphanManifestSweep, EligibleAndUnownedIsDeleted) registerNamespaceRaw(*backend, store->layout(), ns); const ManifestRef r = ref(5, 0xAB); writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); // body, no owner - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active*/6); // 6 > 5 => eligible + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active_build_sequence*/6); // 6 > 5 => eligible seedConsumedSealCursor(*backend, store->layout(), ns); seedEmptyRecoveryAuthority(*backend, store->layout(), ns); @@ -214,7 +214,7 @@ TEST(CASOrphanManifestSweep, CheckpointSnapshotAtOlderEpochSealSkipsDeletion) const ManifestRef candidate = ref(5, 0xAC); const String candidate_key = layout.manifestKey(ManifestId{ns, candidate}); writeManifestRaw(*backend, layout, ns, candidate, {blobEntryFor("a", DB::UInt128(1))}); - setWatermarkMinActive(*backend, layout, kServerRoot, kWriterEpoch, /*min_active=*/6); + setWatermarkMinActive(*backend, layout, kServerRoot, kWriterEpoch, /*min_active_build_sequence=*/6); seedConsumedSealCursor(*backend, layout, ns); std::vector warnings; @@ -299,7 +299,7 @@ TEST(CASOrphanManifestSweep, CursorPageAdvancesAndWrapsWithListBudget) const ManifestRef r2 = ref(5, 0xE2); writeManifestRaw(*backend, store->layout(), ns, r1, {blobEntryFor("a", DB::UInt128(1))}); writeManifestRaw(*backend, store->layout(), ns, r2, {blobEntryFor("b", DB::UInt128(2))}); - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, /*min_active_build_sequence*/6); const ManifestSweepResult first = sweepManifestCursorPageForTest(*store, "", /*list_budget*/1, /*delete_budget*/0); EXPECT_EQ(first.listed, 1u); @@ -553,11 +553,11 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip .ops = publishCommittedOps("phantom", phantom), .prev_epoch_seal = RefTxnId{6, 1}}; String malformed_later_link = encodeRefLogTxn(direct_later_link); - const String encoded_predecessor{R"("!pse":"6")"}; + const String encoded_predecessor{R"("!prev_epoch":"6")"}; const size_t predecessor_pos = malformed_later_link.find(encoded_predecessor); ASSERT_NE(predecessor_pos, String::npos); malformed_later_link.replace( - predecessor_pos, encoded_predecessor.size(), R"("!pse":"2")"); + predecessor_pos, encoded_predecessor.size(), R"("!prev_epoch":"2")"); ASSERT_EQ(backend->putIfAbsent( store->layout().refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, malformed_later_link)).outcome, PutOutcome::Done); diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index 43963c40e800..6285f171e3f3 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -150,7 +150,7 @@ ReadyFixture makeReadyFixture() .last_epoch_seal = RefTxnId{1, 2}, }); EXPECT_TRUE(runRegularRoundReclaiming(*f.gc).acquired_lease); - setWatermarkMinActive(*f.backend, f.store->layout(), "test", kCandidateEpoch, /*min_active=*/6); + setWatermarkMinActive(*f.backend, f.store->layout(), "test", kCandidateEpoch, /*min_active_build_sequence=*/6); std::vector entries; std::vector seeded_edges; diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index ac14519e04c5..e966b87a332a 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -5,6 +5,8 @@ #include #include +#include + using namespace DB::Cas; namespace @@ -67,11 +69,11 @@ TEST(CASFormatBattery, PartManifest) /// stays self-consistent with whatever sample() produces, now that decode verifies payload_digest. const String golden = currentFormatHeader("cas_part_manifest") + - "{\"me\":\"5\",\"mb\":\"15\",\"mo\":1,\"ns\":\"00/aa@cas@\",\"pd\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. - "{\"p\":\"a/b.bin\",\"pm\":\"blob\",\"ha\":\"ch128\",\"h\":\"00112233445566778899aabbccddeeff\",\"sz\":4096}\n" - "{\"p\":\"c/small.txt\",\"pm\":\"inline\",\"il\":12}\n" + "{\"epoch\":\"5\",\"build\":\"15\",\"ord\":1,\"root_namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. + "{\"path\":\"a/b.bin\",\"place\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\",\"size\":4096}\n" + "{\"path\":\"c/small.txt\",\"place\":\"inline\",\"size\":12}\n" "{\"n\":2}\n" - "==> \"c/small.txt\" il=12 <==\n" + "==> \"c/small.txt\" size=12 <==\n" "hello world!\n"; runFormatBattery({FormatId::PartManifest, [&] { return sealObject(FormatId::PartManifest, encodePartManifest(m)); }, @@ -115,17 +117,88 @@ TEST(CASPartManifestFormat, EmptyEntriesRoundTrips) TEST(CASPartManifestFormat, PlacementWordsRenderAndRejectUnknown) { const String text = encodePartManifest(sample()); - EXPECT_NE(text.find("\"pm\":\"blob\""), String::npos); - EXPECT_NE(text.find("\"pm\":\"inline\""), String::npos); + EXPECT_NE(text.find("\"place\":\"blob\""), String::npos); + EXPECT_NE(text.find("\"place\":\"inline\""), String::npos); /// An unknown placement word fails closed. String bad = text; - const size_t pos = bad.find(R"("pm":"blob")"); + const size_t pos = bad.find(R"("place":"blob")"); ASSERT_NE(pos, String::npos); - bad.replace(pos, String(R"("pm":"blob")").size(), R"("pm":"bogus")"); + bad.replace(pos, String(R"("place":"blob")").size(), R"("place":"bogus")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } +/// Closed-set pin: the two `EntryPlacement` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASPartManifestFormat, ClosedSetPinsEntryPlacementWords) +{ + EXPECT_EQ(entryPlacementToWireWord(EntryPlacement::Inline), "inline"); + EXPECT_EQ(entryPlacementToWireWord(EntryPlacement::Blob), "blob"); + for (const auto p : magic_enum::enum_values()) + EXPECT_EQ(entryPlacementFromWireWord(entryPlacementToWireWord(p)), p); +} + +TEST(CASPartManifestFormat, SizeBeforePlaceIsAcceptedForBothPlacements) +{ + String blob_first = encodePartManifest(sample()); + const String blob_record = R"("place":"blob","algo":"ch128","digest":"00112233445566778899aabbccddeeff","size":4096)"; + const size_t blob_pos = blob_first.find(blob_record); + ASSERT_NE(blob_pos, String::npos); + blob_first.replace(blob_pos, blob_record.size(), + R"("size":4096,"place":"blob","algo":"ch128","digest":"00112233445566778899aabbccddeeff")"); + EXPECT_EQ(decodePartManifest(blob_first).entries[0].blob_size, 4096u); + + String inline_first = encodePartManifest(sample()); + const String inline_record = R"("place":"inline","size":12)"; + const size_t inline_pos = inline_first.find(inline_record); + ASSERT_NE(inline_pos, String::npos); + inline_first.replace(inline_pos, inline_record.size(), R"("size":12,"place":"inline")"); + EXPECT_EQ(decodePartManifest(inline_first).entries[1].inline_bytes, "hello world!"); +} + +TEST(CASPartManifestFormat, MissingSizeIsRejectedForBothPlacements) +{ + /// The MESSAGE is asserted, not just the code: a manifest whose entry lost its size also fails + /// the payload-digest check (blob) and the banner rebuild (inline), both of which raise the same + /// code, so a code-only assertion would still pass with the per-placement fences deleted. + const auto expect_message = [](const String & text, std::string_view expected) + { + try + { + static_cast(decodePartManifest(text)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), expected); + } + }; + + String blob_missing_size = encodePartManifest(sample()); + const size_t blob_pos = blob_missing_size.find(",\"size\":4096"); + ASSERT_NE(blob_pos, String::npos); + blob_missing_size.erase(blob_pos, String(",\"size\":4096").size()); + expect_message(blob_missing_size, "PartManifest: blob entry 'a/b.bin' missing size"); + + String inline_missing_size = encodePartManifest(sample()); + const size_t inline_pos = inline_missing_size.find(",\"size\":12"); + ASSERT_NE(inline_pos, String::npos); + inline_missing_size.erase(inline_pos, String(",\"size\":12").size()); + expect_message(inline_missing_size, "PartManifest: inline entry 'c/small.txt' missing size"); +} + +TEST(CASPartManifestFormat, DuplicateSizeIsRejected) +{ + String text = encodePartManifest(sample()); + const String size = R"("size":4096)"; + const size_t pos = text.find(size); + ASSERT_NE(pos, String::npos); + text.replace(pos, size.size(), R"("size":1,"size":4096)"); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(text); }); +} + /// Proves the payload zone, not JSON-string escaping: an Inline entry whose bytes contain an /// embedded '\n', a NUL byte, and a '"' character round-trip byte-faithfully. If this content were /// carried as a JSON string value it would need escaping (or would be flatly invalid for the NUL @@ -198,7 +271,7 @@ TEST(CASPartManifestFormat, InlineBannerCarriesTheEscapedPath) m.entries = {e}; m.payload_digest = computePayloadDigest(m); - EXPECT_NE(encodePartManifest(m).find("==> \"p\\nq.proj/c.txt\" il=1 <=="), String::npos); + EXPECT_NE(encodePartManifest(m).find("==> \"p\\nq.proj/c.txt\" size=1 <=="), String::npos); } TEST(CASPartManifestFormat, ByteDeterminism) @@ -322,8 +395,8 @@ TEST(CASPartManifestFormat, DecodeRejectsOutOfOrderEntries) m.payload_digest = computePayloadDigest(m); const String text = encodePartManifest(m); - const size_t pos_a = text.find(R"("p":"a/one.bin")"); - const size_t pos_b = text.find(R"("p":"b/two.bin")"); + const size_t pos_a = text.find(R"("path":"a/one.bin")"); + const size_t pos_b = text.find(R"("path":"b/two.bin")"); ASSERT_NE(pos_a, String::npos); ASSERT_NE(pos_b, String::npos); @@ -366,10 +439,10 @@ TEST(CASPartManifestFormat, DecodeRejectsNonAdjacentDuplicatePath) m.payload_digest = computePayloadDigest(m); String forged = encodePartManifest(m); - const String needle = R"("p":"ccc/three.bin")"; + const String needle = R"("path":"ccc/three.bin")"; const size_t pos = forged.find(needle); ASSERT_NE(pos, String::npos); - forged.replace(pos, needle.size(), R"("p":"aaa/one.bin")"); + forged.replace(pos, needle.size(), R"("path":"aaa/one.bin")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(forged); }); } @@ -377,10 +450,10 @@ TEST(CASPartManifestFormat, DecodeRejectsNonAdjacentDuplicatePath) TEST(CASPartManifestFormat, UnknownEntryAlgoFailsClosed) { String bad = encodePartManifest(sample()); - const String needle = R"("ha":"ch128")"; + const String needle = R"("algo":"ch128")"; const size_t pos = bad.find(needle); ASSERT_NE(pos, String::npos); - bad.replace(pos, needle.size(), R"("ha":"bogus")"); + bad.replace(pos, needle.size(), R"("algo":"bogus")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } @@ -390,7 +463,7 @@ TEST(CASPartManifestFormat, UnknownEntryAlgoFailsClosed) TEST(CASPartManifestFormat, DigestHexWidthMismatchFailsClosedNotBadArguments) { String bad = encodePartManifest(sample()); - const String key = R"("h":")"; + const String key = R"("digest":")"; const size_t key_pos = bad.find(key); ASSERT_NE(key_pos, String::npos); const size_t hex_start = key_pos + key.size(); @@ -429,18 +502,18 @@ TEST(CASPartManifestFormat, TrailingByteAfterPayloadZoneFailsClosed) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } -/// An Inline entry's record "il" disagrees with what the payload zone's banner+bytes actually -/// declare (the banner and bytes are left as originally written; only the record line's "il" is -/// edited). The record's declared `il` is what decode uses both to build the expected banner text +/// An Inline entry's record `size` disagrees with what the payload zone's banner+bytes actually +/// declare (the banner and bytes are left as originally written; only the record line's `size` is +/// edited). The record's declared `size` is what decode uses both to build the expected banner text /// and to know how many bytes to read from the zone, so this must fail closed rather than silently /// reading the wrong byte count. -TEST(CASPartManifestFormat, InlineRecordIlMismatchWithPayloadZoneBannerFailsClosed) +TEST(CASPartManifestFormat, InlineRecordSizeMismatchWithPayloadZoneBannerFailsClosed) { String bad = encodePartManifest(sample()); - const String needle = "\"il\":12"; + const String needle = "\"size\":12"; const size_t pos = bad.find(needle); ASSERT_NE(pos, String::npos); - bad.replace(pos, needle.size(), "\"il\":13"); + bad.replace(pos, needle.size(), "\"size\":13"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 7a426676cbc8..a92d8f9bc4d1 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -1260,7 +1260,7 @@ TEST(CASPartWriteTxn, AbandonRemovesStagedDebrisAndDisables) { /// Port of AbandonLeavesDebrisAndDisables to the new abandon semantics (CasPartWriteTxn.cpp abandon): /// abandon best-effort exact-token-DELETEs this build's STAGED manifest debris, leaves blob bodies - /// (full GC's job via min_active), and disables the build (further ops throw via requireAlive). + /// (full GC's job via min_active_build_sequence), and disables the build (further ops throw via requireAlive). auto b = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"srv1/tbl"}; diff --git a/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp b/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp index b1022505585d..1f86c6fe7d47 100644 --- a/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp +++ b/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp @@ -62,7 +62,7 @@ ManifestEntry blobEntry(const String & name, const String & payload) /// public PartWriteTxn/Pool/Gc API (no snap injection): /// /// PartWriteTxn A uploads blob P and publishes refA -> t1 -> { data.bin: P }. A is then RELEASED (dtor), -/// retiring its build_seq so the GC watermark `min_active` advances PAST A. P now carries A's +/// retiring its build_seq so the GC watermark `min_active_build_sequence` advances PAST A. P now carries A's /// `cas_owner` and is no longer protected by any in-flight build. /// /// PartWriteTxn B starts and ADOPTS the same blob P via tokenless evidence (adoptEvidence — the cross-node @@ -83,7 +83,7 @@ TEST(CASPartWriteTxnRootDangle, SharedBlobSurvivesSourceDropDuringBuild) const String P = "shared-blob-payload-P"; /// PartWriteTxn A: upload P, publish refA -> manifest -> { data.bin: P }, then release A so its build_seq - /// retires and min_active advances past it. + /// retires and min_active_build_sequence advances past it. { PartWriteInfo info; info.intended_ref = ns.string() + "/refA"; @@ -93,7 +93,7 @@ TEST(CASPartWriteTxnRootDangle, SharedBlobSurvivesSourceDropDuringBuild) a->putBlob(idOf(P), BlobSource::fromString(P)); a->promote(ns, "refA", a->buildId(), id); } - s->renewWatermarkOnce(); /// A is gone; min_active now advances past A's build_seq + s->renewWatermarkOnce(); /// A is gone; min_active_build_sequence now advances past A's build_seq /// PartWriteTxn B: adopt the SAME blob P (cross-node adopt — tokenless evidence via adoptEvidence), assemble /// its manifest, and precommitAdd it. The precommit pins P's closure (fold +1 edge) for the build. @@ -145,7 +145,7 @@ TEST(CASPartWriteTxnRootDangle, PrematureReclaimCommitFailsClosed) const RootNamespace ns{"test/tbl"}; const String P = "shared-blob-payload-P-reclaim"; - /// PartWriteTxn A: upload P, publish refA -> manifest, retire A so min_active advances past it. + /// PartWriteTxn A: upload P, publish refA -> manifest, retire A so min_active_build_sequence advances past it. { PartWriteInfo info; info.intended_ref = ns.string() + "/refA"; @@ -232,7 +232,7 @@ TEST(CASPartWriteTxnRoot, LivePrecommitNotReclaimed) const String Q = "live-build-blob-payload-Q"; /// PartWriteTxn B stays ALIVE: upload Q, assemble, precommitAdd — and we DO NOT retire its seq. So - /// `min_active <= build_seq` (B is in-flight) and the watermark keeps a live, advancing seq. + /// `min_active_build_sequence <= build_seq` (B is in-flight) and the watermark keeps a live, advancing seq. PartWriteInfo binfo; binfo.intended_ref = ns.string() + "/refLive"; auto b = s->beginPartWrite(binfo); @@ -240,7 +240,7 @@ TEST(CASPartWriteTxnRoot, LivePrecommitNotReclaimed) b->precommitAdd(ns, "refLive", t); b->putBlob(idOf(Q), BlobSource::fromString(Q)); s->renewWatermarkOnce(); - ASSERT_LE(s->minActive(), b->buildSeq()) << "precondition: B must be in-flight (min_active <= seq)"; + ASSERT_LE(s->minActive(), b->buildSeq()) << "precondition: B must be in-flight (min_active_build_sequence <= seq)"; /// GC to fixpoint while B is live. Gc gc(s, u128Of("gc-b8-live")); diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index c960c0547037..a74baf05fa61 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -680,22 +680,15 @@ TEST(CASPluggableHash, ForeignAlgoSegmentIsDebrisNotOurs) } /// ============================================================================================ -/// CAS reader-generation gate (`Core/Formats/CasFormat.h`'s `G_BUILD`) was raised to 4 for -/// per-namespace contiguous ref-log ids (INV-1) and has since moved again, to 5, for Stage B's -/// namespace-life-keyed ref layer ("format bump B", `kNamespaceLifeKeyedGeneration`) -- this test's -/// assertions read `G_BUILD` itself rather than a hardcoded generation number for exactly that reason, -/// so a THIRD bump does not silently make them false. `PoolMeta::createOrValidate`'s open-time -/// CAS-raise targets `G_BUILD`, and `decodePoolMeta` fail-closes BOTH on a FUTURE -/// `min_reader_generation` AND on a BACKWARD pool whose header `compatibility_version` is below -/// `kNamespaceLifeKeyedGeneration` (which, being the LATER of the two historical breaking-change -/// floors, subsumes `kContiguousRefStreamsGeneration` -- see `CasPoolMetaFormat.cpp`). +/// CAS reader-generation gate (`CasFormat.h`'s `G_BUILD`). This test's assertions read `G_BUILD` +/// itself rather than a hardcoded generation number, so a future bump does not silently make them +/// false. `PoolMeta::createOrValidate`'s open-time CAS-raise targets `G_BUILD`, and `decodePoolMeta` +/// fail-closes BOTH on a FUTURE `min_reader_generation` AND on a BACKWARD pool whose header +/// `compatibility_version` is below the format-generation baseline (see `CasPoolMetaFormat.cpp`). /// ============================================================================================ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { - EXPECT_GE(G_BUILD, kNamespaceLifeKeyedGeneration) - << "the reader-generation gate must be at least the namespace-life-keyed floor it enforces"; - /// A freshly opened/created pool records `min_reader_generation == G_BUILD` (the open-time /// CAS-raise, `PoolMeta::createOrValidate`, always targets this build's own floor). { @@ -709,8 +702,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) } /// FORWARD gate: a pool-meta carrying `min_reader_generation == G_BUILD + 1` (one generation past - /// THIS build's floor) still fails closed at open -- the startup gate (`decodePoolMeta`) rejects it - /// even though generation 4 is now understood. + /// THIS build's floor) fails closed at open -- the startup gate (`decodePoolMeta`) rejects it. { auto backend = std::make_shared(); const Layout layout("p"); @@ -722,12 +714,9 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); }); } - /// BACKWARD floor: a pool whose header `v` (compatibility_version) is BELOW `G_BUILD` was written - /// by an older build this reader can no longer trust -- today that is one generation short of - /// `kNamespaceLifeKeyedGeneration`, a pool whose ref-object keys carry no incarnation segment, - /// which this build's parsers refuse as corruption rather than read. Craft it at the text layer: - /// take a fresh pool-meta and rewrite its line-1 version gate down to `G_BUILD - 1` (an older - /// build would have stamped exactly that). + /// BACKWARD floor: a pool whose header `v` (compatibility_version) is below the format-generation + /// baseline predates every build this reader can trust. Craft it at the text layer: take a fresh + /// pool-meta and rewrite its line-1 version gate down to `G_BUILD - 1`. { auto backend = std::make_shared(); const Layout layout("p"); diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index b0ad1b340afd..8e07048439f8 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -511,6 +511,7 @@ TEST(CASPoolMeta, RejectsBadConstantsOnDecode) PoolMeta bad_pm; bad_pm.pool_id = hexToU128("00000000000000000000000000000001"); bad_pm.blob_header_len = 100; /// violates 8-alignment invariant + bad_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(bad_pm)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { PoolMeta::createOrValidate(*b, layout, 256); }); @@ -2009,7 +2010,7 @@ TEST(CASPoolShutdown, CleanStopDrainsAndWritesFarewell) const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()); const MountLease lease = decodeMountLease(got->bytes); - EXPECT_EQ(lease.min_active, std::numeric_limits::max()) + EXPECT_EQ(lease.min_active_build_sequence, std::numeric_limits::max()) << "a clean drain (no in-flight ref-log PUT) must write the farewell marker"; } @@ -2046,7 +2047,7 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) const auto got = backend->get(mount_key); ASSERT_TRUE(got.has_value()); const MountLease lease = decodeMountLease(got->bytes); - EXPECT_NE(lease.min_active, std::numeric_limits::max()) + EXPECT_NE(lease.min_active_build_sequence, std::numeric_limits::max()) << "an unresolved ref-log PUT must skip the clean-release farewell marker"; EXPECT_FALSE(lease.gc_fenced); @@ -2074,7 +2075,7 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) Layout l{"p"}; DB::Cas::tests::seedPoolMetaForRestart(*b); /// Predecessor: claim epoch 7, no farewell (simulate crash: just drop the keeper) -- a bare - /// `claimMount` plants the lease directly, with no clean-farewell `min_active` marker and no + /// `claimMount` plants the lease directly, with no clean-farewell `min_active_build_sequence` marker and no /// `gc_fenced`, so the successor below has no certificate of death until it observes one itself. ASSERT_EQ(claimMount(*b, l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, MountClaimResult::Claimed); @@ -2121,7 +2122,7 @@ TEST(CASMountOpenWaits, CleanOpenSkipsAllWaits) { auto b = std::make_shared(); /// Predecessor released cleanly (drain + farewell from Task 5): open, then reset() drives ~Pool(), - /// which -- with nothing in flight -- writes the farewell marker (min_active == UINT64_MAX). + /// which -- with nothing in flight -- writes the farewell marker (min_active_build_sequence == UINT64_MAX). auto predecessor = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test"}); predecessor.reset(); @@ -2677,7 +2678,7 @@ TEST(CASPoolRemount, TeardownJoinsBothWorkersBeforeRelease) runtime.stopBackgroundWorkers(); EXPECT_EQ(worker_exits.load(), 2u); runtime.finishTeardown(true); - EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active, + EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active_build_sequence, std::numeric_limits::max()); } @@ -3372,7 +3373,7 @@ TEST(CASPoolShutdown, PreSendCancellationAllowsFarewellButAmbiguityDoesNot) runtime.stopBackgroundWorkers(); } runtime.finishTeardown(true); - return decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active; + return decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active_build_sequence; }; EXPECT_EQ(run(false), std::numeric_limits::max()); diff --git a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp index ee1b2a4d4d99..07a9591e6fc6 100644 --- a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp +++ b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp @@ -431,7 +431,7 @@ TEST(CASRebuildCondemnNothing, CarriesHoldsVerbatimWhileCondemningNothing) const CasFoldSeal seal = decodeFoldSeal(backend->get(layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); const auto it = seal.ref_lives.find(catalogLifeIdForTest(*backend, layout, kNsA)); ASSERT_NE(it, seal.ref_lives.end()); - EXPECT_EQ(it->second.coverage.classification, 4); + EXPECT_EQ(it->second.coverage.classification, CoverageClass::Clamped); ASSERT_TRUE(it->second.coverage.hold.has_value()); EXPECT_EQ(*it->second.coverage.hold, planted) << "a rebuild retried nothing, so it rewrites nothing about the hold"; diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 2c2805f49310..355ec21a2126 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -7,6 +7,8 @@ #include #include +#include + using namespace DB; using namespace DB::Cas; @@ -76,7 +78,7 @@ TEST(CASFormatBattery, RunFile) [&] { return sealObject(FormatId::RunFile, encodeRun(records)); }, [](std::string_view s) { decodeRun(std::string(openObject(FormatId::RunFile, s))); }, fmt::format("{{\"type\":\"cas_run\",\"v\":{},\"kind\":\"source_edge\"}}\n", currentCompatibilityVersion()) + - "{\"b\":\"0100000000000000000000000000000002\",\"s\":\"00000000000000000000000000000005\",\"m\":\"edge\"}\n" + "{\"ref\":\"0100000000000000000000000000000002\",\"src\":\"00000000000000000000000000000005\",\"mark\":\"edge\"}\n" "{\"n\":1}\n"}); } @@ -126,6 +128,69 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) EXPECT_EQ(back[2].marker, RunMarker::Zero); } +/// Closed-set pin: the three `RunMarker` words, walked through `magic_enum::enum_values`, which is what proves the +/// renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASRecordStream, ClosedSetPinsRunMarkerWords) +{ + EXPECT_EQ(runMarkerToWireWord(RunMarker::Zero), "zero"); + EXPECT_EQ(runMarkerToWireWord(RunMarker::Edge), "edge"); + EXPECT_EQ(runMarkerToWireWord(RunMarker::Condemned), "condemned"); + for (const auto m : magic_enum::enum_values()) + EXPECT_EQ(runMarkerFromWireWord(runMarkerToWireWord(m)), m); +} + +/// The condemned row's six fields are all-or-nothing: a row that says `condemned` but drops one of +/// them would decode with a silently defaulted value (a zero size, an empty token, `pending` false), +/// which is a different retention decision than the writer recorded. +TEST(CASRecordStream, CondemnedRowMissingOneOfItsSixFieldsFailsClosed) +{ + const String good = encodeRun({condemned(chRef(2), Token{"e-1", TokenType::ETag}, 4242, 7, /*pend*/ true)}); + for (const std::string_view field : {R"(,"pending":true)", R"(,"token_type":"etag")", R"(,"token":"e-1")", + R"(,"size":4242)", R"(,"condemn_round":"7")", R"(,"confirmed":false)"}) + { + String bytes = good; + const size_t at = bytes.find(field); + ASSERT_NE(at, String::npos) << "fixture does not carry " << field; + bytes.erase(at, field.size()); + try + { + static_cast(decodeRun(bytes)); + FAIL() << "expected CORRUPTED_DATA after dropping " << field; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + const String expected_message = field == R"(,"token_type":"etag")" || field == R"(,"token":"e-1")" + ? "CAS cas_run: token missing token_type/token" + : "CAS cas_run: condemned record missing pending/size/condemn_round/confirmed"; + EXPECT_EQ(e.message(), expected_message); + } + } +} + +/// The mirror fence: an active row carrying any condemned field is a row whose two halves disagree +/// about what it is, and the reader must not pick one half. +TEST(CASRecordStream, ActiveRowCarryingACondemnedFieldFailsClosed) +{ + String bytes = encodeRun({edge(chRef(1), 10)}); + const String needle = R"(,"mark":"edge")"; + const size_t at = bytes.find(needle); + ASSERT_NE(at, String::npos); + bytes.insert(at + needle.size(), R"(,"size":4242)"); + + try + { + static_cast(decodeRun(bytes)); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), "CAS cas_run: non-condemned record carries condemned fields"); + } +} + TEST(CASRecordStream, WriterIsByteDeterministic) { std::vector recs = { @@ -136,6 +201,24 @@ TEST(CASRecordStream, WriterIsByteDeterministic) EXPECT_EQ(encodeRun(recs), encodeRun(recs)); /// pure function of the sorted record set } +/// The run `ref` carries the algorithm as a raw leading BYTE, a second representation of the same +/// closed set the `algo` WORD spells elsewhere. The word side is proven exhaustive at compile time by +/// its wire table; the byte side is a hand-written switch, so nothing but this walk stops a new +/// algorithm from being written by `renderB` and rejected by the reader -- an asymmetry that would +/// appear as unreadable runs rather than as a failing build. +TEST(CASRecordStream, EveryBlobHashAlgoRoundTripsThroughTheRunRefByte) +{ + for (const BlobHashAlgo algo : magic_enum::enum_values()) + { + BlobDigest digest{}; + digest.bytes[0] = 0x10; + const BlobRef ref{algo, digest}; + const std::vector back = decodeRun(encodeRun({edge(ref, 1)})); + ASSERT_EQ(back.size(), 1u) << "algo " << magic_enum::enum_name(algo); + EXPECT_EQ(back[0].ref.algo, algo) << "the leading byte did not survive the round trip"; + } +} + TEST(CASRecordStream, SortOrderAcrossAlgosFollowsAlgoByte) { /// b = . The algo byte leads, so string-sorting b reproduces the @@ -173,9 +256,9 @@ TEST(CASRecordStream, SourceIdRendersAs32Hex) { const String bytes = encodeRun({edge(chRef(1), 10)}); /// The source id 10 is a 32-char lowercase hex string ending in 'a'. - EXPECT_NE(bytes.find("\"s\":\"0000000000000000000000000000000a\""), String::npos); - /// The record key `b` for a ch128 ref is the algo byte 01 + a 32-hex digest (34 chars total). - EXPECT_NE(bytes.find("\"b\":\"01"), String::npos); + EXPECT_NE(bytes.find("\"src\":\"0000000000000000000000000000000a\""), String::npos); + /// The record key `ref` for a ch128 ref is the algo byte 01 + a 32-hex digest (34 chars total). + EXPECT_NE(bytes.find("\"ref\":\"01"), String::npos); } TEST(CASRecordStream, SealChecksumMismatchFailsClosed) @@ -248,12 +331,13 @@ TEST(CASRecordStream, HeaderGates) { /// Wrong type. { - const String s = "{\"type\":\"cas_pool_meta\",\"v\":3,\"kind\":\"source_edge\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_pool_meta\",\"v\":1,\"kind\":\"source_edge\"}\n{\"n\":0}\n"; EXPECT_THROW(decodeRun(s), DB::Exception); } - /// Wrong kind. + /// Wrong kind. `v:1` is the baseline generation, so it always passes the header gate before the + /// kind check runs. { - const String s = "{\"type\":\"cas_run\",\"v\":3,\"kind\":\"blob_delta\"}\n{\"n\":0}\n"; + const String s = "{\"type\":\"cas_run\",\"v\":1,\"kind\":\"blob_delta\"}\n{\"n\":0}\n"; EXPECT_THROW(decodeRun(s), DB::Exception); } /// Future version -> UNKNOWN_FORMAT_VERSION. diff --git a/src/Disks/tests/gtest_cas_recovery_grounding.cpp b/src/Disks/tests/gtest_cas_recovery_grounding.cpp index 5362813bdfa6..d908038a153f 100644 --- a/src/Disks/tests/gtest_cas_recovery_grounding.cpp +++ b/src/Disks/tests/gtest_cas_recovery_grounding.cpp @@ -269,9 +269,9 @@ TEST(CASRecoveryGrounding, RejectsLifeEpochAboveCommittedFrontierOnDecodeAndGrou { const RefCkpt invalid = ckpt(2, RefTxnId{1, 5}); String encoded = encodeRefCkpt(ckpt(1, RefTxnId{1, 5})); - const size_t life_epoch = encoded.find(R"("le":"1")"); + const size_t life_epoch = encoded.find(R"("life_epoch":"1")"); ASSERT_NE(life_epoch, String::npos); - encoded.replace(life_epoch, String{R"("le":"1")"}.size(), R"("le":"2")"); + encoded.replace(life_epoch, String{R"("life_epoch":"1")"}.size(), R"("life_epoch":"2")"); expectCode([&] { (void)decodeRefCkpt(encoded); }, DB::ErrorCodes::CORRUPTED_DATA); expectCode([&] { (void)chooseRecoveryGrounding(catalog(NsState::Live), invalid); }, @@ -568,9 +568,9 @@ TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = RefTxnId{1, 2}}); - const size_t frontier_sequence = malformed_ckpt.find(R"("cts":"2")"); + const size_t frontier_sequence = malformed_ckpt.find(R"("committed_seq":"2")"); ASSERT_NE(frontier_sequence, String::npos); - malformed_ckpt.replace(frontier_sequence, String{R"("cts":"2")"}.size(), R"("cts":"3")"); + malformed_ckpt.replace(frontier_sequence, String{R"("committed_seq":"2")"}.size(), R"("committed_seq":"3")"); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -635,9 +635,9 @@ TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); - const size_t frontier_epoch = malformed_ckpt.find(R"("cte":"1")"); + const size_t frontier_epoch = malformed_ckpt.find(R"("committed_epoch":"1")"); ASSERT_NE(frontier_epoch, String::npos); - malformed_ckpt.replace(frontier_epoch, String{R"("cte":"1")"}.size(), R"("cte":"2")"); + malformed_ckpt.replace(frontier_epoch, String{R"("committed_epoch":"1")"}.size(), R"("committed_epoch":"2")"); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index 41d88fa57487..a458bbb52b6b 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -16,6 +16,8 @@ #include #include +#include + using namespace DB::Cas; namespace ProfileEvents @@ -56,27 +58,27 @@ namespace DB::ErrorCodes namespace { -/// Hand-builds one raw "ent" line, bypassing `encodeRefCatalog` entirely -- used by the decode-side +/// Hand-builds one raw `entry` line, bypassing `encodeRefCatalog` entirely -- used by the decode-side /// rejection tests, which must exercise bytes the encoder itself would refuse to produce. -String rawEntLine(const String & ns, const String & state, const String & inc_hex, +String rawEntryLine(const String & ns, const String & state, const String & inc_hex, std::optional> creator = std::nullopt) { if (!creator) - return fmt::format(R"({{"k":"ent","ns":"{}","st":"{}","inc":"{}"}})", ns, state, inc_hex); + return fmt::format(R"({{"kind":"entry","ns":"{}","state":"{}","life":"{}"}})", ns, state, inc_hex); const auto & [srid, we, fg] = *creator; - return fmt::format(R"({{"k":"ent","ns":"{}","st":"{}","inc":"{}","csr":"{}","cwe":"{}","cfg":"{}"}})", + return fmt::format(R"({{"kind":"entry","ns":"{}","state":"{}","life":"{}","creator":"{}","creator_epoch":"{}","creator_fence":"{}"}})", ns, state, inc_hex, srid, we, fg); } -/// Wraps `ent_lines` in the header/trailer a real `cas_ref_catalog` object carries. `v:1` always +/// Wraps `entry_lines` in the header/trailer a real `cas_ref_catalog` object carries. `v:1` always /// passes the header gate (any version <= the build's `G_BUILD` does), matching the convention /// `gtest_cas_fold_seal_format.cpp`'s `RejectsOutOfRangeNsCleanupState` uses for the same reason. -String rawCatalog(const std::vector & ent_lines) +String rawCatalog(const std::vector & entry_lines) { String out = R"({"type":"cas_ref_catalog","v":1})" "\n"; - for (const String & l : ent_lines) + for (const String & l : entry_lines) out += l + "\n"; - out += fmt::format("{{\"n\":{}}}\n", ent_lines.size()); + out += fmt::format("{{\"n\":{}}}\n", entry_lines.size()); return out; } @@ -84,7 +86,7 @@ String withRemovalStartedRound(String line, uint64_t round) { const size_t close = line.rfind('}'); EXPECT_NE(close, String::npos); - line.insert(close, fmt::format(R"(,"rsr":"{}")", round)); + line.insert(close, fmt::format(R"(,"remove_round":"{}")", round)); return line; } @@ -223,12 +225,25 @@ TEST(CASFormatBattery, RefCatalog) [&] { return sealObject(FormatId::RefCatalog, encodeRefCatalog(c)); }, [](std::string_view s) { decodeRefCatalog(std::string(openObject(FormatId::RefCatalog, s))); }, currentFormatHeader("cas_ref_catalog") + - "{\"k\":\"ent\",\"ns\":\"a\",\"st\":\"creating\",\"inc\":\"00000000000000000000000000000001\"," - "\"csr\":\"srv1\",\"cwe\":\"5\",\"cfg\":\"2\"}\n" - "{\"k\":\"ent\",\"ns\":\"b\",\"st\":\"live\",\"inc\":\"00000000000000000000000000000002\"}\n" + "{\"kind\":\"entry\",\"ns\":\"a\",\"state\":\"creating\",\"life\":\"00000000000000000000000000000001\"," + "\"creator\":\"srv1\",\"creator_epoch\":\"5\",\"creator_fence\":\"2\"}\n" + "{\"kind\":\"entry\",\"ns\":\"b\",\"state\":\"live\",\"life\":\"00000000000000000000000000000002\"}\n" "{\"n\":2}\n"}); } +/// Closed-set pin: the three `NsState` words, walked through `magic_enum::enum_values`, which is what +/// proves the renderer and the parser consult the SAME table: a table entry missing altogether is +/// already a build error at the coverage assert, but two delegates drifting onto different tables is +/// not. +TEST(CASRefCatalogFormat, ClosedSetPinsNsStateWords) +{ + EXPECT_EQ(nsStateToWord(NsState::Creating), "creating"); + EXPECT_EQ(nsStateToWord(NsState::Live), "live"); + EXPECT_EQ(nsStateToWord(NsState::Removing), "removing"); + for (const auto s : magic_enum::enum_values()) + EXPECT_EQ(nsStateFromWord(nsStateToWord(s)), s); +} + /// ---------- codec round-trip ---------- TEST(CASRefCatalogFormat, RoundTripsAllThreeStates) @@ -262,16 +277,16 @@ TEST(CASRefCatalogFormat, RemovalStartedRoundIsRequiredExactlyForRemoving) .removal_started_round = 19}; const RefCatalog catalog{.entries = {removing}}; const String encoded = encodeRefCatalog(catalog); - EXPECT_NE(encoded.find("\"rsr\":\"19\""), String::npos); - EXPECT_NE(encoded.find("\"st\":\"removing\""), String::npos); + EXPECT_NE(encoded.find("\"remove_round\":\"19\""), String::npos); + EXPECT_NE(encoded.find("\"state\":\"removing\""), String::npos); EXPECT_EQ(decodeRefCatalog(encoded), catalog); const String inc = "00000000000000000000000000000009"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { (void)decodeRefCatalog(rawCatalog({rawEntLine("missing", "removing", inc)})); }); + [&] { (void)decodeRefCatalog(rawCatalog({rawEntryLine("missing", "removing", inc)})); }); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - (void)decodeRefCatalog(rawCatalog({withRemovalStartedRound(rawEntLine("forbidden", "live", inc), 21)})); + (void)decodeRefCatalog(rawCatalog({withRemovalStartedRound(rawEntryLine("forbidden", "live", inc), 21)})); }); } @@ -477,7 +492,7 @@ TEST(CASRefCatalogFormatDeathTest, EncodeRejectsLiveWithRemovalStartedRoundAbort #endif /// A namespace + creator server_root_id that both max out at their respective byte bounds (512 + -/// 255), escaped worst-case, land one "ent" line over the 4 KiB line cap (~4.7 KiB) -- reachable +/// 255), escaped worst-case, land one `entry` line over the 4 KiB line cap (~4.7 KiB) -- reachable /// because neither this codec nor `validateServerRootId` restricts the charset, only the length. /// The refusal must be `LIMIT_EXCEEDED` (a capacity refusal), not `LOGICAL_ERROR` (a bug report) -- /// `encodeFoldSeal`'s own `checkLineBytes` raises `LIMIT_EXCEEDED` for the identical shape of gate. @@ -496,60 +511,76 @@ TEST(CASRefCatalogFormat, EncodeLineOverCapRaisesLimitExceeded) TEST(CASRefCatalogFormat, DecodeRejectsDuplicateNamespace) { - const String bad = rawCatalog({rawEntLine("a", "live", u128ToHex(UInt128(1))), - rawEntLine("a", "live", u128ToHex(UInt128(2)))}); + const String bad = rawCatalog({rawEntryLine("a", "live", u128ToHex(UInt128(1))), + rawEntryLine("a", "live", u128ToHex(UInt128(2)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsNonCanonicalOrder) { - const String bad = rawCatalog({rawEntLine("b", "live", u128ToHex(UInt128(1))), - rawEntLine("a", "live", u128ToHex(UInt128(2)))}); + const String bad = rawCatalog({rawEntryLine("b", "live", u128ToHex(UInt128(1))), + rawEntryLine("a", "live", u128ToHex(UInt128(2)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsCreatorPresentOnLive) { - const String bad = rawCatalog({rawEntLine("a", "live", u128ToHex(UInt128(1)), + const String bad = rawCatalog({rawEntryLine("a", "live", u128ToHex(UInt128(1)), std::make_tuple(String("srv"), uint64_t(1), uint64_t(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsCreatorAbsentOnCreating) { - const String bad = rawCatalog({rawEntLine("a", "creating", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine("a", "creating", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsZeroIncarnation) { - const String bad = rawCatalog({rawEntLine("a", "live", u128ToHex(UInt128(0)))}); + const String bad = rawCatalog({rawEntryLine("a", "live", u128ToHex(UInt128(0)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsNameOverByteBound) { const String too_long_ns(kMaxNamespaceBytes + 1, 'a'); - const String bad = rawCatalog({rawEntLine(too_long_ns, "live", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine(too_long_ns, "live", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsUnknownState) { - const String bad = rawCatalog({rawEntLine("a", "bogus", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine("a", "bogus", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } +TEST(CASRefCatalogFormat, DecodeRejectsUnknownEntryKey) +{ + const String bad = rawCatalog( + {R"({"kind":"entry","ns":"a","state":"live","life":"00000000000000000000000000000001","unknown":"x"})"}); + try + { + (void)decodeRefCatalog(bad); + FAIL() << "expected CORRUPTED_DATA"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_NE(e.message().find("unknown entry key"), String::npos) << e.message(); + } +} + TEST(CASRefCatalogFormat, DecodeRejectsEmptyNamespace) { - const String bad = rawCatalog({rawEntLine("", "live", u128ToHex(UInt128(1)))}); + const String bad = rawCatalog({rawEntryLine("", "live", u128ToHex(UInt128(1)))}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } TEST(CASRefCatalogFormat, DecodeRejectsMissingNamespaceKey) { /// No "ns" key at all -- must be refused exactly like an explicit empty one, not read as "". - const String bad = rawCatalog({R"({"k":"ent","st":"live","inc":")" + u128ToHex(UInt128(1)) + "\"}"}); + const String bad = rawCatalog({R"({"kind":"entry","state":"live","life":")" + u128ToHex(UInt128(1)) + "\"}"}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCatalog(bad); }); } @@ -596,7 +627,7 @@ TEST(CASRefCatalogFormat, RegistryRowIsControlStrictWithRawStorage) EXPECT_EQ(traits.compression, CompressionPolicy::Never); } -/// ---------- capacity admission: per-predicate boundary tests [codex r2/r3 finding 9] ---------- +/// ---------- capacity admission: per-predicate boundary tests ---------- TEST(CASRefCatalogAdmission, Predicate1AcceptsEqualityRefusesCapPlusOne) { @@ -688,7 +719,7 @@ TEST(CASRefCatalogAdmission, ReservationCoversActualWidestLegalRowsAcrossDecimal { seal.ref_lives.emplace(std::numeric_limits::max() - i, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{max, max}, .hold = RefHold{ .reason = HoldReason::UnconsumedSealCrossing, @@ -699,7 +730,7 @@ TEST(CASRefCatalogAdmission, ReservationCoversActualWidestLegalRowsAcrossDecimal } for (uint64_t shard = 0; shard < gc_shards; ++shard) { - /// Predicate 2 charges exactly `gc_shards` widest `btr` rows. This fixture is the maximum + /// Predicate 2 charges exactly `gc_shards` widest `blob_run` rows. This fixture is the maximum /// legal cardinality, not an optimistic producer convention: authoritative fold-seal /// grammar permits at most one run per shard and requires its canonical key to use seq 0. seal.blob_target_runs.push_back(RunRef{ @@ -1113,7 +1144,7 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CasFoldSeal held_parent; held_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .last_folded_ref_id = RefTxnId{1, 2}, .hold = RefHold{.offending_position = RefTxnId{1, 3}}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); @@ -1124,7 +1155,7 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CasFoldSeal mismatched_parent; mismatched_parent.ref_lives.emplace(UInt128{8}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( backend, layout, removing, mismatched_parent, 5, @@ -1133,7 +1164,7 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); CatalogEntry live = removing; @@ -1196,7 +1227,7 @@ TEST(CASRefCatalogRemoval, ExactDeletionRefusesChangedEntryAndAdmissionCannotCar CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( backend, layout, removing, ready_parent, 5, @@ -1243,7 +1274,7 @@ TEST(CASRefCatalogRemoval, FenceLossRemainsControlOutcomeWhenWinnerRemovesOrRepl CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); std::optional replacement; if (replace) @@ -1290,7 +1321,7 @@ TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesBeforeEraseCas) PutOutcome::Done); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] @@ -1322,7 +1353,7 @@ TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesAfterEraseResolut PutOutcome::Done); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); size_t authority_checks = 0; @@ -1359,7 +1390,7 @@ TEST(CASRefCatalogRemoval, CasPutExceptionPropagatesAfterMandatoryResolution) PutOutcome::Done); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 2}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); backend.armCasPutThrow(layout.refCatalogKey()); @@ -1432,12 +1463,12 @@ TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuil RefScanSummary ordinary_scan; ordinary_scan.parent_ref_lives.emplace(UInt128{1}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}}); ordinary_scan.parent_ref_lives.emplace(UInt128{3}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{3, 3}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{3, 3}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{3, 3}}}); ordinary_scan.parent_ref_lives.emplace(UInt128{4}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{4, 4}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{4, 4}}}); ordinary_scan.listed_lives = {UInt128{1}, UInt128{2}, UInt128{4}}; ordinary_scan.holds.emplace(UInt128{1}, RefHold{.offending_position = RefTxnId{1, 2}}); ordinary_scan.holds.emplace(UInt128{2}, RefHold{.offending_position = RefTxnId{2, 2}}); @@ -1449,7 +1480,7 @@ TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuil RefScanSummary rebuild_scan; rebuild_scan.parent_ref_lives.emplace(UInt128{1}, ordinary_scan.parent_ref_lives.at(UInt128{1})); rebuild_scan.parent_ref_lives.emplace(UInt128{5}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{5, 5}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{5, 5}}}); rebuild_scan.listed_lives = {UInt128{1}, UInt128{3}, UInt128{5}}; rebuild_scan.holds.emplace(UInt128{1}, RefHold{.offending_position = RefTxnId{1, 3}}); rebuild_scan.holds.emplace(UInt128{3}, RefHold{.offending_position = RefTxnId{3, 4}}); @@ -1541,7 +1572,7 @@ TEST(CASGCStuckRemoval, BoundaryAndAbsentVersusUnreadableMessagesAreExact) EXPECT_NE(absent->find("terminal has not folded"), String::npos); EXPECT_EQ(absent->find("/_log/"), String::npos) << "an absent terminal has no exact id to name"; - row.fold_state.coverage.classification = 4; + row.fold_state.coverage.classification = CoverageClass::Clamped; row.fold_state.coverage.hold = RefHold{ .reason = HoldReason::BodyUndecodable, .offending_position = RefTxnId{5, 6}}; @@ -1601,7 +1632,7 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) seal.generation = 1; seal.ref_lives.emplace(life_id, RefLifeFoldState{ .coverage = RefCoverage{ - .classification = 4, + .classification = CoverageClass::Clamped, .hold = RefHold{ .reason = HoldReason::BodyUndecodable, .offending_position = RefTxnId{5, 6}, @@ -1659,9 +1690,9 @@ TEST(CASGCRefWalkPlan, UnmatchedAdoptedParentLifeIsObservedWithoutEnteringThePla .catalog = catalog, .token = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary scan; scan.parent_ref_lives.emplace(current_life, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{2, 3}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{2, 3}}}); scan.parent_ref_lives.emplace(unmatched_life, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{9, 9}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{9, 9}}}); const uint64_t events_before = ProfileEvents::global_counters[ProfileEvents::CASGCUnmatchedAdoptedParentLives].load(); @@ -1697,7 +1728,7 @@ TEST(CASGCRefPlan, RoundInputOwnsObservationsAndSuccessorStateCannotChangePlan) RefScanSummary observations; observations.max_log_by_life.emplace(UInt128{2}, RefTxnId{2, 7}); observations.parent_ref_lives.emplace(UInt128{2}, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{2, 3}}}); + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{2, 3}}}); const RefPlan plan = tests::buildRefWalkPlanForTest(observations, cut); diff --git a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp index 1ea4b8096e4d..5954f9930f5a 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp @@ -341,13 +341,13 @@ TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsR const RootNamespace ns{"srv1/reconciled"}; /// A dead predecessor's `Creating` entry: its mount lease carries the clean-farewell sentinel - /// (`min_active == UINT64_MAX`), one of `isCreatorFenceTerminal`'s three certificates of death. + /// (`min_active_build_sequence == UINT64_MAX`), one of `isCreatorFenceTerminal`'s three certificates of death. const CreatorFence dead_creator{.server_root_id = "dead-server", .writer_epoch = 3, .fence_generation = 1}; const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(0xbeef), .creator = dead_creator}; CasRefCatalog::casAdmitEntry(*backend, layout, 1, entry); setWatermarkMinActive(*backend, layout, "dead-server", /*writer_epoch=*/3, - /*min_active=*/std::numeric_limits::max()); + /*min_active_build_sequence=*/std::numeric_limits::max()); /// The production path resumes creation itself: reconciles the stale entry onto THIS mount's own /// fence and completes it to `Live`, over the SAME incarnation the dead creator minted. diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index b864587eff7d..7582d9670355 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -276,8 +276,8 @@ TEST(CASRefCheckpoint, CommittedThroughHasCanonicalExactWireEncoding) .committed_through = RefTxnId{9, 11}, .checkpoint_snapshot_id = RefTxnId{9, 10}, .last_epoch_seal = RefTxnId{8, 12}}; - const String expected = R"({"type":"cas_ref_ckpt","v":10} -{"le":"7","cte":"9","cts":"11","cse":"9","css":"10","lse":"8","lss":"12"} + const String expected = R"({"type":"cas_ref_ckpt","v":1} +{"life_epoch":"7","committed_epoch":"9","committed_seq":"11","snapshot_epoch":"9","snapshot_seq":"10","seal_epoch":"8","seal_seq":"12"} )"; EXPECT_EQ(encodeRefCkpt(ckpt), expected); @@ -296,7 +296,7 @@ TEST(CASFormatBattery, RefCkpt) [&] { return sealObject(FormatId::RefCkpt, encodeRefCkpt(ckpt)); }, [](std::string_view s) { decodeRefCkpt(std::string(openObject(FormatId::RefCkpt, s))); }, currentFormatHeader("cas_ref_ckpt") + - "{\"le\":\"7\",\"cte\":\"9\",\"cts\":\"11\",\"cse\":\"9\",\"css\":\"10\",\"lse\":\"8\",\"lss\":\"12\"}\n"}); + "{\"life_epoch\":\"7\",\"committed_epoch\":\"9\",\"committed_seq\":\"11\",\"snapshot_epoch\":\"9\",\"snapshot_seq\":\"10\",\"seal_epoch\":\"8\",\"seal_seq\":\"12\"}\n"}); } /// `last_epoch_seal` is chain evidence, not an arbitrary lower bound. It either names the frontier @@ -323,9 +323,9 @@ TEST(CASRefCheckpoint, CodecRejectsIncoherentCommittedFrontierAndSealEpochs) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefCkpt(unsealed_non_genesis); }); String malformed = encodeRefCkpt(valid); - const size_t cte = malformed.find(R"("cte":"8")"); - ASSERT_NE(cte, String::npos); - malformed.replace(cte, String{R"("cte":"8")"}.size(), R"("cte":"10")"); + const size_t committed_epoch = malformed.find(R"("committed_epoch":"8")"); + ASSERT_NE(committed_epoch, String::npos); + malformed.replace(committed_epoch, String{R"("committed_epoch":"8")"}.size(), R"("committed_epoch":"10")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(malformed); }); } @@ -347,13 +347,30 @@ TEST(CASRefCheckpoint, RejectsAnUnknownKey) expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { decodeRefCkpt(with_critical); }); } +/// Replacing the abbreviated key is a format cut, not an alias. Treating it as an optional partial +/// pair would make an old writer's checkpoint appear to have no committed frontier. +TEST(CASRefCheckpoint, RejectsOldCommittedEpochKeyRatherThanAliasingIt) +{ + /// The values are chosen so ALIASING would be harmless: the spliced `"cte":"9"` re-assigns the + /// epoch the object already carries, leaving a valid checkpoint. A reader that honoured the old + /// spelling would therefore DECODE, and this test fails; only the strict unknown-key rejection + /// makes it throw. Values under which aliasing corrupts the object would let the invariant + /// checker throw the same code and hide the alias. + String with_old_key = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{9}, + .committed_through = RefTxnId{9, 1}, + .checkpoint_snapshot_id = RefTxnId{9, 1}, + .last_epoch_seal = std::nullopt}); + with_old_key.replace(with_old_key.rfind('}'), 1, R"(,"cte":"9"})"); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(with_old_key); }); +} + /// A duplicate key has no single meaning, so it can never be resolved by a reader's preference. TEST(CASRefCheckpoint, RejectsADuplicateKey) { const String good = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{7}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); String duplicated = good; - duplicated.replace(duplicated.rfind('}'), 1, R"(,"le":"9"})"); + duplicated.replace(duplicated.rfind('}'), 1, R"(,"life_epoch":"9"})"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(duplicated); }); } @@ -377,13 +394,13 @@ TEST(CASRefCheckpoint, RejectsTruncation) const String empty_body = good.substr(0, good.find('\n') + 1) + "{}\n"; EXPECT_EQ(decodeRefCkpt(empty_body), RefCkpt{}); - const String half_pair = good.substr(0, good.find('\n') + 1) + R"({"le":"7","cse":"1"})" + "\n"; + const String half_pair = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","snapshot_epoch":"1"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(half_pair); }); - const String other_half = good.substr(0, good.find('\n') + 1) + R"({"le":"7","lss":"2"})" + "\n"; + const String other_half = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","seal_seq":"2"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(other_half); }); - const String frontier_half = good.substr(0, good.find('\n') + 1) + R"({"le":"7","cte":"1"})" + "\n"; + const String frontier_half = good.substr(0, good.find('\n') + 1) + R"({"life_epoch":"7","committed_epoch":"1"})" + "\n"; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(frontier_half); }); } @@ -414,9 +431,9 @@ TEST(CASRefCheckpoint, RejectsInvalidFieldsOnEncodeAndOnDecode) const String header = encodeRefCkpt(RefCkpt{.life_epoch = std::optional{7}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); const String prefix = header.substr(0, header.find('\n') + 1); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(prefix + R"({"le":"0"})" + "\n"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefCkpt(prefix + R"({"life_epoch":"0"})" + "\n"); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { decodeRefCkpt(prefix + R"({"le":"7","cse":"1","css":"0"})" + "\n"); }); + [&] { decodeRefCkpt(prefix + R"({"life_epoch":"7","snapshot_epoch":"1","snapshot_seq":"0"})" + "\n"); }); } /// The registry row is part of the contract: Control/Strict decides how the decoder treats unknown @@ -1227,26 +1244,21 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) /// The BODY is checked as exact length plus a 128-bit SipHash of it -- not literally byte for byte, /// but any change that survives both is a 128-bit collision at a fixed length, which is the trade for /// keeping the assertion readable. It is a function of `{format generation, ns, id, ops, - /// chain_link}` only -- no incarnation reaches it. Generation 10 changed the shared format header; - /// the plaintext discriminator below removes only that change and pins every remaining byte to the - /// generation-9 fixture before accepting the new deterministic compressed size and hash. + /// chain_link}` only -- no incarnation reaches it. const auto got = backend->get(key); ASSERT_TRUE(got.has_value()) << "the birth chunk must be durable at its canonical key"; - String as_generation_9 = openObject(FormatId::RefLog, got->bytes); - const String generation_10_header = R"({"type":"cas_ref_log","v":10})"; - ASSERT_TRUE(as_generation_9.starts_with(generation_10_header)); - as_generation_9.replace(0, generation_10_header.size(), R"({"type":"cas_ref_log","v":9})"); - EXPECT_EQ(as_generation_9, R"({"type":"cas_ref_log","v":9} -{"ns":"test/golden@cas@","we":"1","rs":"1"} + const String plaintext = openObject(FormatId::RefLog, got->bytes); + EXPECT_EQ(plaintext, R"({"type":"cas_ref_log","v":1} +{"namespace":"test/golden@cas@","txn_epoch":"1","txn_seq":"1"} {"op":"namespace_birth"} -{"op":"owner_transition","nbk":"precommit","nrn":"gold_ref","nme":"1","nmb":"7","nmo":1} -{"op":"owner_transition","obk":"precommit","orn":"gold_ref","ome":"1","omb":"7","omo":1,"nbk":"committed","nrn":"gold_ref","nme":"1","nmb":"7","nmo":1} +{"op":"owner_transition","new_kind":"precommit","new_ref":"gold_ref","new_epoch":"1","new_build":"7","new_ord":1} +{"op":"owner_transition","old_kind":"precommit","old_ref":"gold_ref","old_epoch":"1","old_build":"7","old_ord":1,"new_kind":"committed","new_ref":"gold_ref","new_epoch":"1","new_build":"7","new_ord":1} {"n":3} -)") << "generation 10 must change only the self-describing header of this ref-log fixture"; - EXPECT_EQ(got->bytes.size(), 179u) << "the sealed ref-log body changed size"; +)") << "the sealed ref-log plaintext changed"; + EXPECT_EQ(got->bytes.size(), 206u) << "the sealed ref-log body changed size"; SipHash body_hash; body_hash.update(got->bytes.data(), got->bytes.size()); - EXPECT_EQ(getHexUIntLowercase(body_hash.get128()), "ada75a83638e933c98d731183a46b7b7") + EXPECT_EQ(getHexUIntLowercase(body_hash.get128()), "21c275ad44a6b47a4d6c389c0d71bb34") << "the sealed ref-log body changed content -- preparation must seal the same bytes it sealed " "before the extraction"; } diff --git a/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp b/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp index e50603d9258d..0ae7eac58160 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp @@ -41,7 +41,7 @@ /// so the fence would not fire on the very change it exists to catch. Only a real producer /// populates a real field. /// - TRANSACTIONS and WRITER EPOCHS enter as the DECIMAL WIDTH of the two id pairs. That is not -/// equality: `{cse=1,css=1}` and `{cse=1,css=10000}` differ by four bytes. It is `O(1)` because +/// equality: `{snapshot_epoch=1,snapshot_seq=1}` and `{snapshot_epoch=1,snapshot_seq=10000}` differ by four bytes. It is `O(1)` because /// the fields are `uint64_t` and so the width is ceilinged at twenty digits, which is a bound a /// test asserts on a constructed worst case -- `EncodedCkptSizeHasAConstantCeiling...` below. @@ -73,9 +73,10 @@ constexpr uint64_t U64_MAX = std::numeric_limits::max(); /// Constraint 15's bound, as a number: the encoded size of the WIDEST `_ckpt` this build can produce /// (all three fields present, every integer component at `UINT64_MAX`). Pinned as a literal so that -/// adding a field, or widening one, fails a test rather than quietly moving the bound. Generation 10 -/// added one byte to the shared format-version header (`9` became `10`); the scalar body is unchanged. -constexpr size_t CKPT_WORST_CASE_ENCODED_BYTES = 235; +/// adding a field, or widening one, fails a test rather than quietly moving the bound. The shared +/// format-version header is the single-digit `v:1` baseline; a future generation bump that widens it +/// moves this constant too. +constexpr size_t CKPT_WORST_CASE_ENCODED_BYTES = 296; /// The high-cardinality side of the size fence, in ONE transaction. Bounded above by the append lane's /// 5000-operation cap on a normal-class item (`publishCommittedOps` emits two ops per ref), and kept at diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index bc911e4ad6db..13437baf17a8 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -289,114 +289,6 @@ TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) EXPECT_EQ(state.getGreatestApplied(), (RefTxnId{kEpoch + 1, 1})); } -/// The format floor. A pool written before contiguous ref streams holds ref logs whose ids this build -/// would read as a corrupt (holed) chain, so opening it must fail closed at the pool metadata, naming -/// recreation as the migration -- CAS is pre-release and has no in-place migration path. -TEST(CASRefContiguousAlloc, OldPoolFormatIsRefusedNamingRecreation) -{ - PoolMeta pm; - pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; - pm.min_reader_generation = G_BUILD; - pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - - const String current = encodePoolMeta(pm); - EXPECT_NO_THROW(decodePoolMeta(current)); - - /// Rewrite the header-line generation to the last pre-contiguous one, exactly as an older build - /// would have stamped it. - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(kContiguousRefStreamsGeneration - 1); - const size_t at = current.find(from); - ASSERT_NE(at, String::npos); - String old_format = current; - old_format.replace(at, from.size(), to); - - try - { - decodePoolMeta(old_format); - FAIL() << "a pre-contiguous pool must not open"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find(fmt::format("CAS pool format {} predates generation-10 mount-attempt-identity floor", - kContiguousRefStreamsGeneration - 1)), String::npos) - << "the message must name the migration: " << e.message(); - } -} - -/// Generation 6 is a recreate-only physical-layout cut. A generation-5 pool has contiguous, -/// incarnation-qualified streams but still repeats the logical namespace in every key; accepting it -/// would silently run the generation-6 parsers over a different grammar. -TEST(CASRefContiguousAlloc, GenerationFiveNamespaceBearingPoolIsRefusedNamingRecreation) -{ - PoolMeta pm; - pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; - pm.min_reader_generation = G_BUILD; - pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - - const String current = encodePoolMeta(pm); - EXPECT_NO_THROW(decodePoolMeta(current)); - - /// Rewrite the header to the immediately preceding generation, which used - /// `cas/refs///...`. - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":" + std::to_string(kNamespaceLifeKeyedGeneration); - const size_t at = current.find(from); - ASSERT_NE(at, String::npos); - String old_format = current; - old_format.replace(at, from.size(), to); - ASSERT_EQ(kNamespaceLifeKeyedGeneration + 1, kOpaqueNamespaceLifeLayoutGeneration) - << "this test pins the immediately preceding namespace-bearing generation"; - - try - { - decodePoolMeta(old_format); - FAIL() << "a generation-5 namespace-bearing pool must not open"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find(fmt::format("CAS pool format {} predates generation-10 mount-attempt-identity floor", - kNamespaceLifeKeyedGeneration)), String::npos) - << "the message must name the migration: " << e.message(); - } -} - -/// Mutation caught: leaving the pool floor at generation 6 would admit a seal whose independent -/// name-keyed coverage and cleanup collections this build no longer has. Generation 7 is a -/// recreate-only grammar cut, so the immediately preceding generation must fail at pool open. -TEST(CASRefContiguousAlloc, GenerationSixSplitFoldSealPoolIsRefusedNamingRecreation) -{ - PoolMeta pm; - pm.pool_id = UInt128{1, 2}; - pm.blob_header_len = 256; - pm.min_reader_generation = G_BUILD; - pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - - const String current = encodePoolMeta(pm); - const String from = "\"v\":" + std::to_string(G_BUILD); - const String to = "\"v\":6"; - const size_t at = current.find(from); - ASSERT_NE(at, String::npos); - String old_format = current; - old_format.replace(at, from.size(), to); - - try - { - decodePoolMeta(old_format); - FAIL() << "a generation-6 split ref-life fold seal pool must not open"; - } - catch (const DB::Exception & e) - { - EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); - EXPECT_NE(e.message().find("CAS pool format 6 predates generation-10 mount-attempt-identity floor"), String::npos) - << "the message must name the recreate-only grammar cut: " << e.message(); - } -} - TEST(CASPoolMeta, GcShardsIsPersistedAndOverridesMismatchedReopenConfig) { InMemoryBackend backend; diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index cd9ce6c74291..c265618996ee 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -224,11 +224,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"2")"; + const String needle = R"("txn_seq":"2")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"1","!pss":"1")"); + tampered.insert(pos + needle.size(), R"(,"!prev_epoch":"1","!prev_seq":"1")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -255,8 +255,8 @@ TEST(CASRefEpochSealFormat, EncodeRejectsPrevEpochSealWithZeroRefSequence) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefLogTxn(txn); }); } -/// Decode-side splice: `prev_epoch_seal` present as only one of its two wire fields ("!pse" without -/// "!pss") -- a shape only reachable via corrupted bytes, since the encoder always writes both +/// Decode-side splice: `prev_epoch_seal` present as only one of its two wire fields ("!prev_epoch" without +/// "!prev_seq") -- a shape only reachable via corrupted bytes, since the encoder always writes both /// together. Boundary-plus-one for the additive-field decode contract (Constraint 7). TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) { @@ -267,7 +267,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) txn.ops.push_back(epochSealOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"(,"!pss":"9")"; + const String needle = R"(,"!prev_seq":"9")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; @@ -326,11 +326,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealSkippingImmediateEpochSpli txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"1")"; + const String needle = R"("txn_seq":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"3","!pss":"1")"); + tampered.insert(pos + needle.size(), R"(,"!prev_epoch":"3","!prev_seq":"1")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -346,11 +346,11 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealPointingAtSameOrFutureEpoc txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); /// valid: sequence 1, no prev_epoch_seal - const String needle = R"("rs":"1")"; + const String needle = R"("txn_seq":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; - tampered.insert(pos + needle.size(), R"(,"!pse":"5","!pss":"1")"); /// self-pointer + tampered.insert(pos + needle.size(), R"(,"!prev_epoch":"5","!prev_seq":"1")"); /// self-pointer expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } @@ -421,13 +421,13 @@ TEST(CASRefEpochSealFormat, ContextualPassesThroughNonSequenceOneAtOrBelowLifeEp /// Criticality of the prev_epoch_seal wire fields (review finding M4) /// =================================================================================== -/// `!pse`/`!pss` are `!`-prefixed CRITICAL keys: `prev_epoch_seal` is INV-2 chain evidence, and a +/// `!prev_epoch`/`!prev_seq` are `!`-prefixed CRITICAL keys: `prev_epoch_seal` is INV-2 chain evidence, and a /// build that silently dropped it would still pass the structural grammar (absent field => no check) /// while losing the chain link. Proven here by splicing in a DIFFERENT, genuinely-unrecognized -/// `!`-key (simulating a future critical field this build predates) rather than `!pse`/`!pss` +/// `!`-key (simulating a future critical field this build predates) rather than `!prev_epoch`/`!prev_seq` /// themselves, which this build DOES recognize: `JsonObjectReader::skipUnknown` rejects any /// unrecognized `!`-prefixed key with `UNKNOWN_FORMAT_VERSION` (never a silent skip), so this pins -/// the general mechanism the meta-line reader relies on to keep `!pse`/`!pss` safe against a decoder +/// the general mechanism the meta-line reader relies on to keep `!prev_epoch`/`!prev_seq` safe against a decoder /// that doesn't (yet, or anymore) understand them. TEST(CASRefEpochSealFormat, DecodeRejectsUnknownCriticalKeyInMetaLine) { @@ -437,7 +437,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsUnknownCriticalKeyInMetaLine) txn.ops.push_back(namespaceBirthOp()); const String bytes = encodeRefLogTxn(txn); - const String needle = R"("rs":"1")"; + const String needle = R"("txn_seq":"1")"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); String tampered = bytes; @@ -486,8 +486,8 @@ TEST(CASRefEpochSealFormat, FormatBatteryEpochSeal) runFormatBattery({FormatId::RefLog, [txn] { return sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); }, [ns, id](std::string_view s) { decodeRefLogTxn(openObject(FormatId::RefLog, s), ns, id); }, - "{\"type\":\"cas_ref_log\",\"v\":10}\n" - "{\"ns\":\"ns\",\"we\":\"3\",\"rs\":\"1\",\"!pse\":\"2\",\"!pss\":\"9\"}\n" + "{\"type\":\"cas_ref_log\",\"v\":1}\n" + "{\"namespace\":\"ns\",\"txn_epoch\":\"3\",\"txn_seq\":\"1\",\"!prev_epoch\":\"2\",\"!prev_seq\":\"9\"}\n" "{\"op\":\"epoch_seal\"}\n" "{\"n\":1}\n"}); } diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index ed6ed34ae599..cef428ab1d87 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -6,6 +6,8 @@ #include #include +#include + /// v3 text codec tests for `cas_ref_log` (codecs-v3 phase 3). Split out of the retired /// `gtest_cas_ref_codecs.cpp` and re-pointed at the TEXT codec: the encoder-side validation tests are /// format-agnostic (they only assert `encodeRefLogTxn` throws) and carry over verbatim; the old @@ -140,6 +142,20 @@ TEST(CASRefCodec, OrderMatchesLexicalOrderOfRender) /// RefLogTxn: round trip /// =================================================================================== +/// Closed-set pin: the five `RefOpKind` words, walked through `magic_enum::enum_values`, which is +/// what proves the renderer and the parser consult the SAME table: a table entry missing altogether is already a +/// build error at the coverage assert, but two delegates drifting onto different tables is not. +TEST(CASRefCodec, ClosedSetPinsRefOpKindWords) +{ + EXPECT_EQ(refOpKindToWireWord(RefOpKind::NamespaceBirth), "namespace_birth"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::OwnerTransition), "owner_transition"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::SetPublishedAt), "set_published_at"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::RemoveNamespace), "remove_namespace"); + EXPECT_EQ(refOpKindToWireWord(RefOpKind::EpochSeal), "epoch_seal"); + for (const auto k : magic_enum::enum_values()) + EXPECT_EQ(refOpKindFromWireWord(refOpKindToWireWord(k)), k); +} + TEST(CASRefCodec, RoundTripNamespaceBirth) { RefLogTxn txn; @@ -185,9 +201,9 @@ TEST(CASRefCodec, RoundTripSetPublishedAt) EXPECT_EQ(decoded, txn); } -/// No-tolerance decode pin (codex round-2, finding 3): the `"pl"` (payload) field was removed from the +/// No-tolerance decode pin: the `"pl"` (payload) field was removed from the /// ref-op wire in stage-1 T12. Although the retired `set_payload` op WORD is already rejected by -/// `opKindFromWord`, the generic op-record reader reads all field keys before switching on kind, so a +/// `refOpKindFromWireWord`, the generic op-record reader reads all field keys before switching on kind, so a /// `"pl"` field paired with a still-recognized op word would otherwise be `skipUnknown`'d. It is a /// removed field, not a genuinely-unknown one: decoding an op record that still carries `"pl"` must FAIL /// with `CORRUPTED_DATA` naming the removed field. @@ -204,8 +220,8 @@ TEST(CASRefCodec, DecodeRejectsRemovedPayloadFieldInOpRecord) txn.ops.push_back(op); const String bytes = encodeRefLogTxn(txn); - /// Splice the retired `"pl"` field back into the op record, just before its `"ts"` field. - const String needle = ",\"ts\":"; + /// Splice the retired `"pl"` field back into the op record, just before its `"published_ms"` field. + const String needle = ",\"published_ms\":"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); const String tampered = bytes.substr(0, pos) + R"(,"pl":"deadbeef")" + bytes.substr(pos); @@ -297,7 +313,7 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) txn.ops.push_back(op); const String bytes = encodeRefLogTxn(txn); - const String old_group = R"(,"obk":"precommit","orn":"old","ome":"1","omb":"1","omo":1)"; + const String old_group = R"(,"old_kind":"precommit","old_ref":"old","old_epoch":"1","old_build":"1","old_ord":1)"; const auto old_group_pos = bytes.find(old_group); ASSERT_NE(old_group_pos, String::npos); String old_absent = bytes; @@ -307,14 +323,14 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) EXPECT_FALSE(without_old.ops[0].old_binding.has_value()); EXPECT_TRUE(without_old.ops[0].new_binding.has_value()); - const String old_ref = R"(,"orn":"old")"; + const String old_ref = R"(,"old_ref":"old")"; const auto old_ref_pos = bytes.find(old_ref); ASSERT_NE(old_ref_pos, String::npos); String incomplete_old = bytes; incomplete_old.erase(old_ref_pos, old_ref.size()); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_old, txn.ns, txn.txn_id); }); - const String new_group = R"(,"nbk":"committed","nrn":"new","nme":"1","nmb":"1","nmo":1)"; + const String new_group = R"(,"new_kind":"committed","new_ref":"new","new_epoch":"1","new_build":"1","new_ord":1)"; const auto new_group_pos = bytes.find(new_group); ASSERT_NE(new_group_pos, String::npos); String new_absent = bytes; @@ -324,7 +340,7 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) EXPECT_TRUE(without_new.ops[0].old_binding.has_value()); EXPECT_FALSE(without_new.ops[0].new_binding.has_value()); - const String new_ref = R"(,"nrn":"new")"; + const String new_ref = R"(,"new_ref":"new")"; const auto new_ref_pos = bytes.find(new_ref); ASSERT_NE(new_ref_pos, String::npos); String incomplete_new = bytes; @@ -332,6 +348,35 @@ TEST(CASRefCodec, OwnerTransitionBindingGroupsAreAbsentOrComplete) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(incomplete_new, txn.ns, txn.txn_id); }); } +/// The anomaly diagnostic identifies an object found at a key it should not occupy by reading the +/// meta line's three identity fields. It reads them through the codec's own key constants, so this +/// test is what proves the reader did not quietly stop matching when those keys were renamed: with a +/// stale spelling the tolerant reader skips every real key and the peek answers nullopt on a +/// perfectly good ref-log. +TEST(CASRefCodec, PeekReadsTheMetaIdentityOfASealedRefLog) +{ + RefLogTxn txn; + txn.ns = "srv1/db/table@cas@"; + txn.txn_id = RefTxnId{4, 9}; + RefOp birth; + birth.kind = RefOpKind::NamespaceBirth; + txn.ops.push_back(birth); + + const auto peek = peekRefLogMeta(sealObject(FormatId::RefLog, encodeRefLogTxn(txn))); + ASSERT_TRUE(peek.has_value()) << "a well-formed ref-log must identify its own writer"; + EXPECT_EQ(peek->ns, "srv1/db/table@cas@"); + EXPECT_EQ(peek->writer_epoch, 4u); + EXPECT_EQ(peek->ref_sequence, 9u); +} + +/// The other half of its contract: it identifies a writer, it never certifies an object, so anything +/// it cannot read is `nullopt` rather than an exception escaping into the anomaly report. +TEST(CASRefCodec, PeekAnswersNulloptForBytesThatAreNotARefLog) +{ + EXPECT_FALSE(peekRefLogMeta("not a sealed cas object at all").has_value()); + EXPECT_FALSE(peekRefLogMeta(sealObject(FormatId::RefLog, "{\"type\":\"cas_ref_log\",\"v\":1}\n")).has_value()); +} + TEST(CASRefCodec, RoundTripMultipleOpsInOneTransaction) { RefLogTxn txn; @@ -833,7 +878,7 @@ TEST(CASFormatBattery, RefLog) [txn] { return sealObject(FormatId::RefLog, encodeRefLogTxn(txn)); }, [ns, id](std::string_view s) { decodeRefLogTxn(openObject(FormatId::RefLog, s), ns, id); }, currentFormatHeader("cas_ref_log") + - "{\"ns\":\"ns\",\"we\":\"1\",\"rs\":\"1\"}\n" - "{\"op\":\"set_published_at\",\"rn\":\"all_1_1_0\",\"me\":\"1\",\"mb\":\"1\",\"mo\":1,\"ts\":42}\n" + "{\"namespace\":\"ns\",\"txn_epoch\":\"1\",\"txn_seq\":\"1\"}\n" + "{\"op\":\"set_published_at\",\"ref\":\"all_1_1_0\",\"epoch\":\"1\",\"build\":\"1\",\"ord\":1,\"published_ms\":42}\n" "{\"n\":1}\n"}); } diff --git a/src/Disks/tests/gtest_cas_ref_read_contract.cpp b/src/Disks/tests/gtest_cas_ref_read_contract.cpp index a86d2530d4e8..7917d59231e7 100644 --- a/src/Disks/tests/gtest_cas_ref_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ref_read_contract.cpp @@ -77,7 +77,7 @@ void deleteCatalogLife(Backend & backend, const Layout & layout, const Namespace CasFoldSeal parent; parent.ref_lives.emplace(life.incarnation, RefLifeFoldState{ - .coverage = RefCoverage{.classification = 2, .last_folded_ref_id = RefTxnId{1, 1}}, + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); if (CasRefCatalog::deleteCompletedRemoving( backend, layout, *it, parent, 1, diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 2fea715879e2..1c532c5add83 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -65,7 +65,7 @@ TEST(CASRefSnapshotCodec, DecodeRequiresLifecycleField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String field = R"(,"lc":"live")"; + const String field = R"(,"lifecycle":"live")"; const size_t at = bytes.find(field); ASSERT_NE(at, String::npos); bytes.erase(at, field.size()); @@ -78,10 +78,10 @@ TEST(CASRefSnapshotCodec, DecodeRejectsTerminalLifecycleWord) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); - bytes.replace(at, live.size(), R"("lc":"removed")"); + bytes.replace(at, live.size(), R"("lifecycle":"removed")"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); @@ -91,7 +91,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnEpochField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); bytes.replace(at, live.size(), live + R"(,"rte":"7")"); @@ -104,7 +104,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnSequenceField) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); bytes.replace(at, live.size(), live + R"(,"rts":"9")"); @@ -117,7 +117,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnFieldPair) { const RefTableSnapshot s = makeLiveSnapshot(); String bytes = encodeRefTableSnapshot(s); - const String live = R"("lc":"live")"; + const String live = R"("lifecycle":"live")"; const size_t at = bytes.find(live); ASSERT_NE(at, String::npos); bytes.replace(at, live.size(), live + R"(,"rte":"7","rts":"9")"); @@ -142,8 +142,8 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) s.committed.push_back(c); const String bytes = encodeRefTableSnapshot(s); - /// Splice the retired `"pl"` field back into the committed record, just before its `"ts"` field. - const String needle = ",\"ts\":"; + /// Splice the retired `"pl"` field back into the committed record, just before its `"published_ms"` field. + const String needle = ",\"published_ms\":"; const auto pos = bytes.find(needle); ASSERT_NE(pos, String::npos); const String tampered = bytes.substr(0, pos) + R"(,"pl":"deadbeef")" + bytes.substr(pos); @@ -152,6 +152,30 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) [&] { decodeRefTableSnapshot(tampered, s.ns, s.snapshot_id); }); } +/// Row kinds are the owner-kind vocabulary, so an unknown kind word must fail closed at the word +/// table rather than being silently skipped as an unrecognized row -- a skipped row would lose a ref +/// from a snapshot the reader still reports as complete. +TEST(CASRefSnapshotCodec, DecodeRejectsUnknownRowKindWord) +{ + RefTableSnapshot s; + s.ns = "ns"; + s.snapshot_id = RefTxnId{1, 1}; + RefCommittedRow c; + c.ref_name = "all_1_1_0"; + c.manifest_ref = manifestRef(5, 10, 1); + c.published_at_ms = 1717000000000ULL; + s.committed.push_back(c); + + const String bytes = encodeRefTableSnapshot(s); + const String needle = "\"kind\":\"committed\""; + const auto pos = bytes.find(needle); + ASSERT_NE(pos, String::npos); + const String tampered = bytes.substr(0, pos) + "\"kind\":\"archived\"" + bytes.substr(pos + needle.size()); + + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { decodeRefTableSnapshot(tampered, s.ns, s.snapshot_id); }); +} + TEST(CASRefSnapshotCodec, RoundTripLiveEmpty) { RefTableSnapshot s; @@ -207,7 +231,7 @@ TEST(CASRefSnapshotFormat, MaximalRefSequenceRoundTripsAsADecimalString) const String text = encodeRefTableSnapshot(m); const RefTableSnapshot back = decodeRefTableSnapshot(text, m.ns, m.snapshot_id); EXPECT_EQ(back.snapshot_id.ref_sequence, std::numeric_limits::max()); - EXPECT_NE(text.find("\"rs\":\"18446744073709551615\""), String::npos); + EXPECT_NE(text.find("\"snapshot_seq\":\"18446744073709551615\""), String::npos); } /// =================================================================================== @@ -430,9 +454,9 @@ TEST(CASFormatBattery, RefSnapshot) [s] { return sealObject(FormatId::RefSnapshot, encodeRefTableSnapshot(s)); }, [ns, id](std::string_view d) { decodeRefTableSnapshot(openObject(FormatId::RefSnapshot, d), ns, id); }, currentFormatHeader("cas_ref_snap") + - "{\"ns\":\"srv1/db/table@cas@\",\"we\":\"5\",\"rs\":\"200\",\"lc\":\"live\"}\n" - "{\"k\":\"c\",\"rn\":\"all_1_1_0\",\"me\":\"5\",\"mb\":\"10\",\"mo\":1,\"ts\":1717000000000}\n" - "{\"k\":\"c\",\"rn\":\"all_2_2_0\",\"me\":\"5\",\"mb\":\"11\",\"mo\":1,\"ts\":1717000000001}\n" - "{\"k\":\"p\",\"rn\":\"all_3_3_0\",\"me\":\"5\",\"mb\":\"12\",\"mo\":1}\n" + "{\"namespace\":\"srv1/db/table@cas@\",\"snapshot_epoch\":\"5\",\"snapshot_seq\":\"200\",\"lifecycle\":\"live\"}\n" + "{\"kind\":\"committed\",\"ref\":\"all_1_1_0\",\"epoch\":\"5\",\"build\":\"10\",\"ord\":1,\"published_ms\":1717000000000}\n" + "{\"kind\":\"committed\",\"ref\":\"all_2_2_0\",\"epoch\":\"5\",\"build\":\"11\",\"ord\":1,\"published_ms\":1717000000001}\n" + "{\"kind\":\"precommit\",\"ref\":\"all_3_3_0\",\"epoch\":\"5\",\"build\":\"12\",\"ord\":1}\n" "{\"n\":3}\n"}); } diff --git a/src/Disks/tests/gtest_cas_server_root_format.cpp b/src/Disks/tests/gtest_cas_server_root_format.cpp index e302a4d81573..bf04753c9b9f 100644 --- a/src/Disks/tests/gtest_cas_server_root_format.cpp +++ b/src/Disks/tests/gtest_cas_server_root_format.cpp @@ -18,7 +18,7 @@ TEST(CASFormatBattery, Owner) OwnerObject o; o.server_uuid = hexToU128("0123456789abcdeffedcba9876543210"); const String golden = currentFormatHeader("cas_owner") + - "{\"su\":\"0123456789abcdeffedcba9876543210\"}\n"; + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\"}\n"; EXPECT_EQ(encodeOwner(o), golden); EXPECT_FALSE(decodeOwner(golden).retired_at_ms.has_value()); runFormatBattery({FormatId::Owner, @@ -34,7 +34,7 @@ TEST(CASOwnerFormat, RetiredAtRoundTrip) o.retired_at_ms = 1752537600000ULL; EXPECT_EQ(encodeOwner(o), currentFormatHeader("cas_owner") - + "{\"su\":\"0123456789abcdeffedcba9876543210\",\"rt\":1752537600000}\n"); + + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"retired_at_ms\":1752537600000}\n"); const OwnerObject back = decodeOwner(encodeOwner(o)); EXPECT_EQ(back.server_uuid, o.server_uuid); EXPECT_EQ(back.retired_at_ms, o.retired_at_ms); @@ -49,7 +49,7 @@ TEST(CASFormatBattery, ServerEpoch) runFormatBattery({FormatId::ServerEpoch, [&] { return sealObject(FormatId::ServerEpoch, encodeServerEpoch(e)); }, [](std::string_view s) { decodeServerEpoch(std::string(openObject(FormatId::ServerEpoch, s))); }, - currentFormatHeader("cas_epoch") + "{\"nwe\":\"7\"}\n"}); + currentFormatHeader("cas_epoch") + "{\"next_writer_epoch\":\"7\"}\n"}); } CAS_BATTERY_COVERS(MountLease); @@ -63,8 +63,8 @@ TEST(CASFormatBattery, MountLease) [&] { return sealObject(FormatId::MountLease, encodeMountLease(m)); }, [](std::string_view s) { decodeMountLease(std::string(openObject(FormatId::MountLease, s))); }, currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"host-1\",\"pid\":4242," - "\"sat\":1752537600000,\"seq\":\"5\",\"eat\":1752537630000,\"ma\":\"9\",\"fen\":false," + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"host-1\",\"pid\":4242," + "\"started_at_ms\":1752537600000,\"seq\":\"5\",\"expires_at_ms\":1752537630000,\"min_active_build_sequence\":\"9\",\"gc_fenced\":false," "\"write_attempt_id\":\"00112233445566778899aabbccddeeff\"}\n"}); } @@ -74,7 +74,7 @@ TEST(CASMountLeaseFormat, FarewellSentinelAndFencedSurvive) 1, 5, 2, std::numeric_limits::max(), true, hexToU128("00112233445566778899aabbccddeeff")}; const MountLease back = decodeMountLease(encodeMountLease(m)); - EXPECT_EQ(back.min_active, std::numeric_limits::max()); + EXPECT_EQ(back.min_active_build_sequence, std::numeric_limits::max()); EXPECT_TRUE(back.gc_fenced); EXPECT_EQ(back.hostname, "h"); EXPECT_EQ(back.writer_epoch, 7u); @@ -93,8 +93,8 @@ TEST(CASMountLeaseFormat, WriteAttemptIdIsRequiredAndCanonical) EXPECT_EQ(decodeMountLease(encoded).write_attempt_id, m.write_attempt_id); const String without_attempt_id = currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"\",\"pid\":0," - "\"sat\":0,\"seq\":\"0\",\"eat\":0,\"ma\":\"0\",\"fen\":false}\n"; + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"\",\"pid\":0," + "\"started_at_ms\":0,\"seq\":\"0\",\"expires_at_ms\":0,\"min_active_build_sequence\":\"0\",\"gc_fenced\":false}\n"; try { decodeMountLease(without_attempt_id); @@ -109,8 +109,8 @@ TEST(CASMountLeaseFormat, WriteAttemptIdIsRequiredAndCanonical) TEST(CASMountLeaseFormat, ZeroWriteAttemptIdIsRejected) { const String data = currentFormatHeader("cas_mount_lease") + - "{\"su\":\"0123456789abcdeffedcba9876543210\",\"we\":\"7\",\"hn\":\"\",\"pid\":0," - "\"sat\":0,\"seq\":\"0\",\"eat\":0,\"ma\":\"0\",\"fen\":false," + "{\"server_uuid\":\"0123456789abcdeffedcba9876543210\",\"writer_epoch\":\"7\",\"hostname\":\"\",\"pid\":0," + "\"started_at_ms\":0,\"seq\":\"0\",\"expires_at_ms\":0,\"min_active_build_sequence\":\"0\",\"gc_fenced\":false," "\"write_attempt_id\":\"00000000000000000000000000000000\"}\n"; try { @@ -138,11 +138,18 @@ TEST(CASMountLeaseFormat, UnknownFieldsRemainTolerated) TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) { - const String header = "{\"type\":\"cas_mount_lease\",\"v\":3}\n"; - const String fields = "\"hn\":\"host-1\",\"pid\":4242,\"sat\":1752537600000," - "\"seq\":\"5\",\"eat\":1752537630000,\"ma\":\"9\",\"fen\":false}"; - - const auto expectCorrupted = [](const String & data) + /// Each arm drops exactly ONE identity and keeps the other two, and each asserts the message that + /// names the dropped one. A body missing two of them would satisfy whichever clause runs first, so + /// a shared fixture and a shared message together would let two of the three checks be deleted + /// with this test still green. + const String header = "{\"type\":\"cas_mount_lease\",\"v\":1}\n"; + const String uuid = R"("server_uuid":"0123456789abcdeffedcba9876543210",)"; + const String epoch = R"("writer_epoch":"7",)"; + const String attempt = R"("write_attempt_id":"00112233445566778899aabbccddeeff",)"; + const String rest = "\"hostname\":\"host-1\",\"pid\":4242,\"started_at_ms\":1752537600000," + "\"seq\":\"5\",\"expires_at_ms\":1752537630000,\"min_active_build_sequence\":\"9\",\"gc_fenced\":false}"; + + const auto expectMessage = [](const String & data, std::string_view expected) { try { @@ -152,9 +159,11 @@ TEST(CASMountLeaseFormat, RejectsMissingIdentityFields) catch (const DB::Exception & e) { EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), expected); } }; - expectCorrupted(header + R"({"we":"7",)" + fields + "\n"); - expectCorrupted(header + R"({"su":"0123456789abcdeffedcba9876543210",)" + fields + "\n"); + expectMessage(header + "{" + epoch + attempt + rest + "\n", "CAS mount-lease: missing server_uuid"); + expectMessage(header + "{" + uuid + attempt + rest + "\n", "CAS mount-lease: missing writer_epoch"); + expectMessage(header + "{" + uuid + epoch + rest + "\n", "CAS mount-lease: missing or zero write_attempt_id"); } diff --git a/src/Disks/tests/gtest_cas_shutdown_context.cpp b/src/Disks/tests/gtest_cas_shutdown_context.cpp index 6ceb58b1188b..5884d8b47860 100644 --- a/src/Disks/tests/gtest_cas_shutdown_context.cpp +++ b/src/Disks/tests/gtest_cas_shutdown_context.cpp @@ -90,7 +90,7 @@ void emitTestEvent(DB::ContentAddressedMetadataStorage & storage) /// successor skip the observation window, so a phase-2 failure must leave it absent. const auto mount = backend->get(Layout(config.pool_prefix).mountKey(config.server_root_id)); const bool clean_release = mount - && decodeMountLease(mount->bytes).min_active == std::numeric_limits::max(); + && decodeMountLease(mount->bytes).min_active_build_sequence == std::numeric_limits::max(); const bool marker_must_be_absent = phase == 2; std::_Exit(marker_must_be_absent && clean_release ? 1 : 0); } diff --git a/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp b/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp index 4c0000b46236..71f936c7a3c7 100644 --- a/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp +++ b/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp @@ -59,9 +59,9 @@ struct OrphanFixture /// exercising the independent sweep-deletion premise. casAdmitRecoverableEntry(*backend, store->layout(), ns); writeManifestRaw(*backend, store->layout(), ns, orphan, {blobEntryFor("a", DB::UInt128(1))}); - /// min_active 6 > build_sequence 5: the durable watermark fact makes the prefix ELIGIBLE, which + /// min_active_build_sequence 6 > build_sequence 5: the durable watermark fact makes the prefix ELIGIBLE, which /// is the half the premise sits on top of. - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); } String orphanKey() const { return store->layout().manifestKey(ManifestId{ns, orphan}); } @@ -81,7 +81,7 @@ struct UndecodableOrphanFixture { store = openPoolForTest(backend); casAdmitRecoverableEntry(*backend, store->layout(), ns); - setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, store->layout(), kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); } String orphanKey() const { return store->layout().manifestKey(ManifestId{ns, orphan}); } @@ -179,7 +179,7 @@ TEST(CASSweepDeletionPremise, AnUnconsumedTailRemovalRetainsItsTarget) NamespaceFoldView view; RefCoverage cov; - cov.classification = 2; + cov.classification = CoverageClass::Folded; cov.last_folded_ref_id = RefTxnId{kBuildEpoch + 1, 1}; /// rule (1) satisfied view.coverage = cov; view.tail_removal_targets.insert(key); @@ -318,7 +318,7 @@ TEST(CASSweepDeletionPremise, DistinctRetainReasonsLandInDistinctCounters) .retry_count = 1, .next_retry_round = 4}; seedFoldCursorForTest(*backend, layout, ns_b, RefTxnId{kBuildEpoch + 1, 8}, hold); - setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget*/100, /*delete_budget*/10); @@ -403,9 +403,9 @@ TEST(CASSweepDeletionPremise, RecoveryWorkBudgetRetainsAndConvergesWithoutWedgin /// takes the fresh-`_ckpt` `putIfAbsent` path instead of `advanceRecoverableCkptForRawFixture`'s /// monotonic-advance-from-existing-value path (which throws on a null `committed_through`). casAdmitEntry(*backend, layout, ns); - setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active*/1000); + setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active_build_sequence*/1000); - /// Six orphan candidates, all eligible (build_sequence << min_active), none owned by any ref. + /// Six orphan candidates, all eligible (build_sequence << min_active_build_sequence), none owned by any ref. constexpr int kCandidates = 6; for (int i = 1; i <= kCandidates; ++i) writeManifestRaw(*backend, layout, ns, ref(i, 1), @@ -484,7 +484,7 @@ TEST(CASSweepDeletionPremise, NamespaceWorkBudgetCapsDistinctViewsPerPage) /// at all (`_ckpt.committed_through` unset), so absent the namespace cap BOTH would delete. seedFoldCursorForTest(*backend, layout, ns_a, RefTxnId{kBuildEpoch + 1, 1}); seedFoldCursorForTest(*backend, layout, ns_b, RefTxnId{kBuildEpoch + 1, 1}); - setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active*/6); + setWatermarkMinActive(*backend, layout, kServerRoot, kBuildEpoch, /*min_active_build_sequence*/6); GcRoundWorkBudget budget; budget.max_sweep_namespaces = 1; diff --git a/src/Disks/tests/gtest_cas_text_format.cpp b/src/Disks/tests/gtest_cas_text_format.cpp index 625271435b24..c300e522b739 100644 --- a/src/Disks/tests/gtest_cas_text_format.cpp +++ b/src/Disks/tests/gtest_cas_text_format.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include using namespace DB::Cas; @@ -140,6 +141,28 @@ TEST(CASJsonVocab, WriteAndReadBack) EXPECT_FALSE(r.nextKey(key)); } +TEST(CASJsonVocab, WordArrayFieldAndReaderRejectInvalidValues) +{ + CasJsonWriter out; + bool first = true; + const std::array words{"ch128", "sha256"}; + writeWordArrayField(out, WireKey{"algos_used"}, words, first); + closeObject(out, first); + EXPECT_EQ(std::move(out).take(), "{\"algos_used\":[\"ch128\",\"sha256\"]}"); + + const auto read = [](std::string_view text) + { + DB::ReadBufferFromMemory in(text.data(), text.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "test"); + String key; + EXPECT_TRUE(r.nextKey(key)); + return r.readStringArray(); + }; + EXPECT_EQ(read(R"({"algos_used":["ch128","sha256"]})"), (std::vector{"ch128", "sha256"})); + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { read(R"({"algos_used":"ch128"})"); }); + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { read(R"({"algos_used":["ch128",1]})"); }); +} + TEST(CASJsonVocab, FailClosedRules) { auto reader = [](std::string_view text, KeyStrictness s, auto && consume) @@ -180,12 +203,12 @@ TEST(CASJsonVocab, FailClosedRules) r.nextKey(k); }); }); /// bad hex width / junk in u64 string - expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"h":"0102"})", KeyStrictness::Tolerant, [](auto & r) + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"digest":"0102"})", KeyStrictness::Tolerant, [](auto & r) { String k; r.nextKey(k); r.readHex128(); }); }); - expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"s":"12x"})", KeyStrictness::Tolerant, [](auto & r) + expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { reader(R"({"u64_string_field":"12x"})", KeyStrictness::Tolerant, [](auto & r) { String k; r.nextKey(k); r.readU64String(); @@ -213,9 +236,9 @@ TEST(CASTextHeader, WriteExpectSniffGate) EXPECT_FALSE(sniffHeaderLine("PAR1 not a cas object").has_value()); /// wrong type -> CORRUPTED_DATA; future v -> UNKNOWN_FORMAT_VERSION - /// `v:3` is deliberate and must NOT follow a future `G_BUILD` bump: any version <= G_BUILD passes - /// the header gate, which is the point — the BODY is what has to fail here. - const String wrong = "{\"type\":\"cas_owner\",\"v\":3}\n"; + /// `v:1` is the baseline generation, so it always passes the header gate -- the type mismatch is + /// what has to fail here. + const String wrong = "{\"type\":\"cas_owner\",\"v\":1}\n"; DB::ReadBufferFromMemory in2(wrong.data(), wrong.size()); expectCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { expectHeaderLine(in2, FormatId::PoolMeta); }); const String future = fmt::format("{{\"type\":\"cas_pool_meta\",\"v\":{}}}\n", currentCompatibilityVersion() + 1); @@ -251,19 +274,16 @@ TEST(CASZstdArm, SealOpenPolicyAndCaps) { /// Always types compress regardless of size (no threshold — the .zst key must be /// constructible without knowing the body); a raw body is still readable (repair path). - /// `v:3` here is NOT the "any version <= G_BUILD passes" case the other negative bodies rely on: - /// `cas_ref_snap`'s own `changePoints` floor is generation 4, so a generation-3 ref snapshot is not - /// readable by this build in principle. It passes the header gate only because nothing consults - /// `changePoints` at decode time yet -- the gate is `v > G_BUILD` alone. Once a per-class floor is - /// wired in, this literal must move to `G_BUILD`; the test's subject is the truncated BODY, not the - /// version. - const String small = "{\"type\":\"cas_ref_snap\",\"v\":3}\n{}\n"; + /// `sealObject`/`openObject` are the storage-wrapper layer and never invoke the version gate + /// (that happens at decode, e.g. `decodeRefSnapshot`'s `expectHeaderLine`), so `v:1` here is just + /// the baseline header -- the test's subject is the compression arm, not the version. + const String small = "{\"type\":\"cas_ref_snap\",\"v\":1}\n{}\n"; const String sealed_small = sealObject(FormatId::RefSnapshot, small); ASSERT_TRUE(looksZstd(sealed_small)); EXPECT_EQ(openObject(FormatId::RefSnapshot, sealed_small), small); EXPECT_EQ(openObject(FormatId::RefSnapshot, small), small); - String big = "{\"type\":\"cas_ref_snap\",\"v\":3}\n{\"pad\":\""; + String big = "{\"type\":\"cas_ref_snap\",\"v\":1}\n{\"pad\":\""; big += String(8192, 'a'); big += "\"}\n"; const String sealed = sealObject(FormatId::RefSnapshot, big); diff --git a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp index 522b7a738e68..2f15a9d9df95 100644 --- a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp +++ b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp @@ -75,7 +75,7 @@ ManifestId publishPart2( /// (condemn -> graduate -> delete) is in flight while this is true. bool anyRetiredPending(const PoolPtr & s) { - /// Retired-in-snapshot (T4): condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a + /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); } diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index 207ad24a295f..93385f38177a 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -1,10 +1,13 @@ #include #include +#include #include #include #include #include +#include + using namespace DB::Cas; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; } @@ -44,6 +47,23 @@ TEST(CASWireVocab, EnumTablesPinTheCurrentWords) EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::XXH3_128, "t"), "xxh3"); EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::Sha256, "t"), "sha256"); EXPECT_EQ(kObjectKindWords.toWord(ObjectKind::Blob, "t"), "blob"); + EXPECT_EQ(refOwnerKindToWord(RefOwnerKind::Committed), "committed"); + EXPECT_EQ(refOwnerKindToWord(RefOwnerKind::Precommit), "precommit"); +} + +/// Every enum wire table's closed set, walked through `magic_enum::enum_values` rather than a +/// hand-copied list -- a future enumerator the encoder can construct but no table entry covers +/// would otherwise round-trip silently through the untested value. +TEST(CASWireVocab, ClosedSetsRoundTripEveryEnumeratorExhaustively) +{ + for (const auto t : magic_enum::enum_values()) + EXPECT_EQ(tokenTypeFromWord(tokenTypeToWord(t), "t"), t); + for (const auto k : magic_enum::enum_values()) + EXPECT_EQ(objectKindFromWord(objectKindToWord(k), "k"), k); + for (const auto a : magic_enum::enum_values()) + EXPECT_EQ(blobHashAlgoFromWord(blobHashAlgoName(a), "a"), a); + for (const auto k : magic_enum::enum_values()) + EXPECT_EQ(refOwnerKindFromWord(refOwnerKindToWord(k), "k"), k); } TEST(CASWireVocab, EnumWordsRoundTrip) @@ -67,7 +87,7 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) closeObject(out, first); const String rendered = std::move(out).take(); EXPECT_EQ(rendered, - R"({"tt":"etag","tv":"etag-abc\"x","ha":"ch128","h":"00112233445566778899aabbccddeeff"})"); + R"({"token_type":"etag","token":"etag-abc\"x","algo":"ch128","digest":"00112233445566778899aabbccddeeff"})"); DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); @@ -78,10 +98,10 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) TokenType tt{}; while (r.nextKey(key)) { - if (key == "tt") tt = tokenTypeFromWord(r.readString(), "t"); - else if (key == "tv") tv = r.readString(); - else if (key == "ha") ha = r.readString(); - else if (key == "h") h = r.readString(); + if (key == "token_type") tt = tokenTypeFromWord(r.readString(), "t"); + else if (key == "token") tv = r.readString(); + else if (key == "algo") ha = r.readString(); + else if (key == "digest") h = r.readString(); else r.skipUnknown(key); } EXPECT_EQ(tt, TokenType::ETag); @@ -97,13 +117,13 @@ TEST(CASWireVocab, ManifestRefBundleWritesTheOldPrefixedKeys) bool first = true; writeManifestRefFields(w, first, kOldManifestRefKeys, ManifestRef{1, 2, 3}); w.closeObject(first); - EXPECT_EQ(std::move(w).take(), R"({"ome":"1","omb":"2","omo":3})"); + EXPECT_EQ(std::move(w).take(), R"({"old_epoch":"1","old_build":"2","old_ord":3})"); } TEST(CASWireVocab, MatchAndBuildRoundTripsABlobRef) { using namespace DB::Cas; - const String rendered = R"({"ha":"ch128","h":"00112233445566778899aabbccddeeff"})"; + const String rendered = R"({"algo":"ch128","digest":"00112233445566778899aabbccddeeff"})"; DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); BlobRefFields fields; @@ -143,10 +163,10 @@ TEST(CASWireVocab, BlobRefBuildFailsClosedOnRightWidthNonHexDigest) TEST(CASWireVocab, MatchManifestRefFieldsAndBuildRefRoundTripInAnyKeyOrder) { using namespace DB::Cas; - /// Fed out of writer order (mo, me, mb) to pin key-order independence. `me`/`mb` are quoted - /// decimal strings and `mo` is a bare number -- a swapped read primitive between the two shapes + /// Fed out of writer order (ord, epoch, build) to pin key-order independence. `epoch`/`build` are quoted + /// decimal strings and `ord` is a bare number -- a swapped read primitive between the two shapes /// would fail to parse this literal. - const String rendered = R"({"mo":3,"me":"7","mb":"9"})"; + const String rendered = R"({"ord":3,"epoch":"7","build":"9"})"; DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); ManifestRefFields fields; @@ -168,10 +188,10 @@ TEST(CASWireVocab, ManifestRefFieldsBuildRefFailsClosedOnHalfAGroup) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { fields.buildRef("t", "ctx"); }); } -TEST(CASWireVocab, MatchTokenFieldsConsumesTtTvAndLeavesUnrelatedKeyUnmatched) +TEST(CASWireVocab, MatchTokenFieldsConsumesSemanticKeysAndLeavesUnrelatedKeyUnmatched) { using namespace DB::Cas; - const String rendered = R"({"tt":"etag","tv":"abc","zz":1})"; + const String rendered = R"({"token_type":"etag","token":"abc","zz":1})"; DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); TokenFields fields; @@ -190,3 +210,49 @@ TEST(CASWireVocab, MatchTokenFieldsConsumesTtTvAndLeavesUnrelatedKeyUnmatched) EXPECT_EQ(*fields.value, "abc"); EXPECT_TRUE(saw_unmatched); } + +TEST(CASWireVocab, TokenFieldsBuildsInAnyKeyOrderAndRequiresBothFields) +{ + const String rendered = R"({"token":"abc","token_type":"etag"})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + TokenFields fields; + String key; + while (r.nextKey(key)) + { + if (matchTokenFields(key, r, fields)) + continue; + r.skipUnknown(key); + } + EXPECT_EQ(fields.build("t"), (Token{"abc", TokenType::ETag})); + + TokenFields only_type; + only_type.type_word = "etag"; + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { only_type.build("t"); }); +} + +TEST(CASWireVocab, OldManifestEpochKeyDoesNotAliasTheSemanticKey) +{ + const String rendered = R"({"me":"1","build":"2","ord":3})"; + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Tolerant, "t"); + ManifestRefFields fields; + String key; + while (r.nextKey(key)) + { + if (matchManifestRefFields(key, r, kBareManifestRefKeys, fields)) + continue; + r.skipUnknown(key); + } + + try + { + fields.buildRef("RefTableSnapshot", "committed"); + FAIL() << "expected DB::Exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::CORRUPTED_DATA); + EXPECT_EQ(e.message(), "CAS RefTableSnapshot: committed manifest_ref missing epoch/build/ord"); + } +} diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index 662c0daa165e..5539d58dda80 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -354,7 +354,7 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem const auto mount = backend->get(mount_key); ASSERT_TRUE(mount.has_value()); - EXPECT_NE(decodeMountLease(mount->bytes).min_active, std::numeric_limits::max()) + EXPECT_NE(decodeMountLease(mount->bytes).min_active_build_sequence, std::numeric_limits::max()) << "a live writer-cleanup duty forbids the clean-release certificate"; uint64_t fake_boot = 0; diff --git a/src/Storages/System/StorageSystemContentAddressedMounts.cpp b/src/Storages/System/StorageSystemContentAddressedMounts.cpp index 2a1b82a01ee7..e856acad2286 100644 --- a/src/Storages/System/StorageSystemContentAddressedMounts.cpp +++ b/src/Storages/System/StorageSystemContentAddressedMounts.cpp @@ -187,7 +187,7 @@ Pipe StorageSystemContentAddressedMounts::read( col_seq->insert(m.lease.seq); assert_cast(*col_started).insertValue(static_cast(m.lease.started_at_ms)); assert_cast(*col_expires).insertValue(static_cast(m.lease.expires_at_ms)); - col_min_active->insert(m.lease.min_active); + col_min_active->insert(m.lease.min_active_build_sequence); col_fenced->insert(static_cast(m.lease.gc_fenced)); col_state->insert(m.state); diff --git a/tests/integration/test_cas_gc_sharded/test.py b/tests/integration/test_cas_gc_sharded/test.py index a7c56edf2705..fca5d64eea51 100644 --- a/tests/integration/test_cas_gc_sharded/test.py +++ b/tests/integration/test_cas_gc_sharded/test.py @@ -122,13 +122,13 @@ def get_rustfs_object(key): # `gc/state`'s wire format is a plain JSON-like text object (CasGcStateFormat.cpp), not a binary -# blob: the two fields this test needs are literally spelled `"sg":""` (snap_generation) -# and `"sa":""` (snap_attempt) in the object bytes, so a direct regex read is exact without +# blob: the two fields this test needs are literally spelled `"snap_generation":""` +# and `"snap_attempt":""` in the object bytes, so a direct regex read is exact without # needing the C++ decoder. This mirrors the production reader that resolves "the adopted seal" # (Gc/CasOrphanManifestSweep.cpp): read gc/state, take (snap_generation, snap_attempt), then look up # that exact fold seal -- the only two-hop lookup that names one authoritative adopted pair. -_SNAP_GENERATION_RE = re.compile(r'"sg":"(\d+)"') -_SNAP_ATTEMPT_RE = re.compile(r'"sa":"(\d+)"') +_SNAP_GENERATION_RE = re.compile(r'"snap_generation":"(\d+)"') +_SNAP_ATTEMPT_RE = re.compile(r'"snap_attempt":"(\d+)"') def read_adopted_generation_and_attempt(): @@ -144,7 +144,13 @@ def read_adopted_generation_and_attempt(): sg_match = _SNAP_GENERATION_RE.search(text) sa_match = _SNAP_ATTEMPT_RE.search(text) if not sg_match or not sa_match: - return None + # A `gc/state` that exists but does not spell both fields is a wire-format change this + # reader has not followed, NOT "nothing adopted yet" -- say so instead of returning the + # absent-sentinel, which would poll to a timeout and blame the server. + raise AssertionError( + "gc/state exists but neither snap_generation nor snap_attempt could be read from it; " + "the object's key spelling changed and this regex reader is stale: " + text[:400] + ) generation = int(sg_match.group(1)) if generation == 0: return None diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 74294cf95fa5..7d63be5769db 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -784,7 +784,7 @@ def handle_control(path, method, query): return _no_such_key(meta_key) text = entry["body"].decode("utf-8", "strict") rewritten, replacements = re.subn( - r'"st":"clean","cr":"[0-9]+"', '"st":"condemned","cr":"1"', text, count=1 + r'"state":"clean","condemn_round":"[0-9]+"', '"state":"condemned","condemn_round":"1"', text, count=1 ) if replacements != 1: return _bad_request("blob metadata is not Clean: " + meta_key) diff --git a/tests/integration/test_cas_gcs/test.py b/tests/integration/test_cas_gcs/test.py index 285da574ac17..7df52e4dbbd3 100644 --- a/tests/integration/test_cas_gcs/test.py +++ b/tests/integration/test_cas_gcs/test.py @@ -469,7 +469,7 @@ def test_blob_publication_request_budget_and_default_mode(disk): for r in meta if r["method"] == "PUT" and r["headers"].get("x-goog-if-generation-match") == "0" - and '"st":"clean"' in r["request_body"] + and '"state":"clean"' in r["request_body"] ] assert len(creates) == 1, (key, meta) @@ -524,7 +524,7 @@ def test_blob_publication_request_budget_and_default_mode(disk): for r in _meta_requests(retry, target) if r["method"] == "PUT" and r["headers"].get("x-goog-if-generation-match", "0") != "0" - and '"st":"clean"' in r["request_body"] + and '"state":"clean"' in r["request_body"] ] assert len(clean_cas) == 1, clean_cas diff --git a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh index ef5e2377cc21..1cc521a210ca 100755 --- a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh +++ b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh @@ -4,17 +4,17 @@ # own unique local-object-storage pool and a per-run CAS disk name, so unlike 04290_cas_no_leftovers # it does not need no-parallel. -# FINDING #2 regression test: `DROP TABLE ... SYNC` on a content-addressed MergeTree used to leave the +# Regression test: `DROP TABLE ... SYNC` on a content-addressed MergeTree used to leave the # table's CAS ref-catalog row `live` forever whenever `DirShape::TableDir`'s `existsDirectory` observed # zero committed refs -- an empty table, or one whose last part was just removed. `dropAllData`'s own # `existsDirectory` precheck skipped `removeRecursive`/`dropNamespace` entirely in that shape, so the # SQL-level drop completed normally while the CAS catalog row leaked, one per create/drop cycle. # # The primary oracle is the pool's OWN plain-text `cas/ref_catalog` object, read directly off disk: the -# exact `st` (lifecycle) field recorded for the table's logical namespace. `SYSTEM CAS FSCK`'s +# exact `state` (lifecycle) field recorded for the table's logical namespace. `SYSTEM CAS FSCK`'s # unreachable/dangling counts are a secondary check only -- fsck correctly regards a `live` leak as # CONSISTENT (nothing is unreachable; the row simply never dies), so it cannot detect this defect on its -# own; `04290_cas_no_leftovers.sh`'s fsck-only oracle is exactly why FINDING #2 shipped unnoticed. +# own; `04290_cas_no_leftovers.sh`'s fsck-only oracle is exactly why that leak shipped unnoticed. CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # shellcheck source=../shell_config.sh @@ -34,7 +34,7 @@ catalog_line() { grep -F "\"ns\":\"$1\"" "${CATALOG_FILE}" 2>/dev/null || true } -# The `st` (lifecycle) word recorded for namespace $1: "live"/"creating"/"removing", or "absent" if the +# The `state` (lifecycle) word recorded for namespace $1: "live"/"creating"/"removing", or "absent" if the # namespace has no catalog row (matches `04290`'s field-by-name discipline: never assume a position). catalog_state() { local line @@ -43,7 +43,16 @@ catalog_state() { echo "absent" return fi - echo "${line}" | grep -o '"st":"[a-z]*"' | head -1 | sed -E 's/"st":"([a-z]*)"/\1/' + # The pipeline's exit status is `sed`'s, which is 0 even on empty input, so emptiness is the only + # usable signal that the row exists but its state field could not be read -- a stale key spelling + # here must be loud, never mistaken for "absent". + local state + state=$(echo "${line}" | grep -o '"state":"[a-z]*"' | head -1 | sed -E 's/"state":"([a-z]*)"/\1/') + if [ -z "${state}" ]; then + echo "catalog row for namespace $1 exists but has no readable state field" >&2 + return 1 + fi + echo "${state}" } # ClickHouse's own store// fanout with the CAS archive boundary marker, exactly as @@ -91,7 +100,7 @@ echo "empty_table_has_no_ref_stream_before_drop $([ "${STREAM_HITS_BEFORE}" -eq $CLICKHOUSE_CLIENT --query "DROP TABLE t_dropns_empty SYNC" -# The current branch fails here by leaving st:"live"; the fix must show "removing" (a terminal stream +# The current branch fails here by leaving state:"live"; the fix must show "removing" (a terminal stream # record now exists but the catalog row itself is not deleted until GC folds and reclaims it). echo "empty_table_state_after_sync_drop $(catalog_state "${EMPTY_NS}")" STREAM_HITS_AFTER=$(find "${POOL_DIR}/ca/cas/ns/stream" -type f 2>/dev/null | wc -l) @@ -145,7 +154,10 @@ for i in 1 2 3; do CYCLE_NS_LIST+=("${CYCLE_NS}") $CLICKHOUSE_CLIENT --query "DROP TABLE t_dropns_cycle SYNC" - if [ "$(catalog_state "${CYCLE_NS}")" = "live" ]; then + # Read the state into a variable first: `$(...)` inside a test swallows a non-zero return, so an + # unreadable row would count as "not live" and this counter would pass while the reader is broken. + CYCLE_STATE=$(catalog_state "${CYCLE_NS}") || exit 1 + if [ "${CYCLE_STATE}" = "live" ]; then CYCLE_LEAK_COUNT=$((CYCLE_LEAK_COUNT + 1)) fi done From 23e2d8bc8b52db09feede4cf701e7c0ac4f0071e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:03:25 +0200 Subject: [PATCH 05/81] =?UTF-8?q?cas:=20wire-keys=20phase=203=20=E2=80=94?= =?UTF-8?q?=20proof,=20review=20polish,=20and=20the=20ref-protocol=20bench?= =?UTF-8?q?mark?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Small correctness and review follow-ups to the wire-key cut: the part manifest names its namespace field the way every other object does, the algorithm set is read from the proven `EnumWireTable` instead of two independent hand-kept lists, the GC lease and heartbeat keep their separate owner spellings (documented, not merged), and the wire-format word writer gets the contract it was always assumed to have. Extends the `benchmark_cas_ref_protocol` harness to cover every format and direction the wire-keys design measures, plus a review-round fix to that harness. Also fixes `c++expr`: the generated work function needs internal linkage, without which ClickHouse-mode compilation did not work. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- .../Formats/CasBlobEnvelopeFormat.cpp | 2 + .../Formats/CasBlobEnvelopeFormat.h | 9 + .../Formats/CasFoldSealFormat.cpp | 10 +- .../Formats/CasGcOutcomesFormat.cpp | 4 +- .../ContentAddressed/Formats/CasLayout.cpp | 15 +- .../Formats/CasPartManifestFormat.cpp | 4 +- .../Formats/CasPoolMetaFormat.h | 2 +- .../Formats/CasRecordStreamFormat.cpp | 17 +- .../Formats/CasRefCatalogFormat.h | 4 +- .../Formats/CasRefLogFormat.cpp | 2 +- .../Formats/CasRefSnapshotFormat.cpp | 2 +- .../Formats/CasTextFormat.cpp | 15 +- .../ContentAddressed/Formats/CasTextFormat.h | 12 +- .../ContentAddressed/Formats/README.md | 2 +- .../ContentAddressed/Gc/CasGc.cpp | 2 +- .../ContentAddressed/Pool/CasPool.cpp | 7 +- .../ContentAddressed/Pool/CasPoolMeta.cpp | 2 +- .../benchmarks/benchmark_cas_ref_protocol.cpp | 528 +++++++++++++++++- src/Disks/tests/cas_test_helpers.h | 9 +- src/Disks/tests/gtest_ca_wiring.cpp | 4 +- .../tests/gtest_cas_backend_generation.cpp | 2 +- src/Disks/tests/gtest_cas_blob_digest.cpp | 4 +- .../tests/gtest_cas_blob_envelope_format.cpp | 35 ++ src/Disks/tests/gtest_cas_encoding_pins.cpp | 2 +- .../tests/gtest_cas_fence_generation.cpp | 2 +- .../tests/gtest_cas_gc_frontier_gate.cpp | 9 +- src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 7 +- src/Disks/tests/gtest_cas_gc_round.cpp | 4 +- src/Disks/tests/gtest_cas_gc_round_defer.cpp | 8 +- ...est_cas_namespace_file_request_profile.cpp | 4 +- .../tests/gtest_cas_namespace_life_id.cpp | 15 +- src/Disks/tests/gtest_cas_parallel_commit.cpp | 2 +- .../tests/gtest_cas_part_manifest_format.cpp | 8 +- src/Disks/tests/gtest_cas_pluggable_hash.cpp | 16 +- src/Disks/tests/gtest_cas_ref_catalog.cpp | 9 +- .../tests/gtest_cas_ref_chunked_flush.cpp | 5 +- .../tests/gtest_cas_ref_contiguous_alloc.cpp | 7 +- .../tests/gtest_cas_ref_epoch_seal_format.cpp | 16 +- .../tests/gtest_cas_ref_install_safety.cpp | 6 +- src/Disks/tests/gtest_cas_ref_log_format.cpp | 2 +- .../tests/gtest_cas_ref_snapshot_format.cpp | 3 +- ...test_cas_ref_snapshot_publish_ordering.cpp | 2 +- .../gtest_cas_ref_wedge_every_attempt.cpp | 4 +- src/Disks/tests/gtest_cas_ref_writer.cpp | 5 +- utils/c++expr | 4 +- 45 files changed, 708 insertions(+), 125 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index 129714d6b9e9..ff9fda182b26 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -153,6 +153,8 @@ void writeEnvelopeRefField(String & json, size_t budget, std::string_view raw_re } +const size_t mandatory_descriptor_worst_case = kMandatoryDescriptorWorstCase; + std::string_view provenanceOpToWireWord(ProvenanceOp op) { return kProvenanceOpWords.toWord(op, "CAS blob envelope"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index 0aa26e22e182..e1a9a228433d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -33,6 +34,14 @@ enum class ProvenanceOp : uint8_t }; /// Returns the persisted wire word for a validated provenance operation. +/// The largest descriptor `encodeEnvelopeHeader` can ever produce before the diagnostic `ref` gets any +/// budget: every mandatory field at its type maximum, the longest provenance word, the `ref` framing +/// with empty quotes, the closing brace and the trailing newline. A `static_assert` beside its +/// definition proves it fits under `kMinBlobHeaderLen`; this declaration exists so the boundary test +/// can confirm the SAME number against bytes the encoder actually produced, which is the half a +/// compile-time proof cannot do — an understated formula satisfies the assert quite happily. +extern const size_t mandatory_descriptor_worst_case; + std::string_view provenanceOpToWireWord(ProvenanceOp op); /// Its fail-closed inverse: an unknown word is `CORRUPTED_DATA`. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index 129c4b5c3493..77c3d1dcece4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -104,7 +104,7 @@ void insertRecordOnce(Map & map, const Key & key, Value && value, std::string_vi void writeRun(CasJsonWriter & out, std::string_view kind, const RunRef & r) { bool first = true; - writeStringField(out, FoldSealWire::kind, kind, first); + writeWordField(out, FoldSealWire::kind, kind, first); writeStringField(out, FoldSealWire::run_key, r.key, first); writeHex128Field(out, FoldSealWire::checksum, r.checksum, first); writeNumberField(out, FoldSealWire::shard, r.shard, first); @@ -311,14 +311,14 @@ String encodeFoldSeal(const CasFoldSeal & seal) life_state.cleanup_evidence->remove_txn_id.ref_sequence); bool first = true; - writeStringField(out, FoldSealWire::kind, kRefLifeTag, first); + writeWordField(out, FoldSealWire::kind, kRefLifeTag, first); writeHex128Field(out, FoldSealWire::life, life_id, first); - writeStringField(out, FoldSealWire::classification, classification_word, first); + writeWordField(out, FoldSealWire::classification, classification_word, first); writeU64StringField(out, FoldSealWire::fold_epoch, cov.last_folded_ref_id.writer_epoch, first); writeU64StringField(out, FoldSealWire::fold_seq, cov.last_folded_ref_id.ref_sequence, first); if (cov.hold) { - writeStringField(out, FoldSealWire::hold_reason, holdReasonToWord(cov.hold->reason), first); + writeWordField(out, FoldSealWire::hold_reason, holdReasonToWord(cov.hold->reason), first); writeU64StringField(out, FoldSealWire::hold_epoch, cov.hold->offending_position.writer_epoch, first); writeU64StringField(out, FoldSealWire::hold_seq, cov.hold->offending_position.ref_sequence, first); writeNumberField(out, FoldSealWire::retries, cov.hold->retry_count, first); @@ -349,7 +349,7 @@ String encodeFoldSeal(const CasFoldSeal & seal) for (const auto & [shard, s] : seal.condemned_summary) { bool first = true; - writeStringField(out, FoldSealWire::kind, kCondemnedTag, first); + writeWordField(out, FoldSealWire::kind, kCondemnedTag, first); writeNumberField(out, FoldSealWire::shard, shard, first); writeNumberField(out, FoldSealWire::condemned_total, s.condemned_total, first); writeNumberField(out, FoldSealWire::pending_total, s.pending_total, first); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index 1b19256d6d4b..9ae1ffd6282d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -53,10 +53,10 @@ String encodeOutcomeLog(const OutcomeLog & log) for (const OutcomeEntry & e : log.entries) { bool first = true; - writeStringField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); + writeWordField(out, GcOutcomesWire::kind, objectKindToWord(e.kind), first); writeBlobRefFields(out, first, e.ref); /// algo + digest writeTokenFields(out, first, e.token); /// token_type + token - writeStringField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); + writeWordField(out, GcOutcomesWire::outcome, outcomeKindToWireWord(e.outcome), first); closeObject(out, first); writeChar('\n', out); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp index 9466a8af735e..d3c92e767c12 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -67,14 +68,16 @@ std::optional Layout::parseBlobKey(std::string_view key) const if (shard.size() != 2 || hex.size() < 2 || shard != hex.substr(0, 2)) return std::nullopt; /// malformed shard/hex shape -- not ours - /// `` -> `BlobHashAlgo`: the small enum-value set makes a linear scan against - /// `blobHashAlgoName` (the ONE name authority) cheaper and safer than a second name table that - /// could drift from it. + /// `` -> `BlobHashAlgo` through the wire table itself, whose coverage is proven against + /// the enum at compile time. A hand-written candidate list here would be a second enumeration that + /// a new algorithm could silently outgrow: the parser would reject a segment the writer emits. + /// This path answers "is this key ours?", so an unknown segment is `nullopt` -- debris, not + /// corruption -- which is why it scans rather than calling the throwing `fromWord`. std::optional algo; - for (BlobHashAlgo candidate : {BlobHashAlgo::CityHash128, BlobHashAlgo::XXH3_128, BlobHashAlgo::Sha256}) - if (algo_name == blobHashAlgoName(candidate)) + for (const auto & entry : kBlobHashAlgoWords.entries) + if (algo_name == entry.word) { - algo = candidate; + algo = entry.value; break; } if (!algo) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index e8ff121dfee7..ac70c2c00b23 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -25,7 +25,7 @@ namespace namespace PartManifestWire { - constexpr WireKey ns{"root_namespace"}; + constexpr WireKey ns{"namespace"}; constexpr WireKey payload_digest{"payload_digest"}; constexpr WireKey path{"path"}; constexpr WireKey place{"place"}; @@ -161,7 +161,7 @@ PartManifest decodePartManifest(std::string_view data) else r.skipUnknown(key); } if (!ns) - throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing root_namespace"); + throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing namespace"); if (!pd) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: descriptor missing payload_digest"); m.ref = fields.buildRef("PartManifest", "descriptor"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h index e25e767fb5e3..f17cb5b87fb1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h @@ -38,7 +38,7 @@ struct PoolMeta /// because changing it would move the blob payload offset for existing objects; a new hash algorithm /// is rejected unless `allow_new` is set, and concurrent admission is retried from fresh metadata. /// - /// `allow_mint` (spec §2 [C4][D2]) gates the create-if-absent path: minting a fresh `_pool_meta` is a + /// `allow_mint` gates the create-if-absent path: minting a fresh `_pool_meta` is a /// consequential write that establishes a brand-new pool identity, so it is permitted ONLY on the /// writable startup path that has just passed the zero-write residual proof (`Pool::open`). Every /// non-bootstrap caller — a read-only/observe open, `openForDecommission` — passes `false`; an absent diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index 857e428538f6..00d20988ac03 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -54,16 +54,17 @@ int hexNibble(char c) return -1; } +/// The run `ref` carries the algorithm as a raw leading byte, so this is the byte-side counterpart of +/// the word table -- and it walks that same table rather than listing the enumerators again. A second +/// list is how the writer and the reader come to disagree about which algorithms exist: `renderB` +/// writes whatever the enum holds, and a hand-written switch here would reject exactly what a new +/// enumerator adds. BlobHashAlgo algoFromByte(uint8_t b, std::string_view what) { - switch (b) - { - case static_cast(BlobHashAlgo::CityHash128): return BlobHashAlgo::CityHash128; - case static_cast(BlobHashAlgo::XXH3_128): return BlobHashAlgo::XXH3_128; - case static_cast(BlobHashAlgo::Sha256): return BlobHashAlgo::Sha256; - default: - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown algo byte {} in record key", what, b); - } + for (const auto & entry : kBlobHashAlgoWords.entries) + if (static_cast(entry.value) == b) + return entry.value; + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown algo byte {} in record key", what, b); } /// `ref` = the algo byte as two lowercase hex chars, then the digest hex at the algo's width. The diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h index 9515b00b425f..0e93845731f4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.h @@ -103,8 +103,8 @@ struct RefCatalog /// instead) -- but deliberately does NOT enforce the whole-object cap itself: that predicate must /// name the namespace under admission, which only a caller of `checkCatalogAdmission` knows. /// -/// These bytes go to and come from the backend DIRECTLY, exactly like `cas_ref_ckpt`: the Pool-side -/// `CasRefCatalog::read`/`casUpdateImpl` (`Pool/CasRefCatalog.cpp`) bypass `sealObject`/`openObject`, +/// These bytes go to and come from the backend DIRECTLY: the catalog read and update paths bypass +/// `sealObject`/`openObject`, /// which are the identity under this class's `CompressionPolicy::Never` and would add nothing. A /// policy flip to `Always` therefore breaks this silently -- and is caught, because `storedSuffix` /// would stop being empty and the registry test asserting `storedSuffix(FormatId::RefCatalog) == ""` diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index 4c6dc9fdf053..8d60260c8b53 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -126,7 +126,7 @@ struct BindingFields /// `!`-prefixed: `prev_epoch_seal` is INV-2 chain evidence, not cosmetic metadata -- a decoder that /// doesn't understand it must refuse the object rather than silently drop the chain link while /// otherwise passing the structural grammar (`JsonObjectReader::skipUnknown` rejects any unrecognized -/// `!`-key with `UNKNOWN_FORMAT_VERSION`, tolerant or not; see task-1 review finding M4). +/// `!`-key with `UNKNOWN_FORMAT_VERSION`, tolerant or not). void writeLogMeta(CasJsonWriter & out, const String & ns, const RefTxnId & txn_id, const std::optional & prev_epoch_seal) { bool first = true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index 78caf9c5632c..04ecef15cd53 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -108,7 +108,7 @@ void writeSnapshotMeta(CasJsonWriter & out, const RefTableSnapshot & snapshot) /// A snapshot object exists only for a live namespace -- `RefLifecycle::Removed` has no snapshot /// representation -- so the wire carries exactly one lifecycle word. The reader keeps the /// fail-closed half: any other word, or none, is rejected there. - writeStringField(out, RefSnapWire::lifecycle, kLiveLifecycleWord, first); + writeWordField(out, RefSnapWire::lifecycle, kLiveLifecycleWord, first); closeObject(out, first); writeChar('\n', out); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index 8438ac786478..d2f25aab615f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -109,6 +109,19 @@ void CasJsonWriter::stringValue(std::string_view s) appendChar('"'); } +void CasJsonWriter::wordValue(std::string_view word) +{ + /// The contract, checked where it is cheap to check: a vocabulary word carries no byte the JSON + /// escaper would rewrite. Violating it would emit malformed JSON rather than a mis-escaped + /// string, so this is a programming error and belongs in the debug build, not a runtime branch on + /// the encode hot path. + chassert(std::none_of(word.begin(), word.end(), + [](char c) { return isSpecialJsonByte(static_cast(c)); })); + appendChar('"'); + buf.append(word.data(), word.size()); + appendChar('"'); +} + void CasJsonWriter::wordArray(std::span words) { appendChar('['); @@ -118,7 +131,7 @@ void CasJsonWriter::wordArray(std::span words) if (!first) appendChar(','); first = false; - stringValue(word); + wordValue(word); } appendChar(']'); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index 48f473c33d65..bd353f864d17 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -63,6 +63,13 @@ class CasJsonWriter /// Quoted JSON string with full escaping (bulk-run scan). Defined in CasTextFormat.cpp. void stringValue(std::string_view s); + /// A value from a wire VOCABULARY -- an enum table's word or a record tag. Those are drawn from + /// `[a-z0-9_]` by construction, so this writes the bytes as they are instead of running them + /// through the escaper's byte scan and state machine. Output is identical to `stringValue` for + /// every input the contract admits; a caller that passes an arbitrary string is the bug this + /// asserts against, and `writeWordField` is the only intended way in. + void wordValue(std::string_view word); + /// JSON array of canonical word strings, emitted without intermediate storage. void wordArray(std::span words); @@ -163,10 +170,13 @@ inline void writeKey(CasJsonWriter & out, WireKey key, bool & first) writeKey(out, key.text, first); } +/// For a value that comes from a wire vocabulary (an enum table's word, a record tag). It is NOT +/// interchangeable with `writeStringField`: this one promises its value needs no JSON escaping and +/// skips the escaper accordingly, which is why an open string must never be routed through it. inline void writeWordField(CasJsonWriter & out, WireKey key, std::string_view word, bool & first) { writeKey(out, key, first); - writeStringValue(out, word); + out.wordValue(word); } inline void writeWordArrayField(CasJsonWriter & out, WireKey key, std::span words, bool & first) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index ea7a73ab41bf..52cd954377fd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -24,7 +24,7 @@ trailer, followed by a banner-framed raw payload zone for inline file bytes. | `cas/ns/state//_ckpt` | mutable life checkpoint (`life_epoch`, `committed_epoch`/`committed_seq`, `snapshot_epoch`/`snapshot_seq`, `seal_epoch`/`seal_seq`) | `CasRefCkptFormat` | writer/GC fold | | `cas/ns/state//_files/…​` | namespace-owned raw files | — | upper layers | | `cas/ref_catalog` | namespace lifecycle catalog (`kind:"entry"`, `ns`, `state`, `life`, `remove_round`, `creator`, `creator_epoch`, `creator_fence`) | `CasRefCatalogFormat` | namespace admission/removal | -| `cas/manifests//-/.zst` | part manifest (`root_namespace`, `payload_digest`; entry `path`, `place`, `size`) | `CasPartManifestFormat` | part build | +| `cas/manifests//-/.zst` | part manifest (`namespace`, `payload_digest`; entry `path`, `place`, `size`) | `CasPartManifestFormat` | part build | | blob keys (`CasLayout::blobKey`) | blob envelope (`type`, `v`, `tag`, `build`, `time_ms`, `creator`, `op`, `chver`, `ref`) + payload | `CasBlobEnvelopeFormat` | uploads | | blob-meta keys (`CasLayout::blobMetaKey`) | freshness sidecar (`state`, `condemn_round`, `size`) | `CasBlobMetaFormat` | dedup/GC | | `gc/state`, `gc/hb` | GC state (`round`, `gc_shards`, `snap_generation`, `snap_pruned_through`, `snap_attempt`, `manifest_sweep_cursor`, `lease_owner`, `lease_seq`) / heartbeat (`owner`, `hb_seq`) | `CasGcStateFormat` | GC | diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 46db0402ed7a..baaf7b86fdc8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -2674,7 +2674,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// It counts the CUT ARITHMETICALLY, not by listed ids. Under arithmetic intake a listed-id count is /// not even the right question: a hint hole means a round legitimately applies records the listing /// never mentioned, so the old recomputation would report fewer logs than folded and fail every - /// healthy round on a lying store -- it would have made this task's own fix unshippable. + /// healthy round on a lying store. /// /// BE HONEST ABOUT WHAT IS LEFT. The old formula could disagree with reality because it was derived /// from a different source (the listing) than the counter. This one is derived from the runs the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index ed6a957cbde6..f5563abf4306 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -342,8 +342,9 @@ String Pool::lifecycleReasonDetail(PoolLifecycle lc) const void Pool::throwIfLifecycleTerminal() const { /// The typed error carries the sub-state in its message so a wrong diagnosis is impossible from the - /// first error line (spec §1 [D5]). `Live`/`TransientNotLive` proceed here — the transient class is - /// still gated only by the write fence in this task (the full six-class gate is Task 8). + /// first error line. `Live`/`TransientNotLive` proceed here — the transient class is + /// still gated only by the write fence; the destructive gate additionally requires the other + /// lifecycle proofs. const PoolLifecycle lc = mount_runtime.lifecycle(); if (lc == PoolLifecycle::Live || lc == PoolLifecycle::TransientNotLive) return; @@ -1627,7 +1628,7 @@ void Pool::reportImpossibleInterference(const String & key, const String & reaso /// requests -- never the caller's thread, and never blocking this call's own return. try { - /// The lease owns the pool reference for this task's lifetime; capturing one here as well would + /// The lease owns the pool reference until it releases it; capturing one here as well would /// put it outside the lease's release ordering. const bool dispatched = tryDispatchDetached([this, key](DetachedStopToken token) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp index 34aaeaa37308..8b1e4bd9ba02 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp @@ -132,7 +132,7 @@ PoolMeta PoolMeta::createOrValidate( /// build at all), so the reader-generation floor is stamped at THIS build's `G_BUILD` at /// creation, not left at 0. /// - /// BOOTSTRAP GATE (spec §2 [C4][D2]): minting is permitted ONLY on the verified bootstrap path. A + /// BOOTSTRAP GATE: minting is permitted ONLY on the verified bootstrap path. A /// non-bootstrap caller (a read-only/observe open, `openForDecommission`) passes `allow_mint=false` /// and fails closed here — never minting a fresh identity outside that path (an observe scan that /// minted would poison the next writable mount's residual check). The residual EMPTINESS proof itself diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp index f23f4dc06bae..41365299eaf4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp @@ -1,15 +1,38 @@ #include #include +#include +#include +#include #include +#include + #include +#include +#include #include #include +#include +#include +#include +#include +#include #include +#include +#include +#include +#include #include #include +namespace DB::ErrorCodes +{ + extern const int CORRUPTED_DATA; + extern const int LIMIT_EXCEEDED; + extern const int LOGICAL_ERROR; +} + /// Pure measurement, no pass/fail assertions -- see the cas-gc-rebuild BACKLOG.md entries /// "OPTIMIZATION OPPORTUNITY -- ref-ledger JSON encoding writes byte-by-byte" and the (now /// RESOLVED) "admits() re-encodes the WHOLE ref table once per state-growing op" entry for the @@ -38,6 +61,16 @@ /// ladder, rung 2 was NOT attempted either (it trades readability and needs a human decision); /// reported as DONE_WITH_CONCERNS. CasEncodingPins.* stayed byte-identical (green) throughout. /// +/// NOTE (2026-08, wire-key-rename campaign): the "Phase B baselines" table immediately below measures +/// a DIFFERENT investigation (the `RefTableState` encapsulation refactor) and predates the five-format, +/// both-directions wire-key-cut design entirely. It is NOT the "before" side for that campaign's +/// measurement, and a later reader must not diff against it for that purpose. The actual before side +/// is the pre-cut worktree pinned at commit `65ec8688cdb`; the recorded patch that builds this file +/// there lives under `docs/superpowers/cas/bench-wire-keys-phase3/`. The table is kept exactly as +/// written because it is real history for the investigation it belongs to, not because it answers this +/// one -- see the "Wire-key-cut instrument" section further down for the five new formats this +/// campaign added. +/// /// Phase B baselines, 2026-07-21, pre-encapsulation (this binary; `--benchmark_repetitions=3 /// --benchmark_report_aggregates_only=true`; medians reported). Recorded ahead of the /// `RefTableState` encapsulation refactor so later phases can re-run this exact suite unchanged and @@ -118,6 +151,15 @@ RefLogTxn makeSamplePromoteTxn() /// A synthetic snapshot of `n` committed rows plus one pending precommit ready to promote. /// Built as a RefTableSnapshot and materialized via the public `replay` entry point, so this /// helper keeps compiling unchanged when RefTableState's fields become private (Phase A). +/// +/// Committed-row field widths (load-bearing for the `cas_ref_snap` wire-key-cut benchmarks and byte +/// oracle, which measure the RELATIVE cost of a key rename against the encoded VALUE bytes as the +/// denominator): `published_at_ms` is a real 13-digit epoch-ms rather than the default `0`, and +/// `manifest_ref`'s `writer_epoch`/`build_sequence` are multi-digit (a pool old enough to have +/// restarted its writer decades of times, and a build counter past its 89811th commit -- the same +/// order of magnitude as the real ref-ledger key at the top of this file, `kSafeKeyLikeString`, and +/// `makeSamplePromoteTxn`'s ref name). A minimal `0`/`1`/`1` shrinks the value-byte denominator a key +/// rename is measured against and inflates the rename's apparent percentage cost. RefTableSnapshot makeSyntheticSnapshot(size_t n) { RefTableSnapshot snapshot; @@ -127,7 +169,8 @@ RefTableSnapshot makeSyntheticSnapshot(size_t n) { RefCommittedRow row; row.ref_name = "part_" + std::to_string(i) + "_20260719_0_1000_1"; - row.manifest_ref = ManifestRef{1, 1, static_cast(i + 1)}; + row.manifest_ref = ManifestRef{42, 89811 + static_cast(i), static_cast(i + 1)}; + row.published_at_ms = 1752900000000ULL + i; snapshot.committed.push_back(row); } std::sort(snapshot.committed.begin(), snapshot.committed.end(), @@ -550,4 +593,485 @@ static void BM_Materialize(benchmark::State & state) } BENCHMARK(BM_Materialize)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); -BENCHMARK_MAIN(); +/// ------------------------------------------------------------------------------------------------- +/// Wire-key-cut instrument (Task 7): encode AND decode for the five formats the campaign's wire-key +/// rename touched most (`cas_run`, `cas_ref_snap`, `cas_part_manifest`, `cas_fold_seal`, +/// `cas_ref_catalog`), plus a byte/cap oracle (below `reportFormatCaps`). This section only BUILDS the +/// instrument -- it does not take the before/after measurement itself, which is a later task run +/// against this same binary built on both sides of the cut. The "before" side is the pre-cut worktree +/// at `/home/mfilimonov/workspace/ClickHouse/cas-p2-before`, pinned at commit `65ec8688cdb`; the +/// recorded patch that adapts this file's one incompatible call site (`foldedClassification`/ +/// `clampedClassification` below) for that build lives under +/// `docs/superpowers/cas/bench-wire-keys-phase3/`. Every other line in this section is byte-identical +/// on both sides -- confirmed against the before-side headers, which differ from these only in +/// comment text (the retired terse wire spellings) and in `RefCoverage::classification`'s type. +/// ------------------------------------------------------------------------------------------------- + +namespace +{ + +/// `cas_run` fixture: `n` distinct blobs in strictly ascending digest order (`SourceEdgeRunWriter` +/// requires non-decreasing `(ref, source_id)` keys, and a monotonically increasing digest alone +/// satisfies that regardless of `source_id`). Marker mix models one healthy in-degree run: the +/// overwhelming majority of tracked blobs simply carry a live edge this generation (98% `Edge`); a +/// blob losing its LAST edge (`Zero`) or actually condemned for deletion (`Condemned`, carrying the +/// full retired-incarnation token) is comparatively rare at any one round -- 1% each here, not 0 and +/// not half. `source_id` is a synthetic per-record counter rather than a real backend id: the codec's +/// cost is driven by the DIGEST's hex width, not the id's numeric value. The condemned token mirrors a +/// real S3 ETag's width (a quoted 32-hex value) and `size` a realistic single-blob byte count (64 KiB, +/// a typical compressed column chunk). Record count ranges 100 to 100,000 (`RangeMultiplier(10)`), +/// matching every `Complexity()` benchmark already in this file. +std::vector makeSourceEdgeRecords(size_t n) +{ + std::vector records; + records.reserve(n); + for (size_t i = 0; i < n; ++i) + { + const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(i + 1))}; + SourceEdgeRecord rec; + rec.ref = ref; + if (i % 100 == 0) + { + rec.source_id = UInt128(0); + rec.marker = RunMarker::Condemned; + rec.delete_pending = (i % 200 == 0); + rec.token = Token{"\"e1b2c3d4e5f6071829300a0b0c0d0e0f\"", TokenType::ETag}; + rec.size = 64 * 1024; + rec.condemn_round = 7; + } + else if (i % 100 == 50) + { + rec.source_id = UInt128(0); + rec.marker = RunMarker::Zero; + } + else + { + rec.source_id = UInt128(i + 1); + rec.marker = RunMarker::Edge; + } + records.push_back(rec); + } + return records; +} + +/// Runs the real `SourceEdgeRunWriter` over `records`, exactly as `CASRecordStream`'s own +/// `encodeRun` helper does -- so decode below always consumes real encoder output, never a +/// hand-built string. +String encodeSourceEdgeRun(const std::vector & records) +{ + DB::WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + for (const auto & r : records) + writer.append(r); + writer.finish(); + /// `str()` returns a `std::string &`, so returning it plainly would copy-construct the whole + /// encoded run on every call (no NRVO is available for a reference) -- `std::move` here moves it + /// instead, matching the four other encoders, which all end with `std::move(out).take()` and copy + /// nothing. `str()` finalizes `out` itself, so no separate `finalize()` call is needed first. + return std::move(out.str()); +} + +/// The ONE call site whose TYPE differs across the wire-key-rename cut this benchmark spans: on this +/// (AFTER) side `RefCoverage::classification` is the closed `CoverageClass` enum; at the pre-cut +/// commit it is a raw `uint8_t` whose CLAMPED value is ALSO renumbered (4 there, 3 here -- see +/// `CasFoldSealFormat.h`'s own history comment on `CoverageClass`). A bare numeric literal at the call +/// site would therefore silently measure the WRONG row shape on the before-side build, so the Step-4 +/// patch touches only this pair of one-line functions; every benchmark body in this file stays +/// byte-identical on both sides. +CoverageClass foldedClassification() { return CoverageClass::Folded; } +CoverageClass clampedClassification() { return CoverageClass::Clamped; } + +/// Not the record axis under test (`n` below is `ref_lives` row count): fixed at a representative +/// multi-shard pool size. A single-shard fixture would fold `blob_target_runs`/`condemned_summary` to +/// one degenerate entry each, understating the per-shard fan-out a real multi-shard pool carries in +/// both sections. +constexpr uint64_t kFoldSealGcShards = 4; + +/// `cas_fold_seal` fixture: `n` `ref_lives` rows keyed by ascending life id, split base/hold-bearing/ +/// cleanup-evidence 90%/5%/5%. Per the spec's byte table, a hold-bearing row adds 33 bytes and a +/// cleanup-evidence row adds 16 bytes over a base row's 22-plus-class-word bytes; at this 90/5/5 mix +/// the recovered uplift over an all-base fixture is 0.05*33 + 0.05*16 = 2.45 bytes/row, about 8% over +/// a base row's own ~30 bytes (the full one-third the spec's deltas imply is the all-clamped extreme, +/// not this mix) -- still enough that omitting the two minority shapes entirely would misstate the +/// row-average cost in the wrong direction. The 90/5/5 split models a healthy pool: most namespaces +/// fold cleanly every round (base: `Folded`, no hold, no cleanup evidence); a minority sit behind a +/// transient barrier (hold-bearing: `Clamped`, `ManifestBodyMissing`); a minority are mid-teardown +/// (cleanup evidence: `Folded` plus a terminal `remove_namespace` fold). Neither minority shape is the +/// common case, but neither is negligible either -- both recur every round in a live pool. +/// `RefTxnId` epoch/sequence pairs and the hold's `retry_count`/`next_retry_round` are multi-digit +/// (a pool old enough to have restarted its writer dozens of times and folded past its 100,000th +/// ref-log transaction; a hold retried past its first round but nowhere near abandoned) rather than +/// the single-digit illustrative values the spec's byte table uses to name the three row SHAPES -- +/// matching the shapes, not the spec table's example digits, is what keeps the value-byte denominator +/// realistic (see `makeSyntheticSnapshot`'s doc comment for why that denominator matters). Record +/// count ranges 100 to 100,000, matching every `Complexity()` benchmark in this file. +CasFoldSeal makeFoldSeal(size_t n) +{ + CasFoldSeal seal; + seal.generation = 7; + seal.parent_generation = 6; + for (size_t i = 0; i < n; ++i) + { + RefLifeFoldState row; + if (i % 20 == 0) + { + row.coverage = RefCoverage{ + .classification = clampedClassification(), + .last_folded_ref_id = RefTxnId{42, 103482}, + .hold = RefHold{ + .reason = HoldReason::ManifestBodyMissing, + .offending_position = RefTxnId{42, 103500}, + .retry_count = 14, + .next_retry_round = 1042}}; + } + else if (i % 20 == 1) + { + row.coverage = RefCoverage{.classification = foldedClassification(), .last_folded_ref_id = RefTxnId{42, 118203}}; + row.cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{42, 118190}}; + } + else + { + row.coverage = RefCoverage{.classification = foldedClassification(), .last_folded_ref_id = RefTxnId{42, 100123}}; + } + seal.ref_lives.emplace(UInt128(i + 1), std::move(row)); + } + for (uint64_t shard = 0; shard < kFoldSealGcShards; ++shard) + { + seal.blob_target_runs.push_back(RunRef{ + .key = fmt::format("p/gc/gen/7/attempt/1/blob_target/{}/0", shard), + .checksum = UInt128(0x1000 + shard), .shard = shard, .key_generation = 7}); + seal.condemned_summary[shard] = CondemnedSummary{ + .condemned_total = 1000 + shard, .pending_total = 10 + shard, .oldest_nonpending_condemn_round = 4}; + } + return seal; +} + +/// `cas_part_manifest` fixture: `n` entries in path order, 90% `Blob` (the column/mark/index files +/// that dominate a real MergeTree part) and every 10th `Inline` (small metadata files like +/// `count.txt`/`checksums.txt` that get embedded rather than stored as a separate blob). Blob sizes +/// cycle 4-64 KiB across 16 steps to resemble the spread of real column-chunk sizes rather than one +/// repeated constant; inline bytes are a fixed 48-byte payload, resembling a small metadata file. +/// `ref`/`root_namespace_id` are fixed -- they do not scale with entry count in a real manifest +/// either. `encodePartManifest` sorts entries itself, so input order need not be canonical. Record +/// count ranges 100 to 100,000, matching every `Complexity()` benchmark in this file (a real part +/// rarely reaches the top of that range; it stress-tests a pathologically wide/many-column part). +PartManifest makePartManifest(size_t n) +{ + PartManifest m; + m.ref = ManifestRef{5, 15, 1}; + m.root_namespace_id = RootNamespace("00/aa@cas@"); + m.entries.reserve(n); + for (size_t i = 0; i < n; ++i) + { + ManifestEntry e; + if (i % 10 == 9) + { + e.path = fmt::format("{:06}_meta.txt", i); + e.placement = EntryPlacement::Inline; + e.inline_bytes = String(48, 'x'); + } + else + { + e.path = fmt::format("{:06}_data.bin", i); + e.placement = EntryPlacement::Blob; + e.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(i + 1))}; + e.blob_size = 4096 * (1 + (i % 16)); + } + m.entries.push_back(std::move(e)); + } + m.payload_digest = computePayloadDigest(m); + return m; +} + +/// `cas_ref_catalog` fixture: `n` entries in ascending namespace order (a 7-digit zero-padded ordinal +/// keeps ascending lexical order across the whole 100..100,000 range, well under `kMaxNamespaceBytes`). +/// The mix resembles one whole-pool catalog snapshot: most namespaces are simply `Live` (96%), with a +/// small steady trickle of admission (`Creating`, 2%) and teardown (`Removing`, 2%) in flight at any +/// moment -- neither churn state is the common case, but neither is negligible either. Record count +/// ranges 100 to 100,000, matching every `Complexity()` benchmark in this file. +RefCatalog makeRefCatalog(size_t n) +{ + RefCatalog catalog; + catalog.entries.reserve(n); + for (size_t i = 0; i < n; ++i) + { + CatalogEntry e; + e.ns = RootNamespace(fmt::format("roots/ca_tbl_{:07}", i)); + e.incarnation = UInt128(i + 1); + if (i % 50 == 0) + { + e.state = NsState::Creating; + e.creator = CreatorFence{"srv-bench", 1, 1}; + } + else if (i % 50 == 25) + { + e.state = NsState::Removing; + e.removal_started_round = 42; + } + else + { + e.state = NsState::Live; + } + catalog.entries.push_back(std::move(e)); + } + return catalog; +} + +} + +/// `cas_run` is streamed (`object_cap == 0`; see `CasRecordStreamFormat.h`) and never materialized +/// whole in production, but the benchmark still needs one complete encoded run to time and to decode: +/// `encodeSourceEdgeRun` drives the real `SourceEdgeRunWriter`/`SourceEdgeRunReader` pair over an +/// in-memory buffer, the same pair the streaming production path uses over its own `WriteBuffer`/ +/// `ReadBuffer`. +static void BM_CasRunEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const std::vector records = makeSourceEdgeRecords(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeSourceEdgeRun(records)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRunEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRunDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const String encoded = encodeSourceEdgeRun(makeSourceEdgeRecords(n)); + for (auto _ : state) + { + DB::ReadBufferFromMemory in(encoded.data(), encoded.size()); + SourceEdgeRunReader reader(in); + SourceEdgeRecord rec; + size_t count = 0; + while (reader.next(rec)) + ++count; + benchmark::DoNotOptimize(count); + } + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRunDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +/// `BM_SnapshotEncode` above already exists (for the E4 contiguous-scan investigation) and has no +/// decode counterpart. This pair is the one the wire-key-cut measurement uses: same fixture, but named +/// and shaped to match the other four formats' encode/decode pairs in this section. +static void BM_CasRefSnapEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefTableSnapshot snapshot = makeSyntheticSnapshot(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeRefTableSnapshot(snapshot)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefSnapEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRefSnapDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefTableSnapshot snapshot = makeSyntheticSnapshot(n); + const String encoded = encodeRefTableSnapshot(snapshot); + for (auto _ : state) + benchmark::DoNotOptimize(decodeRefTableSnapshot(encoded, snapshot.ns, snapshot.snapshot_id)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefSnapDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasPartManifestEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const PartManifest m = makePartManifest(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodePartManifest(m)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasPartManifestEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasPartManifestDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const PartManifest m = makePartManifest(n); + const String encoded = encodePartManifest(m); + for (auto _ : state) + benchmark::DoNotOptimize(decodePartManifest(encoded)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasPartManifestDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasFoldSealEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const CasFoldSeal seal = makeFoldSeal(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeFoldSeal(seal)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasFoldSealEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasFoldSealDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const CasFoldSeal seal = makeFoldSeal(n); + const String encoded = encodeFoldSeal(seal); + for (auto _ : state) + benchmark::DoNotOptimize(decodeFoldSeal(encoded)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasFoldSealDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRefCatalogEncode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefCatalog catalog = makeRefCatalog(n); + for (auto _ : state) + benchmark::DoNotOptimize(encodeRefCatalog(catalog)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefCatalogEncode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +static void BM_CasRefCatalogDecode(benchmark::State & state) +{ + const size_t n = static_cast(state.range(0)); + const RefCatalog catalog = makeRefCatalog(n); + const String encoded = encodeRefCatalog(catalog); + for (auto _ : state) + benchmark::DoNotOptimize(decodeRefCatalog(encoded)); + state.SetComplexityN(static_cast(n)); +} +BENCHMARK(BM_CasRefCatalogDecode)->RangeMultiplier(10)->Range(100, 100000)->Complexity(); + +namespace +{ + +/// Binary search on record count with the real encode -> `sealObject` -> `openObject` pipeline as the +/// oracle. `openObject` (`CasTextFormat.cpp`) enforces the registry's `object_cap` on BOTH the raw and +/// the zstd-frame-header path, so this one pipeline works whether or not the format compresses; a +/// format's OWN pre-put gate (e.g. fold-seal's `checkFoldSealObjectBytes`) may throw earlier, at the +/// encode step itself. Either `LIMIT_EXCEEDED` or `CORRUPTED_DATA` at this boundary means "does not +/// fit" and steers the search; any other exception is a fixture bug, not a capacity signal, and is +/// left to propagate rather than being misread as "found the cap". +/// +/// `known_fits_n`/`known_fits_bytes` seed the exponential search from a bytes-per-record estimate +/// measured at a small `n` -- NOT a hardcoded delta table -- purely to reduce how many large, +/// expensive encodes the search performs before bisecting. The estimate never becomes the answer: the +/// real encoder confirms every step of both the exponential growth and the final exact bisection. +template +uint64_t maxRecordCountUnderCap(FormatId id, uint64_t known_fits_n, uint64_t known_fits_bytes, Encode encode) +{ + auto fits = [&](uint64_t n) -> bool + { + try + { + const String stored = sealObject(id, encode(n)); + benchmark::DoNotOptimize(openObject(id, stored)); + return true; + } + catch (const DB::Exception & e) + { + if (e.code() == DB::ErrorCodes::CORRUPTED_DATA || e.code() == DB::ErrorCodes::LIMIT_EXCEEDED) + return false; + throw; + } + }; + + /// The bisection below is correct only if `fits(lo) == true`. The caller's `known_fits_n` comes + /// from an encode IT ran itself -- never through `openObject`, which is what actually enforces + /// `object_cap` (the raw-size check, or the zstd frame's declared decompressed size) -- so this + /// verifies the bound directly rather than trusting that claim. If `known_fits_n` itself is + /// already over the cap (e.g. a future, much larger report size), halve downward until a verified + /// fit is found; if even `n == 1` does not fit, that is a fixture/format bug, not a capacity + /// signal, and is raised loudly rather than silently reported as a wrong maximum. + uint64_t lo = known_fits_n; + while (lo > 1 && !fits(lo)) + lo /= 2; + if (!fits(lo)) + throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, + "maxRecordCountUnderCap: format {} does not fit its object cap even at n=1", static_cast(id)); + + const FormatTraits & traits = traitsFor(id); + const uint64_t per_record = std::max(1, known_fits_bytes / std::max(1, known_fits_n)); + uint64_t hi = std::max(lo * 2, traits.object_cap / per_record); + while (fits(hi)) + { + lo = hi; + hi *= 2; + } + while (hi - lo > 1) + { + const uint64_t mid = lo + (hi - lo) / 2; + (fits(mid) ? lo : hi) = mid; + } + return lo; +} + +/// One format's report line: decompressed bytes at `report_n`, stored bytes under the format's REAL +/// registered compression policy (or `n/a` for a policy that stores raw -- `Never`/`PinnedRaw` -- since +/// there is no separate compressed form to report, and a `0` there would read as a measurement rather +/// than "not applicable"), and the largest record count the real encoder admits under the format's +/// object cap. +template +void reportSealedFormat(std::string_view name, FormatId id, uint64_t report_n, Encode encode) +{ + const FormatTraits & traits = traitsFor(id); + const String decompressed = encode(report_n); + const String stored = sealObject(id, decompressed); + const bool stores_raw = traits.compression != CompressionPolicy::Always; + const uint64_t max_n = maxRecordCountUnderCap(id, report_n, decompressed.size(), encode); + + fmt::print("{:<18} decompressed={:>10} bytes (n={}) stored={} max_n_under_object_cap={}\n", + name, decompressed.size(), report_n, + stores_raw ? "n/a (stored raw, no compression)" : (std::to_string(stored.size()) + " bytes (zstd)"), + max_n); +} + +/// Step 3's byte and cap oracle: a small, main-less, flag-invoked harness (see `main` below) rather +/// than a benchmark or a gtest, so it never engages the timing loop and never needs a second `main` in +/// this binary. Reports, per format, at the stated `kReportN`: decompressed bytes, stored bytes under +/// the real compression policy (or `n/a`), and the maximum record count the real encoder admits under +/// the object cap (or `n/a` where none applies). +void reportFormatCaps() +{ + constexpr uint64_t kReportN = 1000; + fmt::print("=== cas format byte/cap report (n={}) ===\n", kReportN); + + /// `cas_run` is `object_cap == 0` (streamed, `RunFile` family): never materialized whole in + /// production, so there is no whole-object cap to search for and no compressed form to report. + { + const String encoded = encodeSourceEdgeRun(makeSourceEdgeRecords(kReportN)); + fmt::print("{:<18} decompressed={:>10} bytes (n={}) stored=n/a (PinnedRaw, never compressed) " + "max_n_under_object_cap=n/a (object_cap=0: streamed one line at a time, never materialized whole)\n", + "cas_run", encoded.size(), kReportN); + } + + reportSealedFormat("cas_ref_snap", FormatId::RefSnapshot, kReportN, + [](uint64_t n) { return encodeRefTableSnapshot(makeSyntheticSnapshot(n)); }); + reportSealedFormat("cas_part_manifest", FormatId::PartManifest, kReportN, + [](uint64_t n) { return encodePartManifest(makePartManifest(n)); }); + reportSealedFormat("cas_fold_seal", FormatId::FoldSeal, kReportN, + [](uint64_t n) { return encodeFoldSeal(makeFoldSeal(n)); }); + reportSealedFormat("cas_ref_catalog", FormatId::RefCatalog, kReportN, + [](uint64_t n) { return encodeRefCatalog(makeRefCatalog(n)); }); +} + +} + +/// Hand-written in place of `BENCHMARK_MAIN()` so `--report_format_caps` can dispatch to Step 3's +/// oracle BEFORE `benchmark::Initialize` ever sees argv -- keeping the byte/cap report in this same +/// binary without a second `main` or a separate gtest target, and without the report's args tripping +/// `ReportUnrecognizedArguments`. Absent that flag, behavior is exactly `BENCHMARK_MAIN()`'s. +int main(int argc, char ** argv) +{ + for (int i = 1; i < argc; ++i) + { + if (std::string_view(argv[i]) == "--report_format_caps") + { + reportFormatCaps(); + return 0; + } + } + + benchmark::Initialize(&argc, argv); + if (benchmark::ReportUnrecognizedArguments(argc, argv)) + return 1; + benchmark::RunSpecifiedBenchmarks(); + return 0; +} diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index df987e1325ac..0f1a610359aa 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -474,7 +474,7 @@ inline String encodeMinimalGcState(uint64_t round) } /// Inject condemned bookkeeping + gc/state directly (bypassing a real GC round) so a test can seed the -/// GC ledger's condemned state at an arbitrary round. Retired-in-snapshot: the condemned entries are +/// GC ledger's condemned state at an arbitrary round. The condemned entries are /// seeded the way a real round leaves them — as `RunMarker::Condemned` sentinel rows inside an adopted fold seal's /// shard run (there is no separate retired-list object). A synthetic +edge/-edge pair nets each blob to /// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `RunMarker::Condemned` row. @@ -542,7 +542,7 @@ inline void injectRetire( backend.putOverwrite(layout.gcStateKey(), state, head.token); } -/// Adopt a fold seal carrying a given per-gc-shard `condemned_summary` (retired-in-snapshot T4) and point +/// Adopt a fold seal carrying a given per-gc-shard `condemned_summary` and point /// gc/state at it (snap_generation / snap_attempt / gc_shards), bypassing a real GC round. If a seal /// already exists at (generation, attempt) it is overwritten with the new summary (its other fields are /// preserved); otherwise a fresh minimal seal is created. Read-modify-CAS on gc/state preserves the lease. @@ -620,7 +620,7 @@ inline bool runRoundsUntilAbsent( } /// The CURRENT condemned entries for `shard`, read from the adopted fold seal's `blob_target_runs` -/// (retired-in-snapshot T4): the round no longer writes a separate retired-list object — condemned +///: the round no longer writes a separate retired-list object — condemned /// entries RIDE the source-edge run as `RunMarker::Condemned` sentinel rows at the zero-sentinel key. This reads /// the seal at (snap_generation, snap_attempt), opens every run for `shard`, and reconstructs the /// `RetiredEntry` shape (hash from the run key, the rest from the decoded `CondemnedRow`). Empty when @@ -669,8 +669,7 @@ inline std::vector currentRetiredSet( } /// True iff ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row — the ack-floor deletion -/// pipeline is in flight while this is true (retired-in-snapshot T4 replacement for the old -/// "iterate gc/state.retired_refs" probe). `gc_shards` is read from gc/state when 0 is passed. +/// pipeline is in flight while this is true. `gc_shards` is read from gc/state when 0 is passed. inline bool anyCondemnedInSeal( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, uint64_t gc_shards = 0) { diff --git a/src/Disks/tests/gtest_ca_wiring.cpp b/src/Disks/tests/gtest_ca_wiring.cpp index 9d12f9ccb759..6d34e444d603 100644 --- a/src/Disks/tests/gtest_ca_wiring.cpp +++ b/src/Disks/tests/gtest_ca_wiring.cpp @@ -1903,7 +1903,7 @@ TEST(CASWiringExchange, AdoptIntoADetachedTargetPublishesADetachedRefAndNoLiveRe EXPECT_TRUE(storage->existsFile(detached_tmp_path + "/p.proj/data.bin")); EXPECT_TRUE(storage->existsFile(detached_tmp_path + "/uuid.txt")); - /// Finalization, unchanged by this task: `IMergeTreeDataPart::renameTo(detached/)` is a + /// Finalization: `IMergeTreeDataPart::renameTo(detached/)` is a /// moveDirectory of the staged dir to its final detached name, which on a content-addressed disk is /// a ref repoint WITHIN the same namespace -- the same shape the active path's /// `renameTempPartAndReplace` uses, and the reason the relinked detached part needs no new @@ -2825,7 +2825,7 @@ DB::Cas::PoolPtr openResurrectStore(std::shared_ptr & } /// Condemn (kind=Blob, hash, token) by seeding gc/state + a per-shard retired set (the durable GC ledger -/// shape — RetiredEntry, exact-token delete, unchanged by this task) AND condemning the per-hash freshness +/// shape — `RetiredEntry`, exact-token delete) AND condemning the per-hash freshness /// meta, which is what the writer's condemned decision ACTUALLY point-reads (spec §meta-protocols v3). /// Bumps the round so the retirement is a fresh one; leaves the object itself in place (condemn, NOT delete). void seedCondemnBlobToken(DB::Cas::Pool & store, const DB::UInt128 & hash, diff --git a/src/Disks/tests/gtest_cas_backend_generation.cpp b/src/Disks/tests/gtest_cas_backend_generation.cpp index b6f9502c45fa..04da7d8aba63 100644 --- a/src/Disks/tests/gtest_cas_backend_generation.cpp +++ b/src/Disks/tests/gtest_cas_backend_generation.cpp @@ -151,7 +151,7 @@ TEST(CASBackendGeneration, NativeHeadUsesNativeTokenMetadataApi) ASSERT_EQ(b->putIfAbsent("p/native-head/key", "v1").outcome, PutOutcome::Done); - /// putIfAbsent's own HEAD-fallback stamping path calls the ordinary API (untouched by this task); + /// `putIfAbsent`'s HEAD-fallback stamping path calls the ordinary API; /// reset the counters so only nativeHead's call, below, is observed. storage->ordinary_calls = 0; storage->native_calls = 0; diff --git a/src/Disks/tests/gtest_cas_blob_digest.cpp b/src/Disks/tests/gtest_cas_blob_digest.cpp index 9a87f08ccda2..b6f8c9f3f972 100644 --- a/src/Disks/tests/gtest_cas_blob_digest.cpp +++ b/src/Disks/tests/gtest_cas_blob_digest.cpp @@ -1,7 +1,7 @@ #include -/// CAS pluggable-blob-hash Phase 2, Task 1: `BlobDigest` (the pool-scoped variable-length content digest, ADDITIVE-ONLY -- no -/// existing `UInt128 blob_hash` field is migrated in this task) + the ONE `PoolMeta`-scoped +/// `BlobDigest` is the pool-scoped variable-length content digest. It is additive: existing +/// `UInt128 blob_hash` fields retain their representation. The `PoolMeta`-scoped /// `DigestCodec` all digest<->hex/bytes conversion must route through. /// /// THE KEY GATE (`ShardOfBitIdenticalToOldHighBitsOver200RandomValues` below): `DigestCodec`'s diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 25cd9577fe76..98bd9ef57ac3 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -172,6 +172,41 @@ TEST(CASBlobEnvelopeFormat, MandatoryWorstCaseBoundary) << "ref budget reachable through the real encoder at the default header length"; } +/// The half a `static_assert` cannot do. The compile-time bound proves the FORMULA fits under the +/// floor; it cannot notice a formula that understates the encoder — shrink any component and the +/// assert only grows happier. So this reconstructs the same number from bytes the real encoder +/// produced, and the only quantity it borrows is the version field's type width: +/// +/// what the encoder wrote at max-width values, with an empty ref +/// + the digits the version field did NOT use at this generation +/// == the mandatory worst case +/// +/// Every other field in the fixture is already at its type maximum, so nothing else is missing from +/// the measured side. An understated key cost, or a shrunken `kMaxU32DecimalLen`, moves the formula +/// without moving the encoder and lands here. +TEST(CASBlobEnvelopeFormat, WorstCaseFormulaMatchesTheEncoder) +{ + EnvelopeHeader h = maxReachableHeader(""); + const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen)); + /// The mandatory shape is everything up to and including the closing brace, plus the newline the + /// encoder reserves at the last byte; the padding between them is the ref budget this measures. + const size_t json_len = head.find_last_not_of(' ', kMinBlobHeaderLen - 2) + 1; + const size_t mandatory_at_current_version = json_len + 1; /// + the reserved '\n' + + size_t version_digits = 0; + for (uint32_t v = currentCompatibilityVersion(); ; v /= 10) + { + ++version_digits; + if (v < 10) + break; + } + const size_t unused_version_digits = std::numeric_limits::digits10 + 1 - version_digits; + + EXPECT_EQ(mandatory_at_current_version + unused_version_digits, mandatory_descriptor_worst_case) + << "the formula and the encoder disagree about the mandatory descriptor: encoder wrote " + << mandatory_at_current_version << " bytes at a " << version_digits << "-digit version"; +} + TEST(CASBlobEnvelopeFormat, CriticalKeyDescriptorStillFitsAtDefaultLength) { /// The test-only `!x` critical key is written BEFORE `ref`; even at max-reachable field values diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index 45a075e50d3d..7d7808d70ecc 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -71,7 +71,7 @@ TEST(CASEncodingPins, RefLogTxnAllOpKinds) /// not quote/newline/control bytes/U+2028, so `ref_name` -- the only free-form string `RefOp` /// still carries now that `payload` is gone -- exercises quote, newline, a bare control byte, /// and the three-byte U+2028 sequence. Backslash escaping is pinned separately, over an - /// unrestricted string, by `gtest_cas_json_writer.cpp`'s `CASJsonWriterEscaping` suite. + /// unrestricted string, by the JSON-writer escaping suite. set_published_at.ref_name = String("20260101_0_1_1_1\"c\nd") + "\x01" "e" + "\xE2\x80\xA8" "f"; set_published_at.expected_manifest_ref = ManifestRef{1, 2, 3}; set_published_at.published_at_ms = 1234; diff --git a/src/Disks/tests/gtest_cas_fence_generation.cpp b/src/Disks/tests/gtest_cas_fence_generation.cpp index be621ddd9dd0..dcc1f7b3c4b9 100644 --- a/src/Disks/tests/gtest_cas_fence_generation.cpp +++ b/src/Disks/tests/gtest_cas_fence_generation.cpp @@ -327,7 +327,7 @@ TEST(CASFenceGeneration, PlainObjectRemoveAbortsWhenFenceTripsBetweenAdmissionAn store->removeNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "victim"); }); - /// The durable delete never ran -- the object survives (reads are not fence-gated by this task). + /// The durable delete never ran, so the object survives; reads are not fence-gated. const auto still_there = store->getNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "victim"); ASSERT_TRUE(still_there.has_value()); EXPECT_EQ(*still_there, "still here"); diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 55f7247d87b3..8109fc851605 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -32,10 +32,9 @@ namespace DB::ErrorCodes /// /// Reachability is a property of the WHOLE POOL. A blob is unreferenced only if no namespace anywhere /// owns an edge to it, so a round that deletes one is asserting something about every namespace at -/// once -- including the ones it never looked at. Task 7 made the per-namespace half of that assertion -/// cheap and exact: one `GET` at the cursor's arithmetic successor, absent means end-of-stream. Task 8 -/// made a namespace that could NOT be walked say so durably. What neither can supply is the SET those -/// proofs have to cover, and that is what this task is about. +/// once -- including the ones it never looked at. A `GET` at the cursor's arithmetic successor makes +/// the per-namespace proof cheap and exact, and a namespace that cannot be walked reports that fact +/// durably. Neither supplies the SET those proofs have to cover. /// /// So the gate has three terms, and a round destroys only when all three are clear: /// @@ -1423,7 +1422,7 @@ TEST(CASGCFrontierGate, TheHandOffReclaimIsInertUnderSuppression) EXPECT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) << "the superseded generation's prefix survives a suppressed round intact"; - /// AND THE OPPORTUNITY IS CONSUMED, NOT DEFERRED -- the one place in this task where the gate + /// AND THE OPPORTUNITY IS CONSUMED, NOT DEFERRED -- the gate /// costs something permanent, so it is asserted here rather than left to be discovered later. /// /// The hand-off is a one-shot DIFFERENCE: it compares the PARENT seal's runs against the new diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index 9cc892165f14..ce6b09099487 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -24,8 +24,8 @@ /// DURABLE HOLDS (spec 2026-07-27 "ref chain complete cut" §5). /// /// A namespace whose ref-log walk meets an IMPOSSIBLE shape stops there, and that stop has to survive -/// the round. Before this task the stop was a single bit — `classification == CoverageClass::Clamped` — -/// and everything that explained it (what went wrong, and exactly WHERE) lived in a log line and an +/// the round. A classification alone cannot preserve the cause and position of the stop; without +/// durable hold evidence, that information lives only in a log line and an /// in-memory anomaly, both gone by the next round. That is not enough for three separate reasons: /// /// * the next round could not RETRY the exact position, so a hold only survived while the round's @@ -298,8 +298,7 @@ std::vector> illFormedSealsTheEncoderMustRe "durable", clamped_without_hold); /// The closed set is now the enum's declared values, so only an explicit cast reaches outside it. - /// 4 is the sharpest value to plant: it was the wire value for Clamped before this task's dense - /// renumbering, and under the new table it is simply out of range. + /// 4 is the sharpest value to plant: it is outside the closed wire vocabulary. CasFoldSeal classification_retired_wire_value = cleanSeal("ns/0"); fixtureCoverage(classification_retired_wire_value, "ns/0").classification = static_cast(4); diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index b54b5e71d648..aa0a8f914972 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -129,7 +129,7 @@ GcState readState(InMemoryBackend & b, const Pool & s) return decodeGcState(got->bytes); } -/// Whether ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row (retired-in-snapshot T4: the +/// Whether ANY gc-shard's adopted-seal run still holds a `RunMarker::Condemned` row (the /// retired state rides the snapshot run, not a separate retired-list object) — the ack-floor deletion /// pipeline is still in flight while this is true. bool anyRetiredPending(InMemoryBackend & b, const Pool & s) @@ -540,7 +540,7 @@ TEST(CASGCRound, PublishDropReclaimsBlobAndManifestToFixpoint) EXPECT_FALSE(blobExists(*backend, store->layout(), DB::UInt128(1))); } -/// retired-in-snapshot T4: after a round condemns one blob, the ADOPTED fold seal's per-shard +/// After a round condemns one blob, the ADOPTED fold seal's per-shard /// condemned_summary reflects it (condemned_total == 1, pending_total == 0) — distilled zero-I/O from the /// RunMarker::Condemned rows the fold sealed into the snapshot run. TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) diff --git a/src/Disks/tests/gtest_cas_gc_round_defer.cpp b/src/Disks/tests/gtest_cas_gc_round_defer.cpp index a33816a8bac2..adbccce614ec 100644 --- a/src/Disks/tests/gtest_cas_gc_round_defer.cpp +++ b/src/Disks/tests/gtest_cas_gc_round_defer.cpp @@ -41,7 +41,7 @@ TEST(CASGCRoundDefer, PredicateTruthTable) EXPECT_FALSE(shouldDeferRound(2, false, 8, 3, 8)); // bound reached => force fold } -/// graduationDue (retired-in-snapshot T4): read ZERO-I/O from the adopted seal's condemned_summary. An +/// `graduationDue` reads ZERO-I/O from the adopted seal's `condemned_summary`. An /// entry whose oldest non-pending condemn round crosses current_round forces it true; a delete_pending /// entry forces it true regardless of the round; otherwise false. TEST(CASGCRoundDefer, GraduationDueDetectsDuePendingAndRoundCrossing) @@ -584,9 +584,9 @@ TEST(CASGCRoundDefer, DueGraduationIsSoleFoldTriggerAtHighThreshold) writeBlobBody(*backend, layout, blob); - /// Seed the adopted fold seal's condemned_summary with B already `delete_pending` (pending_total = 1), - /// mirroring `CASGCRoundDefer.GraduationDueDetectsDuePendingAndRoundCrossing`. Retired-in-snapshot - /// (T4): graduationDue reads this summary ZERO-I/O off the adopted seal — a delete_pending entry forces + /// Seed the adopted fold seal's condemned_summary with B already `delete_pending` (pending_total = 1). + /// `graduationDue` + /// reads this summary ZERO-I/O from the adopted seal — a `delete_pending` entry forces /// it true regardless of the round. At `gc_fold_threshold = 1000` a real condemn -> graduate pipeline of /// `runRegularRound` calls is not usable to set this up: every round before graduation would ITSELF /// defer (nothing due yet, and changed_shards never nears 1000), so the due-pending summary is injected diff --git a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp index bfb62f817ecb..b8f803e514f8 100644 --- a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp +++ b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp @@ -500,8 +500,8 @@ TEST(CASNamespaceFileDiskProfile, TheLifeResolutionIsPaidOncePerTableOpen) } -/// THE REMOVAL PATHS MUST NOT CREATE A NAMESPACE — the case that regressed silently in this task's first -/// round, so it is pinned on the catalog rather than on the file outcome. +/// THE REMOVAL PATHS MUST NOT CREATE A NAMESPACE: the catalog, rather than the file outcome, proves +/// that invariant. /// /// Why the file outcome cannot pin it: `unlinkFile`/`removeRecursive` against a never-opened table /// answer "absent" both before and after the defect, because a freshly minted namespace has no files diff --git a/src/Disks/tests/gtest_cas_namespace_life_id.cpp b/src/Disks/tests/gtest_cas_namespace_life_id.cpp index 9c2156d52aea..65c116d89f07 100644 --- a/src/Disks/tests/gtest_cas_namespace_life_id.cpp +++ b/src/Disks/tests/gtest_cas_namespace_life_id.cpp @@ -205,8 +205,8 @@ TEST(CASNamespaceLifeIdDeathTest, ZeroIncarnationIsUnconstructibleAborts) } #endif -/// Generation-5 namespace-bearing keys are outside the generation-6 parser roots altogether. Pool -/// admission rejects their generation before any listed-key parser is involved. +/// Namespace-bearing keys outside the opaque-life layout are rejected before any listed-key parser is +/// involved. TEST(CASNamespaceLifeId, GenerationFiveNamespaceBearingKeysAreOutsideTheFinalGrammar) { Layout l("p"); @@ -312,8 +312,8 @@ TEST(CASNamespaceLifeId, NamespaceFileKeysCarryTheIncarnationSegment) EXPECT_FALSE(l.namespaceFileKey(second, "format_version.txt").starts_with(l.namespaceFilesPrefix(life))); } -/// Generation-5 namespace-bearing file keys are outside the final parser root. Malformed ids under the -/// final state root are corruption and name the offending key. +/// Namespace-bearing file keys outside the opaque-life layout are rejected. Malformed life ids under +/// the state root are corruption and name the offending key. TEST(CASNamespaceLifeId, NamespaceFileParserRefusesLegacyAndMalformedIncarnations) { Layout l("p"); @@ -373,8 +373,7 @@ TEST(CASNamespaceLifeId, PhysicalFileKeysIgnoreLogicalNamespaceSpelling) EXPECT_EQ(parsed->relative_name, nested_name); } -/// The "cannot compile" half of spec §9 r9-5 #3: after this task there is no way to reach a ref-layer -/// key from a namespace alone, so dropping the incarnation is a compile error rather than an aliasing +/// A ref-layer key cannot be reached from a namespace alone, so dropping the incarnation is a compile error rather than an aliasing /// bug. Each helper is asserted twice -- the namespace-only form absent, the incarnation form present. TEST(CASNamespaceLifeId, NamespaceOnlyKeyHelpersDoNotExist) { @@ -413,8 +412,8 @@ TEST(CASNamespaceLifeId, NamespaceLifeIdAndRootNamespaceDoNotInterconvert) SUCCEED(); } -/// The out-of-scope fences, and they are POSITIVE on purpose: Constraint 12 keeps loose mountpoint -/// objects and part manifests on the identity they have today, so this task must NOT have qualified +/// The out-of-scope fences are POSITIVE on purpose: loose mountpoint objects and part manifests keep +/// their namespace identity, so they must NOT be qualified /// them. If a negative here fails, someone added a life-scoped overload to a family the amendment /// explicitly excluded; if a positive fails, someone removed the un-scoped one those callers use. TEST(CASNamespaceLifeId, MountpointObjectsAndManifestsStayUnqualified) diff --git a/src/Disks/tests/gtest_cas_parallel_commit.cpp b/src/Disks/tests/gtest_cas_parallel_commit.cpp index 7ae5ab676712..8cdb007d6623 100644 --- a/src/Disks/tests/gtest_cas_parallel_commit.cpp +++ b/src/Disks/tests/gtest_cas_parallel_commit.cpp @@ -160,7 +160,7 @@ namespace /// Fixture for the `CasCommitRollback` suite: wraps a real `ContentAddressedMetadataStorage` and /// drives ordinary `ContentAddressedTransaction`s through disk paths, so the fault seams under test /// (`ContentAddressedMetadataStorage::armPromoteFailureForTest`/`setAfterPromoteHookForTest`, the -/// minimal test-only hooks this task adds) fire from the SAME `publishStaging` call path production +/// minimal test-only hooks) fire from the SAME `publishStaging` call path production /// `commit()` uses -- unlike `CaWiringFixture` above, which pokes the bare pool primitives directly. /// Every part in one fixture instance shares ONE fixed table uuid (and therefore one `RootNamespace`), /// matching every test's single `fx.ns()`. diff --git a/src/Disks/tests/gtest_cas_part_manifest_format.cpp b/src/Disks/tests/gtest_cas_part_manifest_format.cpp index e966b87a332a..dd6eab02c003 100644 --- a/src/Disks/tests/gtest_cas_part_manifest_format.cpp +++ b/src/Disks/tests/gtest_cas_part_manifest_format.cpp @@ -30,8 +30,7 @@ void expectThrowsCode(int expected_code, F && fn) } } -/// One Blob + one Inline entry, matching the plan's §text-shape illustration verbatim (codecs-v3 -/// phase 6): deliberately NOT path-sorted on input, so the round trip also exercises canonical +/// One Blob + one Inline entry, deliberately NOT path-sorted on input, so the round trip also exercises canonical /// path-order encoding. PartManifest sample() { @@ -69,7 +68,7 @@ TEST(CASFormatBattery, PartManifest) /// stays self-consistent with whatever sample() produces, now that decode verifies payload_digest. const String golden = currentFormatHeader("cas_part_manifest") + - "{\"epoch\":\"5\",\"build\":\"15\",\"ord\":1,\"root_namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. + "{\"epoch\":\"5\",\"build\":\"15\",\"ord\":1,\"namespace\":\"00/aa@cas@\",\"payload_digest\":\"" + u128ToHex(m.payload_digest) + "\"}\n" // NOLINT(modernize-raw-string-literal): mixes '\"' quoting with '\n' line endings across this concatenated literal; a raw string can't hold the newline as-is. "{\"path\":\"a/b.bin\",\"place\":\"blob\",\"algo\":\"ch128\",\"digest\":\"00112233445566778899aabbccddeeff\",\"size\":4096}\n" "{\"path\":\"c/small.txt\",\"place\":\"inline\",\"size\":12}\n" "{\"n\":2}\n" @@ -517,8 +516,7 @@ TEST(CASPartManifestFormat, InlineRecordSizeMismatchWithPayloadZoneBannerFailsCl expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodePartManifest(bad); }); } -/// ==== migrated from gtest_cas_manifest_codec.cpp (deleted in the phase-6 binary->text cutover, -/// Task 3): these exercise refMatchesBody/manifestNamespaceMatches/findEntry/entryRange, pure +/// ==== Migrated manifest helpers: these exercise `refMatchesBody`, `manifestNamespaceMatches`, `findEntry`, and `entryRange`, pure /// functions carried over verbatim from the retired binary codec (untouched by the wire-shape /// migration) — reusing this file's own sample() fixture instead of reintroducing a second one. ==== diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index a74baf05fa61..6c7eb1d59667 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -439,15 +439,9 @@ TEST(CASPluggableHash, Sha256BlobSeenByCondemnSweepAndFsckNotSilentlySkipped) } /// ============================================================================================ -/// CAS pluggable-blob-hash Phase 2 Task 6 -- end-to-end sha256 WRITE path (in-memory; the real -/// wiring-level integration + soak is Task 7). -/// -/// Before this task, `PartWriteTxn`'s OWN write-path internals stayed a fixed 128-bit representation -/// downstream of the mint (`poolContentHash`/`PartWriteTxn::putBlob`'s `logical_hash`, the `deps` map key, the -/// event-log `object_hash` render, and `objectKey`) -- safe only because the disk-config factory guard -/// (`MetadataStorageFactory.cpp`) blocked any real sha256 pool from reaching `PartWriteTxn` at all (see the -/// Task 5 report and the "Task 6+" comments this task removes). Task 6 finishes those sites AND lifts -/// the guard in the SAME commit. This test drives a REAL `PartWriteTxn` (`putBlob` -> `stageManifest` -> +/// End-to-end SHA-256 write path. Every `PartWriteTxn` representation downstream of digest creation +/// must preserve the pool's variable-width digest; truncation would address a different blob. This +/// test drives a REAL `PartWriteTxn` (`putBlob` -> `stageManifest` -> /// `precommitAdd` -> `promote`) on a `Sha256` pool and asserts: /// 1. the blob lands under `blobs/sha256/<64-hex>` and the manifest entry's `blob_hash`, read back via /// `decodePartManifest`, is the FULL 32-byte digest (bytes beyond 16 are non-zero for a real sha256 @@ -510,7 +504,7 @@ TEST(CASPluggableHash, Sha256BuildWritesFullWidthDigestAndInlineEqualsBlob) /// THE CRUX (blob side): the blob body lands under the sha256-segmented path, addressed by the /// FULL 64-hex key -- `PartWriteTxn::putBlob`'s internal `logical_hash` must not have silently narrowed it - /// to a 32-hex (128-bit) key before this task. + /// to a 32-hex (128-bit) key. const String blob_key = store->layout().blobKey(id); EXPECT_NE(blob_key.find("/blobs/sha256/"), String::npos) << blob_key; ASSERT_TRUE(backend->head(blob_key).exists); @@ -545,7 +539,7 @@ TEST(CASPluggableHash, Sha256BuildWritesFullWidthDigestAndInlineEqualsBlob) /// validation at `foldManifestEdges` with refresh-on-miss. /// ============================================================================================ -/// spec §9.8 -- THE race regression this task exists to close. Each `Pool`'s `admitted_algos` cache +/// Each `Pool`'s `admitted_algos` cache /// is a MONOTONE snapshot seeded once at `Pool::open` and never re-read on its own; if node A admits /// a brand-new algo and publishes a manifest naming it, node B's stale cache must NOT fail the fold /// closed forever -- `foldManifestEdges` must refresh `_pool_meta` on the very first miss and accept diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index a458bbb52b6b..f2a3ec1d6570 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -71,8 +71,7 @@ String rawEntryLine(const String & ns, const String & state, const String & inc_ } /// Wraps `entry_lines` in the header/trailer a real `cas_ref_catalog` object carries. `v:1` always -/// passes the header gate (any version <= the build's `G_BUILD` does), matching the convention -/// `gtest_cas_fold_seal_format.cpp`'s `RejectsOutOfRangeNsCleanupState` uses for the same reason. +/// passes the header gate because any version <= the build's `G_BUILD` does. String rawCatalog(const std::vector & entry_lines) { String out = R"({"type":"cas_ref_catalog","v":1})" "\n"; @@ -618,11 +617,11 @@ TEST(CASRefCatalogFormat, RegistryRowIsControlStrictWithRawStorage) EXPECT_EQ(traits.object_cap, 256u * 1024u * 1024u); EXPECT_EQ(traits.line_cap, 4u * 1024u); EXPECT_EQ(traitsForType("cas_ref_catalog"), &traits); - /// Raw, so the key has no suffix: `Pool/CasRefCatalog.cpp` hands bytes to/from the backend - /// directly, bypassing `sealObject`/`openObject` because both are the identity under + /// Raw, so the key has no suffix: the catalog hands bytes directly to/from the backend, + /// bypassing `sealObject`/`openObject` because both are the identity under /// `CompressionPolicy::Never`. This line is the TRIPWIRE for that shortcut -- a policy flip to /// `Always` would silently write uncompressed bodies under a `.zst` key, which this assertion - /// catches first (see `CasRefCatalogFormat.h`'s comment on `encodeRefCatalog`). + /// catches first. EXPECT_EQ(storedSuffix(FormatId::RefCatalog), ""); EXPECT_EQ(traits.compression, CompressionPolicy::Never); } diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index 2cdc32ff1851..1afff867b905 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -51,9 +51,8 @@ extern const Event CASRefSnapshotPublishDispatched; /// single item whose own op count, or whose one op's encoded size, exceeds its cap fails ALONE; a /// neighbor co-batched into the same flush still commits. `ref_txn_max_ops` is checked exactly (the /// `build_ops` result's size), and the per-op cap is checked by encoding exactly one op at a time -- -/// no accumulation, matching the admission machinery this replaces. T9 (removal-class detection by -/// op inspection) and T10 (chunked flush across a whole-batch op-count overflow) extend this file; -/// this task adds only the per-item / per-op isolation tests and the canonical round-trip leg of +/// no accumulation, matching the admission machinery. The per-item / per-op isolation tests and +/// the canonical round-trip leg cover /// test 12 (the maximum legally-admissible normal-class transaction). /// /// The suite name is prefixed `RefWriter` so it is covered by the `RefWriter*` unit-test gate filter. diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index 13437baf17a8..a81339967eaa 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -242,9 +242,8 @@ TEST(CASRefContiguousAlloc, EpochChangeRestartsTheSequenceAtOne) } /// The read side is what makes INV-1 an invariant rather than a convention: a transaction whose id is -/// not the successor of `greatest_applied` is CORRUPTED_DATA, naming both ids. Before this task the -/// state machine checked strict increase only, so a stream with a hole applied cleanly and no reader -/// could tell a complete chain from a truncated one. +/// not the successor of `greatest_applied` is CORRUPTED_DATA, naming both ids. Strict increase alone +/// admits holes, so it cannot distinguish a complete chain from a truncated one. TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) { const String ns = "srv1/contig_density"; @@ -253,7 +252,7 @@ TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) RefTableState state = replay(DB::Cas::tests::minimalLiveSnapshot(ns, RefTxnId{kEpoch, 1}), {}); ASSERT_EQ(state.getGreatestApplied(), (RefTxnId{kEpoch, 1})); - /// Strictly greater, but skips {7,2}: admitted before this task, rejected now. + /// Strictly greater, but skips {7,2}: not the required successor. try { applyRefLogTxn(state, RefLogTxn{ns, RefTxnId{kEpoch, 3}, publishCommittedOps("r", ManifestRef{1, 1, 1}), std::nullopt}); diff --git a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp index c265618996ee..3db61594d3ee 100644 --- a/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_epoch_seal_format.cpp @@ -153,7 +153,7 @@ TEST(CASRefEpochSealFormat, EncodeRejectsSealTxnWithSecondNonSealOp) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefLogTxn(txn); }); } -/// Decode-side pin for the same op-count rule (review finding I1): `encodeRefLogTxn` can never +/// Decode-side pin for the same op-count rule: `encodeRefLogTxn` can never /// produce a 2-op seal body, so only a decode-only splice proves `decodeRefLogTxn` independently /// re-derives the rule rather than trusting whatever the encoder produced -- deleting the structural /// validator's call site inside `decodeRefLogTxn` would leave this the only failing test. @@ -213,7 +213,7 @@ TEST(CASRefEpochSealFormat, EncodeRejectsPrevEpochSealAtNonUnitSequence) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { encodeRefLogTxn(txn); }); } -/// Decode-side pin for the sequence-1-only rule (review finding I1). `prev_epoch_seal`'s +/// Decode-side pin for the sequence-1-only rule. `prev_epoch_seal`'s /// writer_epoch (1) is strictly below the transaction's own (5), satisfying the I3 chain-direction /// rule, so this isolates the sequence-1 rule specifically rather than incidentally also tripping I3. TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) @@ -233,7 +233,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealAtNonUnitSequenceSpliced) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } -/// Well-formedness (review finding M2): a zero component inside `prev_epoch_seal` is rejected the +/// Well-formedness: a zero component inside `prev_epoch_seal` is rejected the /// same way a zero component in the primary `txn_id` is (`checkRefTxnIdNonzero`, shared code path). TEST(CASRefEpochSealFormat, EncodeRejectsPrevEpochSealWithZeroWriterEpoch) { @@ -276,7 +276,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealMissingPssComponent) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } -/// Chain direction (review finding I3): a seal closing epoch E always has id `{E, T+1}`, and the +/// Chain direction: a seal closing epoch E always has id `{E, T+1}`, and the /// sequence-1 transaction in the next numeric epoch must name it. This remains context-free (a /// property of one transaction), so it belongs in the structural half; Tasks 2/6 walk this pointer /// backwards over untrusted decoded bodies and must not have to re-derive the rule themselves. @@ -335,7 +335,7 @@ TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealSkippingImmediateEpochSpli expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { decodeRefLogTxn(tampered, txn.ns, txn.txn_id); }); } -/// Decode-side pin for the chain-direction rule (review finding I3): the encoder's own check would +/// Decode-side pin for the chain-direction rule: the encoder's own check would /// refuse to produce this shape (the two Encode* tests above pin that direction), so a splice into an /// otherwise-valid sequence-1 body proves decode re-derives the rule independently. TEST(CASRefEpochSealFormat, DecodeRejectsPrevEpochSealPointingAtSameOrFutureEpochSpliced) @@ -382,7 +382,7 @@ TEST(CASRefEpochSealFormat, ContextualRejectsPrevEpochSealWhenForbidden) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { validateEpochSealGrammarContextual(txn, /*life_epoch=*/5); }); } -/// codex r2 finding 2: "genesis" is per-namespace. A namespace first born at global epoch 5 (not +/// "Genesis" is per-namespace. A namespace first born at global epoch 5 (not /// epoch 1) appends {5, 1} with NO prev_epoch_seal -- that IS its genesis, not a transition. TEST(CASRefEpochSealFormat, ContextualAllowsGenesisBirthAboveEpochOneWithoutPrevEpochSeal) { @@ -393,7 +393,7 @@ TEST(CASRefEpochSealFormat, ContextualAllowsGenesisBirthAboveEpochOneWithoutPrev EXPECT_NO_THROW(validateEpochSealGrammarContextual(txn, /*life_epoch=*/5)); } -/// Review finding I2: the `ref_sequence != 1` early return is load-bearing for Task 4's encode call +/// The `ref_sequence != 1` early return is load-bearing for the encode call /// site, which calls this on every txn it mints, including ordinary sequence->=2 transactions in a /// post-transition epoch that legitimately carry no `prev_epoch_seal`. Pinned on both sides of the /// life_epoch relation to prove the early return fires regardless of it. @@ -418,7 +418,7 @@ TEST(CASRefEpochSealFormat, ContextualPassesThroughNonSequenceOneAtOrBelowLifeEp } /// =================================================================================== -/// Criticality of the prev_epoch_seal wire fields (review finding M4) +/// Criticality of the `prev_epoch_seal` wire fields /// =================================================================================== /// `!prev_epoch`/`!prev_seq` are `!`-prefixed CRITICAL keys: `prev_epoch_seal` is INV-2 chain evidence, and a diff --git a/src/Disks/tests/gtest_cas_ref_install_safety.cpp b/src/Disks/tests/gtest_cas_ref_install_safety.cpp index 3c3c58b480d8..b44ce523465c 100644 --- a/src/Disks/tests/gtest_cas_ref_install_safety.cpp +++ b/src/Disks/tests/gtest_cas_ref_install_safety.cpp @@ -107,7 +107,7 @@ PoolPtr openPoolSingleAttempt(const BackendPtr & backend) /// With `openPoolFenceControlled`'s budget below that margin is 100 + 100 = 200 ms, so a 100 ms /// remaining lease sits BETWEEN them: the flush is admitted and then its very first pre-attempt gate /// refuses. That is -/// exactly the production shape this task is about (a lease too short to start a write, not a lost +/// exactly the production shape of a lease too short to start a write, not a lost /// one), and it needs no fault injection at all -- which is the point: nothing is sent. constexpr uint64_t FENCE_DEADLINE_HEALTHY_MS = 30000; constexpr uint64_t FENCE_DEADLINE_REFUSES_ATTEMPT_MS = 100; @@ -330,8 +330,8 @@ TEST(CASRefInstallSafety, PreAttemptRefusalDoesNotWedgeTheLane) << "the refused drop must not have taken effect"; /// The availability half of the claim: the lane is usable the moment the lease is healthy again -- - /// no remount, no wedge resolution, nothing to clear. Before this task the same sequence left a - /// wedge over a key that was never written, and this append would have failed forever. + /// no remount, no wedge resolution, nothing to clear. A refused pre-attempt never writes a key, + /// so it cannot leave a wedge to block this append. store->setMountDeadline(FENCE_DEADLINE_HEALTHY_MS); store->dropRef(ns, "part_a"); EXPECT_FALSE(store->resolveRef(ns, "part_a", /*allow_stale=*/false).has_value()) diff --git a/src/Disks/tests/gtest_cas_ref_log_format.cpp b/src/Disks/tests/gtest_cas_ref_log_format.cpp index cef428ab1d87..1ef57e5005d5 100644 --- a/src/Disks/tests/gtest_cas_ref_log_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_log_format.cpp @@ -202,7 +202,7 @@ TEST(CASRefCodec, RoundTripSetPublishedAt) } /// No-tolerance decode pin: the `"pl"` (payload) field was removed from the -/// ref-op wire in stage-1 T12. Although the retired `set_payload` op WORD is already rejected by +/// ref-op wire. Although the retired `set_payload` op WORD is already rejected by /// `refOpKindFromWireWord`, the generic op-record reader reads all field keys before switching on kind, so a /// `"pl"` field paired with a still-recognized op word would otherwise be `skipUnknown`'d. It is a /// removed field, not a genuinely-unknown one: decoding an op record that still carries `"pl"` must FAIL diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp index 1c532c5add83..942fc170e139 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_format.cpp @@ -126,8 +126,7 @@ TEST(CASRefSnapshotCodec, DecodeRejectsRetiredRemoveTxnFieldPair) [&] { (void)decodeRefTableSnapshot(bytes, s.ns, s.snapshot_id); }); } -/// No-tolerance decode pin (codex round-2, finding 3): the `"pl"` (payload) field was removed from the -/// committed-row wire in stage-1 T12. It is NOT a genuinely-unknown future field the tolerant reader may +/// No-tolerance decode pin: the `"pl"` (payload) field is not a genuinely-unknown future field the tolerant reader may /// skip -- silently discarding a persisted payload would lose data -- so decoding a committed row that /// still carries `"pl"` must FAIL with `CORRUPTED_DATA` naming the removed field, not `skipUnknown` it. TEST(CASRefSnapshotCodec, DecodeRejectsRemovedPayloadFieldInCommittedRow) diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp index ca34681f7b93..341484e8f84c 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp @@ -180,7 +180,7 @@ TEST(CASRefSnapshotPublishOrdering, AdoptionHappensLastAndOnlyAfterBothDurableEf /// 3. `NeedsRecovery` ("Poisoned") lane: recovery precedes any snapshot publication /// --------------------------------------------------------------------------------------------- -/// `Poisoned` is this task's plan's name for what the code spells `RefLaneState::NeedsRecovery` -- the +/// `RefLaneState::NeedsRecovery` is the state for a transaction known durable but not installable in the cache -- the /// state the header documents as "a transaction is known durable but cannot be installed in this cache /// ... a hard write and certification fence until replay completes". Recorded here as the vocabulary /// correction for later tasks: there is no state literally named `Poisoned` anywhere in `CasRefLedger`. diff --git a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp index ace02d24836e..c696323b3d80 100644 --- a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp +++ b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp @@ -802,8 +802,8 @@ TEST(CASRefWedgeEveryAttempt, SuccessorSealAtTheWedgedKeyRejectsConclusivelyAndS } /// The wire round trip of the same rule, driven from the OTHER producer of `last_epoch_seal`: -/// recovery's CAS-walk (Task 6), stood in for here by its test seam. The point is the encode call -/// site, which is this task's. +/// recovery's CAS-walk, represented here by its test seam. The point is the encode call +/// site. TEST(CASRefWedgeEveryAttempt, OrdinaryFirstAppendAfterASealedTransitionCarriesTheExactPrevEpochSeal) { auto backend = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index 48414871269b..c4e173fb96c5 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -1711,8 +1711,7 @@ TEST(CASRefWriterAppendLane, WedgedRefLaneCountTracksExactlyTheWedgedTableThroug /// The reaction is now the mount's, not the table's [review I5]: a foreign object at a key that /// mount-lease exclusivity says is exclusively ours contradicts the exclusivity itself, so the append /// site routes through `reportImpossibleInterference` exactly as the wedge-resolve site does -- fence -/// closed, remount scheduled. Before this task it failed closed and stayed closed, blocking the table -/// until somebody remounted by hand. So there are two separate scopes to keep straight, and this test +/// closed, remount scheduled. The fence is released only after remount, so there are two separate scopes to keep straight, and this test /// pins both: /// the FENCE is mount-wide -- while it is closed EVERY lane is refused, including untouched ones; /// the DAMAGE is per-namespace -- a real remount replaces both immutable runtimes, then recovery of @@ -3947,7 +3946,7 @@ TEST(CASRefWriterNamespaceRemoval, RemovalPublishesTerminalLogWithoutTerminalSna EXPECT_EQ(terminal_logs, 1u); } -/// Review fix (prerequisite to this task's dropNamespace rewiring): `flushRefBatch`'s per-item +/// `flushRefBatch`'s per-item /// validation previously previewed each op as its OWN single-op trial transaction, so a /// whole-transaction-shape rule ("remove_namespace must be the FINAL op") trivially passed on every /// singleton slice regardless of an item's REAL combined shape -- a malformed item would only have diff --git a/utils/c++expr b/utils/c++expr index 08ddff2e1363..54c44c8ecdbd 100755 --- a/utils/c++expr +++ b/utils/c++expr @@ -235,7 +235,9 @@ size_t max_tests = $BENCHMARK_TESTS; size_t max_steps = $BENCHMARK_STEPS; $GLOBAL -int work(int thread_id = 0) { +/// Internal linkage: the ClickHouse build enables -Werror,-Wmissing-prototypes, which rejects a +/// definition of an external function that has no preceding declaration. +static int work(int thread_id = 0) { (void)thread_id; try { EOF From b55e44595e652e03f820791fcaa2abdae846e986 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 2 Sep 2026 01:03:45 +0200 Subject: [PATCH 06/81] cas: reuse the JSON object/row reader across a stream's rows; decode speedup Full before/after measurement of the wire-key cut found decode of four of the five formats barely slower, with `cas_fold_seal` the exception (its short strings make the longer keys dominate). Chasing that, the JSON object reader and the per-format row reader are now reused across a stream's rows instead of rebuilt for each one, cutting decode time 57-81% (53-79% net of the key-length cost). A separate copy-free string-read attempt was measured at a 6-7% regression on `cas_ref_catalog` and is not included here. Also lets the full stateless test suite run locally: `functional_tests.py` turns on verbose output for the dataset-attach step (so a `DNS_ERROR` that only fires outside CI doesn't get swallowed and misread as a Kafka failure downstream) and extends the "skip stateful tests when running locally" guard to a local run with no test selector at all. Co-Authored-By: Claude Fable 5 Signed-off-by: Mikhail Filimonov --- ci/jobs/functional_tests.py | 8 ++ ci/jobs/scripts/clickhouse_proc.py | 4 +- .../Formats/CasBlobEnvelopeFormat.cpp | 25 ++-- .../Formats/CasBlobEnvelopeFormat.h | 9 +- .../Formats/CasFoldSealFormat.cpp | 12 +- .../Formats/CasGcOutcomesFormat.cpp | 11 +- .../Formats/CasPartManifestFormat.cpp | 12 +- .../Formats/CasRecordStreamFormat.cpp | 25 ++-- .../Formats/CasRecordStreamFormat.h | 7 ++ .../Formats/CasRefCatalogFormat.cpp | 30 +++-- .../Formats/CasRefCkptFormat.cpp | 5 +- .../Formats/CasRefLogFormat.cpp | 11 +- .../Formats/CasRefSnapshotFormat.cpp | 12 +- .../Formats/CasTextFormat.cpp | 118 +++++++++++++----- .../ContentAddressed/Formats/CasTextFormat.h | 28 ++++- .../ContentAddressed/Formats/CasWireVocab.cpp | 21 ++-- src/Disks/tests/cas_format_test_battery.h | 15 ++- .../tests/gtest_cas_blob_envelope_format.cpp | 40 ++++-- src/Disks/tests/gtest_cas_format.cpp | 15 +++ 19 files changed, 287 insertions(+), 121 deletions(-) diff --git a/ci/jobs/functional_tests.py b/ci/jobs/functional_tests.py index b73f6f308211..227bc50bfc17 100644 --- a/ci/jobs/functional_tests.py +++ b/ci/jobs/functional_tests.py @@ -506,6 +506,14 @@ def main(): # for local run check if stateful tests are present to skip prepare_stateful_data and start faster if not has_stateful_tests = True + if info.is_local_run and not tests: + # A local run of the WHOLE suite cannot prepare the stateful datasets: `create.sql` attaches + # them from a web disk on `dockerhub-proxy.dockerhub-proxy-zone`, which resolves only inside + # CI, so the step dies with DNS_ERROR before a single test runs. Skipping it lets the + # stateless suite run locally; the tests that genuinely need `test.hits`/`test.visits` fail + # and are triaged as environment rather than taking the whole job with them. + print("Local full-suite run: skipping stateful data preparation (datasets are CI-hosted)") + has_stateful_tests = False if tests and info.is_local_run: from glob import glob diff --git a/ci/jobs/scripts/clickhouse_proc.py b/ci/jobs/scripts/clickhouse_proc.py index d4ce6313bf24..4a7740603c3e 100644 --- a/ci/jobs/scripts/clickhouse_proc.py +++ b/ci/jobs/scripts/clickhouse_proc.py @@ -839,7 +839,9 @@ def prepare_stateful_data(self, with_s3_storage, is_db_replicated): command = bootstrap_vars + command if with_s3_storage: command = "USE_S3_STORAGE_FOR_MERGE_TREE=1\n" + command - return Shell.check(command) + # verbose: this step loads the stateful datasets and it is the only place in the job + # that can fail without printing anything at all, which is exactly what happened. + return Shell.check(command, verbose=True) def insert_system_zookeeper_config(self): for _ in range(10): diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp index ff9fda182b26..523912c519c4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp @@ -35,6 +35,8 @@ namespace EnvelopeWire constexpr WireKey op{"op"}; constexpr WireKey chver{"chver"}; constexpr WireKey ref{"ref"}; + /// Not a field this build understands: written only to exercise the reader's `!`-key policy. + constexpr WireKey unknown_critical{"!x"}; } constexpr EnumWireTable kProvenanceOpWords{{{ @@ -165,7 +167,8 @@ ProvenanceOp provenanceOpFromWireWord(std::string_view w) return kProvenanceOpWords.fromWord(w, "CAS blob envelope"); } -String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) +String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len, + std::optional version_override) { if (header.kind != ObjectKind::Blob) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -177,22 +180,20 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len) { CasJsonWriter buf(256); bool first = true; - writeKey(buf, EnvelopeWire::type, first); writeStringValue(buf, kBlobType); - writeKey(buf, EnvelopeWire::version, first); writeIntText(currentCompatibilityVersion(), buf); - writeKey(buf, EnvelopeWire::tag, first); writeHex128Value(buf, header.incarnation_tag); - writeKey(buf, EnvelopeWire::build, first); writeHex128Value(buf, header.build_id); + writeStringField(buf, EnvelopeWire::type, kBlobType, first); + writeNumberField(buf, EnvelopeWire::version, version_override.value_or(currentCompatibilityVersion()), first); + writeHex128Field(buf, EnvelopeWire::tag, header.incarnation_tag, first); + writeHex128Field(buf, EnvelopeWire::build, header.build_id, first); if (header.provenance) { - writeKey(buf, EnvelopeWire::time_ms, first); writeIntText(header.provenance->created_at_ms, buf); - writeKey(buf, EnvelopeWire::creator, first); writeHex128Value(buf, header.provenance->creator_server_id); - writeKey(buf, EnvelopeWire::op, first); writeStringValue(buf, provenanceOpToWireWord(header.provenance->op)); - writeKey(buf, EnvelopeWire::chver, first); writeIntText(header.provenance->ch_version, buf); + writeNumberField(buf, EnvelopeWire::time_ms, header.provenance->created_at_ms, first); + writeHex128Field(buf, EnvelopeWire::creator, header.provenance->creator_server_id, first); + writeStringField(buf, EnvelopeWire::op, provenanceOpToWireWord(header.provenance->op), first); + writeNumberField(buf, EnvelopeWire::chver, header.provenance->ch_version, first); } /// Test-only critical extension: an unknown `!`-key BEFORE `ref`. if (header.emit_unknown_critical_key) - { - writeKey(buf, "!x", first); writeStringValue(buf, "1"); - } + writeStringField(buf, EnvelopeWire::unknown_critical, "1", first); json = std::move(buf).take(); /// e.g. {"type":"cas_blob","v":1,...,"chver":26006001 (no ref, no closing brace) } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h index e1a9a228433d..e968bbec5dbc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.h @@ -94,7 +94,14 @@ struct EnvelopeHeader /// diagnostic `ref` is the only truncatable field and is shortened, never dropped, when necessary to /// preserve the fixed layout. The header is built without payload bytes, so an upload can stage the /// header before the payload is streamed. -String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len); +/// `version_override` exists for one caller: the boundary test that has to see what the descriptor +/// costs at the WIDEST version the budget reserves room for. The budget is sized for a ten-digit +/// version; production has only ever written a one-digit one, so a test that encodes at the current +/// version and adds the missing digits arithmetically never sends the boundary through the encoder +/// at all -- it re-derives the formula it is supposed to be checking. Production passes nothing and +/// gets `currentCompatibilityVersion()`. +String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len, + std::optional version_override = {}); /// Parses and validates the JSON descriptor, its expected `type`, and its compatibility version. /// Derives `header_len` from the terminating '\n' and requires every preceding byte in the pad zone to diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp index 77c3d1dcece4..b2c0bf1f9756 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp @@ -391,11 +391,17 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional expect } uint64_t seen = 0; + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "fold seal"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Strict, "fold seal"); + readLineInto(in, row_line, line_cap, "fold seal"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Strict, "fold seal"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp index 9ae1ffd6282d..2f9aa0e141d2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp @@ -71,11 +71,16 @@ OutcomeLog decodeOutcomeLog(std::string_view data) const uint64_t line_cap = traitsFor(FormatId::GcOutcomes).line_cap; OutcomeLog log; + /// One line scratch and one reader for the whole loop, as the other row decoders do: + /// rebuilding them per row costs an allocation per row for the seen-key store and the line. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "outcome log"); - ReadBufferFromMemory line_in(line.data(), line.size()); - JsonObjectReader r(line_in, KeyStrictness::Tolerant, "outcome log"); + readLineInto(in, row_line, line_cap, "outcome log"); + ReadBufferFromMemory line_in(row_line.data(), row_line.size()); + row_reader.reset(line_in, KeyStrictness::Tolerant, "outcome log"); + JsonObjectReader & r = row_reader; String key; /// The first key distinguishes a trailer (`n`) from a record (`kind`). diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp index ac70c2c00b23..e0722add9bdc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp @@ -176,11 +176,17 @@ PartManifest decodePartManifest(std::string_view data) /// Index-aligned with `m.entries` (Blob entries push an unused 0 placeholder). std::vector inline_lens; String blob_ref_what; /// reused across Blob entries so the error context does not allocate per row + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "cas_part_manifest"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Tolerant, "cas_part_manifest"); + readLineInto(in, row_line, line_cap, "cas_part_manifest"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Tolerant, "cas_part_manifest"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp index 00d20988ac03..78dedf30a4b2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.cpp @@ -31,6 +31,13 @@ namespace RunWire constexpr WireKey confirmed{"confirmed"}; } +namespace RunHeaderWire +{ + constexpr WireKey type{"type"}; + constexpr WireKey version{"v"}; + constexpr WireKey kind{"kind"}; +} + constexpr EnumWireTable kRunMarkerWords{{{ {RunMarker::Zero, "zero"}, {RunMarker::Edge, "edge"}, @@ -119,12 +126,9 @@ void writeRunHeaderLine(WriteBuffer & out, std::string_view kind) const FormatTraits & t = traitsFor(FormatId::RunFile); CasJsonWriter line(64); bool first = true; - writeKey(line, "type", first); - writeStringValue(line, t.type); - writeKey(line, "v", first); - writeIntText(currentCompatibilityVersion(), line); - writeKey(line, "kind", first); - writeStringValue(line, kind); + writeStringField(line, RunHeaderWire::type, t.type, first); + writeNumberField(line, RunHeaderWire::version, currentCompatibilityVersion(), first); + writeStringField(line, RunHeaderWire::kind, kind, first); closeObject(line, first); writeChar('\n', line); const std::string_view line_view = line.view(); @@ -242,9 +246,12 @@ bool SourceEdgeRunReader::next(SourceEdgeRecord & rec) if (done) return false; - const String line = readLine(hashing, traitsFor(FormatId::RunFile).line_cap, "cas_run"); - ReadBufferFromMemory line_in(line.data(), line.size()); - JsonObjectReader r(line_in, KeyStrictness::Strict, "cas_run"); + readLineInto(hashing, scratch, traitsFor(FormatId::RunFile).line_cap, "cas_run"); + ReadBufferFromMemory line_in(scratch.data(), scratch.size()); + /// Re-point the reader rather than building one per row: a fresh reader re-allocates its + /// seen-key store and value scratch every row, and this loop runs once per record. + reader.reset(line_in, KeyStrictness::Strict, "cas_run"); + JsonObjectReader & r = reader; String key; if (!r.nextKey(key)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index fcc216b121ee..dbf9fd8fc9d8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -1,6 +1,7 @@ #pragma once #include #include +#include #include #include #include @@ -187,6 +188,12 @@ class SourceEdgeRunReader HashingReadBuffer hashing; uint64_t seen = 0; bool done = false; + /// Reused line scratch, mirroring the writer's: `readLineInto` clears it without releasing its + /// buffer, so a run of any length allocates only up to the longest line it has actually seen. + String scratch; + /// Reused object reader, for the same reason: its per-object buffers then cost one allocation + /// for the whole run rather than one per row. It starts unbound and every row re-points it. + JsonObjectReader reader; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp index f2cea307342c..6cf44651508e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCatalogFormat.cpp @@ -147,19 +147,17 @@ String encodeRefCatalog(const RefCatalog & catalog) e.ns.string(), nsStateToWord(e.state), e.removal_started_round ? "carries" : "lacks"); bool first = true; - writeKey(out, RefCatalogWire::kind, first); writeStringValue(out, kEntryTag); - writeKey(out, RefCatalogWire::ns, first); writeStringValue(out, e.ns.string()); - writeKey(out, RefCatalogWire::state, first); writeStringValue(out, nsStateToWord(e.state)); - writeKey(out, RefCatalogWire::life, first); writeHex128Value(out, e.incarnation); + writeStringField(out, RefCatalogWire::kind, kEntryTag, first); + writeStringField(out, RefCatalogWire::ns, e.ns.string(), first); + writeStringField(out, RefCatalogWire::state, nsStateToWord(e.state), first); + writeHex128Field(out, RefCatalogWire::life, e.incarnation, first); if (e.removal_started_round) - { - writeKey(out, RefCatalogWire::remove_round, first); writeU64StringValue(out, *e.removal_started_round); - } + writeU64StringField(out, RefCatalogWire::remove_round, *e.removal_started_round, first); if (e.creator) { - writeKey(out, RefCatalogWire::creator, first); writeStringValue(out, e.creator->server_root_id); - writeKey(out, RefCatalogWire::creator_epoch, first); writeU64StringValue(out, e.creator->writer_epoch); - writeKey(out, RefCatalogWire::creator_fence, first); writeU64StringValue(out, e.creator->fence_generation); + writeStringField(out, RefCatalogWire::creator, e.creator->server_root_id, first); + writeU64StringField(out, RefCatalogWire::creator_epoch, e.creator->writer_epoch, first); + writeU64StringField(out, RefCatalogWire::creator_fence, e.creator->fence_generation, first); } closeObject(out, first); closeLine("entry"); @@ -180,11 +178,17 @@ RefCatalog decodeRefCatalog(std::string_view data) RefCatalog catalog; uint64_t seen = 0; + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; for (;;) { - const String line = readLine(in, line_cap, "ref catalog"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Strict, "ref catalog"); + readLineInto(in, row_line, line_cap, "ref catalog"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Strict, "ref catalog"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp index dbc93315a99a..4034f5c244c1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefCkptFormat.cpp @@ -105,10 +105,7 @@ String encodeRefCkpt(const RefCkpt & ckpt) /// written by the one shared `RefTxnId` writer the `_log` and `_snap` formats also use, so the /// three ref formats cannot disagree on the encoding. if (ckpt.life_epoch) - { - writeKey(out, RefCkptWire::life_epoch, first); - writeU64StringValue(out, *ckpt.life_epoch); - } + writeU64StringField(out, RefCkptWire::life_epoch, *ckpt.life_epoch, first); if (ckpt.committed_through) writeRefTxnIdFields(out, first, RefCkptWire::committed_epoch, RefCkptWire::committed_seq, *ckpt.committed_through); if (ckpt.checkpoint_snapshot_id) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp index 8d60260c8b53..8c9d010ee678 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefLogFormat.cpp @@ -365,11 +365,16 @@ RefLogTxn decodeRefLogTxn(std::string_view data, const String & expected_ns, con expected_ns, expected_txn_id.writer_epoch, expected_txn_id.ref_sequence); /// op record lines, until the trailer + /// One line scratch and one reader for the whole loop, as the other row decoders do: + /// rebuilding them per row costs an allocation per row for the seen-key store and the line. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "cas_ref_log"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Tolerant, "cas_ref_log"); + readLineInto(in, row_line, line_cap, "cas_ref_log"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Tolerant, "cas_ref_log"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefLogTxn: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp index 04ecef15cd53..8d8b9e6f8244 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRefSnapshotFormat.cpp @@ -180,11 +180,17 @@ RefTableSnapshot decodeRefTableSnapshot( } /// record lines (committed then precommit), until the trailer + /// One line scratch and one reader for the whole loop: a decoder that rebuilds them per + /// row pays an allocation per row for the seen-key store and the line, which profiling put + /// at about a fifth of the instructions executed inside a row. + String row_line; + JsonObjectReader row_reader; while (true) { - const String line = readLine(in, line_cap, "cas_ref_snap"); - ReadBufferFromMemory l(line.data(), line.size()); - JsonObjectReader r(l, KeyStrictness::Tolerant, "cas_ref_snap"); + readLineInto(in, row_line, line_cap, "cas_ref_snap"); + ReadBufferFromMemory l(row_line.data(), row_line.size()); + row_reader.reset(l, KeyStrictness::Tolerant, "cas_ref_snap"); + JsonObjectReader & r = row_reader; String key; if (!r.nextKey(key)) throw Exception(ErrorCodes::CORRUPTED_DATA, "RefTableSnapshot: empty line"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp index d2f25aab615f..aefa25c871c6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -166,13 +167,36 @@ auto JsonObjectReader::guarded(F && f) } JsonObjectReader::JsonObjectReader(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_) - : in(in_), strictness(strictness_), what(what_) + : in(&in_), strictness(strictness_), what(what_) { - guarded([&] { assertChar('{', in); }); + guarded([&] { assertChar('{', *in); }); +} + +void JsonObjectReader::reset(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_) +{ + in = &in_; + strictness = strictness_; + what = what_; + /// `clear` on both keeps their buffers: that is the whole point of reusing the reader. + seen_keys.clear(); + scratch.clear(); + first = true; + done = false; + guarded([&] { assertChar('{', *in); }); +} + +std::string_view JsonObjectReader::readStringIntoScratch() +{ + scratch.clear(); + readJSONStringInto(scratch, *in, jsonReadSettings()); + return scratch; } bool JsonObjectReader::nextKey(String & key) { + /// A default-constructed reader is unbound until `reset`; using one is a programming error, so + /// this belongs in the debug build rather than as a branch on the decode hot path. + chassert(in != nullptr); return guarded([&]() -> bool { if (done) @@ -180,7 +204,7 @@ bool JsonObjectReader::nextKey(String & key) if (first) { first = false; - if (checkChar('}', in)) + if (checkChar('}', *in)) { done = true; return false; @@ -188,15 +212,15 @@ bool JsonObjectReader::nextKey(String & key) } else { - if (checkChar('}', in)) + if (checkChar('}', *in)) { done = true; return false; } - assertChar(',', in); + assertChar(',', *in); } - readJSONString(key, in, jsonReadSettings()); - assertChar(':', in); + readJSONString(key, *in, jsonReadSettings()); + assertChar(':', *in); if (std::find(seen_keys.begin(), seen_keys.end(), key) != seen_keys.end()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: duplicate key '{}'", what, key); seen_keys.push_back(key); @@ -209,7 +233,7 @@ String JsonObjectReader::readString() return guarded([&] { String s; - readJSONString(s, in, jsonReadSettings()); + readJSONString(s, *in, jsonReadSettings()); return s; }); } @@ -219,18 +243,18 @@ std::vector JsonObjectReader::readStringArray() return guarded([&] { std::vector words; - assertChar('[', in); - if (checkChar(']', in)) + assertChar('[', *in); + if (checkChar(']', *in)) return words; while (true) { String word; - readJSONString(word, in, jsonReadSettings()); + readJSONString(word, *in, jsonReadSettings()); words.push_back(std::move(word)); - if (checkChar(']', in)) + if (checkChar(']', *in)) return words; - assertChar(',', in); + assertChar(',', *in); } }); } @@ -239,7 +263,7 @@ UInt128 JsonObjectReader::readHex128() { return guarded([&] { - const String hex = readString(); + const std::string_view hex = readStringIntoScratch(); if (hex.size() != 32 || std::any_of(hex.begin(), hex.end(), [](char c) { return !isLowercaseHexChar(c); })) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: expected 32 lowercase hex chars, got '{}'", what, hex); return unhexUInt(hex.data()); @@ -250,7 +274,7 @@ uint64_t JsonObjectReader::readU64String() { return guarded([&] { - const String s = readString(); + const std::string_view s = readStringIntoScratch(); ReadBufferFromMemory buf(s.data(), s.size()); uint64_t v = 0; readIntText(v, buf); @@ -265,7 +289,7 @@ uint64_t JsonObjectReader::readU64Number() return guarded([&] { uint64_t v = 0; - readIntText(v, in); + readIntText(v, *in); return v; }); } @@ -282,9 +306,9 @@ bool JsonObjectReader::readBool() { return guarded([&] { - if (checkString("true", in)) + if (checkString("true", *in)) return true; - assertString("false", in); + assertString("false", *in); return false; }); } @@ -298,20 +322,28 @@ void JsonObjectReader::skipUnknown(const String & key) "CAS {}: critical key '{}' is not understood by this build", what, key); if (strictness == KeyStrictness::Strict) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown key '{}' in a strict format", what, key); - skipJSONField(in, key, jsonReadSettings()); + skipJSONField(*in, key, jsonReadSettings()); }); } /// ---- header line / trailer line / raw line access ---- +namespace +{ +namespace ContainerWire +{ + constexpr WireKey type{"type"}; + constexpr WireKey version{"v"}; + constexpr WireKey count{"n"}; +} +} + void writeHeaderLine(CasJsonWriter & out, FormatId id) { const FormatTraits & t = traitsFor(id); bool first = true; - writeKey(out, "type", first); - writeStringValue(out, t.type); - writeKey(out, "v", first); - writeIntText(currentCompatibilityVersion(), out); + writeStringField(out, ContainerWire::type, t.type, first); + writeNumberField(out, ContainerWire::version, currentCompatibilityVersion(), first); closeObject(out, first); writeChar('\n', out); } @@ -319,29 +351,49 @@ void writeHeaderLine(CasJsonWriter & out, FormatId id) void writeTrailerLine(CasJsonWriter & out, uint64_t n) { bool first = true; - writeKey(out, "n", first); - writeIntText(n, out); + writeNumberField(out, ContainerWire::count, n, first); closeObject(out, first); writeChar('\n', out); } -String readLine(ReadBuffer & in, uint64_t line_cap, std::string_view what) +void readLineInto(ReadBuffer & in, String & line, uint64_t line_cap, std::string_view what) { - String line; + /// `clear` keeps the capacity, so a caller that reuses one scratch across a stream's rows stops + /// allocating after the longest line it has seen -- the same bound the writer's scratch has. + line.clear(); while (true) { if (in.eof()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: truncated object (line without terminator)", what); - const char c = *in.position(); - ++in.position(); - if (c == '\n') - return line; - line.push_back(c); - if (line.size() > line_cap) + + /// Take the whole run up to the terminator in one append rather than a byte at a time: a + /// per-character `push_back` also re-checks the cap on every character, and a stream row is + /// hundreds of characters long. + const char * const from = in.position(); + const char * const found = find_first_symbols<'\n'>(from, in.buffer().end()); + const size_t taken = static_cast(found - from); + if (line.size() + taken > line_cap) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: line exceeds the {}-byte cap", what, line_cap); + line.append(from, taken); + in.position() += taken; + + /// `find_first_symbols` stops either at the terminator or at the end of the buffered window. + /// Only the first case ends the line; the second needs the next window. + if (found != in.buffer().end()) + { + ++in.position(); + return; + } } } +String readLine(ReadBuffer & in, uint64_t line_cap, std::string_view what) +{ + String line; + readLineInto(in, line, line_cap, what); + return line; +} + namespace { TextHeader parseHeaderObject(std::string_view line, std::string_view what) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h index bd353f864d17..8bbbf58a4efd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasTextFormat.h @@ -235,6 +235,19 @@ class JsonObjectReader public: /// Consumes the opening `{`; throws `CORRUPTED_DATA` when the object does not start there. JsonObjectReader(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_); + + /// An unbound reader, for a decoder that wants one reader outside its row loop and re-points it + /// per row. `reset` must be called before any read; nothing else is valid on it. + JsonObjectReader() = default; + + /// Re-point an existing reader at another object, as the constructor would, but WITHOUT + /// releasing the buffers it has already grown. A stream decoder reads one object per row, and a + /// reader built fresh each time re-allocates its seen-key store and its value scratch on every + /// row; measured on the `cas_run` decode path, allocation accounting is about a fifth of all + /// instructions executed inside the decoder. Reusing one reader amortises that away. The + /// object-level state -- the key set and the position in the object -- is reset in full, so a + /// reused reader accepts and rejects exactly what a fresh one would. + void reset(ReadBuffer & in_, KeyStrictness strictness_, std::string_view what_); /// Advances to the next key; false when the closing '}' was consumed. The caller must /// consume the value (one read* / skipUnknown) before the next call. Duplicate keys are /// rejected with `CORRUPTED_DATA`. @@ -263,10 +276,16 @@ class JsonObjectReader template auto guarded(F && f); - ReadBuffer & in; - KeyStrictness strictness; + /// Reads one JSON string into `scratch` and returns a view of it, so a value that is parsed and + /// discarded -- a hex digest, a decimal counter -- costs no allocation once the scratch has + /// grown. The view is valid until the next read on this reader. + std::string_view readStringIntoScratch(); + + ReadBuffer * in = nullptr; + KeyStrictness strictness = KeyStrictness::Strict; String what; std::vector seen_keys; + String scratch; bool first = true; bool done = false; }; @@ -288,6 +307,11 @@ TextHeader expectHeaderLine(ReadBuffer & in, FormatId id); std::optional sniffHeaderLine(std::string_view bytes); /// Reads one line (excluding the '\n' terminator); CORRUPTED_DATA on missing terminator or a line /// longer than `line_cap`. +/// Read one terminator-delimited line into `line`, replacing its contents and KEEPING its capacity, +/// so a caller streaming many rows can reuse one scratch and stop allocating after the longest line. +void readLineInto(ReadBuffer & in, String & line, uint64_t line_cap, std::string_view what); + +/// Allocating form, for callers that read a single line. String readLine(ReadBuffer & in, uint64_t line_cap, std::string_view what); /// Position of the next byte `stringValue` treats specially (control byte, '"', '\\', or the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 2b30d730aba5..39191bad1a43 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -47,28 +47,21 @@ ObjectKind objectKindFromWord(std::string_view w, std::string_view what) void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t) { - writeKey(out, SharedWire::token_type, first); - writeStringValue(out, tokenTypeToWord(t.type)); - writeKey(out, SharedWire::token, first); - writeStringValue(out, t.value); + writeStringField(out, SharedWire::token_type, tokenTypeToWord(t.type), first); + writeStringField(out, SharedWire::token, t.value, first); } void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r) { - writeKey(out, SharedWire::algo, first); - writeStringValue(out, blobHashAlgoName(r.algo)); - writeKey(out, SharedWire::digest, first); - writeStringValue(out, codecFor(r.algo).toHex(r.digest)); + writeStringField(out, SharedWire::algo, blobHashAlgoName(r.algo), first); + writeStringField(out, SharedWire::digest, codecFor(r.algo).toHex(r.digest), first); } void writeManifestRefFields(CasJsonWriter & out, bool & first, const ManifestRefWireKeys & keys, const ManifestRef & r) { - writeKey(out, keys.epoch, first); - out.u64StringValue(r.writer_epoch); - writeKey(out, keys.build, first); - out.u64StringValue(r.build_sequence); - writeKey(out, keys.ord, first); - out.u64Number(r.manifest_ordinal); + writeU64StringField(out, keys.epoch, r.writer_epoch, first); + writeU64StringField(out, keys.build, r.build_sequence, first); + writeNumberField(out, keys.ord, r.manifest_ordinal, first); } ManifestRef manifestRefFromFields(uint64_t writer_epoch, uint64_t build_sequence, uint64_t manifest_ordinal, diff --git a/src/Disks/tests/cas_format_test_battery.h b/src/Disks/tests/cas_format_test_battery.h index 361947968f3c..efed9ce44a74 100644 --- a/src/Disks/tests/cas_format_test_battery.h +++ b/src/Disks/tests/cas_format_test_battery.h @@ -29,12 +29,19 @@ struct FormatBatteryCase std::function make_future_version = {}; }; -/// Canonical object headers track the current compatibility generation. The type remains an -/// explicit test literal at every call site, so a registry/type mismatch cannot be hidden by a -/// self-derived expectation. +/// The canonical object header, spelled literally. +/// +/// The version is the LITERAL 1, not `currentCompatibilityVersion()`. Deriving it from production +/// was the defect: encoder output and expected bytes would then move together across a generation +/// bump, and a golden that tracks the code it is meant to pin cannot fail. The type was already a +/// literal at every call site for the same reason; the version had been left behind. +/// +/// A future generation bump is therefore SUPPOSED to break every test that uses this. That is the +/// point: the new bytes get read, agreed to, and written down, rather than being adopted silently. +/// `HeaderVersionIsTheLiteralThisBatteryPins` below fails first and says so. inline String currentFormatHeader(std::string_view type) { - return fmt::format("{{\"type\":\"{}\",\"v\":{}}}\n", type, DB::Cas::currentCompatibilityVersion()); + return fmt::format("{{\"type\":\"{}\",\"v\":1}}\n", type); } namespace cas_battery_detail diff --git a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp index 98bd9ef57ac3..edfa45abf828 100644 --- a/src/Disks/tests/gtest_cas_blob_envelope_format.cpp +++ b/src/Disks/tests/gtest_cas_blob_envelope_format.cpp @@ -186,25 +186,39 @@ TEST(CASBlobEnvelopeFormat, MandatoryWorstCaseBoundary) /// without moving the encoder and lands here. TEST(CASBlobEnvelopeFormat, WorstCaseFormulaMatchesTheEncoder) { + /// Drive the encoder at the WIDEST version the budget reserves room for, rather than encoding at + /// today's one-digit version and adding the missing digits by arithmetic. Doing the arithmetic + /// here would re-derive the very formula this test exists to check, and would never send the + /// ten-digit boundary through the encoder's own number formatting. EnvelopeHeader h = maxReachableHeader(""); - const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen)); + const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen), + std::numeric_limits::max()); + /// The mandatory shape is everything up to and including the closing brace, plus the newline the /// encoder reserves at the last byte; the padding between them is the ref budget this measures. const size_t json_len = head.find_last_not_of(' ', kMinBlobHeaderLen - 2) + 1; - const size_t mandatory_at_current_version = json_len + 1; /// + the reserved '\n' + const size_t mandatory_at_max_version = json_len + 1; /// + the reserved '\n' - size_t version_digits = 0; - for (uint32_t v = currentCompatibilityVersion(); ; v /= 10) - { - ++version_digits; - if (v < 10) - break; - } - const size_t unused_version_digits = std::numeric_limits::digits10 + 1 - version_digits; + EXPECT_EQ(mandatory_at_max_version, mandatory_descriptor_worst_case) + << "the formula and the encoder disagree about the mandatory descriptor at the widest " + "version: encoder wrote " << mandatory_at_max_version << " bytes, formula says " + << mandatory_descriptor_worst_case; - EXPECT_EQ(mandatory_at_current_version + unused_version_digits, mandatory_descriptor_worst_case) - << "the formula and the encoder disagree about the mandatory descriptor: encoder wrote " - << mandatory_at_current_version << " bytes at a " << version_digits << "-digit version"; + /// And the whole point of the budget: even at that width one byte remains spare under the floor. + EXPECT_LE(mandatory_descriptor_worst_case, kMinBlobHeaderLen - 1); +} + +/// The version really is rendered at its full width by the encoder above, not merely accounted for. +/// Without this, an encoder that silently clamped or dropped the override would still satisfy the +/// equality it feeds. +TEST(CASBlobEnvelopeFormat, MaxWidthVersionIsActuallyRendered) +{ + EnvelopeHeader h = maxReachableHeader(""); + const String head = encodeEnvelopeHeader(h, static_cast(kMinBlobHeaderLen), + std::numeric_limits::max()); + EXPECT_NE(head.find("\"v\":4294967295"), String::npos) + << "the max-width version was not rendered; the boundary above proves nothing. Header: " + << head; } TEST(CASBlobEnvelopeFormat, CriticalKeyDescriptorStillFitsAtDefaultLength) diff --git a/src/Disks/tests/gtest_cas_format.cpp b/src/Disks/tests/gtest_cas_format.cpp index 6c5ad84cc0cc..b3236797176c 100644 --- a/src/Disks/tests/gtest_cas_format.cpp +++ b/src/Disks/tests/gtest_cas_format.cpp @@ -84,3 +84,18 @@ TEST(CASFormat, CheckCompatibilityFailsClosedOnFuture) EXPECT_EQ(e.code(), DB::ErrorCodes::UNKNOWN_FORMAT_VERSION); } } + +/// The battery's goldens spell their header version as the literal 1 rather than asking production +/// for it, so that a generation bump cannot move the expectation and the encoder output together. +/// The cost of that is a golden set which goes stale silently if nobody notices the bump; this test +/// is what notices. It fails FIRST and says what to do, so the failure that greets a generation bump +/// is one explanatory test rather than every exact-encoding golden at once. +TEST(CASFormat, HeaderVersionIsTheLiteralThisBatteryPins) +{ + EXPECT_EQ(currentCompatibilityVersion(), 1u) + << "The compatibility version has moved away from the literal 1 that " + "`currentFormatHeader` in cas_format_test_battery.h writes into every golden header. " + "That is a deliberate wire change: read the new bytes, agree to them, and update the " + "literal and the goldens together. Do NOT make the helper derive the version from " + "production again -- a golden that tracks the code it pins cannot fail."; +} From acede29cd44e448b12f4b8840171123191581a02 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 31 Aug 2026 13:37:57 +0200 Subject: [PATCH 07/81] docs: recommend single-replica merges for `CAS` Signed-off-by: Mikhail Filimonov --- docs/en/antalya/cas/index.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/en/antalya/cas/index.md b/docs/en/antalya/cas/index.md index 2bc71046494b..cb1d563eebf0 100644 --- a/docs/en/antalya/cas/index.md +++ b/docs/en/antalya/cas/index.md @@ -64,6 +64,14 @@ Two consequences for planning: Each prefix is a fully independent pool (its own refs, leases, and `GC`), so rounds stay short regardless of the total fleet size. +:::tip +For replicated tables on `CAS`, enable +[`execute_merges_on_single_replica_time_threshold`](/operations/settings/merge-tree-settings#execute_merges_on_single_replica_time_threshold). +This lets one replica perform each merge while the others wait for and fetch the resulting part, +avoiding redundant merge work across replicas. Set the threshold higher than the usual merge +duration for your workload. +::: + ## Status {#status} `CAS` is **experimental**. It ships in Altinity Antalya builds. Experimental means the on-disk From f7c64c8cbb4b92e7e4cda7f0f8e903b395a97913 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 31 Aug 2026 19:56:11 +0200 Subject: [PATCH 08/81] Allow EXPORT PARTITION from a source on a CAS disk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ALTER TABLE ... EXPORT PARTITION` was refused with `SUPPORT_IS_DISABLED` on a content-addressed disk because it is absent from the partition-command allowlist in `MergeTreeData`. The rejection it fell into says the command "clones parts file-by-file with no transaction, which would corrupt the clone", and that reason does not describe exporting. `ExportPartTask` reads the source part through `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) under `readLockParts` and writes rows into the destination through a `SinkToStorage` on an ordinary query pipeline. Nothing is hard-linked or copied on the source disk; the command's own bookkeeping is in ZooKeeper. So the allowlist was rejecting it by omission rather than by an argument that applies to it, which the code around it already half concedes: `EXPORT_PARTITION` is listed among the commands permitted to target `PARTITION ALL` a few lines above. Verified end to end rather than by inspection, on a server built from this change: a `ReplicatedMergeTree` source on a CAS disk holding (1,2020), (2,2020), (3,2021), exported to an `IcebergLocal` destination. `EXPORT PARTITION ID '2020'` succeeds and the destination holds exactly (1,2020) and (2,2020) — the right partition, and the 2021 row correctly absent. Two limitations surfaced on the way and are NOT addressed here, because neither is about CAS. Export is implemented only for `ReplicatedMergeTree`: a plain `MergeTree` source now returns `Code: 48 NOT_IMPLEMENTED` instead of the CAS refusal, so the reproduction in the report — which uses a plain MergeTree — will still fail, just for its real reason. And the operation remains behind the server setting `allow_experimental_export_merge_tree_partition`. Closes: https://github.com/Altinity/ClickHouse/issues/2291 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KiHKrvEVy8u4nA1A8qYFUY Signed-off-by: Mikhail Filimonov --- src/Storages/MergeTree/MergeTreeData.cpp | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/Storages/MergeTree/MergeTreeData.cpp b/src/Storages/MergeTree/MergeTreeData.cpp index a507f0695d4c..e60e7fbd1a87 100644 --- a/src/Storages/MergeTree/MergeTreeData.cpp +++ b/src/Storages/MergeTree/MergeTreeData.cpp @@ -6848,6 +6848,14 @@ void MergeTreeData::checkAlterPartitionIsPossible( /// `FORGET PARTITION` is SUPPORTED on CA — it only manipulates ZooKeeper partition metadata /// (removes block-number nodes from ZooKeeper) and does not write, clone, or touch any part /// files on disk, so it is safe on a content-addressed disk. + /// `EXPORT PARTITION` is SUPPORTED because the reason this list exists does not apply + /// to it. The rejection below is about commands that clone parts file-by-file; + /// exporting does not clone at all. `ExportPartTask` reads the source part through + /// `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) and writes + /// rows into the destination through a `SinkToStorage` on an ordinary query + /// pipeline, so the source's part files are only READ, under `readLockParts`, and + /// nothing is hard-linked or copied on the content-addressed disk. The command's + /// own bookkeeping is ZooKeeper-side. /// NOTE: `MOVE_PARTITION` also admits cross-disk /// `MOVE ... TO DISK/VOLUME` (this check cannot distinguish the destination); that uses /// the byte-copy `clonePart` path (NOT the corrupting per-file hardlink), but only @@ -6864,6 +6872,7 @@ void MergeTreeData::checkAlterPartitionIsPossible( PartitionCommand::FREEZE_ALL_PARTITIONS, PartitionCommand::UNFREEZE_PARTITION, PartitionCommand::UNFREEZE_ALL_PARTITIONS, + PartitionCommand::EXPORT_PARTITION, }; if (!std::ranges::contains(supported_commands, command.type)) From 572c10a99bcae99149bd04a1daa52b9369a41182 Mon Sep 17 00:00:00 2001 From: Konstantin Morozov Date: Wed, 2 Sep 2026 18:10:40 +0200 Subject: [PATCH 09/81] add settings Signed-off-by: Konstantin Morozov --- docs/en/antalya/cas/architecture/mounts-and-leases.md | 4 ++-- docs/en/antalya/cas/configuration.md | 2 ++ .../ContentAddressed/ContentAddressedMetadataStorage.cpp | 6 ++++++ .../ContentAddressed/ContentAddressedMetadataStorage.h | 2 ++ .../ContentAddressed/ContentAddressedSettings.cpp | 2 ++ 5 files changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index d778756ce323..71abeb9c749b 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -85,8 +85,8 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve controller checks that one configured attempt still fits before each backend `PUT` or resolving `GET`, after each interruptible backoff, and before accepting success. A retry, `GET`, response timestamp, or wall-clock step never extends authority. -- **Cadence.** The runtime normally starts a logical renewal every `mount_renew_period` (default - 10 s), with TTL `mount_lease_ttl_ms` (default 30 s, TTL/3 renewal ratio). The next beat is anchored +- **Cadence.** The runtime normally starts a logical renewal every `cas_mount_renew_period_ms` (default + 10 s), with TTL `cas_mount_lease_ttl_ms` (default 30 s, TTL/3 renewal ratio). The next beat is anchored at the committed body's pre-I/O BOOTTIME start. A slow recovery therefore causes an immediate catch-up beat when the nominal cadence has elapsed; it does not wait a fresh full period after the response. diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index d7ee12130993..7d52106692cd 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -92,6 +92,8 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_blob_hash` | `cityhash128` | Pool blob content-hash function (`cityhash128` \| `xxh3-128` \| `sha256`). Recorded in the pool at creation; a mismatching config is refused at mount | | `cas_blob_hash_allow_new` | `false` | Explicit opt-in to admit a new hash algorithm into an existing pool. One-way: once admitted, the pool carries both algorithms permanently | | `skip_access_check` | `false` | Skip the boot-time capability probe (start now, fix later). Only the preflight probe is skipped — the conditional-write correctness check still runs on every writable mount. **Not available on a writable generation-token (GCS) disk**, which refuses to mount with it: there, the probe battery is the only proof that a token-exact delete carries its generation precondition. Mount such a disk read-only if you need to defer the check | +| `cas_mount_lease_ttl_ms` | `30000` | Milliseconds for which a mount lease remains valid after a successful claim or renewal. Lower values shorten stale-mount recovery but reduce tolerance for object-storage and scheduling delays | +| `cas_mount_renew_period_ms` | `10000` | Milliseconds between background mount-lease renewals. It must leave enough time for one request attempt and the lease safety margin before the TTL expires | | `cas_gc_snapshot_generations_to_keep` | `3` | GC snapshot generations retained | | `cas_gc_shards` | `1` | Blob-hash-prefix reducer shards (≥ 1). Recorded in the pool at creation; a mismatching config is refused at mount | | `gcs_max_conditional_put_bytes` | 1 GiB | Largest conditional non-blob `PUT` on a generation-token store, including create-if-absent metadata/control artifacts and conditional replacements. Blob publication is unconditional, uses ordinary multipart, and is not subject to this cap | diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index c9cf2b166389..b585b3b3e6dd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -77,6 +77,8 @@ namespace ContentAddressedSetting extern const ContentAddressedSettingsUInt64 gc_round_prefix_wholesale_budget; extern const ContentAddressedSettingsUInt64 gc_round_handoff_prefix_wholesale_budget; extern const ContentAddressedSettingsUInt64 gc_round_outcome_entry_budget; + extern const ContentAddressedSettingsUInt64 mount_lease_ttl_ms; + extern const ContentAddressedSettingsUInt64 mount_renew_period_ms; extern const ContentAddressedSettingsUInt64 part_folder_cache_bytes; extern const ContentAddressedSettingsUInt64 part_folder_cache_max_entries; extern const ContentAddressedSettingsUInt64 part_folder_cache_max_entry_bytes; @@ -294,6 +296,8 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage( , gc_round_prefix_wholesale_budget(settings_[ContentAddressedSetting::gc_round_prefix_wholesale_budget].value) , gc_round_handoff_prefix_wholesale_budget(settings_[ContentAddressedSetting::gc_round_handoff_prefix_wholesale_budget].value) , gc_round_outcome_entry_budget(settings_[ContentAddressedSetting::gc_round_outcome_entry_budget].value) + , mount_lease_ttl(std::chrono::milliseconds(settings_[ContentAddressedSetting::mount_lease_ttl_ms].value)) + , mount_renew_period(std::chrono::milliseconds(settings_[ContentAddressedSetting::mount_renew_period_ms].value)) , cas_part_folder_cache_bytes(settings_[ContentAddressedSetting::part_folder_cache_bytes].value) , cas_part_folder_cache_max_entries(settings_[ContentAddressedSetting::part_folder_cache_max_entries].value) , cas_part_folder_cache_max_entry_bytes(settings_[ContentAddressedSetting::part_folder_cache_max_entry_bytes].value) @@ -790,6 +794,8 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.gc_round_handoff_prefix_wholesale_budget = gc_round_handoff_prefix_wholesale_budget; pool_config.gc_round_outcome_entry_budget = gc_round_outcome_entry_budget; pool_config.gc_meta_pool_size = gc_meta_pool_size; + pool_config.mount_lease_ttl_ms = mount_lease_ttl; + pool_config.mount_renew_period = mount_renew_period; pool_config.event_sink = makeCasEventSink(); PoolView view; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 043ce8f9280b..b1d30b6724d9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -611,6 +611,8 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC const uint64_t gc_round_prefix_wholesale_budget; const uint64_t gc_round_handoff_prefix_wholesale_budget; const uint64_t gc_round_outcome_entry_budget; + const std::chrono::milliseconds mount_lease_ttl; + const std::chrono::milliseconds mount_renew_period; /// Part-folder view cache settings. `cas_part_folder_cache_bytes == 0` disables retention. const uint64_t cas_part_folder_cache_bytes; const uint64_t cas_part_folder_cache_max_entries; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index bf6ab3b90e33..4a8b7a82d04f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -70,6 +70,8 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, gc_round_prefix_wholesale_budget, 20000, "Generation-prefix wholesale delete (prune only) object cap per round (0 = unbounded)", 0) \ DECLARE(UInt64, gc_round_handoff_prefix_wholesale_budget, 5000, "Post-CAS hand-off generation-prefix reclaim object cap per round, reserved separately from gc_round_prefix_wholesale_budget so a prune-heavy round cannot starve the one-shot hand-off (0 = unbounded)", 0) \ DECLARE(UInt64, gc_round_outcome_entry_budget, 5000, "GcOutcomes per-round entry cap across the redelete/spared audit log (0 = unbounded)", 0) \ + DECLARE(UInt64, mount_lease_ttl_ms, 30000, "Mount lease validity after a successful claim or renewal, in milliseconds", 0) \ + DECLARE(UInt64, mount_renew_period_ms, 10000, "Interval between background mount lease renewals, in milliseconds", 0) \ DECLARE(String, server_root_id, "", "REQUIRED explicit layout subtree identity; macros expand as in the s3 endpoint", 0) \ DECLARE(UInt64, part_folder_cache_bytes, 64ULL << 20, "Part-folder view cache byte budget (0 disables retention)", 0) \ DECLARE(UInt64, part_folder_cache_max_entries, 10000, "Part-folder view cache entry cap", 0) \ From e4240830ccc91a4d30012e416b31a91789c8e967 Mon Sep 17 00:00:00 2001 From: Konstantin Morozov Date: Wed, 2 Sep 2026 18:37:04 +0200 Subject: [PATCH 10/81] add validation Signed-off-by: Konstantin Morozov --- docs/en/antalya/cas/configuration.md | 4 ++-- .../ContentAddressed/ContentAddressedSettings.cpp | 10 ++++++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 7d52106692cd..8770000811ba 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -92,8 +92,8 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_blob_hash` | `cityhash128` | Pool blob content-hash function (`cityhash128` \| `xxh3-128` \| `sha256`). Recorded in the pool at creation; a mismatching config is refused at mount | | `cas_blob_hash_allow_new` | `false` | Explicit opt-in to admit a new hash algorithm into an existing pool. One-way: once admitted, the pool carries both algorithms permanently | | `skip_access_check` | `false` | Skip the boot-time capability probe (start now, fix later). Only the preflight probe is skipped — the conditional-write correctness check still runs on every writable mount. **Not available on a writable generation-token (GCS) disk**, which refuses to mount with it: there, the probe battery is the only proof that a token-exact delete carries its generation precondition. Mount such a disk read-only if you need to defer the check | -| `cas_mount_lease_ttl_ms` | `30000` | Milliseconds for which a mount lease remains valid after a successful claim or renewal. Lower values shorten stale-mount recovery but reduce tolerance for object-storage and scheduling delays | -| `cas_mount_renew_period_ms` | `10000` | Milliseconds between background mount-lease renewals. It must leave enough time for one request attempt and the lease safety margin before the TTL expires | +| `cas_mount_lease_ttl_ms` | `30000` | Milliseconds for which a mount lease remains valid after a successful claim or renewal (≥ 1). Lower values shorten stale-mount recovery but reduce tolerance for object-storage and scheduling delays | +| `cas_mount_renew_period_ms` | `10000` | Milliseconds between background mount-lease renewals (≥ 1). It must leave enough time for one request attempt and the lease safety margin before the TTL expires | | `cas_gc_snapshot_generations_to_keep` | `3` | GC snapshot generations retained | | `cas_gc_shards` | `1` | Blob-hash-prefix reducer shards (≥ 1). Recorded in the pool at creation; a mismatching config is refused at mount | | `gcs_max_conditional_put_bytes` | 1 GiB | Largest conditional non-blob `PUT` on a generation-token store, including create-if-absent metadata/control artifacts and conditional replacements. Blob publication is unconditional, uses ordinary multipart, and is not subject to this cap | diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index 4a8b7a82d04f..6fed1a6a99f0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -230,6 +230,16 @@ void ContentAddressedSettings::validate() "content_addressed disk: cas_gc_interval_sec and cas_gc_shards must be >= 1 (got {}, {})", settings[ContentAddressedSetting::gc_interval_sec].value, settings[ContentAddressedSetting::gc_shards].value); + if (settings[ContentAddressedSetting::mount_lease_ttl_ms] == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: cas_mount_lease_ttl_ms must be >= 1 (got {})", + settings[ContentAddressedSetting::mount_lease_ttl_ms].value); + + if (settings[ContentAddressedSetting::mount_renew_period_ms] == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: cas_mount_renew_period_ms must be >= 1 (got {})", + settings[ContentAddressedSetting::mount_renew_period_ms].value); + /// The layout subtree identity is explicit and REQUIRED — no default, so an ABSENT key throws a /// typed `NO_ELEMENTS_IN_CONFIG` (mirroring the `metadata_type` check in `MetadataStorageFactory`), /// distinct from a PRESENT-but-invalid value, which falls through to `validateServerRootId`'s From f6a788182b83f2b91a112a595afc6336b45d4361 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:01:04 +0200 Subject: [PATCH 11/81] cas: don't refuse a GCS mount just because the versioning probe can't answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On a generation-dialect (GCS) mount, `ObjectStorageBackend::checkPoolPreconditions` refused to mount both when the bucket was verified to have versioning enabled and when the probe simply couldn't get an answer. The first live run against a real GCS bucket hit the second case: the service account lacked `storage.buckets.get`, `GetBucketVersioning` returned 403, and every writable CAS mount on that bucket failed with `NOT_IMPLEMENTED` at server start — a missing IAM grant turned into a hard outage, even though an unreadable bucket configuration is not evidence the bucket is actually versioned. The probe now logs a warning naming what it couldn't verify and how to fix it (grant `storage.buckets.get`, or confirm by hand) and lets the mount proceed. A bucket confirmed versioned still refuses, because a token-exact `DELETE` there archives a noncurrent generation instead of reclaiming storage. That first credentialed run against Google (HMAC groups) also surfaced three test-suite assumptions the real service doesn't meet (`system.cas_log.token` isn't always a numeric generation for build-lifecycle rows; a second `COUNT()` over a Parquet object is answered from the per-file row-count cache, not the Parquet metadata cache; a disk over an absent bucket refuses at `CREATE TABLE`, not the first `INSERT`) and one open question — whether process-wide `system.events` deltas can be attributed to one statement when a mount-lease renewal shares the same counters. The suite now attributes every counter it asserts through `system.query_log.ProfileEvents` instead. Also documents GCS's request-rate limits in the CAS bucket requirements. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- docs/en/antalya/cas/architecture/backend.md | 14 +- docs/en/antalya/cas/bucket-requirements.md | 74 +++++++- .../ContentAddressed/Backend/CasBackend.h | 9 +- .../Backend/CasObjectStorageBackend.cpp | 29 ++-- .../Backend/CasObjectStorageBackend.h | 5 +- .../ContentAddressed/Primitives/CasEvent.h | 11 +- .../ObjectStorages/IObjectStorage.h | 5 +- .../ObjectStorages/S3/S3ObjectStorage.cpp | 5 + .../tests/gtest_cas_backend_generation.cpp | 66 ++++++-- tests/integration/test_gcs_live/test.py | 160 +++++++++++------- 10 files changed, 267 insertions(+), 111 deletions(-) diff --git a/docs/en/antalya/cas/architecture/backend.md b/docs/en/antalya/cas/architecture/backend.md index b1d88843471e..cc0740ea73da 100644 --- a/docs/en/antalya/cas/architecture/backend.md +++ b/docs/en/antalya/cas/architecture/backend.md @@ -108,15 +108,17 @@ storage, and GC would silently stop reclaiming. `runCapabilityProbe` (`Backend/CasProbe.cpp`) runs a throwaway-key battery against every writable mount, described in full on the [bucket requirements](/antalya/cas/bucket-requirements) page. It is fail-closed: any check that does not pass throws `NOT_IMPLEMENTED` naming the specific failure, and -the mount refuses to become writable. Two further gates run as the battery's opening steps, and one +the mount refuses to become writable. The one tolerated exception is a versioning probe that cannot +answer at all, described in the first bullet below. Two further gates run as the battery's opening steps, and one sits genuinely alongside it. The distinction matters: because the versioning check runs *inside* the battery, skipping the battery used to skip it too, which is exactly why the third gate exists. -- `checkPoolPreconditions` — inside the battery. On the `GCS`-dialect combination only, requires bucket versioning to be - *verifiably* off. A confirmed `Enabled` and an inconclusive probe both throw: `CAS` cannot assume - the safe answer here, because what it would do on a versioned bucket is delete objects it believes - it reclaimed. A probe is inconclusive when the credential may not read the bucket's versioning - configuration, or when the backend cannot answer at all. +- `checkPoolPreconditions` — inside the battery. On the `GCS`-dialect combination only, checks that + bucket versioning is off. A confirmed `Enabled` throws: what `CAS` would do on a versioned bucket is + delete objects it believes it reclaimed. An inconclusive probe — the credential may not read the + bucket's versioning configuration, or the backend cannot answer at all — logs a warning and lets + the mount proceed, since it is not evidence of a versioned bucket; verifying it then falls to the + operator, as it already does for soft delete. - `checkSkipAccessCheckSupport` — alongside the battery, in the skip branch of `Pool::open`, since it is the gate that decides whether the battery may be skipped at all. It asks whether the backend may serve a writable mount that skips the battery at all. The `GCS`-dialect combination refuses, so `skip_access_check = true` cannot reach a diff --git a/docs/en/antalya/cas/bucket-requirements.md b/docs/en/antalya/cas/bucket-requirements.md index 67a3795ab682..700d0a94d371 100644 --- a/docs/en/antalya/cas/bucket-requirements.md +++ b/docs/en/antalya/cas/bucket-requirements.md @@ -32,11 +32,13 @@ Bucket **versioning is not required** — in fact it must be **disabled** on the dialect (see below), because a token-exact delete on a versioned bucket archives a noncurrent generation instead of reclaiming storage, silently stopping GC reclamation. -On the generation-token dialect that requirement is checked, and checked strictly: a writable mount -proceeds only when the probe *confirms* versioning is disabled. A bucket reported as versioned and a -probe that could not answer — the credential may not read the bucket's versioning configuration, or -the backend cannot report it — both refuse the mount. `CAS` does not assume the safe answer, because -the failure it would be assuming away is `GC` deleting objects it believes it reclaimed. +On the generation-token dialect that requirement is checked at mount. A bucket reported as versioned +refuses the mount. A probe that could not answer — the credential may not read the bucket's versioning +configuration (`storage.buckets.get` on GCS), or the backend cannot report it — does not: the mount +proceeds and logs a warning naming what it could not verify, because an unreadable configuration is +not evidence of a versioned bucket, and refusing on it would turn a missing IAM grant into an outage. +In that case confirming that versioning is disabled is your responsibility, exactly as soft delete is +below; grant the permission if you want the mount to verify it for you. Because that check is part of the mount battery, `skip_access_check = true` is refused on a writable generation-token disk. Mount the disk read-only if you need to start before the access check can @@ -54,6 +56,68 @@ Soft delete does not leave the deleted generation live, so it does not break exa the way versioning does. What it does is delay physical reclamation until the retention period expires: `GC` reports space as reclaimed while the bill still reflects it. +## Request rate, and the limit that is not the one you expect {#request-rate} + +Google Cloud Storage publishes two kinds of ceiling, and the one that constrains `CAS` is the +smaller and less-known of them. + +A bucket starts at roughly **1000 object writes per second** — uploads, updates and deletes — and +roughly **5000 object reads per second**, counting listings and metadata reads as reads. Those +ceilings are not fixed: Cloud Storage raises them by splitting the index range behind the bucket, +which it says takes "on the order of minutes" to detect and act on. Buckets with a hierarchical +namespace start up to eight times higher. + +Separately, Cloud Storage applies **a much smaller limit to repeated writes to the same object +name**. Google documents that this limit exists but does not publish its value. Measured against a +live bucket from this codebase, it begins to bite at approximately one mutation per second on a +single key, and it does not participate in the auto-scaling above — splitting an index range cannot +help a single name. + +That second limit is the one `CAS` meets first, because two of its objects are single fixed names +written on a hot path: + +| Object | One per | Written on | +|---|---|---| +| `cas/ns/state//_ckpt` | table | every durable ref-log transaction, plus namespace birth, epoch seal and snapshot | +| `cas/ref_catalog` | pool | twice per `CREATE TABLE` and twice per `DROP TABLE` | + +Blob bodies and their metadata sidecars are named by content hash and are therefore spread the way +Google's own guidance asks for: it recommends "completely random object names" for the best load +distribution, and a hashed prefix where names would otherwise be sequential. Ref-log transactions are +sequential within a namespace but are written under a per-namespace prefix, so they scale with the +number of tables rather than sharing one index range. + +### What this means for a deployment {#rate-consequences} + +- **A single table commits at about one transaction per second** on Google Cloud Storage. Inserts, + merges and mutations on that table queue behind the checkpoint write; they do not fail, but the + lane's throughput is capped and each flush's tail takes longer than it would on a store without + the per-name limit. +- **A pool performs about one table lifecycle transition per second.** Concurrent `CREATE TABLE` or + `DROP TABLE` beyond that rate contends on the pool-wide catalog. Test suites that create hundreds + of tables in parallel are the case that provokes this; ordinary production DDL is not. +- **Ramping up gradually is Google's documented expectation.** Its guidance is to increase the + request rate "no faster than doubling the rate over a period of 20 minutes", and to pause or + reduce the rate when latency or error rates rise. A pool that goes from idle to full write load in + one step will see throttling before the bucket has redistributed the load. + +### Throttling is a retryable condition, not a failure {#rate-errors} + +Cloud Storage signals a rate it will not serve with HTTP `429`, `408`, or a `5xx` status, and its +retry guidance names all three, together with socket timeouts and TCP disconnects, as retryable with +exponential backoff and jitter. Every mutable-object write `CAS` issues carries a generation +precondition, which places it in Google's *conditionally idempotent* class — a retry either applies +exactly once or fails the precondition, never applies twice. Retrying them is therefore safe by +Google's own rule, not merely by ours. + +### Reads over a wide-area link want a cache disk {#rate-reads} + +The read ceiling is high enough that `CAS` does not approach it, but latency is a separate matter. +A cacheless `CAS` disk pays a round trip per column file per part: measured against a bucket in +another region, a `SELECT` issued about 725 ranged reads and took 3.6 seconds at the median and 15.7 +seconds at the ninety-ninth percentile. Put a `cache` disk in front of the `CAS` disk for any +deployment where the bucket is not local to the server. + ## Platform support {#platform-support} The deterministic request-construction coverage is green, but the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index e40521134377..579c33a1c355 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -315,10 +315,11 @@ class Backend virtual bool supportsListTokens() const = 0; /// Pool-level preconditions beyond per-op conditional semantics — checked by the capability - /// probe BEFORE the op battery. Default: nothing to check. The S3 backend fails closed here - /// unless a generation-dialect (GCS) bucket is VERIFIABLY free of object versioning: a - /// token-exact DELETE against a versioned bucket archives a noncurrent generation instead of - /// reclaiming storage, so GC "reclaim" would silently stop reclaiming. + /// probe BEFORE the op battery. Default: nothing to check. The S3 backend refuses here when a + /// generation-dialect (GCS) bucket is verified to have object versioning enabled: a token-exact + /// DELETE against a versioned bucket archives a noncurrent generation instead of reclaiming + /// storage, so GC "reclaim" would silently stop reclaiming. A probe that cannot answer is not + /// evidence of that, so it warns and the mount proceeds. virtual void checkPoolPreconditions() {} /// Fail-closed precondition: may this backend serve a WRITABLE mount that skips the access-check diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index d27e01b412b0..fab29d8ae0a0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -16,6 +16,7 @@ #include #include +#include #include "config.h" @@ -51,8 +52,11 @@ ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mod /// See Backend::checkPoolPreconditions. Only the Native, generation-dialect (GCS) combination has /// anything to check: a token-exact DELETE on a versioned bucket archives a noncurrent generation -/// instead of reclaiming storage, so GC "reclaim" would silently stop reclaiming. Both an enabled -/// bucket and an unverifiable probe refuse the mount. +/// instead of reclaiming storage, so GC "reclaim" would silently stop reclaiming. A bucket VERIFIED +/// to have versioning enabled refuses the mount. A probe that cannot answer does not: it is not +/// evidence of a versioned bucket, its usual cause is a credential without permission to read the +/// bucket configuration, and refusing on it would turn a missing IAM grant into a hard outage. The +/// mount proceeds with a warning that names what was not verified and how the operator can verify it. void ObjectStorageBackend::checkPoolPreconditions() { if (mode != Mode::Native || native_token_type != TokenType::Generation) @@ -61,20 +65,15 @@ void ObjectStorageBackend::checkPoolPreconditions() const auto versioned = object_storage->isBucketVersioningEnabled(); if (!versioned.has_value()) { - /// An unverifiable probe fails the mount, exactly like a confirmed Enabled below. Proceeding - /// on the ASSUMPTION that versioning is off was the earlier behaviour and it is not - /// defensible: what GC does on a versioned bucket is delete objects it believes it reclaimed, - /// so the assumption is silently wrong in precisely the case that matters, and it is wrong - /// without bound (a warning at mount does not stop the next round). The operator can prove - /// the bucket's state with one call and grant the permission the probe needs. - throw Exception(ErrorCodes::NOT_IMPLEMENTED, + LOG_WARNING(getLogger("CasObjectStorageBackend"), "CAS on GCS: could not VERIFY the bucket-versioning precondition (the versioning check " - "request failed — e.g. the credential lacks permission to read it — or this backend " - "cannot answer it) — refusing to mount writable. CAS cannot assume versioning is off: if " - "it is actually enabled, token-exact DELETEs archive noncurrent generations instead of " - "reclaiming storage and GC silently stops reclaiming space. Grant the credential " - "permission to read the bucket's versioning configuration, confirm versioning is " - "disabled, and retry the mount."); + "request failed, e.g. the credential lacks permission to read the bucket configuration, " + "or this backend cannot answer it). Mounting anyway. If versioning IS enabled on this " + "bucket, token-exact DELETEs archive noncurrent generations instead of reclaiming storage " + "and GC silently stops reclaiming space. Confirm by hand that versioning is disabled, or " + "grant the credential permission to read the bucket's versioning configuration " + "(storage.buckets.get on GCS) so the next mount can verify it."); + return; } if (*versioned) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index 7344c8fbede4..7bf65f67849f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -103,8 +103,9 @@ class ObjectStorageBackend final : public Backend /// Return a page after `cursor`; the next cursor is the last returned key and is empty at the end. ListPage list(const String & prefix, const String & cursor, size_t limit) override; - /// Pool-level precondition: on a Native, generation-dialect (GCS) backend, reject the pool unless - /// object versioning is VERIFIABLY disabled — see Backend::checkPoolPreconditions. + /// Pool-level precondition: on a Native, generation-dialect (GCS) backend, reject the pool when + /// object versioning is verified ENABLED; warn and continue when the probe cannot answer — see + /// Backend::checkPoolPreconditions. void checkPoolPreconditions() override; /// Fail-closed precondition: a Native, generation-dialect (GCS) backend refuses a writable mount diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h index 752ac5cf3f84..05b9b75a8967 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasEvent.h @@ -54,9 +54,12 @@ enum class CasEventObjectKind { None, Blob, Manifest, Root, Snap }; /// Pure-data event passed from the content-addressed core to the metadata-storage audit-log sink. /// Fields that do not apply to an event remain empty or zero. `reason` is mandatory for decisions /// and must explain why the operation took its outcome; `detail` carries structured facts needed -/// to reconstruct the event without parsing the free-form reason. Hashes are lowercase hexadecimal, -/// tokens identify object incarnations, and the numeric fields identify GC rounds, snapshot -/// generations, or the manifest journal version as applicable. +/// to reconstruct the event without parsing the free-form reason. Hashes are lowercase hexadecimal. +/// `token` identifies an object incarnation on events about a stored object (`object_kind` is Blob or +/// Manifest); the part-build lifecycle events that carry a token at all (`BuildStart`, `Precommit`, +/// `BuildPublish`, `BuildAbort`) reuse it for the 128-bit build id in hex and leave `object_kind` at +/// `None`. The numeric fields +/// identify GC rounds, snapshot generations, or the manifest journal version as applicable. struct CasEvent { CasEventType type = CasEventType::BlobPut; @@ -64,7 +67,7 @@ struct CasEvent String ref_name; /// the ref name — a mutable directory handle, git-style (empty if N/A) CasEventObjectKind object_kind = CasEventObjectKind::None; String object_hash; /// lowercase hex (empty if N/A) - String token; /// incarnation token (empty if N/A) + String token; /// incarnation token; the hex build id on build lifecycle events (empty if N/A) UInt64 round = 0; UInt64 gen = 0; UInt64 at_version = 0; diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index 1bddf3b07c0c..00f7b63a934d 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -427,8 +427,9 @@ class IObjectStorage virtual void pinConditionalOpsGenerationDialect(bool /*expect_generation_tokens*/) {} /// Whether the underlying bucket has object versioning enabled; nullopt when unknown or not - /// applicable. Used by the CAS capability probe to fail closed on GCS: on a versioned bucket - /// a token-exact DELETE archives a noncurrent generation instead of reclaiming storage. + /// applicable. Used by the CAS capability probe on GCS: a bucket verified as versioned refuses + /// the mount (a token-exact DELETE there archives a noncurrent generation instead of reclaiming + /// storage), an unknown answer is logged and tolerated. virtual std::optional isBucketVersioningEnabled() const { return std::nullopt; } /// True when this object storage can execute writes under the given retry profile. diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 3d514e371266..c0cbf7ceb5f8 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -574,7 +574,12 @@ std::optional S3ObjectStorage::isBucketVersioningEnabled() const auto outcome = client.get()->GetBucketVersioning(request); if (!outcome.IsSuccess()) + { + /// The caller only learns "unknown"; the reason is what the operator needs to act on. + LOG_WARNING(log, "GetBucketVersioning for bucket `{}` failed: {} ({})", + uri.bucket, outcome.GetError().GetMessage(), outcome.GetError().GetExceptionName()); return std::nullopt; + } return outcome.GetResult().GetStatus() == Aws::S3::Model::BucketVersioningStatus::Enabled; } diff --git a/src/Disks/tests/gtest_cas_backend_generation.cpp b/src/Disks/tests/gtest_cas_backend_generation.cpp index 04da7d8aba63..7bcd4768f0d7 100644 --- a/src/Disks/tests/gtest_cas_backend_generation.cpp +++ b/src/Disks/tests/gtest_cas_backend_generation.cpp @@ -24,6 +24,8 @@ #include #include #include +#include +#include #include #include @@ -123,9 +125,41 @@ std::shared_ptr makeVersioningObjectStorageForTest(std: return std::make_shared(std::move(settings), versioned); } +/// Captures what `ObjectStorageBackend` logs at WARNING and above, so a test can assert both that a +/// warning was raised and that none was. Same shape as the capture in gtest_cas_settings.cpp. +class ScopedBackendLogCapture +{ +public: + ScopedBackendLogCapture() + : logger(getLogger("CasObjectStorageBackend")) + , channel(new Poco::StreamChannel(stream)) + , old_channel(logger->getChannel(), /*shared=*/true) + , old_level(logger->getLevel()) + { + logger->setChannel(channel.get()); + logger->setLevel("warning"); + } + + ~ScopedBackendLogCapture() + { + logger->setChannel(old_channel); + logger->setLevel(old_level); + } + + String captured() const { return stream.str(); } + +private: + LoggerPtr logger; + std::ostringstream stream; + Poco::AutoPtr channel; + /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. + Poco::AutoPtr old_channel; + int old_level; +}; + /// Every refusal reached from these mount gates is `NOT_IMPLEMENTED`, so the code alone cannot tell /// which one fired. Match a phrase unique to the intended message as well, or a test asserting the -/// unverifiable-versioning refusal would pass on the enabled-bucket refusal and vice versa. +/// enabled-versioning refusal would pass on the skip-access-check refusal and vice versa. template void expectThrowsNotImplementedSaying(const std::string & needle, F && fn) { @@ -180,17 +214,25 @@ TEST(CASBackendGeneration, StampedTokenTypeFollowsNativeKind) EXPECT_EQ(hr.token.type, TokenType::Generation); } -/// A generation-dialect (GCS) mount needs bucket versioning to be VERIFIABLY off: a token-exact +/// A generation-dialect (GCS) mount wants bucket versioning to be verifiably off: a token-exact /// DELETE against a versioned bucket archives a noncurrent generation, so GC would delete objects it -/// believes it reclaimed. A probe that cannot answer therefore refuses the mount rather than -/// assuming the safe answer. -TEST(CASBackendGeneration, CheckPoolPreconditionsFailsClosedOnUnverifiableVersioning) +/// believes it reclaimed. A probe that cannot answer is not evidence of a versioned bucket, though: +/// the usual cause is a credential without permission to read the bucket configuration, and +/// refusing on it turns a missing IAM grant into a hard outage. So the mount proceeds, and says +/// loudly what it could not verify and how the operator can. +TEST(CASBackendGeneration, CheckPoolPreconditionsWarnsAndContinuesOnUnverifiableVersioning) { auto b = std::make_shared( makeVersioningObjectStorageForTest(std::nullopt), ObjectStorageBackend::Mode::Native); b->setNativeTokenTypeForTest(TokenType::Generation); - expectThrowsNotImplementedSaying("could not VERIFY", [&] { b->checkPoolPreconditions(); }); + ScopedBackendLogCapture capture; + EXPECT_NO_THROW(b->checkPoolPreconditions()); + + const auto logged = capture.captured(); + EXPECT_NE(logged.find("could not VERIFY"), String::npos) << logged; + EXPECT_NE(logged.find("versioning"), String::npos) << logged; + EXPECT_NE(logged.find("storage.buckets.get"), String::npos) << logged; } TEST(CASBackendGeneration, CheckPoolPreconditionsRejectsEnabledVersioning) @@ -202,27 +244,31 @@ TEST(CASBackendGeneration, CheckPoolPreconditionsRejectsEnabledVersioning) expectThrowsNotImplementedSaying("VERSIONING enabled", [&] { b->checkPoolPreconditions(); }); } -/// The one accepting case: a probe that answered, and answered "disabled". -TEST(CASBackendGeneration, CheckPoolPreconditionsAcceptsVerifiedDisabledVersioning) +/// The fully verified case: a probe that answered, and answered "disabled". Nothing to warn about. +TEST(CASBackendGeneration, CheckPoolPreconditionsAcceptsVerifiedDisabledVersioningSilently) { auto b = std::make_shared( makeVersioningObjectStorageForTest(false), ObjectStorageBackend::Mode::Native); b->setNativeTokenTypeForTest(TokenType::Generation); + ScopedBackendLogCapture capture; EXPECT_NO_THROW(b->checkPoolPreconditions()); + EXPECT_TRUE(capture.captured().empty()) << capture.captured(); } /// The ETag-dialect (AWS-compatible) backend never consults bucket versioning at all — the check is /// a silent no-op for any backend that is not Native + TokenType::Generation. Driven over a storage -/// whose probe is unverifiable, which is what a generation-dialect backend now refuses: dropping the -/// dialect guard from checkPoolPreconditions would fail this test. +/// whose probe is unverifiable, which is what a generation-dialect backend warns about: dropping the +/// dialect guard from checkPoolPreconditions would fail the silence assertion. TEST(CASBackendGeneration, CheckPoolPreconditionsNoOpOnEtagDialect) { auto b = std::make_shared( makeVersioningObjectStorageForTest(std::nullopt), ObjectStorageBackend::Mode::Native); ASSERT_EQ(b->nativeTokenType(), TokenType::ETag); + ScopedBackendLogCapture capture; EXPECT_NO_THROW(b->checkPoolPreconditions()); + EXPECT_TRUE(capture.captured().empty()) << capture.captured(); } /// A writable generation-dialect (GCS) mount may not skip the mutating capability battery: that diff --git a/tests/integration/test_gcs_live/test.py b/tests/integration/test_gcs_live/test.py index f2fdd5c83ef6..8c8e5e47024a 100644 --- a/tests/integration/test_gcs_live/test.py +++ b/tests/integration/test_gcs_live/test.py @@ -63,36 +63,35 @@ non-CAS requests retain their ETag-based contract, that CAS records generations rather than ETags, and that each named body-publication action was actually selected. The Task 10 cases use a statement query id in `system.cas_log` and `system.query_log`, so unrelated background work cannot satisfy them. -The older ordinary characterization still uses process-wide `system.events`; its limitations remain -spelled out below rather than being silently hidden. +The ordinary characterization does the same since the first live run; the section below records why +process-wide `system.events` deltas were not acceptable evidence in this configuration. -## OPEN QUESTION FOR WHOEVER FIRST RUNS THIS WITH CREDENTIALS +## The process-wide counter hazard, and how the first live run settled it `system.events` counters are PROCESS-WIDE, and this configuration also holds several CAS disks whose control writers issue object-storage requests of their own. Their GC schedulers are stopped before the -tests, but mount leases and other control work still exist. So every ordinary counter delta asserted here -is only as sound as the assumption that no CAS activity moved that counter inside the measured window. -Where that assumption fails, the assertion still passes — for a reason that has nothing to do with the -statement it names. - -This is an open question, not a known defect: which counters CAS can actually move during these -windows is not determinable without a real run. It is written here rather than in a tracked item -because the first run is when it matters and this docstring is what its reader will have in front of -them. **On that run, check each counter individually instead of trusting a pass** — for any counter CAS -can move, a green assertion is not evidence that the statement under test issued the operation. - -One test is EXEMPT, and the reason is the template for clearing the others: +tests, but the mount-lease renewal keeps running: each renewal is a conditional overwrite of the lease +object (`S3PutObject`), followed by a `get` (`S3GetObject`) when the outcome is unresolved; the one-time +claim at mount issued a `head` and a `putIfAbsent` as well. A process-wide delta on `S3PutObject` or +`S3GetObject` therefore cannot say which disk issued the request, and nothing rules out other control +writers on other counters. + +Settled on the first credentialed run (2026-09-02): the ordinary operation-set test attributes every +counter it asserts to the statement that must have issued it, through `system.query_log.ProfileEvents` +by query id, exactly as the publication scenarios already did. The delete shapes are proven by log +lines filtered to this run's own key prefix rather than by a counter. Nothing in this file asserts a +process-wide S3 request counter any more. + +The one exemption that never needed it: `test_default_gcs_client_parquet_metadata_cache_keys_on_the_ordinary_etag` uses `ParquetMetadataCacheMisses` and `ParquetMetadataCacheHits`, which only a Parquet read moves. No CAS -disk can touch either, so those two deltas mean exactly what they say. Clearing a counter means showing -that same thing about it — not observing it pass. +disk can touch either, so those two deltas mean exactly what they say. -One instance is already settled and serves as the pattern for the other direction. `S3ListObjects` was -asserted here and has been removed: an ordinary MergeTree lifecycle on a local-metadata disk never lists, so it could not -have been satisfied by this workload at all — but the CAS disks in this same configuration DO list, so -a background collection round inside the window could have satisfied it anyway. That is exactly the -failure mode above, and it is why "make something list somehow" would have produced a test passing for -the wrong reason rather than a working one. +`S3ListObjects` was asserted here once and has been removed: an ordinary MergeTree lifecycle on a +local-metadata disk never lists, so it could not have been satisfied by this workload at all — but the +CAS disks in this same configuration DO list, so a background collection round inside the window could +have satisfied it anyway. "Make something list somehow" would have produced a test passing for the +wrong reason rather than a working one. It does NOT assert the outbound header set — that `x-goog-if-generation-match` appears on the wire, that `x-amz-date` / `x-amz-content-sha256` / `x-amz-security-token` / `x-amz-api-version` are absent, @@ -544,9 +543,13 @@ def _opaque_generation_evidence(node, query): def _cas_generation_domain(node, disk): """Whether this disk recorded generations and every recorded value belongs to the numeric domain.""" __tracebackhide__ = True + # Only rows about an OBJECT carry an incarnation token. The part-build lifecycle events that carry + # a token at all (`build_start`, `precommit`, `build_publish`, `build_abort`) reuse the column for + # the 128-bit build id in hex and have `object_kind = 'none'`; seen on live GCS 2026-09-02, where + # they were the only non-numeric values and every manifest/blob token was a generation. count, all_numeric, _digests = _opaque_generation_evidence( node, - "SELECT DISTINCT token FROM system.cas_log WHERE disk_name = '{}' AND token != '' FORMAT TSV".format(disk), + "SELECT DISTINCT token FROM system.cas_log WHERE disk_name = '{}' AND token != '' AND object_kind != 'none' FORMAT TSV".format(disk), ) return count > 0, all_numeric @@ -751,9 +754,10 @@ def _wait_for_driver_phase(path, scenario_id, phase, timeout=60): def test_default_gcs_client_accepts_the_ordinary_object_storage_operation_set(auth_mode, policy, path_fragment): """Every S3 operation an ordinary disk issues is accepted under either GCS client. - The `system.events` deltas are what make this non-vacuous: each named operation must have been - issued at least once, so a statement that quietly stopped reaching object storage — because a - default changed, or because a part stayed in memory — cannot leave the assertion true. + Per-statement `ProfileEvents` are what make this non-vacuous: each named operation must have been + issued by the statement that is supposed to issue it, so a statement that quietly stopped reaching + object storage — because a default changed, or because a part stayed in memory — cannot leave the + assertion true, and a CAS disk's lease renewal in the same process cannot satisfy it either. The statement-to-operation mapping is deliberately NOT pinned. Which statement produces a batch delete rather than singular ones is a ClickHouse implementation detail that moves between versions; @@ -761,7 +765,7 @@ def test_default_gcs_client_accepts_the_ordinary_object_storage_operation_set(au test fail on refactors that say nothing about GCS. Object LISTING is not covered by THIS test — an ordinary MergeTree lifecycle on a local-metadata - disk never issues one, see the comment on `counters` below. + disk never issues one, see the `S3ListObjects` comment in the body. `test_default_gcs_client_accepts_an_object_listing` covers it on the same authenticated client through the table-engine path, which is a lister. @@ -779,55 +783,63 @@ def test_default_gcs_client_accepts_the_ordinary_object_storage_operation_set(au # metadata directory and issue no S3 listing at all. An ordinary lifecycle on a local-metadata disk # never lists. # - # Worse than merely unsatisfiable, it would be unsound: `system.events` is process-wide, and the - # CAS disks in this same configuration DO list, so a background GC round landing inside the delta - # window could satisfy it for a reason that has nothing to do with this test's workload. - counters = [ - "S3PutObject", - "S3GetObject", - "S3HeadObject", - "S3CopyObject", - "S3DeleteObjects", - "S3CreateMultipartUpload", - "S3UploadPart", - "S3CompleteMultipartUpload", - ] - before = _events(node, counters) - + # Worse than merely unsatisfiable, a process-wide `system.events` delta on it would be unsound: the + # CAS disks in this same configuration DO list, so a background GC round landing inside the window + # could satisfy it for a reason that has nothing to do with this test's workload. + run_tag = "task10-ordinary-{}-{}".format(auth_mode, RUN_ID) table = _create(node, policy, "task10_plain_{}".format(auth_mode)) - - # A single-part PUT with custom metadata, then the HEAD that `s3_check_objects_after_upload` - # issues to verify it. + # A single-part PUT, then the HEAD that `s3_check_objects_after_upload` + # issues to verify it. Both are attributed to this INSERT. + put_query_id = run_tag + "-put" node.query( "INSERT INTO {} SELECT number, toString(number) FROM numbers(500)".format(table), settings={"s3_check_objects_after_upload": 1}, + query_id=put_query_id, ) + put_profile = _query_profile_events(node, put_query_id, ["S3PutObject", "S3HeadObject"]) + assert put_profile["S3PutObject"] > 0, put_profile + assert put_profile["S3HeadObject"] > 0, put_profile # A multipart upload: a tiny single-part ceiling rather than a large body, so the run does not # depend on how large a default part happens to be. + multipart_query_id = run_tag + "-multipart" node.query( "INSERT INTO {} SELECT number, repeat('x', 4096) FROM numbers(500, 4000)".format(table), settings={ "s3_min_upload_part_size": 5 * 1024 * 1024, "s3_max_single_part_upload_size": 1024, }, + query_id=multipart_query_id, ) + multipart_profile = _query_profile_events( + node, multipart_query_id, ["S3CreateMultipartUpload", "S3UploadPart", "S3CompleteMultipartUpload"] + ) + for name, value in multipart_profile.items(): + assert value > 0, "{} was never issued by the multipart INSERT: {}".format(name, multipart_profile) assert int(node.query("SELECT count() FROM {}".format(table))) == 4500 - assert int(node.query("SELECT sum(id) FROM {}".format(table))) > 0 - + # A read of column data, attributed: `count()` alone can be answered from part metadata. + get_query_id = run_tag + "-get" + assert int(node.query("SELECT sum(id) FROM {}".format(table), query_id=get_query_id)) > 0 + get_profile = _query_profile_events(node, get_query_id, ["S3GetObject"]) + assert get_profile["S3GetObject"] > 0, get_profile # A server-side copy: moving a partition between the two volumes of one policy copies each object # and then deletes the source. This is the only statement here that reaches `CopyObject`. - node.query("ALTER TABLE {} MOVE PARTITION tuple() TO VOLUME 'cold'".format(table)) + copy_query_id = run_tag + "-copy" + # Pinned synchronous: an asynchronous move runs on the background assignee, outside this query's + # thread group, and the copy would not be attributable to it. + node.query( + "ALTER TABLE {} MOVE PARTITION tuple() TO VOLUME 'cold'".format(table), + settings={"alter_move_to_space_execute_async": 0}, + query_id=copy_query_id, + ) + copy_profile = _query_profile_events(node, copy_query_id, ["S3CopyObject"]) + assert copy_profile["S3CopyObject"] > 0, copy_profile assert int(node.query("SELECT count() FROM {}".format(table))) == 4500 - - # A merge (more reads and writes), then the deletes. + # A merge (more reads and writes), then the deletes. Part removal runs in the background, so the + # deletes are not attributable to a statement; the two shapes are proven from the log below, filtered + # to this run's own keys. node.query("OPTIMIZE TABLE {} FINAL".format(table)) node.query("ALTER TABLE {} DROP PARTITION tuple()".format(table)) assert int(node.query("SELECT count() FROM {}".format(table))) == 0 - - after = _events(node, counters) - for name in counters: - assert after[name] > before[name], "{} was never issued ({} -> {}), so GCS acceptance of it is unproven".format(name, before[name], after[name]) - # `S3DeleteObjects` counts the singular and batch forms together, so the counter alone cannot say # the batch form was accepted. The two paths log differently, which separates them: # `deleteFileFromS3` logs "Object with path was removed from S3" and `deleteFilesFromS3` logs @@ -881,11 +893,29 @@ def test_default_gcs_hmac_reports_a_typed_error_for_a_refused_request(): rest of this group already exercises. """ node = cluster.instances["node"] - table = _create(node, HMAC_ABSENT_BUCKET_DISK) - error = node.query_and_get_error("INSERT INTO {} SELECT number, toString(number) FROM numbers(10)".format(table)) - # A parsed S3 error names the bucket problem. An unparsed one surfaces as a bare transport or - # timeout failure, which is what must not appear. - assert ("NoSuchBucket" in error) or ("S3_ERROR" in error) or ("ACCESS_DENIED" in error), error + table = "t_" + HMAC_ABSENT_BUCKET_DISK + node.query("DROP TABLE IF EXISTS {} SYNC".format(table)) + # `MergeTreeData`'s constructor writes `format_version.txt` to the policy's first writable disk, so + # on a live endpoint the refusal already arrives at CREATE (seen on GCS 2026-09-02: `The specified + # bucket does not exist`). This disk sets `skip_access_check`, so no startup access check runs + # before that. `query_and_get_answer_with_error` does not raise on success, so the INSERT is + # reached only when CREATE was accepted. + _, error = node.query_and_get_answer_with_error( + """ + CREATE TABLE {} (id Int64, data String) + ENGINE = MergeTree() ORDER BY id + SETTINGS storage_policy = '{}' + """.format(table, HMAC_ABSENT_BUCKET_DISK) + ) + if not error.strip(): + _, error = node.query_and_get_answer_with_error( + "INSERT INTO {} SELECT number, toString(number) FROM numbers(10)".format(table) + ) + # A parsed S3 error names the bucket problem in the error document's own words. An unparsed one + # surfaces as a bare transport or timeout failure, which is what must not appear. The raw text is + # kept out of the assertion message: a signature-mismatch body echoes the credential scope. + parsed = ("NoSuchBucket" in error) or ("specified bucket does not exist" in error) or ("ACCESS_DENIED" in error) + assert parsed, "the refusal did not arrive as a parsed S3 error document (see the server log)" @pytest.mark.parametrize("auth_mode,named_collection", NAMED_COLLECTION_CASES) @@ -956,13 +986,17 @@ def test_default_gcs_client_parquet_metadata_cache_keys_on_the_ordinary_etag(aut ) before = _events(node, events) + # Both reads aggregate a COLUMN rather than `count()`: after the first read, `count()` is answered + # from the per-file row-count cache (`use_cache_for_count_from_files`) without opening the object, + # so the second read would never reach the Parquet metadata cache and could not hit it. Seen live + # on GCS 2026-09-02: the second `count()` logged neither a hit nor a miss. # First read: cold, so the metadata is fetched from the object and the key is minted. - assert int(node.query("SELECT count() FROM {}".format(table_function))) == 1000 + assert int(node.query("SELECT sum(id) FROM {}".format(table_function))) == 499500 after_cold = _events(node, events) assert after_cold["ParquetMetadataCacheMisses"] > before["ParquetMetadataCacheMisses"], "no Parquet metadata cache miss, so the read never reached the object and nothing below is meaningful" # Second read of the same unchanged object: the key must be rebuilt identically and hit. - assert int(node.query("SELECT count() FROM {}".format(table_function))) == 1000 + assert int(node.query("SELECT sum(id) FROM {}".format(table_function))) == 499500 after_warm = _events(node, events) assert after_warm["ParquetMetadataCacheHits"] > after_cold["ParquetMetadataCacheHits"], "the second read of an unchanged object missed the cache, so the key is not stable" From 5740d2a2953896c4ccdf5aec1c26cfc39e80f5e6 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:01:30 +0200 Subject: [PATCH 12/81] cas: scope the relink-confirm refusal to the asked-about ref, not the whole namespace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same-pool replication transfers only a part's manifest: the receiver publishes its own ref over the sender's blobs, then asks the sender a read-only question — "do you still hold exactly this manifest for this part?" — and promotes only on `Yes`. Rule 3 of `CasRefLedger::confirmExactRef` answered `Unknown` whenever ANY mutation of the same namespace was queued, in flight, or awaiting its checkpoint frontier, not just a mutation of the asked-about ref. On the live GCS stand, two replicas answered each other `Unknown` almost every time for forty minutes: every replica is also a receiver, and each failed fetch appends two records to its own lane (a precommit, then its removal on abort), so under load neither side ever observed the other's lane quiet. Both replication queues wedged at 1.5-1.7k entries, the replicas diverged to 123k against 166k rows, and the soak died on `SYSTEM SYNC REPLICA`. Nothing was lost — once one side stopped fetching, the other drained in two minutes — but the lane-wide refusal made every sustained-write workload look like data loss in progress. On RustFS in a LAN the window closing this fast never showed the defect; GCS limits checkpoint publication to about one mutation per second per object, so the window is long enough to matter. Rule 3 now refuses only when the asked-about ref itself has a queued or in-flight mutation, via `RefTableRuntime::carved` mirroring the tenure's carved items and validating a ref-scoped item's ops against its `MutationScope` before durability. Covered by a two-node liveness case against a fake GCS with delayed `_ckpt` writes (`test_cas_gcs_relink_liveness`), and every confirm refusal is now attributed and counted rather than silent. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- src/Common/ProfileEvents.cpp | 5 + .../ContentAddressed/Pool/CasPool.h | 10 + .../ContentAddressed/Pool/CasRefLedger.cpp | 185 +++++- .../ContentAddressed/Pool/CasRefLedger.h | 73 ++- .../ContentAddressed/Pool/CasRefProtocol.h | 4 + .../tests/gtest_cas_confirm_exact_ref.cpp | 588 +++++++++++++++++- .../tests/gtest_cas_ref_chunked_flush.cpp | 93 ++- .../test_cas_gcs/gcs_mocks/server.py | 45 +- .../test_cas_gcs_relink_liveness/__init__.py | 0 .../configs/storage_conf.xml | 25 + .../test_cas_gcs_relink_liveness/test.py | 305 +++++++++ 11 files changed, 1239 insertions(+), 94 deletions(-) create mode 100644 tests/integration/test_cas_gcs_relink_liveness/__init__.py create mode 100644 tests/integration/test_cas_gcs_relink_liveness/configs/storage_conf.xml create mode 100644 tests/integration/test_cas_gcs_relink_liveness/test.py diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 87e11d082281..aac53e612a4c 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -808,6 +808,11 @@ The server successfully detected this situation and will download merged part fr M(CASRefAppendDefiniteFailure, "Number of CAS ref-log appends rejected with certainty. A non-zero value indicates invalid requests or backend rejection requiring investigation.", ValueType::Number) \ M(CASRefAppendSealRejected, "Number of CAS ref-log transactions conclusively rejected by a successor's epoch seal occupying the id they derived. This is the protocol working -- the writer was deposed and its operation was never acknowledged -- but a lane that keeps counting here is a writer that has lost its mount and does not yet know it.", ValueType::Number) \ M(CASRefAppendOccupantUnreadable, "Number of CAS ref-log appends that met a DIFFERENT object at the id they derived and could not read it to tell a successor's epoch seal from a breach of mount write-exclusivity. The decision is deferred to the next attempt, which re-derives the same id. Sustained growth means a real breach may be going unreported: the loud interference path is only reached once the occupant can be read.", ValueType::Number) \ + M(CASRelinkConfirmRefusedRefMutationInFlight, "Number of CAS fetch-by-relink confirms this server answered Unknown because a queued or in-flight ref-lane mutation names the asked-about ref or the whole namespace. Expected under write load; the receiver retries the fetch.", ValueType::Number) \ + M(CASRelinkConfirmRefusedLaneWedged, "Number of CAS fetch-by-relink confirms answered Unknown because the namespace's ref lane holds an unresolved append (a wedge). Lasts until the next flush or a remount resolves it.", ValueType::Number) \ + M(CASRelinkConfirmRefusedLaneBroken, "Number of CAS fetch-by-relink confirms answered Unknown because the namespace's ref lane is in NeedsRecovery, Closed or Faulted state, or is Writing with nothing carved. A growing value outside induced faults is a lane defect, not load.", ValueType::Number) \ + M(CASRelinkConfirmRefusedStateLockBusy, "Number of CAS fetch-by-relink confirms answered Unknown because the ref table's state lock was held. Under write load the usual holder is the table's own append leader, arming or installing a chunk; otherwise a recovery, a listing or a snapshot publish. The confirm never waits for it.", ValueType::Number) \ + M(CASRelinkConfirmRefusedMountCannotSpeak, "Number of CAS fetch-by-relink confirms answered Unknown because this mount cannot speak for the namespace: its ref table is unrecovered or mid-recovery, its catalog life was invalidated, its runtime was superseded by a remount, or its mount fence is no longer held. Neither a lane defect nor write load. A growing value means this writer is losing, or has already lost, its claim to the namespace.", ValueType::Number) \ M(CASRefNeedsRecovery, "Number of CAS ref append lanes moved to `NeedsRecovery` because a known-durable transaction could not be installed. Such a lane refuses writes, snapshots, and confirmation until durable replay completes.", ValueType::Number) \ M(CASRefSweepDeferred, "Number of stale-precommit sweeps deferred after a read-only failure. A non-zero value indicates cleanup is waiting for a later trigger.", ValueType::Number) \ M(CASRefSweepRearmed, "Number of failed or partial stale-precommit sweeps scheduled for retry. Growing values indicate persistent cleanup or backend errors.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index a56588dcb305..f6af8d4c7ab3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -1037,6 +1037,16 @@ class Pool : public std::enable_shared_from_this /// `refQueuePendingForTest`; used to assert the baton is not stranded on a pre-tenure fault. bool refLeaderActiveForTest(const RootNamespace & ns) { return ref_ledger.refLeaderActiveForTest(ns); } + /// Test-only: the carved-item mirror's size for `ns` (see `CasRefLedger::refCarvedForTest`). + size_t refCarvedForTest(const RootNamespace & ns) { return ref_ledger.refCarvedForTest(ns); } + + /// Test-only: whether the carved item named `ref_name` is already completed (see + /// `CasRefLedger::refCarvedItemDoneForTest`). + bool refCarvedItemDoneForTest(const RootNamespace & ns, const String & ref_name) + { + return ref_ledger.refCarvedItemDoneForTest(ns, ref_name); + } + /// Test seam: how many concurrent `ensureRefTableRecovered` callers for `ns` are /// PARKED right now waiting on the leader's in-flight recovery (see `RefTableRuntime:: /// recovery_waiters_for_test`) -- lets a test `yield()`-poll for "a second caller actually reached diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp index 0f6200c90553..318978c9b54e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp @@ -57,6 +57,11 @@ namespace ProfileEvents extern const Event CASRefAppendDefiniteFailure; extern const Event CASRefAppendSealRejected; extern const Event CASRefAppendOccupantUnreadable; + extern const Event CASRelinkConfirmRefusedRefMutationInFlight; + extern const Event CASRelinkConfirmRefusedLaneWedged; + extern const Event CASRelinkConfirmRefusedLaneBroken; + extern const Event CASRelinkConfirmRefusedStateLockBusy; + extern const Event CASRelinkConfirmRefusedMountCannotSpeak; extern const Event CASRefNeedsRecovery; extern const Event CASRefSweepDeferred; extern const Event CASRefSweepRearmed; @@ -81,6 +86,16 @@ namespace DB::Cas namespace { +/// The relink confirm's logger, resolved once. `LOG_IMPL` evaluates its logger argument BEFORE testing +/// the level, so `getLogger` at the call site would take the global logger-registry lock on every +/// refusal even with tracing off -- and `confirmExactRef` refuses while holding pool-wide append +/// admission, on a path a remote peer drives. +const LoggerPtr & confirmLogger() +{ + static const LoggerPtr logger = getLogger("CasRefLedger"); + return logger; +} + /// Classifies whether an exception thrown out of a ref-table recovery attempt (checkpoint/snapshot/log /// GETs, or the seal PUT) is a TRANSIENT object-store transport failure worth retrying, /// vs. a terminal condition (corruption, decode failure, logic error, resource limit) that must fail @@ -423,15 +438,17 @@ ConfirmAnswer CasRefLedger::confirmExactRef(const RootNamespace & ns, const Stri /// `sweepStalePrecommitsForRead` and `maybeScheduleSnapshotPublish`, the three maintenance calls /// `resolveRef` performs and all three of which can do I/O. /// - /// ONE snapshot across BOTH lane mutexes. `pending`/`leader_active` live under + /// ONE snapshot across BOTH lane mutexes. `pending`/`carved` live under /// `ref_queue_mutex`, the rows and the wedge under `state_mutex`, and the whole point of the /// rules is their CONJUNCTION -- read at different instants they would prove nothing. The lock /// ORDER is the one the rest of this file already establishes (`enforceRefTableCacheBudget` /// nests `state_mutex` under `ref_queue_mutex`, and nothing anywhere takes them the other way /// round). Because admission (`appendRefOps`' `pending.push_back`) happens under - /// `ref_queue_mutex`, an append is either entirely before this snapshot -- and then visible as a - /// pending item -- or entirely after it. There is no interleaving in which a removal is admitted - /// and this function still answers `Yes`. + /// `ref_queue_mutex`, an append is either entirely before this snapshot -- and then visible in + /// `pending` or, once carved, in `carved` -- or entirely after it. There is no interleaving in + /// which a mutation of the asked-about ref is admitted and this function still answers `Yes`; a + /// mutation of another ref may be admitted, and the answer is still right, because it cannot move + /// this ref's row. /// /// What a `Yes` does NOT prove, stated so nobody has to rediscover it: that this runtime's /// recovered view is a COMPLETE replay of the durable log. Completeness is recovery's contract, not @@ -443,6 +460,12 @@ ConfirmAnswer CasRefLedger::confirmExactRef(const RootNamespace & ns, const Stri /// Rule 2 (residency). Direct slot lookup, never a catalog observation or exact-runtime acquisition: /// a read-only query must not let a peer grow this writer's cache or make the next reader pay for a /// recovery it invented. A cold or evicted table is simply unknown here. + /// + /// These two arms are the ONLY refusals in this function that are deliberately not counted: a table + /// this mount has never touched, or has dropped under cache-budget pressure, is ordinary cache + /// behaviour rather than the lane, mount or load condition each counter below separates. Counting + /// it would put a number that moves with cache size next to numbers that describe this writer's + /// health. There is also no runtime here to attribute the refusal to. const auto it = ref_name_slots.find(ns.string()); if (it == ref_name_slots.end()) return ConfirmAnswer::Unknown; @@ -450,16 +473,31 @@ ConfirmAnswer CasRefLedger::confirmExactRef(const RootNamespace & ns, const Stri return ConfirmAnswer::Unknown; RefTableRuntime & rt = *it->second.current; - /// `try_to_lock`, not a blocking acquire: `ensureRefTableRecovered` holds `state_mutex` across its - /// whole exact replay, so blocking here would make a confirm WAIT on someone else's recovery -- - /// up to the full retry envelope -- while holding `ref_queue_mutex`, which is pool-wide append - /// admission. That is the zero-I/O contract broken by proxy: the query would not issue a request, - /// it would merely be paid for by one, and it would stall every table's lane meanwhile. Failing to - /// take the lock is just one more ambiguity, so it answers like every other one. (Same technique, - /// and same non-blocking rationale, as `enforceRefTableCacheBudget`'s candidate loop.) + /// Every refusal below is attributed here, on the node that computed it: `ConfirmAnswer` crosses + /// two interfaces as a three-value enum and stays that way, so the counters and this trace line are + /// the only way a live gate can tell load (`RefMutationInFlight`) from a fault (`LaneWedged`, + /// `LaneBroken`), from contention (`StateLockBusy`), or from this mount losing its claim to the + /// namespace (`MountCannotSpeak`). The invariant to keep when editing below: every + /// `return ConfirmAnswer::Unknown` past this point goes through `refuse`, and the only uncounted + /// refusals in this function are the two residency arms above, which say why. + const auto refuse = [&](ProfileEvents::Event reason, std::string_view why) + { + ProfileEvents::increment(reason); + LOG_TRACE(confirmLogger(), "Relink confirm for ref '{}' in namespace '{}' is unknown: {}", + ref_name, ns.string(), why); + return ConfirmAnswer::Unknown; + }; + + /// `try_to_lock`, not a blocking acquire: this function already holds `ref_queue_mutex`, which is + /// pool-wide append admission, so blocking here would stall EVERY table's lane for as long as + /// whoever holds `state_mutex` keeps it. That is the zero-I/O contract broken by proxy: the query + /// would not issue a request, it would merely wait on one, and it would hold up admission + /// meanwhile. Failing to take the lock is just one more ambiguity, so it answers like every other + /// one. (Same technique, and same non-blocking rationale, as `enforceRefTableCacheBudget`'s + /// candidate loop.) std::unique_lock slock(rt.state_mutex, std::try_to_lock); if (!slock.owns_lock()) - return ConfirmAnswer::Unknown; + return refuse(ProfileEvents::CASRelinkConfirmRefusedStateLockBusy, "the table's state lock is held"); /// Rule 2 (warm). An unrecovered or mid-recovery runtime has an EMPTY `state`, which would read as /// "the ref does not exist" -- knowledge it does not have. `superseded_by_remount` is the same @@ -469,16 +507,40 @@ ConfirmAnswer CasRefLedger::confirmExactRef(const RootNamespace & ns, const Stri if (!rt.recovered || rt.recovery_in_progress || rt.catalog_life_invalidated.load(std::memory_order_acquire) || rt.superseded_by_remount.load(std::memory_order_acquire)) - return ConfirmAnswer::Unknown; - - /// Rule 3 (lane quiescent). A wedge is "an object that may be durable and is not applied" -- it may - /// BE the removal being asked about. A pending item or an active leader tenure is a mutation this - /// table has already admitted; mid-tenure, a chunked flush has committed some of its transactions - /// and not others, and `leader_active` spans the whole tenure, so that partially-durable window is - /// covered too. None of the three says anything about WHICH ref is affected, so all three are - /// table-scoped refusals. - if (rt.lane_state != RefLaneState::Ready || !rt.pending.empty() || rt.leader_active) - return ConfirmAnswer::Unknown; + return refuse(ProfileEvents::CASRelinkConfirmRefusedMountCannotSpeak, + "the table is unrecovered, recovering, retired or superseded by a remount"); + + /// Rule 3 (no admitted mutation of THIS ref). The hazard is a committed row that lags a transaction + /// of the asked-about ref: the leader does not hold `state_mutex` across the `PUT`, so between + /// "durable" and "installed" that ref's row is stale, and a `Yes` read off it would authorize a + /// receiver to promote over a blob the transaction may already have retired. A mutation of ANOTHER + /// ref cannot change this ref's binding or the blobs its manifest protects, so its row is exactly as + /// authoritative as on an idle lane; refusing for it is what starved two replicas of each other on a + /// slow control plane. Every admitted mutation names its scope (`MutationScope`, recorded at + /// admission under `ref_queue_mutex` and validated against its ops at flush), and it is visible in + /// `pending` from admission to carve and in `carved` from carve to the tenure's exit guard, so "a + /// change of this ref is queued or in flight" is read from those two. The lane states other than + /// `Ready`/`Writing` refuse table-wide: `Wedged` holds a transaction that may be durable, and once + /// its tenure exits nothing but the attempt and the lane state records WHICH ref it touched -- the + /// exit guard clears the carved mirror, and the chunk's items were completed with an error before + /// that -- and `NeedsRecovery`, `Closed`, `Faulted` are fences on the whole view. + /// `Writing` with nothing carved cannot happen; it fails closed. + if (rt.lane_state == RefLaneState::Wedged) + return refuse(ProfileEvents::CASRelinkConfirmRefusedLaneWedged, "the lane holds an unresolved append"); + if (rt.lane_state != RefLaneState::Ready && rt.lane_state != RefLaneState::Writing) + return refuse(ProfileEvents::CASRelinkConfirmRefusedLaneBroken, "the lane is neither Ready nor Writing"); + if (rt.lane_state == RefLaneState::Writing && rt.carved.empty()) + return refuse(ProfileEvents::CASRelinkConfirmRefusedLaneBroken, "the lane is Writing with nothing carved"); + const auto covers = [&](const MutationScope & scope) + { + return scope.kind == MutationScope::Kind::WholeShard || scope.ref_name == ref_name; + }; + for (const auto & item : rt.pending) + if (covers(item->scope)) + return refuse(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight, "a queued mutation names this ref"); + for (const auto & item : rt.carved) + if (covers(item->scope)) + return refuse(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight, "a carved mutation names this ref"); /// Rule 5 (exact row equality) -- the only rule that can answer `No` at all. On a table that passed /// rules 2-4 the committed map is this writer's view, so a missing row or a different `ManifestRef` @@ -503,7 +565,8 @@ ConfirmAnswer CasRefLedger::confirmExactRef(const RootNamespace & ns, const Stri if (!fence_ok_fn() || rt.catalog_life_invalidated.load(std::memory_order_acquire) || rt.superseded_by_remount.load(std::memory_order_acquire)) - return ConfirmAnswer::Unknown; + return refuse(ProfileEvents::CASRelinkConfirmRefusedMountCannotSpeak, + "this mount no longer holds the namespace's write fence"); return ConfirmAnswer::Yes; } @@ -2162,6 +2225,11 @@ void CasRefLedger::completeOwnedItemsAndReleaseLeadership( /// no-op for them; it only matters for an item the leader owned but never got to carve. std::erase(rt->pending, owned); } + /// The tenure is over: every carved item is completed (above, or by its chunk's commit) and its + /// effect is either installed, or its failure is recorded by the lane state (`Wedged` for an + /// ambiguous `PUT`, `NeedsRecovery` for a durable-but-not-installed chunk), so the confirm no + /// longer needs to see it. + rt->carved.clear(); rt->leader_active = false; rt->cv.notify_all(); } @@ -2678,6 +2746,31 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptrref_name != scope_ref) + return &op.old_binding->ref_name; + if (op.new_binding && op.new_binding->ref_name != scope_ref) + return &op.new_binding->ref_name; + return nullptr; + } + if (op.kind == RefOpKind::SetPublishedAt && op.ref_name != scope_ref) + return &op.ref_name; + return nullptr; +} +} + void CasRefLedger::flushRefBatch(const RootNamespace & ns, const std::shared_ptr & rt, std::vector> & owned_items) { @@ -2863,16 +2956,18 @@ void CasRefLedger::flushRefBatch(const RootNamespace & ns, const std::shared_ptr /// `seen_refs`/`batch` growth and only recorded the batch into `owned_items` afterwards, so any throw /// after the first pop stranded already-popped items -- neither in `pending` nor in `owned_items` -- /// and their waiters hung forever. Instead: - /// PLAN (may throw, mutates NOTHING): under `ref_queue_mutex`, scan `pending` WITHOUT popping and - /// build the selection count, reserving every container (`batch`, `owned_items`) that the publish - /// below grows. A throw here leaves `pending`/`owned_items` byte-for-byte unchanged, so the - /// leadership-exit guard completes only the leader's own item and the untouched followers stay - /// queued for a later leader. + /// PLAN (may throw, mutates no CONTENT): under `ref_queue_mutex`, scan `pending` WITHOUT popping + /// and build the selection count, reserving CAPACITY in every container (`batch`, `owned_items`, + /// `rt->carved`) that the publish below grows -- a capacity change, never a size or content + /// change. A throw here leaves `pending`/`owned_items`/`carved` byte-for-byte unchanged in + /// content, so the leadership-exit guard completes only the leader's own item and the untouched + /// followers stay queued for a later leader. /// PUBLISH (no-throw): still under the SAME continuous `ref_queue_mutex` hold (no TOCTOU by /// construction), pop the selected front items and append them to `batch` and `owned_items` using /// only non-throwing operations (capacity pre-reserved; `shared_ptr` copies and `deque::pop_front` - /// never throw). ProfileEvents increments are deferred past the plan so the plan is literally - /// non-mutating. + /// never throw). ProfileEvents increments are deferred past the plan so the plan performs no + /// observable mutation beyond the reserved capacity above. The same items are appended to + /// `rt->carved`, the confirm-visible mirror the exit guard clears. std::vector> batch; { std::lock_guard g(ref_queue_mutex); @@ -2916,6 +3011,9 @@ void CasRefLedger::flushRefBatch(const RootNamespace & ns, const std::shared_ptr if (carve_hook_for_test) carve_hook_for_test(CarvePhaseForTest::PlanReserveOwned); owned_items.reserve(owned_items.size() + selected); + /// Reserve the confirm-visible mirror too, for the same reason: the publish appends into it and + /// must not throw. + rt->carved.reserve(rt->carved.size() + selected); /// --- PUBLISH (no-throw) --- if (carve_hook_for_test) @@ -2924,6 +3022,7 @@ void CasRefLedger::flushRefBatch(const RootNamespace & ns, const std::shared_ptr { batch.push_back(rt->pending.front()); /// shared_ptr copy, capacity reserved owned_items.push_back(rt->pending.front()); /// same item into the responsibility set + rt->carved.push_back(rt->pending.front()); /// and into the confirm-visible mirror rt->pending.pop_front(); } @@ -3047,10 +3146,34 @@ void CasRefLedger::flushRefBatch(const RootNamespace & ns, const std::shared_ptr /// this item. `item_ops` was built in step 1 against the pre-boundary state; the carve /// deduplicates ref names within a batch, so the overflowing item operates on a ref distinct /// from the just-committed chunk's and re-validating it against the reseeded `working` is - /// consistent. + /// consistent. The scope is validated here as well, because the confirm relies on it (see + /// `confirmExactRef`, rule 3). RefTableState item_scratch = working; try { + /// Scope validation. `MutationScope` is what `confirmExactRef` reads to decide whether a + /// queued or in-flight mutation may change the ref it is asked about, so a `Ref{name}` item + /// whose ops mutate ANOTHER ref would let the confirm answer `Yes` off a row this very item is + /// about to change. Checked before anything durable and failing only this item: every + /// production caller names the exact ref its ops mutate, so a mismatch is a programming error. + if (it->scope.kind == MutationScope::Kind::Ref) + { + for (const RefOp & op : item_ops) + { + /// A namespace removal names no ref and moves every row, so it is outside every + /// `Ref` scope. The confirm's answer about every OTHER ref rests on this scope + /// check alone, so it is rejected here rather than left to any later one. + if (op.kind == RefOpKind::RemoveNamespace) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "ref mutation on namespace '{}' is scoped to ref '{}' but its {} op moves every ref", + ns.string(), it->scope.ref_name, refOpKindToWireWord(op.kind)); + if (const String * other = refNamedOutsideScope(op, it->scope.ref_name)) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "ref mutation on namespace '{}' is scoped to ref '{}' but its {} op names ref '{}'", + ns.string(), it->scope.ref_name, refOpKindToWireWord(op.kind), *other); + } + } + /// Whole-item shape validation (prerequisite to `dropNamespace`): the /// per-op loop below previews each op as its OWN single-op trial transaction, so a /// whole-transaction-shape rule like "remove_namespace must be the FINAL op" trivially diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h index 163956607278..60c3066b29ed 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h @@ -37,11 +37,14 @@ enum class ResolveAudit : uint8_t { Emit, Deferred }; /// there is no independent apply marker or durable-id floor whose combinations form a second, /// implicit state machine. /// -/// `Ready` is the only state that admits a new append or certifies a cached row. `Writing` owns the +/// `Ready` is the state that admits a new append. A cached row is certified (`confirmExactRef`) in +/// `Ready` and in `Writing` alike, and in both only while no queued or carved mutation names that row's +/// ref -- a `Ready` lane with such a mutation queued refuses too. `Writing` owns the /// exact attempt before its first possible send. `Wedged` owns that same attempt after an ambiguous -/// result. `NeedsRecovery` means a transaction is known durable but cannot be installed in this cache; -/// it is a hard write and certification fence until replay completes. `Closed` records a successor's -/// epoch seal, and `Faulted` records foreign or internally inconsistent durable state. +/// result and certifies nothing. `NeedsRecovery` means a transaction is known durable but cannot be +/// installed in this cache; it is a hard write and certification fence until replay completes. `Closed` +/// records a successor's epoch seal, and `Faulted` records foreign or internally inconsistent durable +/// state. enum class RefLaneState : uint8_t { Ready, @@ -57,7 +60,9 @@ enum class RefLaneState : uint8_t /// `Yes` is the only answer that AUTHORIZES anything, so it is the only one that must be earned: it is /// returned exclusively when every rule of the lane snapshot holds. `Unknown` is the catch-all for /// every ambiguity, and it is the answer this primitive is biased towards: a cold, evicted, recovering, -/// busy or non-`Ready` table answers `Unknown` rather than doing any work to find out. +/// busy, fenced-out, wedged or otherwise broken table answers `Unknown` rather than doing any work to +/// find out, and so does a table with a queued or in-flight mutation of the asked-about ref (or of the +/// whole namespace); a mutation of another ref does not refuse. /// /// `No` means "this runtime's committed row for that ref is not the manifest you asked about" -- and /// nothing more. It is NOT a proof of the negative about the durable table, because the mount fence is @@ -154,7 +159,8 @@ class CasRefLedger /// receiver drives, so it must never be able to make this writer do work. /// /// The rules are evaluated as one snapshot spanning both lane mutexes, in this order: table warm - /// and resident; lane state `Ready`; exact committed-row equality; mount fence live last. Every + /// and resident; lane state `Ready` or `Writing`, with no queued or carved mutation whose + /// `MutationScope` covers the ref; exact committed-row equality; mount fence live last. Every /// ambiguity answers `Unknown` -- see `ConfirmAnswer`, and the .cpp for why the order and the /// two-mutex hold are what make a `Yes` a linearization point rather than a guess. ConfirmAnswer confirmExactRef(const RootNamespace & ns, const String & ref_name, @@ -481,6 +487,30 @@ class CasRefLedger return it != ref_name_slots.end() && it->second.current->leader_active; } + /// Returns the number of items carved by the current tenure and not yet released by its exit guard. + /// Under the queue mutex, like `refQueuePendingForTest`. + size_t refCarvedForTest(const RootNamespace & ns) + { + std::lock_guard g(ref_queue_mutex); + const auto it = ref_name_slots.find(ns.string()); + return it == ref_name_slots.end() ? 0 : it->second.current->carved.size(); + } + + /// Returns whether the `carved` entry for `ref_name` (if any) is already completed. Lets a test + /// PROVE an earlier chunk's item is done rather than infer it from carve-hook ordering. Under the + /// queue mutex, like `refCarvedForTest`; `done` itself is guarded by the same mutex. + bool refCarvedItemDoneForTest(const RootNamespace & ns, const String & ref_name) + { + std::lock_guard g(ref_queue_mutex); + const auto it = ref_name_slots.find(ns.string()); + if (it == ref_name_slots.end()) + return false; + for (const auto & item : it->second.current->carved) + if (item->scope.kind == MutationScope::Kind::Ref && item->scope.ref_name == ref_name) + return item->done; + return false; + } + /// Returns the number of callers currently waiting for `ns` recovery under its state mutex. uint64_t refRecoveryWaitersForTest(const RootNamespace & ns) { @@ -712,7 +742,7 @@ class CasRefLedger /// One coherent decoded `RefTableState` and append runtime for a namespace. It is recovered lazily /// and evicted only as a whole. `state_mutex` is separate from - /// `ref_queue_mutex` (which only ever guards `pending`/`leader_active`) so a reader (resolveRef/ + /// `ref_queue_mutex` (which only ever guards `pending`/`carved`/`leader_active`) so a reader (resolveRef/ /// listRefs) can observe `state` without contending with the flush leader's network round trip -- /// the leader only holds `state_mutex` for the brief copy-out-before-validate and the /// apply-after-commit steps, never for the `putIfAbsentControlled` call itself. @@ -844,6 +874,18 @@ class CasRefLedger uint64_t publish_backoff_ms = 0; std::deque> pending; /// guarded by ref_queue_mutex + /// The current tenure's carved items, from the carve (`flushRefBatch`'s PUBLISH phase) to the + /// tenure's exit guard (`completeOwnedItemsAndReleaseLeadership`), both under `ref_queue_mutex`. + /// The carve pops an item out of `pending`, so `pending` alone cannot show a mutation between + /// carve and install -- the window in which its transaction may be durable while the committed + /// row still lags it. The mirror makes that item visible, under the same mutex, to a reader + /// (such as `confirmExactRef`) that holds only the runtime. An item is completed by its chunk's + /// install or earlier by an error, often long before the exit guard; the mirror keeps it + /// regardless. + /// Over-inclusive on purpose: an installed item and an item that failed validation before any + /// send both stay here until the exit guard; that is one tenure of over-refusal for their refs, + /// never an under-refusal. + std::vector> carved; /// guarded by ref_queue_mutex bool leader_active = false; /// guarded by ref_queue_mutex /// Set before the exact `Live -> Removing` catalog CAS and retained until that life is deleted. /// New positive mutations check it in the same queue critical section as admission; the one @@ -1108,9 +1150,10 @@ class CasRefLedger /// and apply-after-commit ordering -- the LIVE state is still only ever advanced once the object is /// durable; `commitRefChunk`'s pre-`PUT` apply targets a private candidate that nothing else can /// observe. Every item it carves out of `pending` is appended to - /// `owned_items` (the leader's responsibility set) at the moment it is carved. When a batch's total - /// op count exceeds `ref_txn_max_ops`, the validation loop emits SEVERAL ref-log transactions in one - /// tenure via `commitRefChunk` -- each a complete commit boundary. + /// `owned_items` (the leader's responsibility set) and to `rt->carved` (the confirm-visible mirror, + /// see `RefTableRuntime::carved`) at the moment it is carved. When a batch's total op count exceeds + /// `ref_txn_max_ops`, the validation loop emits SEVERAL ref-log transactions in one tenure via + /// `commitRefChunk` -- each a complete commit boundary. void flushRefBatch(const RootNamespace & ns, const std::shared_ptr & rt, std::vector> & owned_items); @@ -1179,10 +1222,12 @@ class CasRefLedger /// Leadership-exit guard for `appendRefOps`: under `ref_queue_mutex`, completes every still-unfinished /// item this leader owned (with `flush_exception` when unwinding, or a fail-closed `LOGICAL_ERROR` - /// otherwise), removes each owned item from `pending` so no future leader can carve it, and releases - /// leadership (`leader_active = false` + `cv.notify_all`). On the normal path every owned item is - /// already `done`, so only the leadership release has effect. This is the single authority that - /// resets `leader_active` on any exit from the leader loop. + /// otherwise), removes each owned item from `pending` so no future leader can carve it, clears + /// `rt->carved` (the confirm-visible mirror, now that every carved item's fate -- installed or a + /// recorded lane failure -- no longer needs to be seen), and releases leadership + /// (`leader_active = false` + `cv.notify_all`). On the normal path every owned item is already + /// `done`, so only the leadership release and the mirror clear have effect. This is the single + /// authority that resets `leader_active` on any exit from the leader loop. void completeOwnedItemsAndReleaseLeadership( const RootNamespace & ns, const std::shared_ptr & rt, const std::vector> & owned_items, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h index db061bb7ebcb..76a40bcc143d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h @@ -60,6 +60,10 @@ enum class RootMutationOrigin : uint8_t /// call touches. The flat-combining batch builder admits at most ONE mutation per ref name into a /// single flush (per-ref durable histories stay bit-identical to the unbatched protocol) and flushes /// `WholeShard` calls SOLO (dropNamespace and anything touching multiple refs wholesale). +/// +/// It is also safety-bearing: `CasRefLedger::confirmExactRef` refuses to certify a ref while a queued +/// or carved item's scope covers it, and `flushRefBatch` fails an item whose ops name a ref outside its +/// declared scope, so a caller must name exactly the ref its ops mutate. struct MutationScope { enum class Kind : uint8_t { Ref, WholeShard }; diff --git a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp index 4b19f9ae967e..f129949c0590 100644 --- a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp +++ b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp @@ -2,6 +2,8 @@ #include "config.h" +#include + #include #include #include @@ -10,9 +12,11 @@ #include #include #include +#include #include #include #include +#include #include #include @@ -42,8 +46,10 @@ /// recover from storage to answer, and it must not even MATERIALIZE a runtime -- a read-only /// interserver query must never be able to make this writer do work. /// - The snapshot spans BOTH lane mutexes, so an append admitted concurrently is ordered strictly -/// after it: there is no window in which the confirm says `Yes` while a removal of that ref is -/// already admitted. +/// after it: there is no window in which the confirm says `Yes` while a mutation OF THAT REF is +/// already admitted. A queued or in-flight mutation of another SINGLE ref does not refuse -- rule 3 +/// reads each item's `MutationScope`, and refusing for the whole table starved two replicas of each +/// other under load on a slow control plane. A `WholeShard`-scoped mutation still refuses every ref. /// /// The suite name is prefixed `Cas` so it is covered by the `Cas*` unit-test gate filter. @@ -51,6 +57,15 @@ namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; extern const int MEMORY_LIMIT_EXCEEDED; +extern const int LOGICAL_ERROR; +} + +namespace ProfileEvents +{ + extern const Event CASRelinkConfirmRefusedRefMutationInFlight; + extern const Event CASRelinkConfirmRefusedLaneWedged; + extern const Event CASRelinkConfirmRefusedLaneBroken; + extern const Event CASRelinkConfirmRefusedMountCannotSpeak; } using namespace DB::Cas; @@ -174,16 +189,17 @@ struct CaseSync bool entered = false; }; -/// `num_pairs` add-then-remove precommit op pairs for distinct refs, each naming a distinct manifest. -/// Every pair is undone immediately, so the LIVE state stays ~empty and validating thousands of ops -/// stays linear -- it is the OP COUNT, not the resident state, that drives the chunk split under test. -std::vector precommitAddRemovePairs(const String & prefix, size_t num_pairs, uint64_t manifest_epoch) +/// `num_pairs` add-then-remove precommit op pairs on ONE ref, each pair naming a distinct manifest, +/// so an item scoped `MutationScope::ref(ref)` names exactly the ref its ops mutate (the flush +/// validates that). Every pair is undone immediately, so the LIVE state stays ~empty and validating +/// thousands of ops stays linear -- it is the OP COUNT, not the resident state, that drives the chunk +/// split under test. +std::vector precommitAddRemovePairs(const String & ref, size_t num_pairs, uint64_t manifest_epoch) { std::vector ops; ops.reserve(num_pairs * 2); for (size_t i = 0; i < num_pairs; ++i) { - const String ref = prefix + std::to_string(i); const ManifestRef manifest{manifest_epoch, i + 1, 1}; RefOp add; add.kind = RefOpKind::OwnerTransition; @@ -233,6 +249,16 @@ uint64_t backendRequests(const CountingBackend & b) return b.headTotal() + b.getTotal() + b.getStreamTotal() + b.putTotal() + b.listTotal(); } +/// One refusal counter's current value. `confirmExactRef` attributes every `Unknown` to exactly one of +/// these, and a live gate reads them to tell load from a fault from a lost mount -- distinctions the +/// three-value `ConfirmAnswer` cannot carry. A test that checks only the ANSWER passes just as happily +/// when two of them are swapped, so the tests that reach a refusal deterministically assert the +/// attribution as a DELTA around the confirm, never an absolute (the suite shares one process). +uint64_t refusalCount(ProfileEvents::Event event) +{ + return ProfileEvents::global_counters[event].load(); +} + /// One-shot throwing probe in the post-durable install region -- the only way to reach `NeedsRecovery` /// transition now that §A1 made every install region allocation-free. Copied in shape from /// `gtest_cas_ref_install_safety.cpp`: the exception is built OUTSIDE the region (constructing one @@ -394,7 +420,15 @@ TEST(CASConfirmExactRef, UnrecoveredResidentTableIsUnknownWithZeroBackendRequest ASSERT_FALSE(store->refTableCachedForTest(ns)) << "that runtime must be UNRECOVERED"; backend->resetCounts(); + /// An unrecovered table is not a lane fault and not load: it is this mount being unable to speak + /// for the namespace, and the live gate must be able to tell those apart. + const uint64_t cannot_speak_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedMountCannotSpeak); + const uint64_t broken_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); EXPECT_EQ(store->confirmExactRef(ns, "x", ManifestRef{1, 1, 1}), ConfirmAnswer::Unknown); + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedMountCannotSpeak) - cannot_speak_before, 1u) + << "an unrecovered view must be reported as this mount being unable to speak for the table"; + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken) - broken_before, 0u) + << "an unrecovered table is not a lane defect; the live gate asserts LaneBroken is zero"; EXPECT_EQ(backendRequests(*backend), 0u) << "an unrecovered table must answer Unknown without driving recovery"; EXPECT_FALSE(store->refTableCachedForTest(ns)) @@ -467,8 +501,8 @@ TEST(CASConfirmExactRef, RecoveryInProgressIsUnknownWithZeroBackendRequests) /// Rule 3, the in-flight case: an append is admitted and its leader is parked in the pre-carve window. /// Nothing is durable yet and the committed row still matches EXACTLY -- which is precisely why a /// naive implementation answers `Yes` here, and precisely why that is the TOCTOU this design closes. -/// The apply-state is still `Clean` at this point, so rule 3 is what produces the `Unknown`, not -/// rule 4. +/// The lane state is still `Ready` and the item is in `pending`, so rule 3 reading the item's scope is +/// what produces the `Unknown`. TEST(CASConfirmExactRef, InFlightAppendIsUnknown) { auto backend = std::make_shared(); @@ -498,18 +532,20 @@ TEST(CASConfirmExactRef, InFlightAppendIsUnknown) EXPECT_EQ(apply_state, RefLaneState::Ready) << "the pre-carve window is before any PUT, so rule 4 must not be what answers here"; EXPECT_EQ(while_in_flight, ConfirmAnswer::Unknown) - << "an admitted append makes the whole table's committed view provisional"; + << "an admitted mutation of THIS ref makes its committed row provisional"; EXPECT_EQ(store->confirmExactRef(ns, "x", id.ref), ConfirmAnswer::No); } -/// Rule 3, mid-tenure (spec §testing "mid-tenure chunked flush", `CarvePhaseForTest::ChunkReseed`): -/// one leader tenure commits MULTIPLE durable transactions, so at a chunk boundary the table is -/// PARTIALLY durable. `leader_active` covers the whole tenure, so this is already `Unknown` -- a wider -/// unknown window under load, never a hole. The confirm is issued on the leader's own thread, which is -/// safe because the boundary holds neither lane mutex. -TEST(CASConfirmExactRef, MidTenureChunkBoundaryIsUnknown) +/// Rule 3 at a chunk boundary (`CarvePhaseForTest::ChunkReseed`): one leader tenure commits MULTIPLE +/// durable transactions, so between two chunks the table is PARTIALLY durable -- for the refs those +/// chunks mutate. The seed ref is touched by neither, so its row is exactly as authoritative as on an +/// idle lane and it confirms; BOTH carved items' own refs refuse, because their transactions may be +/// durable and not installed -- the second one is what makes a rule that read only the front of the +/// mirror visible. The confirm is issued on the leader's own thread, which is safe because the +/// boundary holds neither lane mutex. +TEST(CASConfirmExactRef, UntouchedRefConfirmsMidTenure) { auto backend = std::make_shared(); auto store = openPool(backend); @@ -519,7 +555,9 @@ TEST(CASConfirmExactRef, MidTenureChunkBoundaryIsUnknown) ASSERT_EQ(store->confirmExactRef(ns, "seed", id.ref), ConfirmAnswer::Yes); std::atomic boundaries{0}; - std::atomic unknown_at_boundary{0}; + std::atomic yes_for_seed_at_boundary{0}; + std::atomic unknown_for_carved_ref_at_boundary{0}; + std::atomic unknown_for_second_carved_ref_at_boundary{0}; std::atomic requests_at_boundary{0}; store->setCarveHookForTest([&](CasRefLedger::CarvePhaseForTest phase) { @@ -527,8 +565,17 @@ TEST(CASConfirmExactRef, MidTenureChunkBoundaryIsUnknown) return; boundaries.fetch_add(1); const uint64_t before = backendRequests(*backend); - if (store->confirmExactRef(ns, "seed", id.ref) == ConfirmAnswer::Unknown) - unknown_at_boundary.fetch_add(1); + if (store->confirmExactRef(ns, "seed", id.ref) == ConfirmAnswer::Yes) + yes_for_seed_at_boundary.fetch_add(1); + /// "aaa_" has no committed row (rule 5 would say `No`), so an `Unknown` here can only come from + /// rule 3 reading the carved item's scope. + if (store->confirmExactRef(ns, "aaa_", id.ref) == ConfirmAnswer::Unknown) + unknown_for_carved_ref_at_boundary.fetch_add(1); + /// "bbb_" is the mirror's SECOND entry and likewise has no committed row, so this is the same + /// assertion made about an entry a rule that stopped at the front of `carved` would never + /// reach. + if (store->confirmExactRef(ns, "bbb_", id.ref) == ConfirmAnswer::Unknown) + unknown_for_second_carved_ref_at_boundary.fetch_add(1); requests_at_boundary.fetch_add(static_cast(backendRequests(*backend) - before)); }); @@ -549,10 +596,10 @@ TEST(CASConfirmExactRef, MidTenureChunkBoundaryIsUnknown) [&] { return store->refQueuePendingForTest(ns) >= 2; }); }); - auto append = [&store, &ns](const String & prefix, uint64_t manifest_epoch) + auto append = [&store, &ns](const String & ref, uint64_t manifest_epoch) { - std::vector item_ops = precommitAddRemovePairs(prefix, 1500, manifest_epoch); - store->appendRefOps(ns, MutationScope::ref(prefix), + std::vector item_ops = precommitAddRemovePairs(ref, 1500, manifest_epoch); + store->appendRefOps(ns, MutationScope::ref(ref), [ops = std::move(item_ops)](const RefTableState &) { return ops; }, RootMutationOrigin::Writer, RootMutationKind::Publish); }; @@ -574,11 +621,16 @@ TEST(CASConfirmExactRef, MidTenureChunkBoundaryIsUnknown) store->setCarveHookForTest(nullptr); ASSERT_GE(boundaries.load(), 1) << "the flush did not chunk -- the mid-tenure window was not exercised"; - EXPECT_EQ(unknown_at_boundary.load(), boundaries.load()) - << "a mid-tenure, partially-durable table must never confirm"; + EXPECT_EQ(yes_for_seed_at_boundary.load(), boundaries.load()) + << "a ref no carved item names must confirm mid-tenure -- that is the liveness this rule exists for"; + EXPECT_EQ(unknown_for_carved_ref_at_boundary.load(), boundaries.load()) + << "a ref a carved item names must not confirm while its transaction may be durable and not installed"; + EXPECT_EQ(unknown_for_second_carved_ref_at_boundary.load(), boundaries.load()) + << "rule 3 must scan the whole carved mirror: 'bbb_' is its second entry and has no committed " + "row, so a rule that examined only the front entry would answer No here"; EXPECT_EQ(requests_at_boundary.load(), 0) << "the mid-tenure confirm must still be I/O-free"; - /// The tenure is over: the seed ref is untouched by it and confirms again. + /// The tenure is over: the seed ref confirms as before. EXPECT_EQ(store->confirmExactRef(ns, "seed", id.ref), ConfirmAnswer::Yes); } @@ -607,6 +659,260 @@ TEST(CASConfirmExactRef, WedgedLaneIsUnknown) } +/// Rule 3, a REAL wedge: the removal of x is sent, the response is lost, the single-attempt budget is +/// exhausted, and the lane wedges. `commitRefChunk` completes the chunk's items with an error before +/// the tenure ends, so the transaction that may be durable is recorded nowhere but in the attempt and +/// the lane state -- `pending` and `carved` are both empty. Every ref refuses: x because its removal +/// may be durable, `other` because nothing but the lane state records WHICH ref the wedged transaction +/// touched. +TEST(CASConfirmExactRef, WedgedTransactionRefusesEveryRef) +{ + auto backend = std::make_shared(); + PoolConfig cfg; + /// Single-attempt budget: one ambiguous PUT is a conclusive wedge, no inter-attempt sleep. The + /// operation deadline is deliberately far wider than one attempt, so the pre-send gate never + /// refuses before the injected fault is reached and the outcome is decided by the fault, not by + /// how loaded the machine is. + CasRequestBudget budget; + budget.max_attempts = 1; + budget.attempt_timeout_ms = 100; + budget.operation_deadline_ms = 5000; + budget.lease_safety_margin_ms = 100; + cfg.cas_request_budget = budget; + auto store = openPoolWithConfig(backend, cfg); + const RootNamespace ns{"srv1/confirm_real_wedge"}; + /// Pins the namespace to the fixture life BEFORE its first real touch, so the fault key computed + /// from that same life below is the key production actually writes to. + DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns); + + const ManifestId id_x = publishEmptyPart(store, ns, "x"); + const ManifestId id_other = publishEmptyPart(store, ns, "other"); + ASSERT_EQ(store->confirmExactRef(ns, "x", id_x.ref), ConfirmAnswer::Yes); + ASSERT_EQ(store->confirmExactRef(ns, "other", id_other.ref), ConfirmAnswer::Yes); + + backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; + backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; + backend->fault_skip = 0; + backend->fault_count = 1; + EXPECT_THROW(store->dropRef(ns, "x"), DB::Exception); + ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + ASSERT_EQ(store->refQueuePendingForTest(ns), 0u); + ASSERT_EQ(store->refCarvedForTest(ns), 0u) + << "the wedged item was completed and released; only the lane state records its transaction"; + + backend->resetCounts(); + /// A wedge and a broken lane are two DIFFERENT things to a live gate -- one is an unresolved append + /// that the next flush or a remount clears, the other is a lane defect. Both refuse here, and + /// without these deltas the test would pass just as happily if the wedge branch were deleted and + /// the broken-lane branch answered for it. + const uint64_t wedged_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneWedged); + const uint64_t broken_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + EXPECT_EQ(store->confirmExactRef(ns, "x", id_x.ref), ConfirmAnswer::Unknown) << "x's removal may be durable"; + EXPECT_EQ(store->confirmExactRef(ns, "other", id_other.ref), ConfirmAnswer::Unknown) + << "a wedge refuses table-wide: no per-ref record of the wedged transaction survives the tenure"; + EXPECT_EQ(backendRequests(*backend), 0u) << "a wedged lane must answer without trying to resolve the wedge"; + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneWedged) - wedged_before, 2u) + << "both refusals must be reported as a wedge"; + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken) - broken_before, 0u) + << "a wedge is not a lane defect: reporting it as one would send the live gate hunting a bug"; +} + + +/// `carved` bookkeeping: a carved item leaves `pending` at the carve and is completed by its chunk's +/// install (or earlier, by an error), while the mirror is cleared only at the tenure's exit guard, so +/// the confirm reads the item from `rt.carved` from carve to tenure end. Sampled at +/// `PostDurableInstall` -- the transaction is durable, nothing is installed, `pending` is already +/// empty -- and again after the tenure: the mirror must hold exactly the carved item during, and be +/// empty after. The hook runs on the leader's own thread with neither lane mutex held, so the seams +/// (which take `ref_queue_mutex`) are safe to call from it. +TEST(CASConfirmExactRef, CarvedItemIsVisibleFromCarveToTenureEnd) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_carved"}; + publishEmptyPart(store, ns, "x"); + + std::atomic samples{0}; + std::atomic carved_during{0}; + std::atomic pending_during{0}; + store->setCarveHookForTest([&](CasRefLedger::CarvePhaseForTest phase) + { + if (phase != CasRefLedger::CarvePhaseForTest::PostDurableInstall) + return; + samples.fetch_add(1); + carved_during.store(store->refCarvedForTest(ns)); + pending_during.store(store->refQueuePendingForTest(ns)); + }); + store->dropRef(ns, "x"); + store->setCarveHookForTest(nullptr); + + ASSERT_EQ(samples.load(), 1) << "the drop must commit exactly one chunk"; + EXPECT_EQ(carved_during.load(), 1u) + << "the carved removal must be visible while its transaction is durable but not installed"; + EXPECT_EQ(pending_during.load(), 0u) + << "the carve popped the item out of pending -- carved is the only place it can be seen"; + EXPECT_EQ(store->refCarvedForTest(ns), 0u) << "the exit guard must release the mirror"; +} + +/// Scope validation: `MutationScope` is what the confirm reads to decide whether an in-flight mutation +/// may change the ref it is asked about, so an item scoped to ref X must fail, alone, before anything +/// is durable, both when its ops mutate ref Y and when they carry a namespace removal, which names no +/// ref and moves every row. It throws `LOGICAL_ERROR`, which aborts the process in debug and +/// sanitizer builds instead of behaving like a catchable exception -- +/// `CASConfirmExactRefDeathTest.MisScopedItemAborts` below proves the abort positively in those builds. +#ifndef DEBUG_OR_SANITIZER_BUILD +TEST(CASConfirmExactRef, MisScopedItemFailsBeforeAnythingIsDurable) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_misscoped"}; + const ManifestId seed = publishEmptyPart(store, ns, "seed"); /// the namespace is born already + + const uint64_t puts_before = backend->putTotal(); + const uint64_t overwrites_before = backend->putOverwriteTotal(); + const uint64_t cas_puts_before = backend->casPutTotal(); + RefOp add; + add.kind = RefOpKind::OwnerTransition; + add.new_binding = RefOwnerBinding{RefOwnerKind::Precommit, "y", ManifestRef{900000003, 1, 1}}; + try + { + store->appendRefOps(ns, MutationScope::ref("x"), + [add](const RefTableState &) { return std::vector{add}; }, + RootMutationOrigin::Writer, RootMutationKind::Publish); + FAIL() << "an item scoped to ref 'x' whose op binds ref 'y' must be refused"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::LOGICAL_ERROR); + } + /// A namespace removal names no ref at all, so a scope check that only compared names would let it + /// through -- and it moves every row, which is the one thing a `Ref` scope promises the confirm + /// will not happen behind its back. + RefOp remove_namespace; + remove_namespace.kind = RefOpKind::RemoveNamespace; + try + { + store->appendRefOps(ns, MutationScope::ref("x"), + [remove_namespace](const RefTableState &) { return std::vector{remove_namespace}; }, + RootMutationOrigin::Writer, RootMutationKind::Publish); + FAIL() << "an item scoped to ref 'x' carrying a namespace removal must be refused"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::LOGICAL_ERROR) + << "the scope check must be what rejects it"; + } + + /// The ref-log transaction object -- the only thing that would make this item durable -- is a + /// `putIfAbsent`; `putOverwrite` and `casPut` are asserted too so the fence covers every write kind + /// the backend can observe, not just the one this item would have used. + EXPECT_EQ(backend->putTotal(), puts_before) << "the refusal must happen before any object is written"; + EXPECT_EQ(backend->putOverwriteTotal(), overwrites_before) << "the refusal must happen before any object is written"; + EXPECT_EQ(backend->casPutTotal(), cas_puts_before) << "the refusal must happen before any object is written"; + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready) << "a validation failure is not a lane fault"; + EXPECT_EQ(store->confirmExactRef(ns, "seed", seed.ref), ConfirmAnswer::Yes) + << "the failed item must leave the table exactly as it was"; +} +#endif + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASConfirmExactRefDeathTest, MisScopedItemAborts) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_misscoped"}; + publishEmptyPart(store, ns, "seed"); + + RefOp add; + add.kind = RefOpKind::OwnerTransition; + add.new_binding = RefOwnerBinding{RefOwnerKind::Precommit, "y", ManifestRef{900000003, 1, 1}}; + EXPECT_DEATH({ + store->appendRefOps(ns, MutationScope::ref("x"), + [add](const RefTableState &) { return std::vector{add}; }, + RootMutationOrigin::Writer, RootMutationKind::Publish); + }, ""); + + RefOp remove_namespace; + remove_namespace.kind = RefOpKind::RemoveNamespace; + EXPECT_DEATH({ + store->appendRefOps(ns, MutationScope::ref("x"), + [remove_namespace](const RefTableState &) { return std::vector{remove_namespace}; }, + RootMutationOrigin::Writer, RootMutationKind::Publish); + }, ""); +} +#endif + +/// The mirror must survive an item's COMPLETION, not just its carve: an item is completed by its own +/// chunk's commit, often chunks before the tenure ends. Two items whose op counts force a chunk split +/// (mirrors `UntouchedRefConfirmsMidTenure`'s co-batching) are carved together in one tenure: chunk 1 +/// = {aaa_} alone, chunk 2 = {bbb_} alone. Sampled at chunk 2's `PostDurableInstall`, the test PROVES -- +/// via `refCarvedItemDoneForTest`, not by inferring from hook order -- that aaa_ is already done, and +/// that it is still counted in `carved` alongside bbb_ until the tenure's exit guard. +TEST(CASConfirmExactRef, CarvedItemSurvivesEarlierChunkCompletion) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_carved_multi_chunk"}; + publishEmptyPart(store, ns, "seed"); + + std::atomic boundaries{0}; + std::atomic aaa_done_at_second_boundary{false}; + std::atomic carved_at_second_boundary{0}; + store->setCarveHookForTest([&](CasRefLedger::CarvePhaseForTest phase) + { + if (phase != CasRefLedger::CarvePhaseForTest::PostDurableInstall) + return; + if (boundaries.fetch_add(1) + 1 != 2) + return; /// only chunk 2's durable point is of interest + aaa_done_at_second_boundary.store(store->refCarvedItemDoneForTest(ns, "aaa_")); + carved_at_second_boundary.store(store->refCarvedForTest(ns)); + }); + + /// Co-batching setup identical to `UntouchedRefConfirmsMidTenure`: the pre-carve hook parks the + /// first caller until the second is queued, so both items are carved together deterministically. + auto sync = std::make_shared(); + store->setRefPreCarveHookForTest([sync, store, ns] + { + std::unique_lock lk(sync->m); + if (sync->entered) + return; + sync->entered = true; + sync->cv.notify_all(); + sync->cv.wait_for(lk, std::chrono::seconds(20), + [&] { return store->refQueuePendingForTest(ns) >= 2; }); + }); + + auto append = [&store, &ns](const String & ref, uint64_t manifest_epoch) + { + std::vector item_ops = precommitAddRemovePairs(ref, 1500, manifest_epoch); + store->appendRefOps(ns, MutationScope::ref(ref), + [ops = std::move(item_ops)](const RefTableState &) { return ops; }, + RootMutationOrigin::Writer, RootMutationKind::Publish); + }; + std::thread a([&] { append("aaa_", 900000001); }); + { + std::unique_lock lk(sync->m); + sync->cv.wait_for(lk, std::chrono::seconds(20), [&] { return sync->entered; }); + } + std::thread b([&] { append("bbb_", 900000002); }); + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20); + while (store->refQueuePendingForTest(ns) < 2 && std::chrono::steady_clock::now() < deadline) + std::this_thread::yield(); + sync->cv.notify_all(); + a.join(); + b.join(); + store->setRefPreCarveHookForTest(nullptr); + store->setCarveHookForTest(nullptr); + + ASSERT_EQ(boundaries.load(), 2) << "the flush must chunk into exactly two transactions"; + ASSERT_TRUE(aaa_done_at_second_boundary.load()) + << "chunk 1's item must already be done by the time chunk 2 goes durable"; + EXPECT_EQ(carved_at_second_boundary.load(), 2u) + << "a completed item must still be counted in the mirror until the tenure's exit guard"; + EXPECT_EQ(store->refCarvedForTest(ns), 0u) << "the exit guard must release the mirror after both chunks"; +} + + /// `NeedsRecovery` is table-scoped, so confirmation refuses even a row that still looks perfect. TEST(CASConfirmExactRef, NeedsRecoveryIsUnknown) { @@ -625,8 +931,16 @@ TEST(CASConfirmExactRef, NeedsRecoveryIsUnknown) ASSERT_FALSE(store->refLaneWedgedForTest(ns)); ASSERT_FALSE(store->refLeaderActiveForTest(ns)); + /// The positive side of the wedge case's negative: `NeedsRecovery` is the lane defect + /// `LaneBroken` is FOR, so this is the one refusal that must be reported as one. + const uint64_t broken_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + const uint64_t wedged_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneWedged); EXPECT_EQ(store->confirmExactRef(ns, "keep", keep.ref), ConfirmAnswer::Unknown) << "a table that may be missing a durable transaction cannot confirm ANY of its rows"; + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken) - broken_before, 1u) + << "a NeedsRecovery lane is exactly what LaneBroken reports"; + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneWedged) - wedged_before, 0u) + << "the lane is not wedged here, and the two counters must not be interchangeable"; } @@ -647,7 +961,12 @@ TEST(CASConfirmExactRef, LostMountFenceIsUnknown) ASSERT_TRUE(store->refTableCachedForTest(ns)) << "the table must still be resident, so it is the FENCE that refuses, not residency"; backend->resetCounts(); + /// The other arm of the same counter: rule 6's fence check. Losing the mount is the most + /// safety-relevant refusal this function has, so it must not be the silent one. + const uint64_t cannot_speak_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedMountCannotSpeak); EXPECT_EQ(store->confirmExactRef(ns, "x", id.ref), ConfirmAnswer::Unknown); + EXPECT_EQ(refusalCount(ProfileEvents::CASRelinkConfirmRefusedMountCannotSpeak) - cannot_speak_before, 1u) + << "a refusal for a lost mount fence must be counted, not invisible"; EXPECT_EQ(backendRequests(*backend), 0u); /// The fence is checked LAST, so it gates only the `Yes`: a token that does not match the committed @@ -709,6 +1028,225 @@ TEST(CASConfirmExactRef, ConcurrentAppendIsOrderedAfterTheSnapshot) } +/// The livelock shape: a mutation of ANOTHER ref is queued and its leader is parked before the carve, +/// so the lane has a pending item and an active tenure. A confirm about an untouched committed ref must +/// answer `Yes` -- the queued mutation cannot change this ref's binding or the blobs its manifest +/// protects -- while the queued ref itself answers `Unknown`. +/// +/// A SECOND queued mutation, of a third ref, sits behind the leader's own item, so the queue holds two +/// and the ref asked about last is not at its front: that is what makes a rule reading only +/// `pending`'s front item visible, which every other test in this file would pass. +TEST(CASConfirmExactRef, UntouchedRefConfirmsWhileAnotherRefIsQueued) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_liveness"}; + + const ManifestId id_x = publishEmptyPart(store, ns, "x"); + const ManifestId id_other = publishEmptyPart(store, ns, "other"); + const ManifestId id_third = publishEmptyPart(store, ns, "third"); + ASSERT_EQ(store->confirmExactRef(ns, "x", id_x.ref), ConfirmAnswer::Yes); + + LeaderLatch latch; + latch.arm(store); + std::thread dropper([&] { store->dropRef(ns, "other"); }); + latch.awaitEntered(); + /// The leader pushes its own item before it takes the baton, so the queue is [other, third] and + /// `third` is reachable only by a scan that goes past the front. + std::thread second_dropper([&] { store->dropRef(ns, "third"); }); + const auto queued_by = std::chrono::steady_clock::now() + std::chrono::seconds(20); + while (store->refQueuePendingForTest(ns) < 2 && std::chrono::steady_clock::now() < queued_by) + std::this_thread::yield(); + + /// Sampled while parked, asserted after the join (a failed assertion here would skip the release). + const bool leader_active = store->refLeaderActiveForTest(ns); + const size_t pending = store->refQueuePendingForTest(ns); + /// The refusal counters are the only way a live gate can read WHY a confirm said `Unknown`, so the + /// attribution is pinned here rather than left to the reader of the .cpp: this pair of confirms is + /// the one place where a `Yes` and a scope-driven `Unknown` are produced back to back from the same + /// lane state. + const uint64_t in_flight_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const uint64_t broken_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + const ConfirmAnswer untouched = store->confirmExactRef(ns, "x", id_x.ref); + const uint64_t in_flight_after_untouched + = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const ConfirmAnswer touched = store->confirmExactRef(ns, "other", id_other.ref); + const uint64_t in_flight_after_touched + = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const ConfirmAnswer touched_behind_the_front = store->confirmExactRef(ns, "third", id_third.ref); + const uint64_t in_flight_after_behind_the_front + = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const uint64_t broken_after = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + + latch.release(); + dropper.join(); + second_dropper.join(); + store->setRefPreCarveHookForTest(nullptr); + + EXPECT_TRUE(leader_active); + EXPECT_EQ(pending, 2u) << "the second dropper must be queued behind the parked leader's own item"; + EXPECT_EQ(in_flight_after_untouched - in_flight_before, 0u) + << "a confirm that answers Yes must not be counted as a refusal"; + EXPECT_EQ(in_flight_after_touched - in_flight_after_untouched, 1u) + << "the refused confirm must be attributed to the ref-scoped mutation, which is what a live " + "gate reads to tell load from a lane fault"; + EXPECT_EQ(in_flight_after_behind_the_front - in_flight_after_touched, 1u) + << "the second queued item's ref must be refused for the same reason as the first"; + EXPECT_EQ(broken_after - broken_before, 0u) + << "the lane is Ready here: a refusal attributed to a broken lane would misreport a fault"; + EXPECT_EQ(untouched, ConfirmAnswer::Yes) + << "a queued mutation of another ref must not refuse this one"; + EXPECT_EQ(touched, ConfirmAnswer::Unknown) + << "the queued ref's own row is provisional"; + EXPECT_EQ(touched_behind_the_front, ConfirmAnswer::Unknown) + << "rule 3 must scan the whole pending queue, not only its front item"; + EXPECT_EQ(store->confirmExactRef(ns, "x", id_x.ref), ConfirmAnswer::Yes); + EXPECT_EQ(store->confirmExactRef(ns, "other", id_other.ref), ConfirmAnswer::No); + EXPECT_EQ(store->confirmExactRef(ns, "third", id_third.ref), ConfirmAnswer::No); +} + + +/// Rule 3's `WholeShard` arm. A mutation that declares no ref is one that may move EVERY row, so it +/// refuses every ref for as long as it is queued or carved -- unlike a `Ref`-scoped neighbour, which +/// refuses only its own. In production `dropNamespaceImpl` and `sweepStalePrecommitsNow` are the two +/// appenders that declare `wholeShard()`. +/// +/// The two confirms of "x" differ in exactly one thing: whether the whole-shard item has been queued. +/// The first is the liveness answer this rule was narrowed to give, the second the refusal the arm +/// exists for, so deleting the arm turns the second into the first. +TEST(CASConfirmExactRef, WholeShardScopedMutationRefusesAnUntouchedRef) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_whole_shard"}; + + const ManifestId id_x = publishEmptyPart(store, ns, "x"); + publishEmptyPart(store, ns, "other"); + ASSERT_EQ(store->confirmExactRef(ns, "x", id_x.ref), ConfirmAnswer::Yes); + + LeaderLatch latch; + latch.arm(store); + std::thread dropper([&] { store->dropRef(ns, "other"); }); + latch.awaitEntered(); + + /// Sampled while parked, asserted after the join (a failed assertion here would skip the release). + const ConfirmAnswer before_whole_shard = store->confirmExactRef(ns, "x", id_x.ref); + + /// Queued BEHIND the parked leader's own `Ref`-scoped item. The ops are an add/remove precommit + /// pair on a ref of its own, so the item is ordinary work that happens to declare no ref -- the + /// scope, not the ops, is what rule 3 reads. + std::thread whole_shard([&] + { + store->appendRefOps(ns, MutationScope::wholeShard(), + [](const RefTableState &) { return precommitAddRemovePairs("zzz_", 1, 900000004); }, + RootMutationOrigin::Writer, RootMutationKind::Publish); + }); + const auto queued_by = std::chrono::steady_clock::now() + std::chrono::seconds(20); + while (store->refQueuePendingForTest(ns) < 2 && std::chrono::steady_clock::now() < queued_by) + std::this_thread::yield(); + const size_t pending = store->refQueuePendingForTest(ns); + + const uint64_t in_flight_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const uint64_t broken_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + const ConfirmAnswer with_whole_shard = store->confirmExactRef(ns, "x", id_x.ref); + const uint64_t in_flight_after = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const uint64_t broken_after = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + + latch.release(); + dropper.join(); + whole_shard.join(); + store->setRefPreCarveHookForTest(nullptr); + + EXPECT_EQ(pending, 2u) << "the whole-shard item must be queued behind the parked leader's own item"; + EXPECT_EQ(before_whole_shard, ConfirmAnswer::Yes) + << "with only a Ref-scoped mutation of another ref queued, 'x' must still confirm"; + EXPECT_EQ(with_whole_shard, ConfirmAnswer::Unknown) + << "a queued mutation that declares no ref may move every row, so no ref may confirm"; + EXPECT_EQ(in_flight_after - in_flight_before, 1u) + << "the refusal must be attributed to an in-flight mutation, not to a lane or mount condition"; + EXPECT_EQ(broken_after - broken_before, 0u) + << "the lane is Ready here: a refusal attributed to a broken lane would misreport a fault"; + + /// The tenure is over and the whole-shard item is applied: 'x' confirms again. + EXPECT_EQ(store->confirmExactRef(ns, "x", id_x.ref), ConfirmAnswer::Yes); +} + + +/// The stale-row hazard rule 3 exists for, on the same ref: a repoint of x from m1 to m2 is DURABLE +/// and NOT installed, so the committed row still says m1. A `Yes` here would let a receiver promote a +/// manifest whose blobs the durable repoint may already have retired. The leader is parked at the +/// SECOND `PostDurableInstall` of the repointing publish (the first is its precommit), with no lane +/// mutex held; `pending` is already empty, so only the carved mirror can refuse. +TEST(CASConfirmExactRef, SameRefRepointDurableButNotInstalledIsUnknown) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/confirm_stale_row"}; + const ManifestId m1 = publishEmptyPart(store, ns, "x"); + ASSERT_EQ(store->confirmExactRef(ns, "x", m1.ref), ConfirmAnswer::Yes); + + struct Hold + { + std::mutex m; + std::condition_variable cv; + int seen = 0; + bool parked = false; + bool released = false; + }; + auto hold = std::make_shared(); + store->setCarveHookForTest([hold](CasRefLedger::CarvePhaseForTest phase) + { + if (phase != CasRefLedger::CarvePhaseForTest::PostDurableInstall) + return; + std::unique_lock lk(hold->m); + if (++hold->seen != 2) + return; /// 1 = the precommit's chunk; 2 = the promote (the repoint) -- park here + hold->parked = true; + hold->cv.notify_all(); + hold->cv.wait_for(lk, std::chrono::seconds(20), [&] { return hold->released; }); + }); + + ManifestId m2; + std::thread repointer([&] { m2 = publishEmptyPart(store, ns, "x", /*allow_repoint=*/true); }); + bool parked = false; + { + std::unique_lock lk(hold->m); + parked = hold->cv.wait_for(lk, std::chrono::seconds(20), [&] { return hold->parked; }); + } + const size_t pending_now = store->refQueuePendingForTest(ns); + const size_t carved_now = store->refCarvedForTest(ns); + const RefLaneState lane_now = store->laneStateForTest(ns); + /// `pending` is empty and the mirror holds one, so this refusal provably comes from the carved + /// loop -- the only place where its attribution can be pinned. + const uint64_t in_flight_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight); + const uint64_t broken_before = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken); + const ConfirmAnswer stale = store->confirmExactRef(ns, "x", m1.ref); + const uint64_t in_flight_delta + = refusalCount(ProfileEvents::CASRelinkConfirmRefusedRefMutationInFlight) - in_flight_before; + const uint64_t broken_delta = refusalCount(ProfileEvents::CASRelinkConfirmRefusedLaneBroken) - broken_before; + { + std::lock_guard lk(hold->m); + hold->released = true; + } + hold->cv.notify_all(); + repointer.join(); + store->setCarveHookForTest(nullptr); + + ASSERT_TRUE(parked) << "the repoint never reached its post-durable window"; + EXPECT_EQ(pending_now, 0u) << "the repoint was carved: pending cannot be what refuses"; + EXPECT_EQ(carved_now, 1u) << "the carved mirror is what the confirm must read"; + EXPECT_EQ(lane_now, RefLaneState::Writing); + EXPECT_EQ(stale, ConfirmAnswer::Unknown) + << "x's durable repoint is not installed: its row is stale and must not confirm m1"; + EXPECT_EQ(in_flight_delta, 1u) + << "a refusal read off the carved mirror is a mutation in flight, not a lane fault"; + EXPECT_EQ(broken_delta, 0u) + << "the lane is Writing with a carved item -- the healthy shape, not a broken one"; + EXPECT_EQ(store->confirmExactRef(ns, "x", m1.ref), ConfirmAnswer::No) << "installed: m1 is no longer x's binding"; + EXPECT_EQ(store->confirmExactRef(ns, "x", m2.ref), ConfirmAnswer::Yes); +} + + /// =========================================================================================== /// Task 11: the EXCHANGE-level confirm -- `IContentAddressedExchange::ownsNamespace` (routing) and /// `::confirmExactRef` (the storage forward of gate 1, plus the token text and the disk lifecycle). diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index 1afff867b905..1099764d5500 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -2,6 +2,8 @@ #include "config.h" +#include + #include #include #include @@ -36,6 +38,7 @@ namespace DB::ErrorCodes extern const int LIMIT_EXCEEDED; extern const int CORRUPTED_DATA; extern const int NETWORK_ERROR; +extern const int LOGICAL_ERROR; } namespace ProfileEvents @@ -245,6 +248,9 @@ TEST(CASRefWriterChunkedFlush, OversizedItemFailsAlone) auto sync = std::make_shared(); armPreCarveBlock(store, ns, sync, 2); + /// `fillerOps` returns default `RefOp{}` of kind `NamespaceBirth`, which names no ref, so the + /// item's scope name is never compared against anything and the scope check (step 3) would pass + /// this item even with the op-count cap removed -- the cap is what's under test here, not the scope. Caller oversized = launchAppend(store, ns, MutationScope::ref("oversized"), [](const RefTableState &) -> std::vector { return fillerOps(ref_txn_max_ops + 1); }); waitEntered(sync); @@ -282,7 +288,7 @@ TEST(CASRefWriterChunkedFlush, OversizedOpFailsItsItemAlone) auto sync = std::make_shared(); armPreCarveBlock(store, ns, sync, 2); - Caller oversized = launchAppend(store, ns, MutationScope::ref("oversized_op"), + Caller oversized = launchAppend(store, ns, MutationScope::ref(oversized_op.ref_name), [oversized_op](const RefTableState &) -> std::vector { return {oversized_op}; }); waitEntered(sync); Caller neighbor = launchDrop(store, ns, "neighbor"); @@ -302,6 +308,49 @@ TEST(CASRefWriterChunkedFlush, OversizedOpFailsItsItemAlone) EXPECT_FALSE(store->resolveRef(ns, "neighbor").has_value()) << "neighbor's drop must have committed"; } +/// Per-item isolation of the `MutationScope` validation (`CasRefLedger.cpp`'s `flushRefBatch` step 3): +/// a mis-scoped item co-batched with an innocent neighbor must fail ALONE, and the neighbor's mutation +/// must COMMIT -- not merely avoid throwing -- exactly the batch-isolation shape +/// `OversizedOpFailsItsItemAlone` proves for the step-1 admission caps. Release-arm only: in a debug or +/// sanitizer build the mis-scoped item's `LOGICAL_ERROR` aborts the whole process, taking the co-batched +/// neighbor down with it before either assertion can run. +#ifndef DEBUG_OR_SANITIZER_BUILD +TEST(CASRefWriterChunkedFlush, MisScopedItemFailsAloneNeighborCommits) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const RootNamespace ns{"srv1/chunked_misscoped"}; + publishEmptyPart(store, ns, "neighbor"); + ASSERT_TRUE(store->resolveRef(ns, "neighbor").has_value()); + + RefOp add; + add.kind = RefOpKind::OwnerTransition; + add.new_binding = RefOwnerBinding{RefOwnerKind::Precommit, "y", ManifestRef{900000004, 1, 1}}; + + auto sync = std::make_shared(); + armPreCarveBlock(store, ns, sync, 2); + + Caller misscoped = launchAppend(store, ns, MutationScope::ref("x"), + [add](const RefTableState &) -> std::vector { return {add}; }); + waitEntered(sync); + Caller neighbor = launchDrop(store, ns, "neighbor"); + waitPendingAtLeast(store, ns, 2); + sync->cv.notify_all(); + + ASSERT_EQ(misscoped.fut.wait_for(std::chrono::seconds(10)), std::future_status::ready) << "mis-scoped item must not hang"; + ASSERT_EQ(neighbor.fut.wait_for(std::chrono::seconds(10)), std::future_status::ready) << "neighbor must not hang"; + const std::exception_ptr misscoped_err = misscoped.fut.get(); + const std::exception_ptr neighbor_err = neighbor.fut.get(); + misscoped.t.join(); + neighbor.t.join(); + store->setRefPreCarveHookForTest(nullptr); + + expectFailedWithCode(misscoped_err, DB::ErrorCodes::LOGICAL_ERROR, "mis-scoped item"); + EXPECT_TRUE(neighbor_err == nullptr) << "the co-batched neighbor must commit despite the mis-scoped item"; + EXPECT_FALSE(store->resolveRef(ns, "neighbor").has_value()) << "neighbor's drop must have committed"; +} +#endif + /// Test 12, canonical round-trip leg: the maximum legally-admissible normal-class transaction under /// the new counts-only caps -- `ref_txn_max_ops` ops, each padded to exactly `ref_op_max_bytes` -- /// round-trips comfortably under the whole-transaction `ref_txn_max_bytes` decode cap (5000 * 4096 = @@ -449,22 +498,22 @@ PoolPtr openPoolWith(const BackendPtr & backend, PoolConfig cfg) return Pool::open(backend, cfg); } -/// `num_pairs` add-then-remove precommit op pairs (2 * `num_pairs` ops total) for distinct refs -/// (`prefix` + zero-padded index) each naming a distinct valid manifest. Every pair adds a precommit -/// binding and immediately removes it, so the LIVE state (the `precommits` set, the committed COW map, -/// the owned-manifest index) stays ~empty throughout the whole transaction -- keeping the per-op -/// `admits` preview and the sanitizer-only body-counter assert O(1), so validating a maximal chunk of -/// thousands of ops stays O(ops), not O(ops^2). It is the OP COUNT (not the resident state) that drives -/// the chunk split under test; each op is tiny (well under `ref_op_max_bytes`), so the whole run is -/// admissible on a `Live` namespace. The durable transaction still carries every op verbatim, so a -/// chunk's ops can be compared against the exact expected vector. -std::vector addRemovePrecommitPairs(const String & prefix, size_t num_pairs, uint64_t manifest_epoch) +/// `num_pairs` add-then-remove precommit op pairs (2 * `num_pairs` ops total) on ONE ref, each pair +/// naming a distinct valid manifest, so an item scoped `MutationScope::ref(ref)` names exactly the ref +/// its ops mutate (the flush validates that). Every pair adds a precommit binding and immediately +/// removes it, so the LIVE state (the `precommits` set, the committed COW map, the owned-manifest +/// index) stays ~empty throughout the whole transaction -- keeping the per-op `admits` preview and the +/// sanitizer-only body-counter assert O(1), so validating a maximal chunk of thousands of ops stays +/// O(ops), not O(ops^2). It is the OP COUNT (not the resident state) that drives the chunk split under +/// test; each op is tiny (well under `ref_op_max_bytes`), so the whole run is admissible on a `Live` +/// namespace. The durable transaction still carries every op verbatim, so a chunk's ops can be compared +/// against the exact expected vector. +std::vector addRemovePrecommitPairs(const String & ref, size_t num_pairs, uint64_t manifest_epoch) { std::vector ops; ops.reserve(num_pairs * 2); for (size_t i = 0; i < num_pairs; ++i) { - const String ref = prefix + paddedRefName(i); const ManifestRef manifest{manifest_epoch, i + 1, 1}; RefOp add; add.kind = RefOpKind::OwnerTransition; @@ -577,9 +626,9 @@ TEST(CASRefWriterChunkedFlush, ChunkedFlushCommitsPerChunk) /// 2000 ops per item (1000 add/remove pairs) -> 6000 > ref_txn_max_ops (5000): chunk 1 = /// {item_a,item_b} (4000), chunk 2 = {item_c} (2000). - const std::vector ops1 = addRemovePrecommitPairs("aaa_", 1000, 900000001); - const std::vector ops2 = addRemovePrecommitPairs("bbb_", 1000, 900000002); - const std::vector ops3 = addRemovePrecommitPairs("ccc_", 1000, 900000003); + const std::vector ops1 = addRemovePrecommitPairs("item_a", 1000, 900000001); + const std::vector ops2 = addRemovePrecommitPairs("item_b", 1000, 900000002); + const std::vector ops3 = addRemovePrecommitPairs("item_c", 1000, 900000003); auto c1 = std::make_shared>(0); auto c2 = std::make_shared>(0); auto c3 = std::make_shared>(0); @@ -700,9 +749,9 @@ ChunkFailureOutcome runChunkFailureCase(const String & ns_suffix, ChunkFaultBack armPreCarveBlock(store, ns, sync, 2); /// 3000 ops per item (1500 add/remove pairs) -> 6000 > ref_txn_max_ops: chunk 1 = {item_a}, /// chunk 2 = {item_b}. - AppendCaller a = launchAppendOps(store, ns, MutationScope::ref("item_a"), addRemovePrecommitPairs("aaa_", 1500, 900000001), nullptr); + AppendCaller a = launchAppendOps(store, ns, MutationScope::ref("item_a"), addRemovePrecommitPairs("item_a", 1500, 900000001), nullptr); waitEntered(sync); - AppendCaller b = launchAppendOps(store, ns, MutationScope::ref("item_b"), addRemovePrecommitPairs("bbb_", 1500, 900000002), nullptr); + AppendCaller b = launchAppendOps(store, ns, MutationScope::ref("item_b"), addRemovePrecommitPairs("item_b", 1500, 900000002), nullptr); waitPendingAtLeast(store, ns, 2); sync->cv.notify_all(); @@ -823,9 +872,9 @@ TEST(CASRefWriterChunkedFlush, LeaderOwnItemCommittedBeforeThrow) auto sync = std::make_shared(); armPreCarveBlock(store, ns, sync, 2); /// 3000 ops per item (1500 add/remove pairs) -> chunk 1 = {item_a}, boundary throw before chunk 2. - AppendCaller a = launchAppendOps(store, ns, MutationScope::ref("item_a"), addRemovePrecommitPairs("aaa_", 1500, 900000001), c1); + AppendCaller a = launchAppendOps(store, ns, MutationScope::ref("item_a"), addRemovePrecommitPairs("item_a", 1500, 900000001), c1); waitEntered(sync); - AppendCaller b = launchAppendOps(store, ns, MutationScope::ref("item_b"), addRemovePrecommitPairs("bbb_", 1500, 900000002), c2); + AppendCaller b = launchAppendOps(store, ns, MutationScope::ref("item_b"), addRemovePrecommitPairs("item_b", 1500, 900000002), c2); waitPendingAtLeast(store, ns, 2); sync->cv.notify_all(); @@ -848,7 +897,7 @@ TEST(CASRefWriterChunkedFlush, LeaderOwnItemCommittedBeforeThrow) if (txn.txn_id == ra.id) chunk1_txn = txn; ASSERT_TRUE(chunk1_txn.has_value()) << "chunk 1 must be durable"; - EXPECT_EQ(chunk1_txn->ops, addRemovePrecommitPairs("aaa_", 1500, 900000001)); + EXPECT_EQ(chunk1_txn->ops, addRemovePrecommitPairs("item_a", 1500, 900000001)); /// item_a's build_ops ran once (chunk 1); item_b's ran once (before the boundary throw preempted its /// validation) and is NOT re-invoked -- the at-most-once contract holds through the failed tenure. EXPECT_EQ(c1->load(), 1); @@ -886,9 +935,9 @@ TEST(CASRefWriterChunkedFlush, SnapshotPublisherLatchedAcrossChunks) auto sync = std::make_shared(); armPreCarveBlock(store, ns, sync, 2); /// 3000 ops per item (1500 add/remove pairs) -> chunk 1 = {item_a}, chunk 2 = {item_b}. - AppendCaller a = launchAppendOps(store, ns, MutationScope::ref("item_a"), addRemovePrecommitPairs("aaa_", 1500, 900000001), nullptr); + AppendCaller a = launchAppendOps(store, ns, MutationScope::ref("item_a"), addRemovePrecommitPairs("item_a", 1500, 900000001), nullptr); waitEntered(sync); - AppendCaller b = launchAppendOps(store, ns, MutationScope::ref("item_b"), addRemovePrecommitPairs("bbb_", 1500, 900000002), nullptr); + AppendCaller b = launchAppendOps(store, ns, MutationScope::ref("item_b"), addRemovePrecommitPairs("item_b", 1500, 900000002), nullptr); waitPendingAtLeast(store, ns, 2); sync->cv.notify_all(); diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 7d63be5769db..349c4068e4e5 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -27,7 +27,14 @@ - ``POST /_control/condemn?bucket=B&key=K`` — rewrite blob ``K``'s existing ``.meta`` sibling from ``Clean`` to ``Condemned`` without adding a captured storage request. This is a deterministic state seam for the writer retry test, not a model of the GC request sequence; - - ``POST /_control/reset`` — drop the capture log and the counters (objects are kept); + - ``POST /_control/reset`` — drop the capture log and the counters (objects are kept), and clear the + delay knob below; + - ``POST /_control/delay?substr=S&ms=N`` — every PUT whose key contains ``S`` sleeps ``N`` + milliseconds before it is served, outside the store lock so other requests keep flowing. A fixed + per-request delay, not a modelled per-object rate cap: it charges an isolated write the same as a + burst. Each delayed PUT also increments the ``DelayedPut`` counter (visible at + ``/_control/counters``), so a caller can prove the knob fired rather than infer it from timing. + ``substr=&ms=0`` clears it; - ``POST /_control/mode?if_match=reject|ignore&omit_generation=0|1`` — select the adversarial behaviours below. Global, not per bucket: the client reuses connections across buckets and a per-bucket switch would invite a test to believe it had isolated something it had not. @@ -124,6 +131,11 @@ def __init__(self): # Adversarial behaviour, off by default — see the module docstring. self.if_match_mode = "reject" self.omit_generation = False + # `/_control/delay`: every PUT whose key contains `delay_substr` sleeps `delay_ms` before it is + # served, outside the store lock so other requests keep flowing. A fixed per-request delay, not + # a modelled rate cap — see the module docstring's `/_control/delay` bullet. + self.delay_substr = "" + self.delay_ms = 0 self._next_generation = _GENERATION_SEED self._next_etag_ordinal = 1 self._next_upload_ordinal = 1 @@ -797,9 +809,19 @@ def handle_control(path, method, query): json.dumps({"bucket": bucket, "key": blob_key, "state": "condemned"}).encode(), {"Content-Type": "application/json"}, ) + if path == "/_control/delay" and method == "POST": + STORE.delay_substr = query.get("substr", [""])[0] + STORE.delay_ms = int(query.get("ms", ["0"])[0]) + return Reply( + 200, + json.dumps({"substr": STORE.delay_substr, "ms": STORE.delay_ms}).encode(), + {"Content-Type": "application/json"}, + ) if path == "/_control/reset" and method == "POST": STORE.requests = [] STORE.counters = {} + STORE.delay_substr = "" + STORE.delay_ms = 0 return Reply(200, b"OK") return Reply(404, _error_xml("NoSuchControl", "unknown control path " + path)) @@ -847,8 +869,18 @@ def _dispatch(self, method, want_body=True): stripped = path.lstrip("/") bucket, _, key = stripped.partition("/") + delayed = method == "PUT" and STORE.delay_ms and STORE.delay_substr and STORE.delay_substr in key + if delayed: + time.sleep(STORE.delay_ms / 1000.0) + with _LOCK: STORE.count("method_" + method) + # A caller that only checks queue drainage cannot tell a delay that fired from a delay + # knob that silently stopped matching (a renamed endpoint, a renamed query param, a + # substring that no longer matches the key) — this counter is the caller's proof the + # sleep above actually ran. + if delayed: + STORE.count("DelayedPut") request_class = _request_class(bucket, key) operation = _request_operation(bucket, request_class, method, query, headers) if method == "PUT": @@ -923,8 +955,17 @@ def do_DELETE(self): def main(): port = int(sys.argv[1]) - server = http.server.ThreadingHTTPServer(("0.0.0.0", port), Handler) + # `bind_and_activate=False`, then raise the listen backlog, then bind+activate by hand: the base + # class calls `socket.listen()` (which locks in the backlog) from inside `__init__` when + # `bind_and_activate` is left at its default, before this line could change it. + server = http.server.ThreadingHTTPServer(("0.0.0.0", port), Handler, bind_and_activate=False) server.daemon_threads = True + # The default listen backlog (5) starves unrelated connections once `/_control/delay` holds a + # few dozen handler threads asleep at once: a caller under `test_cas_gcs_relink_liveness` + # measured `connect timed out` on keys the delay knob was never meant to slow. + server.request_queue_size = 128 + server.server_bind() + server.server_activate() server.serve_forever() diff --git a/tests/integration/test_cas_gcs_relink_liveness/__init__.py b/tests/integration/test_cas_gcs_relink_liveness/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/integration/test_cas_gcs_relink_liveness/configs/storage_conf.xml b/tests/integration/test_cas_gcs_relink_liveness/configs/storage_conf.xml new file mode 100644 index 000000000000..44eceb785560 --- /dev/null +++ b/tests/integration/test_cas_gcs_relink_liveness/configs/storage_conf.xml @@ -0,0 +1,25 @@ + + + + + object_storage + s3 + cas + + __SERVER_ROOT_ID__ + http://fakegcs:8080/hmacbucket/cas/ + gcs_hmac + GOOG1EFAKEACCESSKEYID + fake-goog4-hmac-secret + 1 + 1 + + + + +
disk_cas_gcs_shared
 + + + + diff --git a/tests/integration/test_cas_gcs_relink_liveness/test.py b/tests/integration/test_cas_gcs_relink_liveness/test.py new file mode 100644 index 000000000000..b4f91ce65bee --- /dev/null +++ b/tests/integration/test_cas_gcs_relink_liveness/test.py @@ -0,0 +1,305 @@ +"""Fetch-by-relink liveness on a slow control plane. + +Two replicas of one ReplicatedMergeTree table share one CAS pool over the fake GCS service of +`test_cas_gcs`, with every write to a key containing `_ckpt` delayed. That substring targets the +ref-lane flush's checkpoint publication, but it also matches namespace creation, recovery, and the GC +snapshot publisher's checkpoint contribution — all four are slowed, not just the flush. Both replicas +insert continuously, so each is a sender and a receiver at once and each keeps its own lane busy. A +confirm rule that refuses whenever the sender's lane is busy starves both replication queues here +(finding F11 of the 2026-09-02 live GCS campaign); the ref-scoped rule lets them drain. The unit tests +pin the rule; this is the liveness reproduction they cannot give. +""" +import json +import os +import shlex +import threading +import time + +import pytest + +from helpers.cluster import ClickHouseCluster +from helpers.mock_servers import start_mock_servers + +GCS_HOST = "fakegcs" +GCS_PORT = 8080 +CONFIG_IN_CONTAINER = "/etc/clickhouse-server/config.d/cas_gcs_shared.xml" +MOCK_DIR = os.path.join(os.path.dirname(__file__), "..", "test_cas_gcs", "gcs_mocks") +NODES = ("node1", "node2") +CA_DISK = "disk_cas_gcs_shared" + +# Every `_ckpt` PUT sleeps this long: one second is the real service's own per-object mutation cap +# (see the module docstring above). A full run at this value takes several minutes. +CKPT_DELAY_MS = 1000 +INSERTS_PER_NODE = 80 +ROWS_PER_INSERT = 1000 +DRAIN_TIMEOUT_S = 180 + +cluster = ClickHouseCluster(__file__) + + +@pytest.fixture(scope="module", autouse=True) +def start_cluster(): + # As in test_cas_gcs: a CAS disk mounts at server start and fails closed when its store is + # unreachable, and the fake can only be launched once its container is up. So both nodes start + # without the disk, the disk configuration is installed with the node's own server root, and each + # node is restarted once the fake answers. + for name in NODES: + cluster.add_instance(name, macros={"replica": name}, with_zookeeper=True, stay_alive=True) + cluster.add_instance( + GCS_HOST, hostname=GCS_HOST, image="altinityinfra/python-bottle", tag="latest", stay_alive=True + ) + try: + cluster.start() + start_mock_servers(cluster, MOCK_DIR, [("server.py", GCS_HOST, str(GCS_PORT))]) + for name in NODES: + node = cluster.instances[name] + node.copy_file_to_container( + os.path.join(os.path.dirname(__file__), "configs", "storage_conf.xml"), + CONFIG_IN_CONTAINER, + ) + node.replace_in_config(CONFIG_IN_CONTAINER, "__SERVER_ROOT_ID__", name) + node.restart_clickhouse() + yield cluster + finally: + cluster.shutdown() + + +def _control_post(path): + container = cluster.get_container_id(GCS_HOST) + return cluster.exec_in_container( + container, ["curl", "-sS", "-X", "POST", "http://localhost:{}{}".format(GCS_PORT, path)] + ) + + +def _control_get(path): + container = cluster.get_container_id(GCS_HOST) + return cluster.exec_in_container( + container, ["curl", "-sS", "http://localhost:{}{}".format(GCS_PORT, path)] + ) + + +def _set_delay(substr, ms): + # `curl -sS` exits 0 on an HTTP 404 or 500, and a docker-exec only raises on a non-zero exit + # status — so a renamed endpoint or a renamed query parameter would otherwise go unnoticed here + # and the rest of the test would exercise a fake running at full speed. Checking the echoed body + # is what makes a broken lever fail loudly instead of silently. + reply = _control_post("/_control/delay?substr={}&ms={}".format(substr, ms)) + assert json.loads(reply) == {"substr": substr, "ms": ms}, ( + "fake did not echo back the delay setting it was asked for: {!r}".format(reply) + ) + + +def _delayed_put_count(): + counters = json.loads(_control_get("/_control/counters")) + return counters.get("DelayedPut", 0) + + +def _queue_size(node, table): + return int( + node.query("SELECT count() FROM system.replication_queue WHERE table = '{}'".format(table)) + ) + + +def _queue_breakdown(node, table): + # What a bare queue-size number cannot say: WHICH kind of entry is stuck and why. A future stuck + # run for an unrelated reason (a merge stall, a ZooKeeper hiccup) would otherwise print a + # byte-identical failure message to this test's own liveness symptom. + return node.query( + "SELECT type, count(), any(last_exception) FROM system.replication_queue " + "WHERE table = '{}' GROUP BY type ORDER BY type FORMAT TSV".format(table) + ) + + +def _replica_status(node, table): + row = node.query( + "SELECT queue_size, absolute_delay, log_pointer, log_max_index " + "FROM system.replicas WHERE table = '{}' FORMAT TSV".format(table) + ) + queue_size, absolute_delay, log_pointer, log_max_index = row.split() + return int(queue_size), int(absolute_delay), int(log_pointer), int(log_max_index) + + +def _drained(node, table): + # `queue_size == 0` alone is checked right after the insert threads join, before the + # queue-updating thread is guaranteed to have pulled the peer's latest log entries — so a node + # can read an empty queue with fetches still outstanding, and closing the delay window in that + # instant would let the tail race to completion at full speed and pass for the wrong reason. + # `log_pointer > log_max_index` (every log entry has been copied into the execution queue) and + # `absolute_delay == 0` together rule that race out. + queue_size, absolute_delay, log_pointer, log_max_index = _replica_status(node, table) + return queue_size == 0 and absolute_delay == 0 and log_pointer > log_max_index + + +def _refusal_counters(node): + return node.query( + "SELECT event, value FROM system.events WHERE event LIKE 'CASRelinkConfirmRefused%' " + "ORDER BY event FORMAT TSV" + ) + + +def _refusal_counter(node, event): + # `system.events` has no row for an event that never fired, so an empty result is a zero, not an + # error — which is what lets the caller assert a clean zero rather than having to special-case a + # missing row. A misspelled event name reads the same way, so a zero here is never by itself + # evidence that the named counter exists. + value = node.query( + "SELECT value FROM system.events WHERE event = '{}'".format(event) + ).strip() + return int(value) if value else 0 + + +def _log_lines(node, pattern): + out = node.exec_in_container( + [ + "bash", + "-c", + "grep -a -E {} /var/log/clickhouse-server/clickhouse-server.log || true".format( + shlex.quote(pattern) + ), + ] + ) + return [line for line in out.splitlines() if line.strip()] + + +def _relink_finished_pattern(table, disk): + """The receiver-side proof that a fetch completed by relink, not by byte transfer. + + Reachable only after `Fetcher::relinkPartToDisk`'s confirm step answered yes and `promote()` + returned `Committed` — every other row in that function returns or throws before this line, so its + presence cannot be produced by a fallback to bytes. Same pattern as + `test_cas_replicated_relink.relink_finished_pattern`, generalised to any part name since this test + does not track individual part names. + """ + return r"default\.{} .*Relink of part .* onto disk {} finished \(no bytes transferred\)".format( + table, disk + ) + + +def test_both_queues_drain_under_slow_checkpoints(): + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "relink_liveness" + for node in (node1, node2): + node.query("DROP TABLE IF EXISTS {} SYNC".format(table)) + node.query( + "CREATE TABLE {t} (id Int64, v UInt64) " + "ENGINE = ReplicatedMergeTree('/clickhouse/tables/{t}', '{{replica}}') " + "ORDER BY id SETTINGS storage_policy = 'cas_gcs_shared'".format(t=table) + ) + + _set_delay("_ckpt", CKPT_DELAY_MS) + try: + errors = [] + + def insert_loop(node, base): + try: + for i in range(INSERTS_PER_NODE): + node.query( + "INSERT INTO {} SELECT number, number * 10 FROM numbers({}, {})".format( + table, base + i * ROWS_PER_INSERT, ROWS_PER_INSERT + ) + ) + except Exception as e: # surfaced below, on the test thread + errors.append((node.name, repr(e))) + + threads = [ + threading.Thread(target=insert_loop, args=(node1, 0)), + threading.Thread(target=insert_loop, args=(node2, 10_000_000)), + ] + for t in threads: + t.start() + for t in threads: + t.join() + assert errors == [], errors + + # The lever, checked BEFORE the drain assertion: if the delay knob silently stopped + # matching (a renamed endpoint, a renamed query param, a `_ckpt` key that stopped matching + # the substring), the fake would serve every write at full speed and the drain assertion + # below would pass having exercised nothing. Checking this first means a failure here is + # never confused with the liveness failure this test exists to catch. + delayed = _delayed_put_count() + assert delayed >= 2 * INSERTS_PER_NODE, ( + "the delay knob fired only {} times; expected at least {} — it may have silently " + "stopped matching `_ckpt` PUTs, which would make any drain result meaningless".format( + delayed, 2 * INSERTS_PER_NODE + ) + ) + + # Liveness: the insert threads have already joined, so each replica's lane is kept busy from + # here on by its own fetch bookkeeping alone — the self-sustaining half of the livelock. Both + # queues must still drain. + deadline = time.time() + DRAIN_TIMEOUT_S + while time.time() < deadline and not (_drained(node1, table) and _drained(node2, table)): + time.sleep(1) + drained = (_drained(node1, table), _drained(node2, table)) + sizes = (_queue_size(node1, table), _queue_size(node2, table)) + for node in (node1, node2): + print(node.name, "refusal counters:", _refusal_counters(node)) + assert drained == (True, True), ( + "replication queues did not drain in {} s with slow checkpoints: node1={} node2={}\n" + "node1 queue (type, count, last_exception):\n{}\n" + "node2 queue (type, count, last_exception):\n{}".format( + DRAIN_TIMEOUT_S, + sizes[0], + sizes[1], + _queue_breakdown(node1, table), + _queue_breakdown(node2, table), + ) + ) + + # Transport proof, checked AFTER the drain assertion on purpose: under the old rule the + # parts never arrive at all, so a relink assertion placed before the drain check would fail + # for the second-best reason and muddy the evidence this test exists to produce. "Both + # queues drained" cannot by itself distinguish a relinked fetch from a byte fetch that + # dropped into one of `relinkPartToDisk`'s silent fallback exits — this line is reachable + # only through the intended path. + for node in (node1, node2): + finished = _log_lines(node, _relink_finished_pattern(table, CA_DISK)) + assert finished, ( + "{} drained its queue without a single relink completing — every part that " + "arrived took some route other than fetch-by-relink".format(node.name) + ) + + # No confirm may be refused by lane STATE on a healthy run. These two counters move only on + # `confirmExactRef`'s wedge and broken-lane branches — an unresolved append, or a lane in + # NeedsRecovery, Closed, Faulted, or Writing with nothing carved. This case drives the fake's + # control plane with a delay and never with a fault, so none of those is reachable here and a + # non-zero value is a lane defect the drain assertion above can hide: a confirm refused + # table-wide costs the receiver only a retry, and enough retries still finish inside the drain + # window. Against a real bucket a wedge IS reachable from load alone — sustained throttling can + # exhaust an append's retry budget and leave its outcome unresolved — so this assertion rests + # on the fault-free stand and would have to be rethought before it ran anywhere else. + # + # A zero here is not evidence that the counter exists: `_refusal_counter` reads a missing row + # as a zero, and a misspelled or unregistered name reads the same way. + # + # What is NOT asserted, deliberately: `CASRelinkConfirmRefusedRefMutationInFlight` above zero. + # A refusal there needs a queued or carved mutation naming the very ref the peer is asking + # about, and each node's lane is busy with its own newer parts rather than with the seconds-old + # part its peer is fetching — a run that passed both assertions above left node1 with no + # `CASRelinkConfirmRefused%` row at all. Requiring it would demand the symptom the ref-scoped + # rule removes, so it would pass only while the livelock is present. The two things it was + # meant to show are shown better above: the relink-completion assertion proves confirms were + # asked and answered `Yes`, and the delayed-PUT floor proves the checkpoint publications were + # slowed, which is the contention itself. Its attribution is pinned in the unit suite, by + # `CASConfirmExactRef.UntouchedRefConfirmsWhileAnotherRefIsQueued`. + for node in (node1, node2): + for event in ("CASRelinkConfirmRefusedLaneWedged", "CASRelinkConfirmRefusedLaneBroken"): + assert _refusal_counter(node, event) == 0, ( + "{} refused a relink confirm by lane state ({}), which no fault was injected to " + "produce. All refusal counters on this node:\n{}".format( + node.name, event, _refusal_counters(node) + ) + ) + finally: + try: + _set_delay("", 0) + except Exception as exc: + # Never let a failure here mask a real assertion failure raised above. + print("failed to clear the delay knob:", repr(exc)) + + expected = 2 * INSERTS_PER_NODE * ROWS_PER_INSERT + assert int(node1.query("SELECT count() FROM {}".format(table))) == expected + assert int(node2.query("SELECT count() FROM {}".format(table))) == expected + for node in (node1, node2): + node.query("DROP TABLE IF EXISTS {} SYNC".format(table)) From 0349d57e213208a493aafac6e596c0c5c766ee1f Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:01:52 +0200 Subject: [PATCH 13/81] cas: key the manifest decode cache by id alone, no HEAD on a hit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A manifest id names one content, ever: `stageManifest` mints it from `(writer_epoch, build_sequence, next_manifest_ordinal++)` — all three either a durable counter bumped by a conditional write or strictly increasing per process — and writes the body exactly once. The only other mutations of a manifest are exact-token deletes (writer cleanup, GC's owner-removed cleanup, the orphan sweep). So the token carried in `ManifestCacheKey` distinguished nothing, and the `HEAD` that supplied it (`CasManifestReader::readManifestShared`) was a per-read check of a GC-side invariant, not of the cached content's validity — it cost one serial round trip per uncached or `ForceFresh` access and could only ever detect a protocol violation (something deleting a manifest the ref graph still names), never serve wrong bytes if removed, since id-to-content is a function. The cache now keys by `ManifestId` alone: no `HEAD` on a hit, exactly one `GET` on a miss. Detection of a dangling reference moves from "the next read" to "the first uncached read, or fsck". The `part_folder_validate` setting, which existed only to pace that now-removed `HEAD`, is retired along with it. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- docs/en/antalya/cas/architecture/read-path.md | 39 +-- .../antalya/cas/architecture/replication.md | 3 +- docs/en/antalya/cas/configuration.md | 1 - .../antalya/cas/operations/troubleshooting.md | 2 +- docs/en/operations/storing-data.md | 5 - src/Common/ProfileEvents.cpp | 1 - .../ContentAddressedMetadataStorage.cpp | 36 +-- .../ContentAddressedMetadataStorage.h | 11 - .../ContentAddressedSettings.cpp | 10 +- .../ContentAddressedSettings.h | 19 +- .../ContentAddressedTransaction.cpp | 11 +- .../ContentAddressedTransaction.h | 10 +- .../Parts/PartFolderAccess.cpp | 56 +--- .../ContentAddressed/Parts/PartFolderAccess.h | 53 +--- .../Pool/CasManifestReader.cpp | 39 +-- .../ContentAddressed/Pool/CasManifestReader.h | 40 +-- .../ContentAddressed/Pool/CasPool.h | 15 +- .../ContentAddressed/README.md | 2 +- src/Disks/tests/cas_test_helpers.h | 6 +- src/Disks/tests/gtest_ca_transaction.cpp | 36 ++- src/Disks/tests/gtest_ca_wiring.cpp | 58 ++++ .../tests/gtest_cas_part_folder_access.cpp | 278 +++++------------- .../tests/gtest_cas_part_folder_view.cpp | 3 +- src/Disks/tests/gtest_cas_pool.cpp | 164 ++++++++++- src/Disks/tests/gtest_cas_repoint.cpp | 2 +- src/Disks/tests/gtest_cas_settings.cpp | 27 +- 26 files changed, 442 insertions(+), 485 deletions(-) diff --git a/docs/en/antalya/cas/architecture/read-path.md b/docs/en/antalya/cas/architecture/read-path.md index beb2d87d99fb..49a6864a0109 100644 --- a/docs/en/antalya/cas/architecture/read-path.md +++ b/docs/en/antalya/cas/architecture/read-path.md @@ -36,30 +36,33 @@ decoding the whole body is cheaper than any partial-read machinery would be. | Cache | Keyed by | Setting | Default | What still hits the network | |---|---|---|---|---| -| Manifest decode cache | `(ManifestId, Token)` | `cas_manifest_decode_cache_bytes` | 128 MiB | A mandatory `HEAD` on **every** access, cache hit or miss | -| Part-folder view cache (`Cas::CachedPartFolderAccess`, `Parts/PartFolderAccess.h`) | Part ref key | `cas_part_folder_cache_bytes`, `cas_part_folder_cache_max_entries`, `cas_part_folder_cache_max_entry_bytes` | 64 MiB / 10 000 entries / 16 MiB | Its `ForceFresh` policy re-proves the manifest body via that same mandatory `HEAD`, paced by `cas_part_folder_validate` (`always` \| `never` \| `age `) | +| Manifest decode cache | `ManifestId` | `cas_manifest_decode_cache_bytes` | 128 MiB | Nothing on a hit; one `GET` on a miss | +| Part-folder view cache (`Cas::CachedPartFolderAccess`, `Parts/PartFolderAccess.h`) | Part ref key | `cas_part_folder_cache_bytes`, `cas_part_folder_cache_max_entries`, `cas_part_folder_cache_max_entry_bytes` | 64 MiB / 10 000 entries / 16 MiB | Nothing on a validated hit; a `ForceFresh` access bypasses the retained view and rebuilds from the manifest decode cache | -**The `HEAD` is mandatory even on a cache hit** — the page's most counter-intuitive fact, because it -means a cache hit still costs one object-store round trip: +**A cache hit costs no request.** A manifest id is minted once and its body is written once, so one +id names one content forever and a cached decode can be served without asking the object store: ```mermaid flowchart TD - A["readManifestShared(ManifestId)"] --> B["HEAD the manifest key"] - B -->|"absent"| C["throw FILE_DOESNT_EXIST --
a live ref must never name a missing object"] - B -->|"present, token t"| D{"cache lookup (ManifestId, t)"} - D -->|hit| E["return the cached decode -- no GET"] - D -->|miss| F["GET the body"] - F --> G{"body's own ref and namespace
match the key?"} - G -->|no| H["throw CORRUPTED_DATA"] - G -->|yes| I["decode, insert into cache keyed by (ManifestId, t), return"] + A["readManifestShared(ManifestId)"] --> B{"decode cache lookup by ManifestId"} + B -->|hit| C["return the cached decode -- no request"] + B -->|miss| D["GET the body"] + D -->|"absent"| E["throw FILE_DOESNT_EXIST --
a live ref must never name a missing object"] + D -->|"present"| F{"body's own ref and namespace
match the key?"} + F -->|no| G["throw CORRUPTED_DATA"] + F -->|yes| H["decode, insert into the cache keyed by ManifestId, return"] ``` -The `HEAD` is what proves the live ref still names an existing object — the no-dangle invariant — -and it supplies the token that keys the cache; only then is the decode cache consulted. On a miss, -the `GET` is followed by the two identity checks in the diagram, each `CORRUPTED_DATA` on failure. -Only a fully validated decode enters the cache. Setting either cache's byte budget to `0` disables -retention while leaving the `HEAD`-and-validate sequence intact — a cache is purely an -optimization, never a trust boundary. +On a miss, the `GET` is followed by the two identity checks in the diagram, each `CORRUPTED_DATA` on +failure, and only a fully validated decode enters the cache. A live ref that names a missing body is +detected on a miss, by the garbage collector before it deletes a manifest, and by `fsck`; a reader +holding a cached decode for a manifest the collector has since removed sees a snapshot-consistent +manifest and fails with a typed error when it reads a blob that is gone. Write paths that carry +entries forward from a committed part (hardlinks, renames, single-file rewrites, relink) adopt the +source blobs on the strength of the source ref's live edge, which the collector honours; deleting +objects out of band, behind the collector's back, is outside that contract and is what `fsck` +reports. Setting either cache's byte budget to `0` disables retention while leaving the +`GET`-and-validate sequence intact — a cache is purely an optimization, never a trust boundary. The part-folder view cache is invalidated on every promote and repoint, and is single-flight on a cold build: concurrent readers of the same not-yet-cached view coalesce into one build rather than diff --git a/docs/en/antalya/cas/architecture/replication.md b/docs/en/antalya/cas/architecture/replication.md index 371ea74b39f7..8e8981c404ae 100644 --- a/docs/en/antalya/cas/architecture/replication.md +++ b/docs/en/antalya/cas/architecture/replication.md @@ -109,7 +109,8 @@ content, and that root can confirm its exact refs. `DETACH`, `ATTACH`, `delete_tmp_` cleanup, and merge-result renames all reduce to the same two moves: re-key any *staged* source into the destination, then `republishRef(src → dst)` for any -*committed* source. `republishRef` re-reads the source manifest freshly, publishes an +*committed* source. `republishRef` resolves the source ref freshly and reads its manifest through +the manifest cache, publishes an equivalent-entry manifest under the destination ref — a **new** manifest id, with blobs untouched and adopted by evidence — then drops the source ref. A destination that already exists with identical entries just drops the source, an idempotent re-drive; one with different entries diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index d7ee12130993..2e9c6403b91b 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -98,7 +98,6 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_part_folder_cache_bytes` | 64 MiB | Part-folder view cache byte budget (`0` disables retention) | | `cas_part_folder_cache_max_entries` | `10000` | Part-folder view cache entry cap | | `cas_part_folder_cache_max_entry_bytes` | 16 MiB | Oversized part-folder views bypass retention above this size | -| `cas_part_folder_validate` | `always` | Cache body re-proof policy (`always` \| `never` \| `age `). **Leave at `always`**: the other modes trade the fail-closed body-existence check for an optimization — this is a trust decision about unverified data, not a performance knob | | `cas_manifest_decode_cache_bytes` | 128 MiB | Manifest decode cache byte budget (`0` disables) | | `cas_gc_meta_pool_size` | `16` | Bounded pool size for GC per-hash freshness-meta writes | | `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount | diff --git a/docs/en/antalya/cas/operations/troubleshooting.md b/docs/en/antalya/cas/operations/troubleshooting.md index be541add39f8..fe81f56b64ea 100644 --- a/docs/en/antalya/cas/operations/troubleshooting.md +++ b/docs/en/antalya/cas/operations/troubleshooting.md @@ -25,7 +25,7 @@ tools. | Writes or `ALTER`s on a `CAS` disk fail with a `READONLY`-class error | The disk's metadata storage rejects every mutating entry point; this is deliberate for a disk opened with `true`, used by every offline `clickhouse-disks` tool | Confirm whether the disk was intentionally configured read-only (offline inspection, `cas-fsck`, `cas-gc-dryrun`, `cas-gc-rebuild`, `cas-drop-member` all require it); a production disk serving writes must not carry `true` | | A table stays unavailable after a transient network error during startup | `AsyncLoader` has no retry/requeue path for a failed table load job: a transient S3 `NETWORK_ERROR` during `CAS` ref-table startup recovery can leave the job permanently `FAILED` | Restart the server, or issue a fresh load for the table; this is a one-shot job design, not a `CAS`-specific bug | | A mounted pool directory was removed or renamed out of band | Renewal observes an absent, foreign, successor, or otherwise conflicting mount body and terminates the keeper with a typed fail-closed exception; the runtime closes the local write fence and requests remount rather than adopting the body | Never remove or rename a live pool's storage path. To retire a member permanently use [`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission) instead of raw filesystem operations; collect the `watermark_renew` classification and subsequent `mount_remount` step | -| Stale-looking part metadata after an out-of-band change to the pool | The part-folder view cache may be serving a retained (not re-validated) view | Set the disk-level `cas_part_folder_cache_bytes = 0` as a diagnostic kill switch to disable retention, and run `fsck`/integrity checks with `cas_part_folder_validate = always` so every read re-proves the body | +| Stale-looking part metadata after an out-of-band change to the pool | The part-folder view cache may be serving a retained (not re-validated) view | Set the disk-level `cas_part_folder_cache_bytes = 0` to disable view retention and `cas_manifest_decode_cache_bytes = 0` to make every manifest read fetch the body, then run `fsck`; both are diagnostic kill switches, not steady-state settings | | A wide merge (many thousands of columns) fails with a port-exhaustion error from the network layer | Each column in a wide part can cost a separate object-store operation in one merge, and a very wide part can issue on the order of the column count in requests, exhausting local ephemeral TCP ports under load | Reduce concurrent merge parallelism on that table, or increase the host's ephemeral port range; this is a general high-fan-out-merge limit, not specific to content addressing | ## Mount renewal and remount decision flow {#mount-renewal-remount-flow} diff --git a/docs/en/operations/storing-data.md b/docs/en/operations/storing-data.md index 1f2b85d24244..95e1273f83ea 100644 --- a/docs/en/operations/storing-data.md +++ b/docs/en/operations/storing-data.md @@ -485,7 +485,6 @@ Configuration: 60 1 67108864 - always ``` @@ -542,10 +541,6 @@ disk-level and server-level settings surface. view cache. - `cas_part_folder_cache_max_entry_bytes` — `16` MiB by default. Maximum size of a single cached part-folder view entry. -- `cas_part_folder_validate` — `always` (default), `never`, or `age `. Controls how often a - `ForceFresh` read re-proves a cached manifest body via a `HEAD` request: `always` re-proves every - time (the original, pre-optimization behavior), `never` trusts the cache without re-proving, and - `age ` re-proves only once the cached entry is older than the given number of seconds. - `cas_manifest_decode_cache_bytes` — `128` MiB by default. Byte bound for the decoded-manifest cache. `0` disables decode caching entirely (a diagnostic mode). - `cas_gc_meta_pool_size` — `16` by default. Bounded thread-pool size for the GC's per-hash freshness-meta diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index aac53e612a4c..b0d3e267669b 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -951,7 +951,6 @@ The server successfully detected this situation and will download merged part fr M(CASRefRecoveryStragglerAdopted, "Number of straggler ref-log transactions a recovery compare-and-swap walk met at the slot it tried to seal and adopted, re-sealing at the new T+1. Non-zero means writes from a dying epoch were still materializing when recovery ran.", ValueType::Number) \ M(CASRefRecoveryCancelled, "Number of CAS ref-table recovery attempts abandoned because a self-remount requested cancellation before re-arming the mount fence. Non-zero means remounts are overlapping recoveries; nothing is written or installed on this path.", ValueType::Number) \ M(CASRefRecoveryStreamHole, "Number of times CAS ref-table recovery found a 404 BELOW a durable same-epoch witness -- a hole in a stream INV-1 makes dense. Restarted while the restart budget lasts (a racing cleanup is the innocent explanation), then reported as corruption. Any sustained non-zero value is data loss, not noise.", ValueType::Number) \ - M(CASPartFolderValidateSkipped, "Number of CAS part-folder validation HEADs skipped by policy or a fresh retained view. High values reduce reads but can delay detecting external changes.", ValueType::Number) \ M(CASBlobAdoptTrusted, "Number of CAS blob adoptions trusted through a durable manifest edge without per-file probes. Growth indicates manifest-based relinking.", ValueType::Number) \ M(S3GetObjectTagging, "Number of S3 API GetObjectTagging calls.", ValueType::Number) \ M(S3HeadObjectMicroseconds, "Time of S3 API HeadObject execution.", ValueType::Microseconds) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index 447fe908e45e..5d00341b1d71 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -29,7 +29,6 @@ #include #include #include -#include #include #include #include @@ -137,7 +136,7 @@ const char * casLifecycleReasonWord(Cas::PoolLifecycle lc) /// serverRootId, scratchPath, stagingBackend, objectStorage, gcHealth, /// lifecycleSnapshot (both non-store()-gated introspection reads for system.cas_mounts -- /// readable in EVERY lifecycle state including a not-live/vanished/null pool, spec §7), -/// parseStagingBackend/parsePartFolderValidate/ +/// parseStagingBackend/ /// tryFromDisk (static), checkNotReadOnly, the *ForTest seams, serverPrefix/liveNamespace/ /// shadowNamespace/shadowScope/route/classifyDirectory (pure path computation, no pool I/O), /// ownsNamespace (the relink-confirm routing predicate -- a string comparison against @@ -303,7 +302,6 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage( , blob_hash_algo(settings_.blobHashAlgo()) , blob_hash_allow_new(settings_[ContentAddressedSetting::blob_hash_allow_new].value) , skip_access_check(settings_.skipAccessCheck()) - , part_folder_validate(settings_.partFolderValidate()) { } @@ -328,35 +326,6 @@ Cas::StagingBackend ContentAddressedMetadataStorage::parseStagingBackend( return parseStagingBackend(config.getString(config_prefix + ".staging_backend", "local")); } -Cas::PartFolderValidate ContentAddressedMetadataStorage::parsePartFolderValidate(const std::string & value) -{ - using PartFolderValidate = Cas::PartFolderValidate; - if (value == "always") - return {PartFolderValidate::Mode::Always, 0}; - if (value == "never") - return {PartFolderValidate::Mode::Never, 0}; - if (value.starts_with("age ")) - { - /// `std::from_chars` against an UNSIGNED type never accepts a leading '-' (unlike - /// `std::stoull`, which silently negates modulo 2^64) -- a malformed/negative/non-digit/empty - /// suffix falls through to the terminal throw below instead of wrapping into an astronomical - /// age_seconds that behaves as skip-forever. - const std::string age_str = value.substr(4); - uint64_t age_seconds = 0; - const auto [ptr, ec] = std::from_chars(age_str.data(), age_str.data() + age_str.size(), age_seconds); - if (ec == std::errc{} && ptr == age_str.data() + age_str.size()) - return {PartFolderValidate::Mode::Age, age_seconds}; - } - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "Unknown cas_part_folder_validate value '{}' (expected 'always', 'never', or 'age ')", value); -} - -Cas::PartFolderValidate ContentAddressedMetadataStorage::parsePartFolderValidate( - const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix) -{ - return parsePartFolderValidate(config.getString(config_prefix + ".part_folder_validate", "always")); -} - ContentAddressedMetadataStorage * ContentAddressedMetadataStorage::tryFromDisk(const DiskPtr & disk) { /// The cheap predicate FIRST, never an exception probe: for every non-object-storage disk @@ -849,8 +818,7 @@ void ContentAddressedMetadataStorage::startup() Cas::CachedPartFolderAccess::CacheParams{ .cache_bytes = cas_part_folder_cache_bytes, .max_entries = cas_part_folder_cache_max_entries, - .max_entry_bytes = cas_part_folder_cache_max_entry_bytes, - .validate = part_folder_validate}); + .max_entry_bytes = cas_part_folder_cache_max_entry_bytes}); /// Reclaim this mount's leaked `staging//` debris after an explicit, writable S3 /// staging mount has passed the native-copy check above. The prefix is keyed by this mount's own diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 043ce8f9280b..15f73b32117a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -167,15 +167,6 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// thin wrapper around the string-taking overload for callers that still hold a config reference. static Cas::StagingBackend parseStagingBackend(const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix); - /// Parses a `part_folder_validate` value (`always` | `never` | `age `). The `age` form - /// accepts only a non-negative integer number of seconds; malformed input and unknown modes throw - /// `BAD_ARGUMENTS` instead of silently selecting a policy. - static Cas::PartFolderValidate parsePartFolderValidate(const std::string & value); - - /// Reads `part_folder_validate` from `config`, defaulting to `always`, and parses it. Kept only as - /// a thin wrapper around the string-taking overload for callers that still hold a config reference. - static Cas::PartFolderValidate parsePartFolderValidate(const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix); - /// Returns the content-addressed metadata storage backing `disk`, or nullptr if `disk` is not /// content-addressed. Plain (non-object-storage) disks do not implement `getMetadataStorage` at /// all and throw `NOT_IMPLEMENTED`; that is treated as "not content-addressed" rather than @@ -627,8 +618,6 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC const bool blob_hash_allow_new; /// Per-disk `` policy passed to `Cas::PoolConfig`. const bool skip_access_check; - /// Policy controlling when retained part-folder views revalidate their manifest body. - const Cas::PartFolderValidate part_folder_validate; /// A single coherent snapshot of the pool and its cached part-folder facade, taken under ONE /// `pointer_mutex` acquisition (see `poolAccess()`) so no caller can observe `pool` from one mount /// generation and `part_access` from another -- the two used to be fetched by two separate calls diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index bf6ab3b90e33..11a45c61bb8c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -74,7 +74,6 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, part_folder_cache_bytes, 64ULL << 20, "Part-folder view cache byte budget (0 disables retention)", 0) \ DECLARE(UInt64, part_folder_cache_max_entries, 10000, "Part-folder view cache entry cap", 0) \ DECLARE(UInt64, part_folder_cache_max_entry_bytes, 16ULL << 20, "Oversized part-folder views bypass retention above this size", 0) \ - DECLARE(String, part_folder_validate, "always", "ForceFresh body re-proof policy (always | never | age )", 0) \ DECLARE(UInt64, manifest_decode_cache_bytes, 128ULL << 20, "Manifest DECODE cache byte budget (0 disables)", 0) \ DECLARE(UInt64, gc_meta_pool_size, 16, "Bounded pool size for GC per-hash freshness-meta writes", 0) \ DECLARE(String, staging_backend, "local", "Blob staging backend (local | s3); s3 is opt-in", 0) \ @@ -85,11 +84,10 @@ struct ContentAddressedSettingsImpl : public BaseSettingsblob_hash_algo_cached = Cas::parseBlobHashAlgo(settings[ContentAddressedSetting::blob_hash].value); impl->staging_backend_cached = ContentAddressedMetadataStorage::parseStagingBackend(settings[ContentAddressedSetting::staging_backend].value); - impl->part_folder_validate_cached = ContentAddressedMetadataStorage::parsePartFolderValidate(settings[ContentAddressedSetting::part_folder_validate].value); } Cas::BlobHashAlgo ContentAddressedSettings::blobHashAlgo() const @@ -259,9 +256,4 @@ Cas::StagingBackend ContentAddressedSettings::stagingBackend() const return impl->staging_backend_cached; } -Cas::PartFolderValidate ContentAddressedSettings::partFolderValidate() const -{ - return impl->part_folder_validate_cached; -} - } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.h index f9619ee93e75..47a9943421c6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.h @@ -11,15 +11,11 @@ namespace Poco { namespace Util { class AbstractConfiguration; } } // NOLINT(cpp namespace DB::Cas { -/// Forward declared to keep this header light: the full definitions live in -/// `ContentAddressedMetadataStorage.h` (`StagingBackend`) and `Parts/PartFolderAccess.h` -/// (`PartFolderValidate`), which are heavy and — in `ContentAddressedMetadataStorage.h`'s -/// case — will itself include this header once the metadata storage is rewired onto it. -/// Both are legal opaque declarations: `StagingBackend` fixes no explicit underlying type -/// (matching its definition, which leaves it as the implicit `int`), and `PartFolderValidate` -/// is only ever used here as an incomplete-type function return, never stored by value. +/// Forward declared to keep this header light: the full definition lives in +/// `ContentAddressedMetadataStorage.h`, which is heavy and will itself include this header once the +/// metadata storage is rewired onto it. A legal opaque declaration: `StagingBackend` fixes no +/// explicit underlying type, matching its definition. enum class StagingBackend; -struct PartFolderValidate; } namespace DB @@ -72,8 +68,8 @@ struct ContentAddressedSettings /// Fail-closed checks: `gc_interval_sec` and `gc_shards` must both be >= 1; `server_root_id` must /// be present (an ABSENT key throws a typed `NO_ELEMENTS_IN_CONFIG`, distinct from a /// PRESENT-but-invalid value, which throws `Cas::validateServerRootId`'s `BAD_ARGUMENTS`); and the - /// three enum-valued string settings (`blob_hash`, `staging_backend`, `part_folder_validate`) must - /// parse. The parsed enum values are cached for the typed accessors below. + /// two enum-valued string settings (`blob_hash`, `staging_backend`) must parse. The parsed enum + /// values are cached for the typed accessors below. void validate(); /// Typed accessors for the enum-valued string settings, parsed and cached by `validate`. @@ -83,11 +79,10 @@ struct ContentAddressedSettings /// only; the two scopes are deliberately distinct. bool skipAccessCheck() const; Cas::StagingBackend stagingBackend() const; - Cas::PartFolderValidate partFolderValidate() const; private: /// The parsed enum values live inside `impl` (defined in the .cpp, where the forward-declared - /// `Cas::StagingBackend` / `Cas::PartFolderValidate` types are complete), not as members here. + /// `Cas::StagingBackend` type is complete), not as members here. std::unique_ptr impl; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp index 437c4d536250..8f281eaea842 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp @@ -1208,8 +1208,9 @@ void ContentAddressedTransaction::createHardLink(const std::string & path_from, /// Carry forward from the COMMITTED source part: read the source manifest, find the named entry, /// record a TOKENLESS W-EVIDENCE dep for its blob (no HEAD before precommit; promote re-proves it). - /// ForceFresh getView == resolveRef(allow_stale=false) + readManifestShared, so this is the same - /// request pattern as before, now instrumented via the facade. + /// ForceFresh getView == resolveRef(allow_stale=false) + readManifestShared; the decode is served + /// from the manifest cache when warm, so a burst of hardlinks from one source part costs no + /// manifest request after the first. auto view = metadata_storage.partAccess()->getView(src->refKey(), Cas::Freshness::ForceFresh); if (!view) throw Exception(ErrorCodes::FILE_DOESNT_EXIST, @@ -1615,9 +1616,9 @@ void ContentAddressedTransaction::unlinkFile(const std::string & path, bool if_e std::erase_if(st.entries, [&](const Cas::ManifestEntry & e) { return e.path == r->file; }); if (!staged_here) { - /// One mandatory body-HEAD per (transaction, ref), not per file: the MergeTree fast-removal - /// path unlinks every file of the part through THIS transaction right before removeDirectory. - /// The first unlink re-proves the body ForceFresh; the rest of the burst reuses that proof. + /// One fresh resolve per (transaction, ref), not per file: the MergeTree fast-removal path + /// unlinks every file of the part through THIS transaction right before removeDirectory. + /// The first unlink resolves ForceFresh; the rest of the burst reads the retained view. const String memo_key = r->refKey().cacheKey(); const bool already_proven = force_fresh_validated_refs.contains(memo_key); const auto view = metadata_storage.partAccess()->getView( diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.h index a9ac41cfaac7..60e473ede583 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.h @@ -158,11 +158,11 @@ class ContentAddressedTransaction : public IMetadataTransaction bool committed = false; bool failed = false; - /// Memoizes, per (this transaction, ref), whether `unlinkFile` has already re-proven a committed - /// ref's manifest body `ForceFresh`. The MergeTree fast-removal path unlinks every file of a part - /// through ONE transaction right before `removeDirectory`: the first unlink's `ForceFresh` view - /// proves the body once; the rest of the burst reuse that proof (`Cas::Freshness::CachedForLoad`) - /// instead of paying one manifest-body HEAD per file. Cleared in `commit()`'s epilogue. + /// Memoizes, per (this transaction, ref), whether `unlinkFile` has already resolved a committed + /// ref `ForceFresh`. The MergeTree fast-removal path unlinks every file of a part through ONE + /// transaction right before `removeDirectory`: the first unlink resolves fresh and bypasses the + /// retained view; the rest of the burst reads the retained view (`Cas::Freshness::CachedForLoad`). + /// Cleared in `commit()`'s epilogue. std::unordered_set force_fresh_validated_refs; /// Stage a content part file as a blob without recording a dependency proof, and add/replace its diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.cpp index bce2e9423704..b29fe64b872d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.cpp @@ -29,7 +29,6 @@ namespace ProfileEvents extern const Event CASPartFolderViewOversizedBypasses; extern const Event CASPartFolderViewInvalidations; extern const Event CASRefRollbackBestEffortDropFailed; - extern const Event CASPartFolderValidateSkipped; extern const Event CASRefRepoint; } @@ -43,12 +42,11 @@ namespace DB::Cas { PartFolderView::PartFolderView(PartRefKey key_, Cas::ManifestId manifest_id_, uint64_t manifest_size_, - std::shared_ptr manifest_, uint64_t validated_at_ms_) + std::shared_ptr manifest_) : key(std::move(key_)) , manifest_id(std::move(manifest_id_)) , manifest_size(manifest_size_) , manifest_body(std::move(manifest_)) - , validated_at_ms(validated_at_ms_) { chassert(manifest_body); /// The binary-search contract: entries must be strictly ascending by `path` (sorted and unique) — @@ -62,12 +60,10 @@ PartFolderView::PartFolderView(PartRefKey key_, Cas::ManifestId manifest_id_, ui } std::shared_ptr PartFolderView::make( - PartRefKey key, const Cas::Resolved & resolved, std::shared_ptr manifest, - uint64_t validated_at_ms) + PartRefKey key, const Cas::Resolved & resolved, std::shared_ptr manifest) { return std::make_shared( - std::move(key), resolved.manifest_id, resolved.manifest_size, - std::move(manifest), validated_at_ms); + std::move(key), resolved.manifest_id, resolved.manifest_size, std::move(manifest)); } std::optional PartFolderView::projectionDirPrefix(const std::string & file) @@ -145,11 +141,9 @@ CachedPartFolderAccess::CachedPartFolderAccess(Cas::PoolPtr store_) { } -CachedPartFolderAccess::CachedPartFolderAccess(Cas::PoolPtr store_, CacheParams params_, std::function now_ms_fn_) - : store(std::move(store_)), params(params_), now_ms_fn(std::move(now_ms_fn_)) +CachedPartFolderAccess::CachedPartFolderAccess(Cas::PoolPtr store_, CacheParams params_) + : store(std::move(store_)), params(params_) { - if (!now_ms_fn) - now_ms_fn = []() -> uint64_t { return timeInMilliseconds(std::chrono::system_clock::now()); }; if (params.cache_bytes > 0) view_cache = std::make_unique( "LRU", CurrentMetrics::CASPartFolderCacheBytes, CurrentMetrics::CASPartFolderCacheEntries, @@ -171,8 +165,7 @@ CachedPartFolderAccess::getView(const PartRefKey & key, Freshness freshness) con const String cache_key = key.cacheKey(); /// Retained views serve `CachedForLoad` directly only after their manifest ID matches the fresh - /// resolve. `ForceFresh` must re-prove the manifest body unless the configured validation policy - /// explicitly permits a recent retained view; a fresh ref resolve proves ref currency, not body existence. + /// resolve. `ForceFresh` and `StrictValidate` always rebuild from the pool's manifest cache. if (freshness == Freshness::CachedForLoad && view_cache) { if (auto cached = view_cache->get(cache_key)) @@ -190,28 +183,6 @@ CachedPartFolderAccess::getView(const PartRefKey & key, Freshness freshness) con } } - /// With a non-`Always` validation policy, `ForceFresh` may serve a retained view without another - /// body HEAD when its manifest ID still matches and its validation timestamp is within the age - /// policy. `StrictValidate` bypasses retention. A manifest-ID mismatch always rebuilds, because all - /// part content is represented by the manifest. - if (freshness == Freshness::ForceFresh && view_cache && params.validate.mode != PartFolderValidate::Mode::Always) - { - if (auto cached = view_cache->get(cache_key); - cached && cached->manifestId() == resolved->manifest_id) - { - const bool fresh_enough = params.validate.mode == PartFolderValidate::Mode::Never - || (now_ms_fn() - cached->validatedAtMs()) < params.validate.age_seconds * 1000ULL; - if (fresh_enough) - { - ProfileEvents::increment(ProfileEvents::CASPartFolderViewHits); - ProfileEvents::increment(ProfileEvents::CASPartFolderValidateSkipped); - recordDecision(cache_key, LastDecision::Hit, cached.get(), /*retained=*/true); - emitResolveEvent(key, *resolved); - return cached; - } - } - } - auto view = buildView(key, *resolved, freshness); /// Retain eligible views. `StrictValidate` never populates the cache, and oversized views are @@ -264,10 +235,10 @@ void CachedPartFolderAccess::emitResolveEvent(const PartRefKey & key, const Cas: std::shared_ptr CachedPartFolderAccess::buildView( const PartRefKey & key, const Cas::Resolved & resolved, Freshness freshness) const { - /// Fresh modes do not coalesce: each `ForceFresh`/`StrictValidate` call owns its mandatory HEAD. - /// Only cold `CachedForLoad` builds use single-flight. + /// Fresh modes do not coalesce: each `ForceFresh`/`StrictValidate` call owns its own read (a cache + /// hit costs no request). Only cold `CachedForLoad` builds use single-flight. if (freshness != Freshness::CachedForLoad) - return PartFolderView::make(key, resolved, store->readManifestShared(resolved.manifest_id), now_ms_fn()); + return PartFolderView::make(key, resolved, store->readManifestShared(resolved.manifest_id)); std::promise> promise; std::shared_future> future; @@ -292,7 +263,7 @@ std::shared_ptr CachedPartFolderAccess::buildView( }); try { - auto view = PartFolderView::make(key, resolved, store->readManifestShared(resolved.manifest_id), now_ms_fn()); + auto view = PartFolderView::make(key, resolved, store->readManifestShared(resolved.manifest_id)); promise.set_value(view); return view; } @@ -505,9 +476,10 @@ Cas::CommitOutcome CachedPartFolderAccess::publishEntries(const PartRefKey & dst bool CachedPartFolderAccess::republishRef(const PartRefKey & src, const PartRefKey & dst) { - /// Content addressing has no rename, so move a committed ref by reading the source body freshly, - /// publishing equivalent entries at the destination, and then dropping the source. The source - /// body is re-proved and is never taken from a retained view. + /// Content addressing has no rename, so move a committed ref by reading the source manifest + /// through the pool's manifest cache after a fresh ref resolve, publishing equivalent entries at + /// the destination, and then dropping the source. The source blobs are adopted by evidence of the + /// live source edge, never re-probed; the decode is never taken from a retained view. auto resolved = store->resolveRef(src.ns, src.ref); if (!resolved) return false; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.h index cfa81f83662a..a9f4f3c40665 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.h @@ -52,11 +52,10 @@ struct CommitOutcome bool created = false; }; -/// Read-freshness policy at the part-folder access boundary. The -/// mutable-read-vs-write-evidence distinction is carried by the METHOD, not a fourth value: -/// mutable per-part reads call `resolve` (no manifest involved); write-path source reads call -/// `getView`, which under ForceFresh always re-proves the manifest body (mandatory HEAD in -/// `readManifestShared` — a fresh ref resolve alone proves ref currency, NOT body existence). +/// Read-freshness policy at the part-folder access boundary. The mutable-read-vs-write-evidence +/// distinction is carried by the METHOD, not a fourth value: mutable per-part reads call `resolve` +/// (no manifest involved); write-path source reads call `getView`, which under ForceFresh always +/// resolves fresh and bypasses the retained view. enum class Freshness { CachedForLoad, /// repeated load-window reads; stale-tolerant resolve (allow_stale=true) @@ -79,15 +78,12 @@ class PartFolderView /// must be non-null and its entries must be strictly ascending by canonical path; this is the /// ordering required by the binary-search and range-scan helpers. PartFolderView(PartRefKey key_, Cas::ManifestId manifest_id_, uint64_t manifest_size_, - std::shared_ptr manifest_, uint64_t validated_at_ms_); + std::shared_ptr manifest_); - /// Joins a fresh `Resolved` with its validated shared decode. `validated_at_ms` is supplied by - /// the caller after `readManifestShared` has proven the manifest body with a HEAD. Keeping the - /// timestamp outside this helper lets `CachedPartFolderAccess` use one injectable clock for both - /// the stamp and its age-window comparison. + /// Joins a fresh `Resolved` with its validated shared decode. static std::shared_ptr make( PartRefKey key, const Cas::Resolved & resolved, - std::shared_ptr manifest, uint64_t validated_at_ms); + std::shared_ptr manifest); /// Recognizes a projection directory by its last path component, `.proj` or `.tmp_proj`, and /// returns the corresponding in-tree prefix. The input is the routed file path; unrelated paths @@ -97,10 +93,6 @@ class PartFolderView const PartRefKey & refKey() const { return key; } const Cas::ManifestId & manifestId() const { return manifest_id; } const std::shared_ptr & manifest() const { return manifest_body; } - /// The wall-clock ms at which this view's manifest body was last proven live by a HEAD. A - /// refresh that changes only ref metadata carries the original stamp forward because it did not - /// re-prove the body. - uint64_t validatedAtMs() const { return validated_at_ms; } /// Finds an entry by canonical path using the manifest's sorted-entry invariant. const Cas::ManifestEntry * findFile(const String & path) const; @@ -122,7 +114,6 @@ class PartFolderView Cas::ManifestId manifest_id; uint64_t manifest_size = 0; std::shared_ptr manifest_body; - uint64_t validated_at_ms = 0; }; } @@ -132,17 +123,6 @@ namespace DB::Cas { class PartWriteTxn; } namespace DB::Cas { -/// Controls whether `ForceFresh` must re-prove the manifest body on every access. `Always` (the default) -/// preserves the fail-closed body check; `Age` and `Never` may serve a retained view after a fresh ref -/// resolve when its manifest ID matches. A ref resolve proves ref currency, but not that the manifest -/// body still exists, so these modes trade that additional check for a bounded performance optimization. -struct PartFolderValidate -{ - enum class Mode : uint8_t { Always, Age, Never }; - Mode mode = Mode::Always; - uint64_t age_seconds = 0; /// only meaningful for Mode::Age -}; - class CachedPartFolderAccess; /// A part write that has been staged and PRECOMMITTED but not yet promoted -- the durable-but- @@ -237,8 +217,6 @@ class CachedPartFolderAccess /// path takes a per-disk global mutex and allocates on EVERY read. Off by default so the read /// hit path never pays for it; the disk factory / tests turn it on when they consult `explain`. bool explain_enabled = false; - /// The `ForceFresh` manifest-body re-proof policy. `Always` is the fail-closed default. - PartFolderValidate validate; }; /// `CacheParams params_ = {}` cannot be a default argument here — Clang's complete-class- @@ -247,16 +225,12 @@ class CachedPartFolderAccess /// argument written inside the class body is evaluated too early. Two overloads sidestep it; the /// single-arg form default-constructs `CacheParams` (retention disabled) out-of-line. explicit CachedPartFolderAccess(Cas::PoolPtr store_); - /// `now_ms_fn_`: wall-clock ms, injected (tests) for the age-window comparison AND the - /// retained view's `validated_at_ms` stamp -- the SAME function drives both, so a test controls - /// each side of the comparison exactly. Defaults to `std::chrono::system_clock` (mirrors - /// `Cas::Gc`'s `now_ms_fn` convention) when empty. - CachedPartFolderAccess(Cas::PoolPtr store_, CacheParams params_, std::function now_ms_fn_ = {}); + CachedPartFolderAccess(Cas::PoolPtr store_, CacheParams params_); /// Resolves the ref and, when present, reads and validates its manifest into an immutable view. - /// `nullptr` means the ref is absent. Strict validation and the default `ForceFresh` policy reach - /// `readManifestShared`'s mandatory HEAD because a fresh ref resolve alone does not prove that the - /// manifest body still exists. + /// `nullptr` means the ref is absent. `ForceFresh` and `StrictValidate` bypass the retained view + /// and read through the pool's manifest cache; a manifest is immutable per id, so a retained view + /// can be stale only by naming a different manifest id, which the fresh resolve detects. std::shared_ptr getView(const PartRefKey & key, Freshness freshness) const; /// Ref-only resolution (per-part reads, part-dir existence, publish stamps): no @@ -364,9 +338,6 @@ class CachedPartFolderAccess private: Cas::PoolPtr store; CacheParams params; - /// Wall-clock milliseconds; see the constructor comment. `std::function::operator` is const, so this is - /// callable from const methods (`getView`, `buildView`) without a `mutable` qualifier. - std::function now_ms_fn; /// Supplies the conservative encoded-manifest weight used by `CacheBase` for eviction decisions. struct ViewWeight @@ -384,7 +355,7 @@ class CachedPartFolderAccess mutable std::unordered_map>> inflight; /// Reads a manifest and constructs a view. Cold `CachedForLoad` builds are single-flight per key; - /// fresh modes perform their own read so each call retains its validation guarantee. + /// fresh modes perform their own read. std::shared_ptr buildView( const PartRefKey & key, const Cas::Resolved & resolved, Freshness freshness) const; /// Removes a retained view and records the invalidation for diagnostics. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp index 778814997ca1..a9edd6783c8e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp @@ -40,30 +40,20 @@ CasManifestReader::CasManifestReader( manifest_decode_cache_bytes, /*max_count=*/16384, ManifestDecodeCache::DEFAULT_SIZE_RATIO); } -size_t CasManifestReader::ManifestCacheKeyHash::operator()(const ManifestCacheKey & k) const -{ - /// Combine the manifest-id hash with the token's bytes + type. The token is part of the key so a - /// re-incarnation under the same id misses (the immutable bytes changed identity). - const size_t h1 = std::hash{}(k.manifest_id); - const size_t h2 = std::hash{}(k.token.value); - const size_t h3 = std::hash{}(static_cast(k.token.type)); - size_t h = h1; - h ^= h2 + 0x9e3779b97f4a7c15ULL + (h << 6) + (h >> 2); - h ^= h3 + 0x9e3779b97f4a7c15ULL + (h << 6) + (h >> 2); - return h; -} - std::shared_ptr CasManifestReader::readManifestShared(const ManifestId & id) { + /// One id names one content forever (minted once, written once, only ever deleted), so a cached + /// decode is served without any request. + if (manifest_cache) + if (auto cached = manifest_cache->get(id)) + return cached; + /// A live reference naming a missing manifest body is a dangling-reference violation /// (`INV-NO-DANGLE`). Never substitute an empty manifest: callers must observe the missing object - /// as an exception. + /// as an exception. The `GET` alone carries the absence signal, so no `HEAD` precedes it. const String key = layout.manifestKey(id); - - /// `HEAD` is mandatory even on a cache hit. It proves that the live reference still names an - /// existing object and supplies the token that identifies the immutable bytes being reused. - const HeadResult head = backend.head(key); - if (!head.exists) + std::optional object = backend.get(key); + if (!object) { if (event_sink) { @@ -79,15 +69,6 @@ std::shared_ptr CasManifestReader::readManifestShared(const throw Exception(ErrorCodes::FILE_DOESNT_EXIST, "live ref names manifest at {} but its object is missing — INV-NO-DANGLE", key); } - - if (manifest_cache) - if (auto cached = manifest_cache->get(ManifestCacheKey{.manifest_id = id, .token = head.token})) - return cached; - - std::optional object = backend.get(key); - if (!object) - throw Exception(ErrorCodes::FILE_DOESNT_EXIST, - "manifest at {} vanished between head and get — INV-NO-DANGLE", key); ProfileEvents::increment(ProfileEvents::CASPartFolderManifestGets); PartManifest body = decodePartManifest(openObject(FormatId::PartManifest, object->bytes)); @@ -132,7 +113,7 @@ std::shared_ptr CasManifestReader::readManifestShared(const auto decoded = std::make_shared(std::move(body)); if (manifest_cache) - manifest_cache->set(ManifestCacheKey{.manifest_id = id, .token = head.token}, decoded); + manifest_cache->set(id, decoded); return decoded; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h index af5d31776857..e3aa15955743 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h @@ -21,12 +21,12 @@ struct BlobLocation uint64_t length = 0; }; -/// Reads and validates part manifests, caches immutable decodes, and translates blob entries into -/// ranged object reads. A read first obtains the object's current backend token, then reuses a -/// decode only for the matching `(ManifestId, Token)` pair; a cache miss performs a `GET` and -/// validates both the manifest reference and owning namespace before publication into the cache. -/// Missing or changing objects and failed identity checks are surfaced as exceptions, never as an -/// empty or partially trusted manifest. +/// Reads and validates part manifests, caches immutable decodes by `ManifestId`, and translates blob +/// entries into ranged object reads. A manifest id is minted once and its body is written once, so +/// one id names one content forever: a cache hit is served without any request, and a miss performs +/// one `GET` and validates both the manifest reference and the owning namespace before publication +/// into the cache. A missing body, a decode failure or a failed identity check is surfaced as an +/// exception, never as an empty or partially trusted manifest. /// /// The reader receives its backend, immutable layout and pool metadata, and event sink by reference; /// it has no `Pool` back-reference and owns no `Pool`-level mutex. The decode cache is a @@ -36,14 +36,14 @@ class CasManifestReader { public: /// Binds the reader to the pool environment. A positive cache budget creates the byte-weighted - /// LRU; zero disables caching while leaving the mandatory `HEAD` and validation sequence intact. + /// LRU; zero disables caching while leaving the one-`GET`-and-validate sequence intact. CasManifestReader( Backend & backend_, const Layout & layout_, const PoolMeta & meta_, const CasEventSink & event_sink_, size_t manifest_decode_cache_bytes); /// Reads a manifest by value using the fail-closed sequence described above. A missing body, - /// disappearance between `HEAD` and `GET`, decode failure, or either identity mismatch throws; - /// only a fully validated decode can enter the cache. + /// decode failure, or either identity mismatch throws; only a fully validated decode can enter + /// the cache. PartManifest readManifest(const ManifestId & id); /// Reads a manifest like `readManifest` but returns the immutable shared decode. This preserves @@ -59,25 +59,9 @@ class CasManifestReader size_t manifestDecodeCacheBytes() const { return manifest_cache ? manifest_cache->sizeInBytes() : 0; } private: - /// The cache must include the backend token: a reused manifest identifier can refer to a new - /// object incarnation, and its immutable decoded bytes must not be reused across incarnations. - struct ManifestCacheKey - { - ManifestId manifest_id; - Token token; - bool operator==(const ManifestCacheKey &) const = default; - }; - - /// Hashes both identity components and the token type so cache lookup uses the same complete - /// identity as `ManifestCacheKey::operator==`. - struct ManifestCacheKeyHash - { - size_t operator()(const ManifestCacheKey & k) const; - }; - /// Estimates retained decode memory from fixed object overhead plus entry path and inline-byte - /// storage. Weighting by bytes gives a server reading many parts an honest memory ceiling instead - /// of a count-only bound; the cache key still provides the fail-closed token semantics. + /// storage. Weighting by bytes gives a server reading many parts an honest memory ceiling + /// instead of a count-only bound. struct PartManifestWeight { /// Returns the approximate bytes retained for one decoded manifest by the cache. @@ -89,7 +73,7 @@ class CasManifestReader return bytes; } }; - using ManifestDecodeCache = CacheBase; + using ManifestDecodeCache = CacheBase, PartManifestWeight>; Backend & backend; const Layout & layout; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index f6af8d4c7ab3..b75f34d4092c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -555,15 +555,16 @@ class Pool : public std::enable_shared_from_this { return ref_ledger.confirmExactRef(ns, ref_name, manifest_ref); } - /// Read the single immutable part manifest named by `id`. Derives the key via CasLayout::manifestKey, - /// decodes the body, and fails CLOSED: a committed ref naming a missing body throws FILE_DOESNT_EXIST - /// (INV-NO-DANGLE surfaced on the read path); a body whose `ref` ≠ id.ref (refMatchesBody) or whose + /// Read the single immutable part manifest named by `id`. Serves the cached decode when the id + /// is cached (no request); otherwise derives the key via CasLayout::manifestKey, GETs and decodes + /// the body, and fails CLOSED: an absent body throws FILE_DOESNT_EXIST (a committed ref naming a + /// missing body is a dangling reference); a body whose `ref` ≠ id.ref (refMatchesBody) or whose /// `root_namespace_id` ≠ id.root_namespace (manifestNamespaceMatches) throws CORRUPTED_DATA — the - /// ref is addressing the wrong object, or a cross-namespace dangle. Token-gated decode cache below. + /// ref is addressing the wrong object, or a cross-namespace dangle. Id-keyed decode cache below. PartManifest readManifest(const ManifestId & id); - /// Identical to `readManifest` (same mandatory HEAD, same fail-closed validation, same decode - /// cache) but returns the SHARED immutable decode the manifest cache holds — no per-call copy. - /// The wiring read path uses this variant. + /// Identical to `readManifest` (same fail-closed validation, same decode cache) but returns the + /// SHARED immutable decode the manifest cache holds — no per-call copy. The wiring read path + /// uses this variant. std::shared_ptr readManifestShared(const ManifestId & id); BlobLocation locate(const ManifestEntry & entry) const; /// Blob placement only std::map listRefs(const RootNamespace & ns); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md index 305ffe1b1ff9..dbf5d104d79e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md @@ -112,7 +112,7 @@ Primitives → Formats → Backend → Pool → Gc → Tools ≈ Parts → facad `CasDecommission`, `CasInspect`. - **`Parts/`** — part semantics over the pool: `PartPathParser` (the ClickHouse-path classifier), `PartFolderAccess` (`PartRefKey` + `Freshness` - + `PartFolderValidate` + `PartFolderView` + `CachedPartFolderAccess`). + + `PartFolderView` + `CachedPartFolderAccess`). - **Top level (facade)** — the entry points: `ContentAddressedMetadataStorage` (the `IMetadataStorage` facade), `ContentAddressedTransaction` (the `IMetadataTransaction`, including the write buffers), `ContentAddressedExchange` diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 0f1a610359aa..aea9ec64e3ac 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -127,9 +127,9 @@ inline void ensureBlobUploadPoolForTest(size_t size = 8) /// Minimal `ContentAddressedSettings` for a direct-construction gtest fixture: sets only /// `server_root_id` and `scratch_path` (the two values every positional-ctor call site used to pass -/// explicitly) and validates, so the cached enum-valued accessors (`stagingBackend`, `blobHashAlgo`, -/// `partFolderValidate`) are populated from their (default) string settings exactly as the disk-factory -/// path would populate them. Callers that need a non-default setting (e.g. `staging_backend=s3`) apply +/// explicitly) and validates, so the cached enum-valued accessors (`stagingBackend`, `blobHashAlgo`) +/// are populated from their (default) string settings exactly as the disk-factory path would populate +/// them. Callers that need a non-default setting (e.g. `staging_backend=s3`) apply /// the override via `settings[ContentAddressedSetting::x] = value;` and re-run `settings.validate()` /// themselves before constructing. inline DB::ContentAddressedSettings makeSettingsForTest(const std::string & server_root_id, const std::filesystem::path & scratch_path) diff --git a/src/Disks/tests/gtest_ca_transaction.cpp b/src/Disks/tests/gtest_ca_transaction.cpp index 324e4ad72bd0..ebf2311a8a11 100644 --- a/src/Disks/tests/gtest_ca_transaction.cpp +++ b/src/Disks/tests/gtest_ca_transaction.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -11,7 +12,6 @@ namespace ProfileEvents { extern const Event CASRefRepoint; -extern const Event CASManifestHead; } namespace DB::ErrorCodes @@ -621,13 +621,9 @@ TEST(CASTransactionRemove, UnlinkStormThenDirDropIsOneRefDrop) EXPECT_EQ(rep.dangling, 0u); } -/// Task 22 (URF plan phase 7): the MergeTree fast-removal shape's per-file ForceFresh proof is -/// memoized per (transaction, ref) in `unlinkFile` — the first unlink's `ForceFresh` `getView` re-proves -/// the manifest body with one HEAD; the rest of the burst reuses that proof via `CachedForLoad`. This -/// pins the HEAD-count side of `UnlinkStormThenDirDropIsOneRefDrop` above (which already pins the -/// repoint count): before this memoization, N unlinks of the same part paid N manifest-body HEADs; now -/// the whole storm-then-drop transaction pays exactly one. -TEST(CASTransactionRemove, UnlinkStormMemoizesOneForceFreshHead) +/// Memoizing the per-file `ForceFresh` read per (transaction, ref) saves a ref resolve and a view +/// rebuild on every file of the burst after the first. +TEST(CASTransactionRemove, UnlinkStormMemoizesOneForceFreshResolve) { auto storage = openTxStorage(); @@ -643,8 +639,16 @@ TEST(CASTransactionRemove, UnlinkStormMemoizesOneForceFreshHead) } ASSERT_TRUE(storage->existsDirectory("b09/b09b09b0-0909-4909-8909-090909090909/all_1_1_0")); - const uint64_t heads_before = ProfileEvents::global_counters[ProfileEvents::CASManifestHead].load(); + /// A warm view-cache hit deliberately emits no `RefResolve`, so this counts exactly the calls that + /// did real resolve work on the part. + size_t resolves = 0; + storage->store()->setEventSink([&](DB::Cas::CasEvent e) + { + if (e.type == DB::Cas::CasEventType::RefResolve && e.ref_name == "all_1_1_0") + ++resolves; + }); + size_t resolves_after_unlinks = 0; /// 2. The MergeTree fast-removal shape: unlink every file one-by-one, THEN removeDirectory — all /// in ONE transaction (mirrors UnlinkStormThenDirDropIsOneRefDrop above). { @@ -652,17 +656,19 @@ TEST(CASTransactionRemove, UnlinkStormMemoizesOneForceFreshHead) tx->unlinkFile("b09/b09b09b0-0909-4909-8909-090909090909/all_1_1_0/checksums.txt", /*if_exists=*/false, /*should_remove_objects=*/true); tx->unlinkFile("b09/b09b09b0-0909-4909-8909-090909090909/all_1_1_0/data.bin", /*if_exists=*/false, /*should_remove_objects=*/true); tx->unlinkFile("b09/b09b09b0-0909-4909-8909-090909090909/all_1_1_0/txn_version.txt", /*if_exists=*/false, /*should_remove_objects=*/true); + resolves_after_unlinks = resolves; tx->removeDirectory("b09/b09b09b0-0909-4909-8909-090909090909/all_1_1_0"); tx->commit(DB::NoCommitOptions{}); } - /// 3. The whole part is gone, and the three-file unlink storm paid exactly ONE manifest-body HEAD - /// (the first unlink's ForceFresh proof) — not three. removeDirectory clears the staged removal - /// marks, so publishStaging's own (unmemoized) ForceFresh getView never fires for this ref either - /// (see UnlinkStormThenDirDropIsOneRefDrop's zero-repoints assertion above). + storage->store()->setEventSink(nullptr); + + /// 3. The whole part is gone, and the three-file burst resolved it exactly once: the first unlink's + /// ForceFresh read, with the other two served from the retained view it left behind. Without the + /// memo each unlink would resolve again. EXPECT_FALSE(storage->existsDirectory("b09/b09b09b0-0909-4909-8909-090909090909/all_1_1_0")); - EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASManifestHead].load(), heads_before + 1) - << "unlink-storm-then-dir-drop must pay exactly one ForceFresh manifest-body HEAD, not one per file"; + EXPECT_EQ(resolves_after_unlinks, 1u) + << "an unlink storm over one ref must resolve it once for the whole burst, not once per file"; } /// A create-then-remove of a new part in one transaction must discard both the manifest entries diff --git a/src/Disks/tests/gtest_ca_wiring.cpp b/src/Disks/tests/gtest_ca_wiring.cpp index 6d34e444d603..dad37b10bedc 100644 --- a/src/Disks/tests/gtest_ca_wiring.cpp +++ b/src/Disks/tests/gtest_ca_wiring.cpp @@ -576,6 +576,64 @@ TEST(CASWiringRead, BlobViewPlanRidesTheStandardPipeline) } } +/// A retained view whose blob the collector has since removed still plans the read (no I/O), and the +/// read itself throws a typed exception at the first byte; it never returns an empty payload, and +/// the size it reports comes from the manifest, not from the missing object. +TEST(CASWiringRead, DeletedBlobUnderStaleViewFailsTypedNotEmpty) +{ + auto object_storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + auto settings = DB::Cas::tests::makeSettingsForTest( + "test", std::filesystem::temp_directory_path() / "ca_wiring_stale_view"); + auto storage = std::make_shared( + object_storage, "pool", "srv1", "", nullptr, settings); + storage->startup(); + publishWiredPart(*storage, storage->liveNamespace("a11a11a1-1111-4111-8111-111111111111"), "all_1_1_0"); + + const std::string path = "a11/a11a11a1-1111-4111-8111-111111111111/all_1_1_0/data.bin"; + auto plan_before = storage->getBlobViewPlan(path); + ASSERT_TRUE(plan_before.has_value()); /// warms the view and decode caches + + /// Remove the blob object exactly as GC would once nothing references it. + auto pool = storage->store(); + const String blob_key = pool->layout().blobKey(DB::Cas::tests::idOf("payload-A")); + { + const DB::Cas::HeadResult h = pool->backend().head(blob_key); + ASSERT_TRUE(h.exists); + pool->backend().deleteExact(blob_key, h.token); + } + + /// Planning still succeeds from the cached manifest and names the same object. + auto plan_after = storage->getBlobViewPlan(path); + ASSERT_TRUE(plan_after.has_value()); + EXPECT_EQ(plan_after->object.remote_path, plan_before->object.remote_path); + + const auto objects = storage->getStorageObjects(path); + ASSERT_EQ(objects.size(), 1u); + EXPECT_EQ(objects[0].bytes_size, String("payload-A").size()); /// size comes from the manifest + + const DB::Cas::BlobLocation location{ + .key = objects[0].remote_path, + .offset = pool->poolMeta().blob_header_len, + .length = objects[0].bytes_size}; + /// The local object storage opens the file eagerly and maps ENOENT to FILE_DOESNT_EXIST + /// (ReadBufferFromFile); on S3 the same read raises S3_ERROR at the first byte. Either way the + /// failure is a typed exception, never an empty payload. + String got; + int code = 0; + try + { + auto buf = storage->readBlobPayload(location, path, DB::ReadSettings{}); + DB::readStringUntilEOF(got, *buf); + } + catch (const DB::Exception & e) + { + code = e.code(); + } + EXPECT_EQ(code, DB::ErrorCodes::FILE_DOESNT_EXIST) + << "read of a deleted blob returned " << got.size() << " bytes (code " << code << ")"; + EXPECT_TRUE(got.empty()); +} + TEST(CASWiringRead, ProjectionDirectory) { auto storage = openWiringStorage(); diff --git a/src/Disks/tests/gtest_cas_part_folder_access.cpp b/src/Disks/tests/gtest_cas_part_folder_access.cpp index 69fc028aeda4..dc7928080b4b 100644 --- a/src/Disks/tests/gtest_cas_part_folder_access.cpp +++ b/src/Disks/tests/gtest_cas_part_folder_access.cpp @@ -5,11 +5,9 @@ #include #include #include -#include #include #include #include -#include #include #include #include @@ -28,7 +26,6 @@ namespace DB::ErrorCodes namespace ProfileEvents { extern const Event CASRefRollbackBestEffortDropFailed; -extern const Event CASPartFolderValidateSkipped; } using namespace DB; @@ -64,17 +61,7 @@ Cas::ManifestId publishPart(const Cas::PoolPtr & store, const Cas::RootNamespace Cas::CachedPartFolderAccess::CacheParams cacheOn() { return {.cache_bytes = 64ULL << 20, .max_entries = 10000, .max_entry_bytes = 16ULL << 20, - .explain_enabled = true, .validate = {}}; -} - -/// Mirrors gtest_cas_s3_staging.cpp's helper of the same shape: the shape a real CAS disk config -/// has under `storage_configuration.disks.`, so `config_prefix = "disk"` reads exactly like -/// the disk factory's `config_prefix`. Used to unit-test `parsePartFolderValidate` standalone. -Poco::AutoPtr configWithDiskSection(const std::string & inner_xml) -{ - std::istringstream xml_stream( // STYLE_CHECK_ALLOW_STD_STRING_STREAM - "" + inner_xml + ""); - return new Poco::Util::XMLConfiguration(xml_stream); + .explain_enabled = true}; } /// Every mutating backend op throws once armed — models a correlated backend outage during the @@ -190,7 +177,7 @@ class PromoteDefiniteFailureBackend final : public Cas::InMemoryBackend } -TEST(CASPartFolderAccess, RetainedHitSkipsManifestHead) +TEST(CASPartFolderAccess, RetainedHitCostsNoRequest) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); @@ -201,14 +188,21 @@ TEST(CASPartFolderAccess, RetainedHitSkipsManifestHead) const Cas::PartRefKey key{ns, "part_1"}; const String manifest_key = layout.manifestKey(id); + /// Cold build: warms the retained view and the decode cache. Excluded from the counts below so + /// they measure only the warm hits that follow. + ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); + backend->resetCounts(); - for (int i = 0; i < 5; ++i) + for (int i = 0; i < 4; ++i) ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); - /// The one-GET goal (spec acceptance 4): ONE body GET, ONE mandatory HEAD (the cold build); - /// every subsequent CachedForLoad call is a validated hit — zero manifest ops. - EXPECT_EQ(backend->getCount(manifest_key), 1u); - EXPECT_EQ(backend->headCount(manifest_key), 1u); + /// A retained hit costs no request at all -- not merely no manifest GET on this key, but no + /// backend traffic of ANY kind (GET, HEAD, LIST, streamed GET) against ANY key. + EXPECT_EQ(backend->getCount(manifest_key), 0u); + EXPECT_EQ(backend->getTotal(), 0u); + EXPECT_EQ(backend->headTotal(), 0u); + EXPECT_EQ(backend->listTotal(), 0u); + EXPECT_EQ(backend->getStreamTotal(), 0u); EXPECT_TRUE(access.explain(key).retained); EXPECT_EQ(access.explain(key).last_decision, Cas::CachedPartFolderAccess::LastDecision::Hit); @@ -224,7 +218,7 @@ TEST(CASPartFolderAccess, HitPathJournalEmptyAndCheapWhenExplainDisabled) /// per-disk explain mutex nor write a journal entry (B2). Cas::CachedPartFolderAccess access(store, {.cache_bytes = 64ULL << 20, .max_entries = 10000, .max_entry_bytes = 16ULL << 20, - .explain_enabled = false, .validate = {}}); + .explain_enabled = false}); const auto id = publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); const Cas::PartRefKey key{ns, "part_1"}; const String manifest_key = layout.manifestKey(id); @@ -233,9 +227,8 @@ TEST(CASPartFolderAccess, HitPathJournalEmptyAndCheapWhenExplainDisabled) for (int i = 0; i < 5; ++i) ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); - /// Same request oracle as RetainedHitSkipsManifestHead — one cold build, then validated hits. + /// Same request oracle as RetainedHitCostsNoRequest — one cold build, then retained hits. EXPECT_EQ(backend->getCount(manifest_key), 1u); - EXPECT_EQ(backend->headCount(manifest_key), 1u); /// The journal is never written when disabled. EXPECT_EQ(access.explainJournalSizeForTest(), 0u); /// explain() still reports live retention truthfully, but the decision defaults to Miss (unwritten). @@ -275,12 +268,11 @@ TEST(CASPartFolderAccess, GetViewFailsClosedOnMissingBody) const Cas::RootNamespace ns{"srv/t1"}; const auto id = publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); - /// Physically delete the live manifest body (a protocol violation) — every getView mode must - /// surface INV-NO-DANGLE as FILE_DOESNT_EXIST in Phase 2 (there is no retained view to hit). - /// Retention is off (the single-arg ctor below), so this is the `always` (default) part_folder_validate - /// mode under test regardless — the `never`/`age` skip is proven by the ValidateNever/ValidateAge - /// tests further down, which turn retention ON. + /// Physically delete the live manifest body (a protocol violation). Retention is off and the + /// decode cache is cold (promote reads the body through the backend, not the reader), so every + /// getView mode reaches the reader's miss path: one GET, no HEAD, FILE_DOESNT_EXIST. deleteManifestBody(*backend, layout, id); + backend->resetCounts(); Cas::CachedPartFolderAccess access(store); const Cas::PartRefKey key{ns, "part_1"}; @@ -288,6 +280,7 @@ TEST(CASPartFolderAccess, GetViewFailsClosedOnMissingBody) Cas::Freshness::ForceFresh, Cas::Freshness::StrictValidate}) expectThrowsCode(ErrorCodes::FILE_DOESNT_EXIST, [&] { access.getView(key, freshness); }); + EXPECT_EQ(backend->headCount(layout.manifestKey(id)), 0u); } TEST(CASPartFolderAccess, WritePrimitivesRoundTrip) @@ -678,7 +671,7 @@ TEST(CASPartFolderAccess, ExplainRecordsDecisions) auto backend = std::make_shared(); auto store = openPoolForTest(backend); const Cas::RootNamespace ns{"srv/t1"}; - Cas::CachedPartFolderAccess access(store, {.explain_enabled = true, .validate = {}}); + Cas::CachedPartFolderAccess access(store, {.explain_enabled = true}); publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); const Cas::PartRefKey key{ns, "part_1"}; @@ -717,11 +710,10 @@ TEST(CASPartFolderAccess, BaselineRequestCountsWithoutRetention) for (int i = 0; i < n; ++i) ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); - /// The Phase-3 baseline (retention off): one manifest-body GET (the decode cache absorbs the - /// rest) but a mandatory manifest HEAD per call. Phase 4's validated hits remove the HEADs; - /// this test pins the numbers Phase 4 improves. + /// Retention off: one manifest-body GET (the decode cache absorbs the rest) and no manifest + /// HEAD at all — a cached decode is served without a request. EXPECT_EQ(backend->getCount(manifest_key), 1u); - EXPECT_EQ(backend->headCount(manifest_key), static_cast(n)); + EXPECT_EQ(backend->headCount(manifest_key), 0u); } /// ==== Phase 4 (retention) semantics battery: spec §Testing acceptance criteria ==== @@ -767,7 +759,11 @@ TEST(CASPartFolderAccess, MismatchRebuildAfterRepublish) EXPECT_TRUE(access.explain(key).retained); } -TEST(CASPartFolderAccess, ForceFreshFailsClosedWhileRetainedViewExists) +/// The decode cache is keyed by id and an id names one content forever, so a warm reader serves +/// `ForceFresh` from the immutable decode with no manifest request even after the body object is +/// gone. A retained-view hit is not what is being tested here: `ForceFresh` bypasses the view cache +/// and rebuilds the view from the pool's manifest cache. +TEST(CASPartFolderAccess, ForceFreshServesImmutableDecodeWithoutManifestRequestsAfterBodyDeletion) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); @@ -776,178 +772,53 @@ TEST(CASPartFolderAccess, ForceFreshFailsClosedWhileRetainedViewExists) Cas::CachedPartFolderAccess access(store, cacheOn()); const auto id = publishPart(store, ns, "part_1", {inlineEntry("f", "x")}); const Cas::PartRefKey key{ns, "part_1"}; + const String manifest_key = layout.manifestKey(id); - ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); /// retained + ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); /// warms the decode cache deleteManifestBody(*backend, layout, id); /// protocol violation: live body vanishes + backend->resetCounts(); - /// Write-evidence and strict paths surface INV-NO-DANGLE immediately (mandatory HEAD)... - expectThrowsCode(ErrorCodes::FILE_DOESNT_EXIST, - [&] { access.getView(key, Cas::Freshness::ForceFresh); }); - expectThrowsCode(ErrorCodes::FILE_DOESNT_EXIST, - [&] { access.getView(key, Cas::Freshness::StrictValidate); }); - - /// ...while a validated CachedForLoad hit still serves the immutable decode — the documented - /// residual delta (spec §Staleness Equivalence): detection deferred, never for write evidence. - EXPECT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); + auto view = access.getView(key, Cas::Freshness::ForceFresh); + ASSERT_NE(view, nullptr); + EXPECT_NE(view->findFile("f"), nullptr); + EXPECT_EQ(backend->getCount(manifest_key), 0u); + EXPECT_EQ(backend->headCount(manifest_key), 0u); + EXPECT_EQ(access.explain(key).last_decision, Cas::CachedPartFolderAccess::LastDecision::ForceFreshRead); + + /// `StrictValidate` serves the same immutable decode: an id names one content, so once the id is + /// resolved there is nothing stricter left to prove about the body. It bypasses retention, so its + /// recorded decision differs from the `ForceFresh` one above. + auto strict_view = access.getView(key, Cas::Freshness::StrictValidate); + ASSERT_NE(strict_view, nullptr); + EXPECT_EQ(strict_view->manifest().get(), view->manifest().get()); + EXPECT_EQ(backend->getCount(manifest_key), 0u); + EXPECT_EQ(backend->headCount(manifest_key), 0u); + EXPECT_EQ(access.explain(key).last_decision, Cas::CachedPartFolderAccess::LastDecision::StrictBypass); } -/// ==== §3 (part_folder_validate): the ForceFresh body re-proof HEAD is configurable ==== - -TEST(CASPartFolderAccess, ValidateNeverServesRetainedViewWithoutBodyHead) +/// With the decode cache disabled a prior read leaves nothing behind, so the deleted body surfaces +/// as FILE_DOESNT_EXIST in every mode: the miss path is the same fail-closed path a cold reader takes. +TEST(CASPartFolderAccess, DeletedBodyFailsClosedInEveryModeWhenDecodeCacheDisabled) { auto backend = std::make_shared(); - auto store = openPoolForTest(backend); + auto store = DB::Cas::Pool::open(backend, + DB::Cas::PoolConfig{.pool_prefix = "p", .server_root_id = "test", .manifest_decode_cache_bytes = 0}); const Cas::Layout layout("p"); const Cas::RootNamespace ns{"srv/t1"}; - const auto id = publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); - - auto params = cacheOn(); - params.validate = {Cas::PartFolderValidate::Mode::Never, 0}; - Cas::CachedPartFolderAccess access(store, params); + Cas::CachedPartFolderAccess access(store); /// retention off: every mode reaches the reader + const auto id = publishPart(store, ns, "part_1", {inlineEntry("f", "x")}); const Cas::PartRefKey key{ns, "part_1"}; - /// Prime the retained view (pays the HEAD once). ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); - /// Body vanishes (a protocol violation the net would normally catch)... deleteManifestBody(*backend, layout, id); - const auto skips_before = ProfileEvents::global_counters[ProfileEvents::CASPartFolderValidateSkipped].load(); - /// ...but `never` serves the retained view, no HEAD, no throw. - EXPECT_NO_THROW(access.getView(key, Cas::Freshness::ForceFresh)); - EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASPartFolderValidateSkipped].load() - skips_before, 1); -} - -TEST(CASPartFolderAccess, ValidateAlwaysStillHeadsEveryForceFresh) -{ - auto backend = std::make_shared(); - auto store = openPoolForTest(backend); - const Cas::Layout layout("p"); - const Cas::RootNamespace ns{"srv/t1"}; - const auto id = publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); - - Cas::CachedPartFolderAccess access(store, cacheOn()); /// default = Always - const Cas::PartRefKey key{ns, "part_1"}; - ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); - deleteManifestBody(*backend, layout, id); - /// `always` re-proves the body every ForceFresh — the deleted body surfaces as FILE_DOESNT_EXIST. - expectThrowsCode(ErrorCodes::FILE_DOESNT_EXIST, - [&] { access.getView(key, Cas::Freshness::ForceFresh); }); -} - -TEST(CASPartFolderAccess, ValidateAgeSkipsWithinWindowThenHeadsAfter) -{ - auto backend = std::make_shared(); - auto store = openPoolForTest(backend); - const Cas::Layout layout("p"); - const Cas::RootNamespace ns{"srv/t1"}; - const auto id = publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); - - auto params = cacheOn(); - params.validate = {Cas::PartFolderValidate::Mode::Age, /*age_seconds=*/5}; - /// An injected clock (spec §3 TDD requirement): the SAME function stamps the retained view's - /// validated_at_ms (buildView) and drives the age-window comparison (getView), so the test controls - /// both sides of the comparison deterministically -- no real sleep. - std::atomic fake_now_ms{1'000'000}; - Cas::CachedPartFolderAccess access(store, params, [&] { return fake_now_ms.load(); }); - const Cas::PartRefKey key{ns, "part_1"}; - const String manifest_key = layout.manifestKey(id); - - /// Prime the retained view (pays the HEAD once) at fake_now_ms. - ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); - const uint64_t heads_after_prime = backend->headCount(manifest_key); - - /// +2s: still inside the 5s window — served from the retained view, no new HEAD. - fake_now_ms += 2000; - ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); - EXPECT_EQ(backend->headCount(manifest_key), heads_after_prime); - - /// +6s from the ORIGINAL stamp (past the 5s window): re-proves the body via a fresh HEAD. - fake_now_ms += 4000; - ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); - EXPECT_GT(backend->headCount(manifest_key), heads_after_prime); -} - -/// ==== §3: `parsePartFolderValidate` config parsing, standalone (mirrors CASS3Staging's -/// parseStagingBackend coverage) -- review finding: std::stoull silently accepted a leading '-' -/// (unsigned wraparound), so a malformed `age -5` never hit the parser's own fail-closed throw. -/// These pin the fixed `std::from_chars`-based parsing directly, with no disk/store needed. ==== - -TEST(CASPartFolderValidateParse, DefaultConfigParsesToAlways) -{ - /// No `part_folder_validate` key at all -- the byte-for-byte-pre-§3-behavior default. - auto config = configWithDiskSection("/tmp/whatever"); - const auto v = ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); - EXPECT_EQ(v.mode, Cas::PartFolderValidate::Mode::Always); -} - -TEST(CASPartFolderValidateParse, ParsesAlways) -{ - auto config = configWithDiskSection("always"); - const auto v = ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); - EXPECT_EQ(v.mode, Cas::PartFolderValidate::Mode::Always); -} - -TEST(CASPartFolderValidateParse, ParsesNever) -{ - auto config = configWithDiskSection("never"); - const auto v = ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); - EXPECT_EQ(v.mode, Cas::PartFolderValidate::Mode::Never); -} - -TEST(CASPartFolderValidateParse, ParsesPositiveAge) -{ - auto config = configWithDiskSection("age 5"); - const auto v = ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); - EXPECT_EQ(v.mode, Cas::PartFolderValidate::Mode::Age); - EXPECT_EQ(v.age_seconds, 5u); -} - -TEST(CASPartFolderValidateParse, AcceptsAgeZeroAsADegenerateButValidWindow) -{ - /// `age 0` is accepted, not rejected: it is a well-formed (if degenerate -- effectively an - /// almost-always-expired window) configuration, not malformed input. Only genuinely malformed - /// suffixes (negative, non-digit, empty, trailing garbage) fail closed below. - auto config = configWithDiskSection("age 0"); - const auto v = ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); - EXPECT_EQ(v.mode, Cas::PartFolderValidate::Mode::Age); - EXPECT_EQ(v.age_seconds, 0u); -} - -TEST(CASPartFolderValidateParse, NegativeAgeThrows) -{ - /// The bug this regression-guards: std::stoull("-5") used to return 18446744073709551611 - /// (unsigned wraparound) instead of rejecting the leading '-'. - auto config = configWithDiskSection("age -5"); - expectThrowsCode(ErrorCodes::BAD_ARGUMENTS, - [&] { ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); }); -} - -TEST(CASPartFolderValidateParse, NonDigitAgeThrows) -{ - auto config = configWithDiskSection("age abc"); - expectThrowsCode(ErrorCodes::BAD_ARGUMENTS, - [&] { ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); }); -} - -TEST(CASPartFolderValidateParse, TrailingGarbageAfterAgeThrows) -{ - auto config = configWithDiskSection("age 5abc"); - expectThrowsCode(ErrorCodes::BAD_ARGUMENTS, - [&] { ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); }); -} - -TEST(CASPartFolderValidateParse, EmptyAgeSuffixThrows) -{ - auto config = configWithDiskSection("age "); - expectThrowsCode(ErrorCodes::BAD_ARGUMENTS, - [&] { ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); }); -} + backend->resetCounts(); -TEST(CASPartFolderValidateParse, UnknownValueThrows) -{ - /// Fail-closed: an unrecognized value must NEVER silently become `never`/`always`. - auto config = configWithDiskSection("sometimes"); - expectThrowsCode(ErrorCodes::BAD_ARGUMENTS, - [&] { ContentAddressedMetadataStorage::parsePartFolderValidate(*config, "disk"); }); + for (auto freshness : {Cas::Freshness::CachedForLoad, + Cas::Freshness::ForceFresh, + Cas::Freshness::StrictValidate}) + expectThrowsCode(ErrorCodes::FILE_DOESNT_EXIST, [&] { access.getView(key, freshness); }); + EXPECT_EQ(backend->headCount(layout.manifestKey(id)), 0u); + EXPECT_EQ(backend->getCount(layout.manifestKey(id)), 3u); /// one GET per attempt, nothing cached } TEST(CASPartFolderAccess, AbsenceIsNeverRetained) @@ -985,7 +856,7 @@ TEST(CASPartFolderAccess, GetViewEmitsRefResolveOnlyOnRealResolveWork) std::vector seen; store->setEventSink([&](const Cas::CasEvent & e) { seen.push_back(e); }); - Cas::CachedPartFolderAccess access(store, cacheOn()); /// retention on, validate == Always (default) + Cas::CachedPartFolderAccess access(store, cacheOn()); /// retention on const auto refResolveCount = [&] { @@ -1003,8 +874,7 @@ TEST(CASPartFolderAccess, GetViewEmitsRefResolveOnlyOnRealResolveWork) ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); EXPECT_EQ(refResolveCount(), 1) << "a warm view-cache hit must not add a RefResolve row"; - /// ForceFresh always re-proves the manifest body under the default Always validation policy, so - /// this is real resolve work again -> +1. + /// ForceFresh always bypasses the retained view, so this is real resolve work again -> +1. ASSERT_NE(access.getView(key, Cas::Freshness::ForceFresh), nullptr); EXPECT_EQ(refResolveCount(), 2); @@ -1021,7 +891,7 @@ TEST(CASPartFolderAccess, OversizedViewServedNotRetained) Cas::CachedPartFolderAccess access(store, Cas::CachedPartFolderAccess::CacheParams{ .cache_bytes = 64ULL << 20, .max_entries = 10000, .max_entry_bytes = 1, - .explain_enabled = true, .validate = {}}); + .explain_enabled = true}); const auto id = publishPart(store, ns, "part_1", {inlineEntry("f", "x")}); const Cas::PartRefKey key{ns, "part_1"}; const String manifest_key = layout.manifestKey(id); @@ -1032,10 +902,17 @@ TEST(CASPartFolderAccess, OversizedViewServedNotRetained) EXPECT_EQ(access.explain(key).last_decision, Cas::CachedPartFolderAccess::LastDecision::OversizedBypass); - const uint64_t head_before = backend->headCount(manifest_key); + backend->resetCounts(); auto view2 = access.getView(key, Cas::Freshness::CachedForLoad); ASSERT_NE(view2, nullptr); - EXPECT_GT(backend->headCount(manifest_key), head_before); /// not retained: re-HEADs every call + /// Not retained: every call rebuilds the view (a new view object over the SAME shared decode) and + /// records the bypass again; the rebuild costs no manifest request because the decode is cached. + EXPECT_NE(view1.get(), view2.get()); + EXPECT_EQ(view1->manifest().get(), view2->manifest().get()); + EXPECT_EQ(access.explain(key).last_decision, + Cas::CachedPartFolderAccess::LastDecision::OversizedBypass); + EXPECT_EQ(backend->getCount(manifest_key), 0u); + EXPECT_EQ(backend->headCount(manifest_key), 0u); EXPECT_FALSE(access.explain(key).retained); } @@ -1056,9 +933,10 @@ TEST(CASPartFolderAccess, DisabledModeKeepsBaseline) for (int i = 0; i < n; ++i) ASSERT_NE(access.getView(key, Cas::Freshness::CachedForLoad), nullptr); - /// Exactly the Phase-3 baseline: bytes=0 restores the no-retention call graph byte-for-byte. + /// bytes=0 restores the no-retention call graph: one body GET, then the decode cache serves every + /// rebuild with no manifest request. EXPECT_EQ(backend->getCount(manifest_key), 1u); - EXPECT_EQ(backend->headCount(manifest_key), static_cast(n)); + EXPECT_EQ(backend->headCount(manifest_key), 0u); EXPECT_FALSE(access.explain(key).retained); } diff --git a/src/Disks/tests/gtest_cas_part_folder_view.cpp b/src/Disks/tests/gtest_cas_part_folder_view.cpp index be9b9e91de91..5bd2ab8b7c43 100644 --- a/src/Disks/tests/gtest_cas_part_folder_view.cpp +++ b/src/Disks/tests/gtest_cas_part_folder_view.cpp @@ -47,8 +47,7 @@ std::shared_ptr makeView() return std::make_shared( Cas::PartRefKey{Cas::RootNamespace{"srv/t"}, "part_1"}, Cas::ManifestId{Cas::RootNamespace{"srv/t"}, Cas::ManifestRef{1, 2, 3}}, - /*manifest_size=*/1000, manifest, - /*validated_at_ms=*/42); + /*manifest_size=*/1000, manifest); } std::vector sorted(std::vector v) { std::sort(v.begin(), v.end()); return v; } diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 8e07048439f8..7235765a04a7 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -807,11 +808,11 @@ TEST(CASPool, LookupAndListOverManifestEntries) EXPECT_EQ(all[3].path, "p.proj/data.bin"); } -/// The Phase 1c manifest decode cache is keyed by (ManifestId, Token). Resolve+read the same ref twice: -/// the second readManifest must be served from the cache (no second GET of the body). A fresh publish -/// under a DIFFERENT ref name mints a NEW ManifestId (and a new shard token), so the cache misses and -/// the body is fetched again. A CountingBackend asserts the body GET count. -TEST(CASPool, ManifestCacheIsKeyedByIdAndToken) +/// The manifest decode cache is keyed by ManifestId alone: an id is minted once and its body is +/// written once, so one id names one content forever. Resolve+read the same ref twice: the second +/// readManifest is served from the cache with NO request at all. A fresh publish under a DIFFERENT +/// ref name mints a NEW ManifestId, so the cache misses and the body is fetched once. +TEST(CASPool, ManifestCacheIsKeyedById) { auto b = std::make_shared(); auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); @@ -820,8 +821,9 @@ TEST(CASPool, ManifestCacheIsKeyedByIdAndToken) const ManifestId id1 = publishPart(s, ns.string(), "part_1", "payload-1"); const String key1 = layout.manifestKey(id1); + b->resetCounts(); - /// First read: a body GET populates the (id1, token) cache entry. + /// First read: a body GET populates the id1 cache entry. { auto r = s->resolveRef(ns, "part_1"); ASSERT_TRUE(r.has_value()); @@ -831,7 +833,7 @@ TEST(CASPool, ManifestCacheIsKeyedByIdAndToken) const uint64_t gets_after_first = b->getCount(key1); ASSERT_GE(gets_after_first, 1u); /// the first read DID fetch the body - /// Second read of the SAME id: the (id, token) cache must serve it — NO additional body GET. + /// Second read of the SAME id: the id-keyed cache must serve it — NO additional body GET. { auto r = s->resolveRef(ns, "part_1"); ASSERT_TRUE(r.has_value()); @@ -840,7 +842,8 @@ TEST(CASPool, ManifestCacheIsKeyedByIdAndToken) ASSERT_EQ(m.entries.size(), 1u); } EXPECT_EQ(b->getCount(key1), gets_after_first) - << "second readManifest re-GET the body for the same (ManifestId, Token) — cache miss"; + << "second readManifest re-GET the body for the same ManifestId — cache miss"; + EXPECT_EQ(b->headCount(key1), 0u) << "keyed by id alone: no HEAD on a miss or a hit"; /// A fresh publish under a DIFFERENT ref name mints a NEW ManifestId: the cache (keyed by id) misses. /// (Promoting a different manifest over the SAME committed ref is a distinct promote-over-committed @@ -856,6 +859,7 @@ TEST(CASPool, ManifestCacheIsKeyedByIdAndToken) ASSERT_EQ(m2.entries.size(), 1u); EXPECT_GE(b->getCount(key2), 1u) /// the new id's body WAS fetched (cache miss) << "fresh publish (new ManifestId) should miss the id-keyed manifest cache"; + EXPECT_EQ(b->headCount(key2), 0u); } /// Phase 5 (part-folder cache spec): manifest_cache is now a byte-weighted CacheBase LRU instead of a @@ -2471,11 +2475,153 @@ TEST(CASPool, ReadManifestSharedReturnsSharedDecodeWithoutCopy) auto m2 = store->readManifestShared(resolved->manifest_id); EXPECT_EQ(m1.get(), m2.get()); /// the SAME shared decode, no copy EXPECT_EQ(backend->getCount(manifest_key), 1u); /// one body GET - EXPECT_EQ(backend->headCount(manifest_key), 2u); /// mandatory HEAD per call (unchanged) + EXPECT_EQ(backend->headCount(manifest_key), 0u); /// keyed by id: no HEAD on a miss or a hit ASSERT_EQ(m1->entries.size(), 1u); EXPECT_EQ(m1->entries[0].path, "data.bin"); } +/// A miss whose object is absent is the one dangling-reference case the reader still detects +/// itself: exactly one GET, no HEAD, one `ReadMissing` event, FILE_DOESNT_EXIST. +TEST(CASPool, ReadManifestAbsentBodyEmitsReadMissingWithOneGetAndNoHead) +{ + auto b = std::make_shared(); + auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + Layout layout("p"); + const RootNamespace ns{"srv1/tbl"}; + const ManifestId id{.root_namespace = ns, .ref = manifestRefFor("absent-body-event")}; + const String key = layout.manifestKey(id); + + std::vector events; + s->setEventSink([&](CasEvent e) { events.push_back(std::move(e)); }); + b->resetCounts(); + expectThrowsCode(DB::ErrorCodes::FILE_DOESNT_EXIST, [&] { s->readManifest(id); }); + s->setEventSink(nullptr); + + EXPECT_EQ(b->getCount(key), 1u); + EXPECT_EQ(b->headCount(key), 0u); + size_t read_missing = 0; + for (const auto & e : events) + { + if (e.type != CasEventType::ReadMissing) + continue; + ++read_missing; + EXPECT_EQ(e.object_kind, CasEventObjectKind::Manifest); + EXPECT_EQ(e.detail.at("code"), "FILE_DOESNT_EXIST"); + EXPECT_EQ(e.detail.at("site"), "readManifest"); + } + EXPECT_EQ(read_missing, 1u); +} + +/// A reader holding a decode for a manifest the collector has since removed sees a snapshot-consistent +/// manifest: the second read is the same shared decode with no request, `locate` is pure, and the +/// missing blob is observed only when its key is read. Nothing here is a fallback: the absence is +/// surfaced by the blob read, never masked by the cache. +TEST(CASPool, StaleSnapshotServesCachedManifestAndBlobAbsenceSurfacesOnRead) +{ + auto b = std::make_shared(); + auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + Layout layout("p"); + const RootNamespace ns{"srv1/tbl"}; + + const ManifestId id = publishPart(s, ns.string(), "part_1", "payload-1"); + auto r = s->resolveRef(ns, "part_1"); + ASSERT_TRUE(r.has_value()); + auto m1 = s->readManifestShared(r->manifest_id); + const String manifest_key = layout.manifestKey(id); + const String blob_key = layout.blobKey(idOf("payload-1")); + + /// What GC does after the owner is removed and the decrement is adopted: exact-token deletes of + /// the body and of the now-unreferenced blob. + { + const HeadResult h = b->head(manifest_key); + ASSERT_TRUE(h.exists); + b->deleteExact(manifest_key, h.token); + } + { + const HeadResult h = b->head(blob_key); + ASSERT_TRUE(h.exists); + b->deleteExact(blob_key, h.token); + } + b->resetCounts(); + + auto m2 = s->readManifestShared(r->manifest_id); + EXPECT_EQ(m1.get(), m2.get()); + EXPECT_EQ(b->getCount(manifest_key), 0u); + EXPECT_EQ(b->headCount(manifest_key), 0u); + + ASSERT_EQ(m2->entries.size(), 1u); + const BlobLocation location = s->locate(m2->entries[0]); + EXPECT_EQ(location.key, blob_key); + EXPECT_EQ(b->getCount(blob_key), 0u); /// locate is pure: no I/O until the read + EXPECT_FALSE(b->get(location.key).has_value()); /// the read observes the absence +} + +/// The scoped contract for mutation evidence, executable. A carry-forward from a committed source +/// (what createHardLink, republishRef, repointRef and the relink receiver do) adopts each entry as a +/// tokenless TrustedManifest dependency and promote issues no probe for it: the live source edge is +/// what keeps the blob alive, and under protocol-compliant GC the state "cached source decode, blob +/// gone" cannot be constructed. Out-of-band deletion of BOTH the cached source body and the blob is +/// outside that contract; the carry-forward then commits a ref to an absent blob and fsck's +/// reachable-but-absent scan is the detector. This test pins that documented outcome so a later +/// change that silently alters it is noticed. It is not a defect report. +TEST(CASPool, CachedSourceDecodeLetsAdoptionCommitAnAbsentBlobThatFsckReports) +{ + auto b = std::make_shared(); + auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + Layout layout("p"); + const RootNamespace ns{"srv1/tbl"}; + + const ManifestId src_id = publishPart(s, ns.string(), "part_src", "payload-src"); + auto src = s->resolveRef(ns, "part_src"); + ASSERT_TRUE(src.has_value()); + const auto src_manifest = s->readManifestShared(src->manifest_id); /// warms the decode cache + const String src_manifest_key = layout.manifestKey(src_id); + const String blob_key = layout.blobKey(idOf("payload-src")); + + /// Out of band: both objects gone, the committed source ref untouched. + for (const String & key : {src_manifest_key, blob_key}) + { + const HeadResult h = b->head(key); + ASSERT_TRUE(h.exists); + b->deleteExact(key, h.token); + } + b->resetCounts(); + + /// The carry-forward reaches its source through the reader, the way every production caller does, + /// and the cache answers: the same decode as before, with no request on the body just deleted. + /// Adopting from the `shared_ptr` held across the deletion would prove nothing about the cache -- + /// were the cache to stop retaining, this re-read would fetch and throw, and the rest of this + /// scenario would be unreachable in production for the same reason. + const auto cached_manifest = s->readManifestShared(src->manifest_id); + ASSERT_EQ(cached_manifest.get(), src_manifest.get()); + EXPECT_EQ(b->getCount(src_manifest_key), 0u); + EXPECT_EQ(b->headCount(src_manifest_key), 0u); + + /// The carry-forward, in the order prepareEntries runs it for a committed source: adopt, stage, + /// precommit, promote. No blob body is written. + PartWriteInfo info; + info.intended_ref = ns.string() + "/part_dst"; + info.intended_namespace = ns; + auto build = s->beginPartWrite(info); + ASSERT_EQ(src_manifest->entries.size(), 1u); + build->adoptEvidence(cached_manifest->entries[0]); + const ManifestId dst_id = build->stageManifest({cached_manifest->entries[0]}); + build->precommitAdd(ns, "part_dst", dst_id); + EXPECT_NO_THROW(build->promote(ns, "part_dst", build->buildId(), dst_id)); + EXPECT_EQ(b->headCount(blob_key), 0u); /// a TrustedManifest leaf is not probed, by design + EXPECT_EQ(b->getCount(blob_key), 0u); + + /// The documented outcome: a committed ref names an absent blob, and fsck reports it. + ASSERT_TRUE(s->resolveRef(ns, "part_dst").has_value()); + const FsckReport rep = runFsck(*s, /*detail=*/true); + EXPECT_GE(rep.dangling, 1u); + bool blob_reported = false; + for (const FsckObject & o : rep.objects) + if (o.key == blob_key && o.cls == FsckClass::Dangling) + blob_reported = true; + EXPECT_TRUE(blob_reported) << "fsck must report the adopted-but-absent blob " << blob_key; +} + #if defined(DEBUG_OR_SANITIZER_BUILD) #define EXPECT_RUNTIME_STATE_REJECTION(statement) EXPECT_DEATH({ statement; }, "CAS mount runtime") #else diff --git a/src/Disks/tests/gtest_cas_repoint.cpp b/src/Disks/tests/gtest_cas_repoint.cpp index 9044a8eda895..fab72a86c59e 100644 --- a/src/Disks/tests/gtest_cas_repoint.cpp +++ b/src/Disks/tests/gtest_cas_repoint.cpp @@ -83,7 +83,7 @@ TEST(CASRepoint, AddFileRepoints) const RootNamespace ns{"srv/t1"}; DB::Cas::CachedPartFolderAccess access( store, {.cache_bytes = 64ULL << 20, .max_entries = 10000, .max_entry_bytes = 16ULL << 20, - .explain_enabled = false, .validate = {}}); + .explain_enabled = false}); const auto id_before = publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); const DB::Cas::PartRefKey key{ns, "part_1"}; /// Warm the retained view so the erase-on-success cache discipline is actually exercised. diff --git a/src/Disks/tests/gtest_cas_settings.cpp b/src/Disks/tests/gtest_cas_settings.cpp index 46d18480e55b..1e43c222d71a 100644 --- a/src/Disks/tests/gtest_cas_settings.cpp +++ b/src/Disks/tests/gtest_cas_settings.cpp @@ -130,6 +130,29 @@ TEST(CASContentAddressedSettings, RemovedCacheSettingsAreRejected) } } +/// `cas_part_folder_validate` paced a manifest `HEAD` that no longer exists. A config still asking +/// for it must fail the disk open, not be quietly accepted and ignored. +TEST(CASContentAddressedSettings, RetiredPartFolderValidateIsRejected) +{ + for (const std::string & value : {"always", "never", "age 5"}) + { + SCOPED_TRACE(value); + auto cfg = makeConfig( + "srv1" + "" + value + ""); + ContentAddressedSettings settings; + try + { + settings.loadFromConfig(*cfg, "disk", "/data", "/data/scratch", identity_macros); + FAIL() << "expected the retired setting cas_part_folder_validate to be rejected as unknown"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_SETTING); + } + } +} + TEST(CASContentAddressedSettings, UnknownKeyRejected) { expectLoadFailureWithExactMessage( @@ -164,10 +187,6 @@ TEST(CASContentAddressedSettings, InvalidEnumDiagnosticsNameExternalConfigKeys) "srv1remote", ErrorCodes::BAD_ARGUMENTS, "Unknown cas_staging_backend value 'remote' (expected 'local' or 's3')"); - expectLoadFailureWithExactMessage( - "srv1sometimes", - ErrorCodes::BAD_ARGUMENTS, - "Unknown cas_part_folder_validate value 'sometimes' (expected 'always', 'never', or 'age ')"); } /// The point of this test is that none of these names appears anywhere in CAS code. It is not an From adc0fa7007a3b1db5518327711312f68c93986fe Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:02:33 +0200 Subject: [PATCH 14/81] cas: introduce the CasRequests/CasOperation request engine (core) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six review rounds across two earlier designs found nine classes of defect in how CAS handled its conditional-write tokens, eight of them tracing to one root: `Token` was `struct { String value; TokenType type; }`, anyone could construct one from anything, and the backend accepted whatever it was handed. Concretely this let an empty value pass as a token (and on S3/Azure an empty `If-Match` is *omitted*, so a fenced write silently becomes an unconditional overwrite), let a write commit against a token that was really the result of a later, unrelated `HEAD`, and let `TokenMismatch` — documented as remote evidence that another incarnation is current — be returned for what was actually a local refusal, with GC acting on it and mislabelling live blobs `Replaced`. Every earlier revision patched one symptom at one call site; the next review round found the same root through a different one. A second, independent waste rode along: `Backend::get` always issued both a `HEAD` and a `GET`, though a `GET` already returns everything a `HEAD` does plus the body — doubling the request cost of every control-object read. This introduces the replacement, starting with its core (the migration of every CAS subsystem onto it is the next commit): `Backend` becomes a string-in/string-out transport callable only through a `TransportAccess` key; `Incarnation` replaces the free-form `Token` as a type that can only be minted by the backend from an actual store response; `CasRequests` owns a backend and a `Fence`, and `admit()`/`resume(generation)` hand out a `CasOperation` carrying the admitted generation and an optional liveness predicate. Every verb on that operation (`read`, `head`, `list`, `remove`, `publish`, `create`, `replace`, `readModifyWrite`, ...) takes a `Retry` policy; the engine re-checks admission before every attempt, before every sleep, and once more after a proven commit, settles every conflict and ambiguity by one exact read, and reports one of `Committed | Declined | Conflict | Refused | GaveUp` — never an exception for an ordinary lost race. An upstream slice under `src/IO` and `S3ObjectStorage` adds a `SingleAttempt` request mode so a marked `GET` answers with the same incarnation identity a `HEAD` does (closing the two-request cost) and a reissue that gets back a different ETag is treated as body drift, not silently accepted. The old controller stays in place during the migration; the next commits delete it. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- src/Common/ErrorCodes.cpp | 4 +- src/Common/ProfileEvents.cpp | 6 + .../ContentAddressed/Backend/CasBackend.h | 372 ++++- .../ContentAddressed/Backend/CasFence.h | 35 + .../Backend/CasInMemoryBackend.cpp | 388 +++-- .../Backend/CasInMemoryBackend.h | 202 ++- .../Backend/CasIncarnation.cpp | 34 + .../ContentAddressed/Backend/CasIncarnation.h | 88 + .../Backend/CasInstrumentedBackend.cpp | 10 + .../Backend/CasInstrumentedBackend.h | 129 +- .../Backend/CasObjectStorageBackend.cpp | 659 ++++---- .../Backend/CasObjectStorageBackend.h | 184 +- .../ContentAddressed/Backend/CasRequests.cpp | 870 ++++++++++ .../ContentAddressed/Backend/CasRequests.h | 336 ++++ .../ContentAddressed/Backend/CasRetry.cpp | 30 + .../ContentAddressed/Backend/CasRetry.h | 52 + .../Backend/CasThrottlingBackend.h | 221 +++ .../Backend/CasTransportAccess.h | 21 + .../ContentAddressed/Backend/CasWriteResult.h | 114 ++ .../ContentAddressedMetadataStorage.cpp | 13 +- .../ContentAddressed/Formats/CasFormat.cpp | 15 + .../ContentAddressed/Formats/CasFormat.h | 7 + .../ObjectStorages/IObjectStorage.cpp | 29 + .../ObjectStorages/IObjectStorage.h | 20 + .../ObjectStorages/S3/S3ObjectStorage.cpp | 194 ++- .../ObjectStorages/S3/S3ObjectStorage.h | 52 +- src/Disks/tests/cas_test_helpers.h | 118 +- src/Disks/tests/gtest_cas_backend.cpp | 337 ++-- .../tests/gtest_cas_backend_contract.cpp | 7 +- .../tests/gtest_cas_backend_generation.cpp | 90 +- .../tests/gtest_cas_bootstrap_ordering.cpp | 2 + src/Disks/tests/gtest_cas_decommission.cpp | 15 + .../gtest_cas_decommission_catalog_duties.cpp | 4 + .../tests/gtest_cas_fence_generation.cpp | 6 + src/Disks/tests/gtest_cas_forget.cpp | 3 + src/Disks/tests/gtest_cas_fsck.cpp | 6 + src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 2 + src/Disks/tests/gtest_cas_gc_log.cpp | 7 + .../tests/gtest_cas_holey_list_detector.cpp | 2 + .../tests/gtest_cas_lifecycle_condition.cpp | 5 +- src/Disks/tests/gtest_cas_mount.cpp | 24 +- .../tests/gtest_cas_orphan_manifest_sweep.cpp | 2 + src/Disks/tests/gtest_cas_part_write.cpp | 119 +- src/Disks/tests/gtest_cas_pool.cpp | 92 +- src/Disks/tests/gtest_cas_probe.cpp | 35 +- .../tests/gtest_cas_protocol_scenarios.cpp | 4 +- .../tests/gtest_cas_recovery_streaming.cpp | 6 + src/Disks/tests/gtest_cas_request_control.cpp | 12 +- src/Disks/tests/gtest_cas_requests.cpp | 1481 +++++++++++++++++ .../tests/gtest_cas_retirement_sweep.cpp | 4 + src/Disks/tests/gtest_cas_s3_staging.cpp | 60 +- src/Disks/tests/gtest_cas_sentinel_probe.cpp | 91 +- src/Disks/tests/gtest_cas_upload_detached.cpp | 2 + src/Disks/tests/gtest_cas_upstream_slice.cpp | 743 +++++++++ src/IO/ObjectStorageRequestMode.h | 18 + src/IO/ReadBufferFromS3.cpp | 11 + src/IO/ReadBufferFromS3.h | 7 + src/IO/ReadSettings.h | 10 + src/IO/WriteSettings.h | 15 +- 59 files changed, 6372 insertions(+), 1053 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasFence.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h create mode 100644 src/Disks/tests/gtest_cas_requests.cpp create mode 100644 src/Disks/tests/gtest_cas_upstream_slice.cpp create mode 100644 src/IO/ObjectStorageRequestMode.h diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index f9ecd77e997c..299f3bf4d391 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -679,6 +679,8 @@ M(1009, PENDING_MUTATIONS_NOT_ALLOWED) \ M(1010, EXPORT_PARTITION_ALREADY_EXPORTED) \ M(1011, PARTITION_EXPORT_FAILED) \ + M(1012, CAS_WRITE_UNATTRIBUTED) \ + M(1013, CAS_DELETE_MARKER) \ /* See END */ #ifdef APPLY_FOR_EXTERNAL_ERROR_CODES @@ -695,7 +697,7 @@ namespace ErrorCodes APPLY_FOR_ERROR_CODES(M) #undef M - constexpr ErrorCode END = 1011; + constexpr ErrorCode END = 1013; ErrorPairHolder values[END + 1]{}; struct ErrorCodesNames diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index b0d3e267669b..ad3d22f1925b 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -932,6 +932,12 @@ The server successfully detected this situation and will download merged part fr M(CASConditionalWriteDefiniteFailure, "Number of CAS conditional writes rejected with certainty before applying. A non-zero value indicates invalid requests, oversized entities, or access denial.", ValueType::Number) \ M(CASConditionalWriteUnresolved, "Number of CAS conditional writes with an unknown outcome after conflict, timeout, connection loss, or server error. A non-zero value indicates backend instability or state requiring resolution.", ValueType::Number) \ M(CASConditionalWriteFenceLostPostWrite, "Number of CAS writes that succeeded but lost the final mount-fence check. A non-zero value indicates late responses after the mount lifecycle changed.", ValueType::Number) \ + M(CASRequestAttempt, "Number of physical requests the CAS request contract started. Each one was admitted by the mount fence and reserved against the call's deadline before it was sent.", ValueType::Number) \ + M(CASRequestReissue, "Number of CAS requests re-sent after a jittered backoff sleep. Growth means the object store is throttling, failing, or contended.", ValueType::Number) \ + M(CASRequestResolveRead, "Number of requests the CAS request contract made to settle a refused precondition or an ambiguous write: a body read, or a HEAD where the caller needs only presence. Every conflict and every ambiguity costs one.", ValueType::Number) \ + M(CASRequestGaveUp, "Number of CAS writes that ended without a proven outcome, at a deadline, on a lost mount fence, or unresolved. A non-zero value means callers are being asked to retry later.", ValueType::Number) \ + M(CASRequestRefused, "Number of CAS writes the store itself refused, proving they never applied: a malformed request, an entity too large, or an access or credential denial that no credential refresh was performed for, either because the disk has no refresh mechanism or because this write had already spent its one refresh.", ValueType::Number) \ + M(CASRequestFenceLostPostWrite, "Number of CAS writes that were proven durable but lost the mount fence before the call could claim them. A non-zero value indicates late responses after the mount lifecycle changed.", ValueType::Number) \ M(CASMountRenewalAttempts, "Number of physical conditional renewal PUTs sent for CAS mount leases. This counts transport attempts, not logical renewals.", ValueType::Number) \ M(CASMountRenewalRetries, "Number of physical conditional renewal PUTs sent after the first attempt of one logical CAS mount-lease renewal.", ValueType::Number) \ M(CASMountRenewalResolved, "Number of CAS mount-lease renewals whose committed outcome was proved by an exact resolving GET.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index 579c33a1c355..758faa33daa5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -1,9 +1,16 @@ #pragma once +#include +#include #include #include #include +#include +#include #include #include +#include +#include +#include #include #include #include @@ -13,16 +20,25 @@ #include #include +namespace DB::ErrorCodes +{ + extern const int CAS_WRITE_UNATTRIBUTED; + extern const int CORRUPTED_DATA; + extern const int LOGICAL_ERROR; + extern const int NOT_IMPLEMENTED; +} + namespace DB::Cas { -/// User metadata carried alongside an object (S3 x-amz-meta-*). The CA store uses exactly one entry, -/// "cas_owner" = "::" — the owner triple the GC watermark reads. +/// User metadata carried alongside an object (S3 x-amz-meta-*). RETIRED: the transport neither +/// writes nor reads attributes, and this alias survives only in the legacy signatures below. using ObjectMeta = std::map; /// A byte window requested from an object. An absent length means that the window extends to EOF. -/// Backends use the same semantics for materialized and forward-only reads: the offset is exact, -/// while a backend may expose an advisory end when its underlying read buffer cannot enforce one. +/// RETIRED for materialized reads: `get` accepts only a whole-object window. It survives on +/// `getStream`, which is not a forwarder. The offset is exact, while a backend may expose an advisory +/// end when its underlying read buffer cannot enforce one. struct Range { uint64_t offset = 0; @@ -241,77 +257,108 @@ inline BlobPayloadCopyResult copyBlobPayloadBounded(ReadBuffer & from, WriteBuff class Backend { public: + Backend(); virtual ~Backend() = default; - /// Reads the selected bytes and their token, or returns nullopt when the key is absent. For a - /// mutable object, callers must use this materialized form so the body is fixed before parsing. - virtual std::optional get(const String & key, Range range) = 0; - std::optional get(const String & key) { return get(key, {}); } - - /// Forward-only stream over the object's `range` (default: whole object) for WRITE-ONCE objects - /// (runs, seals). The returned `stream` yields exactly the window's bytes and nothing is - /// materialized whole by the seam — the caller reads at its own pace. MUTABLE objects (root - /// shards, gc/state, mounts) MUST keep using `get`: their bytes can change under an open stream. - /// CAVEAT: the window END is advisory on storages where `setReadUntilPosition` is a hint - /// (LocalObjectStorage) — the stream may yield bytes past the window; consumers MUST bound their - /// own consumption (RunFileReader bounds to its data_end). The window START is always exact. - virtual std::optional getStream(const String & key, Range range) = 0; - std::optional getStream(const String & key) { return getStream(key, {}); } - - /// Returns the current incarnation's existence, size, token, and metadata without reading its - /// body. The result describes one point-in-time observation; a later operation must use the - /// returned token when it needs to protect against replacement. - virtual HeadResult head(const String & key) = 0; - - /// Creates `key` only when it is absent. `PreconditionFailed` leaves the existing object intact; - /// storage failures are reported as exceptions. On success, the returned token identifies the - /// newly created incarnation. - virtual PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) = 0; - PutResult putIfAbsent(const String & key, const String & bytes) { return putIfAbsent(key, bytes, {}); } - - /// Unconditionally publishes one complete blob body. The caller owns every lifecycle decision; - /// this method only executes the selected streaming or native-copy transport and returns after - /// the complete destination becomes visible. - virtual void publishBlob(const BlobPublishRequest & request) = 0; - - /// Replaces the current object only when its token equals `expected`. A mismatch leaves the - /// object unchanged and returns `PreconditionFailed`; the returned token is meaningful only on - /// `Done`. - virtual PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, - const ObjectMeta & meta) = 0; - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected) - { - return putOverwrite(key, bytes, expected, {}); - } - /// expected == nullopt => create-if-absent CAS (the first write of a root manifest). - /// A non-null expected token conditionally replaces that exact current incarnation. Conflicts - /// leave the object unchanged and are returned as an outcome rather than an exception. - virtual CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) = 0; - CasResult casPut(const String & key, const String & bytes, const std::optional & expected) + /// ---- The transport primitives ---- + /// + /// These are the ONLY methods that reach the store. Each takes a `TransportAccess`, which nothing + /// outside `CasRequests` can construct, so no caller can reach the store without the request + /// contract's retry, deadline and fence rules. They deal in the store's own strings: an + /// incarnation VALUE means no more here than "what the store answered", and every grammar, + /// key-binding and dialect check on it belongs to `CasRequests`. + + /// An object's bytes together with the value naming the incarnation they were read from. + struct Raw { String bytes; String value; }; + /// One object's size and incarnation value, without its body. + struct RawMeta { uint64_t size; String value; }; + /// One listed key; `value` is present only on a backend that surfaces per-key incarnations + /// through LIST -- see `supportsListTokens`. + struct RawListedKey { String key; uint64_t size; std::optional value; }; + struct RawListPage { std::vector keys; String next_cursor; }; + /// The store refused the write's precondition. Nothing was written; what the key holds now is + /// whatever a read finds. An expected outcome, never an error. + struct RawConflict {}; + /// `DeleteMarker` is a removal that did NOT reclaim: a versioned bucket archived a noncurrent + /// version instead. Distinct from `Removed` because reclaiming the storage is the point. + enum class RawRemoval : uint8_t { Removed, Gone, Mismatch, DeleteMarker }; + + /// Reads the whole object, or nullopt when the key is absent. + virtual std::optional read (const String & key, TransportAccess &) = 0; + /// One point-in-time observation of the current incarnation's size and value; nullopt when absent. + virtual std::optional head (const String & key, TransportAccess &) = 0; + /// One page of keys under `prefix`, resuming strictly after `cursor`; an empty `next_cursor` + /// marks the end of the enumeration. + virtual RawListPage list (const String & prefix, const String & cursor, size_t limit, TransportAccess &) = 0; + /// Removes ONLY the incarnation whose value equals `expected_value`; a mismatch must leave the + /// object untouched. + virtual RawRemoval remove(const String & key, const String & expected_value, TransportAccess &) = 0; + /// Creates the key (`expected_value == nullopt`) or replaces exactly the incarnation named by + /// `expected_value`. Returns the store's own value for what it just wrote, UNVALIDATED: a value + /// that fails the dialect grammar means the write may still have landed, which the caller settles + /// by reading the key back, and which this seam must never report as corruption. + virtual std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess &) = 0; + /// A forward-only read of a WRITE-ONCE object (runs, seals): nothing is materialized by the seam. + /// MUTABLE objects (root shards, gc/state, mounts) MUST use `read` -- their bytes may change + /// under an open stream. Null when the key is absent. + virtual std::unique_ptr stream(const String & key, TransportAccess &) = 0; + /// Executes one unconditional blob publication and returns once the complete destination is + /// visible. Transport only: it observes no destination state and produces no incarnation. + virtual void publish(const BlobPublishRequest & request, TransportAccess &) = 0; + + /// Authoritative, cache-bypassing probe of one key -- see `ProbeOutcome`. DEFAULT (used by every + /// backend without sharper raw-error evidence, e.g. `InMemoryBackend`): derived from `head`/`read` + /// alone, so it can only distinguish `Present` from `KeyAbsent`, and ANY exception from either + /// call is `Indeterminate` -- never promoted to `KeyAbsent`. A backend able to surface real + /// container/permission evidence (the S3-native and Local paths of `ObjectStorageBackend`) + /// overrides this to sharpen the classification. + virtual SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) { - return casPut(key, bytes, expected, {}); + try + { + const auto meta = head(key, access); + if (!meta) + return {ProbeOutcome::KeyAbsent, std::nullopt}; + auto raw = read(key, access); + /// Vanished between the two: still a clean, authoritative miss, not an error. + if (!raw) + return {ProbeOutcome::KeyAbsent, std::nullopt}; + return {ProbeOutcome::Present, std::move(raw->bytes)}; + } + catch (...) + { + return {ProbeOutcome::Indeterminate, std::nullopt}; + } } - /// Deletes only the current incarnation identified by `token`. A token mismatch must leave the - /// object untouched; the result distinguishes that case from an already absent key. - virtual DeleteOutcome deleteExact(const String & key, const Token & token) = 0; + /// The dialect this backend mints its incarnation values in. + virtual Dialect dialect() const = 0; + + /// Identifies this backend INSTANCE, so an incarnation observed elsewhere can be refused rather + /// than used as a precondition here. Assigned at construction and never reused in this process. + uint64_t backendId() const { return backend_id; } - /// Lists one page of keys under `prefix`, starting after `cursor` and returning at most `limit` - /// entries. `ListPage::next_cursor` is the only supported continuation state. - virtual ListPage list(const String & prefix, const String & cursor, size_t limit) = 0; + /// The budget for one HTTP attempt in milliseconds; 0 when this backend has no such notion. The + /// request contract reserves it before every attempt it starts. + virtual uint64_t attemptTimeoutMs() const { return 0; } + + /// Asks the storage to re-acquire credentials. TRUE when fresh ones were installed, so the + /// caller's reissue can sign with them; FALSE when this backend has no refresh mechanism, which + /// makes an expired-credential failure terminal for the caller's policy rather than retryable. + virtual bool refreshCredentials() { return false; } /// Capability fact about the LIST seam: TRUE iff this backend can surface a per-key incarnation - /// token through `list` (i.e. each `ListedKey` carries a token that uniquely identifies the + /// value through `list` (i.e. each `RawListedKey` carries a value that uniquely identifies the /// current incarnation of that key, matching what `head` would return). /// /// Why this matters: S3 ETags are content-derived and are returned in list responses; the - /// in-memory backend mints a monotonic token it can also surface through `list`. A backend that - /// cannot surface per-key tokens through `list` MUST return FALSE. + /// in-memory backend mints a monotonic value it can also surface through `list`. A backend that + /// cannot surface per-key values through `list` MUST return FALSE. /// - /// FALSE ⇒ GC `discover` must read every root-shard body to learn the current token (fail closed). - /// TRUE ⇒ `discover` may skip an unchanged root-shard body read when the listed token equals - /// the persisted folded token, saving a GET per unchanged shard. + /// FALSE ⇒ GC `discover` must read every root-shard body to learn the current incarnation (fail + /// closed). TRUE ⇒ `discover` may skip an unchanged root-shard body read when the listed value + /// equals the persisted folded one, saving a GET per unchanged shard. virtual bool supportsListTokens() const = 0; /// Pool-level preconditions beyond per-op conditional semantics — checked by the capability @@ -336,33 +383,196 @@ class Backend /// are not gated here — see ObjectStorageBackend's override for the one backend that is). virtual void checkConditionalWriteSingleAttemptSupport() {} - /// Authoritative, cache-bypassing probe of one key — see `ProbeOutcome`. DEFAULT (used by every - /// backend without sharper raw-error evidence, e.g. `InMemoryBackend`): derived from `head`/`get` - /// alone, so it can only distinguish `Present` from `KeyAbsent`, and ANY exception from either - /// call is `Indeterminate` — never promoted to `KeyAbsent`. A backend able to surface real - /// container/permission evidence (the S3-native and Local paths of `ObjectStorageBackend`) - /// overrides this to sharpen the classification. - virtual SentinelProbeResult probeSentinelRaw(const String & key) + /// ---- The legacy Token-typed surface ---- + /// + /// Every one of these obtains the migration key and calls the primitive above, so a fault + /// injection written against a PRIMITIVE intercepts a legacy caller too. They stay virtual while + /// the migration runs, so a test double that overrides one of THESE keeps working until the site + /// it instruments moves; the whole block, and `migrationAccess` with it, is deleted at the lock. + /// `Range` and `ObjectMeta` are already retired: a non-whole window is refused, and object + /// attributes are neither written nor returned. + virtual std::optional get(const String & key, Range range) { - try + if (!range.whole()) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "CAS backend: a ranged get is retired; read the object whole"); + auto access = migrationAccess(); + auto raw = read(key, access); + if (!raw) + return std::nullopt; + return GetResult{std::move(raw->bytes), legacyMintObserved(key, std::move(raw->value)), {}}; + } + std::optional get(const String & key) { return get(key, {}); } + + /// Forward-only stream over the object's `range` (default: whole object) for WRITE-ONCE objects + /// (runs, seals). Not a forwarder: `stream` returns no incarnation, and a forwarder would have to + /// fill `GetStreamResult::token` with a default-constructed `Token` that names nothing. Each + /// backend keeps its own implementation until the last caller moves onto `stream`. + /// CAVEAT: the window END is advisory on storages where `setReadUntilPosition` is a hint + /// (LocalObjectStorage) — the stream may yield bytes past the window; consumers MUST bound their + /// own consumption (RunFileReader bounds to its data_end). The window START is always exact. + virtual std::optional getStream(const String & key, Range range) = 0; + std::optional getStream(const String & key) { return getStream(key, {}); } + + virtual HeadResult head(const String & key) + { + auto access = migrationAccess(); + auto raw = head(key, access); + if (!raw) + return {}; + return HeadResult{true, raw->size, legacyMintObserved(key, std::move(raw->value)), {}}; + } + + virtual PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & /*meta*/) + { + auto access = migrationAccess(); + auto r = write(key, bytes, std::nullopt, access); + if (!r) + return PutResult{PutOutcome::PreconditionFailed, {}}; + return PutResult{PutOutcome::Done, legacyMintWritten(key, std::move(*r))}; + } + PutResult putIfAbsent(const String & key, const String & bytes) { return putIfAbsent(key, bytes, {}); } + + virtual PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, + const ObjectMeta & /*meta*/) + { + if (legacyTokenIsForeign(key, expected)) + return PutResult{PutOutcome::PreconditionFailed, {}}; + auto access = migrationAccess(); + auto r = write(key, bytes, expected.value, access); + if (!r) + return PutResult{PutOutcome::PreconditionFailed, {}}; + return PutResult{PutOutcome::Done, legacyMintWritten(key, std::move(*r))}; + } + PutResult putOverwrite(const String & key, const String & bytes, const Token & expected) + { + return putOverwrite(key, bytes, expected, {}); + } + + virtual CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & /*meta*/) + { + if (expected && legacyTokenIsForeign(key, *expected)) + return CasResult{CasOutcome::Conflict, {}}; + auto access = migrationAccess(); + auto r = write(key, bytes, expected ? std::optional(expected->value) : std::nullopt, access); + if (!r) + return CasResult{CasOutcome::Conflict, {}}; + return CasResult{CasOutcome::Committed, legacyMintWritten(key, std::move(*r))}; + } + CasResult casPut(const String & key, const String & bytes, const std::optional & expected) + { + return casPut(key, bytes, expected, {}); + } + + virtual DeleteOutcome deleteExact(const String & key, const Token & token) + { + if (legacyTokenIsForeign(key, token)) + return DeleteOutcome{DeleteOutcome::Kind::TokenMismatch, false}; + auto access = migrationAccess(); + switch (remove(key, token.value, access)) { - const HeadResult hr = head(key); - if (!hr.exists) - return {ProbeOutcome::KeyAbsent, std::nullopt}; - auto g = get(key); - /// Vanished between head and get: still a clean, authoritative miss, not an error. - if (!g) - return {ProbeOutcome::KeyAbsent, std::nullopt}; - return {ProbeOutcome::Present, std::move(g->bytes)}; + case RawRemoval::Removed: return {DeleteOutcome::Kind::Deleted, false}; + case RawRemoval::Gone: return {DeleteOutcome::Kind::NotFound, false}; + case RawRemoval::Mismatch: return {DeleteOutcome::Kind::TokenMismatch, false}; + case RawRemoval::DeleteMarker: return {DeleteOutcome::Kind::Deleted, true}; } - catch (...) + UNREACHABLE(); + } + + virtual ListPage list(const String & prefix, const String & cursor, size_t limit) + { + auto access = migrationAccess(); + auto raw = list(prefix, cursor, limit, access); + ListPage page; + page.next_cursor = std::move(raw.next_cursor); + page.keys.reserve(raw.keys.size()); + for (auto & k : raw.keys) { - return {ProbeOutcome::Indeterminate, std::nullopt}; + std::optional token; + if (k.value) + token = legacyMintObserved(k.key, std::move(*k.value)); + page.keys.push_back(ListedKey{std::move(k.key), k.size, std::move(token)}); } + return page; + } + + virtual void publishBlob(const BlobPublishRequest & request) + { + auto access = migrationAccess(); + publish(request, access); + } + + virtual SentinelProbeResult probeSentinelRaw(const String & key) + { + auto access = migrationAccess(); + return probeSentinelRaw(key, access); + } + +protected: + /// The migration key. Every legacy forwarder above obtains one; nothing else may, and the whole + /// mechanism is deleted with the forwarders at the lock. + static TransportAccess migrationAccess() { return TransportAccess{}; } + + /// ---- Where a raw response becomes a legacy `Token` ---- + /// + /// The primitives return the store's value as it arrived: judging it belongs to the caller that + /// can act on the judgement. A legacy caller cannot -- it puts the `Token` straight into its next + /// request -- so these two are where a legacy response is judged, and the ONLY places a legacy + /// `Token` is minted. A backend that overrides a legacy method for the migration window mints + /// through them too. + + /// An OBSERVED value (read, head, list) that is not an incarnation means the response fell + /// through unmapped: nothing was changed by it, and the caller must not act on it. + Token legacyMintObserved(const String & key, String value) + { + if (!isIncarnationValue(dialect(), value)) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS backend: the store answered for '{}' with a value '{}' that is not a valid incarnation", + key, value); + return Token{std::move(value), dialect()}; + } + + /// A value from a SUCCESSFUL write that is not an incarnation is the ambiguous case, not the + /// corrupt one: the write may well have landed, and only reading the key back can say. Hence + /// `CAS_WRITE_UNATTRIBUTED`, which names that duty, and never `CORRUPTED_DATA`, which a caller + /// may treat as a deterministic failure and stop. + Token legacyMintWritten(const String & key, String value) + { + if (!isIncarnationValue(dialect(), value)) + throw Exception(ErrorCodes::CAS_WRITE_UNATTRIBUTED, + "CAS backend: the store accepted a write of '{}' but answered with '{}', which is not an " + "incarnation; the write may have committed and must be resolved by reading back", + key, value); + return Token{std::move(value), dialect()}; } + /// The dialect half of the legacy token check: the primitives take a bare VALUE and cannot see + /// the dialect a `Token` declares, so a foreign-dialect token is answered here as an ordinary + /// non-match rather than being forwarded to a wire (or a value space) that was never designed to + /// discriminate it. A MALFORMED value is refused FIRST, under its own declared dialect, so a + /// token that is both malformed and foreign is still reported as the caller bug it is. + bool legacyTokenIsForeign(const String & key, const Token & token) + { + if (!isIncarnationValue(token.type, token.value)) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS backend: refusing a conditional mutation of '{}' with a malformed token '{}' (dialect {}): " + "an empty, wildcard or list token would turn the precondition into an unconditional write", + key, token.value, static_cast(token.type)); + return token.type != dialect(); + } + +private: + uint64_t backend_id; }; +inline Backend::Backend() +{ + /// Per instance, monotonic, never reused: an incarnation carries the id of the backend that + /// observed it, so it can be refused anywhere else. Starts at 1, leaving 0 naming no backend. + static std::atomic next_backend_id{1}; + backend_id = next_backend_id.fetch_add(1, std::memory_order_relaxed); +} + using BackendPtr = std::shared_ptr; /// Walk every key under `prefix` exactly once, resuming by the backend's explicit last-returned-key diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasFence.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasFence.h new file mode 100644 index 000000000000..dd3e23da718a --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasFence.h @@ -0,0 +1,35 @@ +#pragma once +#include +#include + +namespace DB::Cas +{ + +/// The mount fence a write is admitted under, expressed as three closures so a caller (or a test) +/// can swap in a real mount's fence or a fixed, always-open one without a virtual base class. +struct Fence +{ + enum class Admit : uint8_t { Ok, LostOrRearmed, NoBudget }; + + /// The fence's current generation. + std::function generation; + /// Whether a write admitted under `admitted_generation`, expected to still be running + /// `needed_ms` from now, may proceed. + std::function admit; + /// Throw if `admitted_generation` is no longer the fence's live generation. + std::function check_or_throw; + + /// A fence that never trips: generation 0 forever, `admit` always `Ok`, `check_or_throw` never + /// throws. For backends with no mount lease to enforce (in-memory, tests). + static Fence open(); +}; + +inline Fence Fence::open() +{ + return Fence{ + []() -> uint64_t { return 0; }, + [](uint64_t, uint64_t) { return Fence::Admit::Ok; }, + [](uint64_t) {}}; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp index fef7cdaafad8..2ab9adac812d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include @@ -11,6 +12,7 @@ namespace ErrorCodes { extern const int CORRUPTED_DATA; extern const int FILE_DOESNT_EXIST; + extern const int LOGICAL_ERROR; } } @@ -20,9 +22,8 @@ namespace DB::Cas namespace { -/// The windowed slice of `data` for `range`, with the same clamping `get` documents: an offset at or -/// past EOF yields an empty result; an open-ended length runs to EOF. Shared by `get` and `getStream` -/// so the two stay in lockstep. +/// The windowed slice of `data` for `range`, with the clamping `getStream` documents: an offset at or +/// past EOF yields an empty result; an open-ended length runs to EOF. String sliceWindow(const String & data, Range range) { const size_t offset = static_cast(range.offset); @@ -33,6 +34,30 @@ String sliceWindow(const String & data, Range range) return data.substr(offset); } +/// A CALLER bug, refused before it ever reaches the store: an empty, wildcard or list value would +/// turn a conditional mutation into an unconditional one. Stricter than +/// `isIncarnationValue(Dialect::Emulated, ...)` (non-empty only): this backend is a test double +/// reused across the whole CAS gtest suite, so it also refuses `*` and a comma even though no value +/// it currently mints can contain either. +bool isValidEmulatedTokenValue(const String & value) +{ + return !value.empty() && value != "*" && value.find(',') == String::npos; +} + +/// Every conditional mutation refuses a malformed expected value unconditionally, matching the +/// production backend's own guard: this is the test backend for that contract, and must not accept +/// anything production refuses. A caller holding only a HEAD-derived value for a key it has not +/// confirmed exists must gate on presence itself before calling, exactly as every production call +/// site already does. +void checkExpectedValue(const String & key, const String & value) +{ + if (!isValidEmulatedTokenValue(value)) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "InMemoryBackend: refusing a conditional mutation of '{}' with a malformed token '{}': " + "an empty, wildcard or list token would turn the precondition into an unconditional write", + key, value); +} + } Token InMemoryBackend::mintToken() @@ -43,18 +68,35 @@ Token InMemoryBackend::mintToken() return t; } -std::optional InMemoryBackend::get(const String & key, Range range) +std::exception_ptr InMemoryBackend::takeArmedFailure(ArmedFailures & armed, const String & key) +{ + std::lock_guard lock(mutex_); + const auto it = armed.find(key); + if (it == armed.end() || it->second.empty()) + return nullptr; + std::exception_ptr error = it->second.front(); + it->second.erase(it->second.begin()); + return error; +} + +std::function InMemoryBackend::hookFor(const Hooks & hooks, const String & key) const { + std::lock_guard lock(mutex_); + const auto it = hooks.find(key); + return it == hooks.end() ? std::function{} : it->second; +} + +std::optional InMemoryBackend::read(const String & key, TransportAccess &) +{ + if (auto armed = takeArmedFailure(read_failures_, key)) + std::rethrow_exception(armed); + std::lock_guard lock(mutex_); auto it = store_.find(key); if (it == store_.end()) return std::nullopt; - GetResult gr; - gr.bytes = sliceWindow(it->second.bytes, range); - gr.token = it->second.token; - gr.attributes = it->second.meta; - return gr; + return Raw{it->second.bytes, it->second.token.value}; } std::optional InMemoryBackend::getStream(const String & key, Range range) @@ -72,53 +114,142 @@ std::optional InMemoryBackend::getStream(const String & key, Ra return sr; } -HeadResult InMemoryBackend::head(const String & key) +std::unique_ptr InMemoryBackend::stream(const String & key, TransportAccess &) +{ + auto sr = getStream(key, Range{}); + if (!sr) + return nullptr; + return std::move(sr->stream); +} + +std::optional InMemoryBackend::head(const String & key, TransportAccess &) { + if (auto armed = takeArmedFailure(head_failures_, key)) + std::rethrow_exception(armed); + std::lock_guard lock(mutex_); auto it = store_.find(key); if (it == store_.end()) - return HeadResult{}; + return std::nullopt; + + return RawMeta{static_cast(it->second.bytes.size()), it->second.token.value}; +} + +std::expected InMemoryBackend::write( + const String & key, const String & bytes, const std::optional & expected_value, TransportAccess &) +{ + return applyWrite(key, bytes, expected_value, WriteKnobs::All); +} + +PutResult InMemoryBackend::putIfAbsent(const String & key, const String & bytes, const ObjectMeta &) +{ + auto r = applyWrite(key, bytes, std::nullopt, WriteKnobs::AmbiguousPutIfAbsent); + if (!r) + return PutResult{PutOutcome::PreconditionFailed, {}}; + return PutResult{PutOutcome::Done, legacyMintWritten(key, std::move(*r))}; +} + +CasResult InMemoryBackend::casPut(const String & key, const String & bytes, + const std::optional & expected, const ObjectMeta &) +{ + if (expected && legacyTokenIsForeign(key, *expected)) + return CasResult{CasOutcome::Conflict, {}}; + auto r = applyWrite(key, bytes, expected ? std::optional(expected->value) : std::nullopt, + WriteKnobs::FailNextCasPut); + if (!r) + return CasResult{CasOutcome::Conflict, {}}; + return CasResult{CasOutcome::Committed, legacyMintWritten(key, std::move(*r))}; +} + +std::expected InMemoryBackend::applyWrite( + const String & key, const String & bytes, const std::optional & expected_value, WriteKnobs knobs) +{ + if (expected_value) + checkExpectedValue(key, *expected_value); + + if (auto armed = takeArmedFailure(write_failures_, key)) + std::rethrow_exception(armed); + + /// Both hooks run with NO lock held: a hook exists to read and write this backend from inside a + /// write, and `mutex_` is not recursive. + if (auto hook = hookFor(before_write_hooks_, key)) + hook(); - HeadResult hr; - hr.exists = true; - hr.size = static_cast(it->second.bytes.size()); - hr.token = it->second.token; - hr.attributes = it->second.meta; - return hr; + auto result = writeUnderLock(key, bytes, expected_value, knobs); + if (!result.has_value()) + return result; + + if (auto hook = hookFor(write_committed_hooks_, key)) + hook(); + + /// Last, so the object is durable and every observer has run before the response goes missing. + if (knobs == WriteKnobs::All && takeAmbiguousLandedWrite(key)) + throw std::runtime_error("InMemoryBackend: the write of '" + key + "' landed and its response was lost"); + + return result; } -PutResult InMemoryBackend::putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) +std::expected InMemoryBackend::writeUnderLock( + const String & key, const String & bytes, const std::optional & expected_value, WriteKnobs knobs) { std::lock_guard lock(mutex_); - // One-shot injected ambiguous outcome: throw WITHOUT touching the store, modeling a request whose - // own attempt outcome never reached the caller (see the header doc for the classification this - // must produce). std::runtime_error, not DB::Exception, is deliberate: it dodges BOTH - // classification paths in BOTH build configurations -- dynamic_cast fails (so - // isDeterministicLocalFailure is never consulted), and classifyConditionalWriteResult falls through - // to its Unresolved default because it isn't an S3Exception. A DB::Exception would have been - // fragile: picking a code outside isDeterministicLocalFailure's set is a landmine for the next - // person who extends that set. - auto ambiguous_it = ambiguous_put_keys_.find(key); - if (ambiguous_it != ambiguous_put_keys_.end()) + if (!expected_value && (knobs == WriteKnobs::All || knobs == WriteKnobs::AmbiguousPutIfAbsent)) { - ambiguous_put_keys_.erase(ambiguous_it); - throw std::runtime_error("InMemoryBackend: injected ambiguous putIfAbsent outcome for '" + key + "'"); + // One-shot injected ambiguous outcome: throw WITHOUT touching the store, modeling a request + // whose own attempt outcome never reached the caller (see the header doc for the + // classification this must produce). std::runtime_error, not DB::Exception, is deliberate: it + // dodges BOTH classification paths in BOTH build configurations -- dynamic_cast fails (so isDeterministicLocalFailure is never consulted), and + // classifyConditionalWriteResult falls through to its Unresolved default because it isn't an + // S3Exception. A DB::Exception would have been fragile: picking a code outside + // isDeterministicLocalFailure's set is a landmine for the next person who extends that set. + auto ambiguous_it = ambiguous_put_keys_.find(key); + if (ambiguous_it != ambiguous_put_keys_.end()) + { + ambiguous_put_keys_.erase(ambiguous_it); + throw std::runtime_error("InMemoryBackend: injected ambiguous write outcome for '" + key + "'"); + } + } + + // One-shot injected conflict, on EITHER form: the knob is armed against a conditional write, and + // a create-if-absent is one -- the lease acquire this models creates its object. + if (knobs == WriteKnobs::All || knobs == WriteKnobs::FailNextCasPut) + { + auto fail_it = fail_next_cas_.find(key); + if (fail_it != fail_next_cas_.end()) + { + fail_next_cas_.erase(fail_it); + return std::unexpected(RawConflict{}); + } } - if (store_.contains(key)) - return {PutOutcome::PreconditionFailed, {}}; + if (!expected_value) + { + if (store_.contains(key)) + return std::unexpected(RawConflict{}); + + Token t = mintToken(); + Object obj; + obj.bytes = bytes; + obj.token = t; + store_[key] = std::move(obj); + return t.value; + } + + auto it = store_.find(key); + if (it == store_.end()) + return std::unexpected(RawConflict{}); + if (enforce_tokens_ && it->second.token.value != *expected_value) + return std::unexpected(RawConflict{}); Token t = mintToken(); - Object obj; - obj.bytes = bytes; - obj.token = t; - obj.meta = meta; - store_[key] = std::move(obj); - return {PutOutcome::Done, t}; + it->second.bytes = bytes; + it->second.token = t; + return t.value; } -void InMemoryBackend::publishBlob(const BlobPublishRequest & request) +void InMemoryBackend::publish(const BlobPublishRequest & request, TransportAccess &) { if (const auto * streaming = std::get_if(&request.publication)) { @@ -126,7 +257,7 @@ void InMemoryBackend::publishBlob(const BlobPublishRequest & request) if (!payload) throw Exception( ErrorCodes::CORRUPTED_DATA, - "InMemoryBackend::publishBlob: payload source for {} returned no reader", + "InMemoryBackend::publish: payload source for {} returned no reader", request.destination_key); /// Drain before taking the store lock: the source may itself read another object from this @@ -145,7 +276,7 @@ void InMemoryBackend::publishBlob(const BlobPublishRequest & request) if (!copy_result.exact(streaming->payload_size)) throw Exception( ErrorCodes::CORRUPTED_DATA, - "InMemoryBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", + "InMemoryBackend::publish: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", copy_result.has_excess ? "more than " : "", copy_result.copied, request.destination_key, @@ -165,7 +296,7 @@ void InMemoryBackend::publishBlob(const BlobPublishRequest & request) if (source == store_.end()) throw Exception( ErrorCodes::FILE_DOESNT_EXIST, - "InMemoryBackend::publishBlob: staging object {} is absent", + "InMemoryBackend::publish: staging object {} is absent", staged.object_key); Object object; @@ -174,66 +305,6 @@ void InMemoryBackend::publishBlob(const BlobPublishRequest & request) store_[request.destination_key] = std::move(object); } -PutResult InMemoryBackend::putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) -{ - std::lock_guard lock(mutex_); - auto it = store_.find(key); - if (it == store_.end()) - return {PutOutcome::PreconditionFailed, {}}; - - if (enforce_tokens_ && it->second.token != expected) - return {PutOutcome::PreconditionFailed, {}}; - - Token t = mintToken(); - it->second.bytes = bytes; - it->second.token = t; - it->second.meta = meta; - return {PutOutcome::Done, t}; -} - -CasResult InMemoryBackend::casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) -{ - std::lock_guard lock(mutex_); - - // One-shot injected conflict - auto fail_it = fail_next_cas_.find(key); - if (fail_it != fail_next_cas_.end()) - { - fail_next_cas_.erase(fail_it); - return {CasOutcome::Conflict, {}}; - } - - auto it = store_.find(key); - bool exists = (it != store_.end()); - - if (!expected.has_value()) - { - // create-if-absent CAS - if (exists) - return {CasOutcome::Conflict, {}}; - Token t = mintToken(); - Object obj; - obj.bytes = bytes; - obj.token = t; - obj.meta = meta; - store_[key] = std::move(obj); - return {CasOutcome::Committed, t}; - } - else - { - // swap-if-current CAS - if (!exists) - return {CasOutcome::Conflict, {}}; - if (enforce_tokens_ && it->second.token != *expected) - return {CasOutcome::Conflict, {}}; - Token t = mintToken(); - it->second.bytes = bytes; - it->second.token = t; - it->second.meta = meta; - return {CasOutcome::Committed, t}; - } -} - DeleteOutcome InMemoryBackend::applyDelete(const String & key, const Token & token) { // Caller holds the mutex. @@ -259,47 +330,53 @@ DeleteOutcome InMemoryBackend::applyDelete(const String & key, const Token & tok return d; } -DeleteOutcome InMemoryBackend::deleteExact(const String & key, const Token & token) +Backend::RawRemoval InMemoryBackend::remove(const String & key, const String & expected_value, TransportAccess &) { + /// See `checkExpectedValue`: a malformed value is refused as a caller bug, unconditionally -- + /// covering both the immediate delete below and the hold_deletes_ enqueue path, so a queued + /// PendingDelete can never carry a malformed value either. + checkExpectedValue(key, expected_value); + + const Token expected{expected_value, TokenType::Emulated}; + std::lock_guard lock(mutex_); if (hold_deletes_) { - // Validate the key exists (and token matches if enforcing) before queuing, + // Validate the key exists (and the value matches if enforcing) before queuing, // but don't remove yet — just enqueue. auto it = store_.find(key); if (it == store_.end()) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::NotFound; - return d; - } - if (enforce_tokens_ && it->second.token != token) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::TokenMismatch; - return d; - } + return RawRemoval::Gone; + if (enforce_tokens_ && it->second.token != expected) + return RawRemoval::Mismatch; PendingDelete pd; pd.key = key; - pd.token = token; + pd.token = expected; pending_deletes_.push_back(std::move(pd)); - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::Deleted; - d.created_delete_marker = simulate_delete_markers_; - return d; + return simulate_delete_markers_ ? RawRemoval::DeleteMarker : RawRemoval::Removed; } - return applyDelete(key, token); + const DeleteOutcome d = applyDelete(key, expected); + switch (d.kind) + { + case DeleteOutcome::Kind::Deleted: + return d.created_delete_marker ? RawRemoval::DeleteMarker : RawRemoval::Removed; + case DeleteOutcome::Kind::NotFound: + return RawRemoval::Gone; + case DeleteOutcome::Kind::TokenMismatch: + return RawRemoval::Mismatch; + } + UNREACHABLE(); } -ListPage InMemoryBackend::list(const String & prefix, const String & cursor, size_t limit) +Backend::RawListPage InMemoryBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess &) { if (limit == 0) return {}; std::lock_guard lock(mutex_); - ListPage page; + RawListPage page; // Cursor is the last key returned by the previous page. auto it = cursor.empty() ? store_.lower_bound(prefix) : store_.upper_bound(cursor); @@ -310,10 +387,10 @@ ListPage InMemoryBackend::list(const String & prefix, const String & cursor, siz if (!it->first.starts_with(prefix)) break; - ListedKey lk; + RawListedKey lk; lk.key = it->first; lk.size = static_cast(it->second.bytes.size()); - lk.token = it->second.token; /// in-memory backend always surfaces the token (supportsListTokens == true) + lk.value = it->second.token.value; /// in-memory backend always surfaces it (supportsListTokens == true) page.keys.push_back(std::move(lk)); ++count; ++it; @@ -326,6 +403,49 @@ ListPage InMemoryBackend::list(const String & prefix, const String & cursor, siz return page; } +bool InMemoryBackend::refreshCredentials() +{ + std::lock_guard lock(mutex_); + ++refresh_credentials_calls_; + return refresh_credentials_result_; +} + +size_t InMemoryBackend::refreshCredentialsCalls() const +{ + std::lock_guard lock(mutex_); + return refresh_credentials_calls_; +} + +void InMemoryBackend::failNextWriteWith(const String & key, std::exception_ptr error) +{ + std::lock_guard lock(mutex_); + write_failures_[key].push_back(std::move(error)); +} + +void InMemoryBackend::failNextReadWith(const String & key, std::exception_ptr error) +{ + std::lock_guard lock(mutex_); + read_failures_[key].push_back(std::move(error)); +} + +void InMemoryBackend::failNextHeadWith(const String & key, std::exception_ptr error) +{ + std::lock_guard lock(mutex_); + head_failures_[key].push_back(std::move(error)); +} + +void InMemoryBackend::onBeforeWrite(const String & key, std::function hook) +{ + std::lock_guard lock(mutex_); + before_write_hooks_[key] = std::move(hook); +} + +void InMemoryBackend::onWriteCommitted(const String & key, std::function hook) +{ + std::lock_guard lock(mutex_); + write_committed_hooks_[key] = std::move(hook); +} + void InMemoryBackend::setHoldDeletes(bool hold) { std::lock_guard lock(mutex_); @@ -367,6 +487,18 @@ void InMemoryBackend::injectAmbiguousPutIfAbsent(const String & key) ambiguous_put_keys_.insert(key); } +void InMemoryBackend::injectAmbiguousLandedWrite(const String & key) +{ + std::lock_guard lock(mutex_); + ambiguous_landed_keys_.insert(key); +} + +bool InMemoryBackend::takeAmbiguousLandedWrite(const String & key) +{ + std::lock_guard lock(mutex_); + return ambiguous_landed_keys_.erase(key) != 0; +} + void InMemoryBackend::setEnforceTokens(bool enforce) { std::lock_guard lock(mutex_); @@ -379,4 +511,10 @@ void InMemoryBackend::setSimulateDeleteMarkers(bool simulate) simulate_delete_markers_ = simulate; } +void InMemoryBackend::setRefreshCredentialsResult(bool result) +{ + std::lock_guard lock(mutex_); + refresh_credentials_result_ = result; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h index 5e67aca61f04..2f2dd6c281a1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h @@ -1,5 +1,7 @@ #pragma once #include +#include +#include #include #include #include @@ -26,95 +28,144 @@ class InMemoryBackend : public Backend public: InMemoryBackend() = default; - /// Unhide the base convenience overloads (omitted Range/ObjectMeta/expected-token forms): the - /// overrides below would otherwise shadow them for callers holding a concrete backend type. - using Backend::get; + /// Unhide the base overloads this class's own declarations would otherwise shadow: the legacy + /// `head`/`list`/`getStream`/`putIfAbsent`/`casPut` names, and the omitted-`Range`/`ObjectMeta` + /// conveniences. + using Backend::casPut; using Backend::getStream; + using Backend::head; + using Backend::list; using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; // ---- Backend interface ---- - /// Returns the requested byte window, current token, and metadata, or `nullopt` when the key is absent. - std::optional get(const String & key, Range range) override; + /// Returns the stored bytes and the key's current incarnation value, or `nullopt` when absent. + std::optional read(const String & key, TransportAccess & access) override; - /// Returns a forward-only stream over the requested byte window, or `nullopt` when the key is absent. - /// The in-memory implementation copies the window into an owning read buffer while holding the - /// backend lock, so the returned stream remains independent of later backend mutations. - std::optional getStream(const String & key, Range range) override; + /// Returns the current size and incarnation value without materializing the body. + std::optional head(const String & key, TransportAccess & access) override; - /// Returns the current existence, size, token, and metadata without materializing the body. - HeadResult head(const String & key) override; + /// Lists up to `limit` keys under `prefix` in map order. `cursor` is the last key from the + /// previous page; `next_cursor` is set only when more matching keys remain. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override; - /// The in-memory backend mints a monotonic token it surfaces through `list` — TRUE. - bool supportsListTokens() const override { return true; } + /// Removes exactly the incarnation named by `expected_value`, or queues that check for a later + /// `landPendingDelete` when delete holding is enabled. A queued delete is reported as removed, + /// but its expected value is rechecked when it is landed. + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override; - /// Creates `key` only when it is absent. On success stores `bytes` and `meta` under a new token; - /// on a precondition failure leaves the existing object untouched. - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override; + /// Creates the key when `expected_value` is empty, or replaces the incarnation it names. A + /// refused precondition leaves the store unchanged. Value enforcement can be disabled with + /// `setEnforceTokens` to model a backend that incorrectly ignores the condition. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override; + + /// A forward-only reader over a private copy of the bytes; null when the key is absent. + std::unique_ptr stream(const String & key, TransportAccess & access) override; /// Publishes either `[fresh_envelope][payload]` or the complete staged bytes as one atomic /// in-memory replacement. Streaming sources are fully validated before the destination changes. - void publishBlob(const BlobPublishRequest & request) override; + void publish(const BlobPublishRequest & request, TransportAccess & access) override; - /// Replaces the existing object only when `expected` is its current token. Token enforcement can - /// be disabled with `setEnforceTokens` to model a backend that incorrectly ignores this condition. - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, - const ObjectMeta & meta) override; + /// This backend mints its own emulated values. + Dialect dialect() const override { return Dialect::Emulated; } - /// Performs create-if-absent when `expected` is empty, or replace-if-current-token otherwise. - /// Conflicts leave the store unchanged and are returned as an outcome rather than an exception. - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override; + /// The in-memory backend mints a monotonic value it surfaces through `list` — TRUE. + bool supportsListTokens() const override { return true; } - /// Removes exactly the incarnation named by `token`, or queues that token check for a later - /// `landPendingDelete` when delete holding is enabled. A queued delete is reported as accepted, - /// but its token is rechecked when it is landed. - DeleteOutcome deleteExact(const String & key, const Token & token) override; + /// Whatever `setRefreshCredentialsResult` last configured; FALSE by default, so a test that has + /// not opted in models a backend with no refresh mechanism. + bool refreshCredentials() override; - /// Lists up to `limit` keys under `prefix` in map order. `cursor` is the last key from the previous - /// page; returned tokens identify the listed incarnations and `next_cursor` is set only when more - /// matching keys remain. - ListPage list(const String & prefix, const String & cursor, size_t limit) override; + /// Returns a forward-only stream over the requested byte window, or `nullopt` when the key is absent. + /// The in-memory implementation copies the window into an owning read buffer while holding the + /// backend lock, so the returned stream remains independent of later backend mutations. + std::optional getStream(const String & key, Range range) override; + + /// ---- Two legacy verbs, overridden ONLY so each write knob keeps its verb identity ---- + /// + /// A knob is armed against a VERB, but the keyed `write` cannot see which verb its caller used, so + /// the base forwarder would let `failNextCasPut` fire on a `putIfAbsent` and + /// `injectAmbiguousPutIfAbsent` on a create-shaped `casPut`. Each of these consumes only the knob + /// named for it, and neither reaches the keyed primitive -- so, unlike every other legacy verb, an + /// override of `write` in a SUBCLASS of this backend does not intercept these two. Deleted with the + /// rest of the legacy surface at the lock, and the exception goes with them. + PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override; + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override; // ---- Fault-injection controls ---- - /// When true, `deleteExact` validates and enqueues deletes rather than applying them immediately. - /// The caller sees `Deleted` (the send was accepted), but the object remains until - /// `landPendingDelete`, where the token is checked again. + /// When true, `remove` validates and enqueues deletes rather than applying them immediately. + /// The caller sees `Removed` (the send was accepted), but the object remains until + /// `landPendingDelete`, where the expected value is checked again. void setHoldDeletes(bool hold); /// Returns the number of currently held deletes. size_t pendingDeletes() const; - /// Applies and removes the held delete at index `i`. The token is evaluated against the current - /// object at land time; the queue entry is removed whether the result is `TokenMismatch` or - /// `Deleted`. An invalid index returns `NotFound`. + /// Applies and removes the held delete at index `i`. The expected value is evaluated against the + /// current object at land time; the queue entry is removed whether the result is `Mismatch` or + /// `Removed`. An invalid index returns `NotFound`. DeleteOutcome landPendingDelete(size_t i); - /// Injects a one-shot artificial `Conflict` on the next `casPut` for `key`. + /// Injects a one-shot artificial refusal on the next `casPut` of `key`, IN EITHER FORM: a GC lease + /// acquire creates its object, and a test arming this knob for it is testing exactly that create + /// losing its condition. void failNextCasPut(const String & key); - /// Injects a one-shot AMBIGUOUS outcome on the next `putIfAbsent` for `key`: instead of attempting - /// the write, that call throws a plain (non-`DB::Exception`) exception -- classified `Unresolved`, - /// never `DefiniteFailure`, by `classifyConditionalWriteResult` regardless of build flags -- and the - /// store is left exactly as it was. Models a request whose own HTTP attempt outcome is lost (a - /// timeout, a dropped connection) rather than a clean `PreconditionFailed`, for tests of controlled - /// ops (`CasRequestController::slotOccupy` and its callers) that must exercise the "ambiguous - /// attempt, resolve before deciding" path without a live network. One-shot, mirroring - /// `failNextCasPut`'s contract: consumed by the first matching `putIfAbsent` call, whether the key - /// was already present or not. + /// Injects a one-shot AMBIGUOUS outcome on the next CREATING write of `key` (a write with no + /// expected value): instead of attempting it, that call throws a plain (non-`DB::Exception`) + /// exception -- classified `Unresolved`, never `DefiniteFailure`, by + /// `classifyConditionalWriteResult` regardless of build flags -- and the store is left exactly as + /// it was. Models a request whose own HTTP attempt outcome is lost (a timeout, a dropped + /// connection) rather than a clean refusal, for tests that must exercise the "ambiguous attempt, + /// resolve before deciding" path without a live network. One-shot, mirroring `failNextCasPut`'s + /// contract: consumed by the first matching write, whether the key was already present or not. void injectAmbiguousPutIfAbsent(const String & key); - /// Enables or disables token checks for delete, overwrite, and CAS operations. Disabling checks - /// models a backend that reports every expected token as matching. + /// The other ambiguity, and the only one that can prove a resolve read settles a commit: the next + /// write of `key` IS APPLIED and then throws a plain (non-`DB::Exception`) exception, so the object + /// is durable and its incarnation was never returned. One-shot, and consumed by the keyed `write` + /// and by every legacy verb that forwards through it -- `putOverwrite` today. The two verbs that + /// route around the primitive, `putIfAbsent` and `casPut`, do not consume it. + void injectAmbiguousLandedWrite(const String & key); + + /// Enables or disables value checks for remove and replace. Disabling checks models a backend + /// that reports every expected value as matching. void setEnforceTokens(bool enforce); - /// When true, successful deletes report `created_delete_marker = true`, modelling a versioned S3 - /// bucket whose delete creates a marker instead of reclaiming the current object. + /// When true, a successful `remove` answers `DeleteMarker`, modelling a versioned S3 bucket whose + /// delete creates a marker instead of reclaiming the current object. void setSimulateDeleteMarkers(bool simulate); + /// What `refreshCredentials` answers: TRUE models a storage that installed fresh credentials. + void setRefreshCredentialsResult(bool result); + + /// How many times `refreshCredentials` has been called on this backend. + size_t refreshCredentialsCalls() const; + + /// The next `write` naming `key` throws `error` instead of applying it, and the store is left + /// exactly as it was. Each arming is consumed by one write, so arming twice fails two consecutive + /// attempts of the same call. An `exception_ptr` rather than a concrete type because a caller + /// classifies a failed attempt by its exception CLASS, and the classes worth exercising span + /// `S3Exception`, `DB::Exception`, `Poco::Exception` and plain `std::exception`. + void failNextWriteWith(const String & key, std::exception_ptr error); + /// The read-side siblings, for the read loop's own classification. `read` and `head` are armed + /// separately because the two resolve loops differ in exactly which of them they issue: a + /// presence-only caller must be able to fail its HEAD without a body read stealing the arming. + void failNextReadWith(const String & key, std::exception_ptr error); + void failNextHeadWith(const String & key, std::exception_ptr error); + + /// Runs before a write of `key` is applied, with no backend lock held -- so a hook may itself read + /// and write this backend, which is what it exists for: a hook that replaces `key` models a + /// permanently hot key whose incarnation moves under every attempt. A hook that writes the same + /// key re-enters this callback, so a hook must guard its own recursion. + void onBeforeWrite(const String & key, std::function hook); + /// Runs after a write of `key` is durable and BEFORE its value is returned, with no backend lock + /// held -- the point at which a fact outside the store can change while a write is in flight. + void onWriteCommitted(const String & key, std::function hook); + private: /// Complete in-memory incarnation state for one key. All fields are read or modified while /// `mutex_` is held; replacing `token` marks a new incarnation even when the bytes are unchanged. @@ -122,11 +173,10 @@ class InMemoryBackend : public Backend { String bytes; Token token; - ObjectMeta meta; }; - /// Token captured when a held delete is queued. It is intentionally checked again at land time so - /// a replacement between send and land produces `TokenMismatch` rather than deleting the new object. + /// Value captured when a held delete is queued. It is intentionally checked again at land time so + /// a replacement between send and land produces `Mismatch` rather than deleting the new object. struct PendingDelete { String key; @@ -137,10 +187,36 @@ class InMemoryBackend : public Backend /// backend instance, which also makes token equality a safe content-cache identity check in tests. Token mintToken(); - /// Applies an exact-token delete while `mutex_` is already held. Used by immediate deletes and by + /// Applies an exact-value delete while `mutex_` is already held. Used by immediate deletes and by /// `landPendingDelete` after its queue entry has been removed. DeleteOutcome applyDelete(const String & key, const Token & token); + using ArmedFailures = std::map>; + using Hooks = std::map>; + + /// Which of the verb-scoped write knobs one call may consume. + enum class WriteKnobs : uint8_t + { + All, /// the keyed `write`, and the legacy verbs that forward through it + AmbiguousPutIfAbsent, /// legacy `putIfAbsent` + FailNextCasPut, /// legacy `casPut`, either form + }; + + /// Consumes and returns the next failure armed for `key`, or null when none is. + std::exception_ptr takeArmedFailure(ArmedFailures & armed, const String & key); + /// Consumes the landed-then-lost arming for `key`, if there is one. + bool takeAmbiguousLandedWrite(const String & key); + /// A copy of the hook registered for `key`, taken under the lock so the caller can run it without + /// one. + std::function hookFor(const Hooks & hooks, const String & key) const; + /// One write, whichever verb asked for it: armed failure, hooks, the store mutation, and exactly + /// the knobs `knobs` allows. + std::expected applyWrite(const String & key, const String & bytes, + const std::optional & expected_value, WriteKnobs knobs); + /// The part of `applyWrite` that touches the store, run with `mutex_` held. + std::expected writeUnderLock(const String & key, const String & bytes, + const std::optional & expected_value, WriteKnobs knobs); + mutable std::mutex mutex_; std::map store_; uint64_t token_seq_ = 0; @@ -150,8 +226,16 @@ class InMemoryBackend : public Backend std::vector pending_deletes_; std::set fail_next_cas_; std::set ambiguous_put_keys_; + std::set ambiguous_landed_keys_; bool enforce_tokens_ = true; bool simulate_delete_markers_ = false; + bool refresh_credentials_result_ = false; + size_t refresh_credentials_calls_ = 0; + ArmedFailures write_failures_; + ArmedFailures read_failures_; + ArmedFailures head_failures_; + Hooks before_write_hooks_; + Hooks write_committed_hooks_; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp new file mode 100644 index 000000000000..f01a45d95fc0 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp @@ -0,0 +1,34 @@ +#include + +#include + +#include + +namespace DB::Cas +{ + +bool isIncarnationValue(Dialect dialect, const String & value) +{ + switch (dialect) + { + case Dialect::Generation: + { + if (value.empty() || value == "0") + return false; + if (value.size() > 1 && value.front() == '0') + return false; + return std::all_of(value.begin(), value.end(), [](char c) { return c >= '0' && c <= '9'; }); + } + case Dialect::ETag: + { + String trimmed = value; + boost::algorithm::trim(trimmed); + return !trimmed.empty() && trimmed != "*" && trimmed.find(',') == String::npos; + } + case Dialect::Emulated: + return !value.empty(); + } + return false; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h new file mode 100644 index 000000000000..1f8edf1889d6 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h @@ -0,0 +1,88 @@ +#pragma once +#include +#include +#include +#include + +namespace DB::Cas +{ + +using Dialect = TokenType; + +/// The per-dialect grammar a response value must meet to be an incarnation. Generation: canonical +/// positive decimal (no leading zero, not "0" -- zero is the dialect's absence sentinel). ETag: +/// non-empty, not "*" after trimming whitespace, no comma (a list matches any member). Emulated: +/// non-empty. `ObjectStorageBackend::isValidTokenValue` forwards here. +bool isIncarnationValue(Dialect dialect, const String & value); + +/// One backend-observed incarnation of an object: the transport's own token value together with the +/// backend and key it was observed against. Not default-constructible, not constructible from a bare +/// `String`, and minted ONLY by `CasRequests` -- a caller can hold one only by way of an admitted +/// read or write, so an `Incarnation` is always traceable to the request that produced it. +class Incarnation +{ +public: + Incarnation() = delete; + bool operator==(const Incarnation &) const = default; + + /// "etag:" | "generation:" | "emulated:" + String render() const; + const String & key() const { return key_; } + Dialect dialect() const { return dialect_; } + uint64_t backendId() const { return backend_id_; } + +private: + friend class CasRequests; + + Incarnation(uint64_t backend_id, String key, Dialect dialect, String value) + : backend_id_(backend_id), key_(std::move(key)), dialect_(dialect), value_(std::move(value)) + { + } + + /// The transport's own text; CasRequests reads it to build the next conditional request. + const String & value() const { return value_; } + + uint64_t backend_id_; + String key_; + Dialect dialect_; + String value_; +}; + +inline String Incarnation::render() const +{ + switch (dialect_) + { + case Dialect::ETag: return "etag:" + value_; + case Dialect::Generation: return "generation:" + value_; + case Dialect::Emulated: return "emulated:" + value_; + } + UNREACHABLE(); +} + +/// An incarnation as recorded in a persisted manifest/ref: the dialect word and value, without any +/// live backend to check them against. Forward-only: a `PersistedIncarnation` is captured FROM a live +/// `Incarnation`, never the reverse -- a persisted record must never be trusted to mint a live one. +/// `matches` re-derives the same rendering the live incarnation would produce and compares it +/// textually, so the two representations can never drift apart. +struct PersistedIncarnation +{ + String dialect; /// "etag" | "generation" | "emulated" + String value; + + static PersistedIncarnation capture(const Incarnation & live); + bool matches(const Incarnation & live) const; +}; + +inline PersistedIncarnation PersistedIncarnation::capture(const Incarnation & live) +{ + const String rendered = live.render(); + const auto colon = rendered.find(':'); + return PersistedIncarnation{rendered.substr(0, colon), rendered.substr(colon + 1)}; +} + +inline bool PersistedIncarnation::matches(const Incarnation & live) const +{ + return live.render() == dialect + ":" + value; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp index 4749d0c8ca35..5c6a892f8b2b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp @@ -82,6 +82,10 @@ namespace DB::Cas /// Maps `(CasNs, CasOp)` to the corresponding `ProfileEvents::Event`. The table is row-major: the /// outer index is the namespace and the inner index is the operation. Its rows and columns must stay /// in lockstep with the `CasNs` and `CasOp` enum orderings. +/// +/// `CasOp::Read` deliberately keeps the `CAS*Get` event names: renaming a user-visible ProfileEvent +/// is its own change, made when the events are retired, and nothing outside this table would have +/// been improved by doing it here. static const ProfileEvents::Event cas_event_table[CAS_NS_COUNT][CAS_OP_COUNT] = { /* Blob */ {ProfileEvents::CASBlobPut, ProfileEvents::CASBlobPutDeduplicated, ProfileEvents::CASBlobOverwrite, @@ -134,6 +138,12 @@ void incrementCasEvent(CasNs ns, CasOp op) ProfileEvents::increment(cas_event_table[static_cast(ns)][static_cast(op)]); } +void InstrumentedBackend::publish(const BlobPublishRequest & request, TransportAccess & access) +{ + inner->publish(request, access); + incrementCasEvent(classifyCasNs(request.destination_key), CasOp::Put); +} + void InstrumentedBackend::publishBlob(const BlobPublishRequest & request) { inner->publishBlob(request); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h index 5df949883e62..25d2407a57c4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h @@ -35,15 +35,28 @@ enum class CasNs : uint8_t }; static constexpr size_t CAS_NS_COUNT = 6; -/// Operation + outcome class (11 classes), mapped from the `Backend` method and its return value. -/// putIfAbsent → Done ⇒ Put ; PreconditionFailed ⇒ PutDeduplicated -/// putOverwrite → Done ⇒ Overwrite ; PreconditionFailed ⇒ CasConflict -/// casPut → Committed ⇒ Cas ; Conflict ⇒ CasConflict -/// head → exists ⇒ Head ; !exists ⇒ HeadMiss (the 404 signal) -/// get → Get (all calls, hit or miss) -/// getStream → GetStream (all calls, hit or miss) -/// deleteExact → Delete (all outcomes) -/// list → List +/// Operation + outcome class, mapped from the `Backend` method and its result. +/// +/// The decorator counts BOTH surfaces for the migration window. It must: it forwards a legacy call +/// to the inner backend AS a legacy call, so that a double wrapped by a `Pool` still intercepts it, +/// and the conversion to a primitive happens one level down, inside that double. Each request is +/// therefore counted exactly once, on whichever surface its caller used. +/// putIfAbsent → Done ⇒ Put ; PreconditionFailed ⇒ PutDeduplicated +/// putOverwrite → Done ⇒ Overwrite ; PreconditionFailed ⇒ CasConflict +/// casPut → Committed ⇒ Cas ; Conflict ⇒ CasConflict +/// write, no expected value → a value ⇒ Put ; RawConflict ⇒ PutDeduplicated +/// write, an expected value → a value ⇒ Overwrite ; RawConflict ⇒ CasConflict +/// head, head(key) → present ⇒ Head ; absent ⇒ HeadMiss (the 404 signal) +/// read, get → Read (all calls, hit or miss) +/// getStream → GetStream +/// remove, deleteExact → Delete (all outcomes) +/// list → List +/// publish, publishBlob → Put +/// +/// `Cas` therefore counts only what a caller sent as a `casPut`: the primitive cannot tell a +/// compare-and-set from any other replacement, so a migrated replace counts as `Overwrite`. That +/// distinction, and the `CAS*CompareSwap`/`CAS*GetStream` events it feeds, go when the legacy +/// surface does. enum class CasOp : uint8_t { Put = 0, @@ -53,7 +66,7 @@ enum class CasOp : uint8_t CasConflict, Head, HeadMiss, - Get, + Read, GetStream, Delete, List, @@ -74,10 +87,13 @@ void incrementCasEvent(CasNs ns, CasOp op); class InstrumentedBackend final : public Backend { public: - /// Unhide the base convenience overloads (omitted Range/ObjectMeta/expected-token forms): the - /// overrides below would otherwise shadow them for callers holding a concrete backend type. + /// Unhide the base overloads this class's own declarations would otherwise shadow: the + /// convenience forms that omit Range/ObjectMeta/expected-token. using Backend::get; using Backend::getStream; + using Backend::head; + using Backend::list; + using Backend::probeSentinelRaw; using Backend::putIfAbsent; using Backend::putOverwrite; using Backend::casPut; @@ -92,22 +108,69 @@ class InstrumentedBackend final : public Backend /// The typed sentinel probe is a diagnostic/authoritative read, not a routine storage operation — /// deliberately uninstrumented (no ProfileEvent), like the capability checks above. MUST still be /// forwarded explicitly: `Backend::probeSentinelRaw`'s generic default derives its classification from - /// THIS object's own `head`/`get` (virtual dispatch would otherwise resolve back to + /// THIS object's own `head`/`read` (virtual dispatch would otherwise resolve back to /// `InstrumentedBackend`'s plain, non-typed overrides above), silently discarding whatever sharper /// container/permission evidence the wrapped `inner` backend (e.g. `ObjectStorageBackend`'s S3/Local /// classification) is able to provide. + SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override + { + return inner->probeSentinelRaw(key, access); + } SentinelProbeResult probeSentinelRaw(const String & key) override { return inner->probeSentinelRaw(key); } /// Delegate the read and count it after the inner call succeeds or returns absent. Exceptions /// propagate unchanged and therefore do not produce a separate outcome event. - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - auto result = inner->get(key, range); - incrementCasEvent(classifyCasNs(key), CasOp::Get); + auto result = inner->read(key, access); + incrementCasEvent(classifyCasNs(key), CasOp::Read); + return result; + } + + /// Count `Head` or `HeadMiss` from the returned presence after delegating to the backend. + std::optional head(const String & key, TransportAccess & access) override + { + auto result = inner->head(key, access); + incrementCasEvent(classifyCasNs(key), result ? CasOp::Head : CasOp::HeadMiss); + return result; + } + + /// Delegate one paginated listing and classify the prefix used for the request. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + auto page = inner->list(prefix, cursor, limit, access); + incrementCasEvent(classifyCasNs(prefix), CasOp::List); + return page; + } + + /// Delegate the conditional removal and count every returned outcome as `Delete`. + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + auto outcome = inner->remove(key, expected_value, access); + incrementCasEvent(classifyCasNs(key), CasOp::Delete); + return outcome; + } + + /// Count a create and a replacement separately, and each of them separately from its refusal: + /// they cost the same one request, but a pool whose creates are mostly refused and one whose + /// replacements mostly conflict are different problems. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + auto result = inner->write(key, bytes, expected_value, access); + const CasOp op = expected_value ? (result ? CasOp::Overwrite : CasOp::CasConflict) + : (result ? CasOp::Put : CasOp::PutDeduplicated); + incrementCasEvent(classifyCasNs(key), op); return result; } /// Delegate a forward-only read stream and count the request after the stream is acquired. + std::unique_ptr stream(const String & key, TransportAccess & access) override + { + auto result = inner->stream(key, access); + incrementCasEvent(classifyCasNs(key), CasOp::GetStream); + return result; + } std::optional getStream(const String & key, Range range) override { auto result = inner->getStream(key, range); @@ -115,7 +178,18 @@ class InstrumentedBackend final : public Backend return result; } - /// Count `Head` or `HeadMiss` from the returned presence flag after delegating to the backend. + /// ---- The legacy surface, forwarded AS legacy ---- + /// + /// Not inherited from `Backend`: its forwarder would call the primitive on THIS object, so the + /// inner backend would receive a primitive and any legacy override it carries -- which is how + /// almost every fault injection in the test suite is written -- would never run. + std::optional get(const String & key, Range range) override + { + auto result = inner->get(key, range); + incrementCasEvent(classifyCasNs(key), CasOp::Read); + return result; + } + HeadResult head(const String & key) override { HeadResult result = inner->head(key); @@ -123,7 +197,6 @@ class InstrumentedBackend final : public Backend return result; } - /// Count a successful create as `Put` and an existing-key precondition result as `PutDeduplicated`. PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override { PutResult result = inner->putIfAbsent(key, bytes, meta); @@ -131,12 +204,6 @@ class InstrumentedBackend final : public Backend return result; } - /// Count one successful physical blob publication after delegating exactly once. The backend has - /// no lifecycle reason to classify here; decision diagnostics remain with the writer. - void publishBlob(const BlobPublishRequest & request) override; - - /// Count a successful token-conditional overwrite as `Overwrite`; a precondition conflict is - /// counted as `CasConflict`. PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override { @@ -145,7 +212,6 @@ class InstrumentedBackend final : public Backend return result; } - /// Count a committed compare-and-swap as `Cas`; conflicts are counted as `CasConflict`. CasResult casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) override { @@ -154,7 +220,6 @@ class InstrumentedBackend final : public Backend return result; } - /// Delegate token-exact deletion and count every returned deletion outcome as `Delete`. DeleteOutcome deleteExact(const String & key, const Token & token) override { DeleteOutcome outcome = inner->deleteExact(key, token); @@ -162,7 +227,6 @@ class InstrumentedBackend final : public Backend return outcome; } - /// Delegate one paginated listing and classify the prefix used for the request. ListPage list(const String & prefix, const String & cursor, size_t limit) override { ListPage page = inner->list(prefix, cursor, limit); @@ -170,8 +234,17 @@ class InstrumentedBackend final : public Backend return page; } - /// This capability is a property of the wrapped backend, not an operation to count. + void publishBlob(const BlobPublishRequest & request) override; + + /// Count one successful physical blob publication after delegating exactly once. The backend has + /// no lifecycle reason to classify here; decision diagnostics remain with the writer. + void publish(const BlobPublishRequest & request, TransportAccess & access) override; + + /// These are properties of the wrapped backend, not operations to count. + Dialect dialect() const override { return inner->dialect(); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + uint64_t attemptTimeoutMs() const override { return inner->attemptTimeoutMs(); } + bool refreshCredentials() override { return inner->refreshCredentials(); } private: BackendPtr inner; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index fab29d8ae0a0..25575c709a08 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -1,6 +1,8 @@ #include +#include #include +#include #include #include #include @@ -35,15 +37,20 @@ namespace ErrorCodes extern const int CORRUPTED_DATA; extern const int FILE_DOESNT_EXIST; extern const int NOT_IMPLEMENTED; + extern const int LOGICAL_ERROR; + extern const int CAS_WRITE_UNATTRIBUTED; } } namespace DB::Cas { -ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_) +ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_, + bool single_attempt_control_plane_, uint64_t attempt_timeout_ms_) : object_storage(std::move(object_storage_)) , mode(mode_) + , single_attempt_control_plane(single_attempt_control_plane_) + , attempt_timeout_ms(attempt_timeout_ms_) , emu_root(object_storage->getCommonKeyPrefix()) { if (mode == Mode::Native && object_storage->conditionalOpsUseGenerationTokens()) @@ -130,32 +137,21 @@ void ObjectStorageBackend::checkConditionalWriteSingleAttemptSupport() /// Native helpers /// ========================================================================================= -bool ObjectStorageBackend::isValidGenerationTokenValue(const String & value) +bool ObjectStorageBackend::isValidTokenValue(TokenType type, const String & value) { - return !value.empty() && std::all_of(value.begin(), value.end(), [](char c) { return c >= '0' && c <= '9'; }); + return isIncarnationValue(type, value); } -std::optional ObjectStorageBackend::nativeHead(const String & key) +std::optional ObjectStorageBackend::nativeHead( + const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) { - auto metadata = object_storage->tryGetObjectMetadataWithNativeToken(key, /*with_tags=*/false); + auto metadata = object_storage->tryGetObjectMetadataWithNativeToken(key, /*with_tags=*/false, profile, timeout_ms); if (!metadata) return std::nullopt; - HeadResult hr; - hr.exists = true; - hr.size = metadata->size_bytes; - hr.token = tokenForHead(metadata->etag); - /// A generation-token store guarantees a numeric x-goog-generation on every successful HEAD; - /// a missing or non-numeric value (a proxy dropping the header, a service regression) means the - /// ordinary ETag fell through unmapped. There is no follow-up HEAD to patch this over, so surface - /// the failure here rather than minting a token that would poison the first conditional operation - /// that trusts it -- exactly the contract tokenFromWriteResult already enforces on the write path. - if (native_token_type == TokenType::Generation && !isValidGenerationTokenValue(hr.token.value)) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS on GCS: a HEAD of {} succeeded but its response carried no valid generation ({})", - key, metadata->etag); - hr.attributes = ObjectMeta(metadata->attributes.begin(), metadata->attributes.end()); - return hr; + /// Normalized (a generation arrives quoted through the SDK's ETag field) and otherwise as the + /// store gave it. Whether it IS an incarnation is judged where the answer can be acted on. + return RawMeta{metadata->size_bytes, normalizeTokenValue(metadata->etag)}; } /// Finalize a conditional write (the condition rode on the buffer's WriteSettings) and map a @@ -230,22 +226,20 @@ static PutOutcome finalizeConditionalWriteInstrumented(WriteBuffer & buf) /// Issue a conditional PUT (the condition rides on `ws`) and map a precondition loss — see /// finalizeConditionalWrite. The condition is checked by the backend when the object is completed, /// so the precondition loss always surfaces from the buffer's finalize, never from write. -PutResult ObjectStorageBackend::nativeConditionalPut(const String & key, const String & bytes, const WriteSettings & ws, const ObjectMeta & meta) +std::expected ObjectStorageBackend::nativeConditionalPut( + const String & key, const String & bytes, const WriteSettings & ws) { - std::optional attrs; - if (!meta.empty()) - attrs.emplace(meta.begin(), meta.end()); /// ObjectMeta is the same map type as ObjectAttributes auto buf = object_storage->writeObject( - StoredObject(key), WriteMode::Rewrite, attrs, DBMS_DEFAULT_BUFFER_SIZE, ws); + StoredObject(key), WriteMode::Rewrite, /*attributes=*/std::nullopt, DBMS_DEFAULT_BUFFER_SIZE, ws); buf->write(bytes.data(), bytes.size()); if (finalizeConditionalWriteInstrumented(*buf) == PutOutcome::PreconditionFailed) - return {PutOutcome::PreconditionFailed, {}}; + return std::unexpected(RawConflict{}); - /// Attribute the token of the incarnation WE just wrote (model WCreate) -- see - /// tokenFromWriteResult for the exact generation-vs-ETag policy. The S3 write returns its object - /// ETag/generation in the PutObject/CompleteMultipartUpload response, so no follow-up HEAD is - /// needed for most backends — this is ~73% of the CA backend's HEADs. - return {PutOutcome::Done, tokenFromWriteResult(key, buf->getResultObjectETag())}; + /// The response's own value for what it just wrote, normalized and otherwise untouched. An S3 + /// write carries its object ETag/generation in the PutObject/CompleteMultipartUpload response, so + /// no follow-up HEAD is needed -- and when it carries none, an empty value is the honest answer: + /// the write may have landed, which only the caller can resolve by reading the key back. + return normalizeTokenValue(buf->getResultObjectETag().value_or(String{})); } namespace @@ -253,20 +247,26 @@ namespace } -/// True when an exception from `IObjectStorage::readObject` means "the object is simply not there". +/// True when an exception from a read means "the KEY is simply not there". /// Two surfaces: -/// 1. S3/RustFS: `S3Exception` with `S3Errors::NO_SUCH_KEY` (the modeled enum — the primary -/// signal) or `getExceptionName() == "NoSuchKey"` (the canonical XML `` string, present -/// when the SDK was able to parse it; mirrors `finalizeConditionalWrite`'s detection). +/// 1. S3/RustFS: `S3Exception` with `S3Errors::NO_SUCH_KEY` (the modeled enum — the primary +/// signal), `getExceptionName() == "NoSuchKey"` (the canonical XML `` string, present when +/// the SDK was able to parse it; mirrors `finalizeConditionalWrite`'s detection), or +/// `RESOURCE_NOT_FOUND`, the generic code the SDK derives from a 404 whose body it could not +/// parse into a name. /// 2. Local / emulated: `DB::Exception` with `ErrorCodes::FILE_DOESNT_EXIST` (from /// `ReadBufferFromFile` when `open(2)` returns ENOENT). /// +/// `NO_SUCH_BUCKET` is deliberately NOT here even though it is the third member of the store's own +/// 404 family: a vanished CONTAINER is not an absent key, and answering "absent" for it would let a +/// caller read an empty pool out of an outage. It propagates, and `probeSentinelRaw` classifies it. /// Any other error (network, auth, throttle, corruption) propagates unchanged — fail-closed. static bool isObjectNotFound(const std::exception & e) { #if USE_AWS_S3 if (const auto * s3e = dynamic_cast(&e)) return s3e->getS3ErrorCode() == Aws::S3::S3Errors::NO_SUCH_KEY + || s3e->getS3ErrorCode() == Aws::S3::S3Errors::RESOURCE_NOT_FOUND || s3e->getExceptionName() == "NoSuchKey"; #endif if (const auto * dbe = dynamic_cast(&e)) @@ -274,54 +274,20 @@ static bool isObjectNotFound(const std::exception & e) return false; } -/// Read `range` of the object at `path` as a TRUE ranged read: seek to the offset and bound the -/// read window. Seek the storage buffer to the requested offset and bound the returned bytes instead -/// of reading a whole snapshot run and slicing it afterward; snapshot runs can be gigabytes at scale, -/// while the caller's memory budget is O(block). -static String readObjectRanged(IObjectStorage & object_storage, const String & path, Range range, - uint64_t known_size = 0) +/// Read the whole object at `path`. A caller that already knows the size passes it so the read +/// buffer is sized to the body instead of the storage's ~1 MiB default. +static String readWholeObject(IObjectStorage & object_storage, const String & path, uint64_t known_size = 0) { auto buf = object_storage.readObject( StoredObject(path), casSizedReadSettings(getReadSettings(), known_size), /*read_hint=*/std::nullopt); String content; - if (range.whole()) - { - readStringUntilEOF(content, *buf); - return content; - } - - /// An offset at or past EOF yields an empty result, matching the range contract of the previous - /// whole-read implementation. - /// `seek` past the object size may throw depending on the storage, so fail-close the window - /// against the known size before touching the buffer position. - /// Native callers already HEAD the key, so passing its size avoids another metadata round trip. - /// A zero size means the caller does not know it and metadata must be fetched here. - const uint64_t object_size = known_size != 0 ? known_size - : object_storage.getObjectMetadata(path, /*with_tags=*/false).size_bytes; - if (range.offset >= object_size) - return {}; - - /// The readable window, clamped to EOF. `setReadUntilPosition` is only a hint (not every object - /// storage honors it — LocalObjectStorage does not), so the exact byte count below is what bounds - /// the read; the hint lets storages that DO honor it avoid over-fetching. - const uint64_t available = object_size - range.offset; - const uint64_t to_read = range.length.has_value() ? std::min(*range.length, available) : available; - - if (range.length.has_value()) - buf->setReadUntilPosition(range.offset + *range.length); - buf->seek(static_cast(range.offset), SEEK_SET); - - content.resize(to_read); - const size_t got = buf->read(content.data(), to_read); - content.resize(got); + readStringUntilEOF(content, *buf); return content; } /// Open a forward-only stream over `range` of the object at `path`, positioned at the window's first -/// byte and bounded to its last. Mirrors -/// `readObjectRanged`'s seek + bound, but RETURNS the buffer instead of draining it — the caller reads -/// at its own pace, so nothing is materialized whole. Returns nullptr when the offset is at or past EOF -/// (the empty-window clamp), matching the ranged-get contract. +/// byte and bounded to its last. Nothing is materialized whole: the caller reads at its own pace. An +/// offset at or past EOF yields an empty stream rather than an error. static std::unique_ptr openObjectRangedStream(IObjectStorage & object_storage, const String & path, Range range, uint64_t known_size = 0) { @@ -330,11 +296,9 @@ static std::unique_ptr openObjectRangedStream(IObjectStorage & objec if (range.whole()) return buf; - /// Clamp exactly like `readObjectRanged`: an offset at or past EOF yields an empty stream, and /// `seek` past the object size may throw depending on the storage, so fail-close against the known - /// size before touching the buffer position. - /// As in `readObjectRanged`, a caller-supplied size avoids another metadata round trip; zero means - /// that the size is unknown and must be fetched. + /// size before touching the buffer position. A caller-supplied size avoids another metadata round + /// trip; zero means that the size is unknown and must be fetched. const uint64_t object_size = known_size != 0 ? known_size : object_storage.getObjectMetadata(path, /*with_tags=*/false).size_bytes; if (range.offset >= object_size) @@ -459,17 +423,14 @@ bool ObjectStorageBackend::emuExists(const String & key) const return object_storage->exists(StoredObject(emuPath(key))); } -String ObjectStorageBackend::emuRead(const String & key, Range range) const +String ObjectStorageBackend::emuRead(const String & key) const { - return readObjectRanged(*object_storage, emuPath(key), range); + return readWholeObject(*object_storage, emuPath(key)); } -Token ObjectStorageBackend::emuWrite(const String & key, const String & bytes, const ObjectMeta & meta) +Token ObjectStorageBackend::emuWrite(const String & key, const String & bytes) { - std::optional attrs; - if (!meta.empty()) - attrs.emplace(meta.begin(), meta.end()); /// ObjectMeta is the same map type as ObjectAttributes - auto buf = object_storage->writeObject(StoredObject(emuPath(key)), WriteMode::Rewrite, attrs); + auto buf = object_storage->writeObject(StoredObject(emuPath(key)), WriteMode::Rewrite); buf->write(bytes.data(), bytes.size()); buf->finalize(); @@ -482,7 +443,7 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St if (object_storage->getType() != ObjectStorageType::Local) throw Exception( ErrorCodes::NOT_IMPLEMENTED, - "ObjectStorageBackend::publishBlob: atomic emulated publication requires local object storage"); + "ObjectStorageBackend::publish: atomic emulated publication requires local object storage"); const String destination_object = emuPath(key); const String temporary_object = destination_object + ".publish-" + toString(UUIDHelpers::generateV4()) + ".tmp"; @@ -506,7 +467,7 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St out->cancel(); throw Exception( ErrorCodes::CORRUPTED_DATA, - "ObjectStorageBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", + "ObjectStorageBackend::publish: source yielded {}{} payload bytes for {}, declared {} -- nothing was published", copy_result.has_excess ? "more than " : "", copy_result.copied, key, @@ -554,11 +515,20 @@ Token ObjectStorageBackend::emuMintToken(const String & key, const String & etag { emuPruneTokenState(emuNowNs()); - /// Anomalous: the object storage reported no etag at all (LocalObjectStorage always does; this - /// guards a hypothetical future/test double). Mint a fresh, UNPERSISTED value — never worse than - /// the old counter for this case, but never masquerading as a real etag-derived identity. + /// The object storage identified the object with nothing at all (LocalObjectStorage always + /// reports its mtime; this is the anomaly). There is no identity to hand out and none to invent: + /// a minted nonce would name an incarnation the store cannot recognise on the next conditional + /// request. A just-completed write is therefore unattributed -- it may have landed, and the + /// caller resolves that by reading back -- and an observation simply has nothing to report. if (etag.empty()) - return Token{std::to_string(++emu_seq), TokenType::Emulated}; + { + if (just_wrote) + throw Exception(ErrorCodes::CAS_WRITE_UNATTRIBUTED, + "CAS backend: the emulated store accepted a write of '{}' but reports no etag for it; " + "the write may have committed and must be resolved by reading back", key); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS backend: the emulated store reports no etag for '{}', so it names no incarnation", key); + } auto it = emu_token_state.find(key); if (it != emu_token_state.end() && it->second.first == etag) @@ -584,43 +554,45 @@ Token ObjectStorageBackend::emuMintToken(const String & key, const String & etag /// Backend interface /// ========================================================================================= -std::optional ObjectStorageBackend::get(const String & key, Range range) +ReadSettings ObjectStorageBackend::readSettingsFor(ObjectStorageRetryProfile profile, uint64_t timeout_ms) const +{ + ReadSettings rs = getReadSettings(); + /// Mark the request for the store's native conditional dialect, so a GCS read is answered with a + /// generation rather than an MD5-shaped ETag. + rs.object_storage_request_mode = ObjectStorageRequestMode::NativeConditional; + rs.object_storage_retry_profile = profile; + rs.object_storage_attempt_timeout_ms = timeout_ms; + return rs; +} + +std::optional ObjectStorageBackend::read(const String & key, TransportAccess &) +{ + return readUnder(key, controlPlaneProfile(), attempt_timeout_ms); +} + +std::optional ObjectStorageBackend::readUnder( + const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) { if (mode == Mode::Native) { - auto hr = nativeHead(key); - if (!hr) - return std::nullopt; - - /// The object may be deleted between the HEAD above and the GET below (a GC or concurrent - /// writer racing the read window). Catch the not-found signal and honor the `optional` - /// contract — callers such as `Pool::loadShardDecoded` already handle a nullopt return and - /// treat it as "raced a deletion, absent". Any other error (network, auth, corruption) - /// propagates unchanged — fail-closed by construction. - /// - /// A REPLACEMENT racing the same window (HEAD observes token A, GET reads the bytes of a - /// subsequently-written incarnation B) is likewise not a hazard: HEAD strictly precedes GET, so - /// the returned token is never NEWER than the returned bytes — a mixed pair is always - /// (bytes_newer, token_older), never the reverse. Every consumer of this token uses it as a - /// conditional precondition (`casPut`/`putOverwrite`/`deleteExact`), which fails closed EXACTLY - /// in the mixed case, so a stale token costs a retry, never lets a caller act on a - /// bytes/token pair that never coexisted. This also covers `known_size`: content-addressed blob - /// bodies are byte-identical across incarnations (a "replacement" only rotates envelope/token), - /// mutable control objects are read-modify-CAS loops that re-validate on conflict, and write-once - /// objects self-validate their contents on decode. - GetResult gr; + /// ONE request: an S3 GET answers with the incarnation of the bytes it returned, so no HEAD + /// is needed to name them and no HEAD-to-GET window exists in which the two could disagree. + /// The value is returned UNVALIDATED -- `CasRequests` is where a response value becomes an + /// incarnation, and it is the one place that can decide what a malformed one means. try { - gr.bytes = readObjectRanged(*object_storage, key, range, hr->size); + auto got = object_storage->readSmallObjectAndGetObjectMetadata( + StoredObject(key), readSettingsFor(profile, timeout_ms), casMaxStoredObjectBytes()); + return Raw{std::move(got.data), normalizeTokenValue(got.metadata.etag)}; } catch (const std::exception & e) { + /// The object is simply not there; every other error (network, auth, corruption) + /// propagates unchanged -- fail-closed by construction. if (isObjectNotFound(e)) return std::nullopt; throw; } - gr.token = hr->token; - return gr; } std::lock_guard lock(emu_mutex); @@ -631,10 +603,10 @@ std::optional ObjectStorageBackend::get(const String & key, Range ran /// caller in this process can delete the file in between. External deletion (e.g. a test teardown /// racing a read) is still handled: convert FILE_DOESNT_EXIST to nullopt rather than letting it /// escape as an unexplained exception. - GetResult gr; + Raw raw; try { - gr.bytes = emuRead(key, range); + raw.bytes = emuRead(key); } catch (const std::exception & e) { @@ -642,21 +614,21 @@ std::optional ObjectStorageBackend::get(const String & key, Range ran return std::nullopt; throw; } - gr.token = emuObserveToken(key); - return gr; + raw.value = emuObserveToken(key).value; + return raw; } std::optional ObjectStorageBackend::getStream(const String & key, Range range) { if (mode == Mode::Native) { - auto hr = nativeHead(key); + auto hr = nativeHead(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); if (!hr) return std::nullopt; - /// Same HEAD-then-read race as `get`: the object may be deleted between the HEAD above and the - /// stream open below. Honor the `optional` contract on a not-found signal; any other error - /// (network, auth, corruption) propagates unchanged — fail-closed by construction. + /// The object may be deleted between the HEAD above and the stream open below. Honor the + /// `optional` contract on a not-found signal; any other error (network, auth, corruption) + /// propagates unchanged — fail-closed by construction. GetStreamResult sr; try { @@ -668,7 +640,7 @@ std::optional ObjectStorageBackend::getStream(const String & ke return std::nullopt; throw; } - sr.token = hr->token; + sr.token = legacyMintObserved(key, hr->value); return sr; } @@ -676,7 +648,7 @@ std::optional ObjectStorageBackend::getStream(const String & ke if (!emuExists(key)) return std::nullopt; - /// The emulated path holds emu_mutex across the exists-check and the stream open, matching `get`. + /// The emulated path holds emu_mutex across the exists-check and the stream open, matching `read`. /// External deletion still converts to nullopt rather than escaping as an unexplained exception. GetStreamResult sr; try @@ -693,69 +665,102 @@ std::optional ObjectStorageBackend::getStream(const String & ke return sr; } -HeadResult ObjectStorageBackend::head(const String & key) +std::unique_ptr ObjectStorageBackend::stream(const String & key, TransportAccess &) { - if (mode == Mode::Native) + /// No HEAD: this is ONE request, and the caller reserved one. Opening an object-storage buffer + /// issues nothing by itself, so the first GET is forced HERE -- otherwise the request that finds + /// the object absent, or fails, would happen after the call returned and be accounted to no + /// attempt at all. A present but empty object still yields a buffer; only a not-found is null. + /// + /// The buffer carries the storage's ORDINARY read settings, not this mount's control-plane + /// profile. `ReadBufferFromS3::nextImpl` re-reads `max_single_read_retries` on every `next()`, so + /// opening under SingleAttempt would strip the SDK's retries from the whole BODY -- which the + /// caller reads at its own pace, long after this attempt returned -- and stretch a single + /// attempt's timeout across the entire transfer. Only the open is the caller's to bound. Nor is + /// the request marked NativeConditional: a stream observes no incarnation to answer with. + try + { + std::unique_ptr buf; + if (mode == Mode::Native) + buf = object_storage->readObject(StoredObject(key), getReadSettings(), /*read_hint=*/std::nullopt); + else + { + /// Same lock the other emulated paths take, held across the open alone: there is no token + /// to observe here, so nothing else needs to be serialized with it. + std::lock_guard lock(emu_mutex); + buf = object_storage->readObject(StoredObject(emuPath(key)), getReadSettings(), /*read_hint=*/std::nullopt); + } + buf->nextIfAtEnd(); + return buf; + } + catch (const std::exception & e) { - auto hr = nativeHead(key); - return hr ? *hr : HeadResult{}; + if (isObjectNotFound(e)) + return nullptr; + throw; } +} + +std::optional ObjectStorageBackend::head(const String & key, TransportAccess &) +{ + return headUnder(key, controlPlaneProfile(), attempt_timeout_ms); +} + +std::optional ObjectStorageBackend::headUnder( + const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) +{ + if (mode == Mode::Native) + return nativeHead(key, profile, timeout_ms); std::lock_guard lock(emu_mutex); if (!emuExists(key)) - return HeadResult{}; + return std::nullopt; auto metadata = object_storage->tryGetObjectMetadata(emuPath(key), /*with_tags=*/false); /// A path that exists on the Local filesystem but yields no object metadata is a directory, not /// an object (`tryGetObjectMetadata` returns nullopt for a directory). HEAD must report it as - /// not-an-object (exists=false) — otherwise existsFile/getStorageObjects treat a pool sub-dir (e.g. - /// `store`, traversed by system.remote_data_paths) as a file and a later body read throws EISDIR. + /// not-an-object — otherwise existsFile/getStorageObjects treat a pool sub-dir (e.g. `store`, + /// traversed by system.remote_data_paths) as a file and a later body read throws EISDIR. if (!metadata) - return HeadResult{}; - HeadResult hr; - hr.exists = true; - hr.size = metadata->size_bytes; - hr.attributes = ObjectMeta(metadata->attributes.begin(), metadata->attributes.end()); - hr.token = emuObserveToken(key); - return hr; + return std::nullopt; + return RawMeta{metadata->size_bytes, emuObserveToken(key).value}; } /// See Backend::probeSentinelRaw / CasBackend.h's ProbeOutcome for the semantics this classifies. +SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key, TransportAccess &) +{ + return probeSentinelUnder(key, controlPlaneProfile(), attempt_timeout_ms); +} + SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key) +{ + return probeSentinelUnder(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); +} + +SentinelProbeResult ObjectStorageBackend::probeSentinelUnder( + const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) { if (mode == Mode::Native) { try { - /// `getObjectMetadata` (unlike `tryGetObjectMetadata`/`nativeHead`) is the THROWING raw-HEAD - /// primitive — it does NOT collapse NO_SUCH_KEY/NO_SUCH_BUCKET/RESOURCE_NOT_FOUND into one - /// `nullopt` before we get a chance to classify the S3 error. Its result is discarded here; - /// only whether (and how) it throws matters — the body comes from `get` below. - object_storage->getObjectMetadata(key, /*with_tags=*/false); - - /// The raw HEAD proved the key present. Delegate the body read to the existing `get`, which - /// already HEADs again and reads — an extra round trip this authoritative, low-rate probe can - /// afford, in exchange for reusing its already-correct HEAD→GET race handling. Kept INSIDE - /// this try: a transient failure here must also classify Indeterminate, never escape unclassified. - auto g = get(key); - if (!g) - return {ProbeOutcome::KeyAbsent, std::nullopt}; /// raced a deletion right after the raw HEAD - return {ProbeOutcome::Present, std::move(g->bytes)}; + /// One `read`: unlike a bodyless HEAD 404, a GET 404 carries a response body, so the + /// SDK can parse its `` and a missing key and a missing bucket arrive as different + /// errors -- which is the whole distinction this probe exists to make. + auto raw = readUnder(key, profile, timeout_ms); + if (!raw) + return {ProbeOutcome::KeyAbsent, std::nullopt}; + return {ProbeOutcome::Present, std::move(raw->bytes)}; } #if USE_AWS_S3 catch (const S3Exception & e) { + /// `read` already answers the key-absent half of the store's 404 family (see + /// `isObjectNotFound`), so what reaches here is what it deliberately does not flatten. + /// The limit of the one-read shape: every case below classifies an error a GET raised, so + /// a store whose HEAD and GET answer differently for the same key is classified by its GET. switch (e.getS3ErrorCode()) { - case Aws::S3::S3Errors::NO_SUCH_KEY: - return {ProbeOutcome::KeyAbsent, std::nullopt}; - case Aws::S3::S3Errors::RESOURCE_NOT_FOUND: - /// A HEAD response carries no body, so the SDK cannot parse a `NoSuchKey` `` - /// and instead derives this generic code straight from the HTTP 404 status (see - /// `isNotFoundError`, `src/IO/S3/getObjectInfo.cpp`) — this is what a REAL S3 HEAD - /// on an absent key actually throws. The container/key distinction is deliberately - /// NOT attempted here (a bodyless 404 cannot carry it). - return {ProbeOutcome::KeyAbsent, std::nullopt}; case Aws::S3::S3Errors::NO_SUCH_BUCKET: return {ProbeOutcome::ContainerAbsent, std::nullopt}; case Aws::S3::S3Errors::ACCESS_DENIED: @@ -773,7 +778,7 @@ SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key) } } - /// EmulatedSingleProcess (Local): stat the configured container directory FIRST — `emuExists`/`get` + /// EmulatedSingleProcess (Local): stat the configured container directory FIRST — `emuExists`/`read` /// alone cannot distinguish "this key is absent" from "the whole pool directory is gone" (Local /// listing is best-effort and silently reports zero either way, see LocalObjectStorage::listObjects). try @@ -781,10 +786,10 @@ SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key) if (!object_storage->existsOrHasAnyChild(emu_root)) return {ProbeOutcome::ContainerAbsent, std::nullopt}; - auto g = get(key); - if (!g) + auto raw = readUnder(key, profile, timeout_ms); + if (!raw) return {ProbeOutcome::KeyAbsent, std::nullopt}; - return {ProbeOutcome::Present, std::move(g->bytes)}; + return {ProbeOutcome::Present, std::move(raw->bytes)}; } catch (...) { @@ -814,74 +819,52 @@ WriteSettings ObjectStorageBackend::conditionalWriteSettings() const /// profile to its own single-attempt client. A backend that cannot honor it is rejected for /// writable Native mounts by checkConditionalWriteSingleAttemptSupport (fail closed). ws.object_storage_retry_profile = ObjectStorageRetryProfile::SingleAttempt; + /// And that attempt is bounded by the same budget the caller reserved for it. Without this the + /// write would run under the storage's own timeout while its caller waited on a shorter one. + ws.object_storage_attempt_timeout_ms = attempt_timeout_ms; return ws; } -/// See the declaration in the header for the policy. Centralizes the generation-vs-ETag attribution -/// decision for all successful conditional non-blob writes, including create-if-absent artifacts -/// and conditional replacements. -/// -/// The strict Generation-dialect check below is gated on `etag.has_value()`, not merely on -/// `native_token_type`: `WriteBufferFromS3` unconditionally assigns `object_etag = outcome.GetResult().GetETag()` -/// on BOTH of its success paths -- `makeSinglepartUpload` (WriteBufferFromS3.cpp) and -/// `completeMultipartUpload` (WriteBufferFromS3.cpp) -- so a successful S3 write always leaves -/// `getResultObjectETag()` holding a value, empty string included; `has_value()` is exactly "this was -/// a real S3-style write response", the only case Step 7's "a missing x-goog-generation is an -/// exception" rule is ABOUT. `S3ObjectStorage::writeObject` returns that `WriteBufferFromS3` directly, -/// undecorated, so this holds for the whole CAS-over-S3 write path with no wrapping in between. A -/// backend with no write-time-token concept at all (local files, or a non-S3 `IObjectStorage` -/// exercising Generation dialect purely for a unit test, see -/// `CASBackendGeneration.StampedTokenTypeFollowsNativeKind`) reports `nullopt` structurally, not a -/// broken response, and keeps falling back to a fresh HEAD exactly like the ETag dialect. A future -/// change that wraps the returned write buffer in a decorator would need to re-derive or preserve this -/// chain -- `WriteBufferFromFileDecorator::getResultObjectETag` returns `nullopt` for a wrapped impl -/// that is not itself a `WriteBufferFromFileBase`, which would silently turn a hard failure back into -/// a HEAD fallback. -/// -Token ObjectStorageBackend::tokenFromWriteResult(const String & key, const std::optional & etag) +std::expected ObjectStorageBackend::write( + const String & key, const String & bytes, const std::optional & expected_value, TransportAccess &) { - if (native_token_type == TokenType::Generation && etag.has_value()) - { - /// Validate the MINTED value, not the raw one: the HTTP boundary presents the generation - /// through the SDK's ETag field and therefore quotes it, and `tokenForHead` is what strips - /// that transport syntax. Validating before the strip would reject every real GCS write. - /// The message still reports the raw arrival, since that is what needs diagnosing. - const Token token = tokenForHead(*etag); - if (!isValidGenerationTokenValue(token.value)) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS on GCS: a conditional write to {} succeeded but its response carried no " - "valid generation ({}) -- there is no follow-up HEAD to patch this over, so the write " - "cannot be attributed to an incarnation", - key, *etag); - return token; - } - - /// ETag dialect (and any backend with no write-time token at all, e.g. local files): unchanged - /// pre-existing behavior -- an absent/empty value falls back to a fresh HEAD of `key`. - if (etag && !etag->empty()) - return tokenForHead(*etag); + /// An empty, wildcard or list value would turn the precondition into an unconditional write -- + /// refuse it as a caller bug before anything else runs. + if (expected_value && !isValidTokenValue(dialect(), *expected_value)) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS backend: refusing a conditional mutation of '{}' with a malformed token '{}' (dialect {}): " + "an empty, wildcard or list token would turn the precondition into an unconditional write", + key, *expected_value, static_cast(dialect())); - auto hr = nativeHead(key); - return hr ? hr->token : Token{}; -} - -PutResult ObjectStorageBackend::putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) -{ if (mode == Mode::Native) { WriteSettings ws = conditionalWriteSettings(); - ws.object_storage_write_if_none_match = "*"; - return nativeConditionalPut(key, bytes, ws, meta); + if (expected_value) + ws.object_storage_write_if_match = *expected_value; + else + ws.object_storage_write_if_none_match = "*"; + return nativeConditionalPut(key, bytes, ws); } std::lock_guard lock(emu_mutex); - if (emuExists(key)) - return {PutOutcome::PreconditionFailed, {}}; + const bool exists = emuExists(key); + if (!expected_value) + { + if (exists) + return std::unexpected(RawConflict{}); + } + else + { + if (!exists) + return std::unexpected(RawConflict{}); + if (!tokenMatches(emuObserveToken(key), Token{*expected_value, TokenType::Emulated})) + return std::unexpected(RawConflict{}); + } - return {PutOutcome::Done, emuWrite(key, bytes, meta)}; + return emuWrite(key, bytes).value; } -void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) +void ObjectStorageBackend::publish(const BlobPublishRequest & request, TransportAccess &) { if (const auto * streaming = std::get_if(&request.publication)) { @@ -889,7 +872,7 @@ void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) if (!payload) throw Exception( ErrorCodes::CORRUPTED_DATA, - "ObjectStorageBackend::publishBlob: payload source for {} returned no reader", + "ObjectStorageBackend::publish: payload source for {} returned no reader", request.destination_key); if (mode != Mode::Native) @@ -924,7 +907,7 @@ void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) out->cancel(); throw Exception( ErrorCodes::CORRUPTED_DATA, - "ObjectStorageBackend::publishBlob: source yielded {}{} payload bytes for {}, declared {} -- upload aborted, nothing published", + "ObjectStorageBackend::publish: source yielded {}{} payload bytes for {}, declared {} -- upload aborted, nothing published", copy_result.has_excess ? "more than " : "", copy_result.copied, request.destination_key, @@ -938,14 +921,14 @@ void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) if (mode != Mode::Native) throw Exception( ErrorCodes::NOT_IMPLEMENTED, - "ObjectStorageBackend::publishBlob: verbatim staged publication requires Native mode"); + "ObjectStorageBackend::publish: verbatim staged publication requires Native mode"); WriteSettings write_settings; write_settings.object_storage_copy_mode = ObjectStorageCopyMode::NativeOnly; if (!object_storage->supportsCopyMode(write_settings.object_storage_copy_mode)) throw Exception( ErrorCodes::NOT_IMPLEMENTED, - "ObjectStorageBackend::publishBlob: object storage {} does not support native-only same-store copy", + "ObjectStorageBackend::publish: object storage {} does not support native-only same-store copy", object_storage->getName()); object_storage->copyObject( @@ -955,123 +938,52 @@ void ObjectStorageBackend::publishBlob(const BlobPublishRequest & request) write_settings); } -PutResult ObjectStorageBackend::putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) +Backend::RawRemoval ObjectStorageBackend::remove(const String & key, const String & expected_value, TransportAccess &) { - /// §3.18 №19: reject a wrong-dialect expected token before it ever reaches the wire (Native) or - /// the emu compare (Emulated) — see mintingTypeMatches. - if (!mintingTypeMatches(expected.type)) - return {PutOutcome::PreconditionFailed, {}}; - - if (mode == Mode::Native) - { - WriteSettings ws = conditionalWriteSettings(); - ws.object_storage_write_if_match = expected.value; - return nativeConditionalPut(key, bytes, ws, meta); - } - - std::lock_guard lock(emu_mutex); - if (!emuExists(key)) - return {PutOutcome::PreconditionFailed, {}}; - if (!tokenMatches(emuObserveToken(key), expected)) - return {PutOutcome::PreconditionFailed, {}}; - - return {PutOutcome::Done, emuWrite(key, bytes, meta)}; -} - -CasResult ObjectStorageBackend::casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) -{ - /// §3.18 №19: a create-if-absent CAS (expected == nullopt) has no token to validate; only the - /// swap form carries one, and it must match this backend's own minting dialect before anything - /// else runs. - if (expected.has_value() && !mintingTypeMatches(expected->type)) - return {CasOutcome::Conflict, {}}; - - if (mode == Mode::Native) - { - WriteSettings ws = conditionalWriteSettings(); - if (expected.has_value()) - ws.object_storage_write_if_match = expected->value; - else - ws.object_storage_write_if_none_match = "*"; - - /// The PUT-side outcomes (Done / PreconditionFailed) collapse onto CAS outcomes 1:1: a lost - /// condition — whether a mismatched If-Match or a 404 on an If-Match PUT — is a Conflict. - PutResult put = nativeConditionalPut(key, bytes, ws, meta); - return put.outcome == PutOutcome::Done - ? CasResult{CasOutcome::Committed, put.token} - : CasResult{CasOutcome::Conflict, {}}; - } - - std::lock_guard lock(emu_mutex); - const bool exists = emuExists(key); - - if (!expected.has_value()) - { - if (exists) - return {CasOutcome::Conflict, {}}; - } - else - { - if (!exists) - return {CasOutcome::Conflict, {}}; - if (!tokenMatches(emuObserveToken(key), *expected)) - return {CasOutcome::Conflict, {}}; - } - - return {CasOutcome::Committed, emuWrite(key, bytes, meta)}; + return removeUnder(key, expected_value, controlPlaneProfile(), attempt_timeout_ms); } -DeleteOutcome ObjectStorageBackend::deleteExact(const String & key, const Token & token) +Backend::RawRemoval ObjectStorageBackend::removeUnder( + const String & key, const String & expected_value, ObjectStorageRetryProfile profile, uint64_t timeout_ms) { - /// §3.18 №19: same local dialect guard as putOverwrite/casPut — never forward a foreign-dialect - /// value as the removeObjectIfTokenMatches argument. - if (!mintingTypeMatches(token.type)) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::TokenMismatch; - return d; - } + /// Same grammar guard as `write`, and for the same reason: an empty, wildcard or list value would + /// turn the condition into an unconditional delete. + if (!isValidTokenValue(dialect(), expected_value)) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS backend: refusing a conditional mutation of '{}' with a malformed token '{}' (dialect {}): " + "an empty, wildcard or list token would turn the precondition into an unconditional write", + key, expected_value, static_cast(dialect())); if (mode == Mode::Native) { - /// `removeObjectIfTokenMatches` maps onto `DeleteOutcome` one-to-one. `NOT_IMPLEMENTED` from a - /// backend that does not enforce conditional removal propagates — fail-closed by construction. - auto result = object_storage->removeObjectIfTokenMatches(StoredObject(key), token.value); - DeleteOutcome d; - d.created_delete_marker = result.created_delete_marker; + /// `NOT_IMPLEMENTED` from a storage that does not enforce conditional removal propagates — + /// fail-closed by construction. + auto result = object_storage->removeObjectIfTokenMatches(StoredObject(key), expected_value, profile, timeout_ms); switch (result.outcome) { case ConditionalRemoveOutcome::Removed: - d.kind = DeleteOutcome::Kind::Deleted; - break; + /// A delete marker means the storage archived a noncurrent version instead of + /// reclaiming the current object -- a removal that did not reclaim. + return result.created_delete_marker ? RawRemoval::DeleteMarker : RawRemoval::Removed; case ConditionalRemoveOutcome::TokenMismatch: - d.kind = DeleteOutcome::Kind::TokenMismatch; - break; + return RawRemoval::Mismatch; case ConditionalRemoveOutcome::NotFound: - d.kind = DeleteOutcome::Kind::NotFound; - break; + return RawRemoval::Gone; } - return d; + UNREACHABLE(); } std::lock_guard lock(emu_mutex); - DeleteOutcome d; if (!emuExists(key)) - { - d.kind = DeleteOutcome::Kind::NotFound; - return d; - } - if (!tokenMatches(emuObserveToken(key), token)) - { - d.kind = DeleteOutcome::Kind::TokenMismatch; - return d; - } + return RawRemoval::Gone; + if (!tokenMatches(emuObserveToken(key), Token{expected_value, TokenType::Emulated})) + return RawRemoval::Mismatch; object_storage->removeObjectIfExists(StoredObject(emuPath(key))); /// Keep the deleted incarnation's last-minted etag around ONLY while a same-mtime-quantum /// collision with an immediate recreate is still possible (emuMintToken) — once it is /// comfortably old, erase it so `emu_token_state` does not grow for the lifetime of the backend - /// instance (codex-review-triage §3.18, Important #1). + /// instance. if (auto it = emu_token_state.find(key); it != emu_token_state.end()) { const uint64_t now_ns = emuNowNs(); @@ -1080,11 +992,16 @@ DeleteOutcome ObjectStorageBackend::deleteExact(const String & key, const Token else emu_token_expiry.push_back(EmuTokenExpiry{now_ns, key, it->second}); } - d.kind = DeleteOutcome::Kind::Deleted; - return d; + return RawRemoval::Removed; } -ListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit) +Backend::RawListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess &) +{ + return listUnder(prefix, cursor, limit, controlPlaneProfile(), attempt_timeout_ms); +} + +Backend::RawListPage ObjectStorageBackend::listUnder( + const String & prefix, const String & cursor, size_t limit, ObjectStorageRetryProfile profile, uint64_t timeout_ms) { /// Use the lazy object-storage iterator instead of `listObjects(..., max_keys=0)`: the latter /// materialized the whole prefix, then sliced client-side, so a paginated walk re-fetched the full @@ -1103,32 +1020,29 @@ ListPage ObjectStorageBackend::list(const String & prefix, const String & cursor object_storage->listObjects(physical_prefix, children, /*max_keys=*/0); /// Hold emu_mutex across the whole scan: emuMintToken below reads/updates emu_token_state, the - /// same per-key state get/head/put*/delete* mutate under this lock (see the "caller holds + /// same per-key state read/head/write/remove mutate under this lock (see the "caller holds /// emu_mutex" contract on the private emu* helpers). std::lock_guard lock(emu_mutex); - std::vector all; + std::vector all; all.reserve(children.size()); for (const auto & child : children) { if (!child->relative_path.starts_with(physical_prefix)) continue; - ListedKey lk; + RawListedKey lk; lk.key = child->relative_path.substr(strip.size()); lk.size = child->metadata ? child->metadata->size_bytes : 0; - /// §3.18 №18: mint DIRECTLY as TokenType::Emulated — do NOT call tokenForList, which always - /// stamps native_token_type (ETag/Generation) regardless of mode and would surface a token - /// of the wrong dialect for every Emulated consumer (head/get mint Emulated). if (child->metadata) - lk.token = emuMintToken(lk.key, child->metadata->etag, /*just_wrote=*/false); + lk.value = emuMintToken(lk.key, child->metadata->etag, /*just_wrote=*/false).value; all.push_back(std::move(lk)); } - std::sort(all.begin(), all.end(), [](const ListedKey & a, const ListedKey & b) { return a.key < b.key; }); + std::sort(all.begin(), all.end(), [](const RawListedKey & a, const RawListedKey & b) { return a.key < b.key; }); - ListPage page; + RawListPage page; auto all_it = cursor.empty() - ? std::lower_bound(all.begin(), all.end(), prefix, [](const ListedKey & a, const String & s) { return a.key < s; }) - : std::upper_bound(all.begin(), all.end(), cursor, [](const String & s, const ListedKey & a) { return s < a.key; }); + ? std::lower_bound(all.begin(), all.end(), prefix, [](const RawListedKey & a, const String & s) { return a.key < s; }) + : std::upper_bound(all.begin(), all.end(), cursor, [](const String & s, const RawListedKey & a) { return s < a.key; }); while (all_it != all.end() && page.keys.size() < limit) { page.keys.push_back(*all_it); @@ -1143,26 +1057,28 @@ ListPage ObjectStorageBackend::list(const String & prefix, const String & cursor ? std::nullopt : std::optional(cursor); - ListPage page; - auto it = object_storage->iterate(physical_prefix, /*max_keys=*/0, /*with_tags=*/false, start_after); + RawListPage page; + auto it = object_storage->iterate(physical_prefix, /*max_keys=*/0, /*with_tags=*/false, start_after, profile, timeout_ms); for (; it->isValid(); it->next()) { const auto child = it->current(); if (!child->relative_path.starts_with(physical_prefix)) continue; - ListedKey lk; + RawListedKey lk; lk.key = child->relative_path.substr(strip.size()); if (!cursor.empty() && lk.key <= cursor) continue; lk.size = child->metadata ? child->metadata->size_bytes : 0; - /// Surface the per-key incarnation token (matching what `head` would return, see above) so the - /// `supportsListTokens() == true` capability is honest. A listing without an etag leaves the - /// token unset, which GC discover treats as Read (fail closed). The supportsListTokens()+ - /// empty-etag gate now lives in tokenForList. + /// Surface the per-key incarnation value (matching what `head` would return) so the + /// `supportsListTokens() == true` capability is honest. A listing without an etag leaves it + /// unset, which GC discover treats as Read (fail closed). The supportsListTokens()+ + /// empty-etag gate lives in tokenForList; whether the value it passes IS an incarnation is + /// judged where the answer can be acted on, not here. if (child->metadata) - lk.token = tokenForList(child->metadata->etag); + if (const auto token = tokenForList(child->metadata->etag)) + lk.value = token->value; if (page.keys.size() == limit) { @@ -1175,4 +1091,61 @@ ListPage ObjectStorageBackend::list(const String & prefix, const String & cursor return page; } +/// ========================================================================================= +/// The legacy surface — see the declarations for why these are not the base's forwarders. +/// Every one of them issues its request under the storage's own retry profile, and mints through +/// the base's legacy mint so a malformed response is judged in exactly one place. +/// ========================================================================================= + +std::optional ObjectStorageBackend::get(const String & key, Range range) +{ + if (!range.whole()) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "CAS backend: a ranged get is retired; read the object whole"); + + auto raw = readUnder(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); + if (!raw) + return std::nullopt; + return GetResult{std::move(raw->bytes), legacyMintObserved(key, std::move(raw->value)), {}}; +} + +HeadResult ObjectStorageBackend::head(const String & key) +{ + auto raw = headUnder(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); + if (!raw) + return {}; + return HeadResult{true, raw->size, legacyMintObserved(key, std::move(raw->value)), {}}; +} + +ListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit) +{ + auto raw = listUnder(prefix, cursor, limit, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); + + ListPage page; + page.next_cursor = std::move(raw.next_cursor); + page.keys.reserve(raw.keys.size()); + for (auto & k : raw.keys) + { + std::optional token; + if (k.value) + token = legacyMintObserved(k.key, std::move(*k.value)); + page.keys.push_back(ListedKey{std::move(k.key), k.size, std::move(token)}); + } + return page; +} + +DeleteOutcome ObjectStorageBackend::deleteExact(const String & key, const Token & token) +{ + if (legacyTokenIsForeign(key, token)) + return DeleteOutcome{DeleteOutcome::Kind::TokenMismatch, false}; + + switch (removeUnder(key, token.value, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0)) + { + case RawRemoval::Removed: return {DeleteOutcome::Kind::Deleted, false}; + case RawRemoval::Gone: return {DeleteOutcome::Kind::NotFound, false}; + case RawRemoval::Mismatch: return {DeleteOutcome::Kind::TokenMismatch, false}; + case RawRemoval::DeleteMarker: return {DeleteOutcome::Kind::Deleted, true}; + } + UNREACHABLE(); +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index 7bf65f67849f..1c491058802d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -48,13 +48,13 @@ PutOutcome finalizeConditionalWrite(WriteBuffer & buf); class ObjectStorageBackend final : public Backend { public: - /// Unhide the base convenience overloads (omitted Range/ObjectMeta/expected-token forms): the - /// overrides below would otherwise shadow them for callers holding a concrete backend type. + /// Unhide the base overloads this class's own declarations would otherwise shadow: the + /// convenience forms that omit `Range`, and the keyed primitives that share a legacy name. using Backend::get; using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; + using Backend::head; + using Backend::list; + using Backend::probeSentinelRaw; enum class Mode { Native, EmulatedSingleProcess }; @@ -62,17 +62,65 @@ class ObjectStorageBackend final : public Backend /// operations and native token dialect; `EmulatedSingleProcess` serializes operations locally for /// tests and local development. A Native generation-token store must use a single PUT because /// its multipart completion path does not enforce the precondition. - ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_); + /// + /// `single_attempt_control_plane` selects the SingleAttempt retry profile for the READ-class + /// requests below: a writable Native mount owns its own retry policy and a transparently retried + /// request would outlive the caller's deadline, while a read-only mount has no such deadline and + /// keeps the storage's default. `attempt_timeout_ms` bounds ONE attempt of those requests; 0 + /// leaves the storage's own timeout in place. Both are supplied by the mount that opens the pool; + /// the defaults are what a narrow unit test constructing a bare backend gets. + ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_, + bool single_attempt_control_plane_ = false, uint64_t attempt_timeout_ms_ = 0); - /// Read an object or return `nullopt` if it is absent. Native mode HEADs first so the returned - /// token identifies the incarnation whose bytes are read; a not-found race is also reported as + /// Read the whole object, or return `nullopt` if it is absent. Native mode reads the incarnation + /// value out of the GET response itself, so no HEAD precedes it; a not-found race is reported as /// `nullopt`, while unrelated storage errors propagate. + std::optional read(const String & key, TransportAccess & access) override; + /// Return the current size and incarnation value, or `nullopt` when the key is absent. + std::optional head(const String & key, TransportAccess & access) override; + /// Return a page after `cursor`; the next cursor is the last returned key and is empty at the end. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override; + /// Remove only the incarnation named by `expected_value`, preserving the object on a mismatch and + /// reporting a versioned bucket's delete marker as the non-reclaiming removal it is. + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override; + /// Create the key (`expected_value == nullopt`) or replace exactly the incarnation it names. The + /// returned value is the write response's own; a response that carries none at all is + /// `CAS_WRITE_UNATTRIBUTED`, never patched over by a follow-up read. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override; + /// Open a forward-only whole-object stream for a write-once object, or null when the key is + /// absent. Nothing is materialized; mutable objects must use `read` because their contents may + /// change while the stream is open. + std::unique_ptr stream(const String & key, TransportAccess & access) override; + /// Execute the selected unconditional blob transport without observing destination state or + /// returning a write-response value. Streaming uses ordinary write settings; staged bytes require + /// a native same-store copy. + void publish(const BlobPublishRequest & request, TransportAccess & access) override; + /// Native mints its store's own dialect (ETag or GCS generation); the emulated adapter mints its + /// own values. + Dialect dialect() const override { return mode == Mode::Native ? native_token_type : Dialect::Emulated; } + /// The budget for one attempt of a read-class request, as configured by the mount. + uint64_t attemptTimeoutMs() const override { return attempt_timeout_ms; } + /// Ask the storage to re-acquire credentials through its refresh callback. + bool refreshCredentials() override { return object_storage->tryRefreshCredentialsViaCallback(); } + + /// ---- The legacy surface, kept for the migration window ---- + /// + /// Not inherited from `Backend`: its forwarders would issue these requests under the profile the + /// keyed primitives use, which on a writable Native mount is SingleAttempt. That is right for a + /// primitive -- `CasRequests` is the retry loop around it -- and wrong for a legacy caller, which + /// has no loop at all and would lose the storage's own retries on the first blip. These keep the + /// storage's default profile until their callers move onto the engine, and are deleted at the + /// lock. Each mints its `Token` through the base's legacy mint, so a malformed response is + /// judged in exactly one place. std::optional get(const String & key, Range range) override; - /// Open a forward-only ranged stream for a write-once object. The stream is not materialized in - /// memory; mutable objects must use `get` because their contents may change while it is open. - std::optional getStream(const String & key, Range range) override; - /// Return the current size, attributes, and incarnation token, or an absent `HeadResult`. HeadResult head(const String & key) override; + ListPage list(const String & prefix, const String & cursor, size_t limit) override; + DeleteOutcome deleteExact(const String & key, const Token & token) override; + SentinelProbeResult probeSentinelRaw(const String & key) override; + /// Open a forward-only ranged stream for a write-once object. See `Backend::getStream` for why it + /// is not a forwarder. + std::optional getStream(const String & key, Range range) override; /// S3 ETags are content-derived and surfaced in list responses — TRUE for ETag-token Native /// and EmulatedSingleProcess modes. FALSE on a generation-token store (GCS): the XML LIST /// surfaces MD5-style ETags in the response BODY, which the header-level response adaptation @@ -82,27 +130,6 @@ class ObjectStorageBackend final : public Backend /// shard — a cost, not a correctness change). bool supportsListTokens() const override { return native_token_type != TokenType::Generation; } - /// Create `key` only if it is absent. On a precondition failure the object is untouched and the - /// result has no token; on success the token identifies the newly written incarnation. - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override; - - /// Execute the selected unconditional blob transport without observing destination state or - /// returning a write-response token. Streaming uses ordinary write settings; staged bytes require - /// a native same-store copy. - void publishBlob(const BlobPublishRequest & request) override; - /// Replace `key` only when its current token exactly equals `expected`; a mismatch leaves the - /// existing incarnation untouched. Storage exceptions propagate instead of being reported as a - /// successful or failed precondition. - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override; - /// Perform a compare-and-set: `expected == nullopt` means create-if-absent. A conflict leaves the - /// object untouched; a committed result carries the new incarnation token. - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) override; - /// Remove only the incarnation matching `token`, preserving the object on a mismatch and exposing - /// whether the storage created a delete marker. - DeleteOutcome deleteExact(const String & key, const Token & token) override; - /// Return a page after `cursor`; the next cursor is the last returned key and is empty at the end. - ListPage list(const String & prefix, const String & cursor, size_t limit) override; - /// Pool-level precondition: on a Native, generation-dialect (GCS) backend, reject the pool when /// object versioning is verified ENABLED; warn and continue when the probe cannot answer — see /// Backend::checkPoolPreconditions. @@ -120,11 +147,12 @@ class ObjectStorageBackend final : public Backend /// `EmulatedSingleProcess`. void checkConditionalWriteSingleAttemptSupport() override; - /// See Backend::probeSentinelRaw. Native: a raw HEAD via `IObjectStorage::getObjectMetadata` (the - /// THROWING variant — unlike `tryGetObjectMetadata`/`nativeHead`, it never swallows the S3 error), - /// classified by S3 error code. EmulatedSingleProcess (Local): stats the configured container - /// directory (`emu_root`) first — `ContainerAbsent` if it is gone — then the key. - SentinelProbeResult probeSentinelRaw(const String & key) override; + /// See Backend::probeSentinelRaw. Native: ONE `read`, classified by the S3 error it throws. A GET + /// 404 carries a response body, so the SDK can parse its `` and tell `NoSuchKey` from + /// `NoSuchBucket` -- a distinction a bodyless HEAD 404 cannot make. + /// EmulatedSingleProcess (Local): stats the configured container directory (`emu_root`) first -- + /// `ContainerAbsent` if it is gone -- then the key. + SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override; /// The token kind this backend's object storage mints: TokenType::ETag for AWS-compatible /// stores, TokenType::Generation when the storage mints GCS generations (the @@ -151,12 +179,9 @@ class ObjectStorageBackend final : public Backend return etag; } - /// Mint the incarnation token for a key we just HEAD'd or wrote: the object ETag/generation - /// string carried under this backend's native dialect (native_token_type). - /// - /// This is the ONLY site that mints a Generation token: `tokenForList` is the sole other - /// `native_token_type` mint, and `supportsListTokens` above returns false for Generation, so it - /// cannot produce one. + /// The `Token` form of an observed ETag/generation: normalized, and stamped with this backend's + /// native dialect. The transport itself deals in bare values; this is the normalize-and-stamp + /// step on its own, with `tokenForList` as its LIST-side sibling. Token tokenForHead(const String & etag) const { return Token{normalizeTokenValue(etag), native_token_type}; @@ -179,20 +204,18 @@ class ObjectStorageBackend final : public Backend return observed == expected; } + /// The per-dialect grammar a response value must meet to be an incarnation. Generation: canonical + /// positive decimal AFTER the SDK ETag-field quote strip (no leading zero, not "0" — zero is the + /// dialect's absence sentinel). ETag: non-empty, not "*" after trimming whitespace, no comma (a + /// list matches any member). Emulated: non-empty. + static bool isValidTokenValue(TokenType type, const String & value); + /// Settings for a Native COMPARE/CREATE write (create-if-absent, compare-and-set): mark the request /// conditional, make exactly one attempt at every retry layer, skip the racy post-upload /// existence/size check, and force a single PUT on generation stores because GCS does not /// enforce the condition on multipart completion. WriteSettings conditionalWriteSettings() const; WriteSettings conditionalWriteSettingsForTest() const { return conditionalWriteSettings(); } - /// Convert a successful write/copy response's incarnation-identifying string into this backend's - /// token -- the ONE place that decides how strictly to trust it ("Exact successful-write token"). - /// Generation dialect (GCS): the response MUST carry a non-empty, purely numeric generation; a - /// missing or non-numeric value is an exception -- there is no follow-up HEAD, so a broken or - /// lying response can never be silently patched over by a later, unrelated read. Every other - /// dialect (ETag, and any backend with no write-time token at all, e.g. local files) keeps the - /// pre-existing behavior: an absent value falls back to a fresh HEAD of `key`. - Token tokenFromWriteResult(const String & key, const std::optional & etag); /// Override the emulated backend's wall clock for deterministic expiry tests. void setEmuNowNsForTest(uint64_t now_ns); /// Return the guarded per-key token-state size for expiry tests. @@ -202,6 +225,27 @@ class ObjectStorageBackend final : public Backend const ObjectStoragePtr object_storage; const Mode mode; TokenType native_token_type = TokenType::ETag; + /// See the constructor: what the READ-class requests (read, head, list, remove) carry. + const bool single_attempt_control_plane; + const uint64_t attempt_timeout_ms; + ObjectStorageRetryProfile controlPlaneProfile() const + { + return single_attempt_control_plane ? ObjectStorageRetryProfile::SingleAttempt : ObjectStorageRetryProfile::Default; + } + /// The read settings a request carries: the native conditional dialect, plus the retry profile and + /// per-attempt bound its caller is entitled to. + ReadSettings readSettingsFor(ObjectStorageRetryProfile profile, uint64_t timeout_ms) const; + + /// The bodies a keyed primitive and its legacy override share. They differ in one thing: the + /// keyed call passes `controlPlaneProfile(), attempt_timeout_ms`, the legacy one the storage's + /// defaults. The legacy arguments disappear with the legacy methods. + std::optional readUnder(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); + std::optional headUnder(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); + RawListPage listUnder(const String & prefix, const String & cursor, size_t limit, + ObjectStorageRetryProfile profile, uint64_t timeout_ms); + RawRemoval removeUnder(const String & key, const String & expected_value, + ObjectStorageRetryProfile profile, uint64_t timeout_ms); + SentinelProbeResult probeSentinelUnder(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); /// EmulatedSingleProcess state: per-key {etag, disambiguator} — see emuMintToken. A successfully /// deleted entry is retained only while its etag is recent enough that an immediate recreate could /// land in the same mtime quantum. `deleteExact` erases already-old entries immediately and queues @@ -218,26 +262,17 @@ class ObjectStorageBackend final : public Backend }; std::deque emu_token_expiry; uint64_t emu_now_ns_for_test = 0; - /// Fallback nonce for the (anomalous) case where the object storage reports an EMPTY etag: mints a - /// fresh, unpersisted value each time — never worse than the old counter for that case, but never - /// masquerading as a real etag-derived identity either. - uint64_t emu_seq = 0; - /// Look up Native metadata and convert the storage ETag or generation to this backend's token. On - /// a generation-token store, the minted token is validated exactly like a write result (see - /// isValidGenerationTokenValue) before this returns it: a missing/malformed x-goog-generation on an - /// otherwise-successful HEAD would otherwise mint an invalid token here with no check at all, one - /// layer before tokenFromWriteResult's own check on the write path. - std::optional nativeHead(const String & key); + /// Look up Native metadata and normalize the storage ETag or generation into an incarnation + /// value. The value is returned as the store gave it: whether it IS an incarnation is judged by + /// whoever can act on the answer, never here. + std::optional nativeHead(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); - /// True iff `value` is a well-formed generation: non-empty and every character an ASCII digit. - /// Shared by nativeHead and tokenFromWriteResult so the two places that mint a Generation token - /// from a remote response cannot drift apart on what "valid" means. Deliberately NOT folded into - /// tokenForHead, which stays a pure minter with no opinion on the value it is handed. - static bool isValidGenerationTokenValue(const String & value); - /// Write a body with the condition already encoded in `ws`, finalize it, classify a lost - /// precondition, and return the new token when the write succeeds. - PutResult nativeConditionalPut(const String & key, const String & bytes, const WriteSettings & ws, const ObjectMeta & meta); + /// Write a body with the condition already encoded in `ws`, finalize it, map a lost precondition + /// onto `RawConflict`, and return the write response's own value on success -- normalized, and + /// otherwise untouched: an empty one means the response named no incarnation, which is the + /// caller's to resolve, not this seam's to refuse. + std::expected nativeConditionalPut(const String & key, const String & bytes, const WriteSettings & ws); /// §3.18 №19 hardening: whether `t` is the dialect this backend itself mints (native_token_type /// for Native mode, always TokenType::Emulated for EmulatedSingleProcess). Every conditional @@ -258,10 +293,10 @@ class ObjectStorageBackend final : public Backend /// The caller holds `emu_mutex` for all five helpers below, preserving the exists/read and /// observe/write checks as one process-local operation. bool emuExists(const String & key) const; - String emuRead(const String & key, Range range) const; + String emuRead(const String & key) const; /// Write a body as the new incarnation of `key` and return its freshly minted token (the /// object's own post-write etag — see emuMintToken). - Token emuWrite(const String & key, const String & bytes, const ObjectMeta & meta); + Token emuWrite(const String & key, const String & bytes); /// Write a complete blob body to a sibling temporary local object, then atomically replace `key` /// and advance any existing same-ETag disambiguator. A failure before the rename leaves the old /// destination and its token state untouched and cleans the temporary. @@ -279,8 +314,9 @@ class ObjectStorageBackend final : public Backend /// Single source of truth for minting an emulated token from an observed `etag`: the wire value IS /// the etag while it is the first thing minted for `key` at that etag, or `etag#N` once a SAME-etag /// rewrite forces a disambiguator (`just_wrote` — see the mtime-quantum note in emu_token_state's - /// declaration and codex-review-triage §3.18 19c step 4). An empty `etag` (the storage could not - /// report one) falls back to a fresh, UNPERSISTED monotonic value from emu_seq. + /// declaration). An empty `etag` means the storage could not identify the object at all, and + /// there is nothing to invent from: a just-completed write cannot be attributed + /// (`CAS_WRITE_UNATTRIBUTED`) and an observation has no incarnation to report (`CORRUPTED_DATA`). Token emuMintToken(const String & key, const String & etag, bool just_wrote); }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp new file mode 100644 index 000000000000..04b5afc5b338 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -0,0 +1,870 @@ +#include + +#include +#include +#include +#include +#include + +#include "config.h" + +#if USE_AWS_S3 +#include +#endif + +#include + +#include + +#include +#include +#include + +namespace ProfileEvents +{ + extern const Event CASRequestAttempt; + extern const Event CASRequestReissue; + extern const Event CASRequestResolveRead; + extern const Event CASRequestGaveUp; + extern const Event CASRequestRefused; + extern const Event CASRequestFenceLostPostWrite; +} + +namespace DB::ErrorCodes +{ + extern const int BAD_ARGUMENTS; + extern const int CAS_DELETE_MARKER; + extern const int CORRUPTED_DATA; + extern const int LOGICAL_ERROR; + extern const int NOT_IMPLEMENTED; +} + +namespace DB::Cas +{ + +namespace detail +{ + +void recordAttempt() +{ + ProfileEvents::increment(ProfileEvents::CASRequestAttempt); +} + +void recordReissue() +{ + ProfileEvents::increment(ProfileEvents::CASRequestReissue); +} + +} + +namespace +{ + +/// `CLOCK_BOOTTIME` milliseconds -- the clock a mount lease deadline is expressed on, reproduced here +/// rather than shared because `Backend/` does not depend on the mount plane. +uint64_t bootClockMs() +{ + return clock_gettime_ns(CLOCK_BOOTTIME) / 1000000; +} + +uint64_t saturatingAdd(uint64_t lhs, uint64_t rhs) +{ + return lhs > std::numeric_limits::max() - rhs ? std::numeric_limits::max() : lhs + rhs; +} + +/// The failure class a fresh credential could fix, named here rather than taken from +/// `S3Exception::isAccessTokenExpiredError`, which also fires on `S3Errors::UNKNOWN` -- the SDK's code +/// for EVERY error it does not model. Borrowing it would put throttling codes an S3-compatible store +/// reports under a non-AWS name into the credential class, and would carve a real access denial out of +/// `isDefinitelyRefusedWrite` so the write wedges its caller instead of being refused. `UNKNOWN` is +/// therefore never matched by code alone; a store that spells the error out by name still matches. +bool isRefreshableCredentialError([[maybe_unused]] const std::exception & e) +{ +#if USE_AWS_S3 + const auto * s3 = dynamic_cast(&e); + if (!s3) + return false; + const Aws::S3::S3Errors code = s3->getS3ErrorCode(); + if (code == Aws::S3::S3Errors::INVALID_ACCESS_KEY_ID || code == Aws::S3::S3Errors::ACCESS_DENIED + || code == Aws::S3::S3Errors::INVALID_SIGNATURE || code == Aws::S3::S3Errors::INVALID_CLIENT_TOKEN_ID) + return true; + const String & name = s3->getExceptionName(); + return name == "ExpiredToken" || name == "InvalidToken" || name == "InvalidAccessKeyId" + || name == "SignatureDoesNotMatch" || name == "AccessDenied" || name == "AccountProblem"; +#else + return false; +#endif +} + +/// An answer from the store rather than a fault in reaching it: reissuing replays it unchanged. +bool isDefiniteStoreRefusal([[maybe_unused]] const std::exception & e) +{ +#if USE_AWS_S3 + if (const auto * s3 = dynamic_cast(&e)) + return !s3->isRetryableError(); +#endif + return false; +} + +GaveUp::Source sourceFor(const Retry::Bound & bound) +{ + return bound.lease_bound ? GaveUp::Source::Lease : GaveUp::Source::Policy; +} + +/// Drop the body from an observation the write engine had to fetch to prove whose bytes were at the +/// key. The presence-only loop is defined by what it reports, so the demotion happens on its results +/// rather than being trusted to every branch that builds one. +Observation withoutBody(Observation seen) +{ + if (const auto * obj = std::get_if(&seen)) + return Meta{obj->bytes.size(), obj->incarnation}; + return seen; +} + +WriteResult withoutBody(WriteResult result) +{ + if (auto * conflict = std::get_if(&result)) + conflict->seen = withoutBody(std::move(conflict->seen)); + else if (auto * declined = std::get_if(&result)) + declined->seen = withoutBody(std::move(declined->seen)); + else if (auto * gave_up = std::get_if(&result)) + gave_up->last_seen = withoutBody(std::move(gave_up->last_seen)); + return result; +} + +} + +bool isDeterministicLocalFailure(int code) +{ + return code == ErrorCodes::LOGICAL_ERROR || code == ErrorCodes::NOT_IMPLEMENTED + || code == ErrorCodes::BAD_ARGUMENTS || code == ErrorCodes::CORRUPTED_DATA; +} + +bool isDefinitelyRefusedWrite([[maybe_unused]] const std::exception & e) +{ +#if USE_AWS_S3 + if (const auto * s3 = dynamic_cast(&e)) + /// The refresh class is included rather than left to overlap: the two name lists agree, but + /// `InvalidSignature` carries a code `isAccessDeniedError` does not match, and an error that is + /// refreshable without being refusable would spend the whole deadline whenever no refresh is + /// available -- which is every CAS disk today. + return S3::isMalformedRequestError(*s3) || S3::isEntityTooLargeError(*s3) + || S3::isAccessDeniedError(*s3) || isRefreshableCredentialError(e); +#endif + return false; +} + +CasRequests::CasRequests(BackendPtr backend_, Fence fence_, + std::function now_ms_, std::function sleep_ms_) + : backend(std::move(backend_)) + , fence(std::move(fence_)) + , now_ms(now_ms_ ? std::move(now_ms_) : std::function(bootClockMs)) + , sleep_ms(sleep_ms_ ? std::move(sleep_ms_) : std::function(sleepForMilliseconds)) + , attempt_reservation_ms(backend->attemptTimeoutMs()) +{ +} + +void CasRequests::setNowFnForTest(std::function now_ms_) +{ + now_ms = now_ms_ ? std::move(now_ms_) : std::function(bootClockMs); +} + +void CasRequests::setSleepFnForTest(std::function sleep_ms_) +{ + sleep_ms = sleep_ms_ ? std::move(sleep_ms_) : std::function(sleepForMilliseconds); +} + +CasOperation CasRequests::admit(Liveness liveness) +{ + return CasOperation(*this, fence.generation(), std::move(liveness)); +} + +CasOperation CasRequests::resume(uint64_t admitted_generation, Liveness liveness) +{ + return CasOperation(*this, admitted_generation, std::move(liveness)); +} + +std::optional CasRequests::tryMint(const String & key, String value) const +{ + if (!isIncarnationValue(backend->dialect(), value)) + return std::nullopt; + return Incarnation(backend->backendId(), key, backend->dialect(), std::move(value)); +} + +Incarnation CasRequests::mint(const String & key, String value) const +{ + if (auto minted = tryMint(key, value)) + return std::move(*minted); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS: the store answered for '{}' with a value '{}' that is not a valid incarnation", key, value); +} + +const String & CasRequests::valueFor(const String & key, const Incarnation & inc) const +{ + if (inc.key() != key || inc.backendId() != backend->backendId()) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS: an incarnation of '{}' observed on backend {} cannot be the precondition for '{}' on backend {}", + inc.key(), inc.backendId(), key, backend->backendId()); + return inc.value(); +} + +CasOperation::Gate CasOperation::gate(uint64_t needed_ms) const +{ + switch (owner.fence.admit(admitted_generation, needed_ms)) + { + case Fence::Admit::LostOrRearmed: return Gate::FenceLost; + case Fence::Admit::NoBudget: return Gate::NoBudget; + case Fence::Admit::Ok: break; + } + /// The caller's own facts are the second half of admission, and the engine does not need to know + /// which of the two refused: a stopping task and a lost lease end the operation the same way. + if (liveness && !liveness()) + return Gate::FenceLost; + return Gate::Ok; +} + +uint64_t CasOperation::reservedFor(uint64_t sleep_ms, uint32_t envelopes) const +{ + uint64_t total = sleep_ms; + for (uint32_t i = 0; i < envelopes; ++i) + total = saturatingAdd(total, owner.attempt_reservation_ms); + return total; +} + +bool CasOperation::fits(uint64_t needed_ms, const Retry::Bound & bound) const +{ + /// Strict at the boundary. A backend with no attempt timeout reserves 0 and full jitter can draw a + /// 0 sleep, so a `needed <= remaining` test would keep issuing requests at and past the deadline -- + /// the one thing "no request after the boundary" promises never happens. + const uint64_t now = owner.now_ms(); + if (now >= bound.deadline_ms) + return false; + return needed_ms <= bound.deadline_ms - now; +} + +bool CasOperation::refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted) +{ + if (const auto * db_e = dynamic_cast(&e); db_e && isDeterministicLocalFailure(db_e->code())) + return true; + /// A local failure -- a bad allocation, a logic error raised inside the attempt -- is not a + /// transport fault, and reissuing it would spend the whole deadline replaying the same bug. Every + /// exception the transport raises is a `Poco::Exception`, `DB::Exception` included. + if (!dynamic_cast(&e)) + return true; + /// A credential failure gets ONE refresh per call -- the storage hands back a fresh client every + /// time it is asked, so refreshing per attempt would reissue a permanent denial to the deadline. + /// Without new credentials nothing would sign differently, so a read propagates rather than + /// spending its policy on a request that cannot start succeeding. + if (isRefreshableCredentialError(e)) + { + if (refresh_attempted) + return true; + refresh_attempted = true; + return !owner.backend->refreshCredentials(); + } + /// The store's own answer decides. A definite refusal replays identically whether the store proved + /// the request never applied or merely named an error it will keep naming; everything else -- a + /// throttle, a 5xx, an unmodeled name an S3-compatible store reports -- may still be transient. + return isDefinitelyRefusedWrite(e) || isDefiniteStoreRefusal(e); +} + +void CasOperation::giveUpReadFenceLost(std::string_view verb, const String & subject, std::string_view when) +{ + last_read_stop = ReadStop::FenceLost; + throwCasTransientUnavailable(fmt::format("CAS {} of '{}'", verb, subject), + fmt::format("mount fence tripped {}", when)); +} + +void CasOperation::giveUpReadNoBudget(std::string_view verb, const String & subject, std::string_view what) +{ + last_read_stop = ReadStop::NoBudgetLease; + throwCasWriteRetryLater(fmt::format("{} of '{}': no lease budget {}", verb, subject, what)); +} + +void CasOperation::giveUpReadDeadline(std::string_view verb, const String & subject, + const Retry::Bound & bound, uint32_t attempts_made) +{ + last_read_stop = ReadStop::PolicyExhausted; + throwCasWriteRetryLater(fmt::format("{} of '{}': gave up at the {} deadline after {} attempt(s)", + verb, subject, bound.lease_bound ? "lease" : "policy", attempts_made)); +} + +std::optional CasOperation::readUnder(const String & key, const Retry & policy, const Retry::Bound & bound) +{ + return readLoop("read", key, policy, bound, [&](auto & access) -> std::optional + { + auto raw = owner.backend->read(key, access); + if (!raw) + return std::nullopt; + return Object{std::move(raw->bytes), owner.mint(key, std::move(raw->value))}; + }); +} + +std::optional CasOperation::headUnder(const String & key, const Retry & policy, const Retry::Bound & bound) +{ + return readLoop("head", key, policy, bound, [&](auto & access) -> std::optional + { + auto raw = owner.backend->head(key, access); + if (!raw) + return std::nullopt; + return Meta{raw->size, owner.mint(key, std::move(raw->value))}; + }); +} + +KeyPage CasOperation::listUnder(const String & prefix, const String & cursor, size_t limit, + const Retry & policy, const Retry::Bound & bound) +{ + return readLoop("list", prefix, policy, bound, [&](auto & access) + { + Backend::RawListPage raw = owner.backend->list(prefix, cursor, limit, access); + KeyPage page; + page.next_cursor = std::move(raw.next_cursor); + page.keys.reserve(raw.keys.size()); + for (auto & listed : raw.keys) + { + KeyEntry entry{std::move(listed.key), listed.size, std::nullopt}; + if (listed.value) + entry.incarnation = owner.mint(entry.key, std::move(*listed.value)); + page.keys.push_back(std::move(entry)); + } + return page; + }); +} + +Removal CasOperation::removeUnder(const String & key, const String & expected_value, + const Retry & policy, const Retry::Bound & bound) +{ + const Backend::RawRemoval raw = readLoop("remove", key, policy, bound, [&](auto & access) + { + return owner.backend->remove(key, expected_value, access); + }); + switch (raw) + { + case Backend::RawRemoval::Removed: return Removal::Removed; + case Backend::RawRemoval::Gone: return Removal::Gone; + case Backend::RawRemoval::Mismatch: return Removal::Mismatch; + case Backend::RawRemoval::DeleteMarker: break; + } + /// Thrown outside the attempt loop: a versioned bucket answers this way every time, so reissuing + /// would spend the whole deadline to be told the same thing. + throw Exception(ErrorCodes::CAS_DELETE_MARKER, + "CAS remove of '{}' archived a noncurrent version instead of reclaiming the object: " + "the bucket has object versioning enabled", key); +} + +std::optional CasOperation::read(const String & key, const Retry & policy) +{ + return readUnder(key, policy, policy.bind(owner.now_ms())); +} + +std::optional CasOperation::head(const String & key, const Retry & policy) +{ + return headUnder(key, policy, policy.bind(owner.now_ms())); +} + +KeyPage CasOperation::list(const String & prefix, const String & cursor, size_t limit, const Retry & policy) +{ + return listUnder(prefix, cursor, limit, policy, policy.bind(owner.now_ms())); +} + +void CasOperation::forEachListedKey(const String & prefix, const KeyEntryFn & fn, const Retry & per_page, + size_t page_limit, const std::function & on_page_fetched) +{ + String cursor; + for (;;) + { + KeyPage page = list(prefix, cursor, page_limit, per_page); + if (on_page_fetched) + on_page_fetched(); + for (const KeyEntry & entry : page.keys) + if (!fn(entry)) + return; + if (page.next_cursor.empty()) + return; + cursor = std::move(page.next_cursor); + } +} + +Removal CasOperation::remove(const String & key, const Incarnation & seen, const Retry & policy) +{ + return removeUnder(key, owner.valueFor(key, seen), policy, policy.bind(owner.now_ms())); +} + +Removal CasOperation::removeCurrent(const String & key, const Retry & policy) +{ + const Retry::Bound bound = policy.bind(owner.now_ms()); + for (uint32_t attempt = 1;; ++attempt) + { + const std::optional seen = headUnder(key, policy, bound); + if (!seen) + return Removal::Gone; + const Removal removed = removeUnder(key, owner.valueFor(key, seen->incarnation), policy, bound); + if (removed != Removal::Mismatch) + return removed; + + /// A `Mismatch` is a VALUE, so it never reaches the fault check inside the two loops above: this + /// verb has to honour `once` itself, and its contract is that it never hands a `Mismatch` back. + /// The message says what actually stopped it, which is the policy and not the deadline. + if (policy.single_attempt) + throwCasWriteRetryLater(fmt::format( + "removeCurrent of '{}': the observed incarnation was replaced and the policy allows no reissue", key)); + + /// Another incarnation became current between the observation and the delete. Re-observe, paced + /// like every other reissue: the reservation covers the next `head` and the `remove` after it. + const uint64_t pause_ms = Retry::backoff(attempt); + const uint64_t needed = reservedFor(pause_ms, 2); + switch (gate(needed)) + { + case Gate::FenceLost: giveUpReadFenceLost("removeCurrent", key, "before the reissue"); + case Gate::NoBudget: giveUpReadNoBudget("removeCurrent", key, "for the reissue"); + case Gate::Ok: break; + } + if (!fits(needed, bound)) + giveUpReadDeadline("removeCurrent", key, bound, attempt); + detail::recordReissue(); + owner.sleep_ms(pause_ms); + } +} + +SentinelProbeResult CasOperation::probeSentinel(const String & key, const Retry & policy) +{ + const Retry::Bound bound = policy.bind(owner.now_ms()); + bool refresh_attempted = false; + for (uint32_t attempt = 1;; ++attempt) + { + const uint64_t reservation = reservedFor(0, 1); + switch (gate(reservation)) + { + case Gate::FenceLost: giveUpReadFenceLost("probeSentinel", key, "before the request"); + case Gate::NoBudget: giveUpReadNoBudget("probeSentinel", key, "for one more request"); + case Gate::Ok: break; + } + /// Before the first attempt nothing has been probed, so the caller is told the request never + /// happened rather than being handed an outcome no request produced. + if (!fits(reservation, bound)) + giveUpReadDeadline("probeSentinel", key, bound, attempt - 1); + + detail::recordAttempt(); + SentinelProbeResult result{ProbeOutcome::Indeterminate, std::nullopt}; + try + { + result = owner.withTransportAccess([&](auto & access) + { + return owner.backend->probeSentinelRaw(key, access); + }); + } + catch (const std::exception & e) + { + if (refreshAndClassifyReadFault(e, refresh_attempted)) + throw; + /// The probe reports every transport failure as `Indeterminate` rather than by throwing, so + /// a decorator that does throw is folded onto the same inconclusive outcome. + } + + /// The reissue is driven by the OUTCOME: this is the one primitive whose failures never reach a + /// `catch`, and `Indeterminate` means "inconclusive", which is exactly what a retry resolves. + /// The other four outcomes are authoritative answers and return on the first attempt. + if (result.outcome != ProbeOutcome::Indeterminate || policy.single_attempt) + return result; + + const uint64_t pause_ms = Retry::backoff(attempt); + const uint64_t needed = reservedFor(pause_ms, 1); + /// A lost fence and a spent lease budget are not things the store said, so they are reported the + /// way every other read verb reports them rather than being dressed up as an outcome. + switch (gate(needed)) + { + case Gate::FenceLost: giveUpReadFenceLost("probeSentinel", key, "before the reissue"); + case Gate::NoBudget: giveUpReadNoBudget("probeSentinel", key, "for the reissue"); + case Gate::Ok: break; + } + /// Only the policy's own bound ends the loop with a value: a probe DID run, and what it saw is + /// more than a give-up exception could say. Every consumer treats `Indeterminate` fail-closed. + if (!fits(needed, bound)) + return result; + detail::recordReissue(); + owner.sleep_ms(pause_ms); + } +} + +std::unique_ptr CasOperation::stream(const String & key, const Retry & policy) +{ + const Retry::Bound bound = policy.bind(owner.now_ms()); + return readLoop("stream", key, policy, bound, [&](auto & access) + { + return owner.backend->stream(key, access); + }); +} + +void CasOperation::publish(const BlobPublishRequest & request, const Retry & policy) +{ + const Retry::Bound bound = policy.bind(owner.now_ms()); + readLoop("publish", request.destination_key, policy, bound, [&](auto & access) + { + owner.backend->publish(request, access); + }); +} + +CasOperation::Resolved CasOperation::observe(const String & key, const Retry & policy, const Retry::Bound & bound) +{ + last_read_stop.reset(); + try + { + auto got = readUnder(key, policy, bound); + if (!got) + return {ProvenAbsent{}, std::nullopt}; + return {std::move(*got), std::nullopt}; + } + catch (const Exception & e) + { + /// A local bug replays identically on every reissue, so it is never swallowed into "nothing + /// observed". Anything else settled nothing -- and `last_read_stop` says whether a bound + /// refused the read or the transport itself failed, which the exception cannot. + if (isDeterministicLocalFailure(e.code())) + throw; + return {NotObserved{}, last_read_stop}; + } + catch (const std::exception & e) + { + /// A failure that is not the transport's is not an observation: it is the same local bug the + /// read loop refuses to reissue, and swallowing it here would hide it just as thoroughly. + if (!dynamic_cast(&e)) + throw; + return {NotObserved{}, last_read_stop}; + } +} + +CasOperation::Resolved CasOperation::observePresence(const String & key, const Retry & policy, const Retry::Bound & bound) +{ + last_read_stop.reset(); + try + { + auto got = headUnder(key, policy, bound); + if (!got) + return {ProvenAbsent{}, std::nullopt}; + return {std::move(*got), std::nullopt}; + } + catch (const Exception & e) + { + if (isDeterministicLocalFailure(e.code())) + throw; + return {NotObserved{}, last_read_stop}; + } + catch (const std::exception & e) + { + if (!dynamic_cast(&e)) + throw; + return {NotObserved{}, last_read_stop}; + } +} + +WriteResult CasOperation::gaveUp(GaveUp::Why why, GaveUp::Source source, WriteState & state) const +{ + ProfileEvents::increment(ProfileEvents::CASRequestGaveUp); + return GaveUp{why, source, state.sent_any, state.last_seen}; +} + +WriteResult CasOperation::gaveUpForReadStop(ReadStop stop, WriteState & state, const Retry::Bound & bound) const +{ + switch (stop) + { + case ReadStop::FenceLost: + return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); + case ReadStop::NoBudgetLease: + /// The fence's budget IS the mount lease, so the lease is the bound that ended this call. + return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); + case ReadStop::PolicyExhausted: + return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); + } + UNREACHABLE(); +} + +WriteResult CasOperation::gaveUpAfterFailedObservation(std::optional stop, WriteState & state, + const Retry::Bound & bound) const +{ + if (stop) + return gaveUpForReadStop(*stop, state, bound); + /// No bound refused; the read itself failed. That is what `Unresolved` names, and it is the honest + /// answer -- claiming a deadline the clock never reached would misreport which bound to widen. + return gaveUp(GaveUp::Why::Unresolved, sourceFor(bound), state); +} + +WriteResult CasOperation::postCommit(Incarnation inc, bool resolved_by_read, WriteState & state, const Retry::Bound & bound) +{ + /// Admission once more, now that the write is proven durable: a fence lost here means the object + /// may well exist, but this call must never claim it -- the caller has to resolve the key instead. + switch (gate(0)) + { + case Gate::FenceLost: + ProfileEvents::increment(ProfileEvents::CASRequestFenceLostPostWrite); + return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); + case Gate::NoBudget: + /// The fence's budget IS the mount lease, so its refusal names the lease as the bound that + /// ended this call, whichever bound produced the policy's own deadline. + return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); + case Gate::Ok: break; + } + return Committed{std::move(inc), state.attempts_sent, resolved_by_read}; +} + +std::optional CasOperation::pauseAndReissue(WriteState & state, const Retry::Bound & bound) +{ + const uint64_t pause_ms = Retry::backoff(++state.reissues); + const uint64_t needed = reservedFor(pause_ms, 2); + switch (gate(needed)) + { + case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); + case Gate::NoBudget: return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); + case Gate::Ok: break; + } + if (!fits(needed, bound)) + return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); + detail::recordReissue(); + owner.sleep_ms(pause_ms); + return std::nullopt; +} + +WriteResult CasOperation::writeLoop(const String & key, const String & bytes, const std::optional & expected, + const Retry & policy, const Retry::Bound & bound, WriteState & state, + ResolveWith resolve_refusal_with) +{ + std::optional expected_value; + if (expected) + expected_value = owner.valueFor(key, *expected); + + for (;;) + { + /// A write reserves TWO envelopes: the attempt, and the exact read that settles it. That is + /// what keeps "every conflict is settled by one read" true at the deadline edge. + const uint64_t reservation = reservedFor(0, 2); + switch (gate(reservation)) + { + case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); + case Gate::NoBudget: return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); + case Gate::Ok: break; + } + if (!fits(reservation, bound)) + return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); + + detail::recordAttempt(); + ++state.attempts_sent; + state.sent_any = true; + + /// Disengaged means the attempt threw: its fate is unproven, and nothing may be read out of it. + std::optional> outcome; + /// A credential answer is given BEFORE the store applies anything, so the attempt provably did + /// not land. It is the one failure that is neither a commit nor an ambiguity, and keeping it out + /// of `any_ambiguous` is what lets a second credential failure of the same call be refused + /// instead of resolved by a read and reissued to the deadline. + bool credential_answer = false; + bool refreshed = false; + try + { + outcome = owner.withTransportAccess([&](auto & access) + { + return owner.backend->write(key, bytes, expected_value, access); + }); + } + catch (const Exception & e) + { + if (isDeterministicLocalFailure(e.code())) + throw; + /// ONE refresh per call, and only for the class a credential could explain -- so an + /// oversized entity or a malformed request never triggers a re-acquisition, and a denial + /// that fresh credentials do not fix is refused on the second look rather than reissued to + /// the deadline (the storage hands back a new client every time it is asked). + credential_answer = isRefreshableCredentialError(e); + if (credential_answer && !state.refresh_attempted) + { + state.refresh_attempted = true; + refreshed = owner.backend->refreshCredentials(); + } + /// Fresh credentials only help if there is a reissue to sign with them, so under `once` the + /// store's answer stands even when a refresh succeeded. A refusal that FOLLOWS an ambiguous + /// attempt of this call proves nothing about that attempt, so it is settled by the read + /// below instead of ending the call here. + if ((!refreshed || policy.single_attempt) && isDefinitelyRefusedWrite(e) && !state.any_ambiguous) + { + ProfileEvents::increment(ProfileEvents::CASRequestRefused); + return Refused{e.code(), e.message()}; + } + } + catch (const std::exception &) + { + /// An unmodeled failure may still have landed: leave `outcome` disengaged and settle it by + /// reading, never by reporting a refusal the store never gave. + } + + if (outcome && outcome->has_value()) + { + if (auto inc = owner.tryMint(key, std::move(**outcome))) + return postCommit(std::move(*inc), /*resolved_by_read=*/false, state, bound); + /// A 2xx carrying a value no grammar accepts: the write may well have landed, so this is an + /// ambiguity to settle by reading, never a corruption verdict about the object. + outcome.reset(); + } + if (!outcome && !credential_answer) + state.any_ambiguous = true; + + /// Nothing for a read to settle: this attempt did not apply, and no EARLIER attempt of the call + /// is unresolved either. Re-send it under the credentials the refresh installed. The policy is + /// named here rather than inherited from the refusal above, so a gap in what counts as a + /// definite refusal can never turn `once` into a sleeping loop. + if (refreshed && !policy.single_attempt && !state.any_ambiguous) + { + if (auto given_up = pauseAndReissue(state, bound)) + return *given_up; + continue; + } + + /// Every refused precondition and every ambiguous attempt is settled by ONE exact read, under + /// every policy: a refused precondition does not say WHO holds the key, and a 404 and a 412 + /// reach here as the same answer. A refused precondition needs only to know WHAT is there, so a + /// presence-only caller settles it with a HEAD; proving an ambiguous attempt landed needs the + /// bytes, and there the body read is unavoidable. + ProfileEvents::increment(ProfileEvents::CASRequestResolveRead); + const Resolved resolved = resolve_refusal_with == ResolveWith::Presence && !state.any_ambiguous + ? observePresence(key, policy, bound) + : observe(key, policy, bound); + state.last_seen = resolved.seen; + /// A bound refused the resolve, so say WHICH. Erasing it here is what let a lost fence be + /// reported as an ordinary conflict and a lease refusal as a policy deadline. + if (resolved.stop) + return gaveUpForReadStop(*resolved.stop, state, bound); + if (const auto * obj = std::get_if(&state.last_seen)) + { + /// Our own bytes prove an earlier ambiguous attempt landed. Without an ambiguity there was + /// nothing of ours to land, so identical bytes are somebody else's object and the caller + /// that owns the key's meaning decides what that means. + if (state.any_ambiguous && obj->bytes == bytes) + return postCommit(obj->incarnation, /*resolved_by_read=*/true, state, bound); + return Conflict{state.last_seen}; + } + if (!state.any_ambiguous) + return Conflict{state.last_seen}; + if (policy.single_attempt) + return gaveUp(GaveUp::Why::Unresolved, sourceFor(bound), state); + if (auto given_up = pauseAndReissue(state, bound)) + return *given_up; + } +} + +WriteResult CasOperation::create(const String & key, const String & bytes, const Retry & policy) +{ + WriteState state; + return writeLoop(key, bytes, std::nullopt, policy, policy.bind(owner.now_ms()), state, ResolveWith::Body); +} + +WriteResult CasOperation::replace(const String & key, const String & bytes, const Incarnation & seen, const Retry & policy) +{ + WriteState state; + return writeLoop(key, bytes, seen, policy, policy.bind(owner.now_ms()), state, ResolveWith::Body); +} + +WriteResult CasOperation::readModifyWrite(const String & key, const DecideOnObject & decide, const Retry & policy) +{ + const Retry::Bound bound = policy.bind(owner.now_ms()); + WriteState state; + + Resolved resolved = observe(key, policy, bound); + state.last_seen = resolved.seen; + std::optional current; + if (const auto * obj = std::get_if(&state.last_seen)) + current = *obj; + else if (!std::holds_alternative(state.last_seen)) + return gaveUpAfterFailedObservation(resolved.stop, state, bound); + + for (;;) + { + /// Outside every classification: `decide` is the caller's own control flow, and an exception + /// from it is theirs to see unchanged. + const std::optional next = decide(current); + if (!next) + return Declined{state.last_seen}; + + WriteResult result = writeLoop(key, *next, + current ? std::optional(current->incarnation) : std::nullopt, policy, bound, state, + ResolveWith::Body); + if (!std::holds_alternative(result)) + return result; + + /// The write's own resolve read IS this iteration's read: a hot key never costs a second GET + /// per conflict. + if (const auto * obj = std::get_if(&state.last_seen)) + current = *obj; + else if (std::holds_alternative(state.last_seen)) + current.reset(); + + if (policy.single_attempt) + return result; + if (auto given_up = pauseAndReissue(state, bound)) + return *given_up; + + /// Only when the resolve settled nothing is a fresh read owed; otherwise `current` already is + /// what the store held. + if (std::holds_alternative(state.last_seen)) + { + resolved = observe(key, policy, bound); + state.last_seen = resolved.seen; + if (const auto * obj = std::get_if(&state.last_seen)) + current = *obj; + else if (std::holds_alternative(state.last_seen)) + current.reset(); + else + return gaveUpAfterFailedObservation(resolved.stop, state, bound); + } + } +} + +WriteResult CasOperation::readModifyWriteOnPresence(const String & key, const DecideOnMeta & decide, const Retry & policy) +{ + const Retry::Bound bound = policy.bind(owner.now_ms()); + WriteState state; + + Resolved resolved = observePresence(key, policy, bound); + state.last_seen = resolved.seen; + std::optional current; + if (const auto * meta = std::get_if(&state.last_seen)) + current = *meta; + else if (!std::holds_alternative(state.last_seen)) + return gaveUpAfterFailedObservation(resolved.stop, state, bound); + + for (;;) + { + const std::optional next = decide(current); + if (!next) + return Declined{state.last_seen}; + + WriteResult result = writeLoop(key, *next, + current ? std::optional(current->incarnation) : std::nullopt, policy, bound, state, + ResolveWith::Presence); + /// A refused precondition was settled by a HEAD, but proving an ambiguous attempt landed needs + /// the bytes; this loop is presence-only by contract, so that body stops here. + state.last_seen = withoutBody(std::move(state.last_seen)); + if (!std::holds_alternative(result)) + return withoutBody(std::move(result)); + + if (const auto * meta = std::get_if(&state.last_seen)) + current = *meta; + else if (std::holds_alternative(state.last_seen)) + current.reset(); + + if (policy.single_attempt) + return Conflict{state.last_seen}; + if (auto given_up = pauseAndReissue(state, bound)) + return *given_up; + + if (std::holds_alternative(state.last_seen)) + { + resolved = observePresence(key, policy, bound); + state.last_seen = resolved.seen; + if (const auto * meta = std::get_if(&state.last_seen)) + current = *meta; + else if (std::holds_alternative(state.last_seen)) + current.reset(); + else + return gaveUpAfterFailedObservation(resolved.stop, state, bound); + } + } +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h new file mode 100644 index 000000000000..8b29fe82eeae --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -0,0 +1,336 @@ +#pragma once +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB::Cas +{ + +/// TRUE when the store's own answer proves this write never applied: a malformed request, an entity +/// too large, an access denial, or a credential failure. Whether a STALE CREDENTIAL explains it is not +/// asked here -- the engine asks the backend for fresh credentials first, and refuses only when no +/// refresh HELPED: the error was outside the credential class, the one refresh this call is allowed +/// installed nothing, or there is no reissue left to sign with what it did install. A refresh that +/// helps re-sends the attempt, which is known not to have applied. A non-S3 exception is never a +/// refusal: an unmodeled error may have landed. +bool isDefinitelyRefusedWrite(const std::exception & e); + +/// Deterministic caller/local bugs, surfaced unchanged by every loop here: reissuing only replays the +/// same failure and buries the root cause behind a retryable exception. The set is `LOGICAL_ERROR`, +/// `NOT_IMPLEMENTED`, `BAD_ARGUMENTS` and `CORRUPTED_DATA`. +bool isDeterministicLocalFailure(int code); + +/// Facts the fence cannot see, sampled by the caller. Non-throwing; FALSE ends the operation exactly +/// like a lost fence, because the engine does not need to know which of the two refused. +using Liveness = std::function; +/// `decide` sees the current object (or absence) and returns the bytes to write, or nullopt for +/// "nothing to do". It may throw: the exception is the caller's control flow and propagates unchanged. +using DecideOnObject = std::function(const std::optional &)>; +using DecideOnMeta = std::function(const std::optional &)>; + +/// One key returned by `list`. `incarnation` is present only on a backend that surfaces per-key +/// incarnations through LIST -- see `Backend::supportsListTokens`. +struct KeyEntry +{ + String key; + uint64_t size; + std::optional incarnation; +}; +/// One page of an enumeration. `next_cursor` resumes strictly after the last returned key; empty +/// marks the end. +struct KeyPage +{ + std::vector keys; + String next_cursor; +}; +/// The walk's callback: FALSE stops the walk. +using KeyEntryFn = std::function; + +namespace detail +{ +/// The engine's per-attempt counters, behind functions so the header need not declare the events. +void recordAttempt(); +void recordReissue(); +} + +class CasOperation; + +/// The only caller of `Backend`. It owns the three things a physical request must be measured +/// against -- the transport, the mount fence, and the clock -- and it is the sole minter of +/// `Incarnation`, so a caller can hold one only by way of a request this class admitted. +/// +/// It is constructed with a fence because a fence is a property of whoever holds the lease, not of a +/// call: the mount plane passes the mount fence, the GC plane and the offline tools an open one. A +/// verb is never called here; verbs live on `CasOperation`, which carries the generation the caller +/// was admitted under. +class CasRequests +{ +public: + /// `now_ms` defaults to `CLOCK_BOOTTIME` milliseconds -- the same clock a mount lease deadline is + /// expressed on, so `Retry::untilLeaseSafe` and this engine compare like with like. `sleep_ms` + /// defaults to a real sleep. `attempt_reservation_ms` is taken from the backend's own attempt + /// timeout: it is what the engine reserves before it starts anything. + CasRequests(BackendPtr backend_, Fence fence_, + std::function now_ms_ = {}, std::function sleep_ms_ = {}); + + /// Admitted now, under the fence's current generation. + CasOperation admit(Liveness liveness = {}); + /// Admitted earlier: the generation came from a persisted runtime record, and an operation that + /// resumes under a generation the fence has since moved past gives up rather than writing. + CasOperation resume(uint64_t admitted_generation, Liveness liveness = {}); + + /// The capability predicates and `dialect()`. + Backend & backendForCapabilityPredicates() { return *backend; } + + void setNowFnForTest(std::function now_ms_); + void setSleepFnForTest(std::function sleep_ms_); + void setAttemptReservationForTest(uint64_t ms) { attempt_reservation_ms = ms; } + +private: + friend class CasOperation; + + /// The one place a transport key is created. Every verb reaches the store through this, so no + /// engine code -- and nothing outside it -- can name the key's type, let alone construct one. + template + auto withTransportAccess(Fn && fn) + { + TransportAccess access; + return std::forward(fn)(access); + } + + /// The store's answer for `key`, as an incarnation. Throws `CORRUPTED_DATA` naming the key when + /// the value fails this backend's dialect grammar. + Incarnation mint(const String & key, String value) const; + /// `mint` without the verdict, for the one caller that must treat a malformed value as an + /// ambiguity to settle by reading rather than as corruption: a write's own 2xx response. + std::optional tryMint(const String & key, String value) const; + /// The transport value to send as a precondition. Throws `LOGICAL_ERROR` when the incarnation + /// names another key or another backend -- a precondition built from it would silently mean + /// something else. + const String & valueFor(const String & key, const Incarnation & inc) const; + + BackendPtr backend; + Fence fence; + std::function now_ms; + std::function sleep_ms; + uint64_t attempt_reservation_ms; +}; + +/// One admitted operation: the unit a policy, a fence generation and a liveness predicate apply to. +/// Move-only, and every request it makes re-checks its admission -- before each attempt, before each +/// sleep, and once more after a proven commit, so a write whose fence was lost while it was in flight +/// is never reported as committed. +class CasOperation +{ +public: + CasOperation(CasOperation &&) = default; + CasOperation(const CasOperation &) = delete; + CasOperation & operator=(const CasOperation &) = delete; + + uint64_t generation() const { return admitted_generation; } + /// The verdict point: is this operation still admitted? For the sites that guard a decision rather + /// than a request. + bool admitted() const { return gate(0) == Gate::Ok; } + + std::optional read(const String & key, const Retry & policy); + std::optional head(const String & key, const Retry & policy); + KeyPage list(const String & prefix, const String & cursor, size_t limit, const Retry & policy); + /// Walks every key under `prefix` exactly once. The policy governs EACH PAGE, not the walk: a walk + /// is an unbounded number of requests, and a silently truncated enumeration is the error a + /// coverage record exists to prevent. `on_page_fetched` fires once per page DELIVERED; a page that + /// took several reissues still fires once, and `CASRequestAttempt` is the physical count. + void forEachListedKey(const String & prefix, const KeyEntryFn & fn, const Retry & per_page, + size_t page_limit = 1000, const std::function & on_page_fetched = {}); + Removal remove(const String & key, const Incarnation & seen, const Retry & policy); + /// `head` then `remove` of what it saw, repeating on `Mismatch`. `Gone` when the key is already + /// absent; never returns `Mismatch` -- under `once`, where there is no reissue to resolve one, a + /// `Mismatch` is the retry-later throw the read verbs use when their policy is exhausted. + Removal removeCurrent(const String & key, const Retry & policy); + /// The one primitive that reports failure as a value, so the policy reissues on the OUTCOME: + /// `Indeterminate` is retried, the four authoritative outcomes return at once, and an + /// `Indeterminate` that outlives the bound is returned rather than thrown. Admission refused before + /// the first attempt still throws -- nothing was probed. + SentinelProbeResult probeSentinel(const String & key, const Retry & policy); + /// The OPEN is under the policy; the body is the SDK's. + std::unique_ptr stream(const String & key, const Retry & policy); + /// The INITIATION is under the policy; the transfer is the SDK's. + void publish(const BlobPublishRequest & request, const Retry & policy); + + WriteResult create(const String & key, const String & bytes, const Retry & policy); + WriteResult replace(const String & key, const String & bytes, const Incarnation & seen, const Retry & policy); + /// Read, decide, write, and re-decide on conflict against what the write's own resolve read + /// already observed. `decide` returning nullopt is `Declined`. + WriteResult readModifyWrite(const String & key, const DecideOnObject & decide, const Retry & policy); + /// The same loop over `head`, settling a refused precondition with a `head` too. Proving that an + /// ambiguous attempt landed needs the bytes, so that one path does read a body; either way the verb + /// reports a `Meta` and never an `Object`. + WriteResult readModifyWriteOnPresence(const String & key, const DecideOnMeta & decide, const Retry & policy); + +private: + friend class CasRequests; + + CasOperation(CasRequests & owner_, uint64_t admitted_generation_, Liveness liveness_) + : owner(owner_), admitted_generation(admitted_generation_), liveness(std::move(liveness_)) + { + } + + enum class Gate : uint8_t { Ok, FenceLost, NoBudget }; + /// The admission point: the fence for `needed_ms` from now, then the caller's own facts. + Gate gate(uint64_t needed_ms) const; + + /// Everything ONE logical write call accumulates. It outlives each attempt, and for + /// `readModifyWrite` it outlives each inner write, so a `GaveUp` reports what the whole call did + /// rather than what its last attempt did. + struct WriteState + { + uint32_t attempts_sent = 0; + bool sent_any = false; + /// Did ANY attempt of this call end without proof of whether it applied? A credential answer + /// does not qualify: the store gives it before applying anything. + bool any_ambiguous = false; + Observation last_seen = NotObserved{}; + uint32_t reissues = 0; + bool refresh_attempted = false; + }; + + /// Why a read-class request stopped without an answer. Every give-up below throws the same + /// `NETWORK_ERROR`, so a caller that SWALLOWS the exception -- only the resolve read does -- cannot + /// recover from it which bound refused, and reporting a lease refusal as a policy deadline is the + /// confusion `GaveUp::Source` exists to prevent. `PolicyExhausted` names the `Retry` bound, whose + /// own source is `Bound::lease_bound`. + enum class ReadStop : uint8_t { FenceLost, NoBudgetLease, PolicyExhausted }; + + /// What the resolve read saw, and why it stopped when it saw nothing. `stop` is set ONLY when a + /// bound refused; a read that failed at the transport leaves it empty, and that is the one case + /// `NotObserved` is still the whole story. + struct Resolved + { + Observation seen; + std::optional stop; + }; + + /// One read-class request under the policy: admission, attempt, classification, jittered reissue. + /// Returns whatever `once` returns, or throws -- the read surface reports failure by exception. + template + auto readLoop(std::string_view verb, const String & subject, const Retry & policy, + const Retry::Bound & bound, Fn && once); + + std::optional readUnder(const String & key, const Retry & policy, const Retry::Bound & bound); + std::optional headUnder(const String & key, const Retry & policy, const Retry::Bound & bound); + KeyPage listUnder(const String & prefix, const String & cursor, size_t limit, + const Retry & policy, const Retry::Bound & bound); + Removal removeUnder(const String & key, const String & expected_value, + const Retry & policy, const Retry::Bound & bound); + + /// How a write settles a REFUSED PRECONDITION, which needs only to know what is at the key. An + /// ambiguous attempt always reads the body, whichever this says, because only the bytes can prove + /// the attempt landed. + enum class ResolveWith : uint8_t { Body, Presence }; + + /// The write engine: one call, any policy. Settles every refused precondition and every ambiguity + /// by an exact read before it reports anything. + WriteResult writeLoop(const String & key, const String & bytes, const std::optional & expected, + const Retry & policy, const Retry::Bound & bound, WriteState & state, + ResolveWith resolve_refusal_with); + /// The resolve read: an exact read under the same policy and deadline, reporting what it saw and, + /// when it saw nothing, which bound stopped it. + Resolved observe(const String & key, const Retry & policy, const Retry::Bound & bound); + /// The presence-only sibling, for the one loop that must not fetch a body. + Resolved observePresence(const String & key, const Retry & policy, const Retry::Bound & bound); + + WriteResult postCommit(Incarnation inc, bool resolved_by_read, WriteState & state, const Retry::Bound & bound); + WriteResult gaveUp(GaveUp::Why why, GaveUp::Source source, WriteState & state) const; + /// The bound that refused the resolve read, reported as the outcome it actually is. + WriteResult gaveUpForReadStop(ReadStop stop, WriteState & state, const Retry::Bound & bound) const; + /// A resolve read that produced nothing. The fence is NOT resampled: a second sample reports a + /// state the read never saw, which is how a lease refusal used to be reported as a policy deadline. + WriteResult gaveUpAfterFailedObservation(std::optional stop, WriteState & state, + const Retry::Bound & bound) const; + /// Admission, then the jittered sleep. A value means the call ended during it; nullopt means the + /// caller may send another attempt. + std::optional pauseAndReissue(WriteState & state, const Retry::Bound & bound); + + /// `sleep_ms` plus `envelopes` attempt reservations, saturating. + uint64_t reservedFor(uint64_t sleep_ms, uint32_t envelopes) const; + /// Is there room to START something needing `needed_ms` before the bound? The guarantee is on the + /// start side: nothing is begun that could not finish inside it. + bool fits(uint64_t needed_ms, const Retry::Bound & bound) const; + + /// One failed read-class attempt, classified. A credential failure is refreshed HERE so the reissue + /// signs with the new client, at most once per call -- `refresh_attempted` is the caller's, and a + /// second credential failure under the same call is classified as if no refresh were available. + /// TRUE means the failure must surface unchanged. + bool refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted); + + /// Each records its cause in `last_read_stop` before throwing, so the resolve read can report it. + [[noreturn]] void giveUpReadFenceLost(std::string_view verb, const String & subject, std::string_view when); + [[noreturn]] void giveUpReadNoBudget(std::string_view verb, const String & subject, std::string_view what); + [[noreturn]] void giveUpReadDeadline(std::string_view verb, const String & subject, + const Retry::Bound & bound, uint32_t attempts_made); + + CasRequests & owner; + uint64_t admitted_generation; + Liveness liveness; + /// Written immediately before a read-class give-up throws, cleared and read only by the resolve + /// read that swallows it. Every other caller lets the exception carry the verdict. + std::optional last_read_stop; +}; + +template +auto CasOperation::readLoop(std::string_view verb, const String & subject, const Retry & policy, + const Retry::Bound & bound, Fn && once) +{ + bool refresh_attempted = false; + for (uint32_t attempt = 1;; ++attempt) + { + const uint64_t reservation = reservedFor(0, 1); + switch (gate(reservation)) + { + case Gate::FenceLost: giveUpReadFenceLost(verb, subject, "before the request"); + case Gate::NoBudget: giveUpReadNoBudget(verb, subject, "for one more request"); + case Gate::Ok: break; + } + if (!fits(reservation, bound)) + giveUpReadDeadline(verb, subject, bound, attempt - 1); + + detail::recordAttempt(); + try + { + return owner.withTransportAccess([&](auto & access) { return once(access); }); + } + catch (const std::exception & e) + { + if (refreshAndClassifyReadFault(e, refresh_attempted) || policy.single_attempt) + throw; + } + + const uint64_t pause_ms = Retry::backoff(attempt); + const uint64_t needed = reservedFor(pause_ms, 1); + switch (gate(needed)) + { + case Gate::FenceLost: giveUpReadFenceLost(verb, subject, "before the reissue"); + case Gate::NoBudget: giveUpReadNoBudget(verb, subject, "for the reissue"); + case Gate::Ok: break; + } + if (!fits(needed, bound)) + giveUpReadDeadline(verb, subject, bound, attempt); + detail::recordReissue(); + owner.sleep_ms(pause_ms); + } +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp new file mode 100644 index 000000000000..e138afb5f39e --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp @@ -0,0 +1,30 @@ +#include + +#include + +#include +#include + +namespace DB::Cas +{ + +uint64_t Retry::backoff(uint32_t attempt) +{ + if (attempt == 0) + return 0; + const uint32_t doublings = std::min(attempt - 1, 20); + const uint64_t ceiling = std::min(5000, 200ull << doublings); + return thread_local_rng() % (ceiling + 1); /// full jitter: uniform(0, ceiling) +} + +Retry::Bound Retry::bind(uint64_t now_ms) const +{ + const uint64_t policy_deadline_ms = now_ms > std::numeric_limits::max() - window_ms + ? std::numeric_limits::max() + : now_ms + window_ms; + if (lease_deadline_ms && *lease_deadline_ms < policy_deadline_ms) + return {*lease_deadline_ms, true}; + return {policy_deadline_ms, false}; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h new file mode 100644 index 000000000000..6e9b32d9c293 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h @@ -0,0 +1,52 @@ +#pragma once +#include +#include + +namespace DB::Cas +{ + +/// A retry policy for one logical CAS write, expressed WITHOUT touching a clock: `window_ms` is the +/// policy's own budget measured from the call's start, and `lease_deadline_ms` (already reduced by +/// the caller's safety margin) is an absolute bound on whatever clock the caller's mount lease is +/// tracked against. Binding a policy to an absolute deadline is deferred to `bind`, which takes the +/// caller's own `now_ms` -- `CasRequests` runs on an injected clock in tests, so a `Retry` built at +/// construction time from the real clock could never be exercised deterministically, and `GaveUp` +/// could not tell a lease-caused deadline from a policy-caused one without recording which bound won. +struct Retry +{ + uint64_t window_ms; + std::optional lease_deadline_ms; + bool single_attempt; + + /// Full jitter: uniform(0, min(5000, 200 << (attempt-1))) milliseconds. `attempt` is 1-based; + /// `attempt == 0` returns 0. + static uint64_t backoff(uint32_t attempt); + + /// A policy with `ms` milliseconds of its own budget and no lease bound. + static Retry within(uint64_t ms) { return {ms, std::nullopt, false}; } + /// `within(90'000)` -- the default write policy. + static Retry standard() { return within(90'000); } + /// The standard policy, additionally bound by the mount lease: `lease_deadline_ms` minus + /// `margin`, clamped at 0 -- never risk a write landing after this node's fence may already be + /// gone. + static Retry untilLeaseSafe(uint64_t lease_deadline_ms, uint64_t margin) + { + return {90'000, lease_deadline_ms > margin ? lease_deadline_ms - margin : 0, false}; + } + /// The standard policy, but at most one attempt is ever sent. + static Retry once() { return {90'000, std::nullopt, true}; } + + /// A policy bound to an absolute deadline on the caller's own clock, plus which bound produced + /// it -- the caller's own budget, or the (smaller) lease bound. + struct Bound + { + uint64_t deadline_ms; + bool lease_bound; + }; + /// Bind this policy to `now_ms`: `deadline_ms = min(now_ms + window_ms, lease_deadline_ms)`, + /// `lease_bound` true exactly when the lease bound was the smaller of the two. Called once at + /// call entry with the owner's `now_ms()`. + Bound bind(uint64_t now_ms) const; +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h new file mode 100644 index 000000000000..96e9545832b8 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h @@ -0,0 +1,221 @@ +#pragma once +#include + +#include "config.h" + +#if USE_AWS_S3 + +#include + +#include +#include + +namespace DB::ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + +namespace DB::Cas +{ + +/// A `Backend` decorator that refuses a chosen share of the requests passing through it, so a test +/// can drive the request contract's reissue, resolve and give-up paths against a real backend with +/// no network in the picture. +/// +/// The refusal is an `S3Exception` carrying `SLOW_DOWN` (HTTP 429) or `SERVICE_UNAVAILABLE` (503). +/// Both are RETRYABLE by `S3Exception::isRetryableError` -- its unretryable set holds neither -- and +/// that is the property being modelled: a retryable store refusal leaves the attempt AMBIGUOUS, so +/// the caller must resolve it by reading rather than treat it as a definite failure. +class ThrottlingBackend final : public Backend +{ +public: + /// `FirstPerKey` refuses the first request naming each key and forwards every later one; + /// `EveryNth` refuses every n-th request across all keys. + enum class Mode : uint8_t { FirstPerKey, EveryNth }; + + ThrottlingBackend(BackendPtr inner_, Mode mode_, size_t n_, int status) + : inner(std::move(inner_)) + , mode(mode_) + , every_nth(n_) + , error(status == 429 ? Aws::S3::S3Errors::SLOW_DOWN : Aws::S3::S3Errors::SERVICE_UNAVAILABLE) + { + if (status != 429 && status != 503) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "ThrottlingBackend: refusal status must be 429 or 503, got {}", status); + if (mode == Mode::EveryNth && every_nth == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "ThrottlingBackend: EveryNth needs a period of at least 1"); + } + + /// How many requests naming `key` this backend has refused. A `list` counts under its prefix and + /// a `publish` under its destination key. + size_t refusals(const String & key) const + { + std::lock_guard lock(mutex); + const auto it = refusal_counts.find(key); + return it == refusal_counts.end() ? 0 : it->second; + } + + std::optional read(const String & key, TransportAccess & access) override + { + refuseOrPass(key); + return inner->read(key, access); + } + + std::optional head(const String & key, TransportAccess & access) override + { + refuseOrPass(key); + return inner->head(key, access); + } + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + refuseOrPass(prefix); + return inner->list(prefix, cursor, limit, access); + } + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + refuseOrPass(key); + return inner->remove(key, expected_value, access); + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + refuseOrPass(key); + return inner->write(key, bytes, expected_value, access); + } + + std::unique_ptr stream(const String & key, TransportAccess & access) override + { + refuseOrPass(key); + return inner->stream(key, access); + } + + void publish(const BlobPublishRequest & request, TransportAccess & access) override + { + refuseOrPass(request.destination_key); + inner->publish(request, access); + } + + SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override + { + refuseOrPass(key); + return inner->probeSentinelRaw(key, access); + } + + /// ---- The legacy surface, refused and forwarded AS legacy ---- + /// + /// Not inherited from `Backend`: its forwarder would call the primitive on THIS object, so the + /// inner backend would receive a primitive and any legacy override it carries would never run. + /// Each request is refused (or not) exactly once, on whichever surface its caller used. + std::optional getStream(const String & key, Range range) override + { + refuseOrPass(key); + return inner->getStream(key, range); + } + + std::optional get(const String & key, Range range) override + { + refuseOrPass(key); + return inner->get(key, range); + } + + HeadResult head(const String & key) override + { + refuseOrPass(key); + return inner->head(key); + } + + PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + { + refuseOrPass(key); + return inner->putIfAbsent(key, bytes, meta); + } + + PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, + const ObjectMeta & meta) override + { + refuseOrPass(key); + return inner->putOverwrite(key, bytes, expected, meta); + } + + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override + { + refuseOrPass(key); + return inner->casPut(key, bytes, expected, meta); + } + + DeleteOutcome deleteExact(const String & key, const Token & token) override + { + refuseOrPass(key); + return inner->deleteExact(key, token); + } + + ListPage list(const String & prefix, const String & cursor, size_t limit) override + { + refuseOrPass(prefix); + return inner->list(prefix, cursor, limit); + } + + void publishBlob(const BlobPublishRequest & request) override + { + refuseOrPass(request.destination_key); + inner->publishBlob(request); + } + + SentinelProbeResult probeSentinelRaw(const String & key) override + { + refuseOrPass(key); + return inner->probeSentinelRaw(key); + } + + /// Unhide the base overloads this class's own declarations would otherwise shadow: the + /// convenience forms that omit Range/ObjectMeta/expected-token. + using Backend::get; + using Backend::getStream; + using Backend::head; + using Backend::list; + using Backend::probeSentinelRaw; + using Backend::putIfAbsent; + using Backend::putOverwrite; + using Backend::casPut; + + /// Facts about the wrapped backend, not requests to refuse. + Dialect dialect() const override { return inner->dialect(); } + bool supportsListTokens() const override { return inner->supportsListTokens(); } + uint64_t attemptTimeoutMs() const override { return inner->attemptTimeoutMs(); } + bool refreshCredentials() override { return inner->refreshCredentials(); } + void checkPoolPreconditions() override { inner->checkPoolPreconditions(); } + void checkSkipAccessCheckSupport() override { inner->checkSkipAccessCheckSupport(); } + void checkConditionalWriteSingleAttemptSupport() override { inner->checkConditionalWriteSingleAttemptSupport(); } + +private: + /// Decide this request and record a refusal, then throw outside the lock. + void refuseOrPass(const String & key) + { + bool refusing = false; + { + std::lock_guard lock(mutex); + refusing = mode == Mode::FirstPerKey ? refused_keys.insert(key).second : (++requests % every_nth) == 0; + if (refusing) + ++refusal_counts[key]; + } + if (refusing) + throw S3Exception(error, "throttled by ThrottlingBackend: {}", key); + } + + const BackendPtr inner; + const Mode mode; + const size_t every_nth; + const Aws::S3::S3Errors error; + + mutable std::mutex mutex; + std::set refused_keys; + std::map refusal_counts; + size_t requests = 0; +}; + +} + +#endif diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h new file mode 100644 index 000000000000..4b54449a147f --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h @@ -0,0 +1,21 @@ +#pragma once + +namespace DB::Cas +{ + +/// A capability token: holding one proves the holder is `CasRequests` (or, during the migration, +/// `Backend` -- deleted at the lock, Task 20). Not copyable, not constructible outside those two +/// friends, and carries no data -- its only job is to gate access at compile time to the backend +/// entry points that must not be called except through the contract. +class TransportAccess +{ + friend class CasRequests; + friend class Backend; /// migration only; deleted at the lock (Task 20) + TransportAccess() = default; + +public: + TransportAccess(const TransportAccess &) = delete; + TransportAccess & operator=(const TransportAccess &) = delete; +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h new file mode 100644 index 000000000000..eac6be8562f9 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h @@ -0,0 +1,114 @@ +#pragma once +#include +/// `throwCasWriteRetryLater` / `throwCasTransientUnavailable` are declared here today; the lock moves +/// them into `CasRequests.h` alongside the rest of this contract. +#include +#include +#include + +#include + +#include +#include +#include +#include +#include + +namespace DB::ErrorCodes +{ + extern const int ABORTED; +} + +namespace DB::Cas +{ + +struct Object { String bytes; Incarnation incarnation; }; +struct Meta { uint64_t size; Incarnation incarnation; }; +enum class Removal : uint8_t { Removed, Gone, Mismatch }; + +/// What a write attempt observed of the key's current state before giving up, so a caller (or the +/// message built by `orThrow`) can report exactly what was seen instead of just that something failed. +struct NotObserved {}; +struct ProvenAbsent {}; +using Observation = std::variant; + +/// A durable write landed: `incarnation` names the incarnation it created (or, for a retried write +/// resolved by a read, the incarnation already present), `attempts_sent` counts the HTTP attempts this +/// call made, and `resolved_by_read` is true when the commit was proven by a read rather than by the +/// attempt's own response. +struct Committed { Incarnation incarnation; uint32_t attempts_sent; bool resolved_by_read; }; +/// The write was never attempted or never needed -- e.g. `putIfAbsent` finding the key already +/// present under the caller's intended content. `seen` is whatever the resolve read observed. +struct Declined { Observation seen; }; +/// A competing write won: the key's current state does not match what this call expected. +struct Conflict { Observation seen; }; +/// The store itself refused the request (not a lost precondition) -- `store_error` is a ClickHouse +/// error code and `message` explains it. +struct Refused { int store_error; String message; }; +/// No attempt landed and none can be proven safe to keep making. +struct GaveUp +{ + enum class Why : uint8_t { Deadline, FenceLost, Unresolved }; + enum class Source : uint8_t { Policy, Lease }; + Why why; Source deadline_source; bool sent_any; Observation last_seen; +}; +using WriteResult = std::variant; + +namespace detail +{ + +/// Helper for std::visit with multiple lambdas; no shared one exists in the tree yet. +template +struct Overload : Ts... +{ + using Ts::operator()...; +}; +template +Overload(Ts...) -> Overload; + +inline String renderObservation(const Observation & seen) +{ + return std::visit(Overload{ + [](const NotObserved &) -> String { return "nothing observed"; }, + [](const ProvenAbsent &) -> String { return "absent"; }, + [](const Meta &) -> String { return "present (meta)"; }, + [](const Object & o) -> String { return "present (" + o.incarnation.render() + ")"; }}, seen); +} + +} + +/// Collapse a `WriteResult` into the incarnation a caller can act on: `nullopt` for a declined write +/// (nothing changed, nothing to report), the committed incarnation otherwise -- or throw, mapping +/// every non-success alternative to the error class its meaning already implies. `what` names the +/// call for the thrown message. +inline std::optional orThrow(WriteResult && result, std::string_view what) +{ + using detail::Overload; + using detail::renderObservation; + return std::visit(Overload{ + [](Committed & c) -> std::optional { return std::move(c.incarnation); }, + [](Declined &) -> std::optional { return std::nullopt; }, + [&](Conflict & c) -> std::optional + { + throw Exception(ErrorCodes::ABORTED, "{}: conflict, observed {}", what, renderObservation(c.seen)); + }, + [&](Refused & r) -> std::optional + { + throw Exception(r.store_error, "{}: the store refused the write: {}", what, r.message); + }, + [&](GaveUp & g) -> std::optional + { + switch (g.why) + { + case GaveUp::Why::FenceLost: + throwCasTransientUnavailable(String(what), "mount fence tripped: the durable write is refused because this node no longer holds the mount incarnation it was admitted under"); + case GaveUp::Why::Deadline: + throwCasWriteRetryLater(fmt::format("{}: gave up at the {} deadline after {} attempt(s)", what, g.deadline_source == GaveUp::Source::Lease ? "lease" : "policy", g.sent_any ? "one or more" : "zero")); + case GaveUp::Why::Unresolved: + throwCasWriteRetryLater(fmt::format("{}: the write is unresolved (sent, resolve read found {})", what, renderObservation(g.last_seen))); + } + UNREACHABLE(); + }}, result); +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index 5d00341b1d71..42f7f2c6684e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -681,8 +681,6 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP const auto mode = object_storage->getType() == ObjectStorageType::Local ? Cas::ObjectStorageBackend::Mode::EmulatedSingleProcess : Cas::ObjectStorageBackend::Mode::Native; - auto backend = std::make_shared(object_storage, mode); - const Cas::TokenType backend_token_type = backend->nativeTokenType(); /// EmulatedSingleProcess emulates the conditional-op / exact-token semantics in-process (local /// object storage has none). That emulation is per-process: two servers pointed at the SAME local @@ -765,10 +763,19 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.gc_meta_pool_size = gc_meta_pool_size; pool_config.event_sink = makeCasEventSink(); + /// Built here rather than above so it carries the budget the pool was configured with. Only a + /// WRITABLE Native mount takes the single-attempt profile for its control-plane requests: it owns + /// its own deadline and retry policy, and a transparently retried request would outlive them. A + /// read-only mount has no such deadline, so it keeps the storage's default. + auto backend = std::make_shared( + object_storage, mode, + /*single_attempt_control_plane_=*/!read_only && mode == Cas::ObjectStorageBackend::Mode::Native, + pool_config.cas_request_budget.attempt_timeout_ms); + PoolView view; view.physical_key_prefix = physical_key_prefix_local; view.pool_prefix = pool_prefix; - view.native_token_type = backend_token_type; + view.native_token_type = backend->nativeTokenType(); view.pool = Cas::Pool::open(std::move(backend), std::move(pool_config)); return view; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp index 23b19ddc3508..4ec24eeae61e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp @@ -1,6 +1,9 @@ #include #include +#include + +#include #include namespace DB @@ -133,6 +136,18 @@ constexpr FormatTraits TRAITS[] = }; } +uint64_t casMaxStoredObjectBytes() +{ + static const uint64_t bound = [] + { + uint64_t largest_uncompressed = 0; + for (const FormatTraits & t : TRAITS) + largest_uncompressed = std::max(largest_uncompressed, t.object_cap); + return static_cast(ZSTD_compressBound(largest_uncompressed)); + }(); + return bound; +} + const FormatTraits & traitsFor(FormatId id) { for (const FormatTraits & t : TRAITS) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h index 7b287a01ae9d..bfcb8cf652fb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h @@ -126,6 +126,13 @@ const FormatTraits & traitsFor(FormatId id); /// because callers use this result to classify the input before decoding it. const FormatTraits * traitsForType(std::string_view type); std::span allRegisteredFormatIds(); + +/// The largest number of STORED bytes any materialized object may occupy: the largest whole-object +/// cap in the registry, expanded by zstd's worst-case bound because the `Always` policy stores those +/// objects compressed. A materialized read refuses anything larger rather than allocating it. A +/// format with no whole-object cap is streamed rather than materialized, so it does not raise this +/// bound. +uint64_t casMaxStoredObjectBytes(); /// Returns the storage-key suffix for `id`: `.zst` for `Always`, and an empty suffix otherwise. /// Key builders use this policy directly so a point lookup never has to inspect the object body or /// try multiple keys. diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp index 2d4088d813b0..fe8401e2d3b2 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp @@ -67,6 +67,35 @@ ObjectStorageIteratorPtr IObjectStorage::iterate( return std::make_shared(std::move(files)); } +ObjectStorageIteratorPtr IObjectStorage::iterate( + const std::string & path_prefix, + size_t max_keys, + bool with_tags, + const std::optional & start_after, + ObjectStorageRetryProfile profile, + uint64_t /*request_timeout_ms*/) const +{ + if (profile == ObjectStorageRetryProfile::SingleAttempt) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support single-attempt listing requests", getName()); + return iterate(path_prefix, max_keys, with_tags, start_after); +} + +std::optional IObjectStorage::tryGetObjectMetadataWithNativeToken( + const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t /*request_timeout_ms*/) const +{ + if (profile == ObjectStorageRetryProfile::SingleAttempt) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support single-attempt metadata requests", getName()); + return tryGetObjectMetadataWithNativeToken(path, with_tags); +} + +ConditionalRemoveResult IObjectStorage::removeObjectIfTokenMatches( + const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t /*request_timeout_ms*/) +{ + if (profile == ObjectStorageRetryProfile::SingleAttempt) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support single-attempt removal requests", getName()); + return removeObjectIfTokenMatches(object, etag); +} + ThreadPool & IObjectStorage::getThreadPoolWriter() { auto context = Context::getGlobalContextInstance(); diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index 00f7b63a934d..f770ce25c0ac 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -272,6 +272,18 @@ class IObjectStorage bool with_tags, const std::optional & start_after) const; + /// Same, under a chosen retry profile, with `request_timeout_ms` bounding one attempt of it + /// (0 = the storage's own timeout). A storage that cannot execute the profile must refuse: a + /// caller that asked for one attempt has its own deadline, and a transparently retried request + /// would outlive it. + virtual ObjectStorageIteratorPtr iterate( + const std::string & path_prefix, + size_t max_keys, + bool with_tags, + const std::optional & start_after, + ObjectStorageRetryProfile profile, + uint64_t request_timeout_ms) const; + /// Get object metadata if supported. It should be possible to receive at least size of object virtual ObjectMetadata getObjectMetadata(const std::string & path, bool with_tags) const = 0; @@ -286,6 +298,10 @@ class IObjectStorage return tryGetObjectMetadata(path, with_tags); } + /// Same, under a chosen retry profile; see the note on `iterate`. + virtual std::optional tryGetObjectMetadataWithNativeToken( + const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const; + /// Read single object virtual std::unique_ptr readObject( /// NOLINT const StoredObject & object, @@ -352,6 +368,10 @@ class IObjectStorage "Conditional (token-exact) object removal is not implemented for {} object storage", getName()); } + /// Same, under a chosen retry profile; see the note on `iterate`. + virtual ConditionalRemoveResult removeObjectIfTokenMatches( + const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms); + /// Copy object with different attributes if required virtual void copyObject( /// NOLINT const StoredObject & object_from, diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index c0cbf7ceb5f8..392ede659844 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -81,6 +81,7 @@ namespace S3RequestSetting extern const S3RequestSettingsUInt64 min_upload_part_size; extern const S3RequestSettingsUInt64 max_unexpected_write_error_retries; extern const S3RequestSettingsUInt64 max_single_operation_copy_size; + extern const S3RequestSettingsUInt64 max_single_read_retries; } @@ -94,6 +95,7 @@ namespace S3AuthSetting namespace ErrorCodes { extern const int BAD_ARGUMENTS; + extern const int CANNOT_READ_ALL_DATA; extern const int LOGICAL_ERROR; extern const int NOT_IMPLEMENTED; extern const int S3_ERROR; @@ -249,10 +251,29 @@ class S3IteratorAsync final : public IObjectStorageIteratorAsync } +template +auto S3ObjectStorage::refreshAndRetryOnExpiredCredentials(Fn && fn) const +{ + try + { + return fn(); + } + catch (const S3Exception & e) + { + if (!e.isAccessTokenExpiredError() || !credentials_refresh_callback) + throw; + auto new_client = credentials_refresh_callback(); + if (!new_client) + throw; + client->set(std::move(new_client)); + return fn(); + } +} + bool S3ObjectStorage::exists(const StoredObject & object) const { auto settings_ptr = s3_settings.get(); - const bool e = S3::objectExists(*client.get(), uri.bucket, object.remote_path, {}); + const bool e = S3::objectExists(*client->get(), uri.bucket, object.remote_path, {}); return e; } @@ -283,8 +304,31 @@ std::unique_ptr S3ObjectStorage::readObject( /// NOLINT blob_storage_log->local_path = object.local_path; } + const bool single_attempt = read_settings.object_storage_retry_profile == ObjectStorageRetryProfile::SingleAttempt; + S3CredentialsRefreshCallback refresh_callback = credentials_refresh_callback; + if (single_attempt) + { + request_settings[S3RequestSetting::max_single_read_retries] = 1; + if (credentials_refresh_callback) + { + /// Captures the client SLOT and a copy of the callback, never `this`: the buffer this + /// returns can outlive the storage, and a credential expiry firing afterwards would + /// otherwise install a fresh client into a destroyed object. + refresh_callback = [client_slot = client, refresh = credentials_refresh_callback]() + -> std::unique_ptr + { + auto new_client = refresh(); + if (new_client) + client_slot->set(std::move(new_client)); + /// The buffer will not reissue this read, so it has no use for a client; refreshing + /// the disk's is what lets the caller's next request sign with the new credentials. + return nullptr; + }; + } + } + return std::make_unique( - client.get(), + clientForRetryProfile(read_settings.object_storage_retry_profile, read_settings.object_storage_attempt_timeout_ms), uri.bucket, object.remote_path, uri.version_id, @@ -295,7 +339,7 @@ std::unique_ptr S3ObjectStorage::readObject( /// NOLINT /* read_until_position */0, restrict_seek, object.bytes_size ? std::optional(object.bytes_size) : std::nullopt, - credentials_refresh_callback, + refresh_callback, std::move(blob_storage_log)); } @@ -311,7 +355,15 @@ SmallObjectDataWithMetadata S3ObjectStorage::readSmallObjectAndGetObjectMetadata copyDataMaxBytes(*buffer, out, max_size_bytes); out.finalize(); - result.metadata = dynamic_cast(buffer.get())->getObjectMetadataFromTheLastRequest(); + auto * s3_buffer = dynamic_cast(buffer.get()); + if (s3_buffer->responseIdentityChanged()) + throw Exception( + ErrorCodes::CANNOT_READ_ALL_DATA, + "Object '{}' response identity changed between reissued GET requests; " + "the bytes read are not from one incarnation", + object.remote_path); + + result.metadata = s3_buffer->getObjectMetadataFromTheLastRequest(); return result; } @@ -370,13 +422,9 @@ std::unique_ptr S3ObjectStorage::writeObject( /// NOLIN /// The SingleAttempt profile (e.g. CAS conditional writes, RFC cas-s3-timeout-retry-control) rides /// on WriteSettings instead of changing this disk's shared client — every other write keeps using - /// client.get() and its normal retry policy unchanged. getSingleAttemptClient() is only invoked - /// when actually selected, so a plain write never pays for building/locking the clone. - std::shared_ptr used_client; - if (write_settings.object_storage_retry_profile == ObjectStorageRetryProfile::SingleAttempt) - used_client = getSingleAttemptClient(); - else - used_client = client.get(); + /// client->get() and its normal retry policy unchanged. + auto used_client = clientForRetryProfile( + write_settings.object_storage_retry_profile, write_settings.object_storage_attempt_timeout_ms); return std::make_unique( used_client, @@ -396,11 +444,23 @@ ObjectStorageIteratorPtr S3ObjectStorage::iterate( size_t max_keys, bool with_tags, const std::optional & start_after) const +{ + return iterate(path_prefix, max_keys, with_tags, start_after, ObjectStorageRetryProfile::Default, /*request_timeout_ms=*/0); +} + +ObjectStorageIteratorPtr S3ObjectStorage::iterate( + const std::string & path_prefix, + size_t max_keys, + bool with_tags, + const std::optional & start_after, + ObjectStorageRetryProfile profile, + uint64_t request_timeout_ms) const { auto settings_ptr = s3_settings.get(); if (!max_keys) max_keys = settings_ptr->request_settings[S3RequestSetting::list_object_keys_size]; - return std::make_shared(uri.bucket, path_prefix, client.get(), max_keys, with_tags, start_after); + return std::make_shared( + uri.bucket, path_prefix, clientForRetryProfile(profile, request_timeout_ms), max_keys, with_tags, start_after); } void S3ObjectStorage::listObjects(const std::string & path, RelativePathsWithMetadata & children, size_t max_keys) const @@ -423,7 +483,7 @@ void S3ObjectStorage::listObjects(const std::string & path, RelativePathsWithMet { ProfileEventTimeIncrement watch(ProfileEvents::S3ListObjectsMicroseconds); - outcome = client.get()->ListObjectsV2(request); + outcome = client->get()->ListObjectsV2(request); } throwIfError(outcome, "while listing objects in bucket '{}' with prefix '{}' on disk '{}'", uri.bucket, path, disk_name); @@ -461,7 +521,7 @@ void S3ObjectStorage::removeObjectImpl(const StoredObject & object, bool if_exis { auto blob_storage_log = BlobStorageLogWriter::create(disk_name); - deleteFileFromS3(client.get(), uri.bucket, object.remote_path, if_exists, + deleteFileFromS3(client->get(), uri.bucket, object.remote_path, if_exists, blob_storage_log, object.local_path, object.bytes_size, ProfileEvents::DiskS3DeleteObjects); } @@ -489,7 +549,7 @@ void S3ObjectStorage::removeObjectsImpl(const StoredObjects & objects, bool if_e auto settings_ptr = s3_settings.get(); - deleteFilesFromS3(client.get(), uri.bucket, keys, if_exists, + deleteFilesFromS3(client->get(), uri.bucket, keys, if_exists, s3_capabilities, settings_ptr->request_settings[S3RequestSetting::objects_chunk_size_to_delete], blob_storage_log, local_paths_for_blob_storage_log, file_sizes_for_blob_storage_log, ProfileEvents::DiskS3DeleteObjects); @@ -506,6 +566,19 @@ void S3ObjectStorage::removeObjectsIfExist(const StoredObjects & objects) } ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches(const StoredObject & object, const std::string & etag) +{ + return removeObjectIfTokenMatchesImpl(object, etag, client->get()); +} + +ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches( + const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) +{ + return refreshAndRetryOnExpiredCredentials( + [&] { return removeObjectIfTokenMatchesImpl(object, etag, clientForRetryProfile(profile, request_timeout_ms)); }); +} + +ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatchesImpl( + const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client) { S3::DeleteObjectRequest request; request.SetBucket(uri.bucket); @@ -517,7 +590,7 @@ ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches(const Stored ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); - auto outcome = client.get()->DeleteObject(request); + auto outcome = used_client->DeleteObject(request); /// Mirror removeObjectImpl (deleteFileFromS3): every conditional delete lands in /// system.blob_storage_log too — GC reclaim was invisible there otherwise. TokenMismatch @@ -552,7 +625,7 @@ ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches(const Stored bool S3ObjectStorage::conditionalOpsUseGenerationTokens() const { - return client.get()->supportsGcsNativeConditionalRequests(); + return client->get()->supportsGcsNativeConditionalRequests(); } bool S3ObjectStorage::supportsCopyMode(ObjectStorageCopyMode mode) const @@ -572,7 +645,7 @@ std::optional S3ObjectStorage::isBucketVersioningEnabled() const S3::GetBucketVersioningRequest request; request.SetBucket(uri.bucket); - auto outcome = client.get()->GetBucketVersioning(request); + auto outcome = client->get()->GetBucketVersioning(request); if (!outcome.IsSuccess()) { /// The caller only learns "unknown"; the reason is what the operator needs to act on. @@ -652,24 +725,42 @@ static void putObjectsTagOnS3( void S3ObjectStorage::tagObjects(const StoredObjects & objects, const std::string & tag_key, const std::string & tag_value) { Strings keys = collectRemotePaths(objects); - putObjectsTagOnS3(client.get(), uri.bucket, keys, tag_key, tag_value); + putObjectsTagOnS3(client->get(), uri.bucket, keys, tag_key, tag_value); } std::optional S3ObjectStorage::tryGetObjectMetadata(const std::string & path, bool with_tags) const { - return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::Default); + return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::Default, client->get()); } std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeToken(const std::string & path, bool with_tags) const { - return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::NativeConditional); + return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::NativeConditional, client->get()); +} + +std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeToken( + const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const +{ + return refreshAndRetryOnExpiredCredentials( + [&] + { + return tryGetObjectMetadataImpl( + path, + with_tags, + ObjectStorageRequestMode::NativeConditional, + clientForRetryProfile(profile, request_timeout_ms)); + }); } -std::optional S3ObjectStorage::tryGetObjectMetadataImpl(const std::string & path, bool with_tags, ObjectStorageRequestMode request_mode) const +std::optional S3ObjectStorage::tryGetObjectMetadataImpl( + const std::string & path, + bool with_tags, + ObjectStorageRequestMode request_mode, + const std::shared_ptr & used_client) const { auto settings_ptr = s3_settings.get(); auto object_info = S3::getObjectInfoIfExists( - *client.get(), uri.bucket, path, {}, /* with_metadata= */ true, with_tags, request_mode); + *used_client, uri.bucket, path, {}, /* with_metadata= */ true, with_tags, request_mode); if (object_info.size == 0 && object_info.last_modification_time == 0 && object_info.metadata.empty()) return {}; @@ -691,7 +782,7 @@ ObjectMetadata S3ObjectStorage::getObjectMetadata(const std::string & path, bool S3::ObjectInfo object_info; try { - object_info = S3::getObjectInfo(*client.get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); + object_info = S3::getObjectInfo(*client->get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); } catch (DB::Exception & e) { @@ -701,8 +792,8 @@ ObjectMetadata S3ObjectStorage::getObjectMetadata(const std::string & path, bool auto new_client = credentials_refresh_callback(); if (new_client) { - client.set(std::move(new_client)); - object_info = S3::getObjectInfo(*client.get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); + client->set(std::move(new_client)); + object_info = S3::getObjectInfo(*client->get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); updated = true; } } @@ -735,9 +826,9 @@ void S3ObjectStorage::copyObjectToAnotherObjectStorage( // NOLINT /// Shortcut for S3 if (auto * dest_s3 = dynamic_cast(&object_storage_to); dest_s3 != nullptr) { - auto current_client = dest_s3->client.get(); + auto current_client = dest_s3->client->get(); auto settings_ptr = s3_settings.get(); - auto size = S3::getObjectSize(*client.get(), uri.bucket, object_from.remote_path, {}); + auto size = S3::getObjectSize(*client->get(), uri.bucket, object_from.remote_path, {}); auto scheduler = threadPoolCallbackRunnerUnsafe(getThreadPoolWriter(), ThreadName::S3_COPY_POOL); const auto read_settings_to_use = patchSettings(read_settings); @@ -777,7 +868,7 @@ void S3ObjectStorage::copyObjectToAnotherObjectStorage( // NOLINT if (new_client) { updated = true; - client.set(std::move(new_client)); + client->set(std::move(new_client)); } } if (!updated) @@ -811,7 +902,7 @@ void S3ObjectStorage::copyObject( // NOLINT "(allow_native_copy=false) for object storage {}", getName()); - auto current_client = client.get(); + auto current_client = client->get(); auto settings_ptr = s3_settings.get(); auto size = S3::getObjectSize(*current_client, uri.bucket, object_from.remote_path, {}); auto scheduler = threadPoolCallbackRunnerUnsafe(getThreadPoolWriter(), ThreadName::S3_COPY_POOL); @@ -841,13 +932,13 @@ void S3ObjectStorage::shutdown() /// If S3 request is failed and the method below is executed S3 client immediately returns the last failed S3 request outcome. /// If S3 is healthy nothing wrong will be happened and S3 requests will be processed in a regular way without errors. /// This should significantly speed up shutdown process if S3 is unhealthy. - const_cast(*client.get()).DisableRequestProcessing(); + const_cast(*client->get()).DisableRequestProcessing(); } void S3ObjectStorage::startup() { /// Need to be enabled if it was disabled during shutdown() call. - const_cast(*client.get()).EnableRequestProcessing(); + const_cast(*client->get()).EnableRequestProcessing(); } void S3ObjectStorage::applyNewSettings( @@ -926,7 +1017,7 @@ void S3ObjectStorage::applyNewSettings( && (current_settings->auth_settings.hasUpdates(modified_settings->auth_settings) || for_disk_s3)) { auto new_client = getClient(uri, *modified_settings, context, for_disk_s3, disk_name); - client.set(std::move(new_client)); + client->set(std::move(new_client)); } s3_settings.set(std::move(modified_settings)); } @@ -941,20 +1032,26 @@ ObjectStorageKeyGeneratorPtr S3ObjectStorage::createKeyGenerator() const std::shared_ptr S3ObjectStorage::getS3StorageClient() { - return client.get(); + return client->get(); } std::shared_ptr S3ObjectStorage::tryGetS3StorageClient() { - return client.get(); + return client->get(); } -std::shared_ptr S3ObjectStorage::getSingleAttemptClient() const +std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64_t request_timeout_ms) const { - auto base = client.get(); + auto base = client->get(); std::lock_guard lock(single_attempt_client_mutex); - if (single_attempt_client && single_attempt_client_base == base) - return single_attempt_client; + if (single_attempt_client_base != base) + { + single_attempt_clients.clear(); + single_attempt_client_base = base; + } + + if (auto it = single_attempt_clients.find(request_timeout_ms); it != single_attempt_clients.end()) + return it->second; auto cfg = base->getClientConfiguration(); cfg.retry_strategy.max_retries = 0; @@ -967,9 +1064,20 @@ std::shared_ptr S3ObjectStorage::getSingleAttemptClient() cons if (cfg.expect_continue_min_bytes == 0) cfg.expect_continue_min_bytes = fallback_expect_continue_min_bytes; - single_attempt_client = base->cloneWithConfigurationOverride(cfg); - single_attempt_client_base = base; - return single_attempt_client; + if (request_timeout_ms != 0) + cfg.requestTimeoutMs = static_cast(request_timeout_ms); + + return single_attempt_clients.emplace(request_timeout_ms, base->cloneWithConfigurationOverride(cfg)).first->second; +} + +std::shared_ptr S3ObjectStorage::clientForRetryProfile( + ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const +{ + /// getSingleAttemptClient is only invoked when actually selected, so an ordinary request never + /// pays for building or locking the clone. + if (profile == ObjectStorageRetryProfile::SingleAttempt) + return getSingleAttemptClient(request_timeout_ms); + return client->get(); } bool S3ObjectStorage::tryRefreshCredentialsViaCallback() @@ -981,7 +1089,7 @@ bool S3ObjectStorage::tryRefreshCredentialsViaCallback() auto new_client = credentials_refresh_callback(); if (!new_client) return false; - client.set(std::move(new_client)); + client->set(std::move(new_client)); return true; } } diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 208c83abec81..866d9ab9fdd3 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -5,6 +5,7 @@ #if USE_AWS_S3 #include +#include #include #include #include @@ -44,7 +45,7 @@ class S3ObjectStorage : public IObjectStorage const S3CredentialsRefreshCallback & credentials_refresh_callback_ = [] -> std::unique_ptr{ return nullptr; }) : uri(uri_) , disk_name(disk_name_) - , client(std::move(client_)) + , client(std::make_shared>(std::move(client_))) , s3_settings(std::move(s3_settings_)) , s3_capabilities(s3_capabilities_) , key_generator(std::move(key_generator_)) @@ -104,6 +105,14 @@ class S3ObjectStorage : public IObjectStorage bool with_tags, const std::optional & start_after) const override; + ObjectStorageIteratorPtr iterate( + const std::string & path_prefix, + size_t max_keys, + bool with_tags, + const std::optional & start_after, + ObjectStorageRetryProfile profile, + uint64_t request_timeout_ms) const override; + /// Uses `DeleteObjectRequest`. void removeObjectIfExists(const StoredObject & object) override; @@ -114,6 +123,9 @@ class S3ObjectStorage : public IObjectStorage /// Uses `DeleteObjectRequest` with `If-Match` (token-exact removal for content-addressed disks). ConditionalRemoveResult removeObjectIfTokenMatches(const StoredObject & object, const std::string & etag) override; + ConditionalRemoveResult removeObjectIfTokenMatches( + const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) override; + void tagObjects(const StoredObjects & objects, const std::string & tag_key, const std::string & tag_value) override; ObjectMetadata getObjectMetadata(const std::string & path, bool with_tags) const override; @@ -124,6 +136,9 @@ class S3ObjectStorage : public IObjectStorage /// `nativeHead` can read a GCS generation token where the client's HTTP layer supports one. std::optional tryGetObjectMetadataWithNativeToken(const std::string & path, bool with_tags) const override; + std::optional tryGetObjectMetadataWithNativeToken( + const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const override; + void copyObject( /// NOLINT const StoredObject & object_from, const StoredObject & object_to, @@ -181,20 +196,40 @@ class S3ObjectStorage : public IObjectStorage /// (SingleAttemptRetryStrategy, max_retries=0, Expect:100-continue floor). Rebuilt whenever the /// disk client rotates (applyNewSettings/credentials refresh) — the cached clone is keyed by the /// base client's identity, so a stale clone can never outlive a rotation. - std::shared_ptr getSingleAttemptClient() const; + /// `request_timeout_ms` overrides the clone's send/receive inactivity bound; 0 keeps the disk's. + std::shared_ptr getSingleAttemptClient(uint64_t request_timeout_ms) const; private: void removeObjectImpl(const StoredObject & object, bool if_exists); void removeObjectsImpl(const StoredObjects & objects, bool if_exists); /// Shared by tryGetObjectMetadata/tryGetObjectMetadataWithNativeToken: the only difference between /// the two public overrides is which ObjectStorageRequestMode the HEAD wrapper carries. - std::optional tryGetObjectMetadataImpl(const std::string & path, bool with_tags, ObjectStorageRequestMode request_mode) const; + std::optional tryGetObjectMetadataImpl( + const std::string & path, + bool with_tags, + ObjectStorageRequestMode request_mode, + const std::shared_ptr & used_client) const; + + ConditionalRemoveResult removeObjectIfTokenMatchesImpl( + const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client); + + std::shared_ptr clientForRetryProfile(ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const; + + /// Runs `fn` and, if it failed because the vended credentials expired, refreshes this disk's + /// client and runs it once more. `fn` must re-read the client itself, so the second run signs + /// with the refreshed one. + template + auto refreshAndRetryOnExpiredCredentials(Fn && fn) const; const S3::URI uri; std::string disk_name; - mutable MultiVersion client; + /// The slot this disk's client lives in, and the only thing a credential refresh installs into. + /// Held by `shared_ptr` so a read buffer -- which can outlive this storage -- carries the SLOT + /// rather than a pointer to the storage: a refresh that arrives late then replaces a client + /// nobody will read again, instead of writing into a destroyed object. + const std::shared_ptr> client; MultiVersion s3_settings; S3Capabilities s3_capabilities; @@ -211,13 +246,16 @@ class S3ObjectStorage : public IObjectStorage std::atomic pinned_generation_dialect{-1}; /// -1 unpinned, 0 pinned ETag, 1 pinned generation mutable std::mutex single_attempt_client_mutex; - mutable std::shared_ptr single_attempt_client; - /// The base client the cached clone above was built from. Deliberately held as a shared_ptr (not + /// One clone per requested timeout: the verbs of one operation ask for different bounds, and a + /// single slot would rebuild a whole S3 client (and lose its connection pool) on every + /// alternation between them. + mutable std::map> single_attempt_clients; + /// The base client the cached clones above were built from. Deliberately held as a shared_ptr (not /// a raw pointer): a raw pointer would be compared for identity AFTER the object it once pointed /// to could have been freed and a new client reallocated at the same address by an unrelated /// rotation (ABA), which would false-match and serve a stale clone (e.g. built from retired /// credentials) indefinitely. Holding the shared_ptr pins at most one retired client version — - /// released as soon as the next rotation is observed and the clone is rebuilt — which is what + /// released as soon as the next rotation is observed and the clones are dropped — which is what /// makes the identity comparison in getSingleAttemptClient sound. mutable std::shared_ptr single_attempt_client_base; }; diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index aea9ec64e3ac..e63ce4ae5b1c 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -141,6 +141,27 @@ inline DB::ContentAddressedSettings makeSettingsForTest(const std::string & serv return settings; } +/// A clock that only ever moves when something sleeps on it, plus the record of every sleep it +/// served. Injected into `CasRequests` so a policy's whole 90-second deadline is exercised in a test +/// that takes no wall-clock time, and so the schedule itself -- how many pauses, how long -- becomes +/// an assertion rather than a wait. Single-threaded by construction: two threads sharing one would +/// race on both fields, so a concurrency test uses the real clock instead. +struct FakeClock +{ + uint64_t now = 1'000'000; + std::vector sleeps; + + std::function nowFn() { return [this] { return now; }; } + std::function sleepFn() + { + return [this](uint64_t ms) + { + sleeps.push_back(ms); + now += ms; + }; + } +}; + /// Run `fn`, expect a DB::Exception with EXACTLY `expected_code` (CORRUPTED_DATA-vs-NOT_IMPLEMENTED /// is part of the fail-closed contract: an unknown future format must be NOT_IMPLEMENTED, never /// misreported as corruption). @@ -1448,14 +1469,92 @@ inline std::vector publishCommittedOps(const String & ref_name, class CountingBackend : public DB::Cas::InMemoryBackend { public: - /// Unhide the base convenience overloads (omitted Range/ObjectMeta/expected-token forms): the - /// overrides below would otherwise shadow them for callers holding a concrete backend type. + /// Unhide the base overloads the legacy overrides below would otherwise shadow: the convenience + /// forms that omit Range/ObjectMeta/expected-token, and the transport primitives that share the + /// `head` and `list` names. using DB::Cas::Backend::get; using DB::Cas::Backend::getStream; + using DB::Cas::Backend::head; + using DB::Cas::Backend::list; using DB::Cas::Backend::putIfAbsent; using DB::Cas::Backend::putOverwrite; using DB::Cas::Backend::casPut; + /// ---- The transport primitives: every PHYSICAL request, whichever surface it entered through ---- + /// + /// Counted apart from the legacy counters below, which split by the surface the CALLER used. A + /// `CasRequests` call speaks only these, so an engine test asserting `putTotal` would assert on a + /// surface the engine never touches. Each counter ticks BEFORE the request is served, so an + /// injected failure still counts as a request issued. + /// + /// A legacy call is counted here TOO, because it reaches the store through its primitive -- with + /// the two exceptions `InMemoryBackend` documents, `putIfAbsent` and `casPut`, which route around + /// the primitive to keep their write knobs' verb identity and so land only on the legacy counters. + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(count_mutex); + ++read_requests; + ++read_request_counts[key]; + } + return InMemoryBackend::read(key, access); + } + + std::optional head(const String & key, DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(count_mutex); + ++head_requests; + ++head_request_counts[key]; + } + return InMemoryBackend::head(key, access); + } + + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(count_mutex); + ++list_requests; + ++list_request_counts[prefix]; + } + return InMemoryBackend::list(prefix, cursor, limit, access); + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(count_mutex); + ++write_requests; + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, + DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(count_mutex); + ++remove_requests; + } + return InMemoryBackend::remove(key, expected_value, access); + } + + /// Per-key counterparts of the three READ primitives. The aggregate counters above cannot say + /// WHICH key a request went to, and the legacy per-key counters below never see a caller that + /// speaks the primitives -- `probeSentinelRaw` is one, so a probe is invisible to `headCount`. + uint64_t readRequestCount(const String & key) const { return lookup(read_request_counts, key); } + uint64_t headRequestCount(const String & key) const { return lookup(head_request_counts, key); } + uint64_t listRequestCount(const String & prefix) const { return lookup(list_request_counts, prefix); } + + uint64_t readRequests() const { std::lock_guard lock(count_mutex); return read_requests; } + uint64_t headRequests() const { std::lock_guard lock(count_mutex); return head_requests; } + uint64_t listRequests() const { std::lock_guard lock(count_mutex); return list_requests; } + uint64_t writeRequests() const { std::lock_guard lock(count_mutex); return write_requests; } + uint64_t removeRequests() const { std::lock_guard lock(count_mutex); return remove_requests; } + DB::Cas::HeadResult head(const String & key) override { { @@ -1651,7 +1750,10 @@ class CountingBackend : public DB::Cas::InMemoryBackend whole_get_counts.clear(); head_total = get_total = put_total = cas_put_total = get_stream_total = list_total = delete_total = 0; put_overwrite_total = 0; - + read_requests = head_requests = list_requests = write_requests = remove_requests = 0; + read_request_counts.clear(); + head_request_counts.clear(); + list_request_counts.clear(); } private: @@ -1681,6 +1783,14 @@ class CountingBackend : public DB::Cas::InMemoryBackend uint64_t get_stream_total = 0; uint64_t list_total = 0; uint64_t delete_total = 0; + std::map read_request_counts; + std::map head_request_counts; + std::map list_request_counts; + uint64_t read_requests = 0; + uint64_t head_requests = 0; + uint64_t list_requests = 0; + uint64_t write_requests = 0; + uint64_t remove_requests = 0; }; /// Records the ORDER of body-PUT / `_ckpt`-CAS operations (so a test can compare indices) and lets a @@ -1809,6 +1919,8 @@ template class HintHoleBackendOn : public Base { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using Base::list; /// Hide every key under `prefix` from LIST -- a whole namespace, including objects a later publish /// adds. void hidePrefix(const String & prefix) diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index 6a01821d9472..ccef312ef908 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -10,6 +10,10 @@ #include #include #include +#include +#include +#include +#include #include #include @@ -18,13 +22,10 @@ #include #if USE_AWS_S3 -#include #include #include -#include #include #include -#include #include #include #include @@ -38,6 +39,7 @@ namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; extern const int NOT_IMPLEMENTED; +extern const int LOGICAL_ERROR; } namespace @@ -168,6 +170,28 @@ struct NullBackend final : Backend return ListPage{}; } + /// The primitives, equally trivial. The legacy overrides above still answer the legacy calls -- + /// that is what this double is for -- so these exist to make the class concrete and to pin that + /// implementing the primitive surface alone is enough. + std::optional read(const String & /*key*/, TransportAccess &) override { return std::nullopt; } + std::optional head(const String & /*key*/, TransportAccess &) override { return std::nullopt; } + RawListPage list(const String & /*prefix*/, const String & /*cursor*/, size_t /*limit*/, TransportAccess &) override + { + return RawListPage{}; + } + RawRemoval remove(const String & /*key*/, const String & /*expected_value*/, TransportAccess &) override + { + return RawRemoval::Gone; + } + std::expected write(const String & /*key*/, const String & /*bytes*/, + const std::optional &, TransportAccess &) override + { + return std::unexpected(RawConflict{}); + } + std::unique_ptr stream(const String & /*key*/, TransportAccess &) override { return nullptr; } + void publish(const BlobPublishRequest & /*request*/, TransportAccess &) override {} + Dialect dialect() const override { return Dialect::Emulated; } + bool supportsListTokens() const override { return false; } }; @@ -211,13 +235,6 @@ TEST(CASBackend, NullBackendShapeAndDefaults) ListPage page = b.list("p/", "", 10); EXPECT_TRUE(page.keys.empty()); EXPECT_TRUE(page.next_cursor.empty()); - - // Range::whole() helper - EXPECT_TRUE(Range{}.whole()); - Range r1; r1.offset = 1; - EXPECT_FALSE(r1.whole()); - Range r2; r2.length = 5u; - EXPECT_FALSE(r2.whole()); } // ===================================================================== @@ -278,13 +295,13 @@ TEST(CASInMemory, DeleteExactEnforced) EXPECT_EQ(b.deleteExact("k", t1).kind, DeleteOutcome::Kind::NotFound); } -TEST(CASInMemory, RangeGetAndHeadAndList) +TEST(CASInMemory, GetAndHeadAndList) { InMemoryBackend b; b.putIfAbsent("p/a", "0123456789"); b.putIfAbsent("p/b", "xy"); b.putIfAbsent("q/c", "z"); - EXPECT_EQ(b.get("p/a", Range{.offset = 2, .length = 3})->bytes, "234"); + EXPECT_EQ(b.get("p/a")->bytes, "0123456789"); auto h = b.head("p/a"); EXPECT_TRUE(h.exists); EXPECT_EQ(h.size, 10u); @@ -467,21 +484,6 @@ TEST(CASInMemoryFaults, VersioningMarkerMode) EXPECT_TRUE(b.deleteExact("k", t1).created_delete_marker); // probe must reject this pool } -TEST(CASInMemoryBackend, RoundTripsUserMetadata) -{ - DB::Cas::InMemoryBackend backend; - const DB::Cas::ObjectMeta meta{{"cas_owner", "ab:7:42"}}; - ASSERT_EQ(backend.putIfAbsent("k/key", "body", meta).outcome, DB::Cas::PutOutcome::Done); - - const auto hr = backend.head("k/key"); - ASSERT_TRUE(hr.exists); - ASSERT_EQ(hr.attributes.at("cas_owner"), "ab:7:42"); - - const auto gr = backend.get("k/key"); - ASSERT_TRUE(gr.has_value()); - ASSERT_EQ(gr->attributes.at("cas_owner"), "ab:7:42"); -} - // ===================================================================== // getStream seam (forward-only reads of write-once objects) // ===================================================================== @@ -594,6 +596,70 @@ TEST(CASInstrumentedBackend, PublishBlobDelegatesOnceAndRecordsOnePhysicalBlobWr // M-C2 Task 2: typed S3 precondition signal // ===================================================================== +/// The per-dialect grammar in isolation, independent of any backend fixture. +TEST(CASBackendGrammar, GenerationDialectAcceptsOnlyCanonicalPositiveDecimal) +{ + using DB::Cas::ObjectStorageBackend; + using DB::Cas::TokenType; + EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "123")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "0")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "00123")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "\"123\"")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "12a")); + EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(TokenType::ETag, "\"abc\"")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::ETag, " * ")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::ETag, "a,b")); + EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(TokenType::Emulated, "7")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Emulated, "")); +} + +/// §1 (opt round-B): the fold/point GETs read tiny bodies but a default `ReadBufferFromS3` preallocates +/// ~1 MiB. `casSizedReadSettings` shrinks the buffer to the known body size + slack, capped at the +/// caller's default — never larger than before, regardless of the reported size. +TEST(CASSizedReadSettings, CapsToKnownSizePlusSlackButNeverAboveBase) +{ + DB::ReadSettings base; + base.remote_fs_settings.buffer_size = 1ULL << 20; /// 1 MiB default + base.local_fs_settings.buffer_size = 1ULL << 20; + + /// A ~3.7 KB fold body: buffer shrinks to size + slack, far below the 1 MiB default. + const auto small = DB::Cas::casSizedReadSettings(base, 3700); + EXPECT_EQ(small.remote_fs_settings.buffer_size, 3700 + DB::Cas::CAS_FOLD_READ_SLACK_BYTES); + EXPECT_EQ(small.local_fs_settings.buffer_size, 3700 + DB::Cas::CAS_FOLD_READ_SLACK_BYTES); + + /// A body larger than the default is capped AT the default (never grown). + const auto big = DB::Cas::casSizedReadSettings(base, 8ULL << 20); + EXPECT_EQ(big.remote_fs_settings.buffer_size, 1ULL << 20); + + /// Unknown size (0) = leave the base untouched (the metadata-fetch fallback path). + const auto unknown = DB::Cas::casSizedReadSettings(base, 0); + EXPECT_EQ(unknown.remote_fs_settings.buffer_size, 1ULL << 20); +} + +/// The CountingBackend request-shape recorders that the streaming-memory gates consume: per-key and +/// total getStream counts, and the whole-object get flag that marks a resident-memory violation for a +/// run object. The ranged-window recorder is no longer exercised here: a materialized read is always +/// whole now, so only `getStream` still carries a window. +TEST(CASCountingBackendShape, RecordsGetStreamAndWholeGetShape) +{ + DB::Cas::tests::CountingBackend backend; + backend.putIfAbsent("k", String(1000, 'x')); + + backend.get("k"); + EXPECT_EQ(backend.wholeGetCount("k"), 1u); + + /// getStream counters (per-key and total). + backend.getStream("k", DB::Cas::Range{.offset = 2, .length = 5}); + backend.getStream("k"); + backend.getStream("absent"); + EXPECT_EQ(backend.getStreamCount("k"), 2u); + EXPECT_EQ(backend.getStreamTotal(), 3u); + + backend.resetCounts(); + EXPECT_EQ(backend.wholeGetCount("k"), 0u); + EXPECT_EQ(backend.getStreamTotal(), 0u); +} + #if USE_AWS_S3 namespace @@ -1041,97 +1107,6 @@ TEST(CASS3Signal, FinalizeClassifierMapsPreconditionLossExactly) namespace { -/// A `LocalObjectStorage` that round-trips user metadata in-process. The production -/// `LocalObjectStorage` deliberately drops the `attributes` argument of `writeObject` and never -/// populates `ObjectMetadata::attributes` (local files carry no `x-amz-meta-*`), so it cannot stand -/// in for S3/RustFS when verifying the metadata threading. This test-only subclass records the -/// attributes passed on write, keyed by physical path, and injects them back on metadata reads — -/// exactly what a real object store does for `x-amz-meta-*`. It exercises the `EmulatedSingleProcess` -/// `ObjectStorageBackend` threading (`putIfAbsent` → `writeObject` attributes → `head` attributes) -/// without a live S3 backend; the real S3/RustFS round trip is verified empirically out-of-band. -class AttributePreservingLocalObjectStorage final : public DB::LocalObjectStorage -{ -public: - using DB::LocalObjectStorage::LocalObjectStorage; - - std::unique_ptr writeObject( - const DB::StoredObject & object, - DB::WriteMode mode, - std::optional attributes, - size_t buf_size, - const DB::WriteSettings & write_settings) override - { - if (attributes.has_value()) - { - std::lock_guard lock(mutex); - saved_attributes[object.remote_path] = *attributes; - } - return DB::LocalObjectStorage::writeObject(object, mode, attributes, buf_size, write_settings); - } - - std::optional tryGetObjectMetadata(const std::string & path, bool with_tags) const override - { - auto metadata = DB::LocalObjectStorage::tryGetObjectMetadata(path, with_tags); - if (metadata) - inject(path, *metadata); - return metadata; - } - - DB::ObjectMetadata getObjectMetadata(const std::string & path, bool with_tags) const override - { - auto metadata = DB::LocalObjectStorage::getObjectMetadata(path, with_tags); - inject(path, metadata); - return metadata; - } - -private: - void inject(const std::string & path, DB::ObjectMetadata & metadata) const - { - std::lock_guard lock(mutex); - if (auto it = saved_attributes.find(path); it != saved_attributes.end()) - metadata.attributes = it->second; - } - - mutable std::mutex mutex; - mutable std::map saved_attributes; -}; - -DB::ObjectStoragePtr makeAttributePreservingStorageForTest() -{ - static std::atomic counter{0}; - const auto unique = std::to_string(::getpid()) + "_" + std::to_string(counter.fetch_add(1)); - const auto root = (std::filesystem::temp_directory_path() / ("cas_meta_unit_" + unique)).string(); - - std::error_code ec; - std::filesystem::remove_all(root, ec); - std::filesystem::create_directories(root, ec); - - DB::LocalObjectStorageSettings settings("test", root, /*read_only_=*/false); - return std::make_shared(std::move(settings)); -} - -} - -/// The `EmulatedSingleProcess` `ObjectStorageBackend` must thread user metadata through to the -/// underlying object storage's `writeObject` attributes on `putIfAbsent` and read it back into -/// `HeadResult::attributes` on `head`. Verified here over an attribute-preserving object storage -/// (the production `LocalObjectStorage` drops attributes); the live S3/RustFS round trip is verified -/// empirically out-of-band. -TEST(CASObjectStorageBackend, EmulatedRoundTripsUserMetadata) -{ - ObjectStorageBackend backend(makeAttributePreservingStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); - - const DB::Cas::ObjectMeta meta{{"cas_owner", "ab:7:42"}}; - ASSERT_EQ(backend.putIfAbsent("k/key", "body", meta).outcome, DB::Cas::PutOutcome::Done); - - const auto hr = backend.head("k/key"); - ASSERT_TRUE(hr.exists); - ASSERT_EQ(hr.attributes.at("cas_owner"), "ab:7:42"); -} - -namespace -{ - /// A `LocalObjectStorage` whose `readObject` throws `S3Exception(NO_SUCH_KEY)` for a configured /// physical key, while `tryGetObjectMetadata` still reports that key as PRESENT. /// This simulates the HEAD→GET race window: the HEAD succeeds, then the object is deleted before @@ -1225,31 +1200,6 @@ TEST(CASObjectStorageBackend, NativeModeGetReturnsNulloptOnMidGetNoSuchKey) EXPECT_FALSE(result.has_value()); } -/// A ranged `get` over a real `LocalObjectStorage` returns exactly the requested window, with the -/// same clamping the old read-whole-then-substr path had: a window whose offset is at or past EOF -/// yields an empty result. The only-the-window I/O property (no whole-object read) is enforced by -/// the `readObjectRanged` rewrite and cross-checked by the request-size gate in a later task. -TEST(CASObjectStorageBackend, RangedGetReadsOnlyTheWindow) -{ - auto backend = std::make_shared( - tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); - - const String payload = String(300000, 'a') + String(300000, 'b') + String(300000, 'c'); - backend->putIfAbsent("p/obj", payload); - - const auto mid = backend->get("p/obj", DB::Cas::Range{.offset = 300000, .length = 300000}); - ASSERT_TRUE(mid.has_value()); - EXPECT_EQ(mid->bytes, String(300000, 'b')); - - const auto tail = backend->get("p/obj", DB::Cas::Range{.offset = 600000, .length = std::nullopt}); - ASSERT_TRUE(tail.has_value()); - EXPECT_EQ(tail->bytes, String(300000, 'c')); - - const auto past = backend->get("p/obj", DB::Cas::Range{.offset = 1000000, .length = 10}); - ASSERT_TRUE(past.has_value()); - EXPECT_TRUE(past->bytes.empty()); -} - /// codex-review-triage §3.18, finding 19c: the `EmulatedSingleProcess` adapter used to mint tokens /// from a plain in-process counter (`emu_seq`), NOT actually seeded from the underlying object's etag /// despite the class comment's claim. After a process restart (modeled here as a fresh @@ -1605,8 +1555,18 @@ TEST(CASObjectStorageBackend, NativeRejectsWrongDialectTokenBeforeTouchingTheWir auto storage = std::static_pointer_cast(makeCallCountingStorageForTest()); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - ASSERT_EQ(backend.putIfAbsent("k/dialect", "v1").outcome, PutOutcome::Done); - const Token live = backend.head("k/dialect").token; + /// Placed through the object storage rather than through the backend: a Native write over a local + /// storage has no response incarnation to attribute itself to. Native passes the key to the + /// storage verbatim, so this is the same object the backend reads below -- anchored under the + /// storage's own root, since a bare relative key would resolve beside the test process. + const String key = DB::Cas::tests::nativeKeyUnder(storage, "k/dialect"); + { + auto out = storage->writeObject( + DB::StoredObject(key), DB::WriteMode::Rewrite, {}, DB::DBMS_DEFAULT_BUFFER_SIZE, DB::WriteSettings{}); + DB::writeString(String("v1"), *out); + out->finalize(); + } + const Token live = backend.head(key).token; ASSERT_EQ(live.type, TokenType::ETag); storage->write_calls = 0; @@ -1615,65 +1575,58 @@ TEST(CASObjectStorageBackend, NativeRejectsWrongDialectTokenBeforeTouchingTheWir /// Same wire VALUE, wrong dialect TYPE (Emulated instead of this backend's native ETag dialect). const Token wrong_type_token{live.value, TokenType::Emulated}; - EXPECT_EQ(backend.putOverwrite("k/dialect", "v2", wrong_type_token).outcome, PutOutcome::PreconditionFailed); - EXPECT_EQ(backend.casPut("k/dialect", "v2", wrong_type_token).outcome, CasOutcome::Conflict); - EXPECT_EQ(backend.deleteExact("k/dialect", wrong_type_token).kind, DeleteOutcome::Kind::TokenMismatch); + EXPECT_EQ(backend.putOverwrite(key, "v2", wrong_type_token).outcome, PutOutcome::PreconditionFailed); + EXPECT_EQ(backend.casPut(key, "v2", wrong_type_token).outcome, CasOutcome::Conflict); + EXPECT_EQ(backend.deleteExact(key, wrong_type_token).kind, DeleteOutcome::Kind::TokenMismatch); EXPECT_EQ(storage->write_calls.load(), 0); EXPECT_EQ(storage->remove_if_matches_calls.load(), 0); /// The live incarnation must be untouched by all three rejected attempts. - EXPECT_EQ(backend.head("k/dialect").token, live); + EXPECT_EQ(backend.head(key).token, live); } -/// §1 (opt round-B): the fold/point GETs read tiny bodies but a default `ReadBufferFromS3` preallocates -/// ~1 MiB. `casSizedReadSettings` shrinks the buffer to the known body size + slack, capped at the -/// caller's default — never larger than before, regardless of the reported size. -TEST(CASSizedReadSettings, CapsToKnownSizePlusSlackButNeverAboveBase) -{ - DB::ReadSettings base; - base.remote_fs_settings.buffer_size = 1ULL << 20; /// 1 MiB default - base.local_fs_settings.buffer_size = 1ULL << 20; - - /// A ~3.7 KB fold body: buffer shrinks to size + slack, far below the 1 MiB default. - const auto small = DB::Cas::casSizedReadSettings(base, 3700); - EXPECT_EQ(small.remote_fs_settings.buffer_size, 3700 + DB::Cas::CAS_FOLD_READ_SLACK_BYTES); - EXPECT_EQ(small.local_fs_settings.buffer_size, 3700 + DB::Cas::CAS_FOLD_READ_SLACK_BYTES); - - /// A body larger than the default is capped AT the default (never grown). - const auto big = DB::Cas::casSizedReadSettings(base, 8ULL << 20); - EXPECT_EQ(big.remote_fs_settings.buffer_size, 1ULL << 20); +/// The incarnation grammar: an empty, wildcard or list token would turn a conditional mutation into +/// an unconditional one, so every mutation refuses it as a caller bug (LOGICAL_ERROR) rather than +/// forwarding it to the wire or the emu compare -- distinct from a WRONG-dialect token (see +/// NativeRejectsWrongDialectTokenBeforeTouchingTheWire above), which is a graceful non-match, not a +/// malformed value. Under a debug or sanitizer build, constructing a LOGICAL_ERROR exception ABORTS +/// at construction (Exception::handle_error_code), so the same table is asserted as a death +/// expectation there instead; the contract ("a malformed token is refused") is what both forms pin. +#ifndef DEBUG_OR_SANITIZER_BUILD +TEST(CASBackendGrammar, RejectsEmptyStarAndListTokensOnEveryMutation) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - /// Unknown size (0) = leave the base untouched (the metadata-fetch fallback path). - const auto unknown = DB::Cas::casSizedReadSettings(base, 0); - EXPECT_EQ(unknown.remote_fs_settings.buffer_size, 1ULL << 20); + const DB::Cas::Token empty{"", DB::Cas::TokenType::ETag}; + const DB::Cas::Token star{"*", DB::Cas::TokenType::ETag}; + const DB::Cas::Token list{"\"a\", \"b\"", DB::Cas::TokenType::ETag}; + for (const auto & bad : {empty, star, list}) + { + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { backend.putOverwrite("k", "v", bad); }); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { backend.casPut("k", "v", bad); }); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { backend.deleteExact("k", bad); }); + } } +#endif -/// The CountingBackend request-shape recorders that the streaming-memory gates (Task 3/4) consume: -/// per-key/total getStream counts, the max ranged-get window per key, and the whole-object get flag. -TEST(CASCountingBackendShape, RecordsGetStreamAndRangeShape) +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASBackendGrammarDeathTest, RejectsEmptyStarAndListTokensOnEveryMutation) { - DB::Cas::tests::CountingBackend backend; - backend.putIfAbsent("k", String(1000, 'x')); - - /// A whole-object get flags the resident-memory violation; a ranged get tracks the max window. - backend.get("k"); - backend.get("k", DB::Cas::Range{.offset = 0, .length = 100}); - backend.get("k", DB::Cas::Range{.offset = 10, .length = 400}); - EXPECT_EQ(backend.wholeGetCount("k"), 1u); - EXPECT_EQ(backend.maxRangedGetLen("k"), 400u); - - /// getStream counters (per-key and total). - backend.getStream("k", DB::Cas::Range{.offset = 2, .length = 5}); - backend.getStream("k"); - backend.getStream("absent"); - EXPECT_EQ(backend.getStreamCount("k"), 2u); - EXPECT_EQ(backend.getStreamTotal(), 3u); + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - backend.resetCounts(); - EXPECT_EQ(backend.wholeGetCount("k"), 0u); - EXPECT_EQ(backend.maxRangedGetLen("k"), 0u); - EXPECT_EQ(backend.getStreamTotal(), 0u); + const DB::Cas::Token empty{"", DB::Cas::TokenType::ETag}; + const DB::Cas::Token star{"*", DB::Cas::TokenType::ETag}; + const DB::Cas::Token list{"\"a\", \"b\"", DB::Cas::TokenType::ETag}; + for (const auto & bad : {empty, star, list}) + { + EXPECT_DEATH({ (void)backend.putOverwrite("k", "v", bad); }, ""); + EXPECT_DEATH({ (void)backend.casPut("k", "v", bad); }, ""); + EXPECT_DEATH({ (void)backend.deleteExact("k", bad); }, ""); + } } +#endif #endif diff --git a/src/Disks/tests/gtest_cas_backend_contract.cpp b/src/Disks/tests/gtest_cas_backend_contract.cpp index bf5c2652ca2a..4e1855ed1a87 100644 --- a/src/Disks/tests/gtest_cas_backend_contract.cpp +++ b/src/Disks/tests/gtest_cas_backend_contract.cpp @@ -76,14 +76,17 @@ TEST_P(CASBackendContract, DeleteNotFound) EXPECT_EQ(b->deleteExact("k", t1).kind, DeleteOutcome::Kind::NotFound); } -TEST_P(CASBackendContract, RangeGet) +/// `Range` is retired for materialized reads: a non-whole window is REFUSED rather than served, so +/// no caller can silently receive a partial body where it expected the object. +TEST_P(CASBackendContract, RangedGetIsRefusedAndTheWholeReadStillServes) { auto b = GetParam()(); b->putIfAbsent("k", "0123456789"); Range r; r.offset = 2; r.length = 3u; - EXPECT_EQ(b->get("k", r)->bytes, "234"); + EXPECT_THROW(b->get("k", r), DB::Exception); + EXPECT_EQ(b->get("k")->bytes, "0123456789"); } TEST_P(CASBackendContract, Head) diff --git a/src/Disks/tests/gtest_cas_backend_generation.cpp b/src/Disks/tests/gtest_cas_backend_generation.cpp index 7bcd4768f0d7..d0856c44f647 100644 --- a/src/Disks/tests/gtest_cas_backend_generation.cpp +++ b/src/Disks/tests/gtest_cas_backend_generation.cpp @@ -1,5 +1,7 @@ #include #include +#include +#include #include #include @@ -36,6 +38,7 @@ using namespace DB::Cas; namespace DB::ErrorCodes { extern const int NOT_IMPLEMENTED; + extern const int CAS_WRITE_UNATTRIBUTED; } #if USE_AWS_S3 @@ -183,35 +186,49 @@ TEST(CASBackendGeneration, NativeHeadUsesNativeTokenMetadataApi) auto storage = makeRecordingObjectStorageForTest(); auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); - ASSERT_EQ(b->putIfAbsent("p/native-head/key", "v1").outcome, PutOutcome::Done); + /// Placed through the object storage: a Native write over a local storage has no response + /// incarnation to attribute itself to. Native passes the key verbatim, so this is the object the + /// HEAD below reads -- anchored under the storage's own root, since a bare relative key would + /// resolve beside the test process. + const String key = DB::Cas::tests::nativeKeyUnder(storage, "p/native-head/key"); + { + auto out = storage->writeObject( + DB::StoredObject(key), DB::WriteMode::Rewrite, {}, DB::DBMS_DEFAULT_BUFFER_SIZE, DB::WriteSettings{}); + DB::writeString(String("v1"), *out); + out->finalize(); + } - /// `putIfAbsent`'s HEAD-fallback stamping path calls the ordinary API; - /// reset the counters so only nativeHead's call, below, is observed. + /// Only nativeHead's own call may be observed. storage->ordinary_calls = 0; storage->native_calls = 0; - const auto hr = b->head("p/native-head/key"); + const auto hr = b->head(key); ASSERT_TRUE(hr.exists); EXPECT_EQ(storage->native_calls, 1); EXPECT_EQ(storage->ordinary_calls, 0); } -/// Every Token{...} the backend mints must carry native_token_type instead of a hardcoded -/// TokenType::ETag (Task 5). Mode::Native over a LocalObjectStorage has no write-time ETag, so -/// putIfAbsent's PutResult falls back to a HEAD internally — that HEAD is also a stamping site, -/// so the assertion below exercises both the direct-etag and the HEAD-fallback mint paths. +/// Every token the backend mints carries native_token_type rather than a hardcoded TokenType::ETag. +/// The HEAD mint is the site exercised here; the write-response mint has its own tests over the fake +/// S3 client below, which is the only place a Native write can produce a response incarnation. TEST(CASBackendGeneration, StampedTokenTypeFollowsNativeKind) { - auto b = std::make_shared( - DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); b->setNativeTokenTypeForTest(TokenType::Generation); - const auto put = b->putIfAbsent("p/gen/tok", "v1"); - EXPECT_EQ(put.token.type, TokenType::Generation); + /// A local file's etag is its mtime in nanoseconds, which is also a valid generation value. + const String key = DB::Cas::tests::nativeKeyUnder(storage, "p/gen/tok"); + { + auto out = storage->writeObject(DB::StoredObject(key), DB::WriteMode::Rewrite); + DB::writeString(String("v1"), *out); + out->finalize(); + } - const auto hr = b->head("p/gen/tok"); + const auto hr = b->head(key); ASSERT_TRUE(hr.exists); EXPECT_EQ(hr.token.type, TokenType::Generation); + EXPECT_EQ(b->dialect(), TokenType::Generation); } /// A generation-dialect (GCS) mount wants bucket versioning to be verifiably off: a token-exact @@ -666,6 +683,27 @@ TEST(CASBackendGeneration, PublishBlobSucceedsWithoutResponseGeneration) EXPECT_EQ(client->objects.at("p/gen/publish-no-generation"), "freshpayload"); } +/// The write-response half of the incarnation grammar: a write response that carries no ETag at all +/// must not fall back to a HEAD -- there is no HEAD that can attribute the write with certainty, since +/// the object it would read back might not even be the one this call just wrote. This is the +/// ETag-dialect sibling of PublishBlobSucceedsWithoutResponseGeneration above: a publication has no +/// incarnation to attribute in the first place, so it is unaffected by this guard. Default (ETag) +/// dialect here, deliberately NOT stamped Generation, whose own two cases are covered by +/// CASBackendGenerationS3.WriteEmptyGenerationIsUnattributed and WriteNonNumericGenerationIsUnattributed. +TEST(CASBackendGrammar, NamelessWriteResponseThrowsWriteUnattributed) +{ + (void)getContext(); + FakeGenerationS3Client * client = nullptr; + auto storage = makeGenerationS3ObjectStorageForTest(client); + ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); + ASSERT_EQ(backend.nativeTokenType(), TokenType::ETag); + client->put_returns_no_etag = true; + + DB::Cas::tests::expectThrowsCode( + DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, [&] { backend.putIfAbsent("p/gen/nameless-write", "v"); }); + EXPECT_EQ(client->put_object_calls, 1u); +} + /// The moved cap, end to end: a conditional write on a generation store stays in ONE PUT up to the /// cap the OBJECT STORAGE carries, and refuses rather than silently taking the multipart path above /// it -- GCS enforces no precondition on CompleteMultipartUpload. @@ -682,7 +720,7 @@ TEST(CASBackendGeneration, ConditionalWriteHonoursTheObjectStorageConditionalPut EXPECT_NO_THROW(backend.casPut("p/gen/under-cap", small, std::nullopt, ObjectMeta{})); EXPECT_EQ(client->put_object_calls, 1u); EXPECT_EQ(client->create_multipart_calls, 0u); - const auto single_attempt_client = storage->getSingleAttemptClient(); + const auto single_attempt_client = storage->getSingleAttemptClient(/*request_timeout_ms=*/0); EXPECT_NE(dynamic_cast(single_attempt_client.get()), nullptr); EXPECT_NE( dynamic_cast( @@ -711,20 +749,26 @@ TEST(CASBackendGeneration, ConditionalWriteHonoursTheObjectStorageConditionalPut /// the CAS layer receives in the shape production actually produces, which is why a mount that could /// never succeed passed every unit test. These three tests are that crossing. -TEST_F(CASBackendGenerationS3, WriteEmptyGenerationThrows) +TEST_F(CASBackendGenerationS3, WriteEmptyGenerationIsUnattributed) { backend = makeBackend(); + client->next_put_etag = ""; + /// The write may well have landed -- an empty response value says nothing about that -- so this is + /// the resolve-by-reading class, not the corrupt-response one. DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::CORRUPTED_DATA, - [&] { backend->tokenFromWriteResult("p/gen/no-etag", String{}); }); + DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, + [&] { backend->putIfAbsent("p/gen/no-etag", "v"); }); } -TEST_F(CASBackendGenerationS3, WriteNonNumericGenerationThrows) +TEST_F(CASBackendGenerationS3, WriteNonNumericGenerationIsUnattributed) { backend = makeBackend(); + /// An MD5-shaped ETag where a generation belongs: the store answered, but not with an incarnation + /// this dialect can use, and no follow-up read can attribute the write on its behalf. + client->next_put_etag = "\"d41d8cd98f00b204e9800998ecf8427e\""; DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::CORRUPTED_DATA, - [&] { backend->tokenFromWriteResult("p/gen/bad-etag", "\"d41d8cd98f00b204e9800998ecf8427e\""); }); + DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, + [&] { backend->putIfAbsent("p/gen/bad-etag", "v"); }); } /// A mutable conditional write whose response generation arrives quoted -- exactly what @@ -733,8 +777,10 @@ TEST_F(CASBackendGenerationS3, WriteNonNumericGenerationThrows) TEST_F(CASBackendGenerationS3, WriteGenerationTokenStripsTransportQuoting) { backend = makeBackend(); - const Token tok = backend->tokenFromWriteResult("p/gen/quoted-write", "\"1783078552147137\""); - EXPECT_EQ(tok, (Token{"1783078552147137", TokenType::Generation})); + client->next_put_etag = "\"1783078552147137\""; + const auto put = backend->putIfAbsent("p/gen/quoted-write", "v"); + ASSERT_EQ(put.outcome, PutOutcome::Done); + EXPECT_EQ(put.token, (Token{"1783078552147137", TokenType::Generation})); } /// The same crossing on the read side: a marked HEAD whose ETag field carries a quoted generation diff --git a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp index cfb150401cf2..5e2b94a4da5f 100644 --- a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp +++ b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp @@ -44,6 +44,8 @@ const String kProbeUid2 = "fedcba9876543210fedcba9876543210"; class RecordingBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; using Backend::get; using Backend::getStream; using Backend::putIfAbsent; diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 4f7d37e2b4fa..9701330e3406 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -1217,6 +1217,21 @@ class FailDeletesUnderPrefixBackend : public Backend ListPage list(const String & prefix, const String & cursor, size_t limit) override { return inner->list(prefix, cursor, limit); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + /// The transport primitives forward to `inner`; the legacy overrides above are what this + /// double injects through. Declared because `Backend` declares them pure. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + DB::Cas::Dialect dialect() const override { return inner->dialect(); } + private: std::shared_ptr inner; String fail_prefix; diff --git a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp index a674f13667b6..93d23f6a8cb7 100644 --- a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp +++ b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp @@ -56,6 +56,8 @@ bool slotObjectExists(Backend & backend, const String & leaf) class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; void arm() { armed = true; } bool fired() const { return added; } @@ -89,6 +91,8 @@ class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend class MutateCatalogBetweenRetirementReadsBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; void arm() { armed = true; } bool fired() const { return added; } diff --git a/src/Disks/tests/gtest_cas_fence_generation.cpp b/src/Disks/tests/gtest_cas_fence_generation.cpp index dcc1f7b3c4b9..105925ea33d8 100644 --- a/src/Disks/tests/gtest_cas_fence_generation.cpp +++ b/src/Disks/tests/gtest_cas_fence_generation.cpp @@ -43,6 +43,8 @@ namespace class TripOnHeadBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; HeadResult head(const String & key) override { if (trigger) @@ -59,6 +61,8 @@ class TripOnHeadBackend final : public InMemoryBackend class TripOnSecondHeadBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; using Backend::putIfAbsent; HeadResult head(const String & key) override @@ -148,6 +152,8 @@ PartWriteTxnPtr precommittedBuildForBlob( class BlobPublicationFenceBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; enum class TripPoint : uint8_t { OnHead, diff --git a/src/Disks/tests/gtest_cas_forget.cpp b/src/Disks/tests/gtest_cas_forget.cpp index b1c19993e06c..571632bdfb4d 100644 --- a/src/Disks/tests/gtest_cas_forget.cpp +++ b/src/Disks/tests/gtest_cas_forget.cpp @@ -82,6 +82,9 @@ void fenceOutMount(DB::Cas::Backend & backend, const String & mount_key) class ToggleableTransportFaultBackend final : public DB::Cas::InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using DB::Cas::InMemoryBackend::head; + using DB::Cas::InMemoryBackend::list; using Backend::get; using Backend::getStream; using Backend::putIfAbsent; diff --git a/src/Disks/tests/gtest_cas_fsck.cpp b/src/Disks/tests/gtest_cas_fsck.cpp index 77c490656911..41475abdb98c 100644 --- a/src/Disks/tests/gtest_cas_fsck.cpp +++ b/src/Disks/tests/gtest_cas_fsck.cpp @@ -45,6 +45,8 @@ ManifestRef ref(uint64_t seq, uint64_t inst) class RepublishOnListBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; void armOnFirstList(String prefix, std::function mutation) { std::lock_guard lock(arm_mutex); @@ -125,6 +127,8 @@ enum class FsckListingMode : uint8_t class FsckListingBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; void distort(String prefix_, FsckListingMode mode_) { prefix = std::move(prefix_); @@ -500,6 +504,8 @@ namespace class AdmitLifeAfterNamespaceListingBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; explicit AdmitLifeAfterNamespaceListingBackend(NamespaceLifeId life_) : protected_life(std::move(life_)) {} ListPage list(const String & prefix, const String & cursor, size_t limit) override diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index ce6b09099487..0a66240d4ca9 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -1485,6 +1485,8 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa class BroadListHoleBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; String hide_under_prefix; String hidden_key_infix; size_t holes_served = 0; diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index e48ad0e2ff1f..d751aad975d2 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -183,6 +183,9 @@ namespace class ThrowingBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; + using InMemoryBackend::list; ListPage list(const String & prefix, const String & cursor, size_t limit) override { if (arm) @@ -312,6 +315,8 @@ TEST(CASGCLog, AbortedFinishOnThrowingRound) class NetworkThrowingBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; ListPage list(const String & prefix, const String & cursor, size_t limit) override { if (arm) @@ -412,6 +417,8 @@ TEST(CASGCLog, AbortedFinishCarriesProgressiveCounters) class ModalThrowingBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; enum Mode : int { Off = 0, Transient = 1, Logic = 2 }; ListPage list(const String & prefix, const String & cursor, size_t limit) override { diff --git a/src/Disks/tests/gtest_cas_holey_list_detector.cpp b/src/Disks/tests/gtest_cas_holey_list_detector.cpp index 6c1728d3ba31..0c4779c1a672 100644 --- a/src/Disks/tests/gtest_cas_holey_list_detector.cpp +++ b/src/Disks/tests/gtest_cas_holey_list_detector.cpp @@ -56,6 +56,8 @@ namespace class HoleyListBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; /// Omit `key` from the `nth` (0-based) subsequent qualifying `list` call. Resets the counter. void omitFromNthListCall(const String & key, size_t nth) { diff --git a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp index b93166c88eac..be5cda9f9954 100644 --- a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp +++ b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp @@ -63,6 +63,9 @@ void fenceOutMount(Backend & backend, const String & mount_key) class ToggleableTransportFaultBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; + using InMemoryBackend::list; /// Unhide the base convenience overloads, matching every other Backend subclass in this suite. using Backend::get; using Backend::getStream; @@ -129,7 +132,7 @@ TEST(CASLifecycleCondition, SentinelsDeletedEntersIdentityLostTerminal) EXPECT_FALSE(store->tryRemountOnce()); EXPECT_EQ(store->lifecycle(), PoolLifecycle::IdentityLost); EXPECT_EQ(backend->putTotal(), 0u) << "a terminal-IdentityLost gate probe must never claim, allocate, or write"; - EXPECT_GE(backend->headCount(meta_key), 1u) << "the gate still probes _pool_meta authoritatively"; + EXPECT_GE(backend->headRequestCount(meta_key), 1u) << "the gate still probes _pool_meta authoritatively"; } /// (a2) rev.8 worker-exit: `IdentityLost` is terminal, so the persistent self-remount worker must self-exit diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 7a21b24985fd..e0e96bab87b0 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -493,9 +493,12 @@ TEST(CASServerRootEpoch, AllocatorIsMonotoneAndSurvivesMountConcept) EXPECT_GE(e1, 1u); // 0 is a reserved sentinel EXPECT_GT(e2, e1); // strictly increasing - /// Deleting the (separate) mount object must NOT reset the epoch. No mount has been written in - /// Task 4, so deleteExact of a non-existent mount is a NotFound no-op that touches nothing. - const auto del = b->deleteExact(l.mountKey("r"), b->head(l.mountKey("r")).token); + /// Deleting the (separate) mount object must NOT reset the epoch. No mount has been written yet, + /// so deleteExact of it is a NotFound no-op that touches nothing -- exercised with a well-formed + /// placeholder token, not the absent HeadResult's empty one: InMemoryBackend refuses a malformed + /// token as a caller bug before it ever looks the key up, exactly like the production backend. + ASSERT_FALSE(b->head(l.mountKey("r")).exists); + const auto del = b->deleteExact(l.mountKey("r"), Token{"absent", TokenType::Emulated}); EXPECT_EQ(del.kind, DeleteOutcome::Kind::NotFound); EXPECT_GT(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), e2); } @@ -984,6 +987,21 @@ class AlwaysVanishesBackend final : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + /// The transport primitives forward to `inner`; the legacy overrides above are what this + /// double injects through. Declared because `Backend` declares them pure. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } + private: std::shared_ptr inner; }; diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index 0a19b2418a00..e73a37fb4030 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -110,6 +110,8 @@ class CatalogChangingOnSecondReadBackend : public InMemoryBackend class ReplacingManifestAfterObservationBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; using Backend::get; void arm(const Layout & layout, String manifest_key_) diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index a92d8f9bc4d1..4ccd5cb0ba96 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -175,6 +175,21 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + /// The transport primitives forward to `inner`; the legacy overrides above are what this + /// double injects through. Declared because `Backend` declares them pure. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } + private: BackendPtr inner; String target_key; @@ -207,6 +222,29 @@ class KeyCountingBackend final : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + /// The primitives count too: `Backend::probeSentinelRaw` reaches the store through them, so a + /// per-key observation on the primitive path would otherwise go uncounted. + std::optional read(const String & key, TransportAccess & access) override + { + ++get_counts[key]; + return inner->read(key, access); + } + std::optional head(const String & key, TransportAccess & access) override + { + ++head_counts[key]; + return inner->head(key, access); + } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } + private: BackendPtr inner; std::map head_counts; @@ -217,6 +255,8 @@ class KeyCountingBackend final : public DB::Cas::Backend class RacingBlobPublicationBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; void watch(String key_) { std::lock_guard lock(mutex); @@ -849,6 +889,26 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) DB::Cas::CasResult casPut(const String & k, const String & bts, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, bts, e, m); } DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & tok) override { return inner->deleteExact(k, tok); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + + /// `read` is a GET, so it counts on the watched key too -- otherwise the INV-1 fence below + /// would stop seeing a revival read the moment its caller takes the primitive path. + std::optional read(const String & key, TransportAccess & access) override + { + if (key == watched_key) + ++get_count; + return inner->read(key, access); + } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } private: BackendPtr inner; String watched_key; @@ -927,6 +987,26 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupPresentNeverGetsTheDyingObject) DB::Cas::CasResult casPut(const String & k, const String & bts, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, bts, e, m); } DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & tok) override { return inner->deleteExact(k, tok); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + + /// `read` is a GET, so it counts on the watched key too -- otherwise the INV-1 fence below + /// would stop seeing a revival read the moment its caller takes the primitive path. + std::optional read(const String & key, TransportAccess & access) override + { + if (key == watched_key) + ++get_count; + return inner->read(key, access); + } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } private: BackendPtr inner; String watched_key; @@ -1327,9 +1407,10 @@ TEST(CASPartWriteTxn, PublishHappyPathRoundTrip) const auto * entry = findEntry(manifest.entries, "data.bin"); ASSERT_TRUE(entry != nullptr); const auto loc = s->locate(*entry); - auto got = b->get(loc.key, Range{loc.offset, loc.length}); + /// The located window of the blob object, sliced by the test: the seam reads whole objects. + auto got = b->get(loc.key); ASSERT_TRUE(got.has_value()); - EXPECT_EQ(got->bytes, "hello world"); + EXPECT_EQ(got->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), "hello world"); } TEST(CASPartWriteTxn, PromoteCrossNamespaceManifestFailsClosed) @@ -1495,6 +1576,34 @@ TEST(CASPartWriteTxn, AdoptEvidenceRecordsTrustedManifestDependencyProofWithoutI CasResult casPut(const String & k, const String & bts, const std::optional & e, const ObjectMeta & m) override { return inner->casPut(k, bts, e, m); } DeleteOutcome deleteExact(const String & k, const Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + + /// The primitives count on the same three counters, so "no backend op" stays a total claim + /// whichever path a caller takes. + std::optional read(const String & key, TransportAccess & access) override + { + ++gets; + return inner->read(key, access); + } + std::optional head(const String & key, TransportAccess & access) override + { + ++heads; + return inner->head(key, access); + } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + ++puts; + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override + { + ++puts; + inner->publish(request, access); + } + Dialect dialect() const override { return inner->dialect(); } private: BackendPtr inner; }; @@ -1670,9 +1779,9 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) const auto * entry = findEntry(manifest.entries, "f"); ASSERT_TRUE(entry != nullptr); const auto loc = s->locate(*entry); - const auto got = b->get(loc.key, Range{loc.offset, loc.length}); + const auto got = b->get(loc.key); ASSERT_TRUE(got.has_value()); - EXPECT_EQ(got->bytes, content); + EXPECT_EQ(got->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), content); } /// BUG 1 (WPromote owner==bld): promote is a PURE owner MOVE (Δ=0 — it restores no blob in-degree). The @@ -2355,6 +2464,8 @@ namespace class BlobPutFaultBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; int fault_count = 0; /// remaining ambiguous faults on matching create attempts bool land_despite_fault = false; /// the faulted attempt's own write actually lands (response lost) int publish_stream_attempts = 0; /// unconditional streaming publications observed diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 7235765a04a7..c529e2f5126b 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -79,6 +79,30 @@ class WriteCountingBackend final : public DB::Cas::Backend DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & meta) override { ++writes; return inner->casPut(k, b, e, meta); } DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { ++writes; return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + + /// The primitives count too: `Backend::probeSentinelRaw` reaches the store through them, so a + /// write that took the primitive path would otherwise go unseen by `writes`. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + ++writes; + return inner->remove(key, expected_value, access); + } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + ++writes; + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override + { + ++writes; + inner->publish(request, access); + } + Dialect dialect() const override { return inner->dialect(); } private: std::shared_ptr inner; }; @@ -197,6 +221,30 @@ class ProbeWatchingBackend final : public DB::Cas::Backend DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & m) override { note(k); return inner->casPut(k, b, e, m); } DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { note(k); return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + + /// The primitives note too: `Backend::probeSentinelRaw` reaches the store through them, so a + /// probe-key mutation on the primitive path would otherwise go unseen. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + note(key); + return inner->remove(key, expected_value, access); + } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + note(key); + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override + { + note(request.destination_key); + inner->publish(request, access); + } + Dialect dialect() const override { return inner->dialect(); } private: void note(const String & k) { if (k.find("/_probe/") != String::npos) probe_touched = true; } std::shared_ptr inner; @@ -266,6 +314,21 @@ class ForwardingBackend : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + /// The transport primitives forward to `inner`; the legacy overrides above are what this + /// double injects through. Declared because `Backend` declares them pure. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } + private: std::shared_ptr inner; }; @@ -690,9 +753,10 @@ TEST(CASPool, ResolveReturnsManifestId) EXPECT_EQ(loc.offset, s->poolMeta().blob_header_len); EXPECT_EQ(loc.length, payload.size()); - auto bytes = b->get(loc.key, Range{loc.offset, loc.length}); + auto bytes = b->get(loc.key); ASSERT_TRUE(bytes.has_value()); - EXPECT_EQ(bytes->bytes, payload); /// ranged read, no header touch + /// The located window holds exactly the payload: the envelope header is outside it. + EXPECT_EQ(bytes->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), payload); const auto * small = findEntry(manifest.entries, "small.txt"); ASSERT_TRUE(small != nullptr); @@ -1371,6 +1435,21 @@ class FenceInAdoptWindowBackend final : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } + /// The transport primitives forward to `inner`; the legacy overrides above are what this + /// double injects through. Declared because `Backend` declares them pure. + std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + return inner->write(key, bytes, expected_value, access); + } + std::unique_ptr stream(const String & key, TransportAccess & access) override { return inner->stream(key, access); } + void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } + Dialect dialect() const override { return inner->dialect(); } + private: std::shared_ptr inner; }; @@ -1871,21 +1950,22 @@ TEST(CASPoolRemount, ThrowingForeignConflictSinkCannotReplaceTerminalOutcome) class RemountStepBackend final : public DB::Cas::tests::CountingBackend { public: - using DB::Cas::tests::CountingBackend::get; - void failNextGet(String key) { failed_key = std::move(key); } - std::optional get(const String & key, Range range) override + /// The fault sits on the PRIMITIVE, not on legacy `get`: the lifecycle gate reads `_pool_meta` + /// through `probeSentinelRaw`, which speaks the primitives. A legacy caller reaches this anyway, + /// through the forwarder, so arming it here covers both surfaces rather than only one. + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (!failed_key.empty() && key == failed_key) { failed_key.clear(); throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected remount probe failure"); } - return DB::Cas::tests::CountingBackend::get(key, range); + return DB::Cas::tests::CountingBackend::read(key, access); } private: diff --git a/src/Disks/tests/gtest_cas_probe.cpp b/src/Disks/tests/gtest_cas_probe.cpp index 8c2beb7d4503..1aa2467611f1 100644 --- a/src/Disks/tests/gtest_cas_probe.cpp +++ b/src/Disks/tests/gtest_cas_probe.cpp @@ -5,6 +5,8 @@ #include #include #include +#include +#include #include namespace DB @@ -162,8 +164,14 @@ TEST(CASProbe, MissingSingleAttemptClientFailsCapabilityProbe) EXPECT_TRUE(b->list(probe_prefix, "", 10).keys.empty()); /// The same LIST can see a key that IS under the prefix — otherwise the emptiness above would be - /// indistinguishable from a prefix this backend can never enumerate. - ASSERT_EQ(b->putIfAbsent(probe_prefix + "/token", "probe-v1").outcome, PutOutcome::Done); + /// indistinguishable from a prefix this backend can never enumerate. Placed through the object + /// storage: a Native write over a local storage has no response incarnation to attribute itself + /// to, and Native passes the key verbatim, so this lands exactly where the LIST looks. + { + auto out = storage->writeObject(DB::StoredObject(probe_prefix + "/token"), DB::WriteMode::Rewrite); + DB::writeString(String("probe-v1"), *out); + out->finalize(); + } EXPECT_FALSE(b->list(probe_prefix, "", 10).keys.empty()); } @@ -315,6 +323,29 @@ class DialectGatedCountingBackend final : public Backend return p; } + /// The transport primitives forward verbatim. Nothing in this suite calls them -- the capability + /// probe speaks the legacy Token-typed surface this double gates -- so they exist to make the + /// class concrete; the dialect gate above is what the tests exercise. + std::optional read(const String & key, TransportAccess & a) override { return inner.read(key, a); } + std::optional head(const String & key, TransportAccess & a) override { return inner.head(key, a); } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & a) override + { + return inner.list(prefix, cursor, limit, a); + } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & a) override + { + return inner.remove(key, expected_value, a); + } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & a) override + { + return inner.write(key, bytes, expected_value, a); + } + std::unique_ptr stream(const String & key, TransportAccess & a) override { return inner.stream(key, a); } + void publish(const BlobPublishRequest & request, TransportAccess & a) override { inner.publish(request, a); } + /// The dialect its legacy surface stamps every token with. + Dialect dialect() const override { return TokenType::ETag; } + /// Number of times putOverwrite/casPut(with expected)/deleteExact actually delegated to `inner` /// (i.e. reached the real enforcement) rather than being short-circuited by the dialect gate. int overwrite_reached = 0; diff --git a/src/Disks/tests/gtest_cas_protocol_scenarios.cpp b/src/Disks/tests/gtest_cas_protocol_scenarios.cpp index d70cca40a8ea..924732bfc65d 100644 --- a/src/Disks/tests/gtest_cas_protocol_scenarios.cpp +++ b/src/Disks/tests/gtest_cas_protocol_scenarios.cpp @@ -120,9 +120,9 @@ void assertPartReads( const auto * entry = findEntry(manifest.entries, path); ASSERT_TRUE(entry != nullptr); auto loc = s->locate(*entry); - auto got = b->get(loc.key, Range{loc.offset, loc.length}); + auto got = b->get(loc.key); ASSERT_TRUE(got.has_value()); - EXPECT_EQ(got->bytes, payload); + EXPECT_EQ(got->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), payload); } } diff --git a/src/Disks/tests/gtest_cas_recovery_streaming.cpp b/src/Disks/tests/gtest_cas_recovery_streaming.cpp index ad2dbbcb9342..ff83de9240e7 100644 --- a/src/Disks/tests/gtest_cas_recovery_streaming.cpp +++ b/src/Disks/tests/gtest_cas_recovery_streaming.cpp @@ -143,6 +143,8 @@ bool pollUntil(Pred pred) class VanishMidTailOnceBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; using InMemoryBackend::get; /// keep the one-arg convenience overload visible past our override String target_log_key; @@ -172,6 +174,8 @@ class VanishMidTailOnceBackend : public InMemoryBackend class CorruptLogOnGetBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; using InMemoryBackend::get; /// keep the one-arg convenience overload visible past our override String target_log_key; @@ -202,6 +206,8 @@ class CorruptLogOnGetBackend : public InMemoryBackend class BlockingFirstLogGetBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; using InMemoryBackend::get; String refs_prefix; diff --git a/src/Disks/tests/gtest_cas_request_control.cpp b/src/Disks/tests/gtest_cas_request_control.cpp index ad7801634f39..69b6da4c2c96 100644 --- a/src/Disks/tests/gtest_cas_request_control.cpp +++ b/src/Disks/tests/gtest_cas_request_control.cpp @@ -21,6 +21,7 @@ namespace DB::ErrorCodes { extern const int NETWORK_ERROR; extern const int ABORTED; + extern const int CAS_WRITE_UNATTRIBUTED; } namespace ProfileEvents @@ -192,16 +193,19 @@ TEST(CASRequestControl, CountersHookupIncrementsPerClass) /// Wiring smoke test: a real conditional write through ObjectStorageBackend (Native mode) counts one /// attempt and one Committed outcome via the SAME instrumented call site nativeConditionalPut uses — -/// see finalizeConditionalWriteInstrumented in CasObjectStorageBackend.cpp. +/// see finalizeConditionalWriteInstrumented in CasObjectStorageBackend.cpp. The write itself lands +/// and is counted at that site; a local object storage then returns no incarnation for it, so the +/// call refuses to attribute the write rather than reading one back. The counters are what this pins. TEST(CASRequestControl, NativeConditionalPutCountsOneAttemptAndCommitted) { using ProfileEvents::global_counters; const auto attempts_before = global_counters[ProfileEvents::CASConditionalWriteAttempts].load(); const auto committed_before = global_counters[ProfileEvents::CASConditionalWriteCommitted].load(); - auto b = std::make_shared( - DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - EXPECT_EQ(b->putIfAbsent("p/rc/one", "v1").outcome, PutOutcome::Done); + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); + const String key = DB::Cas::tests::nativeKeyUnder(storage, "p/rc/one"); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, [&] { b->putIfAbsent(key, "v1"); }); #if !WITH_COVERAGE EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteAttempts].load() - attempts_before, 1u); diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp new file mode 100644 index 000000000000..2d583f6a144a --- /dev/null +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -0,0 +1,1481 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "cas_test_helpers.h" + +#include "config.h" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB::ErrorCodes +{ +extern const int ABORTED; +extern const int CAS_DELETE_MARKER; +extern const int CORRUPTED_DATA; +extern const int LOGICAL_ERROR; +extern const int S3_ERROR; +extern const int NETWORK_ERROR; +} + +using namespace DB::Cas; + +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::FakeClock; +using DB::Cas::tests::expectThrowsCode; + +namespace +{ + +/// Every engine test drives `CasRequests` on an injected clock, so a ninety-second policy is exercised +/// in no wall-clock time and the retry schedule itself becomes an assertion. +CasRequests makeRequests(BackendPtr backend, FakeClock & clock, Fence fence = Fence::open()) +{ + return CasRequests(std::move(backend), std::move(fence), clock.nowFn(), clock.sleepFn()); +} + +} + +static_assert(!std::is_default_constructible_v); +static_assert(!std::is_constructible_v); +static_assert(!std::is_constructible_v); +static_assert(!std::is_default_constructible_v); +static_assert(!std::is_copy_constructible_v); + +TEST(CASIncarnation, GrammarRefusesTheNineWays) +{ + EXPECT_FALSE(isIncarnationValue(Dialect::ETag, "")); + EXPECT_FALSE(isIncarnationValue(Dialect::ETag, "*")); + EXPECT_FALSE(isIncarnationValue(Dialect::ETag, " * ")); + EXPECT_FALSE(isIncarnationValue(Dialect::ETag, "\"a\",\"b\"")); + EXPECT_TRUE(isIncarnationValue(Dialect::ETag, "\"abc\"")); + EXPECT_FALSE(isIncarnationValue(Dialect::Generation, "0")); + EXPECT_FALSE(isIncarnationValue(Dialect::Generation, "00123")); + EXPECT_FALSE(isIncarnationValue(Dialect::Generation, "\"123\"")); + EXPECT_FALSE(isIncarnationValue(Dialect::Generation, "123 ")); /// the ninth: decimal is not "decimal, trimmed" + EXPECT_TRUE(isIncarnationValue(Dialect::Generation, "123")); + EXPECT_FALSE(isIncarnationValue(Dialect::Emulated, "")); +} + +TEST(CASRetry, BackoffIsFullJitterUnderTheCap) +{ + for (uint32_t attempt = 1; attempt <= 12; ++attempt) + { + const uint64_t ceiling = std::min(5000, 200ull << (attempt - 1)); + uint64_t sum = 0; + std::set seen; + bool low = false; + bool high = false; + for (int i = 0; i < 1000; ++i) + { + const uint64_t s = Retry::backoff(attempt); + ASSERT_LE(s, ceiling); + sum += s; + seen.insert(s); + low = low || s < ceiling / 4; + high = high || s > ceiling * 3 / 4; + } + const double mean = static_cast(sum) / 1000.0; + EXPECT_GT(mean, static_cast(ceiling) * 0.35) << "attempt " << attempt; + EXPECT_LT(mean, static_cast(ceiling) * 0.65) << "attempt " << attempt; + /// The mean alone cannot tell full jitter from a constant half the ceiling, so the SPREAD is + /// asserted too: many distinct values, reaching into both the bottom and the top quarter. + EXPECT_GE(seen.size(), 3u) << "attempt " << attempt; + EXPECT_TRUE(low) << "attempt " << attempt; + EXPECT_TRUE(high) << "attempt " << attempt; + } +} + +TEST(CASRetry, PoliciesAreShapedAsSpecified) +{ + const uint64_t now = 1'000'000; + EXPECT_EQ(Retry::standard().bind(now).deadline_ms, now + 90'000); + EXPECT_FALSE(Retry::standard().bind(now).lease_bound); + EXPECT_FALSE(Retry::standard().single_attempt); + EXPECT_TRUE(Retry::once().single_attempt); + const Retry::Bound lease = Retry::untilLeaseSafe(now + 10'000, 2'000).bind(now); + EXPECT_EQ(lease.deadline_ms, now + 8'000); + EXPECT_TRUE(lease.lease_bound); + EXPECT_EQ(Retry::within(1'000).bind(now).deadline_ms, now + 1'000); +} + +TEST(CASWriteResult, OrThrowMapsEveryAlternative) +{ + /// The two that are not failures: a commit hands back its incarnation, a decline hands back + /// nothing, and neither throws. Minting one needs a real write, since nothing else may mint. + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + WriteResult committed = op.create("k", "v", Retry::standard()); + ASSERT_TRUE(std::holds_alternative(committed)); + const Incarnation landed = std::get(committed).incarnation; + const auto returned = orThrow(std::move(committed), "create"); + ASSERT_TRUE(returned.has_value()); + EXPECT_EQ(*returned, landed); + EXPECT_FALSE(orThrow(WriteResult{Declined{ProvenAbsent{}}}, "declined").has_value()); + + expectThrowsCode(DB::ErrorCodes::ABORTED, [&] { orThrow(WriteResult{Conflict{ProvenAbsent{}}}, "t"); }); + expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { orThrow(WriteResult{Refused{DB::ErrorCodes::S3_ERROR, "denied"}}, "t"); }); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{GaveUp{GaveUp::Why::Deadline, GaveUp::Source::Policy, true, NotObserved{}}}, "t"); }); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{GaveUp{GaveUp::Why::Unresolved, GaveUp::Source::Policy, true, ProvenAbsent{}}}, "t"); }); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{GaveUp{GaveUp::Why::FenceLost, GaveUp::Source::Lease, false, NotObserved{}}}, "t"); }); +} + +TEST(CASFence, OpenFenceAdmitsEverythingAndNeverMoves) +{ + Fence f = Fence::open(); + EXPECT_EQ(f.generation(), 0u); + EXPECT_EQ(f.admit(0, 1'000'000), Fence::Admit::Ok); + EXPECT_NO_THROW(f.check_or_throw(0)); +} + +/// ================================================================================================ +/// The backend's keyed string primitives +/// ================================================================================================ + +/// The door the primitives are reachable through until `CasRequests` lands: `Backend` is a friend of +/// `TransportAccess`, so a `Backend` subclass can hand out the migration key. Both go at the lock. +struct RawDoor : DB::Cas::Backend +{ + static DB::Cas::TransportAccess key() { return migrationAccess(); } +}; + +TEST(CASBackendPrimitives, InMemoryWriteReadRemoveRoundTripInStrings) +{ + auto b = std::make_shared(); + auto key = RawDoor::key(); + auto w1 = b->write("k", "v1", std::nullopt, key); + ASSERT_TRUE(w1.has_value()); + auto r = b->read("k", key); + ASSERT_TRUE(r); + EXPECT_EQ(r->bytes, "v1"); + EXPECT_EQ(r->value, *w1); + + auto h = b->head("k", key); + ASSERT_TRUE(h); + EXPECT_EQ(h->size, 2u); + EXPECT_EQ(h->value, *w1); + + auto w2 = b->write("k", "v2", std::nullopt, key); /// must be absent → refused + EXPECT_FALSE(w2.has_value()); + auto w3 = b->write("k", "v2", *w1, key); + ASSERT_TRUE(w3.has_value()); + EXPECT_NE(*w3, *w1); /// values never repeat + + EXPECT_EQ(b->remove("k", *w1, key), Backend::RawRemoval::Mismatch); + EXPECT_EQ(b->remove("k", *w3, key), Backend::RawRemoval::Removed); + EXPECT_EQ(b->remove("k", *w3, key), Backend::RawRemoval::Gone); + EXPECT_FALSE(b->read("k", key).has_value()); +} + +TEST(CASBackendPrimitives, ListSurfacesTheIncarnationValueAndPaginates) +{ + auto b = std::make_shared(); + auto key = RawDoor::key(); + const String a = *b->write("p/a", "0123456789", std::nullopt, key); + b->write("p/b", "xy", std::nullopt, key); + b->write("q/c", "z", std::nullopt, key); + + const auto page = b->list("p/", "", 10, key); + ASSERT_EQ(page.keys.size(), 2u); /// sorted, prefix-scoped + EXPECT_EQ(page.keys[0].key, "p/a"); + EXPECT_EQ(page.keys[0].size, 10u); + ASSERT_TRUE(page.keys[0].value.has_value()); + EXPECT_EQ(*page.keys[0].value, a); + EXPECT_TRUE(page.next_cursor.empty()); + + const auto first = b->list("p/", "", 1, key); + ASSERT_EQ(first.keys.size(), 1u); + EXPECT_EQ(first.next_cursor, "p/a"); + const auto second = b->list("p/", first.next_cursor, 1, key); + ASSERT_EQ(second.keys.size(), 1u); + EXPECT_EQ(second.keys[0].key, "p/b"); +} + +TEST(CASBackendPrimitives, EveryBackendInstanceHasItsOwnId) +{ + auto a = std::make_shared(); + auto b = std::make_shared(); + EXPECT_NE(a->backendId(), b->backendId()); + EXPECT_NE(a->backendId(), 0u); + EXPECT_EQ(a->dialect(), Dialect::Emulated); +} + +namespace +{ + +/// Counts every call that reaches the primitive `write`, whichever surface it entered through. +struct WriteCountingBackend : InMemoryBackend +{ + size_t writes = 0; + + std::expected write(const String & k, const String & v, const std::optional & e, + TransportAccess & a) override + { + ++writes; + return InMemoryBackend::write(k, v, e, a); + } +}; + +} + +TEST(CASBackendPrimitives, ALegacyCallReachesAnOverrideOfThePrimitiveItForwardsTo) +{ + /// The migration rule: the new methods are the primitives, and a fault injection written against a + /// NEW signature intercepts a legacy caller too, because the legacy verb forwards through the + /// virtual. `putIfAbsent` and `casPut` are this backend's two documented exceptions -- they route + /// around the primitive to keep their knobs' verb identity -- so the rule is asserted on a verb + /// that does forward. + auto b = std::make_shared(); + auto door = RawDoor::key(); + const String first = *b->write("k", "v", std::nullopt, door); + b->writes = 0; + + b->putOverwrite("k", "w", Token{first, Dialect::Emulated}); /// legacy call + EXPECT_EQ(b->writes, 1u); + + /// And the negative half of the same ruling: the two exceptions really do route around the + /// primitive. Both writes LAND, so this cannot pass by the calls having done nothing. + b->writes = 0; + EXPECT_EQ(b->putIfAbsent("k2", "v").outcome, PutOutcome::Done); + EXPECT_EQ(b->casPut("k3", "v", std::nullopt).outcome, CasOutcome::Committed); + EXPECT_EQ(b->writes, 0u); +} + +TEST(CASBackendPrimitives, EachWriteKnobFiresOnlyForTheVerbItNames) +{ + /// A knob is armed against a VERB. The keyed `write` cannot see which verb its caller used, so + /// consuming both knobs there is right and consuming the other one from a legacy verb is not: a + /// test that arms an ambiguity for `putIfAbsent` must not have it fire on a `casPut`. + auto b = std::make_shared(); + auto door = RawDoor::key(); + + b->failNextCasPut("k"); + EXPECT_EQ(b->putIfAbsent("k", "v").outcome, PutOutcome::Done); /// not casPut's knob to consume + const Token present = b->head("k").token; + EXPECT_EQ(b->casPut("k", "w", present).outcome, CasOutcome::Conflict); + EXPECT_EQ(b->get("k")->bytes, "v"); /// the refusal changed nothing + + b->injectAmbiguousPutIfAbsent("k2"); + EXPECT_EQ(b->casPut("k2", "v", std::nullopt).outcome, CasOutcome::Committed); /// not casPut's knob either + EXPECT_THROW(b->putIfAbsent("k2", "w"), std::runtime_error); + + /// The keyed primitive is the one caller every knob is armed against, and each is still one-shot. + b->injectAmbiguousPutIfAbsent("k3"); + b->failNextCasPut("k3"); + EXPECT_THROW((void)b->write("k3", "v", std::nullopt, door), std::runtime_error); + EXPECT_FALSE(b->write("k3", "v", std::nullopt, door).has_value()); + EXPECT_TRUE(b->write("k3", "v", std::nullopt, door).has_value()); +} + +TEST(CASBackendPrimitives, LegacyGetRefusesAValueThatIsNotAnIncarnation) +{ + /// `read` hands back whatever the store said, malformed included -- settling that is the caller's. + /// The legacy forwarder has no caller to settle it: it would hand the value on as a `Token` that + /// the next conditional operation refuses as a caller bug, one layer too late to name the key. + struct EmptyValueBackend : InMemoryBackend + { + std::optional read(const String &, TransportAccess &) override { return Raw{"body", ""}; } + }; + auto b = std::make_shared(); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { b->get("k"); }); +} + +namespace +{ + +/// A double whose fault injection is written against the LEGACY surface, the way almost every one in +/// this suite is. A `Pool` hands its callers a decorator, so a legacy call reaches the decorator +/// first: if the decorator converted it to a primitive before forwarding, this override would never +/// run and the injection would be silently dead. +struct LegacyCasPutFlaggingBackend : DB::Cas::tests::CountingBackend +{ + bool legacy_cas_put_ran = false; + + CasResult casPut(const String & key, const String & bytes, const std::optional & expected, + const ObjectMeta & meta) override + { + legacy_cas_put_ran = true; + return CountingBackend::casPut(key, bytes, expected, meta); + } +}; + +} + +TEST(CASBackendPrimitives, InstrumentedBackendPassesALegacyCallThroughAsLegacy) +{ + auto inner = std::make_shared(); + InstrumentedBackend instrumented(inner); + EXPECT_EQ(instrumented.casPut("k", "v", std::nullopt).outcome, CasOutcome::Committed); + EXPECT_TRUE(inner->legacy_cas_put_ran); + EXPECT_EQ(inner->casPutCount("k"), 1u); +} + +TEST(CASBackendPrimitives, RefreshCredentialsIsOffUntilAskedFor) +{ + auto b = std::make_shared(); + EXPECT_FALSE(b->refreshCredentials()); + b->setRefreshCredentialsResult(true); + EXPECT_TRUE(b->refreshCredentials()); +} + +#if USE_AWS_S3 + +TEST(CASThrottlingBackend, FirstPerKeyRefusesOnceThenForwards) +{ + auto inner = std::make_shared(); + auto t = std::make_shared(inner, ThrottlingBackend::Mode::FirstPerKey, 0, 429); + auto key = RawDoor::key(); + EXPECT_THROW(t->read("k", key), DB::S3Exception); + EXPECT_NO_THROW(t->read("k", key)); + EXPECT_EQ(t->refusals("k"), 1u); + EXPECT_THROW(t->write("k2", "v", std::nullopt, key), DB::S3Exception); + EXPECT_TRUE(t->write("k2", "v", std::nullopt, key).has_value()); +} + +TEST(CASThrottlingBackend, RefusalsAreRetryableUnderBothStatuses) +{ + auto key = RawDoor::key(); + for (const int status : {429, 503}) + { + auto t = std::make_shared( + std::make_shared(), ThrottlingBackend::Mode::FirstPerKey, 0, status); + try + { + t->head("k", key); + FAIL() << "expected a refusal for status " << status; + } + catch (const DB::S3Exception & e) + { + /// The property the seam exists for: the engine must see an AMBIGUOUS attempt, which is + /// what a retryable store error means, not a definite failure. + EXPECT_TRUE(e.isRetryableError()) << "status " << status; + } + } +} + +TEST(CASThrottlingBackend, PassesALegacyCallThroughAsLegacy) +{ + auto inner = std::make_shared(); + /// A period no call here reaches, so nothing is refused: what this pins is the pass-through. + auto t = std::make_shared(inner, ThrottlingBackend::Mode::EveryNth, 1000, 503); + EXPECT_EQ(t->casPut("k", "v", std::nullopt).outcome, CasOutcome::Committed); + EXPECT_TRUE(inner->legacy_cas_put_ran); + EXPECT_EQ(inner->casPutCount("k"), 1u); +} + +TEST(CASThrottlingBackend, EveryNthRefusesOnThePeriodAcrossKeys) +{ + auto inner = std::make_shared(); + auto t = std::make_shared(inner, ThrottlingBackend::Mode::EveryNth, 3, 503); + auto key = RawDoor::key(); + EXPECT_NO_THROW(t->read("a", key)); + EXPECT_NO_THROW(t->read("b", key)); + EXPECT_THROW(t->read("c", key), DB::S3Exception); /// the third request, whatever it names + EXPECT_EQ(t->refusals("c"), 1u); + EXPECT_EQ(t->refusals("a"), 0u); + EXPECT_NO_THROW(t->read("c", key)); +} + +#endif + +/// ================================================================================================ +/// The request engine +/// ================================================================================================ + +namespace +{ + +/// A type nothing in the engine catches, so a `decide` that throws it can only reach the caller by +/// propagating unchanged. +struct DecideMarker +{ +}; + +/// Answers the FIRST remove with a mismatch without reaching the store, so `removeCurrent` has to +/// re-observe. Counts its own requests: an answer given here never reaches the counting base. +struct MismatchOnceOnRemoveBackend : InMemoryBackend +{ + using InMemoryBackend::head; + + size_t heads = 0; + size_t removes = 0; + bool refuse_next_remove = true; + + std::optional head(const String & key, TransportAccess & access) override + { + ++heads; + return InMemoryBackend::head(key, access); + } + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + ++removes; + if (std::exchange(refuse_next_remove, false)) + return RawRemoval::Mismatch; + return InMemoryBackend::remove(key, expected_value, access); + } +}; + +/// Answers `Indeterminate` for its first `indeterminate_answers` probes, then delegates -- a store +/// briefly out of reach, whose absence was never established. +struct IndeterminateProbeBackend : InMemoryBackend +{ + using Backend::probeSentinelRaw; + + size_t probes = 0; + size_t indeterminate_answers = 2; + + SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override + { + if (++probes <= indeterminate_answers) + return {ProbeOutcome::Indeterminate, std::nullopt}; + return InMemoryBackend::probeSentinelRaw(key, access); + } +}; + +/// Refuses the FIRST `list` naming each distinct cursor -- one refusal per page -- and charges every +/// list a fixed slice of the caller's clock, so what a page costs is a fact rather than a jitter draw. +/// `always_refuse_cursor` keeps one page refused for good. +struct PagedThrottleBackend : InMemoryBackend +{ + using InMemoryBackend::list; + + std::function charge_latency; + std::set refused_cursors; + std::optional always_refuse_cursor; + size_t list_calls = 0; + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + ++list_calls; + if (charge_latency) + charge_latency(); + if ((always_refuse_cursor && *always_refuse_cursor == cursor) || refused_cursors.insert(cursor).second) + throw Poco::TimeoutException("the list resuming after '" + cursor + "' timed out"); + return InMemoryBackend::list(prefix, cursor, limit, access); + } +}; + +/// Runs `on_read` after every read. The resolve read is where a caller's own facts can change +/// between an attempt and the pause that would precede the next one. +struct FlipOnReadBackend : CountingBackend +{ + std::function on_read; + + std::optional read(const String & key, TransportAccess & access) override + { + auto raw = CountingBackend::read(key, access); + if (on_read) + on_read(); + return raw; + } +}; + +} + +TEST(CASIncarnation, RenderAndPersistedCompare) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + const Incarnation first = *orThrow(op.create("k", "v", Retry::standard()), "create"); + EXPECT_EQ(first.render(), "emulated:1"); + EXPECT_EQ(first.key(), "k"); + EXPECT_EQ(first.dialect(), Dialect::Emulated); + + const PersistedIncarnation persisted = PersistedIncarnation::capture(first); + EXPECT_EQ(persisted.dialect, "emulated"); + EXPECT_EQ(persisted.value, "1"); + EXPECT_TRUE(persisted.matches(first)); + + const Incarnation second = *orThrow(op.replace("k", "w", first, Retry::standard()), "replace"); + EXPECT_EQ(second.render(), "emulated:2"); + EXPECT_FALSE(persisted.matches(second)); /// a captured record never re-matches a later incarnation + EXPECT_TRUE(PersistedIncarnation::capture(second).matches(second)); +} + +TEST(CASRetry, BindSaturatesAndLeavesAnEqualLeaseOffTheLeaseSource) +{ + constexpr uint64_t largest = std::numeric_limits::max(); + /// A window one short of the whole range, so any `now` above 1 overflows a naive addition. + EXPECT_EQ(Retry::within(largest - 1).bind(2).deadline_ms, largest); + EXPECT_EQ(Retry::within(largest - 1).bind(1).deadline_ms, largest); + EXPECT_FALSE(Retry::within(largest - 1).bind(2).lease_bound); + + const uint64_t now = 1'000'000; + /// The lease bound lands exactly on the policy deadline. The lease is taken only when it is + /// STRICTLY smaller, so the tie belongs to the policy and `GaveUp` will not name the lease. + const Retry::Bound tie = Retry::untilLeaseSafe(now + 92'000, 2'000).bind(now); + EXPECT_EQ(tie.deadline_ms, now + 90'000); + EXPECT_FALSE(tie.lease_bound); + + const Retry::Bound lease = Retry::untilLeaseSafe(now + 91'999, 2'000).bind(now); + EXPECT_EQ(lease.deadline_ms, now + 89'999); + EXPECT_TRUE(lease.lease_bound); +} + +TEST(CASRequests, CreateThenReplaceThenRemove) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + const Incarnation first = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + const auto seen = op.read("k", Retry::standard()); + ASSERT_TRUE(seen.has_value()); + EXPECT_EQ(seen->bytes, "v1"); + EXPECT_EQ(seen->incarnation, first); + + const Incarnation second = *orThrow(op.replace("k", "v2", first, Retry::standard()), "replace"); + EXPECT_NE(second, first); + + EXPECT_EQ(op.remove("k", first, Retry::standard()), Removal::Mismatch); /// the incarnation is stale + EXPECT_EQ(op.remove("k", second, Retry::standard()), Removal::Removed); + EXPECT_EQ(op.remove("k", second, Retry::standard()), Removal::Gone); + EXPECT_FALSE(op.read("k", Retry::standard()).has_value()); +} + +/// An incarnation observed for one key is refused as the precondition for another, before the write +/// loop starts anything. Constructing a `LOGICAL_ERROR` exception ABORTS under a debug or sanitizer +/// build, so the same contract is asserted there as a death expectation; both forms pin that the +/// refusal happens, and the non-death form additionally pins that it costs no request. +#ifndef DEBUG_OR_SANITIZER_BUILD +TEST(CASRequests, KeyBindingThrowsBeforeAnyRequest) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation of_a = *orThrow(op.create("a", "v", Retry::standard()), "create"); + backend->resetCounts(); + + expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { (void)op.replace("b", "w", of_a, Retry::standard()); }); + EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_TRUE(clock.sleeps.empty()); +} +#else +TEST(CASRequestsDeathTest, KeyBindingThrowsBeforeAnyRequest) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation of_a = *orThrow(op.create("a", "v", Retry::standard()), "create"); + + EXPECT_DEATH({ (void)op.replace("b", "w", of_a, Retry::standard()); }, ""); +} +#endif + +TEST(CASRequests, EveryConflictIsSettledByOneReadAndCarriesTheOccupant) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "theirs", Retry::standard()), "create"); + backend->resetCounts(); + + WriteResult result = op.create("k", "mine", Retry::once()); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + const auto * occupant = std::get_if(&conflict->seen); + ASSERT_NE(occupant, nullptr); + EXPECT_EQ(occupant->bytes, "theirs"); + + /// The refused precondition says only that the key is taken; ONE exact read says by whom. + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 1u); +} + +TEST(CASRequests, AmbiguousCreateThatLandedIsCommittedByTheResolveRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + /// The object becomes durable and THEN the response is lost, so the store holds bytes the caller + /// never learned it wrote -- the only ambiguity a resolve read can settle as a commit. + backend->injectAmbiguousLandedWrite("k"); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_TRUE(committed->resolved_by_read); + EXPECT_EQ(committed->attempts_sent, 1u); + /// Settled by reading, never by writing again: a second create would have conflicted with the + /// first one's own object. + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, AmbiguousCreateThatNeverLandedIsReissued) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->injectAmbiguousPutIfAbsent("k"); /// the attempt's outcome is lost and the store is untouched + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_EQ(backend->readRequests(), 1u); /// the resolve proved absence, and only then did a reissue follow + EXPECT_EQ(clock.sleeps.size(), 1u); +} + +TEST(CASRequests, OnceSendsOneWriteAndAtMostOneResolveRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("the read timed out"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); + /// One attempt is one attempt, but the read that would have settled it is still owed and sent. + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, DecideMayThrowAndTheExceptionPropagatesUnchanged) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + EXPECT_THROW( + op.readModifyWrite("k", [](const std::optional &) -> std::optional { throw DecideMarker{}; }, + Retry::standard()), + DecideMarker); + EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_EQ(backend->readRequests(), 1u); /// the key was read, and nothing was decided about it +} + +TEST(CASRequests, OnPresenceIssuesHeadsAndNoGet) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.readModifyWriteOnPresence("k", + [](const std::optional & current) -> std::optional + { + return current ? std::nullopt : std::optional("v"); + }, + Retry::standard()); + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_GE(backend->headRequests(), 1u); +} + +TEST(CASRequests, OnPresenceSettlesARefusedPreconditionWithAHead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextCasPut("k"); /// the store refuses the precondition, writing nothing + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.readModifyWriteOnPresence("k", + [](const std::optional & current) -> std::optional + { + return current ? std::nullopt : std::optional("v"); + }, + Retry::standard()); + ASSERT_TRUE(std::holds_alternative(result)); + /// A refused precondition needs only to know WHAT is at the key, so this loop never fetches a body. + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_GE(backend->headRequests(), 2u); + EXPECT_EQ(backend->writeRequests(), 2u); +} + +TEST(CASRequests, ForEachListedKeyStopsEarlyAndBudgetsPerPage) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + for (int i = 0; i < 25; ++i) + orThrow(op.create("p/" + std::to_string(i), "v", Retry::standard()), "create"); + backend->resetCounts(); + + size_t seen = 0; + size_t pages = 0; + op.forEachListedKey("p/", [&](const KeyEntry &) { return ++seen < 3; }, Retry::standard(), + /*page_limit=*/10, [&] { ++pages; }); + EXPECT_EQ(seen, 3u); + /// The walk stops where the caller stops it: the remaining two pages are never fetched. + EXPECT_EQ(pages, 1u); + EXPECT_EQ(backend->listRequests(), 1u); +} + +TEST(CASRequests, DeleteMarkerIsANamedException) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation inc = *orThrow(op.create("k", "v", Retry::standard()), "create"); + + backend->setSimulateDeleteMarkers(true); + expectThrowsCode(DB::ErrorCodes::CAS_DELETE_MARKER, [&] { (void)op.remove("k", inc, Retry::standard()); }); + EXPECT_TRUE(clock.sleeps.empty()); /// a versioned bucket answers this way every time +} + +TEST(CASRequests, RemoveCurrentReObservesAMismatchAndRefusesUnderOnce) +{ + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "create"); + + EXPECT_EQ(op.removeCurrent("k", Retry::standard()), Removal::Removed); + /// Another incarnation became current between the observation and the delete: observe again, + /// paced like every other reissue, and delete what the second look saw. + EXPECT_EQ(backend->heads, 2u); + EXPECT_EQ(backend->removes, 2u); + EXPECT_EQ(clock.sleeps.size(), 1u); + } + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "create"); + + /// `once` has no reissue with which to settle a mismatch, and this verb never hands one back. + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.removeCurrent("k", Retry::once()); }); + EXPECT_EQ(backend->heads, 1u); + EXPECT_EQ(backend->removes, 1u); + EXPECT_TRUE(clock.sleeps.empty()); + } +} + +TEST(CASRequests, ProbeSentinelRetriesOnlyTheIndeterminateOutcome) +{ + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "create"); + + const SentinelProbeResult result = op.probeSentinel("k", Retry::standard()); + EXPECT_EQ(result.outcome, ProbeOutcome::Present); + ASSERT_TRUE(result.body.has_value()); + EXPECT_EQ(*result.body, "v"); + EXPECT_EQ(backend->probes, 3u); /// inconclusive twice, then an authoritative answer + EXPECT_EQ(clock.sleeps.size(), 2u); + } + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "create"); + + /// With no reissue left, the inconclusive outcome IS the answer: reported, never thrown. + const SentinelProbeResult result = op.probeSentinel("k", Retry::once()); + EXPECT_EQ(result.outcome, ProbeOutcome::Indeterminate); + EXPECT_EQ(backend->probes, 1u); + EXPECT_TRUE(clock.sleeps.empty()); + } +} + +TEST(CASRequests, AdmissionIsCheckedAtThreePoints) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto door = RawDoor::key(); + uint64_t generation = 1; + bool lost = false; + Fence fence{ + [&] { return generation; }, + [&](uint64_t admitted, uint64_t) + { + return (lost || admitted != generation) ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; + }, + [&](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + + /// (1) before the first attempt, on a handle resumed under a generation the fence has moved past + { + auto op = requests.resume(0); + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_FALSE(backend->read("k", door).has_value()); + } + /// (2) before the next verb of an admitted handle, after a re-arm between two verbs + { + auto op = requests.admit(); + EXPECT_FALSE(op.head("k", Retry::standard()).has_value()); + generation = 2; + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_FALSE(backend->read("k", door).has_value()); + } + /// (3) after a proven commit: the write landed, then the fence tripped before the call returned + { + auto op = requests.admit(); + backend->onWriteCommitted("k2", [&] { lost = true; }); + WriteResult result = op.create("k2", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any); + /// The object IS durable. This call refuses to CLAIM it; it does not undo it. + EXPECT_TRUE(backend->read("k2", door).has_value()); + } +} + +TEST(CASRequests, TheGateBeforeTheSleepEndsTheCallWithoutASecondWrite) +{ + FakeClock clock; + auto backend = std::make_shared(); + bool alive = true; + backend->on_read = [&] { alive = false; }; + backend->injectAmbiguousPutIfAbsent("k"); + auto requests = makeRequests(backend, clock); + auto op = requests.admit([&] { return alive; }); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any); + /// The ambiguous attempt was resolved, and the pause before the reissue was refused rather than + /// served: no sleep, and no second attempt after it. + EXPECT_TRUE(clock.sleeps.empty()); + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 1u); +} + +TEST(CASRequests, AResolveReadRefusedForLeaseBudgetIsReportedAsTheLeaseDeadline) +{ + FakeClock clock; + auto backend = std::make_shared(); + bool lease_spent = false; + Fence fence{ + [] { return uint64_t{0}; }, + [&](uint64_t, uint64_t) { return lease_spent ? Fence::Admit::NoBudget : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + auto op = requests.admit(); + const Incarnation seen = *orThrow(op.create("k", "v", Retry::standard()), "create"); + + /// The store refuses the precondition, and the lease budget is gone by the time the read that + /// would say WHO holds the key is due. The call learned nothing about the key, so what it reports + /// is the bound that stopped it -- not a conflict it never observed. + backend->failNextCasPut("k"); + backend->onBeforeWrite("k", [&] { lease_spent = true; }); + const uint64_t lease_deadline = clock.now + 10'000; + WriteResult result = op.replace("k", "w", seen, Retry::untilLeaseSafe(lease_deadline, 2'000)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Lease); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); + EXPECT_TRUE(clock.sleeps.empty()); + EXPECT_EQ(backend->writeRequests(), 2u); /// the create and the one refused replace + EXPECT_EQ(backend->readRequests(), 0u); /// the resolve read never started +} + +TEST(CASRequests, AFenceWithNoBudgetForTheRequestSendsNothingAndNamesTheLease) +{ + FakeClock clock; + auto backend = std::make_shared(); + const uint64_t budget_ms = 500; + Fence fence{ + [] { return uint64_t{0}; }, + [&](uint64_t, uint64_t needed_ms) { return needed_ms > budget_ms ? Fence::Admit::NoBudget : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + /// One attempt reserves more than the lease has left, so nothing may be started under it. + requests.setAttemptReservationForTest(1'000); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + /// The policy's own window is untouched; what ran out is the fence's budget, which IS the lease. + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Lease); + EXPECT_FALSE(gave_up->sent_any); + + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, AnRmwWhoseFirstReadFailsGivesUpUnresolvedWithoutWriting) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("the read timed out"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.readModifyWrite("k", + [](const std::optional &) -> std::optional { return String("v"); }, Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + /// No BOUND refused this read; the read itself failed. Claiming a deadline the clock never reached + /// would send its reader to widen the wrong thing. + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); + EXPECT_EQ(backend->writeRequests(), 0u); +} + +TEST(CASRequests, AnOnPresenceRmwWhoseFirstHeadFailsGivesUpUnresolvedWithoutWriting) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextHeadWith("k", std::make_exception_ptr(Poco::TimeoutException("the head timed out"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.readModifyWriteOnPresence("k", + [](const std::optional &) -> std::optional { return String("v"); }, Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_EQ(backend->readRequests(), 0u); /// the presence loop does not fall back to a body read +} + +TEST(CASRequests, AConflictWhoseResolveReadFailsIsReportedWithNothingObserved) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "theirs", Retry::standard()), "create"); + + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("the read timed out"))); + WriteResult result = op.create("k", "mine", Retry::once()); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + /// The precondition was refused, so the key IS taken; the read that would have said by whom failed, + /// and the caller is told exactly that rather than handed a guess about the occupant. + EXPECT_TRUE(std::holds_alternative(conflict->seen)); +} + +TEST(CASRequests, AFenceLostDuringTheResolveReadIsAFenceLossNotAConflict) +{ + FakeClock clock; + auto backend = std::make_shared(); + bool alive = true; + /// The fence trips while the ambiguous attempt is in flight: the write's own hook runs before the + /// store is touched, so the resolve read is the first request to meet the closed gate. + backend->onBeforeWrite("k", [&] { alive = false; }); + backend->injectAmbiguousPutIfAbsent("k"); + auto requests = makeRequests(backend, clock); + auto op = requests.admit([&] { return alive; }); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + /// A lost fence is not an observation. Reporting it as an ordinary conflict would tell the caller + /// somebody else holds the key, when what happened is that this node stopped being allowed to ask. + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 0u); /// refused before the resolve read was issued + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, OnPresenceFetchesTheBodyToProveAnAmbiguousAttemptLanded) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->injectAmbiguousLandedWrite("k"); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.readModifyWriteOnPresence("k", + [](const std::optional & current) -> std::optional + { + return current ? std::nullopt : std::optional("v"); + }, + Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_TRUE(committed->resolved_by_read); + /// Presence-only is what this loop REPORTS, not a promise about what it may read: only the bytes + /// can prove the ambiguous attempt was this call's own. + EXPECT_EQ(backend->readRequests(), 1u); +} + +TEST(CASRequests, OnPresenceReportsMetaEvenWhenItHadToFetchTheBody) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "theirs", Retry::standard()), "create"); + + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + WriteResult result = op.readModifyWriteOnPresence("k", + [](const std::optional &) -> std::optional { return String("mine"); }, Retry::once()); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + /// The ambiguity forced a body read, and the body stops at this boundary: a caller of the + /// presence loop can never come to depend on bytes the loop does not promise. + EXPECT_TRUE(std::holds_alternative(conflict->seen)); + EXPECT_FALSE(std::holds_alternative(conflict->seen)); + EXPECT_GE(backend->readRequests(), 1u); +} + +TEST(CASRequests, ForEachListedKeyGivesEachPageItsOwnPolicyWindow) +{ + FakeClock clock; + auto backend = std::make_shared(); + /// Every list costs the caller 300ms, so a page's cost is a fact and not a jitter draw. + backend->charge_latency = [&clock] { clock.now += 300; }; + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + for (int i = 0; i < 25; ++i) + orThrow(op.create("p/" + std::to_string(i), "v", Retry::standard()), "create"); + + size_t seen = 0; + size_t pages = 0; + const uint64_t start = clock.now; + /// A window that comfortably covers ONE page's refusal and its reissue, and could not have covered + /// the walk: the policy governs each page, because a walk is an unbounded number of requests. + op.forEachListedKey("p/", [&](const KeyEntry &) { ++seen; return true; }, Retry::within(1'000), + /*page_limit=*/10, [&] { ++pages; }); + EXPECT_EQ(seen, 25u); + EXPECT_EQ(pages, 3u); + EXPECT_EQ(backend->list_calls, 6u); /// each page refused once, then delivered + EXPECT_GT(clock.now - start, 1'000u); /// the walk outlived the window every page was given +} + +TEST(CASRequests, ForEachListedKeyThrowsRatherThanTruncateWhenAPageNeverArrives) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->charge_latency = [&clock] { clock.now += 300; }; + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + for (int i = 0; i < 25; ++i) + orThrow(op.create("p/" + std::to_string(i), "v", Retry::standard()), "create"); + + const KeyPage first = op.list("p/", "", 10, Retry::within(1'000)); + ASSERT_FALSE(first.next_cursor.empty()); + backend->always_refuse_cursor = first.next_cursor; /// the second page never arrives + backend->refused_cursors.clear(); + + size_t seen = 0; + size_t pages = 0; + /// A silently truncated enumeration is the error a coverage record exists to prevent, so the walk + /// reports the page it could not fetch instead of returning what it managed to read. + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + { + op.forEachListedKey("p/", [&](const KeyEntry &) { ++seen; return true; }, Retry::within(1'000), + /*page_limit=*/10, [&] { ++pages; }); + }); + EXPECT_EQ(pages, 1u); + EXPECT_EQ(seen, 10u); +} + +TEST(CASRequests, LivenessPredicateEndsTheOperationLikeAFenceLoss) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + bool alive = true; + auto op = requests.admit([&] { return alive; }); + EXPECT_TRUE(op.admitted()); + + alive = false; + EXPECT_FALSE(op.admitted()); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(backend->writeRequests(), 0u); + + /// The read surface reports the same refusal the only way it can: by exception. + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_EQ(backend->readRequests(), 0u); +} + +TEST(CASRequests, ReadModifyWriteLosesNoIncrementUnderContentionAndBoundsAHotKey) +{ + auto backend = std::make_shared(); + const auto increment = [](const std::optional & current) -> std::optional + { + return std::to_string(std::stoi(current ? current->bytes : "0") + 1); + }; + + /// The real clock and the real sleep: two threads share this engine, and a `FakeClock` would be a + /// data race on both of its fields. + CasRequests contended(backend, Fence::open()); + { + auto seed = contended.admit(); + orThrow(seed.create("ctr", "0", Retry::standard()), "create"); + } + const auto fifty_increments = [&] + { + auto op = contended.admit(); + for (int i = 0; i < 50; ++i) + orThrow(op.readModifyWrite("ctr", increment, Retry::standard()), "increment"); + }; + std::thread first(fifty_increments); + std::thread second(fifty_increments); + first.join(); + second.join(); + + auto reader = contended.admit(); + const auto counted = reader.read("ctr", Retry::standard()); + ASSERT_TRUE(counted.has_value()); + EXPECT_EQ(counted->bytes, "100"); /// every conflict re-decided against what the resolve read saw + + /// A key rewritten under EVERY attempt is bounded by the deadline instead of looping forever. + FakeClock clock; + auto hot = makeRequests(backend, clock); + bool inside_hook = false; + backend->onBeforeWrite("ctr", [&] + { + if (inside_hook) /// the hook's own write re-enters this callback + return; + inside_hook = true; + auto door = RawDoor::key(); + if (auto raw = backend->read("ctr", door)) + (void)backend->write("ctr", "999", raw->value, door); + inside_hook = false; + }); + + auto op = hot.admit(); + WriteResult result = op.readModifyWrite("ctr", increment, Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_FALSE(clock.sleeps.empty()); /// it paced its retries rather than spinning +} + +TEST(CASRequests, ADeterministicLocalFailureSurfacesUnchangedWithoutAReissue) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextReadWith("k", std::make_exception_ptr( + DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "the object at 'k' is not decodable"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + /// Reissuing would replay the same bug and bury it behind a retryable exception at the deadline. + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, ATransportTimeoutIsReissuedAndALocalFailureIsNot) +{ + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "create"); + backend->resetCounts(); + + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("the read timed out"))); + const auto seen = op.read("k", Retry::standard()); + ASSERT_TRUE(seen.has_value()); + EXPECT_EQ(seen->bytes, "v"); + EXPECT_EQ(backend->readRequests(), 2u); + EXPECT_EQ(clock.sleeps.size(), 1u); + } + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + /// Not a `Poco::Exception`, so it did not come from the transport: reissuing it would spend the + /// whole deadline replaying a local bug. + backend->failNextReadWith("k", std::make_exception_ptr(std::logic_error("a local bug"))); + EXPECT_THROW((void)op.read("k", Retry::standard()), std::logic_error); + EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); + } +} + +#if USE_AWS_S3 + +namespace +{ + +/// An `S3Exception` carrying a canonical `` name. The name is how the request contract tells +/// one store answer from another: the SDK reports every error it does not model as `UNKNOWN`, so the +/// code alone can never stand for a particular failure. +std::exception_ptr s3Error(Aws::S3::S3Errors code, const String & name) +{ + return std::make_exception_ptr(DB::S3Exception("the store answered " + name, code, name)); +} + +} + +TEST(CASRequests, DeadlineIsTheOnlyBoundUnderZeroLatencyThrottling) +{ + FakeClock clock; + auto throttled = std::make_shared( + std::make_shared(), ThrottlingBackend::Mode::EveryNth, 1, 429); + auto requests = makeRequests(throttled, clock); + auto op = requests.admit(); + + const uint64_t start = clock.now; + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Policy); + EXPECT_TRUE(gave_up->sent_any); + /// What ends the call is the policy's own deadline, not a count of attempts: it kept issuing to + /// within one backoff of that deadline, and paused many more times than a small fixed budget allows. + EXPECT_GE(clock.now - start, 85'000u); + EXPECT_GT(clock.sleeps.size(), 16u); +} + +TEST(CASRequests, LeaseBoundPolicyIssuesNothingPastTheBoundary) +{ + FakeClock clock; + auto throttled = std::make_shared( + std::make_shared(), ThrottlingBackend::Mode::EveryNth, 1, 429); + auto requests = makeRequests(throttled, clock); + requests.setAttemptReservationForTest(1'000); + + const uint64_t lease_deadline = clock.now + 10'000; + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::untilLeaseSafe(lease_deadline, 2'000)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + /// The lease was the smaller of the two bounds, and the give-up names it rather than the policy. + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Lease); + /// Nothing is STARTED that could not finish inside the bound: the last request began at least one + /// attempt reservation before lease minus margin. + EXPECT_LE(clock.now, lease_deadline - 2'000 - 1'000); +} + +TEST(CASRequests, AMalformedRequestIsRefusedWithoutAReissue) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::UNKNOWN, "MalformedXML")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * refused = std::get_if(&result); + ASSERT_NE(refused, nullptr); + EXPECT_EQ(refused->store_error, DB::ErrorCodes::S3_ERROR); + /// The store's own answer proves the request never applied: nothing to resolve, nothing to reissue, + /// and no credential the refusal could be about. + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 0u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, AnAccessDenialNoRefreshCanFixIsRefusedOnTheFirstAttempt) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(false); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + ASSERT_TRUE(std::holds_alternative(result)); + /// A refresh is asked for once and installs nothing, and THAT is what makes the denial terminal. + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, ASecondCredentialAnswerAfterTheOneRefreshIsRefused) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + /// The store answers a denial BEFORE it applies anything, so neither attempt landed and no read + /// has anything to settle. A call gets one refresh, so the denial that survives it is the answer. + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + EXPECT_EQ(backend->writeRequests(), 2u); + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(clock.sleeps.size(), 1u); /// the one paced re-send under the credentials it installed +} + +TEST(CASRequests, UnderOnceTheStoreAnswerStandsEvenWhenTheRefreshSucceeded) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + /// Fresh credentials only help a reissue, and `once` has none to sign. Reporting the attempt as + /// unresolved instead would turn a policy that sends one request into one that sleeps. + WriteResult result = op.create("k", "v", Retry::once()); + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, ACredentialAnswerAfterAnAmbiguousAttemptStillOwesTheResolveRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + /// The first attempt's fate is unknown and it may yet land; the second is a proven non-application. + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 3u); + /// The refresh does not license a direct re-send here: the OTHER attempt is still unresolved, so + /// the read that settles it is still owed. + EXPECT_EQ(backend->readRequests(), 2u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); +} + +TEST(CASRequests, AnExpiredTokenARefreshFixesIsResentWithoutAResolveRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::INVALID_CLIENT_TOKEN_ID, "ExpiredToken")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_FALSE(committed->resolved_by_read); + /// The credential answer proves its OWN attempt never applied, and no earlier attempt of this call + /// is unresolved, so the re-send under the fresh credentials owes no read. + EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + EXPECT_EQ(clock.sleeps.size(), 1u); +} + +TEST(CASRequests, AnExpiredTokenNoRefreshCanFixIsRefusedRatherThanRiddenToTheDeadline) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::INVALID_CLIENT_TOKEN_ID, "ExpiredToken")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + WriteResult result = op.create("k", "v", Retry::standard()); + /// The refusal class CONTAINS the refresh class: an expired credential that no refresh installed + /// would otherwise spend the whole deadline being reissued. + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, ANameOnlyAccessDenialOnAReadPropagatesWhenNoRefreshIsAvailable) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(false); + /// Matched by NAME alone: the SDK reports this store's denial under its catch-all code, so the + /// name is the only thing that says a credential could explain it. + backend->failNextReadWith("k", s3Error(Aws::S3::S3Errors::UNKNOWN, "AccessDenied")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + /// One refresh is asked for and installs nothing, so nothing would sign differently: the read + /// propagates instead of spending its policy on a request that cannot start succeeding. + expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, AnUnmodeledStoreErrorOnAReadIsReissuedNotSurfaced) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "create"); + backend->resetCounts(); + + /// An S3-compatible store's own vendor code. The SDK models it as `UNKNOWN`, which is its code for + /// EVERY error it does not know, so it can never stand for "this will not start succeeding". + backend->failNextReadWith("k", s3Error(Aws::S3::S3Errors::UNKNOWN, "SomeVendorCode")); + const auto seen = op.read("k", Retry::standard()); + ASSERT_TRUE(seen.has_value()); + EXPECT_EQ(seen->bytes, "v"); + EXPECT_EQ(backend->readRequests(), 2u); + EXPECT_EQ(clock.sleeps.size(), 1u); +} + +#endif diff --git a/src/Disks/tests/gtest_cas_retirement_sweep.cpp b/src/Disks/tests/gtest_cas_retirement_sweep.cpp index 7300f02d0c7c..6deb241ec7a4 100644 --- a/src/Disks/tests/gtest_cas_retirement_sweep.cpp +++ b/src/Disks/tests/gtest_cas_retirement_sweep.cpp @@ -63,6 +63,8 @@ namespace class HoleyListBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; void omitFromNthListCall(const String & key, size_t nth) { std::lock_guard lock(m); @@ -114,6 +116,8 @@ class HoleyListBackend : public InMemoryBackend class RefPrefixListCountingBackend : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::list; String refs_prefix; String janitor_prefix; std::atomic ref_prefix_lists{0}; diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 92c4542b1a32..6beb6a08dabd 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -202,6 +202,10 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend explicit EtagFaithfulPublicationBackend(FaultScript script_) : script(script_) {} + /// Unhide the transport primitive that shares this name; the legacy override below is what the + /// production sites this double instruments still call. + using DB::Cas::Backend::head; + DB::Cas::HeadResult head(const String & key) override { DB::Cas::HeadResult result = DB::Cas::InMemoryBackend::head(key); @@ -1035,6 +1039,47 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage return tryGetObjectMetadata(path, with_tags); } + /// This fake advertises every retry profile, and an in-memory store has no retry behaviour to + /// vary, so the profile-aware overloads simply forward. A storage that claimed the capability + /// without implementing them would refuse every control-plane request of a writable mount. + std::optional tryGetObjectMetadataWithNativeToken( + const std::string & path, bool with_tags, DB::ObjectStorageRetryProfile, uint64_t) const override + { + return tryGetObjectMetadata(path, with_tags); + } + + DB::ObjectStorageIteratorPtr iterate( + const std::string & path_prefix, size_t max_keys, bool with_tags, const std::optional & start_after, + DB::ObjectStorageRetryProfile, uint64_t) const override + { + return DB::LocalObjectStorage::iterate(path_prefix, max_keys, with_tags, start_after); + } + + DB::ConditionalRemoveResult removeObjectIfTokenMatches( + const DB::StoredObject & object, const std::string & etag, DB::ObjectStorageRetryProfile, uint64_t) override + { + return removeObjectIfTokenMatches(object, etag); + } + using DB::LocalObjectStorage::removeObjectIfTokenMatches; + + /// A real S3 GET answers with the object's incarnation, which is what the backend reads its + /// bytes AND its generation from in one request. Quoted, the way the SDK's ETag field carries a + /// generation across the HTTP boundary. + DB::SmallObjectDataWithMetadata readSmallObjectAndGetObjectMetadata( + const DB::StoredObject & object, const DB::ReadSettings &, size_t, std::optional) const override + { + std::lock_guard lock(mutex); + auto it = objects.find(object.remote_path); + if (it == objects.end()) + throw DB::S3Exception("FakeGenerationObjectStorage: object does not exist", + Aws::S3::S3Errors::RESOURCE_NOT_FOUND); + DB::SmallObjectDataWithMetadata result; + result.data = it->second.bytes; + result.metadata.size_bytes = it->second.bytes.size(); + result.metadata.etag = "\"" + std::to_string(it->second.generation) + "\""; + return result; + } + void removeObjectIfExists(const DB::StoredObject & object) override { std::lock_guard lock(mutex); @@ -1093,7 +1138,9 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage /// stores `bytes` and mints the next generation. Throws an `S3Exception` naming `PreconditionFailed` /// on a lost condition -- the one signal `finalizeConditionalWrite` classifies as /// `PutOutcome::PreconditionFailed` rather than an ordinary failure. - void commitConditionalWrite(const std::string & key, const std::string & bytes, + /// Returns the generation it minted, the way a real store returns it in the write response: the + /// backend attributes the write to that generation and nothing reads it back. + uint64_t commitConditionalWrite(const std::string & key, const std::string & bytes, const std::string & if_none_match, const std::string & if_match) { std::lock_guard lock(mutex); @@ -1106,7 +1153,9 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage throw DB::S3Exception("FakeGenerationObjectStorage: if-match precondition failed", Aws::S3::S3Errors::UNKNOWN, "PreconditionFailed"); - objects[key] = Entry{bytes, next_generation++}; + const uint64_t generation = next_generation++; + objects[key] = Entry{bytes, generation}; + return generation; } private: @@ -1132,6 +1181,10 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage void sync() override {} std::string getFileName() const override { return key; } + /// The write response's own incarnation, quoted the way the SDK's ETag field carries a GCS + /// generation across the HTTP boundary -- the backend is what strips that transport syntax. + std::optional getResultObjectETag() const override { return committed_generation; } + protected: void nextImpl() override { @@ -1143,7 +1196,7 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage void finalizeImpl() override { next(); - storage.commitConditionalWrite(key, buffered, if_none_match, if_match); + committed_generation = "\"" + std::to_string(storage.commitConditionalWrite(key, buffered, if_none_match, if_match)) + "\""; } private: @@ -1152,6 +1205,7 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage std::string if_none_match; std::string if_match; std::string buffered; + std::optional committed_generation; }; mutable std::mutex mutex; diff --git a/src/Disks/tests/gtest_cas_sentinel_probe.cpp b/src/Disks/tests/gtest_cas_sentinel_probe.cpp index 05411f0c3848..85896cf43d05 100644 --- a/src/Disks/tests/gtest_cas_sentinel_probe.cpp +++ b/src/Disks/tests/gtest_cas_sentinel_probe.cpp @@ -8,7 +8,9 @@ #include #include #include +#include #include +#include #include #include @@ -32,39 +34,37 @@ namespace using DB::Cas::tests::nativeKeyUnder; -/// A Backend decorator whose head/get/list all throw an untyped runtime error when armed — modelling +/// A Backend decorator whose read/head/list all throw an untyped runtime error when armed — modelling /// a backend with no sharper evidence than "something went wrong" (a network timeout, a 5xx, an -/// unclassifiable failure). Mirrors the existing MetaWriteFaultBackend fault-injection pattern -/// (cas_test_helpers.h): every other operation delegates to InMemoryBackend unchanged. +/// unclassifiable failure). The fault is injected on the PRIMITIVES, which is what +/// `Backend::probeSentinelRaw`'s default derives its answer from; every other operation delegates to +/// InMemoryBackend unchanged. class TransportFaultBackend final : public InMemoryBackend { public: - /// Unhide the base convenience overloads, matching every other Backend subclass in this suite. - using Backend::get; using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; + using Backend::head; + using Backend::list; - HeadResult head(const String & key) override + std::optional head(const String & key, TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } std::atomic fail{true}; @@ -129,7 +129,14 @@ TEST(CASSentinelProbe, NativePresentKeyReturnsPresentWithBody) ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); const String key = nativeKeyUnder(storage, "some/key"); - ASSERT_EQ(backend.putIfAbsent(key, "native body").outcome, PutOutcome::Done); + /// Placed through the object storage: a Native write over a local storage has no response + /// incarnation to attribute itself to. Native passes the key verbatim, so this is the object the + /// probe reads. + { + auto out = storage->writeObject(DB::StoredObject(key), DB::WriteMode::Rewrite); + DB::writeString(String("native body"), *out); + out->finalize(); + } const auto result = probeSentinel(backend, key); EXPECT_EQ(result.outcome, ProbeOutcome::Present); @@ -152,30 +159,44 @@ TEST(CASSentinelProbe, TransportErrorNeverClassifiesAsAbsent) namespace { -/// A `LocalObjectStorage` whose `getObjectMetadata` can be armed to throw a configurable synthetic +/// A `LocalObjectStorage` whose object ACCESS can be armed to throw a configurable synthetic /// `S3Exception` — the same technique `gtest_cas_backend.cpp`'s `NativeReadThrowsNoSuchKeyObjectStorage` /// uses to exercise S3 error codes without a live S3 endpoint. Constructing `ObjectStorageBackend` in /// `Mode::Native` over this fake is the established pattern for testing the Native/S3 raw-error classifier /// in isolation (see also `gtest_cas_backend.cpp`'s `NativeRejectsWrongDialectTokenBeforeTouchingTheWire`). -class ThrowingS3MetadataObjectStorage final : public DB::LocalObjectStorage +/// Both the read and the metadata surface throw: the Native sentinel probe issues one READ, and a +/// store that answers an error for a key answers it however the key is touched. +class ThrowingS3ObjectStorage final : public DB::LocalObjectStorage { public: using DB::LocalObjectStorage::LocalObjectStorage; - void throwOnGetObjectMetadata(Aws::S3::S3Errors code) { metadata_error = code; } + void throwOnObjectAccess(Aws::S3::S3Errors code) { access_error = code; } + + std::unique_ptr readObject( + const DB::StoredObject & object, + const DB::ReadSettings & read_settings, + std::optional read_hint, + bool use_external_buffer, + bool restrict_seek) const override + { + if (access_error) + throw DB::S3Exception("injected fault: " + object.remote_path, *access_error); + return DB::LocalObjectStorage::readObject(object, read_settings, read_hint, use_external_buffer, restrict_seek); + } DB::ObjectMetadata getObjectMetadata(const std::string & path, bool with_tags) const override { - if (metadata_error) - throw DB::S3Exception("injected fault: " + path, *metadata_error); + if (access_error) + throw DB::S3Exception("injected fault: " + path, *access_error); return DB::LocalObjectStorage::getObjectMetadata(path, with_tags); } private: - std::optional metadata_error; + std::optional access_error; }; -DB::ObjectStoragePtr makeThrowingS3MetadataStorageForTest() +DB::ObjectStoragePtr makeThrowingS3StorageForTest() { static std::atomic counter{0}; const auto unique = std::to_string(::getpid()) + "_" + std::to_string(counter.fetch_add(1)); @@ -186,7 +207,7 @@ DB::ObjectStoragePtr makeThrowingS3MetadataStorageForTest() std::filesystem::create_directories(root, ec); DB::LocalObjectStorageSettings settings("test", root, /*read_only_=*/false); - return std::make_shared(std::move(settings)); + return std::make_shared(std::move(settings)); } } @@ -195,8 +216,8 @@ DB::ObjectStoragePtr makeThrowingS3MetadataStorageForTest() /// error must classify EXACTLY, and anything unmodeled must fail closed to Indeterminate. TEST(CASSentinelProbe, NativeClassifiesNoSuchKeyAsKeyAbsent) { - auto storage = std::static_pointer_cast(makeThrowingS3MetadataStorageForTest()); - storage->throwOnGetObjectMetadata(Aws::S3::S3Errors::NO_SUCH_KEY); + auto storage = std::static_pointer_cast(makeThrowingS3StorageForTest()); + storage->throwOnObjectAccess(Aws::S3::S3Errors::NO_SUCH_KEY); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::KeyAbsent); @@ -208,8 +229,8 @@ TEST(CASSentinelProbe, NativeClassifiesNoSuchKeyAsKeyAbsent) /// `NO_SUCH_KEY`. Without classifying it, every real-S3 absence would be `Indeterminate` forever. TEST(CASSentinelProbe, NativeClassifiesResourceNotFoundAsKeyAbsent) { - auto storage = std::static_pointer_cast(makeThrowingS3MetadataStorageForTest()); - storage->throwOnGetObjectMetadata(Aws::S3::S3Errors::RESOURCE_NOT_FOUND); + auto storage = std::static_pointer_cast(makeThrowingS3StorageForTest()); + storage->throwOnObjectAccess(Aws::S3::S3Errors::RESOURCE_NOT_FOUND); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::KeyAbsent); @@ -217,8 +238,8 @@ TEST(CASSentinelProbe, NativeClassifiesResourceNotFoundAsKeyAbsent) TEST(CASSentinelProbe, NativeClassifiesNoSuchBucketAsContainerAbsent) { - auto storage = std::static_pointer_cast(makeThrowingS3MetadataStorageForTest()); - storage->throwOnGetObjectMetadata(Aws::S3::S3Errors::NO_SUCH_BUCKET); + auto storage = std::static_pointer_cast(makeThrowingS3StorageForTest()); + storage->throwOnObjectAccess(Aws::S3::S3Errors::NO_SUCH_BUCKET); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::ContainerAbsent); @@ -226,8 +247,8 @@ TEST(CASSentinelProbe, NativeClassifiesNoSuchBucketAsContainerAbsent) TEST(CASSentinelProbe, NativeClassifiesAccessDeniedAsAccessDenied) { - auto storage = std::static_pointer_cast(makeThrowingS3MetadataStorageForTest()); - storage->throwOnGetObjectMetadata(Aws::S3::S3Errors::ACCESS_DENIED); + auto storage = std::static_pointer_cast(makeThrowingS3StorageForTest()); + storage->throwOnObjectAccess(Aws::S3::S3Errors::ACCESS_DENIED); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::AccessDenied); @@ -235,8 +256,8 @@ TEST(CASSentinelProbe, NativeClassifiesAccessDeniedAsAccessDenied) TEST(CASSentinelProbe, NativeClassifiesUnmodeledErrorAsIndeterminate) { - auto storage = std::static_pointer_cast(makeThrowingS3MetadataStorageForTest()); - storage->throwOnGetObjectMetadata(Aws::S3::S3Errors::SERVICE_UNAVAILABLE); + auto storage = std::static_pointer_cast(makeThrowingS3StorageForTest()); + storage->throwOnObjectAccess(Aws::S3::S3Errors::SERVICE_UNAVAILABLE); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::Indeterminate); @@ -252,8 +273,8 @@ TEST(CASSentinelProbe, NativeClassifiesUnmodeledErrorAsIndeterminate) /// reached through the wrapper. TEST(CASSentinelProbe, InstrumentedBackendForwardsToInnerClassification) { - auto storage = std::static_pointer_cast(makeThrowingS3MetadataStorageForTest()); - storage->throwOnGetObjectMetadata(Aws::S3::S3Errors::NO_SUCH_BUCKET); + auto storage = std::static_pointer_cast(makeThrowingS3StorageForTest()); + storage->throwOnObjectAccess(Aws::S3::S3Errors::NO_SUCH_BUCKET); auto inner = std::make_shared(storage, ObjectStorageBackend::Mode::Native); InstrumentedBackend instrumented(inner); diff --git a/src/Disks/tests/gtest_cas_upload_detached.cpp b/src/Disks/tests/gtest_cas_upload_detached.cpp index 6f6f341f4bb6..0f158475f269 100644 --- a/src/Disks/tests/gtest_cas_upload_detached.cpp +++ b/src/Disks/tests/gtest_cas_upload_detached.cpp @@ -106,6 +106,8 @@ std::optional metaStateAt(InMemoryBackend & b, const Layout & layout, class ProtocolRecordingBackend final : public InMemoryBackend { public: + /// Unhide the primitive overload that the legacy override below would otherwise hide. + using InMemoryBackend::head; void watch(String blob_key_, String meta_key_) { blob_key = std::move(blob_key_); diff --git a/src/Disks/tests/gtest_cas_upstream_slice.cpp b/src/Disks/tests/gtest_cas_upstream_slice.cpp new file mode 100644 index 000000000000..73f894b28b9e --- /dev/null +++ b/src/Disks/tests/gtest_cas_upstream_slice.cpp @@ -0,0 +1,743 @@ +#include + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "config.h" + +#if USE_AWS_S3 +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include +#include +#endif + +namespace DB::ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +#if USE_AWS_S3 + extern const int CANNOT_READ_ALL_DATA; + extern const int NETWORK_ERROR; +#endif +} + +namespace +{ + +/// Same unique-temp-root convention as the other CAS unit tests, so parallel runs never share a root. +std::shared_ptr makeLocalObjectStorageForRetryProfileTest() +{ + static std::atomic counter{0}; + const auto unique = std::to_string(::getpid()) + "_" + std::to_string(counter.fetch_add(1)); + const auto root = (std::filesystem::temp_directory_path() / ("cas_unit_upstream_slice_" + unique)).string(); + + std::error_code ec; + std::filesystem::remove_all(root, ec); + std::filesystem::create_directories(root, ec); + + return std::make_shared(DB::LocalObjectStorageSettings("test", root, /*read_only_=*/false)); +} + +/// Every refusal below is NOT_IMPLEMENTED, and so is the pre-existing refusal of conditional removal, +/// so the code alone cannot tell which one fired. Match a phrase unique to the intended message too. +template +void expectThrowsNotImplementedSaying(const std::string & needle, F && fn) +{ + try + { + fn(); + FAIL() << "expected DB::Exception saying '" << needle << "'"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find(needle), std::string::npos) << "actual message: " << e.message(); + } +} + +} + +/// The base `IObjectStorage` bodies forward `Default` and refuse `SingleAttempt`: a caller that asked +/// for one attempt has its own deadline, and a transparently retried request would outlive it. +TEST(CASUpstreamSlice, HeadListRemoveOverloadsRefuseSingleAttemptOnTheBaseStorage) +{ + auto local = makeLocalObjectStorageForRetryProfileTest(); + + expectThrowsNotImplementedSaying( + "single-attempt metadata requests", + [&] { local->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::SingleAttempt, 0); }); + expectThrowsNotImplementedSaying( + "single-attempt listing requests", + [&] { local->iterate("", 1, false, {}, DB::ObjectStorageRetryProfile::SingleAttempt, 0); }); + expectThrowsNotImplementedSaying( + "single-attempt removal requests", + [&] { local->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::SingleAttempt, 0); }); + + /// `Default` must keep reaching the ordinary implementation. For removal that is still a refusal, + /// but the pre-existing one — matching its wording proves the profile overload forwarded. + EXPECT_NO_THROW(local->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::Default, 0)); + EXPECT_NO_THROW(local->iterate("", 1, false, {}, DB::ObjectStorageRetryProfile::Default, 0)); + expectThrowsNotImplementedSaying( + "Conditional (token-exact) object removal", + [&] { local->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::Default, 0); }); +} + +#if USE_AWS_S3 + +namespace +{ + +/// One scripted answer to a `GetObject`. `fail_mid_body` makes the response stream throw after it has +/// already delivered bytes, which is what drives `ReadBufferFromS3` to reissue the request. +struct ScriptedGetObjectStep +{ + bool ok = true; + Aws::S3::S3Errors error = Aws::S3::S3Errors::SLOW_DOWN; + std::string exception_name; + std::string etag; + std::string body; + bool fail_mid_body = false; +}; + +ScriptedGetObjectStep okStep(const std::string & etag, const std::string & body, bool fail_mid_body = false) +{ + return ScriptedGetObjectStep{ + .ok = true, + .error = Aws::S3::S3Errors::SLOW_DOWN, + .exception_name = "", + .etag = etag, + .body = body, + .fail_mid_body = fail_mid_body}; +} + +ScriptedGetObjectStep throttleStep() +{ + return ScriptedGetObjectStep{ + .ok = false, + .error = Aws::S3::S3Errors::SLOW_DOWN, + .exception_name = "SlowDown", + .etag = "", + .body = "", + .fail_mid_body = false}; +} + +/// `S3Exception::isAccessTokenExpiredError` keys on the error CODE, not the name. +ScriptedGetObjectStep expiredTokenStep() +{ + return ScriptedGetObjectStep{ + .ok = false, + .error = Aws::S3::S3Errors::ACCESS_DENIED, + .exception_name = "ExpiredToken", + .etag = "", + .body = "", + .fail_mid_body = false}; +} + +/// One scripted answer to a control-plane request (HEAD or conditional DELETE). `retryable` is what +/// the SDK's retry strategy consults, so it is what decides whether the client's own attempt loop +/// reissues the request — which is how a single-attempt clone is told apart from the disk client. +struct ScriptedControlStep +{ + bool ok = true; + Aws::S3::S3Errors error = Aws::S3::S3Errors::SLOW_DOWN; + std::string exception_name; + bool retryable = false; +}; + +ScriptedControlStep controlOk() +{ + return ScriptedControlStep{.ok = true, .error = Aws::S3::S3Errors::SLOW_DOWN, .exception_name = "", .retryable = false}; +} + +ScriptedControlStep controlExpiredToken() +{ + return ScriptedControlStep{ + .ok = false, .error = Aws::S3::S3Errors::ACCESS_DENIED, .exception_name = "ExpiredToken", .retryable = false}; +} + +ScriptedControlStep controlThrottle() +{ + return ScriptedControlStep{ + .ok = false, .error = Aws::S3::S3Errors::SLOW_DOWN, .exception_name = "SlowDown", .retryable = true}; +} + +/// `ReadBufferFromIStream` reads through `Poco::Net::HTTPBasicStreamBuf::readFromDevice`, so a fake +/// response body has to be one of those rather than a plain `std::stringstream`. +class ScriptedBodyStreamBuf : public Poco::Net::HTTPBasicStreamBuf +{ +public: + ScriptedBodyStreamBuf(std::string body_, bool fail_mid_body_) + : Poco::Net::HTTPBasicStreamBuf(256, std::ios::in), body(std::move(body_)), fail_mid_body(fail_mid_body_) + { + } + +private: + int readFromDevice(char * buffer, std::streamsize length) override + { + if (fail_mid_body && position > 0) + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "scripted failure part-way through the response body"); + + const size_t available = body.size() - position; + const size_t n = std::min(static_cast(length), available); + std::memcpy(buffer, body.data() + position, n); + position += n; + return static_cast(n); + } + + const std::string body; + const bool fail_mid_body; + size_t position = 0; +}; + +class ScriptedBodyStreamHolder +{ +protected: + ScriptedBodyStreamHolder(std::string body, bool fail_mid_body) : buf(std::move(body), fail_mid_body) { } + ScriptedBodyStreamBuf buf; +}; + +/// The holder base is listed first so `buf` is constructed before `std::iostream` is handed its address. +class ScriptedBodyStream : private ScriptedBodyStreamHolder, public std::iostream +{ +public: + ScriptedBodyStream(std::string body, bool fail_mid_body) + : ScriptedBodyStreamHolder(std::move(body), fail_mid_body), std::iostream(&buf) + { + } +}; + +/// An `S3::Client` whose `GetObject` answers from a script, recording how many times it was called and +/// whether each request carried the native-conditional mark. Clones share the state, so the counters +/// still see the requests issued through the single-attempt clone. +class ScriptedGetObjectClient : public DB::S3::Client +{ +private: + struct State + { + std::vector script; + std::vector head_script; + std::vector delete_script; + + size_t get_object_calls = 0; + std::vector native_conditional_marks; + + /// The `requestTimeoutMs` of the client each request was actually issued through, which is + /// what proves a request rode the clone built for the bound its caller asked for. + std::vector head_request_timeouts_ms; + std::vector delete_request_timeouts_ms; + /// Every configuration this client was asked to clone with — a request-free way to see which + /// client a verb selected. + std::vector clone_request_timeouts_ms; + + std::mutex mutex; + }; + + const std::shared_ptr state; + +public: + ScriptedGetObjectClient() : ScriptedGetObjectClient(std::make_shared(), GetClientConfiguration()) { } + + static DB::S3::PocoHTTPClientConfiguration GetClientConfiguration() + { + DB::RemoteHostFilter remote_host_filter; + /// max_retries is deliberately nonzero: it is the disk client's own attempt loop, and the + /// only thing that distinguishes it from the single-attempt clone. The two slow-down flags + /// are off so that loop spins without waiting out a real backoff. + return DB::S3::ClientFactory::instance().createClientConfiguration( + "some-region", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 2}, + /* s3_slow_all_threads_after_network_error = */ false, + /* s3_slow_all_threads_after_retryable_error = */ false, + /* enable_s3_requests_logging = */ true, + /* for_disk_s3 = */ false, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); + } + + void script(std::vector steps) const + { + std::lock_guard lock(state->mutex); + state->script = std::move(steps); + } + + size_t getObjectCalls() const + { + std::lock_guard lock(state->mutex); + return state->get_object_calls; + } + + std::vector nativeConditionalMarks() const + { + std::lock_guard lock(state->mutex); + return state->native_conditional_marks; + } + + void scriptHead(std::vector steps) const + { + std::lock_guard lock(state->mutex); + state->head_script = std::move(steps); + } + + void scriptDelete(std::vector steps) const + { + std::lock_guard lock(state->mutex); + state->delete_script = std::move(steps); + } + + std::vector headRequestTimeouts() const + { + std::lock_guard lock(state->mutex); + return state->head_request_timeouts_ms; + } + + std::vector deleteRequestTimeouts() const + { + std::lock_guard lock(state->mutex); + return state->delete_request_timeouts_ms; + } + + std::vector cloneRequestTimeouts() const + { + std::lock_guard lock(state->mutex); + return state->clone_request_timeouts_ms; + } + + std::unique_ptr cloneWithConfigurationOverride( + const DB::S3::PocoHTTPClientConfiguration & client_configuration_override) const override + { + { + std::lock_guard lock(state->mutex); + state->clone_request_timeouts_ms.push_back(client_configuration_override.requestTimeoutMs); + } + return std::unique_ptr(new ScriptedGetObjectClient(state, client_configuration_override)); + } + + Aws::S3::Model::GetObjectOutcome GetObject(const Aws::S3::Model::GetObjectRequest & request) const override + { + std::lock_guard lock(state->mutex); + + const auto * marked = dynamic_cast(&request); + state->native_conditional_marks.push_back(marked != nullptr && marked->isNativeConditional()); + + const size_t index = state->get_object_calls++; + if (index >= state->script.size()) + { + return Aws::S3::Model::GetObjectOutcome(Aws::Client::AWSError( + Aws::S3::S3Errors::NO_SUCH_KEY, "NoSuchKey", "the script has no answer for this request", false)); + } + + const auto & step = state->script[index]; + if (!step.ok) + { + return Aws::S3::Model::GetObjectOutcome(Aws::Client::AWSError( + step.error, step.exception_name, "scripted error", false)); + } + + Aws::S3::Model::GetObjectResult result; + result.SetETag(step.etag); + result.SetContentLength(static_cast(step.body.size())); + result.ReplaceBody(new ScriptedBodyStream(step.body, step.fail_mid_body)); + return Aws::S3::Model::GetObjectOutcome(std::move(result)); + } + + Aws::S3::Model::HeadObjectOutcome HeadObject(const Aws::S3::Model::HeadObjectRequest & /*request*/) const override + { + std::lock_guard lock(state->mutex); + state->head_request_timeouts_ms.push_back(getClientConfiguration().requestTimeoutMs); + + const auto step = nextControlStep(state->head_script, state->head_request_timeouts_ms.size()); + if (!step.ok) + return Aws::S3::Model::HeadObjectOutcome(makeError(step)); + + Aws::S3::Model::HeadObjectResult result; + /// Any nonzero size: tryGetObjectMetadataImpl reads an all-zero HeadObjectResult as a miss. + result.SetContentLength(scripted_head_object_size); + result.SetETag(scripted_head_object_etag); + return Aws::S3::Model::HeadObjectOutcome(std::move(result)); + } + + Aws::S3::Model::DeleteObjectOutcome DeleteObject(const Aws::S3::Model::DeleteObjectRequest & /*request*/) const override + { + std::lock_guard lock(state->mutex); + state->delete_request_timeouts_ms.push_back(getClientConfiguration().requestTimeoutMs); + + const auto step = nextControlStep(state->delete_script, state->delete_request_timeouts_ms.size()); + if (!step.ok) + return Aws::S3::Model::DeleteObjectOutcome(makeError(step)); + + Aws::S3::Model::DeleteObjectResult result; + result.SetDeleteMarker(false); + return Aws::S3::Model::DeleteObjectOutcome(std::move(result)); + } + +private: + static constexpr long long scripted_head_object_size = 7; + static constexpr const char * scripted_head_object_etag = "\"h1\""; + + /// A script shorter than the number of requests keeps answering with its last step, so a test + /// that means "this error, however many attempts the client makes" says it in one entry. + static ScriptedControlStep nextControlStep(const std::vector & script, size_t call_number) + { + if (script.empty()) + return controlOk(); + return script[std::min(call_number - 1, script.size() - 1)]; + } + + static Aws::Client::AWSError makeError(const ScriptedControlStep & step) + { + return Aws::Client::AWSError(step.error, step.exception_name, "scripted error", step.retryable); + } + + ScriptedGetObjectClient(std::shared_ptr state_, const DB::S3::PocoHTTPClientConfiguration & client_configuration) + : DB::S3::Client( + 100, + DB::S3::ServerSideEncryptionKMSConfig(), + std::make_shared("", ""), + client_configuration, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + DB::S3::ClientSettings{ + .use_virtual_addressing = true, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }) + , state(std::move(state_)) + { + } +}; + +std::shared_ptr makeScriptedS3ObjectStorage( + ScriptedGetObjectClient *& out_client, + DB::S3ObjectStorage::S3CredentialsRefreshCallback credentials_refresh_callback = {}) +{ + auto owned_client = std::make_unique(); + out_client = owned_client.get(); + + DB::S3::URI uri; + uri.bucket = "cas-upstream-slice-bucket"; + DB::S3Capabilities capabilities; + DB::ObjectStorageKeyGeneratorPtr key_generator; + + return std::make_shared( + std::move(owned_client), + std::make_unique(), + std::move(uri), + capabilities, + key_generator, + "cas-upstream-slice-disk", + /*for_disk_s3_=*/true, + credentials_refresh_callback); +} + +template +void expectThrowsCodeSaying(int expected_code, const std::string & needle, F && fn) +{ + try + { + fn(); + FAIL() << "expected DB::Exception saying '" << needle << "'"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), expected_code); + EXPECT_NE(e.message().find(needle), std::string::npos) << "actual message: " << e.message(); + } +} + +/// Builds the storage over a refresh callback that vends one fresh scripted client, so a test can +/// both script that client up front and assert the storage ended up holding that exact object. +std::shared_ptr makeScriptedS3ObjectStorageWithRefresh( + ScriptedGetObjectClient *& out_client, + ScriptedGetObjectClient *& out_refreshed, + std::function script_refreshed) +{ + return makeScriptedS3ObjectStorage( + out_client, + [&out_refreshed, script_refreshed]() -> std::unique_ptr + { + auto fresh = std::make_unique(); + script_refreshed(*fresh); + out_refreshed = fresh.get(); + return fresh; + }); +} + +} + +/// A plain `GET` must carry the same native-conditional mark a `HEAD` does when the read asks for it, +/// so that on a generation-token store both answer with the same incarnation identity. +TEST(CASUpstreamSlice, NativeConditionalReadSettingMarksTheGetRequest) +{ + (void)getContext(); + + ScriptedGetObjectClient * marked_client = nullptr; + auto marked_storage = makeScriptedS3ObjectStorage(marked_client); + marked_client->script({okStep("\"e1\"", "AAAA")}); + + DB::ReadSettings marked_settings; + marked_settings.object_storage_request_mode = DB::ObjectStorageRequestMode::NativeConditional; + marked_storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), marked_settings, 1 << 20); + + ASSERT_EQ(marked_client->nativeConditionalMarks().size(), 1u); + EXPECT_TRUE(marked_client->nativeConditionalMarks().at(0)); + + ScriptedGetObjectClient * plain_client = nullptr; + auto plain_storage = makeScriptedS3ObjectStorage(plain_client); + plain_client->script({okStep("\"e1\"", "AAAA")}); + + plain_storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), DB::ReadSettings{}, 1 << 20); + + ASSERT_EQ(plain_client->nativeConditionalMarks().size(), 1u); + EXPECT_FALSE(plain_client->nativeConditionalMarks().at(0)); +} + +/// The buffer's own retry loop can straddle a replacement of the object: the first response is the old +/// incarnation, the reissue the new one. The bytes handed back are then from neither one alone. +TEST(CASUpstreamSlice, ReadSmallObjectThrowsWhenAReissueAnswersWithADifferentETag) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + client->script({okStep("\"e1\"", "AAAA", /*fail_mid_body=*/true), okStep("\"e2\"", "BBBB")}); + + DB::ReadSettings read_settings; + read_settings.object_storage_request_mode = DB::ObjectStorageRequestMode::NativeConditional; + /// Default profile here: the buffer's own multi-attempt loop is what straddles the replacement. + expectThrowsCodeSaying( + DB::ErrorCodes::CANNOT_READ_ALL_DATA, + "response identity changed", + [&] { storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), read_settings, 1 << 20); }); + + EXPECT_EQ(client->getObjectCalls(), 2u); +} + +/// Scoped to an identity CHANGE: a reissue is ordinary, and refusing every retried read would turn a +/// dropped connection into a hard error. +TEST(CASUpstreamSlice, ReadSmallObjectAcceptsAReissueThatAnswersWithTheSameETag) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + client->script({okStep("\"e1\"", "AAAA", /*fail_mid_body=*/true), okStep("\"e1\"", "AAAA")}); + + const auto result = storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), DB::ReadSettings{}, 1 << 20); + EXPECT_EQ(result.data, "AAAA"); + EXPECT_EQ(client->getObjectCalls(), 2u); +} + +/// Under `SingleAttempt` the read must not retry at all: the caller owns the retry decision and its +/// own deadline. A throttle answer is retryable, so an unpinned buffer would reissue it. +TEST(CASUpstreamSlice, SingleAttemptProfileIssuesExactlyOneGetOnThrottle) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + client->script({throttleStep(), okStep("\"e1\"", "AAAA")}); + + DB::ReadSettings read_settings; + read_settings.object_storage_retry_profile = DB::ObjectStorageRetryProfile::SingleAttempt; + EXPECT_ANY_THROW(storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), read_settings, 1 << 20)); + + EXPECT_EQ(client->getObjectCalls(), 1u); +} + +TEST(CASUpstreamSlice, SingleAttemptClientCarriesTheRequestedTimeout) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + + const auto base_timeout = storage->getS3StorageClient()->getClientConfiguration().requestTimeoutMs; + + auto kept = storage->getSingleAttemptClient(0); + EXPECT_EQ(kept->getClientConfiguration().requestTimeoutMs, base_timeout); + + auto bounded = storage->getSingleAttemptClient(1234); + EXPECT_EQ(bounded->getClientConfiguration().requestTimeoutMs, 1234); + EXPECT_EQ(bounded->getClientConfiguration().retry_strategy.max_retries, 0u); + /// The cached clone is keyed by the timeout too, so one built for another bound is never served. + EXPECT_NE(bounded.get(), kept.get()); +} + +/// The buffer installs a refreshed client in itself only. A single-attempt read never retries, so that +/// copy is never used; what makes the caller's next attempt sign with the new credentials is the disk +/// client having been replaced. +TEST(CASUpstreamSlice, ExpiredTokenOnSingleAttemptReadInstallsTheRefreshedClientIntoTheStorage) +{ + (void)getContext(); + + ScriptedGetObjectClient * expired_client = nullptr; + const DB::S3::Client * refreshed_client = nullptr; + auto storage = makeScriptedS3ObjectStorage( + expired_client, + [&]() -> std::unique_ptr + { + auto fresh = std::make_unique(); + refreshed_client = fresh.get(); + return fresh; + }); + expired_client->script({expiredTokenStep()}); + + const auto * client_before = storage->getS3StorageClient().get(); + + DB::ReadSettings read_settings; + read_settings.object_storage_retry_profile = DB::ObjectStorageRetryProfile::SingleAttempt; + EXPECT_ANY_THROW(storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), read_settings, 1 << 20)); + + ASSERT_NE(refreshed_client, nullptr); + EXPECT_NE(storage->getS3StorageClient().get(), client_before); + EXPECT_EQ(storage->getS3StorageClient().get(), refreshed_client); +} + +/// `refreshAndRetryOnExpiredCredentials` on the HEAD path: the vended credentials expire, the callback +/// hands over a fresh client, and the request is reissued through it. Installing that client into the +/// storage is what stops the next request repeating the failure. +TEST(CASUpstreamSlice, NativeTokenHeadRecoversFromAnExpiredTokenAndInstallsTheRefreshedClient) +{ + (void)getContext(); + + ScriptedGetObjectClient * expired = nullptr; + ScriptedGetObjectClient * refreshed = nullptr; + auto storage = makeScriptedS3ObjectStorageWithRefresh( + expired, refreshed, [](const ScriptedGetObjectClient & fresh) { fresh.scriptHead({controlOk()}); }); + /// Installing the refreshed client drops the storage's last reference to this one, so the raw + /// pointer would dangle before the assertions below read its counters. + const auto expired_owner = storage->getS3StorageClient(); + expired->scriptHead({controlExpiredToken()}); + + const auto metadata = storage->tryGetObjectMetadataWithNativeToken( + "k", /*with_tags=*/false, DB::ObjectStorageRetryProfile::Default, /*request_timeout_ms=*/0); + + ASSERT_TRUE(metadata.has_value()); + ASSERT_NE(refreshed, nullptr); + EXPECT_EQ(storage->getS3StorageClient().get(), refreshed); + EXPECT_EQ(expired->headRequestTimeouts().size(), 1u); + EXPECT_EQ(refreshed->headRequestTimeouts().size(), 1u); +} + +/// The same for the conditional DELETE, which is the other verb that issues inline. +TEST(CASUpstreamSlice, ConditionalRemoveRecoversFromAnExpiredTokenAndInstallsTheRefreshedClient) +{ + (void)getContext(); + + ScriptedGetObjectClient * expired = nullptr; + ScriptedGetObjectClient * refreshed = nullptr; + auto storage = makeScriptedS3ObjectStorageWithRefresh( + expired, refreshed, [](const ScriptedGetObjectClient & fresh) { fresh.scriptDelete({controlOk()}); }); + /// See the HEAD test: the storage's last reference to this client goes away when the refreshed + /// one is installed. + const auto expired_owner = storage->getS3StorageClient(); + expired->scriptDelete({controlExpiredToken()}); + + const auto result = storage->removeObjectIfTokenMatches( + DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::Default, /*request_timeout_ms=*/0); + + EXPECT_EQ(result.outcome, DB::ConditionalRemoveOutcome::Removed); + ASSERT_NE(refreshed, nullptr); + EXPECT_EQ(storage->getS3StorageClient().get(), refreshed); + EXPECT_EQ(expired->deleteRequestTimeouts().size(), 1u); + EXPECT_EQ(refreshed->deleteRequestTimeouts().size(), 1u); +} + +/// The client the conditional DELETE selects is what decides whether the SDK reissues a throttled +/// request. The `Default` half is what makes "exactly one" mean something: the disk client here does +/// retry a throttle, so a single attempt is a property of the clone, not of the fake. +/// +/// Only the DELETE is counted. `S3::Client::HeadObject` does not use the SDK attempt loop at all — it +/// calls the virtual once and returns — so a HEAD is one request under either profile, and what the +/// profile changes for it is the transport bound, which the timeout test below pins. +TEST(CASUpstreamSlice, SingleAttemptConditionalRemoveIssuesExactlyOneRequestOnThrottle) +{ + (void)getContext(); + + ScriptedGetObjectClient * retrying = nullptr; + auto retrying_storage = makeScriptedS3ObjectStorage(retrying); + retrying->scriptDelete({controlThrottle()}); + + EXPECT_ANY_THROW(retrying_storage->removeObjectIfTokenMatches( + DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::Default, 0)); + EXPECT_EQ(retrying->deleteRequestTimeouts().size(), 3u); /// max_retries = 2, so three attempts + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + client->scriptDelete({controlThrottle()}); + + EXPECT_ANY_THROW(storage->removeObjectIfTokenMatches( + DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::SingleAttempt, 0)); + EXPECT_EQ(client->deleteRequestTimeouts().size(), 1u); +} + +/// The reservation the caller budgets for an attempt is only real if the transport is built to it, so +/// the two verbs must ride a clone carrying the timeout they asked for — and two different bounds must +/// coexist, or every alternation between verbs would rebuild a whole S3 client. +TEST(CASUpstreamSlice, HeadAndRemoveUnderSingleAttemptRideTheClientBoundToTheRequestedTimeout) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + + storage->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::SingleAttempt, 4321); + ASSERT_EQ(client->headRequestTimeouts().size(), 1u); + EXPECT_EQ(client->headRequestTimeouts().at(0), 4321); + + storage->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::SingleAttempt, 8765); + ASSERT_EQ(client->deleteRequestTimeouts().size(), 1u); + EXPECT_EQ(client->deleteRequestTimeouts().at(0), 8765); + + EXPECT_EQ(client->cloneRequestTimeouts(), (std::vector{4321, 8765})); + + /// Asking again for a bound already built must reuse that clone rather than evict the other one. + storage->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::SingleAttempt, 4321); + EXPECT_EQ(client->cloneRequestTimeouts(), (std::vector{4321, 8765})); + EXPECT_EQ(client->headRequestTimeouts().at(1), 4321); +} + +/// `iterate` issues nothing itself, so its client selection is observed through the clone it causes. +/// The async iterator fetches its first batch lazily, so constructing one sends no request. +TEST(CASUpstreamSlice, IterateUnderSingleAttemptSelectsTheClientBoundToTheRequestedTimeout) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + + (void)storage->iterate("p", 1, false, {}, DB::ObjectStorageRetryProfile::Default, 0); + EXPECT_TRUE(client->cloneRequestTimeouts().empty()); + + (void)storage->iterate("p", 1, false, {}, DB::ObjectStorageRetryProfile::SingleAttempt, 4321); + EXPECT_EQ(client->cloneRequestTimeouts(), (std::vector{4321})); +} + +#endif diff --git a/src/IO/ObjectStorageRequestMode.h b/src/IO/ObjectStorageRequestMode.h new file mode 100644 index 000000000000..39301ae953a9 --- /dev/null +++ b/src/IO/ObjectStorageRequestMode.h @@ -0,0 +1,18 @@ +#pragma once + +#include + +namespace DB +{ + +/// How a request is issued to an object storage. `NativeConditional` marks a request as carrying (or +/// eligible to carry) a storage-native conditional header, which lets a provider-specific client +/// translate `If-Match`/`ETag` into its own vocabulary (e.g. a GCS generation) on the request and the +/// response. Reads use it so that a plain GET answers with the same incarnation identity a HEAD does. +enum class ObjectStorageRequestMode : uint8_t +{ + Default, + NativeConditional, +}; + +} diff --git a/src/IO/ReadBufferFromS3.cpp b/src/IO/ReadBufferFromS3.cpp index a0184a1a067e..cf92f9c6f2a2 100644 --- a/src/IO/ReadBufferFromS3.cpp +++ b/src/IO/ReadBufferFromS3.cpp @@ -545,6 +545,14 @@ std::unique_ptr ReadBufferFromS3::initialize( Stopwatch watch{CLOCK_MONOTONIC}; auto read_result = sendRequest(attempt, offset, right_offset); + /// Compared per reissue rather than only against the first response, so an A -> B -> A' sequence + /// is caught at B: a later response equal to the first is not evidence that nothing changed. + const String etag = read_result.GetETag(); + if (!first_response_etag) + first_response_etag = etag; + else if (*first_response_etag != etag) + response_identity_changed = true; + size_t buffer_size = use_external_buffer ? 0 : read_settings.remote_fs_settings.buffer_size; return std::make_unique(std::move(read_result), buffer_size, std::move(watch)); } @@ -559,6 +567,9 @@ Aws::S3::Model::GetObjectResult ReadBufferFromS3::sendRequest(size_t attempt, si S3::setClickhouseAttemptNumber(req, attempt); + if (read_settings.object_storage_request_mode == ObjectStorageRequestMode::NativeConditional) + req.setNativeConditional(); + if (range_end_incl) { req.SetRange(fmt::format("bytes={}-{}", range_begin, *range_end_incl)); diff --git a/src/IO/ReadBufferFromS3.h b/src/IO/ReadBufferFromS3.h index 4c084d408916..bc37ce7e3606 100644 --- a/src/IO/ReadBufferFromS3.h +++ b/src/IO/ReadBufferFromS3.h @@ -91,6 +91,10 @@ class ReadBufferFromS3 : public ReadBufferFromFileBase /// This method returns metadata from the last request. If there were no requests, it will throw exception. ObjectMetadata getObjectMetadataFromTheLastRequest() const; + /// True when a reissued GET answered with a different ETag than the first one did, i.e. the bytes + /// this buffer produced may come from more than one incarnation of the object. + bool responseIdentityChanged() const { return response_identity_changed; } + size_t getReadUntilPosition() const { return read_until_position; } std::string getStopReason() const { return stop_reason; } @@ -111,6 +115,9 @@ class ReadBufferFromS3 : public ReadBufferFromFileBase Aws::S3::Model::GetObjectResult sendRequest(size_t attempt, size_t range_begin, std::optional range_end_incl) const; + std::optional first_response_etag; + bool response_identity_changed = false; + ReadSettings read_settings; bool use_external_buffer; diff --git a/src/IO/ReadSettings.h b/src/IO/ReadSettings.h index d74844481674..00f3b649a95d 100644 --- a/src/IO/ReadSettings.h +++ b/src/IO/ReadSettings.h @@ -3,7 +3,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -159,6 +161,14 @@ struct ReadSettings bool read_through_distributed_cache = false; DistributedCacheSettings distributed_cache_settings; + /// Selects the object storage request mode this read should carry; see ObjectStorageRequestMode. + ObjectStorageRequestMode object_storage_request_mode = ObjectStorageRequestMode::Default; + + /// Selects the retry profile the object storage should execute this read under, and the request + /// timeout of the client it picks for it; see ObjectStorageRetryProfile. 0 = the storage's own. + ObjectStorageRetryProfile object_storage_retry_profile = ObjectStorageRetryProfile::Default; + uint64_t object_storage_attempt_timeout_ms = 0; + ReadSettings adjustBufferSize(size_t file_size) const; /// Verification/metadata-read mode: disable every read-side cache (and the diff --git a/src/IO/WriteSettings.h b/src/IO/WriteSettings.h index 514d97fa9a99..b68da174ab44 100644 --- a/src/IO/WriteSettings.h +++ b/src/IO/WriteSettings.h @@ -3,6 +3,7 @@ #include #include #include +#include #include @@ -28,16 +29,6 @@ enum class ObjectStorageCopyMode : uint8_t NativeOnly, }; -/// Per-request GCS conditional-dialect opt-in, carried alongside the write itself so it survives -/// into the object storage request that ends up on the wire (see `RequestWithNativeConditionalMode`). -/// NativeConditional: this write is content-addressed-storage-owned and may use GCS generation -/// tokens instead of the AWS-style ETag plumbing, when the client's HTTP layer supports it. -enum class ObjectStorageRequestMode : uint8_t -{ - Default, - NativeConditional, -}; - /// Settings to be passed to IDisk::writeFile() struct WriteSettings { @@ -90,6 +81,10 @@ struct WriteSettings /// ObjectStorageRetryProfile. ObjectStorageRetryProfile object_storage_retry_profile = ObjectStorageRetryProfile::Default; + /// Request timeout (send/receive inactivity bound) for the single-attempt client selected by + /// `object_storage_retry_profile == SingleAttempt`. 0 = the storage's configured timeout. + uint64_t object_storage_attempt_timeout_ms = 0; + /// Selects the transport requirement for an object storage copy; see `ObjectStorageCopyMode`. ObjectStorageCopyMode object_storage_copy_mode = ObjectStorageCopyMode::Default; From 37c9bd4356b9ac6d2bde522cc07168085f5d1592 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:03:07 +0200 Subject: [PATCH 15/81] cas: migrate every CAS subsystem onto CasRequests/CasOperation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows the previous commit's engine introduction by moving every production caller off the old ad-hoc backend controller and onto `CasOperation`: pool bootstrap (sentinel probe, capability probe, plain objects, pool meta, manifest and ref-protocol readers), GC (maintenance state, namespace janitor, decommission, the GC core's lease/heartbeat/commits/ folds/persisted redelete), part-write (blob meta, the part-write transaction, create-first marker reconciliation), the ref lane (catalog and checkpoint publisher, namespace creation lifecycle, the resumed-operations arms, the catalog erase loop), and mount (renew, farewell, claim, epoch allocation, the heartbeat floor, remount re-anchoring). `PersistedIncarnation` replaces the ad-hoc token in the wire vocabulary, the record stream, the outcomes and the condemned rows. Each subsystem's move keeps its behavior but inherits the engine's guarantees for free: every write is admitted under a fence and re-checked before each attempt/sleep/commit, every conflict is settled by one exact read instead of an assumed outcome, and a credential refresh mid-attempt is never mistaken for a landed write. Along the way this fixes real bugs the engine surfaces mechanically rather than by inspection — e.g. two double-counting fault-injection doubles in the GC maintenance-state path, and several sites that treated an unobserved conflict as corruption instead of "vanished or a competing leader also wrote". The bulk of the diff is the matching migration of every test double (the `cp4` series) off the legacy backend overrides and onto the primitives the production code now actually calls — direct-Backend doubles for the primitives, virtualized clocks for every retry/backoff path that used to sleep for real, and fault injection that latches instead of pinning `max_attempts`, so a shut gate can no longer hang the test binary. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../Backend/CasInMemoryBackend.cpp | 78 +- .../Backend/CasInMemoryBackend.h | 79 +- .../ContentAddressed/Backend/CasProbe.cpp | 302 ++-- .../ContentAddressed/Backend/CasProbe.h | 33 +- .../Backend/CasRequestBudget.cpp | 69 + .../Backend/CasRequestBudget.h | 96 ++ .../Backend/CasRequestControl.cpp | 52 - .../Backend/CasRequestControl.h | 84 +- .../ContentAddressed/Backend/CasRequests.cpp | 100 +- .../ContentAddressed/Backend/CasRequests.h | 36 +- .../Backend/CasSentinelProbe.cpp | 43 +- .../Backend/CasSentinelProbe.h | 20 +- .../ContentAddressed/Backend/CasWriteResult.h | 15 +- .../ContentAddressedMetadataStorage.cpp | 8 +- .../ContentAddressedTransaction.cpp | 15 +- .../Formats/CasGcOutcomesFormat.h | 7 +- .../Formats/CasPoolMetaFormat.h | 8 +- .../Formats/CasRecordStreamFormat.h | 8 +- .../ContentAddressed/Formats/CasWireVocab.cpp | 29 +- .../ContentAddressed/Formats/CasWireVocab.h | 31 +- .../ContentAddressed/Gc/CasBlobInDegree.cpp | 74 +- .../ContentAddressed/Gc/CasBlobInDegree.h | 72 +- .../ContentAddressed/Gc/CasGc.cpp | 567 ++++---- .../ContentAddressed/Gc/CasGc.h | 44 +- .../Gc/CasGcMaintenanceState.cpp | 22 +- .../Gc/CasGcMaintenanceState.h | 21 +- .../ContentAddressed/Gc/CasGcMetaWriter.cpp | 62 +- .../ContentAddressed/Gc/CasGcMetaWriter.h | 26 +- .../ContentAddressed/Gc/CasGcShardPlan.cpp | 8 +- .../ContentAddressed/Gc/CasGcShardPlan.h | 15 +- .../Gc/CasNamespaceJanitor.cpp | 66 +- .../ContentAddressed/Gc/CasNamespaceJanitor.h | 18 +- .../Gc/CasOrphanManifestSweep.cpp | 138 +- .../Gc/CasOrphanManifestSweep.h | 13 +- .../Gc/CatalogLifecycleReconciler.cpp | 23 +- .../Gc/CatalogLifecycleReconciler.h | 14 +- .../ContentAddressed/Pool/CasBlobMeta.cpp | 26 +- .../ContentAddressed/Pool/CasBlobMeta.h | 75 +- .../Pool/CasManifestReader.cpp | 7 +- .../ContentAddressed/Pool/CasManifestReader.h | 15 +- .../ContentAddressed/Pool/CasMountRuntime.cpp | 124 +- .../ContentAddressed/Pool/CasMountRuntime.h | 61 +- .../ContentAddressed/Pool/CasPartWriteTxn.cpp | 246 ++-- .../ContentAddressed/Pool/CasPartWriteTxn.h | 11 +- .../ContentAddressed/Pool/CasPlainObjects.cpp | 112 +- .../ContentAddressed/Pool/CasPlainObjects.h | 67 +- .../ContentAddressed/Pool/CasPool.cpp | 254 +++- .../ContentAddressed/Pool/CasPool.h | 77 +- .../ContentAddressed/Pool/CasPoolMeta.cpp | 103 +- .../ContentAddressed/Pool/CasRefCatalog.cpp | 405 +++--- .../ContentAddressed/Pool/CasRefCatalog.h | 187 ++- .../ContentAddressed/Pool/CasRefCkpt.cpp | 206 +-- .../ContentAddressed/Pool/CasRefCkpt.h | 103 +- .../ContentAddressed/Pool/CasRefLedger.cpp | 1284 ++++++++-------- .../ContentAddressed/Pool/CasRefLedger.h | 111 +- .../ContentAddressed/Pool/CasRefProtocol.cpp | 25 +- .../ContentAddressed/Pool/CasRefProtocol.h | 33 +- .../ContentAddressed/Pool/CasServerRoot.cpp | 1286 ++++++++--------- .../ContentAddressed/Pool/CasServerRoot.h | 177 ++- .../Tools/CasDecommission.cpp | 196 +-- .../ContentAddressed/Tools/CasFsck.cpp | 110 +- .../ContentAddressed/Tools/CasInspect.cpp | 13 +- .../ContentAddressed/Tools/CasInspect.h | 4 +- .../benchmarks/benchmark_cas_ref_protocol.cpp | 2 +- src/Disks/tests/cas_sweep_test_support.h | 11 +- src/Disks/tests/cas_test_helpers.h | 1115 ++++++++------ src/Disks/tests/gtest_ca_wiring.cpp | 36 +- src/Disks/tests/gtest_cas_backend.cpp | 113 +- .../tests/gtest_cas_backend_contract.cpp | 19 +- src/Disks/tests/gtest_cas_blob_digest.cpp | 14 +- src/Disks/tests/gtest_cas_blob_indegree.cpp | 352 +++-- src/Disks/tests/gtest_cas_blob_meta.cpp | 110 +- .../tests/gtest_cas_bootstrap_ordering.cpp | 59 +- .../tests/gtest_cas_confirm_exact_ref.cpp | 134 +- src/Disks/tests/gtest_cas_decommission.cpp | 272 ++-- .../gtest_cas_decommission_catalog_duties.cpp | 82 +- src/Disks/tests/gtest_cas_detached_work.cpp | 176 ++- src/Disks/tests/gtest_cas_encoding_pins.cpp | 6 +- src/Disks/tests/gtest_cas_event_log.cpp | 260 ++-- .../tests/gtest_cas_fence_generation.cpp | 65 +- src/Disks/tests/gtest_cas_forget.cpp | 39 +- src/Disks/tests/gtest_cas_fsck.cpp | 114 +- src/Disks/tests/gtest_cas_gc_ack_floor.cpp | 266 ++-- .../tests/gtest_cas_gc_arithmetic_intake.cpp | 5 +- src/Disks/tests/gtest_cas_gc_attempt.cpp | 34 +- src/Disks/tests/gtest_cas_gc_bounded_walk.cpp | 6 +- src/Disks/tests/gtest_cas_gc_fold.cpp | 4 +- .../tests/gtest_cas_gc_frontier_gate.cpp | 592 +++++--- src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 24 +- src/Disks/tests/gtest_cas_gc_log.cpp | 93 +- .../gtest_cas_gc_maintenance_state_format.cpp | 165 ++- src/Disks/tests/gtest_cas_gc_meta_writer.cpp | 22 +- .../tests/gtest_cas_gc_outcomes_format.cpp | 14 +- src/Disks/tests/gtest_cas_gc_rebuild.cpp | 24 +- src/Disks/tests/gtest_cas_gc_resume.cpp | 36 +- src/Disks/tests/gtest_cas_gc_round.cpp | 315 +++- src/Disks/tests/gtest_cas_gc_round_defer.cpp | 10 +- .../tests/gtest_cas_gc_shard_incarnation.cpp | 45 +- src/Disks/tests/gtest_cas_gc_shard_plan.cpp | 16 +- .../tests/gtest_cas_gc_undercount_repro.cpp | 37 +- src/Disks/tests/gtest_cas_heartbeat.cpp | 465 +++--- .../tests/gtest_cas_holey_list_detector.cpp | 22 +- src/Disks/tests/gtest_cas_inspect.cpp | 10 +- .../tests/gtest_cas_lifecycle_condition.cpp | 42 +- src/Disks/tests/gtest_cas_manifest_reader.cpp | 49 + src/Disks/tests/gtest_cas_mount.cpp | 1199 ++++++++------- .../tests/gtest_cas_mount_claim_conflicts.cpp | 105 +- src/Disks/tests/gtest_cas_mount_runtime.cpp | 156 ++ ...est_cas_namespace_file_request_profile.cpp | 14 +- .../tests/gtest_cas_namespace_janitor.cpp | 534 ++++--- .../tests/gtest_cas_ns_creation_lifecycle.cpp | 306 ++-- .../tests/gtest_cas_ns_file_incarnation.cpp | 23 +- .../tests/gtest_cas_ns_file_read_contract.cpp | 19 +- src/Disks/tests/gtest_cas_observability.cpp | 32 +- .../tests/gtest_cas_orphan_manifest_sweep.cpp | 59 +- .../tests/gtest_cas_orphan_nomination.cpp | 51 +- .../tests/gtest_cas_part_folder_access.cpp | 102 +- src/Disks/tests/gtest_cas_part_write.cpp | 475 ++++-- src/Disks/tests/gtest_cas_plain_objects.cpp | 88 ++ src/Disks/tests/gtest_cas_pluggable_hash.cpp | 39 +- src/Disks/tests/gtest_cas_pool.cpp | 549 +++++-- src/Disks/tests/gtest_cas_pool_meta.cpp | 76 + src/Disks/tests/gtest_cas_probe.cpp | 358 ++--- .../tests/gtest_cas_protocol_scenarios.cpp | 76 +- .../gtest_cas_rebuild_condemn_nothing.cpp | 15 +- .../tests/gtest_cas_record_stream_format.cpp | 11 +- .../tests/gtest_cas_recovery_grounding.cpp | 107 +- .../tests/gtest_cas_recovery_streaming.cpp | 41 +- src/Disks/tests/gtest_cas_ref_catalog.cpp | 869 +++++++---- .../gtest_cas_ref_catalog_birth_wiring.cpp | 282 ++-- .../tests/gtest_cas_ref_chunked_flush.cpp | 125 +- src/Disks/tests/gtest_cas_ref_ckpt.cpp | 754 ++++++---- src/Disks/tests/gtest_cas_ref_ckpt_join.cpp | 165 ++- .../tests/gtest_cas_ref_contiguous_alloc.cpp | 41 +- src/Disks/tests/gtest_cas_ref_gc.cpp | 48 +- .../tests/gtest_cas_ref_install_safety.cpp | 379 +++-- src/Disks/tests/gtest_cas_ref_protocol.cpp | 51 + .../tests/gtest_cas_ref_read_contract.cpp | 37 +- .../tests/gtest_cas_ref_recovery_cas_walk.cpp | 443 ++++-- ...test_cas_ref_snapshot_publish_ordering.cpp | 136 +- .../gtest_cas_ref_wedge_every_attempt.cpp | 780 ++++++---- src/Disks/tests/gtest_cas_ref_writer.cpp | 862 ++++++----- src/Disks/tests/gtest_cas_requests.cpp | 599 +++++--- .../tests/gtest_cas_retirement_sweep.cpp | 46 +- src/Disks/tests/gtest_cas_s3_staging.cpp | 46 +- src/Disks/tests/gtest_cas_sentinel_probe.cpp | 71 +- src/Disks/tests/gtest_cas_slot_occupy.cpp | 446 +++--- .../tests/gtest_cas_truncate_reclaim.cpp | 4 +- .../tests/gtest_cas_txn_apply_ledger.cpp | 7 +- src/Disks/tests/gtest_cas_upload_detached.cpp | 14 +- src/Disks/tests/gtest_cas_upload_fanout.cpp | 4 +- src/Disks/tests/gtest_cas_wire_vocab.cpp | 100 +- src/Disks/tests/gtest_cas_writer_duties.cpp | 81 +- .../StorageSystemContentAddressedMounts.cpp | 6 +- 154 files changed, 14128 insertions(+), 9592 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h create mode 100644 src/Disks/tests/gtest_cas_manifest_reader.cpp create mode 100644 src/Disks/tests/gtest_cas_mount_runtime.cpp create mode 100644 src/Disks/tests/gtest_cas_plain_objects.cpp create mode 100644 src/Disks/tests/gtest_cas_pool_meta.cpp create mode 100644 src/Disks/tests/gtest_cas_ref_protocol.cpp diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp index 2ab9adac812d..f7f51b5b45d8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp @@ -3,8 +3,8 @@ #include #include #include +#include #include -#include namespace DB { @@ -138,31 +138,11 @@ std::optional InMemoryBackend::head(const String & key, Transp std::expected InMemoryBackend::write( const String & key, const String & bytes, const std::optional & expected_value, TransportAccess &) { - return applyWrite(key, bytes, expected_value, WriteKnobs::All); -} - -PutResult InMemoryBackend::putIfAbsent(const String & key, const String & bytes, const ObjectMeta &) -{ - auto r = applyWrite(key, bytes, std::nullopt, WriteKnobs::AmbiguousPutIfAbsent); - if (!r) - return PutResult{PutOutcome::PreconditionFailed, {}}; - return PutResult{PutOutcome::Done, legacyMintWritten(key, std::move(*r))}; -} - -CasResult InMemoryBackend::casPut(const String & key, const String & bytes, - const std::optional & expected, const ObjectMeta &) -{ - if (expected && legacyTokenIsForeign(key, *expected)) - return CasResult{CasOutcome::Conflict, {}}; - auto r = applyWrite(key, bytes, expected ? std::optional(expected->value) : std::nullopt, - WriteKnobs::FailNextCasPut); - if (!r) - return CasResult{CasOutcome::Conflict, {}}; - return CasResult{CasOutcome::Committed, legacyMintWritten(key, std::move(*r))}; + return applyWrite(key, bytes, expected_value); } std::expected InMemoryBackend::applyWrite( - const String & key, const String & bytes, const std::optional & expected_value, WriteKnobs knobs) + const String & key, const String & bytes, const std::optional & expected_value) { if (expected_value) checkExpectedValue(key, *expected_value); @@ -175,7 +155,7 @@ std::expected InMemoryBackend::applyWrite( if (auto hook = hookFor(before_write_hooks_, key)) hook(); - auto result = writeUnderLock(key, bytes, expected_value, knobs); + auto result = writeUnderLock(key, bytes, expected_value); if (!result.has_value()) return result; @@ -183,45 +163,33 @@ std::expected InMemoryBackend::applyWrite( hook(); /// Last, so the object is durable and every observer has run before the response goes missing. - if (knobs == WriteKnobs::All && takeAmbiguousLandedWrite(key)) - throw std::runtime_error("InMemoryBackend: the write of '" + key + "' landed and its response was lost"); + if (takeAmbiguousLandedWrite(key)) + throw Poco::TimeoutException("InMemoryBackend: the write of '" + key + "' landed and its response was lost"); return result; } std::expected InMemoryBackend::writeUnderLock( - const String & key, const String & bytes, const std::optional & expected_value, WriteKnobs knobs) + const String & key, const String & bytes, const std::optional & expected_value) { std::lock_guard lock(mutex_); - if (!expected_value && (knobs == WriteKnobs::All || knobs == WriteKnobs::AmbiguousPutIfAbsent)) + // One-shot injected ambiguous outcome: throw WITHOUT touching the store, modeling a request + // whose own attempt outcome never reached the caller. Poco::TimeoutException, not + // DB::Exception, is deliberate: a client-side timeout is the real shape of this failure, and + // its class is what every caller classifies by -- ambiguous, in both build configurations. + auto ambiguous_it = ambiguous_write_keys_.find(key); + if (ambiguous_it != ambiguous_write_keys_.end()) { - // One-shot injected ambiguous outcome: throw WITHOUT touching the store, modeling a request - // whose own attempt outcome never reached the caller (see the header doc for the - // classification this must produce). std::runtime_error, not DB::Exception, is deliberate: it - // dodges BOTH classification paths in BOTH build configurations -- dynamic_cast fails (so isDeterministicLocalFailure is never consulted), and - // classifyConditionalWriteResult falls through to its Unresolved default because it isn't an - // S3Exception. A DB::Exception would have been fragile: picking a code outside - // isDeterministicLocalFailure's set is a landmine for the next person who extends that set. - auto ambiguous_it = ambiguous_put_keys_.find(key); - if (ambiguous_it != ambiguous_put_keys_.end()) - { - ambiguous_put_keys_.erase(ambiguous_it); - throw std::runtime_error("InMemoryBackend: injected ambiguous write outcome for '" + key + "'"); - } + ambiguous_write_keys_.erase(ambiguous_it); + throw Poco::TimeoutException("InMemoryBackend: injected ambiguous write outcome for '" + key + "'"); } - // One-shot injected conflict, on EITHER form: the knob is armed against a conditional write, and - // a create-if-absent is one -- the lease acquire this models creates its object. - if (knobs == WriteKnobs::All || knobs == WriteKnobs::FailNextCasPut) + auto refuse_it = refuse_next_write_keys_.find(key); + if (refuse_it != refuse_next_write_keys_.end()) { - auto fail_it = fail_next_cas_.find(key); - if (fail_it != fail_next_cas_.end()) - { - fail_next_cas_.erase(fail_it); - return std::unexpected(RawConflict{}); - } + refuse_next_write_keys_.erase(refuse_it); + return std::unexpected(RawConflict{}); } if (!expected_value) @@ -475,16 +443,16 @@ DeleteOutcome InMemoryBackend::landPendingDelete(size_t i) return applyDelete(pd.key, pd.token); } -void InMemoryBackend::failNextCasPut(const String & key) +void InMemoryBackend::refuseNextWrite(const String & key) { std::lock_guard lock(mutex_); - fail_next_cas_.insert(key); + refuse_next_write_keys_.insert(key); } -void InMemoryBackend::injectAmbiguousPutIfAbsent(const String & key) +void InMemoryBackend::injectAmbiguousWrite(const String & key) { std::lock_guard lock(mutex_); - ambiguous_put_keys_.insert(key); + ambiguous_write_keys_.insert(key); } void InMemoryBackend::injectAmbiguousLandedWrite(const String & key) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h index 2f2dd6c281a1..e4e5bbe5bb03 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h @@ -17,7 +17,7 @@ namespace DB::Cas /// /// The backend also exposes fault-injection controls for probe tests and CAS correctness tests: /// - `setHoldDeletes` / `landPendingDelete`: simulate async/delayed conditional deletes -/// - `failNextCasPut`: inject a one-shot conflict +/// - `refuseNextWrite`: inject a one-shot conflict /// - `setEnforceTokens(false)`: mimic a "dumb" backend that ignores token checks /// - `setSimulateDeleteMarkers`: mimic S3 versioning-enabled buckets /// @@ -29,13 +29,10 @@ class InMemoryBackend : public Backend InMemoryBackend() = default; /// Unhide the base overloads this class's own declarations would otherwise shadow: the legacy - /// `head`/`list`/`getStream`/`putIfAbsent`/`casPut` names, and the omitted-`Range`/`ObjectMeta` - /// conveniences. - using Backend::casPut; + /// `head`/`list`/`getStream` names, and the omitted-`Range` convenience. using Backend::getStream; using Backend::head; using Backend::list; - using Backend::putIfAbsent; // ---- Backend interface ---- @@ -70,6 +67,13 @@ class InMemoryBackend : public Backend /// This backend mints its own emulated values. Dialect dialect() const override { return Dialect::Emulated; } + /// Zero unless a fixture sets one. The engine reserves this before every attempt it starts, so a + /// pool fixture that configures `CasRequestBudget::attempt_timeout_ms` must set the SAME value + /// here: production pairs the two (`ContentAddressedMetadataStorage` builds its backend from the + /// pool's budget), and a fixture that sets only the budget leaves the engine reserving nothing. + uint64_t attemptTimeoutMs() const override { return attempt_timeout_ms; } + void setAttemptTimeoutMs(uint64_t ms) { attempt_timeout_ms = ms; } + /// The in-memory backend mints a monotonic value it surfaces through `list` — TRUE. bool supportsListTokens() const override { return true; } @@ -82,18 +86,6 @@ class InMemoryBackend : public Backend /// backend lock, so the returned stream remains independent of later backend mutations. std::optional getStream(const String & key, Range range) override; - /// ---- Two legacy verbs, overridden ONLY so each write knob keeps its verb identity ---- - /// - /// A knob is armed against a VERB, but the keyed `write` cannot see which verb its caller used, so - /// the base forwarder would let `failNextCasPut` fire on a `putIfAbsent` and - /// `injectAmbiguousPutIfAbsent` on a create-shaped `casPut`. Each of these consumes only the knob - /// named for it, and neither reaches the keyed primitive -- so, unlike every other legacy verb, an - /// override of `write` in a SUBCLASS of this backend does not intercept these two. Deleted with the - /// rest of the legacy surface at the lock, and the exception goes with them. - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override; - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override; - // ---- Fault-injection controls ---- /// When true, `remove` validates and enqueues deletes rather than applying them immediately. @@ -109,26 +101,20 @@ class InMemoryBackend : public Backend /// `Removed`. An invalid index returns `NotFound`. DeleteOutcome landPendingDelete(size_t i); - /// Injects a one-shot artificial refusal on the next `casPut` of `key`, IN EITHER FORM: a GC lease - /// acquire creates its object, and a test arming this knob for it is testing exactly that create - /// losing its condition. - void failNextCasPut(const String & key); - - /// Injects a one-shot AMBIGUOUS outcome on the next CREATING write of `key` (a write with no - /// expected value): instead of attempting it, that call throws a plain (non-`DB::Exception`) - /// exception -- classified `Unresolved`, never `DefiniteFailure`, by - /// `classifyConditionalWriteResult` regardless of build flags -- and the store is left exactly as - /// it was. Models a request whose own HTTP attempt outcome is lost (a timeout, a dropped - /// connection) rather than a clean refusal, for tests that must exercise the "ambiguous attempt, - /// resolve before deciding" path without a live network. One-shot, mirroring `failNextCasPut`'s - /// contract: consumed by the first matching write, whether the key was already present or not. - void injectAmbiguousPutIfAbsent(const String & key); + /// Refuses the next write of `key` once, as a clean precondition failure that leaves the store + /// unchanged -- whatever the write's shape and whichever surface issued it. + void refuseNextWrite(const String & key); + + /// Injects a one-shot AMBIGUOUS outcome on the next write of `key`: instead of attempting it, that + /// call throws `Poco::TimeoutException` and the store is left exactly as it was. Models a request + /// whose own HTTP attempt outcome is lost (a timeout, a dropped connection) rather than a clean + /// refusal, for tests that must exercise the "ambiguous attempt, resolve before deciding" path + /// without a live network. + void injectAmbiguousWrite(const String & key); /// The other ambiguity, and the only one that can prove a resolve read settles a commit: the next - /// write of `key` IS APPLIED and then throws a plain (non-`DB::Exception`) exception, so the object - /// is durable and its incarnation was never returned. One-shot, and consumed by the keyed `write` - /// and by every legacy verb that forwards through it -- `putOverwrite` today. The two verbs that - /// route around the primitive, `putIfAbsent` and `casPut`, do not consume it. + /// write of `key` IS APPLIED and then throws `Poco::TimeoutException`, so the object is durable and + /// its incarnation was never returned. One-shot. void injectAmbiguousLandedWrite(const String & key); /// Enables or disables value checks for remove and replace. Disabling checks models a backend @@ -194,14 +180,6 @@ class InMemoryBackend : public Backend using ArmedFailures = std::map>; using Hooks = std::map>; - /// Which of the verb-scoped write knobs one call may consume. - enum class WriteKnobs : uint8_t - { - All, /// the keyed `write`, and the legacy verbs that forward through it - AmbiguousPutIfAbsent, /// legacy `putIfAbsent` - FailNextCasPut, /// legacy `casPut`, either form - }; - /// Consumes and returns the next failure armed for `key`, or null when none is. std::exception_ptr takeArmedFailure(ArmedFailures & armed, const String & key); /// Consumes the landed-then-lost arming for `key`, if there is one. @@ -209,23 +187,26 @@ class InMemoryBackend : public Backend /// A copy of the hook registered for `key`, taken under the lock so the caller can run it without /// one. std::function hookFor(const Hooks & hooks, const String & key) const; - /// One write, whichever verb asked for it: armed failure, hooks, the store mutation, and exactly - /// the knobs `knobs` allows. + /// One write, whichever verb asked for it: armed failure, hooks, the store mutation and the knobs. std::expected applyWrite(const String & key, const String & bytes, - const std::optional & expected_value, WriteKnobs knobs); + const std::optional & expected_value); /// The part of `applyWrite` that touches the store, run with `mutex_` held. std::expected writeUnderLock(const String & key, const String & bytes, - const std::optional & expected_value, WriteKnobs knobs); + const std::optional & expected_value); mutable std::mutex mutex_; std::map store_; uint64_t token_seq_ = 0; + /// Set before any operation runs, and read without the lock for the same reason the engine reads + /// it once at construction: it belongs to setup, not to a request. + uint64_t attempt_timeout_ms = 0; + // Fault-injection state. These fields are protected by `mutex_` just like `store_`. bool hold_deletes_ = false; std::vector pending_deletes_; - std::set fail_next_cas_; - std::set ambiguous_put_keys_; + std::set refuse_next_write_keys_; + std::set ambiguous_write_keys_; std::set ambiguous_landed_keys_; bool enforce_tokens_ = true; bool simulate_delete_markers_ = false; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp index bb653e90fcea..091b3ae39754 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp @@ -5,6 +5,7 @@ namespace DB { namespace ErrorCodes { + extern const int CAS_DELETE_MARKER; extern const int NOT_IMPLEMENTED; } } @@ -12,238 +13,187 @@ namespace ErrorCodes namespace DB::Cas { -void runCapabilityProbe(Backend & backend, const String & probe_prefix) +namespace +{ + +/// `op.remove`'s own delete-marker signal (`CAS_DELETE_MARKER`, thrown outside the attempt loop because +/// a versioned bucket answers this way every time) is re-signalled here as the operator-facing message +/// the probe has always thrown for it — everything else propagates unchanged. Every `remove` the battery +/// issues against a live incarnation goes through this: a store that ignores the delete precondition AND +/// mints delete markers must still be reported with this message, not the engine's terse one. +Removal removeOrReportDeleteMarker(CasOperation & op, const String & key, const Incarnation & seen) +{ + try + { + return op.remove(key, seen, Retry::standard()); + } + catch (const DB::Exception & e) + { + if (e.code() != DB::ErrorCodes::CAS_DELETE_MARKER) + throw; + throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, + "CasProbe: remove succeeded but created a versioning delete marker — the bucket has " + "object VERSIONING enabled, and a content-addressed pool cannot run on a versioned bucket: " + "every GC delete would archive a noncurrent version instead of reclaiming storage (the bucket " + "grows forever), and the constantly-rewritten ref objects would pile up versions on every " + "commit. This is NOT ignorable and has no override. Use a bucket where versioning was NEVER " + "enabled — note that merely SUSPENDING versioning is not enough (deletes on a " + "versioning-suspended bucket still mint delete markers, so this probe will refuse again)"); + } +} + +} + +void runCapabilityProbe(CasOperation & op, const String & probe_prefix) { - // Probe key used for the primary battery steps. // Sub-directory style ("probe_prefix/token") ensures that list(probe_prefix, …) works for both the // in-memory backend (prefix match) and the LocalObjectStorage backend (directory listing). const String key = probe_prefix + "/token"; - // Probe key used for the casPut chain. - const String cas_key = probe_prefix + "/cas"; // Best-effort cleanup — runs at function exit regardless of outcome. - // We capture the keys we need to clean up. auto cleanup = [&]() noexcept { - // Skip the delete when HEAD says the key is already gone (the happy path: step 8 deleted - // it). A deleteExact with the absent HeadResult's EMPTY token is a malformed conditional - // op — AWS S3 answers 400 InvalidArgument ("If-Match cannot be empty"), which lands as a - // scary AWSClient log line on every mount even though the catch swallows it. - for (const auto & k : {key, cas_key}) + // Skip the remove when HEAD says the key is already gone (the happy path: the battery's own + // delete already ran). `Incarnation` can only be minted from an actual HEAD/read observation, so + // an unconditional "delete with whatever precondition" this backend never saw is not + // constructible here — the gate below is the only way to reach `remove` at all. + try { - try - { - const auto h = backend.head(k); - if (h.exists) - backend.deleteExact(k, h.token); - } - catch (...) {} /// NOLINT(bugprone-empty-catch) + const auto h = op.head(key, Retry::standard()); + if (h) + op.remove(key, h->incarnation, Retry::standard()); } + catch (...) {} /// NOLINT(bugprone-empty-catch) }; try { - // ---- Step 0: store-level preconditions (backend-specific; throws = mount refused). ---- - backend.checkPoolPreconditions(); - - // ---- Step 0b: conditional writes must use one underlying HTTP attempt. Transparent SDK - // retries can outlive the writer's mount lease and hide whether a conditional operation - // committed; CAS retries must instead be explicit and state-aware. Throws = mount refused. - // Keep this separate from Step 0 so each precondition remains independently unit-testable. ---- - backend.checkConditionalWriteSingleAttemptSupport(); - - // ---- Step 1: putIfAbsent fresh → Done; read-after-write returns the bytes. ---- - Token t1; + // ---- Step 1: create fresh -> Committed; read-after-write returns the bytes. ---- + Incarnation t1 = [&] { - const auto res = backend.putIfAbsent(key, "probe-v1"); - t1 = res.token; - if (res.outcome != PutOutcome::Done) + WriteResult r = op.create(key, "probe-v1", Retry::standard()); + if (!std::holds_alternative(r)) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putIfAbsent on a fresh key returned PreconditionFailed — backend is unexpectedly occupied or broken"); - } + "CasProbe: create on a fresh key did not commit — backend is unexpectedly occupied or broken"); + return std::get(r).incarnation; + }(); { - const auto g = backend.get(key); - if (!g.has_value() || g->bytes != "probe-v1") + const auto g = op.read(key, Retry::standard()); + if (!g || g->bytes != "probe-v1") throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: read-after-write failed — putIfAbsent succeeded but the object is not readable"); + "CasProbe: read-after-write failed — create succeeded but the object is not readable"); } - // ---- Step 2: putIfAbsent same key → PreconditionFailed; bytes intact. ---- + // ---- Step 2: create the same key again -> Conflict; bytes intact. ---- { - const auto outcome = backend.putIfAbsent(key, "should-not-land").outcome; - if (outcome != PutOutcome::PreconditionFailed) + WriteResult r = op.create(key, "should-not-land", Retry::standard()); + const auto * conflict = std::get_if(&r); + if (!conflict) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putIfAbsent on an existing key was not rejected (PreconditionFailed expected) — " + "CasProbe: create on an existing key was not rejected (a conflict was expected) — " "backend does not enforce conditional create"); - const auto g = backend.get(key); - if (!g.has_value() || g->bytes != "probe-v1") + const auto * seen = std::get_if(&conflict->seen); + if (!seen || seen->bytes != "probe-v1") throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putIfAbsent conflict was 'reported' but the original bytes were clobbered — " + "CasProbe: create's conflict was reported but the original bytes were clobbered — " "backend does not enforce conditional create"); } - // ---- Step 3: putOverwrite wrong token → PreconditionFailed; bytes intact. ---- + // ---- Step 3: replace against the CURRENT incarnation (t1) -> Committed; incarnation changed; + // bytes replaced. Every "wrong incarnation" step below reuses THIS key's own prior + // incarnations (never a synthetic value) — an `Incarnation` is minted only from an + // actual backend observation, so there is no other way to name one that is + // guaranteed wrong yet dialect-valid. ---- + Incarnation t2 = [&] { - /// Wrong-token values are NUMERIC on purpose: a generation-dialect backend (GCS) - /// validates the If-Match FORMAT client-side and throws on a non-numeric value (an - /// ETag-kind token leaking into a generation dialect) — the probe's synthetic wrong - /// tokens must be format-valid for EVERY token kind, merely guaranteed-wrong. A huge - /// numeric is a wrong ETag on AWS (412), a wrong generation on GCS (412), and a wrong - /// sequence on the emulated backends (TokenMismatch). - /// - /// The TYPE must be the LIVE dialect (t1.type, just observed from this same backend), - /// never a hardcoded TokenType::Emulated: a backend that mints a different dialect - /// (e.g. Native/ETag) rejects a foreign-dialect token locally, before the wrong VALUE - /// ever reaches the wire — which would make this check pass vacuously against a - /// non-enforcing store instead of proving enforcement (codex-review-triage §3.18, - /// Critical: the №19 local dialect guard must not defeat this probe). - Token wrong_token{"900000000000000001", t1.type}; - const auto outcome = backend.putOverwrite(key, "clobbered", wrong_token).outcome; - if (outcome != PutOutcome::PreconditionFailed) + WriteResult r = op.replace(key, "probe-v2", t1, Retry::standard()); + if (!std::holds_alternative(r)) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putOverwrite with a wrong token was not rejected (PreconditionFailed expected) — " - "backend does not enforce conditional overwrite"); - const auto g = backend.get(key); - if (!g.has_value() || g->bytes != "probe-v1") + "CasProbe: replace with the correct incarnation was rejected — backend does not accept a valid overwrite"); + Incarnation next = std::get(r).incarnation; + if (next == t1) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putOverwrite with wrong token was 'rejected' but the original bytes were clobbered"); - } - - // ---- Step 4: putOverwrite correct token → Done; bytes replaced; token changed. ---- - Token t2; + "CasProbe: replace succeeded but did not mint a new incarnation — an incarnation must " + "change on every write"); + return next; + }(); { - const auto res = backend.putOverwrite(key, "probe-v2", t1); - t2 = res.token; - if (res.outcome != PutOutcome::Done) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putOverwrite with the correct token was rejected — backend does not accept valid overwrite"); - if (t2 == t1) + const auto g = op.read(key, Retry::standard()); + if (!g || g->bytes != "probe-v2") throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putOverwrite succeeded but did not mint a new token — tokens must change on every write"); - const auto g = backend.get(key); - if (!g.has_value() || g->bytes != "probe-v2") - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: putOverwrite succeeded but the new bytes are not readable"); + "CasProbe: replace succeeded but the new bytes are not readable"); } - // ---- Step 5: casPut chain. ---- - // 5a: create-if-absent (nullopt expected). - Token ct1; - { - const auto res = backend.casPut(cas_key, "cas-s1", std::nullopt); - ct1 = res.token; - if (res.outcome != CasOutcome::Committed) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: casPut create-if-absent (nullopt expected) was not committed — " - "backend does not support CAS create-if-absent"); - } - // 5b: conflict on existing (nullopt expected, but key exists). - { - const auto outcome = backend.casPut(cas_key, "cas-s1x", std::nullopt).outcome; - if (outcome != CasOutcome::Conflict) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: casPut with nullopt expected against an existing key was not Conflict — " - "backend does not enforce create-if-absent semantics on casPut"); - } - // 5c: conflict on stale token. - { - Token stale{"900000000000000002", ct1.type}; /// numeric + live dialect: see step 3 - const auto outcome = backend.casPut(cas_key, "cas-s1y", stale).outcome; - if (outcome != CasOutcome::Conflict) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: casPut with a stale token was not Conflict — " - "backend does not enforce token-exact CAS"); - } - // Bytes must still be the original. + // ---- Step 4: replace against t1, now STALE (the key committed to t2 in step 3) -> Conflict; + // bytes intact. ---- { - const auto g = backend.get(cas_key); - if (!g.has_value() || g->bytes != "cas-s1") + WriteResult r = op.replace(key, "clobbered", t1, Retry::standard()); + const auto * conflict = std::get_if(&r); + if (!conflict) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: casPut conflicts were reported but the original bytes were altered"); - } - // 5d: commit on current token. - { - const auto res = backend.casPut(cas_key, "cas-s2", ct1); - if (res.outcome != CasOutcome::Committed) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: casPut with the current token was not committed — " - "backend does not honor casPut with matching token"); - const auto g = backend.get(cas_key); - if (!g.has_value() || g->bytes != "cas-s2") + "CasProbe: replace with a stale incarnation was not rejected (a conflict was expected) — " + "backend does not enforce conditional overwrite"); + const auto * seen = std::get_if(&conflict->seen); + if (!seen || seen->bytes != "probe-v2") throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: casPut committed but new bytes are not readable"); + "CasProbe: replace with a stale incarnation was 'rejected' but the original bytes were clobbered"); } - // ---- Step 6: deleteExact wrong token → TokenMismatch AND the object still readable. ---- + // ---- Step 5: remove with a STALE incarnation (t1) -> Mismatch; the object survives. A store + // that ignores this precondition removes the object here, so this path can reach a + // delete marker exactly like step 7's — route it through the same reporter. ---- { - Token wrong_token{"900000000000000003", t2.type}; /// numeric + live dialect: see step 3 - const auto d = backend.deleteExact(key, wrong_token); - if (d.kind != DeleteOutcome::Kind::TokenMismatch) + const Removal d = removeOrReportDeleteMarker(op, key, t1); + if (d != Removal::Mismatch) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: deleteExact with a wrong token was not TokenMismatch — " - "delete with mismatching token was honored — backend does not enforce conditional deletes"); - const auto g = backend.get(key); - if (!g.has_value()) + "CasProbe: remove with a stale incarnation was not rejected (a mismatch was expected) — " + "backend does not enforce conditional deletes"); + const auto g = op.read(key, Retry::standard()); + if (!g) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: deleteExact with a wrong token was rejected (correctly) but the object was deleted anyway — " + "CasProbe: remove with a stale incarnation was rejected (correctly) but the object was deleted anyway — " "backend does not enforce conditional deletes"); } - // ---- Step 7: list(probe_prefix) contains the probe key (list-after-write). ---- + // ---- Step 6: list(probe_prefix) contains the probe key (list-after-write). ---- { - const auto page = backend.list(probe_prefix, "", 100); bool found = false; - for (const auto & listed : page.keys) + op.forEachListedKey(probe_prefix, [&](const KeyEntry & listed) -> bool { - if (listed.key == key) - { - found = true; - break; - } - } + if (listed.key != key) + return true; + found = true; + return false; + }, Retry::standard()); if (!found) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "CasProbe: list-after-write failed — the probe key '{}' is not visible in the listing under prefix '{}'", key, probe_prefix); } - // ---- Step 8: deleteExact correct token → Deleted; object gone; no delete marker; - // list no longer contains the key. ---- + // ---- Step 7: remove with the CORRECT incarnation (t2) -> Removed; no delete marker; object + // gone; list no longer contains the key. ---- { - const auto d = backend.deleteExact(key, t2); - if (d.kind != DeleteOutcome::Kind::Deleted) + const Removal d = removeOrReportDeleteMarker(op, key, t2); + if (d != Removal::Removed) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: deleteExact with the correct token was not Deleted — backend rejected a valid token-exact delete"); - if (d.created_delete_marker) + "CasProbe: remove with the correct incarnation was not Removed — backend rejected a valid incarnation-exact delete"); + const auto g = op.read(key, Retry::standard()); + if (g) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: deleteExact succeeded but created a versioning delete marker — the bucket has " - "object VERSIONING enabled, and a content-addressed pool cannot run on a versioned bucket: " - "every GC delete would archive a noncurrent version instead of reclaiming storage (the bucket " - "grows forever), and the constantly-rewritten ref objects would pile up versions on every " - "commit. This is NOT ignorable and has no override. Use a bucket where versioning was NEVER " - "enabled — note that merely SUSPENDING versioning is not enough (deletes on a " - "versioning-suspended bucket still mint delete markers, so this probe will refuse again)"); - const auto g = backend.get(key); - if (g.has_value()) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: deleteExact succeeded (Deleted) but the object is still readable — backend delete is not effective"); - // List-after-delete. - const auto page = backend.list(probe_prefix, "", 100); - for (const auto & listed : page.keys) + "CasProbe: remove succeeded (Removed) but the object is still readable — backend delete is not effective"); + bool still_listed = false; + op.forEachListedKey(probe_prefix, [&](const KeyEntry & listed) -> bool { - if (listed.key == key) - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "CasProbe: list-after-delete failed — the deleted probe key '{}' is still visible in the listing under prefix '{}'", - key, probe_prefix); - } - } - - // ---- Step 9: cleanup (best-effort; also deletes cas_key). ---- - // cas_key is still alive — clean it up via its current token. - { - const auto h = backend.head(cas_key); - if (h.exists) - backend.deleteExact(cas_key, h.token); + if (listed.key != key) + return true; + still_listed = true; + return false; + }, Retry::standard()); + if (still_listed) + throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, + "CasProbe: list-after-delete failed — the deleted probe key '{}' is still visible in the listing under prefix '{}'", + key, probe_prefix); } } catch (...) @@ -253,8 +203,8 @@ void runCapabilityProbe(Backend & backend, const String & probe_prefix) throw; } - // Normal-exit cleanup (cas_key was cleaned inside the try; key was deleted in step 8). - // Call cleanup anyway to handle any partial state edge cases — it is a no-op if keys are gone. + // Normal-exit cleanup (the key was already deleted in step 7). Call cleanup anyway to handle any + // partial state edge cases — it is a no-op if the key is gone. cleanup(); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h index 9ac92529f70b..99ec36581af2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h @@ -1,23 +1,26 @@ #pragma once -#include +#include #include namespace DB::Cas { -/// Run the capability battery against `backend`, using throwaway keys under `probe_prefix`. +/// Run the capability battery against `op`, using throwaway keys under `probe_prefix`. /// -/// The probe validates the backend preconditions required by a writable content-addressed pool: -/// 1. Store-level safety checks pass, including the requirement that conditional writes use one -/// underlying HTTP attempt. Hidden SDK retries can outlive the writer's mount lease and obscure -/// whether a conditional operation committed; retries must therefore be explicit CAS state-machine -/// transitions rather than transparent client behavior. -/// 2. Conditional-create and conditional-overwrite are enforced (`putIfAbsent` prevents overwrites, -/// and `putOverwrite` rejects a wrong-token update). -/// 3. `casPut` supports create-if-absent, conflict-on-existing, conflict-on-stale, and commit-on-current. -/// 4. Conditional-delete is enforced (`deleteExact` with a wrong token is rejected and the object survives). -/// 5. Listing reflects both creation and deletion of a probe object. -/// 6. Successful deletion does not create a versioning delete marker. A content-addressed pool cannot +/// `op` must already be admitted. The two store-level precondition hooks — +/// `Backend::checkPoolPreconditions` and `Backend::checkConditionalWriteSingleAttemptSupport` — are the +/// CALLER's responsibility, run through `CasRequests::backendForCapabilityPredicates()` before admitting +/// `op` and calling this: an admitted `CasOperation` carries no route back to the raw backend, by design, +/// so this function cannot reach them itself. +/// +/// The battery validates the backend preconditions required by a writable content-addressed pool: +/// 1. `create` and `replace` are conditional and exact: `create` refuses an occupied key +/// (create-if-absent, conflict-on-existing) and `replace` refuses a stale incarnation +/// (conflict-on-stale), while a matching incarnation commits (commit-on-current). +/// 2. Conditional-delete is enforced (`remove` with a stale incarnation is rejected and the object +/// survives). +/// 3. Listing reflects both creation and deletion of a probe object. +/// 4. Successful deletion does not create a versioning delete marker. A content-addressed pool cannot /// reclaim storage correctly from a versioned bucket: garbage-collection deletes would archive old /// versions instead of removing objects, and repeated ref updates would accumulate versions. /// @@ -25,9 +28,9 @@ namespace DB::Cas /// specific failed check. This is fail-closed: a backend that does not pass the battery MUST NOT be /// used to coordinate a content-addressed pool. /// -/// Cleanup of probe keys is best-effort and runs unconditionally: after the battery completes, or on +/// Cleanup of the probe key is best-effort and runs unconditionally: after the battery completes, or on /// the failure path immediately before the check-failing exception is rethrown. Cleanup itself suppresses /// exceptions so that it cannot hide the capability-check failure. -void runCapabilityProbe(Backend & backend, const String & probe_prefix); +void runCapabilityProbe(CasOperation & op, const String & probe_prefix); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp new file mode 100644 index 000000000000..536518abc7c1 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp @@ -0,0 +1,69 @@ +#include + +#include +#include + +namespace DB +{ +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} +} + +namespace DB::Cas +{ + +void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms) +{ + if (budget.max_attempts < 1) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS request budget rejected: max_attempts must be at least 1 (got {}) — zero would let " + "putIfAbsentControlled return Unresolved without ever sending an attempt.", + budget.max_attempts); + + /// Overflow-safe: `attempt_timeout_ms + lease_safety_margin_ms` could wrap uint64 for absurd config + /// values, which would make the sum spuriously small and the inequality below pass when it should + /// fail closed. Compare via subtraction against the (unsigned, so already non-negative) TTL instead + /// of computing the sum directly. + if (!(budget.attempt_timeout_ms < mount_lease_ttl_ms + && budget.lease_safety_margin_ms < mount_lease_ttl_ms - budget.attempt_timeout_ms)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS request budget rejected: attempt_timeout_ms ({}) + lease_safety_margin_ms ({}) must be " + "strictly less than the mount lease TTL ({} ms). A writable mount refuses to open with " + "this budget.", + budget.attempt_timeout_ms, budget.lease_safety_margin_ms, mount_lease_ttl_ms); + /// STRICTLY less, and the strictness is the load-bearing half. `attempt_timeout_ms > + /// operation_deadline_ms` is the obvious error — a single attempt cannot outlast the logical + /// operation it belongs to. EQUALITY is the subtle one, and it is worse than useless: the deadline + /// is captured as `now + operation_deadline_ms` and every pre-send gate below asks + /// `now + attempt_timeout_ms > deadline_ms`, so equal values collapse that to `now_2 > now_1` and + /// ONE elapsed millisecond between the two clock reads refuses the operation having sent NOTHING. + /// The resulting behaviour is "mostly works, occasionally refuses with nothing sent", decided by + /// the scheduler rather than by the budget — exactly the flakiness this validation exists to catch, + /// and observed three times in tests before it was forbidden. A caller that wants one attempt says + /// `max_attempts = 1`; the equality adds only the race. + if (!(budget.attempt_timeout_ms < budget.operation_deadline_ms)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS request budget rejected: attempt_timeout_ms ({}) must be strictly less than " + "operation_deadline_ms ({}) — equality turns the pre-send gate into a wall-clock race that " + "refuses after a single elapsed tick, having sent nothing. Use max_attempts to bound the " + "number of attempts.", + budget.attempt_timeout_ms, budget.operation_deadline_ms); + if (!(budget.retry_initial_backoff_ms <= budget.retry_max_backoff_ms)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS request budget rejected: retry_initial_backoff_ms ({}) must not exceed " + "retry_max_backoff_ms ({}) — the capped-exponential backoff cap cannot sit below its own " + "starting value. Set both to 0 to disable inter-attempt backoff.", + budget.retry_initial_backoff_ms, budget.retry_max_backoff_ms); + + LOG_INFO(getLogger("CasRequestControl"), + "CAS request budget in effect: attempt_timeout_ms={} operation_deadline_ms={} max_attempts={} " + "lease_safety_margin_ms={} retry_initial_backoff_ms={} retry_max_backoff_ms={} " + "(mount_lease_ttl_ms={} mount_renew_period_ms={})", + budget.attempt_timeout_ms, budget.operation_deadline_ms, budget.max_attempts, + budget.lease_safety_margin_ms, budget.retry_initial_backoff_ms, budget.retry_max_backoff_ms, + mount_lease_ttl_ms, mount_renew_period_ms); +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h new file mode 100644 index 000000000000..84c251b5c2af --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h @@ -0,0 +1,96 @@ +#pragma once +#include + +namespace DB::Cas +{ + +/// The limits a writable mount is configured with. `attempt_timeout_ms` and `lease_safety_margin_ms` +/// are what `CasMountRuntime::admit` measures a request against, and the three `recovery_retry_*` fields +/// bound a whole ref-table recovery; see `validateCasRequestBudget` for the relationship a writable +/// mount enforces at startup. +/// +/// The four fields below them -- `operation_deadline_ms`, `max_attempts` and the two inter-attempt +/// backoff bounds -- belong to the retiring `CasRequestController` and have no consumer in the request +/// contract, which expresses the same bounds as a `Retry` policy per call. They are kept only so the +/// controller and its tests still compile, and they go when it does; their doc comments still describe +/// the controller's own loop. +struct CasRequestBudget +{ + /// Maximum client wait budgeted for one HTTP attempt. `CasRequestController` uses this ONLY as a + /// per-attempt scheduling check (an attempt is not started unless it could still finish inside the + /// operation deadline) — the actual socket-level wait is configured on the object storage's client + /// (the object storage backend's single-attempt client), not by this struct. + uint64_t attempt_timeout_ms = 5000; + /// Maximum wall-clock time for the COMPLETE logical operation — every attempt, every exact-key + /// resolution, and every inter-attempt backoff sleep — counted from the first call to + /// `putIfAbsentControlled`. A DURATION, not an absolute deadline: each call establishes its own + /// `now + operation_deadline_ms` bound. + /// + /// This deadline is the authoritative bound on how long a CAS conditional write keeps riding an S3 + /// disruption server-side before the caller sees an abort. 90s absorbs a ~60s object-store outage + /// with margin (see the arithmetic on `max_attempts` below) — PROVIDED the mount fence stays alive. + /// The fence, not this deadline, is + /// what binds under a TOTAL outage: lease renewals are conditional writes against the same store, + /// so when everything is unreachable the fence deadline freezes at `last_renew + mount_lease_ttl` + /// and `fence_ok` stops the loop ≈ TTL−attempt_timeout−margin (~23s) after the last successful + /// renewal — the required fail-closed behavior (never an attempt past the lease), not a + /// budget limitation. While renewals DO land (blips, throttling, partial outages — the runtime-owned + /// renewal worker keeps extending the fence deadline), the op is NOT bounded by + /// the lease TTL and rides the full deadline here. + uint64_t operation_deadline_ms = 90000; + /// Maximum number of controlled attempts for one logical operation (the first attempt counts as 1). + /// Sized so the operation deadline above — never this count — is what binds under the observed + /// failure shape (~3s adaptive first-attempt PUT timeout per failed attempt + capped-exponential + /// backoff): 16 attempts × ~3s + Σ backoff (0.2+0.4+0.8+1.6+3.2 + 10×5 = 56.2s) ≈ 104s > 90s. + uint32_t max_attempts = 16; + /// Startup-only margin folded into `validateCasRequestBudget`'s inequality against the mount lease + /// TTL. Not consulted at runtime by the controller itself — the caller's `fence_ok` callback (backed + /// by the local write fence's own deadline) is what actually gates lease-relative timing per attempt. + uint64_t lease_safety_margin_ms = 2000; + /// Inter-attempt backoff (`cas_s3_retry_initial_backoff_ms` / + /// `cas_s3_retry_max_backoff_ms`): the sleep before reissuing + /// after an ambiguous attempt whose resolve observed the key absent, capped exponential — + /// `initial · 2^(reissues-1)`, never above `retry_max_backoff_ms`. 0 disables backoff (immediate + /// reissue — the pre-backoff behavior, and what most exhaustion-path unit tests configure). The + /// controller checks the fence BEFORE every sleep and never sleeps past the operation deadline. + uint64_t retry_initial_backoff_ms = 200; + uint64_t retry_max_backoff_ms = 5000; + + /// Recovery-level retry (`CasRefLedger::ensureRefTableRecovered`): a whole ref-table recovery + /// attempt (LIST + snapshot/log GETs + seal PUT) that fails with a transient NETWORK_ERROR is + /// retried, with capped-exponential backoff, until this total wall-clock budget is spent — then the + /// error propagates and the table's load fails for this touch (the `lazy_load_tables` database + /// setting makes the NEXT touch retry). This sits ON TOP of the per-request `operation_deadline_ms` + /// envelope above: one recovery attempt may itself burn ~90s inside a single seal PUT. Independent + /// of the mount-lease invariants validated in `validateCasRequestBudget` — not part of that + /// inequality set. + uint64_t recovery_retry_budget_ms = 120000; + uint64_t recovery_retry_initial_backoff_ms = 1000; + uint64_t recovery_retry_max_backoff_ms = 30000; +}; + +/// Startup validation: a writable mount refuses to open with an inconsistent budget rather than +/// silently falling back to an unbounded or unsafe retry policy. Throws +/// `BAD_ARGUMENTS` unless ALL hold: +/// attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms +/// attempt_timeout_ms < operation_deadline_ms (STRICTLY — see below) +/// retry_initial_backoff_ms <= retry_max_backoff_ms +/// +/// The middle one is strict on purpose. Equality does not mean "one attempt's worth of budget": the +/// deadline is captured as `now + operation_deadline_ms` and each pre-send gate asks +/// `now + attempt_timeout_ms > deadline_ms`, so equal values reduce it to `now_2 > now_1` and a single +/// elapsed millisecond refuses the operation having sent NOTHING. Bound the attempt COUNT with +/// `max_attempts`, never by starving the deadline. +/// `mount_renew_period_ms` takes no part in the inequality (the renewer keeps the fence deadline +/// refreshed well ahead of the TTL by construction) — it is accepted only so the effective-values log +/// line records the full picture in one place. +/// +/// A successor mounting over an unclean predecessor waits at least one lease TTL, plus its +/// materialization grace period, before trusting recovery listings. This is long enough for any +/// conditional PUT still in flight at the predecessor to either land or be abandoned by its own +/// exhausted retry budget. The predecessor's budget is constrained by +/// `attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms`, so no additional handover +/// check is needed here. +void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms); + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp index 0e81b0e7415c..30f6a9f06475 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp @@ -160,58 +160,6 @@ uint64_t saturatingAdd(uint64_t lhs, uint64_t rhs) } -void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms) -{ - if (budget.max_attempts < 1) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: max_attempts must be at least 1 (got {}) — zero would let " - "putIfAbsentControlled return Unresolved without ever sending an attempt.", - budget.max_attempts); - - /// Overflow-safe: `attempt_timeout_ms + lease_safety_margin_ms` could wrap uint64 for absurd config - /// values, which would make the sum spuriously small and the inequality below pass when it should - /// fail closed. Compare via subtraction against the (unsigned, so already non-negative) TTL instead - /// of computing the sum directly. - if (!(budget.attempt_timeout_ms < mount_lease_ttl_ms - && budget.lease_safety_margin_ms < mount_lease_ttl_ms - budget.attempt_timeout_ms)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: attempt_timeout_ms ({}) + lease_safety_margin_ms ({}) must be " - "strictly less than the mount lease TTL ({} ms). A writable mount refuses to open with " - "this budget.", - budget.attempt_timeout_ms, budget.lease_safety_margin_ms, mount_lease_ttl_ms); - /// STRICTLY less, and the strictness is the load-bearing half. `attempt_timeout_ms > - /// operation_deadline_ms` is the obvious error — a single attempt cannot outlast the logical - /// operation it belongs to. EQUALITY is the subtle one, and it is worse than useless: the deadline - /// is captured as `now + operation_deadline_ms` and every pre-send gate below asks - /// `now + attempt_timeout_ms > deadline_ms`, so equal values collapse that to `now_2 > now_1` and - /// ONE elapsed millisecond between the two clock reads refuses the operation having sent NOTHING. - /// The resulting behaviour is "mostly works, occasionally refuses with nothing sent", decided by - /// the scheduler rather than by the budget — exactly the flakiness this validation exists to catch, - /// and observed three times in tests before it was forbidden. A caller that wants one attempt says - /// `max_attempts = 1`; the equality adds only the race. - if (!(budget.attempt_timeout_ms < budget.operation_deadline_ms)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: attempt_timeout_ms ({}) must be strictly less than " - "operation_deadline_ms ({}) — equality turns the pre-send gate into a wall-clock race that " - "refuses after a single elapsed tick, having sent nothing. Use max_attempts to bound the " - "number of attempts.", - budget.attempt_timeout_ms, budget.operation_deadline_ms); - if (!(budget.retry_initial_backoff_ms <= budget.retry_max_backoff_ms)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: retry_initial_backoff_ms ({}) must not exceed " - "retry_max_backoff_ms ({}) — the capped-exponential backoff cap cannot sit below its own " - "starting value. Set both to 0 to disable inter-attempt backoff.", - budget.retry_initial_backoff_ms, budget.retry_max_backoff_ms); - - LOG_INFO(getLogger("CasRequestControl"), - "CAS request budget in effect: attempt_timeout_ms={} operation_deadline_ms={} max_attempts={} " - "lease_safety_margin_ms={} retry_initial_backoff_ms={} retry_max_backoff_ms={} " - "(mount_lease_ttl_ms={} mount_renew_period_ms={})", - budget.attempt_timeout_ms, budget.operation_deadline_ms, budget.max_attempts, - budget.lease_safety_margin_ms, budget.retry_initial_backoff_ms, budget.retry_max_backoff_ms, - mount_lease_ttl_ms, mount_renew_period_ms); -} - namespace { /// Shared by both public entry points below so the log line and the exception's message text can diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h index e01534d48a2f..643da6f46246 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h @@ -1,5 +1,6 @@ #pragma once #include +#include #include #include #include @@ -139,89 +140,6 @@ void recordConditionalWriteAttemptStarted(); /// throwing — see ObjectStorageBackend::nativeConditionalPut). void recordConditionalWriteOutcome(CasWriteOutcome outcome); -/// The three separate limits a CAS-owned retry controller enforces for ONE logical conditional-write -/// operation. Never represented by a single `request_timeout_ms` value — see `validateCasRequestBudget` -/// for the relationship a writable mount enforces at startup, and `CasRequestController` for the -/// runtime use. -struct CasRequestBudget -{ - /// Maximum client wait budgeted for one HTTP attempt. `CasRequestController` uses this ONLY as a - /// per-attempt scheduling check (an attempt is not started unless it could still finish inside the - /// operation deadline) — the actual socket-level wait is configured on the object storage's client - /// (the object storage backend's single-attempt client), not by this struct. - uint64_t attempt_timeout_ms = 5000; - /// Maximum wall-clock time for the COMPLETE logical operation — every attempt, every exact-key - /// resolution, and every inter-attempt backoff sleep — counted from the first call to - /// `putIfAbsentControlled`. A DURATION, not an absolute deadline: each call establishes its own - /// `now + operation_deadline_ms` bound. - /// - /// This deadline is the authoritative bound on how long a CAS conditional write keeps riding an S3 - /// disruption server-side before the caller sees an abort. 90s absorbs a ~60s object-store outage - /// with margin (see the arithmetic on `max_attempts` below) — PROVIDED the mount fence stays alive. - /// The fence, not this deadline, is - /// what binds under a TOTAL outage: lease renewals are conditional writes against the same store, - /// so when everything is unreachable the fence deadline freezes at `last_renew + mount_lease_ttl` - /// and `fence_ok` stops the loop ≈ TTL−attempt_timeout−margin (~23s) after the last successful - /// renewal — the required fail-closed behavior (never an attempt past the lease), not a - /// budget limitation. While renewals DO land (blips, throttling, partial outages — the runtime-owned - /// renewal worker keeps extending the fence deadline), the op is NOT bounded by - /// the lease TTL and rides the full deadline here. - uint64_t operation_deadline_ms = 90000; - /// Maximum number of controlled attempts for one logical operation (the first attempt counts as 1). - /// Sized so the operation deadline above — never this count — is what binds under the observed - /// failure shape (~3s adaptive first-attempt PUT timeout per failed attempt + capped-exponential - /// backoff): 16 attempts × ~3s + Σ backoff (0.2+0.4+0.8+1.6+3.2 + 10×5 = 56.2s) ≈ 104s > 90s. - uint32_t max_attempts = 16; - /// Startup-only margin folded into `validateCasRequestBudget`'s inequality against the mount lease - /// TTL. Not consulted at runtime by the controller itself — the caller's `fence_ok` callback (backed - /// by the local write fence's own deadline) is what actually gates lease-relative timing per attempt. - uint64_t lease_safety_margin_ms = 2000; - /// Inter-attempt backoff (`cas_s3_retry_initial_backoff_ms` / - /// `cas_s3_retry_max_backoff_ms`): the sleep before reissuing - /// after an ambiguous attempt whose resolve observed the key absent, capped exponential — - /// `initial · 2^(reissues-1)`, never above `retry_max_backoff_ms`. 0 disables backoff (immediate - /// reissue — the pre-backoff behavior, and what most exhaustion-path unit tests configure). The - /// controller checks the fence BEFORE every sleep and never sleeps past the operation deadline. - uint64_t retry_initial_backoff_ms = 200; - uint64_t retry_max_backoff_ms = 5000; - - /// Recovery-level retry (`CasRefLedger::ensureRefTableRecovered`): a whole ref-table recovery - /// attempt (LIST + snapshot/log GETs + seal PUT) that fails with a transient NETWORK_ERROR is - /// retried, with capped-exponential backoff, until this total wall-clock budget is spent — then the - /// error propagates and the table's load fails for this touch (the `lazy_load_tables` database - /// setting makes the NEXT touch retry). This sits ON TOP of the per-request `operation_deadline_ms` - /// envelope above: one recovery attempt may itself burn ~90s inside a single seal PUT. Independent - /// of the mount-lease invariants validated in `validateCasRequestBudget` — not part of that - /// inequality set. - uint64_t recovery_retry_budget_ms = 120000; - uint64_t recovery_retry_initial_backoff_ms = 1000; - uint64_t recovery_retry_max_backoff_ms = 30000; -}; - -/// Startup validation: a writable mount refuses to open with an inconsistent budget rather than -/// silently falling back to an unbounded or unsafe retry policy. Throws -/// `BAD_ARGUMENTS` unless ALL hold: -/// attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms -/// attempt_timeout_ms < operation_deadline_ms (STRICTLY — see below) -/// retry_initial_backoff_ms <= retry_max_backoff_ms -/// -/// The middle one is strict on purpose. Equality does not mean "one attempt's worth of budget": the -/// deadline is captured as `now + operation_deadline_ms` and each pre-send gate asks -/// `now + attempt_timeout_ms > deadline_ms`, so equal values reduce it to `now_2 > now_1` and a single -/// elapsed millisecond refuses the operation having sent NOTHING. Bound the attempt COUNT with -/// `max_attempts`, never by starving the deadline. -/// `mount_renew_period_ms` takes no part in the inequality (the renewer keeps the fence deadline -/// refreshed well ahead of the TTL by construction) — it is accepted only so the effective-values log -/// line records the full picture in one place. -/// -/// A successor mounting over an unclean predecessor waits at least one lease TTL, plus its -/// materialization grace period, before trusting recovery listings. This is long enough for any -/// conditional PUT still in flight at the predecessor to either land or be abandoned by its own -/// exhausted retry budget. The predecessor's budget is constrained by -/// `attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms`, so no additional handover -/// check is needed here. -void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms); - /// Throw the recoverable "CAS write could not be committed, retry later" condition. /// /// WHY NETWORK_ERROR (this replaces an earlier ABORTED throw): diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 04b5afc5b338..3fdf7e417611 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -72,12 +72,17 @@ uint64_t saturatingAdd(uint64_t lhs, uint64_t rhs) return lhs > std::numeric_limits::max() - rhs ? std::numeric_limits::max() : lhs + rhs; } +} + /// The failure class a fresh credential could fix, named here rather than taken from /// `S3Exception::isAccessTokenExpiredError`, which also fires on `S3Errors::UNKNOWN` -- the SDK's code /// for EVERY error it does not model. Borrowing it would put throttling codes an S3-compatible store /// reports under a non-AWS name into the credential class, and would carve a real access denial out of /// `isDefinitelyRefusedWrite` so the write wedges its caller instead of being refused. `UNKNOWN` is /// therefore never matched by code alone; a store that spells the error out by name still matches. +/// +/// Declared in the header: a caller whose OWN loop makes the next physical attempt has to tell this +/// class apart from the rest of `isDefinitelyRefusedWrite`. bool isRefreshableCredentialError([[maybe_unused]] const std::exception & e) { #if USE_AWS_S3 @@ -96,6 +101,9 @@ bool isRefreshableCredentialError([[maybe_unused]] const std::exception & e) #endif } +namespace +{ + /// An answer from the store rather than a fault in reaching it: reissuing replays it unchanged. bool isDefiniteStoreRefusal([[maybe_unused]] const std::exception & e) { @@ -111,6 +119,20 @@ GaveUp::Source sourceFor(const Retry::Bound & bound) return bound.lease_bound ? GaveUp::Source::Lease : GaveUp::Source::Policy; } +/// Could the precondition this write was built with still be met by what the resolve read saw? A +/// create needs the key absent; a replace needs the incarnation it named to still be current. +bool preconditionStillSatisfiable(const Observation & seen, const std::optional & expected) +{ + return std::visit(detail::Overload{ + /// The read itself failed, so it proved nothing either way and an ambiguous attempt may still + /// be alive. Reporting a conflict on it would name an occupant nobody observed. + [](const NotObserved &) { return true; }, + [&](const ProvenAbsent &) { return !expected.has_value(); }, + [&](const Meta & m) { return expected.has_value() && m.incarnation == *expected; }, + [&](const Object & o) { return expected.has_value() && o.incarnation == *expected; }}, + seen); +} + /// Drop the body from an observation the write engine had to fetch to prove whose bytes were at the /// key. The presence-only loop is defined by what it reports, so the demotion happens on its results /// rather than being trusted to every branch that builds one. @@ -560,7 +582,7 @@ CasOperation::Resolved CasOperation::observePresence(const String & key, const R WriteResult CasOperation::gaveUp(GaveUp::Why why, GaveUp::Source source, WriteState & state) const { ProfileEvents::increment(ProfileEvents::CASRequestGaveUp); - return GaveUp{why, source, state.sent_any, state.last_seen}; + return GaveUp{why, source, state.sent_any, state.last_seen, state.attempts_sent}; } WriteResult CasOperation::gaveUpForReadStop(ReadStop stop, WriteState & state, const Retry::Bound & bound) const @@ -631,6 +653,11 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co if (expected) expected_value = owner.valueFor(key, *expected); + /// This inner write's own bytes are what an ambiguity of it could have landed. A previous inner + /// write of the same call sent DIFFERENT bytes and ended in a conflict, which proved its attempts + /// dead -- carrying its ambiguity forward is how a competitor's identical object gets claimed. + state.any_ambiguous = false; + for (;;) { /// A write reserves TWO envelopes: the attempt, and the exact read that settles it. That is @@ -653,7 +680,7 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co std::optional> outcome; /// A credential answer is given BEFORE the store applies anything, so the attempt provably did /// not land. It is the one failure that is neither a commit nor an ambiguity, and keeping it out - /// of `any_ambiguous` is what lets a second credential failure of the same call be refused + /// of `any_ambiguous` is what lets a second credential failure of this inner write be refused /// instead of resolved by a read and reissued to the deadline. bool credential_answer = false; bool refreshed = false; @@ -668,30 +695,35 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co { if (isDeterministicLocalFailure(e.code())) throw; - /// ONE refresh per call, and only for the class a credential could explain -- so an - /// oversized entity or a malformed request never triggers a re-acquisition, and a denial - /// that fresh credentials do not fix is refused on the second look rather than reissued to - /// the deadline (the storage hands back a new client every time it is asked). + /// ONE refresh per call, only for the class a credential could explain, and only when a + /// reissue could sign with what it installs. So an oversized entity never triggers a + /// re-acquisition, a denial fresh credentials do not fix is refused on the second look + /// rather than reissued to the deadline (the storage hands back a new client every time it + /// is asked), and under a single-attempt policy the refusal below is literally "no refresh + /// installed credentials and no earlier ambiguity". credential_answer = isRefreshableCredentialError(e); - if (credential_answer && !state.refresh_attempted) + if (credential_answer && !state.refresh_attempted && !policy.single_attempt) { state.refresh_attempted = true; refreshed = owner.backend->refreshCredentials(); } - /// Fresh credentials only help if there is a reissue to sign with them, so under `once` the - /// store's answer stands even when a refresh succeeded. A refusal that FOLLOWS an ambiguous - /// attempt of this call proves nothing about that attempt, so it is settled by the read - /// below instead of ending the call here. - if ((!refreshed || policy.single_attempt) && isDefinitelyRefusedWrite(e) && !state.any_ambiguous) + /// A refusal that FOLLOWS an ambiguous attempt of this inner write proves nothing about that + /// attempt, so it is settled by the read below instead of ending the call here. + if (!refreshed && isDefinitelyRefusedWrite(e) && !state.any_ambiguous) { ProfileEvents::increment(ProfileEvents::CASRequestRefused); - return Refused{e.code(), e.message()}; + return Refused{e.code(), e.message(), state.attempts_sent}; } } - catch (const std::exception &) + catch (const std::exception & e) { - /// An unmodeled failure may still have landed: leave `outcome` disengaged and settle it by - /// reading, never by reporting a refusal the store never gave. + /// Only the transport can have landed anything, and every exception it raises is a + /// `Poco::Exception`. A local fault -- a bad allocation, a logic error raised inside the + /// attempt -- is not a store answer, and settling it by a read would bury the bug behind an + /// outcome the store never gave. What is left is an unmodeled transport failure that may + /// still have landed, so `outcome` stays disengaged and the read below settles it. + if (!dynamic_cast(&e)) + throw; } if (outcome && outcome->has_value()) @@ -705,10 +737,8 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co if (!outcome && !credential_answer) state.any_ambiguous = true; - /// Nothing for a read to settle: this attempt did not apply, and no EARLIER attempt of the call - /// is unresolved either. Re-send it under the credentials the refresh installed. The policy is - /// named here rather than inherited from the refusal above, so a gap in what counts as a - /// definite refusal can never turn `once` into a sleeping loop. + /// Nothing for a read to settle: this attempt did not apply, and no EARLIER attempt of this + /// inner write is unresolved either. Re-send it under the credentials the refresh installed. if (refreshed && !policy.single_attempt && !state.any_ambiguous) { if (auto given_up = pauseAndReissue(state, bound)) @@ -730,17 +760,29 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// reported as an ordinary conflict and a lease refusal as a policy deadline. if (resolved.stop) return gaveUpForReadStop(*resolved.stop, state, bound); - if (const auto * obj = std::get_if(&state.last_seen)) + + /// No attempt of this inner write is unresolved, so the refused precondition IS the answer. + /// Identical bytes here are somebody else's object, and the caller that owns the key's meaning + /// decides what that means. + if (!state.any_ambiguous) + return Conflict{state.last_seen, state.attempts_sent}; + + if (!preconditionStillSatisfiable(state.last_seen, expected)) { - /// Our own bytes prove an earlier ambiguous attempt landed. Without an ambiguity there was - /// nothing of ours to land, so identical bytes are somebody else's object and the caller - /// that owns the key's meaning decides what that means. - if (state.any_ambiguous && obj->bytes == bytes) + /// The precondition has MOVED, and THAT is what makes byte equality a proof: an attempt + /// that never applied leaves the key exactly as it found it, so under an incarnation that + /// did not move our own bytes cannot be told from the bytes already there. A create reaches + /// here for every object it sees -- an occupied key never satisfies its precondition. + if (const auto * obj = std::get_if(&state.last_seen); obj && obj->bytes == bytes) return postCommit(obj->incarnation, /*resolved_by_read=*/true, state, bound); - return Conflict{state.last_seen}; + /// Nothing of this inner write's is at the key, and a reissue would be refused too. + return Conflict{state.last_seen, state.attempts_sent}; } - if (!state.any_ambiguous) - return Conflict{state.last_seen}; + + /// Unresolved but repeatable: the precondition would still be met -- or nothing was observed at + /// all, which proves neither way and leaves this write's ambiguity alive. A reissue is what + /// settles it, and re-sending the same bytes under the same precondition is safe: it ends at + /// the store's own answer. A policy with no reissue has to say it settled nothing. if (policy.single_attempt) return gaveUp(GaveUp::Why::Unresolved, sourceFor(bound), state); if (auto given_up = pauseAndReissue(state, bound)) @@ -849,7 +891,7 @@ WriteResult CasOperation::readModifyWriteOnPresence(const String & key, const De current.reset(); if (policy.single_attempt) - return Conflict{state.last_seen}; + return Conflict{state.last_seen, state.attempts_sent}; if (auto given_up = pauseAndReissue(state, bound)) return *given_up; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index 8b29fe82eeae..75aa3fee87c6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -34,6 +34,12 @@ bool isDefinitelyRefusedWrite(const std::exception & e); /// `NOT_IMPLEMENTED`, `BAD_ARGUMENTS` and `CORRUPTED_DATA`. bool isDeterministicLocalFailure(int code); +/// The failure class a FRESH CREDENTIAL could fix -- a subset of `isDefinitelyRefusedWrite`, exposed +/// because a caller whose own loop makes the next physical attempt must not treat it as terminal: the +/// engine refreshes once before it gives the answer, and the caller's next attempt signs with what the +/// refresh installed. +bool isRefreshableCredentialError(const std::exception & e); + /// Facts the fence cannot see, sampled by the caller. Non-throwing; FALSE ends the operation exactly /// like a lost fence, because the engine does not need to know which of the two refused. using Liveness = std::function; @@ -87,6 +93,10 @@ class CasRequests CasRequests(BackendPtr backend_, Fence fence_, std::function now_ms_ = {}, std::function sleep_ms_ = {}); + /// Both may be called concurrently on one `CasRequests`: neither writes a member, and the only + /// state either reads is the backend and the fence -- whose closures must therefore be thread-safe + /// too. The test setters below DO write members, and belong to setup, before any operation runs. + /// /// Admitted now, under the fence's current generation. CasOperation admit(Liveness liveness = {}); /// Admitted earlier: the generation came from a persisted runtime record, and an operation that @@ -96,6 +106,11 @@ class CasRequests /// The capability predicates and `dialect()`. Backend & backendForCapabilityPredicates() { return *backend; } + /// A caller's own inter-iteration wait, paced through the same clock the engine's own sleeps use -- + /// so a test that replaces the sleep sees no real time pass in either. `CasOperation` carries the + /// same call for the loops that hold an operation rather than the plane it was admitted on. + void pause(uint64_t ms) { sleep_ms(ms); } + void setNowFnForTest(std::function now_ms_); void setSleepFnForTest(std::function sleep_ms_); void setAttemptReservationForTest(uint64_t ms) { attempt_reservation_ms = ms; } @@ -134,6 +149,9 @@ class CasRequests /// Move-only, and every request it makes re-checks its admission -- before each attempt, before each /// sleep, and once more after a proven commit, so a write whose fence was lost while it was in flight /// is never reported as committed. +/// +/// SINGLE-THREADED: it carries mutable per-call state, so one operation belongs to one task. A caller +/// that fans work out gives each task its own, built from `generation()` through `CasRequests::resume`. class CasOperation { public: @@ -142,6 +160,10 @@ class CasOperation CasOperation & operator=(const CasOperation &) = delete; uint64_t generation() const { return admitted_generation; } + /// A caller's own inter-iteration wait, paced through the same clock the engine's own sleeps use -- + /// so a test that replaces the sleep sees no real time pass in either. For the hand-written loops + /// that reissue something the engine must not reissue for them. + void pause(uint64_t ms) { owner.sleep_ms(ms); } /// The verdict point: is this operation still admitted? For the sites that guard a decision rather /// than a request. bool admitted() const { return gate(0) == Gate::Ok; } @@ -192,15 +214,19 @@ class CasOperation /// The admission point: the fence for `needed_ms` from now, then the caller's own facts. Gate gate(uint64_t needed_ms) const; - /// Everything ONE logical write call accumulates. It outlives each attempt, and for - /// `readModifyWrite` it outlives each inner write, so a `GaveUp` reports what the whole call did - /// rather than what its last attempt did. + /// Everything ONE logical write call accumulates. `attempts_sent`, `sent_any`, `last_seen`, + /// `reissues` and `refresh_attempted` outlive each attempt and, for `readModifyWrite`, each inner + /// write, so a `GaveUp` reports what the whole call did rather than what its last attempt did. struct WriteState { uint32_t attempts_sent = 0; bool sent_any = false; - /// Did ANY attempt of this call end without proof of whether it applied? A credential answer - /// does not qualify: the store gives it before applying anything. + /// The one field that belongs to the INNER write instead: did any of ITS attempts end without + /// proof of whether it applied? Every attempt of one inner write sends the same bytes, which is + /// what makes "the resolve read found our bytes" a statement about an attempt of ours; an inner + /// write that ended in `Conflict` saw the precondition move, which proves its ambiguous + /// attempts dead. A credential answer never sets it: the store gives one before applying + /// anything. bool any_ambiguous = false; Observation last_seen = NotObserved{}; uint32_t reissues = 0; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp index af52f2854bfd..0f17ef3c4be1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp @@ -6,9 +6,9 @@ namespace DB::Cas { -SentinelProbeResult probeSentinel(Backend & backend, const String & key) +SentinelProbeResult probeSentinel(CasOperation & op, const String & key, const Retry & policy) { - return backend.probeSentinelRaw(key); + return op.probeSentinel(key, policy); } namespace @@ -29,7 +29,7 @@ bool isProbeSubtreeDebris(const String & probe_root, const String & key) } -BootstrapResidual probePoolBootstrapResidual(Backend & backend, const Layout & layout) +BootstrapResidual probePoolBootstrapResidual(CasOperation & op, const Layout & layout) { const String pool_meta_key = layout.poolMetaKey(); const String catalog_key = layout.refCatalogKey(); @@ -40,31 +40,28 @@ BootstrapResidual probePoolBootstrapResidual(Backend & backend, const Layout & l /// `_pool_meta` anywhere is decisive. It relies on lexicographic LIST order only for COST — `_pool_meta` /// sorts first under `/`, so a healthy pool short-circuits on the first page rather than /// enumerating its whole content on every open. + bool has_pool_meta = false; bool has_residual = false; bool has_catalog = false; try { - String cursor; - for (;;) + op.forEachListedKey(prefix, [&](const KeyEntry & listed) -> bool { - const ListPage page = backend.list(prefix, cursor, 1000); - for (const ListedKey & listed : page.keys) + if (listed.key == pool_meta_key) { - if (listed.key == pool_meta_key) - return BootstrapResidual::PoolMetaPresent; /// decisive — the pool is authoritative - if (isProbeSubtreeDebris(probe_root, listed.key)) - continue; /// crash leftover / concurrent opener's battery — ignore ([D2]) - if (listed.key == catalog_key) - { - has_catalog = true; - continue; - } - has_residual = true; /// a non-`_probe` object, and no `_pool_meta` seen (so far) + has_pool_meta = true; + return false; /// decisive — the pool is authoritative; stop the walk } - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; - } + if (isProbeSubtreeDebris(probe_root, listed.key)) + return true; /// crash leftover / concurrent opener's battery — ignore ([D2]) + if (listed.key == catalog_key) + { + has_catalog = true; + return true; + } + has_residual = true; /// a non-`_probe` object, and no `_pool_meta` seen (so far) + return true; + }, Retry::standard()); } catch (...) { @@ -77,6 +74,8 @@ BootstrapResidual probePoolBootstrapResidual(Backend & backend, const Layout & l prefix, getCurrentExceptionMessage(/*with_stacktrace=*/false)); return BootstrapResidual::Indeterminate; } + if (has_pool_meta) + return BootstrapResidual::PoolMetaPresent; if (has_residual) return BootstrapResidual::ResidualWithoutMeta; if (!has_catalog) @@ -88,7 +87,7 @@ BootstrapResidual probePoolBootstrapResidual(Backend & backend, const Layout & l /// license to mint `_pool_meta`. try { - const auto got = backend.get(catalog_key); + const auto got = op.read(catalog_key, Retry::standard()); if (!got) return BootstrapResidual::ResidualWithoutMeta; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h index c0d926f1c31d..c4e15ee014c4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include namespace DB::Cas @@ -9,11 +9,17 @@ namespace DB::Cas /// timeouts / 5xx / connection errors => Indeterminate; permission errors => AccessDenied; /// missing container/bucket/prefix-parent => ContainerAbsent; a clean authoritative miss => KeyAbsent. /// -/// Free-function entry point (spec §2) — a thin dispatch to the backend's own typed-evidence -/// classification (`Backend::probeSentinelRaw`; see there for the per-backend semantics: the -/// S3-native raw HEAD error, the Local container-directory stat, or the generic head/get-based -/// default for a backend without sharper evidence). -SentinelProbeResult probeSentinel(Backend & backend, const String & key); +/// Free-function entry point (spec §2) — a thin dispatch to `op`'s own typed-evidence classification +/// (`CasOperation::probeSentinel`, which in turn reaches `Backend::probeSentinelRaw`; see there for the +/// per-backend semantics: the S3-native raw HEAD error, the Local container-directory stat, or the +/// generic head/get-based default for a backend without sharper evidence). +/// +/// `policy` is required and has no default, because `Indeterminate` is the one outcome the request +/// contract REISSUES on and the right number of reissues is the caller's question, not this function's. +/// A caller that owns an outer retry loop -- the lifecycle gate, whose inconclusive verdict IS +/// `StayTransient` for the recovery loop to retry -- passes `Retry::once()`, or it pays its own loop's +/// interval inside every probe. A caller with no loop of its own passes `Retry::standard()`. +SentinelProbeResult probeSentinel(CasOperation & op, const String & key, const Retry & policy); /// Verdict of the zero-write startup bootstrap residual check ("Startup ordered vs the capability /// probe"). Before a writable `Pool::open` runs @@ -50,6 +56,6 @@ enum class BootstrapResidual : uint8_t /// bootstraps cleanly. On non-strong-LIST backends this is the single best-effort authoritative check the /// weaker guarantee allows — still fail-closed on any residual object found. Used by `Pool::open` BEFORE /// the capability battery so that no probe write ever precedes the emptiness proof. -BootstrapResidual probePoolBootstrapResidual(Backend & backend, const Layout & layout); +BootstrapResidual probePoolBootstrapResidual(CasOperation & op, const Layout & layout); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h index eac6be8562f9..d6a8a172007e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h @@ -41,16 +41,25 @@ struct Committed { Incarnation incarnation; uint32_t attempts_sent; bool resolve /// present under the caller's intended content. `seen` is whatever the resolve read observed. struct Declined { Observation seen; }; /// A competing write won: the key's current state does not match what this call expected. -struct Conflict { Observation seen; }; +/// `attempts_sent` counts the HTTP attempts this call made, the same count `Committed` and `GaveUp` +/// carry: an operator's attempt counters sum over ALL the endings of a write, and losing the key is +/// one an operator wants counted rather than dropped. +struct Conflict { Observation seen; uint32_t attempts_sent = 0; }; /// The store itself refused the request (not a lost precondition) -- `store_error` is a ClickHouse -/// error code and `message` explains it. -struct Refused { int store_error; String message; }; +/// error code and `message` explains it. `attempts_sent` is the same count `Conflict` carries, for +/// the same reason. +struct Refused { int store_error; String message; uint32_t attempts_sent = 0; }; /// No attempt landed and none can be proven safe to keep making. struct GaveUp { enum class Why : uint8_t { Deadline, FenceLost, Unresolved }; enum class Source : uint8_t { Policy, Lease }; Why why; Source deadline_source; bool sent_any; Observation last_seen; + /// The HTTP attempts this call made, the same count `Committed` carries. Operator counters -- the + /// mount renewal's attempt and retry counters among them -- have to count the attempts of a write + /// that GAVE UP as well as of one that committed, and `sent_any` cannot say how many. It stays + /// beside this because the readers that only branch on "was anything sent" branch on it by name. + uint32_t attempts_sent = 0; }; using WriteResult = std::variant; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index 42f7f2c6684e..fedde91b1c9a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -1235,9 +1235,15 @@ void ContentAddressedMetadataStorage::confirmPoolIdentityForEmptyEnumeration(con const Cas::PoolPtr pool = store(); /// Live here (past the op gate's Live, non-terminal admission). ++empty_proof_probe_count_for_test; + /// The open plane: this probe is what authorizes an empty answer, and a mount whose lease has + /// blipped must still be able to ask it. + Cas::CasOperation probe_op = pool->gcRequests().admit(); const Cas::SentinelProbeResult probe = empty_proof_probe_override_for_test ? empty_proof_probe_override_for_test() - : Cas::probeSentinel(pool->backend(), pool->layout().poolMetaKey()); + /// `once`: this probe is the gate that authorises an empty answer, and an inconclusive one is a + /// refusal the caller retries -- reissuing here would stall a directory listing for the whole + /// retry window instead. + : Cas::probeSentinel(probe_op, pool->layout().poolMetaKey(), Cas::Retry::once()); switch (probe.outcome) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp index 8f281eaea842..95eea5cd319b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp @@ -289,7 +289,8 @@ void ContentAddressedTransaction::uploadPendingBlobs(PartStaging & st) const uint64_t payload_size = pb.size; source.open = [store, staging_key, header_len, payload_size]() -> std::unique_ptr { - auto staged = store->backend().getStream(staging_key); + Cas::CasOperation op = store->mountRequests().admit(); + auto staged = op.stream(staging_key, Cas::Retry::standard()); if (!staged) throw Exception( ErrorCodes::FILE_DOESNT_EXIST, @@ -297,7 +298,7 @@ void ContentAddressedTransaction::uploadPendingBlobs(PartStaging & st) staging_key); String encoded_header(header_len, '\0'); - staged->stream->readStrict(encoded_header.data(), encoded_header.size()); + staged->readStrict(encoded_header.data(), encoded_header.size()); const Cas::EnvelopeHeader decoded = Cas::decodeEnvelopeHeader( encoded_header, header_len + payload_size, @@ -309,7 +310,7 @@ void ContentAddressedTransaction::uploadPendingBlobs(PartStaging & st) staging_key, decoded.header_len, header_len); - return std::move(staged->stream); + return staged; }; } else @@ -906,10 +907,12 @@ std::unique_ptr ContentAddressedTransaction::writeFile( /// buffer writes this header first, UNHASHED and excluded from the reported size, so the /// content key stays the pool's hash of `payload` and `blob_size` stays the payload size. std::string envelope_header = buildS3StagingBlobHeader(*r); - /// rev.7 [C2]: capture the fence generation now, re-checked immediately before the durable - /// `sink->finalize()` in `finalizeImpl` (the streaming upload becomes durable there). + /// The staged upload becomes durable in `sink->finalize()`, outside this request contract by + /// design, so its admission is re-checked there through the callback below. The generation + /// comes off the operation that admitted it, so the two can never name different + /// incarnations. const Cas::PoolPtr pool = metadata_storage.store(); - const uint64_t admitted_generation = pool->fenceGeneration(); + const uint64_t admitted_generation = pool->mountRequests().admit().generation(); return std::make_unique( std::move(object_sink), staging_key, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index edefc816cff2..c8fd6d8919c9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -15,8 +16,8 @@ namespace DB::Cas /// `gc/gen/{g}/attempt/{a}/outcomes/{round}/{shard}`. It contains the results of exact-token deletes /// for entries that were already published as `delete_pending`, as well as candidates spared when /// the one-pass merge found a live in-degree. The log is written before the round's single state CAS; -/// `putIfAbsent` adopts an existing durable log on replay rather than treating a byte difference as -/// an error. The uncompressed payload is a header line, one flat JSON record per entry in insertion +/// An existing durable log is adopted on replay rather than treated as an error on a byte +/// difference. The uncompressed payload is a header line, one flat JSON record per entry in insertion /// order, and an `{"n":count}` trailer. `FormatId::GcOutcomes` stores the sealed payload in one zstd /// frame, so its object-storage key has the `.zst` suffix. enum class OutcomeKind : uint8_t @@ -40,7 +41,7 @@ struct OutcomeEntry { ObjectKind kind = ObjectKind::Blob; BlobRef ref{}; - Token token; + PersistedIncarnation token; OutcomeKind outcome = OutcomeKind::Spared; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h index f17cb5b87fb1..80acffecde30 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.h @@ -10,7 +10,7 @@ namespace DB::Cas { -class Backend; +class CasOperation; class Layout; /// `_pool_meta` — the pool identity and the pool-wide constants that every reader and writer must @@ -50,19 +50,19 @@ struct PoolMeta /// pool-lifecycle entry point cannot silently re-arm the observe-mint footgun by omission. The two /// production callers pass it explicitly; only test minting sites opt in with `allow_mint=true`. static PoolMeta createOrValidate( - Backend &, const Layout &, uint64_t blob_header_len, uint64_t gc_shards, + CasOperation &, const Layout &, uint64_t blob_header_len, uint64_t gc_shards, BlobHashAlgo blob_hash_algo = BlobHashAlgo::CityHash128, bool allow_new = false, bool allow_mint = false); /// Convenience for single-shard callers. Production pool opening passes the configured value to /// the explicit overload above; this preserves compact single-shard codec/unit fixtures. static PoolMeta createOrValidate( - Backend & backend, const Layout & layout, uint64_t blob_header_len, + CasOperation & op, const Layout & layout, uint64_t blob_header_len, BlobHashAlgo blob_hash_algo = BlobHashAlgo::CityHash128, bool allow_new = false, bool allow_mint = false) { return createOrValidate( - backend, layout, blob_header_len, /*gc_shards=*/1, blob_hash_algo, allow_new, allow_mint); + op, layout, blob_header_len, /*gc_shards=*/1, blob_hash_algo, allow_new, allow_mint); } }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index dbf9fd8fc9d8..e18da0869b0a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -56,8 +57,9 @@ inline RunMarker runMarkerFromByte(char byte, std::string_view what) /// whole — streamed one line at a time over a `ReadBuffer`), `line_cap = 4 KiB`, `PinnedRaw` (no /// compression) + `Strict` (byte-deterministic for `putDeterministicArtifact` adoption). /// -/// This file is backend-free: it accepts caller-owned `ReadBuffer`/`WriteBuffer` objects and never -/// includes backend or GC subsystem headers. The GC layer owns the stream lifetime and the bridge to +/// This file is backend-free: it accepts caller-owned `ReadBuffer`/`WriteBuffer` objects and reaches +/// no backend or GC machinery -- `PersistedIncarnation` is a value type with no live backend behind +/// it, which is exactly why a persisted row may hold one. The GC layer owns the stream lifetime and the bridge to /// packed keys and condemned rows; this codec owns only the durable text representation and its /// identifier-layer types. Keeping that boundary physical prevents storage or GC dependencies from /// leaking into the format implementation. @@ -85,7 +87,7 @@ struct SourceEdgeRecord UInt128 source_id{}; RunMarker marker = RunMarker::Edge; bool delete_pending = false; - Token token{}; + PersistedIncarnation token{}; uint64_t size = 0; uint64_t condemn_round = 0; bool marker_confirmed = false; /// durable Condemned meta confirmed for this entry (graduation gate) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 39191bad1a43..93bda6fd1b91 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -20,14 +20,25 @@ namespace DB::Cas static_assert(casEnumTableCoversEnum()); static_assert(casEnumTableCoversEnum()); -std::string_view tokenTypeToWord(TokenType t) +std::string_view dialectWordFromString(std::string_view w, std::string_view what) { - return kTokenTypeWords.toWord(t, "CAS wire: TokenType"); + /// Parse then re-render, so what a record carries is the table's own spelling rather than the + /// caller's copy of it. + return kTokenTypeWords.toWord(kTokenTypeWords.fromWord(w, what), what); } -TokenType tokenTypeFromWord(std::string_view w, std::string_view what) +uint8_t dialectByteFromWord(std::string_view w, std::string_view what) { - return kTokenTypeWords.fromWord(w, what); + return static_cast(kTokenTypeWords.fromWord(w, what)); +} + +std::string_view dialectWordFromByte(uint8_t byte, std::string_view what) +{ + for (const auto & entry : kTokenTypeWords.entries) + if (static_cast(entry.value) == byte) + return entry.word; + /// The byte comes off persisted media, so an unknown one is malformed data, not a caller bug. + throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: unknown dialect byte {}", what, byte); } BlobHashAlgo blobHashAlgoFromWord(std::string_view w, std::string_view what) @@ -45,10 +56,10 @@ ObjectKind objectKindFromWord(std::string_view w, std::string_view what) return kObjectKindWords.fromWord(w, what); } -void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t) +void writeTokenFields(CasJsonWriter & out, bool & first, const PersistedIncarnation & inc) { - writeStringField(out, SharedWire::token_type, tokenTypeToWord(t.type), first); - writeStringField(out, SharedWire::token, t.value, first); + writeStringField(out, SharedWire::token_type, dialectWordFromString(inc.dialect, "wire: dialect"), first); + writeStringField(out, SharedWire::token, inc.value, first); } void writeBlobRefFields(CasJsonWriter & out, bool & first, const BlobRef & r) @@ -107,11 +118,11 @@ BlobRef BlobRefFields::build(std::string_view what) const return BlobRef{algo, codecFor(algo).fromHex(*digest_hex)}; } -Token TokenFields::build(std::string_view what) const +PersistedIncarnation TokenFields::build(std::string_view what) const { if (!type_word || !value) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: token missing token_type/token", what); - return Token{*value, tokenTypeFromWord(*type_word, what)}; + return PersistedIncarnation{String(dialectWordFromString(*type_word, what)), *value}; } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h index 5ca327462192..c370e60db464 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -17,7 +18,9 @@ namespace DB::Cas /// unrecognized value with `CORRUPTED_DATA`; silently choosing a default would turn malformed /// persisted data into a different valid-looking record. -/// The `TokenType` wire vocabulary; coverage is proven in `CasWireVocab.cpp`. +/// The incarnation-dialect wire vocabulary; coverage is proven in `CasWireVocab.cpp`. It has two +/// persisted encodings -- the word, used by every JSON codec here, and the one byte the condemned-row +/// payload stores -- and both go through this one table so they can never name different sets. inline constexpr EnumWireTable kTokenTypeWords{{{ {TokenType::ETag, "etag"}, {TokenType::Generation, "generation"}, @@ -29,13 +32,15 @@ inline constexpr EnumWireTable kObjectKindWords{{{ {ObjectKind::Blob, "blob"}, }}}; -/// Convert a token discriminator to its canonical wire word. Throws `LOGICAL_ERROR` for an -/// out-of-range enum value. -std::string_view tokenTypeToWord(TokenType t); +/// Validate a persisted dialect word and return its canonical spelling. `what` identifies the +/// containing codec or field in the `CORRUPTED_DATA` exception; an unrecognized word is rejected +/// rather than carried into a record no reader could decode. +std::string_view dialectWordFromString(std::string_view w, std::string_view what); -/// Parse a canonical token-type word. `what` identifies the containing codec or field in the -/// `CORRUPTED_DATA` exception; unknown words are rejected rather than treated as a default type. -TokenType tokenTypeFromWord(std::string_view w, std::string_view what); +/// The same vocabulary in the one-byte form the condemned-row payload stores. Both directions are +/// fail-closed: an unrecognized word or byte is `CORRUPTED_DATA`. +uint8_t dialectByteFromWord(std::string_view w, std::string_view what); +std::string_view dialectWordFromByte(uint8_t byte, std::string_view what); /// Parse a canonical blob-hash algorithm word. The write side uses `blobHashAlgoName` directly, so /// this is its fail-closed inverse. `what` identifies the containing codec or field in the @@ -51,8 +56,9 @@ std::string_view objectKindToWord(ObjectKind k); ObjectKind objectKindFromWord(std::string_view w, std::string_view what); /// Append the sibling fields `token_type` and `token` to an in-progress JSON object. The caller owns `first`, -/// which must describe the fields already written to that object; the token value is JSON-escaped. -void writeTokenFields(CasJsonWriter & out, bool & first, const Token & t); +/// which must describe the fields already written to that object; the value is JSON-escaped. The +/// dialect is validated on the way out, so a record can never persist a word its reader would reject. +void writeTokenFields(CasJsonWriter & out, bool & first, const PersistedIncarnation & inc); /// Append the sibling fields `algo` and `digest` to an in-progress JSON object. The algorithm word and /// lowercase digest are canonical, and the digest is rendered at the width required by `r.algo`. @@ -145,15 +151,16 @@ struct BlobRefFields BlobRef build(std::string_view what) const; }; -/// Collector for one `Token`'s two flat fields (`token_type`/`token`), filled in by `matchTokenFields`. +/// Collector for one persisted incarnation's two flat fields (`token_type`/`token`), filled in by +/// `matchTokenFields`. struct TokenFields { std::optional type_word; std::optional value; - /// Requires both fields and parses the token type word. `what` identifies the enclosing codec + /// Requires both fields and validates the dialect word. `what` identifies the enclosing codec /// in `CORRUPTED_DATA` exceptions. - Token build(std::string_view what) const; + PersistedIncarnation build(std::string_view what) const; }; /// Each `match*Fields` helper tests `key` against the one or two field names it owns, consumes the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp index 2700aec0291a..9343906d7202 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -53,8 +54,8 @@ class PriorEdgeCursor { public: /// The key codec is stateless and self-describing, so a run may freely mix supported hash algorithms. - PriorEdgeCursor(Backend & backend_, const std::vector & segments_) - : backend(backend_), segments(segments_) + PriorEdgeCursor(CasOperation & op_, const std::vector & segments_) + : op(op_), segments(segments_) { advance(); } @@ -137,12 +138,12 @@ class PriorEdgeCursor } /// Typed open validates the NDJSON header before any row is consumed. Each row carries its /// own algorithm byte, so no separate width gate is needed. - reader = openSourceEdgeRun(backend, segments[seg_idx].key); + reader = openSourceEdgeRun(op, segments[seg_idx].key); } } private: - Backend & backend; + CasOperation & op; const std::vector & segments; size_t seg_idx = 0; @@ -190,7 +191,7 @@ String encodeCondemnedRow(const CondemnedRow & row) String out; out.push_back(runMarkerByte(RunMarker::Condemned)); out.push_back(static_cast((row.delete_pending ? 1 : 0) | (row.marker_confirmed ? 2 : 0))); - out.push_back(static_cast(row.token.type)); + out.push_back(static_cast(dialectByteFromWord(row.token.dialect, "condemned row"))); auto beU64 = [&](uint64_t v) { for (int i = 7; i >= 0; --i) out += static_cast((v >> (8 * i)) & 0xFF); }; beU64(row.condemn_round); beU64(row.size); @@ -214,10 +215,7 @@ CondemnedRow decodeCondemnedRow(std::string_view p) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS condemned row: unknown flags 0x{:02x}", flags); row.delete_pending = flags & 1; row.marker_confirmed = flags & 2; - const uint8_t type = static_cast(p[2]); - if (type < 1 || type > 3) - throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS condemned row: unknown token_type {}", type); - row.token.type = static_cast(type); + row.token.dialect = String(dialectWordFromByte(static_cast(p[2]), "condemned row")); auto beU64 = [&](size_t off) { uint64_t v = 0; for (int i = 0; i < 8; ++i) v = (v << 8) | static_cast(p[off + i]); return v; }; row.condemn_round = beU64(3); row.size = beU64(11); @@ -274,14 +272,14 @@ SourceEdgeRunView openSourceEdgeRun(std::string_view bytes) return SourceEdgeRunView(std::make_unique(bytes.data(), bytes.size())); } -SourceEdgeRunView openSourceEdgeRun(Backend & backend, const String & key) +SourceEdgeRunView openSourceEdgeRun(CasOperation & op, const String & key) { - /// Streaming: `getStream` is a forward-only read of the write-once run — nothing is - /// materialized whole (cas_run is object_cap = 0). Absent object => fail-closed. - auto sr = backend.getStream(key); - if (!sr) + /// Streaming: a forward-only read of the write-once run — nothing is materialized whole + /// (cas_run is object_cap = 0). Absent object => fail-closed. + auto stream = op.stream(key, Retry::standard()); + if (!stream) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS source-edge run: object {} is absent", key); - return SourceEdgeRunView(std::move(sr->stream)); + return SourceEdgeRunView(std::move(stream)); } namespace @@ -332,27 +330,39 @@ void SourceEdgeKeyCodec::parse(std::string_view key, BlobRef & ref, UInt128 & so source_id = u128FromBytesBE(String(key.substr(1 + digest_len, 16)), "src-edge run key source_id"); } -void putDeterministicArtifact(Backend & backend, const String & key, const String & bytes) +void putDeterministicArtifact(CasOperation & op, const String & key, const String & bytes) { - if (backend.putIfAbsent(key, bytes).outcome == PutOutcome::PreconditionFailed) + WriteResult result = op.create(key, bytes, Retry::standard()); + if (const auto * conflict = std::get_if(&result)) { - const auto existing = backend.get(key); - if (!existing || existing->bytes != bytes) + /// Only something the resolve read actually OBSERVED can support a corruption verdict. + if (const auto * occupant = std::get_if(&conflict->seen)) + { + if (occupant->bytes == bytes) + return; /// our own deterministic replay; adopt (no-op). throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc: deterministic artifact at {} occupied by divergent bytes (impossible under " "correct operation; refusing to proceed)", key); - /// byte-equal => our own deterministic replay; adopt (no-op). + } + if (std::holds_alternative(conflict->seen)) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS gc: deterministic artifact at {} refused the write but reads as absent (impossible " + "under correct operation; refusing to proceed)", key); + /// Nothing was observed, so the key's state is unknown. Reporting that as corruption would be a + /// deterministic local failure, which nothing above this retries -- it falls through instead. } + const String what = "CAS gc: deterministic artifact at " + key; + orThrow(std::move(result), what); } -void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, +void foldDeltasIntoGeneration(CasOperation & op, const Layout & layout, const std::vector & prior_runs, uint64_t new_generation, uint64_t attempt, uint64_t shard, std::vector scattered, std::vector & out_runs, uint64_t current_round, uint64_t condemn_round, - const std::function(const BlobRef &)> & head_blob, - const std::function(const BlobRef &)> & peek_head, + const BlobHeadFn & head_blob, + const BlobHeadFn & peek_head, const std::function & confirm_condemned_marker, RetiredMergeResult * out_retired, bool suppress_destructive, @@ -383,7 +393,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, return a.source_id < b.source_id; }); - PriorEdgeCursor cursor(backend, prior_runs); + PriorEdgeCursor cursor(op, prior_runs); DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); // sorted NDJSON; byte-deterministic for write-once adoption @@ -421,7 +431,7 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, /// 1. the round-paced floor: a blob condemned in round R cannot graduate before R+1, so a `+1` /// that lands in the same round as the condemnation is always folded before any delete; /// 2. the exact-token delete: a writer that resurrected the blob replaced its incarnation, so a - /// stale token's delete finds a TokenMismatch and removes nothing; + /// stale entry's delete matches nothing and removes nothing; /// 3. THIS: the entry is settled against `indeg` recomputed by the merge that just ran, so an edge /// folded after the condemnation but before the delete pass spares the blob outright -- /// `indeg > 0` wins over `delete_pending`, unconditionally and past the floor. @@ -531,12 +541,12 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, if (cur_edges == 0 && cur_touched && peek_head) { if (const auto hr = peek_head(cur_blob); - hr && hr->exists && hr->token != stale.token) + hr && !stale.token.matches(hr->incarnation)) { RetiredEntry fresh; fresh.kind = ObjectKind::Blob; fresh.ref = cur_blob; - fresh.token = hr->token; + fresh.token = PersistedIncarnation::capture(hr->incarnation); fresh.size = hr->size; fresh.condemn_round = condemn_round; ReplacedEntry re; @@ -554,12 +564,12 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, /// incarnation token for the later exact-token delete; an absent object needs no entry. else if (cur_edges == 0 && cur_touched && head_blob) { - if (const auto hr = head_blob(cur_blob); hr && hr->exists) + if (const auto hr = head_blob(cur_blob); hr) { RetiredEntry fresh; fresh.kind = ObjectKind::Blob; fresh.ref = cur_blob; - fresh.token = hr->token; + fresh.token = PersistedIncarnation::capture(hr->incarnation); fresh.size = hr->size; fresh.condemn_round = condemn_round; rmr.still_retired.push_back(std::move(fresh)); @@ -681,12 +691,12 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, /// seal's RunRef.checksum and verified before any consumer acts on the run. const UInt128 run_checksum = sourceEdgeRunChecksum(run_bytes); const String run_key = layout.blobTargetRunKey(new_generation, attempt, shard, 0); - putDeterministicArtifact(backend, run_key, run_bytes); + putDeterministicArtifact(op, run_key, run_bytes); out_runs.push_back(RunRef{.key = run_key, .checksum = run_checksum, .shard = shard, .key_generation = new_generation}); } -std::vector zeroInDegree(Backend & backend, const std::vector & runs) +std::vector zeroInDegree(CasOperation & op, const std::vector & runs) { std::vector result; for (const RunRef & run : runs) @@ -695,7 +705,7 @@ std::vector zeroInDegree(Backend & backend, const std::vector +#include #include #include #include @@ -26,7 +26,7 @@ struct RetiredEntry { ObjectKind kind = ObjectKind::Blob; BlobRef ref{}; - Token token; /// the exact incarnation token GC observed (exact-token delete) + PersistedIncarnation token; /// the exact incarnation GC observed; the delete re-heads and compares it uint64_t size = 0; uint64_t condemn_round = 0; /// the GC round that condemned this incarnation (round-paced /// graduation: an entry graduates only once condemn_round < the @@ -44,6 +44,10 @@ struct RetiredEntry /// graduation (delete_pending rows always carry it). }; +/// The fold's HEAD hooks (`head_blob`, `peek_head`): the caller issues the request on its own admitted +/// operation and reports what it saw, or nothing when the object is absent. +using BlobHeadFn = std::function(const BlobRef &)>; + /// Backend-independent codec for source-edge keys. A key is `algo` (u8), the digest at that algorithm's /// native width, and `source_id` (16 bytes, big-endian). The packed byte order is exactly /// `(BlobRef, source_id)` order, which lets the fold merge compare keys directly. The leading algorithm @@ -67,24 +71,34 @@ UInt128 sourceEdgeId(const ManifestId & id, const String & path); /// producers of real source edges must fail closed on a hash collision with it. void assertValidSourceEdgeId(const UInt128 & source_id); -/// Serialized payload of a condemned source-edge sentinel. The payload retains the full incarnation -/// token, including its type, because deletion must remain exact-token guarded. Its fixed prefix is -/// `[0x02][flags][token_type][round BE64][size BE64][token_len BE16]`, followed by token bytes. +/// Serialized payload of a condemned source-edge sentinel. The payload retains the full incarnation, +/// dialect included, because deletion must remain exact-token guarded. Its fixed prefix is +/// `[0x02][flags][token_type][round BE64][size BE64][token_len BE16]`, followed by token bytes; +/// `token_type` is the dialect byte of `CasWireVocab`'s vocabulary. /// `flags` bit 0 is `delete_pending`, bit 1 is `marker_confirmed`. struct CondemnedRow { bool delete_pending = false; - Token token; // {value, type} — the full token required by exact-token deletion + PersistedIncarnation token; // the incarnation the exact-token delete must re-observe uint64_t size = 0; uint64_t condemn_round = 0; bool marker_confirmed = false; // durable Condemned meta confirmed (graduation gate) - bool operator==(const CondemnedRow &) const = default; + /// Spelled out rather than defaulted because `PersistedIncarnation` carries no equality of its + /// own. A field added above belongs here too. + bool operator==(const CondemnedRow & o) const + { + return delete_pending == o.delete_pending + && token.dialect == o.token.dialect && token.value == o.token.value + && size == o.size && condemn_round == o.condemn_round + && marker_confirmed == o.marker_confirmed; + } }; -/// Encode a condemned-row payload. Throws `CORRUPTED_DATA` if the token cannot fit in its u16 length. +/// Encode a condemned-row payload. Throws `CORRUPTED_DATA` if the token cannot fit in its u16 length +/// or the dialect is not one this vocabulary knows. String encodeCondemnedRow(const CondemnedRow & row); -/// Decode and validate a condemned-row payload. Unknown flags, token types, or inconsistent lengths +/// Decode and validate a condemned-row payload. Unknown flags, dialect bytes, or inconsistent lengths /// throw `CORRUPTED_DATA`. CondemnedRow decodeCondemnedRow(std::string_view payload); @@ -112,7 +126,7 @@ class SourceEdgeRunView private: friend SourceEdgeRunView openSourceEdgeRun(std::string_view bytes); - friend SourceEdgeRunView openSourceEdgeRun(Backend & backend, const String & key); + friend SourceEdgeRunView openSourceEdgeRun(CasOperation & op, const String & key); /// Keep the underlying stream alive for the reader, which borrows it rather than owning it. explicit SourceEdgeRunView(std::unique_ptr stream_); @@ -122,19 +136,21 @@ class SourceEdgeRunView /// Open a typed source-edge run. The NDJSON header must identify a `cas_run` of kind `source_edge`; /// otherwise opening fails closed. The memory overload borrows caller-owned bytes. The backend overload -/// streams the write-once object through `getStream`, retaining only one record-sized buffer. +/// streams the write-once object: the reader itself holds one record-sized buffer, but the open stream +/// underneath it buffers on its own terms, unbounded and unmeasured here. SourceEdgeRunView openSourceEdgeRun(std::string_view bytes); -SourceEdgeRunView openSourceEdgeRun(Backend & backend, const String & key); +SourceEdgeRunView openSourceEdgeRun(CasOperation & op, const String & key); /// Store a deterministic write-once artifact (same inputs => byte-identical bytes): the blob in-degree -/// runs and fold seals. `putIfAbsent`; on a `PreconditionFailed` the key is already -/// occupied — `get` it and compare bytes: byte-equal means our own deterministic replay (adopt, no-op), -/// divergent bytes are impossible under correct operation and we fail closed with `CORRUPTED_DATA` -/// rather than let a divergent artifact disagree with the adopted snapshot. Deterministic artifacts are -/// therefore byte-equal-or-`CORRUPTED_DATA`. It is +/// runs and fold seals. `create`; a `Conflict` means the write was refused, and only what its resolve +/// read OBSERVED decides what that means: byte-equal bytes are our own deterministic replay (adopt, +/// no-op), divergent bytes or a key that reads as absent are impossible under correct operation and +/// fail closed with `CORRUPTED_DATA` rather than let a divergent artifact disagree with the adopted +/// snapshot, and an unobserved key proves nothing and is reported as the ordinary refusal it is -- +/// a corruption verdict there would be a deterministic local failure no caller retries. It is /// NOT for observation-bearing artifacts (outcome logs) — those carry HEAD-observed -/// tokens that two observers may legitimately differ on and keep first-durable-write-wins semantics. -void putDeterministicArtifact(Backend & backend, const String & key, const String & bytes); +/// incarnations that two observers may legitimately differ on and keep first-durable-write-wins semantics. +void putDeterministicArtifact(CasOperation & op, const String & key, const String & bytes); /// One source-edge update before merging: the edge `(ref, source_id)`, and whether it is an activation /// (+edge) or a removal (−edge). Idempotent under re-fold at the merge (set membership, not a counter). @@ -193,13 +209,13 @@ struct BlobCandidate /// seal's exact reference is authoritative and key construction is not used. `new_generation`, `attempt`, /// and `shard` name only the output run's key namespace. /// The fresh entry that re-condemns the current -/// token, paired with the STALE entry's token it superseded. Kept as its own struct (rather than a +/// incarnation, paired with the STALE entry's it superseded. Kept as its own struct (rather than a /// field bolted onto `RetiredEntry`) so the common merge element stays slim — only replaced entries -/// carry the extra superseded token. +/// carry the extra superseded incarnation. struct ReplacedEntry { - RetiredEntry fresh; /// the freshly condemned CURRENT token (also pushed into still_retired byte-identically) - Token old_token; /// the superseded (stale) entry's token — what republication replaced + RetiredEntry fresh; /// the freshly condemned CURRENT incarnation (also pushed into still_retired byte-identically) + PersistedIncarnation old_token; /// the superseded (stale) entry's — what republication replaced }; /// One example of an unmatched-remove delta, kept for the caller's single once-per-round WARNING @@ -218,7 +234,7 @@ struct RetiredMergeResult std::vector still_retired; /// carried + newly-condemned + newly-PENDING entries (the next list) std::vector graduated; /// newly floor-passed this pass — published pending, deleted NEXT pass std::vector spared; /// in-degree recovered — entry dropped - std::vector redelete; /// pending in the PRIOR list — execute deleteExact pre-CAS, drop + std::vector redelete; /// pending in the PRIOR list — execute the exact-incarnation delete pre-CAS, drop std::vector replaced; /// re-condemned CURRENT tokens that superseded a stale entry after republication; caller emits blob_retire_replaced /// Count of `remove == true` deltas that matched no presence for their `(BlobRef, source_id)` key — @@ -381,14 +397,14 @@ struct GcRoundWorkBudget /// for merge-mechanics unit tests only; the real GC round always passes the gate. /// The merge comparator is exactly `(ref.algo, ref.digest, source_id)` (that is, `BlobRef::operator<` /// followed by `source_id`), which is also the raw key order produced by `SourceEdgeKeyCodec`. -void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, +void foldDeltasIntoGeneration(CasOperation & op, const Layout & layout, const std::vector & prior_runs, uint64_t new_generation, uint64_t attempt, uint64_t shard, std::vector scattered, std::vector & out_runs, uint64_t current_round = 0, uint64_t condemn_round = 0, - const std::function(const BlobRef &)> & head_blob = {}, - const std::function(const BlobRef &)> & peek_head = {}, + const BlobHeadFn & head_blob = {}, + const BlobHeadFn & peek_head = {}, const std::function & confirm_condemned_marker = {}, RetiredMergeResult * out_retired = nullptr, bool suppress_destructive = false, @@ -408,6 +424,6 @@ void foldDeltasIntoGeneration(Backend & backend, const Layout & layout, /// shard) and return every blob written at in-degree 0 (the candidates that transitioned to zero). An /// empty `runs` is an empty baseline. Each `RunRef` supplies the exact object key, so resolution never /// reconstructs a key from generation metadata. -std::vector zeroInDegree(Backend & backend, const std::vector & runs); +std::vector zeroInDegree(CasOperation & op, const std::vector & runs); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index baaf7b86fdc8..283805bd892e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -26,6 +27,7 @@ #include #include #include +#include namespace ProfileEvents { @@ -81,9 +83,28 @@ void onGcEnumerationPage() /// Defined below; forward-declared so the post-CAS hand-off delete in `runRegularRound` can /// reach the same wholesale LIST-delete helper the retention prune uses. -uint64_t deletePrefixWholesale(Backend & backend, const String & prefix, uint64_t bounded_remaining, +uint64_t deletePrefixWholesale(CasOperation & op, const String & prefix, uint64_t bounded_remaining, bool * out_fully_drained = nullptr); +/// The text an incarnation carries into the event log. Persisted and live incarnations render the +/// same way, so the column speaks one vocabulary whichever half of the pipeline wrote the row. +String renderIncarnation(const PersistedIncarnation & token) +{ + return token.dialect + ":" + token.value; +} + +/// The label one removal carries into the event log and the outcomes audit rows. +std::string_view removalName(Removal removal) +{ + switch (removal) + { + case Removal::Removed: return "deleted"; + case Removal::Gone: return "absent"; + case Removal::Mismatch: return "replaced"; + } + UNREACHABLE(); +} + } std::set RefPlan::lifeIds() const @@ -321,18 +342,13 @@ void Gc::runNamespaceJanitorPage( NamespaceJanitorResult janitor_result; try { - Backend & backend = store->backend(); + CasRequests & requests = store->gcRequests(); const Layout & layout = store->layout(); - NamespaceJanitor janitor(backend, layout, 1000); - const uint64_t admitted_generation = leased_state.lease.seq; - janitor_result = janitor.runOnePage(suppress_destructive, [&] - { - const auto got = backend.get(layout.gcStateKey()); - if (!got) - return false; - const GcState current = decodeGcState(got->bytes); - return current.lease.owner == gc_id && current.lease.seq == admitted_generation; - }); + NamespaceJanitor janitor(requests, layout, 1000); + /// ONE authority read per page, made here rather than from the predicate: the janitor's + /// operation samples its liveness before every request, and a page walks up to a thousand keys. + refreshAuthority(leased_state.lease.seq); + janitor_result = janitor.runOnePage(suppress_destructive, [this] { return authority_held; }); for (const String & anomaly : janitor_result.anomalies) LOG_WARNING(logger, "CAS namespace janitor: {}", anomaly); if (janitor_result.leaked) @@ -355,7 +371,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al RoundReport & report = progress ? *progress : local_report; report = RoundReport{}; GcState state; - Token state_token; + std::optional state_incarnation; /// Every exit path waits for this round's meta jobs. The throwing `meta_pool_wait` phase below is /// a protocol barrier -- this round's condemns must be durable no later than the ledger they are @@ -370,7 +386,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// are correlated by `round_id` and not by the round number a follower never learns. { GcPhaseTimer t(phase_sink, "lease"); - report.acquired_lease = acquireOrRenewLease(state, state_token, allow_steal); + report.acquired_lease = acquireOrRenewLease(state, state_incarnation, allow_steal); t.metric("acquired", report.acquired_lease ? 1 : 0); t.metric("steal_allowed", allow_steal ? 1 : 0); } @@ -396,7 +412,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// attempt is idempotent. const Layout & layout = store->layout(); - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const uint64_t new_round = state.round + 1; /// ONE budget instance for the WHOLE round, threaded into every destructive-or-observability-write @@ -430,7 +446,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al t.metric("deleted", drain_result.deleted); } - /// Token-guarded fence-out of dead mounts (liveness only — graduation itself paces on GC + /// Incarnation-guarded fence-out of dead mounts (liveness only — graduation itself paces on GC /// rounds via `new_round`, not on heartbeat acks). Fencing no longer trusts a predecessor's stamped /// `expires_at_ms` against our wall clock — it /// fences ONLY once `mount_obs` has watched the mount's write-token hold unchanged for the full @@ -446,7 +462,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// PUT per newly-fenced mount. { GcPhaseTimer t(phase_sink, "heartbeat_floor"); - const HeartbeatFloor floor = computeHeartbeatFloor(backend, layout, now_ms_fn(), mono_ms_fn(), + const HeartbeatFloor floor = computeHeartbeatFloor(op, layout, now_ms_fn(), mono_ms_fn(), stable_threshold_ms, mount_obs); report.fence_outs = floor.fenced_now; if (floor.fenced_now > 0) @@ -542,7 +558,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al { ++rounds_since_last_fold_; report.deferred = true; - /// A DEFER round mints no new round -- unlike the fold path below (CasGc.cpp:642), which sets + /// A DEFER round mints no new round -- unlike the fold path below, which sets /// `report.round = state.round` only AFTER the round's single `gc/state` CAS has committed /// `next.round = new_round` and `state` was reassigned to that committed `next` (so on that /// path `state.round` reads the FRESH round number). Here the round CAS never runs, so `state` @@ -591,7 +607,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al }); /// Capture the PARENT seal's run refs BEFORE fold mutates - /// `state.snap_generation`/`snap_attempt` in-memory (CasGc.cpp:838). We compare these against the + /// `state.snap_generation`/`snap_attempt` in-memory. We compare these against the /// NEW seal's refs post-CAS to detect a ref that moved OFF an already-pruned generation (the /// wholesale prune skipped it while it was still referenced and its cursor advanced past it), and /// hand-off delete that generation's now-unreferenced leftover. Absent parent seal => empty. @@ -610,7 +626,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// The pass performs discovery, windowing, and the three-cursor merge (spare / graduate / condemn). /// It emits phases 5..10 of its own. - FoldResult folded = fold(state, state_token, report, new_round, *walk_plan, policy, round_work_budget); + FoldResult folded = fold(state, state_incarnation, report, new_round, *walk_plan, policy, round_work_budget); /// THE ROUND'S DESTRUCTIVE GATE, read once, here, and consulted at EVERY destructive site below. /// It is available this early because `fold` computes it (see `FoldResult::suppress_destructive`), @@ -668,52 +684,37 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al suppress_destructive ? kNothingToDelete : merge.redelete; for (const RetiredEntry & entry : redelete_now) { - DeleteOutcome del = backend.deleteExact(layout.blobKey(entry.ref), entry.token); - if (del.created_delete_marker) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS gc: delete of blob {} created a delete marker — versioning is enabled " - "on the pool (mis-provisioned; the capability probe must reject this)", blobIdOf(entry.ref)); - - /// A RustFS quirk: a conditional delete (`If-Match`) against an ABSENT - /// object can answer HTTP 412 (precondition failed) instead of 404 — we map that 412 to - /// TokenMismatch. Backend-agnostically disambiguate here: a genuine TokenMismatch means the - /// object exists under a different (fresh) token; if a follow-up HEAD shows the object is - /// gone, the "mismatch" was actually the object being absent — treat it as Absent (NotFound) - /// end-to-end so the `.meta` cleanup below still runs. - bool absent_on_mismatch_quirk = false; - if (del.kind == DeleteOutcome::Kind::TokenMismatch) - { - const HeadResult head = backend.head(layout.blobKey(entry.ref)); - if (!head.exists) - { - del.kind = DeleteOutcome::Kind::NotFound; - absent_on_mismatch_quirk = true; - } - } - - const DeleteClass del_class = classifyDeleteOutcome(del); - const OutcomeKind outcome_kind = del_class == DeleteClass::Deleted ? OutcomeKind::Deleted - : del_class == DeleteClass::Absent ? OutcomeKind::Absent - : OutcomeKind::Replaced; + /// The condemned incarnation is a PERSISTED pair and cannot itself be a precondition, so + /// the round observes the blob and compares the two renderings. Observing first also + /// settles the absent case without spending a conditional delete against a key that is + /// already gone. + const String blob_key = layout.blobKey(entry.ref); + const std::optional observed = op.head(blob_key, Retry::standard()); + Removal del = Removal::Gone; + if (observed) + del = entry.token.matches(observed->incarnation) + ? op.remove(blob_key, observed->incarnation, Retry::standard()) + : Removal::Mismatch; + + const OutcomeKind outcome_kind = del == Removal::Removed ? OutcomeKind::Deleted + : del == Removal::Gone ? OutcomeKind::Absent + : OutcomeKind::Replaced; OutcomeEntry outcome{.kind = entry.kind, .ref = entry.ref, .token = entry.token, .outcome = outcome_kind}; - const String del_outcome{deleteClassName(del_class)}; - /// The single content-delete site is attributable per row. TokenMismatch (a writer - /// recreated the incarnation) is terminal-OK: the fresh incarnation is a live object. + const String del_outcome{removalName(del)}; + /// The single content-delete site is attributable per row. A mismatch (a writer recreated + /// the incarnation) is terminal-OK: the fresh incarnation is a live object. EventEmitter{*store}.emit([&](CasEvent & e) { e.type = CasEventType::BlobDelete; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(entry.ref); - e.token = entry.token.value; + e.token = renderIncarnation(entry.token); e.round = new_round; e.gen = generation; e.outcome = del_outcome; - e.reason = absent_on_mismatch_quirk - ? "delete_pending published by a prior pass; exact-token delete (pre-CAS) " - "(delete returned token-mismatch but the object is absent — backend 412-on-absent quirk)" - : "delete_pending published by a prior pass; exact-token delete (pre-CAS)"; + e.reason = "delete_pending published by a prior pass; exact-incarnation delete (pre-CAS)"; e.detail = {{"condemn_round", std::to_string(entry.condemn_round)}, - {"key", layout.blobKey(entry.ref)}}; + {"key", blob_key}}; }); /// The audit row is observability only -- the delete above already executed regardless of /// this cap. Skipping it here bounds the per-shard `GcOutcomes` body without skipping or @@ -725,12 +726,12 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al } ++report.redeleted; ProfileEvents::increment(ProfileEvents::CASGCRetiredRedeleted); - /// Drop the per-hash meta only on Deleted/NotFound — a Replaced (TokenMismatch) outcome - /// means a writer already resurrected a fresh incarnation at this hash (INV-1), and that - /// writer's own republication path already flipped the meta back to Clean; blindly deleting here - /// would race that legitimate Clean write for no reason (the meta is advisory, but there is no - /// reason to touch it on that path at all). - if (del_class == DeleteClass::Deleted || del_class == DeleteClass::Absent) + /// Drop the per-hash meta only on a removal or a proven absence — a mismatch means a + /// writer already resurrected a fresh incarnation at this hash (INV-1), and that writer's + /// own republication path already flipped the meta back to Clean; blindly deleting here + /// would race that legitimate Clean write for no reason (the meta is advisory, but there is + /// no reason to touch it on that path at all). + if (del == Removal::Removed || del == Removal::Gone) { meta_writer->scheduleConfirmedMetaDelete(entry.ref); } @@ -753,7 +754,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al e.type = CasEventType::GcRecheckVerdict; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(entry.ref); - e.token = entry.token.value; + e.token = renderIncarnation(entry.token); e.round = new_round; e.gen = generation; e.outcome = "spared"; @@ -793,7 +794,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al e.type = CasEventType::GcRecheckVerdict; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(entry.ref); - e.token = entry.token.value; + e.token = renderIncarnation(entry.token); e.round = new_round; e.gen = generation; e.outcome = "pending"; @@ -815,13 +816,13 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al e.type = CasEventType::BlobRetireReplaced; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(entry.ref); - e.token = entry.token.value; + e.token = renderIncarnation(entry.token); e.round = new_round; e.gen = generation; e.outcome = "replaced"; e.reason = "current object token differs from the retired entry — republication replaced the " "incarnation; superseded the stale entry and re-condemned the current token"; - e.detail = {{"superseded_token", replaced.old_token.value}}; + e.detail = {{"superseded_token", renderIncarnation(replaced.old_token)}}; }); /// The supersede is ALSO a blob entering the retired set fresh (a re-condemn of the /// CURRENT token) — write the meta Condemned exactly like a fresh `head_blob` condemn would, @@ -840,12 +841,14 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al { const String key = layout.outcomesKey(generation, attempt, new_round, shard); const String body = sealObject(FormatId::GcOutcomes, encodeOutcomeLog(log)); - if (backend.putIfAbsent(key, body).outcome == PutOutcome::PreconditionFailed) + WriteResult written = op.create(key, body, Retry::standard()); + if (const auto * conflict = std::get_if(&written)) { - const auto existing = backend.get(key); + /// The conflict's observation IS the read that used to follow the refused create. + const auto * existing = std::get_if(&conflict->seen); if (!existing) throw Exception(ErrorCodes::ABORTED, - "CAS gc: outcome log at {} vanished between putIfAbsent and read", key); + "CAS gc: outcome log at {} vanished between the create and the read that settled it", key); if (existing->bytes != body) { try { log = decodeOutcomeLog(openObject(FormatId::GcOutcomes, existing->bytes)); } @@ -856,6 +859,8 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al } } } + else + orThrow(std::move(written), fmt::format("CAS gc: outcome log at {}", key)); for (const OutcomeEntry & o : log.entries) { switch (o.outcome) @@ -941,12 +946,13 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al round_commit_timer->metric("generations_visited", next.snap_pruned_through - pruned_through_before); round_commit_timer->metric("pruned_through", next.snap_pruned_through); round_commit_timer->metric("generations_referenced", referenced_generations.size()); - const CasResult res = backend.casPut(layout.gcStateKey(), encodeGcState(next), state_token); - if (res.outcome != CasOutcome::Committed) + WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_incarnation, + Retry::standard()); + if (std::holds_alternative(commit)) throw Exception(ErrorCodes::ABORTED, "CAS gc round: gc/state moved during the round (another leader advanced it); retry next round"); + state_incarnation = orThrow(std::move(commit), "CAS gc round commit"); state = std::move(next); - state_token = res.token; report.round = state.round; round_commit_timer->metric("round", report.round); round_commit_timer->metric("generation", generation); @@ -964,7 +970,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// Post-CAS reference-parent HAND-OFF DELETE. `pruneSupersededGenerations` SKIPS a /// generation the live seal still references AND advances `snap_pruned_through` PAST it - /// (CasGc.cpp:1066 computes the cursor as `g - 1` after the loop increments `g` over every skipped + /// (`pruneSupersededGenerations` computes the cursor as `g - 1` after the loop increments `g` over every skipped /// generation). So once a skipped generation is behind the cursor, the wholesale prune NEVER revisits /// it — a ref that later moves off it would strand that generation's WHOLE prefix (fold seal, retired/ /// outcomes sets, all shards' runs), not just the single carried run object. Reclaim it HERE, now that @@ -1021,7 +1027,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al if (remaining == 0) break; const uint64_t reclaimed = deletePrefixWholesale( - backend, layout.gcGenPrefix(old_ref.key_generation), remaining); + op, layout.gcGenPrefix(old_ref.key_generation), remaining); round_work_budget.handoff_prefix_wholesale_objects_used += reclaimed; objects_reclaimed += reclaimed; LOG_TRACE(logger, @@ -1051,16 +1057,17 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// sealed AND taken on a round that could prove its frontier -- an unprovable round's `-1` may /// itself be the observation that is missing an owner elsewhere, so deleting the body on it is /// exactly the irreversible step the gate exists to withhold. - static const std::map kNoManifestCleanup; - const std::map & mf_cleanup_now = + static const std::map kNoManifestCleanup; + const std::map & mf_cleanup_now = suppress_destructive ? kNoManifestCleanup : folded.mf_cleanup; uint64_t attempted = 0; - for (const auto & [id, token] : mf_cleanup_now) + for (const auto & [id, incarnation] : mf_cleanup_now) { ++attempted; - const DeleteOutcome mdel = backend.deleteExact(layout.manifestKey(id), token); /// NotFound/TokenMismatch tolerated - const DeleteClass mdel_class = classifyDeleteOutcome(mdel); - if (mdel_class == DeleteClass::Deleted) + /// Gone and Mismatch are both tolerated: the body is already reclaimed, or a live + /// incarnation replaced the one this round's fold observed. + const Removal mdel = op.remove(layout.manifestKey(id), incarnation, Retry::standard()); + if (mdel == Removal::Removed) ++report.manifests_deleted; EventEmitter{*store}.emit([&](CasEvent & e) { @@ -1068,11 +1075,11 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al e.namespace_ = id.root_namespace.string(); e.object_kind = CasEventObjectKind::Manifest; e.object_hash = manifestRefDebugString(id.ref); - e.token = token.value; + e.token = incarnation.render(); e.round = new_round; e.gen = generation; - e.outcome = String{deleteClassName(mdel_class)}; - e.reason = "owner-removed manifest body; exact-token delete after decrements adopted"; + e.outcome = String{removalName(mdel)}; + e.reason = "owner-removed manifest body; exact-incarnation delete after decrements adopted"; }); } t.metric("attempted", attempted); @@ -1108,25 +1115,31 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al ManifestSweepResult & sweep = folded.orphan_sweep; for (const ManifestSweepResult::Nomination & nomination : sweep.nominations) { - const DeleteOutcome outcome = backend.deleteExact(nomination.key, nomination.token); - const DeleteClass outcome_class = classifyDeleteOutcome(outcome); + /// The nominated incarnation is persisted, so the body is observed and the two renderings + /// compared before the removal names a precondition. + const std::optional observed = op.head(nomination.key, Retry::standard()); + Removal outcome = Removal::Gone; + if (observed) + outcome = nomination.token.matches(observed->incarnation) + ? op.remove(nomination.key, observed->incarnation, Retry::standard()) + : Removal::Mismatch; EventEmitter{*store}.emit([&](CasEvent & e) { e.type = CasEventType::ManifestDelete; e.namespace_ = nomination.id.root_namespace.string(); e.object_kind = CasEventObjectKind::Manifest; e.object_hash = nomination.key; - e.token = nomination.token.value; + e.token = renderIncarnation(nomination.token); e.round = new_round; e.gen = generation; - e.outcome = String{deleteClassName(outcome_class)}; - e.reason = "orphan-manifest sweep: source edges retired and adopted before exact-token delete"; + e.outcome = String{removalName(outcome)}; + e.reason = "orphan-manifest sweep: source edges retired and adopted before exact-incarnation delete"; }); - if (outcome.kind == DeleteOutcome::Kind::TokenMismatch) + if (outcome == Removal::Mismatch) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS orphan sweep: manifest key {} changed token after exact GET; immutable manifest " - "identity suffered illegal ABA, retained replacement", nomination.key); - if (outcome_class == DeleteClass::Deleted) + "CAS orphan sweep: manifest key {} changed incarnation after the exact read; immutable " + "manifest identity suffered illegal ABA, retained replacement", nomination.key); + if (outcome == Removal::Removed) ++sweep.deleted; else ++sweep.skipped; @@ -1167,10 +1180,9 @@ void Gc::reportStuckRemovals(const RefPlan & plan, uint64_t current_round) } } -bool Gc::foldManifestEdges(const ManifestId & id, int sign, std::vector & deltas, - std::map & mf_cleanup, uint32_t txn_ordinal) +bool Gc::foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, std::vector & deltas, + std::map & mf_cleanup, uint32_t txn_ordinal) { - Backend & backend = store->backend(); const Layout & layout = store->layout(); const String key = layout.manifestKey(id); @@ -1180,7 +1192,7 @@ bool Gc::foldManifestEdges(const ManifestId & id, int sign, std::vector fail closed) ProfileEvents::increment(ProfileEvents::CASRefManifestBodyFoldGets); /// one body GET per manifest fold @@ -1246,7 +1258,7 @@ bool Gc::foldManifestEdges(const ManifestId & id, int sign, std::vectortoken); /// owner removed: defer exact-token body delete to recheck + mf_cleanup.emplace(id, got->incarnation); /// owner removed: defer the exact body delete to recheck return true; } @@ -1262,7 +1274,7 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::mapbackend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); std::set witness_namespaces; @@ -1291,7 +1303,7 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::map got = backend.get(ckpt_key); + const std::optional got = op.read(ckpt_key, Retry::standard()); /// ABSENT IS NORMAL AND IS NOT A WITNESS: a namespace has no `_ckpt` until its first snapshot /// publication commits, and one that 404s mid-round is a namespace being reclaimed. Neither says /// anything about which ids exist, so neither may hold the walk -- and neither may throw @@ -1332,7 +1344,7 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::map> Gc::newestFoldSealRef() { - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); const String gen_prefix = layout.gcGenPrefix(0); const String top = gen_prefix.substr(0, gen_prefix.size() - 2); /// ".../gc/gen/" @@ -1344,13 +1356,13 @@ std::optional> Gc::newestFoldSealRef() std::set listed_generations; bool listed_anything = false; std::optional> newest; - forEachListedKey(backend, top, [&](const ListedKey & k) + op.forEachListedKey(top, [&](const KeyEntry & k) { listed_anything = true; const size_t from = top.size(); const size_t gen_end = k.key.find('/', from); if (gen_end == String::npos) - return; + return true; uint64_t generation = 0; try { @@ -1358,10 +1370,11 @@ std::optional> Gc::newestFoldSealRef() } catch (...) // NOLINT(bugprone-empty-catch) { - return; /// foreign key shape under `gc/gen` is debris, not a generation number + return true; /// foreign key shape under `gc/gen` is debris, not a generation number } listed_generations.insert(generation); - }, 1000, onGcEnumerationPage); + return true; + }, Retry::standard(), 1000, onGcEnumerationPage); const uint64_t listed_max_generation = listed_generations.empty() ? 0 : *listed_generations.rbegin(); /// STEP DOWN THROUGH THE GENERATIONS THE LISTING ITSELF REPORTED until one carries a seal. The @@ -1463,11 +1476,11 @@ std::optional> Gc::newestFoldSealRef() Gc::GenerationSealProbe Gc::probeGenerationForSeal(uint64_t generation) { - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); GenerationSealProbe probe; - forEachListedKey(backend, layout.gcGenPrefix(generation), [&](const ListedKey & k) + op.forEachListedKey(layout.gcGenPrefix(generation), [&](const KeyEntry & k) { probe.generation_exists = true; /// ANY object proves this generation was minted /// Parse a candidate attempt out of the path and then PROVE it by rebuilding the key: only a @@ -1477,11 +1490,11 @@ Gc::GenerationSealProbe Gc::probeGenerationForSeal(uint64_t generation) static constexpr std::string_view kAttempt = "/attempt/"; const size_t a_begin = k.key.find(kAttempt); if (a_begin == String::npos) - return; + return true; const size_t a_from = a_begin + kAttempt.size(); const size_t a_end = k.key.find('/', a_from); if (a_end == String::npos) - return; + return true; uint64_t attempt = 0; try { @@ -1489,13 +1502,14 @@ Gc::GenerationSealProbe Gc::probeGenerationForSeal(uint64_t generation) } catch (...) // NOLINT(bugprone-empty-catch) { - return; /// foreign key shape is debris, not an attempt + return true; /// foreign key shape is debris, not an attempt } if (layout.foldSealKey(generation, attempt) != k.key) - return; + return true; if (!probe.seal_attempt || *probe.seal_attempt < attempt) probe.seal_attempt = attempt; - }, 1000, onGcEnumerationPage); + return true; + }, Retry::standard(), 1000, onGcEnumerationPage); return probe; } @@ -1539,11 +1553,12 @@ void Gc::FoldResult::FrontierDeficit::count(FrontierUnproven reason) } } -Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & report, +Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_incarnation*/, + RoundReport & report, uint64_t current_round, const RefPlan & walk_plan, UniversePolicy policy, GcRoundWorkBudget & work_budget) { - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); FoldResult result; @@ -1586,13 +1601,13 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// `cleanupRefObjects` and terminal-evidence attribution -- retains both the chosen incarnation and /// the lifecycle/absence distinction instead of re-reading or reducing the catalog independently. result.catalog_cut = catalog_snapshot; - /// THE POSITIVE EMPTY-UNIVERSE PROOF (see the destructive gate below). `token` is guaranteed by + /// THE POSITIVE EMPTY-UNIVERSE PROOF (see the destructive gate below). `incarnation` is guaranteed by /// `CasRefCatalog::read` on every operational path -- absence there is `CORRUPTED_DATA`, never an /// empty snapshot -- but the check stays here so this fails closed if a bootstrap/test snapshot /// ever reaches this line. `entries` (not `live_incarnation`, which drops `Creating`) is the right /// source: a catalog holding only `Creating` rows must NOT read as an empty universe, and `entries` /// is the one view that still carries those rows. - result.catalog_cut_proved_empty = catalog_snapshot.token.has_value() && catalog_snapshot.catalog.entries.empty(); + result.catalog_cut_proved_empty = catalog_snapshot.incarnation.has_value() && catalog_snapshot.catalog.entries.empty(); /// A malformed ref-object key or namespace aborts ref folding for the whole round: the /// round produces no ref delta, advances no cursor, and authorizes no destructive work -- recorded as @@ -1675,7 +1690,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// Condemn-time observation: ONE HEAD per new zero-transition captures the exact incarnation token /// the eventual delete carries (absent => a prior landed delete => nothing to condemn). Emits the /// Candidate trail (IndegZero / GcRetireObserve / BlobRetire) exactly where the decision is made. - const auto head_blob = [&](const BlobRef & ref) -> std::optional + const auto head_blob = [&](const BlobRef & ref) -> std::optional { EventEmitter{*store}.emit([&](CasEvent & e) { @@ -1686,19 +1701,19 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & e.gen = state.snap_generation + 1; e.reason = "last folded owner edge dropped; in-degree reached 0"; }); - const HeadResult observed = backend.head(layout.blobKey(ref)); + const std::optional observed = op.head(layout.blobKey(ref), Retry::standard()); EventEmitter{*store}.emit([&](CasEvent & e) { e.type = CasEventType::GcRetireObserve; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(ref); - e.token = observed.exists ? observed.token.value : ""; + e.token = observed ? observed->incarnation.render() : ""; e.round = condemn_round; e.gen = state.snap_generation + 1; - e.outcome = observed.exists ? "present" : "absent"; - e.reason = "zero-in-degree candidate; HEAD-observe the current token"; + e.outcome = observed ? "present" : "absent"; + e.reason = "zero-in-degree candidate; observe the current incarnation"; }); - if (!observed.exists) + if (!observed) return std::nullopt; ++report.candidates; ++report.condemned; @@ -1708,21 +1723,22 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & e.type = CasEventType::BlobRetire; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(ref); - e.token = observed.token.value; + e.token = observed->incarnation.render(); e.round = condemn_round; e.gen = state.snap_generation + 1; e.outcome = "retired"; e.reason = "condemned zero-in-degree candidate; entering the current retired list"; }); - HeadResult adjusted = observed; - adjusted.size = retiredLogicalSize(ObjectKind::Blob, observed.size, store->poolMeta().blob_header_len); + Meta adjusted = *observed; + adjusted.size = retiredLogicalSize(ObjectKind::Blob, observed->size, store->poolMeta().blob_header_len); /// This candidate unconditionally becomes a fresh `RetiredEntry` in `closeBlob` (the ONLY /// caller of `head_blob`) whenever this lambda returns a value — so this is exactly the round's /// side-effecting condemn site. Write the meta Condemned so the writer's point-read gate /// sees it; a successful write records the in-process (hash, token) confirmation the graduation /// gate consumes (`scheduleCondemnMarkerWrite` captures everything BY VALUE — never by reference /// to `cur_blob`, which the fold's tight streaming loop mutates while the job is queued). - meta_writer->scheduleCondemnMarkerWrite(ref, observed.token, condemn_round, adjusted.size); + meta_writer->scheduleCondemnMarkerWrite(ref, PersistedIncarnation::capture(observed->incarnation), + condemn_round, adjusted.size); return adjusted; }; @@ -1732,12 +1748,12 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// hook is `head_blob` above, reserved for a genuinely NEW zero-in-degree candidate. A supersede's /// own event is `blob_retire_replaced`, emitted once below from `merge.replaced`. Plain HEAD, no /// events, no counters. - const auto peek_head = [&](const BlobRef & ref) -> std::optional + const auto peek_head = [&](const BlobRef & ref) -> std::optional { - HeadResult hr = backend.head(layout.blobKey(ref)); - if (!hr.exists) + std::optional hr = op.head(layout.blobKey(ref), Retry::standard()); + if (!hr) return std::nullopt; - hr.size = retiredLogicalSize(ObjectKind::Blob, hr.size, store->poolMeta().blob_header_len); + hr->size = retiredLogicalSize(ObjectKind::Blob, hr->size, store->poolMeta().blob_header_len); return hr; }; @@ -1757,7 +1773,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & return true; try { - if (const auto lm = loadMeta(backend, layout, entry.ref); lm && lm->meta.state == MetaState::Condemned) + if (const auto lm = loadMeta(op, layout, entry.ref); lm && lm->meta.state == MetaState::Condemned) { meta_writer->noteCondemnMarkerDurable(entry.ref, entry.token); /// memoize for a round-CAS-abort replay return true; @@ -1775,7 +1791,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// resurrected this exact hash under a FRESH token between the original swallowed write and this /// retry, the retry stamps `Condemned` over that writer's live, uncondemned incarnation. This is /// never destructive -- the eventual exact-token delete is a no-op against the fresh token - /// (`DeleteOutcome::TokenMismatch`/`NotFound`) -- worst case the resurrecting writer's later + /// (it finds a different incarnation, or none) -- worst case the resurrecting writer's later /// same-token adopter sees stale `Condemned` metadata and republishes once unnecessarily. meta_writer->scheduleCondemnMarkerWrite(entry.ref, entry.token, entry.condemn_round, entry.size); return false; @@ -2196,7 +2212,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & const RefTxnId & witness) -> std::optional { const EpochCrossResult crossing = - crossEpochFromSeal(backend, layout, ns, from_seal, seal_proven, witness, life); + crossEpochFromSeal(op, layout, ns, from_seal, seal_proven, witness, life); intake_absent_probes += crossing.absent_probes; /// a failed crossing pays its reads too ProfileEvents::increment(ProfileEvents::CASRefLogBodyGets, crossing.body_gets); if (crossing.outcome == EpochCrossOutcome::StartInvalid) @@ -2303,7 +2319,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// GET + decode the expected record. Absence is the decision point of the whole walk, and /// an invalid body is a per-namespace hold: the key belongs to exactly one namespace, so it /// can never be grounds for discarding another namespace's fold. - const auto got = backend.get(layout.refLogKey(life, *expected)); + const auto got = op.read(layout.refLogKey(life, *expected), Retry::standard()); if (!got) { ++intake_absent_probes; @@ -2395,11 +2411,11 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// CLAMP (barrier), never a round abort: keep the cursor below THIS log and re-read it next /// round. A removed precommit whose body never existed emitted no edge -- skip, no clamp. std::vector log_deltas; - std::map log_mf_cleanup; + std::map log_mf_cleanup; for (const RefManifestEdge & edge : edges) { ProfileEvents::increment(ProfileEvents::CASRefEmittedEdges); /// one manifest-edge event - if (foldManifestEdges(edge.manifest_id, edge.change, log_deltas, log_mf_cleanup, + if (foldManifestEdges(op, edge.manifest_id, edge.change, log_deltas, log_mf_cleanup, txn_ordinal)) continue; @@ -3022,7 +3038,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & { /// Either a real delta or a non-empty retired input: run the merge (empty deltas still settle /// the RunMarker::Condemned rows riding the parent run). The prior runs are the parent seal's shard-0 refs. - foldDeltasIntoGeneration(backend, layout, priorRunsFor(0), + foldDeltasIntoGeneration(op, layout, priorRunsFor(0), new_generation, attempt, /*shard*/0, std::move(deltas), result.fold_seal.blob_target_runs, current_round, condemn_round, head_blob, peek_head, @@ -3065,7 +3081,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & /// shards concurrently (CasGcScheduler ownership); their run-key namespaces never collide. std::vector shard_runs; foldDeltasIntoGeneration( - backend, layout, priorRunsFor(shard), new_generation, attempt, shard, + op, layout, priorRunsFor(shard), new_generation, attempt, shard, std::move(buckets[shard]), shard_runs, current_round, condemn_round, head_blob, peek_head, confirm_condemned_marker, @@ -3192,7 +3208,7 @@ Gc::FoldResult Gc::fold(GcState & state, Token & /*state_token*/, RoundReport & t.metric("seal_cleanup_evidence", std::count_if( result.fold_seal.ref_lives.begin(), result.fold_seal.ref_lives.end(), [](const auto & item) { return item.second.cleanup_evidence.has_value(); })); - putDeterministicArtifact(backend, layout.foldSealKey(new_generation, attempt), seal_body); + putDeterministicArtifact(op, layout.foldSealKey(new_generation, attempt), seal_body); } /// One-pass round: the fold NO LONGER CASes gc/state. (new_generation, attempt) are adopted @@ -3249,7 +3265,7 @@ void Gc::cleanupRefObjects( if (suppress_destructive) return; - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); if (!folded.catalog_cut) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS GC ref cleanup: fold result carries no catalog cut"); @@ -3272,25 +3288,26 @@ void Gc::cleanupRefObjects( /// Current-life ref cleanup is not the dead-life janitor: every irreversible key delete must /// still be licensed by the SAME complete catalog observation and GC lease that adopted the - /// fold. Re-read both after the target HEAD and immediately before `deleteExact`. A moved token, - /// changed row/life, missing or unreadable authority object, or changed owner/sequence stops the - /// whole cleanup pass. Continuing with another row/key would turn a refusal into a fallback. + /// fold. Re-read both after the target observation and immediately before the removal. A moved + /// incarnation, changed row/life, missing or unreadable authority object, or changed + /// owner/sequence stops the whole cleanup pass. Continuing with another row/key would turn a + /// refusal into a fallback. const auto deleteRefObject = [&](const String & key) { - const HeadResult h = backend.head(key); - if (!h.exists) + const std::optional h = op.head(key, Retry::standard()); + if (!h) return true; try { - const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(op, layout); current_catalog.life_index.throwIfAmbiguous("CAS GC ref cleanup revalidation"); const auto current_entry_it = std::lower_bound( current_catalog.catalog.entries.begin(), current_catalog.catalog.entries.end(), ns, [](const CatalogEntry & entry, const RootNamespace & needle) { return entry.ns < needle; }); const std::optional current_life = current_catalog.life_index.resolve(life.incarnation); - if (current_catalog.token != folded.catalog_cut->token + if (current_catalog.incarnation != folded.catalog_cut->incarnation || current_entry_it == current_catalog.catalog.entries.end() || current_entry_it->ns != ns || *current_entry_it != observed_entry || !current_life || *current_life != life) @@ -3301,7 +3318,7 @@ void Gc::cleanupRefObjects( return false; } - const auto current_state_object = backend.get(layout.gcStateKey()); + const auto current_state_object = op.read(layout.gcStateKey(), Retry::standard()); if (!current_state_object) { LOG_WARNING(logger, @@ -3327,7 +3344,7 @@ void Gc::cleanupRefObjects( return false; } - backend.deleteExact(key, h.token); + op.remove(key, h->incarnation, Retry::standard()); ProfileEvents::increment(ProfileEvents::CASRefCleanupObjectsDeleted); /// cleanup object deletion return true; }; @@ -3351,7 +3368,7 @@ void Gc::cleanupRefObjects( std::optional retained_log_proof; try { - retained_log_proof = readCheckpointSnapshotBase(backend, layout, life, *checkpoint).predecessor_seal_id; + retained_log_proof = readCheckpointSnapshotBase(op, layout, life, *checkpoint).predecessor_seal_id; } catch (const Exception & e) { @@ -3396,12 +3413,12 @@ namespace /// GC-metadata wholesale delete of every object under `prefix`. Returns the number of objects deleted. /// `bounded_remaining` caps how many objects this call may delete (0 => stop immediately, deleting none). /// -/// Token source: the in-memory and S3 backends surface a per-key token through `list` -/// (`supportsListTokens()`), so `deleteExact` straight from the listed token; otherwise HEAD first. +/// Precondition source: the in-memory and S3 backends surface a per-key incarnation through `list` +/// (`supportsListTokens()`), so the removal goes straight from the listed one; otherwise observe first. /// -/// 404 / NotFound is FAIL-OPEN: an object that vanished between LIST and delete (a concurrent crashed +/// An absence is FAIL-OPEN: an object that vanished between LIST and delete (a concurrent crashed /// attempt, or a racing prune) is already reclaimed — never throw on a benign missing GC-internal object -/// during a prune (it would only wedge GC). A genuine TokenMismatch is +/// during a prune (it would only wedge GC). A genuine mismatch is /// likewise tolerated here: the object was rewritten under us (another attempt is live at this key) — the /// safe direction during a best-effort prune is to leave it for a later round, never to force-delete. /// `out_fully_drained`, when set, reports whether the WHOLE prefix was exhausted (every listed key @@ -3410,7 +3427,7 @@ namespace /// remain, and the cursor must stay put so a later round's fresh budget can finish the same prefix /// instead of stranding the remainder permanently. `bounded_remaining == 0` conservatively reports /// `false` (nothing was even examined, so completeness cannot be claimed). -uint64_t deletePrefixWholesale(Backend & backend, const String & prefix, uint64_t bounded_remaining, +uint64_t deletePrefixWholesale(CasOperation & op, const String & prefix, uint64_t bounded_remaining, bool * out_fully_drained) { if (out_fully_drained) @@ -3420,22 +3437,22 @@ uint64_t deletePrefixWholesale(Backend & backend, const String & prefix, uint64_ String cursor; while (deleted < bounded_remaining) { - ListPage page = backend.list(prefix, cursor, kListPageLimit); + KeyPage page = op.list(prefix, cursor, kListPageLimit, Retry::standard()); /// One page fetched, not one increment per listed key below. ProfileEvents::increment(ProfileEvents::CASGCEnumerationPages); for (const auto & listed : page.keys) { if (deleted >= bounded_remaining) return deleted; - if (listed.token.has_value()) + if (listed.incarnation.has_value()) { - /// deleteExact tolerates NotFound (returns Kind::NotFound) and TokenMismatch — both are - /// benign here (already gone / rewritten by a live attempt); do not throw. - backend.deleteExact(listed.key, *listed.token); + /// `Gone` and `Mismatch` are both benign here (already gone / rewritten by a live + /// attempt); do not throw. + op.remove(listed.key, *listed.incarnation, Retry::standard()); } - else if (const auto head = backend.head(listed.key); head.exists) + else if (const auto head = op.head(listed.key, Retry::standard())) { - backend.deleteExact(listed.key, head.token); + op.remove(listed.key, head->incarnation, Retry::standard()); } ++deleted; } @@ -3466,7 +3483,7 @@ void Gc::pruneSupersededGenerations(uint64_t adopted_generation, uint64_t attemp if (keep == 0) return; /// keep ALL (debug/forensics — replay GC's in-degree view as-of a past round) - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); static constexpr uint64_t kMaxPrunePerRound = 64; /// bound the per-round prune burst @@ -3523,7 +3540,7 @@ void Gc::pruneSupersededGenerations(uint64_t adopted_generation, uint64_t attemp break; bool fully_drained = false; const uint64_t reclaimed = deletePrefixWholesale( - backend, layout.gcGenPrefix(g), remaining, &fully_drained); + op, layout.gcGenPrefix(g), remaining, &fully_drained); work_budget.prefix_wholesale_objects_used += reclaimed; if (!fully_drained) break; @@ -3550,7 +3567,8 @@ void Gc::pruneSupersededGenerations(uint64_t adopted_generation, uint64_t attemp std::optional Gc::readFoldSeal(uint64_t generation, uint64_t attempt) { - if (const auto got = store->backend().get(store->layout().foldSealKey(generation, attempt))) + CasOperation op = store->gcRequests().admit(); + if (const auto got = op.read(store->layout().foldSealKey(generation, attempt), Retry::standard())) return decodeFoldSeal( got->bytes, store->layout(), store->poolConfig().gc_shards, generation); return std::nullopt; @@ -3599,7 +3617,8 @@ std::vector Gc::discoverUniverse() /// The filter itself lives in `CasRefCatalog::liveUniverse` (review Important C) -- fsck's own /// reachability walk needed the identical catalog-authoritative set and is not this class, so the /// filter moved to where both can share it rather than grow a second copy that could disagree. - return CasRefCatalog::liveUniverse(store->backend(), store->layout()); + CasOperation op = store->gcRequests().admit(); + return CasRefCatalog::liveUniverse(op, store->layout()); } bool Gc::graduationDue(const GcState & state, uint64_t current_round) @@ -3646,12 +3665,12 @@ RefScanSummary Gc::enumerateRefPrefix() /// name is absorbed per key by `parseRefObjectKeyForEnumeration`, which is what keeps this /// enumeration -- which runs before the fold, outside its catch -- unable to wedge the round. const Layout & layout = store->layout(); - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); RefScanSummary scan; static constexpr size_t kListPageLimit = 1000; size_t count_in_page = 0; - forEachListedKey(backend, layout.casRefsPrefix(), [&](const ListedKey & lk) + op.forEachListedKey(layout.casRefsPrefix(), [&](const KeyEntry & lk) { scan.keys.push_back(lk.key); const auto parsed = parseRefObjectKeyForEnumeration(layout, lk.key); @@ -3671,8 +3690,9 @@ RefScanSummary Gc::enumerateRefPrefix() count_in_page = 0; ProfileEvents::increment(ProfileEvents::CASRefGlobalListPages); } - }, kListPageLimit, onGcEnumerationPage); - /// The walk's `backend.list` lands at least once even for an empty/undersized final page -- + return true; + }, Retry::standard(), kListPageLimit, onGcEnumerationPage); + /// The walk's list lands at least once even for an empty/undersized final page -- /// count it (one increment per physical LIST call). if (count_in_page > 0 || scan.keys.empty()) ProfileEvents::increment(ProfileEvents::CASRefGlobalListPages); @@ -3686,7 +3706,8 @@ RoundInput Gc::listRefPrefix(const GcState & state) /// plan. A listed id absent from the later cut is dead, inert debris: it contributes no work and /// cannot force DEFER. RefScanSummary scan = enumerateRefPrefix(); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(store->backend(), store->layout()); + CasOperation op = store->gcRequests().admit(); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, store->layout()); /// TEST SEAM: see `setPostHotScanCatalogReadHookForTest`. Moved into a local before invoking (the /// same reason `create_namespace_step1_pre_read_hook_for_test` is swapped rather than called /// directly): a hook that reassigns the member from inside its own body would otherwise reassign @@ -3724,7 +3745,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// Writes ONLY the GC plane; namespace streams/state, manifests, and blobs are read-only inputs; /// the rebuild never deletes them. RebuildReport rep; - Backend & backend = store->backend(); + CasOperation op = store->gcRequests().admit(); const Layout & layout = store->layout(); /// Read bookkeeping health before the lease (the lease acquire on an absent state CREATES a @@ -3736,7 +3757,7 @@ RebuildReport Gc::rebuildBaseline(bool force) bool healthy = false; bool validate_generation_zero_ref_baseline = false; { - const auto got = backend.get(layout.gcStateKey()); + const auto got = op.read(layout.gcStateKey(), Retry::standard()); /// The state's own decode stays inside its own try: an undecodable `gc/state` IS scenario (а), /// the disaster this command exists for. The prior-seal refusal below must NOT be swallowed by /// that catch, so the seal is read outside it. @@ -3797,7 +3818,7 @@ RebuildReport Gc::rebuildBaseline(bool force) "to rebuild; this pool must be recreated.", st.snap_generation, st.snap_attempt); for (const RunRef & r : seal->blob_target_runs) - if (!backend.head(r.key).exists) + if (!op.head(r.key, Retry::standard())) healthy = false; prior_seal = std::move(seal); rep.adopted_seal_generation = st.snap_generation; @@ -3875,8 +3896,8 @@ RebuildReport Gc::rebuildBaseline(bool force) /// has_observation==false always takes the non-steal branch on its one and only call), pass it /// explicitly rather than rely on that invariant. GcState state; - Token state_token; - if (!acquireOrRenewLease(state, state_token, /*allow_steal=*/false)) + std::optional state_incarnation; + if (!acquireOrRenewLease(state, state_incarnation, /*allow_steal=*/false)) { rep.refusal = "another GC leader holds the lease"; return rep; @@ -3893,7 +3914,7 @@ RebuildReport Gc::rebuildBaseline(bool force) || drain_result.catalog_resolution != CatalogResolution::DrainComplete) throwCasWriteRetryLater("CAS GC rebuild lost authority before the catalog settled"); const RefScanSummary rebuild_ref_scan = enumerateRefPrefix(); - const CasRefCatalog::Snapshot rebuild_work_catalog_cut = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot rebuild_work_catalog_cut = CasRefCatalog::read(op, layout); RefScanSummary rebuild_round_scan = rebuild_ref_scan; if (prior_seal) @@ -3915,8 +3936,9 @@ RebuildReport Gc::rebuildBaseline(bool force) for (const NamespaceLifeId & life : rebuild_walk_universe) { std::vector table_keys; - forEachListedKey(backend, layout.namespaceStreamPrefix(life), - [&](const ListedKey & lk) { table_keys.push_back(lk.key); }, 1000, onGcEnumerationPage); + op.forEachListedKey(layout.namespaceStreamPrefix(life), + [&](const KeyEntry & lk) { table_keys.push_back(lk.key); return true; }, + Retry::standard(), 1000, onGcEnumerationPage); std::map grouped; try { @@ -3952,12 +3974,12 @@ RebuildReport Gc::rebuildBaseline(bool force) { const String gen_prefix = layout.gcGenPrefix(0); const String top = gen_prefix.substr(0, gen_prefix.size() - 2); /// ".../gc/gen/" - forEachListedKey(backend, top, [&](const ListedKey & k) + op.forEachListedKey(top, [&](const KeyEntry & k) { const size_t from = top.size(); const size_t slash = k.key.find('/', from); if (slash == String::npos) - return; + return true; try { max_gen = std::max(max_gen, static_cast(std::stoull(k.key.substr(from, slash - from)))); @@ -3966,7 +3988,8 @@ RebuildReport Gc::rebuildBaseline(bool force) { /// Foreign key shape under `gc/gen` is debris, not a numbering input. } - }, 1000, onGcEnumerationPage); + return true; + }, Retry::standard(), 1000, onGcEnumerationPage); } const uint64_t generation = max_gen + 1; const uint64_t budget = rebuild_edge_budget_override ? rebuild_edge_budget_override @@ -3987,7 +4010,7 @@ RebuildReport Gc::rebuildBaseline(bool force) if (buckets[shard].empty()) return; std::vector out; - foldDeltasIntoGeneration(backend, layout, prior_runs[shard], generation, ++attempt_of[shard], + foldDeltasIntoGeneration(op, layout, prior_runs[shard], generation, ++attempt_of[shard], shard, std::move(buckets[shard]), out, /*current_round*/0, /*condemn_round*/0, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, @@ -4026,7 +4049,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// round once it is known, and nothing else is touched. std::set minted_hold_lives; uint64_t max_fence_round = 0; - std::map mf_cleanup_unused; + std::map mf_cleanup_unused; for (const NamespaceLifeId & life : rebuild_walk_universe) { @@ -4053,7 +4076,7 @@ RebuildReport Gc::rebuildBaseline(bool force) checkpoint_it != rebuild_checkpoints.recovery_checkpoints.end()) checkpoint = checkpoint_it->second; const RecoveredRefTable recovered = recoverRefTableDetailedFromAuthority( - backend, layout, *entry_it, checkpoint); + op, layout, *entry_it, checkpoint); const RefTableState & st = recovered.state; RefCoverage cov; @@ -4073,7 +4096,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { const ManifestId id{ns, row.manifest_ref}; owned_manifest_keys.insert(layout.manifestKey(id)); - if (!foldManifestEdges(id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) + if (!foldManifestEdges(op, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) { rep.refusal = "committed ref '" + ns.string() + "/" + ref_name + "' names a missing or invalid part manifest — that is DATA LOSS the rebuild " @@ -4089,7 +4112,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { const ManifestId id{ns, manifest_ref}; owned_manifest_keys.insert(layout.manifestKey(id)); - if (foldManifestEdges(id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) + if (foldManifestEdges(op, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) ++rep.live_precommits; else { @@ -4148,26 +4171,27 @@ RebuildReport Gc::rebuildBaseline(bool force) { const RootNamespace ns{ns_str}; std::vector deltas; - forEachListedKey(backend, layout.manifestNamespacePrefix(ns), [&](const ListedKey & k) + op.forEachListedKey(layout.manifestNamespacePrefix(ns), [&](const KeyEntry & k) { if (owned_manifest_keys.contains(k.key)) - return; + return true; /// The one shared manifest-path parser for the canonical hexadecimal manifest identifier, /// also used by fsck's parseBuildPrefix and the orphan sweep's parseListedManifestObject. const auto parsed = layout.parseManifestKey(k.key); if (!parsed) - return; /// foreign key shape — debris + return true; /// foreign key shape — debris const ManifestRef & mref = parsed->ref; if (prefixEligible(*store, ns, BuildPrefix{mref.writer_epoch, mref.build_sequence})) - return; /// provably dead — the orphan sweep's territory, never an edge + return true; /// provably dead — the orphan sweep's territory, never an edge const ManifestId id{ns, mref}; - if (foldManifestEdges(id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) + if (foldManifestEdges(op, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) { ++rep.unowned_alive_manifests; route_deltas(deltas); } /// A missing/invalid UNOWNED body is debris (no owner claims it) — skip, never refuse. - }, 1000, onGcEnumerationPage); + return true; + }, Retry::standard(), 1000, onGcEnumerationPage); } /// A REBUILD CONDEMNS NOTHING (spec §7). @@ -4216,7 +4240,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// `mountObservationThresholdMs` -- see its doc comment (CasServerRoot.h). const uint64_t stable_threshold_ms = mountObservationThresholdMs( ttl_ms, static_cast(store->poolConfig().mount_renew_period.count())); - computeHeartbeatFloor(backend, layout, now_ms_fn(), mono_ms_fn(), stable_threshold_ms, mount_obs); + computeHeartbeatFloor(op, layout, now_ms_fn(), mono_ms_fn(), stable_threshold_ms, mount_obs); /// Retired-in-snapshot: the rebuilt seal's `condemned_summary` must be TOTAL over gc_shards so a /// subsequent regular round reads graduation/carry decisions zero-I/O off it (and its `carryParentRefs` @@ -4232,7 +4256,7 @@ RebuildReport Gc::rebuildBaseline(bool force) for (uint64_t a : attempt_of) seal_attempt = std::max(seal_attempt, a); validateFoldSealForWrite(seal, layout, gc_shards); - putDeterministicArtifact(backend, layout.foldSealKey(generation, seal_attempt), encodeFoldSeal(seal)); + putDeterministicArtifact(op, layout.foldSealKey(generation, seal_attempt), encodeFoldSeal(seal)); GcState next = state; next.round = round; @@ -4243,12 +4267,14 @@ RebuildReport Gc::rebuildBaseline(bool force) /// family independently of that, and the two reasons are stated apart on purpose — a future reader /// must not take this line as evidence that REBUILD still produces condemnations somewhere. next.manifest_sweep_cursor = ""; - const CasResult res = backend.casPut(layout.gcStateKey(), encodeGcState(next), state_token); - if (res.outcome != CasOutcome::Committed) + WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_incarnation, + Retry::standard()); + if (std::holds_alternative(commit)) { rep.refusal = "gc/state changed under the rebuild (a competing writer) — re-run"; return rep; } + orThrow(std::move(commit), "CAS gc rebuild: gc/state commit"); rep.performed = true; rep.round = round; @@ -4277,13 +4303,13 @@ std::vector Gc::previewDeletes() { std::vector out; - const auto state_bytes = store->backend().get(store->layout().gcStateKey()); + CasOperation op = store->gcRequests().admit(); + const auto state_bytes = op.read(store->layout().gcStateKey(), Retry::standard()); if (!state_bytes) return out; const GcState state = decodeGcState(state_bytes->bytes); const Layout & layout = store->layout(); - Backend & backend = store->backend(); /// Resolve the run objects THROUGH the adopted seal's refs, never by /// `blobTargetRunKey` construction: with reference-parent carry a shard's current run may physically @@ -4301,27 +4327,27 @@ std::vector Gc::previewDeletes() const auto it = runs_by_shard.find(shard); static const std::vector kEmptyRuns; const std::vector & shard_runs = it != runs_by_shard.end() ? it->second : kEmptyRuns; - for (const BlobCandidate & cand : zeroInDegree(backend, shard_runs)) + for (const BlobCandidate & cand : zeroInDegree(op, shard_runs)) { - const HeadResult observed = backend.head(layout.blobKey(cand.ref)); - if (!observed.exists) + const std::optional observed = op.head(layout.blobKey(cand.ref), Retry::standard()); + if (!observed) continue; PreviewEntry e; e.kind = ObjectKind::Blob; e.ref = cand.ref; e.key = layout.blobKey(cand.ref); - e.size = observed.size; + e.size = observed->size; e.reason = "unreachable"; out.push_back(std::move(e)); } /// Retired-in-snapshot: stream the SAME adopted seal runs and emit every `RunMarker::Condemned` - /// sentinel row. The stored token IS the authority — NO HEAD here (a HEAD would defeat the point - /// and cost I/O). `delete_pending` rows are deleted next fold; the rest await graduation. Preview + /// sentinel row. The stored incarnation IS the authority — no observation here (one would defeat + /// the point and cost I/O). `delete_pending` rows are deleted next fold; the rest await graduation. Preview /// stays WRITE-FREE (`openSourceEdgeRun` is a pure reader). Output is a superset of the above. for (const RunRef & run : shard_runs) { - SourceEdgeRunView reader = openSourceEdgeRun(backend, run.key); + SourceEdgeRunView reader = openSourceEdgeRun(op, run.key); String key; String payload; while (reader.next(key, payload)) @@ -4357,31 +4383,59 @@ void Gc::rememberObservation(const GcLease & lease) last_seen_seq = lease.seq; } +void Gc::refreshAuthority(uint64_t admitted_generation) +{ + /// Fail-closed first, so every early exit below leaves this leader deposed. + authority_held = false; + try + { + CasOperation op = store->gcRequests().admit(); + const auto got = op.read(store->layout().gcStateKey(), Retry::standard()); + if (!got) + return; + const GcState current = decodeGcState(got->bytes); + authority_held = current.lease.owner == gc_id && current.lease.seq == admitted_generation; + } + catch (...) + { + tryLogCurrentException(logger, + "CAS gc: the leader-authority probe failed; this round's destructive operations treat the " + "lease as lost"); + } +} + void Gc::pulseHeartbeat(Pool & store, UInt128 gc_id) { + CasOperation op = store.gcRequests().admit(); const String key = store.layout().gcHbKey(); - const auto got = store.backend().get(key); + const auto got = op.read(key, Retry::standard()); GcHeartbeat hb; - std::optional expected; if (got) - { hb = decodeGcHeartbeat(got->bytes); - expected = got->token; - } hb.owner = gc_id; ++hb.hb_seq; - store.backend().casPut(key, encodeGcHeartbeat(hb), expected); + /// ONE attempt, and the outcome is discarded: a pulse that loses its race is replaced by the next + /// one on cadence, so a deposed leader must never spend a whole retry budget fighting for this key. + const String body = encodeGcHeartbeat(hb); + if (got) + op.replace(key, body, got->incarnation, Retry::once()); + else + op.create(key, body, Retry::once()); } -bool Gc::acquireOrRenewLease(GcState & state, Token & state_token, bool allow_steal) +bool Gc::acquireOrRenewLease(GcState & state, std::optional & state_incarnation, bool allow_steal) { + CasOperation op = store->gcRequests().admit(); const String key = store->layout().gcStateKey(); - for (int attempt = 0; attempt < 2; ++attempt) + /// What the decision that actually landed wrote. `readModifyWrite` re-decides on every conflict + /// against what the losing write's own resolve read observed, so the last decision is the committed + /// one, and a competing leader's write makes the next decision see a moved lease tuple. + GcState decided; + const std::optional committed = orThrow(op.readModifyWrite(key, + [&](const std::optional & current_object) -> std::optional { - const auto got = store->backend().get(key); - - if (!got) + if (!current_object) { if (has_observation) throw Exception(ErrorCodes::CORRUPTED_DATA, @@ -4394,18 +4448,11 @@ bool Gc::acquireOrRenewLease(GcState & state, Token & state_token, bool allow_st /// the authoritative value from the persisted GcState (pool is authoritative on reopen). /// PoolConfig carries the configured value from the disk XML. fresh.gc_shards = store->poolConfig().gc_shards; - const CasResult acquire_res = store->backend().casPut(key, encodeGcState(fresh), std::nullopt); - if (acquire_res.outcome == CasOutcome::Committed) - { - rememberObservation(fresh.lease); - state = std::move(fresh); - state_token = acquire_res.token; - return true; - } - continue; + decided = fresh; + return encodeGcState(fresh); } - GcState current = decodeGcState(got->bytes); + const GcState current = decodeGcState(current_object->bytes); if (current.gc_shards != store->poolConfig().gc_shards) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state gc_shards {} disagrees with the pool-authoritative _pool_meta value {}", @@ -4415,19 +4462,12 @@ bool Gc::acquireOrRenewLease(GcState & state, Token & state_token, bool allow_st { GcState next = current; ++next.lease.seq; - const CasResult renew_res = store->backend().casPut(key, encodeGcState(next), got->token); - if (renew_res.outcome == CasOutcome::Committed) - { - rememberObservation(next.lease); - state = std::move(next); - state_token = renew_res.token; - return true; - } - continue; + decided = next; + return encodeGcState(next); } GcHeartbeat hb; - if (const auto hb_got = store->backend().get(store->layout().gcHbKey())) + if (const auto hb_got = op.read(store->layout().gcHbKey(), Retry::standard())) hb = decodeGcHeartbeat(hb_got->bytes); /// Observation-based heartbeat liveness, symmetric with the frozen-lease-tuple check below: /// ANY movement of the observed (owner, hb_seq) pair between this contender's two ticks is @@ -4463,27 +4503,23 @@ bool Gc::acquireOrRenewLease(GcState & state, Token & state_token, bool allow_st last_seen_hb_owner = hb.owner; last_seen_hb_seq = hb.hb_seq; } - return false; + return std::nullopt; } GcState next = current; next.lease.owner = gc_id; ++next.lease.seq; - const CasResult steal_res = store->backend().casPut(key, encodeGcState(next), got->token); - if (steal_res.outcome == CasOutcome::Committed) - { - rememberObservation(next.lease); - state = std::move(next); - state_token = steal_res.token; - return true; - } + decided = next; + return encodeGcState(next); + }, Retry::standard()), "CAS gc lease"); - if (const auto reread = store->backend().get(key)) - rememberObservation(decodeGcState(reread->bytes).lease); - return false; - } + if (!committed) + return false; /// declined: a live incumbent holds the lease - return false; + rememberObservation(decided.lease); + state = std::move(decided); + state_incarnation = committed; + return true; } CatalogLifecycleReconcileResult Gc::drainCompletedRemoving(const GcState & leased_state) @@ -4503,28 +4539,17 @@ CatalogLifecycleReconcileResult Gc::drainCompletedRemoving(const GcState & lease "CAS GC pre-fold drain: adopted parent seal (generation {}, attempt {}) is missing", leased_state.snap_generation, leased_state.snap_attempt); - Backend & backend = store->backend(); - const Layout & layout = store->layout(); + /// The drain erases catalog rows, so its operation carries this leader's authority as its + /// liveness: `CatalogLifecycleReconciler` and `deleteCompletedRemovingAtSnapshot` decide + /// `FencedOut` from `op.admitted()`, and the GC plane's fence is open, so without this the verdict + /// would be a constant TRUE and a deposed leader would keep erasing. The refresh below runs at the + /// top of every erase attempt, so a leader deposed between two erases of one drain stops before the + /// second -- the single reading taken here would otherwise authorise all of them. const uint64_t admitted_generation = leased_state.lease.seq; - const auto check_fence = [&](uint64_t expected_generation) - { - if (expected_generation != admitted_generation) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS GC pre-fold drain: internal leader generation mismatch (expected {}, admitted {})", - expected_generation, admitted_generation); - const auto got = backend.get(layout.gcStateKey()); - if (!got) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS GC pre-fold drain: gc/state vanished while checking leader generation {}", - admitted_generation); - const GcState current = decodeGcState(got->bytes); - if (current.lease.owner != gc_id || current.lease.seq != admitted_generation) - return CasRefCatalog::LeaderFenceStatus::Moved; - return CasRefCatalog::LeaderFenceStatus::Held; - }; - - return CatalogLifecycleReconciler( - backend, layout, *parent, admitted_generation, check_fence).reconcile(); + refreshAuthority(admitted_generation); + CasOperation op = store->gcRequests().admit([this] { return authority_held; }); + return CatalogLifecycleReconciler(op, store->layout(), *parent) + .reconcile([this, admitted_generation] { refreshAuthority(admitted_generation); }); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index 93209b70a3ff..f6bf38fdbaea 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -1,4 +1,5 @@ #pragma once +#include #include #include #include @@ -453,7 +454,7 @@ class Gc String key; uint64_t size = 0; String reason; /// "unreachable" | "delete_pending" | "awaiting_graduation" - Token token; /// stored condemn-time token (empty for "unreachable") + PersistedIncarnation token; /// stored condemn-time incarnation (empty for "unreachable") uint64_t condemn_round = 0; }; @@ -509,10 +510,10 @@ class Gc private: /// Lease acquire/renew/steal per the documented observation protocol. On success `state` holds the - /// committed gc/state (with our lease) and `state_token` its backend token. `allow_steal=false` - /// suppresses only the steal CAS (see runRegularRound's doc comment) — acquiring a free lease and - /// renewing our own are unaffected. - bool acquireOrRenewLease(GcState & state, Token & state_token, bool allow_steal); + /// committed gc/state (with our lease) and `state_incarnation` the incarnation that write created. + /// `allow_steal=false` suppresses only the steal (see runRegularRound's doc comment) — acquiring a + /// free lease and renewing our own are unaffected. + bool acquireOrRenewLease(GcState & state, std::optional & state_incarnation, bool allow_steal); /// Catalog-only helping barrier run immediately after lease acquisition. It validates the adopted /// parent and delegates deterministic `Removing`-row settlement to `CatalogLifecycleReconciler`. @@ -531,13 +532,13 @@ class Gc /// What one fold produced. The blob deltas are sealed /// into a write-once generation; `fold_seal` is the durable index of WHAT WAS FOLDED (a CasFoldSeal), /// `root_shards` the discovered universe, `mf_cleanup` the part-manifest cleanup work keyed by - /// ManifestId (owner-removed bodies whose exact-token delete is deferred until their decrements are - /// sealed), and `retired_merge` the per-gc-shard ack-floor retired-cursor outcome. + /// ManifestId (owner-removed bodies whose exact-incarnation delete is deferred until their + /// decrements are sealed), and `retired_merge` the per-gc-shard ack-floor retired-cursor outcome. struct FoldResult { CasFoldSeal fold_seal; std::vector> root_shards; - std::map mf_cleanup; + std::map mf_cleanup; /// Bounded orphan candidates exact-read before reduce. Their source retirements ride this /// fold's runs; their manifest tokens become deletable only after the round CAS adopts them. ManifestSweepResult orphan_sweep; @@ -688,8 +689,6 @@ class Gc /// missing body or a true-removal old body missing at removal-fold => fail-closed FOR THAT DECISION /// (clamp the shard's last_folded_ref_id below it, record the anomaly, stop folding THIS shard) — /// never guess a delta and never wedge the round on a missing body. - /// On success `state` carries the committed snap_generation and `state_token` the committed gc/state - /// token. The committed pair is THREADED into retire, never re-read (zombie-steal protection). /// Round-paced graduation: `current_round` (= state.round + 1, the SAME basis condemn_round is /// stamped at) is the threshold the fold's two-cursor merge graduates/condemns against — an entry /// graduates once `condemn_round < current_round`, i.e. it survived at least one full round after @@ -697,7 +696,8 @@ class Gc /// in-memory; the SINGLE round CAS commits them. /// `walk_plan` owns the round's one enumeration of `cas/ns/stream/` (see `RefScanSummary`) and /// its catalog cut; the fold regroups those keys strictly rather than listing the prefix again. - FoldResult fold(GcState & state, Token & state_token, RoundReport & report, uint64_t current_round, + FoldResult fold(GcState & state, std::optional & state_incarnation, + RoundReport & report, uint64_t current_round, const RefPlan & walk_plan, UniversePolicy policy, /// One instance for the WHOLE round, owned by `runRegularRound` and threaded through /// every destructive-work family the round touches — see `GcRoundWorkBudget`. @@ -780,13 +780,13 @@ class Gc std::optional> newestFoldSealRef(); /// Read ONE part manifest named by `id`, validate it, and append sign*(+1) blob deltas for each - /// blob entry to `deltas`. On sign<0 queue (id -> token) into mf_cleanup. Returns whether a body was + /// blob entry to `deltas`. On sign<0 queue (id -> incarnation) into mf_cleanup. Returns whether a body was /// read+validated: false => ABSENT body (404; the caller decides per the 404 rule). A body that is /// PRESENT but fails refMatchesBody / manifestNamespaceMatches throws CORRUPTED_DATA. /// `txn_ordinal` stamps every delta this call pushes with the round-local ordinal of the ref /// transaction that emitted it (probe B2 — see `TxnApplyLedger`). - bool foldManifestEdges(const ManifestId & id, int sign, std::vector & deltas, - std::map & mf_cleanup, uint32_t txn_ordinal); + bool foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, std::vector & deltas, + std::map & mf_cleanup, uint32_t txn_ordinal); @@ -880,6 +880,10 @@ class Gc /// Update the remembered observation (steal protocol step 3/4). void rememberObservation(const GcLease & lease); + /// Re-read `gc/state` and record whether this leader still holds the lease it was admitted under. + /// Fail-closed: an absent, unreadable or undecodable state reads as deposed. + void refreshAuthority(uint64_t admitted_generation); + PoolPtr store; /// Where `GcPhaseTimer` sends one record per GC phase. Empty unless a `CasGcScheduler` installed one /// for the current round, in which case every phase of that round emits a row. @@ -903,6 +907,18 @@ class Gc /// a round folds; incremented on every DEFER. Bounds batching via `gc_fold_max_defer_rounds`. uint64_t rounds_since_last_fold_ = 0; + /// THIS LEADER'S OWN AUTHORITY VERDICT, and the reason it is a cached bool rather than a probe. + /// + /// It is what the `Liveness` predicates of the round's destructive operations sample -- the pre-fold + /// catalog drain and the namespace janitor page, both of which erase objects a deposed leader must + /// not touch. The engine samples a `Liveness` before EVERY request and before every sleep, so a + /// predicate that read `gc/state` itself would put one `GET` on the hot path of every listed key. + /// The read that sets this flag is therefore made by the round, at the granularity the old + /// hand-written fence check had (once per drain, once per janitor page), never from inside the + /// predicate. The staleness that buys is bounded by that granularity and stated where each caller + /// refreshes it. + bool authority_held = false; + /// the contender's observation window (steal protocol) bool has_observation = false; UInt128 last_seen_owner{}; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp index 19ff16bb956c..8fba83c8f027 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp @@ -9,32 +9,32 @@ namespace DB::ErrorCodes namespace DB::Cas { -GcMaintenanceReadResult readGcMaintenanceState(Backend & backend, const Layout & layout) +GcMaintenanceReadResult readGcMaintenanceState(CasOperation & op, const Layout & layout) { - const auto got = backend.get(layout.gcMaintenanceStateKey()); + const auto got = op.read(layout.gcMaintenanceStateKey(), Retry::standard()); if (!got) - return {.status = GcMaintenanceReadStatus::Absent, .state = std::nullopt, .token = std::nullopt, .diagnostic = {}}; + return {.status = GcMaintenanceReadStatus::Absent, .state = std::nullopt, .incarnation = std::nullopt, .diagnostic = {}}; try { return {.status = GcMaintenanceReadStatus::Valid, .state = decodeGcMaintenanceState(got->bytes), - .token = got->token, .diagnostic = {}}; + .incarnation = got->incarnation, .diagnostic = {}}; } catch (const DB::Exception & e) { if (e.code() != ErrorCodes::CORRUPTED_DATA) throw; return {.status = GcMaintenanceReadStatus::Corrupt, .state = std::nullopt, - .token = got->token, .diagnostic = e.message()}; + .incarnation = got->incarnation, .diagnostic = e.message()}; } } -GcMaintenanceCasResult casGcMaintenanceState( - Backend & backend, const Layout & layout, const std::optional & expected, const GcMaintenanceState & next) +WriteResult casGcMaintenanceState( + CasOperation & op, const Layout & layout, const std::optional & expected, + const GcMaintenanceState & next, const Retry & policy) { - const CasResult result = backend.casPut(layout.gcMaintenanceStateKey(), encodeGcMaintenanceState(next), expected); - if (result.outcome == CasOutcome::Committed) - return {.outcome = GcMaintenanceCasOutcome::Committed, .token = result.token}; - return {.outcome = GcMaintenanceCasOutcome::Conflict, .token = {}}; + const String key = layout.gcMaintenanceStateKey(); + const String bytes = encodeGcMaintenanceState(next); + return expected ? op.replace(key, bytes, *expected, policy) : op.create(key, bytes, policy); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h index 6d941177d19f..5839bbfc6914 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -12,18 +12,17 @@ struct GcMaintenanceReadResult { GcMaintenanceReadStatus status; std::optional state; - std::optional token; + std::optional incarnation; String diagnostic; }; -enum class GcMaintenanceCasOutcome : uint8_t { Committed, Conflict }; -struct GcMaintenanceCasResult -{ - GcMaintenanceCasOutcome outcome = GcMaintenanceCasOutcome::Conflict; - Token token; -}; -GcMaintenanceReadResult readGcMaintenanceState(Backend & backend, const Layout & layout); -GcMaintenanceCasResult casGcMaintenanceState( - Backend & backend, const Layout & layout, const std::optional & expected, const GcMaintenanceState & next); +GcMaintenanceReadResult readGcMaintenanceState(CasOperation & op, const Layout & layout); +/// `create`s the maintenance-state key on absence, `replace`s it when `expected` names the +/// incarnation last observed. `policy` is named by the caller because a write issued from inside an +/// already-failed step (a reset after a failed enumeration) must send at most one attempt rather than +/// spend the round's remaining time retrying a write nothing downstream is waiting on. +WriteResult casGcMaintenanceState( + CasOperation & op, const Layout & layout, const std::optional & expected, + const GcMaintenanceState & next, const Retry & policy); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp index b60df45da424..b157fca7cba1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp @@ -1,9 +1,11 @@ #include +#include #include #include #include +#include namespace ProfileEvents { @@ -24,6 +26,13 @@ namespace DB::Cas namespace { +/// The registry key for one condemned incarnation: the persisted pair rendered the way a live +/// `Incarnation` renders itself, so two dialects can never collide on a shared value. +String condemnMarkerKey(const PersistedIncarnation & token) +{ + return token.dialect + ":" + token.value; +} + /// The per-hash freshness-meta operations GC schedules on the bounded pool are /// best-effort/idempotent by design. The meta is only a point-read freshness marker for the writer/ /// promote gate; the ledger retired-set + the exact-token body delete remain the actual safety @@ -46,8 +55,8 @@ namespace /// a writer reading `Clean` would reuse the exact condemned token, which a stale pre-CAS exact-token /// redelete then deletes -- live-blob data loss (INV_NO_LOSS). Removing the clear restores the exact-token /// delete argument in full: once a hash is `Condemned`, observing `Clean` means EITHER the condemned body -/// is absent OR a writer already changed its incarnation token, so every stale `deleteExact(t1)` finds the -/// body absent or `TokenMismatch`. +/// is absent OR a writer already changed its incarnation, so every stale exact-incarnation delete of the +/// condemned incarnation finds the body absent or holding a different one. /// Write the per-hash meta to Condemned: a blob newly entering the retired set this round (either the /// fresh zero-in-degree condemn, or a republication-supersede re-condemn of the current token). Absent meta @@ -55,17 +64,18 @@ namespace /// alone rather than clobbering a possibly-newer condemn_round. /// /// Returns whether durable Condemned evidence exists after the call: the conditional write committed, -/// or an already-Condemned meta was observed. A lost CAS reports false and writes nothing further (the -/// loser re-reads next time); a thrown backend error propagates (the scheduling wrapper swallows it) — -/// either way the entry stays UNCONFIRMED and the graduation gate carries it. -bool writeCondemnedMeta(Pool & pool, const BlobRef & ref, uint64_t condemn_round, uint64_t size) +/// or an already-Condemned meta was observed. Any other write outcome reports false and writes nothing +/// further (the loser re-reads next time); a thrown backend error propagates (the scheduling wrapper +/// swallows it) — either way the entry stays UNCONFIRMED and the graduation gate carries it. +bool writeCondemnedMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, + uint64_t condemn_round, uint64_t size) { - const auto lm = loadMeta(pool.backend(), pool.layout(), ref); + const auto lm = loadMeta(op, layout, ref); const BlobMeta desired{.state = MetaState::Condemned, .condemn_round = condemn_round, .size = size}; if (!lm) - return putMetaIfAbsent(pool, ref, desired).outcome == CasOverwriteOutcome::Committed; + return std::holds_alternative(putMetaIfAbsent(op, layout, ref, desired)); if (lm->meta.state != MetaState::Condemned) - return casMeta(pool, ref, lm->etag, desired).outcome == CasOverwriteOutcome::Committed; + return std::holds_alternative(casMeta(op, layout, ref, lm->incarnation, desired)); return true; } @@ -73,12 +83,12 @@ bool writeCondemnedMeta(Pool & pool, const BlobRef & ref, uint64_t condemn_round /// exact-token delete. NO tombstone -- an absent meta reads exactly like a Clean one (absent /// means not condemned"). Idempotent: an already-absent meta, or one a racing writer/GC pass already /// moved, is a silent no-op. -void deleteConfirmedMeta(Backend & backend, const Layout & layout, const BlobRef & ref) +void deleteConfirmedMeta(CasOperation & op, const Layout & layout, const BlobRef & ref) { - const auto lm = loadMeta(backend, layout, ref); + const auto lm = loadMeta(op, layout, ref); if (!lm) return; - deleteMetaExact(backend, layout, ref, lm->etag); + deleteMetaExact(op, layout, ref, lm->incarnation); } } @@ -134,12 +144,15 @@ void GcMetaWriter::submit(std::function op) } } -void GcMetaWriter::scheduleCondemnMarkerWrite(const BlobRef & ref, const Token & token, +void GcMetaWriter::scheduleCondemnMarkerWrite(const BlobRef & ref, const PersistedIncarnation & token, uint64_t condemn_round, uint64_t size) { + /// The job admits its OWN operation: a `CasOperation` carries per-call state and belongs to one + /// task, while several of these run concurrently on the pool. submit([st = state, ref, token, condemn_round, size]() { - if (writeCondemnedMeta(*st->store, ref, condemn_round, size)) + CasOperation op = st->store->gcRequests().admit(); + if (writeCondemnedMeta(op, st->store->layout(), ref, condemn_round, size)) st->noteCondemnMarkerDurable(ref, token); }); } @@ -148,7 +161,8 @@ void GcMetaWriter::scheduleConfirmedMetaDelete(const BlobRef & ref) { submit([st = state, ref]() { - deleteConfirmedMeta(st->store->backend(), st->store->layout(), ref); + CasOperation op = st->store->gcRequests().admit(); + deleteConfirmedMeta(op, st->store->layout(), ref); }); } @@ -187,35 +201,35 @@ uint64_t GcMetaWriter::completed() const return state->completed.load(std::memory_order_relaxed); } -void GcMetaWriter::State::noteCondemnMarkerDurable(const BlobRef & ref, const Token & token) +void GcMetaWriter::State::noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token) { std::lock_guard lock(condemn_marker_mutex); - condemn_markers_confirmed.emplace(ref, token.value); + condemn_markers_confirmed.emplace(ref, condemnMarkerKey(token)); } -bool GcMetaWriter::State::condemnMarkerConfirmedInProcess(const BlobRef & ref, const Token & token) +bool GcMetaWriter::State::condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token) { std::lock_guard lock(condemn_marker_mutex); - return condemn_markers_confirmed.contains({ref, token.value}); + return condemn_markers_confirmed.contains({ref, condemnMarkerKey(token)}); } -void GcMetaWriter::State::forgetCondemnMarker(const BlobRef & ref, const Token & token) +void GcMetaWriter::State::forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token) { std::lock_guard lock(condemn_marker_mutex); - condemn_markers_confirmed.erase({ref, token.value}); + condemn_markers_confirmed.erase({ref, condemnMarkerKey(token)}); } -void GcMetaWriter::noteCondemnMarkerDurable(const BlobRef & ref, const Token & token) +void GcMetaWriter::noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token) { state->noteCondemnMarkerDurable(ref, token); } -bool GcMetaWriter::condemnMarkerConfirmedInProcess(const BlobRef & ref, const Token & token) +bool GcMetaWriter::condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token) { return state->condemnMarkerConfirmedInProcess(ref, token); } -void GcMetaWriter::forgetCondemnMarker(const BlobRef & ref, const Token & token) +void GcMetaWriter::forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token) { state->forgetCondemnMarker(ref, token); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h index d1332854f416..95646762d61f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h @@ -1,7 +1,7 @@ #pragma once +#include #include -#include #include #include @@ -29,11 +29,11 @@ class GcMetaWriter GcMetaWriter(const GcMetaWriter &) = delete; GcMetaWriter & operator=(const GcMetaWriter &) = delete; - /// Publish durable Condemned evidence for one (blob, exact incarnation-token) pair. On success the + /// Publish durable Condemned evidence for one (blob, exact incarnation) pair. On success the /// pair is recorded in the in-process confirmation registry, which the graduation gate reads. A - /// lost CAS or a thrown error leaves the pair UNCONFIRMED: the gate then carries the entry and a - /// later round retries the write. - void scheduleCondemnMarkerWrite(const BlobRef & ref, const Token & token, + /// refused write or a thrown error leaves the pair UNCONFIRMED: the gate then carries the entry and + /// a later round retries the write. + void scheduleCondemnMarkerWrite(const BlobRef & ref, const PersistedIncarnation & token, uint64_t condemn_round, uint64_t size); /// Drop the freshness meta of a blob whose body is confirmed deleted or absent. @@ -51,12 +51,14 @@ class GcMetaWriter uint64_t scheduled() const; uint64_t completed() const; - /// The in-process condemn-marker confirmation registry, keyed (blob, exact token value). Pool + /// The in-process condemn-marker confirmation registry, keyed (blob, rendered incarnation). It is + /// keyed by the PERSISTED pair because every entry that consults it arrives from a durable + /// condemned row; a live observation enters through `PersistedIncarnation::capture`. Pool /// completions insert concurrently with the round thread's reads, and the round thread also /// inserts directly when it re-checks a marker synchronously. - void noteCondemnMarkerDurable(const BlobRef & ref, const Token & token); - bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const Token & token); - void forgetCondemnMarker(const BlobRef & ref, const Token & token); + void noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token); + bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token); + void forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token); private: /// Everything a job reaches. Held by `shared_ptr` and captured by value into every job. @@ -69,9 +71,9 @@ class GcMetaWriter std::mutex condemn_marker_mutex; std::set> condemn_markers_confirmed; - void noteCondemnMarkerDurable(const BlobRef & ref, const Token & token); - bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const Token & token); - void forgetCondemnMarker(const BlobRef & ref, const Token & token); + void noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token); + bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token); + void forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token); }; /// Catch each meta-operation exception, count the job, and put it on the pool -- running it diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.cpp index c3b44569d074..7f102a2b86d3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.cpp @@ -40,13 +40,13 @@ bool ShardReducer::owns(const BlobRef & ref) const return blobShard(ref, gc_shards) == shard; } -std::vector ShardReducer::reduce(Backend & backend, const Layout & layout, +std::vector ShardReducer::reduce(CasOperation & op, const Layout & layout, const std::vector & prior_runs, uint64_t new_generation, uint64_t attempt, std::vector shard_deltas, uint64_t current_round, uint64_t condemn_round, - const std::function(const BlobRef &)> & head_blob, - const std::function(const BlobRef &)> & peek_head, + const BlobHeadFn & head_blob, + const BlobHeadFn & peek_head, const std::function & confirm_condemned_marker, RetiredMergeResult * out_retired, bool suppress_destructive, @@ -54,7 +54,7 @@ std::vector ShardReducer::reduce(Backend & backend, const Layout & layou GcRoundWorkBudget * work_budget) const { std::vector out_runs; - foldDeltasIntoGeneration(backend, layout, prior_runs, new_generation, attempt, shard, + foldDeltasIntoGeneration(op, layout, prior_runs, new_generation, attempt, shard, std::move(shard_deltas), out_runs, current_round, condemn_round, head_blob, peek_head, confirm_condemned_marker, out_retired, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.h index 20edc54dfb88..86e698dc6a63 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcShardPlan.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -72,7 +72,7 @@ uint64_t manifestCleanupShard(const ManifestId & id, uint64_t gc_shards); /// uses with `shard == 0`), so `gc_shards == 1` with `shard == 0` reproduces the non-sharded fold /// byte-for-byte. This keeps the one-shard configuration compatible with the original fold path. /// -/// NOTE on durable writes: `reduce` writes the per-shard in-degree run directly via `backend` +/// NOTE on durable writes: `reduce` writes the per-shard in-degree run directly through `op` /// (under `blobTargetRunKey(new_generation, shard, 0)`), exactly as `foldDeltasIntoGeneration` /// does. Returning the durable write here (rather than an in-memory map) keeps the round driver /// stateless: it simply constructs a `ShardReducer` per shard, calls `reduce`, and the sealed @@ -90,8 +90,9 @@ class ShardReducer /// Merge `shard_deltas` (the caller's per-shard `BlobDelta` slice produced by `foldManifestEdges` /// and bucketed by `blobShard`) into a new in-degree generation for this shard. Writes the sealed - /// run under `blobTargetRunKey(new_generation, shard, 0)` via `backend`, appends its `RunRef` to - /// `out_runs`, and returns the `RunRef`. The call is idempotent (write-once via `putIfAbsent`). + /// run under `blobTargetRunKey(new_generation, shard, 0)` through `op`, appends its `RunRef` to + /// `out_runs`, and returns the `RunRef`. The call is idempotent: the run is written once, and a + /// second call finds the identical object already there. /// /// `prior_runs` are the parent generation's run segments for this shard, resolved BY THE CALLER from /// the parent fold seal's `blob_target_runs` filtered to `shard`. An empty vector is the @@ -100,13 +101,13 @@ class ShardReducer /// PRECONDITION: every `BlobDelta` in `shard_deltas` must be owned by this reducer /// (`blobShard(d.ref, gc_shards) == shard`). This is a caller contract; there is no /// underflow throw backstopping it — pass a misbucketed delta and the fold silently misroutes it. - std::vector reduce(Backend & backend, const Layout & layout, + std::vector reduce(CasOperation & op, const Layout & layout, const std::vector & prior_runs, uint64_t new_generation, uint64_t attempt, std::vector shard_deltas, uint64_t current_round = 0, uint64_t condemn_round = 0, - const std::function(const BlobRef &)> & head_blob = {}, - const std::function(const BlobRef &)> & peek_head = {}, + const BlobHeadFn & head_blob = {}, + const BlobHeadFn & peek_head = {}, const std::function & confirm_condemned_marker = {}, RetiredMergeResult * out_retired = nullptr, bool suppress_destructive = false, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp index e2eb6fc761e4..40839952a2b5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp @@ -6,33 +6,50 @@ namespace DB::Cas { -NamespaceJanitorResult NamespaceJanitor::runOnePage( - bool suppress_deletes, const std::function & fence_held) +namespace +{ + +/// The legacy `casPut` this write replaces reported a definite conflict as a value (never a failure to +/// this caller) and reported a store failure -- a refusal, an exhausted policy -- by throwing. Only +/// `Refused`/`GaveUp` are the alternatives a thrown exception used to carry, so only those propagate; +/// `Committed`/`Declined`/`Conflict` stay silent exactly as they did before. +void throwOnRefusedOrGaveUp(WriteResult && result, std::string_view what) +{ + if (std::holds_alternative(result) || std::holds_alternative(result)) + (void)orThrow(std::move(result), what); +} + +} + +NamespaceJanitorResult NamespaceJanitor::runOnePage(bool suppress_deletes, Liveness liveness) { NamespaceJanitorResult result; - const GcMaintenanceReadResult progress = readGcMaintenanceState(backend, layout); + CasOperation op = requests.admit(std::move(liveness)); + const GcMaintenanceReadResult progress = readGcMaintenanceState(op, layout); if (progress.status == GcMaintenanceReadStatus::Corrupt) { result.anomalies.push_back(progress.diagnostic); - (void)casGcMaintenanceState(backend, layout, progress.token, GcMaintenanceState{}); + throwOnRefusedOrGaveUp( + casGcMaintenanceState(op, layout, progress.incarnation, GcMaintenanceState{}, Retry::standard()), + "CAS namespace janitor: corrupt maintenance-state reset"); return result; } const String cursor = progress.state ? progress.state->janitor_cursor : String{}; - ListPage page; + KeyPage page; try { - page = backend.list(layout.namespaceRootPrefix(), cursor, page_budget); + page = op.list(layout.namespaceRootPrefix(), cursor, page_budget, Retry::standard()); } catch (...) { - (void)casGcMaintenanceState(backend, layout, progress.token, GcMaintenanceState{}); + (void)casGcMaintenanceState(op, layout, progress.incarnation, GcMaintenanceState{}, Retry::once()); throw; } result.pages = 1; result.keys = page.keys.size(); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); bool ambiguous = false; try { @@ -45,13 +62,16 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage( } /// A valid page is complete only when the round had deletion authority for every dead-life /// candidate on it. Advancing while the global gate is closed can phase-lock a dead page onto - /// every suppressed round and a different page onto every bounded forced fold. Ambiguous cuts and - /// observed fence loss have the same shape: retain the old cursor so an authoritative round - /// retries the exact page. Malformed keys, absent objects and token mismatches are final per-key - /// outcomes and therefore do not by themselves prevent progress. + /// every suppressed round and a different page onto every bounded forced fold. An ambiguous cut + /// retains the old cursor so an authoritative round retries the exact page; a lost liveness sample + /// only reaches this retained-cursor path when it is caught between the two `op.admitted()` checks + /// below -- a sample lost earlier throws out of a read verb (the maintenance read, the list, or a + /// HEAD) before this line is ever reached, ending the page by exception instead. Malformed keys, + /// absent objects and token mismatches are final per-key outcomes and therefore do not by + /// themselves prevent progress. bool page_decided = !ambiguous && !suppress_deletes; - for (const ListedKey & listed : page.keys) + for (const KeyEntry & listed : page.keys) { std::optional life_id; try @@ -83,15 +103,15 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage( if (ambiguous || suppress_deletes || catalog_cut.life_index.resolve(*life_id)) continue; - std::optional token = listed.token; - if (!token) + std::optional incarnation = listed.incarnation; + if (!incarnation) { try { - const HeadResult current = backend.head(listed.key); - if (!current.exists) + const std::optional current = op.head(listed.key, Retry::standard()); + if (!current) continue; - token = current.token; + incarnation = current->incarnation; } catch (const std::exception & e) { @@ -101,14 +121,14 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage( continue; } } - if (!fence_held()) + if (!op.admitted()) { page_decided = false; break; } try { - if (backend.deleteExact(listed.key, *token).kind == DeleteOutcome::Kind::Deleted) + if (op.remove(listed.key, *incarnation, Retry::standard()) == Removal::Removed) ++result.deleted; } catch (const std::exception & e) @@ -122,7 +142,7 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage( /// Recheck even when the page had no dead candidate. A tenure that observes fence loss after LIST /// or after the last exact delete must not publish progress. Loss after this check may still race /// with the leak-only maintenance CAS; already completed exact deletes remain safe to repeat. - if (page_decided && !fence_held()) + if (page_decided && !op.admitted()) page_decided = false; if (page_decided) @@ -130,7 +150,9 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage( const GcMaintenanceState next{.janitor_cursor = page.next_cursor}; try { - (void)casGcMaintenanceState(backend, layout, progress.token, next); + const WriteResult published = casGcMaintenanceState(op, layout, progress.incarnation, next, Retry::standard()); + if (std::holds_alternative(published) || std::holds_alternative(published)) + result.anomalies.push_back("cursor publication did not commit"); } catch (const std::exception & e) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.h index e60f87e2e6a0..d23d21402c73 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -20,13 +20,21 @@ struct NamespaceJanitorResult class NamespaceJanitor { public: - NamespaceJanitor(Backend & backend_, const Layout & layout_, size_t page_budget_) - : backend(backend_), layout(layout_), page_budget(page_budget_) {} + NamespaceJanitor(CasRequests & requests_, const Layout & layout_, size_t page_budget_) + : requests(requests_), layout(layout_), page_budget(page_budget_) {} - NamespaceJanitorResult runOnePage(bool suppress_deletes, const std::function & fence_held); + /// `liveness` is admitted once for the whole page (one `CasOperation` covers the read, the list, + /// every delete and the cursor publication): a fact the fence cannot see, such as "this tenure + /// still holds the GC round's own lease" -- see `CasRequests::admit`. It is SAMPLED BEFORE EVERY + /// REQUEST the page makes (and before every reissue of one), not just at the two points this + /// function itself checks `op.admitted()` -- so it must be cheap and must never throw. A sample + /// that returns false ends whichever request was about to be sent: a read verb (the maintenance + /// read, the list, a HEAD) throws out of this call, and a write verb (a delete, the cursor + /// publication) reports it as `GaveUp` rather than sending anything. + NamespaceJanitorResult runOnePage(bool suppress_deletes, Liveness liveness); private: - Backend & backend; + CasRequests & requests; const Layout & layout; size_t page_budget; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp index 1a4cdf2e551c..e9c7cde963ef 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp @@ -1,5 +1,5 @@ #include -#include +#include #include #include #include @@ -42,7 +42,7 @@ void onGcEnumerationPage() /// mount there is no deletion authority, so the caller must leave the prefix untouched. The mount's /// `writer_epoch` and `min_active_build_sequence` are the single durable epoch/floor pair used for eligibility, including /// across process replacement and the retired sentinel. -std::optional floorForNamespace(Pool & store, const RootNamespace & ns) +std::optional floorForNamespace(CasOperation & op, const Layout & layout, const RootNamespace & ns) { const String & value = ns.string(); size_t pos = value.size(); @@ -55,7 +55,7 @@ std::optional floorForNamespace(Pool & store, const RootNamespace & const String server_root_id = value.substr(0, pos); if (!server_root_id.empty()) { - if (const auto got = store.backend().get(store.layout().mountKey(server_root_id))) + if (const auto got = op.read(layout.mountKey(server_root_id), Retry::standard())) return decodeMountLease(got->bytes); } if (pos == 0) @@ -91,14 +91,13 @@ std::optional parseListedManifestObject(const Layout & lay /// The fold seal `gc/state` currently adopts, or `nullopt` when the pool has no `gc/state` or no seal /// at `(snap_generation, snap_attempt)` — a pool whose GC has never completed a round. It is read ONCE /// per sweep pass; every namespace the pass touches takes its coverage row out of the same object. -std::optional readAdoptedFoldSeal(Pool & store) +std::optional readAdoptedFoldSeal(CasOperation & op, const Layout & layout) { - const Layout & layout = store.layout(); - const auto state_got = store.backend().get(layout.gcStateKey()); + const auto state_got = op.read(layout.gcStateKey(), Retry::standard()); if (!state_got) return std::nullopt; const GcState state = decodeGcState(state_got->bytes); - const auto got = store.backend().get(layout.foldSealKey(state.snap_generation, state.snap_attempt)); + const auto got = op.read(layout.foldSealKey(state.snap_generation, state.snap_attempt), Retry::standard()); if (!got) return std::nullopt; return decodeFoldSeal(got->bytes, state.snap_generation); @@ -181,7 +180,7 @@ struct NamespaceProtection /// Reaching the budget before even calling `recoverRefTableDetailedFromAuthority` (already spent by an /// earlier namespace) skips that call entirely and reports incomplete immediately. NamespaceProtection activeManifestKeys( - Pool & store, const CatalogEntry & catalog_entry, const RefCkpt & ckpt, + CasOperation & op, const Layout & layout, const CatalogEntry & catalog_entry, const RefCkpt & ckpt, const std::optional & coverage, GcRoundWorkBudget * work_budget = nullptr) { NamespaceProtection protection; @@ -191,8 +190,6 @@ NamespaceProtection activeManifestKeys( return protection; } std::set & active = protection.active; - const Layout & layout = store.layout(); - Backend & backend = store.backend(); const RootNamespace & ns = catalog_entry.ns; const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(catalog_entry.ns, catalog_entry.incarnation); @@ -200,7 +197,7 @@ NamespaceProtection activeManifestKeys( /// The exact row and `_ckpt` come from the caller's frozen catalog cut. Do not resolve `ns` here: /// a later catalog cut can name a reborn life and turn this old life into an apparent orphan. const RecoveredRefTable recovered = recoverRefTableDetailedFromAuthority( - backend, layout, catalog_entry, ckpt); + op, layout, catalog_entry, ckpt); if (work_budget) ++work_budget->sweep_recovery_ops_used; /// one coarse unit for the snapshot+tail recovery itself const RefTableState & state = recovered.state; @@ -228,7 +225,7 @@ NamespaceProtection activeManifestKeys( renderRefTxnId(from_cursor)); const RefTxnId exact_next_epoch{from_cursor.writer_epoch + 1, 1}; const EpochCrossResult crossing = crossEpochFromSeal( - backend, layout, ns, from_cursor, std::nullopt, exact_next_epoch, life); + op, layout, ns, from_cursor, std::nullopt, exact_next_epoch, life); if (!crossing.proved()) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS orphan sweep: exact next-epoch record {} does not prove that folded cursor {} was its seal " @@ -252,7 +249,7 @@ NamespaceProtection activeManifestKeys( /// cursor, first try its ordinary same-epoch successor; only a 404 there can ask the shared /// chain proof to establish a cross with kind unknown. That preserves tails after a cleaned /// cursor without guessing that a missing cursor was a seal. - const auto cursor_got = backend.get(layout.refLogKey(life, cursor)); + const auto cursor_got = op.read(layout.refLogKey(life, cursor), Retry::standard()); if (cursor_got) { const RefLogTxn cursor_txn = decodeRefLogTxn( @@ -301,7 +298,7 @@ NamespaceProtection activeManifestKeys( protection.recovery_incomplete = true; break; } - const auto got = backend.get(layout.refLogKey(life, id)); + const auto got = op.read(layout.refLogKey(life, id), Retry::standard()); if (work_budget) ++work_budget->sweep_recovery_ops_used; if (!got) @@ -353,10 +350,11 @@ NamespaceProtection activeManifestKeys( NamespaceFoldView namespaceFoldView(Pool & store, const RootNamespace & ns) { + CasOperation op = store.gcRequests().admit(); NamespaceFoldView view; - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(store.backend(), store.layout()); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, store.layout()); catalog_cut.life_index.throwIfAmbiguous("CAS orphan manifest sweep"); - view.coverage = coverageOf(readAdoptedFoldSeal(store), catalog_cut, ns); + view.coverage = coverageOf(readAdoptedFoldSeal(op, store.layout()), catalog_cut, ns); return view; } @@ -473,13 +471,16 @@ bool manifestDeletionPremise(const NamespaceFoldView & view, const ManifestKey & return true; } -bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix) +namespace { - /// Eligibility comes only from the durable mount-lease floor. A missing floor means NOT eligible; - /// do not replace that authority check with a frozen-sequence or judged-dead guess. Compare - /// `writer_epoch` first, then `build_sequence`, so old-epoch - /// debris drains after a process restart even when its build_sequence is above the current min_active_build_sequence. - const auto floor = floorForNamespace(store, ns); + +/// Eligibility comes only from the durable mount-lease floor. A missing floor means NOT eligible; +/// do not replace that authority check with a frozen-sequence or judged-dead guess. Compare +/// `writer_epoch` first, then `build_sequence`, so old-epoch +/// debris drains after a process restart even when its build_sequence is above the current min_active_build_sequence. +bool prefixEligibleOn(CasOperation & op, const Layout & layout, const RootNamespace & ns, const BuildPrefix & prefix) +{ + const auto floor = floorForNamespace(op, layout, ns); if (!floor) return false; @@ -493,14 +494,21 @@ bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & return w.min_active_build_sequence > prefix.build_sequence; } +} + +bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix) +{ + CasOperation op = store.gcRequests().admit(); + return prefixEligibleOn(op, store.layout(), ns, prefix); +} + uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix, std::vector * warnings) { - if (!prefixEligible(store, ns, prefix)) - return 0; /// not eligible by the durable watermark fact — delete nothing (controls #8/#9) - const Layout & layout = store.layout(); - Backend & backend = store.backend(); + CasOperation op = store.gcRequests().admit(); + if (!prefixEligibleOn(op, layout, ns, prefix)) + return 0; /// not eligible by the durable watermark fact — delete nothing (controls #8/#9) /// Build the protection view. A missing snapshot body, an invalid transaction, or an incomplete /// ordered view throws, causing the sweep to skip deletion and surface the error; it never substitutes @@ -512,9 +520,9 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi /// the same way the periodic sweep always has: skip and retry next round. /// The §6 premise's durable half, read before the protection view so both share one seal read. NamespaceFoldView view; - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(store.backend(), store.layout()); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); catalog_cut.life_index.throwIfAmbiguous("CAS orphan manifest sweep"); - view.coverage = coverageOf(readAdoptedFoldSeal(store), catalog_cut, ns); + view.coverage = coverageOf(readAdoptedFoldSeal(op, layout), catalog_cut, ns); const CatalogEntry * catalog_entry = catalogEntryOf(catalog_cut, ns); if (!catalog_entry || catalog_entry->state == NsState::Creating) return 0; /// absent/Creating names have no recovery authority and therefore no deletion authority @@ -523,7 +531,7 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi try { const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(catalog_entry->ns, catalog_entry->incarnation); - const std::optional ckpt = readCkpt(backend, layout, life); + const std::optional ckpt = readCkpt(op, layout, life); if (!ckpt) { const String warning = "CAS orphan sweep: namespace " + ns.string() @@ -533,7 +541,7 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi warnings->push_back(warning); return 0; } - protection = activeManifestKeys(store, *catalog_entry, ckpt->ckpt, view.coverage); + protection = activeManifestKeys(op, layout, *catalog_entry, ckpt->ckpt, view.coverage); view.tail_removal_targets = protection.tail_removal_targets; } @@ -554,10 +562,10 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi + renderRefTxnId(RefTxnId{prefix.writer_epoch, prefix.build_sequence}) + "/"; uint64_t deleted = 0; - forEachListedKey(backend, prefix_key, [&](const ListedKey & listed) + op.forEachListedKey(prefix_key, [&](const KeyEntry & listed) { if (protection.active.contains(listed.key)) - return; /// owned by a committed or precommit owner — never sweep + return true; /// owned by a committed or precommit owner — never sweep /// THE §6 SAFETY FLOOR, under the watermark eligibility already established above. The /// watermark says the build is retired; the premise says the ref stream can be shown not to @@ -569,22 +577,23 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi "CAS orphan sweep: retaining {} -- {}", listed.key, retain_reason); if (warnings) warnings->push_back("CAS orphan sweep: retained " + listed.key + " -- " + retain_reason); - return; + return true; } - /// Exact-token delete: HEAD for the current token, then deleteExact. A 404 between HEAD and - /// delete (or a TokenMismatch — a fresh owner reclaimed it) is tolerated (record-and-continue), - /// same as always, regardless of `warnings` -- that is the normal "someone else already reclaimed - /// it" race, not a failure. A THROWN exception (a transient backend hiccup) is the one thing - /// `warnings` changes: opted-in (non-null), it is recorded and the sweep moves to the next key; - /// opted-out (nullptr, every pre-existing caller), it propagates exactly as before (fail-close). + /// Exact-token delete: HEAD for the current incarnation, then remove exactly it -- never + /// `removeCurrent`, whose re-head would delete whatever a fresh owner put there instead. A + /// `Gone` (a 404 between the two) or a `Mismatch` (a fresh owner reclaimed the key) is + /// tolerated, same as always, regardless of `warnings` -- that is the normal "someone else + /// already reclaimed it" race, not a failure. A THROWN exception (a transient backend hiccup) + /// is the one thing `warnings` changes: opted-in (non-null), it is recorded and the sweep moves + /// to the next key; opted-out (nullptr, every pre-existing caller), it propagates exactly as + /// before (fail-close). try { - const HeadResult head = backend.head(listed.key); - if (!head.exists) - return; - const DeleteOutcome outcome = backend.deleteExact(listed.key, head.token); /// NotFound/TokenMismatch spared - if (classifyDeleteOutcome(outcome) == DeleteClass::Deleted) + const std::optional head = op.head(listed.key, Retry::standard()); + if (!head) + return true; + if (op.remove(listed.key, head->incarnation, Retry::standard()) == Removal::Removed) ++deleted; } catch (...) @@ -594,7 +603,8 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi warnings->push_back("CAS orphan sweep: " + listed.key + " delete failed: " + getCurrentExceptionMessage(/*with_stacktrace=*/false)); } - }, 1000, onGcEnumerationPage); + return true; + }, Retry::standard(), 1000, onGcEnumerationPage); return deleted; } @@ -611,35 +621,36 @@ ManifestSweepResult planManifestCursorPage( if (list_budget == 0) return result; - Backend & backend = store.backend(); const Layout & layout = store.layout(); - const ListPage page = backend.list(layout.casManifestsPrefix(), cursor, list_budget); + CasOperation op = store.gcRequests().admit(); + const KeyPage page = op.list(layout.casManifestsPrefix(), cursor, list_budget, Retry::standard()); /// This pass fetches exactly one page per round (the cursor advances across rounds, not within this /// call), so the metric increments once per call, not once per listed key. ProfileEvents::increment(ProfileEvents::CASGCEnumerationPages); /// Freeze every possible destructive candidate BEFORE the later catalog cut. A same-name rebirth can /// replace this logical manifest key between the observations; classifying the old bytes against the - /// later lifecycle cut is safe only when deletion retains the old exact token, so the replacement - /// loses `deleteExact`. Do not take a fresh GET after the catalog read: that would splice new-life - /// bytes into old candidate selection and authorize their deletion with the new token. + /// later lifecycle cut is safe only when deletion retains the old exact incarnation, so the + /// replacement fails the caller's re-observation. Do not take a fresh read after the catalog read: + /// that would splice new-life bytes into old candidate selection and authorize their deletion with + /// the new incarnation. /// /// Bounded to `nomination_budget` well-formed keys — never the whole `list_budget`-sized /// page — since `nomination_budget` is the hard ceiling on how many of them this call can ever /// nominate. A well-formed key beyond this cap has no frozen body; it is retained where its absence /// is discovered below, in the exact same "budget exhausted, cursor does not step over it" shape the /// nomination-count exhaustion already uses. - std::map> observed_candidates; + std::map> observed_candidates; if (nomination_budget > 0) { uint64_t frozen = 0; - for (const ListedKey & listed : page.keys) + for (const KeyEntry & listed : page.keys) { if (frozen >= nomination_budget) break; if (parseListedManifestObject(layout, listed.key)) { - observed_candidates.emplace(listed.key, backend.get(listed.key)); + observed_candidates.emplace(listed.key, op.read(listed.key, Retry::standard())); ++frozen; } } @@ -647,8 +658,8 @@ ManifestSweepResult planManifestCursorPage( /// One seal and one later catalog cut for the whole page; every namespace joins through those same /// immutable observations. - const std::optional adopted_seal = readAdoptedFoldSeal(store); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(backend, layout); + const std::optional adopted_seal = readAdoptedFoldSeal(op, layout); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); catalog_cut.life_index.throwIfAmbiguous("CAS orphan manifest sweep"); std::map eligible_by_prefix; @@ -657,12 +668,12 @@ ManifestSweepResult planManifestCursorPage( std::set errored_namespaces; /// protection view unavailable => skip, never delete /// The key of the last candidate this page actually DECIDED on. The cursor resumes strictly after - /// it (`ListPage::next_cursor` is the last returned key), so a candidate the page never decided on + /// it (`KeyPage::next_cursor` is the last returned key), so a candidate the page never decided on /// stays ahead of the cursor and is examined next pass. See the budget rule below. String decided_through; bool budget_exhausted = false; - for (const ListedKey & listed : page.keys) + for (const KeyEntry & listed : page.keys) { ++result.listed; @@ -767,7 +778,7 @@ ManifestSweepResult planManifestCursorPage( { const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry( catalog_entry->ns, catalog_entry->incarnation); - const std::optional ckpt = readCkpt(backend, layout, life); + const std::optional ckpt = readCkpt(op, layout, life); if (!ckpt) { LOG_WARNING(getLogger("CasOrphanManifestSweep"), @@ -778,7 +789,7 @@ ManifestSweepResult planManifestCursorPage( else { NamespaceProtection protection = activeManifestKeys( - store, *catalog_entry, ckpt->ckpt, view_it->second.coverage, work_budget); + op, layout, *catalog_entry, ckpt->ckpt, view_it->second.coverage, work_budget); if (protection.recovery_incomplete) { /// The committed-tail walk stopped early: `active`/`tail_removal_targets` @@ -854,8 +865,9 @@ ManifestSweepResult planManifestCursorPage( } } - /// This exact token and bytes were captured before the catalog cut (see above). A missing body - /// has no deletion authority; a later replacement loses the old token at `deleteExact`. + /// This exact incarnation and bytes were captured before the catalog cut (see above). A missing + /// body has no deletion authority; a later replacement no longer matches what is recorded here, + /// so the caller's re-observation refuses the delete. /// /// A well-formed key can legitimately be ABSENT here: the freeze loop above caps /// fan-out at `nomination_budget` candidates, so a key beyond that cap was never frozen. Treat @@ -868,7 +880,7 @@ ManifestSweepResult planManifestCursorPage( ++result.skipped; continue; } - const std::optional & got = observed_it->second; + const std::optional & got = observed_it->second; if (!got) { ++result.skipped; @@ -904,7 +916,7 @@ ManifestSweepResult planManifestCursorPage( ManifestSweepResult::Nomination nomination{ .id = id, .key = parsed->key, - .token = got->token, + .token = PersistedIncarnation::capture(got->incarnation), .source_retirements = {}}; for (const ManifestEntry & entry : body->entries) if (entry.placement == EntryPlacement::Blob) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h index e5ae9e3176a4..1d5b9a0dbcb8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h @@ -132,12 +132,13 @@ struct ManifestSweepResult uint64_t retained_work_budget = 0; /// Exact-GET/decode candidates. The reducer must adopt every `source_retirements` entry before the - /// caller may exact-token-delete `key` with `token`. + /// caller may delete `key`, and only after re-observing `token` at it: a key whose incarnation + /// moved on belongs to a fresh owner and must be left alone. struct Nomination { ManifestId id; String key; - Token token; + PersistedIncarnation token; std::vector source_retirements; }; std::vector nominations; @@ -161,19 +162,19 @@ struct ManifestSweepResult /// - a 404 between listing and deletion is record-and-continue, never a throw; /// - never GETs a condemned body to revive it — eligibility + /// exact-token delete only. -/// Returns the number of bodies actually deleted (a `DeleteClass::Deleted`-classified exact-token -/// delete only, never a spared `NotFound`/`TokenMismatch`) — the decommission manifest-debris drain +/// Returns the number of bodies actually deleted (a `Removal::Removed` exact-token +/// delete only, never a spared `Gone`/`Mismatch`) — the decommission manifest-debris drain /// (`Core/CasDecommission.cpp`) sums this across every eligible build prefix into /// `DecommissionReport::manifest_debris_removed`. /// /// `warnings`, when non-null, opts in to the decommission drain's tolerate-and-continue contract: a -/// per-key transient failure (a thrown backend exception on `head`/`deleteExact`) +/// per-key transient failure (a thrown backend exception on `head`/`remove`) /// is pushed onto `*warnings` and the sweep continues with the next key, instead of throwing out of /// this call; likewise a protection-view-unavailable namespace (the pre-existing corrupt-snapshot skip /// below) also pushes a "cannot confirm emptiness" warning, not just a `LOG_WARNING`. `warnings == /// nullptr` (the default, every pre-existing caller) preserves the original behaviour exactly: a /// per-key failure propagates as an exception (fail-close default), and the protection-view skip is -/// log-only. `NotFound`/`TokenMismatch` delete outcomes stay silently spared either way — those are the +/// log-only. `Gone`/`Mismatch` delete outcomes stay silently spared either way — those are the /// normal "a fresh owner reclaimed it" race the periodic sweep expects, not a failure to warn about. /// This direct decommission path relies on the caller's held server-root claim/fence: while that claim /// is held, a same-server-root rebirth cannot become live between its catalog cut and exact-token delete. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp index 899f93504fde..321e6fdb467e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp @@ -18,14 +18,10 @@ namespace DB::Cas { CatalogLifecycleReconciler::CatalogLifecycleReconciler( - Backend & backend_, const Layout & layout_, const CasFoldSeal & adopted_parent_, - uint64_t admitted_generation_, - std::function check_fence_) - : backend(backend_) + CasOperation & op_, const Layout & layout_, const CasFoldSeal & adopted_parent_) + : op(op_) , layout(layout_) , adopted_parent(adopted_parent_) - , admitted_generation(admitted_generation_) - , check_fence(std::move(check_fence_)) { } @@ -62,7 +58,8 @@ CatalogResolution CatalogLifecycleReconciler::resolveExactRow( return CatalogResolution::ExactRowStillPresent; } -CatalogLifecycleReconcileResult CatalogLifecycleReconciler::reconcile() +CatalogLifecycleReconcileResult CatalogLifecycleReconciler::reconcile( + const std::function & refresh_authority) { CatalogLifecycleReconcileResult result{ .authority_status = AuthorityStatus::Authoritative, @@ -70,14 +67,19 @@ CatalogLifecycleReconcileResult CatalogLifecycleReconciler::reconcile() .retired_lives = {}, .final_catalog_cut = std::nullopt, .deleted = 0}; - CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); + CasRefCatalog::Snapshot catalog = CasRefCatalog::read(op, layout); for (;;) { const std::optional eligible = selectEligible(catalog); if (!eligible) { - if (check_fence(admitted_generation) == CasRefCatalog::LeaderFenceStatus::Moved) + /// The verdict gets its own refresh, not just each erase: a deposition landing after the + /// last erase is invisible to the reading that erase took, so without this the drain hands + /// a deposed leader `Authoritative` and the round only learns better at its `gc/state` + /// commit -- after a ref walk and a fold seal it never had the authority to build. + refresh_authority(); + if (!op.admitted()) { result.authority_status = AuthorityStatus::FencedOut; return result; @@ -89,8 +91,7 @@ CatalogLifecycleReconcileResult CatalogLifecycleReconciler::reconcile() CasRefCatalog::CompletedRemovingDeleteResult delete_result = CasRefCatalog::deleteCompletedRemovingAtSnapshot( - backend, layout, std::move(catalog), *eligible, adopted_parent, - admitted_generation, check_fence); + op, layout, std::move(catalog), *eligible, adopted_parent, refresh_authority); if (!delete_result.catalog_snapshot) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS catalog lifecycle reconciliation returned no catalog resolution snapshot"); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.h index fd34011c64ef..1069bc9d63f5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.h @@ -43,22 +43,22 @@ class CatalogLifecycleReconciler { public: CatalogLifecycleReconciler( - Backend & backend_, const Layout & layout_, const CasFoldSeal & adopted_parent_, - uint64_t admitted_generation_, - std::function check_fence_); + CasOperation & op_, const Layout & layout_, const CasFoldSeal & adopted_parent_); - CatalogLifecycleReconcileResult reconcile(); + /// `refresh_authority` is forwarded to each erase, which runs it at the top of every attempt, so + /// every erase is authorised by a reading taken in its own attempt, and it is run once more before + /// the drain-complete verdict, which would otherwise report from the reading its last erase left. + /// It is a refresh, not a verdict: the verdict stays `op.admitted()`. + CatalogLifecycleReconcileResult reconcile(const std::function & refresh_authority); private: std::optional selectEligible(const CasRefCatalog::Snapshot & catalog) const; static CatalogResolution resolveExactRow( const CasRefCatalog::Snapshot & catalog, const CatalogEntry & observed); - Backend & backend; + CasOperation & op; const Layout & layout; const CasFoldSeal & adopted_parent; - uint64_t admitted_generation; - std::function check_fence; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp index d1c44d2b168f..f483cc204f19 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp @@ -1,5 +1,4 @@ #include -#include #include @@ -13,34 +12,33 @@ namespace ProfileEvents namespace DB::Cas { -std::optional loadMeta(Backend & backend, const Layout & layout, const BlobRef & ref) +std::optional loadMeta(CasOperation & op, const Layout & layout, const BlobRef & ref) { - const String key = layout.blobMetaKey(ref); - auto got = backend.get(key); + auto got = op.read(layout.blobMetaKey(ref), Retry::standard()); if (!got) return std::nullopt; - return LoadedMeta{.meta = decodeBlobMeta(got->bytes), .etag = got->token}; + return LoadedMeta{.meta = decodeBlobMeta(got->bytes), .incarnation = std::move(got->incarnation)}; } -CasOverwriteResult putMetaIfAbsent(Pool & pool, const BlobRef & ref, const BlobMeta & meta) +WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const BlobRef & ref, + const BlobMeta & meta) { ProfileEvents::increment(ProfileEvents::CASMetaPut); - const String key = pool.layout().blobMetaKey(ref); - return pool.stagingPutIfAbsentMutable(key, encodeBlobMeta(meta)); + return op.create(layout.blobMetaKey(ref), encodeBlobMeta(meta), Retry::standard()); } -CasOverwriteResult casMeta(Pool & pool, const BlobRef & ref, const Token & expected, const BlobMeta & meta) +WriteResult casMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, + const Incarnation & expected, const BlobMeta & meta) { ProfileEvents::increment(ProfileEvents::CASMetaCompareSwap); - const String key = pool.layout().blobMetaKey(ref); - return pool.stagingConditionalOverwrite(key, encodeBlobMeta(meta), expected); + return op.replace(layout.blobMetaKey(ref), encodeBlobMeta(meta), expected, Retry::standard()); } -DeleteOutcome deleteMetaExact(Backend & backend, const Layout & layout, const BlobRef & ref, const Token & expected) +Removal deleteMetaExact(CasOperation & op, const Layout & layout, const BlobRef & ref, + const Incarnation & expected) { ProfileEvents::increment(ProfileEvents::CASMetaDelete); - const String key = layout.blobMetaKey(ref); - return backend.deleteExact(key, expected); + return op.remove(layout.blobMetaKey(ref), expected, Retry::standard()); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h index c6faa5021519..b10364d6c445 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h @@ -1,7 +1,6 @@ #pragma once -#include -#include +#include #include #include #include @@ -11,55 +10,49 @@ namespace DB::Cas { -class Pool; - -/// A decoded blob meta record together with the backend token observed for the same incarnation. -/// The token is returned with the decoded record because the next conditional update or exact delete -/// must be guarded by the version that was actually read; comparing encoded meta bytes would not -/// provide that protection. +/// A decoded blob meta record together with the incarnation the same read observed. The incarnation +/// travels with the record because the next conditional update or exact delete must be guarded by the +/// version that was actually read; comparing encoded meta bytes would not provide that protection. struct LoadedMeta { BlobMeta meta; - Token etag; + Incarnation incarnation; }; /// Shared lifecycle operations for the blob freshness marker used by the writer and GC. The key is /// built from the complete `BlobRef`, so each algorithm uses its own digest representation and no /// pool-wide digest width is threaded through these functions. The marker is a point-read hint rather -/// than the blob lifetime's linearization point: the blob body's incarnation tag and exact-token body -/// deletion provide the safety guarantee, while a stale marker can at most make a writer re-upload. +/// than the blob lifetime's linearization point: the blob body's incarnation tag and exact-incarnation +/// body deletion provide the safety guarantee, while a stale marker can at most make a writer +/// re-upload. /// /// `loadMeta` is used in the adopt path, so its backend must provide strong read-after-write -/// consistency: after a successful meta write, the one subsequent GET must observe that write. -/// Conditional updates and deletion use the backend token, not the encoded meta bytes. +/// consistency: after a successful meta write, the one subsequent read must observe that write. +/// Conditional updates and deletion use the observed incarnation, not the encoded meta bytes. /// -/// Returns the current decoded marker and its conditional token, or nullopt when the meta key is -/// absent. Decoding errors propagate as exceptions. -std::optional loadMeta(Backend & backend, const Layout & layout, const BlobRef & ref); - -/// Creates the marker only when its key is absent, controlled: a SlowDown/429/5xx on the attempt is -/// resolved-and-reissued within budget rather than escaping as a raw client error (triage: S22 RCA). -/// A precondition failure (another -/// writer already created the marker -- possibly with a DIFFERENT record, e.g. a stale `Condemned` -/// marker still present when a vanished body is freshly re-uploaded) is reported as -/// `CasOverwriteOutcome::Conflict`, never thrown -- this uses `putIfAbsentControlledMutable`, NOT the -/// ref-log lane's `putIfAbsentControlled` (that method's resolve throws `CORRUPTED_DATA` on any -/// different bytes at the key, which is correct for the ref-log's immutable content-addressed keys -/// but wrong for this mutable marker, where a pre-existing different value is an expected, non-corrupt -/// outcome). -CasOverwriteResult putMetaIfAbsent(Pool & pool, const BlobRef & ref, const BlobMeta & meta); - -/// Replaces the marker only when its current backend token equals `expected`, controlled (same -/// budgeted resolve-and-reissue as putMetaIfAbsent). A genuine conflict (current token AND bytes both -/// differ from what this call intended) is reported as `CasOverwriteOutcome::Conflict`, never thrown -- -/// exactly like the previous uncontrolled `CasResult` contract -- so the caller's existing -/// reload-and-retry metadata reconciliation in `PartWriteTxn::ensureBlobPresent` keeps working unchanged. -CasOverwriteResult casMeta(Pool & pool, const BlobRef & ref, const Token & expected, const BlobMeta & meta); - -/// Deletes only the marker incarnation identified by `expected`. A token mismatch leaves the current -/// marker untouched; `NotFound` is distinct from that case so callers can tell absence from a raced -/// replacement. The backend's complete `DeleteOutcome` is returned, including any storage-specific -/// delete-marker status. -DeleteOutcome deleteMetaExact(Backend & backend, const Layout & layout, const BlobRef & ref, const Token & expected); +/// Returns the current decoded marker and the incarnation to guard the next write with, or nullopt +/// when the meta key is absent. Decoding errors propagate as exceptions. +std::optional loadMeta(CasOperation & op, const Layout & layout, const BlobRef & ref); + +/// Creates the marker only when its key is absent, on the plane `op` belongs to -- like its siblings, +/// so one caller's decision cannot end up split across two fences. Anything at the key that this call +/// did not itself write -- a stale `Condemned` marker still present when a vanished body is freshly +/// re-uploaded, or a racing writer's byte-identical marker -- comes back as `Conflict` carrying what +/// was observed, never as a throw: this marker is mutable, so a pre-existing different value is an +/// expected outcome rather than corruption. +WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const BlobRef & ref, + const BlobMeta & meta); + +/// Replaces the marker only when its current incarnation is `expected`, on the plane `op` belongs to. +/// A competing write is reported as `Conflict` carrying what the resolve read observed, never thrown, +/// so the caller's own reload-and-retry reconciliation decides what to do about it. +WriteResult casMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, + const Incarnation & expected, const BlobMeta & meta); + +/// Deletes only the marker incarnation named by `expected`. `Mismatch` leaves the current marker +/// untouched and is distinct from `Gone`, so callers can tell absence from a raced replacement. A +/// versioned bucket that archives instead of reclaiming raises `CAS_DELETE_MARKER`. +Removal deleteMetaExact(CasOperation & op, const Layout & layout, const BlobRef & ref, + const Incarnation & expected); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp index a9edd6783c8e..dc43ce62f347 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.cpp @@ -30,9 +30,9 @@ namespace DB::Cas { CasManifestReader::CasManifestReader( - Backend & backend_, const Layout & layout_, const PoolMeta & meta_, + CasRequests & requests_, const Layout & layout_, const PoolMeta & meta_, const CasEventSink & event_sink_, size_t manifest_decode_cache_bytes) - : backend(backend_), layout(layout_), meta(meta_), event_sink(event_sink_) + : requests(requests_), layout(layout_), meta(meta_), event_sink(event_sink_) { if (manifest_decode_cache_bytes > 0) manifest_cache = std::make_unique( @@ -52,7 +52,8 @@ std::shared_ptr CasManifestReader::readManifestShared(const /// (`INV-NO-DANGLE`). Never substitute an empty manifest: callers must observe the missing object /// as an exception. The `GET` alone carries the absence signal, so no `HEAD` precedes it. const String key = layout.manifestKey(id); - std::optional object = backend.get(key); + CasOperation op = requests.admit(); + std::optional object = op.read(key, Retry::standard()); if (!object) { if (event_sink) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h index e3aa15955743..67a124a16889 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasManifestReader.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -28,17 +28,18 @@ struct BlobLocation /// into the cache. A missing body, a decode failure or a failed identity check is surfaced as an /// exception, never as an empty or partially trusted manifest. /// -/// The reader receives its backend, immutable layout and pool metadata, and event sink by reference; -/// it has no `Pool` back-reference and owns no `Pool`-level mutex. The decode cache is a -/// byte-weighted `CacheBase` LRU whose synchronization is internal to `CacheBase`; a null cache -/// means caching is disabled (`manifest_decode_cache_bytes == 0`). +/// The reader receives its `CasRequests`, immutable layout and pool metadata, and event sink by +/// reference; it has no `Pool` back-reference and owns no `Pool`-level mutex. Each cache-miss read +/// admits its own operation, so a lost mount lease refuses a miss the same way any other read does. +/// The decode cache is a byte-weighted `CacheBase` LRU whose synchronization is internal to +/// `CacheBase`; a null cache means caching is disabled (`manifest_decode_cache_bytes == 0`). class CasManifestReader { public: /// Binds the reader to the pool environment. A positive cache budget creates the byte-weighted /// LRU; zero disables caching while leaving the one-`GET`-and-validate sequence intact. CasManifestReader( - Backend & backend_, const Layout & layout_, const PoolMeta & meta_, + CasRequests & requests_, const Layout & layout_, const PoolMeta & meta_, const CasEventSink & event_sink_, size_t manifest_decode_cache_bytes); /// Reads a manifest by value using the fail-closed sequence described above. A missing body, @@ -75,7 +76,7 @@ class CasManifestReader }; using ManifestDecodeCache = CacheBase, PartManifestWeight>; - Backend & backend; + CasRequests & requests; const Layout & layout; const PoolMeta & meta; const CasEventSink & event_sink; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp index 1b6a57d331dd..c5e61f3a1471 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp @@ -34,7 +34,6 @@ namespace ProfileEvents namespace DB::Cas { -void reportMountRenewProgress(const CasOverwriteProgress & progress) noexcept; void reportMountRenewCompletion(const MountRenewResult & result) noexcept; void configureMountRenewObservability( const String * server_root_id, const CasEventSink * event_sink, bool deferred) noexcept; @@ -53,6 +52,8 @@ int64_t wallClockNowSeconds() CasMountRuntime::CasMountRuntime( BackendPtr backend_ptr_, + CasRequests & mount_requests_, + CasRequests & farewell_requests_, const Layout & layout_, MountConfig config_, String server_root_id_, @@ -60,6 +61,8 @@ CasMountRuntime::CasMountRuntime( CasRequestBudget cas_request_budget_, std::function remount_attempt_) : backend_ptr(std::move(backend_ptr_)) + , mount_requests(mount_requests_) + , farewell_requests(farewell_requests_) , layout(layout_) , config(std::move(config_)) , server_root_id(std::move(server_root_id_)) @@ -132,18 +135,27 @@ void CasMountRuntime::checkFenceOrThrow(uint64_t admitted_generation) const "system.cas_mounts for the disk's lifecycle before retrying"); } -bool CasMountRuntime::refAppendFenceOk() const +Fence::Admit CasMountRuntime::admit(uint64_t admitted_generation, uint64_t needed_ms) const { - /// `mayMutate` checks the latch and deadline. The additional budget check prevents starting a - /// controlled request that cannot plausibly finish, including its safety margin, before expiry. - if (mount_fence.lost.load(std::memory_order_acquire)) - return false; + if (mount_fence.lost.load(std::memory_order_acquire) || fenceGeneration() != admitted_generation) + return Fence::Admit::LostOrRearmed; const uint64_t now = bootMsNow(); const uint64_t deadline = mount_fence.deadline_boot_ms.load(std::memory_order_acquire); if (now >= deadline) - return false; - const uint64_t margin = cas_request_budget.attempt_timeout_ms + cas_request_budget.lease_safety_margin_ms; - return margin < deadline - now; + return Fence::Admit::NoBudget; + /// Compared by subtraction rather than as the sum `needed_ms + margin`, which can wrap for an + /// absurd configuration and then read as if there were room. + const uint64_t remaining = deadline - now; + if (needed_ms >= remaining || cas_request_budget.lease_safety_margin_ms >= remaining - needed_ms) + return Fence::Admit::NoBudget; + return Fence::Admit::Ok; +} + +bool CasMountRuntime::refAppendFenceOk() const +{ + /// One attempt's worth of room under the live generation: a ref-log attempt is not started when it + /// cannot plausibly finish, safety margin included, before the lease expires. + return admit(fenceGeneration(), cas_request_budget.attempt_timeout_ms) == Fence::Admit::Ok; } void CasMountRuntime::setMountDeadline(uint64_t deadline_boot_ms) @@ -351,7 +363,7 @@ void CasMountRuntime::installKeeper( const std::function & now_ms) { auto replacement = std::make_unique( - backend_ptr, layout, server_root_id, our_uuid, writer_epoch, + mount_requests, farewell_requests, layout, server_root_id, our_uuid, writer_epoch, config.mount_lease_ttl_ms, now_ms, [this] { return minActive(); }, [this](CasEvent e) { emitEvent(std::move(e)); }, @@ -398,7 +410,7 @@ uint64_t CasMountRuntime::startKeeper() } DriverLease lease(*this, active); - const uint64_t anchor = keeper->start(); + const uint64_t anchor = keeper->start([this] { return !renewalCancelled(); }); (void)lease.finish(destination); return anchor; } @@ -407,70 +419,36 @@ MountRenewOperationEnvironment CasMountRuntime::renewalEnvironment(bool worker_c { return MountRenewOperationEnvironment{ .boot_ms = [this] { return bootMsNow(); }, - .stop_cause = [this, worker_call] + .live = [this, worker_call] { - return config.renewal_stop_cause_for_test - ? config.renewal_stop_cause_for_test() - : renewalStopCause(worker_call); - }, - .wait_before_retry = [this, worker_call](uint64_t wait_ms) { return waitForRetry(wait_ms, worker_call); }, - .observe = [](const CasOverwriteProgress & progress) - { - switch (progress.kind) - { - case CasOverwriteProgressKind::PutStarted: - ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalAttempts); - break; - case CasOverwriteProgressKind::RetryStarted: - ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalRetries); - break; - case CasOverwriteProgressKind::ResolvedByGet: - ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalResolved); - break; - case CasOverwriteProgressKind::BecameAmbiguous: - case CasOverwriteProgressKind::ResolveStarted: - break; - } - reportMountRenewProgress(progress); + return config.renewal_live_for_test ? config.renewal_live_for_test() : renewalLive(worker_call); }, + .cancelled = [this] { return renewalCancelled(); }, }; } -CasOverwriteStopCause CasMountRuntime::renewalStopCause(bool worker_call) const +bool CasMountRuntime::renewalLive(bool worker_call) const { std::lock_guard lock(driver_mutex); if (workers_stop_requested) - return CasOverwriteStopCause::Cancelled; - if (worker_call + return false; + return !(worker_call && (renewal_driver_state == RenewalDriverState::ParkRequested || renewal_driver_state == RenewalDriverState::Parked || lifecycle() != PoolLifecycle::Live - || mount_fence.lost.load(std::memory_order_acquire))) - return CasOverwriteStopCause::FenceOrLifecycleLost; - return CasOverwriteStopCause::Continue; + || mount_fence.lost.load(std::memory_order_acquire))); +} + +bool CasMountRuntime::renewalCancelled() const +{ + std::lock_guard lock(driver_mutex); + return workers_stop_requested; } -bool CasMountRuntime::waitForRetry(uint64_t wait_ms, bool worker_call) +void CasMountRuntime::sleepInterruptibly(uint64_t ms) { std::unique_lock lock(driver_mutex); - driver_cv.wait_for(lock, std::chrono::milliseconds(wait_ms), [this, worker_call] - { - return workers_stop_requested - || (worker_call - && (renewal_driver_state == RenewalDriverState::ParkRequested - || renewal_driver_state == RenewalDriverState::Parked - || lifecycle() != PoolLifecycle::Live - || mount_fence.lost.load(std::memory_order_acquire))); - }); - if (workers_stop_requested) - return false; - if (worker_call - && (renewal_driver_state == RenewalDriverState::ParkRequested - || renewal_driver_state == RenewalDriverState::Parked - || lifecycle() != PoolLifecycle::Live - || mount_fence.lost.load(std::memory_order_acquire))) - return false; - return true; + driver_cv.wait_for(lock, std::chrono::milliseconds(ms), [this] { return workers_stop_requested; }); } void CasMountRuntime::consumeRenewResult( @@ -481,14 +459,20 @@ void CasMountRuntime::consumeRenewResult( { /// Driver ownership has already been restored by `DriverLease::finish`; this is the single logical /// consumption boundary and it runs without `driver_mutex` or keeper access. - if (result.outcome == MountRenewOutcome::Committed - && (result.diagnostics.attempts_sent > 1 || result.diagnostics.resolved_by_get)) + /// The physical counters come off the result rather than off a per-attempt callback, so they count + /// the same on every ending: a renewal that gave up still sent what it sent. + if (result.attempts_sent > 0) + { + ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalAttempts, result.attempts_sent); + ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalRetries, result.attempts_sent - 1); + } + if (result.resolved_by_read) + ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalResolved); + + if (result.outcome == MountRenewOutcome::Committed && (result.attempts_sent > 1 || result.resolved_by_read)) ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalRecovered); if (result.outcome == MountRenewOutcome::Terminal - && result.diagnostics.deadline_source == CasOverwriteDeadlineSource::ExternalLeaseSafety - && result.diagnostics.stop_cause == CasOverwriteStopCause::Continue - && (result.diagnostics.unresolved_reason == CasUnresolvedReason::NoAttemptSent - || result.diagnostics.unresolved_reason == CasUnresolvedReason::DeadlineMidWay)) + && result.deadline_source == GaveUp::Source::Lease) ProfileEvents::incrementNoTrace(ProfileEvents::CASMountRenewalDeadlineExceeded); if (result.outcome == MountRenewOutcome::Committed) @@ -541,7 +525,13 @@ uint64_t CasMountRuntime::renewKeeperOnce( /// whole-chain finalizer to deliver after `remount_mutex` is released. configureMountRenewObservability( &server_root_id, &event_sink, active == RenewalDriverState::RemountCall); - const MountRenewResult result = call.keeper->renew(cas_request_budget, renewalEnvironment(worker_call)); + /// The remount redo re-anchors the lease BEFORE `armMountFence`, with the fence still latched lost, + /// so it renews on the keeper's open plane: admitted under the mount fence it could only ever give + /// up, and every remount would fail at this step. `RemountCall` is reached from + /// `renewKeeperForRemountOnce` alone. + const MountRenewResult result = active == RenewalDriverState::RemountCall + ? call.keeper->renewForRemount(renewalEnvironment(worker_call)) + : call.keeper->renew(renewalEnvironment(worker_call)); const RenewalDriverState destination = active == RenewalDriverState::WorkerCall ? RenewalDriverState::WorkerIdle : (active == RenewalDriverState::RemountCall ? RenewalDriverState::Parked : RenewalDriverState::Dormant); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h index 8a6c2cea2f8f..3bb3a929a9ad 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h @@ -1,6 +1,7 @@ #pragma once #include -#include +#include +#include #include #include #include @@ -98,8 +99,9 @@ struct MountConfig std::function terminal_publication_driver_lock_acquired_hook_for_test = {}; /// Deterministic failure injection at the vanished-reason preparation boundary. std::function vanished_reason_prepare_hook_for_test = {}; - /// Test-only override for exact pre/post-send controller gate interleavings. - std::function renewal_stop_cause_for_test = {}; + /// Test-only override of the renewal's liveness predicate, for exact pre/post-send gate + /// interleavings. FALSE ends the renewal exactly as a lost fence does. + std::function renewal_live_for_test = {}; }; /// Local, in-memory write fence. It is deliberately not checked by reading the object store for every @@ -136,6 +138,10 @@ class CasMountRuntime public: CasMountRuntime( BackendPtr backend_ptr_, + /// The two planes the `MountLeaseKeeper` runs on: renewals under the mount fence, the farewell + /// on an open one. Owned by `Pool` and outliving this runtime. + CasRequests & mount_requests_, + CasRequests & farewell_requests_, const Layout & layout_, MountConfig config_, String server_root_id_, @@ -295,6 +301,32 @@ class CasMountRuntime /// it cannot plausibly finish before the fence expires. bool refAppendFenceOk() const; + /// TRUE once the pool has reached — or is being driven toward — a state on which the self-remount + /// worker must stop: a published terminal `Vanished` intent (`vanished_intent` — set early by + /// FORGET, or by a natural `enterVanished`, and already subsuming every settled `Vanished*` state since + /// it is published before the state store) OR `IdentityLost` (rev.8: a fail-loud TERMINAL state — no + /// demoted observer; recovery is restart or FORGET). Consulted by `scheduleRemount` before arming and by + /// the remount loop at every step boundary. (The GC scheduler applies the same three-way test through + /// `Pool`, spec §9 rev.8 item 8.) + bool remountTerminal() const + { + return vanished_intent.load(std::memory_order_acquire) + || lifecycle() == PoolLifecycle::IdentityLost; + } + + /// The inter-attempt sleep the mount plane runs on. A plain sleep would hold a parked or stopping + /// renewal for the whole capped backoff; this one wakes on the same stop signal the workers watch. + /// It shortens a stop, not a fence loss: the fence cannot see a stop request, so a woken operation + /// still reissues unless its own liveness predicate refuses. + void sleepInterruptibly(uint64_t ms); + + /// The mount fence's admission verdict, as `Fence::admit` expects it: may a request admitted under + /// `admitted_generation`, still expected to be running `needed_ms` from now, proceed? + /// `LostOrRearmed` when the fence is latched lost or a fresh lease incarnation replaced the one the + /// caller was admitted under; `NoBudget` when the live lease has no room left for `needed_ms` plus + /// the safety margin, so nothing is begun that could land after this node's fence may be gone. + Fence::Admit admit(uint64_t admitted_generation, uint64_t needed_ms) const; + /// The `writer_epoch` of the live mount incarnation. Bumped by `tryRemountOnce` (self-remount after a /// GC fence-out) — a `PartWriteTxn` minted under an older epoch fails closed on its next step. uint64_t liveWriterEpoch() const { return live_writer_epoch.load(std::memory_order_acquire); } @@ -398,26 +430,19 @@ class CasMountRuntime void renewalLoop(); void remountLoop(); ThreadFromGlobalPool makeWorker(std::function body); - CasOverwriteStopCause renewalStopCause(bool worker_call) const; - bool waitForRetry(uint64_t wait_ms, bool worker_call); + /// The renewal's liveness: facts the mount fence cannot see -- a shutdown request, a parked or + /// park-requested driver, a pool that left `Live`. FALSE ends the renewal. + bool renewalLive(bool worker_call) const; + /// Whether this node has already been asked to stop. Sampled ONCE, before the write, so a refusal + /// caused by the stop cannot be mistaken for one that preceded it. + bool renewalCancelled() const; void tripFenceWithoutOperationalLoss(); std::unique_lock lockTerminalPublication(); - /// TRUE once the pool has reached — or is being driven toward — a state on which the self-remount - /// worker must stop: a published terminal `Vanished` intent (`vanished_intent` — set early by - /// FORGET, or by a natural `enterVanished`, and already subsuming every settled `Vanished*` state since - /// it is published before the state store) OR `IdentityLost` (rev.8: a fail-loud TERMINAL state — no - /// demoted observer; recovery is restart or FORGET). Consulted by `scheduleRemount` before arming and by - /// the remount loop at every step boundary. (The GC scheduler applies the same three-way test through - /// `Pool`, spec §9 rev.8 item 8.) - bool remountTerminal() const - { - return vanished_intent.load(std::memory_order_acquire) - || lifecycle() == PoolLifecycle::IdentityLost; - } - /// ---- injected environment (no `Pool` back-reference); initialized first, in this order ---- BackendPtr backend_ptr; + CasRequests & mount_requests; + CasRequests & farewell_requests; const Layout & layout; MountConfig config; String server_root_id; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp index d5424b4786a6..1abb2f991638 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -22,6 +23,8 @@ namespace ProfileEvents { extern const Event CASBlobBodyPutAvoided; extern const Event CASBlobAdoptTrusted; + extern const Event CASMetaPut; + extern const Event CASMetaCompareSwap; extern const Event CASMetaCreateClean; extern const Event CASMetaAdoptBackfill; extern const Event CASMetaResurrectClean; @@ -72,7 +75,10 @@ uint64_t nowMs() bool isDeterministicBlobPublicationFailure(const std::exception & error) { - if (classifyConditionalWriteResult(error) == CasWriteOutcome::DefiniteFailure) + /// A refused write, EXCEPT the class a fresh credential fixes: the engine refreshes once before it + /// hands the failure back, so this loop's next physical attempt signs with what the refresh + /// installed, and `max_publication_attempts` is what bounds it if the refresh did not help. + if (isDefinitelyRefusedWrite(error) && !isRefreshableCredentialError(error)) return true; if (const auto * db_error = dynamic_cast(&error)) @@ -125,6 +131,7 @@ BlobSource BlobSource::fromString(String bytes) PartWriteTxn::PartWriteTxn(PoolPtr store_, UInt128 build_id_, uint64_t build_seq_, uint64_t epoch_, PartWriteInfo info_) : store(std::move(store_)) + , txn_generation(store->mountRequests().admit().generation()) , build_id(build_id_) , build_seq(build_seq_) , epoch(epoch_) @@ -259,17 +266,39 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) ErrorCodes::LOGICAL_ERROR, "PartWriteTxn::ensureBlobPresent: durable precommit required before materializing {}", blobIdOf(req.ref)); - /// This generation belongs to the operation, not to one observation/publication attempt. In - /// particular, an outer retry after ambiguous I/O must not adopt a re-armed incarnation, and a - /// trip-and-rearm hidden inside the mandatory `HEAD` must still invalidate the original writer. - const uint64_t admitted_generation = store->fenceGeneration(); + /// One operation per upload task, never shared: `fanOutBlobUploads` runs these concurrently and the + /// handle carries per-call state. It RESUMES on the generation the build was admitted under rather + /// than sampling a fresh one, so an outer retry after ambiguous I/O cannot adopt a re-armed + /// incarnation, a trip-and-rearm hidden inside the mandatory `HEAD` still invalidates the original + /// writer, and a re-arm between the precommit and this upload refuses the upload instead of proving + /// a dependency under an incarnation the precommit never saw. The build's own facts -- cancellation + /// and a superseded writer epoch -- stay in `requireAlive`, where each states which one refused. + CasOperation op = store->mountRequests().resume(txn_generation); + /// One policy value, `Retry::standard()`, named here for the requests this function issues + /// directly; the shared helpers it calls construct the same value themselves. + const Retry policy = Retry::standard(); const BlobRef & ref = req.ref; const BlobSource & source = req.source; const String key = store->layout().blobKey(ref); + const String meta_key = store->layout().blobMetaKey(ref); const PoolMeta & pool_meta = store->poolMeta(); const PoolConfig & pool_config = store->poolConfig(); + /// The verdict points. A decision that produces durable metadata or dependency readiness is refused + /// once the operation is no longer admitted, even where the mount has already re-armed and is + /// writable again by the time the request that crossed the boundary returned. + auto requireAdmitted = [&](std::string_view verdict) + { + if (!op.admitted()) + throwCasTransientUnavailable( + fmt::format("PartWriteTxn::ensureBlobPresent of '{}'", key), + fmt::format("the mount no longer admits this build {} -- either a lease loss the disk " + "auto-recovers from, or a FORGET decommission / lost identity that does NOT " + "recover; consult system.cas_mounts for the disk's lifecycle before retrying", + verdict)); + }; + auto buildHeader = [&]() { EnvelopeHeader header; @@ -281,18 +310,24 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) return encodeEnvelopeHeader(header, static_cast(pool_meta.blob_header_len)); }; - auto validateMetaSize = [&](const LoadedMeta & loaded) + auto validateMetaSize = [&](const BlobMeta & observed) { - if (loaded.meta.size != source.size) + if (observed.size != source.size) throw Exception( ErrorCodes::CORRUPTED_DATA, "PartWriteTxn::ensureBlobPresent: metadata for {} declares logical size {}, expected {}", key, - loaded.meta.size, + observed.size, source.size); }; - auto reconcileMetaClean = [&](std::optional loaded, BlobPublicationReason reason) + /// Bring the freshness marker to `Clean`. A publication that followed an ABSENT observation has + /// nothing at the marker key to decide from, so its create IS the whole reconciliation; routing it + /// through a read-decide-write would spend a GET on every insert to learn what the create settles + /// for itself. Only a create that loses -- a racing writer's marker, or the stale `Condemned` one a + /// resurrect always finds -- needs the read, and there the engine's own loop is what bounds the + /// retries at the policy's deadline instead of a fixed count of unpaced attempts. + auto reconcileMetaClean = [&](BlobPublicationReason reason) { if (reason == BlobPublicationReason::Absent) ProfileEvents::increment(ProfileEvents::CASMetaCreateClean); @@ -300,48 +335,69 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) ProfileEvents::increment(ProfileEvents::CASMetaResurrectClean); const BlobMeta clean{.state = MetaState::Clean, .condemn_round = 0, .size = source.size}; - constexpr int max_meta_attempts = 8; - for (int attempt = 0; attempt < max_meta_attempts; ++attempt) + const String what = fmt::format( + "PartWriteTxn::ensureBlobPresent: reconciling the freshness metadata of '{}' to `Clean` " + "after blob publication", key); + + if (reason == BlobPublicationReason::Absent) { - if (loaded) - { - validateMetaSize(*loaded); - if (loaded->meta.state == MetaState::Clean) - return; - if (casMeta(*store, ref, loaded->etag, clean).outcome == CasOverwriteOutcome::Committed) - return; - } - else if (putMetaIfAbsent(*store, ref, clean).outcome == CasOverwriteOutcome::Committed) + ProfileEvents::increment(ProfileEvents::CASMetaPut); + WriteResult created = op.create(meta_key, encodeBlobMeta(clean), policy); + /// Anything but a lost race is this call's answer, and `orThrow` maps it exactly as it maps + /// the read-decide-write's own result. + if (!std::holds_alternative(created)) { + orThrow(std::move(created), what); return; } - loaded = loadMeta(store->backend(), store->layout(), ref); } - throwCasWriteRetryLater(fmt::format( - "PartWriteTxn::ensureBlobPresent: freshness metadata for {} did not reconcile to `Clean` " - "within {} attempts after blob publication", - key, - max_meta_attempts)); + + orThrow( + op.readModifyWrite( + meta_key, + [&](const std::optional & current) -> std::optional + { + if (current) + { + const BlobMeta observed = decodeBlobMeta(current->bytes); + validateMetaSize(observed); + if (observed.state == MetaState::Clean) + return std::nullopt; + /// The same two choke points the standalone marker writes count on, so a + /// reconciliation stays visible as a marker create or a marker compare-swap. + ProfileEvents::increment(ProfileEvents::CASMetaCompareSwap); + } + else + ProfileEvents::increment(ProfileEvents::CASMetaPut); + return encodeBlobMeta(clean); + }, + policy), + what); }; constexpr int max_publication_attempts = 8; for (int attempt = 0; attempt < max_publication_attempts; ++attempt) { requireAlive(); - const HeadResult head = store->backend().head(key); - std::optional loaded; + /// Pace the reissues the way the request engine paces its own, and through the engine's own + /// clock: an ambiguous publication is most often a store under load, and a large body + /// republished eight times back to back is what makes that worse. After `requireAlive`, so a + /// cancelled or superseded build fails closed instead of spending a backoff first. + if (attempt > 0) + op.pause(Retry::backoff(attempt)); + const std::optional present = op.head(key, policy); BlobPublicationReason reason = BlobPublicationReason::Absent; - if (head.exists) + if (present) { - if (head.size < pool_meta.blob_header_len) + if (present->size < pool_meta.blob_header_len) throw Exception( ErrorCodes::CORRUPTED_DATA, "PartWriteTxn::ensureBlobPresent: blob {} size {} is below envelope length {}", key, - head.size, + present->size, pool_meta.blob_header_len); - const uint64_t logical_size = head.size - pool_meta.blob_header_len; + const uint64_t logical_size = present->size - pool_meta.blob_header_len; if (logical_size != source.size) throw Exception( ErrorCodes::CORRUPTED_DATA, @@ -350,37 +406,38 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) logical_size, source.size); - loaded = loadMeta(store->backend(), store->layout(), ref); + const std::optional loaded = loadMeta(op, store->layout(), ref); if (loaded) - validateMetaSize(*loaded); + validateMetaSize(loaded->meta); if (!loaded || loaded->meta.state == MetaState::Clean) { - /// Observation can produce durable metadata and dependency readiness too. Refuse both - /// when the mandatory `HEAD` crossed a fence generation, even if the mount has already - /// re-armed and is writable again by the time it returns. - store->checkFenceOrThrow(admitted_generation); + /// Observation can produce durable metadata and dependency readiness too. + requireAdmitted("after the mandatory `HEAD`"); if (!loaded) { + /// A backfilled marker is a point-read hint for the next observer, so a competing + /// writer that got there first settles the same question: its outcome is not read. ProfileEvents::increment(ProfileEvents::CASMetaAdoptBackfill); putMetaIfAbsent( - *store, + op, + store->layout(), ref, BlobMeta{.state = MetaState::Clean, .condemn_round = 0, .size = logical_size}); } - store->checkFenceOrThrow(admitted_generation); + requireAdmitted("before the body-put-avoided observation is recorded"); ProfileEvents::increment(ProfileEvents::CASBlobBodyPutAvoided); EventEmitter{*store}.emit([&](CasEvent & event) { event.type = CasEventType::BlobReuseAdopt; event.object_kind = CasEventObjectKind::Blob; event.object_hash = blobIdOf(ref); - event.token = head.token.value; + event.token = present->incarnation.render(); event.outcome = "observed"; event.reason = "a present non-condemned blob was observed after mandatory `HEAD`"; event.detail = {{"action", "observed"}, {"size", std::to_string(source.size)}}; }); - store->checkFenceOrThrow(admitted_generation); + requireAdmitted("before the observed dependency proof is returned"); return BlobUploadResult{ ref, BlobDepRecord{ObjectKind::Blob, BlobDependencyProof::Materialized, source.size}, @@ -389,7 +446,6 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) reason = BlobPublicationReason::Condemned; } - store->checkFenceOrThrow(admitted_generation); const bool first_publication = source.beginPublication(); BlobPublicationTransport transport; @@ -414,11 +470,20 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) try { - store->backend().publishBlob(BlobPublishRequest{key, std::move(publication)}); + /// A single physical publication, deliberately not under the shared policy: the engine + /// would reissue this exact envelope, and a re-sent envelope re-publishes the same + /// `incarnation_tag`, so on a content-derived-ETag dialect the republished body carries the + /// incarnation GC condemned and the exact-incarnation delete would remove a live body. + /// Every physical publication therefore mints its own envelope -- each iteration of this + /// loop builds one -- and the engine may never reissue one. The verbatim staged copy is + /// additionally a once-only privilege `beginPublication` spends. + op.publish(BlobPublishRequest{key, std::move(publication)}, Retry::once()); } catch (const std::exception & error) { - if (isDeterministicBlobPublicationFailure(error)) + /// A publication this build is no longer admitted to make is not an ambiguity to retry: + /// every further request of this operation refuses the same way. + if (isDeterministicBlobPublicationFailure(error) || !op.admitted()) throw; if (attempt + 1 == max_publication_attempts) throwCasWriteRetryLater(fmt::format( @@ -440,11 +505,10 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) continue; } + reconcileMetaClean(reason); /// A publication may land just as this mount loses its fence. The bytes are harmless debris, /// but they cannot become dependency proof for the fenced transaction. - store->checkFenceOrThrow(admitted_generation); - reconcileMetaClean(loaded, reason); - store->checkFenceOrThrow(admitted_generation); + requireAdmitted("before the publication is recorded"); EventEmitter{*store}.emit([&](CasEvent & event) { event.type = CasEventType::BlobPut; @@ -461,7 +525,7 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) {"size", std::to_string(source.size)}, {"build_id", u128ToHex(build_id)}}; }); - store->checkFenceOrThrow(admitted_generation); + requireAdmitted("before the published dependency proof is returned"); return BlobUploadResult{ ref, BlobDepRecord{ObjectKind::Blob, BlobDependencyProof::Materialized, source.size}, @@ -562,37 +626,48 @@ ManifestId PartWriteTxn::stageManifest(std::vector entries) const ManifestId id{owning_ns, ref}; const String key = store->layout().manifestKey(id); - /// Body PUT through the Pool's shared request controller: - /// budgeted attempts + resolve-before-reissue, replacing the old bare single-attempt write whose + /// Body PUT on the Pool's staging plane, under the mount fence a `precommitAdd` would fail anyway: + /// budgeted attempts with resolve-before-reissue, replacing the old bare single-attempt write whose /// whole S3-blip tolerance was ONE ~3s adaptive-timeout attempt (a 19s object-store pause killed an /// INSERT through it while every plain read/write path survived — v3 soak evidence). Reissuing this - /// conditional PUT is sound: the body bytes are fixed for the whole operation (`encoded` is built - /// once; `encodePartManifest` is canonical/deterministic), so `resolveByExactGet` can prove whether - /// an ambiguous attempt landed. Still NO preliminary HEAD. A DIFFERENT object at this key is a - /// ManifestId collision — the controller's resolve raises CORRUPTED_DATA (a proven conflict, - /// fail-closed before any owner transition can name this id), subsuming the old - /// PreconditionFailed->LOGICAL_ERROR mapping. - /// - /// fence_ok is the ref lane's own mount predicate (`refAppendFenceOk`: fence not lost + enough - /// lease left for one more attempt): staging runs on this writable Pool under that same mount - /// lease, and a fenced writer must not keep PUTting bodies ahead of a precommitAdd that would fail - /// the same fence anyway. There is no ref-table runtime here, so the lane's extra - /// `superseded_by_remount` term does not apply. - Token manifest_token; - const CasWriteOutcome put_outcome = store->stagingPutIfAbsent(key, encoded, &manifest_token); - if (put_outcome == CasWriteOutcome::DefiniteFailure) - throwCasWriteRetryLater(fmt::format( - "stageManifest: part-manifest PUT at '{}' definitively failed (non-retryable rejection); " - "nothing was named — the caller re-stages with a fresh ManifestId", key)); - /// Unresolved = budget exhausted (or fence lost) without a definite outcome. Unlike the ref-log - /// lane there is nothing to wedge: this id was never named by any owner transition - /// (`next_manifest_ordinal` is already past it, so no re-stage ever reuses the key), and a - /// late-landing body is inert unreferenced debris for the orphan-manifest sweep. NETWORK_ERROR = - /// the same retryable abort class the ref lane's exhausted budget maps to. - if (put_outcome == CasWriteOutcome::Unresolved) - throwCasWriteRetryLater(fmt::format( - "stageManifest: part-manifest PUT at '{}' is UNCERTAIN (retry budget exhausted) — " - "nothing conclusive was named; the caller re-stages with a fresh ManifestId", key)); + /// conditional PUT is sound: the body bytes are fixed for the whole call (`encoded` is built once; + /// `encodePartManifest` is canonical/deterministic), so the engine's resolve read can prove whether + /// an ambiguous attempt landed. Still NO preliminary HEAD. + WriteResult staged = store->stagingPutIfAbsent(key, encoded); + const Incarnation manifest_incarnation = std::visit(detail::Overload{ + [](Committed & committed) -> Incarnation { return std::move(committed.incarnation); }, + [&](Conflict & conflict) -> Incarnation + { + /// Our own bytes under our own `ManifestId` name this same body, whoever wrote them; a + /// DIFFERENT object under an id this build minted is a ManifestId collision, fail-closed + /// before any owner transition can name it. + if (const auto * object = std::get_if(&conflict.seen); object && object->bytes == encoded) + return object->incarnation; + throw Exception(ErrorCodes::CORRUPTED_DATA, + "stageManifest: part-manifest key '{}' already holds {} that is not this manifest's body " + "-- a ManifestId collision", key, detail::renderObservation(conflict.seen)); + }, + [&](Declined &) -> Incarnation + { + throw Exception(ErrorCodes::LOGICAL_ERROR, + "stageManifest: the part-manifest create at '{}' declined; a create has nothing to decline", key); + }, + [&](Refused & refused) -> Incarnation + { + throwCasWriteRetryLater(fmt::format( + "stageManifest: part-manifest PUT at '{}' definitively failed ({}); " + "nothing was named — the caller re-stages with a fresh ManifestId", key, refused.message)); + }, + /// Unlike the ref-log lane there is nothing to wedge: this id was never named by any owner + /// transition (`next_manifest_ordinal` is already past it, so no re-stage ever reuses the key), + /// and a late-landing body is inert unreferenced debris for the orphan-manifest sweep. + [&](GaveUp &) -> Incarnation + { + throwCasWriteRetryLater(fmt::format( + "stageManifest: part-manifest PUT at '{}' is UNCERTAIN (retry budget exhausted) — " + "nothing conclusive was named; the caller re-stages with a fresh ManifestId", key)); + }}, + staged); EventEmitter{*store}.emit([&](CasEvent & e) { @@ -600,7 +675,7 @@ ManifestId PartWriteTxn::stageManifest(std::vector entries) e.namespace_ = owning_ns.string(); e.object_kind = CasEventObjectKind::Manifest; e.object_hash = manifestRefDebugString(id.ref); - e.token = manifest_token.value; + e.token = manifest_incarnation.render(); e.reason = "stageManifest: part-manifest body written"; }); @@ -734,7 +809,8 @@ bool PartWriteTxn::promote(const RootNamespace & target_ns, const String & final /// Read + validate the manifest body ONCE (O(manifest entries), one streaming read). Absent or /// invalid ⇒ fail closed: a committed ref must never name a missing/mismatched manifest. const String manifest_key = store->layout().manifestKey(id); - const auto body_got = store->backend().get(manifest_key); + CasOperation op = store->mountRequests().admit(); + const auto body_got = op.read(manifest_key, Retry::standard()); if (!body_got) throwCasWriteRetryLater(fmt::format( "promote: manifest body absent at {} — failing closed (retry with a fresh ManifestId)", manifest_key)); @@ -1117,8 +1193,8 @@ void PartWriteTxn::abandon() void PartWriteTxn::cleanupStagedManifestDebrisBestEffort() { /// Best-effort writer cleanup of THIS build's pre-precommit/staged `_manifests` debris. The common case - /// is writer cleanup; a missed object is benign — the namespace-scoped orphan sweep reclaims it. Exact-token delete only; never - /// throws. SKIP the manifest that became a live precommit owner: its body is a live precommit input + /// is writer cleanup; a missed object is benign — the namespace-scoped orphan sweep reclaims it. Exact-incarnation + /// delete only; never throws. SKIP the manifest that became a live precommit owner: its body is a live precommit input /// whose deletion is GC's job after the sealed decrement (never writer-delete it). /// /// "Became a live precommit owner" is decided from the ATTEMPT, not from a confirmed append: any @@ -1127,6 +1203,7 @@ void PartWriteTxn::cleanupStagedManifestDebrisBestEffort() /// precommit that turns out to be live and whose body is gone clamps GC's fold barrier forever), /// while keeping it is not: an unreferenced body is ordinary orphan-sweep debris. const bool precommit_attempted = precommit_state != PrecommitState::NotAttempted; + CasOperation op = store->mountRequests().admit(); for (const ManifestId & id : staged_manifests) { if (precommit_attempted && id.ref == precommit_manifest && id.root_namespace == precommit_target_ns) @@ -1134,9 +1211,8 @@ void PartWriteTxn::cleanupStagedManifestDebrisBestEffort() try { const String key = store->layout().manifestKey(id); - const HeadResult hr = store->backend().head(key); - if (hr.exists) - store->backend().deleteExact(key, hr.token); + if (const auto observed = op.head(key, Retry::standard())) + op.remove(key, observed->incarnation, Retry::standard()); } catch (...) // NOLINT(bugprone-empty-catch) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.h index f7a8026bec01..953059a76ebc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.h @@ -31,8 +31,10 @@ struct BlobSource std::shared_ptr> publication_attempted = std::make_shared>(false); - /// Atomically consume the logical source's first-publication privilege. Called after the final - /// fence check and immediately before backend publication I/O. + /// Atomically consume the logical source's first-publication privilege -- the verbatim staged copy + /// is available to exactly one physical publication of this source, whichever one gets here first. + /// A fenced-out writer may spend it before the engine refuses its publication; the only effect is + /// that a later publication streams instead of copying. bool beginPublication() const { return !publication_attempted->exchange(true, std::memory_order_acq_rel); @@ -349,6 +351,11 @@ class PartWriteTxn void cleanupStagedManifestDebrisBestEffort(); PoolPtr store; + /// The mount incarnation this BUILD was admitted under, sampled once when it began. Every upload + /// task resumes on it rather than admitting afresh, so a fence re-armed mid-transaction makes the + /// upload give up instead of minting a dependency proof under an incarnation the precommit never + /// saw -- a re-arm that keeps the writer epoch is invisible to `requireAlive`. + uint64_t txn_generation{}; UInt128 build_id{}; uint64_t build_seq{}; /// per-process monotone sequence uint64_t epoch{}; /// owning Pool's process_epoch diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp index 511ff5cc2516..4f1803c9bcbf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp @@ -1,64 +1,32 @@ #include #include #include - -namespace DB -{ -namespace ErrorCodes -{ - extern const int ABORTED; -} -} +#include namespace DB::Cas { -namespace -{ - constexpr size_t MAX_CAS_ATTEMPTS = 100; -} - void CasPlainObjects::casPutObject(const String & full_key, const String & bytes) { - /// The read determines whether this is a conditional create or replacement. The token is only - /// valid for the incarnation returned by that head, so a precondition failure means another - /// writer won the race and the loop must observe the new incarnation before trying again. - /// - /// SINGLE-APPENDER INVARIANT: `bytes` is frozen by the caller before this loop starts (see the - /// append-base note at `ContentAddressedTransaction::writeFile`'s Append branch); the loop only - /// re-reads the TOKEN on conflict, never the base content. This is correct only while nothing - /// concurrently appends to the same key — a losing retry would overwrite the winner's bytes with a - /// stale, pre-conflict payload (a lost update). Implement a real `casAppendObject` (re-reading the - /// base content, not just the token, inside the loop) before adding any concurrent appender. - /// - /// rev.7 [C2]: the fence generation captured at admission is re-checked immediately before EVERY - /// durable PUT below, not just the first attempt. A mismatch (the mount lease was lost, or re-armed - /// under a fresh incarnation, since admission) aborts with the typed transient error before the backend - /// is ever touched. - const uint64_t admitted_generation = fence_generation_fn(); - - for (size_t attempt = 0; attempt < MAX_CAS_ATTEMPTS; ++attempt) - { - HeadResult head = backend.head(full_key); - check_fence_or_throw_fn(admitted_generation); - if (!head.exists) - { - if (backend.putIfAbsent(full_key, bytes).outcome == PutOutcome::Done) - return; - } - else - { - if (backend.putOverwrite(full_key, bytes, head.token).outcome == PutOutcome::Done) - return; - } - /// `PreconditionFailed` means the observed state changed under us; re-head and retry. - } - throw Exception(ErrorCodes::ABORTED, "object CAS contention on '{}'", full_key); + /// SINGLE-APPENDER INVARIANT: `bytes` is frozen by the caller before this call (see the + /// append-base note at `ContentAddressedTransaction::writeFile`'s Append branch); `decide` below + /// always returns the same frozen bytes regardless of what it observes at the key. This is correct + /// only while nothing concurrently appends to the same key -- a losing retry would overwrite the + /// winner's bytes with a stale, pre-conflict payload (a lost update). Implement a real + /// `casAppendObject` (deciding from the current body, not just presence) before adding any + /// concurrent appender. + CasOperation op = requests.admit(); + WriteResult result = op.readModifyWriteOnPresence( + full_key, + [&](const std::optional &) -> std::optional { return bytes; }, + Retry::standard()); + orThrow(std::move(result), fmt::format("object CAS write on '{}'", full_key)); } std::optional CasPlainObjects::casGetObject(const String & full_key) { - std::optional result = backend.get(full_key); + CasOperation op = requests.admit(); + std::optional result = op.read(full_key, Retry::standard()); if (!result) return std::nullopt; return result->bytes; @@ -66,26 +34,11 @@ std::optional CasPlainObjects::casGetObject(const String & full_key) void CasPlainObjects::casRemoveObject(const String & full_key) { - /// Delete only the incarnation observed by the preceding head. A token mismatch leaves the - /// replacement untouched and is retried against a fresh observation; absence is a successful - /// no-op. - /// - /// rev.7 [C2]: same fence-generation admission as `casPutObject` -- the admitted generation is - /// re-checked immediately before every durable delete. - const uint64_t admitted_generation = fence_generation_fn(); - - for (size_t attempt = 0; attempt < MAX_CAS_ATTEMPTS; ++attempt) - { - const HeadResult head = backend.head(full_key); - if (!head.exists) - return; - check_fence_or_throw_fn(admitted_generation); - const DeleteOutcome outcome = backend.deleteExact(full_key, head.token); - if (outcome.kind == DeleteOutcome::Kind::Deleted || outcome.kind == DeleteOutcome::Kind::NotFound) - return; - /// `TokenMismatch` means a concurrent rewrite; re-head and retry. - } - throw Exception(ErrorCodes::ABORTED, "object CAS contention on '{}' (runaway live-lock brake)", full_key); + /// `removeCurrent` re-heads and retries against a concurrent replacement itself, and reports + /// absence (its own no-op) the same way whether the key was never there or just vanished, so the + /// result carries nothing this caller acts on differently. + CasOperation op = requests.admit(); + op.removeCurrent(full_key, Retry::standard()); } void CasPlainObjects::putNamespaceFile(const NamespaceLifeId & life, const String & name, const String & bytes) @@ -102,20 +55,14 @@ std::vector CasPlainObjects::listNamespaceFiles(const NamespaceLifeId & { const String prefix = layout.namespaceFilesPrefix(life); std::vector names; - String cursor; - while (true) + CasOperation op = requests.admit(); + op.forEachListedKey(prefix, [&](const KeyEntry & entry) { - ListPage page = backend.list(prefix, cursor, /*limit*/ 1000); - for (const ListedKey & listed : page.keys) - { - /// Strip the storage prefix so callers receive the bare flat file name. - if (listed.key.starts_with(prefix)) - names.push_back(listed.key.substr(prefix.size())); - } - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; - } + /// Strip the storage prefix so callers receive the bare flat file name. + if (entry.key.starts_with(prefix)) + names.push_back(entry.key.substr(prefix.size())); + return true; + }, Retry::standard()); /// Backends are not required to return pages in the same order, so make the public result /// deterministic instead of relying on `InMemoryBackend` ordering. std::sort(names.begin(), names.end()); @@ -143,7 +90,8 @@ bool CasPlainObjects::mountpointObjectExists(const String & key) /// the `store` pool subdirectory traversed by `system.remote_data_paths`. The local backend /// treats a directory as not an object, so this returns false instead of attempting a body read /// that would raise a filesystem exception for a directory. - return backend.head(layout.mountpointObjectKey(key)).exists; + CasOperation op = requests.admit(); + return op.head(layout.mountpointObjectKey(key), Retry::standard()).has_value(); } void CasPlainObjects::removeMountpointObject(const String & key) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.h index d1eae78d6e57..76af7681a3eb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.h @@ -1,9 +1,7 @@ #pragma once -#include +#include #include #include -#include -#include #include #include #include @@ -16,35 +14,23 @@ namespace DB::Cas /// by the namespace LIFE, never by its bare name -- and mountpoint objects mirrored by path. The object /// bodies are raw passthrough bytes; this component does not decode them as CAS metadata. /// -/// The component holds references to the shared `Backend` and `Layout` only. It owns no pool mutex +/// The component holds references to the shared `CasRequests` and `Layout` only. It owns no pool mutex /// and has no pool back-reference, allowing `Pool` to retain thin forwarding methods with the same /// external interface. The private helpers implement the shared head-plus-conditional-write and -/// head-plus-exact-delete protocols used by both object families. A conditional outcome means that -/// the observed incarnation changed, so the helper re-reads the head and retries; the fixed bound -/// prevents an unexpected continuous conflict from becoming an unbounded operation and reports -/// `ABORTED` when it is reached. -/// -/// Every durable write/delete on this surface is fence-generation-gated (rev.7 [C2]): `Pool` injects -/// two callbacks that reach its `mount_runtime` (declared AFTER this member, hence constructed -/// after it -- these callbacks capture `Pool` itself and are invoked only at runtime, post- -/// construction, exactly like `ref_ledger`'s callbacks in `CasPool.cpp`, so referencing a -/// not-yet-constructed sibling member through them is safe). +/// head-plus-exact-delete protocols used by both object families, over an operation admitted fresh for +/// each call: every attempt inside it re-checks admission before touching the store, so a mount lease +/// lost mid-call is refused rather than written through. class CasPlainObjects { public: - CasPlainObjects( - Backend & backend_, const Layout & layout_, - std::function fence_generation_fn_, - std::function check_fence_or_throw_fn_) - : backend(backend_), layout(layout_) - , fence_generation_fn(std::move(fence_generation_fn_)) - , check_fence_or_throw_fn(std::move(check_fence_or_throw_fn_)) + CasPlainObjects(CasRequests & requests_, const Layout & layout_) + : requests(requests_), layout(layout_) { } /// Stores the raw bytes under ONE LIFE's `_files/` prefix. Existing files are replaced - /// conditionally using the object incarnation observed by `Backend::head`; a storage failure or an - /// exhausted conflict-retry bound is propagated as an exception. + /// conditionally using the object incarnation observed by the admitted operation; a storage failure + /// or an exhausted conflict-retry bound is propagated as an exception. /// /// `life` is supplied by the caller and never re-derived here, so this surface issues no catalog /// request of its own. A stale writer therefore targets its own old incarnation's key and cannot @@ -53,7 +39,7 @@ class CasPlainObjects /// Reads a namespace file of ONE LIFE without interpreting its body. Returns `nullopt` when the /// object is absent and propagates backend read failures. A stale reader may see stale bytes or - /// `NotFound`, never a newer incarnation's data: its key names the life it was given. + /// absence, never a newer incarnation's data: its key names the life it was given. std::optional getNamespaceFile(const NamespaceLifeId & life, const String & name); /// Enumerates the file names directly below ONE LIFE's `_files/` prefix. Fetches all paginated @@ -61,9 +47,9 @@ class CasPlainObjects /// listing order. std::vector listNamespaceFiles(const NamespaceLifeId & life); - /// Removes the current OBJECT incarnation of one of a life's files, if any (the object token, not - /// the namespace incarnation, which `life` fixes). A concurrent replacement is never removed - /// accidentally: the exact-delete helper re-reads and retries with the new token. + /// Removes the current OBJECT incarnation of one of a life's files, if any (the object incarnation, + /// not the namespace incarnation, which `life` fixes). A concurrent replacement is never removed + /// accidentally: the underlying `removeCurrent` re-heads and retries against the new incarnation. void removeNamespaceFile(const NamespaceLifeId & life, const String & name); /// Stores raw bytes for a loose mountpoint file at the path-derived object key. The key is @@ -80,32 +66,27 @@ class CasPlainObjects /// attempting to read a directory as an object. bool mountpointObjectExists(const String & key); - /// Removes the current path-mirrored mountpoint-object incarnation, if present, using exact-token - /// deletion so a concurrent rewrite remains intact. + /// Removes the current path-mirrored mountpoint-object incarnation, if present, using + /// `removeCurrent`'s re-head-and-retry so a concurrent rewrite remains intact. void removeMountpointObject(const String & key); private: - /// Creates or conditionally replaces one raw object. The method re-heads after a conditional - /// conflict and throws `ABORTED` after the bounded retry loop cannot establish a stable token. - /// Fence-generation-gated (rev.7 [C2]): captures the fence generation at admission for the call's - /// whole retry loop; every iteration re-checks it immediately before its durable PUT. + /// Creates or conditionally replaces one raw object. The write always sends `bytes` regardless of + /// what is currently there, settling a refused precondition with a HEAD; only an ambiguous attempt + /// reads the body, because only the bytes can prove it landed. Retries a lost precondition under + /// the engine's own bound; an exhausted retry or a store refusal is propagated as an exception. void casPutObject(const String & full_key, const String & bytes); - /// Reads one raw object by its complete backend key and returns `nullopt` when it is absent. A read, - /// not a durable-effect operation -- NOT fence-gated (rev.7 [C2] scopes the gate to durable writes). + /// Reads one raw object by its complete backend key and returns `nullopt` when it is absent. std::optional casGetObject(const String & full_key); - /// Removes one raw object by exact token. Absence is a successful no-op; a token mismatch causes - /// a fresh head and retry, while a bounded retry failure throws `ABORTED`. Fence-generation-gated - /// the same way as `casPutObject`. + /// Removes one raw object at its current incarnation. Absence is a successful no-op; a concurrent + /// replacement is retried against the freshly observed incarnation, and an exhausted retry is + /// propagated as an exception. void casRemoveObject(const String & full_key); - Backend & backend; + CasRequests & requests; const Layout & layout; - - /// ---- fence-generation admission (injected by `Pool`; see the class doc comment) ---- - std::function fence_generation_fn; - std::function check_fence_or_throw_fn; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index f5563abf4306..d719a9392c93 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -128,10 +128,13 @@ struct LifecycleGate /// `min_reader_generation` are legally mutable and are deliberately not compared (the format gate is the /// decode itself succeeding). LifecycleGate probePoolLifecycleGate( - Backend & backend, const Layout & layout, const String & srid, + CasOperation & op, const Layout & layout, const String & srid, UInt128 expected_pool_id, uint64_t expected_blob_header_len) { - const SentinelProbeResult meta_probe = probeSentinel(backend, layout.poolMetaKey()); + /// `once` on both probes: an inconclusive answer IS this gate's verdict (`StayTransient`), and the + /// recovery loop that called it is what retries. Reissuing here would spend the loop's whole + /// interval inside a single probe and make a terminal transition wait for it. + const SentinelProbeResult meta_probe = probeSentinel(op, layout.poolMetaKey(), Retry::once()); switch (meta_probe.outcome) { case ProbeOutcome::Present: @@ -164,7 +167,7 @@ LifecycleGate probePoolLifecycleGate( /// authoritatively absent ⇒ `IdentityLost` (a fail-loud terminal state), regardless of whatever /// else remains under the prefix. Erasure is never PROVEN by the system — only asserted by the /// operator's `FORGET` — so there is no prefix-emptiness leg here. - const SentinelProbeResult owner_probe = probeSentinel(backend, layout.ownerKey(srid)); + const SentinelProbeResult owner_probe = probeSentinel(op, layout.ownerKey(srid), Retry::once()); if (owner_probe.outcome != ProbeOutcome::KeyAbsent) return {LifecycleGateVerdict::StayTransient, "_pool_meta absent but the owner sentinel was not conclusively absent"}; @@ -182,6 +185,18 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) : pool_backend(std::move(backend_)) , config(std::move(config_)) , meta(std::move(meta_)) + /// The mount plane's fence reaches `mount_runtime`, declared far below: the closures capture + /// `this` and run only after construction, exactly like `ref_ledger`'s callbacks. All three planes + /// take the fence's own clock, so a policy bound to a mount-lease deadline and the fence that + /// enforces it are read from the same source. + , mount_requests(pool_backend, Fence{ + [this] { return mount_runtime.fenceGeneration(); }, + [this](uint64_t g, uint64_t needed) { return mount_runtime.admit(g, needed); }, + [this](uint64_t g) { mount_runtime.checkFenceOrThrow(g); }}, + config.boot_ms_fn, + mountPlaneSleepFn()) + , farewell_requests(pool_backend, Fence::open(), config.boot_ms_fn) + , gc_requests(pool_backend, Fence::open(), config.boot_ms_fn) /// Seed the monotone admitted-algo cache from the pool state `createOrValidate` already /// established (fresh create, steady-state member, or a just-completed admission union) -- /// register-before-first-write means this Pool's own `writeAlgo()` is ALWAYS a @@ -190,36 +205,30 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) /// `Layout` no longer captures a pool algo -- every blob key is built from a /// `BlobRef` (algo + digest) directly, so the constructor takes only the pool prefix. , pool_layout(config.pool_prefix) - /// Plain-object surface component: binds to this Pool's own backend + layout (declared after - /// both, so this reference-holding member is constructed last and destroyed first) plus two - /// fence-generation callbacks reaching `mount_runtime` (declared AFTER `plain_objects`, hence - /// constructed after it -- these callbacks capture `this` and are invoked only at runtime, - /// post-construction, exactly like `ref_ledger`'s callbacks below, so referencing a - /// not-yet-constructed sibling member through them is safe). - , plain_objects( - *pool_backend, pool_layout, - [this] { return mount_runtime.fenceGeneration(); }, - [this] (uint64_t gen) { mount_runtime.checkFenceOrThrow(gen); }) - /// Manifest reader component: backend/layout/meta by reference + the event-sink reference. The - /// sink is installed by the factory before writable mounting starts. Owns the decode cache, - /// built from the same config bytes the Pool ctor used before. - , manifest_reader(*pool_backend, pool_layout, meta, event_sink_, config.manifest_decode_cache_bytes) - /// Ref-log / ref-table subsystem. Injected with backend/layout + the - /// RefLedgerConfig slice + the event-sink reference + the pool `cas_request_budget` + the RAW mount - /// `boot_ms_fn` (for its retry controller), plus callbacks into the mount/watermark state that lives + /// Plain-object surface component, on the MOUNT plane: the namespace-file and mountpoint writes it + /// offers are durable mutations whose right to land is the mount lease. + , plain_objects(mount_requests, pool_layout) + /// Manifest reader component, on the MOUNT plane: a content read on a mount whose lease is gone is + /// refused, which is what the reader's own contract promises and what the metadata storage's op gate + /// already does for every content read. A reader for work that outlives the lease (GC) is a separate + /// reader over that plane, never this one shared across two. The event sink is installed by the + /// factory before writable mounting starts. + , manifest_reader(mount_requests, pool_layout, meta, event_sink_, config.manifest_decode_cache_bytes) + /// Ref-log / ref-table subsystem, on the MOUNT plane: a ref-lane write and a mount-lease renewal + /// are then measured against the same fence and the same clock. Injected with the + /// RefLedgerConfig slice + the event-sink reference + the pool `cas_request_budget`, plus callbacks + /// into the mount/watermark state that lives /// on `mount_runtime` (reached through Pool delegates). The callbacks capture `this`; they are /// invoked only at runtime (post-construction), so referencing `mount_runtime` (declared AFTER /// `ref_ledger`, hence constructed after it) is safe -- exactly as the pre-3.5 layout referenced the /// mount raw-members that also followed `ref_ledger`. Declared/constructed BEFORE `mount_runtime`, /// preserving the original member order verbatim (see the header note). , ref_ledger( - pool_backend, pool_layout, config.refLedgerConfig(), event_sink_, config.cas_request_budget, + mount_requests, pool_layout, config.refLedgerConfig(), event_sink_, config.cas_request_budget, config.server_root_id, - config.boot_ms_fn, [this] { return liveWriterEpoch(); }, [this] { return refAppendFenceOk(); }, [this] { return mount_runtime.fenceGeneration(); }, - [this] (uint64_t gen) { mount_runtime.checkFenceOrThrow(gen); }, [this] { return bootMsNow(); }, [this] { return mayMutate(); }, [this] (const String & key, const String & reason, const std::optional & offending_ns) @@ -229,13 +238,14 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) [this] (const RootNamespace & ns) { cancelInflightBuildsForNamespace(ns); }, config.recovery_pre_first_request_hook_for_test) /// Mount / write-fence / build-watermark / self-remount runtime. Injected with - /// backend/layout + the `MountConfig` slice + `server_root_id` + the event-sink reference + the pool + /// backend/layout + the mount and farewell planes + the `MountConfig` slice + `server_root_id` + the event-sink reference + the pool /// `cas_request_budget` + the `remount_attempt` callback (== `Pool::tryRemountOnce`, whose claim/ /// recovery ORCHESTRATION stays on Pool). The callback captures `this`; it is invoked only at runtime /// (post-construction). Declared/constructed AFTER `ref_ledger`, preserving the original member order /// verbatim (mount destroyed first, ledger last; both orders proven safe -- see the header note). , mount_runtime( - pool_backend, pool_layout, config.mountConfig(), config.server_root_id, event_sink_, + pool_backend, mount_requests, farewell_requests, + pool_layout, config.mountConfig(), config.server_root_id, event_sink_, config.cas_request_budget, [this] { return tryRemountOnce(); }) { @@ -253,7 +263,8 @@ std::vector Pool::refreshAdmittedAlgos() /// A direct GET+decode of `_pool_meta`, not a re-run of `createOrValidate`'s admission logic -- /// this Pool's OWN algo is already admitted, so all this /// needs is the CURRENT authoritative `algos_used`, unioned into the monotone cache. - const auto existing = pool_backend->get(pool_layout.poolMetaKey()); + CasOperation op = gc_requests.admit(); + const auto existing = op.read(pool_layout.poolMetaKey(), Retry::standard()); std::lock_guard lock(admitted_algos_mutex); if (existing) @@ -378,6 +389,9 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// FAIL-CLOSED: the capability probe throws NOT_IMPLEMENTED on any failed check, and /// PoolMeta::createOrValidate is pool-authoritative — the config constants apply only at creation. Layout layout(config.pool_prefix); + /// The whole bootstrap runs on an OPEN fence: there is no mount lease yet, and the claim below is + /// what establishes one. + CasRequests bootstrap_requests(backend, Fence::open(), config.boot_ms_fn); bool initialize_empty_catalog = false; /// The probe writes and deletes throwaway keys to verify conditional-op enforcement. A read-only /// open must never mutate the pool it inspects; fsck only reads, so skip it. (Pool meta below is @@ -392,7 +406,8 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// crash-mid-battery still bootstraps cleanly. This closes the "restart poisons a /// partially-erased pool" hole: a missing `_pool_meta` over residual data now fails startup loud /// with zero writes, instead of minting a fresh identity on top of the old objects. - switch (probePoolBootstrapResidual(*backend, layout)) + CasOperation residual_op = bootstrap_requests.admit(); + switch (probePoolBootstrapResidual(residual_op, layout)) { case BootstrapResidual::PoolMetaPresent: break; /// authoritative existing pool; its catalog is mandatory below. @@ -430,7 +445,8 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// Only on this arm: `EmptyOrProbeOnly` proves there is no slot object to read (a mount /// lease is itself residual), and `PoolMetaPresent` is not a recreation at all -- neither /// pays for the scan. - const std::vector held = probeNonTerminalMountSlots(*backend, layout); + CasOperation slots_op = bootstrap_requests.admit(); + const std::vector held = probeNonTerminalMountSlots(slots_op, layout); if (!held.empty()) { String detail; @@ -460,6 +476,15 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) if (!config.skip_access_check) { + /// The two STORE-LEVEL gates run before the battery writes anything, so a backend this build + /// must refuse is refused without leaving `_probe/` debris behind. They ask the backend + /// directly because an admitted operation deliberately has no route back to it; the predicates + /// come from the same engine the probe operation is admitted from, so they can never be asked + /// of a different backend than the one probed. + Backend & probe_backend = bootstrap_requests.backendForCapabilityPredicates(); + probe_backend.checkPoolPreconditions(); + probe_backend.checkConditionalWriteSingleAttemptSupport(); + /// Give each mount a PER-MOUNT UNIQUE probe key prefix so two servers mounting the SAME /// shared pool concurrently never collide on the (formerly fixed) `/_probe/token` / /// `/_probe/cas` keys. Without this, the loser of the `putIfAbsent` race aborts startup @@ -468,13 +493,14 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// independently. A crashed mount leaves harmless `_probe//...` debris under the `_probe/` /// namespace only (never the content planes) — acceptable. const UInt128 probe_uid = (static_cast(thread_local_rng()) << 64) | thread_local_rng(); - runCapabilityProbe(*backend, config.pool_prefix + "/_probe/" + u128ToHex(probe_uid)); + CasOperation probe_op = bootstrap_requests.admit(); + runCapabilityProbe(probe_op, config.pool_prefix + "/_probe/" + u128ToHex(probe_uid)); } else { - /// skip_access_check: skip the access-check-class probe I/O (store preconditions + the - /// `_probe/` round trip, both folded into runCapabilityProbe above) but NOT the two - /// fail-closed gates below — see `PoolConfig::skip_access_check`. + /// skip_access_check: skip the access-check-class probe I/O (the store-precondition gate and + /// the `_probe/` round trip above) but NOT the two fail-closed gates below — see + /// `PoolConfig::skip_access_check`. /// /// First, whether this backend may skip the battery AT ALL. A generation-dialect (GCS) /// backend may not: the battery is the only thing that proves a token-exact DELETE @@ -494,14 +520,18 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// the narrowly-defined retry path when this opener (or a concurrent opener) completed this step /// but did not reach the pool-meta create. if (initialize_empty_catalog) - CasRefCatalog::initializeEmptyForNewPool(*backend, layout); + { + CasOperation catalog_op = bootstrap_requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(catalog_op, layout); + } /// `allow_mint` = writable open only: a writable `Pool::open` reaches here having just passed the /// zero-write residual proof above, so minting a missing `_pool_meta` is safe. A read-only/observe /// open never ran that proof (and there is no truly-read-only backend — `openPoolView` opens the same /// writable object storage and only sets `read_only`), so it must NEVER mint: an absent meta fails /// closed instead (spec §2 [C4][D2]). + CasOperation meta_op = bootstrap_requests.admit(); PoolMeta meta = PoolMeta::createOrValidate( - *backend, layout, config.blob_header_len, config.gc_shards, config.blob_hash_algo, config.blob_hash_allow_new, + meta_op, layout, config.blob_header_len, config.gc_shards, config.blob_hash_algo, config.blob_hash_allow_new, /*allow_mint=*/!config.read_only); config.gc_shards = meta.gc_shards; const BlobHashAlgo write_algo = config.blob_hash_algo; /// `config` is moved-from just below @@ -551,7 +581,8 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol const ObserveRefCatalog observe_catalog = [s = store.get()]() { - CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(*s->pool_backend, s->pool_layout); + CasOperation op = s->gc_requests.admit(); + CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, s->pool_layout); snapshot.life_index.throwIfAmbiguous("CAS server-root mount safety"); return snapshot.catalog; }; @@ -562,7 +593,11 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// 2. Owner anchor — IDENTITY (clock-free). A foreign uuid fails closed; an absent owner over a /// non-empty subtree is CORRUPTED_DATA; a fresh empty root is claimed. - claimOwnerOrThrow(*store->pool_backend, store->pool_layout, srid, our_uuid, observe_catalog); + /// Every step of this protocol runs on the OPEN plane. These are bootstrap-control writes: they + /// establish the very right to write, and the mount fence they would otherwise be gated on is + /// either unarmed (a first open) or latched lost (a self-remount, which could then never reclaim). + CasOperation owner_op = store->gc_requests.admit(); + claimOwnerOrThrow(owner_op, store->pool_layout, srid, our_uuid, observe_catalog); /// Wall-clock `now_ms`, hoisted above the writer_epoch allocation below: the absent-epoch /// branch's `DecommissionRecovery` policy needs it to judge a surviving mount's liveness before @@ -587,8 +622,9 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol const EpochMintPolicy epoch_policy = (policy == MountClaimPolicy::NoWait) ? EpochMintPolicy::DecommissionRecovery : EpochMintPolicy::NormalMount; + CasOperation epoch_op = store->gc_requests.admit(); uint64_t writer_epoch = allocateWriterEpoch( - *store->pool_backend, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); + epoch_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); store->mount_runtime.setProcessEpoch(writer_epoch, std::memory_order_relaxed); /// 4. Mount lease — LIVENESS. Decide over the current mount object using the wall-clock `now_ms` @@ -658,8 +694,9 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol MountClaimResult claim; if (policy == MountClaimPolicy::WaitForExpiry) { + CasOperation claim_op = store->gc_requests.admit(); claim = claimMountAwaitingExpiry( - *store->pool_backend, store->pool_layout, srid, our_uuid, writer_epoch, + claim_op, store->pool_layout, srid, our_uuid, writer_epoch, [&now_ms]() { return now_ms(); }, [raw] { return raw->bootMsNow(); }, ttl_ms, poll_interval_ms, sleep_ms, on_wait_start, emit_mount_event); } @@ -668,8 +705,9 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// NoWait (decommission gate): a single unobserved attempt -- no bounded wait-and-retry /// for a stale-looking lease to lapse. Anything but Claimed/FencedSelf below is refused /// immediately. - claim = claimMount(*store->pool_backend, store->pool_layout, srid, our_uuid, writer_epoch, - now_ms(), ttl_ms, /*proven_dead_token=*/{}, emit_mount_event); + CasOperation claim_op = store->gc_requests.admit(); + claim = claimMount(claim_op, store->pool_layout, srid, our_uuid, writer_epoch, + now_ms(), ttl_ms, /*proven_dead_incarnation=*/{}, emit_mount_event); } if (claim.kind == MountClaimResult::FencedSelf) { @@ -679,8 +717,9 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol "({} recoveries exhausted) — a fresh writer_epoch kept being fenced before we " "could adopt it. This should not persist; investigate GC fence-out timing.", srid, max_fence_recoveries); + CasOperation reallocate_op = store->gc_requests.admit(); writer_epoch = allocateWriterEpoch( - *store->pool_backend, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); + reallocate_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); store->mount_runtime.setProcessEpoch(writer_epoch, std::memory_order_relaxed); continue; } @@ -717,8 +756,9 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol if (fence_recovery >= max_fence_recoveries) throw; store->mount_runtime.keeperReset(); + CasOperation refenced_epoch_op = store->gc_requests.admit(); writer_epoch = allocateWriterEpoch( - *store->pool_backend, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); + refenced_epoch_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); store->mount_runtime.setProcessEpoch(writer_epoch, std::memory_order_relaxed); continue; } @@ -841,10 +881,14 @@ PoolPtr Pool::openForDecommission(BackendPtr backend, PoolConfig config, const S /// fenced/terminated/clean-farewell lease reclaims; a live lease refuses immediately (no bounded /// observation wait -- see `mountWritable`). Owner anchor absent + mount absent = nothing to /// decommission. - std::optional victim_uuid = readOwnerUuid(*backend, layout, victim_srid); + /// The open plane: this factory impersonates the victim to take its mount, so there is no lease of + /// ours to be gated on until the claim below establishes one. + CasRequests bootstrap_requests(backend, Fence::open(), config.boot_ms_fn); + CasOperation owner_op = bootstrap_requests.admit(); + std::optional victim_uuid = readOwnerUuid(owner_op, layout, victim_srid); if (!victim_uuid) { - if (const auto mount = backend->get(layout.mountKey(victim_srid))) + if (const auto mount = owner_op.read(layout.mountKey(victim_srid), Retry::standard())) victim_uuid = decodeMountLease(mount->bytes).server_uuid; /// partial hand-cleanup: adopt from the lease else throw Exception(ErrorCodes::BAD_ARGUMENTS, @@ -859,8 +903,9 @@ PoolPtr Pool::openForDecommission(BackendPtr backend, PoolConfig config, const S /// `_pool_meta` must already be present. It never bootstraps: `allow_mint=false` so an absent meta /// (a partially-erased pool whose owner anchor survives) fails closed with INVALID_STATE rather than /// minting a fresh identity here (spec §2 [C4][D2]). + CasOperation meta_op = bootstrap_requests.admit(); PoolMeta meta = PoolMeta::createOrValidate( - *backend, layout, config.blob_header_len, config.gc_shards, config.blob_hash_algo, config.blob_hash_allow_new, + meta_op, layout, config.blob_header_len, config.gc_shards, config.blob_hash_algo, config.blob_hash_allow_new, /*allow_mint=*/false); config.gc_shards = meta.gc_shards; const BlobHashAlgo write_algo = config.blob_hash_algo; /// `config` is moved-from just below @@ -1225,8 +1270,37 @@ bool Pool::tryRemountOnce() return false; { step = "pool_identity_probe"; - const LifecycleGate gate = probePoolLifecycleGate( - *pool_backend, pool_layout, config.server_root_id, meta.pool_id, meta.blob_header_len); + /// The open plane: this runs with the mount fence latched lost -- that is what a remount is + /// recovering from -- so an operation admitted under the fence could never issue the probe. + /// + /// Admitted with NO liveness predicate. Refusing the probe once the pool is already terminal + /// would be circular -- this probe is what establishes terminality -- and it would make the + /// verdicts below unreachable in exactly the states they exist for, the mid-FORGET `Replaced` + /// bail among them. A predicate is also the wrong instrument for ending it: the engine samples + /// one before the first request and reports a refusal as a THROWN fence loss, which is not how + /// a remount step reports anything. Both sentinel reads are `once`, so the probe is at most two + /// physical requests and cannot outlast a shutdown. + CasOperation probe_op = gc_requests.admit(); + LifecycleGate gate{LifecycleGateVerdict::StayTransient, {}}; + try + { + gate = probePoolLifecycleGate( + probe_op, pool_layout, config.server_root_id, meta.pool_id, meta.blob_header_len); + } + catch (...) + { + /// The same contract as the startup-protocol steps below: a remount attempt reports failure + /// by returning false, never by throwing at whoever called it. A probe that could not reach + /// the store proved nothing, so the pool stays where it was and the loop retries. + try + { + error = getCurrentExceptionMessage(/*with_stacktrace*/ false); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + } + return false; + } switch (gate.verdict) { case LifecycleGateVerdict::Recover: @@ -1283,7 +1357,8 @@ bool Pool::tryRemountOnce() { const ObserveRefCatalog observe_catalog = [this]() { - CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(*pool_backend, pool_layout); + CasOperation op = gc_requests.admit(); + CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, pool_layout); snapshot.life_index.throwIfAmbiguous("CAS server-root remount safety"); return snapshot.catalog; }; @@ -1292,10 +1367,15 @@ bool Pool::tryRemountOnce() step = "ref_catalog_observe"; (void)observe_catalog(); step = "owner_claim"; - claimOwnerOrThrow(*pool_backend, pool_layout, srid, our_uuid, observe_catalog); + /// The open plane throughout: the mount fence is latched lost here (starting the keeper does + /// not clear it), so an operation admitted under the fence could never make the claim that + /// re-establishes it. + CasOperation owner_op = gc_requests.admit(); + claimOwnerOrThrow(owner_op, pool_layout, srid, our_uuid, observe_catalog); step = "writer_epoch_allocate"; + CasOperation epoch_op = gc_requests.admit(); const uint64_t writer_epoch = allocateWriterEpoch( - *pool_backend, pool_layout, srid, EpochMintPolicy::NormalMount, 0, observe_catalog); + epoch_op, pool_layout, srid, EpochMintPolicy::NormalMount, 0, observe_catalog); result_writer_epoch = writer_epoch; /// Mount-slot writer audit: `this` is already fully open (setEventSink ran long ago), so @@ -1304,8 +1384,9 @@ bool Pool::tryRemountOnce() const auto sleep_ms = [](uint64_t ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); }; step = "mount_claim"; + CasOperation claim_op = gc_requests.admit(); const MountClaimResult claim = claimMountAwaitingExpiry( - *pool_backend, pool_layout, srid, our_uuid, writer_epoch, + claim_op, pool_layout, srid, our_uuid, writer_epoch, now_ms, [this] { return bootMsNow(); }, ttl_ms, poll_interval_ms, sleep_ms, [&srid](const MountLease & held, uint64_t threshold_ms) { @@ -1637,7 +1718,10 @@ void Pool::reportImpossibleInterference(const String & key, const String & reaso return; try { - const auto got = pool_backend->get(key); + /// The open plane: this diagnostic runs after the fence was deliberately tripped a few + /// lines above, and a stop request ends it through the operation's own liveness. + CasOperation op = gc_requests.admit([token] { return !token.stopping(); }); + const auto got = op.read(key, Retry::standard()); if (!got) { LOG_ERROR(getLogger("CasPool"), @@ -1684,7 +1768,8 @@ uint64_t Pool::currentGcRound() const /// Read `gc/state` once (no CAS loop — a point-in-time read is sufficient; a concurrent /// GC advance only makes the returned round larger, which is strictly more conservative for the /// `precommitAdd` self-floor). Returns 0 when absent (pool never GC'd — no round to floor to). - const auto state_bytes = pool_backend->get(pool_layout.gcStateKey()); + CasOperation op = gc_requests.admit(); + const auto state_bytes = op.read(pool_layout.gcStateKey(), Retry::standard()); if (!state_bytes) return 0; return decodeGcState(state_bytes->bytes).round; @@ -1721,7 +1806,10 @@ NamespaceListing Pool::listNamespaces(const String & prefix) /// opaque life id and therefore cannot mint a namespace during discovery. std::unordered_set found; std::vector skipped; - const CasRefCatalog::Snapshot cut = CasRefCatalog::read(*pool_backend, pool_layout); + /// The mount plane, like every other content read: enumerating a namespace on a mount whose lease + /// is gone must be refused, not answered. + CasOperation catalog_op = mount_requests.admit(); + const CasRefCatalog::Snapshot cut = CasRefCatalog::read(catalog_op, pool_layout); for (const CatalogEntry & entry : cut.catalog.entries) { try @@ -1747,7 +1835,9 @@ std::vector Pool::listMirroredChildren(const String & prefix) /// Namespace children come from the catalog. `roots/` is still listed for loose mountpoint files, /// whose logical paths retain path identity. std::unordered_set children; - const CasRefCatalog::Snapshot cut = CasRefCatalog::read(*pool_backend, pool_layout); + /// The mount plane: this is a content enumeration, the same class as `listNamespaces` above. + CasOperation catalog_op = mount_requests.admit(); + const CasRefCatalog::Snapshot cut = CasRefCatalog::read(catalog_op, pool_layout); for (const CatalogEntry & entry : cut.catalog.entries) { if (!entry.ns.string().starts_with(prefix)) @@ -1760,27 +1850,20 @@ std::vector Pool::listMirroredChildren(const String & prefix) } const String roots_full = pool_layout.rootsPrefix() + prefix; + CasOperation roots_op = mount_requests.admit(); + roots_op.forEachListedKey(roots_full, [&](const KeyEntry & listed) { - String cursor; - while (true) + const String & key = listed.key; + if (key.starts_with(roots_full)) { - ListPage page = pool_backend->list(roots_full, cursor, /*limit*/ 1000); - for (const ListedKey & listed : page.keys) - { - const String & key = listed.key; - if (!key.starts_with(roots_full)) - continue; - const std::string_view rest(key.data() + roots_full.size(), key.size() - roots_full.size()); - const size_t slash = rest.find('/'); - const std::string_view seg = slash == std::string_view::npos ? rest : rest.substr(0, slash); - if (!seg.empty()) - children.emplace(seg); - } - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; + const std::string_view rest(key.data() + roots_full.size(), key.size() - roots_full.size()); + const size_t slash = rest.find('/'); + const std::string_view seg = slash == std::string_view::npos ? rest : rest.substr(0, slash); + if (!seg.empty()) + children.emplace(seg); } - } + return true; + }, Retry::standard()); return {children.begin(), children.end()}; } @@ -1791,7 +1874,22 @@ std::vector Pool::listMirroredChildren(const String & prefix) void Pool::setCasRetrySleepForTest(std::function sleep_fn) { - ref_ledger.setCasRetrySleepForTest(std::move(sleep_fn)); + /// All three planes, not just the ledger's: a test that replaces the retry sleep must not be left + /// with a real one on the plane the site under test happens to use. + farewell_requests.setSleepFnForTest(sleep_fn); + gc_requests.setSleepFnForTest(sleep_fn); + ref_ledger.setCasRetrySleepForTest(sleep_fn); + /// The ledger reaches the mount plane too, and `CasRequests` falls back to the engine's plain + /// sleep for an empty argument -- which is not this plane's default. Re-install ours last, so + /// clearing the seam cannot leave a parked or stopping renewal held for a whole capped backoff. + mount_requests.setSleepFnForTest(sleep_fn ? std::move(sleep_fn) : mountPlaneSleepFn()); +} + +void Pool::setCasRequestNowFnForTest(std::function now_fn) +{ + mount_requests.setNowFnForTest(now_fn); + farewell_requests.setNowFnForTest(now_fn); + gc_requests.setNowFnForTest(std::move(now_fn)); } void Pool::setRefRecoveryRetrySleepForTest( @@ -1898,17 +1996,17 @@ size_t Pool::wedgedRefLaneCount() return ref_ledger.wedgedRefLaneCount(); } -CasWriteOutcome Pool::stagingPutIfAbsent(std::string_view key, std::string_view bytes, Token * out_token) +WriteResult Pool::stagingPutIfAbsent(const String & key, const String & bytes) { - return ref_ledger.stagingPutIfAbsent(key, bytes, out_token); + return ref_ledger.stagingPutIfAbsent(key, bytes); } -CasOverwriteResult Pool::stagingConditionalOverwrite(std::string_view key, std::string_view bytes, const Token & expected) +WriteResult Pool::stagingConditionalOverwrite(const String & key, const String & bytes, const Incarnation & expected) { return ref_ledger.stagingConditionalOverwrite(key, bytes, expected); } -CasOverwriteResult Pool::stagingPutIfAbsentMutable(std::string_view key, std::string_view bytes) +WriteResult Pool::stagingPutIfAbsentMutable(const String & key, const String & bytes) { return ref_ledger.stagingPutIfAbsentMutable(key, bytes); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index b75f34d4092c..76ca6ed063f8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -9,7 +9,7 @@ #include #include #include -#include +#include #include #include #include @@ -709,22 +709,36 @@ class Pool : public std::enable_shared_from_this const PoolMeta & poolMeta() const { return meta; } const Layout & layout() const { return pool_layout; } Backend & backend() { return *pool_backend; } + + /// ---- the three request planes ---- + /// The mount plane: the durable writes whose right to land IS this node's mount lease. An + /// operation admitted here is refused the moment the fence trips, is re-armed under a fresh lease + /// incarnation, or runs out of room before the lease expires. + CasRequests & mountRequests() { return mount_requests; } + /// The farewell plane, on an open fence. Releasing the lease is the last thing a departing mount + /// does, and refusing it because the mount fence has already run down would leave the slot looking + /// live until GC fences it out. + CasRequests & farewellRequests() { return farewell_requests; } + /// The open-fence plane: GC, the offline tools, this pool's own reads, and the bootstrap-control + /// claims. None of them hold a mount lease -- the claims are what ESTABLISHES one, so gating them + /// on the fence would make a self-remount, which runs with the fence latched lost, unable ever to + /// reclaim. + CasRequests & gcRequests() { return gc_requests; } /// The owning `BackendPtr` itself (not just a reference into it): the decommission slot-retirement /// decommission step (`CasDecommission.cpp`) must keep the backend alive across `admin.reset()` -- the graceful /// close that stamps the mount's farewell -- to physically delete the control objects afterward. A /// bare `Backend &` from `backend()` would dangle the instant the owning `Pool` is destroyed. BackendPtr poolBackendPtr() const { return pool_backend; } - /// Staging PUT surface for `PartWriteTxn`: the methods wrap the ref-ledger's retry controller - /// AND the ref-lane fence predicate, so `PartWriteTxn` reaches neither directly (the `friend` is gone). - /// Behavior-identical to the previously-inlined controller+fence at CasPartWriteTxn.cpp stageManifest / - /// mutable marker writes; thin delegates to `ref_ledger`. - CasWriteOutcome stagingPutIfAbsent(std::string_view key, std::string_view bytes, Token * out_token = nullptr); - /// Same retry/fence policy as `stagingPutIfAbsent`, for a mutable If-Match overwrite. - CasOverwriteResult stagingConditionalOverwrite(std::string_view key, std::string_view bytes, const Token & expected); + /// Staging write surface for `PartWriteTxn`: thin delegates onto the ref ledger, so a staging write + /// is admitted on the same plane and under the same policy as a ref-lane write and `PartWriteTxn` + /// reaches neither directly. + WriteResult stagingPutIfAbsent(const String & key, const String & bytes); + /// Same retry/fence policy as `stagingPutIfAbsent`, for a mutable exact-incarnation overwrite. + WriteResult stagingConditionalOverwrite(const String & key, const String & bytes, const Incarnation & expected); /// Same retry/fence policy as `stagingPutIfAbsent`, for a mutable marker where an existing - /// DIFFERENT value at the key is a normal Conflict outcome, not corruption. - CasOverwriteResult stagingPutIfAbsentMutable(std::string_view key, std::string_view bytes); + /// DIFFERENT value at the key is a normal `Conflict`, not corruption. + WriteResult stagingPutIfAbsentMutable(const String & key, const String & bytes); /// CAS mixed-algo pools: /// the NODE-LOCAL algo this Pool mints NEW content with (`PoolConfig::blob_hash_algo` -- never @@ -865,10 +879,9 @@ class Pool : public std::enable_shared_from_this /// `cancel_inflight_builds` callback. void cancelInflightBuildsForNamespace(const RootNamespace & ns); - /// Delegate to `mount_runtime`: the write fence moved there. pre-attempt fence check: extends - /// `mayMutate` with the REMAINING budget check -- an attempt is not even started unless there is - /// enough of the mount lease left for one more attempt_timeout plus the lease safety margin. Passed - /// as `fence_ok` to every `CasRequestController` call the ref-log writer path makes. + /// Delegate to `mount_runtime`: the write fence moved there. Extends `mayMutate` with the REMAINING + /// budget check -- work is not started unless there is enough of the mount lease left for one more + /// attempt timeout plus the safety margin. bool refAppendFenceOk() const; /// incidental-detection reaction for a foreign-interference @@ -1019,13 +1032,24 @@ class Pool : public std::enable_shared_from_this ref_ledger.setSnapshotBeforeCkptCasHookForTest(std::move(hook)); } - /// Test-only: replace the request controller's inter-attempt backoff sleep (e.g. with a no-op) — - /// for tests that drive a persistent conditional-write fault to budget exhaustion through a fully - /// wired Pool/disk and must not serve the production capped-exponential sleeps for real (see - /// `CasRequestController::setSleepFnForTest`). Call before driving traffic; empty restores the - /// real sleep. + /// Test-only: replace the inter-attempt backoff sleep (e.g. with a clock-advancing no-op) on all + /// three request planes and on ref-table recovery, for tests that drive a persistent write fault to + /// exhaustion through a fully wired Pool/disk and must not serve the production sleeps for real. + /// Call before driving traffic. On the three request planes an empty function restores each plane's + /// own default, the mount plane's interruptible sleep included. + /// + /// It does NOT bound a reissue the engine refuses to start: the engine's inter-attempt backoff is + /// jittered and drawn before the sleep, and admission compares that drawn duration against the + /// lease. A test that needs a reissue admitted, or refused, deterministically has to arrange the + /// clock, not the sleep. void setCasRetrySleepForTest(std::function sleep_fn); + /// Test-only: replace the request engine's clock on all three planes. A test driving a PERSISTENT + /// transient fault must run the retry window on a clock it advances; the sleep seam alone cannot + /// bound it, because a read the engine keeps reissuing is bounded by the policy deadline and the + /// deadline is read from this clock. + void setCasRequestNowFnForTest(std::function now_fn); + /// Test-only: replace only ref-table recovery's token-aware retry delay seam. void setRefRecoveryRetrySleepForTest( std::function &)> sleep_fn); @@ -1108,10 +1132,25 @@ class Pool : public std::enable_shared_from_this return std::forward(mutation)(); } + /// The mount plane's inter-attempt sleep: interruptible, so a parked or stopping renewal is not + /// held for a whole capped backoff. Named rather than inlined because the test seam has to be able + /// to put it back. + std::function mountPlaneSleepFn() + { + return [this](uint64_t ms) { mount_runtime.sleepInterruptibly(ms); }; + } + BackendPtr pool_backend; PoolConfig config; PoolMeta meta; + /// The three planes' engines, declared before every component that is handed one and after the + /// config they take their clock from. `mutable` because issuing a request is not a change to the + /// pool: a `const` observer still has to read the store. + mutable CasRequests mount_requests; + mutable CasRequests farewell_requests; + mutable CasRequests gc_requests; + std::shared_ptr detached_work = std::make_shared(); mutable std::mutex writer_cleanup_mutex; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp index 8b1e4bd9ba02..fddb27103311 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPoolMeta.cpp @@ -1,9 +1,10 @@ #include -#include +#include #include #include #include #include +#include namespace DB { @@ -62,51 +63,48 @@ String joinAlgoNames(const std::vector & algos_used) } /// The relaxed admission check (replaces an earlier fail-close that required the single pool algo to match): -/// `pm`/`token` are the most-recently-read `_pool_meta` state (present, decoded, valid). Already a -/// member of `algos_used` => OK, no write (steady state). Not a member and `!allow_new` => -/// `BAD_ARGUMENTS` (the pool is never touched). Not a member and `allow_new` => CAS-union `config_algo` -/// into `algos_used` (recomputed from the FRESH value on every retry -- union-only, so there is no -/// ABA) and raises `min_reader_generation` to THIS build's own floor (`G_BUILD`, `CasFormat.h`) in -/// the SAME write (first registration of a schema-3-bearing algo also raises -/// `min_reader_generation` -- a build that cannot decode schema-3 settlement state has an OLDER -/// `G_BUILD` and is correctly refused by the startup gate once a future generation bump lands here). -/// On a CAS conflict, re-read and retry the whole decision (a concurrent admitter may have unioned a -/// DIFFERENT algo, or the very one we wanted, in the meantime). -PoolMeta admitOrValidate( - Backend & backend, const String & key, PoolMeta pm, Token token, - BlobHashAlgo config_algo, bool allow_new) +/// re-reads `key` itself (via `readModifyWrite`'s own observe) rather than trusting a snapshot the +/// caller already holds, so a caller that only knows the key is present -- never a stale decode -- +/// can ask this to settle admission. Already a member of `algos_used` => OK, no write (steady state, +/// `decide` declines). Not a member and `!allow_new` => `BAD_ARGUMENTS` (the pool is never touched). +/// Not a member and `allow_new` => CAS-union `config_algo` into `algos_used` (recomputed from the +/// FRESH value on every retry -- union-only, so there is no ABA) and raises `min_reader_generation` to +/// THIS build's own floor (`G_BUILD`, `CasFormat.h`) in the SAME write (first registration of a +/// schema-3-bearing algo also raises `min_reader_generation` -- a build that cannot decode schema-3 +/// settlement state has an OLDER `G_BUILD` and is correctly refused by the startup gate once a future +/// generation bump lands here). A concurrent admitter's own union is folded in the same way, since +/// `decide` runs again against whatever `readModifyWrite` observes on retry. +PoolMeta admitOrValidate(CasOperation & op, const String & key, BlobHashAlgo config_algo, bool allow_new) { - for (;;) - { - if (isAlgoAdmittedIn(pm, config_algo)) - return pm; - - if (!allow_new) - throwNotAdmitted(pm, config_algo); - - PoolMeta next = pm; - next.algos_used.push_back(static_cast(config_algo)); - std::sort(next.algos_used.begin(), next.algos_used.end()); - next.min_reader_generation = G_BUILD; - - const CasResult res = backend.casPut(key, encodePoolMeta(next), token); - if (res.outcome == CasOutcome::Committed) - return next; - - auto fresh = backend.get(key); - if (!fresh) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS pool meta: '{}' vanished mid-admission (conflicting write then a concurrent delete)", key); - pm = decodePoolMeta(fresh->bytes); - token = fresh->token; - /// loop: re-evaluate membership against the FRESH pm (never re-encode the stale `next`) - } + PoolMeta observed; + WriteResult result = op.readModifyWrite( + key, + [&](const std::optional & current) -> std::optional + { + if (!current) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS pool meta: '{}' vanished mid-admission (conflicting write then a concurrent delete)", key); + observed = decodePoolMeta(current->bytes); + + if (isAlgoAdmittedIn(observed, config_algo)) + return std::nullopt; + if (!allow_new) + throwNotAdmitted(observed, config_algo); + + observed.algos_used.push_back(static_cast(config_algo)); + std::sort(observed.algos_used.begin(), observed.algos_used.end()); + observed.min_reader_generation = G_BUILD; + return encodePoolMeta(observed); + }, + Retry::standard()); + orThrow(std::move(result), fmt::format("CAS pool meta admission on '{}'", key)); + return observed; } } PoolMeta PoolMeta::createOrValidate( - Backend & backend, const Layout & layout, uint64_t blob_header_len, uint64_t gc_shards, + CasOperation & op, const Layout & layout, uint64_t blob_header_len, uint64_t gc_shards, BlobHashAlgo blob_hash_algo, bool allow_new, bool allow_mint) { /// The passed config is the caller's responsibility — reject bad values before any I/O. @@ -120,11 +118,16 @@ PoolMeta PoolMeta::createOrValidate( const String key = layout.poolMetaKey(); /// Present => the pool is authoritative; ignore the passed config's blob_header_len and run the - /// flag-gated admission check rather than the old single-value fail-close. - if (auto existing = backend.get(key)) + /// flag-gated admission check rather than the old single-value fail-close. The steady state (the + /// configured algo is already admitted) is decided from THIS read, so the common open costs one + /// GET; only a union or a `!allow_new` refusal falls through to `admitOrValidate`, which re-reads + /// the key itself as part of its own conditional write. + if (auto existing = op.read(key, Retry::standard())) { PoolMeta pm = decodePoolMeta(existing->bytes); - return admitOrValidate(backend, key, std::move(pm), existing->token, blob_hash_algo, allow_new); + if (isAlgoAdmittedIn(pm, blob_hash_algo)) + return pm; + return admitOrValidate(op, key, blob_hash_algo, allow_new); } /// Absent => mint a pool id and try to create the object with `algos_used = {blob_hash_algo}`. @@ -152,17 +155,17 @@ PoolMeta PoolMeta::createOrValidate( pm.min_reader_generation = G_BUILD; pm.algos_used = {static_cast(blob_hash_algo)}; - if (backend.casPut(key, encodePoolMeta(pm), /*expected*/ std::nullopt).outcome == CasOutcome::Committed) + WriteResult result = op.create(key, encodePoolMeta(pm), Retry::standard()); + if (std::holds_alternative(result)) return pm; /// Lost the race: the winner's object MUST be present now. The loser UNIONS its algo via the SAME /// flag-gated admission path as a reopen, instead of the old unconditional fail-close. - auto winner = backend.get(key); - if (!winner) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS pool meta: create-if-absent reported Conflict but '{}' is absent on re-read", key); - PoolMeta winner_pm = decodePoolMeta(winner->bytes); - return admitOrValidate(backend, key, std::move(winner_pm), winner->token, blob_hash_algo, allow_new); + if (std::holds_alternative(result)) + return admitOrValidate(op, key, blob_hash_algo, allow_new); + + orThrow(std::move(result), fmt::format("CAS pool meta creation on '{}'", key)); + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS pool meta: create-if-absent on '{}' neither committed, conflicted, nor threw", key); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp index b9b952a7321a..e6f0637e94ec 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp @@ -2,9 +2,11 @@ #include #include #include +#include #include #include #include +#include namespace DB { @@ -22,70 +24,90 @@ namespace { -CasRefCatalog::Snapshot readOptionalForBootstrap(Backend & backend, const Layout & layout) +/// The one verdict for an absent mandatory catalog, so the read entry point and the mutation's own +/// `decide` cannot drift apart on what absence means. +[[noreturn]] void throwMandatoryCatalogAbsent(const String & key) { - const auto got = backend.get(layout.refCatalogKey()); + throw Exception(ErrorCodes::CORRUPTED_DATA, + "Mandatory CAS ref catalog '{}' is absent -- refusing to interpret opaque life " + "objects as an empty ownership universe", + key); +} + +/// Every non-committed alternative of a catalog write, as the exception its meaning already implies. +/// `Declined` cannot reach here: `create` never declines, and no `decide` in this file returns +/// nothing to write. +[[noreturn]] void throwCatalogWriteFailure(WriteResult result, const String & what) +{ + orThrow(std::move(result), what); + throw Exception(ErrorCodes::LOGICAL_ERROR, "{}: the write was declined, which this call cannot produce", what); +} + +CasRefCatalog::Snapshot readOptionalForBootstrap(CasOperation & op, const Layout & layout) +{ + const std::optional got = op.read(layout.refCatalogKey(), Retry::standard()); if (!got) { RefCatalog empty; return CasRefCatalog::Snapshot{ - .catalog = empty, .token = std::nullopt, .life_index = CatalogLifeIndex(empty)}; + .catalog = empty, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(empty)}; } RefCatalog catalog = decodeRefCatalog(got->bytes); return CasRefCatalog::Snapshot{ - .catalog = catalog, .token = got->token, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .incarnation = got->incarnation, .life_index = CatalogLifeIndex(catalog)}; } } -CasRefCatalog::Snapshot CasRefCatalog::read(Backend & backend, const Layout & layout) +CasRefCatalog::Snapshot CasRefCatalog::read(CasOperation & op, const Layout & layout) { - Snapshot snapshot = readOptionalForBootstrap(backend, layout); - if (!snapshot.token) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "Mandatory CAS ref catalog '{}' is absent -- refusing to interpret opaque life " - "objects as an empty ownership universe", - layout.refCatalogKey()); + Snapshot snapshot = readOptionalForBootstrap(op, layout); + if (!snapshot.incarnation) + throwMandatoryCatalogAbsent(layout.refCatalogKey()); return snapshot; } -CasRefCatalog::Snapshot CasRefCatalog::initializeEmptyForNewPool(Backend & backend, const Layout & layout) +CasRefCatalog::Snapshot CasRefCatalog::initializeEmptyForNewPool(CasOperation & op, const Layout & layout) { + const String key = layout.refCatalogKey(); RefCatalog empty; const String canonical_empty = encodeRefCatalog(empty); - const PutResult put = backend.putIfAbsent(layout.refCatalogKey(), canonical_empty); - if (put.outcome == PutOutcome::Done) - return Snapshot{.catalog = empty, .token = put.token, .life_index = CatalogLifeIndex(empty)}; - - /// A second opener can win after both proved the prefix empty. Decode its exact object before - /// accepting the race; conflict is never a license to continue with an assumed empty catalog or - /// arbitrary decoded body. - const auto got = backend.get(layout.refCatalogKey()); - if (!got) + WriteResult result = op.create(key, canonical_empty, Retry::standard()); + if (const auto * committed = std::get_if(&result)) + return Snapshot{.catalog = empty, .incarnation = committed->incarnation, .life_index = CatalogLifeIndex(empty)}; + + /// A second opener can win after both proved the prefix empty. The refused precondition was + /// settled by an exact read, so the winner's object is decoded from what that read observed; + /// a conflict is never a license to continue with an assumed empty catalog or an arbitrary body. + const auto * conflict = std::get_if(&result); + if (!conflict) + throwCatalogWriteFailure(std::move(result), fmt::format("CAS ref catalog '{}' bootstrap create", key)); + + const auto * occupant = std::get_if(&conflict->seen); + if (!occupant) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS ref catalog '{}' disappeared after bootstrap create conflict", - layout.refCatalogKey()); - RefCatalog catalog = decodeRefCatalog(got->bytes); - if (!catalog.entries.empty() || got->bytes != canonical_empty) + "CAS ref catalog '{}' disappeared after bootstrap create conflict", key); + RefCatalog catalog = decodeRefCatalog(occupant->bytes); + if (!catalog.entries.empty() || occupant->bytes != canonical_empty) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS ref catalog '{}' conflicts with bootstrap's required canonical empty catalog", - layout.refCatalogKey()); - return Snapshot{.catalog = std::move(catalog), .token = got->token, .life_index = CatalogLifeIndex(empty)}; + "CAS ref catalog '{}' conflicts with bootstrap's required canonical empty catalog", key); + return Snapshot{.catalog = std::move(catalog), .incarnation = occupant->incarnation, + .life_index = CatalogLifeIndex(empty)}; } std::optional CasRefCatalog::lifeIfCataloged( - Backend & backend, const Layout & layout, const RootNamespace & ns) + CasOperation & op, const Layout & layout, const RootNamespace & ns) { - const Snapshot snap = read(backend, layout); + const Snapshot snap = read(op, layout); for (const CatalogEntry & entry : snap.catalog.entries) if (entry.ns.string() == ns.string() && entry.state != NsState::Creating) return snap.life_index.resolve(entry.incarnation); return std::nullopt; } -std::vector CasRefCatalog::liveUniverse(Backend & backend, const Layout & layout) +std::vector CasRefCatalog::liveUniverse(CasOperation & op, const Layout & layout) { - const Snapshot snap = read(backend, layout); + const Snapshot snap = read(op, layout); snap.life_index.throwIfAmbiguous("CAS live namespace discovery"); std::vector universe; universe.reserve(snap.catalog.entries.size()); @@ -101,52 +123,93 @@ std::vector CasRefCatalog::liveUniverse(Backend & backend, cons namespace { -/// Live-lock brake, the same shape and for the same reason as `publishCkpt`'s/`allocateWriterEpoch`'s -/// on their own contended token-CAS singletons: the catalog is ONE object mutated by every lifecycle -/// transition of every namespace in the pool, so persistent contention is a real, not theoretical, -/// exit condition to plan for. +/// Thrown from inside a `casUpdate` `mutate` closure to signal a refusal that must STOP the attempt +/// rather than be treated as a refused precondition to retry: `casUpdateImpl` propagates whatever +/// `mutate` throws straight out, uncaught, which is exactly the behavior these three need. Retrying +/// any of them against a freshly re-read catalog would just re-decide against an entry that is, by +/// definition, no longer `observed` -- token-exactness means the FIRST mismatch is final, not a reason +/// to loop. Each is caught by its own exact type right where it is thrown; deriving from +/// `std::exception` is only so the throw itself is well-formed, never so a caller catches these by +/// base class. +struct CatalogFenceMovedMarker : std::exception {}; +struct CatalogEntryMismatchMarker : std::exception {}; +struct CatalogCreatorStillLiveMarker : std::exception {}; + +/// Live-lock brake for the ONE loop below that is written by hand rather than driven by the engine: +/// the catalog is a single object mutated by every lifecycle transition of every namespace in the +/// pool, so persistent contention is a real, not theoretical, exit condition to plan for. +/// +/// It bounds ATTEMPTS, not time. Each iteration binds its own window per verb and pauses between +/// iterations, so the aggregate wall clock of one call is this cap times a backoff plus two verb +/// windows -- hours in the worst case. That is accepted because the only caller is the GC pre-fold +/// drain: a background round that may take as long as it takes, and whose next round re-derives +/// everything anyway. A foreground path must not adopt this loop without a wall-clock bound. constexpr size_t kMaxCatalogCasAttempts = 100; /// Shared body of `casUpdate`/`casAdmitEntry`. `encode` turns a freshly `mutate`d candidate into the /// bytes to write: the plain path just grammar-checks (`encodeRefCatalog`), the admitting path also -/// runs both admission predicates (`checkCatalogAdmission`) first. Retries on `Conflict` against a -/// FRESH read, exactly like `PoolMeta::admitOrValidate` -- never re-encoding the stale candidate. +/// runs both admission predicates (`checkCatalogAdmission`) first. A refused precondition re-runs +/// `mutate` against the FRESH body -- never re-encoding the stale candidate. RefCatalog casUpdateImpl( - Backend & backend, const Layout & layout, + CasOperation & op, const Layout & layout, const std::function & mutate, const std::function & encode) { const String key = layout.refCatalogKey(); - CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + /// The candidate the LAST `decide` produced, which is the one the engine wrote: every earlier one + /// belongs to an attempt whose precondition was refused. + std::optional written; - for (size_t attempt = 0; attempt < kMaxCatalogCasAttempts; ++attempt) + const auto decide = [&](const std::optional & current) -> std::optional { - snap.life_index.throwIfAmbiguous("CAS ref catalog mutation"); - RefCatalog candidate = mutate(snap.catalog); - const String bytes = encode(candidate); - const CasResult res = backend.casPut(key, bytes, snap.token); - if (res.outcome == CasOutcome::Committed) - return candidate; - - snap = CasRefCatalog::read(backend, layout); - /// `read` treats authoritative absence after a conflict as corruption. Therefore no retry - /// can turn a vanished mandatory catalog into a one-update replacement authority. - } + /// Absence of the mandatory catalog is corruption, not a fresh bootstrap. Refusing here is + /// what stops any mutation from replacing every other namespace with a one-update catalog. + if (!current) + throwMandatoryCatalogAbsent(key); + const RefCatalog durable = decodeRefCatalog(current->bytes); + CatalogLifeIndex(durable).throwIfAmbiguous("CAS ref catalog mutation"); + RefCatalog candidate = mutate(durable); + String bytes = encode(candidate); + written = std::move(candidate); + return bytes; + }; - throwCasWriteRetryLater(fmt::format( - "CAS ref catalog '{}' did not converge after {} attempts", key, kMaxCatalogCasAttempts)); + WriteResult result = op.readModifyWrite(key, decide, Retry::standard()); + /// The fence can be lost in two places and both mean the same to a lifecycle caller: inside + /// `decide`, which throws the marker itself, and between two attempts, where the engine notices it + /// first and no further `decide` runs. Normalising the second onto the first is what keeps "the + /// fence moved" a returned outcome rather than an exception. + if (const auto * gave_up = std::get_if(&result); gave_up && gave_up->why == GaveUp::Why::FenceLost) + throw CatalogFenceMovedMarker{}; + if (!std::holds_alternative(result)) + throwCatalogWriteFailure(std::move(result), fmt::format("CAS ref catalog '{}' update", key)); + return std::move(*written); } -/// Thrown from inside a `casUpdate` `mutate` closure to signal a refusal that must STOP the attempt -/// rather than be treated as a `Conflict` to retry: `casUpdateImpl` propagates whatever `mutate` -/// throws straight out, uncaught, which is exactly the behavior these three need. Retrying any of them -/// against a freshly re-read catalog would just re-decide against an entry that is, by definition, no -/// longer `observed` -- token-exactness means the FIRST mismatch is final, not a reason to loop. -/// Each is caught by its own exact type right where it is thrown; deriving from `std::exception` -/// is only so the throw itself is well-formed, never so a caller catches these by base class. -struct CatalogFenceMovedMarker : std::exception {}; -struct CatalogEntryMismatchMarker : std::exception {}; -struct CatalogCreatorStillLiveMarker : std::exception {}; +/// `casUpdate`'s own guard, shared with the two lifecycle callers that need to catch the fence marker +/// the public entry point translates away. +std::function identityPreserving( + const std::function & mutate) +{ + return [&mutate](const RefCatalog & current) -> RefCatalog + { + RefCatalog candidate = mutate(current); + if (candidate.entries.size() != current.entries.size()) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CasRefCatalog::casUpdate cannot add or delete catalog entries -- use casAdmitEntry, " + "deleteCompletedRemoving, or cancelStalledCreating"); + for (size_t i = 0; i < current.entries.size(); ++i) + { + if (candidate.entries[i].ns != current.entries[i].ns + || candidate.entries[i].incarnation != current.entries[i].incarnation) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CasRefCatalog::casUpdate cannot replace catalog identity at row {} -- namespace " + "and incarnation are immutable outside the narrow admission/deletion APIs", + i); + } + return candidate; + }; +} /// Two `thread_local_rng` draws composed into a `UInt128`, the same pattern already used throughout /// this tree to mint build ids and incarnation tags (`CasPartWriteTxn.cpp`'s `mintU128`, @@ -198,7 +261,7 @@ std::function create_namespace_step1_pre_read_hook_for_test; /// canonical-order/no-duplicate grammar check abort the process with `LOGICAL_ERROR` for what is, at /// this call site only, an ordinary race outcome. RefCatalog createNamespaceStep1( - Backend & backend, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry) + CasOperation & op, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry) { /// Moved into a local before invoking, not called on the global directly: a hook that reassigns /// `create_namespace_step1_pre_read_hook_for_test` from inside its own body (a test driving a @@ -222,7 +285,7 @@ RefCatalog createNamespaceStep1( next.entries.insert(it, entry); return next; }; - return casUpdateImpl(backend, layout, mutate, + return casUpdateImpl(op, layout, mutate, [&entry, gc_shards, &layout](const RefCatalog & c) { return checkCatalogAdmission(c, gc_shards, layout, entry.ns); @@ -232,32 +295,25 @@ RefCatalog createNamespaceStep1( } RefCatalog CasRefCatalog::casUpdate( - Backend & backend, const Layout & layout, const std::function & mutate) + CasOperation & op, const Layout & layout, const std::function & mutate) { - const auto identity_preserving_mutate = [&](const RefCatalog & current) -> RefCatalog + try { - RefCatalog candidate = mutate(current); - if (candidate.entries.size() != current.entries.size()) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CasRefCatalog::casUpdate cannot add or delete catalog entries -- use casAdmitEntry, " - "deleteCompletedRemoving, or cancelStalledCreating"); - for (size_t i = 0; i < current.entries.size(); ++i) - { - if (candidate.entries[i].ns != current.entries[i].ns - || candidate.entries[i].incarnation != current.entries[i].incarnation) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CasRefCatalog::casUpdate cannot replace catalog identity at row {} -- namespace " - "and incarnation are immutable outside the narrow admission/deletion APIs", - i); - } - return candidate; - }; - return casUpdateImpl( - backend, layout, identity_preserving_mutate, [](const RefCatalog & c) { return encodeRefCatalog(c); }); + return casUpdateImpl( + op, layout, identityPreserving(mutate), [](const RefCatalog & c) { return encodeRefCatalog(c); }); + } + catch (const CatalogFenceMovedMarker &) + { + /// The marker is this file's private signal; a caller outside it gets the exception class every + /// other admission refusal raises. + throwCasTransientUnavailable( + fmt::format("CAS ref catalog '{}' update", layout.refCatalogKey()), + "mount fence tripped: the update was admitted under an incarnation this node no longer holds"); + } } RefCatalog CasRefCatalog::casAdmitEntry( - Backend & backend, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry) + CasOperation & op, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry) { if (entry.state == NsState::Removing) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -279,7 +335,7 @@ RefCatalog CasRefCatalog::casAdmitEntry( next.entries.insert(it, entry); return next; }; - return casUpdateImpl(backend, layout, mutate, + return casUpdateImpl(op, layout, mutate, [&entry, gc_shards, &layout](const RefCatalog & c) { return checkCatalogAdmission(c, gc_shards, layout, entry.ns); @@ -287,9 +343,8 @@ RefCatalog CasRefCatalog::casAdmitEntry( } CasRefCatalog::BeginRemovingOutcome CasRefCatalog::beginRemoving( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - uint64_t removal_started_round, uint64_t admitted_generation, - const std::function & check_fence_or_throw) + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + uint64_t removal_started_round) { if (observed.state != NsState::Live || observed.creator || observed.removal_started_round) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -298,8 +353,8 @@ CasRefCatalog::BeginRemovingOutcome CasRefCatalog::beginRemoving( const auto mutate = [&](const RefCatalog & cur) -> RefCatalog { - try { check_fence_or_throw(admitted_generation); } - catch (...) { throw CatalogFenceMovedMarker{}; } + if (!op.admitted()) + throw CatalogFenceMovedMarker{}; const auto it = findEntry(cur, observed.ns); if (it == cur.entries.end() || *it != observed) @@ -314,7 +369,7 @@ CasRefCatalog::BeginRemovingOutcome CasRefCatalog::beginRemoving( try { - casUpdateImpl(backend, layout, mutate, [](const RefCatalog & c) { return encodeRefCatalog(c); }); + casUpdateImpl(op, layout, mutate, [](const RefCatalog & c) { return encodeRefCatalog(c); }); } catch (const CatalogFenceMovedMarker &) { @@ -322,7 +377,7 @@ CasRefCatalog::BeginRemovingOutcome CasRefCatalog::beginRemoving( } catch (const CatalogEntryMismatchMarker &) { - const Snapshot current = read(backend, layout); + const Snapshot current = read(op, layout); const auto it = findEntry(current.catalog, observed.ns); if (it != current.catalog.entries.end() && it->incarnation == observed.incarnation @@ -334,9 +389,9 @@ CasRefCatalog::BeginRemovingOutcome CasRefCatalog::beginRemoving( } CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemoving( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - const CasFoldSeal & authoritative_parent, uint64_t admitted_generation, - const std::function & check_fence) + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + const CasFoldSeal & authoritative_parent, + const std::function & refresh_authority) { if (observed.state != NsState::Removing || !observed.removal_started_round) return { @@ -354,15 +409,13 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov .catalog_snapshot = std::nullopt}; return deleteCompletedRemovingAtSnapshot( - backend, layout, read(backend, layout), observed, authoritative_parent, - admitted_generation, check_fence); + op, layout, read(op, layout), observed, authoritative_parent, refresh_authority); } CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemovingAtSnapshot( - Backend & backend, const Layout & layout, Snapshot catalog_snapshot, + CasOperation & op, const Layout & layout, Snapshot catalog_snapshot, const CatalogEntry & observed, const CasFoldSeal & authoritative_parent, - uint64_t admitted_generation, - const std::function & check_fence) + const std::function & refresh_authority) { if (observed.state != NsState::Removing || !observed.removal_started_round) return { @@ -394,42 +447,45 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov .catalog_snapshot = std::move(catalog_snapshot)}; }; + /// ONE policy value for every erase this loop sends. Each call binds its own window when it is + /// made; what is shared is the policy, not a deadline. + const Retry policy = Retry::standard(); + for (size_t attempt = 0; attempt < kMaxCatalogCasAttempts; ++attempt) { + /// So the admission consulted below is not one reading taken before the first attempt. + if (refresh_authority) + refresh_authority(); + catalog_snapshot.life_index.throwIfAmbiguous("CAS completed-removal deletion"); + /// A caller-supplied cut without an incarnation cannot state a precondition, and an erase that + /// fell back to an unconditional write would delete whatever a concurrent writer had put there. + if (!catalog_snapshot.incarnation) + throwMandatoryCatalogAbsent(layout.refCatalogKey()); const auto observed_it = findEntry(catalog_snapshot.catalog, observed.ns); if (observed_it == catalog_snapshot.catalog.entries.end() || *observed_it != observed) return resolved_result(CompletedRemovingDeleteOutcome::EntryChanged); - bool fence_lost = check_fence(admitted_generation) == LeaderFenceStatus::Moved; - - std::optional cas_result; - std::exception_ptr attempt_failure; - if (!fence_lost) - { - RefCatalog candidate = catalog_snapshot.catalog; - candidate.entries.erase(candidate.entries.begin() + (observed_it - catalog_snapshot.catalog.entries.begin())); - try - { - cas_result = backend.casPut( - layout.refCatalogKey(), encodeRefCatalog(candidate), catalog_snapshot.token); - } - catch (...) - { - attempt_failure = std::current_exception(); - } - } + /// Nothing is attempted without admission, and a refusal here has sent nothing. + if (!op.admitted()) + return resolved_result(CompletedRemovingDeleteOutcome::FencedOut); - /// The response to a conditional erase is not authority for what became durable. Resolve - /// every attempted erase, and a pre-CAS fence refusal, through one complete catalog read. - /// This snapshot is also the next retry/selection cut, so no second read separates them. - catalog_snapshot = read(backend, layout); + RefCatalog candidate = catalog_snapshot.catalog; + candidate.entries.erase(candidate.entries.begin() + (observed_it - catalog_snapshot.catalog.entries.begin())); + WriteResult erase = op.replace(layout.refCatalogKey(), encodeRefCatalog(candidate), + *catalog_snapshot.incarnation, policy); - if (!fence_lost) - fence_lost = check_fence(admitted_generation) == LeaderFenceStatus::Moved; - if (fence_lost) + /// An operation whose admission is gone cannot issue the resolution read either, so the call + /// ends HERE and reports the cut it was given rather than a fresh one. There is nothing further + /// this actor may learn, and nothing further it may do. + if (!op.admitted()) return resolved_result(CompletedRemovingDeleteOutcome::FencedOut); + /// The response to a conditional erase is not authority for what became durable. Resolve every + /// attempted erase through one complete catalog read. This snapshot is also the next + /// retry/selection cut, so no second read separates them. + catalog_snapshot = read(op, layout); + const auto current_it = findEntry(catalog_snapshot.catalog, observed.ns); const bool old_life_still_cataloged = current_it != catalog_snapshot.catalog.entries.end() && current_it->incarnation == observed.incarnation; @@ -438,15 +494,25 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov ? CompletedRemovingDeleteOutcome::Deleted : CompletedRemovingDeleteOutcome::EntryChanged); - if (attempt_failure) - std::rethrow_exception(attempt_failure); - if (cas_result && cas_result->outcome == CasOutcome::Committed) - throwCasWriteRetryLater(fmt::format( - "CAS ref catalog erase for namespace '{}' reported committed, but a complete resolution read " - "still observed incarnation {}", - observed.ns.string(), u128ToHex(observed.incarnation))); - /// A token conflict that leaves the exact old row present retries from this mandatory - /// resolution snapshot. The fence is checked again immediately before the next CAS. + /// The row survived the attempt. Only a refused precondition may be tried again against the + /// mandatory resolution snapshot above; every other alternative is terminal for this call, and + /// a commit the resolution read contradicts is reported rather than believed. + if (!std::holds_alternative(erase)) + { + if (std::holds_alternative(erase)) + throwCasWriteRetryLater(fmt::format( + "CAS ref catalog erase for namespace '{}' reported committed, but a complete resolution read " + "still observed incarnation {}", + observed.ns.string(), u128ToHex(observed.incarnation))); + throwCatalogWriteFailure(std::move(erase), fmt::format( + "CAS ref catalog erase for namespace '{}'", observed.ns.string())); + } + + /// Only a refused precondition reaches here, so this pause paces one contended key's retries. + /// The argument is the number of reissues so far, which is one more than the zero-based + /// iteration: `backoff(0)` is no wait at all, and the first retry is the one most likely to + /// collide with the writer that just won. + op.pause(Retry::backoff(static_cast(attempt) + 1)); } throwCasWriteRetryLater(fmt::format( @@ -455,9 +521,8 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov } CasRefCatalog::StalledCreatingCancelOutcome CasRefCatalog::cancelStalledCreating( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - const std::function & is_creator_fence_terminal, - uint64_t admitted_generation, const std::function & check_fence_or_throw) + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + const std::function & is_creator_fence_terminal) { if (observed.state != NsState::Creating || !observed.creator) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -467,8 +532,8 @@ CasRefCatalog::StalledCreatingCancelOutcome CasRefCatalog::cancelStalledCreating const auto mutate = [&](const RefCatalog & cur) -> RefCatalog { - try { check_fence_or_throw(admitted_generation); } - catch (...) { throw CatalogFenceMovedMarker{}; } + if (!op.admitted()) + throw CatalogFenceMovedMarker{}; const auto it = findEntry(cur, observed.ns); if (it == cur.entries.end() || *it != observed) @@ -483,7 +548,7 @@ CasRefCatalog::StalledCreatingCancelOutcome CasRefCatalog::cancelStalledCreating try { - casUpdateImpl(backend, layout, mutate, [](const RefCatalog & c) { return encodeRefCatalog(c); }); + casUpdateImpl(op, layout, mutate, [](const RefCatalog & c) { return encodeRefCatalog(c); }); } catch (const CatalogFenceMovedMarker &) { return StalledCreatingCancelOutcome::FencedOut; } catch (const CatalogEntryMismatchMarker &) { return StalledCreatingCancelOutcome::EntryChanged; } @@ -492,9 +557,7 @@ CasRefCatalog::StalledCreatingCancelOutcome CasRefCatalog::cancelStalledCreating } CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - uint64_t admitted_generation, const std::function & check_fence_or_throw, - const CkptDeadline & deadline) + CasOperation & op, const Layout & layout, const CatalogEntry & observed) { if (observed.state != NsState::Creating || !observed.creator) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -503,13 +566,14 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( /// Step 2 (spec §3): INV-4's first `_ckpt` writer for this incarnation, and the only writer that /// will ever know its genesis epoch -- see `Pool/CasRefCkpt.h`'s `publishCkpt` doc for the merge - /// discipline this rides on unchanged. `FencedOut` here ends the attempt: nothing durable changed. + /// discipline this rides on unchanged. `FencedOut` here ends the attempt without step 3; the + /// `_ckpt` itself may or may not have become durable, which is why the entry is left `Creating` + /// for whichever actor next reconciles it rather than cleaned up here. const RefCkpt contribution{.life_epoch = std::optional{observed.creator->writer_epoch}, .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - if (publishCkpt(backend, layout, NamespaceLifeId::fromCatalogEntry(observed.ns, observed.incarnation), - contribution, admitted_generation, check_fence_or_throw, - deadline) == CkptPublishOutcome::FencedOut) + if (publishCkpt(op, layout, NamespaceLifeId::fromCatalogEntry(observed.ns, observed.incarnation), + contribution) == CkptPublishOutcome::FencedOut) return NamespaceCreationOutcome::FencedOut; /// Step 3. `mutate` is the fence re-check point `casUpdate`'s header doc names -- checked FIRST, @@ -518,8 +582,10 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( /// (both are truthful refusals of a CAS that was never sent; this is only which one speaks first). const auto mutate = [&](const RefCatalog & cur) -> RefCatalog { - try { check_fence_or_throw(admitted_generation); } - catch (...) { throw CatalogFenceMovedMarker{}; } /// typed, not propagated -- publishCkpt's own precedent + /// Typed, not propagated: the caller asked "did this land", and "the fence moved, so nothing + /// was sent" is an answer, not a failure of the operation. + if (!op.admitted()) + throw CatalogFenceMovedMarker{}; const auto it = findEntry(cur, observed.ns); if (it == cur.entries.end() || *it != observed) @@ -533,7 +599,8 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( try { - casUpdate(backend, layout, mutate); + casUpdateImpl(op, layout, identityPreserving(mutate), + [](const RefCatalog & c) { return encodeRefCatalog(c); }); } catch (const CatalogFenceMovedMarker &) { return NamespaceCreationOutcome::FencedOut; } catch (const CatalogEntryMismatchMarker &) { return NamespaceCreationOutcome::Superseded; } @@ -541,10 +608,8 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( } CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( - Backend & backend, const Layout & layout, uint64_t gc_shards, - const RootNamespace & ns, const CreatorFence & creator, - uint64_t admitted_generation, const std::function & check_fence_or_throw, - const CkptDeadline & deadline) + CasOperation & op, const Layout & layout, uint64_t gc_shards, + const RootNamespace & ns, const CreatorFence & creator) { /// Read-first, per the Task 2 review's own note on `casAdmitEntry`: a namespace that already /// carries an entry is THIS function's job to reject with a clear message, not `casAdmitEntry`'s @@ -552,7 +617,7 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( /// -- true, but useless to a caller trying to understand why its create failed). A concurrent /// insert of the SAME namespace between this read and step 1 is still caught -- `casAdmitEntry`'s /// own grammar check is the backstop, not the only check. - const Snapshot snap = read(backend, layout); + const Snapshot snap = read(op, layout); const auto existing = findEntry(snap.catalog, ns); if (existing != snap.catalog.entries.end()) { @@ -587,13 +652,13 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( /// above) and reports the race as `Superseded` instead. try { - createNamespaceStep1(backend, layout, gc_shards, entry); /// step 1 + createNamespaceStep1(op, layout, gc_shards, entry); /// step 1 } catch (const CatalogEntryAlreadyPresentMarker &) { return NamespaceCreationOutcome::Superseded; } - return completeCreation(backend, layout, entry, admitted_generation, check_fence_or_throw, deadline); + return completeCreation(op, layout, entry); } void CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest(std::function hook) @@ -602,24 +667,25 @@ void CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest(std::function & is_creator_fence_terminal, - uint64_t admitted_generation, const std::function & check_fence_or_throw) + CasOperation & op, const Layout & layout, const CatalogEntry & observed, const CreatorFence & new_creator, + const std::function & is_creator_fence_terminal) { if (observed.state != NsState::Creating || !observed.creator) throw Exception(ErrorCodes::LOGICAL_ERROR, "CasRefCatalog::reconcileStaleCreator: namespace '{}' is not a Creating entry with a " "creator fence -- nothing to reconcile", observed.ns.string()); - /// Review I6: the fence re-check is checked FIRST, on every fresh read this CAS retries -- the same - /// placement `completeCreation` uses for exactly the same reason (see that function's own doc). - /// Token-exactness (the catalog's own entry, by full value) comes next: it is the cheaper, purely + /// The admission check comes FIRST, on every fresh read this retries -- the same placement + /// `completeCreation` uses for exactly the same reason (see that function's own doc). + /// Entry-exactness (the catalog's own entry, by full value) comes next: it is the cheaper, purely /// local comparison, and a mismatch here means the question "is the OLD creator's fence terminal" is /// moot -- `observed` no longer describes anything live to reconcile. const auto mutate = [&](const RefCatalog & cur) -> RefCatalog { - try { check_fence_or_throw(admitted_generation); } - catch (...) { throw CatalogFenceMovedMarker{}; } /// typed, not propagated -- completeCreation's own precedent + /// Typed, not propagated: the caller asked "did this land", and "the fence moved, so nothing + /// was sent" is an answer, not a failure of the operation. + if (!op.admitted()) + throw CatalogFenceMovedMarker{}; const auto it = findEntry(cur, observed.ns); if (it == cur.entries.end() || *it != observed) @@ -634,7 +700,8 @@ CasRefCatalog::ReconcileCreatorOutcome CasRefCatalog::reconcileStaleCreator( try { - casUpdate(backend, layout, mutate); + casUpdateImpl(op, layout, identityPreserving(mutate), + [](const RefCatalog & c) { return encodeRefCatalog(c); }); } catch (const CatalogFenceMovedMarker &) { return ReconcileCreatorOutcome::FencedOut; } catch (const CatalogEntryMismatchMarker &) { return ReconcileCreatorOutcome::EntryChanged; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h index 6eca5f4c985f..40df568bef08 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h @@ -1,6 +1,6 @@ #pragma once #include -#include +#include #include #include #include @@ -13,32 +13,32 @@ namespace DB::Cas { /// The `cas/ref_catalog` object (spec INV-3) as seen from the pool side: reading the current -/// catalog, and the generic token-CAS retry primitive every lifecycle transition rides. This class +/// catalog, and the generic conditional-update primitive every lifecycle transition rides. This class /// builds ONLY that primitive -- the actual lifecycle steps (the three-conditional-write creation /// sequence, the removal terminal-record-then-entry-delete sequence) are later tasks' job, built ON /// TOP of `casUpdate`/`casAdmitEntry`. class CasRefCatalog { public: - /// The catalog snapshot as read from the backend: the decoded object plus the token an update - /// must present to `casPut`. Operational reads always return a token because the catalog is a - /// mandatory control object after pool bootstrap. + /// The catalog snapshot as read from the backend: the decoded object plus the incarnation an + /// update must present as its precondition. Operational reads always carry one because the catalog + /// is a mandatory control object after pool bootstrap. struct Snapshot { RefCatalog catalog; - std::optional token; + std::optional incarnation; CatalogLifeIndex life_index; }; /// Reads and decodes the mandatory current catalog. Absence is corruption, never an empty /// authority set: without the catalog, opaque life keys cannot prove ownership. - static Snapshot read(Backend & backend, const Layout & layout); + static Snapshot read(CasOperation & op, const Layout & layout); /// Materializes the explicit empty catalog for a prefix already proven new by /// `probePoolBootstrapResidual`. This is the only absence-tolerant catalog operation: no /// existing-pool caller can accidentally turn authoritative absence into an empty catalog. A /// concurrent bootstrap winner is accepted only after its object is read and decoded. - static Snapshot initializeEmptyForNewPool(Backend & backend, const Layout & layout); + static Snapshot initializeEmptyForNewPool(CasOperation & op, const Layout & layout); /// The catalog's life for `ns` if a `Live`/`Removing` entry names it, else `nullopt` -- ONE catalog /// read and, crucially, NO WRITE OF ANY KIND. This is the resolution a READ or a REMOVAL uses: it @@ -52,50 +52,43 @@ class CasRefCatalog /// `Creating` is excluded for the same reason `liveUniverse` excludes it: no publication can exist /// under an entry still being created, so there is nothing to resolve to and nothing to read. static std::optional lifeIfCataloged( - Backend & backend, const Layout & layout, const RootNamespace & ns); + CasOperation & op, const Layout & layout, const RootNamespace & ns); /// Every `Live`/`Removing` life the catalog currently names, from this call's own catalog `GET`. /// This helper is for independent readers such as `CasFsck`; a GC fold instead keeps the immutable /// post-LIST snapshot attached to its scan and reuses that exact cut throughout the round. /// `Creating` is excluded: spec §3, no publication can exist yet. - static std::vector liveUniverse(Backend & backend, const Layout & layout); + static std::vector liveUniverse(CasOperation & op, const Layout & layout); - /// The generic token-CAS retry loop shared by every catalog mutation, mirroring - /// `PoolMeta::admitOrValidate`'s loop: read the current snapshot, apply `mutate` to obtain the - /// CANDIDATE next catalog, `casPut` it against the mandatory object's observed token, and on - /// `Conflict` re-read and re-apply `mutate` to the - /// FRESH snapshot -- never re-encoding the stale candidate. `mutate` must return a canonically - /// ordered, grammar-valid candidate; `encodeRefCatalog` (called internally) enforces that. + /// The generic read-modify-write shared by every catalog mutation: `mutate` turns the durable + /// catalog into the CANDIDATE next one, which is encoded and written against the incarnation the + /// same call read. A refused precondition re-runs `mutate` against the FRESH body -- never + /// re-encoding the stale candidate. `mutate` must return a canonically ordered, grammar-valid + /// candidate; `encodeRefCatalog` (called internally) enforces that. /// - /// Bounded (the same live-lock brake `publishCkpt`/`allocateWriterEpoch` use on their own - /// contended token-CAS singletons): after 100 conflicting attempts it gives up and raises the - /// typed retryable error `throwCasWriteRetryLater`, naming the key and the attempt count, rather - /// than spinning forever against a pathologically busy catalog. + /// The engine's own policy is the bound: persistent conflict ends at the write deadline with the + /// typed retryable error `throwCasWriteRetryLater`, rather than spinning forever against a + /// pathologically busy catalog. /// - /// A re-read that finds the object genuinely ABSENT after it was previously observed present is - /// corruption, not a fresh bootstrap. The required `read` throws before another CAS attempt, so - /// no mutation can replace every other namespace with a one-update catalog. + /// A read that finds the object ABSENT is corruption, not a fresh bootstrap -- `mutate` is never + /// offered an absent catalog to build a replacement authority from, so no mutation can replace + /// every other namespace with a one-update catalog. /// /// This primitive runs NO admission check: Constraint 13 (removal is never refused) means /// whether a candidate must clear the additive predicate is the CALLER's decision, not this /// loop's. A caller mutating an entry's state without growing the catalog (a removal transition) /// uses this directly. /// - /// THE FENCE OBLIGATION (Task 3 carry-over from the Task 2 review): this loop has no fence - /// parameter and performs no fence check of its own -- `publishCkpt`'s "AFTER the read, BEFORE the - /// CAS, on every attempt" discipline has no equivalent built in here. The seam a fenced caller + /// THE FENCE OBLIGATION: this loop performs no fence check of its own. The seam a fenced caller /// MUST use is `mutate` itself: it runs, fresh, after EVERY read this loop performs (the very - /// first one and every one after a `Conflict`), immediately before the candidate it returns is - /// encoded and `casPut`. A caller that needs its own write fenced (any catalog mutation minted - /// under a mount incarnation -- which is every one Task 3 onward adds) MUST throw from inside - /// `mutate`, checking on EVERY invocation, not once before calling `casUpdate`: checking once - /// before the call fences against the read this loop is *about* to perform, not the one it just - /// did, and a `Conflict` retry performs an entirely new read `mutate` is never told about except - /// by being called again. `completeCreation`'s own `mutate` (below) is the first production - /// caller to ride this seam, and does so by wrapping its `check_fence_or_throw` call at the top of - /// its `mutate`, exactly where `publishCkpt` places the identical check. + /// first one and every one after a refused precondition), immediately before the candidate it + /// returns is encoded and written. A caller that needs its own write fenced (any catalog mutation + /// minted under a mount incarnation) MUST refuse from inside `mutate`, checking on EVERY + /// invocation, not once before calling `casUpdate`: checking once before the call fences against + /// the read this loop is *about* to perform, not the one it just did, and a retry performs an + /// entirely new read `mutate` is never told about except by being called again. static RefCatalog casUpdate( - Backend & backend, const Layout & layout, + CasOperation & op, const Layout & layout, const std::function & mutate); /// Admits exactly ONE new namespace into the catalog under INV-3's two-predicate gate, inserting @@ -108,7 +101,7 @@ class CasRefCatalog /// `encodeRefCatalog`'s own canonical-order/no-duplicate grammar check, inside /// `checkCatalogAdmission`. static RefCatalog casAdmitEntry( - Backend & backend, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry); + CasOperation & op, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry); enum class BeginRemovingOutcome : uint8_t { @@ -119,13 +112,12 @@ class CasRefCatalog }; /// Exact `Live -> Removing` transition. The immutable observed row is compared by full value on - /// every catalog retry, and the mount fence is checked after every fresh read and before its CAS. - /// A row already `Removing` under the same namespace/life resolves an ambiguous or concurrent - /// transition positively; no caller may change its recorded start round afterward. + /// every catalog retry, and `op`'s admission is checked after every fresh read and before its + /// write. A row already `Removing` under the same namespace/life resolves an ambiguous or + /// concurrent transition positively; no caller may change its recorded start round afterward. static BeginRemovingOutcome beginRemoving( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - uint64_t removal_started_round, uint64_t admitted_generation, - const std::function & check_fence_or_throw); + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + uint64_t removal_started_round); /// Outcome of the only fold-authorized catalog deletion. A refusal never writes the catalog. enum class CompletedRemovingDeleteOutcome : uint8_t @@ -136,23 +128,17 @@ class CasRefCatalog FencedOut, }; - /// Authority result for completed-removal erases. Only an explicit `Moved` is a fence outcome; - /// exceptions mean authority could not be evaluated and propagate to the caller. - enum class LeaderFenceStatus : uint8_t - { - Held, - Moved, - }; - struct CompletedRemovingDeleteResult { CompletedRemovingDeleteOutcome outcome; /// Present only when a mandatory fresh catalog read proves that the exact observed life is no /// longer cataloged, whether this actor's erase committed or another actor removed/replaced it. std::optional invalidated_life; - /// The complete mandatory resolution snapshot after an attempted erase. The GC drain feeds - /// this directly into its next deterministic selection; proof refusal performs no read and - /// leaves it absent. + /// The catalog cut this call ends on. After an attempted erase it is the mandatory resolution + /// read's snapshot, which the GC drain feeds directly into its next deterministic selection. + /// On `FencedOut` it is instead the cut the call was GIVEN: an operation whose admission is + /// gone cannot issue the resolution read, so the erase's own effect is left unreported. Proof + /// refusal performs no read at all and leaves this absent. std::optional catalog_snapshot; bool operator==(CompletedRemovingDeleteOutcome expected) const { return outcome == expected; } @@ -161,21 +147,27 @@ class CasRefCatalog /// Exact-CAS-deletes `observed` only when it is a complete `Removing` row and the authoritative /// adopted parent carries cleanup evidence, but no hold, in the row keyed by the same opaque life /// id. The whole parent seal is consumed so a caller cannot separate the life id from its proof or - /// reduce the proof to a caller-computed boolean. The leader fence is checked after every fresh - /// catalog read and before every attempted CAS. + /// reduce the proof to a caller-computed boolean. `op`'s admission is checked after every fresh + /// catalog read and before every attempted erase. + /// + /// `refresh_authority` runs at the top of every attempt, before the checks that decide whether to + /// send one. It is not a verdict -- the verdict stays `op.admitted()` -- but that liveness may be + /// a cached flag its holder refreshes from a fact this loop cannot see, and one reading taken + /// before the first erase must not authorise the rest. Mandatory: a caller whose liveness needs no + /// refresh passes a no-op and says so, instead of erasing unrefreshed because the argument was + /// easy to omit. static CompletedRemovingDeleteResult deleteCompletedRemoving( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - const CasFoldSeal & authoritative_parent, uint64_t admitted_generation, - const std::function & check_fence); + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + const CasFoldSeal & authoritative_parent, + const std::function & refresh_authority); - /// Same exact deletion, using the caller's complete selected catalog snapshot and token for the - /// one CAS attempt. Its mandatory resolution snapshot is returned in the result so a catalog-only - /// drain can select the next row without an intervening read. + /// Same exact deletion, using the caller's complete selected catalog snapshot and its incarnation + /// for the one erase attempt. Its mandatory resolution snapshot is returned in the result so a + /// catalog-only drain can select the next row without an intervening read. static CompletedRemovingDeleteResult deleteCompletedRemovingAtSnapshot( - Backend & backend, const Layout & layout, Snapshot catalog_snapshot, + CasOperation & op, const Layout & layout, Snapshot catalog_snapshot, const CatalogEntry & observed, const CasFoldSeal & authoritative_parent, - uint64_t admitted_generation, - const std::function & check_fence); + const std::function & refresh_authority); /// Outcome of exact stalled-creation cancellation, the only other exported deletion shape. enum class StalledCreatingCancelOutcome : uint8_t @@ -189,9 +181,8 @@ class CasRefCatalog /// Exact-CAS-deletes one observed `Creating` row only after its complete creator fence is proven /// terminal. This performs no `_ckpt` or other physical cleanup; debris belongs to the janitor. static StalledCreatingCancelOutcome cancelStalledCreating( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - const std::function & is_creator_fence_terminal, - uint64_t admitted_generation, const std::function & check_fence_or_throw); + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + const std::function & is_creator_fence_terminal); /// === Task 3: the §3 creation lifecycle, built on the two primitives above === @@ -203,9 +194,11 @@ class CasRefCatalog { Live, /// the entry reached `Live`; `_ckpt` is durable with this creator's `writer_epoch` /// as `life_epoch` (spec INV-4: the genesis epoch, recorded nowhere else). - FencedOut, /// this caller's OWN admitted generation moved before the `_ckpt` publish or the - /// `Creating -> Live` CAS. Nothing more was written; the caller's own mount - /// incarnation is gone, so it cannot be the one to retry. + FencedOut, /// this caller's OWN admission was lost, at the `_ckpt` publish or at the + /// `Creating -> Live` write. Whether the step it was running landed is + /// UNRESOLVED -- admission is reported lost both before an attempt is sent and + /// after one is proven durable, so a resumer re-reads rather than assumes. The + /// caller's own mount incarnation is gone, so it cannot be the one to retry. Superseded, /// the catalog entry no longer equals what this caller observed -- a concurrent /// reconciler stole it, or a race already carried it to `Live`/`Removing`. Nothing /// was written; a DIFFERENT actor now owns whatever happens to this namespace next. @@ -226,13 +219,13 @@ class CasRefCatalog EntryChanged, /// the catalog's current entry for `observed.ns` no longer equals /// `observed` -- token-exactness failed. Not written; the caller must /// re-read the catalog before trying again. - FencedOut, /// review I6: this caller's OWN admitted generation moved before the CAS - /// -- nothing was written, and the caller's own mount incarnation is gone, - /// so it cannot be the one to retry. Mirrors `NamespaceCreationOutcome:: - /// FencedOut`; without this check a deposed mount could still steal a - /// `Creating` entry onto its own dead fence before the following - /// `completeCreation` refuses it -- the catalog would be mutated by an - /// actor this subsystem otherwise never lets touch it. + FencedOut, /// this caller's OWN admission was lost, and whether its write landed is + /// unresolved; its mount incarnation is gone either way, so it cannot be + /// the one to retry. Mirrors `NamespaceCreationOutcome::FencedOut`; + /// without this check a deposed mount could still steal a `Creating` + /// entry onto its own dead fence before the following `completeCreation` + /// refuses it -- the catalog would be mutated by an actor this subsystem + /// otherwise never lets touch it. }; /// The full, fresh §3 sequence for a namespace that carries NO catalog entry yet: mints a random @@ -250,10 +243,8 @@ class CasRefCatalog /// `Live`/`Removing` IS a caller bug (recreating an existing name is removal's business, not /// creation's) and still throws `LOGICAL_ERROR` naming the observed state. static NamespaceCreationOutcome createNamespace( - Backend & backend, const Layout & layout, uint64_t gc_shards, - const RootNamespace & ns, const CreatorFence & creator, - uint64_t admitted_generation, const std::function & check_fence_or_throw, - const CkptDeadline & deadline); + CasOperation & op, const Layout & layout, uint64_t gc_shards, + const RootNamespace & ns, const CreatorFence & creator); /// Fires once, synchronously, right after `createNamespace`'s own pre-check read observed no /// entry and right before its step 1 performs its own (first) catalog read -- the exact window a @@ -275,9 +266,8 @@ class CasRefCatalog /// this module's own bug, not a race. A `FencedOut` from `publishCkpt` ends the attempt here. /// /// Step 3: `CasRefCatalog::casUpdate`'s `mutate` is the fence re-check point (see the class-level - /// note below) -- `check_fence_or_throw(admitted_generation)` runs FIRST, on every fresh read this - /// retry loop performs, exactly like `publishCkpt`'s own re-check; a throw from it is caught and - /// reported as `FencedOut`, nothing else. ONLY THEN is the fresh entry for `observed.ns` compared + /// note below) -- `op.admitted()` is consulted FIRST, on every fresh read this retry loop performs; + /// a refusal is reported as `FencedOut`, nothing else. ONLY THEN is the fresh entry for `observed.ns` compared /// against `observed` by FULL VALUE equality (`CatalogEntry::operator==`) -- the value-CAS that /// plays the role `publishCkpt`'s object token plays for `_ckpt`, since one catalog object holds /// every namespace's entry and there is no separate per-entry token to CAS against. A mismatch @@ -288,9 +278,7 @@ class CasRefCatalog /// `publishCkpt`); a caller that manages to make BOTH stale sees `FencedOut`, not `Superseded` -- /// both are truthful refusals of a CAS that was never sent. static NamespaceCreationOutcome completeCreation( - Backend & backend, const Layout & layout, const CatalogEntry & observed, - uint64_t admitted_generation, const std::function & check_fence_or_throw, - const CkptDeadline & deadline); + CasOperation & op, const Layout & layout, const CatalogEntry & observed); /// Stale-`Creating` reconciliation (spec INV-3: "stalled creators occupy entries until /// fence-terminal reconciliation"; TLA Task 3 obligation 1: "the call-site is where @@ -315,21 +303,20 @@ class CasRefCatalog /// (token-exactness: a concurrent reconciler, or the original creator finishing on its own, /// invalidates this immediately). /// On success, CASes `creator` to `new_creator` -- `state` and `incarnation` are UNCHANGED, so the - /// caller resumes with `completeCreation(backend, layout, {..., .creator = new_creator}, ...)` over - /// the SAME incarnation, never a fresh one (rebirth under a fresh incarnation is Task 5/removal's - /// business, not a live reconciliation's). + /// caller resumes with `completeCreation(op, layout, {..., .creator = new_creator})` over the SAME + /// incarnation, never a fresh one (rebirth under a fresh incarnation is Task 5/removal's business, + /// not a live reconciliation's). /// - /// `admitted_generation`/`check_fence_or_throw` (review I6): re-checked FIRST on every fresh read - /// this CAS retries, exactly like `completeCreation`'s own placement -- a caller whose OWN mount - /// fence has already moved must not be the one to steal a `Creating` entry onto its own (now dead) - /// fence, even though the following `completeCreation` would go on to refuse it as `FencedOut` - /// anyway: by then the catalog would already have been mutated by a deposed actor, the one posture - /// this subsystem otherwise refuses everywhere else. + /// `op.admitted()` is consulted FIRST on every fresh read this retries, exactly like + /// `completeCreation`'s own placement -- a caller whose OWN mount fence has already moved must not + /// be the one to steal a `Creating` entry onto its own (now dead) fence, even though the following + /// `completeCreation` would go on to refuse it as `FencedOut` anyway: by then the catalog would + /// already have been mutated by a deposed actor, the one posture this subsystem otherwise refuses + /// everywhere else. static ReconcileCreatorOutcome reconcileStaleCreator( - Backend & backend, const Layout & layout, const CatalogEntry & observed, + CasOperation & op, const Layout & layout, const CatalogEntry & observed, const CreatorFence & new_creator, - const std::function & is_creator_fence_terminal, - uint64_t admitted_generation, const std::function & check_fence_or_throw); + const std::function & is_creator_fence_terminal); /// Spec §3: "`Creating` forbids publication -- no ref writes admitted while the entry is /// Creating." Throws `throwCasWriteRetryLater`'s class (transient: `Creating` resolves once the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp index a29b1d5faafe..28923ce51b94 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp @@ -1,8 +1,10 @@ #include #include #include +#include #include #include +#include namespace DB { @@ -19,10 +21,6 @@ namespace DB::Cas namespace { -/// Live-lock brake, the same shape and for the same reason as `CasPlainObjects`': the deadline is the -/// real bound, and this only stops an unexpected continuous conflict from spinning until it elapses. -constexpr size_t MAX_CKPT_CAS_ATTEMPTS = 100; - /// The per-field semantic maximum for an OPTIONAL field: a present value beats an absent one (an /// absence is "this writer knew nothing", never "this writer says none"), and two present values /// resolve by the field's own order -- for `RefTxnId` that is writer_epoch then ref_sequence, the @@ -184,54 +182,41 @@ RecoveryGrounding chooseRecoveryGrounding(const std::optional & ca return result; } -std::optional readCkpt(Backend & backend, const Layout & layout, const NamespaceLifeId & life) +std::optional readCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life) { - std::optional got = backend.get(layout.refCkptKey(life)); + const std::optional got = op.read(layout.refCkptKey(life), Retry::standard()); if (!got) return std::nullopt; /// Materialized read, then decode: the object is MUTABLE, so the body must be fixed before it is - /// parsed, and the token must be the one that labels exactly these bytes. - return CkptSample{decodeRefCkpt(got->bytes), got->token}; + /// parsed, and the incarnation must be the one that labels exactly these bytes. + return CkptSample{decodeRefCkpt(got->bytes), got->incarnation}; } -CkptPublishOutcome publishCkpt(Backend & backend, const Layout & layout, const NamespaceLifeId & life, - const RefCkpt & contribution, uint64_t admitted_generation, - const std::function & check_fence_or_throw, - const CkptDeadline & deadline, - const std::function & admit_request) +CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life, + const RefCkpt & contribution) { const String key = layout.refCkptKey(life); - std::optional current; - bool have_current = false; - const auto request_is_admitted = [&] + + /// Why `decide` had nothing to write. Both answers are declines, and only the fence tells them + /// apart, so the reason is recorded where it is decided instead of re-derived from the outcome. + enum class Decline : uint8_t { - try - { - if (admit_request) - admit_request(); - return true; - } - catch (...) - { - return false; - } + Identical, + Fenced, }; + std::optional decline; + /// A decline on any decision AFTER the first can only follow a refused attempt of this call, and an + /// attempt is only ever refused after it was sent. That distinction is what keeps `IdenticalSkip`'s + /// promise -- no write was issued -- true wherever it is reported. + size_t decisions = 0; - for (size_t attempt = 0; attempt < MAX_CKPT_CAS_ATTEMPTS; ++attempt) + const auto decide = [&](const std::optional & current) -> std::optional { - if (deadline.now_ms() >= deadline.deadline_ms) - break; - - /// Read the WHOLE body every attempt. A retry after a conflict must merge against what is - /// there NOW: reusing the previous attempt's reading is precisely the read-modify-write with - /// the merge left out, one round later. - if (!have_current) - { - if (!request_is_admitted()) - return CkptPublishOutcome::FencedOut; - current = readCkpt(backend, layout, life); - have_current = true; - } + ++decisions; + decline.reset(); + std::optional durable; + if (current) + durable = decodeRefCkpt(current->bytes); /// The one rule the commutative merge cannot state, and it has to be decided HERE, before the /// merge: the semantic maximum turns a decrease into a body identical to the stored one, which @@ -243,125 +228,66 @@ CkptPublishOutcome publishCkpt(Backend & backend, const Layout & layout, const N /// transient control signal every other refusal in this function returns rather than throws. A /// writer that is still admitted and yet contributing a superseded epoch is the fence violation /// this detects, and that one is corruption. - if (current && lifeEpochWouldDecrease(current->ckpt, contribution)) + if (durable && lifeEpochWouldDecrease(*durable, contribution)) { - try - { - check_fence_or_throw(admitted_generation); - } - catch (...) + if (!op.admitted()) { - return CkptPublishOutcome::FencedOut; + decline = Decline::Fenced; + return std::nullopt; } - throwLifeEpochDecrease(current->ckpt, contribution, key); + throwLifeEpochDecrease(*durable, contribution, key); } /// ANY writer may create the object; none of them may invent a field. An absent `_ckpt` is /// created from the contribution as it stands, so a publisher that knows only the checkpoint /// creates one that knows only the checkpoint, and the birth transaction's `life_epoch` merges /// into it whenever it arrives -- in either order, because the merge is a per-field maximum. - const RefCkpt merged = current ? mergeCkpt(current->ckpt, contribution) : contribution; + const RefCkpt merged = durable ? mergeCkpt(*durable, contribution) : contribution; - /// Nothing new: return WITHOUT a CAS. This is not an optimization -- both writers publish on - /// every snapshot and every seal, and most of those carry a checkpoint the object already has, - /// so issuing the write anyway would mint a fresh token per no-op and turn every other writer's - /// in-flight CAS into a conflict, for a body byte-identical to the one already stored. - if (current && merged == current->ckpt) + /// Nothing new: write NOTHING. This is not an optimization -- both writers publish on every + /// snapshot and every seal, and most of those carry a checkpoint the object already has, so + /// issuing the write anyway would mint a fresh incarnation per no-op and turn every other + /// writer's in-flight write into a conflict, for a body byte-identical to the one stored. + if (durable && merged == *durable) { - try - { - check_fence_or_throw(admitted_generation); - } - catch (...) - { - return CkptPublishOutcome::FencedOut; - } - return CkptPublishOutcome::IdenticalSkip; + decline = op.admitted() ? Decline::Identical : Decline::Fenced; + return std::nullopt; } + return encodeRefCkpt(merged); + }; - /// AFTER the read, BEFORE the CAS, on EVERY attempt (spec §3). A generation that moved since - /// admission means this writer's lease incarnation is gone and the body it just merged is - /// stale, so the CAS must never be sent -- and because the check precedes it, nothing was. - try - { - check_fence_or_throw(admitted_generation); - } - catch (...) - { - /// Typed, not propagated: the caller asked "did this land", and "the fence moved, so - /// nothing was sent" is an answer, not a failure of the operation. Only the fence check is - /// wrapped, so nothing else can be mistaken for it. + WriteResult result = op.readModifyWrite(key, decide, Retry::standard()); + if (std::holds_alternative(result)) + return CkptPublishOutcome::Published; + if (std::holds_alternative(result)) + { + if (decline == Decline::Fenced) return CkptPublishOutcome::FencedOut; - } - - const std::optional expected = - current ? std::optional{current->token} : std::nullopt; - /// Encode before entering the ambiguity catch. Allocation or invariant failures happen before - /// any request is sent and must propagate as themselves, not trigger a needless resolution GET. - const String merged_bytes = encodeRefCkpt(merged); - if (!request_is_admitted()) + /// The durable body already carries this contribution and an attempt of this call was sent to + /// get there, so the write is resolved rather than skipped. + return decisions > 1 ? CkptPublishOutcome::Published : CkptPublishOutcome::IdenticalSkip; + } + if (const auto * gave_up = std::get_if(&result)) + { + /// A lost fence is an expected, transient control signal, so it is a value rather than a + /// throw. It says nothing about the object: the engine reports it both before an attempt is + /// sent and after one is proven durable, so the caller must re-read rather than assume. + /// Every other give-up leaves the contribution unpublished, and the caller must be told that + /// rather than left to assume it landed. + if (gave_up->why == GaveUp::Why::FenceLost) return CkptPublishOutcome::FencedOut; - try - { - if (backend.casPut(key, merged_bytes, expected).outcome == CasOutcome::Committed) - return CkptPublishOutcome::Published; - } - catch (...) - { - /// A thrown CAS response does not say whether the object changed. Never retry its bytes - /// from memory: first point-read the exact mutable object, including its fresh token. If - /// that observation includes this contribution under the semantic join, the write is - /// resolved durable; otherwise that exact observation is the only valid base for a retry. - if (!request_is_admitted()) - return CkptPublishOutcome::FencedOut; - try - { - current = readCkpt(backend, layout, life); - have_current = true; - } - catch (...) - { - throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string() - + "': a CAS response was ambiguous and the mandatory exact-read resolution failed (" - + getCurrentExceptionMessage(/*with_stacktrace*/ false) + ")"); - } - - try - { - check_fence_or_throw(admitted_generation); - } - catch (...) - { - return CkptPublishOutcome::FencedOut; - } - - if (current && lifeEpochWouldDecrease(current->ckpt, contribution)) - throwLifeEpochDecrease(current->ckpt, contribution, key); - - const RefCkpt resolved_merge = current ? mergeCkpt(current->ckpt, contribution) : contribution; - if (current && resolved_merge == current->ckpt) - return CkptPublishOutcome::Published; - - /// `current` is the exact observation made after the ambiguous response. The next loop - /// iteration retries the SAME contribution against its token (or expected absence), with - /// no blind CAS and no redundant intervening GET. - continue; - } - /// `Conflict`: the incarnation we read is no longer current, so another writer's merge landed - /// first. Nothing of ours was written; re-read and merge against the winner. - current.reset(); - have_current = false; + throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string() + + "': persistent CAS contention, the checkpoint contribution was not published"); } - - /// Fail closed. Every attempt was all-or-nothing, so there is no partial state -- only an - /// unpublished contribution, which the caller must be told about rather than left to assume. - throwCasWriteRetryLater("CAS _ckpt for namespace '" + life.ns.string() - + "': persistent CAS contention, the checkpoint contribution was not published"); + /// Only `Conflict` and `Refused` are left, and both are an exception. + const String what = "CAS _ckpt for namespace '" + life.ns.string() + "'"; + orThrow(std::move(result), what); + UNREACHABLE(); } -MissingBaseVerdict classifyMissingSampledBase(const Token & sampled_token, const std::optional & current_token) +MissingBaseVerdict classifyMissingSampledBase(const Incarnation & sampled, const std::optional & current) { - if (current_token && !(*current_token == sampled_token)) + if (current && !(*current == sampled)) return MissingBaseVerdict::RestartRecovery; return MissingBaseVerdict::Corrupted; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h index 40b3771611f8..5730e9c98ccb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h @@ -1,13 +1,10 @@ #pragma once -#include +#include #include #include #include -#include #include -#include #include -#include namespace DB::Cas { @@ -46,96 +43,66 @@ RecoveryGrounding chooseRecoveryGrounding(const std::optional & ca /// Compatible contributions still merge commutatively, but the committed frontier is deliberately not /// an unconstrained CRDT maximum: a cross-epoch pair must be numerically adjacent and carry its seal /// evidence. That makes arbitrary regrouping of a corrupt historical set invalid, while the actual -/// publish protocol remains simple: each token-CAS merges one contribution with the one durable body it +/// publish protocol remains simple: each write merges one contribution with the one durable body it /// just read. /// /// It is therefore NOT where `life_epoch`'s may-not-decrease rule lives, and that is a placement /// decision rather than an omission: a commutative function does not know which of its arguments is the /// durable one, so it cannot tell a decrease from an increase. That rule belongs to `publishCkpt`, which -/// does know (see `checkLifeEpochDoesNotDecrease` in the `.cpp`). +/// does know. RefCkpt mergeCkpt(const RefCkpt & a, const RefCkpt & b); /// What one `publishCkpt` call did. enum class CkptPublishOutcome : uint8_t { - Published, /// the merged body is durable -- this call's CAS committed it + Published, /// the contribution is durable, and this call sent at least one write before it + /// was; which write made it durable -- this one or a competitor's -- is not claimed IdenticalSkip, /// the contribution added nothing to what was already there; NO write was issued - FencedOut, /// the admitted fence generation moved before the CAS; NOTHING was written + FencedOut, /// this actor's admission is gone. Whether its last attempt landed is UNRESOLVED: + /// admission can be lost before the write is sent, and equally after the write is + /// proven durable. Re-read before assuming either way }; -/// The retry bound for `publishCkpt`: an absolute point on a monotonic millisecond clock, plus that -/// clock. Both are required and must be the SAME clock -- the caller passes its own injectable boot -/// clock (`CasRefLedger`'s `boot_ms_fn`), so a test drives the exhaustion arm deterministically -/// instead of sleeping, and a VM suspend cannot shorten the window. -struct CkptDeadline -{ - std::function now_ms; - uint64_t deadline_ms = 0; -}; - -/// Merge `contribution` into `ns`'s `_ckpt` and make the result durable. -/// -/// One attempt is: GET the object -> decode it -> merge -> (identical? return without a CAS) -> -/// re-check the fence -> token-CAS. A CAS conflict means another writer's read-modify-write landed -/// between our GET and our CAS, so the whole attempt repeats against the NEW body -- never against the -/// one we already read, which is the point of re-reading rather than retrying the same bytes. -/// A THROWN CAS response is ambiguous rather than a conflict: exact-read the object, validate its body -/// and token, then check admission again. If the durable body semantically includes the contribution, -/// the write is resolved; otherwise retry the same contribution against that exact-read token. An -/// unreadable resolution fails retry-later, and no path issues two CAS attempts without an intervening -/// exact observation. +/// Merge `contribution` into `life`'s `_ckpt` and make the result durable, as ONE read-modify-write +/// on `op`. /// /// An ABSENT object is created from `contribution` as it stands. Every writer may create it and none /// may complete it: a publisher that knows only the checkpoint creates one that knows only the /// checkpoint, and the field a different writer knows merges in whenever it arrives, in either order. /// That is the whole reason each field is optional rather than defaulted. /// -/// FENCE DISCIPLINE (spec §3, the same value at every site of the trio): `check_fence_or_throw` is -/// re-run on EVERY attempt, AFTER that attempt's read and immediately BEFORE its CAS -- not once at -/// entry. A generation that moved means the mount lease incarnation changed since this work was -/// admitted, so this writer's body is stale even if the fence happens to be live again; the CAS must -/// not be sent. That refusal is returned as `FencedOut` rather than thrown: it is an expected, -/// transient control signal (the same class the request controller reports as `Unresolved`), and the -/// snapshot publisher that calls this sits after a durable PUT where an exception would be worse than -/// a value. NOTHING has been written when it is returned -- the check precedes the CAS. +/// Both DECLINE-TIME verdicts consult `op.admitted()` before they speak, because a writer the fence is +/// about to refuse has landed nothing AT THAT POINT: it is told `FencedOut` rather than `IdenticalSkip` +/// or a corruption verdict. `FencedOut` is returned rather than thrown because it is an expected, +/// transient control signal, and the snapshot publisher that calls this sits after a durable PUT where +/// an exception would be worse than a value. It does NOT promise the object is unchanged -- a lost +/// admission is also reported for a write already proven durable. /// /// FAILS CLOSED, never open: /// - an existing `_ckpt` that does not decode PROPAGATES `CORRUPTED_DATA` and is never overwritten. /// It is the only record of recovery's base and of what cleanup may delete; replacing it with a /// body derived from `contribution` alone would erase the base while leaving a well-formed object /// behind -- corruption laundered into something a reader would trust. -/// - a contribution whose `life_epoch` is BELOW the durable one raises `CORRUPTED_DATA`, checked after -/// that attempt's read and before its merge, so no body is built and no CAS is sent. This is the one -/// refusal that HAS to live here rather than in `mergeCkpt`: only this function knows which side is -/// durable. It is reported as corruption ONLY for a writer the fence still admits -- one the fence -/// is about to refuse gets `FencedOut` like every other refusal here, since it landed nothing. -/// - exhausting the deadline (or the live-lock brake) under persistent conflict throws the -/// retry-later class. No partial state exists to clean up: every attempt either committed the -/// complete merged body or changed nothing. -/// -/// `admitted_generation` is the fence generation the CALLER captured when its work was admitted, and -/// `check_fence_or_throw` is the callback the pool wires from `CasMountRuntime::checkFenceOrThrow` -/// (the ledger never owns a `CasMountRuntime`; it receives the pair the way `CasPlainObjects` does). -/// `admit_request` is independent of that post-read fence contract: when supplied, it is checked -/// immediately before every raw backend request, and refusal returns `FencedOut` without starting it. -CkptPublishOutcome publishCkpt(Backend & backend, const Layout & layout, const NamespaceLifeId & life, - const RefCkpt & contribution, uint64_t admitted_generation, - const std::function & check_fence_or_throw, - const CkptDeadline & deadline, - const std::function & admit_request = {}); +/// - a contribution whose `life_epoch` is BELOW the durable one raises `CORRUPTED_DATA`, decided +/// before the merge, so no body is built and no write is sent. This is the one refusal that HAS to +/// live here rather than in `mergeCkpt`: only this function knows which side is durable. +/// - exhausting the policy under persistent conflict throws the retry-later class. No partial state +/// exists to clean up: every attempt either committed the complete merged body or changed nothing. +CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life, + const RefCkpt & contribution); -/// One observation of a namespace's `_ckpt`: the decoded body and the incarnation TOKEN it was read -/// at. The token is what the missing-base revalidation adjudicates against, so a reader that keeps -/// only the body cannot apply the rule. +/// One observation of a namespace's `_ckpt`: the decoded body and the incarnation it was read at. The +/// incarnation is what the missing-base revalidation adjudicates against, so a reader that keeps only +/// the body cannot apply the rule. struct CkptSample { RefCkpt ckpt; - Token token; + Incarnation incarnation; }; /// Point-read of `life`'s `_ckpt`. `nullopt` means the object is absent (a namespace whose creation has /// not published one yet); a present-but-undecodable object throws `CORRUPTED_DATA`. -std::optional readCkpt(Backend & backend, const Layout & layout, const NamespaceLifeId & life); +std::optional readCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life); /// The verdict of INV-4's three-way revalidation, for the one leg that is not simply "it is there". enum class MissingBaseVerdict : uint8_t @@ -145,14 +112,14 @@ enum class MissingBaseVerdict : uint8_t }; /// Adjudicate a sampled recovery anchor that turned out to be unavailable, by comparing the `_ckpt` -/// token this recovery sampled against the token a fresh re-read observes. The anchor is the +/// incarnation this recovery sampled against the one a fresh re-read observes. The anchor is the /// checkpoint-named snapshot and its retained same-id non-seal log witness; the caller supplies this -/// verdict after either exact GET is absent. +/// verdict after either exact read is absent. /// -/// - token ADVANCED -> `RestartRecovery`. Cleanup legitimately advanced the checkpoint and deleted -/// the previous anchor while we were reading. Nothing is wrong; restart from the newer base -/// (bounded by the caller's own restart budget). -/// - token UNCHANGED -> `Corrupted`. The checkpoint still names an object that is not there, and +/// - incarnation ADVANCED -> `RestartRecovery`. Cleanup legitimately advanced the checkpoint and +/// deleted the previous anchor while we were reading. Nothing is wrong; restart from the newer +/// base (bounded by the caller's own restart budget). +/// - incarnation UNCHANGED -> `Corrupted`. The checkpoint still names an object that is not there, and /// the deletion gate makes that unreachable in an honest run: the named snapshot and matching log /// are both retained. Something deleted a live anchor. /// - `_ckpt` itself ABSENT on the re-read -> `Corrupted` for the same reason, and more bluntly: the @@ -161,7 +128,7 @@ enum class MissingBaseVerdict : uint8_t /// /// Pure, so it is decided the same way at every call site; the caller raises `CORRUPTED_DATA` on /// `Corrupted` with its own context. -MissingBaseVerdict classifyMissingSampledBase(const Token & sampled_token, const std::optional & current_token); +MissingBaseVerdict classifyMissingSampledBase(const Incarnation & sampled, const std::optional & current); /// INV-4's snapshot-deletion gate: a snapshot is deletable only STRICTLY BELOW the checkpoint. Strict /// rather than at-or-below because the checkpoint names the snapshot a recovery is entitled to fetch diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp index 318978c9b54e..97e8e03c6577 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp @@ -97,13 +97,11 @@ const LoggerPtr & confirmLogger() } /// Classifies whether an exception thrown out of a ref-table recovery attempt (checkpoint/snapshot/log -/// GETs, or the seal PUT) is a TRANSIENT object-store transport failure worth retrying, -/// vs. a terminal condition (corruption, decode failure, logic error, resource limit) that must fail -/// fast. The recovery reads call the backend directly (not through `ref_request_controller`), so a -/// transient blip surfaces as the object storage's native code -- `S3_ERROR` for the S3 backend, or a -/// socket/timeout/Poco transport code -- NOT the `NETWORK_ERROR` that only the seal PUT's controller -/// re-mints. Retrying only `NETWORK_ERROR` would leave the LIST/GET legs unprotected, which is exactly -/// the exact-read path the recovery retry boundary protects. +/// reads, or the seal write) is a TRANSIENT object-store transport failure worth retrying, vs. a +/// terminal condition (corruption, decode failure, logic error, resource limit) that must fail fast. +/// A read that exhausts its policy surfaces as `NETWORK_ERROR`, but a store's own transport failure +/// can also reach here unclassified -- `S3_ERROR` for the S3 backend, or a socket/timeout/Poco code -- +/// so the set covers both rather than only the one the engine re-mints. bool isTransientRecoveryError(int code) { return code == ErrorCodes::NETWORK_ERROR @@ -198,17 +196,15 @@ Occupant classifyRefLogOccupant(const RootNamespace & ns, const RefTxnId & id, c } CasRefLedger::CasRefLedger( - BackendPtr backend_ptr, + CasRequests & mount_requests_, const Layout & layout_, RefLedgerConfig config_, const CasEventSink & event_sink_, CasRequestBudget cas_request_budget_, String server_root_id_, - std::function controller_boot_ms_fn, std::function live_epoch_fn_, std::function fence_ok_fn_, std::function fence_generation_fn_, - std::function check_fence_or_throw_, std::function boot_ms_now_fn_, std::function may_mutate_, std::function &)> on_impossible_interference_, @@ -216,7 +212,7 @@ CasRefLedger::CasRefLedger( std::function publish_error_hook_, std::function cancel_inflight_builds_, std::function recovery_pre_first_request_hook_for_test_) - : backend(*backend_ptr) + : mount_requests(mount_requests_) , layout(layout_) , config(std::move(config_)) , event_sink(event_sink_) @@ -225,7 +221,6 @@ CasRefLedger::CasRefLedger( , live_epoch_fn(std::move(live_epoch_fn_)) , fence_ok_fn(std::move(fence_ok_fn_)) , fence_generation_fn(std::move(fence_generation_fn_)) - , check_fence_or_throw(std::move(check_fence_or_throw_)) , boot_ms_now_fn(std::move(boot_ms_now_fn_)) , may_mutate(std::move(may_mutate_)) , on_impossible_interference(std::move(on_impossible_interference_)) @@ -234,18 +229,11 @@ CasRefLedger::CasRefLedger( , cancel_inflight_builds(std::move(cancel_inflight_builds_)) , recovery_pre_first_request_hook_for_test(std::move(recovery_pre_first_request_hook_for_test_)) { - /// The ref-log writer path uses the same retry controller and clock seam as the mount's local - /// write fence, so deadline-sensitive tests exercise both paths with one monotonic clock. - /// The raw mount `boot_ms_fn` -- the SAME fake-clock seam the local write fence uses -- is reused - /// here rather than adding a second clock knob; both are monotonic-ms clocks and tests that need - /// deterministic deadline behavior already inject it. - ref_request_controller = std::make_unique(backend_ptr, cas_request_budget, controller_boot_ms_fn); - /// Default backoff sleep for the recovery retry loop (`ensureRefTableRecovered`): sleep in short /// slices and stop early if the mount fence drops (shutdown / lease loss), so teardown never waits /// out a full 30s backoff. This is deliberate, bounded backoff against external object-store I/O - /// failure -- NOT masking a race -- exactly like `CasRequestControl`'s own inter-attempt - /// `threadSleepMs`; the slice loop additionally makes it interruptible, which that one is not. + /// failure -- NOT masking a race -- exactly like the request engine's own inter-attempt sleep; the + /// slice loop additionally makes it interruptible, which that one is not. recovery_retry_sleep_fn = [this](uint64_t total_ms, const std::optional & token) { constexpr uint64_t slice_ms = 200; @@ -259,28 +247,40 @@ CasRefLedger::CasRefLedger( }; } -CasWriteOutcome CasRefLedger::stagingPutIfAbsent(std::string_view key, std::string_view bytes, Token * out_token) +WriteResult CasRefLedger::stagingPutIfAbsent(const String & key, const String & bytes) { - /// The ref lane's mount predicate (`fence_ok_fn` == `Pool::refAppendFenceOk`, with no per-table - /// runtime term) gates every attempt, matching the other staged writes. - return ref_request_controller->putIfAbsentControlled(key, bytes, fence_ok_fn, out_token); + /// Admitted under the mount fence's CURRENT generation: a staged write belongs to the caller in + /// front of it, not to a transaction admitted earlier, so there is nothing to resume under. + CasOperation op = mount_requests.admit(); + return op.create(key, bytes, Retry::standard()); } -CasOverwriteResult CasRefLedger::stagingConditionalOverwrite(std::string_view key, std::string_view bytes, const Token & expected) +WriteResult CasRefLedger::stagingConditionalOverwrite(const String & key, const String & bytes, + const Incarnation & expected) { - /// The supplied write is controlled by the same retry and mount-fence policy as other staged - /// writes. - return ref_request_controller->putOverwriteControlled(key, bytes, expected, fence_ok_fn); + CasOperation op = mount_requests.admit(); + return op.replace(key, bytes, expected, Retry::standard()); } -CasOverwriteResult CasRefLedger::stagingPutIfAbsentMutable(std::string_view key, std::string_view bytes) +WriteResult CasRefLedger::stagingPutIfAbsentMutable(const String & key, const String & bytes) { - return ref_request_controller->putIfAbsentControlledMutable(key, bytes, fence_ok_fn); + CasOperation op = mount_requests.admit(); + return op.create(key, bytes, Retry::standard()); +} + +void CasRefLedger::refuseUnlessAdmitted(const CasOperation & op, std::string_view what) const +{ + if (op.admitted()) + return; + throwCasTransientUnavailable( + fmt::format("content-addressed pool '{}'", server_root_id), + fmt::format("{}: the operation is no longer admitted -- the mount lease has too little time left " + "for another request, or this table's runtime was detached", what)); } void CasRefLedger::setCasRetrySleepForTest(std::function sleep_fn) { - ref_request_controller->setSleepFnForTest(sleep_fn); + mount_requests.setSleepFnForTest(sleep_fn); recovery_retry_sleep_fn = [recovery_sleep_fn = std::move(sleep_fn)]( uint64_t total_ms, const std::optional &) { @@ -581,7 +581,7 @@ std::shared_ptr CasRefLedger::lookupRefTableRunti std::shared_ptr CasRefLedger::acquireRefTableRuntime( const NamespaceLifeId & life, uint64_t admitted_generation) { - check_fence_or_throw(admitted_generation); + refuseUnlessAdmitted(mount_requests.resume(admitted_generation), "ref-table runtime install"); std::shared_ptr result; bool generation_moved = false; @@ -615,7 +615,7 @@ std::shared_ptr CasRefLedger::acquireRefTableRunt } if (generation_moved) - check_fence_or_throw(admitted_generation); + refuseUnlessAdmitted(mount_requests.resume(admitted_generation), "ref-table runtime install"); if (identity_conflict) throwCasWriteRetryLater(fmt::format( "CAS namespace '{}': the cached runtime identity changed while publishing catalog life {}; " @@ -634,7 +634,7 @@ std::shared_ptr CasRefLedger::acquireReadableRefT /// runtime. if (auto current = lookupRefTableRuntime(ns)) { - check_fence_or_throw(current->admitted_fence_generation); + refuseUnlessAdmitted(mount_requests.resume(current->admitted_fence_generation), "resident readable runtime"); { std::lock_guard queue_lock(ref_queue_mutex); if (current->removal_admission_closed) @@ -649,9 +649,9 @@ std::shared_ptr CasRefLedger::acquireReadableRefT } const uint64_t admitted_generation = fence_generation_fn(); - check_fence_or_throw(admitted_generation); - const CasRefCatalog::Snapshot first_catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + CasOperation op = mount_requests.resume(admitted_generation); + const CasRefCatalog::Snapshot first_catalog = CasRefCatalog::read(op, layout); + refuseUnlessAdmitted(op, "cold readable runtime admission"); first_catalog.life_index.throwIfAmbiguous("CAS cold readable runtime admission"); const auto it = std::find_if(first_catalog.catalog.entries.begin(), first_catalog.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); @@ -670,8 +670,8 @@ std::shared_ptr CasRefLedger::acquireReadableRefT /// after this read are caught by `invalidateRemovedCatalogLife` exactly as before. This second GET /// is deliberately immediately before the queue-locked fence/slot recheck in /// `acquireRefTableRuntime`; the held-handle warm path above pays none. - const CasRefCatalog::Snapshot second_catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + const CasRefCatalog::Snapshot second_catalog = CasRefCatalog::read(op, layout); + refuseUnlessAdmitted(op, "cold readable runtime admission"); /// The first read's ambiguity validation does not cover an aliasing incarnation admitted BETWEEN /// the reads; physical life-owned keys use only the incarnation, so an ambiguous second cut must /// refuse admission even when this namespace's own row is untouched. @@ -784,8 +784,8 @@ void CasRefLedger::checkRecoveryStillAdmitted(const RootNamespace & ns, RefTable ProfileEvents::increment(ProfileEvents::CASRefRecoveryCancelled); throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}' was cancelled by a self-remount before the mount " - "fence was re-armed; nothing was written and nothing installed — the next touch recovers under " - "the fresh incarnation", ns.string())); + "fence was re-armed; the last attempt's fate is unresolved and nothing is installed — the next " + "touch recovers under the fresh incarnation", ns.string())); } if (rt.catalog_life_invalidated.load(std::memory_order_acquire)) @@ -802,16 +802,9 @@ void CasRefLedger::checkRecoveryStillAdmitted(const RootNamespace & ns, RefTable "CAS ref-table recovery for namespace '{}': this cached table was superseded by a self-remount " "mid-recovery — retry against the fresh mount incarnation", ns.string())); - /// The FENCE is deliberately NOT checked here, and the omission is the point. `checkFenceOrThrow` - /// asks two things at once -- "is the fence held right now" and "is the generation still mine" -- and - /// the first has no business gating a READ. Most of this walk is reads, and a mount that has - /// transiently lost its lease can still honestly serve them from durable data; refusing at every GET - /// would turn a lease blip into "this table cannot be read at all". - /// - /// The fence gates exactly the three sites that spend it, which is the trio: every `slotOccupy` - /// (through its own `admitted_fence_ok`), the `_ckpt` CAS (inside `publishCkpt`), and the install. - /// A walk that keeps reading after the generation moved simply wastes its own I/O and is then refused - /// at the first of those -- bounded, and strictly better than refusing the reads themselves. + /// The FENCE is deliberately NOT checked here: the walk's own `CasOperation` carries the admitted + /// generation and refuses every request under a generation the fence has moved past, so a second + /// check would only report the same fact from a different sample. } std::optional CasRefLedger::runRecoveryWalkOnce( @@ -828,6 +821,24 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// under the predecessor even if the same logical name is concurrently rebound. const NamespaceLifeId life = rt.life; + /// ONE operation for the whole walk, resumed under the generation this recovery was admitted at: + /// its reads, its seal creates and its `_ckpt` publishes are all measured against that admission, + /// and a result returning after a fence bump can install nothing. + /// + /// The liveness carries EVERY term `checkRecoveryStillAdmitted` polls except the generation, which + /// is the fence's. That makes the poll and the request gate one rule rather than two that can drift: + /// a walk cancelled between two of its own polls used to keep reading until it reached the next one. + /// The predicate is a bool where the poll throws, so the poll still runs at the boundaries -- it is + /// what turns each of these facts into the right exception, and what LATCHES a cancellation for the + /// caller's retry classification. + CasOperation op = mount_requests.resume(admitted_generation, [&rt, &token] + { + return !(token && token->stopping()) + && !rt.recovery_cancel_requested.load(std::memory_order_acquire) + && !rt.catalog_life_invalidated.load(std::memory_order_acquire) + && !rt.superseded_by_remount.load(std::memory_order_acquire); + }); + /// ---- Step 2: immutable runtime authority and checkpoint ---- /// The runtime was admitted for this exact life before entering recovery, so this walk must not take /// another catalog cut. Retirement invalidates the runtime through `catalog_life_invalidated`, which @@ -839,7 +850,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( if (recovery_pre_first_request_hook_for_test) recovery_pre_first_request_hook_for_test(); checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const std::optional sampled_ckpt = readCkpt(backend, layout, life); + const std::optional sampled_ckpt = readCkpt(op, layout, life); std::optional accepted_ckpt_sample = sampled_ckpt; checkRecoveryStillAdmitted(ns, rt, cancelled, token); @@ -858,16 +869,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( try { checkRecoveryStillAdmitted(ns, rt, cancelled, token); - std::function admit_snapshot_base_request; - if (token) - { - admit_snapshot_base_request = [this, &ns, &rt, &cancelled, &token] - { - checkRecoveryStillAdmitted(ns, rt, cancelled, token); - }; - } - CheckpointSnapshotBase base = readCheckpointSnapshotBase( - backend, layout, life, sampled_ckpt->ckpt, admit_snapshot_base_request); + CheckpointSnapshotBase base = readCheckpointSnapshotBase(op, layout, life, sampled_ckpt->ckpt); base_snapshot = std::move(base.snapshot); base_snapshot_bytes = base.bytes; } @@ -881,9 +883,9 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// unchanged checkpoint turns every helper failure (missing, malformed, or seal) into the /// fail-closed corruption it describes. checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const std::optional current = readCkpt(backend, layout, life); - if (classifyMissingSampledBase(sampled_ckpt->token, - current ? std::optional(current->token) : std::nullopt) + const std::optional current = readCkpt(op, layout, life); + if (classifyMissingSampledBase(sampled_ckpt->incarnation, + current ? std::optional(current->incarnation) : std::nullopt) == MissingBaseVerdict::RestartRecovery) return std::nullopt; throw; @@ -920,24 +922,6 @@ std::optional CasRefLedger::runRecoveryWalkOnce( builder.applyOne(std::move(txn), encoded_bytes); }; - const auto check_recovery_write_admitted = [this, &ns, &rt, &cancelled, &token](uint64_t expected_generation) - { - checkRecoveryStillAdmitted(ns, rt, cancelled, token); - check_fence_or_throw(expected_generation); - if (rt.catalog_life_invalidated.load(std::memory_order_acquire)) - throwCasWriteRetryLater(fmt::format( - "CAS ref-table recovery for namespace '{}': catalog retirement invalidated life {} " - "before its checkpoint contribution", - rt.life.ns.string(), renderIncarnation(rt.life.incarnation))); - }; - std::function admit_recovery_request; - if (token) - { - admit_recovery_request = [this, &ns, &rt, &cancelled, &token] - { - checkRecoveryStillAdmitted(ns, rt, cancelled, token); - }; - } const auto publish_recovered_frontier = [&](const RefLogTxn & txn) { const RefCkpt contribution{ @@ -947,9 +931,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( .last_epoch_seal = refLogTxnIsEpochSeal(txn) ? std::optional{txn.txn_id} : txn.prev_epoch_seal}; checkRecoveryStillAdmitted(ns, rt, cancelled, token); - if (publishCkptContribution( - life, contribution, admitted_generation, check_recovery_write_admitted, admit_recovery_request) - == CkptPublishOutcome::FencedOut) + if (publishCkptContribution(op, life, contribution) == CkptPublishOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved before the " "checkpoint could record recovered txn {}-{}; nothing is installed", @@ -960,7 +942,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// one successor between lookahead and our CAS, restart from the exact newer checkpoint so the /// installed state covers every transaction its frontier certifies. checkRecoveryStillAdmitted(ns, rt, cancelled, token); - std::optional exact = readCkpt(backend, layout, life); + std::optional exact = readCkpt(op, layout, life); if (!exact || !exact->ckpt.committed_through || *exact->ckpt.committed_through < txn.txn_id) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': exact checkpoint read after publishing " @@ -969,7 +951,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( if (*exact->ckpt.committed_through != txn.txn_id) return false; - /// This recovery itself may advance `_ckpt`. The just-read token and decoded body are the + /// This recovery itself may advance `_ckpt`. The just-read incarnation and decoded body are the /// latest authority cut the private candidate has validated, so the final install boundary /// compares against this sample rather than the original one. accepted_ckpt_sample = std::move(exact); @@ -995,7 +977,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( checkRecoveryStillAdmitted(ns, rt, cancelled, token); const RefTxnId id{epoch, sequence}; - if (const auto got = backend.get(layout.refLogKey(life, id))) + if (const auto got = op.read(layout.refLogKey(life, id), Retry::standard())) { /// `runRecoveryWalkOnce` is the writer recovery entry point even after a process /// restart, when no in-memory attempt survives. A readable birth checkpoint with no @@ -1038,19 +1020,19 @@ std::optional CasRefLedger::runRecoveryWalkOnce( ? RefTxnId{id.writer_epoch + 1, 1} : RefTxnId{id.writer_epoch, id.ref_sequence + 1}; checkRecoveryStillAdmitted(ns, rt, cancelled, token); - if (backend.get(layout.refLogKey(life, following_id))) + if (op.read(layout.refLogKey(life, following_id), Retry::standard())) { checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const std::optional current = readCkpt(backend, layout, life); - if (!sampled_ckpt || !current || current->token != sampled_ckpt->token) + const std::optional current = readCkpt(op, layout, life); + if (!sampled_ckpt || !current || current->incarnation != sampled_ckpt->incarnation) return std::nullopt; const String frontier_description = sampled_frontier ? fmt::format("{}-{}", sampled_frontier->writer_epoch, sampled_frontier->ref_sequence) : "with only a life epoch"; throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref-table recovery for namespace '{}': exact checkpoint {} " - "had two durable successors through {}-{} while its token remained unchanged; " - "the append lane permits at most one unfrontiered transaction", + "had two durable successors through {}-{} while its incarnation remained " + "unchanged; the append lane permits at most one unfrontiered transaction", ns.string(), frontier_description, following_id.writer_epoch, following_id.ref_sequence); } @@ -1102,12 +1084,12 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// re-read the exact mutable checkpoint to distinguish a concurrent frontier movement /// from durable-data loss under an unchanged authority token. checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const std::optional current = readCkpt(backend, layout, life); - if (!current || current->token != sampled_ckpt->token) + const std::optional current = readCkpt(op, layout, life); + if (!current || current->incarnation != sampled_ckpt->incarnation) return std::nullopt; throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref-table recovery for namespace '{}': committed log id {}-{} is absent while " - "the exact checkpoint frontier {}-{} and its token remain unchanged", + "the exact checkpoint frontier {}-{} and its incarnation remain unchanged", ns.string(), id.writer_epoch, id.ref_sequence, sampled_frontier->writer_epoch, sampled_frontier->ref_sequence); } @@ -1120,7 +1102,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( if (sampled_seal_is_after_hole) { checkRecoveryStillAdmitted(ns, rt, cancelled, token); - if (backend.get(layout.refLogKey(life, *sampled_ckpt->ckpt.last_epoch_seal))) + if (op.read(layout.refLogKey(life, *sampled_ckpt->ckpt.last_epoch_seal), Retry::standard())) { ProfileEvents::increment(ProfileEvents::CASRefRecoveryStreamHole); hole_detail = fmt::format( @@ -1178,92 +1160,91 @@ std::optional CasRefLedger::runRecoveryWalkOnce( validateEpochSealGrammarContextual(seal_txn, *sampled_ckpt->ckpt.life_epoch); const String seal_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(seal_txn)); - /// Presented on EVERY attempt: the generation this recovery was admitted under, never the - /// current one. A seal written by an incarnation that no longer owns the namespace is a write - /// from a dead mount, and refusing pre-attempt leaves the slot provably untouched. - const auto admitted_fence_ok = [this, &rt, admitted_generation, &token] - { - return fence_ok_fn() - && !(token && token->stopping()) - && !rt.catalog_life_invalidated.load(std::memory_order_acquire) - && !rt.superseded_by_remount.load(std::memory_order_acquire) - && fence_generation_fn() == admitted_generation; - }; - + /// One bounded attempt under the generation this recovery was admitted at, never the + /// current one: a seal written by an incarnation that no longer owns the namespace is a + /// write from a dead mount, and refusing pre-attempt leaves the slot provably untouched. checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const SlotOccupyResult occupied = - ref_request_controller->slotOccupy( - layout.refLogKey(life, id), seal_bytes, admitted_fence_ok); + const WriteResult sealed = op.create(layout.refLogKey(life, id), seal_bytes, Retry::once()); - switch (occupied.kind) + if (const auto * conflict = std::get_if(&sealed)) { - case SlotOccupyResult::Kind::Created: + const auto * occupant_object = std::get_if(&conflict->seen); + if (!occupant_object) + /// The create lost the slot and the settling read could not say to what. Continuing + /// would expose a dead epoch that may or may not be closed. + throwCasWriteRetryLater(fmt::format( + "CAS ref-table recovery for namespace '{}': the epoch seal at {}-{} lost its slot " + "to an occupant the settling read could not observe; the table stays unrecovered " + "rather than being exposed with a dead epoch that may or may not be closed", + ns.string(), id.writer_epoch, id.ref_sequence)); + + /// Someone reached this slot first. A DECODE FAILURE here propagates: an object at a + /// key this namespace owns that is not a transaction of this namespace at this id is + /// corruption or a protocol breach, and the one thing recovery must not do is guess + /// past it. + RefLogTxn occupant = decodeRefLogTxn( + openObject(FormatId::RefLog, occupant_object->bytes), ns.string(), id); + const bool occupant_is_seal = refLogTxnIsEpochSeal(occupant); + const RefLogTxn frontier_txn = occupant; + apply_one(std::move(occupant), occupant_object->bytes.size()); + if (!publish_recovered_frontier(frontier_txn)) + return std::nullopt; + if (occupant_is_seal) { - /// The epoch is ours to close and now IS closed. Apply our own seal to the candidate: - /// it is a durable transaction of this stream like any other, and the next recovery - /// will read it back exactly where we put it. - RefLogTxn applied = seal_txn; - apply_one(std::move(applied), seal_bytes.size()); - ProfileEvents::increment(ProfileEvents::CASRefRecoveryEpochSealed); - if (!publish_recovered_frontier(seal_txn)) - return std::nullopt; + /// A concurrent recoverer closed this epoch (or our own earlier attempt did, and + /// its acknowledgment was lost). Either way the epoch is closed by a seal that is + /// as good as ours -- adopt it and continue. Contesting a peer's CORRECT write is + /// how two recoverers of the same table turn a designed race into an incident. + ProfileEvents::increment(ProfileEvents::CASRefRecoveryEpochSealAdopted); ++epoch; sequence = 1; slot_attempts_this_epoch = 0; - break; } - case SlotOccupyResult::Kind::Occupied: - { - /// Someone reached this slot first. A DECODE FAILURE here propagates: an object at a - /// key this namespace owns that is not a transaction of this namespace at this id is - /// corruption or a protocol breach, and the one thing recovery must not do is guess - /// past it. - RefLogTxn occupant = decodeRefLogTxn( - openObject(FormatId::RefLog, occupied.occupant_bytes), ns.string(), id); - const bool occupant_is_seal = refLogTxnIsEpochSeal(occupant); - const RefLogTxn frontier_txn = occupant; - apply_one(std::move(occupant), occupied.occupant_bytes.size()); - if (!publish_recovered_frontier(frontier_txn)) - return std::nullopt; - if (occupant_is_seal) - { - /// A concurrent recoverer closed this epoch (or our own earlier attempt did, and - /// its acknowledgment was lost). Either way the epoch is closed by a seal that is - /// as good as ours -- adopt it and continue. Contesting a peer's CORRECT write is - /// how two recoverers of the same table turn a designed race into an incident. - ProfileEvents::increment(ProfileEvents::CASRefRecoveryEpochSealAdopted); - ++epoch; - sequence = 1; - slot_attempts_this_epoch = 0; - } - else - { - /// A STRAGGLER: an ordinary transaction of the dead epoch landed at `T+1` between - /// our read and our create. Adopt it, advance `T` by exactly ONE, and try the seal - /// again at the NEW `T+1`. Never mint `T+2` around it: ids are state-derived - /// (INV-1/INV-2), and writing past an occupied slot puts a hole in the durable - /// stream that no later reader can distinguish from a lost object. - ProfileEvents::increment(ProfileEvents::CASRefRecoveryStragglerAdopted); - ++sequence; - } - break; - } - case SlotOccupyResult::Kind::Unresolved: + else { - /// The store will not say whether our seal landed. There is no honest way to continue: - /// exposing the table would publish a dead epoch that may or may not be closed, and - /// re-deriving the slot later needs a fresh read anyway. Fail this attempt into the - /// caller's transient-retry loop, which either succeeds on a later attempt or spends - /// its budget and leaves the table unrecovered. - throwCasWriteRetryLater(fmt::format( - "CAS ref-table recovery for namespace '{}': the epoch seal at {}-{} is UNRESOLVED " - "({}); the table stays unrecovered rather than being exposed with a dead epoch that " - "may or may not be closed", - ns.string(), id.writer_epoch, id.ref_sequence, - unresolvedProvesNothingWasSent(occupied.unresolved_reason) - ? "nothing was sent" : "the outcome of the attempt is unknown")); + /// A STRAGGLER: an ordinary transaction of the dead epoch landed at `T+1` between + /// our read and our create. Adopt it, advance `T` by exactly ONE, and try the seal + /// again at the NEW `T+1`. Never mint `T+2` around it: ids are state-derived + /// (INV-1/INV-2), and writing past an occupied slot puts a hole in the durable + /// stream that no later reader can distinguish from a lost object. + ProfileEvents::increment(ProfileEvents::CASRefRecoveryStragglerAdopted); + ++sequence; } + continue; } + + if (const auto * gave_up = std::get_if(&sealed)) + /// The store will not say whether our seal landed. There is no honest way to continue: + /// exposing the table would publish a dead epoch that may or may not be closed, and + /// re-deriving the slot later needs a fresh read anyway. Fail this attempt into the + /// caller's transient-retry loop, which either succeeds on a later attempt or spends + /// its budget and leaves the table unrecovered. + throwCasWriteRetryLater(fmt::format( + "CAS ref-table recovery for namespace '{}': the epoch seal at {}-{} is UNRESOLVED " + "({}); the table stays unrecovered rather than being exposed with a dead epoch that " + "may or may not be closed", + ns.string(), id.writer_epoch, id.ref_sequence, + gave_up->sent_any ? "the outcome of the attempt is unknown" : "nothing was sent")); + + if (const auto * refused = std::get_if(&sealed)) + /// The store proved this attempt never applied. The slot is untouched, so the epoch is + /// still unclosed and the table still may not be exposed. + throwCasWriteRetryLater(fmt::format( + "CAS ref-table recovery for namespace '{}': the store refused the epoch seal at " + "{}-{} ({}); the table stays unrecovered with its dead epoch still open", + ns.string(), id.writer_epoch, id.ref_sequence, refused->message)); + + /// The epoch is ours to close and now IS closed. Apply our own seal to the candidate: + /// it is a durable transaction of this stream like any other, and the next recovery + /// will read it back exactly where we put it. + RefLogTxn applied = seal_txn; + apply_one(std::move(applied), seal_bytes.size()); + ProfileEvents::increment(ProfileEvents::CASRefRecoveryEpochSealed); + if (!publish_recovered_frontier(seal_txn)) + return std::nullopt; + ++epoch; + sequence = 1; + slot_attempts_this_epoch = 0; } } @@ -1277,16 +1258,15 @@ std::optional CasRefLedger::runRecoveryWalkOnce( .committed_through = last_epoch_seal, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = last_epoch_seal}; - if (publishCkptContribution( - life, contribution, admitted_generation, check_recovery_write_admitted, admit_recovery_request) - == CkptPublishOutcome::FencedOut) + if (publishCkptContribution(op, life, contribution) == CkptPublishOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved before the " - "checkpoint could record the epoch seal {}-{}; nothing was written and nothing is installed", + "checkpoint could record the epoch seal {}-{}; the last attempt's fate is unresolved and " + "nothing is installed", ns.string(), last_epoch_seal->writer_epoch, last_epoch_seal->ref_sequence)); checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const std::optional exact = readCkpt(backend, layout, life); + const std::optional exact = readCkpt(op, layout, life); if (!exact || exact->ckpt.committed_through != private_frontier || exact->ckpt.last_epoch_seal != last_epoch_seal) return std::nullopt; accepted_ckpt_sample = exact; @@ -1294,12 +1274,12 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// Final authority validation is the recovery linearization point. The last exact log probe fixed /// the private cut, but another actor could have changed `_ckpt` immediately afterwards. Install - /// only when both the exact object token and its complete decoded body remain equal to the latest - /// authority sample this private candidate accepted. + /// only when both the exact object incarnation and its complete decoded body remain equal to the + /// latest authority sample this private candidate accepted. checkRecoveryStillAdmitted(ns, rt, cancelled, token); - const std::optional final_ckpt = readCkpt(backend, layout, life); + const std::optional final_ckpt = readCkpt(op, layout, life); if (!final_ckpt || !accepted_ckpt_sample - || final_ckpt->token != accepted_ckpt_sample->token + || final_ckpt->incarnation != accepted_ckpt_sample->incarnation || final_ckpt->ckpt != accepted_ckpt_sample->ckpt) return std::nullopt; checkRecoveryStillAdmitted(ns, rt, cancelled, token); @@ -1337,12 +1317,12 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// reconciling a stale creator) converges in a handful of rounds; this guards only against a /// pathologically un-converging sequence of them. static constexpr size_t kMaxResolveAttempts = 32; - const CkptDeadline deadline{boot_ms_now_fn, boot_ms_now_fn() + cas_request_budget.operation_deadline_ms}; const CreatorFence our_fence{server_root_id, live_epoch, admitted_generation}; + CasOperation op = mount_requests.resume(admitted_generation); for (size_t attempt = 0; attempt < kMaxResolveAttempts; ++attempt) { - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); const auto it = std::find_if(snap.catalog.entries.begin(), snap.catalog.entries.end(), [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); @@ -1354,12 +1334,12 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// catalog on the next loop iteration to learn it -- one extra GET, paid once per birth, /// never per write. const auto outcome = CasRefCatalog::createNamespace( - backend, layout, config.gc_shards, ns, our_fence, - admitted_generation, check_fence_or_throw, deadline); + op, layout, config.gc_shards, ns, our_fence); if (outcome == CasRefCatalog::NamespaceCreationOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " - "birthing its catalog entry; nothing was written and nothing installed", ns.string())); + "birthing its catalog entry; the last attempt's fate is unresolved and nothing is " + "installed", ns.string())); continue; /// Live or Superseded: re-read (Superseded means a DIFFERENT actor won birth) } @@ -1384,12 +1364,12 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// not dead), so this case is checked FIRST and unconditionally, before any terminality probe. if (it->creator->server_root_id == server_root_id && it->creator->writer_epoch == live_epoch) { - const auto outcome = CasRefCatalog::completeCreation( - backend, layout, *it, admitted_generation, check_fence_or_throw, deadline); + const auto outcome = CasRefCatalog::completeCreation(op, layout, *it); if (outcome == CasRefCatalog::NamespaceCreationOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " - "resuming its own stalled creation; nothing was written and nothing installed", + "resuming its own stalled creation; the last attempt's fate is unresolved and nothing " + "is installed", ns.string())); continue; /// Live or Superseded: re-read either way } @@ -1398,28 +1378,28 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// fresh read -- never busy-loop this instant) or provably dead, in which case reconciliation /// steals it onto our own fence and this open resumes `completeCreation` itself. const auto reconcile_outcome = CasRefCatalog::reconcileStaleCreator( - backend, layout, *it, our_fence, - [this](const CreatorFence & f) { return isCreatorFenceTerminal(backend, layout, f.server_root_id, f.writer_epoch); }, - admitted_generation, check_fence_or_throw); + op, layout, *it, our_fence, + [&](const CreatorFence & f) { return isCreatorFenceTerminal(op, layout, f.server_root_id, f.writer_epoch); }); switch (reconcile_outcome) { case CasRefCatalog::ReconcileCreatorOutcome::FencedOut: - /// Review I6: our OWN mount fence moved before the steal CAS -- nothing was written, and + /// Our OWN mount fence moved before the steal CAS -- the CAS's fate is unresolved, and /// this mount is the wrong actor to retry (its incarnation is gone). throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " - "reconciling a stalled foreign creator; nothing was written and nothing installed", + "reconciling a stalled foreign creator; the last attempt's fate is unresolved and " + "nothing is installed", ns.string())); case CasRefCatalog::ReconcileCreatorOutcome::Reconciled: { CatalogEntry resumed = *it; resumed.creator = our_fence; - const auto outcome = CasRefCatalog::completeCreation( - backend, layout, resumed, admitted_generation, check_fence_or_throw, deadline); + const auto outcome = CasRefCatalog::completeCreation(op, layout, resumed); if (outcome == CasRefCatalog::NamespaceCreationOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " - "completing a reconciled creation; nothing was written and nothing installed", + "completing a reconciled creation; the last attempt's fate is unresolved and " + "nothing is installed", ns.string())); continue; /// Live or Superseded: re-read either way } @@ -1482,16 +1462,16 @@ void CasRefLedger::ensureRefTableRecovered( }); /// ---- Step 1: capture the admitted generation, ONCE ---- - /// The trio (spec §3, codex finding 7): this ONE value is what the walk's every `slotOccupy` and its - /// `_ckpt` CAS present, and what the install below presents one final time. One capture point, three - /// checks, no re-derivation -- a value re-read midway would let a recovery that lost the mount - /// "recover" its right to write by observing a fresh incarnation it was never admitted under. + /// This ONE value admits the walk's operation, so every + /// request it makes is measured against it, and the install below presents it one final time. One + /// capture point, no re-derivation -- a value re-read midway would let a recovery that lost the + /// mount "recover" its right to write by observing a fresh incarnation it was never admitted under. /// /// Captured for the WHOLE call, not per attempt, for the same reason: the transient-retry loop below /// exists for object-store blips, and a generation that moved is not one. The loop refuses to /// re-drive under a moved generation (below), so the budget is never burned on a doomed retry. const uint64_t admitted_generation = rt.admitted_fence_generation; - check_fence_or_throw(admitted_generation); + refuseUnlessAdmitted(mount_requests.resume(admitted_generation), "ref recovery walk admission"); /// Preserve this runtime's exact writer identity across the unlocked walk. The runtime stays in /// `NeedsRecovery` until the same lock installs a result, so no later append can replace it here. const std::optional retained_attempt = rt.append_attempt; @@ -1573,7 +1553,9 @@ void CasRefLedger::ensureRefTableRecovered( /// of time to come back, and a recovery whose window straddled a fence bump describes a /// mount incarnation that no longer owns this namespace. It must publish NOTHING: the /// table stays unrecovered and the next touch recovers it properly. - check_fence_or_throw(admitted_generation); + if (recovery_install_probe_for_test) + recovery_install_probe_for_test(); + refuseUnlessAdmitted(mount_requests.resume(admitted_generation), "ref recovery install"); if (rt.catalog_life_invalidated.load(std::memory_order_acquire)) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': catalog retirement invalidated life {} " @@ -1614,6 +1596,13 @@ void CasRefLedger::ensureRefTableRecovered( || !isTransientRecoveryError(code)) throw; /// a latched terminal case, or a non-transient failure -- fail fast + /// A cancellation now ends the walk through the operation's liveness, which refuses + /// silently -- so the exception that arrives here carries a transport code and `cancelled` + /// is not yet latched. Take the latch before the backoff rather than one whole attempt + /// later: the self-remount barrier is waiting for this recovery to stop. + if (rt.recovery_cancel_requested.load(std::memory_order_acquire)) + checkRecoveryStillAdmitted(ns, rt, cancelled, token); + const uint64_t elapsed_ms = boot_ms_now_fn() - recovery_start_ms; /// Fail closed BEFORE sleeping: budget spent, mount fence lost, this runtime superseded by a /// self-remount, or the incarnation that admitted this recovery has moved (retrying under a @@ -1626,9 +1615,9 @@ void CasRefLedger::ensureRefTableRecovered( || fence_generation_fn() != admitted_generation) throw; - /// Saturating `initial << recovery_retry_num` (mirrors `CasRequestController::backoffBefore - /// Attempt`): `initial > cap >> n` implies the unshifted product already exceeds the cap, so - /// return the cap without ever computing an overflowing/UB shift for large retry counts. + /// Saturating `initial << recovery_retry_num`: `initial > cap >> n` implies the unshifted + /// product already exceeds the cap, so return the cap without ever computing an + /// overflowing/UB shift for large retry counts. const uint64_t init_backoff = cas_request_budget.recovery_retry_initial_backoff_ms; const uint64_t cap_backoff = cas_request_budget.recovery_retry_max_backoff_ms; const uint64_t backoff_ms = (recovery_retry_num >= 63 || init_backoff > (cap_backoff >> recovery_retry_num)) @@ -2127,7 +2116,7 @@ RefTxnId CasRefLedger::appendRefOpsOnRuntime( /// Build the responsibility set (its own `item`) BEFORE publishing the baton, so becoming /// leader contains NO throwing operation once `leader_active` is set: the only allocation is /// this first `push_back`, done here while still holding `lk` and NOT yet leader. If it throws - /// (a `bad_alloc` at the pre-tenure point; codex stage-1 review, Important), the baton is never + /// (a `bad_alloc` at the pre-tenure point), the baton is never /// taken -- but `item` is already in `pending` (pushed above), so it must be un-enqueued before /// propagating, else a future leader would carve an item whose `build_ops` closure died with /// this unwinding caller (the same use-after-free the exit guard prevents post-publication). @@ -2271,7 +2260,7 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptrcatalog_life_invalidated.load(std::memory_order_acquire) - && !rt->superseded_by_remount.load(std::memory_order_acquire) - && fence_generation_fn() == admitted; - }; + return !rt->catalog_life_invalidated.load(std::memory_order_acquire) + && !rt->superseded_by_remount.load(std::memory_order_acquire); + }); - SlotOccupyResult occupied; + std::optional attempted; try { if (wedge_before_slot_occupy_hook_for_test) wedge_before_slot_occupy_hook_for_test(); - occupied = ref_request_controller->slotOccupy(wedge.key, wedge.bytes, admitted_fence_ok); + attempted = op.create(wedge.key, wedge.bytes, Retry::once()); } catch (...) { - /// `ambiguous-then-definite`, the model-proven control. `slotOccupy` rethrows only a definite - /// refusal of THIS attempt (a whitelisted synchronous rejection, or a deterministic local - /// failure) -- and a definite refusal of a LATER attempt proves nothing whatsoever about the - /// EARLIER ambiguous one, which may still be in flight or may already have landed. So the lane - /// stays wedged: unwedging here is exactly how an acked-then-lost transaction gets written - /// around. The id is not consumed either, so the next attempt re-derives the SAME one. + /// `ambiguous-then-definite`, the model-proven control. A deterministic local failure surfaces + /// unchanged, and it proves nothing whatsoever about the EARLIER ambiguous attempt, which may + /// still be in flight or may already have landed. So the lane stays wedged: unwedging here is + /// exactly how an acked-then-lost transaction gets written around. The id is not consumed + /// either, so the next attempt re-derives the SAME one. result.kind = WedgeResolution::StillWedged; result.survivor_error = makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}': the bounded retry of wedged txn {}-{} was definitively " - "refused ({}), which says nothing about the earlier ambiguous attempt — the lane stays wedged", + "CAS ref-log append for namespace '{}': the bounded retry of wedged txn {}-{} failed ({}), " + "which says nothing about the earlier ambiguous attempt — the lane stays wedged", ns.string(), wedge.txn_id.writer_epoch, wedge.txn_id.ref_sequence, getCurrentExceptionMessage(/*with_stacktrace*/ false))); return result; } + /// A store refusal is the same control as the throw above: it proves only its OWN attempt never + /// applied, and the earlier ambiguous one is what the wedge is about. + if (const auto * refused = std::get_if(&*attempted)) + { + result.kind = WedgeResolution::StillWedged; + result.survivor_error = makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS ref-log append for namespace '{}': the bounded retry of wedged txn {}-{} was definitively " + "refused ({}), which says nothing about the earlier ambiguous attempt — the lane stays wedged", + ns.string(), wedge.txn_id.writer_epoch, wedge.txn_id.ref_sequence, refused->message)); + return result; + } + /// ---- Classify the occupant OFF the lock: pure, and the decode allocates ---- /// The three-way `mine | successor's seal | foreign` adjudication is the CALLER's job by - /// construction (`slotOccupy` never compares bytes), and "mine" means BYTE EQUALITY -- never a - /// shape or generation match, which is the aliasing the phase-0 model rejected. - const Occupant occupant = occupied.kind == SlotOccupyResult::Kind::Occupied - ? classifyRefLogOccupant(ns, wedge.txn_id, occupied.occupant_bytes, wedge.bytes) + /// construction (the engine never compares bytes for meaning), and "mine" means BYTE EQUALITY -- + /// never a shape or generation match, which is the aliasing the phase-0 model rejected. + const auto * conflict = std::get_if(&*attempted); + const auto * occupant_object = conflict ? std::get_if(&conflict->seen) : nullptr; + const auto * gave_up = std::get_if(&*attempted); + /// A conflict whose settling read observed nothing is as unresolved as a give-up: the key holds + /// something this call could not name, so nothing may be adopted or unwedged from it. + const bool unresolved = gave_up || (conflict && !occupant_object); + const Occupant occupant = occupant_object + ? classifyRefLogOccupant(ns, wedge.txn_id, occupant_object->bytes, wedge.bytes) : Occupant::NotOccupied; const bool exact_attempt_is_durable - = occupied.kind == SlotOccupyResult::Kind::Created || occupant == Occupant::Ours; + = std::holds_alternative(*attempted) || occupant == Occupant::Ours; /// Caller holds `state_mutex`. Keeping the identity predicate in one place is part of the safety /// rule: adding a frontier must not create yet another subtly different notion of "same attempt". const auto same_wedge_under_lock = [&] @@ -2402,26 +2406,6 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptrcatalog_life_invalidated.load(std::memory_order_acquire) - || rt->superseded_by_remount.load(std::memory_order_acquire)) - throwCasWriteRetryLater(fmt::format( - "CAS namespace '{}': its captured runtime was retired before wedged-frontier publication", - rt->life.ns.string())); - - bool same_wedge = false; - { - std::lock_guard lock(rt->state_mutex); - same_wedge = same_wedge_under_lock(); - } - if (!same_wedge) - throwCasWriteRetryLater(fmt::format( - "CAS namespace '{}': the captured wedge changed before frontier publication", - rt->life.ns.string())); - }; - const RefCkpt frontier{ .life_epoch = std::nullopt, .committed_through = wedge.txn_id, @@ -2432,8 +2416,7 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptrlife, frontier, wedge.admitted_fence_generation, check_wedge_admitted); + frontier_outcome = publishCkptContribution(op, rt->life, frontier); } catch (...) { @@ -2467,20 +2450,11 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptrstate_mutex); - /// The generation this attempt was admitted under, presented back. `checkFenceOrThrow` reports a - /// moved incarnation by throwing; it is CAUGHT here rather than propagated, because the caller's - /// retry classification keys on the retry-later error class and a routine lease blip must not - /// reach it as a hard failure. Nothing is installed and nothing is unwedged either way, which is - /// the whole meaning of INERT here. - bool fence_moved = false; - try - { - check_fence_or_throw(wedge.admitted_fence_generation); - } - catch (...) - { - fence_moved = true; - } + /// The generation this attempt was admitted under, presented back. A moved incarnation is a + /// verdict here, not an exception: the caller's retry classification keys on the retry-later + /// error class and a routine lease blip must not reach it as a hard failure. Nothing is + /// installed and nothing is unwedged either way, which is the whole meaning of INERT here. + const bool fence_moved = !op.admitted(); /// The remount half of the same question, checked separately because the two are independent /// facts even though today's ordering makes one imply the other: `quiesceRefTablesForRemount` @@ -2510,14 +2484,14 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptrsent_any ? Reason::RefusedPreAttempt : Reason::ResolveFoundNothing; result.kind = WedgeResolution::StillWedged; } @@ -2712,10 +2686,8 @@ CasRefLedger::resolveWedgeOnce(const RootNamespace & ns, const std::shared_ptr> & items, std::exception_ptr e) { std::lock_guard g(ref_queue_mutex); @@ -3373,10 +3345,11 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt } rt->cv.notify_all(); }; - const auto fence_ok = [this, &rt] + /// Everything the append fence cannot see. The generation term is the operation's own, so it is + /// deliberately absent here. + const auto runtime_live = [&rt] { - return fence_ok_fn() - && !rt->catalog_life_invalidated.load(std::memory_order_acquire) + return !rt->catalog_life_invalidated.load(std::memory_order_acquire) && !rt->superseded_by_remount.load(std::memory_order_acquire); }; @@ -3410,8 +3383,9 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt } const uint64_t admitted_generation = rt->admitted_fence_generation; - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + CasOperation catalog_op = mount_requests.resume(admitted_generation, runtime_live); + const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(catalog_op, layout); + refuseUnlessAdmitted(catalog_op, "terminal removal append"); catalog.life_index.throwIfAmbiguous("CAS terminal removal append"); const auto entry_it = std::find_if( catalog.catalog.entries.begin(), catalog.catalog.entries.end(), @@ -3440,8 +3414,9 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt try { const uint64_t admitted_generation = rt->admitted_fence_generation; - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + CasOperation catalog_op = mount_requests.resume(admitted_generation, runtime_live); + const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(catalog_op, layout); + refuseUnlessAdmitted(catalog_op, "removal-class append"); const NamespaceLifeId & life = rt->life; const auto entry_it = std::find_if(catalog.catalog.entries.begin(), catalog.catalog.entries.end(), [&](const CatalogEntry & entry) @@ -3591,7 +3566,7 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// /// THE PLACEMENT IS THE CORRECTNESS ARGUMENT, and it has to hold for BOTH chunk shapes, because /// `commitRefChunk` has two different first durable effects. An ordinary chunk's is the ref-log - /// `putIfAbsentControlled` far below; a `NamespaceBirth` chunk's is the `_ckpt` publish, which is + /// create far below; a `NamespaceBirth` chunk's is the `_ckpt` publish, which is /// EARLIER. This call therefore sits above both, and every statement between here and the lock above /// is in-memory only. A "pure" preparation that published the `_ckpt` itself would be a lie, and /// moving that publish later would change fault semantics the directive says to preserve. @@ -3621,6 +3596,12 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// move is COW-pointer-only and happens here, while nothing is durable. std::optional candidate{std::move(prepared->candidate)}; + /// ONE operation for this chunk's whole durable phase -- the birth `_ckpt`, the log create and the + /// committed-frontier publish -- resumed under the generation the transaction was admitted at. The + /// engine refuses every one of them once that generation moves, so a chunk admitted by an + /// incarnation this mount no longer holds can make nothing durable. + CasOperation op = mount_requests.resume(admitted_fence_generation, runtime_live); + /// INV-4's FIRST `_ckpt` writer, and the ONLY writer anywhere that knows this namespace's /// `life_epoch`: it is the writer epoch of its `namespace_birth`, which is this transaction. No /// later writer can recover it (a table recovered from a snapshot never replays the birth), so if @@ -3660,8 +3641,7 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt { try { - if (publishCkptContribution(rt->life, *prepared->birth_contribution, - admitted_fence_generation, check_fence_or_throw) + if (publishCkptContribution(op, rt->life, *prepared->birth_contribution) == CkptPublishOutcome::FencedOut) { complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( @@ -3725,7 +3705,7 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// append for this table already advanced applied state. That other append can be the birth /// itself, whose `_ckpt` a cleanup call here would then delete out from under it -- the same harm /// class the ambiguous `Writing -> Wedged` branch is deliberately excluded to avoid, against an - /// object with no repair path (BACKLOG `{#ckpt-damage-no-repair-path}`). "`putIfAbsentControlled` + /// object that has no repair path. "the ref-log create /// was never reached" is true of THIS attempt; it says nothing about whether a DIFFERENT attempt /// for this same namespace already made the birth durable. Now that Critical B removed the other /// call sites too, `_ckpt` debris from a never-born namespace is reclaimed only by the future @@ -3738,35 +3718,31 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt } const RefAppendAttempt & active_attempt = *rt->append_attempt; - CasWriteOutcome outcome{}; - /// WHY an Unresolved came back. Two jobs (finding #37 defect 3): the wedge message stops claiming an - /// exhausted retry budget when in fact no request was ever sent, and -- see the `Unresolved` arm -- - /// the one reason that PROVES nothing was sent decides whether the lane wedges at all. - CasUnresolvedReason unresolved_reason = CasUnresolvedReason::NotUnresolved; + std::optional written; try { - outcome = ref_request_controller->putIfAbsentControlled( - active_attempt.key, active_attempt.bytes, fence_ok, /*out_token=*/nullptr, &unresolved_reason); + written = op.create(active_attempt.key, active_attempt.bytes, Retry::standard()); } catch (...) { + /// Every classified outcome comes back as a value, so an exception here is a deterministic + /// local failure raised at a point where an attempt may already have been sent. That is + /// ambiguous, and it therefore transfers ownership from `Writing` to `Wedged`; the exact + /// attempt remains installed. const std::exception_ptr write_error = std::current_exception(); - /// `putIfAbsentControlled` throws CORRUPTED_DATA when resolve-before-reissue observes a DIFFERENT - /// object already at this txn's key -- a proven different-object conflict, not an unresolved PUT. - /// Any other exception after the send boundary is ambiguous and therefore transfers ownership - /// from `Writing` to `Wedged`; the exact attempt remains installed. - if (getCurrentExceptionCode() != ErrorCodes::CORRUPTED_DATA) { - { - std::lock_guard lock(rt->state_mutex); - if (rt->lane_state == RefLaneState::Writing && rt->append_attempt - && rt->append_attempt->txn_id == id) - rt->lane_state = RefLaneState::Wedged; - } - complete_error(chunk_survivors, write_error); - return false; + std::lock_guard lock(rt->state_mutex); + if (rt->lane_state == RefLaneState::Writing && rt->append_attempt + && rt->append_attempt->txn_id == id) + rt->lane_state = RefLaneState::Wedged; } - /// THREE-WAY ADJUDICATION, the same one the wedge resolution owes [review HIGH-2]. "A different + complete_error(chunk_survivors, write_error); + return false; + } + + if (const auto * conflict = std::get_if(&*written)) + { + /// THREE-WAY ADJUDICATION, the same one the wedge resolution owes. "A different /// object at our derived key" is not one situation but two, and they call for opposite /// reactions. One of them is EXPECTED: a successor that sealed our epoch put its epoch-closing /// record at exactly the id we keep re-deriving, and INV-2 says we must keep re-deriving it @@ -3774,37 +3750,60 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// foreign interference would fence the mount and raise an anomaly alarm on the designed path. /// The other is a genuine breach of write-exclusivity and must be exactly as loud as before. /// - /// The occupant is read once, by exact key, here -- `putIfAbsentControlled` proved the mismatch - /// but does not hand back what it saw. One extra request on a path that is already exceptional - /// and already fatal to this attempt. + /// The occupant arrives WITH the conflict: the write's own settling read already observed the + /// key, so no second request is issued here. + const auto * occupant_object = std::get_if(&conflict->seen); + if (!occupant_object) + { + /// The settling read named NO occupant: it either failed, or it proved the key absent after + /// the store had already refused our create. Neither observation can be adjudicated, and + /// neither is terminal -- `resolveWedgeOnce` meets the identical observation and keeps the + /// lane WEDGED, and this site owes the same answer. The wedge is what makes the next flush + /// re-create at this exact key and adjudicate whatever it then finds; faulting instead would + /// spend the table's write availability until a remount on a read that may well succeed on + /// the next attempt. + /// + /// It must be COUNTED, because this arm is quiet by construction: the loud interference + /// report is only reached once the occupant can be NAMED, so a real breach whose occupant + /// keeps failing to be read would otherwise show up as nothing but a throttled log line + /// under load. Sustained growth on this counter is the signal that the loud path is starved. + ProfileEvents::increment(ProfileEvents::CASRefAppendOccupantUnreadable); + { + std::lock_guard lock(rt->state_mutex); + if (rt->lane_state == RefLaneState::Writing && rt->append_attempt + && rt->append_attempt->txn_id == id) + rt->lane_state = RefLaneState::Wedged; + } + ProfileEvents::increment(ProfileEvents::CASRefAppendWedged); + complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS ref-log append for namespace '{}': the store refused txn {}-{}'s create and the " + "settling read named no occupant (observed {}) — the append lane is wedged until the " + "SAME key resolves durable or a conclusive rejection is observed", + ns.string(), id.writer_epoch, id.ref_sequence, detail::renderObservation(conflict->seen)))); + return false; + } + Occupant occupant = Occupant::Foreign; bool classified = false; try { - if (const auto got = backend.get(active_attempt.key)) - { - occupant = classifyRefLogOccupant(ns, id, got->bytes, active_attempt.bytes); - classified = true; - } + occupant = classifyRefLogOccupant(ns, id, occupant_object->bytes, active_attempt.bytes); + classified = true; } catch (...) // NOLINT(bugprone-empty-catch) { - /// Left unclassified deliberately -- see below. The original conflict is what the survivors - /// are told about; this read's own failure is not their business. + /// Left unclassified deliberately -- see below. The conflict itself is what the survivors + /// are told about; the adjudication's own failure is not their business. } if (!classified) { - /// We could not learn WHICH of the two this is, so we decide NEITHER. Reporting foreign - /// interference would fence the mount on a guess, and reporting a conclusive rejection would - /// acknowledge a deposition we did not observe. The id is not consumed and nothing is - /// recorded, so the next append re-derives the same id, meets the same conflict, and - /// classifies again -- deferring costs one round trip and decides nothing wrongly. + /// An occupant WAS observed, and naming it raised something other than the malformed-object + /// codes `classifyRefLogOccupant` answers `Foreign` for. We could not learn WHICH of the two + /// situations this is, so we decide NEITHER: reporting foreign interference would fence the + /// mount on a guess, and reporting a conclusive rejection would acknowledge a deposition we + /// did not observe. /// - /// It must be COUNTED, because deferring is the one arm here that is quiet by construction: - /// the loud interference report is only reached once the occupant can be read, so a real - /// breach whose occupant keeps failing to read would otherwise show up as nothing but a - /// throttled log line under load. Sustained growth on this counter is the signal that the - /// loud path is being starved. + /// It must be COUNTED, for the same reason the arm above is. ProfileEvents::increment(ProfileEvents::CASRefAppendOccupantUnreadable); { std::lock_guard lock(rt->state_mutex); @@ -3814,9 +3813,9 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt complete_error(chunk_survivors, std::make_exception_ptr(Exception( ErrorCodes::CORRUPTED_DATA, "CAS ref-log append for namespace '{}': a DIFFERENT object occupies the id {}-{} this table " - "derived, and reading it to tell a successor's epoch seal from foreign interference did not " - "succeed — the lane is faulted until remount recovery adjudicates durable state", - ns.string(), id.writer_epoch, id.ref_sequence))); + "derived, and telling a successor's epoch seal from foreign interference did not succeed " + "(observed {}) — the lane is faulted until remount recovery adjudicates durable state", + ns.string(), id.writer_epoch, id.ref_sequence, detail::renderObservation(conflict->seen)))); return false; } if (occupant == Occupant::SuccessorSeal) @@ -3840,298 +3839,286 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt ns.string(), id.writer_epoch, id.writer_epoch, id.ref_sequence))); return false; } - /// A genuine breach. This table's appends are now BLOCKED, and that is the intended contract: - /// under mount-lease exclusivity this key is exclusively ours, so a foreign object at it is - /// corruption or a protocol breach, not a race. The id is not consumed, so the next attempt - /// derives the SAME id and hits the SAME conflict, loudly, until a remount-level recovery (a - /// fresh writer epoch is a fresh key namespace) clears it. Advancing past the occupant, which is - /// what the pool-wide allocator did, would have written this table's stream around a foreign - /// object and hidden the violation -- and produced the hole INV-1 exists to forbid. + if (occupant != Occupant::Ours) + { + /// A genuine breach. This table's appends are now BLOCKED, and that is the intended contract: + /// under mount-lease exclusivity this key is exclusively ours, so a foreign object at it is + /// corruption or a protocol breach, not a race. The id is not consumed, so the next attempt + /// derives the SAME id and hits the SAME conflict, loudly, until a remount-level recovery (a + /// fresh writer epoch is a fresh key namespace) clears it. Advancing past the occupant, which is + /// what the pool-wide allocator did, would have written this table's stream around a foreign + /// object and hidden the violation -- and produced the hole INV-1 exists to forbid. + /// + /// Route it through the anomaly policy, exactly as the wedge-resolution site does for the + /// identical observation. Failing closed is right, but failing closed FOREVER is + /// not: without this the mount stays blocked on this table until somebody notices and remounts + /// by hand. One impossibility, one reaction. The report is deliberately BEFORE the survivors are + /// completed, so the fence is closed by the time any caller wakes and can retry. + const String attempt_key = active_attempt.key; + { + std::lock_guard lock(rt->state_mutex); + rt->append_attempt.reset(); + rt->lane_state = RefLaneState::Faulted; + } + const String interference_detail = fmt::format( + "ref-log append for namespace '{}' txn {}-{} observed a DIFFERENT object already at the id it " + "derived, and it is not an epoch seal of this namespace (observed {})", + ns.string(), id.writer_epoch, id.ref_sequence, detail::renderObservation(conflict->seen)); + on_impossible_interference(attempt_key, interference_detail, ns.string()); + complete_error(chunk_survivors, std::make_exception_ptr(Exception( + ErrorCodes::CORRUPTED_DATA, "CAS {}", interference_detail))); + return false; + } + /// `Occupant::Ours`: our OWN bytes at the id we derived, so an earlier attempt of this exact + /// transaction is already durable and byte equality is the proof (never a shape or generation + /// match). This lane's state machine does not produce it -- an attempt that may have landed + /// leaves the lane `Wedged`, and resolving that wedge advances the id -- but it is decided here + /// rather than left to the arm above, because reporting this table's own content as foreign + /// interference would fence the mount. The commit path below installs it, exactly as the wedge + /// site's identical observation does. + } + else if (const auto * refused = std::get_if(&*written)) + { + /// Proof that nothing became durable returns the exact attempt to `Ready`. + { + std::lock_guard lock(rt->state_mutex); + if (rt->lane_state == RefLaneState::Writing && rt->append_attempt + && rt->append_attempt->txn_id == id) + { + rt->append_attempt.reset(); + rt->lane_state = RefLaneState::Ready; + } + } + ProfileEvents::increment(ProfileEvents::CASRefAppendDefiniteFailure); + complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS ref-log append for namespace '{}' definitively failed ({}); " + "cached state is unchanged and txn id {}-{} was never used (a retry re-derives it)", + ns.string(), refused->message, id.writer_epoch, id.ref_sequence))); + return false; + } + else if (const auto * gave_up = std::get_if(&*written)) + { + /// The ONE give-up shape that must NOT wedge. The wedge exists because an unresolved write MAY + /// HAVE LANDED: the durable log may or may not contain this transaction, only a read of that + /// exact key can settle it, and until it does, minting a later id would build on a state that + /// may be missing a landed transaction. All of that presupposes an attempt was SENT. /// - /// Route it through the anomaly policy, exactly as the wedge-resolution site does for the - /// identical observation [review I5]. Failing closed is right, but failing closed FOREVER is - /// not: without this the mount stays blocked on this table until somebody notices and remounts - /// by hand. One impossibility, one reaction. The report is deliberately BEFORE the survivors are - /// completed, so the fence is closed by the time any caller wakes and can retry. - const String attempt_key = active_attempt.key; + /// `sent_any` is the whole call's, not its last attempt's: it is false only when every gate + /// refused before the first request reached the network, so the key is provably unwritten, + /// there is nothing for a wedge to resolve, and wedging is pointless. + /// + /// It is no longer HARMFUL, and the difference is worth stating because the old comment here + /// rested on it: a wedge over a never-written key used to be unclearable, because resolution + /// was a bare read and a read can only ever report absent. The every-attempt rule replaced + /// that with a conditional CREATE, so such a wedge now clears on the next caller's flush by + /// landing the transaction. What remains is that this lane would be blocked until then for no + /// reason at all -- a transient fence blip in the pre-attempt gate would cost the table its + /// write availability, and buy nothing, since there is provably nothing to resolve. + /// + /// The counterexample this argument deliberately excludes: a fence lost or a deadline reached + /// AFTER at least one attempt, and an attempt that COMMITTED but returned under a dropped + /// fence, both report `sent_any` true. Each may have left a durable object, so each keeps + /// wedging. + if (!gave_up->sent_any) + { + { + std::lock_guard lock(rt->state_mutex); + if (rt->lane_state == RefLaneState::Writing && rt->append_attempt + && rt->append_attempt->txn_id == id) + { + rt->append_attempt.reset(); + rt->lane_state = RefLaneState::Ready; + } + } + /// Count it. Before this arm existed these refusals bumped `CASRefAppendWedged`, so + /// removing the wedge also removed the only signal they were happening at all -- and a + /// soak oracle watching that counter fall could not tell "the fix works" from "nothing + /// happened". A separate event keeps both readings available: the wedge counter now means + /// only genuinely ambiguous appends, and this one means availability preserved. + ProfileEvents::increment(ProfileEvents::CASRefAppendPreAttemptRefused); + /// The id is not consumed (INV-1): it was derived from `greatest_applied`, which this + /// refusal leaves exactly as it was, so the next caller on this table derives the SAME id + /// and the durable stream keeps no trace of the refusal. That is the free half of the + /// every-attempt rule -- an attempt that provably sent nothing owes nothing. + complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS ref-log append for namespace '{}' txn {}-{} was refused BEFORE any request was " + "sent — the append lane is NOT wedged (nothing can be durable, so there is " + "nothing to resolve) and the txn id is not consumed (a retry re-derives it)", + ns.string(), id.writer_epoch, id.ref_sequence))); + return false; + } { std::lock_guard lock(rt->state_mutex); - rt->append_attempt.reset(); - rt->lane_state = RefLaneState::Faulted; + if (rt->lane_state == RefLaneState::Writing && rt->append_attempt + && rt->append_attempt->txn_id == id) + rt->lane_state = RefLaneState::Wedged; } - on_impossible_interference(attempt_key, - fmt::format("ref-log append for namespace '{}' txn {}-{} observed a DIFFERENT object already at " - "the id it derived, and it is not an epoch seal of this namespace ({})", - ns.string(), id.writer_epoch, id.ref_sequence, - getCurrentExceptionMessage(/*with_stacktrace*/ false)), - ns.string()); - complete_error(chunk_survivors, write_error); + ProfileEvents::increment(ProfileEvents::CASRefAppendWedged); + complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS ref-log append for namespace '{}' txn {}-{} is UNCERTAIN (last observed {}) — " + "the append lane is wedged until the SAME key resolves durable or a conclusive rejection " + "is observed; this outcome is unproven, not failure", + ns.string(), id.writer_epoch, id.ref_sequence, detail::renderObservation(gave_up->last_seen)))); return false; } - switch (outcome) + { - case CasWriteOutcome::Committed: + /// The log object is durable -- this call's own attempt committed it, or the conflict above + /// proved an earlier attempt of this exact transaction already had. Either way it is not yet + /// admitted to logical history. Publish its exact frontier + /// under the SAME admission generation before any local consequence can make a later id + /// observable or wake a waiter. `Published` and `IdenticalSkip` both prove the contribution + /// durable. `FencedOut`, contention exhaustion, decode failure, or any other unresolved + /// publication leaves the log known durable but uninstalled, which is exactly + /// `NeedsRecovery`; recovery owns resolution of that window. + CkptPublishOutcome frontier_outcome = CkptPublishOutcome::FencedOut; + try + { + frontier_outcome = publishCkptContribution(op, rt->life, prepared->commit_contribution); + } + catch (...) { - /// A durable log object is not yet admitted to logical history. Publish its exact frontier - /// under the SAME admission generation before any local consequence can make a later id - /// observable or wake a waiter. `Published` and `IdenticalSkip` both prove the contribution - /// durable. `FencedOut`, contention exhaustion, decode failure, or any other unresolved - /// publication leaves the log known durable but uninstalled, which is exactly - /// `NeedsRecovery`; recovery owns resolution of that window. - const auto check_commit_admitted = [this, &rt](uint64_t expected_generation) + const std::exception_ptr frontier_error = std::current_exception(); { - check_fence_or_throw(expected_generation); - if (rt->catalog_life_invalidated.load(std::memory_order_acquire) - || rt->superseded_by_remount.load(std::memory_order_acquire)) - throwCasWriteRetryLater(fmt::format( - "CAS namespace '{}': its captured runtime was retired before committed-frontier publication", - rt->life.ns.string())); - }; - CkptPublishOutcome frontier_outcome = CkptPublishOutcome::FencedOut; + std::lock_guard lock(rt->state_mutex); + requireRecovery(*rt, ns, "committed-frontier publication"); + } + complete_error(chunk_survivors, frontier_error); + return false; + } + if (frontier_outcome == CkptPublishOutcome::FencedOut) + { + { + std::lock_guard lock(rt->state_mutex); + requireRecovery(*rt, ns, "committed-frontier publication fence"); + } + complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS ref-log append for namespace '{}': txn {}-{} is durable, but the mount fence " + "moved before its checkpoint frontier was published; the lane needs recovery", + ns.string(), id.writer_epoch, id.ref_sequence))); + return false; + } + + if (carve_hook_for_test) + carve_hook_for_test(CarvePhaseForTest::PostDurableInstall); + bool install_refused = false; + std::exception_ptr install_admission_error; + { + std::lock_guard lock(rt->state_mutex); + /// The checkpoint CAS can succeed and the fence can move before the state lock is + /// reached. Re-present the same admission INSIDE the install hold, immediately before + /// inspecting and swapping the candidate. A stale runtime may leave both log and + /// frontier durable, but it must neither install nor acknowledge them. try { - frontier_outcome = publishCkptContribution( - rt->life, prepared->commit_contribution, admitted_fence_generation, check_commit_admitted); + refuseUnlessAdmitted(op, "committed-frontier publication"); } catch (...) { - const std::exception_ptr frontier_error = std::current_exception(); - { - std::lock_guard lock(rt->state_mutex); - requireRecovery(*rt, ns, "committed-frontier publication"); - } - complete_error(chunk_survivors, frontier_error); - return false; + install_admission_error = std::current_exception(); + requireRecovery(*rt, ns, "post-frontier install admission"); } - if (frontier_outcome == CkptPublishOutcome::FencedOut) + /// Only this leader mutates `rt->state`, so the candidate's base snapshot is still the + /// current one: there is one append-lane leader per table at a time (the `leader_active` + /// baton), the wedge-resolution apply ran earlier in this same flush on this same thread, + /// recovery installs a state exactly once per runtime and has already completed for this + /// table, and every other consumer (readers, the snapshot publisher) only COPIES the + /// state under this mutex. Evaluated here, one statement before the install, and + /// asserted inside it: the comparison allocates nothing, and the identifier is short + /// enough that even the failure path's message is inline-buffered rather than heap + /// allocated, so no build can turn the assert itself into an allocation in the region. + const bool state_unchanged + = !install_admission_error + && rt->lane_state == RefLaneState::Writing + && rt->append_attempt + && rt->append_attempt->txn_id == id + && rt->append_attempt->bytes == active_attempt.bytes + && rt->state.getGreatestApplied() == candidate_base_id; + if (!install_admission_error && !state_unchanged) { - { - std::lock_guard lock(rt->state_mutex); - requireRecovery(*rt, ns, "committed-frontier publication fence"); - } - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}': txn {}-{} is durable, but the mount fence " - "moved before its checkpoint frontier was published; the lane needs recovery", - ns.string(), id.writer_epoch, id.ref_sequence))); - return false; + /// RELEASE-mode counterpart of the `chassert` inside the region below, which is a + /// no-op in a release build and therefore no guard at all for a window that spans a + /// full network round trip. Swapping the candidate in anyway would DISCARD whatever + /// advanced the table. The object is durable and this runtime cannot record it, + /// `LOGICAL_ERROR` here, where the wedge site's identical refusal reports the + /// retry-later class, and the asymmetry is deliberate: THIS one is reachable only by + /// a second writer inside one process -- a bug in this build, which a debug build + /// should abort on and shout about. The wedge site's is reachable by an ordinary + /// remount racing a slow resolution, which is a retryable fact about the world, not a + /// bug. Same refusal, different provenance, so different loudness. + requireRecovery(*rt, ns, "commitRefChunk install"); + install_refused = true; } - - if (carve_hook_for_test) - carve_hook_for_test(CarvePhaseForTest::PostDurableInstall); - bool install_refused = false; - std::exception_ptr install_admission_error; + else if (!install_admission_error) { - std::lock_guard lock(rt->state_mutex); - /// The checkpoint CAS can succeed and the fence can move before the state lock is - /// reached. Re-present the same admission INSIDE the install hold, immediately before - /// inspecting and swapping the candidate. A stale runtime may leave both log and - /// frontier durable, but it must neither install nor acknowledge them. + std::optional completed_attempt; try { - check_commit_admitted(admitted_fence_generation); + DENY_ALLOCATIONS_IN_SCOPE; + if (install_region_probe_for_test) + install_region_probe_for_test(); + chassert(state_unchanged); + rt->state.swap(*candidate); + rt->tail_count_since_snapshot.fetch_add(1, std::memory_order_relaxed); + rt->tail_bytes_since_snapshot.fetch_add(active_attempt.bytes.size(), std::memory_order_relaxed); + rt->append_attempt.swap(completed_attempt); + rt->lane_state = RefLaneState::Ready; } catch (...) { - install_admission_error = std::current_exception(); - requireRecovery(*rt, ns, "post-frontier install admission"); - } - /// Only this leader mutates `rt->state`, so the candidate's base snapshot is still the - /// current one: there is one append-lane leader per table at a time (the `leader_active` - /// baton), the wedge-resolution apply ran earlier in this same flush on this same thread, - /// recovery installs a state exactly once per runtime and has already completed for this - /// table, and every other consumer (readers, the snapshot publisher) only COPIES the - /// state under this mutex. Evaluated here, one statement before the install, and - /// asserted inside it: the comparison allocates nothing, and the identifier is short - /// enough that even the failure path's message is inline-buffered rather than heap - /// allocated, so no build can turn the assert itself into an allocation in the region. - const bool state_unchanged - = !install_admission_error - && rt->lane_state == RefLaneState::Writing - && rt->append_attempt - && rt->append_attempt->txn_id == id - && rt->append_attempt->bytes == active_attempt.bytes - && rt->state.getGreatestApplied() == candidate_base_id; - if (!install_admission_error && !state_unchanged) - { - /// RELEASE-mode counterpart of the `chassert` inside the region below, which is a - /// no-op in a release build and therefore no guard at all for a window that spans a - /// full network round trip. Swapping the candidate in anyway would DISCARD whatever - /// advanced the table. The object is durable and this runtime cannot record it, - /// `LOGICAL_ERROR` here, where the wedge site's identical refusal reports the - /// retry-later class, and the asymmetry is deliberate: THIS one is reachable only by - /// a second writer inside one process -- a bug in this build, which a debug build - /// should abort on and shout about. The wedge site's is reachable by an ordinary - /// remount racing a slow resolution, which is a retryable fact about the world, not a - /// bug. Same refusal, different provenance, so different loudness. requireRecovery(*rt, ns, "commitRefChunk install"); - install_refused = true; + throw; } - else if (!install_admission_error) + candidate.reset(); + completed_attempt.reset(); + try { - std::optional completed_attempt; - try - { - DENY_ALLOCATIONS_IN_SCOPE; - if (install_region_probe_for_test) - install_region_probe_for_test(); - chassert(state_unchanged); - rt->state.swap(*candidate); - rt->tail_count_since_snapshot.fetch_add(1, std::memory_order_relaxed); - rt->tail_bytes_since_snapshot.fetch_add(active_attempt.bytes.size(), std::memory_order_relaxed); - rt->append_attempt.swap(completed_attempt); - rt->lane_state = RefLaneState::Ready; - } - catch (...) - { - requireRecovery(*rt, ns, "commitRefChunk install"); - throw; - } - candidate.reset(); - completed_attempt.reset(); - try - { - rt->state.materializeCommitted(); - } - catch (...) - { - tryLogCurrentException(getLogger("CasPool"), fmt::format( - "CAS ref-log append for namespace '{}': committed txn {}-{} was applied durably, but " - "the post-commit overlay fold failed and was retained coherently for the next flush", - ns.string(), id.writer_epoch, id.ref_sequence)); - } + rt->state.materializeCommitted(); } - } - if (install_admission_error) - { - complete_error(chunk_survivors, install_admission_error); - return false; - } - if (install_refused) - { - complete_error(chunk_survivors, std::make_exception_ptr(Exception( - ErrorCodes::LOGICAL_ERROR, - "CAS ref-log append for namespace '{}': txn {}-{} is durable but this table changed " - "before installation; the lane needs recovery and refuses later writes until replay", - ns.string(), id.writer_epoch, id.ref_sequence))); - return false; - } - if (carve_hook_for_test) - carve_hook_for_test(CarvePhaseForTest::PostInstallPreAck); - ProfileEvents::increment(ProfileEvents::CASRefBatchFlushes); - ProfileEvents::increment(ProfileEvents::CASRefBatchedMutations, chunk_survivors.size()); - { - std::lock_guard g(ref_queue_mutex); - for (const auto & it : chunk_survivors) + catch (...) { - it->committed_id = id; - it->done = true; + tryLogCurrentException(getLogger("CasPool"), fmt::format( + "CAS ref-log append for namespace '{}': committed txn {}-{} was applied durably, but " + "the post-commit overlay fold failed and was retained coherently for the next flush", + ns.string(), id.writer_epoch, id.ref_sequence)); } - rt->cv.notify_all(); } - /// The threshold trigger -- off the lane, - /// dispatched AFTER waking every waiter above so this commit's own callers are never - /// delayed by it. Per chunk (spec §3): each committed chunk schedules its own publication, - /// and settlement coalesces the triggers so a mid-tenure publisher never suppresses a later - /// chunk (`settleSnapshotPublish`). - maybeScheduleSnapshotPublish(ns, rt); - return true; } - case CasWriteOutcome::DefiniteFailure: + if (install_admission_error) { - /// Proof that nothing became durable returns the exact attempt to `Ready`. - { - std::lock_guard lock(rt->state_mutex); - if (rt->lane_state == RefLaneState::Writing && rt->append_attempt - && rt->append_attempt->txn_id == id) - { - rt->append_attempt.reset(); - rt->lane_state = RefLaneState::Ready; - } - } - ProfileEvents::increment(ProfileEvents::CASRefAppendDefiniteFailure); - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}' definitively failed (non-retryable rejection); " - "cached state is unchanged and txn id {}-{} was never used (a retry re-derives it)", + complete_error(chunk_survivors, install_admission_error); + return false; + } + if (install_refused) + { + complete_error(chunk_survivors, std::make_exception_ptr(Exception( + ErrorCodes::LOGICAL_ERROR, + "CAS ref-log append for namespace '{}': txn {}-{} is durable but this table changed " + "before installation; the lane needs recovery and refuses later writes until replay", ns.string(), id.writer_epoch, id.ref_sequence))); return false; } - case CasWriteOutcome::Unresolved: + if (carve_hook_for_test) + carve_hook_for_test(CarvePhaseForTest::PostInstallPreAck); + ProfileEvents::increment(ProfileEvents::CASRefBatchFlushes); + ProfileEvents::increment(ProfileEvents::CASRefBatchedMutations, chunk_survivors.size()); { - /// The ONE `Unresolved` shape that must NOT wedge (finding #37 defect 3). The wedge exists - /// because an `Unresolved` PUT MAY HAVE LANDED: the durable log may or may not contain this - /// transaction, only `resolveByExactGet` on that exact key can settle it, and until it does, - /// minting a later id would build on a state that may be missing a landed transaction. All of - /// that presupposes an attempt was SENT. - /// - /// `unresolvedProvesNothingWasSent` is true only for `NoAttemptSent`, which - /// `putIfAbsentControlled` reports only when a pre-attempt gate -- the mount fence or the - /// operation deadline -- rejected while `attempts_sent == 0`, i.e. strictly before the first - /// `backend->putIfAbsent`. Nothing reached the network, so the key is provably unwritten: - /// there is nothing for a wedge to resolve, and wedging is pointless. - /// - /// It is no longer HARMFUL, and the difference is worth stating because the old comment here - /// rested on it: a wedge over a never-written key used to be unclearable, because resolution - /// was a bare read and a read can only ever report absent. The every-attempt rule replaced - /// that with a conditional CREATE, so such a wedge now clears on the next caller's flush by - /// landing the transaction. What remains is that this lane would be blocked until then for no - /// reason at all -- a transient fence blip in the pre-attempt gate would cost the table its - /// write availability, and buy nothing, since there is provably nothing to resolve. - /// - /// The counterexample this argument deliberately excludes: a fence lost or a deadline reached - /// AFTER at least one attempt is `FenceLostMidWay`/`DeadlineMidWay`, and an attempt that - /// COMMITTED but returned under a dropped fence is `FenceLostPostWrite`. Each of those may - /// have left a durable object, so each keeps wedging -- as does anything a future contributor - /// adds to the enum without classifying it (see the predicate's allow-list construction). - if (unresolvedProvesNothingWasSent(unresolved_reason)) - { - { - std::lock_guard lock(rt->state_mutex); - if (rt->lane_state == RefLaneState::Writing && rt->append_attempt - && rt->append_attempt->txn_id == id) - { - rt->append_attempt.reset(); - rt->lane_state = RefLaneState::Ready; - } - } - /// Count it. Before this arm existed these refusals bumped `CASRefAppendWedged`, so - /// removing the wedge also removed the only signal they were happening at all -- and a - /// soak oracle watching that counter fall could not tell "the fix works" from "nothing - /// happened". A separate event keeps both readings available: the wedge counter now means - /// only genuinely ambiguous appends, and this one means availability preserved. - ProfileEvents::increment(ProfileEvents::CASRefAppendPreAttemptRefused); - /// The id is not consumed (INV-1): it was derived from `greatest_applied`, which this - /// refusal leaves exactly as it was, so the next caller on this table derives the SAME id - /// and the durable stream keeps no trace of the refusal. That is the free half of the - /// every-attempt rule -- an attempt that provably sent nothing owes nothing. - /// The installed attempt is retired below; no request was sent. - /// and is what makes the genuinely ambiguous path below allocation-free. - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}' txn {}-{} was refused BEFORE any request was " - "sent ({}) — the append lane is NOT wedged (nothing can be durable, so there is " - "nothing to resolve) and the txn id is not consumed (a retry re-derives it)", - ns.string(), id.writer_epoch, id.ref_sequence, - describeUnresolvedReason(unresolved_reason)))); - return false; - } + std::lock_guard g(ref_queue_mutex); + for (const auto & it : chunk_survivors) { - std::lock_guard lock(rt->state_mutex); - if (rt->lane_state == RefLaneState::Writing && rt->append_attempt - && rt->append_attempt->txn_id == id) - rt->lane_state = RefLaneState::Wedged; + it->committed_id = id; + it->done = true; } - ProfileEvents::increment(ProfileEvents::CASRefAppendWedged); - complete_error(chunk_survivors, makeCasWriteRetryLaterExceptionPtr(fmt::format( - "CAS ref-log append for namespace '{}' txn {}-{} is UNCERTAIN ({}) — " - "the append lane is wedged until the SAME key resolves durable or a conclusive rejection " - "is observed; this outcome is unproven, not failure", - ns.string(), id.writer_epoch, id.ref_sequence, - describeUnresolvedReason(unresolved_reason)))); - return false; + rt->cv.notify_all(); } + /// The threshold trigger -- off the lane, + /// dispatched AFTER waking every waiter above so this commit's own callers are never + /// delayed by it. Per chunk: each committed chunk schedules its own publication, + /// and settlement coalesces the triggers so a mid-tenure publisher never suppresses a later + /// chunk (`settleSnapshotPublish`). + maybeScheduleSnapshotPublish(ns, rt); + return true; } - /// Unreachable: the switch above covers every `CasWriteOutcome`. Kept explicit so the function has a - /// defined return on all control-flow paths. - return false; } bool CasRefLedger::hasStateBearingSnapshotCandidateUnderStateLock(const RefTableRuntime & rt) const @@ -4408,17 +4395,10 @@ void clampedCounterSub(std::atomic & counter, uint64_t amount) } -CkptPublishOutcome CasRefLedger::publishCkptContribution(const NamespaceLifeId & life, const RefCkpt & contribution, - uint64_t admitted_generation, - const std::function & check_admission, - const std::function & admit_request) +CkptPublishOutcome CasRefLedger::publishCkptContribution(CasOperation & op, const NamespaceLifeId & life, + const RefCkpt & contribution) { - /// The retry window is the SAME budget every other CAS operation of this ledger rides, measured on - /// the ledger's own injectable boot clock -- so a test drives the exhaustion arm without sleeping, - /// and a VM suspend cannot shorten it. - const CkptDeadline deadline{boot_ms_now_fn, boot_ms_now_fn() + cas_request_budget.operation_deadline_ms}; - const CkptPublishOutcome outcome = publishCkpt( - backend, layout, life, contribution, admitted_generation, check_admission, deadline, admit_request); + const CkptPublishOutcome outcome = publishCkpt(op, layout, life, contribution); if (outcome == CkptPublishOutcome::Published) ProfileEvents::increment(ProfileEvents::CASRefCheckpointPublished); else if (outcome == CkptPublishOutcome::IdenticalSkip) @@ -4453,13 +4433,12 @@ bool CasRefLedger::tryPublishSnapshotAndAdvanceCheckpointOnceOnRuntimeImpl( /// recheck discipline -- the same value at every site), so a publish admitted under an incarnation /// that has since been replaced can advance nothing. const uint64_t admitted_generation = rt->admitted_fence_generation; - const auto runtime_still_admitted = [this, &rt, admitted_generation] + CasOperation op = mount_requests.resume(admitted_generation, [&rt] { return !rt->catalog_life_invalidated.load(std::memory_order_acquire) - && !rt->superseded_by_remount.load(std::memory_order_acquire) - && fence_ok_fn() - && fence_generation_fn() == admitted_generation; - }; + && !rt->superseded_by_remount.load(std::memory_order_acquire); + }); + const auto runtime_still_admitted = [&op] { return op.admitted(); }; /// ONE copy of the live state, at a transaction boundary -- no /// replay, no per-entry retention. The tail counters are captured in the SAME critical section so @@ -4546,11 +4525,22 @@ bool CasRefLedger::tryPublishSnapshotAndAdvanceCheckpointOnceOnRuntimeImpl( return false; } const String key = layout.refSnapshotKey(rt->life, candidate_x); - const CasWriteOutcome outcome - = ref_request_controller->putIfAbsentControlled(key, bytes, runtime_still_admitted); - if (outcome != CasWriteOutcome::Committed) + const WriteResult put = op.create(key, bytes, Retry::standard()); + if (const auto * conflict = std::get_if(&put)) + { + /// A snapshot key names the exact state it encodes, so a re-run of this publish -- and only a + /// re-run -- can find its own identical body already there. DIFFERENT bytes under this mount's + /// exclusive lease are corruption, and the publisher must not paper over them with a backoff. + const auto * occupant = std::get_if(&conflict->seen); + if (!occupant || occupant->bytes != bytes) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS ref table '{}': a DIFFERENT object occupies snapshot {}-{} (observed {})", + ns.string(), candidate_x.writer_epoch, candidate_x.ref_sequence, + detail::renderObservation(conflict->seen)); + } + else if (!std::holds_alternative(put)) { - /// DefiniteFailure/Unresolved: DO NOT prune (no durable covering snapshot -- pruning the tail + /// Refused or unresolved: DO NOT prune (no durable covering snapshot -- pruning the tail /// without one is data loss). Arm the bounded per-table backoff so the read path does not /// re-dispatch this full-snapshot encode+PUT until it elapses -- the read-triggered PUT-storm /// latch breaker. A later trigger past the deadline retries. @@ -4572,33 +4562,24 @@ bool CasRefLedger::tryPublishSnapshotAndAdvanceCheckpointOnceOnRuntimeImpl( /// know has already recorded -- in either order. /// /// A checkpoint that does NOT advance leaves the attempt unadopted: the backoff is armed and this - /// returns false, so a later trigger re-runs the whole publish. The re-run's body PUT resolves to - /// `Committed` against its own identical bytes, so retrying costs one conditional PUT and not a - /// second snapshot. Adopting instead would mark this snapshot as the newest -- suppressing every - /// later publish for it -- while the checkpoint still pointed below it, leaving recovery replaying - /// from an older base with nothing scheduled to fix it. + /// returns false, so a later trigger re-runs the whole publish. The re-run's body create meets its + /// own identical bytes as a conflict, which the occupant compare above accepts, so retrying costs + /// one conditional PUT and not a second snapshot. Adopting instead would mark this snapshot as the + /// newest -- suppressing every later publish for it -- while the checkpoint still pointed below it, + /// leaving recovery replaying from an older base with nothing scheduled to fix it. bool ckpt_advanced = false; if (!runtime_still_admitted()) return false; - const auto check_runtime_admission = [this, &rt](uint64_t generation) - { - if (snapshot_before_ckpt_cas_hook_for_test) - snapshot_before_ckpt_cas_hook_for_test(); - check_fence_or_throw(generation); - if (rt->catalog_life_invalidated.load(std::memory_order_acquire) - || rt->superseded_by_remount.load(std::memory_order_acquire)) - throwCasWriteRetryLater(fmt::format( - "CAS namespace '{}': its captured runtime was retired before checkpoint publication", - rt->life.ns.string())); - }; + if (snapshot_before_ckpt_cas_hook_for_test) + snapshot_before_ckpt_cas_hook_for_test(); try { - ckpt_advanced = publishCkptContribution(rt->life, RefCkpt{.life_epoch = std::nullopt, - .committed_through = candidate_x, - .checkpoint_snapshot_id = candidate_x, - .last_epoch_seal = std::nullopt}, - admitted_generation, - check_runtime_admission) != CkptPublishOutcome::FencedOut; + ckpt_advanced = publishCkptContribution(op, rt->life, + RefCkpt{.life_epoch = std::nullopt, + .committed_through = candidate_x, + .checkpoint_snapshot_id = candidate_x, + .last_epoch_seal = std::nullopt}) + != CkptPublishOutcome::FencedOut; } catch (...) { @@ -4924,7 +4905,7 @@ NamespaceLifeId CasRefLedger::namespaceLife(const RootNamespace & ns) auto rt = lookupRefTableRuntime(ns); if (rt) { - check_fence_or_throw(rt->admitted_fence_generation); + refuseUnlessAdmitted(mount_requests.resume(rt->admitted_fence_generation), "resident namespace life"); bool removal_closed = false; { std::lock_guard queue_lock(ref_queue_mutex); @@ -4934,7 +4915,8 @@ NamespaceLifeId CasRefLedger::namespaceLife(const RootNamespace & ns) { /// A lost erase response can leave only the detached predecessor's close bit. Reconcile /// before refusing so an absent/replaced row frees the logical name without rebinding it. - reconcileCatalogCut(CasRefCatalog::read(backend, layout)); + CasOperation reconcile_op = mount_requests.resume(rt->admitted_fence_generation); + reconcileCatalogCut(CasRefCatalog::read(reconcile_op, layout)); const auto refreshed = lookupRefTableRuntime(ns); if (refreshed == rt) throwCasWriteRetryLater(fmt::format( @@ -4951,9 +4933,9 @@ NamespaceLifeId CasRefLedger::namespaceLife(const RootNamespace & ns) /// A cold mutation observes or births the durable identity before allocating any local state. const uint64_t admitted_generation = fence_generation_fn(); - check_fence_or_throw(admitted_generation); - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + CasOperation op = mount_requests.resume(admitted_generation); + const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(op, layout); + refuseUnlessAdmitted(op, "cold mutable namespace admission"); const auto entry_it = std::find_if(catalog.catalog.entries.begin(), catalog.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); if (entry_it != catalog.catalog.entries.end() && entry_it->state == NsState::Removing) @@ -4965,7 +4947,7 @@ NamespaceLifeId CasRefLedger::namespaceLife(const RootNamespace & ns) = entry_it != catalog.catalog.entries.end() && entry_it->state == NsState::Live ? NamespaceLifeId::fromCatalogEntry(entry_it->ns, entry_it->incarnation) : resolveNamespaceLife(ns, admitted_generation, live_epoch_fn()); - check_fence_or_throw(admitted_generation); + refuseUnlessAdmitted(op, "cold mutable namespace admission, after life resolution"); rt = acquireRefTableRuntime(life, admitted_generation); ensureRefTableRecovered(ns, *rt); return rt->life; @@ -4997,7 +4979,7 @@ bool CasRefLedger::namespaceStillLogicallyPresent(const RootNamespace & ns) /// short of that falls through to the exact cold-path observation below. if (const auto current = lookupRefTableRuntime(ns)) { - check_fence_or_throw(current->admitted_fence_generation); + refuseUnlessAdmitted(mount_requests.resume(current->admitted_fence_generation), "resident namespace presence"); bool closed = false; { std::lock_guard queue_lock(ref_queue_mutex); @@ -5018,9 +5000,9 @@ bool CasRefLedger::namespaceStillLogicallyPresent(const RootNamespace & ns) /// the one answer that must never be manufactured by a race, so it alone is re-confirmed by a /// second read of this namespace's row before being trusted. const uint64_t admitted_generation = fence_generation_fn(); - check_fence_or_throw(admitted_generation); - const CasRefCatalog::Snapshot first_catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + CasOperation op = mount_requests.resume(admitted_generation); + const CasRefCatalog::Snapshot first_catalog = CasRefCatalog::read(op, layout); + refuseUnlessAdmitted(op, "namespace presence probe"); if (namespace_presence_probe_after_first_read_hook_for_test) namespace_presence_probe_after_first_read_hook_for_test(); const auto find_entry = [&ns](const CasRefCatalog::Snapshot & snap) -> const CatalogEntry * @@ -5040,8 +5022,8 @@ bool CasRefLedger::namespaceStillLogicallyPresent(const RootNamespace & ns) /// continuously) while adding nothing to this row's proof. A row that appears in between /// answers present: `true` is always the safe direction, and the caller's next poll runs the /// full state dispatch against a fresh observation. - const CasRefCatalog::Snapshot second_catalog = CasRefCatalog::read(backend, layout); - check_fence_or_throw(admitted_generation); + const CasRefCatalog::Snapshot second_catalog = CasRefCatalog::read(op, layout); + refuseUnlessAdmitted(op, "namespace presence probe"); if (find_entry(second_catalog)) return true; return false; /// no catalog row in two atomic observations: proven absent @@ -5075,8 +5057,8 @@ bool CasRefLedger::namespaceStillLogicallyPresent(const RootNamespace & ns) if (namespace_presence_probe_after_terminal_proven_hook_for_test) namespace_presence_probe_after_terminal_proven_hook_for_test(); - check_fence_or_throw(admitted_generation); - const CasRefCatalog::Snapshot post_terminal_catalog = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot post_terminal_catalog = CasRefCatalog::read(op, layout); + refuseUnlessAdmitted(op, "namespace presence probe"); const CatalogEntry * post_terminal_entry = find_entry(post_terminal_catalog); if (!post_terminal_entry || post_terminal_entry->incarnation == entry->incarnation) return false; /// terminal durably proven and nothing has since occupied `ns` under a new life @@ -5101,7 +5083,8 @@ DropNamespaceStats CasRefLedger::dropNamespaceImpl( /// class shares the bigger complete-table byte budget (encodeRefLogTxn's own `checkBudget`, keyed /// off the presence of a `RemoveNamespace` op) and is exempt from the ordinary per-op admission /// check (it only ever shrinks state; see `flushRefBatch`'s `state_growing` filter). - const CasRefCatalog::Snapshot initial_catalog = CasRefCatalog::read(backend, layout); + CasOperation initial_op = mount_requests.admit(); + const CasRefCatalog::Snapshot initial_catalog = CasRefCatalog::read(initial_op, layout); const auto initial_it = std::find_if(initial_catalog.catalog.entries.begin(), initial_catalog.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); if (initial_it == initial_catalog.catalog.entries.end()) @@ -5114,15 +5097,14 @@ DropNamespaceStats CasRefLedger::dropNamespaceImpl( if (initial_it->state == NsState::Creating) { const CatalogEntry & observed = *initial_it; - const uint64_t admitted_generation = fence_generation_fn(); + CasOperation cancel_op = mount_requests.resume(fence_generation_fn()); switch (CasRefCatalog::cancelStalledCreating( - backend, layout, observed, - [this](const CreatorFence & creator) + cancel_op, layout, observed, + [&](const CreatorFence & creator) { return isCreatorFenceTerminal( - backend, layout, creator.server_root_id, creator.writer_epoch); - }, - admitted_generation, check_fence_or_throw)) + cancel_op, layout, creator.server_root_id, creator.writer_epoch); + })) { case CasRefCatalog::StalledCreatingCancelOutcome::Cancelled: invalidateRemovedCatalogLife(NamespaceLifeId::fromCatalogEntry(observed.ns, observed.incarnation)); @@ -5171,13 +5153,14 @@ DropNamespaceStats CasRefLedger::dropNamespaceImpl( } const uint64_t admitted_generation = fence_generation_fn(); + CasOperation removal_op = mount_requests.resume(admitted_generation); std::optional observed_live; if (initial_it->state == NsState::Live) observed_live = *initial_it; bool removing_durable = false; try { - const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(removal_op, layout); const auto entry_it = std::find_if(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); if (entry_it == snapshot.catalog.entries.end()) @@ -5199,12 +5182,11 @@ DropNamespaceStats CasRefLedger::dropNamespaceImpl( { observed_live = *entry_it; uint64_t removal_started_round = 0; - if (const auto got = backend.get(layout.gcStateKey())) + if (const auto got = removal_op.read(layout.gcStateKey(), Retry::standard())) removal_started_round = decodeGcState(got->bytes).round; switch (CasRefCatalog::beginRemoving( - backend, layout, *observed_live, removal_started_round, - admitted_generation, check_fence_or_throw)) + removal_op, layout, *observed_live, removal_started_round)) { case CasRefCatalog::BeginRemovingOutcome::Transitioned: case CasRefCatalog::BeginRemovingOutcome::AlreadyRemoving: @@ -5235,7 +5217,7 @@ DropNamespaceStats CasRefLedger::dropNamespaceImpl( /// changed or fenced case remains closed (fail-close) and propagates the original error. try { - const CasRefCatalog::Snapshot fresh = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot fresh = CasRefCatalog::read(removal_op, layout); const auto fresh_it = std::find_if(fresh.catalog.entries.begin(), fresh.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); if (observed_live @@ -5247,7 +5229,7 @@ DropNamespaceStats CasRefLedger::dropNamespaceImpl( } else if (observed_live && fresh_it != fresh.catalog.entries.end() && *fresh_it == *observed_live) { - check_fence_or_throw(admitted_generation); + refuseUnlessAdmitted(removal_op, "reopening the removal lane"); std::lock_guard queue_lock(ref_queue_mutex); rt->removal_admission_closed = false; rt->cv.notify_all(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h index 60c3066b29ed..f8833b3f6d09 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h @@ -1,6 +1,8 @@ #pragma once #include -#include +#include +#include +#include #include #include #include @@ -89,7 +91,9 @@ class CasRefLedger { public: CasRefLedger( - BackendPtr backend_ptr, + /// The mount plane. Every request this ledger makes is admitted on it, so a ref-lane write and + /// a mount-lease renewal are measured against the same fence and the same clock. + CasRequests & mount_requests_, const Layout & layout_, RefLedgerConfig config_, const CasEventSink & event_sink_, @@ -100,23 +104,19 @@ class CasRefLedger /// unlike the mount-state functions below, because it is a fixed identity for this ledger's /// whole lifetime (mirrors `CasMountRuntime`'s own by-value `server_root_id`). String server_root_id_, - /// Monotonic mount clock used by the retry controller; it may be empty when the controller's - /// default clock is appropriate. - std::function controller_boot_ms_fn, /// Callbacks into mount and watermark state owned by `Pool`, bound for this ledger's lifetime: std::function live_epoch_fn_, std::function fence_ok_fn_, - /// The two fence-GENERATION primitives (`CasMountRuntime::fenceGeneration`/`checkFenceOrThrow`), - /// injected exactly as `CasPlainObjects` takes them. `fence_ok_fn` above answers "may this mount - /// write AT ALL, right now"; these two answer the different question an append lane must ask - /// across an I/O window: "is this still the SAME mount incarnation that admitted the transaction - /// I am about to act on?" A wedge captures the generation at admission and presents it back on - /// every later retry and before every install, so a result that returns after a fence loss or a - /// re-arm is inert for the superseded runtime instead of installing a stale view (spec §3, - /// "the mount-fence generation is captured at admission and required on every slot-occupy and - /// install"). + /// The fence-GENERATION primitive (`CasMountRuntime::fenceGeneration`), injected exactly as + /// `CasPlainObjects` takes it. `fence_ok_fn` above answers "may this mount write AT ALL, right + /// now"; this answers the different question an append lane must ask across an I/O window: "is + /// this still the SAME mount incarnation that admitted the transaction I am about to act on?" A + /// wedge captures the generation at admission and presents it back -- through `mount_requests`, + /// by resuming an operation under it -- on every later retry and before every install, so a + /// result that returns after a fence loss or a re-arm is inert for the superseded runtime instead + /// of installing a stale view (spec §3, "the mount-fence generation is captured at admission and + /// required on every slot-occupy and install"). std::function fence_generation_fn_, - std::function check_fence_or_throw_, std::function boot_ms_now_fn_, std::function may_mutate_, std::function &)> on_impossible_interference_, @@ -288,26 +288,25 @@ class CasRefLedger /// mutation can appear after shutdown has taken its snapshot. bool drainRefLanesForShutdown(uint64_t wait_budget_ms); - /// Performs a staged conditional create through the ledger's retry controller and append-fence - /// predicate. Callers do not access either dependency directly, so every attempt observes the same - /// mount admission rule. - CasWriteOutcome stagingPutIfAbsent(std::string_view key, std::string_view bytes, Token * out_token); + /// A staged conditional create on the mount plane. Callers reach the store only through these, so + /// every attempt observes the same mount admission rule. `Conflict` names what occupies the key; a + /// caller whose key is content-addressed decides for itself whether a different occupant is + /// corruption. + WriteResult stagingPutIfAbsent(const String & key, const String & bytes); - /// Same retry/fence policy as `stagingPutIfAbsent`, for a MUTABLE If-Match overwrite whose bytes - /// are deterministic (safe for GET-based resolution). - CasOverwriteResult stagingConditionalOverwrite(std::string_view key, std::string_view bytes, const Token & expected); + /// The If-Match sibling of `stagingPutIfAbsent`, for a MUTABLE marker. + WriteResult stagingConditionalOverwrite(const String & key, const String & bytes, const Incarnation & expected); - /// Same retry/fence policy as `stagingPutIfAbsent`, for a MUTABLE marker where an existing - /// DIFFERENT value at the key is a normal Conflict outcome, not corruption (see - /// `CasRequestController::putIfAbsentControlledMutable`). - CasOverwriteResult stagingPutIfAbsentMutable(std::string_view key, std::string_view bytes); + /// Create-if-absent for a MUTABLE marker, where a DIFFERENT value already at the key is an + /// ordinary `Conflict` for the caller to act on rather than corruption. + WriteResult stagingPutIfAbsentMutable(const String & key, const String & bytes); /// Hooks required by `EventEmitter`: events are delivered to the injected sink when one is present. bool hasEventSink() const noexcept { return static_cast(event_sink); } void emitEvent(CasEvent && e) const { if (event_sink) event_sink(std::move(e)); } - /// Replaces the retry controller's delay seam for deterministic tests; production callers leave it - /// untouched. + /// Replaces the mount plane's inter-attempt delay seam for deterministic tests; production callers + /// leave it untouched. void setCasRetrySleepForTest(std::function sleep_fn); /// Replaces only ref-table recovery's token-aware retry delay seam for deterministic tests. @@ -407,6 +406,12 @@ class CasRefLedger /// `install_region_probe_for_test`). void setInstallRegionProbeForTest(std::function probe) { install_region_probe_for_test = std::move(probe); } + /// Installs a probe at `ensureRefTableRecovered`'s step 8 -- the LAST admission recheck before a + /// materialized recovery result installs, after the final authority read and O(N) materialization + /// have already run. A test pausing here and then latching a stop token observes whether that stop + /// is honored before install (see `recovery_install_probe_for_test`). + void setRecoveryInstallProbeForTest(std::function probe) { recovery_install_probe_for_test = std::move(probe); } + /// Installs the pre-tenure fault seam (see `ref_pre_tenure_hook_for_test`). void setRefPreTenureHookForTest(std::function hook) { ref_pre_tenure_hook_for_test = std::move(hook); } @@ -630,10 +635,10 @@ class CasRefLedger String bytes; /// `CasMountRuntime::fenceGeneration()` as read at this transaction's ADMISSION -- the same /// critical section that snapshotted the state and derived the id, i.e. one atomic reading of - /// "what this attempt was allowed to do". Every later `slotOccupy` retry is gated on THIS value - /// (never the current one), and every install is preceded by presenting it back through - /// `checkFenceOrThrow`: a retry admitted under a dead incarnation must send nothing, and a - /// result that returns after a fence bump/re-arm must install nothing. + /// "what this attempt was allowed to do". Every later retry of this attempt is admitted by + /// resuming an operation under THIS value (never the current one), and every install is + /// preceded by presenting it back: a retry admitted under a dead incarnation must send + /// nothing, and a result that returns after a fence bump/re-arm must install nothing. uint64_t admitted_fence_generation = 0; }; @@ -697,7 +702,7 @@ class CasRefLedger private: /// Injected storage and mount environment. The member order is part of construction/destruction /// behavior because the callbacks and references are used by the runtime owned below. - Backend & backend; + CasRequests & mount_requests; const Layout & layout; RefLedgerConfig config; const CasEventSink & event_sink; @@ -708,7 +713,6 @@ class CasRefLedger std::function live_epoch_fn; std::function fence_ok_fn; std::function fence_generation_fn; - std::function check_fence_or_throw; std::function boot_ms_now_fn; std::function may_mutate; std::function &)> on_impossible_interference; @@ -745,7 +749,7 @@ class CasRefLedger /// `ref_queue_mutex` (which only ever guards `pending`/`carved`/`leader_active`) so a reader (resolveRef/ /// listRefs) can observe `state` without contending with the flush leader's network round trip -- /// the leader only holds `state_mutex` for the brief copy-out-before-validate and the - /// apply-after-commit steps, never for the `putIfAbsentControlled` call itself. + /// apply-after-commit steps, never for the durable write itself. struct RefTableRuntime { /// An allocator can reuse an evicted predecessor's address for its successor. This monotone id @@ -1000,13 +1004,6 @@ class CasRefLedger return rt.state.nextTxnId(live_epoch_fn()); } - /// The CAS-owned retry controller this Pool's ref-log writer path uses for every conditional - /// log/snapshot `PUT` and uncertain-result resolution. It is also shared by the part-manifest - /// write and mutable freshness-meta writes. The controller is stateless per call (immutable - /// budget/clock/sleep — the sleep fn mutates only through the test-only seam, before traffic), so - /// concurrent lanes and builds use the one instance safely. - std::unique_ptr ref_request_controller; - /// Test-only hook called before a compatible append batch is carved; null in production. std::function ref_pre_carve_hook_for_test; @@ -1030,6 +1027,8 @@ class CasRefLedger /// otherwise non-throwing regions. A test that installs a throwing probe must therefore disarm it /// after the region it targets, or every later install throws too. Null in production. std::function install_region_probe_for_test; + /// See `setRecoveryInstallProbeForTest`. Null in production. + std::function recovery_install_probe_for_test; std::function append_after_runtime_capture_hook_for_test; std::function read_before_state_lock_hook_for_test; std::function readable_catalog_after_observation_hook_for_test; @@ -1102,8 +1101,8 @@ class CasRefLedger /// cleanup could account for. Everything terminal throws. /// /// `admitted_generation` is the ONE fence generation this whole recovery was admitted under: the - /// walk presents it to every `slotOccupy` and to the `_ckpt` CAS, and the caller presents the same - /// value once more immediately before installing. + /// walk resumes its operation under it, so every request the walk makes is measured against it, and + /// the caller presents the same value once more immediately before installing. /// `retained_attempt` is copied under `state_mutex` before the unlocked walk. It is evidence from /// this runtime's admitted writer, not a second recovery authority: only the exact slot it names /// is compared byte-for-byte, and a successor seal remains the existing conclusive-loss case. @@ -1121,8 +1120,7 @@ class CasRefLedger /// independently disqualify it. Throws; remount cancellation raises the retry-later class and /// LATCHES through `cancelled` so the caller's transient loop does not re-drive it. /// - /// The FENCE is deliberately absent: it gates the three sites that spend it (every `slotOccupy`, the - /// `_ckpt` CAS, the install), not every read. See the definition for why. + /// The FENCE is deliberately absent: the walk's own operation carries it. See the definition. void checkRecoveryStillAdmitted( const RootNamespace & ns, RefTableRuntime & rt, bool & cancelled, const std::optional & token = std::nullopt) const; @@ -1173,8 +1171,8 @@ class CasRefLedger }; /// ONE bounded resolution attempt for `rt`'s outstanding wedge (spec INV-1's every-attempt rule): - /// at most one `slotOccupy(wedge.key, wedge.bytes, ...)` per calling flush, gated on the wedge's - /// ORIGINAL `admitted_fence_generation` rather than the current one. There is deliberately NO + /// at most one conditional create of the wedge's exact key and bytes per calling flush, admitted by + /// resuming under the wedge's ORIGINAL `admitted_fence_generation` rather than the current one. There is deliberately NO /// background retry thread and no deadline-resetting loop: a permanently quiet wedged namespace /// waits for its next caller or for a remount, which is acceptable precisely because the wedged /// operation was never acknowledged. @@ -1186,8 +1184,8 @@ class CasRefLedger /// "absent", which is not a rejection: the earlier ambiguous attempt could still land afterwards. /// /// Post-I/O recheck: the outcome is adjudicated on an I/O result, so before ANY action follows from - /// it (adopt, acknowledge, unwedge, fail the survivors) this re-acquires `state_mutex`, presents - /// `admitted_fence_generation` back through `checkFenceOrThrow`, and compares the full wedge + /// it (adopt, acknowledge, unwedge, fail the survivors) this re-acquires `state_mutex`, compares + /// `admitted_fence_generation` against the fence's CURRENT generation, and compares the full wedge /// identity against what is still installed. A result that returns after a fence bump/re-arm, or /// after the wedge it belonged to was replaced, is INERT for this runtime. WedgeResolutionResult resolveWedgeOnce( @@ -1257,10 +1255,15 @@ class CasRefLedger /// - `runRecoveryWalkOnce` contributes `last_epoch_seal` once its own CAS-walk minted or adopted /// one -- it is the only writer that mints seals, so it is the only writer that can record /// where the chain now ends. - CkptPublishOutcome publishCkptContribution(const NamespaceLifeId & life, const RefCkpt & contribution, - uint64_t admitted_generation, - const std::function & check_admission, - const std::function & admit_request = {}); + CkptPublishOutcome publishCkptContribution(CasOperation & op, const NamespaceLifeId & life, + const RefCkpt & contribution); + + /// The verdict points that guard a DECISION rather than a request: the engine refuses a request on + /// its own, but a result already in hand must not be acted on once `op` has stopped being admitted. + /// `op.admitted()` folds every reason together -- a moved mount incarnation, a lease with too little + /// time left, or a caller-supplied liveness term that has since gone false -- because none of them + /// leaves the caller anything more specific to act on than "this admission no longer holds". + void refuseUnlessAdmitted(const CasOperation & op, std::string_view what) const; /// Common candidate predicate for scheduler admission and execution after capture. Caller holds /// `rt.state_mutex`; an epoch seal is not state-bearing and cannot be snapshotted. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp index 5f99e880c968..59210a22092d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp @@ -854,7 +854,7 @@ RefCleanupPlan planRefCleanup(const RefTableListing & listing, const RefTxnId & return plan; } -EpochCrossResult crossEpochFromSeal(Backend & backend, const Layout & layout, const RootNamespace & ns, +EpochCrossResult crossEpochFromSeal(CasOperation & op, const Layout & layout, const RootNamespace & ns, const RefTxnId & from_seal, std::optional seal_proven, const RefTxnId & witness, const NamespaceLifeId & life) { @@ -882,7 +882,7 @@ EpochCrossResult crossEpochFromSeal(Backend & backend, const Layout & layout, co { const RefTxnId start{target_epoch, 1}; result.probed = start; - const auto body = backend.get(layout.refLogKey(life, start)); + const auto body = op.read(layout.refLogKey(life, start), Retry::standard()); if (!body) { ++result.absent_probes; @@ -952,8 +952,7 @@ std::optional nextRefLogIdWithinCommittedFrontier( } CheckpointSnapshotBase readCheckpointSnapshotBase( - Backend & backend, const Layout & layout, const NamespaceLifeId & life, const RefCkpt & checkpoint, - const std::function & admit_request) + CasOperation & op, const Layout & layout, const NamespaceLifeId & life, const RefCkpt & checkpoint) { const RootNamespace & ns = life.ns; if (!checkpoint.checkpoint_snapshot_id) @@ -969,9 +968,7 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( ns.string()); } const RefTxnId snapshot_id = *checkpoint.checkpoint_snapshot_id; - if (admit_request) - admit_request(); - const auto log = backend.get(layout.refLogKey(life, snapshot_id)); + const auto log = op.read(layout.refLogKey(life, snapshot_id), Retry::standard()); if (!log) { throw Exception(ErrorCodes::CORRUPTED_DATA, @@ -1007,9 +1004,7 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( if (base_txn.prev_epoch_seal) { predecessor_seal_id = *base_txn.prev_epoch_seal; - if (admit_request) - admit_request(); - const auto predecessor = backend.get(layout.refLogKey(life, *predecessor_seal_id)); + const auto predecessor = op.read(layout.refLogKey(life, *predecessor_seal_id), Retry::standard()); if (!predecessor) { throw Exception(ErrorCodes::CORRUPTED_DATA, @@ -1030,9 +1025,7 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( } } - if (admit_request) - admit_request(); - const auto snapshot = backend.get(layout.refSnapshotKey(life, snapshot_id)); + const auto snapshot = op.read(layout.refSnapshotKey(life, snapshot_id), Retry::standard()); if (!snapshot) { throw Exception(ErrorCodes::CORRUPTED_DATA, @@ -1047,7 +1040,7 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( } RecoveredRefTable recoverRefTableDetailedFromAuthority( - Backend & backend, const Layout & layout, const std::optional & catalog_entry, + CasOperation & op, const Layout & layout, const std::optional & catalog_entry, const std::optional & ckpt) { /// The frozen catalog row and `_ckpt` supplied by the caller determine every recovery boundary; @@ -1066,7 +1059,7 @@ RecoveredRefTable recoverRefTableDetailedFromAuthority( uint64_t base_snapshot_bytes = 0; if (base_id) { - CheckpointSnapshotBase base = readCheckpointSnapshotBase(backend, layout, life, *ckpt); + CheckpointSnapshotBase base = readCheckpointSnapshotBase(op, layout, life, *ckpt); base_snapshot = std::move(base.snapshot); base_snapshot_bytes = base.bytes; } @@ -1077,7 +1070,7 @@ RecoveredRefTable recoverRefTableDetailedFromAuthority( RefTxnId id = *grounding.walk_from; while (id <= *grounding.committed_through) { - const auto got = backend.get(layout.refLogKey(life, id)); + const auto got = op.read(layout.refLogKey(life, id), Retry::standard()); if (!got) { /// `NamespaceLifeId` is opaque and unique to one logical life. A later birth has a diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h index 76a40bcc143d..4ba61f17f127 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -436,7 +436,7 @@ RefTableState replay(const std::optional & snapshot, std::span /// Everything a successful recovery of one ref table seeds. Produced by streaming replay /// (`RefReplayBuilder::finish`) rather than assigned field-by-field into the runtime, so the whole /// publication is one value installed atomically -- a prose field list would drift, but a struct that -/// the install copies wholesale cannot silently lose a field (Codex review round 4, spec §5). +/// the install copies wholesale cannot silently lose a field. /// /// `finish` populates the fields that are a pure function of `(base snapshot, replayed tail)`: `state`, /// `newest_snapshot_id`, `tail_count`, `tail_bytes`, and `base_snapshot_bytes`. The @@ -470,7 +470,7 @@ struct RecoveryResult std::optional last_epoch_seal; }; -/// The streaming generalisation of `replay` (spec §5): owns a PRIVATE candidate `RefTableState` and +/// The streaming generalisation of `replay`: owns a PRIVATE candidate `RefTableState` and /// applies decoded transactions into it ONE AT A TIME, in place, discarding the candidate on any throw. /// It is the memory fix for a long post-snapshot tail: `replay` takes the whole `tail` materialised in a /// vector (every decoded transaction resident at once, each up to the 20 MiB normal-class cap), whereas @@ -528,7 +528,7 @@ uint64_t decodedRefLogTxnFootprint(const RefLogTxn & txn); /// identical seam. void reportReplayMemoryDelta(int64_t delta_footprint_bytes); -/// Test-only observability for the streaming-recovery memory invariant (spec §5): while a probe is +/// Test-only observability for the streaming-recovery memory invariant: while a probe is /// installed, each recovery loop reports the resident footprint of every decoded transaction it holds, /// for exactly the span it holds it (`reportReplayMemoryDelta` + `decodedRefLogTxnFootprint`). A /// memory-bound test tracks the peak of the summed reported footprint and asserts it stays within a @@ -730,13 +730,13 @@ struct EpochCrossResult /// disagree about when an epoch boundary has been proved -- a rule that says which records a cut /// contains cannot have two implementations. /// -/// `life`: the namespace's life, REQUIRED (review NEW-3 -- a `nullopt`-resolves-internally default was -/// tried once and reintroduced the exact divergence review C3 removed from `Gc::fold`, just relocated -/// into `CasFsck.cpp`'s independent walk, which had its OWN already-resolved `life` in scope one call -/// site above and simply did not pass it). Every caller must resolve `life` itself, ONCE, and pass the -/// SAME value here that it uses for every other read in its own walk -- this function no longer -/// resolves anything on its own, so there is no second resolution left to disagree with the first. -EpochCrossResult crossEpochFromSeal(Backend & backend, const Layout & layout, const RootNamespace & ns, +/// `life`: the namespace's life, REQUIRED -- a `nullopt`-resolves-internally default was tried once +/// and let two callers resolve `life` independently and disagree, even though one of them +/// (`CasFsck.cpp`'s walk) already had its OWN resolved `life` in scope one call site above and simply +/// did not pass it in. Every caller must resolve `life` itself, ONCE, and pass the SAME value here +/// that it uses for every other read in its own walk -- this function no longer resolves anything on +/// its own, so there is no second resolution left to disagree with the first. +EpochCrossResult crossEpochFromSeal(CasOperation & op, const Layout & layout, const RootNamespace & ns, const RefTxnId & from_seal, std::optional seal_proven, const RefTxnId & witness, const NamespaceLifeId & life); @@ -767,8 +767,10 @@ struct RecoveredRefTable /// the named predecessor to be an `EpochSeal`, then read the snapshot. This order prevents a forged /// snapshot at any historical seal or contextually invalid epoch start from becoming state. Cleanup /// retains both the matching log and returned predecessor proof while the checkpoint names this base. -/// When supplied, `admit_request` runs immediately before each raw backend request so a caller may -/// refuse later requests without changing their durable order. Its default preserves read-only callers. +/// Every read below is one of `op`'s own requests, so a caller that needs to refuse a later request +/// mid-walk (a recovery attempt superseded while it runs) folds that fact into `op`'s `Liveness` +/// predicate at admission rather than passing a callback here -- the operation already re-checks it +/// before each request. struct CheckpointSnapshotBase { RefTableSnapshot snapshot; @@ -779,8 +781,7 @@ struct CheckpointSnapshotBase }; CheckpointSnapshotBase readCheckpointSnapshotBase( - Backend & backend, const Layout & layout, const NamespaceLifeId & life, const RefCkpt & checkpoint, - const std::function & admit_request = {}); + CasOperation & op, const Layout & layout, const NamespaceLifeId & life, const RefCkpt & checkpoint); /// Recover a ref table from ONE immutable lifecycle authority cut supplied by the caller. `catalog_entry` /// is either the exact row from that caller's frozen catalog cut or absence from that same cut; `ckpt` is @@ -795,7 +796,7 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( /// is deliberately no self-resolving compatibility overload: every consumer must pass the row from its /// frozen catalog cut explicitly. RecoveredRefTable recoverRefTableDetailedFromAuthority( - Backend & backend, const Layout & layout, const std::optional & catalog_entry, + CasOperation & op, const Layout & layout, const std::optional & catalog_entry, const std::optional & ckpt); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 65bced4938e6..6824312132c6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -1,6 +1,6 @@ #include #include -#include +#include #include #include #include @@ -22,6 +22,7 @@ #include #include #include +#include #include namespace ProfileEvents @@ -38,14 +39,12 @@ namespace ErrorCodes extern const int CORRUPTED_DATA; extern const int FILE_DOESNT_EXIST; extern const int LOGICAL_ERROR; - extern const int NETWORK_ERROR; } } namespace DB::Cas { -void reportMountRenewProgress(const CasOverwriteProgress & progress) noexcept; void reportMountRenewCompletion(const MountRenewResult & result) noexcept; void configureMountRenewObservability( const String * server_root_id, const CasEventSink * event_sink, bool deferred) noexcept; @@ -57,10 +56,22 @@ void deliverDeferredMountRenewObservability(uint64_t remount_attempt_no) noexcep namespace { -/// TRUE iff a `list(prefix, "", 1)` over `prefix` returns at least one key. -bool prefixHasAnyKey(Backend & b, const String & prefix) +/// TRUE iff a one-key listing of `prefix` returns anything. +bool prefixHasAnyKey(CasOperation & op, const String & prefix) { - return !b.list(prefix, /*cursor*/ "", /*limit*/ 1).keys.empty(); + return !op.list(prefix, /*cursor*/ "", /*limit*/ 1, Retry::standard()).keys.empty(); +} + +/// The write's own verdict on whether somebody else holds the key: for a refused precondition, what +/// the write's resolve read saw there; nothing when this write landed. Every other ending failed to +/// reach the store and must surface as itself -- reading it as a rival writer is how a transport +/// outage becomes a "double start" report. +std::optional conflictOrThrow(WriteResult && result, const String & what) +{ + if (Conflict * conflict = std::get_if(&result)) + return std::move(conflict->seen); + orThrow(std::move(result), what); + return std::nullopt; } uint64_t defaultBootMs() @@ -70,17 +81,24 @@ uint64_t defaultBootMs() return static_cast(ts.tv_sec) * 1000 + static_cast(ts.tv_nsec) / 1000000; } +/// Why a renewal ended without a retained lease, in the vocabulary the audit event reports. Each +/// value is assigned from exactly one arm of the write's verdict, so the event never re-derives a +/// reason from state the request engine does not carry. enum class MountRenewTerminalClassification : uint8_t { - FromDiagnostics, + Unclassified, DeterministicFailure, Conflict, Vanished, + Cancelled, + FenceOrLifecycleLost, + ExternalLeaseDeadline, + RequestDeadline, + Unresolved, }; -/// Retry admission (`RetryStarted`/`PutStarted`) touches only this fixed-size state. First ambiguity -/// may deliver its one bounded warning/event before the controller's following pre-resolve gate; it -/// never runs after a pre-request gate. +/// One logical renewal's audit snapshot. Fixed-size and trivially copyable so a reentrant event sink +/// gets a distinct stack slot instead of aliasing the call that is still running. struct MountRenewObservabilityContext { bool active = false; @@ -94,15 +112,10 @@ struct MountRenewObservabilityContext uint64_t observability_start_boot_ms = 0; uint64_t confirmed_deadline_boot_ms = 0; uint64_t initial_confirmed_budget_ms = 0; - CasOverwriteDeadlineSource deadline_source = CasOverwriteDeadlineSource::RequestBudget; - CasOverwriteStopCause stop_cause = CasOverwriteStopCause::Continue; - CasUnresolvedReason unresolved_reason = CasUnresolvedReason::NotUnresolved; MountRenewOutcome outcome = MountRenewOutcome::NotAttempted; - MountRenewTerminalClassification terminal_classification = MountRenewTerminalClassification::FromDiagnostics; + MountRenewTerminalClassification terminal_classification = MountRenewTerminalClassification::Unclassified; uint32_t attempts_sent = 0; - uint32_t ambiguity_attempt_no = 0; - bool resolved_by_get = false; - bool retrying_delivered = false; + bool resolved_by_read = false; }; static_assert(std::is_trivially_copyable_v); @@ -138,6 +151,12 @@ MountRenewObservabilityContext * currentMountRenewObservability() noexcept return &mount_renew_observability.contexts[mount_renew_observability.depth - 1]; } +void markMountRenewTermination(MountRenewTerminalClassification classification) noexcept +{ + if (MountRenewObservabilityContext * context = currentMountRenewObservability()) + context->terminal_classification = classification; +} + enum class MountRenewObservabilityRegistration : uint8_t { Stack, @@ -205,7 +224,6 @@ void initializeMountRenewObservability( UInt128 write_attempt_id, uint64_t attempt_start_boot_ms, uint64_t confirmed_deadline_boot_ms, - CasOverwriteDeadlineSource deadline_source, const CasEventSink & event_sink) noexcept { MountRenewObservabilityContext * context = currentMountRenewObservability(); @@ -228,46 +246,9 @@ void initializeMountRenewObservability( .initial_confirmed_budget_ms = confirmed_deadline_boot_ms > attempt_start_boot_ms ? confirmed_deadline_boot_ms - attempt_start_boot_ms : 0, - .deadline_source = deadline_source, }; } -constexpr std::string_view unresolvedReasonName(CasUnresolvedReason reason) -{ - switch (reason) - { - case CasUnresolvedReason::NotUnresolved: return "not_unresolved"; - case CasUnresolvedReason::NoAttemptSent: return "no_attempt_sent"; - case CasUnresolvedReason::FenceLostMidWay: return "fence_lost_mid_way"; - case CasUnresolvedReason::DeadlineMidWay: return "deadline_mid_way"; - case CasUnresolvedReason::FenceLostPostWrite: return "fence_lost_post_write"; - case CasUnresolvedReason::AttemptsExhausted: return "attempts_exhausted"; - case CasUnresolvedReason::DefiniteFailureAfterAmbiguity: return "definite_failure_after_ambiguity"; - } - return "unknown"; -} - -constexpr std::string_view deadlineSourceName(CasOverwriteDeadlineSource source) -{ - switch (source) - { - case CasOverwriteDeadlineSource::RequestBudget: return "request_budget"; - case CasOverwriteDeadlineSource::ExternalLeaseSafety: return "external_lease_safety"; - } - return "unknown"; -} - -constexpr std::string_view stopCauseName(CasOverwriteStopCause cause) -{ - switch (cause) - { - case CasOverwriteStopCause::Continue: return "continue"; - case CasOverwriteStopCause::Cancelled: return "cancelled"; - case CasOverwriteStopCause::FenceOrLifecycleLost: return "fence_or_lifecycle_lost"; - } - return "unknown"; -} - uint64_t elapsedSince(uint64_t start_boot_ms, uint64_t now_boot_ms) { return now_boot_ms >= start_boot_ms ? now_boot_ms - start_boot_ms : 0; @@ -287,9 +268,6 @@ void emitMountRenewEvent( std::string_view outcome, uint32_t attempts_sent, uint64_t now_boot_ms, - CasUnresolvedReason unresolved_reason, - CasOverwriteDeadlineSource deadline_source, - CasOverwriteStopCause stop_cause, std::string_view classification, uint64_t remount_attempt_no) noexcept { @@ -300,11 +278,9 @@ void emitMountRenewEvent( CasEvent event; event.type = CasEventType::WatermarkRenew; event.outcome = String{outcome}; - event.reason = outcome == "retrying" - ? "CAS mount renewal entered bounded retry after an ambiguous physical attempt" - : (outcome == "recovered" - ? "CAS mount renewal recovered before its confirmed lease-safety deadline" - : "CAS mount renewal ended without retained authority and fenced the mount"); + event.reason = outcome == "recovered" + ? "CAS mount renewal recovered before its confirmed lease-safety deadline" + : "CAS mount renewal ended without retained authority and fenced the mount"; event.detail = { {"server_root_id", *context.server_root_id}, {"writer_epoch", std::to_string(context.writer_epoch)}, @@ -313,9 +289,6 @@ void emitMountRenewEvent( {"attempts_sent", std::to_string(attempts_sent)}, {"elapsed_ms", std::to_string(elapsedSince(context.observability_start_boot_ms, now_boot_ms))}, {"remaining_confirmed_budget_ms", std::to_string(remainingConfirmedBudget(context, now_boot_ms))}, - {"unresolved_reason", String{unresolvedReasonName(unresolved_reason)}}, - {"deadline_source", String{deadlineSourceName(deadline_source)}}, - {"stop_cause", String{stopCauseName(stop_cause)}}, {"classification", String{classification}}, }; if (remount_attempt_no != 0) @@ -328,72 +301,19 @@ void emitMountRenewEvent( } } -void deliverMountRenewRetrying( - const MountRenewObservabilityContext & context, - const String & write_attempt_id, - uint64_t now_boot_ms, - uint64_t remount_attempt_no) noexcept -{ - /// Publish the structured event before the text logger. Either callback may consume recovery - /// budget, but this transition is followed by the controller's pre-resolve gate, so it cannot - /// start backend I/O after that budget has expired. - emitMountRenewEvent( - context, - write_attempt_id, - "retrying", - context.ambiguity_attempt_no, - now_boot_ms, - CasUnresolvedReason::NotUnresolved, - context.deadline_source, - CasOverwriteStopCause::Continue, - "ambiguous", - remount_attempt_no); - try - { - LOG_WARNING( - getLogger("CasMountLeaseKeeper"), - "CAS mount renewal '{}' entered retry after physical attempt {} (writer_epoch={}, seq={}, " - "remaining_confirmed_budget_ms={})", - *context.server_root_id, - context.ambiguity_attempt_no, - context.writer_epoch, - context.seq, - remainingConfirmedBudget(context, now_boot_ms)); - } - catch (...) - { - } -} - -constexpr std::string_view terminalClassificationName(const MountRenewObservabilityContext & context) +constexpr std::string_view terminalClassificationName(MountRenewTerminalClassification classification) { - switch (context.terminal_classification) + switch (classification) { case MountRenewTerminalClassification::DeterministicFailure: return "deterministic_failure"; case MountRenewTerminalClassification::Conflict: return "conflict"; case MountRenewTerminalClassification::Vanished: return "vanished"; - case MountRenewTerminalClassification::FromDiagnostics: break; - } - - switch (context.unresolved_reason) - { - case CasUnresolvedReason::AttemptsExhausted: return "attempts_exhausted"; - case CasUnresolvedReason::DefiniteFailureAfterAmbiguity: return "definite_failure_after_ambiguity"; - case CasUnresolvedReason::FenceLostMidWay: - case CasUnresolvedReason::FenceLostPostWrite: - return context.stop_cause == CasOverwriteStopCause::Cancelled - ? "cancelled" - : "fence_or_lifecycle_lost"; - case CasUnresolvedReason::NoAttemptSent: - case CasUnresolvedReason::DeadlineMidWay: - if (context.stop_cause == CasOverwriteStopCause::Cancelled) - return "cancelled"; - if (context.stop_cause == CasOverwriteStopCause::FenceOrLifecycleLost) - return "fence_or_lifecycle_lost"; - return context.deadline_source == CasOverwriteDeadlineSource::ExternalLeaseSafety - ? "external_lease_deadline" - : "request_deadline"; - case CasUnresolvedReason::NotUnresolved: return "terminal_unclassified"; + case MountRenewTerminalClassification::Cancelled: return "cancelled"; + case MountRenewTerminalClassification::FenceOrLifecycleLost: return "fence_or_lifecycle_lost"; + case MountRenewTerminalClassification::ExternalLeaseDeadline: return "external_lease_deadline"; + case MountRenewTerminalClassification::RequestDeadline: return "request_deadline"; + case MountRenewTerminalClassification::Unresolved: return "unresolved"; + case MountRenewTerminalClassification::Unclassified: return "terminal_unclassified"; } return "terminal_unclassified"; } @@ -409,9 +329,6 @@ void deliverMountRenewObservability( const uint64_t now_boot_ms = defaultBootMs(); const String write_attempt_id = u128ToHex(context.write_attempt_id).substr(0, 12); - if (context.ambiguity_attempt_no != 0 && !context.retrying_delivered) - deliverMountRenewRetrying(context, write_attempt_id, now_boot_ms, remount_attempt_no); - for (uint32_t attempt_no = 2; attempt_no <= context.attempts_sent; ++attempt_no) { try @@ -430,11 +347,11 @@ void deliverMountRenewObservability( } const bool recovered = context.outcome == MountRenewOutcome::Committed - && (context.attempts_sent > 1 || context.resolved_by_get); + && (context.attempts_sent > 1 || context.resolved_by_read); if (recovered) { - const std::string_view classification = context.resolved_by_get - ? "committed_by_get" + const std::string_view classification = context.resolved_by_read + ? "committed_by_read" : "committed_after_retry"; emitMountRenewEvent( context, @@ -442,9 +359,6 @@ void deliverMountRenewObservability( "recovered", context.attempts_sent, now_boot_ms, - context.unresolved_reason, - context.deadline_source, - context.stop_cause, classification, remount_attempt_no); try @@ -465,16 +379,13 @@ void deliverMountRenewObservability( } else if (context.outcome == MountRenewOutcome::Terminal) { - const std::string_view classification = terminalClassificationName(context); + const std::string_view classification = terminalClassificationName(context.terminal_classification); emitMountRenewEvent( context, write_attempt_id, "failed", context.attempts_sent, now_boot_ms, - context.unresolved_reason, - context.deadline_source, - context.stop_cause, classification, remount_attempt_no); try @@ -504,9 +415,9 @@ void deliverMountRenewObservability( /// names the current mount holder in its DecommissionRecovery live-refusal message. String describeMountHolder(const MountLease & m); -std::optional readOwnerObject(Backend & b, const Layout & l, const String & server_root_id) +std::optional readOwnerObject(CasOperation & op, const Layout & l, const String & server_root_id) { - const auto got = b.get(l.ownerKey(server_root_id)); + const auto got = op.read(l.ownerKey(server_root_id), Retry::standard()); if (!got) return std::nullopt; return decodeOwner(got->bytes); @@ -537,48 +448,6 @@ void configureMountRenewObservability( }; } -void reportMountRenewProgress(const CasOverwriteProgress & progress) noexcept -{ - MountRenewObservabilityContext * context = currentMountRenewObservability(); - if (!context || !context->active) - return; - - switch (progress.kind) - { - case CasOverwriteProgressKind::PutStarted: - context->attempts_sent = std::max(context->attempts_sent, progress.attempt_no); - break; - case CasOverwriteProgressKind::BecameAmbiguous: - if (context->ambiguity_attempt_no == 0) - { - context->ambiguity_attempt_no = progress.attempt_no; - if (!context->deferred) - { - /// Mark first, because either diagnostic callback may synchronously renew another - /// Pool. The fixed observation stack keeps this outer snapshot stable. - context->retrying_delivered = true; - try - { - const uint64_t now_boot_ms = defaultBootMs(); - const String write_attempt_id = u128ToHex(context->write_attempt_id).substr(0, 12); - deliverMountRenewRetrying( - *context, write_attempt_id, now_boot_ms, /*remount_attempt_no=*/0); - } - catch (...) - { - /// First-ambiguity observability is diagnostic-only. The controller now runs - /// its pre-resolve gate before starting any additional backend I/O. - } - } - } - break; - case CasOverwriteProgressKind::RetryStarted: - case CasOverwriteProgressKind::ResolveStarted: - case CasOverwriteProgressKind::ResolvedByGet: - break; - } -} - void reportMountRenewCompletion(const MountRenewResult & result) noexcept { if (mount_renew_observability.suppressed_depth != 0) @@ -591,11 +460,8 @@ void reportMountRenewCompletion(const MountRenewResult & result) noexcept return; context->completed = true; context->outcome = result.outcome; - context->attempts_sent = std::max(context->attempts_sent, result.diagnostics.attempts_sent); - context->resolved_by_get = result.diagnostics.resolved_by_get; - context->unresolved_reason = result.diagnostics.unresolved_reason; - context->deadline_source = result.diagnostics.deadline_source; - context->stop_cause = result.diagnostics.stop_cause; + context->attempts_sent = std::max(context->attempts_sent, result.attempts_sent); + context->resolved_by_read = result.resolved_by_read; if (context->deferred) return; @@ -617,7 +483,7 @@ void deliverDeferredMountRenewObservability(uint64_t remount_attempt_no) noexcep } bool serverRootSubtreeEmpty( - Backend & b, const Layout & l, const String & srid, const RefCatalog & catalog_observation) + CasOperation & op, const Layout & l, const String & srid, const RefCatalog & catalog_observation) { const String owned_prefix = srid + "/"; for (const CatalogEntry & entry : catalog_observation.entries) @@ -626,23 +492,23 @@ bool serverRootSubtreeEmpty( /// Manifests and loose roots retain logical path identity. Opaque namespace stream/state debris /// alone is not evidence that this server root owns live work. - if (prefixHasAnyKey(b, l.casManifestsServerPrefix(srid))) + if (prefixHasAnyKey(op, l.casManifestsServerPrefix(srid))) return false; - if (prefixHasAnyKey(b, l.serverRootDataPrefix(srid))) + if (prefixHasAnyKey(op, l.serverRootDataPrefix(srid))) return false; return true; } -std::optional readOwnerUuid(Backend & b, const Layout & l, const String & server_root_id) +std::optional readOwnerUuid(CasOperation & op, const Layout & l, const String & server_root_id) { - const std::optional owner = readOwnerObject(b, l, server_root_id); + const std::optional owner = readOwnerObject(op, l, server_root_id); if (!owner) return std::nullopt; return owner->server_uuid; } void claimOwnerOrThrow( - Backend & b, const Layout & l, const String & srid, UInt128 our_uuid, + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, const ObserveRefCatalog & observe_catalog) { if (!observe_catalog) @@ -651,7 +517,7 @@ void claimOwnerOrThrow( /// Owner present → it is identity: equal UUID is ok, a different UUID fails closed regardless /// of any lease/clock state. - if (const std::optional owner = readOwnerObject(b, l, srid)) + if (const std::optional owner = readOwnerObject(op, l, srid)) { if (owner->server_uuid == our_uuid) { @@ -673,27 +539,33 @@ void claimOwnerOrThrow( /// Owner absent. Claiming is allowed ONLY over a provably-empty subtree; an absent owner over /// existing data means the identity was lost and must never be silently re-claimed. - if (!serverRootSubtreeEmpty(b, l, srid, observe_catalog())) + if (!serverRootSubtreeEmpty(op, l, srid, observe_catalog())) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-root '{}' has no owner anchor but its data subtree is non-empty " "(identity lost over existing data) — refusing to re-claim", srid); - const PutResult put = b.putIfAbsent(key, encodeOwner(OwnerObject{ - .server_uuid = our_uuid, - .retired_at_ms = std::nullopt, - })); - if (put.outcome == PutOutcome::Done) + const std::optional occupant = conflictOrThrow( + op.create(key, encodeOwner(OwnerObject{.server_uuid = our_uuid, .retired_at_ms = std::nullopt}), + Retry::standard()), + fmt::format("CAS server-root '{}' owner claim", srid)); + if (!occupant) return; /// The conditional create conflicted. Recompute the whole catalog + manifest + roots bundle; /// no stale emptiness result is carried across the conflict. - if (!serverRootSubtreeEmpty(b, l, srid, observe_catalog())) + if (!serverRootSubtreeEmpty(op, l, srid, observe_catalog())) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-root '{}' owner claim conflicted and newly visible owned work blocks recreation", srid); - /// Race: another process claimed between our get and our putIfAbsent. Re-read and compare. - const std::optional reread = readOwnerObject(b, l, srid); + /// Race: another process claimed between our read and our create. The write's own resolve read + /// already observed who took the key, and reading again would answer a later question than the one + /// the conflict asked. Only an observation that settled nothing still owes a read. + std::optional reread; + if (const Object * observed = std::get_if(&*occupant)) + reread = decodeOwner(observed->bytes); + else if (!std::holds_alternative(*occupant)) + reread = readOwnerObject(op, l, srid); if (!reread) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS server-root '{}' owner anchor vanished during claim", srid); @@ -709,118 +581,115 @@ void claimOwnerOrThrow( } uint64_t allocateWriterEpoch( - Backend & b, const Layout & l, const String & srid, EpochMintPolicy policy, uint64_t now_ms, + CasOperation & op, const Layout & l, const String & srid, EpochMintPolicy policy, uint64_t now_ms, const ObserveRefCatalog & observe_catalog) { if (!observe_catalog) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS server-root '{}': catalog observer is required", srid); const String key = l.epochKey(srid); - static constexpr int max_attempts = 100; - for (int attempt = 0; attempt < max_attempts; ++attempt) - { - const auto got = b.get(key); + uint64_t allocated = 0; + /// Set when the PREVIOUS decision wrote against an absent epoch. Its conflict means a winner may + /// have installed an epoch while owned work became visible, so the emptiness bundle that + /// authorized that attempt is recomputed before this decision accepts any epoch state at all. + bool previous_decision_saw_no_epoch = false; - ServerEpoch current; - std::optional expected; - if (got) - { - current = decodeServerEpoch(got->bytes); - expected = got->token; - } - else + WriteResult result = op.readModifyWrite(key, + [&](const std::optional & observed) -> std::optional { - /// A missing `epoch` over a non-empty subtree is a reset hazard (durable monotone - /// counter cannot be reconstructed) — fail closed. - if (!serverRootSubtreeEmpty(b, l, srid, observe_catalog())) + if (previous_decision_saw_no_epoch && !serverRootSubtreeEmpty(op, l, srid, observe_catalog())) throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS server-root '{}' has no durable epoch object but its data subtree is " - "non-empty (writer_epoch reset hazard) — refusing to proceed", + "CAS server-root '{}' writer_epoch allocation conflicted and newly visible owned " + "work blocks recreation", srid); + previous_decision_saw_no_epoch = !observed; - /// Same hazard through the CONTROL objects (spec rev.4 Phase C): an absent epoch while - /// a mount object exists means epoch state was lost under a live/recent mount — - /// re-minting epoch 1 there is how a same-(uuid, epoch) twin is born. This is a - /// lifecycle decision, so it uses the authoritative probe, never get-absence. - const SentinelProbeResult mount_probe = b.probeSentinelRaw(l.mountKey(srid)); - switch (mount_probe.outcome) + ServerEpoch current; + if (observed) { - case ProbeOutcome::KeyAbsent: - break; /// authoritative absence — fresh-root bootstrap proceeds below - case ProbeOutcome::Present: + current = decodeServerEpoch(observed->bytes); + } + else + { + /// A missing `epoch` over a non-empty subtree is a reset hazard (durable monotone + /// counter cannot be reconstructed) — fail closed. + if (!serverRootSubtreeEmpty(op, l, srid, observe_catalog())) + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS server-root '{}' has no durable epoch object but its data subtree is " + "non-empty (writer_epoch reset hazard) — refusing to proceed", + srid); + + /// Same hazard through the CONTROL objects: an absent epoch while a mount object + /// exists means epoch state was lost under a live/recent mount — re-minting epoch 1 + /// there is how a same-(uuid, epoch) twin is born. This is a lifecycle decision, so it + /// uses the authoritative probe, never a read's absence (which flattens transport + /// faults into "not found"). + const SentinelProbeResult mount_probe = op.probeSentinel(l.mountKey(srid), Retry::standard()); + switch (mount_probe.outcome) { - if (policy == EpochMintPolicy::DecommissionRecovery) + case ProbeOutcome::KeyAbsent: + break; /// authoritative absence — fresh-root bootstrap proceeds below + case ProbeOutcome::Present: { - chassert(now_ms != 0); /// the decommission caller must pass its clock - const MountLease surviving = decodeMountLease(*mount_probe.body); - /// Deliberately weaker than claimMount's reclaim gate (this file, ~:370-380), - /// which never trusts a bare wall-clock comparison alone (only gc_fenced / - /// the clean-farewell min_active_build_sequence==UINT64_MAX marker / a caller-proven-dead - /// token justify a reclaim there, because clock skew can misjudge liveness). - /// This is still safe: (a) the mint below is DISTINCT from the survivor's - /// epoch by construction, so no same-(uuid, epoch) pair is ever representable - /// even if this liveness read is wrong; (b) claimMount right after this still - /// applies its own STRONG liveness gate and refuses a genuinely live member - /// regardless of what happens here. So a clock-skewed "terminal" misread can - /// only burn one epoch number on a doomed decommission attempt that aborts at - /// claimMount — it can never admit a claim over a live member. - const bool live = !surviving.gc_fenced && surviving.expires_at_ms > now_ms; - if (live) - throw Exception(ErrorCodes::ABORTED, - "CAS decommission '{}': epoch object missing but a LIVE mount lease " - "exists ({}) — refusing to re-mint an epoch under a live member " - "(stop the server or wait for its lease to lapse)", - srid, describeMountHolder(surviving)); - /// Terminal mount: proceed, but mint an epoch DISTINCT from the survivor's - /// by construction — the same-pair state is unrepresentable on this path. - current.next_writer_epoch = std::max(1, surviving.writer_epoch + 1); - break; + if (policy == EpochMintPolicy::DecommissionRecovery) + { + chassert(now_ms != 0); /// the decommission caller must pass its clock + const MountLease surviving = decodeMountLease(*mount_probe.body); + /// Deliberately weaker than claimMount's reclaim gate, which never trusts a + /// bare wall-clock comparison alone (only gc_fenced / the clean-farewell + /// min_active_build_sequence==UINT64_MAX marker / a caller-proven-dead + /// incarnation justify a reclaim there, because clock skew can misjudge + /// liveness). This is still safe: (a) the mint below is DISTINCT from the + /// survivor's epoch by construction, so no same-(uuid, epoch) pair is ever + /// representable even if this liveness read is wrong; (b) claimMount right + /// after this still applies its own STRONG liveness gate and refuses a + /// genuinely live member regardless of what happens here. So a clock-skewed + /// "terminal" misread can only burn one epoch number on a doomed + /// decommission attempt that aborts at claimMount — it can never admit a + /// claim over a live member. + const bool live = !surviving.gc_fenced && surviving.expires_at_ms > now_ms; + if (live) + throw Exception(ErrorCodes::ABORTED, + "CAS decommission '{}': epoch object missing but a LIVE mount lease " + "exists ({}) — refusing to re-mint an epoch under a live member " + "(stop the server or wait for its lease to lapse)", + srid, describeMountHolder(surviving)); + /// Terminal mount: proceed, but mint an epoch DISTINCT from the survivor's + /// by construction — the same-pair state is unrepresentable on this path. + current.next_writer_epoch = std::max(1, surviving.writer_epoch + 1); + break; + } + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS server-root '{}' has no durable epoch object but a mount lease exists — " + "durable epoch state was lost while a mount is live or recently live; " + "refusing to re-mint epoch 1. If no server is live on this root, " + "decommission it or manually remove the stale mount object '{}'.", + srid, l.mountKey(srid)); } - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS server-root '{}' has no durable epoch object but a mount lease exists — " - "durable epoch state was lost while a mount is live or recently live; " - "refusing to re-mint epoch 1. If no server is live on this root, " - "decommission it or manually remove the stale mount object '{}'.", - srid, l.mountKey(srid)); + case ProbeOutcome::ContainerAbsent: + case ProbeOutcome::AccessDenied: + case ProbeOutcome::Indeterminate: + throw Exception(ErrorCodes::CORRUPTED_DATA, + "CAS server-root '{}': cannot verify mount-lease absence before re-minting " + "the writer epoch (probe outcome: {}) — absence was never proven; failing closed", + srid, magic_enum::enum_name(mount_probe.outcome)); } - case ProbeOutcome::ContainerAbsent: - case ProbeOutcome::AccessDenied: - case ProbeOutcome::Indeterminate: - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS server-root '{}': cannot verify mount-lease absence before re-minting " - "the writer epoch (probe outcome: {}) — absence was never proven; failing closed", - srid, magic_enum::enum_name(mount_probe.outcome)); - } - - if (current.next_writer_epoch == 0) - current.next_writer_epoch = 1; - } - const uint64_t next = current.next_writer_epoch; - ServerEpoch new_state; - new_state.next_writer_epoch = next + 1; - - const CasResult res = b.casPut(key, encodeServerEpoch(new_state), expected); - if (res.outcome == CasOutcome::Committed) - return next; - if (!got) - { - /// The absent-epoch create conflicted. A winner may have installed an epoch while owned - /// work became visible, so recompute the complete catalog + manifest + roots bundle - /// before the next iteration is allowed to accept either a present or absent epoch. - if (!serverRootSubtreeEmpty(b, l, srid, observe_catalog())) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS server-root '{}' writer_epoch allocation conflicted and newly visible owned " - "work blocks recreation", - srid); - } - /// Conflict: someone else allocated concurrently — retry against fresh state only after the - /// absent-epoch safety bundle above has been recomputed when required. - } + if (current.next_writer_epoch == 0) + current.next_writer_epoch = 1; + } - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS server-root '{}' writer_epoch allocation did not converge after {} attempts", - srid, max_attempts); + allocated = current.next_writer_epoch; + return encodeServerEpoch(ServerEpoch{.next_writer_epoch = allocated + 1}); + }, + Retry::standard()); + + /// Non-convergence used to be `CORRUPTED_DATA`, on the reasoning that a hundred lost conditional + /// writes really is evidence of something wrong. The bound is a wall-clock deadline now, and ninety + /// seconds of a throttled store is not evidence of anything, so `orThrow`'s retry-later class is + /// the honest verdict. Both fail closed at `Pool::open`. + orThrow(std::move(result), fmt::format("CAS server-root '{}' writer_epoch allocation", srid)); + return allocated; } namespace @@ -901,25 +770,25 @@ void emitMountEvent(const CasEventSink & sink, CasEventType type, const String & } MountClaimResult claimMount( - Backend & b, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, - uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_token, + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, + uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation, const CasEventSink & sink) { const String key = l.mountKey(srid); - const auto got = b.get(key); + const auto got = op.read(key, Retry::standard()); /// Absent → fresh claim. if (!got) { const MountLease body = makeMountBody(our_uuid, our_epoch, /*seq=*/ 1, now_ms, ttl_ms); - const PutResult put = b.putIfAbsent(key, encodeMountLease(body)); - if (put.outcome != PutOutcome::Done) - /// Raced with a concurrent writer between get and putIfAbsent. Treat as a live double - /// start — fail closed; never overwrite a slot that appeared under us. No re-read was - /// done, so no conflicting identity is known to attach to an event. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .token = std::nullopt}; + if (conflictOrThrow(op.create(key, encodeMountLease(body), Retry::standard()), + fmt::format("CAS mount slot claim of '{}'", key))) + /// Raced with a concurrent writer between the read and the create. Treat as a live double + /// start — fail closed; never overwrite a slot that appeared under us. The occupant was + /// not decoded here, so no conflicting identity is known to attach to an event. + return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .incarnation = std::nullopt}; emitMountEvent(sink, CasEventType::MountClaim, srid, "mint", nullptr, "fresh mount slot minted"); - return {.kind = MountClaimResult::Claimed, .body = body, .token = std::nullopt}; + return {.kind = MountClaimResult::Claimed, .body = body, .incarnation = std::nullopt}; } const MountLease existing = decodeMountLease(got->bytes); @@ -930,7 +799,7 @@ MountClaimResult claimMount( { emitMountEvent(sink, CasEventType::MountConflict, srid, "foreign_owner", &existing, "mount slot is held by a foreign server_uuid — refusing to take over across identities"); - return {.kind = MountClaimResult::ForeignOwner, .body = existing, .token = std::nullopt}; + return {.kind = MountClaimResult::ForeignOwner, .body = existing, .incarnation = std::nullopt}; } /// Same uuid + same epoch: it is OUR OWN claim — but a FENCED body is terminal for this @@ -944,19 +813,19 @@ MountClaimResult claimMount( emitMountEvent(sink, CasEventType::MountConflict, srid, "fenced_by_gc", &existing, "own (uuid, epoch) mount slot is GC-fenced — terminal for this incarnation; " "recover with a fresh writer_epoch"); - return {.kind = MountClaimResult::FencedSelf, .body = existing, .token = std::nullopt}; + return {.kind = MountClaimResult::FencedSelf, .body = existing, .incarnation = std::nullopt}; } const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); - const PutResult put = b.putOverwrite(key, encodeMountLease(body), got->token); - if (put.outcome != PutOutcome::Done) - /// The mount changed under us between get and putOverwrite: `got->token` is now KNOWN - /// STALE (that mismatch is exactly why the put failed), not merely unknown -- leaving - /// `.token` unset (rather than handing back a token the caller would wrongly treat as - /// current) is deliberate, matching the identical race below. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .token = std::nullopt}; + if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->incarnation, Retry::standard()), + fmt::format("CAS mount slot refresh of '{}'", key))) + /// The mount changed under us between the read and the write: `got->incarnation` is now + /// KNOWN STALE (that mismatch is exactly why the write was refused), not merely unknown -- + /// leaving `.incarnation` unset (rather than handing back one the caller would wrongly + /// treat as current) is deliberate, matching the identical race below. + return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .incarnation = std::nullopt}; emitMountEvent(sink, CasEventType::MountClaim, srid, "refresh", &existing, "own claim replayed — refreshed seq + expiry"); - return {.kind = MountClaimResult::Claimed, .body = body, .token = std::nullopt}; + return {.kind = MountClaimResult::Claimed, .body = body, .incarnation = std::nullopt}; } /// Same uuid, DIFFERENT epoch: reclaim ONLY on a certificate of death that needs no fresh @@ -967,24 +836,25 @@ MountClaimResult claimMount( /// fence-out) instant instead of an observation wait. /// - the clean marker (`min_active_build_sequence == UINT64_MAX`) → the predecessor's OWN graceful farewell /// (`MountLeaseKeeper::terminate`) — no observation needed either. - /// - `proven_dead_token` matches the token we just read → the CALLER (`claimMountAwaitingExpiry`) - /// already watched this exact token hold stable for the full observation threshold on its own - /// clock; re-deriving that here from a bare wall-clock comparison would be exactly the - /// cross-node trust would make a clock-skewed or delayed observer unsafe. + /// - `proven_dead_incarnation` matches the one we just read → the CALLER + /// (`claimMountAwaitingExpiry`) already watched that exact incarnation hold stable for the full + /// observation threshold on its own clock; re-deriving that here from a bare wall-clock + /// comparison is exactly the cross-node trust that makes a clock-skewed or delayed observer + /// unsafe. /// Anything else → `LiveDoubleStart` (do NOT write): a same-uuid, different-epoch, not fenced, not /// clean-marked, not (yet) proven-dead lease may simply be a live twin, and `expires_at_ms` alone /// can never distinguish that from a dead predecessor across two different clocks. const bool clean_marker = existing.min_active_build_sequence == std::numeric_limits::max(); - const bool proven_dead = proven_dead_token && *proven_dead_token == got->token; + const bool proven_dead = proven_dead_incarnation && *proven_dead_incarnation == got->incarnation; if (existing.gc_fenced || clean_marker || proven_dead) { const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); - const PutResult put = b.putOverwrite(key, encodeMountLease(body), got->token); - if (put.outcome != PutOutcome::Done) - /// The mount changed under us between get and putOverwrite — someone else is racing the - /// reclaim. Fail closed. `got->token` is now KNOWN STALE (that mismatch is exactly why the - /// put failed) -- leaving `.token` unset is deliberate, not an oversight. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .token = std::nullopt}; + if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->incarnation, Retry::standard()), + fmt::format("CAS mount slot reclaim of '{}'", key))) + /// The mount changed under us between the read and the write — someone else is racing the + /// reclaim. Fail closed. `got->incarnation` is now KNOWN STALE (that mismatch is exactly why + /// the write was refused) -- leaving `.incarnation` unset is deliberate, not an oversight. + return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .incarnation = std::nullopt}; const MountPriorState prior = existing.gc_fenced ? MountPriorState::Fenced : clean_marker ? MountPriorState::Clean : MountPriorState::UncleanObserved; @@ -992,17 +862,17 @@ MountClaimResult claimMount( existing.gc_fenced ? "same server_uuid, different writer_epoch, GC-fenced — reclaimed" : clean_marker ? "same server_uuid, different writer_epoch, clean farewell — reclaimed" : "same server_uuid, different writer_epoch, observed dead by " - "token-stability — reclaimed"); - return {.kind = MountClaimResult::Claimed, .body = body, .prior = prior, .token = std::nullopt}; + "incarnation stability — reclaimed"); + return {.kind = MountClaimResult::Claimed, .body = body, .prior = prior, .incarnation = std::nullopt}; } emitMountEvent(sink, CasEventType::MountConflict, srid, "live_double_start", &existing, "same server_uuid, different writer_epoch, not fenced/clean/proven-dead — no wall-clock trust; " - "the caller must run the token-stability observation wait before reclaiming"); - /// No write was attempted on this path -- `got->token` is exactly the CURRENT body's - /// token (what we just read is what's still there), so it is safe to hand back for the caller's - /// observation loop to compare across polls without a redundant re-GET. - return {.kind = MountClaimResult::LiveDoubleStart, .body = existing, .token = got->token}; + "the caller must run the incarnation-stability observation wait before reclaiming"); + /// No write was attempted on this path -- `got->incarnation` is exactly the CURRENT body's + /// incarnation (what we just read is what's still there), so it is safe to hand back for the + /// caller's observation loop to compare across polls without a redundant re-read. + return {.kind = MountClaimResult::LiveDoubleStart, .body = existing, .incarnation = got->incarnation}; } String mountDoubleStartMessage(const String & srid, const MountLease & existing) @@ -1043,7 +913,7 @@ uint64_t mountObservationThresholdMs(uint64_t ttl_ms, uint64_t cadence_ms) } MountClaimResult claimMountAwaitingExpiry( - Backend & b, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, const std::function & now_ms_fn, const std::function & mono_ms_fn, uint64_t ttl_ms, uint64_t poll_interval_ms, @@ -1061,57 +931,57 @@ MountClaimResult claimMountAwaitingExpiry( /// identical. const uint64_t threshold_ms = mountObservationThresholdMs(ttl_ms, poll); - std::optional observed; + std::optional observed; uint64_t observed_since = 0; size_t restarts = 0; while (true) { const bool threshold_met = observed && mono_ms_fn() - observed_since >= threshold_ms; - MountClaimResult r = claimMount(b, l, srid, our_uuid, our_epoch, now_ms_fn(), ttl_ms, + MountClaimResult r = claimMount(op, l, srid, our_uuid, our_epoch, now_ms_fn(), ttl_ms, threshold_met ? observed : std::nullopt, sink); if (r.kind != MountClaimResult::LiveDoubleStart) return r; - /// `claimMount` already read the current body. Reuse `r.token` whenever `claimMount` + /// `claimMount` already read the current body. Reuse `r.incarnation` whenever `claimMount` /// set it (the common case: no write was attempted, so what it read is still current) instead of - /// re-GETting the SAME key here. The rare stale-race branches deliberately leave `.token` unset - /// (see their own comments), so this still falls back to a fresh read exactly there. - std::optional current_token = r.token; - if (!current_token) + /// re-reading the SAME key here. The rare stale-race branches deliberately leave `.incarnation` + /// unset (see their own comments), so this still falls back to a fresh read exactly there. + std::optional current_incarnation = r.incarnation; + if (!current_incarnation) { - const auto got = b.get(l.mountKey(srid)); + const auto got = op.read(l.mountKey(srid), Retry::standard()); if (!got) { /// The slot vanished between claimMount's own GET and ours — normally self-resolving /// within one more `claimMount` call (which re-mints fresh on an absent slot), but under /// slot churn (something else concurrently removing/re-minting it) that resolution could /// keep losing the same race. Pace this like every other iteration and - /// count it toward the SAME bounded restart budget the token-churn case below uses, - /// instead of spinning `get`/`claimMount`/`put` at backend RTT with no sleep and no bound + /// count it toward the SAME bounded restart budget the incarnation-churn case below + /// uses, instead of spinning read/claim/write at backend RTT with no sleep and no bound /// — a persistently vanishing slot is exactly as "alive and contended" as a persistently - /// renewing token. + /// renewing holder. if (++restarts > kMaxObservationRestarts) return r; sleep_ms_fn(poll); continue; } - current_token = got->token; + current_incarnation = got->incarnation; } - if (!observed || *observed != *current_token) + if (!observed || *observed != *current_incarnation) { if (observed && ++restarts > kMaxObservationRestarts) - /// The token kept changing across bounded restarts — the holder is genuinely alive + /// The incarnation kept changing across bounded restarts — the holder is genuinely alive /// (actively renewing), not a dead predecessor. Report it rather than waiting forever. return r; - observed = *current_token; + observed = *current_incarnation; observed_since = mono_ms_fn(); if (on_wait_start) on_wait_start(r.body, threshold_ms); LOG_INFO(getLogger("CasMountLease"), "Attempting to mount content-addressed server root {} after node change or hard " - "restart; waiting ~{} ms (token-stability observation) to confirm the previous " + "restart; waiting ~{} ms (incarnation-stability observation) to confirm the previous " "incarnation's operations are all finalized", srid, threshold_ms); } @@ -1119,7 +989,7 @@ MountClaimResult claimMountAwaitingExpiry( } } -HeartbeatFloor computeHeartbeatFloor(Backend & b, const Layout & l, uint64_t now_ms, +HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64_t now_ms, uint64_t mono_now_ms, uint64_t stable_threshold_ms, MountObservationMap & obs) { @@ -1127,224 +997,198 @@ HeartbeatFloor computeHeartbeatFloor(Backend & b, const Layout & l, uint64_t now /// `obs` is keyed by every srid this leader has EVER observed, but a /// srid removed from the LIST entirely (its `/mount` key gone -- e.g. `SYSTEM CAS - /// DROP POOL MEMBER`) is never visited by the loop below again, so its entry would otherwise linger + /// DROP POOL MEMBER`) is never visited by the walk below again, so its entry would otherwise linger /// forever (~150-250 B/srid, worse on a long-lived leader across many decommissions). Track every /// srid actually seen THIS pass and prune anything else out of `obs` at the end -- disjoint from the - /// mid-loop `obs.erase(srid)` calls below (those fire for a srid seen but now terminal/fenced/gone + /// mid-walk `obs.erase(srid)` calls below (those fire for a srid seen but now terminal/fenced/gone /// this pass; this is for a srid not seen AT ALL). std::set seen_srids; const String prefix = l.serverRootsPrefix(); - String cursor; - while (true) + op.forEachListedKey(prefix, [&](const KeyEntry & listed) { - const ListPage page = b.list(prefix, cursor, /*limit*/ 1000); - for (const auto & listed : page.keys) - { - /// `/owner` and `/epoch` objects share the subtree — only mount bodies gate the floor. - static constexpr std::string_view mount_suffix = "/mount"; - if (!listed.key.ends_with(mount_suffix)) - continue; - - const String & key = listed.key; - - /// The srid is the path segment between `serverRootsPrefix()` and the `/mount` suffix - /// (`/gc/server-roots//mount`). Used both for observability (fenced) and as - /// the key into `obs`. - const String srid = key.substr(prefix.size(), - key.size() - prefix.size() - mount_suffix.size()); - seen_srids.insert(srid); - - /// Fence-out on PreconditionFailed re-GETs and reclassifies from the top; bound the retries - /// so a pathologically contended holder cannot spin forever. On exhaustion the entry is - /// counted as live (conservative — never excluded without a landed fence-out). - constexpr int max_reclassify = 4; - for (int attempt = 0; ; ++attempt) + /// `/owner` and `/epoch` objects share the subtree — only mount bodies gate the floor. + static constexpr std::string_view mount_suffix = "/mount"; + if (!listed.key.ends_with(mount_suffix)) + return true; + + const String & key = listed.key; + + /// The srid is the path segment between `serverRootsPrefix()` and the `/mount` suffix + /// (`/gc/server-roots//mount`). Used both for observability (fenced) and as + /// the key into `obs`. + const String srid = key.substr(prefix.size(), key.size() - prefix.size() - mount_suffix.size()); + seen_srids.insert(srid); + + /// One decision per re-read: a refused fence-out re-enters this lambda with the body the + /// holder's own renewal installed, and the observation check below then sees the new + /// incarnation and restarts the window -- which counts the slot `live` and declines the write. + /// That is why no arm that counts a slot ever also asks for a fence-out body. + WriteResult fenced_out = op.readModifyWrite(key, + [&](const std::optional & observed) -> std::optional { - const auto got = b.get(key); - if (!got) + if (!observed) { obs.erase(srid); - break; /// Raced away (deleted) — nothing to classify. + return std::nullopt; /// raced away (deleted) — nothing to classify } - const MountLease m = decodeMountLease(got->bytes); + const MountLease m = decodeMountLease(observed->bytes); if (m.gc_fenced) { ++floor.already_fenced; obs.erase(srid); /// terminal — no further observation needed - break; + return std::nullopt; } if (m.min_active_build_sequence == std::numeric_limits::max()) { ++floor.terminated; obs.erase(srid); /// terminal — no further observation needed - break; + return std::nullopt; } - /// Observation-based liveness: stable ONLY if the - /// SAME token was already being watched and has now held for the full threshold on our - /// OWN monotonic clock. Anything else — no prior observation, or a changed token (a - /// live renewal, including one raced against our own fence-out attempt below) — - /// (re)starts the observation window and counts as `live` this call. + /// Observation-based liveness: stable ONLY if the SAME incarnation was already being + /// watched and has now held for the full threshold on our OWN monotonic clock. Anything + /// else — no prior observation, or a changed incarnation (a live renewal, including one + /// raced against our own fence-out attempt) — (re)starts the observation window and + /// counts as `live` this call. const auto it = obs.find(srid); - const bool stable = it != obs.end() && it->second.token == got->token + const bool stable = it != obs.end() && it->second.incarnation == observed->incarnation && mono_now_ms - it->second.first_seen_mono_ms >= stable_threshold_ms; if (!stable) { - if (it == obs.end() || it->second.token != got->token) - obs[srid] = MountTokenObservation{got->token, mono_now_ms}; + if (it == obs.end() || it->second.incarnation != observed->incarnation) + obs.insert_or_assign(srid, MountIncarnationObservation{observed->incarnation, mono_now_ms}); ++floor.live; - break; + return std::nullopt; } - const bool exhausted = attempt >= max_reclassify; - if (exhausted) - { - ++floor.live; /// conservative — never exclude without a landed fence-out - break; - } - - /// Stable past the threshold, not yet fenced → token-guarded fence-out preserving the - /// whole body (gc_fenced = true, seq + 1). + /// Stable past the threshold, not yet fenced → fence-out preserving the whole body + /// (gc_fenced = true, seq + 1) against the incarnation this decision observed. MountLease fenced = m; fenced.gc_fenced = true; fenced.seq = m.seq + 1; - const PutResult res = b.putOverwrite(key, encodeMountLease(fenced), got->token); - if (res.outcome == PutOutcome::Done) - { - ++floor.fenced_now; - floor.fenced_srids.push_back(srid); - obs.erase(srid); - LOG_INFO(getLogger("CasHeartbeatFloor"), - "CAS GC fenced out mount lease for content-addressed server root {} at " - "wall-clock ms {}: its write token held unchanged for >= {} ms on the GC " - "leader's own monotonic clock (token-stability observation)", - srid, now_ms, stable_threshold_ms); - break; - } - /// PreconditionFailed: the holder renewed between our GET and PUT — re-GET and - /// reclassify (the observation check above will see the new token and restart it). - } - } - - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; - } + return encodeMountLease(fenced); + }, + Retry::standard()); - /// Prune every `obs` entry for a srid this pass's LIST never saw at all. + if (std::holds_alternative(fenced_out)) + { + ++floor.fenced_now; + floor.fenced_srids.push_back(srid); + obs.erase(srid); + LOG_INFO(getLogger("CasHeartbeatFloor"), + "CAS GC fenced out mount lease for content-addressed server root {} at " + "wall-clock ms {}: its write incarnation held unchanged for >= {} ms on the GC " + "leader's own monotonic clock (incarnation-stability observation)", + srid, now_ms, stable_threshold_ms); + return true; + } + /// Declined: the decision above already classified and counted this slot, and asked for no + /// write. Every remaining verdict means the store was not reached, which is not a + /// classification -- surface it rather than record a floor built on an unread slot. + if (!std::holds_alternative(fenced_out)) + orThrow(std::move(fenced_out), fmt::format("CAS mount fence-out of '{}'", key)); + return true; + }, Retry::standard()); + + /// Prune every `obs` entry for a srid this pass's walk never saw at all. for (auto it = obs.begin(); it != obs.end(); ) it = seen_srids.contains(it->first) ? std::next(it) : obs.erase(it); return floor; } -std::vector probeNonTerminalMountSlots(Backend & b, const Layout & l) +std::vector probeNonTerminalMountSlots(CasOperation & op, const Layout & l) { std::vector slots; - /// Same enumeration as `computeHeartbeatFloor`'s gate -- LIST the server-roots subtree, keep the + /// Same enumeration as `computeHeartbeatFloor`'s gate -- walk the server-roots subtree, keep the /// `/mount` bodies -- but read-only and without any observation state: this answers "is anyone /// still entitled to write here", not "may I fence them out". const String prefix = l.serverRootsPrefix(); - String cursor; - while (true) + op.forEachListedKey(prefix, [&](const KeyEntry & listed) { - const ListPage page = b.list(prefix, cursor, /*limit*/ 1000); - for (const auto & listed : page.keys) - { - static constexpr std::string_view mount_suffix = "/mount"; - if (!listed.key.ends_with(mount_suffix)) - continue; /// `/owner` and `/epoch` share the subtree; only the lease says "live". - - const String srid = listed.key.substr(prefix.size(), - listed.key.size() - prefix.size() - mount_suffix.size()); - - const auto got = b.get(listed.key); - if (!got) - continue; /// raced away between LIST and GET -- there is no slot to be held. + static constexpr std::string_view mount_suffix = "/mount"; + if (!listed.key.ends_with(mount_suffix)) + return true; /// `/owner` and `/epoch` share the subtree; only the lease says "live". - MountLease m; - try - { - m = decodeMountLease(got->bytes); - } - catch (...) - { - /// An undecodable lease is the WORST case for a recreation, not an ignorable one: it is - /// what a slot written by a format this build does not understand looks like, and the - /// holder of that slot is exactly the writer we must not run over. - slots.push_back(NonTerminalMountSlot{srid, fmt::format( - "mount lease could not be decoded by this build ({})", - getCurrentExceptionMessage(/*with_stacktrace=*/false))}); - continue; - } + const String srid = listed.key.substr(prefix.size(), + listed.key.size() - prefix.size() - mount_suffix.size()); - if (m.gc_fenced || m.min_active_build_sequence == std::numeric_limits::max()) - continue; /// terminal: fenced out by GC, or the holder's own graceful farewell. + const auto got = op.read(listed.key, Retry::standard()); + if (!got) + return true; /// raced away between the listing and the read -- there is no slot to be held. + MountLease m; + try + { + m = decodeMountLease(got->bytes); + } + catch (...) + { + /// An undecodable lease is the WORST case for a recreation, not an ignorable one: it is + /// what a slot written by a format this build does not understand looks like, and the + /// holder of that slot is exactly the writer we must not run over. slots.push_back(NonTerminalMountSlot{srid, fmt::format( - "held by server uuid {} (writer_epoch {}, host '{}', pid {}, lease seq {}, stamped " - "expiry {} ms) with neither a graceful farewell nor a GC fence-out", - u128ToHex(m.server_uuid), m.writer_epoch, m.hostname, m.pid, m.seq, m.expires_at_ms)}); + "mount lease could not be decoded by this build ({})", + getCurrentExceptionMessage(/*with_stacktrace=*/false))}); + return true; } - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; - } + if (m.gc_fenced || m.min_active_build_sequence == std::numeric_limits::max()) + return true; /// terminal: fenced out by GC, or the holder's own graceful farewell. + + slots.push_back(NonTerminalMountSlot{srid, fmt::format( + "held by server uuid {} (writer_epoch {}, host '{}', pid {}, lease seq {}, stamped " + "expiry {} ms) with neither a graceful farewell nor a GC fence-out", + u128ToHex(m.server_uuid), m.writer_epoch, m.hostname, m.pid, m.seq, m.expires_at_ms)}); + return true; + }, Retry::standard()); return slots; } -std::vector listMounts(Backend & backend, const Layout & layout, uint64_t now_ms, uint64_t skew_margin_ms) +std::vector listMounts(CasOperation & op, const Layout & layout, uint64_t now_ms, uint64_t skew_margin_ms) { std::vector out; const String prefix = layout.serverRootsPrefix(); - String cursor; - while (true) + op.forEachListedKey(prefix, [&](const KeyEntry & listed) { - const ListPage page = backend.list(prefix, cursor, 1000); - for (const auto & k : page.keys) + static constexpr std::string_view suffix = "/mount"; + if (!listed.key.ends_with(suffix)) + return true; + const auto got = op.read(listed.key, Retry::standard()); + if (!got) + return true; /// raced a delete — read-only view, skip the row + MountInfo info; + /// The srid is the path segment between `serverRootsPrefix()` and the `/mount` suffix — + /// may itself contain `/` (e.g. `shard-01/replica-a`), so slice by prefix length rather + /// than `rfind('/')`, matching `computeHeartbeatFloor`'s extraction. + info.srid = listed.key.substr(prefix.size(), listed.key.size() - prefix.size() - suffix.size()); + try { - static constexpr std::string_view suffix = "/mount"; - if (!k.key.ends_with(suffix)) - continue; - const auto got = backend.get(k.key); - if (!got) - continue; /// raced a delete — read-only view, skip the row - MountInfo info; - /// The srid is the path segment between `serverRootsPrefix()` and the `/mount` suffix — - /// may itself contain `/` (e.g. `shard-01/replica-a`), so slice by prefix length rather - /// than `rfind('/')`, matching `computeHeartbeatFloor`'s extraction. - info.srid = k.key.substr(prefix.size(), k.key.size() - prefix.size() - suffix.size()); - try - { - info.lease = decodeMountLease(got->bytes); - } - catch (...) - { - info.state = "corrupt"; - out.push_back(std::move(info)); - continue; - } - if (info.lease.gc_fenced) - info.state = "fenced"; - else if (info.lease.min_active_build_sequence == std::numeric_limits::max()) - info.state = "terminated"; - else if (now_ms <= info.lease.expires_at_ms + skew_margin_ms) - info.state = "live"; - else - info.state = "expired"; + info.lease = decodeMountLease(got->bytes); + } + catch (...) + { + info.state = "corrupt"; out.push_back(std::move(info)); + return true; } - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; - } + if (info.lease.gc_fenced) + info.state = "fenced"; + else if (info.lease.min_active_build_sequence == std::numeric_limits::max()) + info.state = "terminated"; + else if (now_ms <= info.lease.expires_at_ms + skew_margin_ms) + info.state = "live"; + else + info.state = "expired"; + out.push_back(std::move(info)); + return true; + }, Retry::standard()); return out; } @@ -1375,10 +1219,10 @@ FenceCertificate classifyFenceCertificate(const MountLease & lease, uint64_t fen } -bool isCreatorFenceTerminal(Backend & backend, const Layout & layout, const String & server_root_id, +bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const String & server_root_id, uint64_t writer_epoch) { - const auto got = backend.get(layout.mountKey(server_root_id)); + const auto got = op.read(layout.mountKey(server_root_id), Retry::standard()); if (!got) return false; /// absence proves nothing about liveness -- see the header doc @@ -1416,14 +1260,20 @@ bool isCreatorFenceTerminal(Backend & backend, const Layout & layout, const Stri return terminal; } +/// The farewell's whole budget. It is deliberately short: a departing mount is holding shutdown open, +/// and a slot it fails to hand back is fenced out by the next GC round anyway. +constexpr uint64_t kFarewellBudgetMs = 10'000; + MountLeaseKeeper::MountLeaseKeeper( - BackendPtr backend_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, + CasRequests & mount_requests_, CasRequests & open_requests_, const Layout & layout_, + const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, std::function min_active_build_sequence_fn_, CasEventSink event_sink_, std::chrono::milliseconds lease_safety_margin_, std::function boot_ms_fn_) - : backend(std::move(backend_)) + : mount_requests(mount_requests_) + , open_requests(open_requests_) , key(layout_.mountKey(srid_)) , srid(srid_) , server_uuid(server_uuid_) @@ -1457,28 +1307,35 @@ String MountLeaseKeeper::encodeBody( }); } -Token MountLeaseKeeper::claim(const String & body) +const Incarnation & MountLeaseKeeper::precondition() const { - const HeadResult head = backend->head(key); - if (!head.exists) + if (!last_incarnation) + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "CAS mount-lease: key '{}' has no incarnation to name as a write precondition", key); + return *last_incarnation; +} + +Incarnation MountLeaseKeeper::claim(CasOperation & op, const String & body) +{ + /// One read decides the branch AND supplies the precondition, so both the mint and the adoption + /// are two requests: a separate presence probe would only re-ask what these bytes already answer. + const std::optional got = op.read(key, Retry::standard()); + if (!got) { - const PutResult result = backend->putIfAbsent(key, body); - if (result.outcome != PutOutcome::Done) + WriteResult minted = op.create(key, body, Retry::standard()); + if (std::holds_alternative(minted)) throw Exception( ErrorCodes::ABORTED, - "CAS mount-lease: key '{}' appeared between head and putIfAbsent", key); + "CAS mount-lease: key '{}' appeared between the read and the create", key); + const std::optional incarnation + = orThrow(std::move(minted), fmt::format("CAS mount-lease mint of key '{}'", key)); emitMountEvent( event_sink, CasEventType::MountClaim, srid, "mint", nullptr, "mount slot absent -- keeper minted it directly"); - return result.token; + return *incarnation; } - const auto got = backend->get(key); - if (!got) - throw Exception( - ErrorCodes::ABORTED, - "CAS mount-lease: key '{}' vanished between head and get while claiming", key); - const MountLease observed = decodeMountLease(got->bytes); if (observed.server_uuid != server_uuid) { @@ -1510,13 +1367,13 @@ Token MountLeaseKeeper::claim(const String & body) key, describeMountHolder(observed))); } - const PutResult result = backend->putOverwrite(key, body, got->token); - if (result.outcome != PutOutcome::Done) + WriteResult adopted = op.replace(key, body, got->incarnation, Retry::standard()); + if (const Conflict * conflict = std::get_if(&adopted)) { - const auto current = backend->get(key); - if (current) + /// The write's own resolve read is the re-read: it observed what took the key from us. + if (const Object * occupant = std::get_if(&conflict->seen)) { - const MountLease lease = decodeMountLease(current->bytes); + const MountLease lease = decodeMountLease(occupant->bytes); if (lease.server_uuid == server_uuid && lease.gc_fenced) throw MountFencedException(fmt::format( "CAS mount-lease: key '{}' was fenced by GC inside the adoption window ({})", @@ -1526,18 +1383,21 @@ Token MountLeaseKeeper::claim(const String & body) "CAS mount-lease: key '{}' changed while adopting our own mount slot ({})", key, describeMountHolder(lease)); } - throw Exception( - ErrorCodes::ABORTED, - "CAS mount-lease: key '{}' vanished while adopting our own mount slot", key); + if (std::holds_alternative(conflict->seen)) + throw Exception( + ErrorCodes::ABORTED, + "CAS mount-lease: key '{}' vanished while adopting our own mount slot", key); } + const std::optional incarnation + = orThrow(std::move(adopted), fmt::format("CAS mount-lease adoption of key '{}'", key)); emitMountEvent( event_sink, CasEventType::MountClaim, srid, "adopt", &observed, "adopted our own already-live mount slot"); - return result.token; + return *incarnation; } -uint64_t MountLeaseKeeper::start() +uint64_t MountLeaseKeeper::start(Liveness liveness) { if (keeper_state != MountLeaseKeeperState::New) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: start is allowed only in New state for key '{}'", key); @@ -1545,10 +1405,14 @@ uint64_t MountLeaseKeeper::start() const uint64_t wall_ms = now_ms_fn(); const uint64_t attempt_start_boot_ms = boot_ms_fn(); const String body = encodeBody(/*seq_=*/1, wall_ms, min_active_build_sequence_fn(), newMountWriteAttemptId()); - const Token token = claim(body); + /// Off the mount fence: a self-remount claims with the fence already latched lost, and a claim + /// admitted under it would be refused on every request. What makes the claim safe is that every + /// write below is conditional. + CasOperation op = open_requests.admit(std::move(liveness)); + const Incarnation incarnation = claim(op, body); seq = 1; - last_token = token; + last_incarnation = incarnation; last_committed_attempt_start_boot_ms = attempt_start_boot_ms; const uint64_t ttl_ms = static_cast(ttl.count()); confirmed_deadline_boot_ms = attempt_start_boot_ms > std::numeric_limits::max() - ttl_ms @@ -1558,78 +1422,82 @@ uint64_t MountLeaseKeeper::start() return attempt_start_boot_ms; } -[[noreturn]] void MountLeaseKeeper::throwRenewConflict(const CasOverwriteDiagnostics & diagnostics) const +[[noreturn]] void MountLeaseKeeper::throwRenewConflict(const Observation & seen) const { - if (!diagnostics.resolve_observation_completed) - throw Exception( - ErrorCodes::NETWORK_ERROR, - "CAS mount-lease: key '{}' conflicted but the controller has no authoritative resolve observation", - key); - if (!diagnostics.observed_bytes) + if (const Object * occupant = std::get_if(&seen)) { - emitMountEvent( - event_sink, CasEventType::MountConflict, srid, "vanished", nullptr, - "mount slot vanished while renewing -- failing closed"); - throw Exception( - ErrorCodes::FILE_DOESNT_EXIST, - "CAS mount-lease: key '{}' vanished while renewing -- failing closed", key); - } + markMountRenewTermination(MountRenewTerminalClassification::Conflict); + const MountLease current = decodeMountLease(occupant->bytes); + if (current.server_uuid == server_uuid && current.gc_fenced) + { + emitMountEvent( + event_sink, CasEventType::MountConflict, srid, "fenced_by_gc", ¤t, + "own mount slot was fenced by GC after lease expiry"); + throw MountFencedException(fmt::format( + "CAS mount-lease: key '{}' was fenced by GC after lease expiry ({})", + key, describeMountHolder(current))); + } + if (current.server_uuid == server_uuid && current.writer_epoch == writer_epoch) + { + emitMountEvent( + event_sink, CasEventType::MountConflict, srid, "same_epoch_state_uncertain", ¤t, + "own mount slot advanced past our incarnation -- state uncertain"); + throw Exception( + ErrorCodes::ABORTED, + "CAS mount-lease: key '{}' advanced under our own (uuid, epoch); state uncertain ({} vs our seq={})", + key, describeMountHolder(current), seq); + } + if (current.server_uuid == server_uuid) + { + emitMountEvent( + event_sink, CasEventType::MountConflict, srid, "superseded", ¤t, + "mount slot is held by a newer writer epoch"); + throw Exception( + ErrorCodes::ABORTED, + "CAS mount-lease: key '{}' was superseded by a newer incarnation ({})", + key, describeMountHolder(current)); + } - const MountLease current = decodeMountLease(*diagnostics.observed_bytes); - if (current.server_uuid == server_uuid && current.gc_fenced) - { + /// This decoded authoritative observation is the exact point at which this incarnation learns + /// that a foreign successor owns the slot. Terminal teardown intentionally performs no release + /// I/O, so account the skipped farewell here, once, before the keeper enters its terminal state. + /// The renewal may be parked under `remount_mutex`; keep the increment trace-free. + ProfileEvents::incrementNoTrace(ProfileEvents::CASMountReleaseSkippedForeignOccupant); emitMountEvent( - event_sink, CasEventType::MountConflict, srid, "fenced_by_gc", ¤t, - "own mount slot was fenced by GC after lease expiry"); - throw MountFencedException(fmt::format( - "CAS mount-lease: key '{}' was fenced by GC after lease expiry ({})", - key, describeMountHolder(current))); - } - if (current.server_uuid == server_uuid && current.writer_epoch == writer_epoch) - { - emitMountEvent( - event_sink, CasEventType::MountConflict, srid, "same_epoch_state_uncertain", ¤t, - "own mount slot advanced past our token -- state uncertain"); + event_sink, CasEventType::MountConflict, srid, "foreign_writer", ¤t, + "mount slot is held by a foreign server -- failing closed"); throw Exception( ErrorCodes::ABORTED, - "CAS mount-lease: key '{}' advanced under our own (uuid, epoch); state uncertain ({} vs our seq={})", - key, describeMountHolder(current), seq); + "CAS mount-lease: key '{}' is held by a foreign server ({}) -- failing closed", + key, describeMountHolder(current)); } - if (current.server_uuid == server_uuid) + + if (std::holds_alternative(seen)) { + markMountRenewTermination(MountRenewTerminalClassification::Vanished); emitMountEvent( - event_sink, CasEventType::MountConflict, srid, "superseded", ¤t, - "mount slot is held by a newer writer epoch"); + event_sink, CasEventType::MountConflict, srid, "vanished", nullptr, + "mount slot vanished while renewing -- failing closed"); throw Exception( - ErrorCodes::ABORTED, - "CAS mount-lease: key '{}' was superseded by a newer incarnation ({})", - key, describeMountHolder(current)); + ErrorCodes::FILE_DOESNT_EXIST, + "CAS mount-lease: key '{}' vanished while renewing -- failing closed", key); } - /// This decoded authoritative observation is the exact point at which this incarnation learns - /// that a foreign successor owns the slot. Terminal teardown intentionally performs no release - /// I/O, so account the skipped farewell here, once, before the keeper enters its terminal state. - /// The renewal may be parked under `remount_mutex`; keep the increment trace-free. - ProfileEvents::incrementNoTrace(ProfileEvents::CASMountReleaseSkippedForeignOccupant); - emitMountEvent( - event_sink, CasEventType::MountConflict, srid, "foreign_writer", ¤t, - "mount slot is held by a foreign server -- failing closed"); - throw Exception( - ErrorCodes::ABORTED, - "CAS mount-lease: key '{}' is held by a foreign server ({}) -- failing closed", - key, describeMountHolder(current)); + /// The precondition was refused but nothing identifiable was read back: neither the successor nor + /// an absence is established, so the only honest verdict is that this renewal settled nothing. + markMountRenewTermination(MountRenewTerminalClassification::Unresolved); + throwCasWriteRetryLater(fmt::format( + "CAS mount-lease: key '{}' refused our precondition and the resolving read established neither " + "an occupant nor an absence", key)); } -MountRenewResult MountLeaseKeeper::terminalResult( - uint64_t attempt_start_boot_ms, - CasOverwriteDiagnostics diagnostics, - std::exception_ptr failure) +MountRenewResult MountLeaseKeeper::terminalResult(MountRenewResult result) { - if (!failure) + if (!result.failure) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: terminal renewal requires a failure"); try { - std::rethrow_exception(failure); + std::rethrow_exception(result.failure); } catch (const Exception & e) { @@ -1645,17 +1513,22 @@ MountRenewResult MountLeaseKeeper::terminalResult( "CAS mount-lease: terminal renewal outside Active state (observed state {})", static_cast(keeper_state)); keeper_state = MountLeaseKeeperState::RenewalTerminal; - return MountRenewResult{ - .outcome = MountRenewOutcome::Terminal, - .attempt_start_boot_ms = attempt_start_boot_ms, - .diagnostics = diagnostics, - .failure = std::move(failure), - }; + result.outcome = MountRenewOutcome::Terminal; + return result; } -MountRenewResult MountLeaseKeeper::renew( - const CasRequestBudget & budget, - const MountRenewOperationEnvironment & environment) +MountRenewResult MountLeaseKeeper::renew(const MountRenewOperationEnvironment & environment) +{ + return renewOn(mount_requests, environment); +} + +MountRenewResult MountLeaseKeeper::renewForRemount(const MountRenewOperationEnvironment & environment) +{ + return renewOn(open_requests, environment); +} + +MountRenewResult MountLeaseKeeper::renewOn( + CasRequests & plane, const MountRenewOperationEnvironment & environment) { const MountRenewObservabilityRegistration observability_registration = beginMountRenewObservabilityCall(); const MountRenewObservabilityCallGuard observability_guard(observability_registration); @@ -1668,43 +1541,16 @@ MountRenewResult MountLeaseKeeper::renew( static_cast(keeper_state)); const auto boot_clock = environment.boot_ms ? environment.boot_ms : boot_ms_fn; - const auto stop_cause = environment.stop_cause - ? environment.stop_cause - : [] { return CasOverwriteStopCause::Continue; }; - const auto wait_before_retry = environment.wait_before_retry - ? environment.wait_before_retry - : [](uint64_t) { return true; }; - const auto downstream_observe = environment.observe - ? environment.observe - : [](const CasOverwriteProgress &) {}; - uint32_t physical_attempts_sent = 0; - const auto observe = [&physical_attempts_sent, &downstream_observe](const CasOverwriteProgress & progress) - { - /// This call-stack-owned value is protocol diagnostic truth even when rich observability is - /// intentionally suppressed after deeply reentrant sinks exhaust its bounded TLS slots. - if (progress.kind == CasOverwriteProgressKind::PutStarted) - physical_attempts_sent = std::max(physical_attempts_sent, progress.attempt_no); - downstream_observe(progress); - }; + /// Sampled BEFORE the write. A refused admission is reported as "never attempted" only when this + /// node had already been asked to stop, and reading the flag afterwards could not tell that apart + /// from a flag the refusal itself set. + const bool cancelled = environment.cancelled && environment.cancelled(); const uint64_t wall_ms = now_ms_fn(); const uint64_t attempt_start_boot_ms = boot_clock(); const uint64_t next_seq = seq + 1; const UInt128 write_attempt_id = newMountWriteAttemptId(); const String body = encodeBody(next_seq, wall_ms, min_active_build_sequence_fn(), write_attempt_id); - const Token expected = last_token; - - const uint64_t safety_ms = static_cast(lease_safety_margin.count()); - const uint64_t lease_retry_deadline = confirmed_deadline_boot_ms > safety_ms - ? confirmed_deadline_boot_ms - safety_ms - : 0; - const uint64_t request_deadline = attempt_start_boot_ms > std::numeric_limits::max() - budget.operation_deadline_ms - ? std::numeric_limits::max() - : attempt_start_boot_ms + budget.operation_deadline_ms; - const uint64_t absolute_deadline = std::min(lease_retry_deadline, request_deadline); - const CasOverwriteDeadlineSource deadline_source = lease_retry_deadline <= request_deadline - ? CasOverwriteDeadlineSource::ExternalLeaseSafety - : CasOverwriteDeadlineSource::RequestBudget; if (observability_registration != MountRenewObservabilityRegistration::Ignored) { @@ -1715,110 +1561,121 @@ MountRenewResult MountLeaseKeeper::renew( write_attempt_id, attempt_start_boot_ms, confirmed_deadline_boot_ms, - deadline_source, event_sink); } - CasRequestController controller(backend, budget, boot_clock); - const CasOverwriteOperationContext context{ - .absolute_deadline_ms = absolute_deadline, - .deadline_source = deadline_source, - .stop_cause = stop_cause, - .wait_before_retry = wait_before_retry, - .observe = observe, - }; + MountRenewResult result; + result.attempt_start_boot_ms = attempt_start_boot_ms; - CasOverwriteResult controlled; - controlled.diagnostics.deadline_source = deadline_source; + CasOperation op = plane.admit(environment.live); + std::optional written; try { - controlled = controller.putOverwriteControlled(key, body, expected, context); + written = op.replace(key, body, precondition(), + Retry::untilLeaseSafe(confirmed_deadline_boot_ms, static_cast(lease_safety_margin.count()))); } catch (...) { - /// The controller may propagate a deterministic/non-retryable exception after `PutStarted`. - /// Preserve the physical observer's already-published truth instead of returning the default - /// zero-attempt diagnostics from the result object that was never assigned. This local is not - /// coupled to the bounded rich-event stack and therefore remains truthful at arbitrary nesting. - controlled.diagnostics.attempts_sent = std::max( - controlled.diagnostics.attempts_sent, physical_attempts_sent); - controlled.diagnostics.deadline_source = deadline_source; - if (MountRenewObservabilityContext * observation = currentMountRenewObservability()) - observation->terminal_classification = MountRenewTerminalClassification::DeterministicFailure; - return terminalResult(attempt_start_boot_ms, controlled.diagnostics, std::current_exception()); + /// The engine surfaces a deterministic local failure unchanged rather than reissuing it. + markMountRenewTermination(MountRenewTerminalClassification::DeterministicFailure); + result.failure = std::current_exception(); + return terminalResult(std::move(result)); } - if (controlled.outcome == CasOverwriteOutcome::Committed) + if (Committed * committed = std::get_if(&*written)) { seq = next_seq; - last_token = controlled.token; + last_incarnation = std::move(committed->incarnation); last_committed_attempt_start_boot_ms = attempt_start_boot_ms; const uint64_t ttl_ms = static_cast(ttl.count()); confirmed_deadline_boot_ms = attempt_start_boot_ms > std::numeric_limits::max() - ttl_ms ? std::numeric_limits::max() : attempt_start_boot_ms + ttl_ms; - return MountRenewResult{ - .outcome = MountRenewOutcome::Committed, - .attempt_start_boot_ms = attempt_start_boot_ms, - .diagnostics = controlled.diagnostics, - .failure = nullptr, - }; + result.outcome = MountRenewOutcome::Committed; + result.attempts_sent = committed->attempts_sent; + result.resolved_by_read = committed->resolved_by_read; + result.sent_any = committed->attempts_sent != 0; + return result; } - if (controlled.outcome == CasOverwriteOutcome::Conflict) + if (const Conflict * conflict = std::get_if(&*written)) { - if (MountRenewObservabilityContext * observation = currentMountRenewObservability()) - observation->terminal_classification = MountRenewTerminalClassification::Conflict; + result.sent_any = true; + result.attempts_sent = conflict->attempts_sent; try { - throwRenewConflict(controlled.diagnostics); + throwRenewConflict(conflict->seen); } catch (...) { - return terminalResult(attempt_start_boot_ms, controlled.diagnostics, std::current_exception()); + result.failure = std::current_exception(); } + return terminalResult(std::move(result)); } - if (controlled.diagnostics.attempts_sent == 0 - && controlled.diagnostics.stop_cause == CasOverwriteStopCause::Cancelled) + if (const Refused * refused = std::get_if(&*written)) { - return MountRenewResult{ - .outcome = MountRenewOutcome::NotAttempted, - .attempt_start_boot_ms = attempt_start_boot_ms, - .diagnostics = controlled.diagnostics, - .failure = nullptr, - }; + result.sent_any = true; + result.attempts_sent = refused->attempts_sent; + markMountRenewTermination(MountRenewTerminalClassification::DeterministicFailure); + result.failure = std::make_exception_ptr(Exception( + refused->store_error, + "CAS mount-lease: the store refused the renewal of key '{}': {}", key, refused->message)); + return terminalResult(std::move(result)); } - /// Preserve the typed vanished-slot outcome only when the controller itself completed an exact - /// resolving read. Never start diagnostic backend I/O after its terminal deadline/cancel gate. - if (controlled.diagnostics.resolve_observation_completed - && !controlled.diagnostics.observed_bytes) + if (const GaveUp * gave_up = std::get_if(&*written)) { - if (MountRenewObservabilityContext * observation = currentMountRenewObservability()) - observation->terminal_classification = MountRenewTerminalClassification::Vanished; - emitMountEvent( - event_sink, CasEventType::MountConflict, srid, "vanished", nullptr, - "mount slot vanished while renewing -- failing closed"); - return terminalResult( - attempt_start_boot_ms, - controlled.diagnostics, - std::make_exception_ptr(Exception( - ErrorCodes::FILE_DOESNT_EXIST, - "CAS mount-lease: key '{}' vanished while renewing -- failing closed", - key))); + result.sent_any = gave_up->sent_any; + result.attempts_sent = gave_up->attempts_sent; + if (gave_up->why == GaveUp::Why::Deadline) + result.deadline_source = gave_up->deadline_source; + + /// Nothing was sent and the node was already stopping: the lease is exactly as it was, so this + /// is a renewal that never ran, not one that lost its authority. + if (gave_up->why == GaveUp::Why::FenceLost && !gave_up->sent_any && cancelled) + { + markMountRenewTermination(MountRenewTerminalClassification::Cancelled); + result.outcome = MountRenewOutcome::NotAttempted; + return result; + } + + MountRenewTerminalClassification classification = MountRenewTerminalClassification::Unresolved; + switch (gave_up->why) + { + case GaveUp::Why::FenceLost: + classification = cancelled + ? MountRenewTerminalClassification::Cancelled + : MountRenewTerminalClassification::FenceOrLifecycleLost; + break; + case GaveUp::Why::Deadline: + classification = gave_up->deadline_source == GaveUp::Source::Lease + ? MountRenewTerminalClassification::ExternalLeaseDeadline + : MountRenewTerminalClassification::RequestDeadline; + break; + case GaveUp::Why::Unresolved: + classification = MountRenewTerminalClassification::Unresolved; + break; + } + markMountRenewTermination(classification); + result.failure = makeCasWriteRetryLaterExceptionPtr(fmt::format( + "CAS mount-lease renewal for key '{}' did not retain the lease ({}, {} attempt sent, last " + "observation: {})", + key, + terminalClassificationName(classification), + gave_up->sent_any ? "at least one" : "no", + detail::renderObservation(gave_up->last_seen))); + return terminalResult(std::move(result)); } - const String reason = fmt::format( - "CAS mount-lease renewal for key '{}' is unresolved: {}", - key, describeUnresolvedReason(controlled.diagnostics.unresolved_reason)); - return terminalResult( - attempt_start_boot_ms, - controlled.diagnostics, - makeCasWriteRetryLaterExceptionPtr(reason)); + /// The remaining alternative is `Declined`, which only a decide returning nothing produces; a + /// renewal always has bytes to write. + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "CAS mount-lease: the renewal of key '{}' was declined, which a replace cannot report", key); } -void MountLeaseKeeper::terminate() +void MountLeaseKeeper::terminate(CasOperation & op) { const uint64_t wall_ms = now_ms_fn(); const String body = encodeMountLease(MountLease{ @@ -1832,12 +1689,24 @@ void MountLeaseKeeper::terminate() .min_active_build_sequence = std::numeric_limits::max(), .write_attempt_id = newMountWriteAttemptId(), }); - const PutResult result = backend->putOverwrite(key, body, last_token); - if (result.outcome != PutOutcome::Done) + WriteResult written = op.replace(key, body, precondition(), Retry::within(kFarewellBudgetMs)); + + if (Committed * committed = std::get_if(&written)) { - if (const auto got = backend->get(key)) + seq += 1; + last_incarnation = std::move(committed->incarnation); + emitMountEvent( + event_sink, CasEventType::MountRelease, srid, "farewell", nullptr, + "graceful release -- lease stamped already-expired and watermark retired"); + return; + } + + if (const Conflict * conflict = std::get_if(&written)) + { + /// The write's own resolve read is the re-read this branch used to issue for itself. + if (const Object * occupant = std::get_if(&conflict->seen)) { - const MountLease current = decodeMountLease(got->bytes); + const MountLease current = decodeMountLease(occupant->bytes); if (current.gc_fenced) return; ProfileEvents::increment(ProfileEvents::CASMountExclusivityViolation); @@ -1846,14 +1715,11 @@ void MountLeaseKeeper::terminate() "CAS mount-lease: release of key '{}' found a foreign incarnation ({}) and left it untouched", key, describeMountHolder(current)); } - return; + if (std::holds_alternative(conflict->seen)) + return; /// the slot is already gone; there is nothing left to hand back } - seq += 1; - last_token = result.token; - emitMountEvent( - event_sink, CasEventType::MountRelease, srid, "farewell", nullptr, - "graceful release -- lease stamped already-expired and watermark retired"); + orThrow(std::move(written), fmt::format("CAS mount-lease release of key '{}'", key)); } void MountLeaseKeeper::release() @@ -1861,7 +1727,11 @@ void MountLeaseKeeper::release() if (keeper_state != MountLeaseKeeperState::Active) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: release is allowed only in Active state for key '{}'", key); keeper_state = MountLeaseKeeperState::Released; - terminate(); + /// Off the mount fence, for the same reason the claim is: a departing mount whose lease has already + /// run down still has to hand the slot back, and refusing the write there would leave the slot + /// looking live until GC fences it out. + CasOperation op = open_requests.admit(); + terminate(op); } void sweepOwnMountStaging(IObjectStorage & object_storage, const String & mount_staging_prefix) noexcept diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index 6108141415dc..419f05ac37d2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -1,6 +1,5 @@ #pragma once -#include -#include +#include #include #include #include @@ -51,16 +50,27 @@ struct MountRenewResult { MountRenewOutcome outcome = MountRenewOutcome::Terminal; uint64_t attempt_start_boot_ms = 0; - CasOverwriteDiagnostics diagnostics; + /// Physical attempts this renewal sent, for every terminal ending the engine can count: a commit, + /// a give-up, a conflict, or a store refusal. + uint32_t attempts_sent = 0; + bool resolved_by_read = false; + bool sent_any = false; + /// Which bound ended a renewal that ran out of time; unset for every other ending, including a + /// committed one -- no deadline ended it, so naming one would invent a fact. + std::optional deadline_source; std::exception_ptr failure; }; struct MountRenewOperationEnvironment { std::function boot_ms; - std::function stop_cause; - std::function wait_before_retry; - std::function observe; + /// Facts the mount fence cannot see (park requested, pool no longer live, shutdown). FALSE ends + /// the renewal exactly as a lost fence does; the engine does not need to know which refused. + std::function live; + /// Sampled ONCE before the write. A renewal refused before its first attempt is reported as + /// `NotAttempted` rather than terminal only when this node had already been asked to stop -- + /// sampling it afterwards would read a flag that the refusal itself may have set. + std::function cancelled; }; /// Validate a `server_root_id` — the explicit, configured identity of the content-addressed layout @@ -110,7 +120,6 @@ inline void validateServerRootId(const String & id) /// below can use `OwnerObject`, `ServerEpoch`, `MountLease`, and their `encode`/`decode` functions /// without duplicating the wire-format implementation. -class Backend; class Layout; /// Mount-safety claim logic. These are the identity and epoch-allocation steps a server @@ -121,7 +130,7 @@ class Layout; /// exact-component probes find neither `cas/manifests//` nor `roots//` work. Opaque /// `cas/ns/` debris alone does not identify a logical owner. bool serverRootSubtreeEmpty( - Backend & b, const Layout & l, const String & srid, const RefCatalog & catalog_observation); + CasOperation & op, const Layout & l, const String & srid, const RefCatalog & catalog_observation); /// Supplied by the pool layer so the low-level server-root protocol always observes the mandatory /// catalog. Every absent-control retry obtains a fresh, successfully decoded observation. @@ -131,18 +140,18 @@ using ObserveRefCatalog = std::function; /// a plain GET+decode. nullopt = anchor absent. Pool-member decommission uses this to read the /// victim UUID before mounting writable; `claimOwnerOrThrow` below is the identity-claiming /// counterpart used by normal opens and reuses this GET+decode path. -std::optional readOwnerUuid(Backend & b, const Layout & l, const String & server_root_id); +std::optional readOwnerUuid(CasOperation & op, const Layout & l, const String & server_root_id); /// Claim (or validate) the sticky owner anchor that binds `srid` to a server UUID (identity). /// - owner present, equal `our_uuid`, and not tombstoned → ok (return); /// - owner present and tombstoned → throw `CORRUPTED_DATA` (explicitly retired — fail closed); /// - owner present, different → throw `CORRUPTED_DATA` (foreign owner — fail closed); -/// - owner absent AND the subtree is provably empty → `putIfAbsent` the owner (claim); +/// - owner absent AND the subtree is provably empty → `create` the owner (claim); /// - owner absent BUT the subtree is non-empty → throw `CORRUPTED_DATA` (identity lost over /// existing data — never silently re-claim). /// The owner object is never deleted and never reassigned to a different UUID. void claimOwnerOrThrow( - Backend & b, const Layout & l, const String & srid, UInt128 our_uuid, + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, const ObserveRefCatalog & observe_catalog); /// Which mint policy governs `allocateWriterEpoch`'s absent-epoch branch (see below). @@ -169,9 +178,9 @@ enum class EpochMintPolicy : uint8_t /// distinct from the survivor's by construction (`now_ms` is required, nonzero, here); /// - anything else (`ContainerAbsent`/`AccessDenied`/`Indeterminate`) → throw /// `CORRUPTED_DATA` (absence was never proven; fail closed); -/// - otherwise read `next = current.next_writer_epoch`, `casPut` `{next + 1}` against the -/// observed token, retry on `Conflict` (bounded), and return `next`. -uint64_t allocateWriterEpoch(Backend & b, const Layout & l, const String & srid, +/// - otherwise read `next = current.next_writer_epoch`, write `{next + 1}` against the observed +/// incarnation, re-deciding on conflict, and return `next`. +uint64_t allocateWriterEpoch(CasOperation & op, const Layout & l, const String & srid, EpochMintPolicy policy, uint64_t now_ms, const ObserveRefCatalog & observe_catalog); @@ -183,19 +192,19 @@ enum class MountPriorState None, Clean, /// the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) Fenced, /// the GC leader's own (already threshold-gated) fence-out (`gc_fenced`) - UncleanObserved, /// OUR observation watched the write-token hold stable for the full threshold + UncleanObserved, /// OUR observation watched the incarnation hold stable for the full threshold }; /// Startup decision for the mount lease (`gc/server-roots//mount`), run AFTER the owner gate /// (so `our_uuid` is the established owner). The lease is LIVENESS, not identity — the owner object /// already settled who may write; the lease settles whether a live incarnation currently holds the /// slot. Decision over `get(mountKey)`: -/// - absent → write our body via `putIfAbsent` → `Claimed`; +/// - absent → write our body via `create` → `Claimed`; /// - same `server_uuid` AND same `writer_epoch` as (our_uuid, our_epoch) → it is OUR OWN claim /// (a replay / the keeper adopting it): /// - `gc_fenced` → terminal for THIS (uuid, epoch) — a fence costs an epoch, so refreshing it /// in place would reactivate a fenced incarnation → `FencedSelf` (no write); -/// - otherwise → refresh (`putOverwrite` to bump seq + fresh `expires_at_ms`) → `Claimed`; +/// - otherwise → refresh (`replace` to bump seq + fresh `expires_at_ms`) → `Claimed`; /// - same `server_uuid`, DIFFERENT `writer_epoch` → reclaimed ONLY on a certificate of death that /// needs no fresh wall-clock trust (see /// `claimMountAwaitingExpiry` below for how a plain "looks expired" reading is turned into one): @@ -203,8 +212,8 @@ enum class MountPriorState /// costs an epoch, so its keeper can never renew again) → reclaim, `prior = Fenced`; /// - the clean marker (`min_active_build_sequence == UINT64_MAX`, the predecessor's own graceful farewell) → /// reclaim, `prior = Clean`; -/// - `proven_dead_token` matches the CURRENTLY OBSERVED token (the caller itself watched this -/// exact token hold stable for the full observation threshold) → reclaim, `prior = +/// - `proven_dead_incarnation` matches the CURRENTLY OBSERVED incarnation (the caller itself +/// watched that exact incarnation hold stable for the full observation threshold) → reclaim, `prior = /// UncleanObserved`; /// - none of the above → `LiveDoubleStart` (do NOT write). In particular `expires_at_ms <= /// now_ms` ALONE is never sufficient — comparing a predecessor's stamp against OUR wall clock @@ -229,13 +238,13 @@ struct MountClaimResult /// Which certificate of death justified a same-uuid, different-epoch `Claimed` reclaim (`None` for /// every other `Kind`, and for the absent-slot / same-epoch-refresh `Claimed` cases). MountPriorState prior = MountPriorState::None; - /// The backend token of the body this result observed, for + /// The incarnation of the body this result observed, for /// `LiveDoubleStart` only (a fresh `Claimed`/`FencedSelf`/`ForeignOwner` write/observe has no - /// separate "prior body's token to remember" use). `claimMountAwaitingExpiry`'s observation loop - /// used to re-GET the mount key itself just to recover this token that `claimMount` had already - /// read one line earlier and thrown away -- one wasted GET per iteration. Empty for every other + /// separate "prior body's incarnation to remember" use). `claimMountAwaitingExpiry`'s observation + /// loop would otherwise re-read the mount key just to recover what `claimMount` had already read + /// one line earlier and thrown away -- one wasted read per iteration. Empty for every other /// `Kind` (nothing to compare against). - std::optional token; + std::optional incarnation; }; /// Thrown when a mount operation observes that OUR OWN (uuid, epoch) slot was `gc_fenced` by the GC @@ -251,13 +260,14 @@ class MountFencedException : public DB::Exception : DB::Exception(msg, DB::ErrorCodes::ABORTED) {} }; -/// `proven_dead_token`: the write-token of a same-uuid, different-epoch lease that the CALLER already -/// proved dead by observation (see `claimMountAwaitingExpiry`) — matching it against the CURRENTLY -/// observed token is the ONLY way (besides `gc_fenced` / the clean marker) a same-uuid different-epoch -/// lease is ever reclaimed. Absent (`{}`, the default) for a bare claim attempt with no such proof. +/// `proven_dead_incarnation`: the incarnation of a same-uuid, different-epoch lease that the CALLER +/// already proved dead by observation (see `claimMountAwaitingExpiry`) — matching it against the +/// CURRENTLY observed incarnation is the ONLY way (besides `gc_fenced` / the clean marker) a +/// same-uuid different-epoch lease is ever reclaimed. Absent (`{}`, the default) for a bare claim +/// attempt with no such proof. MountClaimResult claimMount( - Backend & b, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, - uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_token = {}, + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, + uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation = {}, const CasEventSink & sink = {}); /// Format the operator-actionable startup error shown when the mount lease is held by a genuinely @@ -272,15 +282,15 @@ String mountDoubleStartMessage(const String & srid, const MountLease & existing) /// `Clean`), `ForeignOwner`, or `FencedSelf`; /// - a `LiveDoubleStart` from OUR OWN uuid (a stale lease from a prior incarnation of this server, /// OR a genuinely live twin — the two are indistinguishable from a bare read) is resolved by -/// WATCHING the lease's write-token on OUR OWN clock (`mono_ms_fn`), NEVER by comparing the -/// lease's stamped `expires_at_ms` against any clock: once the observed token has held stable for +/// WATCHING the lease's incarnation on OUR OWN clock (`mono_ms_fn`), NEVER by comparing the +/// lease's stamped `expires_at_ms` against any clock: once the observed incarnation has held stable for /// the full rate-bound threshold (`ttl_ms + ttl_ms / 20 + poll_interval_ms` — the lease TTL, a 5% /// clock-drift allowance, and one poll interval of discreteness. This rate bound ensures that a /// holder which last renewed before the observation began can no longer be within its lease. -/// that token is handed to `claimMount` as `proven_dead_token`, which then reclaims token-guarded -/// (`prior = UncleanObserved`). If the token changes DURING the wait (the holder renewed, or a -/// genuine twin is alive) the observation RESTARTS from the new token; bounded to a handful of -/// restarts before giving up and returning the last `LiveDoubleStart` (a holder whose token keeps +/// that incarnation is handed to `claimMount` as `proven_dead_incarnation`, which then reclaims +/// incarnation-guarded (`prior = UncleanObserved`). If the incarnation changes DURING the wait (the +/// holder renewed, or a genuine twin is alive) the observation RESTARTS from the new one; bounded to +/// a handful of restarts before giving up and returning the last `LiveDoubleStart` (a holder whose incarnation keeps /// changing across that many restarts is alive, not dead). /// `now_ms_fn` is WALL clock, used only for stamping the body we (may) write / diagnostics — it never /// participates in the reclaim decision. `mono_ms_fn` is the OBSERVATION clock: monotonic on this @@ -296,7 +306,7 @@ String mountDoubleStartMessage(const String & srid, const MountLease & existing) uint64_t mountObservationThresholdMs(uint64_t ttl_ms, uint64_t cadence_ms); MountClaimResult claimMountAwaitingExpiry( - Backend & b, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, + CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, const std::function & now_ms_fn, const std::function & mono_ms_fn, uint64_t ttl_ms, uint64_t poll_interval_ms, @@ -304,21 +314,21 @@ MountClaimResult claimMountAwaitingExpiry( const std::function & on_wait_start = {}, const CasEventSink & sink = {}); -/// One `server_root_id`'s cross-round token-stability observation, +/// One `server_root_id`'s cross-round incarnation-stability observation, /// owned by the GC leader instance (`Cas::Gc::mount_obs`) and threaded through consecutive /// `computeHeartbeatFloor` calls — one GC round is one observation tick. Mirrors /// `claimMountAwaitingExpiry`'s observation loop, but at heartbeat-gate granularity rather than a /// tight poll loop. -struct MountTokenObservation +struct MountIncarnationObservation { - Token token; + Incarnation incarnation; uint64_t first_seen_mono_ms = 0; }; /// Keyed by `server_root_id`. In-memory only: a fresh leader (after a steal, or a process restart) /// starts with an empty map, which only delays fencing an already-dead mount by one extra round while /// it (re)establishes the observation — safe (never fences early), never unsafe. -using MountObservationMap = std::map; +using MountObservationMap = std::map; /// GC heartbeat gate (GC round protocol step 1). Run by the GC leader at the top of a round: LIST /// `gc/server-roots/` (O(servers), single-digit counts), GET each mount body, and classify + fence out @@ -331,17 +341,17 @@ using MountObservationMap = std::map; /// terminated marker; /// - otherwise, observation-based liveness (the same /// principle `claimMountAwaitingExpiry` uses for a mount's OWN reopen, applied here to the GC's -/// fence-out): `obs` remembers, per srid, the write-token last seen and the leader's OWN -/// monotonic clock reading (`mono_now_ms`) at the moment it first saw that token. A body whose -/// CURRENT token differs from (or is absent from) `obs` is (re)started fresh — counted `live`, +/// fence-out): `obs` remembers, per srid, the incarnation last seen and the leader's OWN +/// monotonic clock reading (`mono_now_ms`) at the moment it first saw it. A body whose +/// CURRENT incarnation differs from (or is absent from) `obs` is (re)started fresh — counted `live`, /// never fenced this call, regardless of what its stamped `expires_at_ms` claims (a bare /// wall-clock stamp is never trusted — see `claimMount`'s "certificate of death" doc). Only once -/// the SAME token has held for `>= stable_threshold_ms` OF THE LEADER'S OWN CLOCK does the body +/// the SAME incarnation has held for `>= stable_threshold_ms` OF THE LEADER'S OWN CLOCK does the body /// become FENCE-eligible; -/// - FENCE-eligible → one token-guarded `putOverwrite` preserving the WHOLE body, setting -/// `gc_fenced = true` and `seq + 1`. On `Done` → excluded (`fenced_now`); on `PreconditionFailed` -/// (the holder renewed concurrently — a live token change) → re-GET and reclassify from the top -/// (bounded retries; the reclassify sees the new token and restarts the observation, counting it +/// - FENCE-eligible → one incarnation-guarded `replace` preserving the WHOLE body, setting +/// `gc_fenced = true` and `seq + 1`. On `Committed` → excluded (`fenced_now`); on `Conflict` +/// (the holder renewed concurrently — a live incarnation change) → re-read and reclassify from the top +/// (bounded retries; the reclassify sees the new incarnation and restarts the observation, counting it /// `live` — conservative, never exclude a heartbeat without a landed fence-out). /// /// `now_ms` is WALL clock, used only for the audit/diagnostic log line — it never participates in the @@ -352,7 +362,7 @@ using MountObservationMap = std::map; /// fencing one round, never fences early). /// /// The fence-out is BOTH safety and liveness. Safety: a sleeper's later renewal permanently fails -/// (its `putOverwrite` now mismatches the fenced token → `tripMountLost`), so it can never re-arm +/// (its `replace` now mismatches the fenced incarnation → `tripMountLost`), so it can never re-arm /// without a fresh `open`. Liveness: a dead server's stale mount slot must not linger forever. /// Preserving the body keeps restart recovery intact: a same-uuid reopen reads the current body and /// reclaims through the normal expired-our-uuid branch. @@ -366,7 +376,7 @@ struct HeartbeatFloor std::vector fenced_srids; }; -HeartbeatFloor computeHeartbeatFloor(Backend & b, const Layout & l, uint64_t now_ms, +HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64_t now_ms, uint64_t mono_now_ms, uint64_t stable_threshold_ms, MountObservationMap & obs); @@ -394,7 +404,7 @@ struct NonTerminalMountSlot /// The caller is pool RECREATION (`Pool::open`'s bootstrap over a prefix with no authoritative /// `_pool_meta`): minting a fresh pool identity while a live writer still holds a slot would leave that /// writer appending its old-format transactions into the new pool. Writes nothing. -std::vector probeNonTerminalMountSlots(Backend & b, const Layout & l); +std::vector probeNonTerminalMountSlots(CasOperation & op, const Layout & l); /// A read-only snapshot of one server's mount slot, for introspection (`system.cas_mounts`). /// state: `live` (lease within TTL+skew), `expired` (lease ran out; the next GC round's heartbeat floor @@ -409,7 +419,7 @@ struct MountInfo /// Enumerate every mount slot under `gc/server-roots/`, decoded and classified — the read-only sibling /// of `computeHeartbeatFloor`: ZERO writes (no fence-out), per-row fail-open. One LIST + one GET per slot. -std::vector listMounts(Backend & backend, const Layout & layout, uint64_t now_ms, uint64_t skew_margin_ms); +std::vector listMounts(CasOperation & op, const Layout & layout, uint64_t now_ms, uint64_t skew_margin_ms); /// Whether the mounted writer identified by `(server_root_id, writer_epoch)` (the two fields of a /// `CatalogEntry::creator` / `CreatorFence`, `ref_catalog`'s spec INV-3 §3, that this predicate actually @@ -447,7 +457,7 @@ std::vector listMounts(Backend & backend, const Layout & layout, uint /// must fail the BUILD (a missing `-Wswitch` case), never silently read as terminal. /// /// Deliberately conservative on the two cases that are NOT proof of death: an ABSENT mount slot -/// (`Backend::get` returning `nullopt` answers nothing about liveness — it is not proof either way) +/// (a read finding the key absent answers nothing about liveness — it is not proof either way) /// and an UNDECODABLE body (an unreadable lease of some other format generation is precisely the case /// that must block, not the one to wave through, mirroring `probeNonTerminalMountSlots`'s own stated /// discipline for that case) both return `false` — refuse reconciliation rather than guess. @@ -461,11 +471,11 @@ std::vector listMounts(Backend & backend, const Layout & layout, uint /// root's OTHER activity, about whether `server_root_id` will ever mount again, or about /// anything beyond this one slot's current body at the instant of this GET. A caller that needs a /// stronger, race-free guarantee (e.g. "and it will never come back") must build that from a WIDER -/// observation, the way `claimMountAwaitingExpiry`'s token-stability window does for its own decision -- +/// observation, the way `claimMountAwaitingExpiry`'s stability window does for its own decision -- /// this function performs no such window and answers from one point-in-time read alone. Answering /// "unknown" (`false`, refuse) is the fail-closed choice on every path already listed above; there is /// no path where this function answers `true` on evidence weaker than one of the three certificates. -bool isCreatorFenceTerminal(Backend & backend, const Layout & layout, const String & server_root_id, +bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const String & server_root_id, uint64_t writer_epoch); /// Synchronous owner of the durable mount lease and merged build-watermark body. The stable @@ -476,16 +486,24 @@ bool isCreatorFenceTerminal(Backend & backend, const Layout & layout, const Stri /// (our_uuid, our_epoch), THEN `keeper.start()`. So `start`'s `claim` hook must ADOPT a live mount /// that is ALREADY ours — same `server_uuid` AND same `writer_epoch` — instead of self-tripping the /// live-double-start guard. The discriminator is the (uuid, epoch) pair: -/// - same uuid + same epoch → our own just-written claim (or a replay) → adopt: `putOverwrite` -/// against the observed token to refresh seq/expiry (no fail); +/// - same uuid + same epoch → our own just-written claim (or a replay) → adopt: `replace` +/// against the observed incarnation to refresh seq/expiry (no fail); /// - same uuid + DIFFERENT live epoch → a newer incarnation superseded us → fail closed; /// - foreign uuid → fail closed; -/// - absent → `putIfAbsent`; expired-our-uuid (any epoch) → `putOverwrite` reclaim. +/// - absent → `create`; expired-our-uuid (any epoch) → `replace` reclaim. +/// +/// PLANES. Only the RENEWAL is admitted under the mount fence, because only a renewal writes under +/// authority the fence is tracking. The claim and the farewell are admitted off it: a self-remount +/// claims with the fence already latched lost, so a claim gated on the fence could never reclaim, and +/// a farewell refused because the fence has run down would leave the slot looking live until GC +/// fences it out. Neither is unguarded: a claim's safety is its own conditional write, and a caller +/// that has shutdown facts hands them over as a `Liveness`. class MountLeaseKeeper { public: MountLeaseKeeper( - BackendPtr backend_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, + CasRequests & mount_requests_, CasRequests & open_requests_, const Layout & layout_, + const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, std::function min_active_build_sequence_fn_, CasEventSink event_sink_ = {}, @@ -494,9 +512,17 @@ class MountLeaseKeeper /// tests and wired by CasMountRuntime::installKeeper. std::function boot_ms_fn_ = {}); - /// Adopt the already-claimed mount. Returns the exact pre-I/O BOOTTIME anchor. - uint64_t start(); - MountRenewResult renew(const CasRequestBudget & budget, const MountRenewOperationEnvironment & environment); + /// Adopt the already-claimed mount. Returns the exact pre-I/O BOOTTIME anchor. `liveness` carries + /// the caller's shutdown terms; the mount fence is deliberately not consulted here. + uint64_t start(Liveness liveness = {}); + /// The steady-state renewal, admitted under the mount fence. + MountRenewResult renew(const MountRenewOperationEnvironment & environment); + /// The remount's re-anchor, which is bootstrap control rather than steady state: a remount renews + /// BEFORE it arms the fence for the new incarnation, so the fence is still latched lost and an + /// operation admitted under it would be refused before its first attempt. It admits on this + /// keeper's own open plane, the one the claim and the farewell use, so there is no plane for a + /// caller to get wrong. Same policy and same verdicts as `renew`. + MountRenewResult renewForRemount(const MountRenewOperationEnvironment & environment = {}); void release(); MountLeaseKeeperState state() const { return keeper_state; } @@ -505,15 +531,18 @@ class MountLeaseKeeper private: String encodeBody(uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const; - Token claim(const String & body); - [[noreturn]] void throwRenewConflict(const CasOverwriteDiagnostics & diagnostics) const; - MountRenewResult terminalResult( - uint64_t attempt_start_boot_ms, - CasOverwriteDiagnostics diagnostics, - std::exception_ptr failure); - void terminate(); - - BackendPtr backend; + /// The incarnation every guarded write of this slot names. Engaged for exactly the states that + /// admit such a write: `start` establishes it and each committed renewal replaces it. + const Incarnation & precondition() const; + /// One renewal admitted on `plane`; `renew` and `renewForRemount` differ only in which they pass. + MountRenewResult renewOn(CasRequests & plane, const MountRenewOperationEnvironment & environment); + Incarnation claim(CasOperation & op, const String & body); + [[noreturn]] void throwRenewConflict(const Observation & seen) const; + MountRenewResult terminalResult(MountRenewResult result); + void terminate(CasOperation & op); + + CasRequests & mount_requests; + CasRequests & open_requests; String key; String srid; @@ -529,7 +558,9 @@ class MountLeaseKeeper std::function boot_ms_fn; MountLeaseKeeperState keeper_state = MountLeaseKeeperState::New; uint64_t seq = 0; - Token last_token; + /// The incarnation our last landed write created; every renewal and the farewell name it as the + /// precondition. Unset only before `start` has landed one. + std::optional last_incarnation; uint64_t confirmed_deadline_boot_ms = 0; uint64_t last_committed_attempt_start_boot_ms = 0; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index 44a86a0c312b..3cf9938cedf9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include #include @@ -7,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -34,68 +36,76 @@ uint64_t nowMs() std::chrono::system_clock::now().time_since_epoch()).count()); } -/// Delete every object listed under `prefix` by its listed (or, absent a list-token backend, HEAD'd) -/// token. This backs the staging and roots drain phases below: the victim's writers are fenced by the -/// decommission claim (`Pool::openForDecommission`), so nothing should be racing these deletes, and a -/// plain exact-token delete of every listed object is race-free. +std::string_view removalName(Removal r) +{ + switch (r) + { + case Removal::Removed: return "removed"; + case Removal::Gone: return "gone"; + case Removal::Mismatch: return "mismatch"; + } + UNREACHABLE(); +} + +/// Delete every object listed under `prefix` by its listed (or, absent a list-incarnation backend, +/// HEAD'd) incarnation. This backs the staging and roots drain phases below: the victim's writers are +/// fenced by the decommission claim (`Pool::openForDecommission`), so nothing should be racing these +/// deletes, and a plain exact-incarnation delete of every listed object is race-free. /// -/// A per-object failure — a backend exception, a `TokenMismatch` or `NotFound` outcome, or an object +/// A per-object failure — a backend exception, a `Mismatch` or `Gone` outcome, or an object /// disappearing between `LIST` and `HEAD` — is recorded as a warning and does not prevent the remaining /// objects from being attempted. The caller keeps the pool slot whenever warnings are present, so the /// terminated slot remains available as a resume anchor instead of being deleted after an unconfirmed -/// drain. Returns only the objects whose exact-token delete was reported as `Deleted`. -uint64_t deleteListedPrefix(Backend & backend, const String & prefix, std::vector & warnings) +/// drain. Returns only the objects whose exact-incarnation delete was reported as `Removed`. +uint64_t deleteListedPrefix(CasOperation & op, const String & prefix, std::vector & warnings) { uint64_t deleted = 0; - forEachListedKey(backend, prefix, [&](const ListedKey & listed) + op.forEachListedKey(prefix, [&](const KeyEntry & listed) { try { - Token token; - if (listed.token) - token = *listed.token; - else + std::optional incarnation = listed.incarnation; + if (!incarnation) { - const HeadResult head = backend.head(listed.key); - if (!head.exists) + const std::optional head = op.head(listed.key, Retry::standard()); + if (!head) { warnings.push_back("decommission drain: " + listed.key + " vanished before delete"); - return; + return true; } - token = head.token; + incarnation = head->incarnation; } - const DeleteOutcome outcome = backend.deleteExact(listed.key, token); - const DeleteClass outcome_class = classifyDeleteOutcome(outcome); - if (outcome_class == DeleteClass::Deleted) + const Removal outcome = op.remove(listed.key, *incarnation, Retry::standard()); + if (outcome == Removal::Removed) ++deleted; else warnings.push_back("decommission drain: " + listed.key + " delete outcome " - + String(deleteClassName(outcome_class))); + + String(removalName(outcome))); } catch (...) { warnings.push_back("decommission drain: " + listed.key + " delete failed: " + getCurrentExceptionMessage(/*with_stacktrace=*/false)); } - }); + return true; + }, Retry::standard()); return deleted; } -/// Delete one slot control object by a token captured at the protocol-defined fence point. Slot -/// retirement is fail-closed: unlike the debris drains above, any non-`Deleted` outcome or exception +/// Delete one slot control object by an incarnation captured at the protocol-defined fence point. Slot +/// retirement is fail-closed: unlike the debris drains above, any non-`Removed` outcome or exception /// stops the tail before it can touch the next control object. -bool deleteSlotObject(Backend & backend, const String & key, const Token & token, std::vector & warnings) +bool deleteSlotObject(CasOperation & op, const String & key, const Incarnation & incarnation, std::vector & warnings) { try { - const DeleteOutcome outcome = backend.deleteExact(key, token); - const DeleteClass outcome_class = classifyDeleteOutcome(outcome); - if (outcome_class == DeleteClass::Deleted) + const Removal outcome = op.remove(key, incarnation, Retry::standard()); + if (outcome == Removal::Removed) return true; warnings.push_back("slot delete failed: " + key + ": delete outcome " - + String(deleteClassName(outcome_class))); + + String(removalName(outcome))); } catch (...) { @@ -122,16 +132,29 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, request_gc_round(); }); + /// Decommission is administrative and non-hot-path: it has no mount-lease fence of its own -- the + /// exact-incarnation compare on every write is the safety mechanism -- so it opens its own + /// always-admitted request engine rather than one of `Pool`'s fenced planes. The pre-impersonation + /// cut below runs against the raw backend passed in, because `Pool::openForDecommission` has not + /// yet wrapped it for instrumentation and no `Pool` exists yet to route through. + CasRequests preflight_requests(backend, Fence::open()); + CasOperation preflight_op = preflight_requests.admit(); + /// Validate one required immutable ownership cut before impersonating the victim. The admin open /// performs its own fresh catalog observation for mount safety, but namespace selection below /// must reuse this exact pre-mutation decision rather than read a later authority set. const Layout catalog_layout(config.pool_prefix); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*backend, catalog_layout); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(preflight_op, catalog_layout); catalog_cut.life_index.throwIfAmbiguous("CAS decommission"); config.event_sink = sink; PoolPtr admin = Pool::openForDecommission(std::move(backend), std::move(config), victim_srid); + /// A second engine over the pool's own (now instrumented) backend: `CasRequests` keeps its own + /// shared_ptr to it, so `op` stays usable after `admin.reset()` retires the `Pool` below. + CasRequests requests(admin->poolBackendPtr(), Fence::open()); + CasOperation op = requests.admit(); + EventEmitter{*admin}.emit([&](CasEvent & e) { e.type = CasEventType::MemberDecommission; @@ -164,7 +187,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// Refuse a same-name lifecycle move that landed after the immutable selection cut. The /// exact-life overloads below also pin recovery to `life`, closing the race after this check: /// a later replacement can never redirect a removal to its new incarnation. - const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(admin->backend(), admin->layout()); + const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(op, admin->layout()); const auto current_entry = std::find_if( current_catalog.catalog.entries.begin(), current_catalog.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns.string() == ns_str; }); @@ -176,7 +199,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, if (selected_entry.state == NsState::Removing) { - if (!admin->backend().head(admin->layout().refCkptKey(life)).exists) + if (!op.head(admin->layout().refCkptKey(life), Retry::standard()).has_value()) throw Exception(ErrorCodes::CORRUPTED_DATA, "ca-decommission: namespace '{}' is Removing but its exact checkpoint is absent; " "the catalog row remains owned and the victim slot cannot be retired", @@ -219,11 +242,12 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, { const String debris_prefix = admin->layout().casManifestsServerPrefix(victim_srid); std::set> groups; /// (namespace, writer epoch, build sequence) - forEachListedKey(admin->backend(), debris_prefix, [&](const ListedKey & listed) + op.forEachListedKey(debris_prefix, [&](const KeyEntry & listed) { if (const auto parsed = admin->layout().parseManifestKey(listed.key)) groups.emplace(parsed->root_namespace.string(), parsed->ref.writer_epoch, parsed->ref.build_sequence); - }); + return true; + }, Retry::standard()); for (const auto & [ns_str, writer_epoch, build_sequence] : groups) report.manifest_debris_removed += sweepNamespace( *admin, RootNamespace(ns_str), BuildPrefix{writer_epoch, build_sequence}, &report.warnings); @@ -233,23 +257,23 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// an `IObjectStorage`, while this command intentionally works at the `Backend` layer, so the same /// prefix is listed and deleted directly. The claim fences the victim's writers during this sweep. report.staging_objects_removed += deleteListedPrefix( - admin->backend(), admin->poolConfig().pool_prefix + "/staging/" + victim_srid + "/", report.warnings); + op, admin->poolConfig().pool_prefix + "/staging/" + victim_srid + "/", report.warnings); /// Drain the victim's mountpoint objects. These are loose, non-content-addressed files under /// `Layout::serverRootDataPrefix`; they have no writer epoch of their own, so the claim is what /// prevents a returning victim from racing this deletion. report.mountpoint_objects_removed += deleteListedPrefix( - admin->backend(), admin->layout().serverRootDataPrefix(victim_srid), report.warnings); + op, admin->layout().serverRootDataPrefix(victim_srid), report.warnings); /// The catalog, not physical debris, owns the slot-retirement decision. A terminal append only /// moves a row to `Removing`; GC must fold/prune/delete it before the member's ownership anchor can - /// disappear. Capture one exact whole-catalog cut after every drain, then revalidate its token and - /// canonical value immediately before entering the retirement tail. The administrative claim fences - /// the victim writer between those observations. + /// disappear. Capture one exact whole-catalog cut after every drain, then revalidate its incarnation + /// and canonical value immediately before entering the retirement tail. The administrative claim + /// fences the victim writer between those observations. std::optional retirement_catalog_cut; if (report.warnings.empty()) { - retirement_catalog_cut = CasRefCatalog::read(admin->backend(), admin->layout()); + retirement_catalog_cut = CasRefCatalog::read(op, admin->layout()); const uint64_t victim_owned_count = std::count_if( retirement_catalog_cut->catalog.entries.begin(), retirement_catalog_cut->catalog.entries.end(), [&](const CatalogEntry & entry) @@ -264,17 +288,15 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, "cleanup — re-run this command afterwards to retire the slot"); } - /// Retire the slot strictly last and only after a clean drain. Copy the layout and shared backend - /// before `admin.reset()`: graceful close destroys the `Pool`, while the backend must remain alive to - /// retire the slot objects afterwards. + /// Retire the slot strictly last and only after a clean drain. Copy the layout before + /// `admin.reset()`: graceful close destroys the `Pool`, while `op` (holding its own shared_ptr to + /// the backend) remains usable to retire the slot objects afterwards. const Layout layout = admin->layout(); - const BackendPtr pool_backend = admin->poolBackendPtr(); if (report.warnings.empty()) { - const CasRefCatalog::Snapshot fresh_retirement_catalog - = CasRefCatalog::read(admin->backend(), admin->layout()); + const CasRefCatalog::Snapshot fresh_retirement_catalog = CasRefCatalog::read(op, admin->layout()); if (!retirement_catalog_cut - || fresh_retirement_catalog.token != retirement_catalog_cut->token + || fresh_retirement_catalog.incarnation != retirement_catalog_cut->incarnation || fresh_retirement_catalog.catalog != retirement_catalog_cut->catalog) { report.warnings.push_back( @@ -287,13 +309,13 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const String epoch_key = layout.epochKey(victim_srid); const String owner_key = layout.ownerKey(victim_srid); - /// Capture both the epoch value and its exact token while the decommission claim still fences - /// the victim. A successor can only bump this object after the farewell below releases the - /// claim, so this token is the epoch-side successor fence for the retirement tail. - std::optional claimed_epoch; + /// Capture both the epoch value and its exact incarnation while the decommission claim still + /// fences the victim. A successor can only bump this object after the farewell below releases + /// the claim, so this incarnation is the epoch-side successor fence for the retirement tail. + std::optional claimed_epoch; try { - claimed_epoch = pool_backend->get(epoch_key); + claimed_epoch = op.read(epoch_key, Retry::standard()); if (!claimed_epoch) report.warnings.push_back("slot capture failed: " + epoch_key + " is absent under the admin claim"); } @@ -308,14 +330,14 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// are removed and its owner anchor is tombstoned. admin.reset(); - /// Read the farewell immediately after `finishTeardown` wrote it. Its exact token is the - /// mount-side fence: deleting by this token can remove only THIS decommission's farewell, not - /// a successor reclaim. Validate the body against the epoch value captured under the claim so - /// a successor that completed before this GET is also recognized and left untouched. - std::optional farewell_mount; + /// Read the farewell immediately after `finishTeardown` wrote it. Its exact incarnation is the + /// mount-side fence: deleting by this incarnation can remove only THIS decommission's farewell, + /// not a successor reclaim. Validate the body against the epoch value captured under the claim + /// so a successor that completed before this read is also recognized and left untouched. + std::optional farewell_mount; try { - farewell_mount = pool_backend->get(mount_key); + farewell_mount = op.read(mount_key, Retry::standard()); if (!farewell_mount) report.warnings.push_back("slot capture failed: " + mount_key + " farewell is absent"); } @@ -352,16 +374,16 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, } /// Mount first: if a successor reclaimed it after the farewell capture, the stale farewell - /// token yields `TokenMismatch` and the tail stops before touching epoch or owner. Epoch second: - /// its under-claim token similarly detects a successor allocation. Before touching owner, re-read - /// both mutable objects: a same-UUID successor can recreate them after both deletes without - /// rewriting the owner identity anchor. Mere presence proves that the slot is live again. Every - /// delete must be explicitly confirmed as `Deleted`, and the final owner tombstone rewrite must - /// succeed against the exact token read immediately before it. + /// incarnation yields `Mismatch` and the tail stops before touching epoch or owner. Epoch + /// second: its under-claim incarnation similarly detects a successor allocation. Before + /// touching owner, re-read both mutable objects: a same-UUID successor can recreate them after + /// both deletes without rewriting the owner identity anchor. Mere presence proves that the slot + /// is live again. Every delete must be explicitly confirmed as `Removed`, and the final owner + /// tombstone rewrite must succeed against the exact incarnation read immediately before it. /// /// ACCEPTED RESIDUAL WINDOW (final review, not closed by this recheck): a same-UUID successor /// can still recreate epoch/mount in the narrow gap strictly AFTER this liveness recheck but - /// BEFORE the owner CAS below reads its own token -- the successor's owner anchor (same + /// BEFORE the owner CAS below reads its own incarnation -- the successor's owner anchor (same /// server_uuid, not yet retired) then gets tombstoned by this decommission run. The successor's /// live process is not deleted (only its owner anchor is marked retired), but a LATER restart of /// that same identity would refuse to reclaim it (claimOwnerOrThrow's tombstone guard). This is @@ -369,15 +391,15 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// (finding #9) intentionally stopped short of making concurrent decommission-vs-recreate /// airtight to the microsecond, since that was explicitly not the priority for this fix. report.slot_removed = false; - if (captures_match && deleteSlotObject(*pool_backend, mount_key, farewell_mount->token, report.warnings) - && deleteSlotObject(*pool_backend, epoch_key, claimed_epoch->token, report.warnings)) + if (captures_match && deleteSlotObject(op, mount_key, farewell_mount->incarnation, report.warnings) + && deleteSlotObject(op, epoch_key, claimed_epoch->incarnation, report.warnings)) { - std::optional current_mount; - std::optional current_epoch; + std::optional current_mount; + std::optional current_epoch; bool liveness_recheck_succeeded = true; try { - current_mount = pool_backend->get(mount_key); + current_mount = op.read(mount_key, Retry::standard()); } catch (...) { @@ -387,7 +409,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, } try { - current_epoch = pool_backend->get(epoch_key); + current_epoch = op.read(epoch_key, Retry::standard()); } catch (...) { @@ -405,33 +427,35 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, { try { - if (const auto owner = pool_backend->get(owner_key)) + if (const auto owner = op.read(owner_key, Retry::standard())) { OwnerObject tombstoned = decodeOwner(owner->bytes); tombstoned.retired_at_ms = nowMs(); - /// Controlled, not a bare putOverwrite: a transient transport error here (or - /// one whose response was simply lost) must not be reported as a hard failure - /// when the write actually landed. A standalone controller (decommission is an - /// administrative, non-hot-path operation; no mount-lease fence applies to it - /// -- the exact-token CAS itself is the safety mechanism, same as the mount/ - /// epoch deletes above) resolves an ambiguous attempt with one GET: unchanged - /// token means the write never applied (legitimately retryable within budget); - /// matching bytes means this exact tombstone already landed (Committed, not a - /// failure); anything else is a genuine successor reclaim (Conflict). - CasRequestController controller(pool_backend, CasRequestBudget{}); - const CasOverwriteResult result = controller.putOverwriteControlled( - owner_key, encodeOwner(tombstoned), owner->token, [] { return true; }); - if (result.outcome == CasOverwriteOutcome::Committed) + /// `op.replace` resolves an ambiguous attempt with a resolve read on its own: + /// a transient transport error here (or one whose response was simply lost) + /// must not be reported as a hard failure when the write actually landed. + /// Unchanged incarnation means the write never applied (legitimately retryable + /// within budget); matching bytes means this exact tombstone already landed + /// (`Committed`, not a failure); a genuine successor reclaim is `Conflict`; + /// `Refused` is the store's own definite answer (a denial, a malformed + /// request, an expired credential) and carries its own code and message, + /// which is worth more here than the generic retry advice below. + WriteResult result = op.replace(owner_key, encodeOwner(tombstoned), owner->incarnation, Retry::standard()); + if (std::holds_alternative(result)) report.slot_removed = true; - else if (result.outcome == CasOverwriteOutcome::Conflict) + else if (std::holds_alternative(result)) report.warnings.push_back( "slot tombstone failed: " + owner_key + ": successor reclaimed the owner anchor before this decommission's tombstone write"); + else if (const Refused * refused = std::get_if(&result)) + report.warnings.push_back( + "slot tombstone failed: " + owner_key + ": the store refused the write (" + + std::to_string(refused->store_error) + "): " + refused->message); else report.warnings.push_back( "slot tombstone failed: " + owner_key + ": tombstone write outcome could not be resolved (retry budget exhausted " - "or the resolve GET itself failed) -- rerun the command to retry"); + "or the resolve read itself failed) -- rerun the command to retry"); } else report.warnings.push_back( diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp index 33af5b0ec1e9..cc0cf437f281 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp @@ -1,4 +1,4 @@ -#include +#include #include #include #include @@ -51,13 +51,13 @@ void checkDeadline(const Deadline & deadline, std::string_view phase) "fsck: exceeded the deadline during '{}' — run against a QUIESCED pool or raise --timeout.", phase); } -void listAll(Backend & backend, const String & prefix, std::unordered_map & out, +void listAll(CasOperation & op, const String & prefix, std::unordered_map & out, const FsckProgress & on_progress, const Deadline & deadline, std::string_view phase) { static constexpr size_t kPageLimit = 1000; uint64_t pages = 0; size_t count_in_page = 0; - forEachListedKey(backend, prefix, [&](const ListedKey & k) + op.forEachListedKey(prefix, [&](const KeyEntry & k) { out[k.key] = k.size; if (++count_in_page == kPageLimit) @@ -68,8 +68,9 @@ void listAll(Backend & backend, const String & prefix, std::unordered_map 0 || pages == 0) { @@ -124,12 +125,12 @@ using RecordRecoveryUnchecked = std::function recoverLateRefTable( - Backend & backend, const Layout & layout, const FsckRecoveryAuthority & authority, + CasOperation & op, const Layout & layout, const FsckRecoveryAuthority & authority, const RecordRecoveryUnchecked & record_unchecked) { try { - const std::optional sampled = readCkpt(backend, layout, authority.life); + const std::optional sampled = readCkpt(op, layout, authority.life); if (!sampled) { record_unchecked(authority.life.ns, layout.refCkptKey(authority.life), @@ -137,7 +138,7 @@ std::optional recoverLateRefTable( return std::nullopt; } return recoverRefTableDetailedFromAuthority( - backend, layout, authority.catalog_entry, sampled->ckpt).state; + op, layout, authority.catalog_entry, sampled->ckpt).state; } catch (const Exception & e) { @@ -153,7 +154,7 @@ std::optional recoverLateRefTable( } } -bool blobStillReferenced(Pool & store, const Layout & layout, +bool blobStillReferenced(CasOperation & op, Pool & store, const Layout & layout, const FsckRecoveryAuthorities & authorities, const String & bkey, const std::vector & labels, const Deadline & deadline, const RecordRecoveryUnchecked & record_unchecked) @@ -181,7 +182,7 @@ bool blobStillReferenced(Pool & store, const Layout & layout, } const RootNamespace rns{ns_part}; const std::optional table = recoverLateRefTable( - store.backend(), layout, authority_it->second, record_unchecked); + op, layout, authority_it->second, record_unchecked); if (!table) return true; const auto rit = table->getCommitted().find(ref_name); @@ -205,7 +206,7 @@ bool blobStillReferenced(Pool & store, const Layout & layout, } /// The manifest sibling of the `blobStillReferenced` recheck above. The ref-walk captures each committed -/// `(ref_name -> manifest_ref)` from a FRESH per-namespace recovery, but the `backend.get(mkey)` that +/// `(ref_name -> manifest_ref)` from a FRESH per-namespace recovery, but the read of `mkey` that /// confirms the manifest body runs LATER in the same (possibly long) namespace loop. A ref republished to /// a DIFFERENT manifest — or DROPPED — in that window, combined with a legitimate GC delete of the OLD /// manifest body, makes the stale captured row look like a committed ref over a missing manifest (a @@ -217,7 +218,7 @@ bool blobStillReferenced(Pool & store, const Layout & layout, /// but a same-life checkpoint advance must be visible. Fails CLOSED on any ambiguity (a throw, a corrupt /// table): treated as "still referenced", the original conservative verdict — the fix can only SHRINK /// false positives, never hide a real loss. -bool manifestStillReferenced(Backend & backend, const Layout & layout, const RootNamespace & ns, +bool manifestStillReferenced(CasOperation & op, const Layout & layout, const RootNamespace & ns, const FsckRecoveryAuthorities & authorities, const String & ref_name, const String & mkey, const Deadline & deadline, const RecordRecoveryUnchecked & record_unchecked) @@ -233,7 +234,7 @@ bool manifestStillReferenced(Backend & backend, const Layout & layout, const Roo return true; /// no original Live/Removing authority -- fail closed } const std::optional table = recoverLateRefTable( - backend, layout, authority_it->second, record_unchecked); + op, layout, authority_it->second, record_unchecked); if (!table) return true; const auto rit = table->getCommitted().find(ref_name); @@ -308,7 +309,7 @@ struct NsVerdicts /// `{life_epoch, 1}`) through that inclusive frontier. A missing required id is therefore a proven hole; /// no above-hole listing witness is needed. An epoch seal advances directly to the next epoch's first id, /// exactly as authoritative read-only recovery does. -void checkRefStream(Backend & backend, const Layout & layout, const NamespaceLifeId & life, +void checkRefStream(CasOperation & op, const Layout & layout, const NamespaceLifeId & life, const CatalogEntry & catalog_entry, const std::optional & checkpoint_sample, const Deadline & deadline, FsckReport & report, NsVerdicts & verdicts) { @@ -323,7 +324,7 @@ void checkRefStream(Backend & backend, const Layout & layout, const NamespaceLif { /// Even when the base IS the frontier and there is no replay tail, a checkpoint may not /// turn an `EpochSeal` into a state snapshot by naming a same-id `_snap`. - (void)readCheckpointSnapshotBase(backend, layout, life, *checkpoint); + (void)readCheckpointSnapshotBase(op, layout, life, *checkpoint); } catch (const Exception & e) { @@ -343,8 +344,8 @@ void checkRefStream(Backend & backend, const Layout & layout, const NamespaceLif checkDeadline(deadline, "checkpoint-base authority revalidation"); try { - const std::optional current = readCkpt(backend, layout, life); - if (!current || !checkpoint_sample || current->token != checkpoint_sample->token) + const std::optional current = readCkpt(op, layout, life); + if (!current || !checkpoint_sample || current->incarnation != checkpoint_sample->incarnation) { verdicts.recordUnchecked(report, ns, key, note + "; checkpoint authority changed while validating its snapshot base"); @@ -375,7 +376,7 @@ void checkRefStream(Backend & backend, const Layout & layout, const NamespaceLif while (expected <= *grounding.committed_through) { checkDeadline(deadline, "ref stream"); - const auto got = backend.get(layout.refLogKey(life, expected)); + const auto got = op.read(layout.refLogKey(life, expected), Retry::standard()); if (!got) { verdicts.recordChainBroken(report, ns, layout.refLogKey(life, expected), @@ -424,7 +425,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co const String & namespace_prefix, FsckReport & report) { const Layout & layout = store.layout(); - Backend & backend = store.backend(); + CasOperation op = store.gcRequests().admit(); /// Path-derived per-object algorithm parsing: every listed blob-tree key -- across every /// admitted algo, not just the pool's node-local write algo -- is classified via /// `Layout::parseBlobKey`, which derives the `BlobRef` from the key's OWN `` path segment @@ -479,7 +480,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// attribution (its physical keys may exist) but is never recovered: only Live/Removing rows have a /// durable publication frontier. A diagnostic records duplicate ids and keeps walking unrelated /// unique lives. - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); struct FsckWalkLife { NamespaceLifeId life; @@ -525,7 +526,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co }; std::vector canonical_candidates; - forEachListedKey(backend, layout.namespaceRootPrefix(), [&](const ListedKey & listed) + op.forEachListedKey(layout.namespaceRootPrefix(), [&](const KeyEntry & listed) { std::optional physical_id; try @@ -539,7 +540,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co else { recordLifelessKeys(NamespaceListing{{}, {{listed.key, "unrecognized key under the namespace ownership tree"}}}); - return; + return true; } } catch (const Exception & e) @@ -547,14 +548,15 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co if (e.code() != ErrorCodes::CORRUPTED_DATA) throw; recordLifelessKeys(NamespaceListing{{}, {{listed.key, e.message()}}}); - return; + return true; } canonical_candidates.push_back(CanonicalNamespaceKey{listed.key, listed.size, *physical_id}); - }); + return true; + }, Retry::standard()); /// The post-observation cut. All three catalog states -- `Creating`, `Live`, `Removing` -- /// protect a life for this purpose; only a life absent from every one of them is residue. - const CasRefCatalog::Snapshot post_listing_cut = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot post_listing_cut = CasRefCatalog::read(op, layout); std::unordered_set pending_lives; for (const CanonicalNamespaceKey & candidate : canonical_candidates) { @@ -615,7 +617,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// One materialized `_ckpt` body is part of this namespace's frozen audit authority. The /// recovery API receives exactly these bytes; `checkRefStream` receives the same decoded /// value, so the two legs cannot quietly choose different frontiers after a concurrent CAS. - const std::optional checkpoint_sample = readCkpt(backend, layout, life); + const std::optional checkpoint_sample = readCkpt(op, layout, life); const std::optional checkpoint = checkpoint_sample ? std::optional{checkpoint_sample->ckpt} : std::nullopt; const auto [authority_it, inserted] = recovery_authorities.emplace( @@ -627,12 +629,12 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// it (`chain-broken`) rather than the downstream `CORRUPTED_DATA` the replay below would /// raise about the same hole. checkRefStream( - backend, layout, life, walk_life.catalog_entry, checkpoint_sample, deadline, report, verdicts); + op, layout, life, walk_life.catalog_entry, checkpoint_sample, deadline, report, verdicts); /// This recovery's finite range comes from the original catalog row and exact `_ckpt`, never /// from a stream listing, a self-resolved name, or an F+1 probe. const RefTableState table = recoverRefTableDetailedFromAuthority( - backend, layout, authority_it->second.catalog_entry, authority_it->second.checkpoint).state; + op, layout, authority_it->second.catalog_entry, authority_it->second.checkpoint).state; for (const auto [ref_name, row] : table.getCommitted()) { const ManifestId id{ns, row.manifest_ref}; @@ -640,7 +642,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co owned_manifest_keys.insert(mkey); const String label = ns_str + "/" + ref_name; - const auto got = backend.get(mkey); + const auto got = op.read(mkey, Retry::standard()); if (!got) { /// A committed ref naming a missing manifest body would be an INV-NO-DANGLE violation — @@ -651,8 +653,8 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// plus a fresh checkpoint from its physical life. Count the dangle ONLY when the exact /// object is HEAD-absent AND that life still names THIS exact manifest — otherwise it is /// LIST/GET lag or a phantom stale-row, never a loss. - if (!backend.head(mkey).exists - && manifestStillReferenced(backend, layout, ns, recovery_authorities, ref_name, mkey, + if (!op.head(mkey, Retry::standard()) + && manifestStillReferenced(op, layout, ns, recovery_authorities, ref_name, mkey, deadline, record_recovery_unchecked)) { ++report.dangling; @@ -725,7 +727,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// misread as an unreferenced blob and fall into the dangling/pending/unaccounted pipeline /// below), and a body must never be misread as a `.meta`. std::unordered_map present_all; - listAll(backend, layout.blobsPrefix(), present_all, on_progress, deadline, "listing blobs"); + listAll(op, layout.blobsPrefix(), present_all, on_progress, deadline, "listing blobs"); std::unordered_map present_blobs; std::unordered_set present_meta_hashes; present_blobs.reserve(present_all.size()); @@ -751,12 +753,11 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co uint64_t size = exists ? it->second : 0; if (!exists) { - const HeadResult h = backend.head(bkey); - if (h.exists) + if (const std::optional h = op.head(bkey, Retry::standard())) { exists = true; - size = h.size; - report.physical_bytes += h.size; + size = h->size; + report.physical_bytes += h->size; } } @@ -766,7 +767,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// Before declaring a loss, re-resolve the referencing refs from the original audit /// authority. A later rebirth must not replace the old owner while this verdict is being /// decided. - const bool still_referenced = blobStillReferenced(store, layout, recovery_authorities, bkey, + const bool still_referenced = blobStillReferenced(op, store, layout, recovery_authorities, bkey, lit != blob_labels.end() ? lit->second : std::vector{}, deadline, record_recovery_unchecked); if (!still_referenced) @@ -814,7 +815,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co if (!unref_hashes.empty()) { - if (const auto state_got = backend.get(layout.gcStateKey())) + if (const auto state_got = op.read(layout.gcStateKey(), Retry::standard())) { have_gc_state = true; const GcState gc_state = decodeGcState(state_got->bytes); @@ -830,7 +831,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// the full identity parsed from the listed blob key. This is required for mixed-algorithm /// pools: a 64-hex digest must not be truncated or compared as though it used the pool's /// local write algorithm, or its true GC state could be hidden as `Unaccounted`. - if (const auto seal_got = backend.get(layout.foldSealKey(gc_state.snap_generation, gc_state.snap_attempt))) + if (const auto seal_got = op.read(layout.foldSealKey(gc_state.snap_generation, gc_state.snap_attempt), Retry::standard())) { uint64_t rows = 0; for (const RunRef & run : decodeFoldSeal(seal_got->bytes, gc_state.snap_generation).blob_target_runs) @@ -839,7 +840,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// Typed open: the source-edge run reader goes through openSourceEdgeRun (the NDJSON /// header gates type == cas_run + kind == source_edge). Fsck keys off the row's hash /// (the record's own algo-prefixed key, never from pool meta). - SourceEdgeRunView reader = openSourceEdgeRun(backend, run.key); + SourceEdgeRunView reader = openSourceEdgeRun(op, run.key); String key; String payload; while (reader.next(key, payload)) @@ -911,7 +912,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co { const RootNamespace ns{ns_str}; std::unordered_map manifest_bodies; - listAll(backend, layout.manifestNamespacePrefix(ns), manifest_bodies, on_progress, deadline, + listAll(op, layout.manifestNamespacePrefix(ns), manifest_bodies, on_progress, deadline, "listing manifests for the stale-edge check"); for (const auto & [mkey, _] : manifest_bodies) { @@ -919,9 +920,9 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co const std::optional id = layout.parseManifestKey(mkey); if (!id) continue; /// foreign/malformed key under `manifests/` — contributes no source edge - const auto got = backend.get(mkey); + const auto got = op.read(mkey, Retry::standard()); if (!got) - continue; /// gone between the LIST and the GET — genuinely not a live source + continue; /// gone between the LIST and the read — genuinely not a live source try { const PartManifest body = decodePartManifest(openObject(FormatId::PartManifest, got->bytes)); @@ -954,11 +955,14 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co FsckClass cls = FsckClass::Unaccounted; String note; - if (const auto rit = retired_by_hash.find(hash); rit != retired_by_hash.end() - && backend.head(bkey).token == rit->second.token) + const auto rit = retired_by_hash.find(hash); + /// HEAD only for a hash the snapshot actually retired, so an unretired blob still costs no request. + const std::optional retired_head + = rit != retired_by_hash.end() ? op.head(bkey, Retry::standard()) : std::nullopt; + if (retired_head && rit->second.token.matches(retired_head->incarnation)) { - /// The PRESENT incarnation is the condemned one — deletion is scheduled. A token - /// mismatch means the listed entry belongs to a displaced older incarnation and says + /// The PRESENT incarnation is the condemned one — deletion is scheduled. A mismatch + /// means the listed entry belongs to a displaced older incarnation and says /// nothing about this object; fall through to the snapshot check. cls = FsckClass::PendingGc; note = rit->second.delete_pending @@ -1053,13 +1057,13 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co for (const String & bkey : reachable_blobs) { checkDeadline(deadline, "head-checking scoped blobs"); - const HeadResult h = backend.head(bkey); + const std::optional h = op.head(bkey, Retry::standard()); const auto lit = blob_labels.find(bkey); - bool exists = h.exists; + const bool exists = h.has_value(); if (!exists) { /// Use the same HEAD-absent re-resolve as the global-mode loop above. - const bool still_referenced = blobStillReferenced(store, layout, recovery_authorities, bkey, + const bool still_referenced = blobStillReferenced(op, store, layout, recovery_authorities, bkey, lit != blob_labels.end() ? lit->second : std::vector{}, deadline, record_recovery_unchecked); if (!still_referenced) @@ -1068,7 +1072,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co if (exists) { ++report.reachable; - report.physical_bytes += h.size; + report.physical_bytes += h->size; } else ++report.dangling; @@ -1077,7 +1081,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co FsckObject o; o.key = bkey; o.kind = ObjectKind::Blob; - o.size = exists ? h.size : 0; + o.size = exists ? h->size : 0; o.cls = exists ? FsckClass::Reachable : FsckClass::Dangling; if (detail && lit != blob_labels.end()) o.reachable_from = lit->second; @@ -1096,7 +1100,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co const RootNamespace ns{ns_str}; const String manifests_prefix = layout.manifestNamespacePrefix(ns); std::unordered_map manifest_bodies; - listAll(backend, manifests_prefix, manifest_bodies, on_progress, deadline, "listing manifests"); + listAll(op, manifests_prefix, manifest_bodies, on_progress, deadline, "listing manifests"); for (const auto & [mkey, sz] : manifest_bodies) { if (owned_manifest_keys.contains(mkey)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index dda1c5921f2d..c6ff7c75f007 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -285,13 +285,14 @@ String renderGcState(const GcState & s) .str(); } -/// `Token::value` is an opaque backend-native string (e.g. an S3 ETag) — NOT a 128-bit hash — so it -/// renders verbatim (escaped), not hex-converted; `type` names which backend family minted it. -String renderToken(const Token & t) +/// A recorded incarnation's value is an opaque backend-native string (e.g. an S3 ETag) — NOT a +/// 128-bit hash — so it renders verbatim (escaped), not hex-converted; `type` is the dialect word, +/// naming which backend family minted it. +String renderPersistedIncarnation(const PersistedIncarnation & inc) { return JsonObj() - .add("value", jsonEscape(t.value)) - .add("type", jsonEscape(tokenTypeToWord(t.type))) + .add("value", jsonEscape(inc.value)) + .add("type", jsonEscape(inc.dialect)) .str(); } @@ -400,7 +401,7 @@ String renderCondemnedRow(const CondemnedRow & r) { return JsonObj() .add("delete_pending", jsonBool(r.delete_pending)) - .add("token", renderToken(r.token)) + .add("token", renderPersistedIncarnation(r.token)) .add("size", jsonUInt(r.size)) .add("condemn_round", jsonUInt(r.condemn_round)) .add("marker_confirmed", jsonBool(r.marker_confirmed)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.h index 0c6bfa3e0cee..7d76f6250de5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.h @@ -17,8 +17,8 @@ namespace DB::Cas /// `cas/ns/stream/` and `cas/ns/state/` roots, `/mount` and `/fold_seal` suffixes, the /// `gc/gen/*/attempt/*/blob_target/*/*` source-edge run segments, then the pool-wide `gc/state` /// and `blobs/` prefix). u128 and hash fields render as lowercase hex strings (matching -/// `u128ToHex`), while backend-native `Token` values render as escaped strings. Neither is exposed -/// as an array of bytes or a raw struct dump. +/// `u128ToHex`), while a recorded incarnation's backend-native value renders as an escaped string +/// beside its dialect word. Neither is exposed as an array of bytes or a raw struct dump. /// /// Throws `ErrorCodes::BAD_ARGUMENTS` when `key` matches none of the recognized CA layouts. Any /// decode failure of a matched key (invalid header, corrupted bytes, future format version, ...) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp index 41365299eaf4..5b425d33c974 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp @@ -635,7 +635,7 @@ std::vector makeSourceEdgeRecords(size_t n) rec.source_id = UInt128(0); rec.marker = RunMarker::Condemned; rec.delete_pending = (i % 200 == 0); - rec.token = Token{"\"e1b2c3d4e5f6071829300a0b0c0d0e0f\"", TokenType::ETag}; + rec.token = PersistedIncarnation{"etag", "\"e1b2c3d4e5f6071829300a0b0c0d0e0f\""}; rec.size = 64 * 1024; rec.condemn_round = 7; } diff --git a/src/Disks/tests/cas_sweep_test_support.h b/src/Disks/tests/cas_sweep_test_support.h index c1b5466a1cb1..37144b65f415 100644 --- a/src/Disks/tests/cas_sweep_test_support.h +++ b/src/Disks/tests/cas_sweep_test_support.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -22,10 +22,15 @@ inline ManifestSweepResult sweepManifestCursorPageForTest( { ManifestSweepResult result = planManifestCursorPage( store, cursor, list_budget, delete_budget, /*catalog_recovery_authoritative=*/true, work_budget); + CasOperation op = store.gcRequests().admit(); for (const ManifestSweepResult::Nomination & nomination : result.nominations) { - const DeleteOutcome outcome = store.backend().deleteExact(nomination.key, nomination.token); - if (classifyDeleteOutcome(outcome) == DeleteClass::Deleted) + /// A nomination records the incarnation it was planned against, so the delete re-observes the + /// key and refuses unless what is there now is still that one: a key a fresh owner has since + /// replaced must survive. + const std::optional seen = op.head(nomination.key, Retry::standard()); + if (seen && nomination.token.matches(seen->incarnation) + && op.remove(nomination.key, seen->incarnation, Retry::standard()) == Removal::Removed) ++result.deleted; else ++result.skipped; diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index e63ce4ae5b1c..141e04c64a38 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -9,12 +9,14 @@ #include #include #include +#include #include #include #include #include #include #include +#include #include #include #include @@ -34,6 +36,7 @@ #include #include #include +#include /// For `ChunkFaultBackend`'s `DefiniteFailure` mode, which needs a real S3-classified error, and for /// the ambiguity it raises otherwise. #include @@ -43,6 +46,7 @@ #include #include #include +#include #include #include #include @@ -179,6 +183,62 @@ void expectThrowsCode(int expected_code, F && fn) } } +/// Asserts the object is present before comparing its body: an absent key would otherwise dereference +/// an empty optional and take the whole binary down instead of failing this one case. +inline void expectBytes(DB::Cas::Backend & backend, const String & key, const String & expected) +{ + const auto got = backend.get(key); + ASSERT_TRUE(got.has_value()) << "object '" << key << "' is absent"; + EXPECT_EQ(got->bytes, expected); +} + +inline void expectBytes(const DB::Cas::BackendPtr & backend, const String & key, const String & expected) +{ + expectBytes(*backend, key, expected); +} + +/// A `CasRequests` over an always-open fence, for a fixture that has a backend but no mounted pool. +/// Every operation admitted from it holds a reference to it, so it must be named and outlive them. +inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::BackendPtr backend) +{ + return DB::Cas::CasRequests(std::move(backend), DB::Cas::Fence::open()); +} + +/// The same, for a fixture holding only a reference. The aliasing `shared_ptr` owns nothing, so the +/// caller keeps the backend alive for as long as the returned object and its operations live. +inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::Backend & backend) +{ + return openRequestsForTest(DB::Cas::BackendPtr(std::shared_ptr(), &backend)); +} + +/// An open-fence operation together with the `CasRequests` it refers to, for a fixture that holds a +/// backend and needs to call a production entry point taking a `CasOperation &`. Neither copyable nor +/// movable: the operation points at the member beside it. +class OperationForTest +{ +public: + explicit OperationForTest(DB::Cas::BackendPtr backend) + : requests(std::move(backend), DB::Cas::Fence::open()), operation(requests.admit()) + { + } + + /// For a fixture holding only a reference: the aliasing `shared_ptr` owns nothing, so the caller + /// keeps the backend alive for as long as this object. + explicit OperationForTest(DB::Cas::Backend & backend) + : OperationForTest(DB::Cas::BackendPtr(std::shared_ptr(), &backend)) + { + } + + OperationForTest(const OperationForTest &) = delete; + OperationForTest & operator=(const OperationForTest &) = delete; + + DB::Cas::CasOperation & operator*() { return operation; } + +private: + DB::Cas::CasRequests requests; + DB::Cas::CasOperation operation; +}; + /// Build a `LocalObjectStorage` rooted at a fresh, unique temporary directory (one per call). /// /// Used by the unit tests that exercise the `Cas::Backend` seam against a real on-disk object storage @@ -355,7 +415,9 @@ inline uint64_t appendRefLogSeed( /// sentinel), exactly as `writeRefLogTxnRaw` below now does -- otherwise this scan can miss a REAL /// incarnation's existing log/snap objects, wrongly conclude the table has none, and prepend a second /// `namespaceBirthOp` on top of a namespace that already has one. - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(backend, layout, ns).value_or(fixture::fixtureLife(ns)); + OperationForTest operation(backend); + const NamespaceLifeId life + = CasRefCatalog::lifeIfCataloged(*operation, layout, ns).value_or(fixture::fixtureLife(ns)); const String prefix = layout.namespaceStreamPrefix(life); uint64_t greatest_seq = 0; bool any_log_or_snap = false; @@ -495,16 +557,21 @@ inline String encodeMinimalGcState(uint64_t round) } /// Inject condemned bookkeeping + gc/state directly (bypassing a real GC round) so a test can seed the -/// GC ledger's condemned state at an arbitrary round. The condemned entries are -/// seeded the way a real round leaves them — as `RunMarker::Condemned` sentinel rows inside an adopted fold seal's -/// shard run (there is no separate retired-list object). A synthetic +edge/-edge pair nets each blob to -/// in-degree 0 and a `seed_head` replays the captured token/size so the fold mints the `RunMarker::Condemned` row. -/// Also sets {round} on gc/state. Entries carry a `condemn_round` (default 0 → uses `round`); callers -/// pass fresh (non-pending) condemns. An empty `entries` set just advances {round}. +/// GC ledger's condemned state at an arbitrary round. The entries are written into the adopted seal's +/// shard run as `RunMarker::Condemned` sentinel rows at the zero source id -- the shape a real round +/// leaves, since there is no separate retired-list object. Also sets {round} on gc/state. An entry's +/// `condemn_round` defaults to `round` when left 0. An empty `entries` set just advances {round}. +/// +/// The rows are written from the caller's entries VERBATIM rather than folded out of synthetic deltas. +/// A fold mints each row's incarnation from a live HEAD of the blob, which can express neither of the +/// two shapes this fixture exists to build: an entry condemning an incarnation NO object carries (a +/// phantom, for the tests that check a condemnation aimed elsewhere spares the live object), and an +/// entry for a blob whose body is already gone. inline void injectRetire( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, uint64_t round, uint64_t shard, std::vector entries) { + OperationForTest operation(backend); DB::Cas::GcState gc_state; const DB::Cas::HeadResult head = backend.head(layout.gcStateKey()); if (head.exists) @@ -515,40 +582,59 @@ inline void injectRetire( { const uint64_t generation = 1; const uint64_t attempt = 1; - uint64_t condemn_round = round; - std::unordered_map seeded; - std::vector synth; - synth.reserve(entries.size() * 2); - for (const DB::Cas::RetiredEntry & e : entries) + + /// The writer's contract: records arrive in non-decreasing `(ref, source_id)` order, and a + /// sentinel row is the only row a blob may have at the zero source id. + std::sort(entries.begin(), entries.end(), + [](const DB::Cas::RetiredEntry & a, const DB::Cas::RetiredEntry & b) { return a.ref < b.ref; }); + + String run_bytes; + /// `UINT64_MAX` is the summary's own "no non-pending entry" value, and round 0 is a real round, + /// so a zero initializer here would claim the oldest possible condemnation instead of none. + uint64_t oldest_nonpending = UINT64_MAX; + uint64_t pending_total = 0; { - if (e.condemn_round) - condemn_round = e.condemn_round; - seeded.emplace(e.ref, DB::Cas::HeadResult{.exists = true, .size = e.size, .token = e.token, .attributes = {}}); - synth.push_back(DB::Cas::BlobDelta{.ref = e.ref, .source_id = DB::UInt128{1}, .remove = false}); - synth.push_back(DB::Cas::BlobDelta{.ref = e.ref, .source_id = DB::UInt128{1}, .remove = true}); + DB::WriteBufferFromString out(run_bytes); + DB::Cas::SourceEdgeRunWriter writer(out); + for (const DB::Cas::RetiredEntry & e : entries) + { + const uint64_t condemn_round = e.condemn_round ? e.condemn_round : round; + if (e.delete_pending) + ++pending_total; + else + oldest_nonpending = std::min(oldest_nonpending, condemn_round); + writer.append(DB::Cas::SourceEdgeRecord{ + .ref = e.ref, + .source_id = DB::UInt128{0}, + .marker = DB::Cas::RunMarker::Condemned, + .delete_pending = e.delete_pending, + .token = e.token, + .size = e.size, + .condemn_round = condemn_round, + .marker_confirmed = e.marker_confirmed}); + } + writer.finish(); + out.finalize(); } - const auto seed_head = [&seeded](const DB::Cas::BlobRef & h) -> std::optional - { - const auto it = seeded.find(h); - return it == seeded.end() ? std::nullopt : std::optional(it->second); - }; - std::vector out; - DB::Cas::foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, generation, attempt, - shard, std::move(synth), out, /*current_round*/0, condemn_round, seed_head, - /*peek_head*/{}, /*confirm_condemned_marker*/{}, - /*out_retired*/nullptr, /*suppress_destructive*/false); + + const String run_key = layout.blobTargetRunKey(generation, attempt, shard, 0); + DB::Cas::orThrow((*operation).create(run_key, run_bytes, DB::Cas::Retry::standard()), + "seed the condemned run at " + run_key); DB::Cas::CasFoldSeal seal; seal.generation = generation; - for (DB::Cas::RunRef & r : out) - seal.blob_target_runs.push_back(std::move(r)); + seal.blob_target_runs.push_back(DB::Cas::RunRef{.key = run_key, + .checksum = DB::Cas::sourceEdgeRunChecksum(run_bytes), + .shard = shard, + .key_generation = generation}); /// Totality over gc_shards so a later real round's graduation/carry reads it zero-I/O. const uint64_t gc_shards = gc_state.gc_shards ? gc_state.gc_shards : 1; for (uint64_t s = 0; s < gc_shards; ++s) seal.condemned_summary[s] = DB::Cas::CondemnedSummary{}; DB::Cas::CondemnedSummary cs; cs.condemned_total = entries.size(); - cs.oldest_nonpending_condemn_round = condemn_round; + cs.pending_total = pending_total; + cs.oldest_nonpending_condemn_round = oldest_nonpending; seal.condemned_summary[shard] = cs; backend.putIfAbsent(layout.foldSealKey(generation, attempt), DB::Cas::encodeFoldSeal(seal)); @@ -649,6 +735,7 @@ inline bool runRoundsUntilAbsent( inline std::vector currentRetiredSet( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, uint64_t shard) { + OperationForTest operation(backend); const auto st = backend.get(layout.gcStateKey()); if (!st) return {}; @@ -665,7 +752,7 @@ inline std::vector currentRetiredSet( { if (run.shard != shard) continue; - auto r = DB::Cas::openSourceEdgeRun(backend, run.key); + auto r = DB::Cas::openSourceEdgeRun(*operation, run.key); String k; String p; while (r.next(k, p)) @@ -709,25 +796,32 @@ inline bool anyCondemnedInSeal( /// incarnation_tag in its envelope header (preserving header_len + payload), putOverwrite against the /// current token, and return the NEW token. Used to drive the W-REVALIDATE adopt branch (current token /// differs from the writer's stale observation). -inline DB::Cas::Token displaceObjectToken( +inline DB::Cas::Incarnation displaceObjectToken( DB::Cas::Backend & backend, const String & key, DB::Cas::ObjectKind kind) { - const auto got = backend.get(key); + OperationForTest operation(backend); + const std::optional got = (*operation).read(key, DB::Cas::Retry::standard()); if (!got) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "displaceObjectToken: object {} absent", key); DB::Cas::EnvelopeHeader header = DB::Cas::decodeEnvelopeHeader(got->bytes, got->bytes.size(), kind); - /// A fresh, distinct incarnation_tag forces a distinct body so the displaced token differs. + /// A fresh, distinct incarnation_tag forces a distinct body so the displaced incarnation differs. header.incarnation_tag = header.incarnation_tag + DB::UInt128(1); /// Re-encode at the SAME header length the object was decoded with (the v3 pad target). const String new_head = DB::Cas::encodeEnvelopeHeader(header, header.header_len); const String body = new_head + got->bytes.substr(header.header_len); - return backend.putOverwrite(key, body, got->token).token; + const std::optional displaced = DB::Cas::orThrow( + (*operation).replace(key, body, got->incarnation, DB::Cas::Retry::standard()), + "displace the object at " + key); + if (!displaced) + throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, + "displaceObjectToken: the replace of {} reported no incarnation", key); + return *displaced; } -inline DB::Cas::Token displaceBlobToken( +inline DB::Cas::Incarnation displaceBlobToken( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::BlobRef & id) { return displaceObjectToken(backend, layout.blobKey(id), DB::Cas::ObjectKind::Blob); @@ -768,13 +862,14 @@ inline void seedPoolMetaForRestart( DB::Cas::Backend & backend, const String & pool_prefix = "p", uint64_t gc_shards = 1) { const DB::Cas::Layout layout(pool_prefix); + OperationForTest operation(backend); DB::Cas::PoolMeta::createOrValidate( - backend, layout, /*blob_header_len=*/256, gc_shards, + *operation, layout, /*blob_header_len=*/256, gc_shards, DB::Cas::BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/true); if (!backend.get(layout.refCatalogKey())) - DB::Cas::CasRefCatalog::initializeEmptyForNewPool(backend, layout); + DB::Cas::CasRefCatalog::initializeEmptyForNewPool(*operation, layout); else - (void)DB::Cas::CasRefCatalog::read(backend, layout); + (void)DB::Cas::CasRefCatalog::read(*operation, layout); } /// Write a blob object (envelope + payload) addressed by `hash`, so a HEAD returns a token. The bytes @@ -824,19 +919,22 @@ inline void condemnMeta(DB::Cas::Backend & backend, const DB::Cas::Layout & layo const DB::UInt128 & hash, uint64_t condemn_round) { const DB::Cas::BlobRef ref = legacyMetaTestRef(hash); - const auto lm = DB::Cas::loadMeta(backend, layout, ref); + OperationForTest operation(backend); + const auto lm = DB::Cas::loadMeta(*operation, layout, ref); ASSERT_TRUE(lm.has_value()); DB::Cas::BlobMeta c = lm->meta; c.state = DB::Cas::MetaState::Condemned; c.condemn_round = condemn_round; - backend.putOverwrite(layout.blobMetaKey(ref), DB::Cas::encodeBlobMeta(c), lm->etag); + ASSERT_TRUE(std::holds_alternative( + DB::Cas::casMeta(*operation, layout, ref, lm->incarnation, c))); } /// Load the meta descriptor for `hash` via the shared ops layer (nullopt = absent). inline std::optional loadMetaForTest(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::UInt128 & hash) { - return DB::Cas::loadMeta(backend, layout, legacyMetaTestRef(hash)); + OperationForTest operation(backend); + return DB::Cas::loadMeta(*operation, layout, legacyMetaTestRef(hash)); } /// The latest GC generation (snap_generation pointer in gc/state), or 0 when absent. @@ -889,10 +987,11 @@ inline std::vector runsForShard( inline int64_t inDegreeInRuns( DB::Cas::Backend & backend, const std::vector & runs, const DB::Cas::BlobRef & ref) { + OperationForTest operation(backend); int64_t degree = 0; for (const DB::Cas::RunRef & run : runs) { - auto r = DB::Cas::openSourceEdgeRun(backend, run.key); + auto r = DB::Cas::openSourceEdgeRun(*operation, run.key); String k; String p; while (r.next(k, p)) @@ -950,8 +1049,9 @@ namespace fixture inline UInt128 catalogLifeIdForTest( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns) { + OperationForTest operation(backend); const std::optional life = - DB::Cas::CasRefCatalog::lifeIfCataloged(backend, layout, ns); + DB::Cas::CasRefCatalog::lifeIfCataloged(*operation, layout, ns); chassert(life.has_value()); return life->incarnation; } @@ -981,8 +1081,9 @@ inline void seedFoldCursorForTest( DB::Cas::RefTxnId cursor, std::optional hold = std::nullopt, uint64_t generation = 1, uint64_t attempt = 1) { + OperationForTest operation(backend); DB::Cas::NamespaceLifeId life = fixture::fixtureLife(ns); - const DB::Cas::CasRefCatalog::Snapshot catalog_cut = DB::Cas::CasRefCatalog::read(backend, layout); + const DB::Cas::CasRefCatalog::Snapshot catalog_cut = DB::Cas::CasRefCatalog::read(*operation, layout); const auto catalog_it = std::find_if( catalog_cut.catalog.entries.begin(), catalog_cut.catalog.entries.end(), [&](const DB::Cas::CatalogEntry & entry) { return entry.ns.string() == ns.string(); }); @@ -992,7 +1093,7 @@ inline void seedFoldCursorForTest( entry.ns = ns; entry.state = DB::Cas::NsState::Live; entry.incarnation = fixture::fixtureLife(ns).incarnation; - DB::Cas::CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); + DB::Cas::CasRefCatalog::casAdmitEntry(*operation, layout, 1, entry); life = DB::Cas::NamespaceLifeId::fromCatalogEntry(ns, entry.incarnation); } else @@ -1046,8 +1147,9 @@ inline uint64_t foldCursorOf( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns, uint64_t shard) { chassert(shard == 0); + OperationForTest operation(backend); const std::optional life = - DB::Cas::CasRefCatalog::lifeIfCataloged(backend, layout, ns); + DB::Cas::CasRefCatalog::lifeIfCataloged(*operation, layout, ns); if (!life) return 0; const uint64_t gen = currentGenerationOf(backend, layout); @@ -1105,7 +1207,9 @@ inline void writeRefSnapshotRaw( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RefTableSnapshot & snapshot) { const DB::Cas::RootNamespace ns{snapshot.ns}; - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(backend, layout, ns).value_or(fixture::fixtureLife(ns)); + OperationForTest operation(backend); + const NamespaceLifeId life + = CasRefCatalog::lifeIfCataloged(*operation, layout, ns).value_or(fixture::fixtureLife(ns)); const String key = layout.refSnapshotKey(life, snapshot.snapshot_id); backend.putIfAbsent(key, DB::Cas::sealObject(DB::Cas::FormatId::RefSnapshot, DB::Cas::encodeRefTableSnapshot(snapshot))); } @@ -1132,7 +1236,8 @@ inline void writeRefSnapshotRaw( /// readable `life_epoch`. Ordinary fixtures use `casAdmitRecoverableEntry` below instead. inline void casAdmitEntry(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns) { - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + OperationForTest operation(backend); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(*operation, layout); for (const CatalogEntry & entry : snap.catalog.entries) if (entry.ns.string() == ns.string()) return; /// already admitted -- by an earlier raw write to the same namespace, or by the @@ -1141,7 +1246,7 @@ inline void casAdmitEntry(DB::Cas::Backend & backend, const DB::Cas::Layout & la entry.ns = ns; entry.state = NsState::Live; entry.incarnation = fixture::fixtureLife(ns).incarnation; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(*operation, layout, 1, entry); } namespace fixture @@ -1160,7 +1265,8 @@ inline void writeRecoverableCkptForRawFixture( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns, const DB::Cas::RefCkpt & ckpt) { - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(backend, layout); + OperationForTest operation(backend); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*operation, layout); const auto it = std::find_if( catalog_cut.catalog.entries.begin(), catalog_cut.catalog.entries.end(), [&] (const CatalogEntry & entry) { return entry.ns == ns; }); @@ -1169,8 +1275,9 @@ inline void writeRecoverableCkptForRawFixture( "raw recovery fixture for namespace '{}' has no catalog entry", ns.string()); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(it->ns, it->incarnation); - const PutResult put = backend.putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(ckpt)); - if (put.outcome != PutOutcome::Done) + const WriteResult put = (*operation).create(layout.refCkptKey(life), encodeRefCkpt(ckpt), + DB::Cas::Retry::standard()); + if (!std::holds_alternative(put)) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' could not publish its checkpoint", ns.string()); } @@ -1181,7 +1288,8 @@ inline void advanceRecoverableCkptForRawFixture( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns, const DB::Cas::RefTxnId & through) { - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(backend, layout); + OperationForTest operation(backend); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*operation, layout); const auto it = std::find_if( catalog_cut.catalog.entries.begin(), catalog_cut.catalog.entries.end(), [&] (const CatalogEntry & entry) { return entry.ns == ns; }); @@ -1190,7 +1298,7 @@ inline void advanceRecoverableCkptForRawFixture( "raw recovery fixture for namespace '{}' has no catalog entry", ns.string()); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(it->ns, it->incarnation); - const std::optional sample = readCkpt(backend, layout, life); + const std::optional sample = readCkpt(*operation, layout, life); if (!sample) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' has no checkpoint to advance", ns.string()); @@ -1202,7 +1310,9 @@ inline void advanceRecoverableCkptForRawFixture( RefCkpt advanced = sample->ckpt; advanced.committed_through = through; - if (backend.casPut(layout.refCkptKey(life), encodeRefCkpt(advanced), sample->token).outcome != CasOutcome::Committed) + if (!std::holds_alternative( + (*operation).replace(layout.refCkptKey(life), encodeRefCkpt(advanced), sample->incarnation, + DB::Cas::Retry::standard()))) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' could not advance its checkpoint", ns.string()); } @@ -1215,7 +1325,8 @@ inline void replaceRecoverableCkptForRawFixture( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns, const DB::Cas::RefCkpt & next) { - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(backend, layout); + OperationForTest operation(backend); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*operation, layout); const auto it = std::find_if( catalog_cut.catalog.entries.begin(), catalog_cut.catalog.entries.end(), [&] (const CatalogEntry & entry) { return entry.ns == ns; }); @@ -1224,7 +1335,7 @@ inline void replaceRecoverableCkptForRawFixture( "raw recovery fixture for namespace '{}' has no catalog entry", ns.string()); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(it->ns, it->incarnation); - const std::optional existing = readCkpt(backend, layout, life); + const std::optional existing = readCkpt(*operation, layout, life); if (!existing) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' has no checkpoint to replace", ns.string()); @@ -1239,7 +1350,9 @@ inline void replaceRecoverableCkptForRawFixture( throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' cannot regress its checkpoint frontier", ns.string()); - if (backend.casPut(layout.refCkptKey(life), encodeRefCkpt(next), existing->token).outcome != CasOutcome::Committed) + if (!std::holds_alternative( + (*operation).replace(layout.refCkptKey(life), encodeRefCkpt(next), existing->incarnation, + DB::Cas::Retry::standard()))) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' could not replace its checkpoint", ns.string()); } @@ -1252,20 +1365,24 @@ inline void publishRecoverableCkptForSemanticWrapper( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::RootNamespace & ns, const DB::Cas::RefTxnId & txn_id) { - const std::optional life = CasRefCatalog::lifeIfCataloged(backend, layout, ns); + OperationForTest operation(backend); + const std::optional life = CasRefCatalog::lifeIfCataloged(*operation, layout, ns); if (!life) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "semantic ref fixture for namespace '{}' was not admitted", ns.string()); - if (!readCkpt(backend, layout, *life)) - { - const PutResult put = backend.putIfAbsent(layout.refCkptKey(*life), encodeRefCkpt(RefCkpt{ - .life_epoch = 1, - .committed_through = txn_id, - .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt, - })); - if (put.outcome == PutOutcome::Done) + if (!readCkpt(*operation, layout, *life)) + { + const WriteResult put = (*operation).create( + layout.refCkptKey(*life), + encodeRefCkpt(RefCkpt{ + .life_epoch = 1, + .committed_through = txn_id, + .checkpoint_snapshot_id = std::nullopt, + .last_epoch_seal = std::nullopt, + }), + DB::Cas::Retry::standard()); + if (std::holds_alternative(put)) return; } @@ -1284,12 +1401,13 @@ inline void casAdmitRecoverableEntry( { casAdmitEntry(backend, layout, ns); - const std::optional life = CasRefCatalog::lifeIfCataloged(backend, layout, ns); + OperationForTest operation(backend); + const std::optional life = CasRefCatalog::lifeIfCataloged(*operation, layout, ns); if (!life) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "recoverable raw fixture for namespace '{}' was not admitted", ns.string()); - if (backend.head(layout.refCkptKey(*life)).exists) + if ((*operation).head(layout.refCkptKey(*life), DB::Cas::Retry::standard())) return; writeRecoverableCkptForRawFixture(backend, layout, ns, RefCkpt{ @@ -1314,15 +1432,16 @@ inline DB::Cas::RecoveredRefTable recoverRefTableDetailedAtCatalogCutForTest( if (it != catalog_cut.catalog.entries.end()) catalog_entry = *it; + OperationForTest operation(backend); std::optional ckpt; if (catalog_entry) { const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(catalog_entry->ns, catalog_entry->incarnation); - if (const std::optional sample = readCkpt(backend, layout, life)) + if (const std::optional sample = readCkpt(*operation, layout, life)) ckpt = sample->ckpt; } - return recoverRefTableDetailedFromAuthority(backend, layout, catalog_entry, ckpt); + return recoverRefTableDetailedFromAuthority(*operation, layout, catalog_entry, ckpt); } /// Writes `txn` at `_log/` (create-if-absent). Admits `txn.ns` into the catalog first @@ -1342,7 +1461,9 @@ inline void writeRefLogTxnRaw( { const DB::Cas::RootNamespace ns{txn.ns}; casAdmitEntry(backend, layout, ns); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(backend, layout, ns).value_or(fixture::fixtureLife(ns)); + OperationForTest operation(backend); + const NamespaceLifeId life + = CasRefCatalog::lifeIfCataloged(*operation, layout, ns).value_or(fixture::fixtureLife(ns)); const String key = layout.refLogKey(life, txn.txn_id); backend.putIfAbsent(key, DB::Cas::sealObject(DB::Cas::FormatId::RefLog, DB::Cas::encodeRefLogTxn(txn))); } @@ -1465,218 +1586,178 @@ inline std::vector publishCommittedOps(const String & ref_name, return {add, promote}; } -/// Counts head/get/putIfAbsent per key for op-count assertions (Pillar B / A1 tests). +/// Serves an inner stream in windows of at most `chunk` bytes, and records the largest window it ever +/// handed out. `InMemoryBackend` materializes the whole object behind its stream, so without this a +/// consumer receives every byte as one contiguous window -- it can hold the object entire and still +/// look like a streaming reader, and nothing at the seam can tell the two apart. +/// +/// What arming this proves is what the consumer then DOES: a reader that assumed one contiguous window +/// fails against a chunked source, so the test's own success is the evidence. The recorded window is +/// the bound it succeeded under, not a measurement of the reader's resident memory -- a consumer that +/// copies every window into a buffer of its own is invisible here, as it is to any `ReadBuffer`. +class ChunkedStreamForTest : public DB::ReadBuffer +{ +public: + ChunkedStreamForTest(std::unique_ptr inner_, size_t chunk, + std::shared_ptr> largest_) + : DB::ReadBuffer(nullptr, 0), inner(std::move(inner_)), storage(chunk), largest(std::move(largest_)) + { + } + +private: + bool nextImpl() override + { + const size_t got = inner->read(storage.data(), storage.size()); + if (got == 0) + return false; + BufferBase::set(storage.data(), got, 0); + uint64_t seen = largest->load(); + while (seen < got && !largest->compare_exchange_weak(seen, got)) + { + } + return true; + } + + std::unique_ptr inner; + std::vector storage; + std::shared_ptr> largest; +}; + +/// Counts every request per key, for the op-count assertions (Pillar B / A1 tests). class CountingBackend : public DB::Cas::InMemoryBackend { public: - /// Unhide the base overloads the legacy overrides below would otherwise shadow: the convenience - /// forms that omit Range/ObjectMeta/expected-token, and the transport primitives that share the - /// `head` and `list` names. - using DB::Cas::Backend::get; + /// Unhide the legacy names the primitive overrides below would otherwise shadow, and the + /// omitted-`Range` convenience. using DB::Cas::Backend::getStream; using DB::Cas::Backend::head; using DB::Cas::Backend::list; - using DB::Cas::Backend::putIfAbsent; - using DB::Cas::Backend::putOverwrite; - using DB::Cas::Backend::casPut; - /// ---- The transport primitives: every PHYSICAL request, whichever surface it entered through ---- + /// ---- The counters live on the transport primitives, so a request is counted once ---- /// - /// Counted apart from the legacy counters below, which split by the surface the CALLER used. A - /// `CasRequests` call speaks only these, so an engine test asserting `putTotal` would assert on a - /// surface the engine never touches. Each counter ticks BEFORE the request is served, so an - /// injected failure still counts as a request issued. + /// Whichever surface a caller used, its request passes through one of the primitives below: a + /// legacy verb reaches them through its forwarder, and a `CasOperation` speaks them directly. + /// Counting here is therefore counting requests rather than callers. /// - /// A legacy call is counted here TOO, because it reaches the store through its primitive -- with - /// the two exceptions `InMemoryBackend` documents, `putIfAbsent` and `casPut`, which route around - /// the primitive to keep their write knobs' verb identity and so land only on the legacy counters. + /// Each counter ticks BEFORE the request is served, so an injected failure still counts as a + /// request issued. A blob publication is not counted: it reaches the store through `publish`, + /// which no counter below observes. std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - { - std::lock_guard lock(count_mutex); - ++read_requests; - ++read_request_counts[key]; - } + tick(get_counts, get_total, key); return InMemoryBackend::read(key, access); } std::optional head(const String & key, DB::Cas::TransportAccess & access) override { - { - std::lock_guard lock(count_mutex); - ++head_requests; - ++head_request_counts[key]; - } + tick(head_counts, head_total, key); return InMemoryBackend::head(key, access); } DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, DB::Cas::TransportAccess & access) override { - { - std::lock_guard lock(count_mutex); - ++list_requests; - ++list_request_counts[prefix]; - } + tick(list_counts, list_total, prefix); return InMemoryBackend::list(prefix, cursor, limit, access); } + /// Create-shaped and replace-shaped writes are counted apart as well as together: whether a write + /// carried a precondition is the only thing about it the transport can still see, and the + /// namespace-file request-profile goldens read the create path off exactly that. std::expected write(const String & key, const String & bytes, const std::optional & expected_value, DB::Cas::TransportAccess & access) override { { std::lock_guard lock(count_mutex); - ++write_requests; + ++write_counts[key]; + ++write_total; + if (expected_value) + { + ++put_overwrite_counts[key]; + ++put_overwrite_total; + } + else + { + ++put_counts[key]; + ++put_total; + } } return InMemoryBackend::write(key, bytes, expected_value, access); } + /// Every ATTEMPTED delete is counted, whatever the backend answers. The destructive gate's tests + /// assert that a suppressed round issues NONE, and an attempt that came back `Gone` is still an + /// attempt -- counting only successful ones would let a gate that leaks deletes over already-absent + /// keys read as green. DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, DB::Cas::TransportAccess & access) override { - { - std::lock_guard lock(count_mutex); - ++remove_requests; - } + tick(delete_counts, delete_total, key); return InMemoryBackend::remove(key, expected_value, access); } - /// Per-key counterparts of the three READ primitives. The aggregate counters above cannot say - /// WHICH key a request went to, and the legacy per-key counters below never see a caller that - /// speaks the primitives -- `probeSentinelRaw` is one, so a probe is invisible to `headCount`. - uint64_t readRequestCount(const String & key) const { return lookup(read_request_counts, key); } - uint64_t headRequestCount(const String & key) const { return lookup(head_request_counts, key); } - uint64_t listRequestCount(const String & prefix) const { return lookup(list_request_counts, prefix); } - - uint64_t readRequests() const { std::lock_guard lock(count_mutex); return read_requests; } - uint64_t headRequests() const { std::lock_guard lock(count_mutex); return head_requests; } - uint64_t listRequests() const { std::lock_guard lock(count_mutex); return list_requests; } - uint64_t writeRequests() const { std::lock_guard lock(count_mutex); return write_requests; } - uint64_t removeRequests() const { std::lock_guard lock(count_mutex); return remove_requests; } - - DB::Cas::HeadResult head(const String & key) override - { - { - std::lock_guard lock(count_mutex); - ++head_counts[key]; - ++head_total; - } - return InMemoryBackend::head(key); - } - - std::optional get(const String & key, DB::Cas::Range range) override - { - { - std::lock_guard lock(count_mutex); - ++get_counts[key]; - ++get_total; - /// Record the request-size shape per key so streaming-memory gates (Task 3/4) can assert - /// the resident-memory bound at the seam: a whole-object read (range.whole()) is a - /// violation for a run object; a ranged read tracks its MAX window length per key. - if (range.whole()) - ++whole_get_counts[key]; - else - { - const uint64_t len = range.length.has_value() ? *range.length : 0; - uint64_t & mx = max_ranged_get_len[key]; - mx = std::max(mx, len); - } - } - return InMemoryBackend::get(key, range); - } - + /// `InMemoryBackend::stream` opens its reader through this, so both the primitive and the legacy + /// stream verb are counted here, once each. std::optional getStream(const String & key, DB::Cas::Range range) override { - { - std::lock_guard lock(count_mutex); - ++get_stream_counts[key]; - ++get_stream_total; - } - return InMemoryBackend::getStream(key, range); - } - - DB::Cas::ListPage list(const String & prefix, const String & cursor, size_t limit) override - { - { - std::lock_guard lock(count_mutex); - ++list_counts[prefix]; - ++list_total; - } - return InMemoryBackend::list(prefix, cursor, limit); - } - - DB::Cas::PutResult putIfAbsent(const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override - { - { - std::lock_guard lock(count_mutex); - ++put_counts[key]; - ++put_total; - } - return InMemoryBackend::putIfAbsent(key, bytes, meta); + tick(get_stream_counts, get_stream_total, key); + std::optional opened = InMemoryBackend::getStream(key, range); + const size_t chunk = stream_chunk.load(); + if (!opened || !opened->stream || chunk == 0) + return opened; + opened->stream = std::make_unique( + std::move(opened->stream), chunk, largestChunkSlot(key)); + return opened; } + /// Serve every stream opened from now on in windows of at most `bytes`, as a network-backed store + /// does. Zero (the default) hands the consumer the whole object at once, which is what this + /// backend's own materialization makes of any stream. A mode rather than a count: `resetCounts` + /// leaves it alone. + void setStreamChunkForTest(size_t bytes) { stream_chunk.store(bytes); } - /// Counted separately from `putIfAbsent` and `casPut`, for the same reason those two are separate: a - /// replacement conditioned on an expected token is its own op with its own cost. The namespace-file - /// request-profile goldens tell the create path from the replace path on exactly this counter. - DB::Cas::PutResult putOverwrite(const String & key, const String & bytes, const DB::Cas::Token & expected, - const DB::Cas::ObjectMeta & meta) override - { - { - std::lock_guard lock(count_mutex); - ++put_overwrite_counts[key]; - ++put_overwrite_total; - } - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); - } - - /// Counted separately from `putIfAbsent`: a token-CAS is a DIFFERENT op with a different cost, and - /// the `_ckpt` no-op contract ("identical merged body issues no write") is asserted on exactly this - /// counter -- a create-if-absent count would not see the replace path at all. - DB::Cas::CasResult casPut(const String & key, const String & bytes, - const std::optional & expected, const DB::Cas::ObjectMeta & meta) override + /// The largest contiguous window any consumer of `key`'s stream was handed. Zero when the key was + /// never streamed. + uint64_t largestStreamChunk(const String & key) const { - { - std::lock_guard lock(count_mutex); - ++cas_put_counts[key]; - ++cas_put_total; - } - return InMemoryBackend::casPut(key, bytes, expected, meta); - } - /// Every ATTEMPTED delete is counted, whatever the backend answers. The destructive gate's tests - /// assert that a suppressed round issues NONE, and an attempt that came back `NotFound` is still an - /// attempt -- counting only successful ones would let a gate that leaks deletes over already-absent - /// keys read as green. - DB::Cas::DeleteOutcome deleteExact(const String & key, const DB::Cas::Token & token) override - { - { - std::lock_guard lock(count_mutex); - ++delete_counts[key]; - ++delete_total; - } - return InMemoryBackend::deleteExact(key, token); - + std::lock_guard lock(count_mutex); + const auto it = largest_stream_chunk.find(key); + return it == largest_stream_chunk.end() ? 0 : it->second->load(); } - uint64_t headCount(const String & key) const { return lookup(head_counts, key); } - uint64_t casPutCount(const String & key) const { return lookup(cas_put_counts, key); } - uint64_t putOverwriteCount(const String & key) const { return lookup(put_overwrite_counts, key); } uint64_t getCount(const String & key) const { return lookup(get_counts, key); } + uint64_t headCount(const String & key) const { return lookup(head_counts, key); } + uint64_t listCount(const String & prefix) const { return lookup(list_counts, prefix); } + uint64_t writeCount(const String & key) const { return lookup(write_counts, key); } uint64_t putCount(const String & key) const { return lookup(put_counts, key); } + uint64_t putOverwriteCount(const String & key) const { return lookup(put_overwrite_counts, key); } uint64_t deleteCount(const String & key) const { return lookup(delete_counts, key); } + uint64_t getStreamCount(const String & key) const { return lookup(get_stream_counts, key); } + + uint64_t getTotal() const { std::lock_guard lock(count_mutex); return get_total; } + uint64_t headTotal() const { std::lock_guard lock(count_mutex); return head_total; } + uint64_t listTotal() const { std::lock_guard lock(count_mutex); return list_total; } + uint64_t writeTotal() const { std::lock_guard lock(count_mutex); return write_total; } + uint64_t putTotal() const { std::lock_guard lock(count_mutex); return put_total; } + uint64_t putOverwriteTotal() const { std::lock_guard lock(count_mutex); return put_overwrite_total; } uint64_t deleteTotal() const { std::lock_guard lock(count_mutex); return delete_total; } + uint64_t getStreamTotal() const { std::lock_guard lock(count_mutex); return get_stream_total; } + /// Attempted deletes against any key whose path CONTAINS `substr` — the per-site assertion the /// destructive-gate tests make ("the generation prune deleted nothing", "the sweep deleted nothing"). uint64_t deleteCountForKeysContaining(const String & substr) const { - std::lock_guard lock(count_mutex); - uint64_t total = 0; - for (const auto & [key, n] : delete_counts) - if (key.find(substr) != String::npos) - total += n; - return total; + return sumForKeysContaining({&delete_counts}, substr); + } + + /// The total number of read + stream + create-shaped write requests against any key whose path + /// CONTAINS `substr` (T0 idle-round gate: zero run I/O touches every `.../blob_target/...` key). + uint64_t ioCountForKeysContaining(const String & substr) const + { + return sumForKeysContaining({&get_counts, &get_stream_counts, &put_counts}, substr); } + /// Every key this backend was ever asked to delete, in sorted order — so a failing zero-delete /// assertion names the sites that leaked instead of just reporting a count. std::vector deletedKeys() const @@ -1688,13 +1769,7 @@ class CountingBackend : public DB::Cas::InMemoryBackend keys.push_back(key); return keys; } - uint64_t getStreamCount(const String & key) const { return lookup(get_stream_counts, key); } - uint64_t listCount(const String & prefix) const { return lookup(list_counts, prefix); } - /// The max ranged-get window length observed for `key` (0 if only whole-object gets, or none). - uint64_t maxRangedGetLen(const String & key) const { return lookup(max_ranged_get_len, key); } - /// How many whole-object gets (range.whole()) hit `key` — nonzero flags a resident-memory - /// violation for a run/seal object that a streaming caller must never read whole. - uint64_t wholeGetCount(const String & key) const { return lookup(whole_get_counts, key); } + /// Every key any counted operation was issued against, plus every LIST prefix, sorted and /// de-duplicated. A request-profile gate asserts the SET, not only the totals, so a new request the /// profile does not allow names its own key in the failure instead of moving an anonymous counter. @@ -1703,8 +1778,7 @@ class CountingBackend : public DB::Cas::InMemoryBackend std::lock_guard lock(count_mutex); std::vector keys; for (const std::map * m : - {&head_counts, &get_counts, &put_counts, &put_overwrite_counts, &cas_put_counts, - &get_stream_counts, &list_counts, &delete_counts}) + {&get_counts, &head_counts, &list_counts, &write_counts, &delete_counts, &get_stream_counts}) for (const auto & [key, n] : *m) keys.push_back(key); std::sort(keys.begin(), keys.end()); @@ -1712,51 +1786,39 @@ class CountingBackend : public DB::Cas::InMemoryBackend return keys; } - uint64_t headTotal() const { std::lock_guard lock(count_mutex); return head_total; } - uint64_t getTotal() const { std::lock_guard lock(count_mutex); return get_total; } - uint64_t putTotal() const { std::lock_guard lock(count_mutex); return put_total; } - uint64_t putOverwriteTotal() const { std::lock_guard lock(count_mutex); return put_overwrite_total; } - uint64_t casPutTotal() const { std::lock_guard lock(count_mutex); return cas_put_total; } - uint64_t getStreamTotal() const { std::lock_guard lock(count_mutex); return get_stream_total; } - uint64_t listTotal() const { std::lock_guard lock(count_mutex); return list_total; } - - /// The total number of get + getStream + putIfAbsent operations against any key whose path - /// CONTAINS `substr` (T0 idle-round gate: zero run I/O touches every `.../blob_target/...` key). - uint64_t ioCountForKeysContaining(const String & substr) const - { - std::lock_guard lock(count_mutex); - uint64_t total = 0; - for (const auto & [key, n] : get_counts) - if (key.find(substr) != String::npos) total += n; - for (const auto & [key, n] : get_stream_counts) - if (key.find(substr) != String::npos) total += n; - for (const auto & [key, n] : put_counts) - if (key.find(substr) != String::npos) total += n; - return total; - } - void resetCounts() { std::lock_guard lock(count_mutex); - head_counts.clear(); get_counts.clear(); + head_counts.clear(); + list_counts.clear(); + write_counts.clear(); put_counts.clear(); put_overwrite_counts.clear(); - cas_put_counts.clear(); - get_stream_counts.clear(); - list_counts.clear(); delete_counts.clear(); - max_ranged_get_len.clear(); - whole_get_counts.clear(); - head_total = get_total = put_total = cas_put_total = get_stream_total = list_total = delete_total = 0; - put_overwrite_total = 0; - read_requests = head_requests = list_requests = write_requests = remove_requests = 0; - read_request_counts.clear(); - head_request_counts.clear(); - list_request_counts.clear(); + get_stream_counts.clear(); + largest_stream_chunk.clear(); + get_total = head_total = list_total = write_total = put_total = put_overwrite_total + = delete_total = get_stream_total = 0; } private: + std::shared_ptr> largestChunkSlot(const String & key) + { + std::lock_guard lock(count_mutex); + auto & slot = largest_stream_chunk[key]; + if (!slot) + slot = std::make_shared>(0); + return slot; + } + + void tick(std::map & per_key, uint64_t & total, const String & key) + { + std::lock_guard lock(count_mutex); + ++per_key[key]; + ++total; + } + uint64_t lookup(const std::map & m, const String & key) const { std::lock_guard lock(count_mutex); @@ -1764,140 +1826,168 @@ class CountingBackend : public DB::Cas::InMemoryBackend return it == m.end() ? 0 : it->second; } + uint64_t sumForKeysContaining(std::initializer_list *> maps, + const String & substr) const + { + std::lock_guard lock(count_mutex); + uint64_t total = 0; + for (const std::map * m : maps) + for (const auto & [key, n] : *m) + if (key.find(substr) != String::npos) + total += n; + return total; + } + mutable std::mutex count_mutex; - std::map head_counts; + /// Held by `shared_ptr` so a stream outliving the map entry's rehash still records into its own slot. + std::map>> largest_stream_chunk; + std::atomic stream_chunk{0}; std::map get_counts; + std::map head_counts; + std::map list_counts; + std::map write_counts; std::map put_counts; std::map put_overwrite_counts; - std::map cas_put_counts; - std::map get_stream_counts; - std::map list_counts; std::map delete_counts; - std::map max_ranged_get_len; - std::map whole_get_counts; - uint64_t head_total = 0; + std::map get_stream_counts; uint64_t get_total = 0; + uint64_t head_total = 0; + uint64_t list_total = 0; + uint64_t write_total = 0; uint64_t put_total = 0; uint64_t put_overwrite_total = 0; - uint64_t cas_put_total = 0; - uint64_t get_stream_total = 0; - uint64_t list_total = 0; uint64_t delete_total = 0; - std::map read_request_counts; - std::map head_request_counts; - std::map list_request_counts; - uint64_t read_requests = 0; - uint64_t head_requests = 0; - uint64_t list_requests = 0; - uint64_t write_requests = 0; - uint64_t remove_requests = 0; + uint64_t get_stream_total = 0; }; -/// Records the ORDER of body-PUT / `_ckpt`-CAS operations (so a test can compare indices) and lets a -/// test inject a persistent `Conflict` on one chosen `_ckpt` key -- the same technique -/// `gtest_cas_ref_writer.cpp`'s `RefWriterTestBackend::ckpt_conflict_key`/`ckpt_conflict_count` uses to -/// drive the ledger into `NeedsRecovery`, reproduced here so this suite has no dependency on that file's -/// internal (non-exported) test type. Delegates every operation to `CountingBackend` unchanged, so the -/// per-key counters (`putCount`/`casPutCount`) remain available as the positive control. +/// Records the ORDER of writes (so a test can compare indices) and lets a test refuse or fail chosen +/// writes by key. Delegates every request to `CountingBackend` unchanged, so the per-key counters +/// remain available as the positive control. +/// +/// The journal records the KEY, not a verb: a write reaches the transport as bytes plus an optional +/// precondition, and the ordering tests it serves already distinguish their two subjects (the snapshot +/// body and the checkpoint) by key. class OrderedFaultBackend : public CountingBackend { public: - using CountingBackend::casPut; - using CountingBackend::get; - using CountingBackend::putIfAbsent; - - enum class Op : uint8_t { Put, Cas }; - struct Entry - { - Op op; - String key; - }; - - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - record(Op::Put, key); - if (fail_put_count > 0 && !fail_put_substr.empty() && key.find(fail_put_substr) != String::npos) + switch (claimFault(key)) { - --fail_put_count; - throw Poco::TimeoutException("OrderedFaultBackend: simulated PUT response lost, nothing landed"); + case Fault::Conflict: + /// A refusal (not a thrown/ambiguous response): the caller's own re-read-and-merge loop + /// treats this exactly like a concurrent writer that landed first. + return std::unexpected(RawConflict{}); + case Fault::ResponseLost: + throw Poco::TimeoutException("OrderedFaultBackend: simulated write response lost, nothing landed"); + case Fault::None: + break; } - return CountingBackend::putIfAbsent(key, bytes, meta); + return CountingBackend::write(key, bytes, expected_value, access); } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + /// Arms a refusal at `key` for the next `count` writes. A COUNT cannot wedge one logical write: + /// the engine reissues an unresolved or refused write until its own retry window closes, so a + /// count the reissues outlive lets the call commit in the end. Use it to bound how much contention + /// a call meets, and `armLatchedWriteConflict` when the call must not commit at all. + void armWriteConflict(const String & key, size_t count) { - record(Op::Cas, key); - if (key == fail_cas_key && fail_cas_count > 0) - { - --fail_cas_count; - /// A `Conflict` (not a thrown/ambiguous response): the caller's own re-read-and-merge loop - /// (`publishCkpt`) treats this exactly like a concurrent writer that landed first, and - /// exhausts `MAX_CKPT_CAS_ATTEMPTS` (100) without ever committing -- deterministically, with - /// no wall-clock wait, since the loop is attempt-bounded rather than only deadline-bounded. - return {CasOutcome::Conflict, {}}; - } - return CountingBackend::casPut(key, bytes, expected, meta); + std::lock_guard lock(mutex); + conflict_key = key; + conflict_count = count; } - /// Arms a persistent CAS conflict at `key` for the next `count` attempts. - void armCasConflict(const String & key, size_t count) + /// Refuses EVERY write of `key` until disarmed with an empty key, so a call meets the refusal on + /// every one of its reissues and reaches its retry deadline without committing. + void armLatchedWriteConflict(const String & key) { - fail_cas_key = key; - fail_cas_count = count; + std::lock_guard lock(mutex); + latched_conflict_key = key; + } + + /// Arms a never-committed write failure for the next `count` writes whose key contains `substr`: + /// the object is never actually written (unlike a real ambiguous response, which may or may not + /// have landed), so a resolve read always finds the key absent and classifies the attempt a + /// definite, non-committed failure. The same count caveat as `armWriteConflict` applies. + void armWriteFailure(const String & substr, int count) + { + std::lock_guard lock(mutex); + failure_substr = substr; + failure_count = count; } - /// Arms a persistent, never-committed PUT failure for the next `count` `putIfAbsent` calls whose key - /// contains `substr`: the object is never actually written (unlike a real ambiguous response, which - /// may or may not have landed), so the resolve-by-exact-GET a controlled `CasRequestBudget` with - /// `max_attempts = 1` performs always finds the key absent and classifies the attempt a definite, - /// non-`Committed` failure -- deterministically, with no internal retry and no wall-clock wait. - void armPutFailure(const String & substr, int count) + /// Loses the response of EVERY write whose key contains `substr` until disarmed with an empty + /// substring -- the latched form of `armWriteFailure`, for a call that must never commit. + void armLatchedWriteFailure(const String & substr) { - fail_put_substr = substr; - fail_put_count = count; + std::lock_guard lock(mutex); + latched_failure_substr = substr; } /// The current length of the journal -- a caller's baseline for `indicesFrom` below, so a query can /// be scoped to "since I last looked" rather than "since the pool opened" (whose earlier entries - /// belong to unrelated setup writes, e.g. the birth transaction's own checkpoint CAS). + /// belong to unrelated setup writes, e.g. the birth transaction's own checkpoint write). size_t journalSize() const { std::lock_guard lock(mutex); return journal.size(); } - /// Every index at or after `from` where `op`/`key` matches, in order. - std::vector indicesFrom(Op op, const String & key, size_t from) const + /// Every index at or after `from` where a write of `key` was issued, in order. + std::vector indicesFrom(const String & key, size_t from) const { std::lock_guard lock(mutex); std::vector result; for (size_t i = from; i < journal.size(); ++i) - if (journal[i].op == op && journal[i].key == key) + if (journal[i] == key) result.push_back(i); return result; } - /// The first index at or after `from` where `op`/`key` matches, if any. - std::optional firstIndexFrom(Op op, const String & key, size_t from) const + /// The first index at or after `from` where a write of `key` was issued, if any. + std::optional firstIndexFrom(const String & key, size_t from) const { - const auto indices = indicesFrom(op, key, from); + const auto indices = indicesFrom(key, from); return indices.empty() ? std::nullopt : std::make_optional(indices.front()); } private: - void record(Op op, const String & key) + enum class Fault : uint8_t { None, Conflict, ResponseLost }; + + /// Journals the write and consumes at most one armed fault, all under one hold: a publisher and a + /// synchronous caller write concurrently in these fixtures, and a counted fault read outside the + /// lock would be handed to both. + Fault claimFault(const String & key) { std::lock_guard lock(mutex); - journal.push_back({op, key}); + journal.push_back(key); + if (!latched_conflict_key.empty() && key == latched_conflict_key) + return Fault::Conflict; + if (key == conflict_key && conflict_count > 0) + { + --conflict_count; + return Fault::Conflict; + } + if (!latched_failure_substr.empty() && key.find(latched_failure_substr) != String::npos) + return Fault::ResponseLost; + if (failure_count > 0 && !failure_substr.empty() && key.find(failure_substr) != String::npos) + { + --failure_count; + return Fault::ResponseLost; + } + return Fault::None; } mutable std::mutex mutex; - std::vector journal; - String fail_cas_key; - size_t fail_cas_count = 0; - String fail_put_substr; - int fail_put_count = 0; + std::vector journal; + String conflict_key; + size_t conflict_count = 0; + String latched_conflict_key; + String failure_substr; + int failure_count = 0; + String latched_failure_substr; }; /// A backend whose LIST permanently omits every key under a chosen prefix while those keys stay fully @@ -1909,7 +1999,7 @@ class OrderedFaultBackend : public CountingBackend /// arithmetic walk that finds it anyway is the property under test -- these fixtures are about the walk, /// not about any one `list` call. /// -/// Erasing keys from a page cannot disturb pagination: `ListPage::next_cursor` is computed by the base +/// Erasing keys from a page cannot disturb pagination: `next_cursor` is computed by the base /// backend before the erase, so the next page still resumes strictly after the last key it returned. /// /// Templated on the base so a suite that also needs request COUNTS composes it over `CountingBackend` @@ -1919,8 +2009,9 @@ template class HintHoleBackendOn : public Base { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy `list` name the primitive override below would otherwise shadow. using Base::list; + /// Hide every key under `prefix` from LIST -- a whole namespace, including objects a later publish /// adds. void hidePrefix(const String & prefix) @@ -1941,8 +2032,8 @@ class HintHoleBackendOn : public Base /// one call, replacing whatever was hidden before. /// /// This is the RustFS defect reproduced as an interface: every one of these keys stays durable and - /// honestly served by `get` / `head` / `putIfAbsent` / `casPut` / `deleteExact`, and only - /// enumeration pretends they are not there. Stating the omission as a SET is what lets a test say + /// honestly served by every other request, and only enumeration pretends they are not there. + /// Stating the omission as a SET is what lets a test say /// the thing the defect report says -- "ids 3 and 4 are invisible while the LATER id 5 is visible" /// -- in one line, instead of assembling it from repeated single-key calls whose combined effect a /// reader has to reconstruct. @@ -1973,14 +2064,15 @@ class HintHoleBackendOn : public Base hidden_prefixes.clear(); } - DB::Cas::ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { - DB::Cas::ListPage page = Base::list(prefix, cursor, limit); + DB::Cas::Backend::RawListPage page = Base::list(prefix, cursor, limit, access); std::lock_guard lock(hide_mutex); if (hidden_keys.empty() && hidden_prefixes.empty()) return page; const size_t before = page.keys.size(); - std::erase_if(page.keys, [&](const DB::Cas::ListedKey & k) + std::erase_if(page.keys, [&](const DB::Cas::Backend::RawListedKey & k) { if (hidden_keys.contains(k.key)) return true; @@ -2016,14 +2108,14 @@ inline void rearmMountFenceAfterAnomalyForTest(const DB::Cas::PoolPtr & store) store->armMountFence(DB::UInt128{0, 1}, store->liveWriterEpoch(), store->bootMsNow() + 600000); } -/// Delegates the FIRST matching `putIfAbsent` to `CountingBackend` -- so the write actually LANDS -- -/// and only THEN throws an ambiguous exception, modelling "our own PUT committed but its response was -/// lost". Every later call behaves normally, so a caller that retries the SAME (key, bytes) meets its -/// OWN earlier write as the occupant: the exact input the every-attempt rule's adoption arm adjudicates -/// (`slotOccupy` reports `Occupied` with bytes equal to the attempt's own). +/// Delegates the FIRST matching create-shaped write to `CountingBackend` -- so the write actually +/// LANDS -- and only THEN throws an ambiguous exception, modelling "our own PUT committed but its +/// response was lost". Every later call behaves normally, so a caller that retries the SAME (key, +/// bytes) meets its OWN earlier write as the occupant: the exact input the every-attempt rule's +/// adoption arm adjudicates (`slotOccupy` reports `Occupied` with bytes equal to the attempt's own). /// -/// `key_substr` empty means "the first putIfAbsent of any key"; set it to scope the fault to one key -/// family when the caller drives a whole Pool (whose bootstrap PUTs would otherwise consume the fault). +/// `key_substr` empty means "the first create of any key"; set it to scope the fault to one key +/// family when the caller drives a whole Pool (whose bootstrap writes would otherwise consume it). /// /// Shared rather than TU-local because two suites need exactly this shape: `gtest_cas_slot_occupy.cpp` /// pins the primitive's same-call resolve, and `gtest_cas_ref_wedge_every_attempt.cpp` drives the @@ -2031,8 +2123,6 @@ inline void rearmMountFenceAfterAnomalyForTest(const DB::Cas::PoolPtr & store) class LandedButAckLostOnceBackend : public CountingBackend { public: - using CountingBackend::putIfAbsent; - using CountingBackend::get; String key_substr; bool fired = false; /// Also lose the caller's IMMEDIATE resolve read of the same key, once. Needed only by a caller @@ -2042,46 +2132,45 @@ class LandedButAckLostOnceBackend : public CountingBackend /// retry loop -- which is why this defaults off and this file's original caller is unaffected. bool lose_resolve_read = false; - DB::Cas::PutResult putIfAbsent(const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - if (!fired && (key_substr.empty() || key.find(key_substr) != String::npos)) + if (!fired && !expected_value && (key_substr.empty() || key.find(key_substr) != String::npos)) { fired = true; - CountingBackend::putIfAbsent(key, bytes, meta); /// the write LANDS + (void)CountingBackend::write(key, bytes, expected_value, access); /// the write LANDS if (lose_resolve_read) - fail_get_once_key = key; + fail_read_once_key = key; throw Poco::TimeoutException("LandedButAckLostOnceBackend: simulated lost PUT response"); } - return CountingBackend::putIfAbsent(key, bytes, meta); + return CountingBackend::write(key, bytes, expected_value, access); } - std::optional get(const String & key, DB::Cas::Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - if (!fail_get_once_key.empty() && key == fail_get_once_key) + if (!fail_read_once_key.empty() && key == fail_read_once_key) { - fail_get_once_key.clear(); + fail_read_once_key.clear(); throw Poco::TimeoutException( "LandedButAckLostOnceBackend: simulated lost GET (read response never arrived)"); } - return CountingBackend::get(key, range); + return CountingBackend::read(key, access); } private: - String fail_get_once_key; + String fail_read_once_key; }; -/// A `CountingBackend` that can fault selected PUTs by key substring (skip the first `fault_skip` -/// matches, then fault the next `fault_count`), and can latch a matching PUT mid-flight. Same class of -/// seam as the wedge tests in `gtest_cas_ref_writer.cpp` use +/// A `CountingBackend` that can fault selected create-shaped writes by key substring (skip the first +/// `fault_skip` matches, then fault the next `fault_count`), and can latch a matching write mid-flight. +/// Same class of seam as the wedge tests in `gtest_cas_ref_writer.cpp` use /// (`fault_key_substr`/`corrupt_key_substr`/`armPutBlock`), narrowed to what the ref-lane tests need. /// Shared (rather than TU-local) because the chunk-boundary tests and the post-durable install-safety /// tests need exactly the same seam. class ChunkFaultBackend : public CountingBackend { public: - using CountingBackend::putIfAbsent; - using CountingBackend::get; - /// Unresolved -> a lost-response ambiguity, NOTHING landed; with a single-attempt budget this /// wedges the lane and a later resolve proves the key ABSENT. /// LandedThenLost -> our OWN exact bytes land and only the acknowledgement is lost, AND the @@ -2105,23 +2194,25 @@ class ChunkFaultBackend : public CountingBackend Mode mode = Mode::None; int fault_skip = 0; int fault_count = 0; - /// One-shot: the next `get` of exactly this key throws, then it is cleared. Armed by + /// One-shot: the next read of exactly this key throws, then it is cleared. Armed by /// `Mode::LandedThenLost` (see above); settable directly for a bare lost-read fault. - String fail_get_once_key; + String fail_read_once_key; - std::optional get(const String & key, DB::Cas::Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - if (!fail_get_once_key.empty() && key == fail_get_once_key) + if (!fail_read_once_key.empty() && key == fail_read_once_key) { - fail_get_once_key.clear(); + fail_read_once_key.clear(); throw Poco::TimeoutException("ChunkFaultBackend: simulated lost GET (read response never arrived)"); } - return CountingBackend::get(key, range); + return CountingBackend::read(key, access); } - DB::Cas::PutResult putIfAbsent(const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - if (mode != Mode::None && !fault_substr.empty() && key.find(fault_substr) != String::npos) + if (mode != Mode::None && !expected_value && !fault_substr.empty() && key.find(fault_substr) != String::npos) { if (fault_skip > 0) { @@ -2141,8 +2232,8 @@ class ChunkFaultBackend : public CountingBackend /// armed to fail ONCE for this key as well, or it would prove the object durable /// inside this very attempt and the lane would never wedge; the wedge-resolution /// GET a flush later then reads it normally. - CountingBackend::putIfAbsent(key, bytes, meta); - fail_get_once_key = key; + (void)CountingBackend::write(key, bytes, expected_value, access); + fail_read_once_key = key; throw Poco::TimeoutException("ChunkFaultBackend: object landed; response lost"); case Mode::Definite: #if USE_AWS_S3 @@ -2155,7 +2246,8 @@ class ChunkFaultBackend : public CountingBackend case Mode::ForeignConflict: /// A foreign writer lands DIFFERENT bytes at this exact key; then our response is /// lost, so resolve-before-reissue GETs foreign bytes -> CORRUPTED_DATA. - CountingBackend::putIfAbsent(key, bytes + String("\x01_FOREIGN_DIFFERENT")); + (void)CountingBackend::write(key, bytes + String("\x01_FOREIGN_DIFFERENT"), + expected_value, access); throw Poco::TimeoutException("ChunkFaultBackend: foreign different object landed; response lost"); case Mode::None: break; @@ -2172,7 +2264,7 @@ class ChunkFaultBackend : public CountingBackend block_cv.wait_for(lk, std::chrono::seconds(20), [&] { return !block_armed; }); } } - return CountingBackend::putIfAbsent(key, bytes, meta); + return CountingBackend::write(key, bytes, expected_value, access); } void armBlock(const String & substr) @@ -2210,49 +2302,76 @@ class ChunkFaultBackend : public CountingBackend bool block_entered = false; }; -/// Fault decorator for the condemn-marker gate tests (codex-review triage 2026-07-17 §3.4): while -/// armed, every conditional-write attempt against a blob `.meta` key throws. The request controller -/// exhausts its budget and reports `Unresolved`, so `writeCondemnedMeta` returns false while the round -/// still commits the unconfirmed retired entry. Every other write passes through. Armed by default; -/// disarm (`fail_meta_writes = false`) to model the backend healing. -class MetaWriteFaultBackend : public DB::Cas::InMemoryBackend +/// `ChunkFaultBackend` COUNTS its faults, and a count can no longer make one conclusive: the write +/// engine settles every ambiguity by an exact read and then REISSUES, so a fault that runs out +/// mid-call is answered by the next attempt instead of by the call's own deadline -- which is the +/// whole difference between a wedge and a commit. This keeps the fault armed until the test clears +/// the latch, on BOTH legs: the write's, and the lost read that `Mode::LandedThenLost` arms. The read +/// leg matters just as much, because a readable key proves the commit inside the very same call. +class LatchedChunkFaultBackend : public ChunkFaultBackend { public: - /// Unhide the base convenience overloads (omitted Range/ObjectMeta/expected-token forms): the - /// overrides below would otherwise shadow them for callers holding a concrete backend type. - using DB::Cas::Backend::get; - using DB::Cas::Backend::getStream; - using DB::Cas::Backend::putIfAbsent; - using DB::Cas::Backend::putOverwrite; - using DB::Cas::Backend::casPut; + /// Set after `mode` / `fault_substr` / `fault_skip`; cleared when the scenario is over, so the + /// test's own out-of-band writes and reads are not caught by it. + bool latched = false; - DB::Cas::PutResult putIfAbsent( - const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - if (fail_meta_writes.load() && key.ends_with(".meta")) - throw std::runtime_error("injected fault: blob meta write lost"); - return InMemoryBackend::putIfAbsent(key, bytes, meta); + if (latched && !fail_read_once_key.empty() && key == fail_read_once_key) + throw Poco::TimeoutException("LatchedChunkFaultBackend: the lost read stays lost"); + return ChunkFaultBackend::read(key, access); } - DB::Cas::PutResult putOverwrite( - const String & key, const String & bytes, const DB::Cas::Token & expected, - const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - if (fail_meta_writes.load() && key.ends_with(".meta")) - throw std::runtime_error("injected fault: blob meta write lost"); - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + if (latched && mode != Mode::None && fault_skip == 0 && !expected_value && !fault_substr.empty() + && key.find(fault_substr) != String::npos) + fault_count = 1; + return ChunkFaultBackend::write(key, bytes, expected_value, access); + } +}; + +/// Fault decorator for the condemn-marker gate tests: while armed, every write against a blob `.meta` +/// key throws. Every other write passes through. Armed by default; disarm +/// (`fail_meta_writes = false`) to model the backend healing. +/// +/// The fault's CLASS is chosen at arming, because the two classes model different failures and the +/// engine treats them differently. `Propagates` is a local error the write loop rethrows on the first +/// attempt, so the caller's own handler sees it at once. `Ambiguous` is a timeout the loop cannot +/// distinguish from a lost response: it resolves by a read and reissues until its policy bound, so a +/// test arming it against a PERMANENT fault must drive the operation's clock or spend the whole +/// retry window in real time. +class MetaWriteFaultBackend : public DB::Cas::InMemoryBackend +{ +public: + enum class FaultKind : uint8_t { Propagates, Ambiguous }; + + /// Fault every `.meta` write with `kind`. Construction arms `Propagates`. + void armWriteFault(FaultKind kind = FaultKind::Propagates) + { + fault_kind.store(kind); + fail_meta_writes.store(true); } - DB::Cas::CasResult casPut(const String & key, const String & bytes, - const std::optional & expected, - const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (fail_meta_writes.load() && key.ends_with(".meta")) + { + if (fault_kind.load() == FaultKind::Ambiguous) + throw Poco::TimeoutException("injected fault: blob meta write response lost"); throw std::runtime_error("injected fault: blob meta write lost"); - return InMemoryBackend::casPut(key, bytes, expected, meta); + } + return InMemoryBackend::write(key, bytes, expected_value, access); } std::atomic fail_meta_writes{true}; + +private: + std::atomic fault_kind{FaultKind::Propagates}; }; /// Blocks INSIDE a blob-meta mutation until `release` is called, so a test can hold a real meta job in @@ -2264,12 +2383,6 @@ class MetaWriteFaultBackend : public DB::Cas::InMemoryBackend class MetaWriteLatchBackend : public DB::Cas::InMemoryBackend { public: - using DB::Cas::Backend::get; - using DB::Cas::Backend::getStream; - using DB::Cas::Backend::putIfAbsent; - using DB::Cas::Backend::putOverwrite; - using DB::Cas::Backend::casPut; - std::atomic entered{false}; void arm() @@ -2284,33 +2397,19 @@ class MetaWriteLatchBackend : public DB::Cas::InMemoryBackend latch_cv.notify_all(); } - DB::Cas::PutResult putIfAbsent( - const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override - { - waitIfMeta(key); - return InMemoryBackend::putIfAbsent(key, bytes, meta); - } - - DB::Cas::PutResult putOverwrite( - const String & key, const String & bytes, const DB::Cas::Token & expected, - const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { waitIfMeta(key); - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); - } - - DB::Cas::CasResult casPut(const String & key, const String & bytes, - const std::optional & expected, - const DB::Cas::ObjectMeta & meta) override - { - waitIfMeta(key); - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } - DB::Cas::DeleteOutcome deleteExact(const String & key, const DB::Cas::Token & token) override + RawRemoval remove(const String & key, const String & expected_value, + DB::Cas::TransportAccess & access) override { waitIfMeta(key); - return InMemoryBackend::deleteExact(key, token); + return InMemoryBackend::remove(key, expected_value, access); } private: @@ -2335,24 +2434,22 @@ class MetaWriteLatchBackend : public DB::Cas::InMemoryBackend class OutcomeLogFaultBackend : public MetaWriteLatchBackend { public: - using DB::Cas::Backend::get; - using DB::Cas::Backend::putIfAbsent; - std::atomic fail_outcome_logs{false}; - DB::Cas::PutResult putIfAbsent( - const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (fail_outcome_logs.load() && key.contains("outcomes/")) - return DB::Cas::PutResult{.outcome = DB::Cas::PutOutcome::PreconditionFailed, .token = {}}; - return MetaWriteLatchBackend::putIfAbsent(key, bytes, meta); + return std::unexpected(RawConflict{}); + return MetaWriteLatchBackend::write(key, bytes, expected_value, access); } - std::optional get(const String & key, DB::Cas::Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (fail_outcome_logs.load() && key.contains("outcomes/")) return std::nullopt; - return DB::Cas::InMemoryBackend::get(key, range); + return DB::Cas::InMemoryBackend::read(key, access); } }; @@ -2375,35 +2472,22 @@ inline void awaitLatchEntered(MetaWriteLatchBackend & backend) class MountSlotRaceBackend : public DB::Cas::InMemoryBackend { public: - using DB::Cas::Backend::get; - using DB::Cas::Backend::getStream; - using DB::Cas::Backend::putIfAbsent; - using DB::Cas::Backend::putOverwrite; - using DB::Cas::Backend::casPut; - std::function before_put_if_absent; std::function before_get; std::function before_put_overwrite; - DB::Cas::PutResult putIfAbsent( - const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - fire(before_put_if_absent); - return InMemoryBackend::putIfAbsent(key, bytes, meta); + fire(expected_value ? before_put_overwrite : before_put_if_absent); + return InMemoryBackend::write(key, bytes, expected_value, access); } - std::optional get(const String & key, DB::Cas::Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { fire(before_get); - return InMemoryBackend::get(key, range); - } - - DB::Cas::PutResult putOverwrite( - const String & key, const String & bytes, const DB::Cas::Token & expected, - const DB::Cas::ObjectMeta & meta) override - { - fire(before_put_overwrite); - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + return InMemoryBackend::read(key, access); } private: @@ -2417,6 +2501,67 @@ class MountSlotRaceBackend : public DB::Cas::InMemoryBackend } }; +/// The engine reissues an unresolved write until its OWN retry window closes, and that window is +/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine +/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline +/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling +/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the +/// closures own the clock, so it outlives everything that can still read it. +class VirtualRetryClock +{ +public: + static std::shared_ptr installOn(const PoolPtr & store) + { + auto clock = std::make_shared(); + store->setCasRequestNowFnForTest(nowFnOf(clock)); + store->setCasRetrySleepForTest(sleepFnOf(clock)); + return clock; + } + + /// The two seams on their own, for a fixture that assembles its own `CasRequests` and ledger + /// rather than a whole `Pool`. Each closure keeps the clock alive. + static std::function nowFnOf(std::shared_ptr owned) + { + return [clock = std::move(owned)] { return clock->nowMs(); }; + } + static std::function sleepFnOf(std::shared_ptr owned) + { + return [clock = std::move(owned)](uint64_t ms) { clock->advance(ms); }; + } + + uint64_t nowMs() const + { + std::lock_guard lock(mutex); + return now_ms; + } + size_t pauseCount() const + { + std::lock_guard lock(mutex); + return pauses; + } + uint64_t longestPause() const + { + std::lock_guard lock(mutex); + return longest_pause; + } + + void advance(uint64_t ms) + { + std::lock_guard lock(mutex); + /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move + /// would leave the loop reissuing for ever against a fault that never clears. + now_ms += ms + 1; + ++pauses; + longest_pause = std::max(longest_pause, ms); + } + +private: + mutable std::mutex mutex; + uint64_t now_ms = 0; + size_t pauses = 0; + uint64_t longest_pause = 0; +}; + /// Expect a DB::Exception with EXACTLY `expected_code` AND a message containing `expected_substring`. /// Needed wherever several distinct branches share one code: the code alone does not identify which /// one ran, so a test that silently takes the wrong branch would still pass. @@ -2437,3 +2582,13 @@ void expectThrowsCodeWithMessage(int expected_code, const String & expected_subs } } + +/// The mount and GC suites call these two unqualified, under `using namespace DB::Cas;`. Argument- +/// dependent lookup does not reach `DB::Cas::tests` from a `shared_ptr`, so the names +/// are re-exported here rather than moved: the qualified `DB::Cas::tests::` spelling the rest of the +/// tree uses keeps working, and there is still one definition. +namespace DB::Cas +{ +using tests::OperationForTest; +using tests::openRequestsForTest; +} diff --git a/src/Disks/tests/gtest_ca_wiring.cpp b/src/Disks/tests/gtest_ca_wiring.cpp index dad37b10bedc..2003b6374c99 100644 --- a/src/Disks/tests/gtest_ca_wiring.cpp +++ b/src/Disks/tests/gtest_ca_wiring.cpp @@ -4,6 +4,24 @@ #include #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include namespace DB::ErrorCodes { @@ -316,20 +334,6 @@ TEST(CASPartPathParser, SplitCacheEvictionStaysCorrect) /// the rewritten ContentAddressedMetadataStorage (real ctor over a Local object storage; the /// backend self-selects EmulatedSingleProcess token semantics). -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include using DB::Cas::tests::idOf; using DB::Cas::tests::u128Of; @@ -2035,10 +2039,6 @@ TEST(CASWiringExchangeDeathTest, PrepareAdoptRefusesATargetThatIsNotAPartDirecto /// ==== Commit atomicity (B122): a publish failing mid-loop must not leave a PARTIAL commit ==== -#include -#include -#include -#include namespace DB::ErrorCodes { diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index ccef312ef908..aabb3f21f0c1 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -5,15 +5,15 @@ #include #include #include +#include +#include #include #include #include +#include #include #include #include -#include -#include -#include #include #include @@ -35,6 +35,8 @@ using namespace DB::Cas; +using DB::Cas::tests::expectBytes; + namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; @@ -261,11 +263,11 @@ TEST(CASInMemory, OverwriteIsTokenExactAndMintsFreshToken) InMemoryBackend b; const Token t1 = b.putIfAbsent("k", "v1").token; EXPECT_EQ(b.putOverwrite("k", "v2", Token{"wrong", TokenType::Emulated}).outcome, PutOutcome::PreconditionFailed); - EXPECT_EQ(b.get("k")->bytes, "v1"); // untouched on mismatch + expectBytes(b, "k", "v1"); // untouched on mismatch const auto overwrite = b.putOverwrite("k", "v2", t1); EXPECT_EQ(overwrite.outcome, PutOutcome::Done); EXPECT_NE(overwrite.token, t1); // tokens never repeat - EXPECT_EQ(b.get("k")->bytes, "v2"); + expectBytes(b, "k", "v2"); } TEST(CASInMemory, CasPutCreateAndSwap) @@ -453,14 +455,14 @@ TEST(CASInMemoryFaults, HeldDeleteLandsLater) b.putOverwrite("k", "v1'", t1); auto landed = b.landPendingDelete(0); // the zombie lands NOW EXPECT_EQ(landed.kind, DeleteOutcome::Kind::TokenMismatch); // 412 — INV-NO-RETURN in miniature - EXPECT_EQ(b.get("k")->bytes, "v1'"); + expectBytes(b, "k", "v1'"); } TEST(CASInMemoryFaults, InjectedCasConflictFiresOnce) { InMemoryBackend b; const Token t1 = b.casPut("m", "s1", std::nullopt).token; - b.failNextCasPut("m"); + b.refuseNextWrite("m"); EXPECT_EQ(b.casPut("m", "s2", t1).outcome, CasOutcome::Conflict); // injected EXPECT_EQ(b.get("m")->bytes, "s1"); EXPECT_EQ(b.casPut("m", "s2", t1).outcome, CasOutcome::Committed); // next attempt is real @@ -636,19 +638,14 @@ TEST(CASSizedReadSettings, CapsToKnownSizePlusSlackButNeverAboveBase) EXPECT_EQ(unknown.remote_fs_settings.buffer_size, 1ULL << 20); } -/// The CountingBackend request-shape recorders that the streaming-memory gates consume: per-key and -/// total getStream counts, and the whole-object get flag that marks a resident-memory violation for a -/// run object. The ranged-window recorder is no longer exercised here: a materialized read is always -/// whole now, so only `getStream` still carries a window. -TEST(CASCountingBackendShape, RecordsGetStreamAndWholeGetShape) +/// The CountingBackend recorders the streaming-memory gates consume: per-key and total stream counts. +/// A window is no longer part of the shape -- a materialized read is always whole, so only `getStream` +/// still carries one, and it is not what the gates measure. +TEST(CASCountingBackendShape, RecordsStreamOpensPerKeyAndInTotal) { DB::Cas::tests::CountingBackend backend; backend.putIfAbsent("k", String(1000, 'x')); - backend.get("k"); - EXPECT_EQ(backend.wholeGetCount("k"), 1u); - - /// getStream counters (per-key and total). backend.getStream("k", DB::Cas::Range{.offset = 2, .length = 5}); backend.getStream("k"); backend.getStream("absent"); @@ -656,10 +653,92 @@ TEST(CASCountingBackendShape, RecordsGetStreamAndWholeGetShape) EXPECT_EQ(backend.getStreamTotal(), 3u); backend.resetCounts(); - EXPECT_EQ(backend.wholeGetCount("k"), 0u); + EXPECT_EQ(backend.getStreamCount("k"), 0u); EXPECT_EQ(backend.getStreamTotal(), 0u); } +/// Armed chunking makes this backend serve a stream the way a network-backed store does, in bounded +/// windows, instead of handing over the materialized object in one piece. The bytes a consumer reads +/// are the same either way; what changes is that a consumer which assumed one contiguous window can no +/// longer get one. +TEST(CASCountingBackendShape, AnArmedChunkBoundsTheWindowAStreamHandsOut) +{ + const String body(10'000, 'x'); + auto backend = std::make_shared(); + ASSERT_EQ(backend->putIfAbsent("run", body).outcome, PutOutcome::Done); + + /// Unarmed: the whole object arrives as one window, which is what this backend's materialization + /// makes of any stream and exactly what the bound exists to remove. + { + auto opened = backend->getStream("run"); + ASSERT_TRUE(opened); + String drained; + readStringUntilEOF(drained, *opened->stream); + EXPECT_EQ(drained, body); + EXPECT_EQ(backend->largestStreamChunk("run"), 0u) << "nothing records a window while chunking is off"; + } + + backend->setStreamChunkForTest(4096); + { + auto opened = backend->getStream("run"); + ASSERT_TRUE(opened); + String drained; + readStringUntilEOF(drained, *opened->stream); + EXPECT_EQ(drained, body) << "chunking changes the window, never the bytes"; + EXPECT_EQ(backend->largestStreamChunk("run"), 4096u); + EXPECT_LT(backend->largestStreamChunk("run"), body.size()) + << "the consumer never held the object entire"; + } + + /// The mode outlives a counter reset, and the recorded window does not. + backend->resetCounts(); + EXPECT_EQ(backend->largestStreamChunk("run"), 0u); + auto reopened = backend->getStream("run"); + ASSERT_TRUE(reopened); + String again; + readStringUntilEOF(again, *reopened->stream); + EXPECT_EQ(backend->largestStreamChunk("run"), 4096u); +} + +/// What makes every request-profile gate in this tree trustworthy: a counter names a PHYSICAL request, +/// so the same request counts once whichever surface issued it. Before the counters moved onto the +/// transport primitives a legacy call and a `CasOperation` call landed on different counters, and a +/// gate written against one was blind to the other. +TEST(CASCountingBackendShape, OneRequestIsCountedOnceWhicheverSurfaceIssuedIt) +{ + auto backend = std::make_shared(); + DB::Cas::CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + DB::Cas::CasOperation op = requests.admit(); + + EXPECT_EQ(backend->putIfAbsent("k", "v").outcome, PutOutcome::Done); /// legacy create + EXPECT_TRUE(std::holds_alternative( + op.create("k2", "v", Retry::standard()))); /// the same request, admitted + EXPECT_EQ(backend->putCount("k"), 1u); + EXPECT_EQ(backend->putCount("k2"), 1u); + EXPECT_EQ(backend->writeTotal(), 2u); + EXPECT_EQ(backend->putOverwriteTotal(), 0u) << "neither write carried a precondition"; + + const Token seen = backend->head("k").token; /// legacy head + EXPECT_TRUE(op.head("k", Retry::standard())); /// admitted head + EXPECT_EQ(backend->headCount("k"), 2u); + + expectBytes(*backend, "k", "v"); /// legacy read + EXPECT_TRUE(op.read("k", Retry::standard())); /// admitted read + EXPECT_EQ(backend->getCount("k"), 2u); + + EXPECT_EQ(backend->putOverwrite("k", "w", seen).outcome, PutOutcome::Done); + EXPECT_EQ(backend->putOverwriteCount("k"), 1u) << "a write with a precondition is the replace shape"; + EXPECT_EQ(backend->writeCount("k"), 2u); + + const std::optional k2_meta = op.head("k2", Retry::standard()); + ASSERT_TRUE(k2_meta); + EXPECT_EQ(op.remove("k2", k2_meta->incarnation, Retry::standard()), Removal::Removed); + EXPECT_EQ(backend->deleteExact("k", backend->head("k").token).kind, DeleteOutcome::Kind::Deleted); + EXPECT_EQ(backend->deleteCount("k"), 1u); + EXPECT_EQ(backend->deleteCount("k2"), 1u); + EXPECT_EQ(backend->deleteTotal(), 2u); +} + #if USE_AWS_S3 namespace diff --git a/src/Disks/tests/gtest_cas_backend_contract.cpp b/src/Disks/tests/gtest_cas_backend_contract.cpp index 4e1855ed1a87..d84757c67573 100644 --- a/src/Disks/tests/gtest_cas_backend_contract.cpp +++ b/src/Disks/tests/gtest_cas_backend_contract.cpp @@ -8,6 +8,13 @@ using namespace DB::Cas; +namespace DB::ErrorCodes +{ +extern const int NOT_IMPLEMENTED; +} + +using DB::Cas::tests::expectBytes; + /// Parameterized contract suite: every case creates a fresh backend from the factory, /// then exercises the Backend seam generically (no InMemoryBackend-specific calls). /// Fault-injection-only features are excluded — those are InMemory-specific tests. @@ -35,11 +42,11 @@ TEST_P(CASBackendContract, OverwriteIsTokenExactAndMintsFreshToken) auto b = GetParam()(); const Token t1 = b->putIfAbsent("k", "v1").token; EXPECT_EQ(b->putOverwrite("k", "v2", Token{"wrong", TokenType::Emulated}).outcome, PutOutcome::PreconditionFailed); - EXPECT_EQ(b->get("k")->bytes, "v1"); // untouched on mismatch + expectBytes(b, "k", "v1"); // untouched on mismatch const auto overwrite = b->putOverwrite("k", "v2", t1); EXPECT_EQ(overwrite.outcome, PutOutcome::Done); EXPECT_NE(overwrite.token, t1); // tokens never repeat - EXPECT_EQ(b->get("k")->bytes, "v2"); + expectBytes(b, "k", "v2"); } TEST_P(CASBackendContract, CasPutCreateAndSwap) @@ -50,9 +57,9 @@ TEST_P(CASBackendContract, CasPutCreateAndSwap) EXPECT_EQ(create.outcome, CasOutcome::Committed); // create-if-absent EXPECT_EQ(b->casPut("m", "s1x", std::nullopt).outcome, CasOutcome::Conflict); // exists now EXPECT_EQ(b->casPut("m", "s2", Token{"stale", TokenType::Emulated}).outcome, CasOutcome::Conflict); - EXPECT_EQ(b->get("m")->bytes, "s1"); + expectBytes(b, "m", "s1"); EXPECT_EQ(b->casPut("m", "s2", t1).outcome, CasOutcome::Committed); - EXPECT_EQ(b->get("m")->bytes, "s2"); + expectBytes(b, "m", "s2"); } TEST_P(CASBackendContract, DeleteExactnessAndSurvival) @@ -85,8 +92,8 @@ TEST_P(CASBackendContract, RangedGetIsRefusedAndTheWholeReadStillServes) Range r; r.offset = 2; r.length = 3u; - EXPECT_THROW(b->get("k", r), DB::Exception); - EXPECT_EQ(b->get("k")->bytes, "0123456789"); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NOT_IMPLEMENTED, [&] { (void)b->get("k", r); }); + expectBytes(b, "k", "0123456789"); } TEST_P(CASBackendContract, Head) diff --git a/src/Disks/tests/gtest_cas_blob_digest.cpp b/src/Disks/tests/gtest_cas_blob_digest.cpp index b6f8c9f3f972..e66ce8239d5e 100644 --- a/src/Disks/tests/gtest_cas_blob_digest.cpp +++ b/src/Disks/tests/gtest_cas_blob_digest.cpp @@ -73,7 +73,8 @@ TEST(CASBlobDigest, ShardOfViaPoolMetaConstructedCodecMatchesOldBlobShard) { auto backend = std::make_shared(); const Layout layout("p"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + const PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); ASSERT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); const DigestCodec codec = codecFor(BlobHashAlgo::CityHash128); @@ -227,26 +228,29 @@ TEST(CASBlobDigest, PoolMetaRecordsCreatingAlgoAndWidthDerivesFromIt) { auto backend = std::make_shared(); const Layout layout("p1"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + const PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); EXPECT_EQ(blobHashLenFor(BlobHashAlgo::CityHash128), 16u); } { auto backend = std::make_shared(); const Layout layout("p2"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + const PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::XXH3_128)})); EXPECT_EQ(blobHashLenFor(BlobHashAlgo::XXH3_128), 16u); } { auto backend = std::make_shared(); const Layout layout("p3"); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + const PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::Sha256, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::Sha256)})); EXPECT_EQ(blobHashLenFor(BlobHashAlgo::Sha256), 32u); /// Reopen (decode path) must re-derive the same recorded algo. - const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256); + const PoolMeta reopened = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::Sha256); EXPECT_EQ(reopened.algos_used, (std::vector{static_cast(BlobHashAlgo::Sha256)})); } } diff --git a/src/Disks/tests/gtest_cas_blob_indegree.cpp b/src/Disks/tests/gtest_cas_blob_indegree.cpp index a340fde248ac..7df585a018a9 100644 --- a/src/Disks/tests/gtest_cas_blob_indegree.cpp +++ b/src/Disks/tests/gtest_cas_blob_indegree.cpp @@ -5,11 +5,16 @@ #include #include #include +#include +#include "config.h" +#if USE_AWS_S3 +#include +#endif #include #include #include -namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; extern const int NOT_IMPLEMENTED; } +namespace DB::ErrorCodes { extern const int ABORTED; extern const int CORRUPTED_DATA; extern const int NOT_IMPLEMENTED; } using namespace DB::Cas; @@ -21,21 +26,18 @@ UInt128 s(uint64_t n) { return UInt128(n); } // source-edge id /// `BlobCandidate.ref` / `inDegreeInRuns` argument is a `BlobRef` as of Phase 3 T3. BlobRef bh(uint64_t n) { return BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(n))}; } -/// Scale thresholds for the "the run genuinely spans several blocks" sanity assertions below. These are -/// NOT format constants — the SourceEdge run is a plain NDJSON stream (`CasRecordStreamFormat`) with no -/// block framing of its own — they only pin the same byte-size scale the (now-deleted, codecs-v3 phase 6) +/// Scale threshold for the "the run genuinely spans several blocks" sanity assertions below. This is +/// NOT a format constant — the SourceEdge run is a plain NDJSON stream (`CasRecordStreamFormat`) with no +/// block framing of its own — it only pins the same byte-size scale the (now-deleted, codecs-v3 phase 6) /// `CasRunFile` block codec used, so the multi-block-sized fixtures below stay meaningfully large. -/// (Previously read straight off `CasRunFile.h`'s own `kRunTargetBlockSize`/`kRunHardCapBlockSize`; this -/// file's `#include` of that header looked removable when `CasRunFile` was deleted in the phase-6 cutover, -/// but these two thresholds turned out to be the only remaining users — hence the local, explicitly-legacy -/// copies here instead of a dangling include. Values unchanged.) constexpr uint32_t kLegacyBlockSize = 256u * 1024u; -constexpr uint32_t kLegacyHardCapBlockSize = 1024u * 1024u; + } TEST(CASBlobInDegree, FoldStartsFromEmptyPriorGeneration) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Generation 1 from empty prior: two distinct edges on b1 and one on b2. @@ -46,29 +48,30 @@ TEST(CASBlobInDegree, FoldStartsFromEmptyPriorGeneration) {bh(2), s(1), false}, }; std::vector runs; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, /*new*/1, /*attempt*/0, /*shard*/0, deltas, runs); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, /*new*/1, /*attempt*/0, /*shard*/0, deltas, runs); ASSERT_FALSE(runs.empty()); - const auto zero = zeroInDegree(backend, runs); + const auto zero = zeroInDegree(*backend_req, runs); EXPECT_TRUE(zero.empty()); /// nothing at zero yet } TEST(CASBlobInDegree, PlusMinusCancelToZeroDetectsCandidate) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Gen 1: activate edge (b1,s1) and (b2,s1). std::vector runs1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, {{bh(1), s(1), false}, {bh(2), s(1), false}}, runs1); /// Generation 2 merges prior gen-1 run (resolved via runs1 refs) with removal of (b1,s1): indeg(b1)=0, indeg(b2)=1. std::vector runs2; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, /*new*/2, /*attempt*/0, 0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, /*new*/2, /*attempt*/0, 0, {{bh(1), s(1), true}}, runs2); - const auto zero = zeroInDegree(backend, runs2); + const auto zero = zeroInDegree(*backend_req, runs2); ASSERT_EQ(zero.size(), 1u); EXPECT_EQ(zero[0].ref, bh(1)); } @@ -76,14 +79,16 @@ TEST(CASBlobInDegree, PlusMinusCancelToZeroDetectsCandidate) TEST(CASBlobInDegree, RunsAreByteDeterministic) { InMemoryBackend a; + DB::Cas::tests::OperationForTest a_req(a); InMemoryBackend b2; + DB::Cas::tests::OperationForTest b2_req(b2); Layout layout{"pool"}; std::vector ra; std::vector rb; /// Same deltas in a DIFFERENT input order must produce the same sealed run bytes (sorted by key). - foldDeltasIntoGeneration(a, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, + foldDeltasIntoGeneration(*a_req, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, {{bh(3), s(1), false}, {bh(1), s(1), false}, {bh(2), s(1), false}}, ra); - foldDeltasIntoGeneration(b2, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, + foldDeltasIntoGeneration(*b2_req, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, {{bh(1), s(1), false}, {bh(2), s(1), false}, {bh(3), s(1), false}}, rb); const auto ga = a.get(layout.blobTargetRunKey(1, /*attempt*/0, 0, 0)); const auto gb = b2.get(layout.blobTargetRunKey(1, /*attempt*/0, 0, 0)); @@ -101,46 +106,91 @@ TEST(CASBlobInDegree, SameEdgeActivatedTwiceCountsOnce) /// The source-edge set is a SET, not a counter — re-adding the same edge is a no-op. /// indeg(b1) must be 1 after both activations, not 2. InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; std::vector deltas{ {bh(1), s(1), false}, // activate (b1,s1) {bh(1), s(1), false}, // same edge again — must deduplicate }; std::vector runs; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, deltas, runs); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, deltas, runs); ASSERT_FALSE(runs.empty()); const int64_t deg = DB::Cas::tests::inDegreeInRuns(backend, runs, bh(1)); EXPECT_EQ(deg, 1); /// deduplicated, not 2 - const auto zero = zeroInDegree(backend, runs); + const auto zero = zeroInDegree(*backend_req, runs); EXPECT_TRUE(zero.empty()); /// b1 still has an active edge } TEST(CASBlobInDegree, FoldDeltaByteEqualReplayAdopts) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; std::vector deltas{{bh(1), s(1), false}}; std::vector runs1; std::vector runs2; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, /*attempt*/7, /*shard*/0, deltas, runs1); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, /*attempt*/7, /*shard*/0, deltas, runs1); /// Same inputs, same attempt => byte-identical run already present => adopt, no throw. - EXPECT_NO_THROW(foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, /*attempt*/7, /*shard*/0, deltas, runs2)); + EXPECT_NO_THROW(foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, /*attempt*/7, /*shard*/0, deltas, runs2)); EXPECT_EQ(runs1, runs2); } TEST(CASBlobInDegree, FoldDeltaDivergentBytesThrowsCorrupted) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Pre-occupy the run key (attempt 7) with junk, then fold => divergent => CORRUPTED_DATA. backend.putIfAbsent(layout.blobTargetRunKey(1, /*attempt*/7, /*shard*/0, /*seq*/0), "not-a-valid-run"); std::vector deltas{{bh(1), s(1), false}}; std::vector runs; - EXPECT_THROW(foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, /*attempt*/7, /*shard*/0, deltas, runs), - DB::Exception); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, /*attempt*/7, /*shard*/0, deltas, runs); }); +} + +#if USE_AWS_S3 +namespace +{ +/// Refuses to serve one key's body. An access denial gets one credential refresh first; it surfaces on +/// the first attempt only because `InMemoryBackend::refreshCredentials` answers false by default, so the +/// write's resolve read ends having observed nothing. +class ReadRefusingBackend : public InMemoryBackend +{ +public: + std::optional read(const String & key, TransportAccess & access) override + { + if (key == refuse_key) + throw DB::S3Exception("injected access denial on the resolve read", Aws::S3::S3Errors::ACCESS_DENIED); + return InMemoryBackend::read(key, access); + } + + String refuse_key; +}; +} + +/// A refused write whose resolve read observed NOTHING says nothing about what is at the key, so it +/// must not be reported as pool corruption. `CORRUPTED_DATA` is a deterministic local failure: no +/// caller above reissues it, so a permission or credential blip during the resolve would wedge every +/// later round on the same artifact. The companion arm is `FoldDeltaDivergentBytesThrowsCorrupted`, +/// where the read DID observe divergent bytes and corruption is the right verdict. +TEST(CASBlobInDegree, DeterministicArtifactWhoseResolveReadObservedNothingIsNotCorruption) +{ + ReadRefusingBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); + Layout layout{"pool"}; + const String key = layout.blobTargetRunKey(1, /*attempt*/0, /*shard*/0, /*seq*/0); + + /// Occupy the key so the create's precondition is refused, THEN arm the refusal, so the failure + /// falls on the resolve read rather than on the setup. + ASSERT_EQ(backend.putIfAbsent(key, "someone else's bytes").outcome, PutOutcome::Done); + backend.refuse_key = key; + + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::ABORTED, + [&] { putDeterministicArtifact(*backend_req, key, "our deterministic bytes"); }); } +#endif /// ==== two-cursor settlement merge (retired-in-snapshot T3, spec §2.1/§3) ==== /// @@ -167,16 +217,16 @@ SourceEdgeRecord edgeRec(UInt128 h, UInt128 sid) .source_id = sid, .marker = RunMarker::Edge}; } -/// head_blob / peek_head stub: present with a fixed token/size. -std::function(const BlobRef &)> headPresent(const String & tok, uint64_t size) +/// `head_blob` / `peek_head` stub. Only the request engine mints an incarnation, so the stub cannot +/// fabricate one: it writes `size` bytes at the blob's own key and hands back the head of what it +/// wrote, which is the incarnation the fold then condemns. +BlobHeadFn headPresent(CasOperation & op, const Layout & layout, uint64_t size) { - return [tok, size](const BlobRef &) -> std::optional + return [&op, &layout, size](const BlobRef & ref) -> std::optional { - HeadResult hr; - hr.exists = true; - hr.size = size; - hr.token = Token{.value = tok, .type = TokenType::Emulated}; - return hr; + const String key = layout.blobKey(ref); + op.create(key, String(size, 'x'), Retry::standard()); + return op.head(key, Retry::standard()); }; } @@ -185,7 +235,7 @@ CondemnedRow condemnedRowFor(uint64_t condemn_round, const String & tok = "t", bool delete_pending = false, uint64_t size = 1) { return CondemnedRow{.delete_pending = delete_pending, - .token = Token{.value = tok, .type = TokenType::Emulated}, + .token = PersistedIncarnation{"emulated", tok}, .size = size, .condemn_round = condemn_round}; } @@ -234,10 +284,10 @@ struct DecodedRun std::vector> edges; /// (blob_hash, source_id) }; -DecodedRun decodeRun(InMemoryBackend & backend, const RunRef & run) +DecodedRun decodeRun(CasOperation & op, const RunRef & run) { DecodedRun d; - auto r = openSourceEdgeRun(backend, run.key); + auto r = openSourceEdgeRun(op, run.key); /// Every run this test helper decodes is CityHash128 (16-byte), so `.toU128()` is a /// provably-exact round trip. String k; @@ -272,6 +322,7 @@ DecodedRun decodeRun(InMemoryBackend & backend, const RunRef & run) TEST(CASBlobInDegree, FoldSealChecksumMismatchFailsClosed) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; const RunRef good = writeSourceEdgeRun(backend, layout, /*gen*/1, /*attempt*/0, /*shard*/0, /*condemned*/{}, /*edges*/{{b(1), s(1)}}); @@ -282,7 +333,7 @@ TEST(CASBlobInDegree, FoldSealChecksumMismatchFailsClosed) /// A delta on a DIFFERENT blob forces the two-cursor merge to stream the prior run to completion, so /// the end-of-segment verifyAgainst fires (not a row-invariant abort). EXPECT_THROW( - foldDeltasIntoGeneration(backend, layout, prior, /*new*/2, /*attempt*/0, /*shard*/0, + foldDeltasIntoGeneration(*backend_req, layout, prior, /*new*/2, /*attempt*/0, /*shard*/0, std::vector{{bh(2), s(1), false}}, out), DB::Exception); } @@ -290,18 +341,20 @@ TEST(CASBlobInDegree, FoldSealChecksumMismatchFailsClosed) TEST(CASBlobInDegree, ZeroInDegreeSealChecksumMismatchFailsClosed) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; const RunRef good = writeSourceEdgeRun(backend, layout, /*gen*/1, /*attempt*/0, /*shard*/0, /*condemned*/{}, /*edges*/{{b(1), s(1)}}); RunRef bad = good; bad.checksum = good.checksum + 1; std::vector runs{bad}; - EXPECT_THROW(zeroInDegree(backend, runs), DB::Exception); + EXPECT_THROW(zeroInDegree(*backend_req, runs), DB::Exception); } TEST(CASThreeCursorMerge, FloorBoundary) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Gen 1's run holds one unrelated surviving edge (b9) plus the carried RunMarker::Condemned rows for A=b1 @@ -312,7 +365,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, /*current_round*/3, /*condemn_round*/4, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); /// Two-phase graduation: the floor-passed entry is REPUBLISHED pending (still in the list); @@ -330,7 +383,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) EXPECT_TRUE(rmr.redelete.empty()); /// still_retired mirrors exactly the RunMarker::Condemned rows written into the output run, in order. - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); ASSERT_EQ(out.condemned.size(), 2u); EXPECT_EQ(out.condemned[0].first, b(1)); EXPECT_TRUE(out.condemned[0].second.delete_pending); @@ -342,6 +395,7 @@ TEST(CASThreeCursorMerge, FloorBoundary) TEST(CASThreeCursorMerge, PendingRedeletesAndDrops) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// A row the PRIOR pass published as delete_pending (carried on gen 1's run): this pass hands it to @@ -351,7 +405,7 @@ TEST(CASThreeCursorMerge, PendingRedeletesAndDrops) std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, /*current_round*/9, /*condemn_round*/9, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); ASSERT_EQ(rmr.redelete.size(), 1u); @@ -361,7 +415,7 @@ TEST(CASThreeCursorMerge, PendingRedeletesAndDrops) EXPECT_TRUE(rmr.spared.empty()); /// The redeleted blob leaves the run entirely (no sentinel carried, no zero marker — untouched). - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); EXPECT_TRUE(out.condemned.empty()); EXPECT_TRUE(out.zero_markers.empty()); } @@ -369,6 +423,7 @@ TEST(CASThreeCursorMerge, PendingRedeletesAndDrops) TEST(CASThreeCursorMerge, RecoverySpares) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// A (=b1) is retired at round 1 and would long since have graduated (current_round = 5) — but this @@ -377,7 +432,7 @@ TEST(CASThreeCursorMerge, RecoverySpares) std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{gen1}, 2, 0, 0, {{bh(1), s(1), false}}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{gen1}, 2, 0, 0, {{bh(1), s(1), false}}, runs2, /*current_round*/5, /*condemn_round*/6, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); ASSERT_EQ(rmr.spared.size(), 1u); @@ -386,7 +441,7 @@ TEST(CASThreeCursorMerge, RecoverySpares) EXPECT_TRUE(rmr.still_retired.empty()); /// b1 recovered its edge: the output run carries the surviving edge and no sentinel for it. - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); EXPECT_TRUE(out.condemned.empty()); ASSERT_EQ(out.edges.size(), 1u); EXPECT_EQ(out.edges[0].first, b(1)); @@ -395,55 +450,59 @@ TEST(CASThreeCursorMerge, RecoverySpares) TEST(CASThreeCursorMerge, NewCandidateCondemned) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Gen 1: C (=b3) has one edge. Gen 2 removes it => transition to zero, not retired => /// condemned with the head-captured token at THIS pass's condemn_round. std::vector runs1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, 0, 0, {{bh(3), s(1), false}}, runs1); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, 0, 0, {{bh(3), s(1), false}}, runs1); std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, 2, 0, 0, {{bh(3), s(1), true}}, runs2, - /*current_round*/0, /*condemn_round*/7, headPresent("t9", 42), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, 2, 0, 0, {{bh(3), s(1), true}}, runs2, + /*current_round*/0, /*condemn_round*/7, headPresent(*backend_req, layout, 42), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); ASSERT_EQ(rmr.still_retired.size(), 1u); EXPECT_EQ(rmr.still_retired[0].ref, bh(3)); - EXPECT_EQ(rmr.still_retired[0].token.value, "t9"); + const std::optional present = (*backend_req).head(layout.blobKey(bh(3)), Retry::standard()); + ASSERT_TRUE(present.has_value()); + EXPECT_TRUE(rmr.still_retired[0].token.matches(present->incarnation)); EXPECT_EQ(rmr.still_retired[0].size, 42u); EXPECT_EQ(rmr.still_retired[0].condemn_round, 7u); EXPECT_TRUE(rmr.graduated.empty()); EXPECT_TRUE(rmr.spared.empty()); /// The fresh condemn is emitted as a RunMarker::Condemned row (not a zero marker) into the output run. - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); ASSERT_EQ(out.condemned.size(), 1u); EXPECT_EQ(out.condemned[0].first, b(3)); - EXPECT_EQ(out.condemned[0].second.token.value, "t9"); + EXPECT_TRUE(out.condemned[0].second.token.matches(present->incarnation)); EXPECT_TRUE(out.zero_markers.empty()); } TEST(CASThreeCursorMerge, AbsentBlobNotCondemned) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Same transition-to-zero as above, but the blob object is already gone at condemn time: /// nothing to delete later, so no entry is minted — a plain zero marker is emitted instead. std::vector runs1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, 0, 0, {{bh(3), s(1), false}}, runs1); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, 0, 0, {{bh(3), s(1), false}}, runs1); std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, 2, 0, 0, {{bh(3), s(1), true}}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, 2, 0, 0, {{bh(3), s(1), true}}, runs2, /*current_round*/0, /*condemn_round*/7, - [](const BlobRef &) -> std::optional { return std::nullopt; }, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); + [](const BlobRef &) -> std::optional { return std::nullopt; }, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); EXPECT_TRUE(rmr.still_retired.empty()); EXPECT_TRUE(rmr.graduated.empty()); EXPECT_TRUE(rmr.spared.empty()); - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); EXPECT_TRUE(out.condemned.empty()); ASSERT_EQ(out.zero_markers.size(), 1u); EXPECT_EQ(out.zero_markers[0], b(3)); @@ -456,11 +515,13 @@ TEST(CASThreeCursorMerge, SnapshotEdgesUnperturbedByRetired) /// The preserved invariant (spec §2.1) is narrower: the retired machinery touches ONLY the sentinel /// namespace — the surviving EDGE rows are byte-identical to a plain fold of the same deltas. InMemoryBackend plain; + DB::Cas::tests::OperationForTest plain_req(plain); InMemoryBackend engaged; + DB::Cas::tests::OperationForTest engaged_req(engaged); Layout layout{"pool"}; std::vector r1; - foldDeltasIntoGeneration(plain, layout, /*prior_runs*/{}, 1, 0, 0, + foldDeltasIntoGeneration(*plain_req, layout, /*prior_runs*/{}, 1, 0, 0, {{bh(1), s(1), false}, {bh(2), s(1), false}, {bh(2), s(2), true}}, r1); /// Engaged: the SAME deltas, but the prior run carries retired rows for b1 (which the delta re-edges @@ -469,12 +530,12 @@ TEST(CASThreeCursorMerge, SnapshotEdgesUnperturbedByRetired) {{b(1), condemnedRowFor(1)}, {b(5), condemnedRowFor(2)}}); std::vector r2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(engaged, layout, /*prior_runs*/{prior}, 2, 0, 0, + foldDeltasIntoGeneration(*engaged_req, layout, /*prior_runs*/{prior}, 2, 0, 0, {{bh(1), s(1), false}, {bh(2), s(1), false}, {bh(2), s(2), true}}, r2, - /*current_round*/9, /*condemn_round*/3, headPresent("t", 1), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); + /*current_round*/9, /*condemn_round*/3, headPresent(*engaged_req, layout, 1), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); - const DecodedRun plain_run = decodeRun(plain, r1[0]); - const DecodedRun engaged_run = decodeRun(engaged, r2[0]); + const DecodedRun plain_run = decodeRun(*plain_req, r1[0]); + const DecodedRun engaged_run = decodeRun(*engaged_req, r2[0]); EXPECT_EQ(plain_run.edges, engaged_run.edges); /// edge rows byte-identical EXPECT_TRUE(plain_run.condemned.empty()); /// The engaged run carries only the retired sentinel(s) on top: b1 spared (no row), b5 graduated. @@ -489,17 +550,18 @@ TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) /// has NO deltas at all: the carried row must (a) survive byte-identically, (b) emit no zero marker, /// (c) never call peek_head (a carried sentinel is not a touch). InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Gen 1: (b,s1) added then removed => net-to-zero => fresh condemn at round 5 (token "tok", size 7). std::vector runs1; RetiredMergeResult rmr1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, 0, 0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, 0, 0, {{bh(2), s(1), false}, {bh(2), s(1), true}}, runs1, - /*current_round*/0, /*condemn_round*/5, headPresent("tok", 7), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr1); + /*current_round*/0, /*condemn_round*/5, headPresent(*backend_req, layout, 7), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr1); ASSERT_EQ(rmr1.still_retired.size(), 1u); { - const DecodedRun g1 = decodeRun(backend, runs1[0]); + const DecodedRun g1 = decodeRun(*backend_req, runs1[0]); ASSERT_EQ(g1.condemned.size(), 1u); EXPECT_EQ(g1.condemned[0].first, b(2)); EXPECT_TRUE(g1.zero_markers.empty()); /// a condemned blob emits RunMarker::Condemned, never a zero marker @@ -507,10 +569,10 @@ TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) /// Gen 2: empty deltas, current_round 1 (< 5 => b carries, does not graduate). peek_head must NOT fire. size_t peek_calls = 0; - auto peek = [&](const BlobRef &) -> std::optional { ++peek_calls; return {}; }; + auto peek = [&](const BlobRef &) -> std::optional { ++peek_calls; return {}; }; std::vector runs2; RetiredMergeResult rmr2; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, 2, 0, 0, {}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, 2, 0, 0, {}, runs2, /*current_round*/1, /*condemn_round*/6, /*head_blob*/{}, peek, /*confirm_condemned_marker*/{}, &rmr2); EXPECT_EQ(peek_calls, 0u); @@ -519,10 +581,12 @@ TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) EXPECT_EQ(rmr2.still_retired[0].condemn_round, 5u); /// carried unchanged EXPECT_TRUE(rmr2.graduated.empty()); - const DecodedRun g2 = decodeRun(backend, runs2[0]); + const DecodedRun g2 = decodeRun(*backend_req, runs2[0]); ASSERT_EQ(g2.condemned.size(), 1u); EXPECT_EQ(g2.condemned[0].first, b(2)); - EXPECT_EQ(g2.condemned[0].second.token.value, "tok"); + const std::optional present = (*backend_req).head(layout.blobKey(bh(2)), Retry::standard()); + ASSERT_TRUE(present.has_value()); + EXPECT_TRUE(g2.condemned[0].second.token.matches(present->incarnation)); EXPECT_EQ(g2.condemned[0].second.size, 7u); EXPECT_TRUE(g2.zero_markers.empty()); } @@ -534,6 +598,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) /// (1) An active edge at the reserved sentinel source_id 0 -> the merge cursor fails closed. { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); writer.append(edgeRec(1, UInt128{0})); // edge at sentinel key @@ -545,13 +610,14 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) backend.putIfAbsent(bad.key, bytes); std::vector runs2; - EXPECT_THROW(foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{bad}, 2, 0, 0, {}, runs2), + EXPECT_THROW(foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{bad}, 2, 0, 0, {}, runs2), DB::Exception); } /// (2) Two sentinel rows for one blob -> duplicate sentinel -> the merge cursor fails closed. { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); /// Same (b,0) key twice (equal keys are allowed by the writer) — two condemned sentinels for b1. @@ -565,22 +631,26 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) backend.putIfAbsent(bad.key, bytes); std::vector runs2; - EXPECT_THROW(foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{bad}, 2, 0, 0, {}, runs2), + EXPECT_THROW(foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{bad}, 2, 0, 0, {}, runs2), DB::Exception); } } -/// A prior run spanning several blocks folds correctly with the streaming prior cursor AND the backend -/// sees only block-bounded ranged/stream requests for it — never a whole-object get of the prior run -/// key. Byte-reproducibility of the merged output is the load-bearing canary (the merge logic is -/// unchanged; only the prior cursor's byte source moved from materialize-whole to stream). -TEST(CASBlobInDegree, FoldStreamsPriorRunBlockBounded) +/// A prior run several times larger than one buffer folds correctly with the streaming prior cursor +/// AND the backend is never asked to READ that run's key — the cursor reaches it only through the +/// streaming open. Byte-reproducibility of the merged output is the load-bearing canary (the merge +/// logic is unchanged; only the prior cursor's byte source moved from materialize-whole to stream). +/// What this does NOT check is how much the open stream buffers: the streaming primitive carries no +/// window, so the seam has nothing to measure. +TEST(CASBlobInDegree, FoldStreamsPriorRunWithoutReadingItWhole) { using DB::Cas::tests::CountingBackend; CountingBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); /// InMemory oracle: the SAME two folds against a plain backend must yield byte-identical runs — /// the streaming cursor changes I/O shape, not bytes. InMemoryBackend oracle; + DB::Cas::tests::OperationForTest oracle_req(oracle); Layout layout{"pool"}; /// Gen 1 from empty prior: enough edges that the SourceEdge run spills across many 256KB blocks. @@ -593,26 +663,28 @@ TEST(CASBlobInDegree, FoldStreamsPriorRunBlockBounded) std::vector runs1_c; std::vector runs1_o; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_c); - foldDeltasIntoGeneration(oracle, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_o); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_c); + foldDeltasIntoGeneration(*oracle_req, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_o); const String gen1_run_key = layout.blobTargetRunKey(1, 0, 0, 0); const auto gen1_run = backend.get(gen1_run_key); ASSERT_TRUE(gen1_run.has_value()); const String gen1_run_bytes = gen1_run->bytes; - /// Sanity: the prior run really spans several blocks (else the block-bounded assertions are - /// vacuous). Blocks seal at kLegacyBlockSize (256KB); ~820KB is 3-4 blocks. + /// Sanity: the prior run is far larger than one read buffer, so "it was never read whole" is a + /// claim about a genuinely large object rather than one a single buffer could have swallowed. ASSERT_GT(gen1_run_bytes.size(), static_cast(kLegacyBlockSize) * 3); /// Reset counters and fold gen 2 with a small delta: remove one edge and add another. The prior - /// gen-1 run must be consumed via the streaming cursor (head + tail get + body getStream + per-seq - /// head probe), NEVER a whole-object get. + /// gen-1 run must be consumed via one streaming open per prior segment, NEVER a whole-object get. backend.resetCounts(); + /// Arm a window smaller than the run so the fold's output is proven correct under chunked delivery, + /// not merely served in one piece: unarmed, `getStream` never installs the recording wrapper at all. + backend.setStreamChunkForTest(kLegacyBlockSize / 4); std::vector gen2{{bh(0), s(1), true}, {bh(19999), s(2), false}}; std::vector runs2_c; std::vector runs2_o; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1_c, 2, 0, 0, gen2, runs2_c); - foldDeltasIntoGeneration(oracle, layout, /*prior_runs*/runs1_o, 2, 0, 0, gen2, runs2_o); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1_c, 2, 0, 0, gen2, runs2_c); + foldDeltasIntoGeneration(*oracle_req, layout, /*prior_runs*/runs1_o, 2, 0, 0, gen2, runs2_o); /// Byte-reproducibility canary: streaming and materialized folds produce identical output bytes. const String gen2_run_key = layout.blobTargetRunKey(2, 0, 0, 0); @@ -625,31 +697,31 @@ TEST(CASBlobInDegree, FoldStreamsPriorRunBlockBounded) ASSERT_EQ(runs2_o.size(), 1u); EXPECT_EQ(runs2_c[0].checksum, runs2_o[0].checksum); - /// The core assertion: no whole-object get of the prior run key — every read carried a Range or a - /// stream (the resident-memory proof at the seam). - EXPECT_EQ(backend.wholeGetCount(gen1_run_key), 0u); - /// The cursor opened the prior run's segment via the streaming reader (head + tail get + getStream). + /// The core assertion: the prior run is never read whole — the cursor reaches it only through the + /// streaming open, so the seam sees no read of that key at all. + EXPECT_EQ(backend.getCount(gen1_run_key), 0u); + /// The cursor opened the prior run's segment through the streaming reader. EXPECT_GE(backend.getStreamCount(gen1_run_key), 1u); - /// Every ranged-get window on the prior run stays within one block + the footer allowance. This - /// bound is strict here because the prior run's footer fits inside the fixed tail probe (only very - /// large runs — ~13k blocks — spill the footer past the probe and add one exact-footer get; a note - /// for that regime lives in the streaming reader's open comment). - EXPECT_LE(backend.maxRangedGetLen(gen1_run_key), - static_cast(kLegacyHardCapBlockSize) + 64u * 1024u); - /// Streaming open touches the prior run's tail probe (and at most one exact-footer get); it is never - /// re-materialized whole. - EXPECT_LE(backend.getCount(gen1_run_key), 2u); + /// The other half of the evidence: a nonzero value here can only come from the recording wrapper + /// `getStream` installs when armed, so this proves the arming actually took effect and the + /// byte-parity check above ran under genuinely chunked delivery, not a no-op setter. + EXPECT_GT(backend.largestStreamChunk(gen1_run_key), 0u); + /// This does NOT bound how much the stream buffers per request: the streaming primitive carries no + /// window for the seam to measure, so resident memory inside the open stream is out of its reach. } -/// The preview consumer `zeroInDegree` streams a multi-block run instead of materializing it whole: the -/// backend sees only block-bounded ranged/stream requests for the run key (never a whole-object get), and -/// the candidate set equals the pre-change (borrowed-mode) result. Byte-parity against an InMemory oracle -/// is the load-bearing canary — the scan logic is unchanged; only the byte source moved to the stream. -TEST(CASBlobInDegree, ZeroInDegreeStreamsBlockBounded) +/// The preview consumer `zeroInDegree` streams a large run instead of materializing it whole: the +/// backend is never asked to READ the run's key, only to open it as a stream, and the candidate set +/// equals the pre-change (borrowed-mode) result. Byte-parity against an InMemory oracle is the +/// load-bearing canary — the scan logic is unchanged; only the byte source moved to the stream. +/// As above, the buffering inside the open stream is not bounded here; nothing at the seam sees it. +TEST(CASBlobInDegree, ZeroInDegreeStreamsRunWithoutReadingItWhole) { using DB::Cas::tests::CountingBackend; CountingBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); InMemoryBackend oracle; + DB::Cas::tests::OperationForTest oracle_req(oracle); Layout layout{"pool"}; /// Gen 1 from empty prior: ~20000 active edges spill the SourceEdge run across several 256KB blocks. @@ -660,26 +732,30 @@ TEST(CASBlobInDegree, ZeroInDegreeStreamsBlockBounded) std::vector runs1_c; std::vector runs1_o; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_c); - foldDeltasIntoGeneration(oracle, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_o); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_c); + foldDeltasIntoGeneration(*oracle_req, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_o); /// Gen 2 removes every edge on two of the blobs => two zero-transition markers in the gen-2 run, /// which is itself multi-block (the surviving-edge rows still span blocks). std::vector gen2{{bh(0), s(1), true}, {bh(19999), s(1), true}}; std::vector runs2_c; std::vector runs2_o; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1_c, 2, 0, 0, gen2, runs2_c); - foldDeltasIntoGeneration(oracle, layout, /*prior_runs*/runs1_o, 2, 0, 0, gen2, runs2_o); + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1_c, 2, 0, 0, gen2, runs2_c); + foldDeltasIntoGeneration(*oracle_req, layout, /*prior_runs*/runs1_o, 2, 0, 0, gen2, runs2_o); const String gen2_run_key = layout.blobTargetRunKey(2, 0, 0, 0); const auto gen2_run = backend.get(gen2_run_key); ASSERT_TRUE(gen2_run.has_value()); - /// Sanity: the run genuinely spans several blocks (else the block-bounded assertions are vacuous). + /// Sanity: the run is far larger than one read buffer, for the same reason as above. ASSERT_GT(gen2_run->bytes.size(), static_cast(kLegacyBlockSize) * 3); backend.resetCounts(); - const auto zero_c = zeroInDegree(backend, runs2_c); - const auto zero_o = zeroInDegree(oracle, runs2_o); + /// Arm a window smaller than the run so the candidate set below is proven correct under chunked + /// delivery, not merely served in one piece: unarmed, `getStream` never installs the recording + /// wrapper at all. + backend.setStreamChunkForTest(kLegacyBlockSize / 4); + const auto zero_c = zeroInDegree(*backend_req, runs2_c); + const auto zero_o = zeroInDegree(*oracle_req, runs2_o); /// Equivalence with the borrowed-mode (InMemory oracle) result: same candidates, in the same order. ASSERT_EQ(zero_c.size(), zero_o.size()); @@ -687,27 +763,31 @@ TEST(CASBlobInDegree, ZeroInDegreeStreamsBlockBounded) for (size_t i = 0; i < zero_c.size(); ++i) EXPECT_EQ(zero_c[i].ref, zero_o[i].ref); - /// The core assertion: no whole-object get of the run key — every read carried a Range or a stream. - EXPECT_EQ(backend.wholeGetCount(gen2_run_key), 0u); - /// The scan opened the run via the streaming reader (head + tail get + getStream). + /// The core assertion: the run is never read whole — the scan reaches it only through the streaming + /// open, so the seam sees no read of that key at all. + EXPECT_EQ(backend.getCount(gen2_run_key), 0u); + /// The scan opened the run through the streaming reader. EXPECT_GE(backend.getStreamCount(gen2_run_key), 1u); - /// Every ranged-get window stays within one block + the footer allowance (the seam memory bound). - EXPECT_LE(backend.maxRangedGetLen(gen2_run_key), - static_cast(kLegacyHardCapBlockSize) + 64u * 1024u); - /// Streaming open touches the tail probe (and at most one exact-footer get); never re-materialized whole. - EXPECT_LE(backend.getCount(gen2_run_key), 2u); + /// The other half of the evidence: a nonzero value here can only come from the recording wrapper + /// `getStream` installs when armed, so this proves the arming actually took effect and the + /// candidate-set check above ran under genuinely chunked delivery, not a no-op setter. + EXPECT_GT(backend.largestStreamChunk(gen2_run_key), 0u); + /// This does NOT bound how much the stream buffers per request: the streaming primitive carries no + /// window for the seam to measure, so resident memory inside the open stream is out of its reach. } /// ==== RunMarker::Condemned row codec + typed source-edge open (retired-in-snapshot T2, spec §2.1) ==== TEST(CASCondemnedRow, RoundTripAllTokenTypes) { - for (auto type : {DB::Cas::TokenType::ETag, DB::Cas::TokenType::Generation, DB::Cas::TokenType::Emulated}) + /// Walked over the vocabulary's own entries rather than a hand-copied list, so a dialect the + /// encoder can construct but this test forgot cannot exist. + for (const auto & entry : DB::Cas::kTokenTypeWords.entries) { DB::Cas::CondemnedRow row; - row.delete_pending = (type == DB::Cas::TokenType::Generation); - row.marker_confirmed = (type == DB::Cas::TokenType::Emulated); - row.token = DB::Cas::Token{.value = "etag-abc-123", .type = type}; + row.delete_pending = (entry.value == DB::Cas::TokenType::Generation); + row.marker_confirmed = (entry.value == DB::Cas::TokenType::Emulated); + row.token = DB::Cas::PersistedIncarnation{String(entry.word), "etag-abc-123"}; row.size = 4096; row.condemn_round = 7; const auto bytes = DB::Cas::encodeCondemnedRow(row); @@ -720,7 +800,7 @@ TEST(CASCondemnedRow, UnknownMarkerByteFailsClosedWithCorruptedData) { /// This pins the condemned-row decoder's own marker validation. DB::Cas::CondemnedRow row; - row.token = DB::Cas::Token{.value = "t", .type = DB::Cas::TokenType::ETag}; + row.token = DB::Cas::PersistedIncarnation{"etag", "t"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes[0] = 0x03; @@ -755,7 +835,7 @@ TEST(CASRecordStream, RunMarkerByteContractFailsClosed) TEST(CASCondemnedRow, UnknownFlagBitsFailClosed) { DB::Cas::CondemnedRow row; - row.token = DB::Cas::Token{.value = "t", .type = DB::Cas::TokenType::ETag}; + row.token = DB::Cas::PersistedIncarnation{"etag", "t"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes[1] = 4; // flags byte: only bits 0 (delete_pending) and 1 (marker_confirmed) are defined EXPECT_THROW(DB::Cas::decodeCondemnedRow(bytes), DB::Exception); @@ -764,7 +844,7 @@ TEST(CASCondemnedRow, UnknownFlagBitsFailClosed) TEST(CASCondemnedRow, UnknownTokenTypeFailsClosed) { DB::Cas::CondemnedRow row; - row.token = DB::Cas::Token{.value = "t", .type = DB::Cas::TokenType::ETag}; + row.token = DB::Cas::PersistedIncarnation{"etag", "t"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes[2] = 99; // token_type byte (offset: [0]=0x02 [1]=flags [2]=token_type) EXPECT_THROW(DB::Cas::decodeCondemnedRow(bytes), DB::Exception); @@ -773,7 +853,7 @@ TEST(CASCondemnedRow, UnknownTokenTypeFailsClosed) TEST(CASCondemnedRow, TruncatedPayloadFailsClosed) { DB::Cas::CondemnedRow row; - row.token = DB::Cas::Token{.value = "0123456789", .type = DB::Cas::TokenType::ETag}; + row.token = DB::Cas::PersistedIncarnation{"etag", "0123456789"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes.resize(bytes.size() - 3); // token bytes shorter than declared token_len EXPECT_THROW(DB::Cas::decodeCondemnedRow(bytes), DB::Exception); @@ -832,6 +912,7 @@ TEST(CASBlobInDegree, TwoAlgoFoldSettlesBothInOneShardRun) /// both settle (edges present, condemn on removal works per ref), mixed rows in one run, no /// algo loop. InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; const BlobRef ch_x{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(11))}; @@ -840,20 +921,20 @@ TEST(CASBlobInDegree, TwoAlgoFoldSettlesBothInOneShardRun) const BlobRef sha_y_ref{BlobHashAlgo::Sha256, sha_y}; std::vector runs1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, {{ch_x, s(1), false}, {sha_y_ref, s(1), false}}, runs1); ASSERT_FALSE(runs1.empty()); EXPECT_EQ(DB::Cas::tests::inDegreeInRuns(backend, runs1, ch_x), 1); EXPECT_EQ(DB::Cas::tests::inDegreeInRuns(backend, runs1, sha_y_ref), 1); - EXPECT_TRUE(zeroInDegree(backend, runs1).empty()); + EXPECT_TRUE(zeroInDegree(*backend_req, runs1).empty()); /// Remove both edges in gen 2: each transitions to zero independently, condemned per its own ref. std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, 2, /*attempt*/0, 0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, 2, /*attempt*/0, 0, {{ch_x, s(1), true}, {sha_y_ref, s(1), true}}, runs2, - /*current_round*/0, /*condemn_round*/1, headPresent("t", 1), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); + /*current_round*/0, /*condemn_round*/1, headPresent(*backend_req, layout, 1), /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); ASSERT_EQ(rmr.still_retired.size(), 2u); std::vector condemned_refs{rmr.still_retired[0].ref, rmr.still_retired[1].ref}; @@ -873,23 +954,24 @@ TEST(CASBlobInDegree, TwoAlgoFoldSettlesBothInOneShardRun) TEST(CASBlobInDegree, UnmatchedRemovalIsAPerKeyNoOpAndSparesSiblingEdges) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Generation 1: blob b1 is referenced by TWO distinct sources (two manifests). std::vector runs1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, /*shard*/0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, /*shard*/0, {{bh(1), s(1), false}, {bh(1), s(2), false}}, runs1); /// Generation 2: fold a removal for a THIRD source that never had an activation folded. std::vector runs2; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, /*new_generation*/2, /*attempt*/0, /*shard*/0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, /*new_generation*/2, /*attempt*/0, /*shard*/0, {{bh(1), s(99), true}}, runs2); /// Both original edges survive: the unmatched removal touched only its own (absent) key. - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); ASSERT_EQ(out.edges.size(), 2u) << "an unmatched removal must not strip sibling edges"; /// And the blob is NOT a deletion candidate. - const auto zero = zeroInDegree(backend, runs2); + const auto zero = zeroInDegree(*backend_req, runs2); EXPECT_TRUE(zero.empty()) << "b1 still has two live source edges"; } @@ -901,23 +983,24 @@ TEST(CASBlobInDegree, UnmatchedRemovalIsAPerKeyNoOpAndSparesSiblingEdges) TEST(CASBlobInDegree, UnmatchedRemovalIsCountedWithAnExample) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Generation 1: blob b1 is referenced by TWO distinct sources (two manifests), same fixture as the /// no-op test above. std::vector runs1; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, /*shard*/0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, /*shard*/0, {{bh(1), s(1), false}, {bh(1), s(2), false}}, runs1); /// Generation 2: fold a removal for a THIRD source that never had an activation folded. std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/runs1, /*new_generation*/2, /*attempt*/0, /*shard*/0, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/runs1, /*new_generation*/2, /*attempt*/0, /*shard*/0, {{bh(1), s(99), true}}, runs2, /*current_round*/0, /*condemn_round*/0, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr); /// The run is byte-identical to the no-op test's outcome for the blob's OTHER edges: both survive. - const DecodedRun out = decodeRun(backend, runs2[0]); + const DecodedRun out = decodeRun(*backend_req, runs2[0]); ASSERT_EQ(out.edges.size(), 2u) << "the counting surface must not perturb the no-op fold outcome"; EXPECT_EQ(out.edges[0].first, b(1)); EXPECT_EQ(out.edges[1].first, b(1)); @@ -951,6 +1034,7 @@ std::vector> condemnedCohort(uint64_t n, uint64 TEST(CASThreeCursorMerge, RedeleteBudgetCapsCohortAndCarriesExcess) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; const RunRef gen1 = writeSourceEdgeRun(backend, layout, 1, 0, 0, condemnedCohort(10, 1, /*delete_pending*/true)); @@ -959,7 +1043,7 @@ TEST(CASThreeCursorMerge, RedeleteBudgetCapsCohortAndCarriesExcess) std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, /*current_round*/9, /*condemn_round*/9, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr, /*suppress_destructive*/false, /*out_applied_by_txn_ordinal*/nullptr, /*source_retirements*/{}, &budget); @@ -979,6 +1063,7 @@ TEST(CASThreeCursorMerge, RedeleteBudgetCapsCohortAndCarriesExcess) TEST(CASThreeCursorMerge, GraduationBudgetCapsCohortAndCarriesExcess) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; const RunRef gen1 = writeSourceEdgeRun(backend, layout, 1, 0, 0, condemnedCohort(10, /*condemn_round*/1, /*delete_pending*/false)); @@ -987,7 +1072,7 @@ TEST(CASThreeCursorMerge, GraduationBudgetCapsCohortAndCarriesExcess) std::vector runs2; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, + foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{gen1}, 2, 0, 0, {}, runs2, /*current_round*/5, /*condemn_round*/6, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr, /*suppress_destructive*/false, /*out_applied_by_txn_ordinal*/nullptr, /*source_retirements*/{}, &budget); @@ -1009,6 +1094,7 @@ TEST(CASThreeCursorMerge, GraduationBudgetCapsCohortAndCarriesExcess) TEST(CASThreeCursorMerge, RedeleteBudgetDrainsCohortToFixpointOverRounds) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; std::vector priors{writeSourceEdgeRun(backend, layout, 1, 0, 0, condemnedCohort(10, 1, /*delete_pending*/true))}; @@ -1020,7 +1106,7 @@ TEST(CASThreeCursorMerge, RedeleteBudgetDrainsCohortToFixpointOverRounds) budget.max_redeletes = 3; std::vector out_runs; RetiredMergeResult rmr; - foldDeltasIntoGeneration(backend, layout, priors, 2 + rounds, 0, 0, {}, out_runs, + foldDeltasIntoGeneration(*backend_req, layout, priors, 2 + rounds, 0, 0, {}, out_runs, /*current_round*/100, /*condemn_round*/100, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &rmr, /*suppress_destructive*/false, /*out_applied_by_txn_ordinal*/nullptr, /*source_retirements*/{}, &budget); diff --git a/src/Disks/tests/gtest_cas_blob_meta.cpp b/src/Disks/tests/gtest_cas_blob_meta.cpp index a0a29bf2e8f4..f38769fb6824 100644 --- a/src/Disks/tests/gtest_cas_blob_meta.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta.cpp @@ -23,49 +23,50 @@ TEST(CASBlobMeta, PutIfAbsentThenCasTransitions) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasOperation op = store->mountRequests().admit(); const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of("hash-a"))}; const BlobMeta clean{.state = MetaState::Clean, .size = 10}; - const CasOverwriteResult created = putMetaIfAbsent(*store, ref, clean); - EXPECT_EQ(created.outcome, CasOverwriteOutcome::Committed); + EXPECT_TRUE(std::holds_alternative(putMetaIfAbsent(op, store->layout(), ref, clean))); - const CasOverwriteResult dup = putMetaIfAbsent(*store, ref, clean); - EXPECT_EQ(dup.outcome, CasOverwriteOutcome::Committed); /// exact-byte resolution adopts the existing marker + /// A create that nothing of its own left unresolved never adopts what is already at the key, even + /// byte-identical: the marker was somebody else's write, and the conflict carries it. + EXPECT_TRUE(std::holds_alternative(putMetaIfAbsent(op, store->layout(), ref, clean))); - const auto lm = loadMeta(*backend, store->layout(), ref); + const auto lm = loadMeta(op, store->layout(), ref); ASSERT_TRUE(lm.has_value()); EXPECT_EQ(lm->meta.state, MetaState::Clean); - const CasOverwriteResult condemned = casMeta(*store, ref, lm->etag, - BlobMeta{.state = MetaState::Condemned, .condemn_round = 5, .size = 10}); - EXPECT_EQ(condemned.outcome, CasOverwriteOutcome::Committed); + EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, lm->incarnation, + BlobMeta{.state = MetaState::Condemned, .condemn_round = 5, .size = 10}))); - const CasOverwriteResult stale = casMeta(*store, ref, lm->etag, /// stale token loses - BlobMeta{.state = MetaState::Clean}); - EXPECT_EQ(stale.outcome, CasOverwriteOutcome::Conflict); + /// the stale incarnation loses + EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, lm->incarnation, + BlobMeta{.state = MetaState::Clean}))); } -TEST(CASBlobMeta, DeleteMetaExactMatchesEtag) +TEST(CASBlobMeta, DeleteMetaExactMatchesTheObservedIncarnation) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasOperation op = store->mountRequests().admit(); const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of("hash-b"))}; - putMetaIfAbsent(*store, ref, BlobMeta{.state = MetaState::Condemned}); - const auto lm = loadMeta(*backend, store->layout(), ref); + putMetaIfAbsent(op, store->layout(), ref, BlobMeta{.state = MetaState::Condemned}); + const auto lm = loadMeta(op, store->layout(), ref); ASSERT_TRUE(lm.has_value()); - EXPECT_EQ(deleteMetaExact(*backend, store->layout(), ref, lm->etag).kind, DeleteOutcome::Kind::Deleted); - EXPECT_FALSE(loadMeta(*backend, store->layout(), ref).has_value()); + EXPECT_EQ(deleteMetaExact(op, store->layout(), ref, lm->incarnation), Removal::Removed); + EXPECT_FALSE(loadMeta(op, store->layout(), ref).has_value()); } /// Phase 3 T3 (mixed-algo pools, was CAS pluggable-blob-hash Phase 2 Task 5 crux Test 2): the `.meta` /// API round-trips a 32-byte (`sha256`-width) `BlobRef` key — the meta object lands under a 64-hex /// key, exercising the SAME `putMetaIfAbsent`/`loadMeta`/`casMeta`/`deleteMetaExact` surface PartWriteTxn/Gc -/// use, just at a wider algo. Writes use the `Pool`'s controller; reads and exact deletion retain their -/// direct `Backend`/`Layout` surface. +/// use, just at a wider algo. TEST(CASBlobMeta, PutLoadCasDeleteRoundTripAtWidth32) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasOperation op = store->mountRequests().admit(); const Layout & layout = store->layout(); /// A distinguishable 32-byte digest (not merely a 16-byte value zero-tailed): every byte set. @@ -76,31 +77,31 @@ TEST(CASBlobMeta, PutLoadCasDeleteRoundTripAtWidth32) const String hex = codecFor(BlobHashAlgo::Sha256).toHex(h); EXPECT_EQ(hex.size(), 64u) << "a 32-byte digest renders 64 hex chars"; - const CasOverwriteResult created = putMetaIfAbsent(*store, ref, - BlobMeta{.state = MetaState::Clean, .size = 555}); - ASSERT_EQ(created.outcome, CasOverwriteOutcome::Committed); - EXPECT_TRUE(backend->head(layout.blobMetaKey(ref)).exists) + ASSERT_TRUE(std::holds_alternative( + putMetaIfAbsent(op, store->layout(), ref, BlobMeta{.state = MetaState::Clean, .size = 555}))); + EXPECT_TRUE(op.head(layout.blobMetaKey(ref), Retry::standard()).has_value()) << "the meta object must land under the 64-hex key, not a truncated 32-hex one"; - const auto lm = loadMeta(*backend, layout, ref); + const auto lm = loadMeta(op, layout, ref); ASSERT_TRUE(lm.has_value()); EXPECT_EQ(lm->meta.state, MetaState::Clean); EXPECT_EQ(lm->meta.size, 555u); - const CasOverwriteResult condemned = casMeta(*store, ref, lm->etag, - BlobMeta{.state = MetaState::Condemned, .condemn_round = 7, .size = 555}); - ASSERT_EQ(condemned.outcome, CasOverwriteOutcome::Committed); - const auto lm2 = loadMeta(*backend, layout, ref); + ASSERT_TRUE(std::holds_alternative(casMeta(op, layout, ref, lm->incarnation, + BlobMeta{.state = MetaState::Condemned, .condemn_round = 7, .size = 555}))); + const auto lm2 = loadMeta(op, layout, ref); ASSERT_TRUE(lm2.has_value()); EXPECT_EQ(lm2->meta.state, MetaState::Condemned); - EXPECT_EQ(deleteMetaExact(*backend, layout, ref, lm2->etag).kind, DeleteOutcome::Kind::Deleted); - EXPECT_FALSE(loadMeta(*backend, layout, ref).has_value()); + EXPECT_EQ(deleteMetaExact(op, layout, ref, lm2->incarnation), Removal::Removed); + EXPECT_FALSE(loadMeta(op, layout, ref).has_value()); } namespace { +/// The fault sits on the transport primitive, which is what every marker write reaches the store +/// through: a create is a `write` with no precondition, a compare-swap a `write` with one. class ControlledMetaWriteFaultBackend : public InMemoryBackend { public: @@ -109,52 +110,57 @@ class ControlledMetaWriteFaultBackend : public InMemoryBackend uint64_t create_attempts = 0; uint64_t overwrite_attempts = 0; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - ++create_attempts; - if (throw_next_create) + if (expected_value) { - throw_next_create = false; - throw Poco::TimeoutException("scripted meta create ambiguity"); + ++overwrite_attempts; + if (throw_next_overwrite) + { + throw_next_overwrite = false; + throw Poco::TimeoutException("scripted meta overwrite ambiguity"); + } } - return InMemoryBackend::putIfAbsent(key, bytes, meta); - } - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override - { - ++overwrite_attempts; - if (throw_next_overwrite) + else { - throw_next_overwrite = false; - throw Poco::TimeoutException("scripted meta overwrite ambiguity"); + ++create_attempts; + if (throw_next_create) + { + throw_next_create = false; + throw Poco::TimeoutException("scripted meta create ambiguity"); + } } - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } }; } -TEST(CASBlobMeta, WritesUsePoolRequestController) +/// An ambiguous marker write is settled by the engine's exact read and reissued, rather than escaping +/// as a raw transport error: the create's resolve proves the key still absent, the compare-swap's +/// proves the expected incarnation still current, and both are repeatable. +TEST(CASBlobMeta, AnAmbiguousMarkerWriteIsResolvedAndReissued) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); store->setCasRetrySleepForTest([](uint64_t) {}); + CasOperation op = store->mountRequests().admit(); backend->create_attempts = 0; backend->overwrite_attempts = 0; const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of("hash-controlled"))}; backend->throw_next_create = true; - EXPECT_EQ( - putMetaIfAbsent(*store, ref, BlobMeta{.state = MetaState::Clean, .size = 10}).outcome, - CasOverwriteOutcome::Committed); + EXPECT_TRUE(std::holds_alternative( + putMetaIfAbsent(op, store->layout(), ref, BlobMeta{.state = MetaState::Clean, .size = 10}))); EXPECT_EQ(backend->create_attempts, 2u); - const auto clean = loadMeta(*backend, store->layout(), ref); + const auto clean = loadMeta(op, store->layout(), ref); ASSERT_TRUE(clean.has_value()); backend->throw_next_overwrite = true; - EXPECT_EQ( - casMeta(*store, ref, clean->etag, BlobMeta{.state = MetaState::Condemned, .condemn_round = 1, .size = 10}).outcome, - CasOverwriteOutcome::Committed); + EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, clean->incarnation, + BlobMeta{.state = MetaState::Condemned, .condemn_round = 1, .size = 10}))); EXPECT_EQ(backend->overwrite_attempts, 2u); } diff --git a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp index 5e2b94a4da5f..d67802db6bcd 100644 --- a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp +++ b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp @@ -41,55 +41,48 @@ const String kProbeUid2 = "fedcba9876543210fedcba9876543210"; /// Records the ORDER of backend operations so a test can assert that the residual LIST precedes the first /// write, and that a fail path performs zero writes. Delegates every operation to `InMemoryBackend` /// unchanged; `Pool::open` wraps this in its `InstrumentedBackend`, which forwards every op here. +/// +/// The `write` primitive covers create, replace and conditional-put alike, so the log distinguishes +/// only writes from removals -- which is all the ordering assertions ask. class RecordingBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; - using Backend::get; - using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - - enum class Op : uint8_t { List, PutIfAbsent, PutOverwrite, CasPut, Delete }; + /// Unhide the legacy `list` overloads the primitive override below would otherwise hide: the tests + /// seed and inspect this store through them. + using Backend::list; + + enum class Op : uint8_t { List, Write, Remove }; struct Entry { Op op; String key; /// the LIST prefix, or the written key }; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + /// Recorded at the PRIMITIVE, which every legacy forwarder reaches too, so an op is logged + /// whichever surface issued it. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { record(Op::List, prefix); - return InMemoryBackend::list(prefix, cursor, limit); - } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - record(Op::PutIfAbsent, key); - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::list(prefix, cursor, limit, access); } - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - record(Op::PutOverwrite, key); - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + record(Op::Write, key); + return InMemoryBackend::write(key, bytes, expected_value, access); } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) override + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { - record(Op::CasPut, key); - return InMemoryBackend::casPut(key, bytes, expected, meta); + record(Op::Remove, key); + return InMemoryBackend::remove(key, expected_value, access); } - DeleteOutcome deleteExact(const String & key, const Token & token) override - { - record(Op::Delete, key); - return InMemoryBackend::deleteExact(key, token); - } - /// The bootstrap path (battery + createOrValidate + mount protocol) issues only whole-String writes, - /// never a streaming create, so recording the four write ops above captures every write `open` can do. + /// `publish` is the one mutating primitive left unrecorded: it writes a blob, and the bootstrap + /// path (battery + `createOrValidate` + mount protocol) publishes none. static bool isWrite(Op op) { - return op == Op::PutIfAbsent || op == Op::PutOverwrite || op == Op::CasPut || op == Op::Delete; + return op == Op::Write || op == Op::Remove; } void clearLog() @@ -127,13 +120,11 @@ class RecordingBackend final : public InMemoryBackend class CatalogMissingAfterListBackend final : public InMemoryBackend { public: - using Backend::get; - - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (key == Layout{kPrefix}.refCatalogKey()) return std::nullopt; - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } }; diff --git a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp index f129949c0590..b2c1542c6017 100644 --- a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp +++ b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp @@ -18,6 +18,9 @@ #include #include +#include + +#include #include #include #include @@ -27,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -80,25 +84,100 @@ namespace /// `recovery_in_progress` set). The failure is deliberately `CORRUPTED_DATA`: /// `isTransientRecoveryError` does not list it, so recovery fails fast instead of burning its retry /// budget. +/// The engine reissues an unresolved write until its OWN retry window closes, and that window is +/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine +/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline +/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling +/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the +/// closures own the clock, so it outlives everything that can still read it. +class VirtualRetryClock +{ +public: + static std::shared_ptr installOn(const PoolPtr & store) + { + auto clock = std::make_shared(); + store->setCasRequestNowFnForTest([clock] { return clock->nowMs(); }); + store->setCasRetrySleepForTest([clock](uint64_t ms) { clock->advance(ms); }); + return clock; + } + + uint64_t nowMs() const + { + std::lock_guard lock(mutex); + return now_ms; + } + size_t pauseCount() const + { + std::lock_guard lock(mutex); + return pauses; + } + uint64_t longestPause() const + { + std::lock_guard lock(mutex); + return longest_pause; + } + + void advance(uint64_t ms) + { + std::lock_guard lock(mutex); + /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move + /// would leave the loop reissuing for ever against a fault that never clears. + now_ms += ms + 1; + ++pauses; + longest_pause = std::max(longest_pause, ms); + } + +private: + mutable std::mutex mutex; + uint64_t now_ms = 0; + size_t pauses = 0; + uint64_t longest_pause = 0; +}; + +/// `ChunkFaultBackend` COUNTS its faults, and a count can no longer make one conclusive: the write +/// engine settles every ambiguity by an exact read and then REISSUES, so a fault that runs out +/// mid-call is answered by the next attempt instead of by the call's own deadline -- which is the +/// whole difference between a wedge and a commit. +class LatchedChunkFaultBackend : public DB::Cas::tests::ChunkFaultBackend +{ +public: + bool latched = false; + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + if (latched && mode != Mode::None && fault_skip == 0 && !expected_value && !fault_substr.empty() + && key.find(fault_substr) != String::npos) + fault_count = 1; + return ChunkFaultBackend::write(key, bytes, expected_value, access); + } + + void disarm() + { + latched = false; + mode = Mode::None; + fault_count = 0; + fault_skip = 0; + fail_read_once_key.clear(); + } +}; + class RecoveryLatchBackend : public CountingBackend { public: - using CountingBackend::get; using CountingBackend::getStream; - using CountingBackend::putIfAbsent; - using CountingBackend::putOverwrite; - using CountingBackend::casPut; - /// Set before the driving call; consumed by the first matching recovery GET. + /// Set before the driving call; consumed by the first matching recovery read. String fail_get_once_key; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (!fail_get_once_key.empty() && key == fail_get_once_key) { fail_get_once_key.clear(); throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, - "RecoveryLatchBackend: simulated non-transient exact GET failure"); + "RecoveryLatchBackend: simulated non-transient exact read failure"); } { std::unique_lock lk(m); @@ -110,7 +189,7 @@ class RecoveryLatchBackend : public CountingBackend cv.wait_for(lk, std::chrono::seconds(20), [&] { return block_key.empty(); }); } } - return CountingBackend::get(key, range); + return CountingBackend::read(key, access); } void armBlockedGet(const String & key) @@ -242,11 +321,14 @@ ManifestId publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const S return id; } -/// Every request class `CountingBackend` observes, summed. The zero-I/O contract is asserted against -/// this total, so a confirm that quietly grew a HEAD or a GET fails the test rather than the review. +/// The reads, heads, stream opens, writes and lists `CountingBackend` observes, summed. The zero-I/O +/// contract is asserted against this total, so a confirm that quietly grew a HEAD or a GET fails the +/// test rather than the review. `writeTotal` and not `putTotal`: a write that carried a precondition +/// is still a write, and counting only the create-shaped ones left the replace path unwatched. +/// Deletes are NOT in this sum. uint64_t backendRequests(const CountingBackend & b) { - return b.headTotal() + b.getTotal() + b.getStreamTotal() + b.putTotal() + b.listTotal(); + return b.headTotal() + b.getTotal() + b.getStreamTotal() + b.writeTotal() + b.listTotal(); } /// One refusal counter's current value. `confirmExactRef` attributes every `Unknown` to exactly one of @@ -667,19 +749,18 @@ TEST(CASConfirmExactRef, WedgedLaneIsUnknown) /// touched. TEST(CASConfirmExactRef, WedgedTransactionRefusesEveryRef) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); PoolConfig cfg; - /// Single-attempt budget: one ambiguous PUT is a conclusive wedge, no inter-attempt sleep. The - /// operation deadline is deliberately far wider than one attempt, so the pre-send gate never - /// refuses before the injected fault is reached and the outcome is decided by the fault, not by - /// how loaded the machine is. + /// The budget bounds the mount lease's own admission arithmetic and nothing else: a write's attempt + /// count is the `Retry` policy's. What makes the injected fault conclusive is that it stays armed + /// for the whole call while the injected clock below carries the call to its own deadline. CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; auto store = openPoolWithConfig(backend, cfg); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/confirm_real_wedge"}; /// Pins the namespace to the fixture life BEFORE its first real touch, so the fault key computed /// from that same life below is the key production actually writes to. @@ -694,7 +775,14 @@ TEST(CASConfirmExactRef, WedgedTransactionRefusesEveryRef) backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_skip = 0; backend->fault_count = 1; + backend->latched = true; EXPECT_THROW(store->dropRef(ns, "x"), DB::Exception); + backend->disarm(); + /// The give-up was the call's OWN retry window: the fault outlasted several reissues and every one + /// of them paced through the injected sleep rather than a real one. + EXPECT_GT(clock->pauseCount(), 1u); + EXPECT_LE(clock->longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; + EXPECT_GE(clock->nowMs(), 60000u); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(store->refQueuePendingForTest(ns), 0u); ASSERT_EQ(store->refCarvedForTest(ns), 0u) @@ -768,9 +856,7 @@ TEST(CASConfirmExactRef, MisScopedItemFailsBeforeAnythingIsDurable) const RootNamespace ns{"srv1/confirm_misscoped"}; const ManifestId seed = publishEmptyPart(store, ns, "seed"); /// the namespace is born already - const uint64_t puts_before = backend->putTotal(); - const uint64_t overwrites_before = backend->putOverwriteTotal(); - const uint64_t cas_puts_before = backend->casPutTotal(); + const uint64_t writes_before = backend->writeTotal(); RefOp add; add.kind = RefOpKind::OwnerTransition; add.new_binding = RefOwnerBinding{RefOwnerKind::Precommit, "y", ManifestRef{900000003, 1, 1}}; @@ -804,11 +890,9 @@ TEST(CASConfirmExactRef, MisScopedItemFailsBeforeAnythingIsDurable) } /// The ref-log transaction object -- the only thing that would make this item durable -- is a - /// `putIfAbsent`; `putOverwrite` and `casPut` are asserted too so the fence covers every write kind - /// the backend can observe, not just the one this item would have used. - EXPECT_EQ(backend->putTotal(), puts_before) << "the refusal must happen before any object is written"; - EXPECT_EQ(backend->putOverwriteTotal(), overwrites_before) << "the refusal must happen before any object is written"; - EXPECT_EQ(backend->casPutTotal(), cas_puts_before) << "the refusal must happen before any object is written"; + /// create, and this counts every write the backend can observe rather than that one shape, so the + /// fence still holds if the durable step ever changes shape. + EXPECT_EQ(backend->writeTotal(), writes_before) << "the refusal must happen before any object is written"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready) << "a validation failure is not a lane fault"; EXPECT_EQ(store->confirmExactRef(ns, "seed", seed.ref), ConfirmAnswer::Yes) << "the failed item must leave the table exactly as it was"; diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 9701330e3406..aa562738649a 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -1,5 +1,6 @@ #include "cas_test_helpers.h" #include +#include #include #include #include @@ -39,13 +40,13 @@ void drainCompletedNamespaceRemovals(const std::shared_ptr & ba ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); } -/// Fails `deleteExact` for one or two designated keys -- either by throwing (a transient backend -/// hiccup) or by returning a synthetic `TokenMismatch` (a "listed but raced" outcome) -- delegating -/// every other key to the base `InMemoryBackend` untouched. Drives the drain phases' per-object -/// fail-close path (`deleteListedPrefix`/`sweepNamespace`, `CasDecommission.cpp`/ -/// `CasOrphanManifestSweep.cpp`): a failure on one listed object must record a warning and let the rest -/// of the sweep proceed, never abort the whole phase. -/// +/// Fails a delete for one or two designated keys -- either by throwing (a transient backend hiccup) or +/// by returning a synthetic `Mismatch` (a "listed but raced" outcome) -- delegating every other key to +/// the base `InMemoryBackend` untouched. Drives the drain phases' per-object fail-close path +/// (`deleteListedPrefix`/`sweepNamespace`, `CasDecommission.cpp`/`CasOrphanManifestSweep.cpp`): a +/// failure on one listed object must record a warning and let the rest of the sweep proceed, never +/// abort the whole phase. Injects on the `remove` PRIMITIVE, not the legacy `deleteExact`, so it +/// intercepts a caller on either surface. class FailingDeleteBackend : public InMemoryBackend { public: @@ -54,13 +55,13 @@ class FailingDeleteBackend : public InMemoryBackend /// Clears every injected failure -- the resume half of a fail-then-retry test (Task 4). void disarm() { throw_key.clear(); mismatch_key.clear(); } - DeleteOutcome deleteExact(const String & key, const Token & token) override + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { if (key == throw_key) throw std::runtime_error("injected transient delete failure for " + key); if (key == mismatch_key) - return DeleteOutcome{.kind = DeleteOutcome::Kind::TokenMismatch}; - return InMemoryBackend::deleteExact(key, token); + return RawRemoval::Mismatch; + return InMemoryBackend::remove(key, expected_value, access); } private: @@ -74,8 +75,6 @@ class FailingDeleteBackend : public InMemoryBackend class CatalogChangesAfterFirstReadBackend : public InMemoryBackend { public: - using Backend::get; - void armCatalogReplacement( const String & key, RefCatalog replacement_, size_t completed_reads_before_replacement = 0) { @@ -87,9 +86,11 @@ class CatalogChangesAfterFirstReadBackend : public InMemoryBackend bool fired() const { return replacement_fired; } - std::optional get(const String & key, Range range) override + /// Injects on the `read` PRIMITIVE rather than the legacy `get`: whichever caller reaches this + /// key -- through the legacy forwarder or through `CasOperation::read` -- funnels through here. + std::optional read(const String & key, TransportAccess & access) override { - auto got = InMemoryBackend::get(key, range); + auto got = InMemoryBackend::read(key, access); if (!armed || replacement_fired || key != catalog_key) return got; if (reads_to_skip > 0) @@ -101,9 +102,8 @@ class CatalogChangesAfterFirstReadBackend : public InMemoryBackend throw std::runtime_error("catalog replacement fixture: catalog is absent"); replacement_fired = true; - const PutResult put = InMemoryBackend::putOverwrite( - key, encodeRefCatalog(replacement), got->token, {}); - if (put.outcome != PutOutcome::Done) + const auto put = InMemoryBackend::write(key, encodeRefCatalog(replacement), got->value, access); + if (!put.has_value()) throw std::runtime_error("catalog replacement fixture: rewrite conflicted"); return got; } @@ -146,24 +146,26 @@ std::vector> snapshotPrefixObjects( class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend { public: - using Backend::get; - using Backend::putOverwrite; - void armForSuccessorReclaim() { armed = true; } - std::optional get(const String & key, Range range) override + /// Injects on the `read` PRIMITIVE: the retirement tail's own reads of `mount_key`/`epoch_key` + /// (`CasDecommission.cpp`) go through `CasOperation::read`, not the legacy `get`. + std::optional read(const String & key, TransportAccess & access) override { - std::optional result = InMemoryBackend::get(key, range); + std::optional result = InMemoryBackend::read(key, access); if (farewell_seen && !successor_injected && (key == mount_key || key == epoch_key)) - injectSuccessor(); + injectSuccessor(access); return result; } - PutResult putOverwrite( - const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override + /// Injects on the `write` PRIMITIVE: `putOverwrite` is not one of the two verb-identity + /// exceptions (`putIfAbsent`/`casPut`), so both the legacy caller and `CasOperation::replace` + /// reach it here. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - const PutResult result = InMemoryBackend::putOverwrite(key, bytes, expected, meta); - if (armed && key == mount_key && result.outcome == PutOutcome::Done) + const auto result = InMemoryBackend::write(key, bytes, expected_value, access); + if (armed && result.has_value() && key == mount_key) { const MountLease mount = decodeMountLease(bytes); if (mount.min_active_build_sequence == std::numeric_limits::max()) @@ -179,10 +181,15 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend const String & successorEpochBytes() const { return successor_epoch_bytes; } private: - void injectSuccessor() + /// Every request here is issued on a PRIMITIVE. A legacy verb would be re-dispatched through the + /// virtual primitive it forwards to -- `get` through `read` -- which is this very hook, so the + /// injection would re-enter itself with `successor_injected` still false. The two `Token`s the + /// tests compare against are minted through `Backend`'s own minter, the one the legacy forwarders + /// use, so they are the values a legacy caller would have received. + void injectSuccessor(TransportAccess & access) { - const auto epoch = InMemoryBackend::get(epoch_key, {}); - const auto mount = InMemoryBackend::get(mount_key, {}); + const auto epoch = InMemoryBackend::read(epoch_key, access); + const auto mount = InMemoryBackend::read(mount_key, access); if (!epoch || !mount) throw std::runtime_error("successor-reclaim fixture: control object disappeared before reclaim"); @@ -190,11 +197,10 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend const uint64_t successor_writer_epoch = epoch_value.next_writer_epoch; ++epoch_value.next_writer_epoch; successor_epoch_bytes = encodeServerEpoch(epoch_value); - const CasResult epoch_put = InMemoryBackend::casPut( - epoch_key, successor_epoch_bytes, std::optional{epoch->token}, {}); - if (epoch_put.outcome != CasOutcome::Committed) + const auto epoch_written = InMemoryBackend::write(epoch_key, successor_epoch_bytes, epoch->value, access); + if (!epoch_written) throw std::runtime_error("successor-reclaim fixture: epoch bump conflicted"); - successor_epoch_token = epoch_put.token; + successor_epoch_token = legacyMintWritten(epoch_key, *epoch_written); MountLease mount_value = decodeMountLease(mount->bytes); mount_value.writer_epoch = successor_writer_epoch; @@ -204,11 +210,10 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend mount_value.min_active_build_sequence = 0; mount_value.gc_fenced = false; successor_mount_bytes = encodeMountLease(mount_value); - const PutResult mount_put = InMemoryBackend::putOverwrite( - mount_key, successor_mount_bytes, mount->token, {}); - if (mount_put.outcome != PutOutcome::Done) + const auto mount_written = InMemoryBackend::write(mount_key, successor_mount_bytes, mount->value, access); + if (!mount_written) throw std::runtime_error("successor-reclaim fixture: mount reclaim conflicted"); - successor_mount_token = mount_put.token; + successor_mount_token = legacyMintWritten(mount_key, *mount_written); successor_injected = true; } @@ -229,19 +234,15 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend { public: - using Backend::get; - using Backend::putOverwrite; - void armForSuccessorReclaim() { armed = true; } - DeleteOutcome deleteExact(const String & key, const Token & token) override + /// Injects on the `remove` PRIMITIVE: `deleteSlotObject`'s epoch delete (`CasDecommission.cpp`) + /// goes through `CasOperation::remove`, not the legacy `deleteExact`. + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { - const DeleteOutcome result = InMemoryBackend::deleteExact(key, token); - if (armed && !successor_injected && key == epoch_key - && classifyDeleteOutcome(result) == DeleteClass::Deleted) - { - injectSuccessor(); - } + const RawRemoval result = InMemoryBackend::remove(key, expected_value, access); + if (armed && !successor_injected && key == epoch_key && result == RawRemoval::Removed) + injectSuccessor(access); return result; } @@ -253,21 +254,30 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend const String & successorEpochBytes() const { return successor_epoch_bytes; } private: - PutResult putOverwrite( - const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override + /// Counts on the `write` PRIMITIVE: the owner tombstone write this test asserts is never + /// attempted (`CasDecommission.cpp`'s `op.replace`) reaches the store through here, not through + /// the legacy `putOverwrite`. Guarded on `expected_value`: the primitive also sees the owner + /// anchor's own CREATE during this fixture's `openVictim`, which `putOverwrite` (a conditional + /// REPLACE only) never did -- counting it would start this counter at 1 before the interesting + /// part of the test begins. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - if (key == owner_key) + if (key == owner_key && expected_value) ++owner_rewrite_attempts; - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } - void injectSuccessor() + /// On the `write` PRIMITIVE, for the reason the sibling fixture above states: a legacy verb is + /// re-dispatched through the virtual primitive it forwards to, so it would re-enter this class's + /// own overrides instead of reaching the store directly. + void injectSuccessor(TransportAccess & access) { successor_epoch_bytes = encodeServerEpoch(ServerEpoch{.next_writer_epoch = 102}); - const PutResult epoch_put = InMemoryBackend::putIfAbsent(epoch_key, successor_epoch_bytes, {}); - if (epoch_put.outcome != PutOutcome::Done) + const auto epoch_written = InMemoryBackend::write(epoch_key, successor_epoch_bytes, std::nullopt, access); + if (!epoch_written) throw std::runtime_error("late-successor fixture: epoch recreation conflicted"); - successor_epoch_token = epoch_put.token; + successor_epoch_token = legacyMintWritten(epoch_key, *epoch_written); successor_mount_bytes = encodeMountLease(MountLease{ .server_uuid = UInt128(0x1234), @@ -279,10 +289,10 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend .expires_at_ms = 31'000, .min_active_build_sequence = 0, }); - const PutResult mount_put = InMemoryBackend::putIfAbsent(mount_key, successor_mount_bytes, {}); - if (mount_put.outcome != PutOutcome::Done) + const auto mount_written = InMemoryBackend::write(mount_key, successor_mount_bytes, std::nullopt, access); + if (!mount_written) throw std::runtime_error("late-successor fixture: mount recreation conflicted"); - successor_mount_token = mount_put.token; + successor_mount_token = legacyMintWritten(mount_key, *mount_written); successor_injected = true; } @@ -304,33 +314,34 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend { public: - using Backend::get; - void armForSuccessorRewrite() { armed = true; } - std::optional get(const String & key, Range range) override + /// Injects on the `read` PRIMITIVE: the tombstone tail's owner read (`CasDecommission.cpp`) + /// goes through `CasOperation::read`, not the legacy `get`. + std::optional read(const String & key, TransportAccess & access) override { - std::optional result = InMemoryBackend::get(key, range); + std::optional result = InMemoryBackend::read(key, access); if (armed && epoch_deleted && !successor_injected && key == owner_key && result) { successor_owner_bytes = encodeOwner(OwnerObject{ .server_uuid = decodeOwner(result->bytes).server_uuid, .retired_at_ms = std::nullopt, }); - const PutResult put = InMemoryBackend::putOverwrite( - owner_key, successor_owner_bytes, result->token, {}); - if (put.outcome != PutOutcome::Done) + const auto put = InMemoryBackend::write(owner_key, successor_owner_bytes, result->value, access); + if (!put.has_value()) throw std::runtime_error("owner-successor fixture: owner rewrite conflicted"); - successor_owner_token = put.token; + successor_owner_token = legacyMintWritten(owner_key, *put); successor_injected = true; } return result; } - DeleteOutcome deleteExact(const String & key, const Token & token) override + /// Injects on the `remove` PRIMITIVE: `deleteSlotObject`'s epoch delete (`CasDecommission.cpp`) + /// goes through `CasOperation::remove`, not the legacy `deleteExact`. + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { - const DeleteOutcome result = InMemoryBackend::deleteExact(key, token); - if (armed && key == epoch_key && classifyDeleteOutcome(result) == DeleteClass::Deleted) + const RawRemoval result = InMemoryBackend::remove(key, expected_value, access); + if (armed && key == epoch_key && result == RawRemoval::Removed) epoch_deleted = true; return result; } @@ -349,35 +360,6 @@ class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend String successor_owner_bytes; }; -/// Models an "ambiguous success" on the final owner tombstone write: the conditional overwrite -/// actually lands (InMemoryBackend applies it), but the response is then lost (a transient -/// exception is thrown on the SAME call, exactly as a real SDK timeout after a landed write would -/// look). Before the fix, decommission caught any exception here and reported failure -/// unconditionally; the controlled overwrite must resolve this via a GET (the current bytes match -/// what was intended) and report Committed instead. -class AmbiguousOwnerTombstoneBackend : public InMemoryBackend -{ -public: - using Backend::putOverwrite; - - void armForAmbiguousTombstone() { armed = true; } - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override - { - const PutResult result = InMemoryBackend::putOverwrite(key, bytes, expected, meta); - if (armed && !fired && key == owner_key && result.outcome == PutOutcome::Done) - { - fired = true; - throw std::runtime_error("ambiguous-tombstone fixture: response lost after the write landed"); - } - return result; - } - -private: - inline static const String owner_key = "p/gc/server-roots/victim/owner"; - bool armed = false; - bool fired = false; -}; /// Seed one victim table with `committed` committed refs and `precommits` dangling precommit bindings, /// via the raw ref-log seeding helpers (fixture idiom of e.g. `gtest_cas_gc_fold.cpp`: `writeManifestRaw` @@ -392,10 +374,16 @@ void makeTableWithRefs(Pool & victim, const String & ns_str, uint64_t committed, Backend & backend = victim.backend(); const Layout & layout = victim.layout(); + /// A throwaway open-fence operation, for the two `CasRefCatalog` calls below only: this fixture + /// writes everything else directly against `backend` via the raw-write helpers, unrelated to any + /// mount fence. + CasRequests requests(victim.poolBackendPtr(), Fence::open()); + CasOperation op = requests.admit(); + /// Final physical ids are pool-wide. The generic raw-write helper intentionally uses one shared /// transition sentinel, so this multi-namespace fixture admits a distinct deterministic test life /// before invoking it; the helper then resolves and preserves that existing catalog identity. - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(op, layout); const auto existing = std::find_if(catalog.catalog.entries.begin(), catalog.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns.string() == ns.string(); }); if (existing == catalog.catalog.entries.end()) @@ -405,7 +393,7 @@ void makeTableWithRefs(Pool & victim, const String & ns_str, uint64_t committed, entry.ns = ns; entry.state = NsState::Live; entry.incarnation = UInt128{next_test_life.fetch_add(1)}; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); } uint64_t last_ref_sequence = 0; @@ -1031,7 +1019,7 @@ TEST(CASDecommission, SuccessorReclaimFencesSlotRetirementTail) EXPECT_FALSE(report.slot_removed); ASSERT_EQ(report.warnings.size(), 1u); EXPECT_NE(report.warnings.front().find("p/gc/server-roots/victim/mount"), String::npos); - EXPECT_NE(report.warnings.front().find("replaced"), String::npos); + EXPECT_NE(report.warnings.front().find("mismatch"), String::npos); const auto mount = backend->get("p/gc/server-roots/victim/mount"); ASSERT_TRUE(mount.has_value()); @@ -1104,6 +1092,40 @@ TEST(CASDecommission, FencedSlotRetirementTailRetiresUncontendedSlot) EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); } +/// One decommission command spans several requests on the SAME open-fence engine: the +/// pre-impersonation catalog cut (before any `Pool` exists), the namespace drop's own catalog re-read, +/// and -- once the row it dropped is no longer owned -- the full retirement tail's reads, deletes and +/// final owner tombstone, all issued after `admin.reset()` destroys the `Pool`. Catalog-row deletion +/// is GC's job (`dropNamespace` only reaches `Removing`), so this is necessarily two commands: the +/// first proves the drop and its catalog read landed, the second (after GC folds the row) proves the +/// retirement tail's requests landed past the `Pool`'s own lifetime. +TEST(CASDecommission, RunsOnAnOpenFence) +{ + auto backend = std::make_shared(); + { + auto victim = openVictim(backend); + makeTableWithRefs(*victim, "victim/db/t1", 1, 0); + } + + const auto pending = decommissionPoolMember( + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + EXPECT_EQ(pending.namespaces_removed, 1u); + EXPECT_FALSE(pending.slot_removed); + EXPECT_FALSE(pending.warnings.empty()); + + drainCompletedNamespaceRemovals(backend); + + const auto report = decommissionPoolMember( + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.slot_removed); + EXPECT_FALSE(backend->get("p/gc/server-roots/victim/mount").has_value()); + EXPECT_FALSE(backend->get("p/gc/server-roots/victim/epoch").has_value()); + const auto owner = backend->get("p/gc/server-roots/victim/owner"); + ASSERT_TRUE(owner.has_value()); + EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); +} + TEST(CASDecommission, SuccessfulDecommissionLeavesTombstonedOwnerAnchor) { auto backend = std::make_shared(); @@ -1148,14 +1170,18 @@ TEST(CASDecommission, SuccessorOwnerRewriteWinsBeforeTombstone) } /// Final whole-branch review finding (Important): -/// a transient exception on the owner tombstone write must not be reported as a hard failure when the -/// write actually landed -- the controlled overwrite resolves this via GET (current bytes already -/// match the intended tombstone) instead of the old bare putOverwrite's "any exception = failure". +/// an ambiguous outcome on the owner tombstone write -- the write lands but its response is lost, the +/// same shape as a real SDK timeout after a landed write -- must not be reported as a hard failure. +/// `op.replace`'s own resolve read settles this (current bytes already match the intended tombstone) +/// and reports `Committed`. `injectAmbiguousLandedWrite` throws `Poco::TimeoutException` after +/// applying the write: the engine's write loop treats a `Poco::Exception` as a transport fault (never +/// a caller bug), which is exactly the class this scenario models -- a bare `std::runtime_error` here +/// would propagate unchanged instead of being resolved. TEST(CASDecommission, OwnerTombstoneAmbiguousSuccessResolvesToCommitted) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); { auto victim = openVictim(backend); } - backend->armForAmbiguousTombstone(); + backend->injectAmbiguousLandedWrite("p/gc/server-roots/victim/owner"); const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); @@ -1168,11 +1194,12 @@ TEST(CASDecommission, OwnerTombstoneAmbiguousSuccessResolvesToCommitted) EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); } -/// Delegates every op to `inner`, except `deleteExact`: while `armed`, any key starting with -/// `fail_prefix` throws an injected transient failure instead of deleting -- models a real backend -/// transiently failing to delete under one whole prefix. `disarm()` clears the failure (the resume -/// half of `FailedDrainKeepsSlotThenResumes`). Forwards every pure-virtual `Backend` member (the -/// `CasBackend.h` list) to `inner` untouched. +/// Delegates every op to `inner`, except `remove`: while `armed`, any key starting with `fail_prefix` +/// throws an injected transient failure instead of deleting -- models a real backend transiently +/// failing to delete under one whole prefix. `disarm()` clears the failure (the resume half of +/// `FailedDrainKeepsSlotThenResumes`). Forwards every pure-virtual `Backend` member (the `CasBackend.h` +/// list) to `inner` untouched. Injects on the `remove` PRIMITIVE, not the legacy `deleteExact`: +/// `deleteListedPrefix`'s mountpoint drain (`CasDecommission.cpp`) goes through `CasOperation::remove`. class FailDeletesUnderPrefixBackend : public Backend { public: @@ -1208,21 +1235,24 @@ class FailDeletesUnderPrefixBackend : public Backend { return inner->casPut(key, bytes, expected, meta); } - DeleteOutcome deleteExact(const String & key, const Token & token) override - { - if (armed && key.starts_with(fail_prefix)) - throw Exception(ErrorCodes::S3_ERROR, "injected transient delete failure for {}", key); - return inner->deleteExact(key, token); - } ListPage list(const String & prefix, const String & cursor, size_t limit) override { return inner->list(prefix, cursor, limit); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The transport primitives forward to `inner`; the legacy overrides above are what this - /// double injects through. Declared because `Backend` declares them pure. + /// The transport primitives forward to `inner`, except `remove`, which is what this double + /// injects through. Declared because `Backend` declares them pure. std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } - RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + if (armed && key.starts_with(fail_prefix)) + /// A caller-bug-shaped exception (never `Poco::Exception`), so the engine surfaces it on + /// the first attempt instead of reissuing it for the whole policy window -- this fixture + /// models a single per-object failure the drain must warn on and move past, not a + /// transport fault the retry loop should absorb. + throw std::runtime_error("injected transient delete failure for " + key); + return inner->remove(key, expected_value, access); + } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { diff --git a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp index 93d23f6a8cb7..5407d4b0f086 100644 --- a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp +++ b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp @@ -24,9 +24,9 @@ PoolPtr openVictim(const std::shared_ptr & backend) return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); } -CatalogEntry catalogEntry(Backend & backend, const Layout & layout, const RootNamespace & ns) +CatalogEntry catalogEntry(CasOperation & op, const Layout & layout, const RootNamespace & ns) { - const RefCatalog catalog = CasRefCatalog::read(backend, layout).catalog; + const RefCatalog catalog = CasRefCatalog::read(op, layout).catalog; const auto it = std::find_if(catalog.entries.begin(), catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); if (it == catalog.entries.end()) @@ -34,9 +34,9 @@ CatalogEntry catalogEntry(Backend & backend, const Layout & layout, const RootNa return *it; } -void makeRemoving(Backend & backend, const Layout & layout, const CatalogEntry & live) +void makeRemoving(CasOperation & op, const Layout & layout, const CatalogEntry & live) { - CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & current) + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find(next.entries.begin(), next.entries.end(), live); @@ -56,19 +56,23 @@ bool slotObjectExists(Backend & backend, const String & leaf) class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; + /// Unhide the legacy `list` overloads the primitive override below would otherwise hide. + using Backend::list; void arm() { armed = true; } bool fired() const { return added; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + /// Intercepted at the PRIMITIVE, which every legacy forwarder reaches too, so the injection fires + /// whichever surface issued the enumeration. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); if (armed && !added && prefix == "p/roots/victim/" && cursor.empty()) { added = true; + CasRequests requests = DB::Cas::tests::openRequestsForTest(*this); + CasOperation op = requests.admit(); CasRefCatalog::casAdmitEntry( - *this, Layout("p"), 1, + op, Layout("p"), 1, CatalogEntry{ .ns = RootNamespace("victim/db/late"), .state = NsState::Live, @@ -85,26 +89,28 @@ class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend /// Admits the late catalog entry between the retirement tail's two exact catalog reads /// (`retirement_catalog_cut`, then `fresh_retirement_catalog`), never before. The mountpoint drain's /// `list("p/roots/victim/", ...)` is the last LIST call in `decommissionPoolMember` before either -/// read, so it orders the two `get("p/cas/ref_catalog")` calls that follow it: the first is +/// read, so it orders the two `read("p/cas/ref_catalog")` calls that follow it: the first is /// `retirement_catalog_cut`, the second is `fresh_retirement_catalog`. Mutating on the second call /// makes that read observe a catalog the first read did not. class MutateCatalogBetweenRetirementReadsBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; + /// Unhide the legacy `list` overloads the primitive override below would otherwise hide. + using Backend::list; void arm() { armed = true; } bool fired() const { return added; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + /// Both hooks sit on the PRIMITIVES, which every legacy forwarder reaches too, so the ordering + /// they observe is the physical request order whichever surface issued each request. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); if (armed && !past_mountpoint_drain && prefix == "p/roots/victim/" && cursor.empty()) past_mountpoint_drain = true; return page; } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (armed && past_mountpoint_drain && !added && key == "p/cas/ref_catalog") { @@ -113,15 +119,17 @@ class MutateCatalogBetweenRetirementReadsBackend final : public InMemoryBackend else { added = true; + CasRequests requests = DB::Cas::tests::openRequestsForTest(*this); + CasOperation op = requests.admit(); CasRefCatalog::casAdmitEntry( - *this, Layout("p"), 1, + op, Layout("p"), 1, CatalogEntry{ .ns = RootNamespace("victim/db/late"), .state = NsState::Live, .incarnation = UInt128{707}}); } } - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } private: @@ -134,6 +142,8 @@ class MutateCatalogBetweenRetirementReadsBackend final : public InMemoryBackend TEST(CASDecommissionCatalogDuties, RemovingWithoutCheckpointIsCorruptionAndKeepsSlot) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); { auto victim = openVictim(backend); const CatalogEntry live{ @@ -141,9 +151,9 @@ TEST(CASDecommissionCatalogDuties, RemovingWithoutCheckpointIsCorruptionAndKeeps .state = NsState::Live, .incarnation = UInt128{701}}; CasRefCatalog::casAdmitEntry( - *backend, victim->layout(), victim->poolConfig().gc_shards, + catalog_op, victim->layout(), victim->poolConfig().gc_shards, live); - makeRemoving(*backend, victim->layout(), live); + makeRemoving(catalog_op, victim->layout(), live); } expectThrowsCode(ErrorCodes::CORRUPTED_DATA, [&] @@ -155,20 +165,22 @@ TEST(CASDecommissionCatalogDuties, RemovingWithoutCheckpointIsCorruptionAndKeeps EXPECT_TRUE(slotObjectExists(*backend, "owner")); EXPECT_TRUE(slotObjectExists(*backend, "epoch")); EXPECT_TRUE(slotObjectExists(*backend, "mount")); - EXPECT_EQ(catalogEntry(*backend, Layout("p"), RootNamespace("victim/db/missing_ckpt")).state, + EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), RootNamespace("victim/db/missing_ckpt")).state, NsState::Removing); } TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndKeepsSlotForGc) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const RootNamespace ns("victim/db/pending_terminal"); std::optional life; { auto victim = openVictim(backend); life = victim->namespaceLife(ns); - const CatalogEntry live = catalogEntry(*backend, victim->layout(), ns); - makeRemoving(*backend, victim->layout(), live); + const CatalogEntry live = catalogEntry(catalog_op, victim->layout(), ns); + makeRemoving(catalog_op, victim->layout(), live); ASSERT_TRUE(backend->head(victim->layout().refCkptKey(*life)).exists); ASSERT_TRUE(backend->list(victim->layout().namespaceStreamPrefix(*life), "", 100).keys.empty()); } @@ -200,22 +212,24 @@ TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndKeeps TEST(CASDecommissionCatalogDuties, PartialRemovalProgressStillWakesGcWhenLaterNamespaceFails) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const RootNamespace progressed_ns("victim/db/a_progressed"); const RootNamespace broken_ns("victim/db/z_missing_ckpt"); std::optional progressed_life; { auto victim = openVictim(backend); progressed_life = victim->namespaceLife(progressed_ns); - const CatalogEntry progressed_live = catalogEntry(*backend, victim->layout(), progressed_ns); - makeRemoving(*backend, victim->layout(), progressed_live); + const CatalogEntry progressed_live = catalogEntry(catalog_op, victim->layout(), progressed_ns); + makeRemoving(catalog_op, victim->layout(), progressed_live); const CatalogEntry broken_live{ .ns = broken_ns, .state = NsState::Live, .incarnation = UInt128{713}}; CasRefCatalog::casAdmitEntry( - *backend, victim->layout(), victim->poolConfig().gc_shards, broken_live); - makeRemoving(*backend, victim->layout(), broken_live); + catalog_op, victim->layout(), victim->poolConfig().gc_shards, broken_live); + makeRemoving(catalog_op, victim->layout(), broken_live); ASSERT_FALSE(backend->head(victim->layout().refCkptKey( NamespaceLifeId::fromCatalogEntry(broken_ns, broken_live.incarnation))).exists); } @@ -239,6 +253,8 @@ TEST(CASDecommissionCatalogDuties, PartialRemovalProgressStillWakesGcWhenLaterNa TEST(CASDecommissionCatalogDuties, VictimEntryAppearingBeforeTheOwnershipCutKeepsSlot) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); { auto victim = openVictim(backend); } backend->arm(); @@ -251,12 +267,14 @@ TEST(CASDecommissionCatalogDuties, VictimEntryAppearingBeforeTheOwnershipCutKeep EXPECT_NE(report.warnings.front().find("pool member decommission underway: 1 namespace(s)"), String::npos) << report.warnings.front(); EXPECT_TRUE(slotObjectExists(*backend, "owner")); - EXPECT_EQ(catalogEntry(*backend, Layout("p"), RootNamespace("victim/db/late")).state, NsState::Live); + EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), RootNamespace("victim/db/late")).state, NsState::Live); } TEST(CASDecommissionCatalogDuties, CatalogTokenMovedBetweenOwnershipCutAndRetirementKeepsSlot) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); { auto victim = openVictim(backend); } backend->arm(); @@ -269,12 +287,14 @@ TEST(CASDecommissionCatalogDuties, CatalogTokenMovedBetweenOwnershipCutAndRetire EXPECT_NE(report.warnings.front().find("catalog changed after the victim ownership check"), String::npos) << report.warnings.front(); EXPECT_TRUE(slotObjectExists(*backend, "owner")); - EXPECT_EQ(catalogEntry(*backend, Layout("p"), RootNamespace("victim/db/late")).state, NsState::Live); + EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), RootNamespace("victim/db/late")).state, NsState::Live); } TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAnotherRound) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const RootNamespace ns("victim/db/folded_terminal"); std::optional life; std::vector stream_before; @@ -291,7 +311,7 @@ TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAn Gc gc(victim, UInt128{811}); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); - ASSERT_EQ(catalogEntry(*backend, victim->layout(), ns).state, NsState::Removing); + ASSERT_EQ(catalogEntry(catalog_op, victim->layout(), ns).state, NsState::Removing); for (const ListedKey & key : backend->list(victim->layout().namespaceStreamPrefix(*life), "", 100).keys) stream_before.push_back(key.key); ASSERT_FALSE(stream_before.empty()); @@ -305,7 +325,7 @@ TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAn EXPECT_EQ(wake_requests.load(), 1u); EXPECT_EQ(report.namespaces_already_removed, 1u); EXPECT_FALSE(report.slot_removed); - EXPECT_EQ(catalogEntry(*backend, Layout("p"), ns).state, NsState::Removing); + EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), ns).state, NsState::Removing); std::vector stream_after; for (const ListedKey & key : backend->list(Layout("p").namespaceStreamPrefix(*life), "", 100).keys) stream_after.push_back(key.key); @@ -316,6 +336,8 @@ TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAn TEST(CASDecommissionCatalogDuties, OpaqueLifeDebrisWithoutCatalogOwnershipDoesNotBlockRetirement) { auto backend = std::make_shared(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); { auto victim = openVictim(backend); } const Layout layout("p"); const NamespaceLifeId dead_life diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index a35e3a5adf31..4f0be05d79f0 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -6,12 +6,14 @@ #include #include +#include #include #include #include #include #include #include +#include #include using namespace DB::Cas; @@ -25,12 +27,17 @@ namespace { /// A gate a test opens explicitly, so a task can be held in flight without a sleep. +/// +/// The wait is BOUNDED and reports a failure rather than blocking for ever, and it names the gate it +/// waited on: an unbounded wait on a premise that stopped holding hung the whole binary here, which +/// hid every test that would have run after it. struct Gate { - void wait() + void wait(std::string_view name) { std::unique_lock lock(m); - cv.wait(lock, [this] { return open_; }); + if (!cv.wait_for(lock, std::chrono::seconds(60), [this] { return open_; })) + ADD_FAILURE() << "timed out waiting for '" << name << "'"; } void open() { @@ -43,13 +50,23 @@ struct Gate bool open_ = false; }; -/// Completes the watched first `GET`, then withholds its return so teardown can latch before the +/// Opens its gate on every exit from the scope, so a failing assertion cannot strand the thread +/// parked behind it: the parked thread is joined during teardown, and a gate that stayed shut turned +/// a reported failure into a whole-binary deadlock. +struct GateOpenedOnExit +{ + explicit GateOpenedOnExit(Gate & gate_) : gate(gate_) {} + GateOpenedOnExit(const GateOpenedOnExit &) = delete; + GateOpenedOnExit & operator=(const GateOpenedOnExit &) = delete; + ~GateOpenedOnExit() { gate.open(); } + Gate & gate; +}; + +/// Completes the watched first read, then withholds its return so teardown can latch before the /// helper is able to issue its next raw request. class BetweenRecoveryGetsBackend : public DB::Cas::tests::OrderedFaultBackend { public: - using DB::Cas::tests::OrderedFaultBackend::get; - void armBetweenGets(String first_key_, std::shared_ptr first_completed_, std::shared_ptr release_first_) { first_key = std::move(first_key_); @@ -58,13 +75,13 @@ class BetweenRecoveryGetsBackend : public DB::Cas::tests::OrderedFaultBackend armed.store(true); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - auto result = DB::Cas::tests::OrderedFaultBackend::get(key, range); + auto result = DB::Cas::tests::OrderedFaultBackend::read(key, access); if (key == first_key && armed.exchange(false)) { first_completed->open(); - release_first->wait(); + release_first->wait("release_first"); } return result; } @@ -77,14 +94,11 @@ class BetweenRecoveryGetsBackend : public DB::Cas::tests::OrderedFaultBackend }; /// Identifies recovery's final authority read without changing the recovery implementation: after the -/// recovered-frontier CAS, its first checkpoint `GET` verifies that contribution and its second is the +/// recovered-frontier CAS, its first checkpoint read verifies that contribution and its second is the /// final authority read immediately preceding materialization. class FinalAuthorityBackend : public DB::Cas::tests::OrderedFaultBackend { public: - using DB::Cas::tests::OrderedFaultBackend::casPut; - using DB::Cas::tests::OrderedFaultBackend::get; - void armFinalAuthorityRead(String checkpoint_key_) { checkpoint_key = std::move(checkpoint_key_); @@ -94,20 +108,22 @@ class FinalAuthorityBackend : public DB::Cas::tests::OrderedFaultBackend armed.store(true); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - auto result = DB::Cas::tests::OrderedFaultBackend::get(key, range); + auto result = DB::Cas::tests::OrderedFaultBackend::read(key, access); if (armed.load() && checkpoint_cas_committed.load() && key == checkpoint_key && gets_after_checkpoint_cas.fetch_add(1) + 1 == 2) final_authority_returned.store(true); return result; } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - CasResult result = DB::Cas::tests::OrderedFaultBackend::casPut(key, bytes, expected, meta); - if (armed.load() && key == checkpoint_key && result.outcome == CasOutcome::Committed) + auto result = DB::Cas::tests::OrderedFaultBackend::write(key, bytes, expected_value, access); + /// A value is the store's committed incarnation; a `RawConflict` is a refused precondition. + if (armed.load() && key == checkpoint_key && result.has_value()) checkpoint_cas_committed.store(true); return result; } @@ -132,16 +148,17 @@ CasRequestBudget oneAttemptBudget() return budget; } -/// A ledger-level fixture keeps the real detached publisher but injects its already-public mount-fence -/// callback. That callback is the existing deterministic boundary after recovery materialized its +/// A ledger-level fixture keeps the real detached publisher but arms `CasRefLedger`'s recovery-install +/// test probe. That probe is the existing deterministic boundary after recovery materialized its /// result and before `installRecoveryResult`. class ManualDetachedLedger { public: ManualDetachedLedger() : backend(std::make_shared()) + , mount_requests(DB::Cas::tests::openRequestsForTest(backend)) , ledger( - backend, + mount_requests, layout, RefLedgerConfig{ .server_root_id = "test", @@ -153,18 +170,9 @@ class ManualDetachedLedger event_sink, oneAttemptBudget(), "test", - [] { return uint64_t{0}; }, [] { return uint64_t{1}; }, [] { return true; }, [] { return uint64_t{1}; }, - [this](uint64_t) - { - if (backend->finalAuthorityReturned() && !final_install_gate_claimed.exchange(true)) - { - final_install_reached.open(); - release_final_install.wait(); - } - }, [] { return uint64_t{0}; }, [] { return true; }, [](const String &, const String &, const std::optional &) {}, @@ -177,7 +185,27 @@ class ManualDetachedLedger {}, [](const RootNamespace &) {}) { - CasRefCatalog::initializeEmptyForNewPool(*backend, layout); + /// The engine's own retry pauses, on a clock the engine reads: one call reaches its retry + /// deadline against a latched fault with no real time passing. `setCasRetrySleepForTest` + /// installs the sleep on both `mount_requests` and the recovery retry loop. + auto clock = std::make_shared(); + mount_requests.setNowFnForTest(DB::Cas::tests::VirtualRetryClock::nowFnOf(clock)); + ledger.setCasRetrySleepForTest(DB::Cas::tests::VirtualRetryClock::sleepFnOf(clock)); + + CasOperation op = mount_requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + + /// The deterministic boundary a test pauses on: after recovery's final authority read and O(N) + /// materialization, immediately before a materialized result installs. A no-op for every test + /// that never arms `backend`'s final-authority read. + ledger.setRecoveryInstallProbeForTest([this] + { + if (backend->finalAuthorityReturned() && !final_install_gate_claimed.exchange(true)) + { + final_install_reached.open(); + release_final_install.wait("release_final_install"); + } + }); } std::function takeDetachedTask() @@ -203,6 +231,9 @@ class ManualDetachedLedger std::shared_ptr registry = std::make_shared(); Gate final_install_reached; Gate release_final_install; + /// The mount plane the ledger admits every request on. Declared before it, and never moved: the + /// ledger holds a reference to this member. + CasRequests mount_requests; CasRefLedger ledger; private: @@ -295,8 +326,13 @@ RefTxnId publishRef(CasRefLedger & ledger, const RootNamespace & ns, const Strin } /// Leaves the runtime in `NeedsRecovery` while its first background publisher is held after capture. -/// Releasing that publisher consumes the armed snapshot failure; zero backoff then makes settlement +/// Releasing that publisher meets the latched snapshot failure; zero backoff then makes settlement /// redispatch the real token-carrying publisher, whose first action is recovery of this exact runtime. +/// +/// Both faults are LATCHED and the engine's retry clock is virtual, because a write here must reach +/// its own retry deadline without committing: the engine reissues one logical write until that +/// deadline, so a counted fault the reissues outlive would let the call commit -- the parked publisher +/// would then succeed, nothing would redispatch it, and no caller would ever reach recovery. void preparePendingRecoveryPublisher( const PoolPtr & store, const std::shared_ptr & backend, @@ -305,6 +341,8 @@ void preparePendingRecoveryPublisher( const std::shared_ptr & release_first_publisher, String & ckpt_key) { + DB::Cas::tests::VirtualRetryClock::installOn(store); + auto capture_calls = std::make_shared>(0); store->setSnapshotAfterCaptureHookForTest( [capture_calls, first_publisher_captured, release_first_publisher] @@ -312,21 +350,23 @@ void preparePendingRecoveryPublisher( if (capture_calls->fetch_add(1) != 0) return; first_publisher_captured->open(); - release_first_publisher->wait(); + release_first_publisher->wait("release_first_publisher"); }); - backend->armPutFailure("_snap/", 1); + backend->armLatchedWriteFailure("_snap/"); ASSERT_NO_THROW(publishRef(store, ns, "ref_1", 1)); - first_publisher_captured->wait(); + first_publisher_captured->wait("first_publisher_captured"); - const auto life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); + const auto life = CasRefCatalog::lifeIfCataloged(catalog_op, store->layout(), ns); ASSERT_TRUE(life); ckpt_key = store->layout().refCkptKey(*life); /// The log lands before the checkpoint conflict, leaving a real unfrontiered durable transaction. - backend->armCasConflict(ckpt_key, 100); + backend->armLatchedWriteConflict(ckpt_key); EXPECT_ANY_THROW(store->dropRef(ns, "ref_1")); - backend->armCasConflict(ckpt_key, 0); + backend->armLatchedWriteConflict({}); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); } @@ -371,9 +411,9 @@ TEST(CASDetachedWork, DrainDoesNotReturnWhileWorkIsInFlight) ASSERT_TRUE(store->tryDispatchDetached([entered, release](DetachedStopToken) { entered->open(); - release->wait(); + release->wait("release"); })); - entered->wait(); + entered->wait("entered"); auto drain = std::async(std::launch::async, [&store] { return store->stopAndDrainDetachedWork(/*deadline_ms=*/60000); }); @@ -398,9 +438,9 @@ TEST(CASDetachedWork, ShutdownDoesNotReturnWhileWorkIsInFlight) ASSERT_TRUE(pool->tryDispatchDetached([entered, release](DetachedStopToken) { entered->open(); - release->wait(); + release->wait("release"); })); - entered->wait(); + entered->wait("entered"); auto done = std::async(std::launch::async, [&storage] { storage->shutdown(); }); awaitStopLatched(pool); @@ -423,10 +463,10 @@ TEST(CASDetachedWork, ImplicitDestructionDrains) ASSERT_TRUE(pool->tryDispatchDetached([entered, release, &finished](DetachedStopToken) { entered->open(); - release->wait(); + release->wait("release"); finished.store(true); })); - entered->wait(); + entered->wait("entered"); auto destroyed = std::async(std::launch::async, [&storage] { storage.reset(); }); awaitStopLatched(pool); @@ -462,9 +502,9 @@ TEST(CASDetachedWork, ExpiredDrainIncrementsTheTimeoutCounter) ASSERT_TRUE(pool->tryDispatchDetached([entered, release](DetachedStopToken) { entered->open(); - release->wait(); + release->wait("release"); })); - entered->wait(); + entered->wait("entered"); const auto before = ProfileEvents::global_counters[ProfileEvents::CASDetachedWorkDrainTimeouts] .load(std::memory_order_relaxed); @@ -490,10 +530,10 @@ TEST(CASDetachedWork, TaskObservesStopTokenOnceLatched) ASSERT_TRUE(store->tryDispatchDetached([entered, release, &saw_stop](DetachedStopToken token) { entered->open(); - release->wait(); + release->wait("release"); saw_stop.store(token.stopping()); })); - entered->wait(); + entered->wait("entered"); auto drain = std::async(std::launch::async, [&store] { return store->stopAndDrainDetachedWork(/*deadline_ms=*/60000); }); @@ -574,7 +614,7 @@ TEST(CASDetachedWork, SettlementSurvivesAThrowingErrorHandler) /// Arm the fault so the publisher's own PUT fails and its `catch` is entered. Use the same arming /// call the snapshot-ordering suite uses against this backend. - backend->armPutFailure("_snap/", 1); + backend->armWriteFailure("_snap/", 1); ASSERT_NO_THROW(publishRef(store, ns, "ref_1", 1)); store->waitForSnapshotPublishSettleForTest(ns); @@ -609,9 +649,11 @@ TEST(CASDetachedWork, StopWakesRecoveryBackoffSleep) }); /// Exhaust one checkpoint publication inside recovery so the outer retry loop enters backoff. - backend->armCasConflict(ckpt_key, 100); + /// Latched: the publication must meet the refusal on every reissue, or it commits and recovery + /// never reaches its backoff. + backend->armLatchedWriteConflict(ckpt_key); release_first_publisher->open(); - sleeping->wait(); + sleeping->wait("sleeping"); const bool drained = store->stopAndDrainDetachedWork(/*deadline_ms=*/5000); release_sleep->store(true); @@ -635,7 +677,7 @@ TEST(CASDetachedWork, StopWakesAConcurrentRecoveryWaiter) if (!recovery_hook_armed->load()) return; recovery_entered->open(); - release_recovery->wait(); + release_recovery->wait("release_recovery"); }; auto store = openPublishingPool(backend, config); const RootNamespace ns{"srv1/concurrent_recovery"}; @@ -650,7 +692,10 @@ TEST(CASDetachedWork, StopWakesAConcurrentRecoveryWaiter) /// drain below waits only for the publisher parked behind it. recovery_hook_armed->store(true); auto first_recovery = std::async(std::launch::async, [&store, &ns] { store->listRefs(ns); }); - recovery_entered->wait(); + /// Declared AFTER the future, so it is destroyed BEFORE it: `first_recovery`'s destructor joins + /// the recovery thread, which cannot leave the hook until this gate is open. + GateOpenedOnExit recovery_released{*release_recovery}; + recovery_entered->wait("recovery_entered"); release_first_publisher->open(); const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); @@ -688,7 +733,7 @@ TEST(CASDetachedWork, StopIsObservedBeforeTheFirstRecoveryRequest) if (!recovery_hook_armed->load()) return; at_boundary->open(); - release->wait(); + release->wait("release"); }; auto store = openPublishingPool(backend, config); const RootNamespace ns{"srv1/pre_first_request"}; @@ -701,7 +746,7 @@ TEST(CASDetachedWork, StopIsObservedBeforeTheFirstRecoveryRequest) recovery_hook_armed->store(true); release_first_publisher->open(); - at_boundary->wait(); + at_boundary->wait("at_boundary"); const uint64_t gets_before = backend->getTotal(); auto drain = std::async(std::launch::async, @@ -769,14 +814,16 @@ TEST(CASDetachedWork, StopBetweenSnapshotBaseRequestsPreventsThePredecessorGet) preparePendingRecoveryPublisher( store, backend, ns, first_publisher_captured, release_first_publisher, ckpt_key); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns).value(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(catalog_op, store->layout(), ns).value(); const String base_log_key = store->layout().refLogKey(life, base_id); const String predecessor_key = store->layout().refLogKey(life, predecessor_seal_id); auto first_get_completed = std::make_shared(); auto release_first_get = std::make_shared(); backend->armBetweenGets(base_log_key, first_get_completed, release_first_get); release_first_publisher->open(); - first_get_completed->wait(); + first_get_completed->wait("first_get_completed"); const uint64_t predecessor_gets_before = backend->getCount(predecessor_key); auto drain = std::async(std::launch::async, @@ -798,11 +845,15 @@ TEST(CASDetachedWork, StopAfterRecoveryMaterializationPreventsFinalInstall) const RootNamespace ns{"srv1/stop_before_recovery_install"}; ASSERT_NO_THROW(publishRef(fixture.ledger, ns, "ref_1", 1)); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*fixture.backend, fixture.layout, ns).value(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(fixture.backend); + CasOperation catalog_op = catalog_requests.admit(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(catalog_op, fixture.layout, ns).value(); const String ckpt_key = fixture.layout.refCkptKey(life); - fixture.backend->armCasConflict(ckpt_key, 100); + /// Latched: the checkpoint write is reissued until its retry window closes, so a counted refusal + /// the reissues outlive would let this drop commit and leave the lane Ready. + fixture.backend->armLatchedWriteConflict(ckpt_key); EXPECT_ANY_THROW(fixture.ledger.dropRef(ns, "ref_1")); - fixture.backend->armCasConflict(ckpt_key, 0); + fixture.backend->armLatchedWriteConflict({}); ASSERT_EQ(fixture.ledger.laneStateForTest(ns), RefLaneState::NeedsRecovery); auto detached_publisher = fixture.takeDetachedTask(); @@ -813,8 +864,11 @@ TEST(CASDetachedWork, StopAfterRecoveryMaterializationPreventsFinalInstall) { task(DetachedStopToken(fixture.registry)); }); + /// Released on every exit, before `running` is joined: the task parks behind this gate, and a + /// failed assertion that left it shut deadlocked the join. + GateOpenedOnExit final_install_released{fixture.release_final_install}; - fixture.final_install_reached.wait(); + fixture.final_install_reached.wait("final_install_reached"); fixture.latchStop(); fixture.release_final_install.open(); EXPECT_NO_THROW(running.get()); diff --git a/src/Disks/tests/gtest_cas_encoding_pins.cpp b/src/Disks/tests/gtest_cas_encoding_pins.cpp index 7d7808d70ecc..c03d3c4ee879 100644 --- a/src/Disks/tests/gtest_cas_encoding_pins.cpp +++ b/src/Disks/tests/gtest_cas_encoding_pins.cpp @@ -132,7 +132,7 @@ TEST(CASEncodingPins, SourceEdgeRunLines) condemned.source_id = UInt128(0); condemned.marker = RunMarker::Condemned; condemned.delete_pending = true; - condemned.token = Token{.value = "token", .type = TokenType::ETag}; + condemned.token = PersistedIncarnation{"etag", "token"}; condemned.size = 9; condemned.condemn_round = 7; condemned.marker_confirmed = true; @@ -170,7 +170,7 @@ TEST(CASWireCutDeltas, ActiveCasRunRow) TEST(CASWireCutDeltas, CondemnedCasRunRow) { - SourceEdgeRecord record{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(3))}, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = true, .token = Token{"token", TokenType::ETag}, .size = 9, .condemn_round = 7, .marker_confirmed = true}; + SourceEdgeRecord record{.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(3))}, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = true, .token = PersistedIncarnation{"etag", "token"}, .size = 9, .condemn_round = 7, .marker_confirmed = true}; WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); writer.append(record); @@ -210,7 +210,7 @@ TEST(CASWireCutDeltas, InlinePartManifestEntry) TEST(CASWireCutDeltas, GcOutcomesRow) { - OutcomeLog log{{OutcomeEntry{ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, Token{"t", TokenType::ETag}, OutcomeKind::Deleted}}}; + OutcomeLog log{{OutcomeEntry{ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, PersistedIncarnation{"etag", "t"}, OutcomeKind::Deleted}}}; /// This literal is the pre-cut baseline this delta is measured against. const String old_bytes = "{\"k\":\"blob\",\"ha\":\"ch128\",\"h\":\"00000000000000000000000000000004\",\"tt\":\"etag\",\"tv\":\"t\",\"oc\":\"deleted\"}\n"; expectDelta(old_bytes, lineAt(encodeOutcomeLog(log), 1), 26); diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index 82920acb177c..519aa0474cad 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -38,17 +38,19 @@ namespace class RenewalEventBackend final : public InMemoryBackend { public: - using InMemoryBackend::get; - using InMemoryBackend::putOverwrite; - - bool throw_before_next_overwrite = false; - bool throw_nonretryable_next_overwrite = false; - bool vanish_on_next_overwrite = false; + bool throw_before_next_write = false; + bool throw_nonretryable_next_write = false; + bool vanish_on_next_write = false; + /// Runs just before an armed fault throws. The engine draws its inter-attempt backoff randomly and + /// admits the reissue against that drawn duration, so a test that needs the ambiguity refused + /// rather than reissued has to move the injected clock here -- from inside the attempt, the only + /// point between admission and the resolve read a test can reach. + std::function before_throw; void armResolveProbe() { std::lock_guard lock(resolve_mutex); - observe_next_get = true; + observe_next_read = true; resolve_started = false; } @@ -58,40 +60,50 @@ class RenewalEventBackend final : public InMemoryBackend return resolve_started; } - std::optional get(const String & key, Range range) override + /// The engine settles an ambiguous write by reading the key back, so the observation belongs on the + /// READ PRIMITIVE -- the resolve read never reaches the legacy `get`. + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { { std::lock_guard lock(resolve_mutex); - if (observe_next_get) + if (observe_next_read) { resolve_started = true; - observe_next_get = false; + observe_next_read = false; } } - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - PutResult putOverwrite( + /// The faults sit on the WRITE PRIMITIVE, and only on a CONDITIONAL write: a lease renewal is a + /// replace, so the pool's own create-if-absent writes must not consume a one-shot fault. + std::expected write( const String & key, const String & bytes, - const Token & expected, - const ObjectMeta & meta) override + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - if (std::exchange(vanish_on_next_overwrite, false)) + if (!expected_value) + return InMemoryBackend::write(key, bytes, expected_value, access); + if (std::exchange(vanish_on_next_write, false)) { - (void)InMemoryBackend::deleteExact(key, expected); - return {PutOutcome::PreconditionFailed, {}}; + (void)InMemoryBackend::remove(key, *expected_value, access); + return std::unexpected(RawConflict{}); } - if (std::exchange(throw_nonretryable_next_overwrite, false)) + if (std::exchange(throw_nonretryable_next_write, false)) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected deterministic renewal rejection"); - if (std::exchange(throw_before_next_overwrite, false)) + if (std::exchange(throw_before_next_write, false)) + { + if (before_throw) + before_throw(); throw Poco::TimeoutException("injected renewal timeout before commit"); - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + } + return InMemoryBackend::write(key, bytes, expected_value, access); } private: std::mutex resolve_mutex; - bool observe_next_get = false; + bool observe_next_read = false; bool resolve_started = false; }; @@ -205,77 +217,81 @@ TEST(CASEvent, WatermarkRenewEventsAreBoundedAndComplete) auto store = openRenewalEventPool(backend, boot_ms); store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); - backend->throw_before_next_overwrite = true; + backend->throw_before_next_write = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); const std::vector renewals = watermarkRenewEvents(events); - ASSERT_EQ(renewals.size(), 2u); - EXPECT_EQ(renewals[0].outcome, "retrying"); - EXPECT_EQ(renewals[1].outcome, "recovered"); - EXPECT_EQ(renewals[0].detail.at("attempts_sent"), "1"); - EXPECT_EQ(renewals[1].detail.at("attempts_sent"), "2"); + /// ONE event per logical renewal, whatever the physical attempts cost: the engine owns its own + /// reissues, and the terminal event carries their count rather than announcing each one. + ASSERT_EQ(renewals.size(), 1u); + EXPECT_EQ(renewals[0].outcome, "recovered"); + EXPECT_EQ(renewals[0].detail.at("attempts_sent"), "2"); + EXPECT_EQ(renewals[0].detail.at("classification"), "committed_after_retry"); EXPECT_EQ(renewals[0].detail.at("server_root_id"), "test"); EXPECT_EQ(renewals[0].detail.at("writer_epoch"), std::to_string(store->writerEpoch())); EXPECT_EQ(renewals[0].detail.at("seq"), "2"); - EXPECT_EQ(renewals[0].detail.at("write_attempt_id"), renewals[1].detail.at("write_attempt_id")); EXPECT_FALSE(renewals[0].detail.at("write_attempt_id").empty()); EXPECT_LT(renewals[0].detail.at("write_attempt_id").size(), 32u); - - for (const CasEvent & event : renewals) - { - for (const String & key : { - "server_root_id", - "writer_epoch", - "seq", - "write_attempt_id", - "attempts_sent", - "elapsed_ms", - "remaining_confirmed_budget_ms", - "unresolved_reason", - "deadline_source", - "stop_cause", - "classification"}) - EXPECT_TRUE(event.detail.contains(key)) << "missing detail key " << key; - } + /// Both attempts sent the same body, so the event names the id the lease actually landed with -- + /// a reissue that minted a fresh id would leave the two disagreeing. + const MountLease landed = decodeMountLease(backend->get(store->layout().mountKey("test"))->bytes); + EXPECT_EQ(renewals[0].detail.at("write_attempt_id"), u128ToHex(landed.write_attempt_id).substr(0, 12)); + + for (const String & key : { + "server_root_id", + "writer_epoch", + "seq", + "write_attempt_id", + "attempts_sent", + "elapsed_ms", + "remaining_confirmed_budget_ms", + "classification"}) + EXPECT_TRUE(renewals[0].detail.contains(key)) << "missing detail key " << key; } -TEST(CASEvent, FirstAmbiguityIsVisibleWhileResolveIsInFlight) +/// An attempt that spends the lease it was admitted under must not START the resolving read. That read +/// is the only thing that can prove the ambiguous attempt landed, and issuing it past the lease-safe +/// bound would be a request made without the authority it was admitted under -- so the renewal reports +/// the deadline that refused it instead of resolving anything. +TEST(CASEvent, AnAmbiguityPastTheLeaseBoundNeverStartsTheResolvingRead) { auto backend = std::make_shared(); uint64_t boot_ms = 100; - std::atomic retrying_events{0}; + std::vector events; auto store = openRenewalEventPool( backend, boot_ms, renewalEventBudget(), "renewal-inflight-ambiguity"); - store->setEventSink([&](CasEvent event) - { - if (event.type == CasEventType::WatermarkRenew && event.outcome == "retrying") - { - retrying_events.fetch_add(1); - /// The diagnostic callback may consume the remaining recovery budget. The controller - /// must re-check its absolute deadline before starting the resolving GET. - boot_ms = 1081; - } - }); + store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); - backend->throw_before_next_overwrite = true; + /// The lease was anchored at 100 with a 1000 ms TTL, so the fence expires at 1100 and holds a 20 ms + /// safety margin. At 1081 only 19 ms remain, and admission refuses the resolve read. + backend->before_throw = [&] { boot_ms = 1'081; }; + backend->throw_before_next_write = true; backend->armResolveProbe(); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); - EXPECT_EQ(retrying_events.load(), 1u) - << "first ambiguity must be externally visible before the pre-resolve deadline gate"; EXPECT_FALSE(backend->resolveStarted()) - << "a diagnostic sink that exhausts the budget must prevent the resolving GET from starting"; - EXPECT_EQ(retrying_events.load(), 1u) << "retrying delivery is bounded to the first ambiguity"; + << "an attempt that consumed the lease must not start the resolving read"; + const std::vector renewals = watermarkRenewEvents(events); + ASSERT_EQ(renewals.size(), 1u); + EXPECT_EQ(renewals[0].outcome, "failed"); + EXPECT_EQ(renewals[0].detail.at("attempts_sent"), "1"); + EXPECT_EQ(renewals[0].detail.at("classification"), "external_lease_deadline"); } +/// Ten renewals nested through each other's conflict sinks, against an eight-slot observation stack. +/// The two calls beyond the stack get no rich event -- and must still report their own physical attempt +/// count, which rides the write result rather than the suppressed observation. TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) { constexpr size_t depth = 10; + constexpr size_t observation_stack_capacity = 8; std::array, depth> backends; std::array, depth> layouts; + std::array, depth> planes; std::array, depth> keepers; std::array server_root_ids; std::array sinks; + std::array renew_events{}; uint64_t wall_ms = 100; uint64_t boot_ms = 100; std::optional deepest_result; @@ -284,16 +300,7 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) renew_at = [&](size_t index) { configureMountRenewObservability(&server_root_ids[index], &sinks[index], /*deferred=*/false); - MountRenewResult result = keepers[index]->renew( - CasRequestBudget{ - .attempt_timeout_ms = 10, - .operation_deadline_ms = 500, - .max_attempts = 1, - .lease_safety_margin_ms = 0, - .retry_initial_backoff_ms = 0, - .retry_max_backoff_ms = 0, - }, - MountRenewOperationEnvironment{}); + MountRenewResult result = keepers[index]->renew(MountRenewOperationEnvironment{}); reportMountRenewCompletion(result); return result; }; @@ -305,6 +312,8 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) server_root_ids[index] = fmt::format("deep-{}", index); sinks[index] = [&, index](CasEvent event) { + if (event.type == CasEventType::WatermarkRenew) + ++renew_events[index]; if (event.type == CasEventType::MountConflict && index + 1 < depth) { MountRenewResult child_result = renew_at(index + 1); @@ -312,8 +321,14 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) deepest_result = std::move(child_result); } }; + /// One open-fence plane per keeper, on the same injected clock the keeper anchors its lease + /// against, and with a sleep that advances it: the deepest renewal reissues, and no unit test + /// may serve the engine's jittered backoff for real. + planes[index] = std::make_unique( + backends[index], Fence::open(), [&] { return boot_ms; }, [&](uint64_t ms) { boot_ms += ms; }); keepers[index] = std::make_unique( - backends[index], + *planes[index], + *planes[index], *layouts[index], server_root_ids[index], UInt128(index + 1), @@ -338,14 +353,21 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) PutOutcome::Done); } } - backends.back()->throw_nonretryable_next_overwrite = true; + /// The deepest slot is the only one nobody took, so its renewal can recover: the attempt is lost + /// before its answer, the resolve read finds the precondition intact, and the reissue commits. + backends.back()->throw_before_next_write = true; const MountRenewResult outer_result = renew_at(0); EXPECT_EQ(outer_result.outcome, MountRenewOutcome::Terminal); ASSERT_TRUE(deepest_result.has_value()); - EXPECT_EQ(deepest_result->outcome, MountRenewOutcome::Terminal); - EXPECT_EQ(deepest_result->diagnostics.attempts_sent, 1u) + EXPECT_EQ(deepest_result->outcome, MountRenewOutcome::Committed); + EXPECT_EQ(deepest_result->attempts_sent, 2u) << "nesting beyond the rich-event stack must not erase physical attempt truth"; + + for (size_t index = 0; index < depth; ++index) + EXPECT_EQ(renew_events[index], index < observation_stack_capacity ? 1u : 0u) + << "renewal " << index << " is " << (index < observation_stack_capacity ? "on" : "beyond") + << " the observation stack"; } TEST(CASEvent, WatermarkRenewSinkFailureCannotChangeOutcome) @@ -361,23 +383,28 @@ TEST(CASEvent, WatermarkRenewSinkFailureCannotChangeOutcome) throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected renewal event sink failure"); }); - backend->throw_before_next_overwrite = true; + backend->throw_before_next_write = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); EXPECT_EQ(decodeMountLease(backend->get(mount_key)->bytes).seq, seq_before + 1); EXPECT_TRUE(store->mayMutate()); } +/// The two terminal endings a renewal reaches without ever settling its write: the store refusing it +/// outright, and the lease refusing to admit it. There is no attempt-count ending -- the engine bounds a +/// write by time, never by a number of tries -- and the deadline ending that DOES send an attempt first +/// is `AnAmbiguityPastTheLeaseBoundNeverStartsTheResolvingRead`. TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) { - const auto one_failed_event = [](const std::vector & events) -> CasEvent + const auto one_failed_event = [](const std::vector & events) -> std::optional { const std::vector renewals = watermarkRenewEvents(events); const auto failed = std::find_if(renewals.begin(), renewals.end(), [](const CasEvent & event) { return event.outcome == "failed"; }); - EXPECT_NE(failed, renewals.end()); - return failed == renewals.end() ? CasEvent{} : *failed; + if (failed == renewals.end()) + return std::nullopt; + return *failed; }; { @@ -386,31 +413,14 @@ TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) std::vector events; auto store = openRenewalEventPool(backend, boot_ms, renewalEventBudget(), "renewal-deterministic-details"); store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); - backend->throw_nonretryable_next_overwrite = true; - - EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); - const CasEvent failed = one_failed_event(events); - EXPECT_EQ(failed.detail.at("attempts_sent"), "1"); - EXPECT_EQ(failed.detail.at("unresolved_reason"), "not_unresolved"); - EXPECT_EQ(failed.detail.at("stop_cause"), "continue"); - EXPECT_EQ(failed.detail.at("classification"), "deterministic_failure"); - } - - { - auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; - CasRequestBudget budget = renewalEventBudget(); - budget.max_attempts = 1; - auto store = openRenewalEventPool(backend, boot_ms, budget, "renewal-exhausted-details"); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); - backend->throw_before_next_overwrite = true; + backend->throw_nonretryable_next_write = true; EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); - const CasEvent failed = one_failed_event(events); - EXPECT_EQ(failed.detail.at("attempts_sent"), "1"); - EXPECT_EQ(failed.detail.at("unresolved_reason"), "attempts_exhausted"); - EXPECT_EQ(failed.detail.at("classification"), "attempts_exhausted"); + const std::optional failed = one_failed_event(events); + ASSERT_TRUE(failed.has_value()) << "the store's refusal must reach the event log"; + /// A deterministic failure reaches the keeper as the exception the engine refuses to reissue, + /// and an exception carries no attempt count -- so the classification is all this ending states. + EXPECT_EQ(failed->detail.at("classification"), "deterministic_failure"); } { @@ -419,14 +429,15 @@ TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) std::vector events; auto store = openRenewalEventPool(backend, boot_ms, renewalEventBudget(), "renewal-deadline-details"); store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); - boot_ms = 1071; + /// The lease was anchored at 100 with a 1000 ms TTL and holds a 20 ms safety margin, so 1090 + /// leaves 10 ms of it and admission refuses the renewal before its first attempt. + boot_ms = 1090; EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); - const CasEvent failed = one_failed_event(events); - EXPECT_EQ(failed.detail.at("attempts_sent"), "0"); - EXPECT_EQ(failed.detail.at("unresolved_reason"), "no_attempt_sent"); - EXPECT_EQ(failed.detail.at("deadline_source"), "external_lease_safety"); - EXPECT_EQ(failed.detail.at("classification"), "external_lease_deadline"); + const std::optional failed = one_failed_event(events); + ASSERT_TRUE(failed.has_value()) << "the refused admission must reach the event log"; + EXPECT_EQ(failed->detail.at("attempts_sent"), "0"); + EXPECT_EQ(failed->detail.at("classification"), "external_lease_deadline"); } } @@ -446,16 +457,17 @@ TEST(CASEvent, ReentrantRenewalSinkPreservesOuterObservationIdentity) store->renewWatermarkOnce(); }); - backend->throw_before_next_overwrite = true; + backend->throw_before_next_write = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); ASSERT_TRUE(reentered); - ASSERT_EQ(events.size(), 2u); - EXPECT_EQ(events[0].outcome, "retrying"); - EXPECT_EQ(events[1].outcome, "recovered"); + /// The nested renewal commits on its first attempt, which is silent, so the outer recovery is the + /// only event -- and it still names the outer renewal's own seq while the durable lease has already + /// moved past it. An observation the nested call reused would report seq 3 here. + ASSERT_EQ(events.size(), 1u); + EXPECT_EQ(events[0].outcome, "recovered"); + EXPECT_EQ(events[0].detail.at("attempts_sent"), "2"); EXPECT_EQ(events[0].detail.at("seq"), "2"); - EXPECT_EQ(events[1].detail.at("seq"), events[0].detail.at("seq")); - EXPECT_EQ(events[1].detail.at("write_attempt_id"), events[0].detail.at("write_attempt_id")); EXPECT_EQ(decodeMountLease(backend->get(store->layout().mountKey("test"))->bytes).seq, 3u) << "the nested first-attempt success must run without replacing the outer observation"; } @@ -469,10 +481,8 @@ TEST(CASEvent, PreCompletionConflictReentrancyPreservesOuterTerminalObservation) auto outer_backend = std::make_shared(); uint64_t outer_boot_ms = 100; - CasRequestBudget outer_budget = renewalEventBudget(); - outer_budget.max_attempts = 1; auto outer = openRenewalEventPool( - outer_backend, outer_boot_ms, outer_budget, "renewal-reentrant-outer", "outer"); + outer_backend, outer_boot_ms, renewalEventBudget(), "renewal-reentrant-outer", "outer"); std::vector outer_events; bool reentered = false; outer->setEventSink([&](CasEvent event) @@ -482,18 +492,18 @@ TEST(CASEvent, PreCompletionConflictReentrancyPreservesOuterTerminalObservation) inner->renewWatermarkOnce(); }); - outer_backend->vanish_on_next_overwrite = true; + /// The inner renewal loses its first attempt's answer and recovers on a reissue, so it has its own + /// identity and its own classification to report. Both must stay off the outer observation. + inner_backend->throw_before_next_write = true; + outer_backend->vanish_on_next_write = true; EXPECT_THROW(outer->renewWatermarkOnce(), DB::Exception); ASSERT_TRUE(reentered); const std::vector renewals = watermarkRenewEvents(outer_events); - ASSERT_EQ(renewals.size(), 2u); - EXPECT_EQ(renewals[0].outcome, "retrying"); - EXPECT_EQ(renewals[1].outcome, "failed"); + ASSERT_EQ(renewals.size(), 1u); + EXPECT_EQ(renewals[0].outcome, "failed"); EXPECT_EQ(renewals[0].detail.at("server_root_id"), "outer"); - EXPECT_EQ(renewals[1].detail.at("server_root_id"), "outer"); - EXPECT_EQ(renewals[1].detail.at("write_attempt_id"), renewals[0].detail.at("write_attempt_id")); - EXPECT_EQ(renewals[1].detail.at("classification"), "vanished"); + EXPECT_EQ(renewals[0].detail.at("classification"), "vanished"); } /// Round-B opt §6: `emitEvent` takes the event BY VALUE (moved-through, not `const &`), so a diff --git a/src/Disks/tests/gtest_cas_fence_generation.cpp b/src/Disks/tests/gtest_cas_fence_generation.cpp index 105925ea33d8..b298c6144029 100644 --- a/src/Disks/tests/gtest_cas_fence_generation.cpp +++ b/src/Disks/tests/gtest_cas_fence_generation.cpp @@ -43,13 +43,14 @@ namespace class TripOnHeadBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy overload the primitive override below would otherwise hide. using InMemoryBackend::head; - HeadResult head(const String & key) override + /// The side effect sits on the HEAD primitive, which is the only path any observation takes. + std::optional head(const String & key, TransportAccess & access) override { if (trigger) std::exchange(trigger, {})(); - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } std::function trigger; @@ -61,32 +62,35 @@ class TripOnHeadBackend final : public InMemoryBackend class TripOnSecondHeadBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy overloads the primitive overrides below would otherwise hide. using InMemoryBackend::head; - using Backend::putIfAbsent; - HeadResult head(const String & key) override + std::optional head(const String & key, TransportAccess & access) override { ++head_calls; if (head_calls == 2 && trigger) std::exchange(trigger, {})(); - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + /// A refused precondition in its value form: nothing was written, and the caller settles what is + /// at the key by reading -- which is the second HEAD this double trips the fence on. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { if (fail_first_put) { fail_first_put = false; - return PutResult{.outcome = PutOutcome::PreconditionFailed, .token = {}}; + return std::unexpected(RawConflict{}); } - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } int head_calls = 0; - /// Default false: `Pool::open`'s own capability probe issues `putIfAbsent` calls before the test - /// gets to arm this, and those must succeed normally. The test flips this to `true` only right - /// before driving the write it actually targets. + /// Default false: `Pool::open`'s own capability probe issues writes before the test gets to arm + /// this, and those must succeed normally. The test flips this to `true` only right before driving + /// the write it actually targets. bool fail_first_put = false; std::function trigger; }; @@ -152,7 +156,7 @@ PartWriteTxnPtr precommittedBuildForBlob( class BlobPublicationFenceBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy overload the primitive override below would otherwise hide. using InMemoryBackend::head; enum class TripPoint : uint8_t { @@ -160,18 +164,20 @@ class BlobPublicationFenceBackend final : public InMemoryBackend AfterPublication, }; - HeadResult head(const String & key) override + /// Both seams sit on the transport primitives: a writer's mandatory HEAD and its publication both + /// reach the store through them. + std::optional head(const String & key, TransportAccess & access) override { - const HeadResult result = InMemoryBackend::head(key); + const std::optional result = InMemoryBackend::head(key, access); if (key == watched_key && trip_point == TripPoint::OnHead && trigger) std::exchange(trigger, {})(); return result; } - void publishBlob(const BlobPublishRequest & request) override + void publish(const BlobPublishRequest & request, TransportAccess & access) override { ++publish_calls; - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); if (request.destination_key == watched_key && trip_point == TripPoint::AfterPublication && trigger) std::exchange(trigger, {})(); } @@ -267,7 +273,10 @@ TEST(CASFenceGeneration, BlobPublicationHeadTripAndRearmCannotAdoptNewFenceGener EXPECT_EQ(backend->publish_calls, 0u); EXPECT_FALSE(backend->head(backend->watched_key).exists); - EXPECT_EQ(loadMeta(*backend, store->layout(), ref), std::nullopt) + /// The mount is live again under a FRESH generation, so this read is admitted where the stale + /// operation's writes were not. + CasOperation probe = store->mountRequests().admit(); + EXPECT_FALSE(loadMeta(probe, store->layout(), ref).has_value()) << "the stale operation must not reconcile freshness metadata after trip-and-rearm"; EXPECT_EQ(build->dependencyProof(ref), std::nullopt); } @@ -310,8 +319,11 @@ TEST(CASFenceGeneration, PlainObjectPutAbortsWhenFenceTripsBetweenAdmissionAndDu store->putNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "somefile", "hello"); }); - /// No durable write ever landed -- assert via the Emulated backend listing. - EXPECT_TRUE(store->listNamespaceFiles(DB::Cas::tests::fixture::fixtureLife(ns)).empty()); + /// No durable write ever landed. Asserted through the RAW backend: every request the pool issues + /// is admitted under the mount fence, which this test has just tripped, so a read through the pool + /// would report that refusal rather than what the store holds. + EXPECT_TRUE(backend->list(store->layout().namespaceFilesPrefix( + DB::Cas::tests::fixture::fixtureLife(ns)), "", 100).keys.empty()); } /// `casRemoveObject`'s delete sibling, same shape: the fence trips between admission and the durable @@ -333,10 +345,12 @@ TEST(CASFenceGeneration, PlainObjectRemoveAbortsWhenFenceTripsBetweenAdmissionAn store->removeNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "victim"); }); - /// The durable delete never ran, so the object survives; reads are not fence-gated. - const auto still_there = store->getNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "victim"); + /// The durable delete never ran, so the object survives -- read raw, since a read through the pool + /// is admitted under the fence this test has tripped and would report that refusal instead. + const auto still_there = backend->get(store->layout().namespaceFileKey( + DB::Cas::tests::fixture::fixtureLife(ns), "victim")); ASSERT_TRUE(still_there.has_value()); - EXPECT_EQ(*still_there, "still here"); + EXPECT_EQ(still_there->bytes, "still here"); } /// The fence re-check must run before EVERY conditional-retry iteration, not just the first attempt @@ -360,7 +374,8 @@ TEST(CASFenceGeneration, PlainObjectPutRechecksFenceOnEveryRetryIterationNotJust }); EXPECT_EQ(backend->head_calls, 2); - EXPECT_TRUE(store->listNamespaceFiles(DB::Cas::tests::fixture::fixtureLife(ns)).empty()); + EXPECT_TRUE(backend->list(store->layout().namespaceFilesPrefix( + DB::Cas::tests::fixture::fixtureLife(ns)), "", 100).keys.empty()); } /// (b) The S3-native staging-buffer finalize: the fence trips AFTER the buffer is constructed diff --git a/src/Disks/tests/gtest_cas_forget.cpp b/src/Disks/tests/gtest_cas_forget.cpp index 571632bdfb4d..c73417fb15c1 100644 --- a/src/Disks/tests/gtest_cas_forget.cpp +++ b/src/Disks/tests/gtest_cas_forget.cpp @@ -76,38 +76,41 @@ void fenceOutMount(DB::Cas::Backend & backend, const String & mount_key) DB::Cas::PutOutcome::Done); } -/// A Backend decorator whose head/get/list throw an untyped transport error while `fail` is armed — so a -/// self-remount attempt verdicts `StayTransient` (fast, no lease-expiry wait) and the remount loop keeps -/// spinning. Starts DISARMED so `Pool::open` succeeds. Mirrors gtest_cas_lifecycle_condition.cpp's decorator. +/// A Backend decorator whose reads, heads and lists throw an untyped transport error while `fail` is +/// armed — so a self-remount attempt verdicts `StayTransient` (fast, no lease-expiry wait) and the remount +/// loop keeps spinning. Starts DISARMED so `Pool::open` succeeds. Mirrors +/// gtest_cas_lifecycle_condition.cpp's decorator. class ToggleableTransportFaultBackend final : public DB::Cas::InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using DB::Cas::InMemoryBackend::head; - using DB::Cas::InMemoryBackend::list; - using Backend::get; - using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - - DB::Cas::HeadResult head(const String & key) override + /// Unhide the LEGACY convenience overloads that the primitive overrides below would otherwise hide. + using Backend::head; + using Backend::list; + + /// The faults sit on the TRANSPORT PRIMITIVES, because that is where every caller reaches the store: + /// the lifecycle gate probes `_pool_meta` through `probeSentinelRaw`, which speaks only these. A + /// legacy caller still reaches the fault, through the forwarder, so arming it here covers both + /// surfaces rather than only one. + std::optional head(const String & key, DB::Cas::TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } - std::optional get(const String & key, DB::Cas::Range range) override + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - DB::Cas::ListPage list(const String & prefix, const String & cursor, size_t limit) override + + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } std::atomic fail{false}; diff --git a/src/Disks/tests/gtest_cas_fsck.cpp b/src/Disks/tests/gtest_cas_fsck.cpp index 41475abdb98c..b1d1696ad0d5 100644 --- a/src/Disks/tests/gtest_cas_fsck.cpp +++ b/src/Disks/tests/gtest_cas_fsck.cpp @@ -9,6 +9,10 @@ #include #include #include "cas_test_helpers.h" +#include "config.h" +#if USE_AWS_S3 +#include +#endif #include #include @@ -45,8 +49,6 @@ ManifestRef ref(uint64_t seq, uint64_t inst) class RepublishOnListBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; void armOnFirstList(String prefix, std::function mutation) { std::lock_guard lock(arm_mutex); @@ -54,7 +56,7 @@ class RepublishOnListBackend : public InMemoryBackend pending_mutation = std::move(mutation); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { std::function to_run; { @@ -67,7 +69,7 @@ class RepublishOnListBackend : public InMemoryBackend } if (to_run) to_run(); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } private: @@ -92,7 +94,7 @@ class MutateOnFirstGetBackend : public InMemoryBackend pending_mutation = std::move(mutation); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { std::function to_run; { @@ -105,7 +107,7 @@ class MutateOnFirstGetBackend : public InMemoryBackend } if (to_run) to_run(); - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } private: @@ -127,17 +129,15 @@ enum class FsckListingMode : uint8_t class FsckListingBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; void distort(String prefix_, FsckListingMode mode_) { prefix = std::move(prefix_); mode = mode_; } - ListPage list(const String & listed_prefix, const String & cursor, size_t limit) override + RawListPage list(const String & listed_prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = InMemoryBackend::list(listed_prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(listed_prefix, cursor, limit, access); if (listed_prefix != prefix) return page; if (mode == FsckListingMode::Empty) @@ -154,8 +154,12 @@ class FsckListingBackend : public InMemoryBackend FsckListingMode mode = FsckListingMode::Full; }; +#if USE_AWS_S3 /// Fail one exact GET without disturbing LIST or any other object read. This keeps the checkpoint /// authority stable while proving that fsck distinguishes a transport failure from durable corruption. +/// An access denial is the class the request engine surfaces on the first attempt instead of reissuing +/// (`InMemoryBackend::refreshCredentials` answers false by default), so the failure needs no retry +/// budget and the caller sees the injected message unchanged. class FailExactGetBackend : public InMemoryBackend { public: @@ -164,16 +168,17 @@ class FailExactGetBackend : public InMemoryBackend key = std::move(key_); } - std::optional get(const String & requested_key, Range range) override + std::optional read(const String & requested_key, TransportAccess & access) override { if (requested_key == key) - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected exact GET failure"); - return InMemoryBackend::get(requested_key, range); + throw DB::S3Exception("injected access denial on exact GET", Aws::S3::S3Errors::ACCESS_DENIED); + return InMemoryBackend::read(requested_key, access); } private: String key; }; +#endif /// Publish the exact `_ckpt` authority an ordinary Live test life would have after its first committed /// record. Raw ref-log helpers deliberately do not do this: several protocol tests need malformed or @@ -181,7 +186,9 @@ class FailExactGetBackend : public InMemoryBackend /// explicit instead of accidentally borrowing the legacy LIST-only recovery rule. void writeFsckCheckpoint(Backend & backend, const Layout & layout, const RootNamespace & ns, RefTxnId committed_through) { - const CasRefCatalog::Snapshot cut = CasRefCatalog::read(backend, layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot cut = CasRefCatalog::read(op, layout); const auto it = std::find_if(cut.catalog.entries.begin(), cut.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); ASSERT_NE(it, cut.catalog.entries.end()); @@ -203,7 +210,9 @@ void writeFsckCheckpointWithBase( Backend & backend, const Layout & layout, const RootNamespace & ns, RefTxnId base, std::optional last_epoch_seal = std::nullopt) { - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); ASSERT_EQ(backend.putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = base, @@ -235,7 +244,9 @@ void expectCheckpointBaseVerdict( /// its catalog cut, precisely the competing-cut mutation that fsck must not splice into its verdict. void replaceCatalogLife(Backend & backend, const Layout & layout, const RootNamespace & ns, UInt128 incarnation) { - CasRefCatalog::Snapshot current = CasRefCatalog::read(backend, layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::Snapshot current = CasRefCatalog::read(op, layout); const auto it = std::find_if(current.catalog.entries.begin(), current.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); ASSERT_NE(it, current.catalog.entries.end()); @@ -243,9 +254,9 @@ void replaceCatalogLife(Backend & backend, const Layout & layout, const RootName it->state = NsState::Live; it->creator.reset(); it->removal_started_round.reset(); - ASSERT_TRUE(current.token.has_value()); - ASSERT_EQ(backend.putOverwrite(layout.refCatalogKey(), encodeRefCatalog(current.catalog), *current.token).outcome, - PutOutcome::Done); + ASSERT_TRUE(current.incarnation.has_value()); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.refCatalogKey(), encodeRefCatalog(current.catalog), *current.incarnation, Retry::standard()))); } FsckReport runFsckWithListingMode(FsckListingMode mode, std::string_view suffix) @@ -266,7 +277,9 @@ FsckReport runFsckWithListingMode(FsckListingMode mode, std::string_view suffix) const uint64_t frontier = publishCommittedTransition(*backend, layout, ns, "tbl", r1, r2); writeFsckCheckpoint(*backend, layout, ns, RefTxnId{1, frontier}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); backend->distort(layout.namespaceStreamPrefix(life), mode); return runFsck(*store, /*detail=*/true); } @@ -327,7 +340,9 @@ FsckReport runCheckpointBaseFsckWithListingMode( writeRefSnapshotRaw(*backend, layout, snapshotOf(base_state, ns.string())); writeFsckCheckpointWithBase(*backend, layout, ns, base); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); if (corrupt_exact_base) { const String base_snapshot_key = layout.refSnapshotKey(life, base); @@ -468,7 +483,8 @@ TEST(CASFsck, CanonicalDeadLifeResidueIsJanitorPendingNotHardFinding) /// protocol this fixture is not driving), so inject the post-deletion catalog snapshot directly, /// mirroring `DuplicateLifeIdIsReportedWhileAnUnrelatedUniqueNamespaceStillProgresses` below. { - CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(*backend, store->layout()); + CasOperation op = store->gcRequests().admit(); + CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, store->layout()); const auto it = std::find_if(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); ASSERT_NE(it, snapshot.catalog.entries.end()); @@ -504,17 +520,19 @@ namespace class AdmitLifeAfterNamespaceListingBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; explicit AdmitLifeAfterNamespaceListingBackend(NamespaceLifeId life_) : protected_life(std::move(life_)) {} - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); if (!published && prefix.ends_with("/cas/ns/")) { published = true; - CasRefCatalog::casAdmitEntry(*this, Layout("p"), /*gc_shards*/1, + /// The base call above has already released the backend's lock, so admitting through this + /// same backend from here cannot deadlock. + CasRequests requests = DB::Cas::tests::openRequestsForTest(*this); + CasOperation op = requests.admit(); + CasRefCatalog::casAdmitEntry(op, Layout("p"), /*gc_shards*/1, CatalogEntry{.ns = protected_life.ns, .state = NsState::Live, .incarnation = protected_life.incarnation}); } @@ -589,7 +607,9 @@ TEST(CASFsck, DuplicateLifeIdIsReportedWhileAnUnrelatedUniqueNamespaceStillProgr const uint64_t sequence = publishCommittedTransition(*backend, layout, unique_ns, "tbl", std::nullopt, r); writeFsckCheckpoint(*backend, layout, unique_ns, RefTxnId{1, sequence}); - CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(*backend, layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout); snapshot.catalog.entries.push_back(CatalogEntry{ .ns = RootNamespace{"bad/a"}, .state = NsState::Live, .incarnation = UInt128{777}}); snapshot.catalog.entries.push_back(CatalogEntry{ @@ -634,7 +654,9 @@ TEST(CASFsck, AmbiguousLifeUnderAPhysicalKeyIsRecordedNotAborted) ASSERT_EQ(backend->putIfAbsent(layout.namespaceFilesPrefix(duplicated_life) + "format_version.txt", "1\n").outcome, PutOutcome::Done); - CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(*backend, layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout); snapshot.catalog.entries.push_back(CatalogEntry{ .ns = RootNamespace{"bad/a"}, .state = NsState::Live, .incarnation = UInt128{777}}); snapshot.catalog.entries.push_back(CatalogEntry{ @@ -811,7 +833,9 @@ TEST(CASFsckAuthority, MissingBurnedEpochSealIsChainBroken) .ns = ns.string(), .txn_id = RefTxnId{1, 2}, .ops = {seal}, .prev_epoch_seal = std::nullopt}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); /// The codec now rejects this skip. Deposit its old on-disk corruption shape by changing only the /// fixed-width epoch token of an otherwise encodable body, so fsck still proves that a missing /// intermediate epoch is reported rather than treated as a sparse legal transition. @@ -848,7 +872,9 @@ TEST(CASFsckAuthority, MissingCheckpointBaseLogIsChainBroken) fixture::admitLive(*backend, layout, ns); const RefTxnId base{1, 1}; - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); writeFsckCheckpointWithBase(*backend, layout, ns, base); const FsckReport report = runFsck(*store, /*detail=*/true); @@ -868,7 +894,9 @@ TEST(CASFsckAuthority, MissingCheckpointBaseSnapshotIsChainBroken) fixture::admitLive(*backend, layout, ns); const RefTxnId base{1, 1}; - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = ns.string(), .txn_id = base, .ops = {namespaceBirthOp()}, .prev_epoch_seal = std::nullopt}); writeFsckCheckpointWithBase(*backend, layout, ns, base); @@ -913,7 +941,9 @@ TEST(CASFsckAuthority, CheckpointSnapshotAtOlderEpochSealIsChainBroken) applyRefLogTxn(through_seal, seal_txn); writeRefSnapshotRaw(*backend, layout, snapshotOf(through_seal, ns.string())); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{2, 1}, @@ -928,8 +958,11 @@ TEST(CASFsckAuthority, CheckpointSnapshotAtOlderEpochSealIsChainBroken) "names an EpochSeal, not a snapshot base"); } -/// An unstable transport failure while exact-reading the same valid checkpoint base proves neither -/// presence nor absence. It remains the honest third answer and must not become a hard finding. +#if USE_AWS_S3 +/// A transport failure while exact-reading the same valid checkpoint base proves neither presence nor +/// absence. It remains the honest third answer and must not become a hard finding. The fault is armed +/// as an access denial so it surfaces on the read's first attempt (see `FailExactGetBackend`), which +/// keeps this test's cost at one request instead of a run through `Retry::standard()`'s deadline. TEST(CASFsckAuthority, CheckpointBaseTransportFailureIsUnchecked) { auto backend = std::make_shared(); @@ -939,7 +972,9 @@ TEST(CASFsckAuthority, CheckpointBaseTransportFailureIsUnchecked) fixture::admitLive(*backend, layout, ns); const RefTxnId base{1, 1}; - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = ns.string(), .txn_id = base, .ops = {namespaceBirthOp()}, .prev_epoch_seal = std::nullopt}); RefTableState state; @@ -952,8 +987,9 @@ TEST(CASFsckAuthority, CheckpointBaseTransportFailureIsUnchecked) const FsckReport report = runFsck(*store, /*detail=*/true); EXPECT_EQ(report.ref_records_walked, 0u); expectCheckpointBaseVerdict( - report, layout.refSnapshotKey(life, base), FsckClass::Unchecked, "injected exact GET failure"); + report, layout.refSnapshotKey(life, base), FsckClass::Unchecked, "injected access denial on exact GET"); } +#endif /// The sampled checkpoint is immutable input, but cleanup may advance `_ckpt` after that sample and /// retire its old base before fsck exact-reads it. The miss is then authority instability, not evidence @@ -968,7 +1004,9 @@ TEST(CASFsckAuthority, CheckpointBaseVanishingAfterAuthorityAdvanceIsUnchecked) fixture::admitLive(*backend, layout, ns); const RefTxnId old_base{1, 1}; - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); writeFsckCheckpointWithBase(*backend, layout, ns, old_base); backend->armOnFirstGet(layout.refLogKey(life, old_base), [&] { diff --git a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp index 9a6f92d3650c..2f8bc102efc8 100644 --- a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp +++ b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp @@ -1,5 +1,6 @@ #include +#include #include #include #include @@ -65,45 +66,43 @@ std::optional currentEntryFor(Backend & backend, const Layout & la return std::nullopt; } -/// Decorator reproducing the rustfs quirk (observed 2026-07-11): a conditional exact-token delete against -/// an object that is ALREADY absent can answer HTTP 412 (precondition failed), which this backend layer -/// maps to `TokenMismatch` -- not the 404-shaped `NotFound` an in-memory backend naturally returns. For -/// keys marked via `quirkOnAbsent`, `deleteExact` forces exactly that answer whenever the underlying -/// object is gone, letting a test drive the GC redelete site through the disambiguation path -/// backend-agnostically (without guessing at real rustfs HTTP mappings). -class TokenMismatchOnAbsentBackend : public InMemoryBackend +/// Counts every conditional removal sent against `watched_key` while that key is ALREADY absent. +/// A store may answer such a removal with a precondition failure rather than a clean miss (rustfs does, +/// observed 2026-07-11), so a caller that sends one cannot tell "somebody replaced it" from "it is +/// gone". The GC redelete site is not allowed to send one: it observes the blob first and compares the +/// condemned incarnation against what it saw. +class AbsentRemovalWatchBackend : public InMemoryBackend { public: - DeleteOutcome deleteExact(const String & key, const Token & token) override + void watch(const String & key) { watched_key = key; } + size_t removalsAgainstAbsent() const { return removals_against_absent; } + + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { - if (quirk_keys.contains(key) && !InMemoryBackend::head(key).exists) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::TokenMismatch; - return d; - } - return InMemoryBackend::deleteExact(key, token); + if (key == watched_key && !InMemoryBackend::head(key, access)) + ++removals_against_absent; + return InMemoryBackend::remove(key, expected_value, access); } - void quirkOnAbsent(const String & key) { quirk_keys.insert(key); } - private: - std::set quirk_keys; + String watched_key; + size_t removals_against_absent = 0; }; class CkptReplacementConflictBackend : public InMemoryBackend { public: - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - if (conflict_once && key == watched_key) + /// Only the CONDITIONAL shape is refused: the fixture's own creation of the object must land. + if (conflict_once && expected_value && key == watched_key) { conflict_once = false; - return CasResult{CasOutcome::Conflict, {}}; + return std::unexpected(RawConflict{}); } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } String watched_key; @@ -115,13 +114,15 @@ TEST(CASSemanticRefFixture, WrapperCreatesInitialRecoverableCheckpoint) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"00/semantic-create@cas@"}; const ManifestRef manifest = ref("srv-a:1", 1, 0xAB); const uint64_t sequence = publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, manifest); const RefTxnId expected_id{manifest.writer_epoch, sequence}; - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); - const auto ckpt = readCkpt(*backend, store->layout(), life); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); + const auto ckpt = readCkpt(op, store->layout(), life); ASSERT_TRUE(ckpt.has_value()); EXPECT_EQ(ckpt->ckpt.life_epoch, 1); @@ -134,24 +135,26 @@ TEST(CASSemanticRefFixture, WrapperAdvancesCheckpointWithoutDiscardingSnapshot) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"00/semantic-advance@cas@"}; const ManifestRef manifest = ref("srv-a:1", 1, 0xAC); const uint64_t publish_sequence = publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, manifest); const RefTxnId publish_id{manifest.writer_epoch, publish_sequence}; writeRefSnapshotRaw(*backend, store->layout(), minimalLiveSnapshot(ns.string(), publish_id, {committedRow("tbl", manifest)})); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); - const auto before_drop = readCkpt(*backend, store->layout(), life); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); + const auto before_drop = readCkpt(op, store->layout(), life); ASSERT_TRUE(before_drop.has_value()); RefCkpt with_snapshot = before_drop->ckpt; with_snapshot.checkpoint_snapshot_id = publish_id; - ASSERT_EQ(backend->casPut( - store->layout().refCkptKey(life), encodeRefCkpt(with_snapshot), before_drop->token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(op.replace( + store->layout().refCkptKey(life), encodeRefCkpt(with_snapshot), before_drop->incarnation, + Retry::once()))); const uint64_t drop_sequence = dropRefTransition(*backend, store->layout(), ns, "tbl", manifest); const RefTxnId drop_id{manifest.writer_epoch, drop_sequence}; - const auto ckpt = readCkpt(*backend, store->layout(), life); + const auto ckpt = readCkpt(op, store->layout(), life); ASSERT_TRUE(ckpt.has_value()); EXPECT_EQ(ckpt->ckpt.committed_through, drop_id); @@ -163,6 +166,8 @@ TEST(CASSemanticRefFixture, CheckpointAdvanceRejectsNonMonotoneAndInvalidState) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"00/semantic-refusal@cas@"}; const ManifestRef manifest = ref("srv-a:1", 1, 0xAD); @@ -172,7 +177,7 @@ TEST(CASSemanticRefFixture, CheckpointAdvanceRejectsNonMonotoneAndInvalidState) const RootNamespace invalid_ns{"00/semantic-invalid@cas@"}; fixture::admitLive(*backend, store->layout(), invalid_ns); - const NamespaceLifeId invalid_life = *CasRefCatalog::lifeIfCataloged(*backend, store->layout(), invalid_ns); + const NamespaceLifeId invalid_life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), invalid_ns); const String invalid_key = store->layout().refCkptKey(invalid_life); ASSERT_EQ(backend->putIfAbsent(invalid_key, "not a checkpoint").outcome, PutOutcome::Done); EXPECT_THROW(advanceRecoverableCkptForRawFixture(*backend, store->layout(), invalid_ns, id), DB::Exception); @@ -183,6 +188,8 @@ TEST(CASRawRefFixture, RawLogWriteDoesNotCreateCheckpoint) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"00/raw-no-ckpt@cas@"}; const RefTxnId id{1, 1}; @@ -193,14 +200,16 @@ TEST(CASRawRefFixture, RawLogWriteDoesNotCreateCheckpoint) .prev_epoch_seal = std::nullopt, }); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); - EXPECT_FALSE(readCkpt(*backend, store->layout(), life).has_value()); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); + EXPECT_FALSE(readCkpt(op, store->layout(), life).has_value()); } TEST(CASRawRefFixture, ReplaceRecoverableCheckpointWritesTheSuppliedFullState) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"00/replace-ckpt@cas@"}; const ManifestRef manifest = ref("srv-a:1", 1, 0xAE); const RefTxnId first_id{manifest.writer_epoch, @@ -217,8 +226,8 @@ TEST(CASRawRefFixture, ReplaceRecoverableCheckpointWritesTheSuppliedFullState) }; replaceRecoverableCkptForRawFixture(*backend, store->layout(), ns, next); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); - const auto replaced = readCkpt(*backend, store->layout(), life); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); + const auto replaced = readCkpt(op, store->layout(), life); ASSERT_TRUE(replaced.has_value()); EXPECT_EQ(replaced->ckpt.life_epoch, next.life_epoch); EXPECT_EQ(replaced->ckpt.committed_through, next.committed_through); @@ -230,12 +239,14 @@ TEST(CASRawRefFixture, ReplaceRecoverableCheckpointRejectsStaleRegressiveAndWron { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"00/replace-ckpt-refusal@cas@"}; const ManifestRef manifest = ref("srv-a:1", 1, 0xAF); const RefTxnId id{manifest.writer_epoch, publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, manifest)}; - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); - const auto existing = readCkpt(*backend, store->layout(), life); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); + const auto existing = readCkpt(op, store->layout(), life); ASSERT_TRUE(existing.has_value()); RefCkpt wrong_life = existing->ckpt; @@ -250,7 +261,7 @@ TEST(CASRawRefFixture, ReplaceRecoverableCheckpointRejectsStaleRegressiveAndWron backend->watched_key = store->layout().refCkptKey(life); backend->conflict_once = true; EXPECT_THROW(replaceRecoverableCkptForRawFixture(*backend, store->layout(), ns, existing->ckpt), DB::Exception); - EXPECT_EQ(readCkpt(*backend, store->layout(), life)->ckpt.committed_through, id); + EXPECT_EQ(readCkpt(op, store->layout(), life)->ckpt.committed_through, id); } /// The owner-removed manifest body is deleted only after a full round (its decrement is sealed — #11). @@ -438,19 +449,24 @@ TEST(CASGCRetire, SpareLeavesMetaCondemned) } /// Two-leader stale-redelete regression — the executable form of the deposed-leader spec §2. A stale -/// leader's pre-CAS exact-token redelete `deleteExact(h, t1)` must never delete a live reuse. With the -/// buggy clear-on-spare, a spare publishes `Clean`; a writer reads `Clean` and REUSES `t1`; the stale -/// `deleteExact(t1)` then deletes the LIVE body (INV_NO_LOSS). Add-only meta closes it: the spare leaves -/// `Condemned`, the writer resurrects to `t2`, and the stale `deleteExact(t1)` is a `TokenMismatch` no-op. +/// leader's pre-CAS redelete of the incarnation `t1` must never delete a live reuse. With the buggy +/// clear-on-spare, a spare publishes `Clean`; a writer reads `Clean` and REUSES `t1`; the stale redelete +/// then deletes the LIVE body (INV_NO_LOSS). Add-only meta closes it: the spare leaves `Condemned`, the +/// writer resurrects to `t2`, and the stale redelete finds a different incarnation and sends nothing. /// /// Interleaving fidelity (APPROXIMATED): the deposed leader's destructive side effect is its pre-CAS -/// exact-token `deleteExact(h, t1)`. We reproduce it deterministically by CAPTURING `t1` at condemn time -/// (exactly the token a paused leader's `delete_pending` snapshot holds) and firing that exact -/// `deleteExact` AFTER the surviving leader's spare and the writer's republication — the faithful destructive -/// op, without a mid-round CAS-interrupt seam on the delete path (which the backend does not expose). +/// redelete of `t1`. We reproduce it deterministically by CAPTURING `t1` at condemn time (exactly the +/// incarnation a paused leader's `delete_pending` snapshot holds) and replaying the round's own +/// observe-compare-remove sequence AFTER the surviving leader's spare and the writer's republication -- +/// the faithful destructive step, without a mid-round interrupt seam on the delete path. +/// +/// The replay really SENDS its removal when the comparison passes, and the removal counter below is +/// what makes the closing fsck mean something: on the clear-on-spare regression the spare publishes +/// `Clean`, the writer's dedup hit keeps `t1`, the comparison passes, the live body is removed and +/// both the counter and the fsck dangle count move. TEST(CASGCRetire, StaleRedeleteAfterSpareDoesNotDeleteLiveReuse) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPoolForTest(backend); const RootNamespace ns{"00/aa@cas@"}; @@ -472,10 +488,14 @@ TEST(CASGCRetire, StaleRedeleteAfterSpareDoesNotDeleteLiveReuse) gc.runRegularRound(); /// -1 => in-degree 0 => condemned at t1 /// The OLD leader L1's planned pre-CAS delete uses the EXACT token it observed at condemn: capture t1. + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const auto condemned_entry = currentEntryFor(*backend, store->layout(), hash); ASSERT_TRUE(condemned_entry.has_value()); - const Token t1 = condemned_entry->token; - ASSERT_EQ(backend->head(blob_key).token, t1); + const PersistedIncarnation t1 = condemned_entry->token; + const std::optional at_condemn = op.head(blob_key, Retry::once()); + ASSERT_TRUE(at_condemn); + ASSERT_TRUE(t1.matches(at_condemn->incarnation)); /// A NEW leader L2 folds a +1 that recovered h's in-degree and adopts a SPARE for h. const ManifestRef r2 = ref("srv-a:1", 2, 0xA2); @@ -495,19 +515,29 @@ TEST(CASGCRetire, StaleRedeleteAfterSpareDoesNotDeleteLiveReuse) /// from the writer's own source — it never reuses t1. const RootNamespace writer_ns{"00/redelete-writer@cas@"}; publishBlobWithDurablePrecommit(store, writer_ns, "writer", id, payload); - const Token t2 = backend->head(blob_key).token; - EXPECT_NE(t2, t1) << "the writer resurrected to a fresh incarnation, not a reuse of t1"; - - /// L1 resumes and executes its stale pre-CAS exact-token redelete `deleteExact(h, t1)`: it must be a - /// TokenMismatch no-op (the live body is now t2), NEVER a Deleted of the live reuse. - const DeleteOutcome stale = backend->deleteExact(blob_key, t1); - EXPECT_EQ(stale.kind, DeleteOutcome::Kind::TokenMismatch) - << "the stale exact-token redelete must miss the live reuse (add-only closes INV_NO_LOSS)"; + const std::optional t2 = op.head(blob_key, Retry::once()); + ASSERT_TRUE(t2); + EXPECT_FALSE(t1.matches(t2->incarnation)) + << "the writer resurrected to a fresh incarnation, not a reuse of t1"; + + /// L1 resumes and replays its stale pre-CAS redelete exactly as the round performs one: observe the + /// key, compare the condemned incarnation against what is there, and remove ONLY on a match. The + /// removal is genuinely attempted on a match, so this step is destructive whenever the writer + /// reused `t1`. + const uint64_t removals_before = backend->deleteCount(blob_key); + const std::optional stale = op.head(blob_key, Retry::once()); + ASSERT_TRUE(stale); + EXPECT_FALSE(t1.matches(stale->incarnation)) + << "the stale redelete must miss the live reuse (add-only closes INV_NO_LOSS)"; + if (t1.matches(stale->incarnation)) + (void)op.remove(blob_key, stale->incarnation, Retry::once()); + EXPECT_EQ(backend->deleteCount(blob_key), removals_before) + << "the comparison failed, so the redelete sent no removal at all against the live body"; /// The live body under t2 survives, stays reachable via the committed r2, and fsck sees no dangle. - const HeadResult hr = backend->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_EQ(hr.token, t2); + const std::optional survivor = op.head(blob_key, Retry::once()); + ASSERT_TRUE(survivor); + EXPECT_EQ(survivor->incarnation, t2->incarnation); replaceRecoverableCkptForRawFixture( *backend, store->layout(), ns, RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 3}, @@ -853,7 +883,9 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) // crashed process: a body that is live-shaped (not terminated, not fenced) but whose write token // never changes again. const String srid2 = "stale-server"; - MountLeaseKeeper srid2_keeper(backend, layout, srid2, DB::UInt128(0x2222), /*writer_epoch=*/1, + CasRequests keeper_requests = openRequestsForTest(backend); + MountLeaseKeeper srid2_keeper(keeper_requests, keeper_requests, layout, srid2, DB::UInt128(0x2222), + /*writer_epoch=*/1, std::chrono::milliseconds(100), [] { return 1000u; }, [] { return 0u; }, {}, std::chrono::milliseconds(0), [] { return 0u; }); srid2_keeper.start(); @@ -912,10 +944,7 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) // srid2's writer comes back and tries to renew: its held token was invalidated by the fence rewrite, // so synchronous renewal returns a terminal failure. (It renews on its own clock; liveness is irrelevant — the token guard // trips regardless.) - const MountRenewResult renewed = srid2_keeper.renew( - CasRequestBudget{.attempt_timeout_ms = 1, .operation_deadline_ms = 10, .max_attempts = 1, - .lease_safety_margin_ms = 0, .retry_initial_backoff_ms = 0, .retry_max_backoff_ms = 0}, - MountRenewOperationEnvironment{}); + const MountRenewResult renewed = srid2_keeper.renew(MountRenewOperationEnvironment{}); ASSERT_EQ(renewed.outcome, MountRenewOutcome::Terminal); ASSERT_NE(renewed.failure, nullptr); EXPECT_THROW(std::rethrow_exception(renewed.failure), DB::Exception); @@ -944,7 +973,9 @@ TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) // A stale mount, exactly as `ExpiredMountFencedOutAndExcluded`: one claim, never renewed again. const String srid2 = "stale-server"; - MountLeaseKeeper srid2_keeper(backend, layout, srid2, DB::UInt128(0x2222), /*writer_epoch=*/1, + CasRequests keeper_requests = openRequestsForTest(backend); + MountLeaseKeeper srid2_keeper(keeper_requests, keeper_requests, layout, srid2, DB::UInt128(0x2222), + /*writer_epoch=*/1, std::chrono::milliseconds(100), [] { return 1000u; }, [&] { return fake_boot; }); srid2_keeper.start(); ASSERT_FALSE(decodeMountLease(backend->get(layout.mountKey(srid2))->bytes).gc_fenced); @@ -968,9 +999,9 @@ TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) EXPECT_TRUE(decodeMountLease(backend->get(layout.mountKey(srid2))->bytes).gc_fenced); } -/// deleteExact against a blob the writer RECREATED (fresh incarnation, different token) between the pending -/// publish and the deleting pass lands TokenMismatch — a terminal-OK outcome recorded as a replace: the -/// fresh incarnation is a live object and survives. report.replaced counts it. +/// A redelete of a blob the writer RECREATED (fresh incarnation) between the pending publish and the +/// deleting pass finds a different incarnation — a terminal-OK outcome recorded as a replace: the fresh +/// incarnation is a live object and survives. report.replaced counts it. TEST(CASGCAckFloor, RecreatedBlobDeleteIsTokenMismatchOk) { auto backend = std::make_shared(); @@ -1005,8 +1036,8 @@ TEST(CASGCAckFloor, RecreatedBlobDeleteIsTokenMismatchOk) // longer matches the pending entry's captured token. displaceBlobToken(*backend, store->layout(), blob_id); - // The deleting pass issues deleteExact(entry.token) → TokenMismatch → Replaced. The fresh incarnation - // survives; the entry is dropped. + // The deleting pass observes the key, finds an incarnation the entry does not name → Replaced. The + // fresh incarnation survives; the entry is dropped. const RoundReport rep = runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); EXPECT_EQ(rep.replaced, 1u); @@ -1059,8 +1090,13 @@ TEST(CASGCAckFloor, ResumeAfterCrashBetweenRetiredPutAndStateCas) // Simulate a crashed deleting pass that DID land the exact-token delete but crashed before the gc/state // CAS. The next (fresh-attempt) pass replays the delete → the object is already gone → NotFound → the // pass records Absent and completes. - ASSERT_EQ(backend->deleteExact(store->layout().blobKey(blob_id), pending_entry.token).kind, - DeleteOutcome::Kind::Deleted); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const String pending_key = store->layout().blobKey(blob_id); + const std::optional doomed = op.head(pending_key, Retry::once()); + ASSERT_TRUE(doomed); + ASSERT_TRUE(pending_entry.token.matches(doomed->incarnation)); + ASSERT_EQ(op.remove(pending_key, doomed->incarnation, Retry::once()), Removal::Removed); const uint64_t round_before = decodeGcState(backend->get(store->layout().gcStateKey())->bytes).round; Gc gc2(store, kGc); @@ -1072,14 +1108,14 @@ TEST(CASGCAckFloor, ResumeAfterCrashBetweenRetiredPutAndStateCas) EXPECT_FALSE(currentEntryFor(*backend, store->layout(), blob).has_value()); } -/// Backend-agnostic regression for the rustfs 412-on-absent quirk: a conditional exact-token delete -/// against an object that is ALREADY absent answers `TokenMismatch`, not `NotFound`, on this backend -/// (`TokenMismatchOnAbsentBackend` reproduces it deterministically). The redelete site must disambiguate -/// via a follow-up HEAD: the object is truly gone, so the outcome must settle as Absent (never Replaced) -/// and the `.meta` cleanup (gated on Deleted/NotFound) must still run. -TEST(CASGCAckFloor, TokenMismatchOnAbsentBlobSettlesAsAbsentAndDropsMeta) +/// A blob whose body a crashed pass already deleted must settle as Absent (never Replaced), its `.meta` +/// cleanup must still run, and -- the part a store can punish -- the round must not send a conditional +/// removal against the absent key at all. A store may answer such a removal with a precondition failure +/// instead of a clean miss (rustfs does), which is indistinguishable from "somebody replaced it"; the +/// round observes first, so it never has to tell the two apart. +TEST(CASGCAckFloor, AbsentBlobSettlesAsAbsentWithoutASpeculativeConditionalRemoval) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPoolForTest(backend); const RootNamespace ns{"00/aa@cas@"}; const ManifestRef r = ref("srv-a:1", 1, 0xAA); @@ -1117,32 +1153,37 @@ TEST(CASGCAckFloor, TokenMismatchOnAbsentBlobSettlesAsAbsentAndDropsMeta) ASSERT_EQ(lm->meta.state, MetaState::Condemned); } - // The object is genuinely gone already (as if a prior crashed pass landed the delete); confirm that, - // then arm the quirk so the NEXT conditional delete against this now-absent key answers TokenMismatch - // instead of NotFound (the rustfs 412-on-absent behavior). + // The object is genuinely gone already (as if a prior crashed pass landed the delete), and from here + // every removal the round sends against this key would be sent against an absent object. + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String blob_key = store->layout().blobKey(blob_id); - ASSERT_EQ(backend->deleteExact(blob_key, pending_entry.token).kind, DeleteOutcome::Kind::Deleted); - ASSERT_FALSE(backend->head(blob_key).exists); - backend->quirkOnAbsent(blob_key); - - // The deleting pass replays deleteExact(entry.token): the backend answers TokenMismatch (quirk), but - // the follow-up HEAD shows the object absent, so the fix disambiguates the outcome to Absent and still - // runs the `.meta` cleanup. + const std::optional doomed = op.head(blob_key, Retry::once()); + ASSERT_TRUE(doomed); + ASSERT_TRUE(pending_entry.token.matches(doomed->incarnation)); + ASSERT_EQ(op.remove(blob_key, doomed->incarnation, Retry::once()), Removal::Removed); + ASSERT_FALSE(op.head(blob_key, Retry::once())); + backend->watch(blob_key); + + // The deleting pass replays the redelete: it observes the absent key and settles Absent without a + // request the store could answer ambiguously, and the `.meta` cleanup still runs. const RoundReport rep = runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); - EXPECT_EQ(rep.absent, 1u) << "the 412-on-absent quirk must settle as Absent, not Replaced"; + EXPECT_EQ(rep.absent, 1u) << "an already-absent blob settles as Absent, not Replaced"; EXPECT_EQ(rep.replaced, 0u); + EXPECT_EQ(backend->removalsAgainstAbsent(), 0u) + << "the redelete observed the key first, so it sent no conditional removal against an absent object"; EXPECT_FALSE(currentEntryFor(*backend, store->layout(), blob).has_value()); EXPECT_FALSE(loadMetaForTest(*backend, store->layout(), blob).has_value()) - << ".meta cleanup (gated on Deleted/NotFound) must still run on the disambiguated Absent outcome"; + << ".meta cleanup (gated on a removal or a proven absence) must still run on the Absent outcome"; } /// ---- condemn-marker gate suite ---- /// /// The per-hash condemn marker is LOAD-BEARING for the delete edge: the writer's adopt gate point-reads /// the meta and an ABSENT meta reads as Clean, so a blob whose condemn-marker write was swallowed can be -/// same-token adopted by a writer landing in the [discovery-LIST, deleteExact] window — invisible to the -/// graduating fold — and the exact-token redelete then deletes a body under a live committed edge +/// same-token adopted by a writer landing in the [discovery-LIST, redelete] window — invisible to the +/// graduating fold — and the redelete then deletes a body under a live committed edge /// (dangling manifest). Graduation to `delete_pending` therefore requires CONFIRMED durable `Condemned` /// evidence for the entry; absent evidence CARRIES the entry to the next round (fail-safe delay, never a /// fail-open delete) and retries the marker so a healed backend restores liveness. @@ -1153,8 +1194,30 @@ TEST(CASGCAckFloor, TokenMismatchOnAbsentBlobSettlesAsAbsentAndDropsMeta) TEST(CASGCCondemnMarker, SwallowedMarkerWriteCarriesEntryInsteadOfDeleting) { auto backend = std::make_shared(); + /// A SWALLOWED write is the premise: the store may have applied it and said nothing, which is the + /// only shape that leaves the round committing an entry whose marker it cannot confirm. The + /// propagating kind never reaches the engine's resolve-and-reissue path at all -- the write loop + /// rethrows a non-`Poco::Exception` on its first attempt. + backend->armWriteFault(MetaWriteFaultBackend::FaultKind::Ambiguous); auto store = openPoolForTest(backend); store->setCasRetrySleepForTest([](uint64_t) {}); + + /// THE FAULT IS PERMANENT, so every condemn-marker write runs the engine's WHOLE retry window, and + /// that window has to run on a clock this test advances. A zeroed sleep alone does not do it: the + /// deadline is still measured against the real clock, so the loop would spin hot for ninety real + /// seconds per marker write. The sleep therefore moves the clock past its own pause -- plus one + /// millisecond, because full-jitter backoff may draw zero and a clock that never moves never closes + /// the window. Scoped to the GC plane, which is where `writeCondemnedMeta` runs, so the mount + /// plane's lease-bound policies keep their real clock. + std::atomic engine_now_ms{0}; + std::atomic engine_sleeps{0}; + store->gcRequests().setNowFnForTest([&] { return engine_now_ms.load(); }); + store->gcRequests().setSleepFnForTest([&](uint64_t pause_ms) + { + engine_sleeps.fetch_add(1); + engine_now_ms.fetch_add(pause_ms + 1); + }); + const RootNamespace ns{"00/aa@cas@"}; const ManifestRef r = ref("srv-a:1", 1, 0xAA); const UInt128 blob = DB::UInt128(1); @@ -1165,9 +1228,18 @@ TEST(CASGCCondemnMarker, SwallowedMarkerWriteCarriesEntryInsteadOfDeleting) runRegularRoundReclaiming(gc); // +1 folds; blob referenced dropRefTransition(*backend, store->layout(), ns, "tbl", r); - runRegularRoundReclaiming(gc); // the condemning round; the controlled marker write exhausts as Unresolved + runRegularRoundReclaiming(gc); // the condemning round; the marker write gives up without committing ASSERT_FALSE(loadMetaForTest(*backend, store->layout(), blob).has_value()) << "precondition: the injected fault must have lost the condemn-marker write"; + /// The marker write was resolved and reissued, then gave up at its own policy window. A give-up + /// needs the next jittered pause not to fit before the deadline and full jitter draws at most five + /// seconds, so it cannot happen before the clock has passed `Retry::standard()`'s window minus + /// that draw. Both assertions pin the REISSUING, which is what the ambiguous kind buys; neither + /// can tell the injected clock from the real one -- that seam bounds the reissuing in real time. + EXPECT_GT(engine_sleeps.load(), 1u) + << "an ambiguous marker write must be resolved and reissued, not surfaced on its first attempt"; + EXPECT_GT(engine_now_ms.load(), 85'000u) + << "the marker write must have spent its whole retry window before reporting failure"; ASSERT_TRUE(currentEntryFor(*backend, store->layout(), blob).has_value()) << "precondition: the retired entry must have been committed despite the lost marker"; diff --git a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp index ed4e76fac722..ed913c2c1a2b 100644 --- a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp +++ b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp @@ -415,15 +415,14 @@ TEST(CASGCArithmeticIntake, EpochStartThatAnswersOnlyEveryOtherReadHoldsInsteadO class AlternatingGetBackend : public InMemoryBackend { public: - using DB::Cas::Backend::get; String flaky; size_t reads = 0; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (key == flaky && ++reads % 2 == 0) return std::nullopt; - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } }; diff --git a/src/Disks/tests/gtest_cas_gc_attempt.cpp b/src/Disks/tests/gtest_cas_gc_attempt.cpp index 0ae235777cbd..e4e5ff635290 100644 --- a/src/Disks/tests/gtest_cas_gc_attempt.cpp +++ b/src/Disks/tests/gtest_cas_gc_attempt.cpp @@ -79,30 +79,36 @@ size_t runGcToFixpoint(const PoolPtr & s, Gc & gc, size_t max_rounds = 64) return rounds; } -/// A backend that throws ONCE on the SINGLE round-commit `gc/state` CAS — the casPut that advances -/// snap_generation (the one-pass round has exactly one such CAS; the lease-acquire CAS does not advance -/// snap_generation, so "advances snap_generation" uniquely picks the round commit). +/// A backend that refuses ONCE the SINGLE round-commit `gc/state` write — the conditional write that +/// advances snap_generation (the one-pass round has exactly one such write; the lease acquire/renew does +/// not advance snap_generation, so "advances snap_generation" uniquely picks the round commit). class InterruptRoundCasBackend : public InMemoryBackend { public: explicit InterruptRoundCasBackend(String gc_state_key_) : gc_state_key(std::move(gc_state_key_)) {} - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - if (arm_interrupt && key == gc_state_key) + if (arm_interrupt && expected_value && key == gc_state_key) { - const auto stored = get(key); - const uint64_t stored_gen = stored ? decodeGcState(stored->bytes).snap_generation : 0; - const uint64_t next_gen = decodeGcState(bytes).snap_generation; - if (next_gen > stored_gen) + const auto stored = InMemoryBackend::read(key, access); + if (stored + && decodeGcState(bytes).snap_generation > decodeGcState(stored->bytes).snap_generation) { - arm_interrupt = false; /// one-shot: only depose the first round-commit CAS - throw DB::Exception(DB::ErrorCodes::ABORTED, - "test-injected: round-commit gc/state CAS denied (leader deposed mid-round; lease lost)"); + arm_interrupt = false; /// one-shot: only depose the first round-commit write + /// A REFUSAL, not a throw: a thrown transport error is an ambiguity the engine settles + /// by an exact read and then reissues while the precondition it named is unmoved, so + /// the round would commit on the reissue. A refused precondition ends the write at + /// once. The object is moved too -- the same bytes under a fresh incarnation -- because + /// a store refuses only what changed; the CONTENT is deliberately left alone, so this + /// round's own lease and cursor are exactly what a deposed round leaves behind. + (void)InMemoryBackend::write(key, stored->bytes, stored->value, access); + return std::unexpected(RawConflict{}); } } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } bool arm_interrupt = false; diff --git a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp index 384e9f29ea35..7fd166eb7805 100644 --- a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp +++ b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp @@ -138,8 +138,6 @@ std::map runRoundCapturingIntake(Gc & gc, UniversePolicy policy class ChasingWriterBackend : public CountingBackend { public: - using CountingBackend::get; - /// Start appending above `published_through` (writer epoch 1) whenever the tail is read, up to /// `max_appends` further records. void arm(const Layout * layout_, const RootNamespace & ns_, uint64_t published_through, uint64_t max_appends) @@ -155,9 +153,9 @@ class ChasingWriterBackend : public CountingBackend uint64_t publishedThrough() const { return published; } - std::optional get(const String & key, DB::Cas::Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - auto result = CountingBackend::get(key, range); + auto result = CountingBackend::read(key, access); if (!layout || appending || published >= limit) return result; if (key != layout->refLogKey(fixture::fixtureLife(ns), RefTxnId{1, published})) diff --git a/src/Disks/tests/gtest_cas_gc_fold.cpp b/src/Disks/tests/gtest_cas_gc_fold.cpp index 78435a6c2d78..5e85ac7de218 100644 --- a/src/Disks/tests/gtest_cas_gc_fold.cpp +++ b/src/Disks/tests/gtest_cas_gc_fold.cpp @@ -533,6 +533,8 @@ TEST(CASGCFold, RoundSideAnomalySuppressesRefLogCleanupWhileRemovalDebrisStaysJa { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); Gc gc(store, kGc); @@ -560,7 +562,7 @@ TEST(CASGCFold, RoundSideAnomalySuppressesRefLogCleanupWhileRemovalDebrisStaysJa /// which is the physical life that owns the eventual janitor work. Spelling the sentinel here instead /// would plant debris under the wrong life and make the retention assertion vacuous. const String debris_key - = layout.namespaceFilesPrefix(CasRefCatalog::lifeIfCataloged(*backend, layout, ns_removed).value()) + = layout.namespaceFilesPrefix(CasRefCatalog::lifeIfCataloged(op, layout, ns_removed).value()) + "leftover_verbatim_file"; backend->putIfAbsent(debris_key, "debris"); const ManifestRef removed_body = ref("srv-r:1", 1, 0xEE); diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 8109fc851605..57839fd4a3fa 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -12,10 +12,12 @@ #include #include "cas_test_helpers.h" +#include #include #include #include +#include #include #include #include @@ -73,6 +75,13 @@ namespace const UInt128 kGc = hexToU128("00000000000000000000000000000001"); +/// The `CASCatalogLifecycleReconciler` suites hand their operation a liveness that reads a local bool +/// directly, and `DrainRaceBackend::afterReadOf` moves that bool at an exact request boundary -- so +/// nothing is cached behind a read and there is nothing for a refresh to re-read. Named rather than an +/// inline no-op because the argument is mandatory, precisely so that erasing without a refresh has to +/// be said out loud. +void noAuthorityRefresh() {} + /// The lying store, shared from `cas_test_helpers.h`: every key is served by exact GET while the /// selected ones are HIDDEN from every LIST. That is the only way to build the cross-namespace /// scenario -- the hidden namespace's records stay durable and readable, so a round that KNOWS to @@ -83,9 +92,8 @@ using CountingHintHoleBackend = DB::Cas::tests::HintHoleBackendOn get(const String & key, Range range) override + /// Runs after every completed read of `key`. It is where a test moves a fact the operation's + /// liveness predicate samples, so admission can be lost at an exact request boundary. + void afterReadOf(const String & key, std::function hook) { - record("get " + key); - return CountingBackend::get(key, range); + std::lock_guard lock(control_mutex); + after_read_key = key; + after_read_hook = std::move(hook); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + std::optional read(const String & key, TransportAccess & access) override { - record("list " + prefix); - return CountingBackend::list(prefix, cursor, limit); + record("get " + key); + std::optional raw = CountingBackend::read(key, access); + std::function hook; + { + std::lock_guard lock(control_mutex); + if (key == after_read_key) + hook = after_read_hook; + } + if (hook) + hook(); + return raw; } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - record("put_begin " + key); - const PutResult result = CountingBackend::putIfAbsent(key, bytes, meta); - record("put_end " + key); - return result; + record("list " + prefix); + return CountingBackend::list(prefix, cursor, limit, access); } - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - record("cas_begin " + key); + /// One primitive now carries both shapes the journal used to name separately: a write with no + /// precondition is the create, a write with one is the conditional replace. + const bool conditional = expected_value.has_value(); + record((conditional ? "cas_begin " : "put_begin ") + key); bool lose_response = false; bool force_conflict = false; { std::unique_lock lock(control_mutex); - if (key == catalog_key && block_next_catalog_cas) + if (conditional && key == catalog_key && block_next_catalog_cas) { block_next_catalog_cas = false; catalog_cas_blocked = true; control_cv.notify_all(); control_cv.wait(lock, [&] { return release_catalog_cas; }); } - if (key == catalog_key && lose_next_catalog_cas_response) + if (conditional && key == catalog_key && lose_next_catalog_cas_response) { lose_next_catalog_cas_response = false; lose_response = true; } - if (key == catalog_key && conflict_next_catalog_cas) + if (conditional && key == catalog_key && conflict_next_catalog_cas) { conflict_next_catalog_cas = false; force_conflict = true; @@ -183,14 +204,18 @@ class DrainRaceBackend final : public CountingBackend if (force_conflict) { record("cas_forced_conflict " + key); - return {.outcome = CasOutcome::Conflict, .token = {}}; + return std::unexpected(RawConflict{}); } - const CasResult result = CountingBackend::casPut(key, bytes, expected, meta); - record("cas_end " + key); - if (lose_response && result.outcome == CasOutcome::Committed) + std::expected result = CountingBackend::write(key, bytes, expected_value, access); + record((conditional ? "cas_end " : "put_end ") + key); + if (lose_response && result.has_value()) { record("cas_response_lost " + key); - throw std::runtime_error("injected lost catalog CAS response"); + /// `Poco::TimeoutException`, because that is the class the write loop cannot distinguish + /// from a lost response: it settles the attempt by an exact read, finds these bytes under a + /// moved incarnation, and reports the write committed. A non-`Poco` exception is rethrown + /// unchanged instead, which would propagate a landed write as a failure. + throw Poco::TimeoutException("injected lost catalog CAS response"); } return result; } @@ -212,25 +237,31 @@ class DrainRaceBackend final : public CountingBackend bool release_catalog_cas = false; bool lose_next_catalog_cas_response = false; bool conflict_next_catalog_cas = false; + String after_read_key; + std::function after_read_hook; }; class PostFoldUnreadableTerminalBackend final : public CountingBackend { public: - ListPage list(const String & prefix, const String & cursor, size_t limit) override + /// Unhide the names the primitive overrides below would otherwise shadow. + using CountingBackend::head; + using CountingBackend::list; + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = CountingBackend::list(prefix, cursor, limit); + RawListPage page = CountingBackend::list(prefix, cursor, limit, access); if (prefix.ends_with("/cas/ns/")) - for (ListedKey & listed : page.keys) - listed.token.reset(); + for (RawListedKey & listed : page.keys) + listed.value.reset(); return page; } - HeadResult head(const String & key) override + std::optional head(const String & key, TransportAccess & access) override { - if (key == unreadable_key) + if (!bypass_fault && key == unreadable_key) throw std::runtime_error("injected post-fold terminal read failure for " + key); - return CountingBackend::head(key); + return CountingBackend::head(key, access); } void makeUnreadable(String key) @@ -238,13 +269,19 @@ class PostFoldUnreadableTerminalBackend final : public CountingBackend unreadable_key = std::move(key); } + /// The test's own look at the key the fault hides, taken through the same primitive with the fault + /// suspended -- there is no second door to the store. bool existsIgnoringFault(const String & key) { - return CountingBackend::head(key).exists; + bypass_fault = true; + const bool present = CountingBackend::head(key).exists; + bypass_fault = false; + return present; } private: String unreadable_key; + bool bypass_fault = false; }; class ScopedCasGcLogCapture @@ -289,7 +326,7 @@ struct CompletedRemovingFixture }; CompletedRemovingFixture seedCompletedRemoving( - DrainRaceBackend & backend, const PoolPtr & store, const UInt128 & lease_owner) + DrainRaceBackend & backend, CasOperation & op, const PoolPtr & store, const UInt128 & lease_owner) { const Layout & layout = store->layout(); CompletedRemovingFixture fixture{ @@ -297,7 +334,7 @@ CompletedRemovingFixture seedCompletedRemoving( .life_id = UInt128{177}, .checkpoint_key = {}, .checkpoint_bytes = {}}; - CasRefCatalog::casAdmitEntry(backend, layout, store->poolConfig().gc_shards, CatalogEntry{ + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{ .ns = fixture.ns, .state = NsState::Live, .incarnation = fixture.life_id}); fixture.checkpoint_key = layout.refCkptKey( NamespaceLifeId::fromCatalogEntry(fixture.ns, fixture.life_id)); @@ -309,7 +346,7 @@ CompletedRemovingFixture seedCompletedRemoving( }); backend.putIfAbsent(fixture.checkpoint_key, fixture.checkpoint_bytes); EXPECT_TRUE(store->namespaceFilesLifeIfReadable(fixture.ns)); - CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & current) + CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { RefCatalog next = current; next.entries[0].state = NsState::Removing; @@ -338,7 +375,8 @@ CompletedRemovingFixture seedCompletedRemoving( } void seedCompletedRemovingBatch( - DrainRaceBackend & backend, const PoolPtr & store, const UInt128 & lease_owner, size_t count) + DrainRaceBackend & backend, CasOperation & op, const PoolPtr & store, const UInt128 & lease_owner, + size_t count) { const Layout & layout = store->layout(); std::vector entries; @@ -349,10 +387,10 @@ void seedCompletedRemovingBatch( .ns = RootNamespace{fmt::format("00/drain-batch-{}@cas@", i)}, .state = NsState::Live, .incarnation = UInt128{200 + i}}; - CasRefCatalog::casAdmitEntry(backend, layout, store->poolConfig().gc_shards, entry); + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, entry); entries.push_back(std::move(entry)); } - CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & current) + CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { RefCatalog next = current; for (CatalogEntry & entry : next.entries) @@ -804,6 +842,8 @@ TEST(CASGCFrontierGate, AnUndecodableCheckpointAnomalySuppressesEveryDeleteFamil { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); Gc gc(store, kGc); @@ -815,12 +855,12 @@ TEST(CASGCFrontierGate, AnUndecodableCheckpointAnomalySuppressesEveryDeleteFamil /// the very object the round's own life resolution will read, or the round folds normally and this /// test measures nothing. const std::optional damaged_life = - CasRefCatalog::lifeIfCataloged(*backend, layout, damaged); + CasRefCatalog::lifeIfCataloged(op, layout, damaged); ASSERT_TRUE(damaged_life.has_value()) << "the publish must have left a catalog entry to resolve"; - const std::optional damaged_ckpt = readCkpt(*backend, layout, *damaged_life); + const std::optional damaged_ckpt = readCkpt(op, layout, *damaged_life); ASSERT_TRUE(damaged_ckpt.has_value()) << "the publish must have left a `_ckpt` to damage"; - ASSERT_EQ(backend->casPut(layout.refCkptKey(*damaged_life), "not a checkpoint", - damaged_ckpt->token).outcome, CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(op.replace( + layout.refCkptKey(*damaged_life), "not a checkpoint", damaged_ckpt->incarnation, Retry::once()))); backend->resetCounts(); std::vector anomaly_counts; @@ -890,6 +930,8 @@ TEST(CASGCFrontierGate, ACarriedHoldSuppressesEveryDeleteFamily) TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSuppressesEveryDeleteFamily) { auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolWithProbeBudget(backend, /*budget*/ 0); const Layout & layout = store->layout(); @@ -908,12 +950,12 @@ TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSuppressesEveryDeleteFamily) << "without a sealed cursor the namespace never becomes a budget-spending probe target"; const std::optional quiet_life = - CasRefCatalog::lifeIfCataloged(*backend, layout, quiet); + CasRefCatalog::lifeIfCataloged(op, layout, quiet); ASSERT_TRUE(quiet_life.has_value()); - const std::optional quiet_ckpt = readCkpt(*backend, layout, *quiet_life); + const std::optional quiet_ckpt = readCkpt(op, layout, *quiet_life); ASSERT_TRUE(quiet_ckpt.has_value()) << "there must be a `_ckpt` to remove"; - ASSERT_EQ(backend->deleteExact(layout.refCkptKey(*quiet_life), quiet_ckpt->token).kind, - DeleteOutcome::Kind::Deleted); + ASSERT_EQ(op.remove(layout.refCkptKey(*quiet_life), quiet_ckpt->incarnation, Retry::once()), + Removal::Removed); backend->hidePrefix(layout.namespaceStreamPrefix(*quiet_life)); backend->resetCounts(); @@ -943,6 +985,8 @@ TEST(CASGCFrontierGate, ADecodedTokenBearingEmptyCatalogCompletesTheFrontierAndD { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const DB::UInt128 blob(0xbead); @@ -952,12 +996,14 @@ TEST(CASGCFrontierGate, ADecodedTokenBearingEmptyCatalogCompletesTheFrontierAndD /// round leaves one (`injectRetire`). writeBlobBody(*backend, layout, blob); const BlobRef blob_ref = legacyMetaTestRef(blob); - const Token blob_token = backend->head(layout.blobKey(blob_ref)).token; + const std::optional blob_observed = op.head(layout.blobKey(blob_ref), Retry::once()); + ASSERT_TRUE(blob_observed) << "the seeded blob body must be present before it is condemned"; + const PersistedIncarnation blob_token = PersistedIncarnation::capture(blob_observed->incarnation); injectRetire(*backend, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = blob_ref, .token = blob_token, .size = 0}}); store->renewWatermarkOnce(); - ASSERT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()) + ASSERT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()) << "the scenario needs a genuinely, provably empty catalog, or this test measures nothing"; Gc gc(store, kGc); @@ -1032,12 +1078,16 @@ TEST(CASGCFrontierGate, AZeroWalkableFrontierWithACreatingCatalogRowIsNotProvedE { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const DB::UInt128 blob(0xbead); writeBlobBody(*backend, layout, blob); const BlobRef blob_ref = legacyMetaTestRef(blob); - const Token blob_token = backend->head(layout.blobKey(blob_ref)).token; + const std::optional blob_observed = op.head(layout.blobKey(blob_ref), Retry::once()); + ASSERT_TRUE(blob_observed) << "the seeded blob body must be present before it is condemned"; + const PersistedIncarnation blob_token = PersistedIncarnation::capture(blob_observed->incarnation); injectRetire(*backend, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = blob_ref, .token = blob_token, .size = 0}}); store->renewWatermarkOnce(); @@ -1049,7 +1099,7 @@ TEST(CASGCFrontierGate, AZeroWalkableFrontierWithACreatingCatalogRowIsNotProvedE entry.incarnation = hexToU128("00000000000000000000000000000042"); entry.creator = CreatorFence{ .server_root_id = "test-stalled-creator", .writer_epoch = 1, .fence_generation = 1}; - CasRefCatalog::casAdmitEntry(*backend, layout, /*gc_shards*/ 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, /*gc_shards*/ 1, entry); Gc gc(store, kGc); backend->resetCounts(); @@ -1190,7 +1240,11 @@ TEST(CASGCFrontierGate, AProvedEmptyCatalogUnderStageASuppressedStaysSuppressed) writeBlobBody(*backend, layout, blob); const BlobRef blob_ref = legacyMetaTestRef(blob); - const Token blob_token = backend->head(layout.blobKey(blob_ref)).token; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const std::optional blob_observed = op.head(layout.blobKey(blob_ref), Retry::once()); + ASSERT_TRUE(blob_observed) << "the seeded blob body must be present before it is condemned"; + const PersistedIncarnation blob_token = PersistedIncarnation::capture(blob_observed->incarnation); injectRetire(*backend, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = blob_ref, .token = blob_token, .size = 0}}); store->renewWatermarkOnce(); @@ -1233,6 +1287,8 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace doomed{"00/doomed@cas@"}; @@ -1269,7 +1325,7 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob remove_op.kind = RefOpKind::RemoveNamespace; const uint64_t remove_seq = appendRefLogSeed(*backend, layout, doomed, {remove_op}); publishRecoverableCkptForSemanticWrapper(*backend, layout, doomed, RefTxnId{1, remove_seq}); - CasRefCatalog::casUpdate(*backend, layout, [&](const RefCatalog & current) -> RefCatalog + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) -> RefCatalog { RefCatalog next = current; const auto it = std::find_if(next.entries.begin(), next.entries.end(), @@ -1299,7 +1355,7 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob gc.setPostHotScanCatalogReadHookForTest([&]() { hook_fired = true; - ASSERT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()) + ASSERT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()) << "the race must land inside the window where the cut itself is already empty"; const RootNamespace newborn{"00/newborn@cas@"}; @@ -1527,6 +1583,8 @@ TEST(CASGCFrontierGate, TheOrphanManifestSweepAndItsCursorAreInertUnderSuppressi TEST(CASGCFrontierGate, APartialProbeBudgetPublishesATallyThatMatchesTheSealedSet) { auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolWithProbeBudget(backend, /*budget*/ 1); const Layout & layout = store->layout(); const RootNamespace a{"00/quiet_a@cas@"}; @@ -1574,8 +1632,8 @@ TEST(CASGCFrontierGate, APartialProbeBudgetPublishesATallyThatMatchesTheSealedSe { EXPECT_NE(sealedCursorOf(*backend, layout, ns), (RefTxnId{})) << "every namespace in the tally must have a sealed cursor: " << ns.string(); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); - const auto checkpoint = readCkpt(*backend, layout, life); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); + const auto checkpoint = readCkpt(op, layout, life); ASSERT_TRUE(checkpoint.has_value()); EXPECT_EQ(checkpoint->ckpt.committed_through, (RefTxnId{1, 1})) << "LIST omission and the probe budget do not alter a valid CTE"; @@ -1634,6 +1692,8 @@ TEST(CASGCFrontierGate, CheckpointFrontierBehindAnInheritedCursorFailsClosed) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/checkpoint-behind-inherited-cursor@cas@"}; @@ -1651,7 +1711,7 @@ TEST(CASGCFrontierGate, CheckpointFrontierBehindAnInheritedCursorFailsClosed) ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); ASSERT_EQ(sealedCursorOf(*backend, layout, ns), (RefTxnId{1, 2})); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String checkpoint_key = layout.refCkptKey(life); const HeadResult checkpoint_head = backend->head(checkpoint_key); ASSERT_TRUE(checkpoint_head.exists); @@ -1683,6 +1743,8 @@ TEST(CASGCFrontierGate, CheckpointFrontierCrossesAnInheritedEpochSeal) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/checkpoint-inherited-seal-crossing@cas@"}; const DB::UInt128 crossed_blob(0xfd); @@ -1703,7 +1765,7 @@ TEST(CASGCFrontierGate, CheckpointFrontierCrossesAnInheritedEpochSeal) publishAt(*backend, layout, ns, RefTxnId{2, 1}, "crossed", 2, crossed_blob, /*birth=*/false, /*prev_epoch_seal=*/RefTxnId{1, 2}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String checkpoint_key = layout.refCkptKey(life); const HeadResult checkpoint_head = backend->head(checkpoint_key); ASSERT_TRUE(checkpoint_head.exists); @@ -1772,6 +1834,8 @@ TEST(CASGCFrontierGate, AWronglyQuietNamespaceIsWalkedTheSameRound) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace quiet{"00/quiet@cas@"}; const DB::UInt128 late_blob(0x77); @@ -1791,7 +1855,7 @@ TEST(CASGCFrontierGate, AWronglyQuietNamespaceIsWalkedTheSameRound) /// A second publish lands, and the store hides the namespace from every LIST at the same moment. publish(*backend, layout, quiet, "ref_2", 2, late_blob); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, quiet); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, quiet); const String checkpoint_key = layout.refCkptKey(life); const HeadResult checkpoint_head = backend->head(checkpoint_key); ASSERT_TRUE(checkpoint_head.exists); @@ -1819,6 +1883,8 @@ TEST(CASGCFrontierGate, CheckpointFrontierBoundsOrdinaryFoldBeforeDurableSuccess { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/checkpoint-bounds-fold@cas@"}; const DB::UInt128 committed_blob(0xf1); @@ -1854,7 +1920,7 @@ TEST(CASGCFrontierGate, CheckpointFrontierBoundsOrdinaryFoldBeforeDurableSuccess ASSERT_TRUE(report.acquired_lease); gc.setPhaseSink({}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); EXPECT_EQ(sealedCursorOf(*backend, layout, ns), (RefTxnId{1, 1})); EXPECT_EQ(inDegreeOf(*backend, layout, beyond_frontier_blob), 0) << "a durable log above `_ckpt.committed_through` is not foldable history"; @@ -1875,6 +1941,8 @@ TEST(CASGCFrontierGate, ConsumedCheckpointFrontierProvesOrdinaryLifeWithoutSucce { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/checkpoint-complete-fold@cas@"}; @@ -1899,7 +1967,7 @@ TEST(CASGCFrontierGate, ConsumedCheckpointFrontierProvesOrdinaryLifeWithoutSucce ASSERT_TRUE(report.acquired_lease); gc.setPhaseSink({}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); EXPECT_EQ(sealedCursorOf(*backend, layout, ns), (RefTxnId{1, 1})); EXPECT_EQ(backend->getCount(layout.refLogKey(life, RefTxnId{1, 2})), 0u) << "the checkpoint boundary proves the cut without a post-frontier 404"; @@ -1915,6 +1983,8 @@ TEST(CASGCFrontierGate, CheckpointFrontierProvesLifeWithHiddenDurableSuccessor) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/checkpoint-hidden-successor@cas@"}; const DB::UInt128 beyond_frontier_blob(0xf4); @@ -1937,7 +2007,7 @@ TEST(CASGCFrontierGate, CheckpointFrontierProvesLifeWithHiddenDurableSuccessor) .last_epoch_seal = std::nullopt, }); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); backend->hide(layout.refLogKey(life, RefTxnId{1, 2})); std::map intake; @@ -1968,6 +2038,8 @@ TEST(CASGCFrontierGate, MissingCommittedCheckpointLogHoldsInsteadOfProvingTheFro { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/missing-committed-checkpoint-log@cas@"}; @@ -1981,7 +2053,7 @@ TEST(CASGCFrontierGate, MissingCommittedCheckpointLogHoldsInsteadOfProvingTheFro .last_epoch_seal = std::nullopt, }); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String missing_key = layout.refLogKey(life, RefTxnId{1, 2}); const HeadResult missing_head = backend->head(missing_key); ASSERT_TRUE(missing_head.exists); @@ -2010,6 +2082,8 @@ TEST(CASGCFrontierGate, HiddenCommittedCheckpointLogIsFoldedThroughTheAuthorityC { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/hidden-committed-checkpoint-log@cas@"}; const DB::UInt128 hidden_blob(0xf8); @@ -2024,7 +2098,7 @@ TEST(CASGCFrontierGate, HiddenCommittedCheckpointLogIsFoldedThroughTheAuthorityC .last_epoch_seal = std::nullopt, }); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); backend->hide(layout.refLogKey(life, RefTxnId{1, 2})); std::map intake; @@ -2079,6 +2153,8 @@ TEST(CASGCFrontierGate, EmptyCheckpointFrontierRejectsAnInheritedCursor) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/empty-checkpoint-after-cursor@cas@"}; @@ -2095,7 +2171,7 @@ TEST(CASGCFrontierGate, EmptyCheckpointFrontierRejectsAnInheritedCursor) ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); ASSERT_EQ(sealedCursorOf(*backend, layout, ns), (RefTxnId{1, 1})); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String checkpoint_key = layout.refCkptKey(life); const HeadResult checkpoint_head = backend->head(checkpoint_key); ASSERT_TRUE(checkpoint_head.exists); @@ -2127,6 +2203,8 @@ TEST(CASGCFrontierGate, CatalogLifeWithoutCheckpointDefersWithoutUsingListedFron { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/missing-checkpoint-fold@cas@"}; const DB::UInt128 blob(0xc7); @@ -2137,8 +2215,8 @@ TEST(CASGCFrontierGate, CatalogLifeWithoutCheckpointDefersWithoutUsingListedFron writeManifestRaw(*backend, layout, ns, manifest, {blobEntryFor("data.bin", blob)}); appendRefLogSeed(*backend, layout, ns, publishCommittedOps("must_remain_unfolded", manifest)); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); - ASSERT_FALSE(readCkpt(*backend, layout, life).has_value()); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); + ASSERT_FALSE(readCkpt(op, layout, life).has_value()); std::map intake; Gc gc(store, kGc); @@ -2169,6 +2247,8 @@ TEST(CASGCFrontierGate, CatalogLifeWithoutCheckpointDefersWithoutUsingListedFron TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSealsCursorsAndDeletesNothing) { auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolWithProbeBudget(backend, /*budget*/ 0); const Layout & layout = store->layout(); const RootNamespace quiet{"00/quiet@cas@"}; @@ -2198,8 +2278,8 @@ TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSealsCursorsAndDeletesNothing) << "the busy life's removal remains reclaimable despite the quiet LIST omission"; EXPECT_EQ(sealedCursorOf(*backend, layout, quiet), quiet_cursor) << "the unprobed namespace's cursor rides verbatim -- it is never dropped"; - const NamespaceLifeId quiet_life = *CasRefCatalog::lifeIfCataloged(*backend, layout, quiet); - const auto quiet_checkpoint = readCkpt(*backend, layout, quiet_life); + const NamespaceLifeId quiet_life = *CasRefCatalog::lifeIfCataloged(op, layout, quiet); + const auto quiet_checkpoint = readCkpt(op, layout, quiet_life); ASSERT_TRUE(quiet_checkpoint.has_value()); EXPECT_EQ(quiet_checkpoint->ckpt.committed_through, quiet_cursor) << "the quiet life's valid CTE is unaffected by LIST omission and a zero probe budget"; @@ -2216,6 +2296,8 @@ TEST(CASGCFrontierGate, ACommittedGapIsRedetectedAndSuppressesEveryRound) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace held{"00/held@cas@"}; const RootNamespace busy{"00/busy@cas@"}; @@ -2256,8 +2338,8 @@ TEST(CASGCFrontierGate, ACommittedGapIsRedetectedAndSuppressesEveryRound) EXPECT_GT(first_intake["tables_held"], 0u); EXPECT_FALSE(first_round.anomalies.empty()); - const NamespaceLifeId held_life = *CasRefCatalog::lifeIfCataloged(*backend, layout, held); - const auto held_checkpoint = readCkpt(*backend, layout, held_life); + const NamespaceLifeId held_life = *CasRefCatalog::lifeIfCataloged(op, layout, held); + const auto held_checkpoint = readCkpt(op, layout, held_life); ASSERT_TRUE(held_checkpoint.has_value()); EXPECT_EQ(held_checkpoint->ckpt.committed_through, (RefTxnId{1, 4})); @@ -2295,7 +2377,7 @@ TEST(CASGCFrontierGate, ACommittedGapIsRedetectedAndSuppressesEveryRound) EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists); EXPECT_EQ(sealedCursorOf(*backend, layout, held), (RefTxnId{1, 2})) << "the committed gap remains unresolved and the cursor cannot advance through it"; - const auto final_checkpoint = readCkpt(*backend, layout, held_life); + const auto final_checkpoint = readCkpt(op, layout, held_life); ASSERT_TRUE(final_checkpoint.has_value()); EXPECT_EQ(final_checkpoint->ckpt.committed_through, (RefTxnId{1, 4})); } @@ -2536,6 +2618,8 @@ TEST(CASGCFrontierGateCleanupRange, ASnapshotAtTheCheckpointSurvivesAndOnlyStric TEST(CASGCFrontierGateCleanupRange, CheckpointBaseValidatorRejectsMissingLogSnapshotAndSeal) { auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const RefTxnId base{1, 1}; const RefCkpt checkpoint{ @@ -2543,36 +2627,38 @@ TEST(CASGCFrontierGateCleanupRange, CheckpointBaseValidatorRejectsMissingLogSnap .committed_through = base, .checkpoint_snapshot_id = base, .last_epoch_seal = std::nullopt}; - CasRefCatalog::initializeEmptyForNewPool(*backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); { const RootNamespace ns{"00/cleanup-missing-base-log@cas@"}; fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, ns).value(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, ns).value(); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base)); - EXPECT_THROW((void)readCheckpointSnapshotBase(*backend, layout, life, checkpoint), DB::Exception); + EXPECT_THROW((void)readCheckpointSnapshotBase(op, layout, life, checkpoint), DB::Exception); } { const RootNamespace ns{"00/cleanup-missing-base-snapshot@cas@"}; fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = ns.string(), .txn_id = base, .ops = {namespaceBirthOp()}, .prev_epoch_seal = std::nullopt}); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, ns).value(); - EXPECT_THROW((void)readCheckpointSnapshotBase(*backend, layout, life, checkpoint), DB::Exception); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, ns).value(); + EXPECT_THROW((void)readCheckpointSnapshotBase(op, layout, life, checkpoint), DB::Exception); } { const RootNamespace ns{"00/cleanup-seal-is-not-base@cas@"}; writeSealAt(*backend, layout, ns, base); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, ns).value(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, ns).value(); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base)); - EXPECT_THROW((void)readCheckpointSnapshotBase(*backend, layout, life, checkpoint), DB::Exception); + EXPECT_THROW((void)readCheckpointSnapshotBase(op, layout, life, checkpoint), DB::Exception); } } TEST(CASGCFrontierGateCleanupRange, LaterEpochBaseWithoutItsContextualBacklinkCannotLicenseDeletion) { auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; - CasRefCatalog::initializeEmptyForNewPool(*backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); const RefTxnId seal_id{1, 2}; const RefTxnId base_id{2, 1}; @@ -2585,12 +2671,12 @@ TEST(CASGCFrontierGateCleanupRange, LaterEpochBaseWithoutItsContextualBacklinkCa fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = ns.string(), .txn_id = base_id, .ops = {}, .prev_epoch_seal = backlink}); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base_id)); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, ns).value(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, ns).value(); std::optional validated_base; try { - (void)readCheckpointSnapshotBase(*backend, layout, life, RefCkpt{ + (void)readCheckpointSnapshotBase(op, layout, life, RefCkpt{ .life_epoch = 1, .committed_through = base_id, .checkpoint_snapshot_id = base_id, @@ -2619,6 +2705,8 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace removed{"00/removed@cas@"}; const RefOp birth_op = namespaceBirthOp(); @@ -2628,8 +2716,8 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito .ns = removed.string(), .txn_id = RefTxnId{1, 1}, .ops = {birth_op}, .prev_epoch_seal = std::nullopt}); fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = removed.string(), .txn_id = RefTxnId{1, 2}, .ops = {remove_op}, .prev_epoch_seal = std::nullopt}); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, removed).value(); - CasRefCatalog::casUpdate(*backend, layout, [&](const RefCatalog & current) + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, removed).value(); + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find_if(next.entries.begin(), next.entries.end(), [&](const CatalogEntry & entry) @@ -2652,7 +2740,7 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito /// The removal evidence must arise from a replay-valid terminal lifecycle, rather than merely /// from a raw terminal record that the recovery state machine refuses. const RecoveredRefTable recovered = recoverRefTableDetailedAtCatalogCutForTest( - *backend, layout, CasRefCatalog::read(*backend, layout), removed); + *backend, layout, CasRefCatalog::read(op, layout), removed); EXPECT_EQ(recovered.state.getLifecycle(), RefLifecycle::Removed); EXPECT_EQ(recovered.state.getRemoveTxnId(), (RefTxnId{1, 2})); @@ -2689,8 +2777,8 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); gc.setPhaseSink({}); - EXPECT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, removed)); + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, removed)); ASSERT_FALSE(janitor_metrics.empty()) << "the namespace_cleanup phase must have run this round"; EXPECT_GE(janitor_metrics.at("janitor_deleted"), 1u) << "the janitor's OWN counter must show the delete -- now that the proved-empty gate has " @@ -2706,6 +2794,8 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace removed{"00/post-fold-unreadable@cas@"}; const RootNamespace progressing{"00/post-fold-progress@cas@"}; @@ -2718,8 +2808,8 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = removed.string(), .txn_id = RefTxnId{1, 2}, .ops = {remove_op}, .prev_epoch_seal = std::nullopt}); - const NamespaceLifeId removed_life = CasRefCatalog::lifeIfCataloged(*backend, layout, removed).value(); - CasRefCatalog::casUpdate(*backend, layout, [&](const RefCatalog & current) + const NamespaceLifeId removed_life = CasRefCatalog::lifeIfCataloged(op, layout, removed).value(); + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find_if(next.entries.begin(), next.entries.end(), [&](const CatalogEntry & entry) @@ -2772,9 +2862,9 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro gc.setPhaseSink({}); ASSERT_TRUE(report.acquired_lease); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, removed)) + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, removed)) << "post-fold physical cleanup cannot gate catalog removal"; - EXPECT_TRUE(CasRefCatalog::lifeIfCataloged(*backend, layout, progressing)); + EXPECT_TRUE(CasRefCatalog::lifeIfCataloged(op, layout, progressing)); EXPECT_EQ(report.manifests_deleted, 1u) << "the janitor leak cannot promote itself into pool-wide destructive suppression"; EXPECT_FALSE(backend->head(layout.manifestKey(manifest_id)).exists); @@ -2843,20 +2933,14 @@ TEST(CASCatalogLifecycleReconciler, EmptyCatalogReturnsAuthoritativeCompleteCut) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - ASSERT_TRUE(CasRefCatalog::initializeEmptyForNewPool(*backend, layout).catalog.entries.empty()); + ASSERT_TRUE(CasRefCatalog::initializeEmptyForNewPool(op, layout).catalog.entries.empty()); CasFoldSeal parent; - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [](uint64_t) - { - return CasRefCatalog::LeaderFenceStatus::Held; - }); - const CatalogLifecycleReconcileResult result = reconciler.reconcile(); + CatalogLifecycleReconciler reconciler(op, layout, parent); + const CatalogLifecycleReconcileResult result = reconciler.reconcile(noAuthorityRefresh); EXPECT_EQ(result.authority_status, AuthorityStatus::Authoritative); EXPECT_EQ(result.catalog_resolution, CatalogResolution::DrainComplete); @@ -2870,25 +2954,19 @@ TEST(CASCatalogLifecycleReconciler, DeletesEligibleRowsFromReturnedResolutionCut { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); constexpr size_t deletes = 3; - seedCompletedRemovingBatch(*backend, store, kGc, deletes); + seedCompletedRemovingBatch(*backend, op, store, kGc, deletes); const auto parent_object = backend->get(layout.foldSealKey(1, 1)); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); backend->clearJournal(); backend->resetCounts(); - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [](uint64_t) - { - return CasRefCatalog::LeaderFenceStatus::Held; - }); - const CatalogLifecycleReconcileResult result = reconciler.reconcile(); + CatalogLifecycleReconciler reconciler(op, layout, parent); + const CatalogLifecycleReconcileResult result = reconciler.reconcile(noAuthorityRefresh); EXPECT_EQ(result.authority_status, AuthorityStatus::Authoritative); EXPECT_EQ(result.catalog_resolution, CatalogResolution::DrainComplete); @@ -2905,33 +2983,34 @@ TEST(CASCatalogLifecycleReconciler, ReturnsRetiredLifeWhenAuthorityMovesAfterRes { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); const auto parent_object = backend->get(layout.foldSealKey(1, 1)); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); - size_t fence_checks = 0; - - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [&fence_checks](uint64_t) - { - ++fence_checks; - return fence_checks == 2 - ? CasRefCatalog::LeaderFenceStatus::Moved - : CasRefCatalog::LeaderFenceStatus::Held; - }); - const CatalogLifecycleReconcileResult result = reconciler.reconcile(); + + /// Admission is lost after the erase has been resolved: the second catalog read of the drain is + /// the resolution cut, so the row is already gone when the loop's next verdict finds no admission. + size_t catalog_reads = 0; + bool authority_held = true; + CasOperation fenced_op = requests.admit([&] { return authority_held; }); + backend->afterReadOf(layout.refCatalogKey(), [&] + { + if (++catalog_reads == 2) + authority_held = false; + }); + + CatalogLifecycleReconciler reconciler(fenced_op, layout, parent); + const CatalogLifecycleReconcileResult result = reconciler.reconcile(noAuthorityRefresh); EXPECT_EQ(result.authority_status, AuthorityStatus::FencedOut); EXPECT_EQ(result.catalog_resolution, CatalogResolution::ExactRowAbsent); ASSERT_EQ(result.retired_lives.size(), 1); EXPECT_EQ(result.retired_lives.front(), NamespaceLifeId::fromCatalogEntry(fixture.ns, fixture.life_id)); - EXPECT_EQ(result.deleted, 0); + EXPECT_EQ(result.deleted, 1); EXPECT_FALSE(result.final_catalog_cut); } @@ -2939,33 +3018,33 @@ TEST(CASCatalogLifecycleReconciler, InitialFenceLossReportsEligibleRowStillPrese { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); const auto parent_object = backend->get(layout.foldSealKey(1, 1)); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); backend->resetCounts(); - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [](uint64_t) - { - return CasRefCatalog::LeaderFenceStatus::Moved; - }); - const CatalogLifecycleReconcileResult result = reconciler.reconcile(); + /// Admission is lost the moment the selection cut has been read, so the erase is never sent. + bool authority_held = true; + CasOperation fenced_op = requests.admit([&] { return authority_held; }); + backend->afterReadOf(layout.refCatalogKey(), [&] { authority_held = false; }); + + CatalogLifecycleReconciler reconciler(fenced_op, layout, parent); + const CatalogLifecycleReconcileResult result = reconciler.reconcile(noAuthorityRefresh); EXPECT_EQ(result.authority_status, AuthorityStatus::FencedOut); EXPECT_EQ(result.catalog_resolution, CatalogResolution::ExactRowStillPresent); EXPECT_TRUE(result.retired_lives.empty()); EXPECT_EQ(result.deleted, 0); EXPECT_FALSE(result.final_catalog_cut); - EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 2) - << "the initial selection and mandatory erase-resolution cuts are the only catalog reads"; - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 0); - EXPECT_EQ(CasRefCatalog::lifeIfCataloged(*backend, layout, fixture.ns), + EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1) + << "an operation whose admission is gone before the erase reports the selection cut it " + "already holds and reads nothing further"; + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0); + EXPECT_EQ(CasRefCatalog::lifeIfCataloged(op, layout, fixture.ns), NamespaceLifeId::fromCatalogEntry(fixture.ns, fixture.life_id)); } @@ -2973,8 +3052,10 @@ TEST(CASCatalogLifecycleReconciler, RetriesFromTheMandatoryConflictResolutionCut { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - seedCompletedRemoving(*backend, store, kGc); + seedCompletedRemoving(*backend, op, store, kGc); const auto parent_object = backend->get(layout.foldSealKey(1, 1)); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); @@ -2982,118 +3063,137 @@ TEST(CASCatalogLifecycleReconciler, RetriesFromTheMandatoryConflictResolutionCut backend->resetCounts(); backend->conflictNextCatalogCas(layout.refCatalogKey()); - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [](uint64_t) - { - return CasRefCatalog::LeaderFenceStatus::Held; - }); - const CatalogLifecycleReconcileResult result = reconciler.reconcile(); + CatalogLifecycleReconciler reconciler(op, layout, parent); + const CatalogLifecycleReconcileResult result = reconciler.reconcile(noAuthorityRefresh); EXPECT_EQ(result.authority_status, AuthorityStatus::Authoritative); EXPECT_EQ(result.catalog_resolution, CatalogResolution::DrainComplete); EXPECT_EQ(result.deleted, 1); const std::vector journal = backend->journalSnapshot(); const String catalog_get = "get " + layout.refCatalogKey(); - EXPECT_EQ(std::count(journal.begin(), journal.end(), catalog_get), 3) - << "the token-conflict retry must reuse its mandatory resolution cut"; + EXPECT_EQ(std::count(journal.begin(), journal.end(), catalog_get), 4) + << "selection, the refused write's own resolve read, the mandatory resolution cut the retry " + "reuses, and the committed erase's resolution -- the catalog takes no cut of its own"; } -TEST(CASCatalogLifecycleReconciler, PropagatesAuthorityFailureBeforeEraseCas) +/// THE DRAIN'S AUTHORITY, end to end. `CatalogLifecycleReconciler` and +/// `deleteCompletedRemovingAtSnapshot` decide `FencedOut` from `CasOperation::admitted()`, and the GC +/// plane's fence is open -- so the only thing that can make that verdict false in production is the +/// `Liveness` the round hands its drain operation. Depose the leader in the window the round leaves +/// between acquiring its lease and the pre-fold drain, and the drain must erase nothing. Without the +/// predicate the verdict is a constant TRUE, the drain completes, and the completed-removal row is +/// gone -- which is what this test catches. +TEST(CASGCFrontierGate, ADeposedLeaderErasesNoCatalogRow) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - seedCompletedRemoving(*backend, store, kGc); - const auto parent_object = backend->get(layout.foldSealKey(1, 1)); - ASSERT_TRUE(parent_object); - const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); - size_t fence_checks = 0; - - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [&fence_checks](uint64_t) - { - if (++fence_checks == 2) - throw std::runtime_error("injected reconciler authority failure before CAS"); - return CasRefCatalog::LeaderFenceStatus::Held; - }); - try - { - (void)reconciler.reconcile(); - FAIL() << "the authority exception must propagate"; - } - catch (const std::runtime_error & e) - { - EXPECT_STREQ(e.what(), "injected reconciler authority failure before CAS"); - } -} + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const uint64_t catalog_writes_before = backend->putOverwriteCount(layout.refCatalogKey()); + + /// Another leader steals `gc/state` after this round's lease renewal and before its drain. + const auto depose = [&] + { + const auto got = op.read(layout.gcStateKey(), Retry::once()); + ASSERT_TRUE(got); + GcState stolen = decodeGcState(got->bytes); + stolen.lease.owner = hexToU128("00000000000000000000000000000099"); + ++stolen.lease.seq; + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.gcStateKey(), encodeGcState(stolen), got->incarnation, Retry::once()))); + }; -TEST(CASCatalogLifecycleReconciler, PropagatesAuthorityFailureAfterMandatoryResolution) + Gc gc(store, kGc); + EXPECT_THROW(gc.runRegularRound(depose), DB::Exception) + << "a deposed leader must give up rather than drain the catalog"; + EXPECT_TRUE(CasRefCatalog::lifeIfCataloged(op, layout, fixture.ns)) + << "the completed-removal row survives a deposed leader's drain"; + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_writes_before) + << "and no catalog write was even attempted"; +} + +/// THE SAME AUTHORITY, now DURING the drain. A drain erases one row per iteration, and what stops a +/// leader deposed between two erases is the refresh the ERASE runs at the top of every attempt (the +/// reconciler only forwards it): the first row goes, the second is never attempted, and the drain +/// reports `FencedOut` from the cut it already holds. One reading taken before the drain would +/// authorise both erases: the erase count and the surviving-row count below are what catch that, +/// since an unrefreshed drain sends a second erase and empties the catalog under a lease this leader +/// no longer owns. +TEST(CASGCFrontierGate, ALeaderDeposedBetweenTwoErasesStopsAfterTheFirst) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); - const auto parent_object = backend->get(layout.foldSealKey(1, 1)); - ASSERT_TRUE(parent_object); - const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); - size_t fence_checks = 0; - - CatalogLifecycleReconciler reconciler( - *backend, - layout, - parent, - /*admitted_generation=*/1, - [&fence_checks](uint64_t) - { - if (++fence_checks == 3) - throw std::runtime_error("injected reconciler authority failure after resolution"); - return CasRefCatalog::LeaderFenceStatus::Held; - }); - try - { - (void)reconciler.reconcile(); - FAIL() << "the authority exception must propagate"; - } - catch (const std::runtime_error & e) - { - EXPECT_STREQ(e.what(), "injected reconciler authority failure after resolution"); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, fixture.ns)); - } + seedCompletedRemovingBatch(*backend, op, store, kGc, /*count=*/2); + const std::vector seeded{ + RootNamespace{"00/drain-batch-0@cas@"}, RootNamespace{"00/drain-batch-1@cas@"}}; + const uint64_t catalog_writes_before = backend->putOverwriteCount(layout.refCatalogKey()); + + /// The hook runs after every catalog read, and the first read it sees with an erase already behind + /// it is the resolution read that closed erase one -- exactly the window between the two erases. + /// Nothing before the drain reads or writes the catalog, so no earlier read can trip this. + bool deposed = false; + backend->afterReadOf(layout.refCatalogKey(), [&] + { + if (deposed || backend->putOverwriteCount(layout.refCatalogKey()) == catalog_writes_before) + return; + deposed = true; + const auto got = op.read(layout.gcStateKey(), Retry::once()); + ASSERT_TRUE(got); + GcState stolen = decodeGcState(got->bytes); + stolen.lease.owner = hexToU128("00000000000000000000000000000099"); + ++stolen.lease.seq; + EXPECT_TRUE(std::holds_alternative( + op.replace(layout.gcStateKey(), encodeGcState(stolen), got->incarnation, Retry::once()))); + }); + + Gc gc(store, kGc); + EXPECT_THROW(gc.runRegularRound(), DB::Exception) + << "a leader deposed inside its own drain must not finish the round"; + EXPECT_TRUE(deposed) << "the round must have reached a catalog read after its first erase"; + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_writes_before + 1) + << "one erase reached the store; the second was refused before it was sent"; + size_t still_cataloged = 0; + for (const RootNamespace & ns : seeded) + if (CasRefCatalog::lifeIfCataloged(op, layout, ns)) + ++still_cataloged; + EXPECT_EQ(still_cataloged, 1u) + << "one row was erased before the deposition; the other survives it"; } TEST(CASGCFrontierGate, HealthyRebuildUsesTheCatalogLifecycleReconciler) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); - const uint64_t catalog_cas_before = backend->casPutCount(layout.refCatalogKey()); + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); Gc gc(store, kGc); const RebuildReport result = gc.rebuildBaseline(/*force=*/true); EXPECT_TRUE(result.performed); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, fixture.ns)); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), catalog_cas_before + 1); + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, fixture.ns)); + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_cas_before + 1); } TEST(CASGCFrontierGate, DamagedStateRebuildDoesNotDeleteCompletedRemovingRows) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/damaged-rebuild-removing@cas@"}; - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, CatalogEntry{ + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{ .ns = ns, .state = NsState::Live, .incarnation = UInt128{901}}); - CasRefCatalog::casUpdate(*backend, layout, [](const RefCatalog & current) + CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { RefCatalog next = current; next.entries.front().state = NsState::Removing; @@ -3106,26 +3206,28 @@ TEST(CASGCFrontierGate, DamagedStateRebuildDoesNotDeleteCompletedRemovingRows) .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, }); - const uint64_t catalog_cas_before = backend->casPutCount(layout.refCatalogKey()); + const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); Gc gc(store, kGc); const RebuildReport result = gc.rebuildBaseline(/*force=*/false); EXPECT_TRUE(result.performed); - EXPECT_TRUE(CasRefCatalog::lifeIfCataloged(*backend, layout, ns)); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), catalog_cas_before); + EXPECT_TRUE(CasRefCatalog::lifeIfCataloged(op, layout, ns)); + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_cas_before); } TEST(CASGCFrontierGate, DeferredRoundDrainsCompletedRemovingBeforeReturning) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/100); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace removed{"00/deferred-removed@cas@"}; const UInt128 life_id{77}; - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, CatalogEntry{ + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{ .ns = removed, .state = NsState::Live, .incarnation = life_id}); - CasRefCatalog::casUpdate(*backend, layout, [&](const RefCatalog & current) + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { RefCatalog next = current; next.entries[0].state = NsState::Removing; @@ -3151,15 +3253,15 @@ TEST(CASGCFrontierGate, DeferredRoundDrainsCompletedRemovingBeforeReturning) const String ckpt_key = layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(removed, life_id)); ASSERT_EQ(backend->putIfAbsent(ckpt_key, "inert checkpoint debris").outcome, PutOutcome::Done); - const uint64_t catalog_cas_before = backend->casPutCount(layout.refCatalogKey()); + const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); Gc gc(store, kGc); const RoundReport report = runRegularRoundReclaiming(gc); ASSERT_TRUE(report.acquired_lease); EXPECT_TRUE(report.deferred); - EXPECT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, removed)); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), catalog_cas_before + 1); + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, removed)); + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_cas_before + 1); EXPECT_TRUE(backend->head(ckpt_key).exists); EXPECT_EQ(backend->deleteCount(ckpt_key), 0); } @@ -3168,9 +3270,11 @@ TEST(CASGCFrontierGate, StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListi { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const UInt128 leader_b = hexToU128("00000000000000000000000000000002"); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); backend->clearJournal(); backend->blockNextCatalogCas(layout.refCatalogKey()); @@ -3222,7 +3326,7 @@ TEST(CASGCFrontierGate, StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListi ASSERT_FALSE(leader_b_failure); ASSERT_TRUE(report_b.acquired_lease); ASSERT_FALSE(report_b.deferred); - ASSERT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()); + ASSERT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); const size_t catalog_cas_end = findJournalAfter(before_a_release, "cas_end " + layout.refCatalogKey(), 0); ASSERT_LT(catalog_cas_end, before_a_release.size()); @@ -3264,7 +3368,7 @@ TEST(CASGCFrontierGate, StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListi EXPECT_LT(successor_seal_put, successor_adoption); ASSERT_TRUE(leader_a_failure); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, fixture.ns)); + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, fixture.ns)); /// Same discrimination as `CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanitor`: leader_b's /// round both drops `fixture.ns`'s catalog row AND, because the resulting cut is genuinely, /// provably empty, opens the destructive gate -- so the namespace janitor reclaims the checkpoint @@ -3282,8 +3386,10 @@ TEST(CASGCFrontierGate, LostCatalogCasResponseIsResolvedBeforeListing) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); backend->clearJournal(); backend->loseNextCatalogCasResponse(layout.refCatalogKey()); @@ -3321,7 +3427,7 @@ TEST(CASGCFrontierGate, LostCatalogCasResponseIsResolvedBeforeListing) EXPECT_LT(conclusive_rescan, stream_list); EXPECT_LT(stream_list, fresh_catalog_cut); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, fixture.ns)); + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, fixture.ns)); /// See the discrimination comment in `CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanitor`: /// attribute the delete to the janitor's own counter, never to end-state absence alone, and never /// assume survival -- both would be indistinguishable from a bug on this exact line (the old @@ -3340,9 +3446,11 @@ TEST_P(CASGCCompletedRemovalFenceRace, FencedLeaderStopsAfterWinnerRemovesOrRepl { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const UInt128 leader_b = hexToU128("00000000000000000000000000000002"); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(fixture.ns, fixture.life_id); ASSERT_TRUE(store->refTableRecoveredForTest(fixture.ns)) @@ -3369,7 +3477,7 @@ TEST_P(CASGCCompletedRemovalFenceRace, FencedLeaderStopsAfterWinnerRemovesOrRepl backend->waitForBlockedCatalogCas(); transferGcLease(*backend, layout, leader_b); - const CasRefCatalog::Snapshot observed = CasRefCatalog::read(*backend, layout); + const CasRefCatalog::Snapshot observed = CasRefCatalog::read(op, layout); RefCatalog winner_catalog; if (GetParam() == CompetingCatalogOutcome::Replacement) { @@ -3387,9 +3495,9 @@ TEST_P(CASGCCompletedRemovalFenceRace, FencedLeaderStopsAfterWinnerRemovesOrRepl .last_epoch_seal = std::nullopt, })); } - ASSERT_EQ(backend->casPut( - layout.refCatalogKey(), encodeRefCatalog(winner_catalog), observed.token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(observed.incarnation); + ASSERT_TRUE(std::holds_alternative(op.replace( + layout.refCatalogKey(), encodeRefCatalog(winner_catalog), *observed.incarnation, Retry::once()))); backend->clearJournal(); const uint64_t plans_before /// NOLINT(clang-analyzer-deadcode.DeadStores) @@ -3443,9 +3551,11 @@ TEST(CASGCFrontierGate, CompletedRemovalDrainUsesNPlusOneCatalogReads) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); constexpr size_t deletes = 3; - seedCompletedRemovingBatch(*backend, store, kGc, deletes); + seedCompletedRemovingBatch(*backend, op, store, kGc, deletes); backend->clearJournal(); backend->resetCounts(); diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index 0a66240d4ca9..f8089907fed0 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -74,6 +74,9 @@ const UInt128 kGc = hexToU128("00000000000000000000000000000001"); class HintHoleCountingBackend : public CountingBackend { public: + /// Unhide the names the primitive overrides below would otherwise shadow. + using CountingBackend::list; + void hide(const String & key) { std::lock_guard lock(m); @@ -86,14 +89,14 @@ class HintHoleCountingBackend : public CountingBackend return served; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = CountingBackend::list(prefix, cursor, limit); + RawListPage page = CountingBackend::list(prefix, cursor, limit, access); std::lock_guard lock(m); if (hidden.empty()) return page; const size_t before = page.keys.size(); - std::erase_if(page.keys, [&](const ListedKey & k) { return hidden.contains(k.key); }); + std::erase_if(page.keys, [&](const RawListedKey & k) { return hidden.contains(k.key); }); if (page.keys.size() != before) ++served; return page; @@ -769,15 +772,14 @@ TEST(CASGCHoldGrammar, AWitnessThatStopsAnsweringIsWitnessDisappeared) class AlternatingGetBackend : public InMemoryBackend { public: - using DB::Cas::Backend::get; String flaky; size_t reads = 0; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (key == flaky && ++reads % 2 == 0) return std::nullopt; - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } }; @@ -1485,20 +1487,22 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa class BroadListHoleBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the name the primitive override below would otherwise shadow. using InMemoryBackend::list; + String hide_under_prefix; String hidden_key_infix; size_t holes_served = 0; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, + TransportAccess & access) override { - ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); if (prefix != hide_under_prefix) return page; const size_t before = page.keys.size(); std::erase_if(page.keys, - [&](const ListedKey & k) { return k.key.find(hidden_key_infix) != String::npos; }); + [&](const RawListedKey & k) { return k.key.find(hidden_key_infix) != String::npos; }); if (page.keys.size() != before) ++holes_served; return page; diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index d751aad975d2..63c672377d90 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -183,28 +183,29 @@ namespace class ThrowingBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the names the primitive overrides below would otherwise shadow. using InMemoryBackend::head; using InMemoryBackend::list; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (arm) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected backend list failure"); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (arm) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected backend get failure"); - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - HeadResult head(const String & key) override + std::optional head(const String & key, TransportAccess & access) override { if (arm) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected backend head failure"); - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } /// Armed only after Pool::open, so opening (which reads/initialises gc state) succeeds. @@ -315,13 +316,14 @@ TEST(CASGCLog, AbortedFinishOnThrowingRound) class NetworkThrowingBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the names the primitive overrides below would otherwise shadow. using InMemoryBackend::list; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (arm) throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected backend outage"); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } std::atomic arm{false}; }; @@ -330,6 +332,11 @@ TEST(CASGCLog, TransientThrowIsClassifiedAborted) { auto backend = std::make_shared(); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + /// A PERSISTENT transient fault is reissued for the whole retry window, so the window has to run + /// on a clock this test advances -- otherwise one read spends ninety real seconds. + std::atomic engine_now_ms{0}; + store->setCasRequestNowFnForTest([&] { return engine_now_ms.fetch_add(10'000) + 10'000; }); + store->setCasRetrySleepForTest([](uint64_t) {}); std::vector rows; DB::Cas::CasGcScheduler sched( @@ -357,21 +364,40 @@ TEST(CASGCLog, TransientErrorClassifierFailsClosed) EXPECT_FALSE(DB::Cas::isTransientGcRoundError(-1)); } -/// A backend that lets the round's FIRST `gc/state` CAS (the lease acquire/renew) through and throws -/// a transient error on the SECOND (the round-closing commit). The round therefore does all of its -/// pre-CAS work -- including condemning the dropped part -- and dies at `round_commit`. -class StateCommitThrowingBackend : public InMemoryBackend +/// A backend that REFUSES the round-closing `gc/state` write -- the one that advances +/// `snap_generation` -- and lets every other write through, the lease acquire/renew included. The round +/// therefore does all of its pre-CAS work, condemning the dropped part included, and dies at +/// `round_commit`. +/// +/// A refusal and not a throw, for two reasons. A thrown transport error is an ambiguity the engine +/// settles by an exact read and, while the precondition it named is unmoved, reissues to the policy +/// deadline -- so an armed fault would spend the whole retry window and end as a transport give-up. And +/// the arm is keyed on the generation rather than on a call count, because the acquire on a fresh pool +/// is an UNCONDITIONAL create that no count of conditional writes can see. +class StateCommitRefusingBackend : public InMemoryBackend { public: - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - if (arm && key.ends_with("gc/state") && ++state_puts_since_arm >= 2) - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected outage on the round-closing CAS"); - return InMemoryBackend::casPut(key, bytes, expected, meta); + if (arm.load() && expected_value && key.ends_with("gc/state")) + { + const auto stored = InMemoryBackend::read(key, access); + if (stored + && decodeGcState(bytes).snap_generation > decodeGcState(stored->bytes).snap_generation) + { + arm.store(false); + /// A store refuses a precondition only when the object moved, so move it: the same + /// bytes under a fresh incarnation is the smallest faithful move, and it leaves the + /// content alone so the assertions below stay about this round. + (void)InMemoryBackend::write(key, stored->bytes, stored->value, access); + return std::unexpected(RawConflict{}); + } + } + return InMemoryBackend::write(key, bytes, expected_value, access); } std::atomic arm{false}; - std::atomic state_puts_since_arm{0}; }; /// The Finish row of a THROWING round must still carry the counters of everything the round did @@ -380,7 +406,7 @@ class StateCommitThrowingBackend : public InMemoryBackend /// indistinguishable from a round that never got past the lease. TEST(CASGCLog, AbortedFinishCarriesProgressiveCounters) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .gc_fold_max_defer_rounds = 0}); const RootNamespace ns{"srv1/tbl"}; @@ -401,8 +427,10 @@ TEST(CASGCLog, AbortedFinishCarriesProgressiveCounters) ASSERT_EQ(round_rows.size(), 2u); const Rec & fin = round_rows[1]; EXPECT_EQ(fin.outcome, Rec::Outcome::Aborted); - EXPECT_EQ(fin.error_code, DB::ErrorCodes::NETWORK_ERROR); - EXPECT_EQ(fin.round, 0u) << "the commit CAS never landed, so the round number must stay unstamped"; + /// A refused precondition is settled by one exact read and reported as a conflict, so the round + /// is dropped whole and names the conflict rather than a transport error. + EXPECT_EQ(fin.error_code, DB::ErrorCodes::ABORTED); + EXPECT_EQ(fin.round, 0u) << "the commit never landed, so the round number must stay unstamped"; EXPECT_GT(fin.candidates_marked + fin.entries_condemned + fin.entries_graduated + fin.entries_redeleted + fin.objects_deleted + fin.fence_outs, 0u) << "the pre-CAS work the round performed must survive into its failure row"; @@ -417,24 +445,26 @@ TEST(CASGCLog, AbortedFinishCarriesProgressiveCounters) class ModalThrowingBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the names the primitive overrides below would otherwise shadow. using InMemoryBackend::list; + enum Mode : int { Off = 0, Transient = 1, Logic = 2 }; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { const int m = mode.load(); if (m == Transient) throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected backend outage"); if (m == Logic) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "injected logic failure"); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { if (key.ends_with("gc/hb")) ++hb_puts; - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } std::atomic mode{Off}; std::atomic hb_puts{0}; @@ -444,6 +474,11 @@ TEST(CASGCScheduler, TransientRoundFailureKeepsLeadershipAndHeartbeat) { auto backend = std::make_shared(); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + /// See `TransientThrowIsClassifiedAborted`: the transient mode is persistent while it is armed, so + /// the retry window runs on a clock this test advances. + std::atomic engine_now_ms{0}; + store->setCasRequestNowFnForTest([&] { return engine_now_ms.fetch_add(10'000) + 10'000; }); + store->setCasRetrySleepForTest([](uint64_t) {}); std::mutex rows_mutex; std::condition_variable rows_cv; diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 47715169db12..4d4dd313dae7 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -18,7 +18,7 @@ namespace class FailingMaintenanceReadBackend : public InMemoryBackend { public: - std::optional get(const String &, Range) override + std::optional read(const String &, DB::Cas::TransportAccess &) override { throw std::runtime_error("injected maintenance read failure"); } @@ -100,115 +100,142 @@ TEST(CASGCMaintenanceStateFormat, RejectsMalformedAndBoundsCursor) TEST(CASGCMaintenanceState, ReadsAndCasWithoutAdoptingConflicts) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); - const GcMaintenanceReadResult absent = readGcMaintenanceState(backend, layout); + auto op = requests.admit(); + + const GcMaintenanceReadResult absent = readGcMaintenanceState(op, layout); EXPECT_EQ(absent.status, GcMaintenanceReadStatus::Absent); EXPECT_FALSE(absent.state); - EXPECT_FALSE(absent.token); + EXPECT_FALSE(absent.incarnation); const GcMaintenanceState first{.janitor_cursor = "cas/ns/first"}; - const GcMaintenanceCasResult created = casGcMaintenanceState(backend, layout, std::nullopt, first); - EXPECT_EQ(created.outcome, GcMaintenanceCasOutcome::Committed); - const GcMaintenanceReadResult valid = readGcMaintenanceState(backend, layout); + ASSERT_TRUE(std::holds_alternative( + casGcMaintenanceState(op, layout, std::nullopt, first, Retry::standard()))); + const GcMaintenanceReadResult valid = readGcMaintenanceState(op, layout); ASSERT_EQ(valid.status, GcMaintenanceReadStatus::Valid); - ASSERT_TRUE(valid.token); + ASSERT_TRUE(valid.incarnation); ASSERT_TRUE(valid.state); EXPECT_EQ(*valid.state, first); - const GcMaintenanceCasResult advanced = casGcMaintenanceState(backend, layout, valid.token, - GcMaintenanceState{.janitor_cursor = "cas/ns/advanced"}); - ASSERT_EQ(advanced.outcome, GcMaintenanceCasOutcome::Committed); - const auto advanced_body = backend.get(key); - ASSERT_TRUE(advanced_body); - - ASSERT_EQ(backend.casPut(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), advanced_body->token).outcome, - CasOutcome::Committed); - const GcMaintenanceCasResult conflict = casGcMaintenanceState(backend, layout, valid.token, - GcMaintenanceState{.janitor_cursor = "loser"}); - EXPECT_EQ(conflict.outcome, GcMaintenanceCasOutcome::Conflict); - EXPECT_EQ(decodeGcMaintenanceState(backend.get(key)->bytes).janitor_cursor, "winner"); + const WriteResult advanced = casGcMaintenanceState(op, layout, valid.incarnation, + GcMaintenanceState{.janitor_cursor = "cas/ns/advanced"}, Retry::standard()); + ASSERT_TRUE(std::holds_alternative(advanced)); + const Incarnation advanced_incarnation = std::get(advanced).incarnation; + + ASSERT_TRUE(std::holds_alternative( + op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), advanced_incarnation, Retry::standard()))); + const WriteResult conflict = casGcMaintenanceState(op, layout, valid.incarnation, + GcMaintenanceState{.janitor_cursor = "loser"}, Retry::standard()); + EXPECT_TRUE(std::holds_alternative(conflict)); + EXPECT_EQ(decodeGcMaintenanceState(backend->get(key)->bytes).janitor_cursor, "winner"); } TEST(CASGCMaintenanceState, ClassifiesCorruptionAndResetsOnlyExactToken) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); - ASSERT_EQ(backend.putIfAbsent(key, "malformed").outcome, PutOutcome::Done); - const GcMaintenanceReadResult corrupt = readGcMaintenanceState(backend, layout); + auto op = requests.admit(); + + ASSERT_EQ(backend->putIfAbsent(key, "malformed").outcome, PutOutcome::Done); + const GcMaintenanceReadResult corrupt = readGcMaintenanceState(op, layout); ASSERT_EQ(corrupt.status, GcMaintenanceReadStatus::Corrupt); - ASSERT_TRUE(corrupt.token); + ASSERT_TRUE(corrupt.incarnation); EXPECT_FALSE(corrupt.state); EXPECT_FALSE(corrupt.diagnostic.empty()); - ASSERT_EQ(casGcMaintenanceState(backend, layout, corrupt.token, {}).outcome, GcMaintenanceCasOutcome::Committed); - EXPECT_EQ(decodeGcMaintenanceState(backend.get(key)->bytes), GcMaintenanceState{}); + ASSERT_TRUE(std::holds_alternative( + casGcMaintenanceState(op, layout, corrupt.incarnation, {}, Retry::standard()))); + EXPECT_EQ(decodeGcMaintenanceState(backend->get(key)->bytes), GcMaintenanceState{}); } TEST(CASGCMaintenanceState, UsesExactlyOneReadOrCasAttempt) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Absent); - EXPECT_EQ(backend.getCount(key), 1u); - - backend.resetCounts(); - ASSERT_EQ(casGcMaintenanceState(backend, layout, std::nullopt, {}).outcome, - GcMaintenanceCasOutcome::Committed); - EXPECT_EQ(backend.casPutCount(key), 1u); - EXPECT_EQ(backend.getCount(key), 0u); - - backend.resetCounts(); - EXPECT_EQ(casGcMaintenanceState(backend, layout, std::nullopt, - GcMaintenanceState{.janitor_cursor = "loser"}).outcome, GcMaintenanceCasOutcome::Conflict); - EXPECT_EQ(backend.casPutCount(key), 1u); - EXPECT_EQ(backend.getCount(key), 0u); - - const auto current = backend.get(key); + auto op = requests.admit(); + + EXPECT_EQ(readGcMaintenanceState(op, layout).status, GcMaintenanceReadStatus::Absent); + EXPECT_EQ(backend->getCount(key), 1u); + + backend->resetCounts(); + ASSERT_TRUE(std::holds_alternative( + casGcMaintenanceState(op, layout, std::nullopt, {}, Retry::standard()))); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getCount(key), 0u); + + backend->resetCounts(); + const WriteResult loser_attempt = casGcMaintenanceState(op, layout, std::nullopt, + GcMaintenanceState{.janitor_cursor = "loser"}, Retry::standard()); + ASSERT_TRUE(std::holds_alternative(loser_attempt)); + EXPECT_EQ(backend->writeTotal(), 1u); + /// Unlike the legacy CAS, a refused precondition is settled by ONE exact read before the write + /// reports the conflict -- `Conflict`'s observation needs to know what is actually there. + EXPECT_EQ(backend->getCount(key), 1u); + + const std::optional current = op.read(key, Retry::standard()); ASSERT_TRUE(current); - ASSERT_EQ(backend.casPut(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), current->token).outcome, - CasOutcome::Committed); - backend.resetCounts(); - EXPECT_EQ(casGcMaintenanceState(backend, layout, current->token, - GcMaintenanceState{.janitor_cursor = "stale"}).outcome, GcMaintenanceCasOutcome::Conflict); - EXPECT_EQ(backend.casPutCount(key), 1u); - EXPECT_EQ(backend.getCount(key), 0u); - EXPECT_EQ(decodeGcMaintenanceState(backend.InMemoryBackend::get(key)->bytes).janitor_cursor, "winner"); + ASSERT_TRUE(std::holds_alternative( + op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), current->incarnation, Retry::standard()))); + + backend->resetCounts(); + const WriteResult stale_attempt = casGcMaintenanceState(op, layout, current->incarnation, + GcMaintenanceState{.janitor_cursor = "stale"}, Retry::standard()); + ASSERT_TRUE(std::holds_alternative(stale_attempt)); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getCount(key), 1u); + EXPECT_EQ(decodeGcMaintenanceState(backend->InMemoryBackend::get(key)->bytes).janitor_cursor, "winner"); } TEST(CASGCMaintenanceState, FutureVersionPropagatesInsteadOfResetting) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); - ASSERT_EQ(backend.putIfAbsent(key, fmt::format( + auto op = requests.admit(); + + ASSERT_EQ(backend->putIfAbsent(key, fmt::format( "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, PutOutcome::Done); + + /// The seed write above lands through the same `write` primitive `CountingBackend` counts, so + /// reset before measuring what the read itself does. + backend->resetCounts(); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, - [&] { (void)readGcMaintenanceState(backend, layout); }); - EXPECT_EQ(backend.casPutCount(key), 0u); + [&] { (void)readGcMaintenanceState(op, layout); }); + EXPECT_EQ(backend->writeTotal(), 0u); - FailingMaintenanceReadBackend failing; - EXPECT_THROW((void)readGcMaintenanceState(failing, layout), std::runtime_error); + auto failing = std::make_shared(); + CasRequests failing_requests(failing, Fence::open()); + auto failing_op = failing_requests.admit(); + EXPECT_THROW((void)readGcMaintenanceState(failing_op, layout), std::runtime_error); } TEST(CASGCMaintenanceState, LosingCorruptResetPreservesConcurrentWinner) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const String key = layout.gcMaintenanceStateKey(); - ASSERT_EQ(backend.putIfAbsent(key, "corrupt").outcome, PutOutcome::Done); - const auto corrupt = readGcMaintenanceState(backend, layout); + auto op = requests.admit(); + + ASSERT_EQ(backend->putIfAbsent(key, "corrupt").outcome, PutOutcome::Done); + const GcMaintenanceReadResult corrupt = readGcMaintenanceState(op, layout); ASSERT_EQ(corrupt.status, GcMaintenanceReadStatus::Corrupt); - ASSERT_TRUE(corrupt.token); - ASSERT_EQ(backend.casPut(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), corrupt.token).outcome, - CasOutcome::Committed); - backend.resetCounts(); - EXPECT_EQ(casGcMaintenanceState(backend, layout, corrupt.token, {}).outcome, - GcMaintenanceCasOutcome::Conflict); - EXPECT_EQ(backend.casPutCount(key), 1u); - EXPECT_EQ(backend.getCount(key), 0u); - EXPECT_EQ(decodeGcMaintenanceState(backend.InMemoryBackend::get(key)->bytes).janitor_cursor, "winner"); + ASSERT_TRUE(corrupt.incarnation); + ASSERT_TRUE(std::holds_alternative( + op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), *corrupt.incarnation, Retry::standard()))); + + backend->resetCounts(); + const WriteResult reset_attempt = casGcMaintenanceState(op, layout, corrupt.incarnation, {}, Retry::standard()); + ASSERT_TRUE(std::holds_alternative(reset_attempt)); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getCount(key), 1u); + EXPECT_EQ(decodeGcMaintenanceState(backend->InMemoryBackend::get(key)->bytes).janitor_cursor, "winner"); } diff --git a/src/Disks/tests/gtest_cas_gc_meta_writer.cpp b/src/Disks/tests/gtest_cas_gc_meta_writer.cpp index 2e98c3477419..ada0deb892c1 100644 --- a/src/Disks/tests/gtest_cas_gc_meta_writer.cpp +++ b/src/Disks/tests/gtest_cas_gc_meta_writer.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -6,6 +7,7 @@ #include #include +#include #include #include #include @@ -13,6 +15,7 @@ using namespace DB::Cas; using DB::Cas::tests::MetaWriteLatchBackend; using DB::Cas::tests::awaitLatchEntered; +using DB::Cas::tests::openRequestsForTest; namespace { @@ -35,7 +38,7 @@ TEST(CASGcMetaWriter, RealCondemnMarkerJobCompletesAcrossOwnerDestruction) auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const BlobRef ref = DB::Cas::tests::idOf("1"); - const Token token{"tok-1"}; + const PersistedIncarnation token{"emulated", "tok-1"}; auto gc = std::make_unique(store, DB::Cas::tests::u128Of(kGcId)); backend->arm(); @@ -47,7 +50,9 @@ TEST(CASGcMetaWriter, RealCondemnMarkerJobCompletesAcrossOwnerDestruction) gc.reset(); releaser.join(); - const auto meta = loadMeta(*backend, store->layout(), ref); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const auto meta = loadMeta(op, store->layout(), ref); ASSERT_TRUE(meta) << "the condemn marker was lost across owner destruction"; EXPECT_EQ(meta->meta.state, MetaState::Condemned); EXPECT_EQ(meta->meta.condemn_round, 1u); @@ -62,7 +67,7 @@ TEST(CASGcMetaWriter, CondemnMarkerConfirmationIsVisibleAfterDrain) auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const BlobRef ref = DB::Cas::tests::idOf("1"); - const Token token{"tok-1"}; + const PersistedIncarnation token{"emulated", "tok-1"}; Gc gc(store, DB::Cas::tests::u128Of(kGcId)); EXPECT_FALSE(gc.metaWriterForTest().condemnMarkerConfirmedInProcess(ref, token)); @@ -83,10 +88,11 @@ TEST(CASGcMetaWriter, RealConfirmedMetaDeleteCompletesAcrossOwnerDestruction) auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const BlobRef ref = DB::Cas::tests::idOf("2"); - ASSERT_EQ( - putMetaIfAbsent(*store, ref, BlobMeta{.state = MetaState::Condemned, .condemn_round = 1, .size = 64}).outcome, - CasOverwriteOutcome::Committed); - ASSERT_TRUE(loadMeta(*backend, store->layout(), ref)); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(putMetaIfAbsent( + op, store->layout(), ref, BlobMeta{.state = MetaState::Condemned, .condemn_round = 1, .size = 64}))); + ASSERT_TRUE(loadMeta(op, store->layout(), ref)); auto gc = std::make_unique(store, DB::Cas::tests::u128Of(kGcId)); backend->arm(); @@ -98,7 +104,7 @@ TEST(CASGcMetaWriter, RealConfirmedMetaDeleteCompletesAcrossOwnerDestruction) gc.reset(); releaser.join(); - EXPECT_FALSE(loadMeta(*backend, store->layout(), ref)) + EXPECT_FALSE(loadMeta(op, store->layout(), ref)) << "the confirmed-meta delete was lost across owner destruction"; } diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index c9c20a7454cd..197cda3c1eab 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -37,7 +37,7 @@ TEST(CASFormatBattery, GcOutcomes) OutcomeEntry e; e.kind = ObjectKind::Blob; e.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}; - e.token = Token{"e-1", TokenType::ETag}; + e.token = PersistedIncarnation{"etag", "e-1"}; e.outcome = OutcomeKind::Deleted; log.entries.push_back(e); runFormatBattery({FormatId::GcOutcomes, @@ -57,13 +57,13 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) { OutcomeLog log; log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("aa00000000000000000000000000000a"))}, - Token{"etag-1", TokenType::ETag}, OutcomeKind::Deleted}); + PersistedIncarnation{"etag", "etag-1"}, OutcomeKind::Deleted}); log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("bb00000000000000000000000000000b"))}, - Token{"7", TokenType::Emulated}, OutcomeKind::Spared}); + PersistedIncarnation{"emulated", "7"}, OutcomeKind::Spared}); log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("cc00000000000000000000000000000c"))}, - Token{"8", TokenType::Emulated}, OutcomeKind::Replaced}); + PersistedIncarnation{"emulated", "8"}, OutcomeKind::Replaced}); log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("dd00000000000000000000000000000d"))}, - Token{"9", TokenType::Emulated}, OutcomeKind::Absent}); + PersistedIncarnation{"emulated", "9"}, OutcomeKind::Absent}); const String text = encodeOutcomeLog(log); const OutcomeLog d = decodeOutcomeLog(text); ASSERT_EQ(d.entries.size(), 4u); @@ -73,7 +73,7 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) EXPECT_EQ(d.entries[2].outcome, OutcomeKind::Replaced); EXPECT_EQ(d.entries[3].outcome, OutcomeKind::Absent); EXPECT_EQ(d.entries[0].token.value, "etag-1"); - EXPECT_EQ(d.entries[0].token.type, TokenType::ETag); + EXPECT_EQ(d.entries[0].token.dialect, "etag"); EXPECT_EQ(d.entries[3].token.value, "9"); /// Insertion order + byte-stable text (the encoder is a pure function of the log). EXPECT_EQ(encodeOutcomeLog(d), text); @@ -97,7 +97,7 @@ TEST(CASGCOutcomesFormat, RecordRequiresCompleteBlobRefAndTokenGroups) OutcomeLog log; log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}, - Token{"e-1", TokenType::ETag}, OutcomeKind::Deleted}); + PersistedIncarnation{"etag", "e-1"}, OutcomeKind::Deleted}); const String bytes = encodeOutcomeLog(log); for (const auto & [field, expected_message] : { diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index 1b6344af63fb..8b6d74834d41 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -237,11 +237,13 @@ TEST(CASGCRebuild, FrozenCheckpointFrontierExcludesVisibleUnfrontieredTail) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/rebuild-frozen-frontier@cas@"}; const UInt128 life_id{0xF001}; const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(ns, life_id); - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = life_id}); const ManifestRef admitted = ref(1, 0xA1); @@ -280,10 +282,12 @@ TEST(CASGCRebuild, LiveCatalogLifeWithoutCheckpointFailsClosed) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/rebuild-missing-checkpoint@cas@"}; const UInt128 life_id{0xF002}; - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = life_id}); const ManifestRef admitted = ref(1, 0xA2); @@ -311,10 +315,12 @@ TEST(CASGCRebuild, CheckpointSnapshotAtOlderEpochSealFailsClosed) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/rebuild-checkpoint-base-seal@cas@"}; const UInt128 life_id{0xF003}; - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = life_id}); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(ns, life_id); @@ -357,12 +363,14 @@ TEST(CASGCRebuild, DamagedGenerationZeroStatePerformsNoCatalogDrainMutation) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const RootNamespace ns{"00/removing-without-parent@cas@"}; const UInt128 life_id{91}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, CatalogEntry{ + CasRefCatalog::casAdmitEntry(op, layout, 1, CatalogEntry{ .ns = ns, .state = NsState::Live, .incarnation = life_id}); - CasRefCatalog::casUpdate(*backend, layout, [](const RefCatalog & current) + CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { RefCatalog next = current; next.entries[0].state = NsState::Removing; @@ -374,7 +382,7 @@ TEST(CASGCRebuild, DamagedGenerationZeroStatePerformsNoCatalogDrainMutation) .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); - const uint64_t catalog_cas_before = backend->casPutCount(layout.refCatalogKey()); + const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); const uint64_t plans_before = ProfileEvents::global_counters[ProfileEvents::CASGCRefWalkPlansBuilt].load(); @@ -382,8 +390,8 @@ TEST(CASGCRebuild, DamagedGenerationZeroStatePerformsNoCatalogDrainMutation) const RebuildReport report = gc.rebuildBaseline(/*force*/ false); ASSERT_TRUE(report.performed) << report.refusal; EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASGCRefWalkPlansBuilt].load() - plans_before, 1u); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), catalog_cas_before); - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(*backend, layout); + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_cas_before); + const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(op, layout); ASSERT_EQ(catalog.catalog.entries.size(), 1u); EXPECT_EQ(catalog.catalog.entries[0].state, NsState::Removing); EXPECT_EQ(catalog.catalog.entries[0].incarnation, life_id); diff --git a/src/Disks/tests/gtest_cas_gc_resume.cpp b/src/Disks/tests/gtest_cas_gc_resume.cpp index 0a8384e644e0..7596ef2cc783 100644 --- a/src/Disks/tests/gtest_cas_gc_resume.cpp +++ b/src/Disks/tests/gtest_cas_gc_resume.cpp @@ -52,31 +52,37 @@ size_t runGcToFixpoint(const PoolPtr & s, Gc & gc, size_t max_rounds = 64) return rounds; } -/// A backend that denies ONCE the SINGLE round-commit `gc/state` CAS — the casPut that advances -/// snap_generation (the one-pass round has exactly one such CAS; the lease-acquire CAS does not advance -/// snap_generation). A denied round leaves only never-adopted attempt-scoped debris (fold seal / retired -/// list under an attempt gc/state never adopted); a fresh-attempt rerun is idempotent. +/// A backend that refuses ONCE the SINGLE round-commit `gc/state` write — the conditional write that +/// advances snap_generation (the one-pass round has exactly one such write; the lease acquire/renew does +/// not advance snap_generation). A refused round leaves only never-adopted attempt-scoped debris (a fold +/// seal under an attempt gc/state never adopted); a fresh-attempt rerun is idempotent. class InterruptRoundCasBackend : public InMemoryBackend { public: explicit InterruptRoundCasBackend(String gc_state_key_) : gc_state_key(std::move(gc_state_key_)) {} - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - if (arm_interrupt && key == gc_state_key) + if (arm_interrupt && expected_value && key == gc_state_key) { - const auto stored = get(key); - const uint64_t stored_gen = stored ? decodeGcState(stored->bytes).snap_generation : 0; - const uint64_t next_gen = decodeGcState(bytes).snap_generation; - if (next_gen > stored_gen) + const auto stored = InMemoryBackend::read(key, access); + if (stored + && decodeGcState(bytes).snap_generation > decodeGcState(stored->bytes).snap_generation) { - arm_interrupt = false; /// one-shot: only depose the first round-commit CAS - throw DB::Exception(DB::ErrorCodes::ABORTED, - "test-injected: round-commit gc/state CAS denied (leader deposed mid-round)"); + arm_interrupt = false; /// one-shot: only depose the first round-commit write + /// A REFUSAL, not a throw: a thrown transport error is an ambiguity the engine settles + /// by an exact read and then reissues while the precondition it named is unmoved, so + /// the round would commit on the reissue. A refused precondition ends the write at + /// once. The object is moved too -- the same bytes under a fresh incarnation -- because + /// a store refuses only what changed; the CONTENT is deliberately left alone, so this + /// round's own lease and cursor are exactly what a deposed round leaves behind. + (void)InMemoryBackend::write(key, stored->bytes, stored->value, access); + return std::unexpected(RawConflict{}); } } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } bool arm_interrupt = false; diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index aa0a8f914972..31ac941282db 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -15,6 +15,7 @@ namespace DB::ErrorCodes extern const int BAD_ARGUMENTS; extern const int CORRUPTED_DATA; extern const int ABORTED; +extern const int NETWORK_ERROR; } namespace ProfileEvents @@ -95,27 +96,23 @@ PoolPtr openTestPoolWithConfig(std::shared_ptr & out_backend, P class GcStateCasFaultBackend : public InMemoryBackend { public: - using Backend::get; - using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - - CasResult casPut(const String & key, const String & bytes, - const std::optional & expected, const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - if (key == faulted_key) + if (expected_value && key == faulted_key) { ++calls_to_faulted_key; if (fail_at_call != 0 && calls_to_faulted_key == fail_at_call) - return CasResult{CasOutcome::Conflict, {}}; + return std::unexpected(RawConflict{}); } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } String faulted_key; size_t calls_to_faulted_key = 0; - size_t fail_at_call = 0; /// 0 = never fault; else fault exactly the Nth casPut to `faulted_key` + /// 0 = never fault; else refuse exactly the Nth conditional write to `faulted_key`. + size_t fail_at_call = 0; }; GcState readState(InMemoryBackend & b, const Pool & s) @@ -162,18 +159,18 @@ size_t driveToFixpoint(InMemoryBackend & backend, const PoolPtr & store, Gc & gc return working_rounds; } -/// A full key -> token snapshot of the backend, for the previewDeletes write-free invariant: any -/// put/casPut/overwrite mints a fresh token (or adds a key) and any delete removes one, so an unchanged -/// map across a call proves it performed NO writes. -std::map snapshotKeyTokens(InMemoryBackend & b) +/// A full key -> incarnation snapshot of the backend, for the previewDeletes write-free invariant: any +/// write mints a fresh incarnation (or adds a key) and any removal drops one, so an unchanged map +/// across a call proves it performed NO writes. +std::map snapshotKeyTokens(CasOperation & op) { std::map out; String cursor; while (true) { - const ListPage page = b.list("", cursor, 100000); - for (const ListedKey & k : page.keys) - out[k.key] = k.token ? k.token->value : String{}; + const KeyPage page = op.list("", cursor, 100000, Retry::once()); + for (const KeyEntry & k : page.keys) + out[k.key] = k.incarnation ? k.incarnation->render() : String{}; if (page.next_cursor.empty()) break; cursor = page.next_cursor; @@ -417,12 +414,13 @@ TEST(CASGCLease, DeadIncumbentThenRevivedIncumbentWinsRace) EXPECT_EQ(readState(*b, *s).lease.owner, kGcA); } -TEST(CASGCLease, ConcurrentStealLosesCas) +/// The steal's refused precondition, case one of two: the re-decide sees the SAME frozen incumbent. +/// `readModifyWrite` does not treat a refusal as terminal -- it re-decides against what the refused +/// write's own resolve read observed and sends another attempt -- so a contender that was steal-eligible +/// still is, and the steal lands inside the SAME round. The two cases are separate tests because they +/// differ only in what the store holds at the re-decide, and that is the whole decision. +TEST(CASGCLease, RefusedStealAgainstAFrozenIncumbentRetriesAndLands) { - /// The CAS-race horn: gc2 is steal-eligible and goes for the CAS, but gc/state moved under it - /// (injected one-shot conflict). It must back off (never acquired=true off a lost CAS) and the - /// owner on storage must be unperturbed. The injected conflict left the object unchanged, so gc2's - /// NEXT round is steal-eligible again and succeeds. std::shared_ptr b; auto s = openTestPool(b); Gc gc1(s, kGcA); @@ -431,25 +429,80 @@ TEST(CASGCLease, ConcurrentStealLosesCas) ASSERT_TRUE(gc1.runRegularRound().acquired_lease); const GcState st0 = readState(*b, *s); EXPECT_FALSE(gc2.runRegularRound().acquired_lease); /// obs #1; gc1 stalls now - b->failNextCasPut(s->layout().gcStateKey()); /// inject: gc2's steal CAS conflicts - EXPECT_FALSE(gc2.runRegularRound().acquired_lease); /// steal attempt loses the CAS => back off + /// `refuseNextWrite` refuses without touching the object, which is what "the incumbent is still + /// frozen" looks like to the re-decide. + b->refuseNextWrite(s->layout().gcStateKey()); + EXPECT_TRUE(gc2.runRegularRound().acquired_lease) + << "a refused steal against an unmoved tuple is retried inside the same call and lands"; const GcState st1 = readState(*b, *s); - EXPECT_EQ(st1.lease.owner, kGcA); /// unchanged - EXPECT_EQ(st1.lease.seq, st0.lease.seq); /// nothing clobbered - EXPECT_TRUE(gc2.runRegularRound().acquired_lease); /// still steal-eligible => succeeds now - EXPECT_EQ(readState(*b, *s).lease.owner, kGcB); + EXPECT_EQ(st1.lease.owner, kGcB); + EXPECT_GT(st1.lease.seq, st0.lease.seq); +} + +/// Case two: the re-decide sees a MOVED tuple. This is what a real refused precondition means -- a +/// store refuses only what changed -- and the incumbent's own renewal is the change. The contender must +/// then decline rather than steal, because a moved tuple is proof of life, and it must leave the +/// incumbent's lease exactly as the incumbent wrote it. +TEST(CASGCLease, RefusedStealWhoseRedecideSeesAMovedTupleDeclines) +{ + class StealRaceBackend : public InMemoryBackend + { + public: + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override + { + if (arm && expected_value && key == gc_state_key) + { + const auto stored = InMemoryBackend::read(key, access); + if (stored) + { + arm = false; + /// The incumbent renews while this contender is deciding: bump its own `seq` under + /// its own owner, then refuse. Written before the refusal is returned, so the + /// resolve read the engine makes next observes the moved tuple. + GcState renewed = decodeGcState(stored->bytes); + ++renewed.lease.seq; + (void)InMemoryBackend::write(key, encodeGcState(renewed), stored->value, access); + return std::unexpected(RawConflict{}); + } + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + String gc_state_key; + bool arm = false; + }; + + auto b = std::make_shared(); + auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + b->gc_state_key = s->layout().gcStateKey(); + Gc gc1(s, kGcA); + Gc gc2(s, kGcB); + + ASSERT_TRUE(gc1.runRegularRound().acquired_lease); + const GcState st0 = readState(*b, *s); + EXPECT_FALSE(gc2.runRegularRound().acquired_lease); /// obs #1; gc1 stalls now + b->arm = true; + EXPECT_FALSE(gc2.runRegularRound().acquired_lease) + << "the re-decide sees a renewed incumbent, which is proof of life: decline, never steal"; + const GcState st1 = readState(*b, *s); + EXPECT_EQ(st1.lease.owner, kGcA) + << "gc2 wrote nothing that landed: a steal would have put kGcB here"; + EXPECT_EQ(st1.lease.seq, st0.lease.seq + 1) + << "the only write that landed is the incumbent's injected renewal"; } TEST(CASGCLease, CreateConflictReReadsWithinTheBound) { - /// The create-Conflict branch: a fresh pool where the create-if-absent CAS conflicts (one-shot). - /// The contender re-reads and falls through within its bounded (2) CAS attempts — the re-read still - /// finds the key absent, so the second attempt creates and acquires. + /// The create-Conflict branch: a fresh pool where the create-if-absent write is refused (one-shot). + /// `readModifyWrite` re-decides against the refused write's own resolve read, which still finds the + /// key absent, so the second attempt creates and acquires. The retry policy's deadline is the bound. std::shared_ptr b; auto s = openTestPool(b); Gc gc(s, hexToU128("0000000000000000000000000000000c")); - b->failNextCasPut(s->layout().gcStateKey()); + b->refuseNextWrite(s->layout().gcStateKey()); EXPECT_TRUE(gc.runRegularRound().acquired_lease); const GcState st = readState(*b, *s); EXPECT_EQ(st.lease.owner, hexToU128("0000000000000000000000000000000c")); @@ -467,15 +520,15 @@ TEST(CASGCLease, CtorFailsClosedOnBadArguments) TEST(CASGCLease, IncumbentRenewConflictRetriesOnceAndAcquires) { - /// The incumbent's own renew CAS conflicts (one-shot). Re-read sees our own ownership => the renew - /// is retried ONCE within the bounded (2) CAS attempts => acquired. Never acquired=true without a - /// Committed CAS — storage must carry the seq the SECOND (committed) attempt wrote. + /// The incumbent's own renew write is refused (one-shot). The re-decide sees our own ownership, so + /// the renew is retried and acquires; the retry policy's deadline is the bound. Never + /// acquired=true without a committed write — storage must carry the seq the SECOND attempt wrote. std::shared_ptr b; auto s = openTestPool(b); Gc gc(s, hexToU128("0000000000000000000000000000000d")); ASSERT_TRUE(gc.runRegularRound().acquired_lease); /// create: seq 1 - b->failNextCasPut(s->layout().gcStateKey()); /// inject: the renew CAS conflicts + b->refuseNextWrite(s->layout().gcStateKey()); /// inject: the renew CAS conflicts EXPECT_TRUE(gc.runRegularRound().acquired_lease); /// re-read (still us) => retried once const GcState st = readState(*b, *s); EXPECT_EQ(st.lease.owner, hexToU128("0000000000000000000000000000000d")); @@ -604,17 +657,19 @@ TEST(CASGCRound, PreviewReportsCondemnedRowsAndIsWriteFree) dropRefTransition(*backend, store->layout(), ns, "tbl", r); runRegularRoundReclaiming(gc); /// condemning round: -1 => in-degree 0 => RunMarker::Condemned row (not pending) - /// Write-free contract: a full key->token snapshot must be identical across the previewDeletes call. - const auto before = snapshotKeyTokens(*backend); + /// Write-free contract: a full key->incarnation snapshot must be identical across the call. + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const auto before = snapshotKeyTokens(op); const std::vector awaiting = gc.previewDeletes(); - const auto after = snapshotKeyTokens(*backend); - EXPECT_EQ(before, after) << "previewDeletes must perform NO writes (put/casPut/overwrite/delete)"; + const auto after = snapshotKeyTokens(op); + EXPECT_EQ(before, after) << "previewDeletes must perform NO writes"; ASSERT_EQ(awaiting.size(), 1u) << "exactly the one condemned blob is previewed"; EXPECT_EQ(awaiting[0].ref, (DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(blob)})); EXPECT_EQ(awaiting[0].key, store->layout().blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(blob)})); EXPECT_EQ(awaiting[0].reason, "awaiting_graduation"); - EXPECT_FALSE(awaiting[0].token.value.empty()) << "must carry the stored condemn-time token"; + EXPECT_FALSE(awaiting[0].token.value.empty()) << "must carry the stored condemn-time incarnation"; EXPECT_GT(awaiting[0].condemn_round, 0u) << "must carry the stored condemn round"; runRegularRoundReclaiming(gc); /// graduation round: entry becomes delete_pending (blob still present) @@ -1801,6 +1856,8 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) /// fold-every-round (Phase-4 Lever A would otherwise defer once the pool quiesces). config.gc_fold_max_defer_rounds = 0; auto store = openTestPoolWithConfig(backend, config); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"test/aa@cas@"}; registerNamespaceRaw(*backend, store->layout(), ns); @@ -1851,7 +1908,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) writeSealAt(*backend, store->layout(), ns, RefTxnId{1, 2}); publishAt(*backend, store->layout(), ns, RefTxnId{2, 1}, "tbl2", /*build_sequence=*/7, DB::UInt128(0xB10B2), /*birth=*/false, /*prev_epoch_seal=*/RefTxnId{1, 2}); - const std::optional life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns); + const std::optional life = CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); ASSERT_TRUE(life.has_value()); const String ckpt_key = store->layout().refCkptKey(*life); const auto old_ckpt = backend->get(ckpt_key); @@ -1997,3 +2054,173 @@ TEST(CASGCRound, TwoManifestsTwoSourceEdgesDropOneSpares) EXPECT_TRUE(blobExists(*backend, store->layout(), DB::UInt128(1))) << "the blob must survive — the second reference still pins it"; } + +/// ===================== THE REQUEST CONTRACT AT THE ROUND'S OWN WRITES ===================== + +/// The advisory pulse sends AT MOST ONE write, and neither of the two ways it can fail reaches the +/// caller: the next pulse comes on cadence, so a deposed leader must never spend a retry budget +/// fighting for this key. Both halves are asserted by the write COUNT, because a policy that reissued +/// would be invisible in the outcome. +TEST(CASGc, HeartbeatPulseIsOnceAndAConflictIsIgnored) +{ + class HeartbeatFaultBackend : public InMemoryBackend + { + public: + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override + { + if (key == hb_key) + { + ++hb_writes; + if (throw_next) + { + throw_next = false; + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected heartbeat write outage"); + } + if (refuse_next) + { + refuse_next = false; + return std::unexpected(RawConflict{}); + } + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + String hb_key; + size_t hb_writes = 0; + bool throw_next = false; + bool refuse_next = false; + }; + + auto backend = std::make_shared(); + auto store = openPoolForTest(backend); + backend->hb_key = store->layout().gcHbKey(); + + Gc::pulseHeartbeat(*store, kGcA); + ASSERT_EQ(backend->hb_writes, 1u) << "one pulse is one write"; + + /// AN UNRESOLVED ATTEMPT IS NOT REISSUED. The key is removed first so the write's own resolving + /// read finds nothing: with nothing at the key the attempt's fate is genuinely unknown, which is + /// the only state a reissuing policy would act on. Under `once` the pulse ends there. + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_EQ(op.removeCurrent(backend->hb_key, Retry::once()), Removal::Removed); + backend->throw_next = true; + const size_t before_unresolved = backend->hb_writes; + EXPECT_NO_THROW(Gc::pulseHeartbeat(*store, kGcA)); + EXPECT_EQ(backend->hb_writes, before_unresolved + 1) + << "an unresolved pulse is abandoned, never reissued"; + + /// A REFUSED PRECONDITION is the ordinary race with another pulser: ignored, never thrown. + Gc::pulseHeartbeat(*store, kGcA); + backend->refuse_next = true; + const size_t before_refused = backend->hb_writes; + EXPECT_NO_THROW(Gc::pulseHeartbeat(*store, kGcA)); + EXPECT_EQ(backend->hb_writes, before_refused + 1); +} + +/// The steal is the one destructive decision the lease machine makes, and it is a CONJUNCTION: the +/// lease tuple unchanged across two of this contender's own observations, the heartbeat pair unchanged +/// across the same window, and a caller allowed to steal. Each conjunct is falsified on its own here, +/// against the same frozen incumbent, so a build that dropped any one of them fails exactly one line. +TEST(CASGc, LeaseDecideStealsOnlyWithAllThreeConjuncts) +{ + auto backend = std::make_shared(); + auto store = openPoolForTest(backend); + + Gc incumbent(store, kGcA); + ASSERT_TRUE(incumbent.runRegularRound().acquired_lease); + + Gc contender(store, kGcB); + EXPECT_FALSE(contender.runRegularRound({}, /*allow_steal=*/true).acquired_lease) + << "the first tick has no earlier observation to freeze against"; + + Gc::pulseHeartbeat(*store, kGcA); + EXPECT_FALSE(contender.runRegularRound({}, /*allow_steal=*/true).acquired_lease) + << "a moved heartbeat pair is proof of life even with the lease tuple frozen"; + + ASSERT_TRUE(incumbent.runRegularRound().acquired_lease); + EXPECT_FALSE(contender.runRegularRound({}, /*allow_steal=*/true).acquired_lease) + << "a moved lease tuple is proof of life even with the heartbeat frozen"; + + EXPECT_FALSE(contender.runRegularRound({}, /*allow_steal=*/false).acquired_lease) + << "both observations are frozen, but this caller may not steal"; + + EXPECT_TRUE(contender.runRegularRound({}, /*allow_steal=*/true).acquired_lease) + << "frozen tuple, frozen heartbeat and a caller allowed to steal"; +} + +/// The round commits everything it did in ONE conditional write of `gc/state`. A refused precondition +/// there means another leader advanced the key, so the round is dropped whole: it throws `ABORTED` and +/// adopts no generation. The lease renewal that OPENED the round is a separate, earlier write and +/// stays committed. +TEST(CASGc, RoundCommitConflictDropsTheRound) +{ + class RoundCommitConflictBackend : public InMemoryBackend + { + public: + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override + { + if (arm && expected_value && key == gc_state_key) + { + const auto stored = InMemoryBackend::read(key, access); + if (stored + && decodeGcState(bytes).snap_generation > decodeGcState(stored->bytes).snap_generation) + { + arm = false; + return std::unexpected(RawConflict{}); + } + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + String gc_state_key; + bool arm = false; + }; + + auto backend = std::make_shared(); + auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + backend->gc_state_key = store->layout().gcStateKey(); + + const RootNamespace ns{"00/aa@cas@"}; + const ManifestRef r = ref(1, 0xAA); + writeBlobBody(*backend, store->layout(), DB::UInt128(1)); + writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); + publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, r); + + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + /// Asserts presence rather than dereferencing: `gc/state` does not exist until a round's own lease + /// acquire creates it, and an empty optional here is undefined behaviour, not a failing assertion. + const auto readState = [&] + { + const auto got = op.read(backend->gc_state_key, Retry::once()); + EXPECT_TRUE(got) << "gc/state must exist once a round has acquired the lease"; + return got ? decodeGcState(got->bytes) : GcState{}; + }; + + Gc gc(store, kGc); + /// One honest round first, so the comparison below is against a committed round rather than + /// against a bootstrap: the double is disarmed, so this round's own commit lands. + ASSERT_TRUE(gc.runRegularRound().acquired_lease); + const GcState before = readState(); + backend->arm = true; + try + { + gc.runRegularRound(); + FAIL() << "a refused round commit must end the round"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::ABORTED); + } + + const GcState after = readState(); + EXPECT_EQ(after.snap_generation, before.snap_generation) + << "a refused commit adopts no generation"; + EXPECT_GT(after.lease.seq, before.lease.seq) + << "the lease renewal that opened the round is a separate, earlier write and stays committed"; +} diff --git a/src/Disks/tests/gtest_cas_gc_round_defer.cpp b/src/Disks/tests/gtest_cas_gc_round_defer.cpp index adbccce614ec..86983a35b1bc 100644 --- a/src/Disks/tests/gtest_cas_gc_round_defer.cpp +++ b/src/Disks/tests/gtest_cas_gc_round_defer.cpp @@ -374,6 +374,8 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/1); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const NamespaceLifeId dead_a = NamespaceLifeId::fromCatalogEntry(RootNamespace{"dead/a"}, UInt128{0xDA}); @@ -387,10 +389,10 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP /// Establish real opaque backend progress rather than fabricating a cursor value. One key remains /// after this page and the durable cursor must be non-empty. const NamespaceJanitorResult first_page - = NamespaceJanitor(*backend, layout, 1).runOnePage(false, [] { return true; }); + = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); ASSERT_EQ(first_page.pages, 1u); ASSERT_EQ(first_page.deleted, 1u); - const GcMaintenanceReadResult partial = readGcMaintenanceState(*backend, layout); + const GcMaintenanceReadResult partial = readGcMaintenanceState(op, layout); ASSERT_EQ(partial.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(partial.state); ASSERT_FALSE(partial.state->janitor_cursor.empty()); @@ -435,7 +437,7 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP EXPECT_GE(cleanup->metrics.at("janitor_keys"), 1u); EXPECT_EQ(cleanup->metrics.at("janitor_deleted"), 0u); - const GcMaintenanceReadResult deferred_progress = readGcMaintenanceState(*backend, layout); + const GcMaintenanceReadResult deferred_progress = readGcMaintenanceState(op, layout); ASSERT_EQ(deferred_progress.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(deferred_progress.state); EXPECT_EQ(deferred_progress.state->janitor_cursor, partial.state->janitor_cursor) @@ -468,7 +470,7 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP EXPECT_EQ(folded_cleanup->metrics.at("janitor_deleted"), 1u); EXPECT_EQ(static_cast(backend->head(key_a).exists) + static_cast(backend->head(key_b).exists), 0u) << "the fold must retry and delete the exact page that DEFER left undecided"; - const GcMaintenanceReadResult completed = readGcMaintenanceState(*backend, layout); + const GcMaintenanceReadResult completed = readGcMaintenanceState(op, layout); ASSERT_EQ(completed.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(completed.state); EXPECT_TRUE(completed.state->janitor_cursor.empty()); diff --git a/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp b/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp index 2efced9934c7..a510b818ee6c 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp @@ -52,6 +52,8 @@ TEST(CASGCShardIncarnation, DiscoveryEqualsPresentShards) { std::shared_ptr backend; auto store = makePoolWithShards(backend, gc_shards); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); Gc gc(store, hexToU128("0000000000000000000000000000000a")); const Layout & layout = store->layout(); @@ -63,7 +65,7 @@ TEST(CASGCShardIncarnation, DiscoveryEqualsPresentShards) fixture::admitLive(*backend, layout, ns_live_empty); /// (b) A genuinely Creating entry, admitted directly (step 1 alone -- never completed to Live). - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns_creating, .state = NsState::Creating, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns_creating, .state = NsState::Creating, .incarnation = UInt128(1), .creator = CreatorFence{.server_root_id = "test", .writer_epoch = 1, .fence_generation = 1}}); /// (c) Ref objects present, but the catalog was never told (or has since forgotten): write @@ -72,7 +74,7 @@ TEST(CASGCShardIncarnation, DiscoveryEqualsPresentShards) writeManifestRaw(*backend, layout, ns_uncataloged, testRef(1), {}); publishCommittedTransition(*backend, layout, ns_uncataloged, "part_1", std::nullopt, testRef(1), /*shard=*/0); { - CasRefCatalog::Snapshot snap = CasRefCatalog::read(*backend, layout); + CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); std::erase_if(snap.catalog.entries, [&](const CatalogEntry & e) { return e.ns.string() == ns_uncataloged.string(); }); const HeadResult h = backend->head(layout.refCatalogKey()); ASSERT_TRUE(h.exists); @@ -97,7 +99,7 @@ TEST(CASGCShardIncarnation, DiscoveryEqualsPresentShards) EXPECT_TRUE(found_live_empty) << "a Live catalog entry with zero ref objects must still be discovered"; /// Confirm (b) really is still Creating (not merely absent from a differently-shaped universe). - const CasRefCatalog::Snapshot final_snap = CasRefCatalog::read(*backend, layout); + const CasRefCatalog::Snapshot final_snap = CasRefCatalog::read(op, layout); const auto creating_it = std::find_if(final_snap.catalog.entries.begin(), final_snap.catalog.entries.end(), [&](const CatalogEntry & e) { return e.ns.string() == ns_creating.string(); }); ASSERT_NE(creating_it, final_snap.catalog.entries.end()); @@ -141,11 +143,13 @@ TEST(CASGCShardIncarnation, DeadLifeStreamIsOpaqueInertDebris) { std::shared_ptr backend; auto store = makePoolWithShards(backend, /*gc_shards=*/1); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); Gc gc(store, hexToU128("0000000000000000000000000000000a")); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/tblIncarnationSwap"}; - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = UInt128(11), .creator = std::nullopt}); // Live forbids a creator fence const ManifestRef dead_ref = testRef(1); writeBlobBody(*backend, layout, UInt128(11)); @@ -157,7 +161,7 @@ TEST(CASGCShardIncarnation, DeadLifeStreamIsOpaqueInertDebris) const NamespaceLifeId dead_life = NamespaceLifeId::fromCatalogEntry(ns, UInt128(11)); { - CasRefCatalog::Snapshot snap = CasRefCatalog::read(*backend, layout); + CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); const auto it = std::find_if(snap.catalog.entries.begin(), snap.catalog.entries.end(), [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); ASSERT_NE(it, snap.catalog.entries.end()); @@ -203,16 +207,18 @@ TEST(CASGCShardIncarnation, CurrentLifeCheckpointIsReadByExactKeyOutsideHotList) auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .gc_shards = 1, .gc_fold_max_defer_rounds = 0}); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); Gc gc(store, hexToU128("0000000000000000000000000000000a")); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/tblOrdinaryRebirth"}; - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = UInt128(11), .creator = std::nullopt}); CatalogEntry after_rebirth{.ns = ns, .state = NsState::Live, .incarnation = UInt128(22), .creator = std::nullopt}; { - CasRefCatalog::Snapshot snap = CasRefCatalog::read(*backend, layout); + CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); const auto it = std::find_if(snap.catalog.entries.begin(), snap.catalog.entries.end(), [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); ASSERT_NE(it, snap.catalog.entries.end()); @@ -269,6 +275,8 @@ TEST(CASGCShardIncarnation, UncatalogedStreamLifeDefersWithoutInventingNamespace { std::shared_ptr backend; auto store = makePoolWithShards(backend, /*gc_shards=*/1); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); Gc gc(store, hexToU128("0000000000000000000000000000000a")); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/tblForgotten"}; @@ -278,7 +286,7 @@ TEST(CASGCShardIncarnation, UncatalogedStreamLifeDefersWithoutInventingNamespace const NamespaceLifeId forgotten_life = store->namespaceLife(ns); { - CasRefCatalog::Snapshot snap = CasRefCatalog::read(*backend, layout); + CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); std::erase_if(snap.catalog.entries, [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); const HeadResult h = backend->head(layout.refCatalogKey()); ASSERT_TRUE(h.exists); @@ -298,13 +306,15 @@ TEST(CASGCShardIncarnation, StateCheckpointsOutsideCatalogAreInertToHotWalk) auto backend = std::make_shared(); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .gc_shards = 1}); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); Gc gc(store, hexToU128("0000000000000000000000000000000a")); const Layout & layout = store->layout(); const RootNamespace creating_ns{"srv1/tblStalledBirth"}; const RootNamespace unrelated_gone_ns{"srv1/tblGenuinelyGone"}; /// Step 1 of createNamespace: insert the Creating entry with a live creator fence. - CasRefCatalog::casAdmitEntry(*backend, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = creating_ns, .state = NsState::Creating, + CasRefCatalog::casAdmitEntry(op, layout, store->poolConfig().gc_shards, CatalogEntry{.ns = creating_ns, .state = NsState::Creating, .incarnation = UInt128(33), .creator = CreatorFence{.server_root_id = "test", .writer_epoch = 1, .fence_generation = 1}}); /// Step 2, without step 3: publish the genesis `_ckpt` directly, at the SAME incarnation the @@ -428,6 +438,8 @@ TEST(CASGCShardIncarnation, NewbornPrecommitProtectsDedupBlobAgainstConcurrentDr { std::shared_ptr backend; auto store = makePoolWithShards(backend, gc_shards); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns_b{"srv1/tblB"}; /// --- Phase 1: Write b1's body before any GC. --- @@ -454,9 +466,9 @@ TEST(CASGCShardIncarnation, NewbornPrecommitProtectsDedupBlobAgainstConcurrentDr store->renewWatermarkOnce(); } const String b1_key = store->layout().blobKey(b1_ref); - ASSERT_TRUE(backend->head(b1_key).exists) - << "b1 body must be present after the seed putBlob"; - const Token b1_token = backend->head(b1_key).token; + const std::optional b1_observed = op.head(b1_key, Retry::once()); + ASSERT_TRUE(b1_observed) << "b1 body must be present after the seed putBlob"; + const PersistedIncarnation b1_token = PersistedIncarnation::capture(b1_observed->incarnation); /// --- Phase 2: Inject gc/state at round 1 with b1 CONDEMNED (body still present). --- /// This simulates GC having advanced to round 1 and retired b1 (condemned token recorded @@ -507,9 +519,12 @@ TEST(CASGCShardIncarnation, NewbornPrecommitProtectsDedupBlobAgainstConcurrentDr EXPECT_TRUE(store->resolveRef(ns_b, "part_b1").has_value()) << "gc_shards=" << gc_shards << ": the ref must commit"; /// The condemned token is bound UNCHANGED — no displacement happens (and none is needed). - EXPECT_EQ(backend->head(b1_key).token, b1_token) - << "gc_shards=" << gc_shards << ": no copy-forward under the Phase-A contract — the token " - "stays; the folded edge will spare it at the next fold (no round runs here to delete it)"; + const std::optional b1_after = op.head(b1_key, Retry::once()); + ASSERT_TRUE(b1_after); + EXPECT_TRUE(b1_token.matches(b1_after->incarnation)) + << "gc_shards=" << gc_shards << ": no copy-forward under the Phase-A contract — the " + "incarnation stays; the folded edge will spare it at the next fold (no round runs " + "here to delete it)"; /// INV-NO-DANGLE: the body is present and no GC round ever runs in this test to fold the /// precommit/committed edge; a real deployment's next fold would see net in-degree >= 1 and diff --git a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp index 96f486781f5e..62f12b94655c 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp @@ -129,6 +129,8 @@ TEST(CASGCShardReducer, MergesDeltasToInDegree) /// Reduce: each reducer merges its shard's deltas into generation 1 (prior = 0 = fresh). auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); ShardReducer r0(0, 2); @@ -139,9 +141,9 @@ TEST(CASGCShardReducer, MergesDeltasToInDegree) EXPECT_TRUE(r1.owns(b2)) << "r1 must own b2"; EXPECT_FALSE(r1.owns(b1)) << "r1 must not own b1"; - const auto runs0 = r0.reduce(*backend, layout, /*prior_runs=*/{}, /*new_generation=*/1, /*attempt=*/0, + const auto runs0 = r0.reduce(op, layout, /*prior_runs=*/{}, /*new_generation=*/1, /*attempt=*/0, std::move(buckets[0])); - const auto runs1 = r1.reduce(*backend, layout, /*prior_runs=*/{}, /*new_generation=*/1, /*attempt=*/0, + const auto runs1 = r1.reduce(op, layout, /*prior_runs=*/{}, /*new_generation=*/1, /*attempt=*/0, std::move(buckets[1])); ASSERT_EQ(runs0.size(), 1u) << "shard-0 reduce must produce exactly one RunRef"; @@ -300,13 +302,15 @@ TEST(CASGCShardCoordinator, ShardedFoldRoutesDeltasToOwningShards) buckets[blobShard(d.ref, kGcShards)].push_back(d); auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); std::vector> shard_runs(kGcShards); for (uint64_t shard = 0; shard < kGcShards; ++shard) { ShardReducer reducer{shard, kGcShards}; - shard_runs[shard] = reducer.reduce(*backend, layout, /*prior_runs=*/{}, /*new_generation=*/1, /*attempt=*/0, + shard_runs[shard] = reducer.reduce(op, layout, /*prior_runs=*/{}, /*new_generation=*/1, /*attempt=*/0, std::move(buckets[shard])); } @@ -459,6 +463,8 @@ TEST(CASGCShardTwoReplica, DisjointShardsConcurrentPerShardRuns) ASSERT_EQ(blobShard(b1, kGcShards), 1u) << "b1 must route to shard 1"; auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); /// (a) DISJOINTNESS — verify `owns` predicate before any reduce. @@ -485,11 +491,11 @@ TEST(CASGCShardTwoReplica, DisjointShardsConcurrentPerShardRuns) /// (b) PER-SHARD RUNS — drive both reducers. /// /// Run shard-0 reducer (simulates the shard-0 replica's work). - const auto runs0 = r0.reduce(*backend, layout, /*prior_runs=*/{}, kNewGen, kAttempt, std::move(bucket0)); + const auto runs0 = r0.reduce(op, layout, /*prior_runs=*/{}, kNewGen, kAttempt, std::move(bucket0)); ASSERT_FALSE(runs0.empty()) << "shard-0 reducer must produce at least one RunRef"; /// Run shard-1 reducer (simulates the shard-1 replica's work, interleaved from the test thread). - const auto runs1 = r1.reduce(*backend, layout, /*prior_runs=*/{}, kNewGen, kAttempt, std::move(bucket1)); + const auto runs1 = r1.reduce(op, layout, /*prior_runs=*/{}, kNewGen, kAttempt, std::move(bucket1)); ASSERT_FALSE(runs1.empty()) << "shard-1 reducer must produce at least one RunRef"; /// The blob-target runs for both shards are durably present (the reducer's write-once `putIfAbsent`), diff --git a/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp b/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp index 430e2faa9293..74410c1794c1 100644 --- a/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp +++ b/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp @@ -155,22 +155,28 @@ TEST(CASGCUndercount, H2DuplicateCommittedRemovalIsIdempotentNoUnderflow) class InterruptRoundCasBackend : public InMemoryBackend { public: - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { - if (arm_interrupt && key == gc_state_key) + if (arm_interrupt && expected_value && key == gc_state_key) { - const auto stored = get(key); - const uint64_t stored_gen = stored ? decodeGcState(stored->bytes).snap_generation : 0; - const uint64_t next_gen = decodeGcState(bytes).snap_generation; - if (next_gen > stored_gen) + const auto stored = InMemoryBackend::read(key, access); + if (stored + && decodeGcState(bytes).snap_generation > decodeGcState(stored->bytes).snap_generation) { - arm_interrupt = false; - throw DB::Exception(DB::ErrorCodes::ABORTED, - "test-injected: round-commit gc/state CAS denied (leader deposed mid-round; lease lost)"); + arm_interrupt = false; /// one-shot: only depose the first round-commit write + /// A REFUSAL, not a throw: a thrown transport error is an ambiguity the engine settles + /// by an exact read and then reissues while the precondition it named is unmoved, so + /// the round would commit on the reissue. A refused precondition ends the write at + /// once. The object is moved too -- the same bytes under a fresh incarnation -- because + /// a store refuses only what changed; the CONTENT is deliberately left alone, so this + /// round's own lease and cursor are exactly what a deposed round leaves behind. + (void)InMemoryBackend::write(key, stored->bytes, stored->value, access); + return std::unexpected(RawConflict{}); } } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } bool arm_interrupt = false; @@ -252,14 +258,15 @@ TEST(CASGCUndercount, H1DrainAfterDeposedRemovalFoldDoesNotUnderflow) class DropAtCommitBackend : public InMemoryBackend { public: - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override { /// The one-pass round has a SINGLE gc/state CAS that advances snap_generation. Fire the injected /// drop ONCE, just before that CAS commits — so the drop event is above this round's sealed cursor. if (arm_drop && key == gc_state_key) { - const auto stored = get(key); + const auto stored = InMemoryBackend::read(key, access); if (stored) { const GcState prev = decodeGcState(stored->bytes); @@ -272,7 +279,7 @@ class DropAtCommitBackend : public InMemoryBackend } } } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } bool arm_drop = false; diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index 84d457b08158..cb852d928618 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -31,12 +31,52 @@ using namespace DB::Cas; namespace { +/// The two request planes this file's keepers run on. Both are open-fence -- the exclusivity these +/// tests exercise is the mount protocol's own, not a fence's -- on the same injected boot clock the +/// keeper's lease deadline is expressed on, so the two never disagree about how much budget is left. +/// `sleep_step_ms`, when set, makes one inter-attempt pause jump the clock past the lease bound: that +/// is how a test asks for exactly one physical attempt without a per-call attempt cap. It depends on +/// the engine checking the bound, sleeping, then checking again -- a reissue that slept first would +/// send a second attempt. `tests::OperationForTest` covers a fixture needing one operation, but +/// neither the two planes a keeper takes nor this clock, which is why this stays local. +class Ops +{ +public: + Ops(std::shared_ptr backend, uint64_t * boot_ms, uint64_t sleep_step_ms = 0) + : mount(openRequestsForTest(backend)) + , farewell(openRequestsForTest(std::move(backend))) + , op(mount.admit()) + { + for (CasRequests * requests : {&mount, &farewell}) + { + requests->setNowFnForTest([boot_ms] { return *boot_ms; }); + requests->setSleepFnForTest( + [boot_ms, sleep_step_ms](uint64_t ms) { *boot_ms += sleep_step_ms ? sleep_step_ms : ms; }); + } + } + + Ops(const Ops &) = delete; + Ops & operator=(const Ops &) = delete; + + CasRequests mount; + CasRequests farewell; + CasOperation op; +}; + +/// A fixture write that must land, so a mis-seeded fixture fails where it is written rather than in +/// the assertion it silently invalidated. +void mustCommit(WriteResult && result, const String & what) +{ + if (!std::holds_alternative(result)) + throw DB::Exception(DB::ErrorCodes::ABORTED, "test fixture write '{}' did not commit", what); +} + /// The normal steady-state flow: `claimMount` writes the live (uuid, epoch) mount, THEN the keeper /// adopts it. Seed that claim so `start` adopts instead of self-tripping the double-start guard. -void seedOwnClaim(Backend & b, const Layout & l, const String & srid, UInt128 uuid, uint64_t epoch, +void seedOwnClaim(CasOperation & op, const Layout & l, const String & srid, UInt128 uuid, uint64_t epoch, uint64_t now_ms, uint64_t ttl_ms) { - ASSERT_EQ(claimMount(b, l, srid, uuid, epoch, now_ms, ttl_ms).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(op, l, srid, uuid, epoch, now_ms, ttl_ms).kind, MountClaimResult::Claimed); } class RenewalScriptBackend final : public InMemoryBackend @@ -55,20 +95,24 @@ class RenewalScriptBackend final : public InMemoryBackend { String key; String bytes; - Token expected; + std::optional expected; }; - using InMemoryBackend::get; - using InMemoryBackend::putOverwrite; - std::deque actions; std::vector attempts; std::function cancel_after_write; - uint64_t get_calls = 0; + uint64_t read_calls = 0; - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override + /// Only a GUARDED write of a mount slot is scripted; the fixture's own seeding and every other + /// key reach the store untouched. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - attempts.push_back({key, bytes, expected}); + if (!expected_value || !key.ends_with("/mount")) + return InMemoryBackend::write(key, bytes, expected_value, access); + + attempts.push_back({key, bytes, expected_value}); const Action action = actions.empty() ? Action::Delegate : actions.front(); if (!actions.empty()) actions.pop_front(); @@ -76,11 +120,11 @@ class RenewalScriptBackend final : public InMemoryBackend if (action == Action::ThrowBefore || action == Action::ThrowBeforeThenLandAfterResolve) { if (action == Action::ThrowBeforeThenLandAfterResolve) - pending = Attempt{key, bytes, expected}; + pending = Attempt{key, bytes, expected_value}; throw Poco::TimeoutException("injected renewal response uncertainty before a result"); } - PutResult result = InMemoryBackend::putOverwrite(key, bytes, expected, meta); + auto result = InMemoryBackend::write(key, bytes, expected_value, access); if (action == Action::LandThenThrow) { if (cancel_after_write) @@ -92,16 +136,16 @@ class RenewalScriptBackend final : public InMemoryBackend return result; } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - ++get_calls; - std::optional result = InMemoryBackend::get(key, range); + ++read_calls; + std::optional result = InMemoryBackend::read(key, access); if (pending && pending->key == key) { const Attempt delayed = *pending; pending.reset(); - const PutResult landed = InMemoryBackend::putOverwrite(delayed.key, delayed.bytes, delayed.expected, {}); - if (landed.outcome != PutOutcome::Done) + const auto landed = InMemoryBackend::write(delayed.key, delayed.bytes, delayed.expected, access); + if (!landed.has_value()) throw DB::Exception(DB::ErrorCodes::ABORTED, "injected delayed renewal did not land"); } return result; @@ -111,27 +155,15 @@ class RenewalScriptBackend final : public InMemoryBackend std::optional pending; }; -CasRequestBudget renewalBudget(uint32_t max_attempts = 3) -{ - return CasRequestBudget{ - .attempt_timeout_ms = 10, - .operation_deadline_ms = 500, - .max_attempts = max_attempts, - .lease_safety_margin_ms = 20, - .retry_initial_backoff_ms = 0, - .retry_max_backoff_ms = 0, - }; -} - MountRenewOperationEnvironment renewalEnvironment( uint64_t & boot_ms, - const std::function & stop_cause = {}) + const std::function & live = {}, + const std::function & cancelled = {}) { return MountRenewOperationEnvironment{ .boot_ms = [&boot_ms] { return boot_ms; }, - .stop_cause = stop_cause ? stop_cause : [] { return CasOverwriteStopCause::Continue; }, - .wait_before_retry = [](uint64_t) { return true; }, - .observe = {}, + .live = live, + .cancelled = cancelled, }; } @@ -156,7 +188,7 @@ DB::Exception terminalException(const MountRenewResult & result) void renewKeeperOrThrow(MountLeaseKeeper & keeper) { - const MountRenewResult result = keeper.renew(renewalBudget(), MountRenewOperationEnvironment{}); + const MountRenewResult result = keeper.renew(MountRenewOperationEnvironment{}); if (result.outcome == MountRenewOutcome::Terminal) std::rethrow_exception(result.failure); if (result.outcome != MountRenewOutcome::Committed) @@ -172,15 +204,18 @@ TEST(CASHeartbeat, AnchorCarriesFloor) const UInt128 uuid(0x1234); uint64_t now_ms = 1000; uint64_t min_active_build_sequence_now = 5; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); - auto hr = backend->head(layout.mountKey(srid)); - ASSERT_TRUE(hr.exists); - auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); + ASSERT_TRUE(ops.op.head(layout.mountKey(srid), Retry::standard()).has_value()); + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); EXPECT_EQ(m.writer_epoch, 9u); EXPECT_EQ(m.min_active_build_sequence, 5u); EXPECT_EQ(m.seq, 1u); @@ -195,10 +230,14 @@ TEST(CASHeartbeat, RenewRereadsCallbackAndBumpsSeq) const UInt128 uuid(0x1234); uint64_t now_ms = 1000; uint64_t min_active_build_sequence_now = 5; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); /// The dynamic field moves; the renewal re-reads it off the callback and bumps seq. @@ -206,7 +245,7 @@ TEST(CASHeartbeat, RenewRereadsCallbackAndBumpsSeq) min_active_build_sequence_now = 8; renewKeeperOrThrow(keeper); - auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); EXPECT_EQ(m.min_active_build_sequence, 8u); EXPECT_EQ(m.seq, 2u); EXPECT_EQ(m.expires_at_ms, 1500u + 100u); @@ -219,16 +258,20 @@ TEST(CASHeartbeat, StopStampsExpiredAndFarewellSentinel) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); now_ms = 2000; keeper.release(); - auto m = decodeMountLease(backend->get(layout.mountKey(srid))->bytes); + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); /// Terminal body stamps the lease already-expired (so a same-server reopen reclaims immediately) /// AND folds the watermark farewell into it (min_active_build_sequence = UINT64_MAX). EXPECT_LE(m.expires_at_ms, now_ms); @@ -246,21 +289,26 @@ TEST(CASHeartbeat, SameEpochUnfencedTouchIsUncertainNotFatal) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); - /// The slot advances past our held token under our own pair (the ambiguous-landed-renewal shape). - const HeadResult h = backend->head(layout.mountKey(srid)); - ASSERT_TRUE(h.exists); + /// The slot advances past the incarnation we hold, under our own pair (the ambiguous-landed-renewal shape). + const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); + ASSERT_TRUE(observed.has_value()); MountLease advanced; advanced.server_uuid = uuid; advanced.writer_epoch = 9; advanced.seq = 99; advanced.write_attempt_id = UInt128{99}; - backend->putOverwrite(layout.mountKey(srid), encodeMountLease(advanced), h.token); + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(advanced), observed->incarnation, + Retry::standard()), "advanced slot"); try { @@ -288,20 +336,25 @@ TEST(CASHeartbeat, SupersededTouchIsFailClosedNotFatal) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); - const HeadResult h = backend->head(layout.mountKey(srid)); - ASSERT_TRUE(h.exists); + const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); + ASSERT_TRUE(observed.has_value()); MountLease successor; successor.server_uuid = uuid; successor.writer_epoch = 10; successor.seq = 1; successor.write_attempt_id = UInt128{1}; - backend->putOverwrite(layout.mountKey(srid), encodeMountLease(successor), h.token); + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(successor), observed->incarnation, + Retry::standard()), "successor slot"); try { @@ -337,20 +390,25 @@ TEST(CASHeartbeat, ForeignUuidTouchFailsClosedWithoutAborting) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); - const HeadResult h = backend->head(layout.mountKey(srid)); - ASSERT_TRUE(h.exists); + const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); + ASSERT_TRUE(observed.has_value()); MountLease foreign; foreign.server_uuid = UInt128(0x9999); foreign.writer_epoch = 1; foreign.seq = 1; foreign.write_attempt_id = UInt128{1}; - backend->putOverwrite(layout.mountKey(srid), encodeMountLease(foreign), h.token); + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(foreign), observed->incarnation, + Retry::standard()), "foreign slot"); /// Restored on every exit: this flag is process-global and every later test in this binary would /// inherit it. @@ -386,9 +444,11 @@ TEST(CASMountAudit, ClaimReleaseAndForeignConflictEmitEvents) std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); const uint64_t now_ms = 1'000'000; /// mint for uuid 1 -> one mount_claim - ASSERT_EQ(claimMount(*backend, layout, "a", UInt128{1}, 1, now_ms, /*ttl_ms=*/10'000, {}, sink).kind, + ASSERT_EQ(claimMount(ops.op, layout, "a", UInt128{1}, 1, now_ms, /*ttl_ms=*/10'000, {}, sink).kind, MountClaimResult::Claimed); ASSERT_EQ(seen.size(), 1u); EXPECT_EQ(seen[0].type, CasEventType::MountClaim); @@ -397,7 +457,7 @@ TEST(CASMountAudit, ClaimReleaseAndForeignConflictEmitEvents) /// a FOREIGN uuid claiming a live slot -> mount_conflict carrying the current holder's identity seen.clear(); - (void)claimMount(*backend, layout, "a", UInt128{2}, 1, now_ms, /*ttl_ms=*/10'000, {}, sink); + (void)claimMount(ops.op, layout, "a", UInt128{2}, 1, now_ms, /*ttl_ms=*/10'000, {}, sink); ASSERT_FALSE(seen.empty()); EXPECT_EQ(seen.back().type, CasEventType::MountConflict); EXPECT_EQ(seen.back().detail.at("server_root_id"), "a"); @@ -416,12 +476,16 @@ TEST(CASMountAudit, KeeperAdoptEmitsClaimAndTerminateEmitsRelease) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); ASSERT_EQ(seen.size(), 1u); @@ -450,12 +514,16 @@ TEST(CASMountAudit, KeeperForeignConflictRefusesAndNamesHolder) const UInt128 uuid_y(0x2222); uint64_t now_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); /// Foreign holder X claims the slot first. - ASSERT_EQ(claimMount(*backend, layout, srid, uuid_x, /*our_epoch=*/1, now_ms, /*ttl_ms=*/100).kind, + ASSERT_EQ(claimMount(ops.op, layout, srid, uuid_x, /*our_epoch=*/1, now_ms, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - MountLeaseKeeper keeper(backend, layout, srid, uuid_y, /*writer_epoch=*/1, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid_y, /*writer_epoch=*/1, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(2000), + [&] { return boot_ms; }); /// The enriched refusal message must name the OBSERVED holder (X), not the caller (Y). const String holder_uuid = u128ToHex(uuid_x); @@ -478,22 +546,26 @@ TEST(CASMountAudit, KeeperAdoptRefusesFencedSelfWithTypedError) const UInt128 uuid(0x1234); uint64_t now_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); /// mint (uuid, epoch 9), then fence it in place (what computeHeartbeatFloor does on expiry): - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); { - auto got = backend->get(layout.mountKey(srid)); + auto got = ops.op.read(layout.mountKey(srid), Retry::standard()); MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - ASSERT_EQ(backend->putOverwrite(layout.mountKey(srid), encodeMountLease(fenced), got->token).outcome, - PutOutcome::Done); + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(fenced), got->incarnation, + Retry::standard()), "fence-out"); } std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; /// A keeper for the SAME (uuid, epoch) tries to adopt the now-fenced slot. - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, sink); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(2000), + [&] { return boot_ms; }); bool threw = false; try @@ -524,25 +596,29 @@ TEST(CASHeartbeat, RenewOverFencedOwnSlotIsClassifiedNotForeign) const String srid = "test"; const UInt128 uuid(0x1234); uint64_t now_ms = 1000; - seedOwnClaim(*backend, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; - MountLeaseKeeper keeper(backend, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), - [&] { return now_ms; }, [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(0), + [&] { return boot_ms; }); keeper.start(); seen.clear(); /// Mid-run: the GC fences our own (uuid, epoch) mount slot in place (as `computeHeartbeatFloor` - /// does on an expired lease), preserving the whole body — a token-guarded putOverwrite, exactly - /// as the GC's own fence-out does it. + /// does on an expired lease), preserving the whole body — a guarded write against the incarnation + /// it observed, exactly as the GC's own fence-out does it. { - const auto got = backend->get(layout.mountKey(srid)); + const auto got = ops.op.read(layout.mountKey(srid), Retry::standard()); MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - ASSERT_EQ(backend->putOverwrite(layout.mountKey(srid), encodeMountLease(fenced), got->token).outcome, - PutOutcome::Done); + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(fenced), got->incarnation, + Retry::standard()), "fence-out"); } /// The renewal must classify the fence honestly — not "foreign writer": @@ -578,13 +654,14 @@ TEST(CASHeartbeat, KeeperStateAllowsOnlyActiveReleaseOrTerminal) auto backend = std::make_shared(); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "released", uuid, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "released", uuid, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "released", uuid, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "released", uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); EXPECT_EQ(keeper.state(), MountLeaseKeeperState::New); - EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalBudget(), renewalEnvironment(boot_ms))); + EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalEnvironment(boot_ms))); EXPECT_KEEPER_STATE_REJECTION(keeper.release()); EXPECT_EQ(keeper.start(), 100u); EXPECT_KEEPER_STATE_REJECTION(keeper.start()); @@ -592,7 +669,7 @@ TEST(CASHeartbeat, KeeperStateAllowsOnlyActiveReleaseOrTerminal) keeper.release(); EXPECT_EQ(keeper.state(), MountLeaseKeeperState::Released); EXPECT_KEEPER_STATE_REJECTION(keeper.start()); - EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalBudget(), renewalEnvironment(boot_ms))); + EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalEnvironment(boot_ms))); EXPECT_KEEPER_STATE_REJECTION(keeper.release()); } @@ -600,19 +677,22 @@ TEST(CASHeartbeat, KeeperStateAllowsOnlyActiveReleaseOrTerminal) auto backend = std::make_shared(); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "terminal", uuid, 9, wall_ms, 1000); + /// One pause jumps the clock past the lease bound, so the ambiguous first attempt is the only + /// one this renewal ever sends and its verdict is the terminal one under test. + Ops ops(backend, &boot_ms, /*sleep_step_ms=*/10'000); + seedOwnClaim(ops.op, layout, "terminal", uuid, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "terminal", uuid, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "terminal", uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); backend->actions = {RenewalScriptBackend::Action::ThrowBefore}; - const MountRenewResult result = keeper.renew(renewalBudget(1), renewalEnvironment(boot_ms)); + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Terminal); EXPECT_NE(result.failure, nullptr); EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); EXPECT_KEEPER_STATE_REJECTION(keeper.start()); - EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalBudget(), renewalEnvironment(boot_ms))); + EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalEnvironment(boot_ms))); EXPECT_KEEPER_STATE_REJECTION(keeper.release()); } @@ -627,16 +707,17 @@ TEST(CASHeartbeat, RenewalRetriesOneImmutableBodyAndAdoptsLostResponse) const UInt128 uuid{0x1234}; uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, srid, uuid, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, srid, uuid, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, srid, uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); backend->attempts.clear(); backend->actions = {RenewalScriptBackend::Action::ThrowBefore, RenewalScriptBackend::Action::Delegate}; - MountRenewResult retried = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)); + MountRenewResult retried = keeper.renew(renewalEnvironment(boot_ms)); ASSERT_EQ(retried.outcome, MountRenewOutcome::Committed); ASSERT_EQ(backend->attempts.size(), 2u); EXPECT_EQ(backend->attempts[0].key, backend->attempts[1].key); @@ -647,11 +728,11 @@ TEST(CASHeartbeat, RenewalRetriesOneImmutableBodyAndAdoptsLostResponse) backend->attempts.clear(); backend->actions = {RenewalScriptBackend::Action::LandThenThrow}; - MountRenewResult adopted = keeper.renew(renewalBudget(1), renewalEnvironment(boot_ms)); + MountRenewResult adopted = keeper.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(adopted.outcome, MountRenewOutcome::Committed); - EXPECT_TRUE(adopted.diagnostics.resolved_by_get); - EXPECT_EQ(adopted.diagnostics.attempts_sent, 1u); - EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey(srid))->bytes).write_attempt_id, + EXPECT_TRUE(adopted.resolved_by_read); + EXPECT_EQ(adopted.attempts_sent, 1u); + EXPECT_EQ(decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes).write_attempt_id, decodeMountLease(backend->attempts.front().bytes).write_attempt_id); } @@ -661,22 +742,26 @@ TEST(CASHeartbeat, DeadlineBeforeSendTerminalizesWithTypedFailure) Layout layout("pool"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 100); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 100); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(100), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(100), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); backend->attempts.clear(); - backend->get_calls = 0; + backend->read_calls = 0; boot_ms = 180; - const MountRenewResult result = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)); + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); const DB::Exception failure = terminalException(result); EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); - EXPECT_NE(failure.message().find("no attempt was sent"), String::npos) << failure.message(); - EXPECT_EQ(result.diagnostics.unresolved_reason, CasUnresolvedReason::NoAttemptSent); + EXPECT_NE(failure.message().find("no attempt sent"), String::npos) << failure.message(); + EXPECT_NE(failure.message().find("external_lease_deadline"), String::npos) << failure.message(); + EXPECT_FALSE(result.sent_any); + ASSERT_TRUE(result.deadline_source.has_value()); + EXPECT_EQ(*result.deadline_source, GaveUp::Source::Lease); EXPECT_TRUE(backend->attempts.empty()); - EXPECT_EQ(backend->get_calls, 0u) << "a pre-send terminal deadline must perform no diagnostic GET"; + EXPECT_EQ(backend->read_calls, 0u) << "a pre-send terminal deadline must perform no diagnostic read"; } TEST(CASHeartbeat, CancellationBeforeSendIsNotAttemptedAndAllowsRelease) @@ -685,16 +770,17 @@ TEST(CASHeartbeat, CancellationBeforeSendIsNotAttemptedAndAllowsRelease) Layout layout("pool"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); backend->attempts.clear(); - backend->get_calls = 0; - const auto cancelled = [] { return CasOverwriteStopCause::Cancelled; }; - const MountRenewResult result = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms, cancelled)); + backend->read_calls = 0; + const MountRenewResult result = keeper.renew(renewalEnvironment( + boot_ms, /*live=*/[] { return false; }, /*cancelled=*/[] { return true; })); EXPECT_EQ(result.outcome, MountRenewOutcome::NotAttempted); EXPECT_EQ(result.failure, nullptr); EXPECT_EQ(keeper.state(), MountLeaseKeeperState::Active); @@ -710,28 +796,27 @@ TEST(CASHeartbeat, CancellationAfterSendIsTerminalAndForbidsRelease) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; bool cancelled = false; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); backend->attempts.clear(); - backend->get_calls = 0; + backend->read_calls = 0; backend->cancel_after_write = [&] { cancelled = true; }; backend->actions = {RenewalScriptBackend::Action::ReturnThenCancel}; const MountRenewResult result = keeper.renew( - renewalBudget(), renewalEnvironment(boot_ms, [&] { - return cancelled ? CasOverwriteStopCause::Cancelled : CasOverwriteStopCause::Continue; - })); + renewalEnvironment(boot_ms, /*live=*/[&] { return !cancelled; }, /*cancelled=*/[&] { return cancelled; })); const DB::Exception failure = terminalException(result); EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); - EXPECT_EQ(result.diagnostics.unresolved_reason, CasUnresolvedReason::FenceLostPostWrite); - EXPECT_EQ(backend->get_calls, 0u) << "post-write cancellation must not start a diagnostic GET"; + EXPECT_TRUE(result.sent_any); + EXPECT_EQ(backend->read_calls, 0u) << "post-write cancellation must not start a diagnostic read"; EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); - const String bytes_before = backend->get(layout.mountKey("test"))->bytes; + const String bytes_before = ops.op.read(layout.mountKey("test"), Retry::standard())->bytes; EXPECT_FALSE(keeper.canRelease()); - EXPECT_EQ(backend->get(layout.mountKey("test"))->bytes, bytes_before); + EXPECT_EQ(ops.op.read(layout.mountKey("test"), Retry::standard())->bytes, bytes_before); } TEST(CASHeartbeat, SlowResolvedSuccessKeepsAttemptStartAnchor) @@ -740,16 +825,17 @@ TEST(CASHeartbeat, SlowResolvedSuccessKeepsAttemptStartAnchor) Layout layout("pool"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); boot_ms = 150; backend->cancel_after_write = [&] { boot_ms = 400; }; backend->actions = {RenewalScriptBackend::Action::LandThenThrow}; - const MountRenewResult result = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)); + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Committed); EXPECT_EQ(result.attempt_start_boot_ms, 150u); EXPECT_EQ(keeper.lastCommittedAttemptStartBootMs(), 150u); @@ -764,26 +850,27 @@ TEST(CASHeartbeat, SamePairTwinAndForeignOrSuccessorStayTerminal) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - seedOwnClaim(*backend, layout, "test", uuid, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", uuid, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", uuid, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); - auto got = backend->get(layout.mountKey("test")); + auto got = ops.op.read(layout.mountKey("test"), Retry::standard()); MountLease current = decodeMountLease(got->bytes); current.server_uuid = current_uuid; current.writer_epoch = current_epoch; current.write_attempt_id = current_attempt; ++current.seq; - ASSERT_EQ(backend->putOverwrite(layout.mountKey("test"), encodeMountLease(current), got->token).outcome, - PutOutcome::Done); - backend->get_calls = 0; - const MountRenewResult result = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)); + mustCommit(ops.op.replace(layout.mountKey("test"), encodeMountLease(current), got->incarnation, + Retry::standard()), "competing slot"); + backend->read_calls = 0; + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); const DB::Exception failure = terminalException(result); EXPECT_NE(failure.code(), DB::ErrorCodes::LOGICAL_ERROR); EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); - EXPECT_EQ(backend->get_calls, 1u) << "the controller's resolving GET must be the only terminal read"; + EXPECT_EQ(backend->read_calls, 1u) << "the write's own resolving read must be the only terminal read"; }; run_case(UInt128{1}, 9, UInt128{0xAAAA}); @@ -797,9 +884,10 @@ TEST(CASHeartbeat, ExpectedPredecessorThenLateLandingIsAdoptedExactly) Layout layout("pool"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); @@ -808,12 +896,12 @@ TEST(CASHeartbeat, ExpectedPredecessorThenLateLandingIsAdoptedExactly) RenewalScriptBackend::Action::ThrowBeforeThenLandAfterResolve, RenewalScriptBackend::Action::Delegate, }; - const MountRenewResult result = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)); + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Committed); - EXPECT_TRUE(result.diagnostics.resolved_by_get); + EXPECT_TRUE(result.resolved_by_read); ASSERT_EQ(backend->attempts.size(), 2u); EXPECT_EQ(backend->attempts[0].bytes, backend->attempts[1].bytes); - EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey("test"))->bytes).write_attempt_id, + EXPECT_EQ(decodeMountLease(ops.op.read(layout.mountKey("test"), Retry::standard())->bytes).write_attempt_id, decodeMountLease(backend->attempts[0].bytes).write_attempt_id); } @@ -825,24 +913,26 @@ TEST(CASHeartbeat, GcFenceAndVanishedMountStayTerminal) Layout layout("pool"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); const String key = layout.mountKey("test"); - auto got = backend->get(key); + auto got = ops.op.read(key, Retry::standard()); if (vanish) - ASSERT_EQ(backend->deleteExact(key, got->token).kind, DeleteOutcome::Kind::Deleted); + ASSERT_EQ(ops.op.remove(key, got->incarnation, Retry::standard()), Removal::Removed); else { MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; ++fenced.seq; - ASSERT_EQ(backend->putOverwrite(key, encodeMountLease(fenced), got->token).outcome, PutOutcome::Done); + mustCommit(ops.op.replace(key, encodeMountLease(fenced), got->incarnation, Retry::standard()), + "fence-out"); } - const DB::Exception failure = terminalException(keeper.renew(renewalBudget(), renewalEnvironment(boot_ms))); + const DB::Exception failure = terminalException(keeper.renew(renewalEnvironment(boot_ms))); EXPECT_NE(failure.code(), DB::ErrorCodes::LOGICAL_ERROR); EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); }; @@ -852,64 +942,74 @@ TEST(CASHeartbeat, GcFenceAndVanishedMountStayTerminal) TEST(CASHeartbeat, LateDeliveryAfterTerminalCannotRearmOrOverwriteSuccessor) { - const auto make_terminal = [](const std::shared_ptr & backend, - const Layout & layout, const String & srid, - uint64_t & wall_ms, uint64_t & boot_ms) + Layout layout("pool"); { - seedOwnClaim(*backend, layout, srid, UInt128{1}, 9, wall_ms, 1000); - auto keeper = std::make_unique( - backend, layout, srid, UInt128{1}, 9, std::chrono::milliseconds(1000), + auto backend = std::make_shared(); + uint64_t wall_ms = 1000; + uint64_t boot_ms = 100; + /// One pause jumps the clock past the lease bound, so the ambiguous first attempt is the only + /// one this renewal sends and the renewal ends terminal with that attempt still in flight. + Ops ops(backend, &boot_ms, /*sleep_step_ms=*/10'000); + seedOwnClaim(ops.op, layout, "before-reclaim", UInt128{1}, 9, wall_ms, 1000); + MountLeaseKeeper keeper( + ops.mount, ops.farewell, layout, "before-reclaim", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, CasEventSink{}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper->start(); + keeper.start(); backend->actions = {RenewalScriptBackend::Action::ThrowBeforeThenLandAfterResolve}; - const MountRenewResult result = keeper->renew(renewalBudget(1), renewalEnvironment(boot_ms)); + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Terminal); - EXPECT_EQ(keeper->state(), MountLeaseKeeperState::RenewalTerminal); - return keeper; - }; - Layout layout("pool"); - { - auto backend = std::make_shared(); - uint64_t wall_ms = 1000; - uint64_t boot_ms = 100; - auto keeper = make_terminal(backend, layout, "before-reclaim", wall_ms, boot_ms); - const MountLease landed = decodeMountLease(backend->get(layout.mountKey("before-reclaim"))->bytes); + /// The delayed write landed during the resolving read. It carries this keeper's own epoch, and + /// it does not put the keeper back in business. + const MountLease landed = decodeMountLease( + ops.op.read(layout.mountKey("before-reclaim"), Retry::standard())->bytes); EXPECT_EQ(landed.writer_epoch, 9u); - EXPECT_EQ(keeper->state(), MountLeaseKeeperState::RenewalTerminal); + EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); } { auto backend = std::make_shared(); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "after-successor", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms, /*sleep_step_ms=*/10'000); + seedOwnClaim(ops.op, layout, "after-successor", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "after-successor", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "after-successor", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); + + /// The incarnation the about-to-be-terminal renewal names as its precondition: a late delivery + /// of that attempt can only ever be replayed against exactly this one. + const Incarnation delayed_precondition + = ops.op.read(layout.mountKey("after-successor"), Retry::standard())->incarnation; + backend->actions = {RenewalScriptBackend::Action::ThrowBefore}; - const MountRenewResult result = keeper.renew(renewalBudget(1), renewalEnvironment(boot_ms)); + const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); ASSERT_EQ(result.outcome, MountRenewOutcome::Terminal); ASSERT_FALSE(backend->attempts.empty()); const auto delayed = backend->attempts.back(); - auto current = backend->get(delayed.key); + + /// The GC fences the slot, then a successor claims it at a fresh epoch and adopts it. + auto current = ops.op.read(delayed.key, Retry::standard()); MountLease fenced = decodeMountLease(current->bytes); fenced.gc_fenced = true; ++fenced.seq; - ASSERT_EQ(backend->InMemoryBackend::putOverwrite(delayed.key, encodeMountLease(fenced), current->token, {}).outcome, - PutOutcome::Done); - ASSERT_EQ(claimMount(*backend, layout, "after-successor", UInt128{1}, 10, wall_ms, 1000).kind, + mustCommit(ops.op.replace(delayed.key, encodeMountLease(fenced), current->incarnation, Retry::standard()), + "fence-out"); + ASSERT_EQ(claimMount(ops.op, layout, "after-successor", UInt128{1}, 10, wall_ms, 1000).kind, MountClaimResult::Claimed); MountLeaseKeeper successor( - backend, layout, "after-successor", UInt128{1}, 10, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "after-successor", UInt128{1}, 10, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); successor.start(); - EXPECT_EQ(backend->InMemoryBackend::putOverwrite(delayed.key, delayed.bytes, delayed.expected, {}).outcome, - PutOutcome::PreconditionFailed); - EXPECT_EQ(decodeMountLease(backend->get(delayed.key)->bytes).writer_epoch, 10u); + + /// Replaying the delayed attempt against the incarnation it named is refused; the successor's + /// body is what stands. + EXPECT_TRUE(std::holds_alternative( + ops.op.replace(delayed.key, delayed.bytes, delayed_precondition, Retry::once()))); + EXPECT_EQ(decodeMountLease(ops.op.read(delayed.key, Retry::standard())->bytes).writer_epoch, 10u); } } @@ -919,21 +1019,22 @@ TEST(CASHeartbeat, WallClockStepsAndBootSuspendCannotExtendAuthority) Layout layout("pool"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - seedOwnClaim(*backend, layout, "test", UInt128{1}, 9, wall_ms, 1000); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); MountLeaseKeeper keeper( - backend, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), + ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); keeper.start(); wall_ms = 9'000'000; - EXPECT_EQ(keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); + EXPECT_EQ(keeper.renew(renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); wall_ms = 1; - EXPECT_EQ(keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); + EXPECT_EQ(keeper.renew(renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); backend->attempts.clear(); boot_ms += 10'000; - const MountRenewResult suspended = keeper.renew(renewalBudget(), renewalEnvironment(boot_ms)); + const MountRenewResult suspended = keeper.renew(renewalEnvironment(boot_ms)); const DB::Exception failure = terminalException(suspended); EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); EXPECT_TRUE(backend->attempts.empty()) << "suspend-sized BOOTTIME overshoot must close admission"; diff --git a/src/Disks/tests/gtest_cas_holey_list_detector.cpp b/src/Disks/tests/gtest_cas_holey_list_detector.cpp index 0c4779c1a672..8841836ebffc 100644 --- a/src/Disks/tests/gtest_cas_holey_list_detector.cpp +++ b/src/Disks/tests/gtest_cas_holey_list_detector.cpp @@ -56,8 +56,8 @@ namespace class HoleyListBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; + /// Unhide the legacy `list` overloads the primitive override below would otherwise hide. + using Backend::list; /// Omit `key` from the `nth` (0-based) subsequent qualifying `list` call. Resets the counter. void omitFromNthListCall(const String & key, size_t nth) { @@ -76,14 +76,16 @@ class HoleyListBackend : public InMemoryBackend return served; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + /// Sabotages the PRIMITIVE, which every legacy forwarder reaches too, so the hole is served + /// whichever surface issued the enumeration. + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); std::lock_guard lock(m); if (omitted.empty()) return page; auto it = std::find_if(page.keys.begin(), page.keys.end(), - [&](const ListedKey & k) { return k.key == omitted; }); + [&](const RawListedKey & k) { return k.key == omitted; }); if (it == page.keys.end()) return page; /// not a qualifying call — do not count it if (seen_calls++ != target_call) @@ -143,9 +145,13 @@ bool blobPresent(const std::shared_ptr & b, const Layout & lay /// than guessing a sequence number. std::set listRefKeys(Backend & b, const Layout & layout, const RootNamespace & ns) { - /// Stage B (Task 4-C): `ns` is born through the REAL append lane here, so its objects sit at a - /// real catalog-minted incarnation, not the Stage-A sentinel. - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(b, layout, ns).value(); + /// `ns` is born through the REAL append lane here, so its objects sit at a real catalog-minted + /// incarnation rather than a fixture-chosen one. The catalog read is made on an open-fence + /// operation of its own: this helper only observes, and shares no admission with the code + /// under test. + CasRequests requests = DB::Cas::tests::openRequestsForTest(b); + CasOperation op = requests.admit(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, ns).value(); std::set keys; forEachListedKey(b, layout.namespaceStreamPrefix(life), [&](const ListedKey & k) { keys.insert(k.key); }); return keys; diff --git a/src/Disks/tests/gtest_cas_inspect.cpp b/src/Disks/tests/gtest_cas_inspect.cpp index c20466591ac4..7d0ce0067c65 100644 --- a/src/Disks/tests/gtest_cas_inspect.cpp +++ b/src/Disks/tests/gtest_cas_inspect.cpp @@ -137,8 +137,8 @@ TEST(CASInspect, RendersRefOwnerKindWireWords) EXPECT_NE(json.find(R"("new_binding":{"kind":"precommit")"), String::npos) << json; } -/// `TokenType` renders as its full wire word; the blob-target-run test below covers `emulated`, so -/// this pins the other two (`etag`/`generation`) via a second condemned-row-only run. +/// A recorded incarnation's dialect renders as its full wire word; the blob-target-run test below +/// covers `emulated`, so this pins the other two (`etag`/`generation`) via a second condemned-row-only run. TEST(CASInspect, RendersTokenTypeWireWordsEtagAndGeneration) { const Layout layout("p"); @@ -147,13 +147,13 @@ TEST(CASInspect, RendersTokenTypeWireWordsEtagAndGeneration) etag_rec.ref = bh(1); etag_rec.source_id = UInt128{0}; etag_rec.marker = RunMarker::Condemned; - etag_rec.token = Token{.value = "v-etag", .type = TokenType::ETag}; + etag_rec.token = PersistedIncarnation{"etag", "v-etag"}; SourceEdgeRecord gen_rec; gen_rec.ref = bh(1); gen_rec.source_id = UInt128{1}; gen_rec.marker = RunMarker::Condemned; - gen_rec.token = Token{.value = "v-gen", .type = TokenType::Generation}; + gen_rec.token = PersistedIncarnation{"generation", "v-gen"}; DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); @@ -209,7 +209,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) condemned_rec.source_id = UInt128{0}; condemned_rec.marker = RunMarker::Condemned; condemned_rec.delete_pending = true; - condemned_rec.token = Token{.value = "etag-1", .type = TokenType::Emulated}; + condemned_rec.token = PersistedIncarnation{"emulated", "etag-1"}; condemned_rec.size = 123; condemned_rec.condemn_round = 7; condemned_rec.marker_confirmed = true; diff --git a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp index be5cda9f9954..db8ab08711bb 100644 --- a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp +++ b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp @@ -57,41 +57,39 @@ void fenceOutMount(Backend & backend, const String & mount_key) ASSERT_EQ(backend.putOverwrite(mount_key, encodeMountLease(m), got->token).outcome, PutOutcome::Done); } -/// A Backend decorator whose head/get/list throw an untyped transport error while `fail` is armed. Starts -/// DISARMED so `Pool::open` succeeds; a test arms it only to make the identity probe inconclusive. Mirrors -/// gtest_cas_sentinel_probe.cpp's `TransportFaultBackend`, but toggleable AFTER open. +/// A Backend decorator whose reads, heads and lists throw an untyped transport error while `fail` is +/// armed. Starts DISARMED so `Pool::open` succeeds; a test arms it only to make the identity probe +/// inconclusive. Mirrors gtest_cas_sentinel_probe.cpp's `TransportFaultBackend`, but toggleable AFTER open. class ToggleableTransportFaultBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::head; - using InMemoryBackend::list; - /// Unhide the base convenience overloads, matching every other Backend subclass in this suite. - using Backend::get; - using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - - HeadResult head(const String & key) override + /// Unhide the LEGACY convenience overloads that the primitive overrides below would otherwise hide. + using Backend::head; + using Backend::list; + + /// The faults sit on the TRANSPORT PRIMITIVES, because that is where every caller reaches the store: + /// the lifecycle gate probes `_pool_meta` through `probeSentinelRaw`, which speaks only these. A + /// legacy caller still reaches the fault, through the forwarder, so arming it here covers both + /// surfaces rather than only one. + std::optional head(const String & key, TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (fail.load()) throw std::runtime_error("injected fault: transport error"); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } std::atomic fail{false}; @@ -131,8 +129,8 @@ TEST(CASLifecycleCondition, SentinelsDeletedEntersIdentityLostTerminal) backend->resetCounts(); EXPECT_FALSE(store->tryRemountOnce()); EXPECT_EQ(store->lifecycle(), PoolLifecycle::IdentityLost); - EXPECT_EQ(backend->putTotal(), 0u) << "a terminal-IdentityLost gate probe must never claim, allocate, or write"; - EXPECT_GE(backend->headRequestCount(meta_key), 1u) << "the gate still probes _pool_meta authoritatively"; + EXPECT_EQ(backend->writeTotal(), 0u) << "a terminal-IdentityLost gate probe must never claim, allocate, or write"; + EXPECT_GE(backend->headCount(meta_key), 1u) << "the gate still probes _pool_meta authoritatively"; } /// (a2) rev.8 worker-exit: `IdentityLost` is terminal, so the persistent self-remount worker must self-exit @@ -244,7 +242,7 @@ TEST(CASLifecycleCondition, ProbeTransportErrorStaysTransientAndRetries) auto store = DB::Cas::tests::openPoolForTest(backend); ASSERT_EQ(store->lifecycle(), PoolLifecycle::Live); - /// Arm the transport fault: the identity probe's head/get/list now throw → Indeterminate. + /// Arm the transport fault: every request the identity probe issues now throws → Indeterminate. backend->fail.store(true); EXPECT_FALSE(store->tryRemountOnce()); diff --git a/src/Disks/tests/gtest_cas_manifest_reader.cpp b/src/Disks/tests/gtest_cas_manifest_reader.cpp new file mode 100644 index 000000000000..03a75c560d7c --- /dev/null +++ b/src/Disks/tests/gtest_cas_manifest_reader.cpp @@ -0,0 +1,49 @@ +#include + +#include +#include "cas_test_helpers.h" + +#include + +namespace DB +{ +namespace ErrorCodes +{ + extern const int FILE_DOESNT_EXIST; +} +} + +using namespace DB::Cas; + +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::FakeClock; +using DB::Cas::tests::expectThrowsCode; + +namespace +{ + +CasRequests makeRequests(BackendPtr backend, FakeClock & clock, Fence fence = Fence::open()) +{ + return CasRequests(std::move(backend), std::move(fence), clock.nowFn(), clock.sleepFn()); +} + +} + +TEST(CASManifestReader, MissingManifestThrowsFileDoesntExist) +{ + FakeClock clock; + auto backend = std::make_shared(); + Layout layout("pool"); + PoolMeta meta; + CasEventSink sink; + auto requests = makeRequests(backend, clock); + CasManifestReader reader(requests, layout, meta, sink, /*manifest_decode_cache_bytes=*/0); + + const ManifestId id{RootNamespace("t"), ManifestRef{1, 1, 1}}; + + /// A live ref naming a missing manifest body is INV-NO-DANGLE: never a substituted empty + /// manifest, always the fail-closed exception -- and this must hold over the migrated + /// `CasOperation`-based read exactly as it did over the raw backend call. + expectThrowsCode(DB::ErrorCodes::FILE_DOESNT_EXIST, [&] { (void)reader.readManifest(id); }); + EXPECT_EQ(backend->getCount(layout.manifestKey(id)), 1u); +} diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index e0e96bab87b0..077312363d71 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -9,8 +9,6 @@ #include #include -#include -#include #include #include #include @@ -29,8 +27,6 @@ namespace ProfileEvents { extern const Event CASMountLeaseLost; extern const Event CASMountExclusivityViolation; - extern const Event CASMountRenewalAttempts; - extern const Event CASMountRenewalRetries; } using namespace DB::Cas; @@ -54,180 +50,202 @@ RefCatalog catalogOwning(const String & ns, NsState state) void renewKeeperOrThrow(MountLeaseKeeper & keeper) { - const MountRenewResult result = keeper.renew( - CasRequestBudget{.attempt_timeout_ms = 1, .operation_deadline_ms = 100, .max_attempts = 2, - .lease_safety_margin_ms = 0, .retry_initial_backoff_ms = 0, .retry_max_backoff_ms = 0}, - MountRenewOperationEnvironment{}); + const MountRenewResult result = keeper.renew(MountRenewOperationEnvironment{}); if (result.outcome == MountRenewOutcome::Terminal) std::rethrow_exception(result.failure); ASSERT_EQ(result.outcome, MountRenewOutcome::Committed); } -class OwnerConflictRevealsManifestBackend : public InMemoryBackend +/// The two request planes a keeper in this file runs on, plus one operation for the protocol calls +/// driven directly. Both planes are open-fence: these fixtures hold no mount lease, so nothing here +/// should be refused by a fence it does not have. The clock and the sleep are ALWAYS injected -- a +/// fixture that drives a lease deadline passes its own so a slow machine cannot run the bound out +/// mid-test, and one that does not still must not sleep for real when a fault sends the engine round +/// again. `tests::OperationForTest` covers the one-operation case but neither the two planes nor the +/// clock, which is why this stays local. +class Ops { public: - using InMemoryBackend::putIfAbsent; + explicit Ops(std::shared_ptr backend) : Ops(std::move(backend), nullptr) {} - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + Ops(std::shared_ptr backend, uint64_t * boot_ms) + : mount(openRequestsForTest(backend)) + , farewell(openRequestsForTest(std::move(backend))) + , op(mount.admit()) { - if (!fired && key == "p/gc/server-roots/root/x/owner") + uint64_t * clock = boot_ms ? boot_ms : &own_clock; + for (CasRequests * requests : {&mount, &farewell}) { - fired = true; - InMemoryBackend::putIfAbsent("p/cas/manifests/root/x/table/debris", "x"); - return {PutOutcome::PreconditionFailed, {}}; + requests->setNowFnForTest([clock] { return *clock; }); + requests->setSleepFnForTest([clock](uint64_t ms) { *clock += ms; }); } - return InMemoryBackend::putIfAbsent(key, bytes, meta); } - bool fired = false; + Ops(const Ops &) = delete; + Ops & operator=(const Ops &) = delete; + + CasRequests mount; + CasRequests farewell; + CasOperation op; + +private: + uint64_t own_clock = 0; }; -class EpochConflictRevealsManifestBackend : public InMemoryBackend +/// The incarnation currently at `key`, for a fixture that has to name it as a precondition. +Incarnation currentIncarnation(CasOperation & op, const String & key) { -public: - using InMemoryBackend::casPut; + const auto got = op.read(key, Retry::standard()); + if (!got) + throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "test fixture read of '{}' found nothing", key); + return got->incarnation; +} - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override +/// A fixture write that must land, so a mis-seeded fixture fails where it is written rather than in +/// the assertion it silently invalidated. +void mustCommit(WriteResult && result, const String & what) +{ + if (!std::holds_alternative(result)) + throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "test fixture write '{}' did not commit", what); +} + +class OwnerConflictRevealsManifestBackend : public InMemoryBackend +{ +public: + std::expected write( + const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - if (!fired && key == "p/gc/server-roots/root/x/epoch") + if (!fired && !expected_value && key == "p/gc/server-roots/root/x/owner") { fired = true; - /// Install the competing allocator's winning epoch before revealing owned work. The - /// retry must not accept that now-present epoch without rechecking the entire emptiness - /// bundle that authorized the original absent-epoch attempt. - const CasResult winner = InMemoryBackend::casPut( - key, encodeServerEpoch(ServerEpoch{.next_writer_epoch = 2}), expected, meta); - winner_installed = winner.outcome == CasOutcome::Committed; - InMemoryBackend::putIfAbsent("p/cas/manifests/root/x/table/debris", "x"); - return {CasOutcome::Conflict, {}}; + InMemoryBackend::write("p/cas/manifests/root/x/table/debris", "x", std::nullopt, access); + return std::unexpected(RawConflict{}); } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } bool fired = false; - bool winner_installed = false; }; -class RenewalLogBackend final : public InMemoryBackend +/// Loses the owner key to a racing claimer between the read and the create: installs `winner`'s owner +/// object, then refuses this write. The subtree stays empty, so the emptiness recompute passes and the +/// claim has to decide the race from what its own write observed. +class OwnerRaceBackend : public InMemoryBackend { public: - using InMemoryBackend::putOverwrite; - - bool throw_before_next_overwrite = false; + explicit OwnerRaceBackend(UInt128 winner_) : winner(winner_) {} - void armBlockedRetry() + /// Counts only reads of the owner key, so an extra re-read the conflict decision no longer needs + /// is visible even though the resolve read on the same key already counts once. + std::optional read(const String & key, TransportAccess & access) override { - std::lock_guard lock(mutex); - blocked_retry_armed = true; - renewal_puts = 0; - second_put_arrived = false; - release_second_put = false; + if (key == "p/gc/server-roots/r/owner") + ++owner_reads; + return InMemoryBackend::read(key, access); } - bool waitForSecondPut() + std::expected write( + const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - std::unique_lock lock(mutex); - return cv.wait_for(lock, std::chrono::seconds(2), [&] { return second_put_arrived; }); + if (!fired && !expected_value && key == "p/gc/server-roots/r/owner") + { + fired = true; + InMemoryBackend::write( + key, encodeOwner(OwnerObject{.server_uuid = winner, .retired_at_ms = std::nullopt}), + std::nullopt, access); + return std::unexpected(RawConflict{}); + } + return InMemoryBackend::write(key, bytes, expected_value, access); } - void releaseSecondPut() - { - std::lock_guard lock(mutex); - release_second_put = true; - cv.notify_all(); - } + bool fired = false; + size_t owner_reads = 0; - PutResult putOverwrite( - const String & key, - const String & bytes, - const Token & expected, - const ObjectMeta & meta) override +private: + UInt128 winner; +}; + +/// Refuses the FIRST write of the epoch key after installing a competing allocator's own epoch, so the +/// absent-epoch decision has to be made a second time. `reveal_owned_work` decides whether owned work +/// becomes visible at that same instant -- the fact the second decision must re-establish. +class EpochConflictBackend : public InMemoryBackend +{ +public: + explicit EpochConflictBackend(bool reveal_owned_work_ = true) : reveal_owned_work(reveal_owned_work_) {} + + std::expected write( + const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { + if (!fired && key == "p/gc/server-roots/root/x/epoch") { - std::unique_lock lock(mutex); - if (blocked_retry_armed) - { - ++renewal_puts; - if (renewal_puts == 1) - throw Poco::TimeoutException("injected renewal timeout before blocked retry"); - if (renewal_puts == 2) - { - second_put_arrived = true; - cv.notify_all(); - if (!cv.wait_for(lock, std::chrono::seconds(20), [&] { return release_second_put; })) - throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "blocked renewal retry was not released"); - blocked_retry_armed = false; - } - } + fired = true; + const auto winner = InMemoryBackend::write( + key, encodeServerEpoch(ServerEpoch{.next_writer_epoch = 2}), expected_value, access); + winner_installed = winner.has_value(); + if (reveal_owned_work) + InMemoryBackend::write("p/cas/manifests/root/x/table/debris", "x", std::nullopt, access); + return std::unexpected(RawConflict{}); } - if (std::exchange(throw_before_next_overwrite, false)) - throw Poco::TimeoutException("injected renewal timeout before commit"); - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } + bool fired = false; + bool winner_installed = false; + private: - std::mutex mutex; - std::condition_variable cv; - bool blocked_retry_armed = false; - uint32_t renewal_puts = 0; - bool second_put_arrived = false; - bool release_second_put = false; + bool reveal_owned_work; }; -class BlockingRenewalDebugChannel final : public Poco::Channel +/// Counts every request that reaches the store, per primitive, so a test can pin how many a protocol +/// step costs rather than only what it produced. +class RequestCountingBackend final : public InMemoryBackend { public: - void log(const Poco::Message & message) override + size_t reads = 0; + size_t heads = 0; + size_t writes = 0; + + std::optional read(const String & key, TransportAccess & access) override { - if (message.getText().find("physical retry attempt") == String::npos) - return; - std::unique_lock lock(mutex); - cv.wait_for(lock, std::chrono::seconds(20), [&] { return released; }); + ++reads; + return InMemoryBackend::read(key, access); } - void unblock() + std::optional head(const String & key, TransportAccess & access) override { - std::lock_guard lock(mutex); - released = true; - cv.notify_all(); + ++heads; + return InMemoryBackend::head(key, access); } -private: - std::mutex mutex; - std::condition_variable cv; - bool released = false; + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override + { + ++writes; + return InMemoryBackend::write(key, bytes, expected_value, access); + } }; -class ScopedBlockingRenewalDebugLog +class RenewalLogBackend final : public InMemoryBackend { public: - ScopedBlockingRenewalDebugLog() - : logger(getLogger("CasMountLeaseKeeper")) - , channel(new BlockingRenewalDebugChannel) - , old_channel(logger->getChannel(), /*shared=*/true) - , old_level(logger->getLevel()) - { - logger->setChannel(channel.get()); - logger->setLevel("debug"); - } + bool throw_before_next_overwrite = false; - ~ScopedBlockingRenewalDebugLog() + /// The fault lives on the primitive every write reaches the store through, keyed to the mount + /// slot so the pool's other conditional writes pass untouched. + std::expected write( + const String & key, + const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - channel->unblock(); - logger->setChannel(old_channel); - logger->setLevel(old_level); + if (expected_value && key.ends_with("/mount") && std::exchange(throw_before_next_overwrite, false)) + throw Poco::TimeoutException("injected renewal timeout before commit"); + return InMemoryBackend::write(key, bytes, expected_value, access); } - - void release() { channel->unblock(); } - -private: - LoggerPtr logger; - Poco::AutoPtr channel; - /// A real reference (shared=true), so the parked previous channel cannot die while ours is installed. - Poco::AutoPtr old_channel; - int old_level; }; class ScopedRenewalLogCapture @@ -312,8 +330,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) backend->throw_before_next_overwrite = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); const String output = capture.captured(); - EXPECT_EQ(countRenewalLogText(output, "CAS mount renewal"), 2u) << output; - EXPECT_EQ(countRenewalLogText(output, "entered retry"), 1u) << output; + EXPECT_EQ(countRenewalLogText(output, "CAS mount renewal"), 1u) << output; EXPECT_EQ(countRenewalLogText(output, "recovered"), 1u) << output; EXPECT_EQ(countRenewalLogText(output, "physical retry attempt"), 0u) << output; } @@ -333,51 +350,17 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) uint64_t boot_ms = 100; auto store = open_store(backend, boot_ms, "renewal-log-fenced"); ScopedRenewalLogCapture capture("information"); - boot_ms = 1071; + /// The lease was claimed at boot 100 with the 1000 ms TTL above, so it expires at 1100. The + /// fence admits only while the remaining time is strictly above the safety margin, and this + /// backend declares no attempt timeout, so the engine reserves nothing on top of that margin. + boot_ms = 1100 - renewalLogBudget().lease_safety_margin_ms; EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); const String output = capture.captured(); EXPECT_EQ(countRenewalLogText(output, "CAS mount renewal"), 1u) << output; EXPECT_EQ(countRenewalLogText(output, "fenced"), 1u) << output; - EXPECT_EQ(countRenewalLogText(output, "entered retry"), 0u) << output; } } -TEST(CASMountAudit, PhysicalRetryCannotBeDelayedByDebugLogging) -{ - auto backend = std::make_shared(); - uint64_t boot_ms = 100; - auto store = Pool::open(backend, PoolConfig{ - .pool_prefix = "renewal-debug-order", - .server_root_id = "test", - .mount_lease_ttl_ms = std::chrono::milliseconds(1000), - .cas_request_budget = renewalLogBudget(), - .boot_ms_fn = [&] { return boot_ms; }, - }); - - const uint64_t attempts_before - = ProfileEvents::global_counters[ProfileEvents::CASMountRenewalAttempts].load(); - const uint64_t retries_before - = ProfileEvents::global_counters[ProfileEvents::CASMountRenewalRetries].load(); - backend->armBlockedRetry(); - ScopedBlockingRenewalDebugLog blocked_log; - auto renewal = std::async(std::launch::async, [&] { store->renewWatermarkOnce(); }); - - const bool retry_reached_backend = backend->waitForSecondPut(); - EXPECT_TRUE(retry_reached_backend) - << "diagnostic logging after retry admission must not delay the backend request"; - EXPECT_EQ( - ProfileEvents::global_counters[ProfileEvents::CASMountRenewalAttempts].load(), - attempts_before + 2) - << "physical attempt visibility must precede completion of the in-flight retry"; - EXPECT_EQ( - ProfileEvents::global_counters[ProfileEvents::CASMountRenewalRetries].load(), - retries_before + 1); - - blocked_log.release(); - backend->releaseSecondPut(); - EXPECT_NO_THROW(renewal.get()); -} - TEST(CASServerRootId, ValidationAcceptsCleanPathsRejectsBad) { EXPECT_NO_THROW(validateServerRootId("replica-a")); @@ -447,11 +430,12 @@ TEST(CASServerRootClaim, OwnerStickyAndForeignFailsClosed) { auto b = std::make_shared(); Layout l("p"); - EXPECT_NO_THROW(claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation())); // fresh empty root → claim - EXPECT_NO_THROW(claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation())); // same uuid → ok + Ops ops(b); + EXPECT_NO_THROW(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation())); // fresh empty root → claim + EXPECT_NO_THROW(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation())); // same uuid → ok try { - claimOwnerOrThrow(*b, l, "r", UInt128(2), emptyCatalogObservation()); + claimOwnerOrThrow(ops.op, l, "r", UInt128(2), emptyCatalogObservation()); FAIL() << "expected a foreign owner to fail closed"; } catch (const DB::Exception & e) @@ -465,14 +449,15 @@ TEST(CASServerRootClaim, TombstonedSameOwnerFailsClosed) { auto b = std::make_shared(); Layout l("p"); - b->putIfAbsent(l.ownerKey("r"), encodeOwner(OwnerObject{ + Ops ops(b); + mustCommit(ops.op.create(l.ownerKey("r"), encodeOwner(OwnerObject{ .server_uuid = UInt128(1), .retired_at_ms = 1752537600000ULL, - })); + }), Retry::standard()), "tombstoned owner"); try { - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); FAIL() << "expected a tombstoned owner claim to fail closed"; } catch (const DB::Exception & e) @@ -487,20 +472,18 @@ TEST(CASServerRootEpoch, AllocatorIsMonotoneAndSurvivesMountConcept) { auto b = std::make_shared(); Layout l("r"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - const uint64_t e1 = allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); - const uint64_t e2 = allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + const uint64_t e1 = allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); + const uint64_t e2 = allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()); EXPECT_GE(e1, 1u); // 0 is a reserved sentinel EXPECT_GT(e2, e1); // strictly increasing /// Deleting the (separate) mount object must NOT reset the epoch. No mount has been written yet, - /// so deleteExact of it is a NotFound no-op that touches nothing -- exercised with a well-formed - /// placeholder token, not the absent HeadResult's empty one: InMemoryBackend refuses a malformed - /// token as a caller bug before it ever looks the key up, exactly like the production backend. - ASSERT_FALSE(b->head(l.mountKey("r")).exists); - const auto del = b->deleteExact(l.mountKey("r"), Token{"absent", TokenType::Emulated}); - EXPECT_EQ(del.kind, DeleteOutcome::Kind::NotFound); - EXPECT_GT(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), e2); + /// so the removal is a no-op that touches nothing. + ASSERT_FALSE(ops.op.head(l.mountKey("r"), Retry::standard()).has_value()); + EXPECT_EQ(ops.op.removeCurrent(l.mountKey("r"), Retry::standard()), Removal::Gone); + EXPECT_GT(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), e2); } /// Phase C (spec rev.4): an ABSENT epoch object over a PRESENT mount object means durable epoch @@ -510,20 +493,22 @@ TEST(CASMount, EpochRemintOverExistingMountRefuses) { auto b = std::make_shared(); Layout l("p"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), /*our_epoch=*/1, /*now_ms=*/1000, /*ttl_ms=*/30000).kind, + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/1, /*now_ms=*/1000, /*ttl_ms=*/30000).kind, MountClaimResult::Claimed); /// The epoch object is ABSENT (never created in this sequence) while the mount exists: - EXPECT_THROW(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); /// CORRUPTED_DATA + EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); /// CORRUPTED_DATA } TEST(CASMount, EpochRemintAuthoritativeAbsenceMints) { auto b = std::make_shared(); Layout l("p"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - EXPECT_EQ(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); /// fresh root: both control objects absent - EXPECT_EQ(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u); /// epoch present now: normal CAS bump, no probe + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); /// fresh root: both control objects absent + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u); /// epoch present now: normal conditional bump, no probe } /// The probe outcome gates the mint: anything short of authoritative KeyAbsent fails closed. @@ -532,15 +517,16 @@ TEST(CASMount, EpochRemintIndeterminateProbeFailsClosed) class IndeterminateProbeBackend final : public InMemoryBackend { public: - SentinelProbeResult probeSentinelRaw(const String &) override + SentinelProbeResult probeSentinelRaw(const String &, TransportAccess &) override { return {.outcome = ProbeOutcome::Indeterminate, .body = std::nullopt}; } }; auto b = std::make_shared(); Layout l("p"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - EXPECT_THROW(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); } /// Decommission over a TERMINAL (expired/fenced) mount with a lost epoch object proceeds and mints @@ -549,11 +535,12 @@ TEST(CASMount, DecommissionRemintOverTerminalMountMintsDistinctEpoch) { auto b = std::make_shared(); Layout l("p"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), /*our_epoch=*/3, /*now_ms=*/1000, /*ttl_ms=*/100).kind, + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/3, /*now_ms=*/1000, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); /// now_ms=5000: the ttl_ms=100 lease above is long expired -> terminal. - EXPECT_EQ(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::DecommissionRecovery, /*now_ms=*/5000, emptyCatalogObservation()), 4u); + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::DecommissionRecovery, /*now_ms=*/5000, emptyCatalogObservation()), 4u); } /// Decommission over a LIVE mount with a lost epoch refuses — the blind bypass would recreate the @@ -562,10 +549,11 @@ TEST(CASMount, DecommissionRemintOverLiveMountRefuses) { auto b = std::make_shared(); Layout l("p"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), /*our_epoch=*/1, /*now_ms=*/1000, /*ttl_ms=*/30000).kind, + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/1, /*now_ms=*/1000, /*ttl_ms=*/30000).kind, MountClaimResult::Claimed); - EXPECT_THROW(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::DecommissionRecovery, /*now_ms=*/2000, emptyCatalogObservation()), + EXPECT_THROW(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::DecommissionRecovery, /*now_ms=*/2000, emptyCatalogObservation()), DB::Exception); /// ABORTED: live member } @@ -577,18 +565,19 @@ TEST(CASMount, EpochBumpWithPresentEpochIssuesNoProbe) { public: int probes = 0; - SentinelProbeResult probeSentinelRaw(const String & k) override + SentinelProbeResult probeSentinelRaw(const String & k, TransportAccess & access) override { ++probes; - return InMemoryBackend::probeSentinelRaw(k); + return InMemoryBackend::probeSentinelRaw(k, access); } }; auto b = std::make_shared(); Layout l("p"); - claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()); - EXPECT_EQ(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); /// bootstrap: ONE probe (absent-epoch branch) + Ops ops(b); + claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); /// bootstrap: ONE probe (absent-epoch branch) const int probes_after_bootstrap = b->probes; - EXPECT_EQ(allocateWriterEpoch(*b, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u); /// epoch present: normal CAS bump... + EXPECT_EQ(allocateWriterEpoch(ops.op, l, "r", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u); /// epoch present: normal conditional bump... EXPECT_EQ(b->probes, probes_after_bootstrap) << "...must not probe the mount key"; } @@ -596,9 +585,10 @@ TEST(CASServerRootClaim, MissingOwnerOverNonEmptyRootIsCorrupted) { auto b = std::make_shared(); Layout l("p"); + Ops ops(b); /// Simulate existing data without an owner (identity lost): plant a key under roots//. - b->putIfAbsent(l.serverRootDataPrefix("r") + "some-data", "x"); - EXPECT_THROW(claimOwnerOrThrow(*b, l, "r", UInt128(1), emptyCatalogObservation()), DB::Exception); + mustCommit(ops.op.create(l.serverRootDataPrefix("r") + "some-data", "x", Retry::standard()), "root debris"); + EXPECT_THROW(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()), DB::Exception); } TEST(CASServerRootSafety, EveryCatalogLifecycleStateBlocksOwnerAndEpochRecreation) @@ -609,39 +599,39 @@ TEST(CASServerRootSafety, EveryCatalogLifecycleStateBlocksOwnerAndEpochRecreatio RefCatalog catalog = catalogOwning("root/x/table", state); const ObserveRefCatalog observe = [catalog] { return catalog; }; - InMemoryBackend owner_backend; - EXPECT_THROW(claimOwnerOrThrow(owner_backend, layout, "root/x", UInt128{1}, observe), DB::Exception); - EXPECT_FALSE(owner_backend.head(layout.ownerKey("root/x")).exists); + Ops owner_ops(std::make_shared()); + EXPECT_THROW(claimOwnerOrThrow(owner_ops.op, layout, "root/x", UInt128{1}, observe), DB::Exception); + EXPECT_FALSE(owner_ops.op.head(layout.ownerKey("root/x"), Retry::standard()).has_value()); - InMemoryBackend epoch_backend; + Ops epoch_ops(std::make_shared()); EXPECT_THROW(allocateWriterEpoch( - epoch_backend, layout, "root/x", EpochMintPolicy::NormalMount, 0, observe), DB::Exception); - EXPECT_FALSE(epoch_backend.head(layout.epochKey("root/x")).exists); + epoch_ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, observe), DB::Exception); + EXPECT_FALSE(epoch_ops.op.head(layout.epochKey("root/x"), Retry::standard()).has_value()); } } TEST(CASServerRootSafety, OwnershipUsesAPathComponentBoundary) { - InMemoryBackend backend; + Ops ops(std::make_shared()); const Layout layout("p"); EXPECT_TRUE(serverRootSubtreeEmpty( - backend, layout, "root/x", catalogOwning("root/xy/table", NsState::Live))); + ops.op, layout, "root/x", catalogOwning("root/xy/table", NsState::Live))); EXPECT_FALSE(serverRootSubtreeEmpty( - backend, layout, "root/x", catalogOwning("root/x/table", NsState::Live))); + ops.op, layout, "root/x", catalogOwning("root/x/table", NsState::Live))); } TEST(CASServerRootSafety, OpaqueStreamAndStateDebrisAloneDoesNotBlockRecreation) { - InMemoryBackend backend; + Ops ops(std::make_shared()); const Layout layout("p"); const NamespaceLifeId dead = NamespaceLifeId::fromCatalogEntry(RootNamespace{"unowned"}, UInt128{99}); - ASSERT_EQ(backend.putIfAbsent(layout.refLogKey(dead, RefTxnId{1, 1}), "debris").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(layout.refCkptKey(dead), "debris").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(layout.namespaceFileKey(dead, "f"), "debris").outcome, PutOutcome::Done); + mustCommit(ops.op.create(layout.refLogKey(dead, RefTxnId{1, 1}), "debris", Retry::standard()), "ref-log debris"); + mustCommit(ops.op.create(layout.refCkptKey(dead), "debris", Retry::standard()), "ckpt debris"); + mustCommit(ops.op.create(layout.namespaceFileKey(dead, "f"), "debris", Retry::standard()), "ns-file debris"); - EXPECT_NO_THROW(claimOwnerOrThrow(backend, layout, "root/x", UInt128{1}, emptyCatalogObservation())); + EXPECT_NO_THROW(claimOwnerOrThrow(ops.op, layout, "root/x", UInt128{1}, emptyCatalogObservation())); EXPECT_EQ(allocateWriterEpoch( - backend, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); + ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 1u); } TEST(CASServerRootSafety, ManifestAndLooseRootDebrisStillBlockRecreation) @@ -651,49 +641,56 @@ TEST(CASServerRootSafety, ManifestAndLooseRootDebrisStillBlockRecreation) layout.casManifestsServerPrefix("root/x") + "table/debris", layout.serverRootDataPrefix("root/x") + "loose"}) { - InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent(key, "x").outcome, PutOutcome::Done); + Ops ops(std::make_shared()); + mustCommit(ops.op.create(key, "x", Retry::standard()), "blocking debris"); EXPECT_THROW(claimOwnerOrThrow( - backend, layout, "root/x", UInt128{1}, emptyCatalogObservation()), DB::Exception); + ops.op, layout, "root/x", UInt128{1}, emptyCatalogObservation()), DB::Exception); EXPECT_THROW(allocateWriterEpoch( - backend, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); } } TEST(CASServerRootSafety, UnreadableCatalogNeverFallsBackToPhysicalGuesses) { - InMemoryBackend backend; + Ops ops(std::make_shared()); const Layout layout("p"); const ObserveRefCatalog unreadable = []() -> RefCatalog { throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "injected unreadable catalog"); }; - EXPECT_THROW(claimOwnerOrThrow(backend, layout, "root/x", UInt128{1}, unreadable), DB::Exception); + EXPECT_THROW(claimOwnerOrThrow(ops.op, layout, "root/x", UInt128{1}, unreadable), DB::Exception); EXPECT_THROW(allocateWriterEpoch( - backend, layout, "root/x", EpochMintPolicy::NormalMount, 0, unreadable), DB::Exception); - EXPECT_FALSE(backend.head(layout.ownerKey("root/x")).exists); - EXPECT_FALSE(backend.head(layout.epochKey("root/x")).exists); + ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, unreadable), DB::Exception); + EXPECT_FALSE(ops.op.head(layout.ownerKey("root/x"), Retry::standard()).has_value()); + EXPECT_FALSE(ops.op.head(layout.epochKey("root/x"), Retry::standard()).has_value()); } TEST(CASServerRootSafety, OwnerConflictRecomputesTheWholeEmptinessBundle) { - OwnerConflictRevealsManifestBackend backend; + auto backend = std::make_shared(); + Ops ops(backend); const Layout layout("p"); - EXPECT_THROW(claimOwnerOrThrow( - backend, layout, "root/x", UInt128{1}, emptyCatalogObservation()), DB::Exception); - EXPECT_TRUE(backend.fired); - EXPECT_FALSE(backend.head(layout.ownerKey("root/x")).exists); + /// The message, not just the code: without the post-conflict recompute the claim still throws + /// `CORRUPTED_DATA`, from the vanished-anchor arm below it, so a bare code assertion would hold + /// with the behaviour this test is named for deleted. + DB::Cas::tests::expectThrowsCodeWithMessage( + DB::ErrorCodes::CORRUPTED_DATA, + "newly visible owned work blocks recreation", + [&] { claimOwnerOrThrow(ops.op, layout, "root/x", UInt128{1}, emptyCatalogObservation()); }); + EXPECT_TRUE(backend->fired); + EXPECT_FALSE(ops.op.head(layout.ownerKey("root/x"), Retry::standard()).has_value()); } TEST(CASServerRootSafety, EpochConflictRecomputesTheWholeEmptinessBundle) { - EpochConflictRevealsManifestBackend backend; + auto backend = std::make_shared(); + Ops ops(backend); const Layout layout("p"); EXPECT_THROW(allocateWriterEpoch( - backend, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); - EXPECT_TRUE(backend.fired); - ASSERT_TRUE(backend.winner_installed); - const auto epoch = backend.get(layout.epochKey("root/x")); + ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + EXPECT_TRUE(backend->fired); + ASSERT_TRUE(backend->winner_installed); + const auto epoch = ops.op.read(layout.epochKey("root/x"), Retry::standard()); ASSERT_TRUE(epoch.has_value()); EXPECT_EQ(decodeServerEpoch(epoch->bytes).next_writer_epoch, 2u) << "the rejected allocator must not consume an epoch from the conflict winner"; @@ -704,14 +701,17 @@ TEST(CASMountLease, AbsentClaimThenRenewBumpsSeq) auto b = std::make_shared(); Layout l("p"); uint64_t now = 1000; - auto r = claimMount(*b, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100); + uint64_t boot = 0; + Ops ops(b, &boot); + auto r = claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100); EXPECT_EQ(r.kind, MountClaimResult::Claimed); - MountLeaseKeeper k(b, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, - [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), + [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), + [&] { return boot; }); k.start(); - EXPECT_EQ(decodeMountLease(b->get(l.mountKey("r"))->bytes).seq, 1u); + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).seq, 1u); renewKeeperOrThrow(k); - EXPECT_EQ(decodeMountLease(b->get(l.mountKey("r"))->bytes).seq, 2u); + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).seq, 2u); } TEST(CASMountLease, HolderBodiesMintFreshAttemptIdsAndFenceCopiesIt) @@ -719,47 +719,51 @@ TEST(CASMountLease, HolderBodiesMintFreshAttemptIdsAndFenceCopiesIt) auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; - ASSERT_EQ(claimMount(*backend, layout, "r", UInt128{1}, 7, now, 100).kind, MountClaimResult::Claimed); + uint64_t boot = 0; + Ops ops(backend, &boot); + ASSERT_EQ(claimMount(ops.op, layout, "r", UInt128{1}, 7, now, 100).kind, MountClaimResult::Claimed); const String key = layout.mountKey("r"); - const MountLease claimed = decodeMountLease(backend->get(key)->bytes); + const MountLease claimed = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); - MountLeaseKeeper keeper(backend, layout, "r", UInt128{1}, 7, std::chrono::milliseconds(100), [&] { return now; }, - [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0)); + MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, "r", UInt128{1}, 7, std::chrono::milliseconds(100), + [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), + [&] { return boot; }); keeper.start(); renewKeeperOrThrow(keeper); - const MountLease renewed = decodeMountLease(backend->get(key)->bytes); + const MountLease renewed = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); EXPECT_NE(claimed.write_attempt_id, UInt128{}); EXPECT_NE(renewed.write_attempt_id, UInt128{}); EXPECT_NE(claimed.write_attempt_id, renewed.write_attempt_id); - auto observed = backend->get(key); + auto observed = ops.op.read(key, Retry::standard()); ASSERT_TRUE(observed.has_value()); MountLease fenced = decodeMountLease(observed->bytes); fenced.gc_fenced = true; ++fenced.seq; - ASSERT_EQ(backend->putOverwrite(key, encodeMountLease(fenced), observed->token).outcome, PutOutcome::Done); - EXPECT_EQ(decodeMountLease(backend->get(key)->bytes).write_attempt_id, renewed.write_attempt_id); + mustCommit(ops.op.replace(key, encodeMountLease(fenced), observed->incarnation, Retry::standard()), "fence-out"); + EXPECT_EQ(decodeMountLease(ops.op.read(key, Retry::standard())->bytes).write_attempt_id, renewed.write_attempt_id); } TEST(CASMountLease, ReclaimAndSuccessorBodiesMintNewAttemptIds) { auto backend = std::make_shared(); Layout layout("p"); + Ops ops(backend); const String key = layout.mountKey("r"); - ASSERT_EQ(claimMount(*backend, layout, "r", UInt128{1}, 7, 1000, 100).kind, MountClaimResult::Claimed); - const MountLease first = decodeMountLease(backend->get(key)->bytes); + ASSERT_EQ(claimMount(ops.op, layout, "r", UInt128{1}, 7, 1000, 100).kind, MountClaimResult::Claimed); + const MountLease first = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); - auto observed = backend->get(key); + auto observed = ops.op.read(key, Retry::standard()); ASSERT_TRUE(observed.has_value()); MountLease fenced = decodeMountLease(observed->bytes); fenced.gc_fenced = true; ++fenced.seq; - ASSERT_EQ(backend->putOverwrite(key, encodeMountLease(fenced), observed->token).outcome, PutOutcome::Done); - const MountLease fence = decodeMountLease(backend->get(key)->bytes); + mustCommit(ops.op.replace(key, encodeMountLease(fenced), observed->incarnation, Retry::standard()), "fence-out"); + const MountLease fence = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); EXPECT_EQ(fence.write_attempt_id, first.write_attempt_id); - ASSERT_EQ(claimMount(*backend, layout, "r", UInt128{1}, 8, 2000, 100).kind, MountClaimResult::Claimed); - const MountLease successor = decodeMountLease(backend->get(key)->bytes); + ASSERT_EQ(claimMount(ops.op, layout, "r", UInt128{1}, 8, 2000, 100).kind, MountClaimResult::Claimed); + const MountLease successor = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); EXPECT_NE(successor.write_attempt_id, first.write_attempt_id); EXPECT_NE(successor.write_attempt_id, UInt128{}); } @@ -774,17 +778,20 @@ TEST(CASMountLease, VanishedBackingStoreStopsRenewalWithoutLogicalError) auto b = std::make_shared(); Layout l("p"); uint64_t now = 1000; - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); - MountLeaseKeeper k(b, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, - [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0)); + uint64_t boot = 0; + Ops ops(b, &boot); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); + MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), + [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), + [&] { return boot; }); k.start(); const String mount_key = l.mountKey("r"); const auto lost_before = ProfileEvents::global_counters[ProfileEvents::CASMountLeaseLost].load(); /// NOLINT(clang-analyzer-deadcode.DeadStores) /// Simulate `rm -rf` of the backing store: the mount slot object is gone, but the keeper still - /// holds a (now stale) token for it. - ASSERT_EQ(b->deleteExact(mount_key, b->head(mount_key).token).kind, DeleteOutcome::Kind::Deleted); + /// names a (now stale) incarnation as its precondition. + ASSERT_EQ(ops.op.removeCurrent(mount_key, Retry::standard()), Removal::Removed); try { @@ -815,38 +822,40 @@ TEST(CASMountLease, TerminateAfterVanishedBackingStoreIsNoOpRelease) auto b = std::make_shared(); Layout l("p"); uint64_t now = 1000; - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); - MountLeaseKeeper k(b, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, - [] { return uint64_t{0}; }); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); + MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), + [&] { return now; }, [] { return uint64_t{0}; }); k.start(); const String mount_key = l.mountKey("r"); const auto lost_before = ProfileEvents::global_counters[ProfileEvents::CASMountLeaseLost].load(); /// Simulate `rm -rf` of the backing store: the mount slot object is gone before we ever attempt - /// a renewal, so `terminate()`'s token-guarded farewell PUT is the first thing to observe it. - ASSERT_EQ(b->deleteExact(mount_key, b->head(mount_key).token).kind, DeleteOutcome::Kind::Deleted); + /// a renewal, so the farewell's guarded write is the first thing to observe it. + ASSERT_EQ(ops.op.removeCurrent(mount_key, Retry::standard()), Removal::Removed); EXPECT_NO_THROW(k.release()) << "clean release against a vanished store must be a no-op, not a LOGICAL_ERROR abort"; EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountLeaseLost].load(), lost_before); } -/// rev.6: a bare `claimMount` (no `proven_dead_token`) NEVER reclaims a same-uuid, different-epoch +/// rev.6: a bare `claimMount` (no `proven_dead_incarnation`) NEVER reclaims a same-uuid, different-epoch /// lease off a wall-clock-looking-expired stamp — only `claimMountAwaitingExpiry`'s observation loop /// can turn that into a reclaim. Renamed from `...ExpiredReclaims` to describe the corrected behavior. TEST(CASMountLease, SameUuidLiveFailsForeignFailsExpiredStillLiveDoubleStart) { auto b = std::make_shared(); Layout l("p"); - claimMount(*b, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100); // A live until 1100 + Ops ops(b); + claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100); // A live until 1100 // same uuid, lease still live → double-start guard: - EXPECT_EQ(claimMount(*b, l, "r", UInt128(1), 8, 1050, 100).kind, MountClaimResult::LiveDoubleStart); + EXPECT_EQ(claimMount(ops.op, l, "r", UInt128(1), 8, 1050, 100).kind, MountClaimResult::LiveDoubleStart); // foreign uuid, even after expiry → fail closed: - EXPECT_EQ(claimMount(*b, l, "r", UInt128(2), 1, 1200, 100).kind, MountClaimResult::ForeignOwner); + EXPECT_EQ(claimMount(ops.op, l, "r", UInt128(2), 1, 1200, 100).kind, MountClaimResult::ForeignOwner); // same uuid, even after the stamp LOOKS expired on our wall clock → still LiveDoubleStart: no - // proven_dead_token was supplied, so there is no certificate of death to reclaim on. - EXPECT_EQ(claimMount(*b, l, "r", UInt128(1), 9, 1200, 100).kind, MountClaimResult::LiveDoubleStart); + // proven_dead_incarnation was supplied, so there is no certificate of death to reclaim on. + EXPECT_EQ(claimMount(ops.op, l, "r", UInt128(1), 9, 1200, 100).kind, MountClaimResult::LiveDoubleStart); } TEST(CASMountMessage, DoubleStartTextHasIdentityAndRemediation) @@ -888,8 +897,9 @@ TEST(CASMountAwaitExpiry, PastExpiryStillPaysTheFullObservationThreshold) { auto b = std::make_shared(); Layout l("p"); + Ops ops(b); /// A prior incarnation (uuid=1, epoch=7) claimed a lease live until 1100. - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); uint64_t wall = 1200; // already past 1100 on wall clock — irrelevant to the decision uint64_t mono = 0; @@ -899,18 +909,19 @@ TEST(CASMountAwaitExpiry, PastExpiryStillPaysTheFullObservationThreshold) auto sleep_fn = [&](uint64_t ms) { wall += ms; mono += ms; ++sleeps; }; const auto r = claimMountAwaitingExpiry( - *b, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 25, sleep_fn); + ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 25, sleep_fn); EXPECT_EQ(r.kind, MountClaimResult::Claimed); EXPECT_GT(sleeps, 0); // NOT instant — no wall-clock trust EXPECT_GE(mono, 100 + 100 / 20 + 25); // full observation threshold paid - EXPECT_EQ(decodeMountLease(b->get(l.mountKey("r"))->bytes).writer_epoch, 8u); // reclaimed as us + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 8u); // reclaimed as us } TEST(CASMountAwaitExpiry, FutureExpiryReclaimsAfterClockAdvances) { auto b = std::make_shared(); Layout l("p"); - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); uint64_t wall = 1000; // lease looks live until 1100, holder does NOT renew uint64_t mono = 0; @@ -919,9 +930,9 @@ TEST(CASMountAwaitExpiry, FutureExpiryReclaimsAfterClockAdvances) auto sleep_fn = [&](uint64_t ms) { wall += ms; mono += ms; }; const auto r = claimMountAwaitingExpiry( - *b, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 50, sleep_fn); + ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 50, sleep_fn); EXPECT_EQ(r.kind, MountClaimResult::Claimed); - const auto body = decodeMountLease(b->get(l.mountKey("r"))->bytes); + const auto body = decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes); EXPECT_EQ(body.writer_epoch, 8u); EXPECT_EQ(body.seq, 2u); // reclaim continues seq (prev 1 + 1) } @@ -932,64 +943,53 @@ TEST(CASMountAwaitExpiry, LiveRenewingTwinTimesOutAsDoubleStart) { auto b = std::make_shared(); Layout l("p"); - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); uint64_t wall = 1000; uint64_t mono = 0; auto now_fn = [&] { return wall; }; auto mono_fn = [&] { return mono; }; /// Each poll: both clocks advance AND the live holder (uuid=1, epoch=7) renews its own lease — - /// the observed write-token changes on EVERY poll, forcing a restart every time. + /// the observed incarnation changes on EVERY poll, forcing a restart every time. auto sleep_fn = [&](uint64_t ms) { wall += ms; mono += ms; - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, wall, 100).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, wall, 100).kind, MountClaimResult::Claimed); }; const auto r = claimMountAwaitingExpiry( - *b, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); + ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); EXPECT_EQ(r.kind, MountClaimResult::LiveDoubleStart); - EXPECT_EQ(decodeMountLease(b->get(l.mountKey("r"))->bytes).writer_epoch, 7u); // still the holder's + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 7u); // still the holder's } namespace { -/// fix-round F5 harness: makes the mount key vanish to EVERY `get()`, unconditionally, while the real -/// underlying object stays put -- forcing `claimMount`'s own internal GET to take the absent-slot race -/// branch every call (its `putIfAbsent` then fails against the real, still-present object, returning -/// `LiveDoubleStart` with no token -- fix-round F8 leaves `.token` unset on exactly this branch, since -/// no re-read was done). That in turn forces `claimMountAwaitingExpiry`'s F8 fallback re-GET, which -/// ALSO sees the slot as vanished -- deterministically reproducing "the slot vanished between -/// claimMount's own GET and ours" on EVERY loop iteration, not just a lucky one-shot race. +/// fix-round F5 harness: makes the mount key vanish to EVERY read, unconditionally, while the real +/// underlying object stays put -- forcing `claimMount`'s own read to take the absent-slot race +/// branch every call (its create then conflicts against the real, still-present object, returning +/// `LiveDoubleStart` with no incarnation -- that branch deliberately leaves `.incarnation` unset, +/// since no re-read was done). That in turn forces `claimMountAwaitingExpiry`'s fallback re-read, +/// which ALSO sees the slot as vanished -- deterministically reproducing "the slot vanished between +/// claimMount's own read and ours" on EVERY loop iteration, not just a lucky one-shot race. class AlwaysVanishesBackend final : public DB::Cas::Backend { public: explicit AlwaysVanishesBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} String watched_key; - std::optional get(const String & k, DB::Cas::Range r) override - { - if (k == watched_key) - return std::nullopt; - return inner->get(k, r); - } std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & m) override { return inner->putIfAbsent(k, b, m); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & m) override { return inner->putOverwrite(k, b, e, m); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, b, e, m); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The transport primitives forward to `inner`; the legacy overrides above are what this - /// double injects through. Declared because `Backend` declares them pure. - std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + /// The fault is on the read primitive, which is the only way anything now reaches the store. + std::optional read(const String & key, TransportAccess & access) override + { + if (key == watched_key) + return std::nullopt; + return inner->read(key, access); + } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } @@ -1016,12 +1016,14 @@ TEST(CASMountAwaitExpiry, PersistentSlotVanishPacesAndBoundsRestartsInsteadOfSpi { auto inner = std::make_shared(); Layout l("p"); - /// A real slot exists underneath (uuid 1, epoch 7) so `claimMount`'s absent-slot `putIfAbsent` - /// genuinely fails every time (never accidentally re-mints). - ASSERT_EQ(claimMount(*inner, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); + Ops inner_ops(inner); + /// A real slot exists underneath (uuid 1, epoch 7) so `claimMount`'s absent-slot create + /// genuinely conflicts every time (never accidentally re-mints). + ASSERT_EQ(claimMount(inner_ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); auto vanishing = std::make_shared(inner); vanishing->watched_key = l.mountKey("r"); + Ops ops(vanishing); uint64_t wall = 1000; uint64_t mono = 0; @@ -1031,20 +1033,21 @@ TEST(CASMountAwaitExpiry, PersistentSlotVanishPacesAndBoundsRestartsInsteadOfSpi auto sleep_fn = [&](uint64_t ms) { wall += ms; mono += ms; ++sleeps; }; const auto r = claimMountAwaitingExpiry( - *vanishing, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); + ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); EXPECT_EQ(r.kind, MountClaimResult::LiveDoubleStart) << "must terminate (bounded), not loop forever"; EXPECT_GT(sleeps, 0) << "a persistently vanishing slot must still pace via sleep_fn, not busy-spin"; - /// The real epoch-7 lease is untouched -- every `putIfAbsent` attempt against it genuinely fails + /// The real epoch-7 lease is untouched -- every create attempt against it genuinely conflicts /// (the object is still there), so it is never accidentally re-minted over. - EXPECT_EQ(decodeMountLease(inner->get(l.mountKey("r"))->bytes).writer_epoch, 7u); + EXPECT_EQ(decodeMountLease(inner_ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 7u); } TEST(CASMountAwaitExpiry, ForeignUuidFailsClosedImmediately) { auto b = std::make_shared(); Layout l("p"); + Ops ops(b); /// A foreign server (uuid=2) holds the mount. - ASSERT_EQ(claimMount(*b, l, "r", UInt128(2), 1, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(2), 1, /*now*/ 1000, /*ttl*/ 100).kind, MountClaimResult::Claimed); uint64_t now = 1000; int sleeps = 0; @@ -1053,7 +1056,7 @@ TEST(CASMountAwaitExpiry, ForeignUuidFailsClosedImmediately) auto sleep_fn = [&](uint64_t ms) { now += ms; ++sleeps; }; const auto r = claimMountAwaitingExpiry( - *b, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 25, sleep_fn); + ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 25, sleep_fn); EXPECT_EQ(r.kind, MountClaimResult::ForeignOwner); EXPECT_EQ(sleeps, 0); // never waits across UUIDs } @@ -1067,7 +1070,8 @@ TEST(CASMountAwaitExpiry, SkewedFarFutureExpiryHasNoEffectOnObservationThreshold { auto b = std::make_shared(); Layout l("p"); - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100000).kind, MountClaimResult::Claimed); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 100000).kind, MountClaimResult::Claimed); uint64_t wall = 1000; uint64_t mono = 0; @@ -1076,10 +1080,10 @@ TEST(CASMountAwaitExpiry, SkewedFarFutureExpiryHasNoEffectOnObservationThreshold auto sleep_fn = [&](uint64_t ms) { wall += ms; mono += ms; }; const auto r = claimMountAwaitingExpiry( - *b, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); + ops.op, l, "r", UInt128(1), /*our_epoch*/ 8, now_fn, mono_fn, /*ttl*/ 100, /*poll*/ 20, sleep_fn); EXPECT_EQ(r.kind, MountClaimResult::Claimed); EXPECT_LE(mono, 100u + 100u / 20 + 20u + 20u); // bounded by OUR threshold, not the predecessor's stamp - EXPECT_EQ(decodeMountLease(b->get(l.mountKey("r"))->bytes).writer_epoch, 8u); // reclaimed + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 8u); // reclaimed } TEST(CASMountLease, KeeperStartAdoptsOurOwnClaimNotDoubleStart) @@ -1087,12 +1091,13 @@ TEST(CASMountLease, KeeperStartAdoptsOurOwnClaimNotDoubleStart) auto b = std::make_shared(); Layout l("p"); uint64_t now = 1000; + Ops ops(b); // The normal flow: claimMount writes the live mount under (uuid=1, epoch=7), THEN keeper.start(). - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); - MountLeaseKeeper k(b, l, "r", UInt128(1), /*epoch*/ 7, std::chrono::milliseconds(100), [&] { return now; }, - [] { return uint64_t{0}; }); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); + MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), /*epoch*/ 7, std::chrono::milliseconds(100), + [&] { return now; }, [] { return uint64_t{0}; }); EXPECT_NO_THROW(k.start()); // adopts our own live (uuid=1,epoch=7) mount — NOT a double-start - EXPECT_EQ(decodeMountLease(b->get(l.mountKey("r"))->bytes).writer_epoch, 7u); + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 7u); } TEST(CASMountFence, SupersededWriterRefusedNoS3Read) @@ -1154,7 +1159,8 @@ TEST(CASMountStartup, FreshWritablePoolBootstrapsAnExplicitEmptyCatalog) .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r", .skip_access_check = true}); - const auto catalog = backend->get(layout.refCatalogKey()); + Ops ops(backend); + const auto catalog = ops.op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog.has_value()); EXPECT_TRUE(decodeRefCatalog(catalog->bytes).entries.empty()); } @@ -1169,19 +1175,17 @@ TEST(CASMountStartup, ExistingPoolWithoutCatalogFailsBeforeSlotMutation) .skip_access_check = true}); } + Ops ops(backend); /// Old raw fixtures did not persist an empty catalog. Make this an explicit existing-pool /// fixture before removing the mandatory object whose loss the mount must reject. - if (!backend->head(layout.refCatalogKey()).exists) - ASSERT_EQ(backend->putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{})).outcome, - PutOutcome::Done); - const HeadResult catalog_head = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(catalog_head.exists); - ASSERT_EQ(backend->deleteExact(layout.refCatalogKey(), catalog_head.token).kind, - DeleteOutcome::Kind::Deleted); - - const auto owner_before = backend->get(layout.ownerKey("r")); - const auto epoch_before = backend->get(layout.epochKey("r")); - const auto mount_before = backend->get(layout.mountKey("r")); + if (!ops.op.head(layout.refCatalogKey(), Retry::standard())) + mustCommit(ops.op.create(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{}), Retry::standard()), + "empty catalog"); + ASSERT_EQ(ops.op.removeCurrent(layout.refCatalogKey(), Retry::standard()), Removal::Removed); + + const auto owner_before = ops.op.read(layout.ownerKey("r"), Retry::standard()); + const auto epoch_before = ops.op.read(layout.epochKey("r"), Retry::standard()); + const auto mount_before = ops.op.read(layout.mountKey("r"), Retry::standard()); ASSERT_TRUE(owner_before.has_value()); ASSERT_TRUE(epoch_before.has_value()); ASSERT_TRUE(mount_before.has_value()); @@ -1190,18 +1194,18 @@ TEST(CASMountStartup, ExistingPoolWithoutCatalogFailsBeforeSlotMutation) .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r", .skip_access_check = true}), DB::Exception); - const auto owner_after = backend->get(layout.ownerKey("r")); - const auto epoch_after = backend->get(layout.epochKey("r")); - const auto mount_after = backend->get(layout.mountKey("r")); + const auto owner_after = ops.op.read(layout.ownerKey("r"), Retry::standard()); + const auto epoch_after = ops.op.read(layout.epochKey("r"), Retry::standard()); + const auto mount_after = ops.op.read(layout.mountKey("r"), Retry::standard()); ASSERT_TRUE(owner_after.has_value()); ASSERT_TRUE(epoch_after.has_value()); ASSERT_TRUE(mount_after.has_value()); EXPECT_EQ(owner_after->bytes, owner_before->bytes); - EXPECT_EQ(owner_after->token, owner_before->token); + EXPECT_EQ(owner_after->incarnation, owner_before->incarnation); EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); - EXPECT_EQ(epoch_after->token, epoch_before->token); + EXPECT_EQ(epoch_after->incarnation, epoch_before->incarnation); EXPECT_EQ(mount_after->bytes, mount_before->bytes); - EXPECT_EQ(mount_after->token, mount_before->token); + EXPECT_EQ(mount_after->incarnation, mount_before->incarnation); } TEST(CASMountReadOnly, ForeignOwnedPoolOpensWithoutMutation) @@ -1213,10 +1217,11 @@ TEST(CASMountReadOnly, ForeignOwnedPoolOpensWithoutMutation) auto a = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r"}); + Ops ops(b); /// Capture the control objects BEFORE the read-only open so we can prove it mutated nothing. - const auto owner_before = b->get(l.ownerKey("r")); - const auto mount_before = b->get(l.mountKey("r")); - const auto epoch_before = b->get(l.epochKey("r")); + const auto owner_before = ops.op.read(l.ownerKey("r"), Retry::standard()); + const auto mount_before = ops.op.read(l.mountKey("r"), Retry::standard()); + const auto epoch_before = ops.op.read(l.epochKey("r"), Retry::standard()); ASSERT_TRUE(owner_before.has_value()); ASSERT_TRUE(mount_before.has_value()); ASSERT_TRUE(epoch_before.has_value()); @@ -1233,9 +1238,9 @@ TEST(CASMountReadOnly, ForeignOwnedPoolOpensWithoutMutation) /// And it mutated nothing: owner still decodes to A's uuid, the mount body is still A's, and the /// raw bytes of owner/epoch/mount are byte-for-byte unchanged (no second owner, no re-claim). - const auto owner_after = b->get(l.ownerKey("r")); - const auto mount_after = b->get(l.mountKey("r")); - const auto epoch_after = b->get(l.epochKey("r")); + const auto owner_after = ops.op.read(l.ownerKey("r"), Retry::standard()); + const auto mount_after = ops.op.read(l.mountKey("r"), Retry::standard()); + const auto epoch_after = ops.op.read(l.epochKey("r"), Retry::standard()); ASSERT_TRUE(owner_after.has_value()); ASSERT_TRUE(mount_after.has_value()); ASSERT_TRUE(epoch_after.has_value()); @@ -1290,16 +1295,18 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) ASSERT_NE(a, nullptr); const uint64_t e1 = a->writerEpoch(); const String mount_key = a->layout().mountKey("r"); - const auto stale_mount = b->get(mount_key); + Ops ops(b); + const auto stale_mount = ops.op.read(mount_key, Retry::standard()); ASSERT_TRUE(stale_mount.has_value()); /// Preserve A's live lease as if its process disappeared without running C++ teardown. Destroying /// the real Pool first keeps the parent process valid; replaying the saved body recreates the exact /// durable stale-lease state that a crashed process would leave behind. a.reset(); - const auto farewell = b->get(mount_key); + const auto farewell = ops.op.read(mount_key, Retry::standard()); ASSERT_TRUE(farewell.has_value()); - ASSERT_EQ(b->putOverwrite(mount_key, stale_mount->bytes, farewell->token).outcome, PutOutcome::Done); + mustCommit(ops.op.replace(mount_key, stale_mount->bytes, farewell->incarnation, Retry::standard()), + "replayed stale lease"); /// A restart of the SAME server (same uuid) must NOT abort: it waits out the stale lease (<= ~300ms) /// and reclaims the mount, coming up with a strictly higher durable writer_epoch. The replayed live @@ -1344,7 +1351,8 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) .wait_sleep_fn = [&overlap_fake_boot](uint64_t ms) { overlap_fake_boot += ms; }}); ASSERT_NE(replacement, nullptr); - const auto reclaimer_slot_before = overlap_backend->get(overlap_mount_key); + Ops overlap_ops(overlap_backend); + const auto reclaimer_slot_before = overlap_ops.op.read(overlap_mount_key, Retry::standard()); ASSERT_TRUE(reclaimer_slot_before.has_value()); const uint64_t overlap_violations_before = ProfileEvents::global_counters[ProfileEvents::CASMountExclusivityViolation].load(); @@ -1353,7 +1361,7 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountExclusivityViolation].load(), overlap_violations_before + 1); - const auto reclaimer_slot_after = overlap_backend->get(overlap_mount_key); + const auto reclaimer_slot_after = overlap_ops.op.read(overlap_mount_key, Retry::standard()); ASSERT_TRUE(reclaimer_slot_after.has_value()); EXPECT_EQ(reclaimer_slot_after->bytes, reclaimer_slot_before->bytes) << "the deposed Pool's release must not retire the reclaimer's lease"; @@ -1400,10 +1408,10 @@ constexpr uint64_t kNowMs = 1'000'000; /// of any lease's stamped `expires_at_ms`. constexpr uint64_t kStableThresholdMs = 10'000; -/// Seed one mount body under mountKey(srid) via the on-storage codec (`encodeMountLease` + -/// `putIfAbsent`) — the same interface the keeper writes through. +/// Seed one mount body under mountKey(srid) via the on-storage codec — the same interface the keeper +/// writes through. MountLease seedMount( - Backend & b, const Layout & l, const String & srid, + CasOperation & op, const Layout & l, const String & srid, uint64_t expires_at_ms, bool gc_fenced, uint64_t min_active_build_sequence, uint64_t seq = 1) { MountLease m; @@ -1417,22 +1425,22 @@ MountLease seedMount( m.min_active_build_sequence = min_active_build_sequence; m.gc_fenced = gc_fenced; m.write_attempt_id = UInt128{1}; - b.putIfAbsent(l.mountKey(srid), encodeMountLease(m)); + mustCommit(op.create(l.mountKey(srid), encodeMountLease(m), Retry::standard()), "seeded mount " + srid); return m; } -/// Simulate a keeper's real renewal between two `computeHeartbeatFloor` calls: a token-guarded -/// overwrite that bumps `seq` (and so mints a fresh backend token), leaving everything else as-is. -/// Models the one thing the observation-based fence cares about: the write token changed, so any -/// in-progress observation of the OLD token must restart. -void renewMount(Backend & b, const Layout & l, const String & srid) +/// Simulate a keeper's real renewal between two `computeHeartbeatFloor` calls: a guarded write that +/// bumps `seq` (and so mints a fresh incarnation), leaving everything else as-is. Models the one +/// thing the observation-based fence cares about: the incarnation changed, so any in-progress +/// observation of the OLD one must restart. +void renewMount(CasOperation & op, const Layout & l, const String & srid) { - const auto got = b.get(l.mountKey(srid)); + const auto got = op.read(l.mountKey(srid), Retry::standard()); ASSERT_TRUE(got.has_value()); MountLease m = decodeMountLease(got->bytes); m.seq += 1; - const PutResult res = b.putOverwrite(l.mountKey(srid), encodeMountLease(m), got->token); - ASSERT_EQ(res.outcome, PutOutcome::Done); + mustCommit(op.replace(l.mountKey(srid), encodeMountLease(m), got->incarnation, Retry::standard()), + "renewed mount " + srid); } } @@ -1441,12 +1449,13 @@ TEST(CASHeartbeatFloor, FirstSightNeverFencesEvenIfStampLooksExpired) auto b = std::make_shared(); Layout l("p"); + Ops ops(b); /// A stamp that would have read as long-expired under the old skew-margin comparison — under /// rev.6 observation the stamp is never even consulted for the fence decision. - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(ops.op, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; - const HeartbeatFloor floor = computeHeartbeatFloor(*b, l, /*now_ms*/ kNowMs, /*mono_now_ms*/ 0, + const HeartbeatFloor floor = computeHeartbeatFloor(ops.op, l, /*now_ms*/ kNowMs, /*mono_now_ms*/ 0, kStableThresholdMs, obs); EXPECT_EQ(floor.fenced_now, 0u); @@ -1455,26 +1464,27 @@ TEST(CASHeartbeatFloor, FirstSightNeverFencesEvenIfStampLooksExpired) EXPECT_EQ(obs.at("s1").first_seen_mono_ms, 0u); } -TEST(CASHeartbeatFloor, StableTokenPastThresholdIsFenced) +TEST(CASHeartbeatFloor, StableIncarnationPastThresholdIsFenced) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + Ops ops(b); + seedMount(ops.op, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; - const HeartbeatFloor floor_before = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); + const HeartbeatFloor floor_before = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); EXPECT_EQ(floor_before.fenced_now, 0u); - const MountLease before = decodeMountLease(b->get(l.mountKey("s1"))->bytes); + const MountLease before = decodeMountLease(ops.op.read(l.mountKey("s1"), Retry::standard())->bytes); - /// No renewal in between: the SAME token, observed since mono 0, is now stable for the full + /// No renewal in between: the SAME incarnation, observed since mono 0, is now stable for the full /// threshold on the leader's own clock. - const HeartbeatFloor floor2 = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ kStableThresholdMs, + const HeartbeatFloor floor2 = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); EXPECT_EQ(floor2.fenced_now, 1u); EXPECT_EQ(floor2.fenced_srids, std::vector{"s1"}); - const MountLease fenced = decodeMountLease(b->get(l.mountKey("s1"))->bytes); + const MountLease fenced = decodeMountLease(ops.op.read(l.mountKey("s1"), Retry::standard())->bytes); EXPECT_TRUE(fenced.gc_fenced); EXPECT_EQ(fenced.seq, before.seq + 1); } @@ -1483,23 +1493,24 @@ TEST(CASHeartbeatFloor, RenewalBetweenRoundsRestartsObservation) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + Ops ops(b); + seedMount(ops.op, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; - computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); + computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); ASSERT_TRUE(obs.contains("s1")); - const Token first_token = obs.at("s1").token; + const Incarnation first_incarnation = obs.at("s1").incarnation; - renewMount(*b, l, "s1"); - const Token renewed_token = b->get(l.mountKey("s1"))->token; - EXPECT_NE(renewed_token, first_token); + renewMount(ops.op, l, "s1"); + const Incarnation renewed_incarnation = currentIncarnation(ops.op, l.mountKey("s1")); + EXPECT_NE(renewed_incarnation, first_incarnation); - const HeartbeatFloor floor2 = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ kStableThresholdMs, + const HeartbeatFloor floor2 = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); EXPECT_EQ(floor2.fenced_now, 0u); ASSERT_TRUE(obs.contains("s1")); - EXPECT_EQ(obs.at("s1").token, renewed_token); + EXPECT_EQ(obs.at("s1").incarnation, renewed_incarnation); EXPECT_EQ(obs.at("s1").first_seen_mono_ms, kStableThresholdMs); } @@ -1512,11 +1523,12 @@ TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) { auto b = std::make_shared(); Layout l("p"); - seedMount(*b, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); - seedMount(*b, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + Ops ops(b); + seedMount(ops.op, l, "s1", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(ops.op, l, "s2", /*expires*/ 10, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; - computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); + computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); ASSERT_TRUE(obs.contains("s1")); ASSERT_TRUE(obs.contains("s2")); @@ -1525,13 +1537,10 @@ TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) /// observation restarts and it stays `live` -- isolating this test to the pruning behavior alone, /// not confounding it with s1 also becoming fence-eligible (which would erase its `obs` entry too, /// for an unrelated reason). - renewMount(*b, l, "s1"); - const auto s2_key = l.mountKey("s2"); - const auto got = b->get(s2_key); - ASSERT_TRUE(got.has_value()); - ASSERT_EQ(b->deleteExact(s2_key, got->token).kind, DeleteOutcome::Kind::Deleted); + renewMount(ops.op, l, "s1"); + ASSERT_EQ(ops.op.removeCurrent(l.mountKey("s2"), Retry::standard()), Removal::Removed); - computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); + computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); EXPECT_TRUE(obs.contains("s1")); EXPECT_FALSE(obs.contains("s2")) << "a srid removed from the LIST entirely must be pruned from obs, not linger forever"; @@ -1544,37 +1553,38 @@ TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) /// two live mounts — genuinely renewing between the two rounds below, so their observation never /// stabilizes. - seedMount(*b, l, "s1", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); - seedMount(*b, l, "s2", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); + Ops ops(b); + seedMount(ops.op, l, "s1", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(ops.op, l, "s2", /*expires*/ kNowMs + 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); /// dead — no renewal between the two rounds below — must be fenced-out by the second call. - seedMount(*b, l, "s3", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); - /// already-fenced — excluded, body byte-identical after both calls (no PUT). - seedMount(*b, l, "s4", /*expires*/ kNowMs - 60'000, /*fenced*/ true, /*min_active_build_sequence*/ 0); + seedMount(ops.op, l, "s3", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); + /// already-fenced — excluded, body byte-identical after both calls (no write). + seedMount(ops.op, l, "s4", /*expires*/ kNowMs - 60'000, /*fenced*/ true, /*min_active_build_sequence*/ 0); /// terminated (min_active_build_sequence == UINT64_MAX) with expired-looking timestamps — excluded, not fenced. - seedMount(*b, l, "s5", /*expires*/ kNowMs - 60'000, /*fenced*/ false, + seedMount(ops.op, l, "s5", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ std::numeric_limits::max()); MountObservationMap obs; /// Round 1 (mono 0): first sight of every non-terminal mount — nothing is fence-eligible yet. - const HeartbeatFloor floor_before = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); + const HeartbeatFloor floor_before = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); EXPECT_EQ(floor_before.live, 3u); // s1, s2, s3: observation just started EXPECT_EQ(floor_before.terminated, 1u); // s5 EXPECT_EQ(floor_before.fenced_now, 0u); EXPECT_EQ(floor_before.already_fenced, 1u); // s4 /// s1 and s2 renew between rounds (as a live keeper would); s3 does not (it crashed). - renewMount(*b, l, "s1"); - renewMount(*b, l, "s2"); + renewMount(ops.op, l, "s1"); + renewMount(ops.op, l, "s2"); - const auto s3_before = b->get(l.mountKey("s3")); - const auto s4_before = b->get(l.mountKey("s4")); + const auto s3_before = ops.op.read(l.mountKey("s3"), Retry::standard()); + const auto s4_before = ops.op.read(l.mountKey("s4"), Retry::standard()); ASSERT_TRUE(s3_before.has_value()); ASSERT_TRUE(s4_before.has_value()); - /// Round 2 (mono == threshold): s1/s2's renewed tokens restart their observation (still live); - /// s3's original token has now held stable for the full threshold -> fenced. - const HeartbeatFloor floor2 = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ kStableThresholdMs, + /// Round 2 (mono == threshold): s1/s2's renewed incarnations restart their observation (still + /// live); s3's original incarnation has now held stable for the full threshold -> fenced. + const HeartbeatFloor floor2 = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); EXPECT_EQ(floor2.live, 2u); // s1, s2: renewed, observation restarted @@ -1583,7 +1593,7 @@ TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) EXPECT_EQ(floor2.already_fenced, 1u); // s4 /// The dead body was fenced: gc_fenced set, seq bumped, the rest of the body preserved. - const auto s3_after = b->get(l.mountKey("s3")); + const auto s3_after = ops.op.read(l.mountKey("s3"), Retry::standard()); ASSERT_TRUE(s3_after.has_value()); const MountLease s3_prev = decodeMountLease(s3_before->bytes); const MountLease s3_now = decodeMountLease(s3_after->bytes); @@ -1594,19 +1604,19 @@ TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) EXPECT_EQ(s3_now.hostname, s3_prev.hostname); EXPECT_EQ(s3_now.expires_at_ms, s3_prev.expires_at_ms); - /// The already-fenced body was not touched (no PUT) across either call. - const auto s4_after = b->get(l.mountKey("s4")); + /// The already-fenced body was not touched (no write) across either call. + const auto s4_after = ops.op.read(l.mountKey("s4"), Retry::standard()); ASSERT_TRUE(s4_after.has_value()); EXPECT_EQ(s4_after->bytes, s4_before->bytes); } namespace { -/// A delegating backend whose `putOverwrite` of the target mount key first performs an inner renewal -/// (a real, token-correct overwrite that pushes expiry far into the future) and THEN delegates — so -/// the caller's fence-out overwrite lands on a stale token and returns PreconditionFailed. The inner -/// renewal runs exactly once (`renewed`), modelling a holder that renews concurrently in the window -/// between the function's GET and its fence-out PUT. +/// A delegating backend whose guarded write of the target mount key first performs an inner renewal +/// (a real, correctly-guarded write that pushes expiry far into the future) and THEN delegates — so +/// the caller's fence-out write lands on a stale precondition and is refused. The inner renewal runs +/// exactly once (`renewed`), modelling a holder that renews concurrently in the window between the +/// function's read and its fence-out write. class RenewOnFenceBackend : public InMemoryBackend { public: @@ -1615,21 +1625,22 @@ class RenewOnFenceBackend : public InMemoryBackend { } - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, - const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - if (key == target_key && !renewed) + if (expected_value && key == target_key && !renewed) { renewed = true; - /// The holder renews under the real current token: fresh far-future expiry. - const auto got = InMemoryBackend::get(key, {}); + /// The holder renews under the real current incarnation: fresh far-future expiry. + const auto got = InMemoryBackend::read(key, access); MountLease m = decodeMountLease(got->bytes); m.seq += 1; m.expires_at_ms = renewed_expires_ms; - const PutResult renew = InMemoryBackend::putOverwrite(key, encodeMountLease(m), got->token); - EXPECT_EQ(renew.outcome, PutOutcome::Done); + const auto renew = InMemoryBackend::write(key, encodeMountLease(m), got->value, access); + EXPECT_TRUE(renew.has_value()); } - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } private: @@ -1639,31 +1650,32 @@ class RenewOnFenceBackend : public InMemoryBackend }; } -TEST(CASHeartbeatFloor, FenceOutLosesTokenRaceReclassifiesLive) +TEST(CASHeartbeatFloor, FenceOutLosesTheIncarnationRaceAndReclassifiesLive) { Layout l("p"); auto b = std::make_shared( l.mountKey("s1"), /*renewed_expires*/ kNowMs + 120'000); + Ops ops(b); - seedMount(*b, l, "s1", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); + seedMount(ops.op, l, "s1", /*expires*/ kNowMs - 60'000, /*fenced*/ false, /*min_active_build_sequence*/ 0); MountObservationMap obs; /// Round 1: first sight, observation starts — never reaches the fence-out path (the race /// decorator stays armed for round 2). - const HeartbeatFloor floor_before = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); + const HeartbeatFloor floor_before = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); EXPECT_EQ(floor_before.fenced_now, 0u); - /// Round 2: the token has been stable past threshold, so the function attempts the fence-out. - /// The decorator renews concurrently under the real token, the PUT hits PreconditionFailed, the - /// function re-GETs and reclassifies it as live (observation restarted on the new token) — never - /// fenced. - const HeartbeatFloor floor2 = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ kStableThresholdMs, + /// Round 2: the incarnation has been stable past threshold, so the function attempts the + /// fence-out. The decorator renews concurrently under the real incarnation, the write is refused, + /// and the re-decision reclassifies the slot as live (observation restarted on the new + /// incarnation) — never fenced. + const HeartbeatFloor floor2 = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); EXPECT_EQ(floor2.fenced_now, 0u); EXPECT_EQ(floor2.live, 1u); - const auto after = b->get(l.mountKey("s1")); + const auto after = ops.op.read(l.mountKey("s1"), Retry::standard()); ASSERT_TRUE(after.has_value()); EXPECT_FALSE(decodeMountLease(after->bytes).gc_fenced); } @@ -1673,8 +1685,9 @@ TEST(CASHeartbeatFloor, EmptyPrefixYieldsNoLiveMounts) auto b = std::make_shared(); Layout l("p"); + Ops ops(b); MountObservationMap obs; - const HeartbeatFloor floor = computeHeartbeatFloor(*b, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); + const HeartbeatFloor floor = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); EXPECT_EQ(floor.live, 0u); EXPECT_EQ(floor.terminated, 0u); @@ -1691,16 +1704,17 @@ TEST(CASListMounts, ClassifiesEveryStateReadOnly) const uint64_t now_ms = 1'000'000; const uint64_t ttl_ms = 10'000; + Ops ops(backend); /// live: fresh claim for srid "a" - ASSERT_EQ(claimMount(*backend, layout, "a", UInt128{1}, /*our_epoch=*/1, now_ms, ttl_ms).kind, + ASSERT_EQ(claimMount(ops.op, layout, "a", UInt128{1}, /*our_epoch=*/1, now_ms, ttl_ms).kind, MountClaimResult::Claimed); /// expired: claim for "b" whose lease ran out long before now_ms - ASSERT_EQ(claimMount(*backend, layout, "b", UInt128{2}, 1, now_ms - 100'000, ttl_ms).kind, + ASSERT_EQ(claimMount(ops.op, layout, "b", UInt128{2}, 1, now_ms - 100'000, ttl_ms).kind, MountClaimResult::Claimed); /// corrupt: garbage bytes in "c"'s mount slot - backend->putIfAbsent(layout.mountKey("c"), "garbage-not-a-proto", {}); + mustCommit(ops.op.create(layout.mountKey("c"), "garbage-not-a-proto", Retry::standard()), "corrupt slot"); - auto mounts = listMounts(*backend, layout, now_ms, /*skew_margin_ms=*/ttl_ms / 2); + auto mounts = listMounts(ops.op, layout, now_ms, /*skew_margin_ms=*/ttl_ms / 2); ASSERT_EQ(mounts.size(), 3u); std::map by_srid; for (const auto & m : mounts) @@ -1711,7 +1725,7 @@ TEST(CASListMounts, ClassifiesEveryStateReadOnly) /// READ-ONLY guarantee: "b" is expired but must NOT be fenced by listMounts /// (computeHeartbeatFloor would stamp gc_fenced=true; the introspection view must not). - auto again = listMounts(*backend, layout, now_ms, ttl_ms / 2); + auto again = listMounts(ops.op, layout, now_ms, ttl_ms / 2); for (const auto & m : again) if (m.srid == "b") { @@ -1730,10 +1744,11 @@ TEST(CASListMounts, NestedSridIsNotTruncated) const uint64_t now_ms = 1'000'000; const uint64_t ttl_ms = 10'000; - ASSERT_EQ(claimMount(*backend, layout, "shard-01/replica-a", UInt128{1}, /*our_epoch=*/1, now_ms, ttl_ms).kind, + Ops ops(backend); + ASSERT_EQ(claimMount(ops.op, layout, "shard-01/replica-a", UInt128{1}, /*our_epoch=*/1, now_ms, ttl_ms).kind, MountClaimResult::Claimed); - auto mounts = listMounts(*backend, layout, now_ms, /*skew_margin_ms=*/ttl_ms / 2); + auto mounts = listMounts(ops.op, layout, now_ms, /*skew_margin_ms=*/ttl_ms / 2); ASSERT_EQ(mounts.size(), 1u); EXPECT_EQ(mounts[0].srid, "shard-01/replica-a"); EXPECT_EQ(mounts[0].state, "live"); @@ -1747,24 +1762,25 @@ TEST(CASClaimMount, SameEpochFencedIsNotRefreshable) using namespace DB::Cas; auto backend = std::make_shared(); Layout layout("pool"); + Ops ops(backend); /// mint for (uuid 1, epoch 1), then fence it in place (what computeHeartbeatFloor does): - ASSERT_EQ(claimMount(*backend, layout, "a", DB::UInt128{1}, 1, 1000, 10'000).kind, + ASSERT_EQ(claimMount(ops.op, layout, "a", DB::UInt128{1}, 1, 1000, 10'000).kind, MountClaimResult::Claimed); { - auto got = backend->get(layout.mountKey("a")); + auto got = ops.op.read(layout.mountKey("a"), Retry::standard()); MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - ASSERT_EQ(backend->putOverwrite(layout.mountKey("a"), encodeMountLease(fenced), got->token).outcome, - PutOutcome::Done); + mustCommit(ops.op.replace(layout.mountKey("a"), encodeMountLease(fenced), got->incarnation, Retry::standard()), + "fence-out"); } /// Same (uuid, epoch) re-claim must NOT refresh a fenced body — a fence costs an epoch: - const auto r = claimMount(*backend, layout, "a", DB::UInt128{1}, 1, 2000, 10'000); + const auto r = claimMount(ops.op, layout, "a", DB::UInt128{1}, 1, 2000, 10'000); EXPECT_EQ(r.kind, MountClaimResult::FencedSelf); /// The body on the backend is still the fenced one (no write happened): - EXPECT_TRUE(decodeMountLease(backend->get(layout.mountKey("a"))->bytes).gc_fenced); + EXPECT_TRUE(decodeMountLease(ops.op.read(layout.mountKey("a"), Retry::standard())->bytes).gc_fenced); /// A DIFFERENT epoch reclaims immediately (existing branch, unchanged): - EXPECT_EQ(claimMount(*backend, layout, "a", DB::UInt128{1}, 2, 2000, 10'000).kind, + EXPECT_EQ(claimMount(ops.op, layout, "a", DB::UInt128{1}, 2, 2000, 10'000).kind, MountClaimResult::Claimed); } @@ -1773,31 +1789,34 @@ TEST(CASClaimMount, SameEpochFencedIsNotRefreshable) /// A same-uuid, different-epoch lease whose STAMPED `expires_at_ms` looks long expired on OUR wall /// clock must NOT be reclaimed by that comparison alone — a clock-skewed or simply late-observing /// caller must never trust a bare wall-clock read across incarnations. `claimMount` (without a -/// `proven_dead_token`) always reports `LiveDoubleStart` for this branch now; only the observation +/// `proven_dead_incarnation`) always reports `LiveDoubleStart` for this branch now; only the observation /// loop (`claimMountAwaitingExpiry`) may turn it into a reclaim, and only after proving death on ITS /// OWN clock. TEST(CASMountObservation, ExpiredLookingLeaseIsNotReclaimedByWallClock) { auto b = std::make_shared(); Layout l{"p"}; + Ops ops(b); /// Predecessor epoch 7 stamped expires_at_ms = 1000; our wall clock says 999999 (long past). - auto first = claimMount(*b, l, "r", UInt128(1), 7, /*now_ms=*/500, /*ttl_ms=*/500); + auto first = claimMount(ops.op, l, "r", UInt128(1), 7, /*now_ms=*/500, /*ttl_ms=*/500); ASSERT_EQ(first.kind, MountClaimResult::Claimed); - auto r = claimMount(*b, l, "r", UInt128(1), /*our_epoch=*/8, /*now_ms=*/999999, 500); + auto r = claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/8, /*now_ms=*/999999, 500); EXPECT_EQ(r.kind, MountClaimResult::LiveDoubleStart); /// no wall-clock trust } -/// The observation loop reclaims once the write-token has held stable for the FULL rate-bound -/// threshold (`ttl_ms + ttl_ms/20 + poll_interval_ms`) on its OWN (injected, fake) clock — never -/// short-circuiting on the wall clock, which this test drives to an irrelevant, already-expired value. -TEST(CASMountObservation, TokenStableForThresholdThenReclaimed) +/// The observation loop reclaims once the observed incarnation has held stable for the FULL +/// rate-bound threshold (`ttl_ms + ttl_ms/20 + poll_interval_ms`) on its OWN (injected, fake) clock — +/// never short-circuiting on the wall clock, which this test drives to an irrelevant, already-expired +/// value. +TEST(CASMountObservation, IncarnationStableForThresholdThenReclaimed) { auto b = std::make_shared(); Layout l{"p"}; - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, 500, 500).kind, MountClaimResult::Claimed); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, 500, 500).kind, MountClaimResult::Claimed); uint64_t mono = 0; std::vector sleeps; - auto r = claimMountAwaitingExpiry(*b, l, "r", UInt128(1), 8, + auto r = claimMountAwaitingExpiry(ops.op, l, "r", UInt128(1), 8, []{ return uint64_t{999999}; }, /// wall clock: irrelevant [&]{ return mono; }, /// observation clock /*ttl_ms=*/500, /*poll_interval_ms=*/50, @@ -1807,28 +1826,31 @@ TEST(CASMountObservation, TokenStableForThresholdThenReclaimed) EXPECT_GE(mono, 500 + 500 / 20 + 50); /// full threshold actually waited } -/// A renewal DURING the observation window (the real holder is still alive) bumps the write-token — -/// the loop must detect the mismatch and RESTART the observation from the new token, never reclaiming -/// off a window that started watching a now-superseded token. +/// A renewal DURING the observation window (the real holder is still alive) mints a new incarnation — +/// the loop must detect the mismatch and RESTART the observation from it, never reclaiming off a +/// window that started watching a now-superseded incarnation. TEST(CASMountObservation, RenewalDuringObservationRestartsIt) { auto b = std::make_shared(); Layout l{"p"}; - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, 500, 500).kind, MountClaimResult::Claimed); + uint64_t keeper_boot = 0; + Ops ops(b, &keeper_boot); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, 500, 500).kind, MountClaimResult::Claimed); /// The real (still-alive) holder's keeper for epoch 7: `start()` adopts the slot `claimMount` just - /// wrote (no seq bump, per the ADOPT RULE), then synchronous renewal bumps the token mid-observation. + /// wrote (no seq bump, per the ADOPT RULE), then a synchronous renewal mints a new incarnation + /// mid-observation. uint64_t keeper_wall = 500; - MountLeaseKeeper keeper(b, l, "r", UInt128(1), 7, std::chrono::milliseconds(500), + MountLeaseKeeper keeper(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(500), [&] { return keeper_wall; }, [] { return uint64_t{0}; }, {}, - std::chrono::milliseconds(0)); + std::chrono::milliseconds(0), [&] { return keeper_boot; }); keeper.start(); const uint64_t threshold_ms = 500 + 500 / 20 + 50; /// = 575 uint64_t mono = 0; bool renewed = false; int wait_starts = 0; - auto r = claimMountAwaitingExpiry(*b, l, "r", UInt128(1), 8, + auto r = claimMountAwaitingExpiry(ops.op, l, "r", UInt128(1), 8, []{ return uint64_t{999999}; }, /// wall clock: irrelevant [&]{ return mono; }, /// observation clock /*ttl_ms=*/500, /*poll_interval_ms=*/50, @@ -1860,22 +1882,23 @@ TEST(CASMountObservation, GcFencedIsReclaimedInstantlyWithPriorFenced) { auto b = std::make_shared(); Layout l{"p"}; - ASSERT_EQ(claimMount(*b, l, "r", UInt128(1), 7, 1000, 500).kind, MountClaimResult::Claimed); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, 1000, 500).kind, MountClaimResult::Claimed); /// Fence it manually (what `computeHeartbeatFloor`'s fence-out does): gc_fenced=true, seq+1, - /// token-guarded. + /// guarded by the observed incarnation. { - auto got = b->get(l.mountKey("r")); + auto got = ops.op.read(l.mountKey("r"), Retry::standard()); ASSERT_TRUE(got.has_value()); MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - ASSERT_EQ(b->putOverwrite(l.mountKey("r"), encodeMountLease(fenced), got->token).outcome, - PutOutcome::Done); + mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(fenced), got->incarnation, Retry::standard()), + "fence-out"); } int sleeps = 0; - auto r = claimMountAwaitingExpiry(*b, l, "r", UInt128(1), /*our_epoch=*/8, + auto r = claimMountAwaitingExpiry(ops.op, l, "r", UInt128(1), /*our_epoch=*/8, []{ return uint64_t{999999}; }, []{ return uint64_t{0}; }, /*ttl_ms=*/500, /*poll_interval_ms=*/50, @@ -1893,60 +1916,62 @@ TEST(CASMountObservation, GcFencedIsReclaimedInstantlyWithPriorFenced) TEST(CASFenceTerminal, AbsentMountSlotIsNotTerminal) { - InMemoryBackend b; + Ops ops(std::make_shared()); Layout l{"p"}; - EXPECT_FALSE(isCreatorFenceTerminal(b, l, "never-mounted", 1)) + EXPECT_FALSE(isCreatorFenceTerminal(ops.op, l, "never-mounted", 1)) << "absence proves nothing about liveness -- never waved through"; } TEST(CASFenceTerminal, UndecodableMountBodyIsNotTerminal) { - InMemoryBackend b; + Ops ops(std::make_shared()); Layout l{"p"}; - b.putIfAbsent(l.mountKey("r"), "garbage-not-a-lease", {}); - EXPECT_FALSE(isCreatorFenceTerminal(b, l, "r", 1)) + mustCommit(ops.op.create(l.mountKey("r"), "garbage-not-a-lease", Retry::standard()), "undecodable lease"); + EXPECT_FALSE(isCreatorFenceTerminal(ops.op, l, "r", 1)) << "an unreadable lease of some other format generation must block, never wave through"; } TEST(CASFenceTerminal, GcFencedIsTerminal) { - InMemoryBackend b; + Ops ops(std::make_shared()); Layout l{"p"}; - ASSERT_EQ(claimMount(b, l, "r", UInt128(1), /*our_epoch=*/7, 1000, 500).kind, MountClaimResult::Claimed); - auto got = b.get(l.mountKey("r")); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/7, 1000, 500).kind, MountClaimResult::Claimed); + auto got = ops.op.read(l.mountKey("r"), Retry::standard()); ASSERT_TRUE(got.has_value()); MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; - ASSERT_EQ(b.putOverwrite(l.mountKey("r"), encodeMountLease(fenced), got->token).outcome, PutOutcome::Done); + mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(fenced), got->incarnation, Retry::standard()), + "fence-out"); - EXPECT_TRUE(isCreatorFenceTerminal(b, l, "r", 7)); + EXPECT_TRUE(isCreatorFenceTerminal(ops.op, l, "r", 7)); } TEST(CASFenceTerminal, CleanFarewellIsTerminal) { - InMemoryBackend b; + Ops ops(std::make_shared()); Layout l{"p"}; - ASSERT_EQ(claimMount(b, l, "r", UInt128(1), /*our_epoch=*/7, 1000, 500).kind, MountClaimResult::Claimed); - auto got = b.get(l.mountKey("r")); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/7, 1000, 500).kind, MountClaimResult::Claimed); + auto got = ops.op.read(l.mountKey("r"), Retry::standard()); ASSERT_TRUE(got.has_value()); MountLease retired = decodeMountLease(got->bytes); retired.min_active_build_sequence = std::numeric_limits::max(); - ASSERT_EQ(b.putOverwrite(l.mountKey("r"), encodeMountLease(retired), got->token).outcome, PutOutcome::Done); + mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(retired), got->incarnation, Retry::standard()), + "farewell"); - EXPECT_TRUE(isCreatorFenceTerminal(b, l, "r", 7)); + EXPECT_TRUE(isCreatorFenceTerminal(ops.op, l, "r", 7)); } TEST(CASFenceTerminal, ADifferentLiveWriterEpochIsTerminalForTheOldOne) { - InMemoryBackend b; + Ops ops(std::make_shared()); Layout l{"p"}; /// Slot now held at epoch 8 -- epoch 7's incarnation is superseded regardless of ITS OWN /// certificate (neither fenced nor farewelled). - ASSERT_EQ(claimMount(b, l, "r", UInt128(1), /*our_epoch=*/8, 1000, 500).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/8, 1000, 500).kind, MountClaimResult::Claimed); - EXPECT_TRUE(isCreatorFenceTerminal(b, l, "r", 7)) + EXPECT_TRUE(isCreatorFenceTerminal(ops.op, l, "r", 7)) << "a different epoch is currently live at this slot -- epoch 7 can never reclaim it"; - EXPECT_FALSE(isCreatorFenceTerminal(b, l, "r", 8)) + EXPECT_FALSE(isCreatorFenceTerminal(ops.op, l, "r", 8)) << "epoch 8 IS the current live epoch -- not terminal"; } @@ -1954,12 +1979,216 @@ TEST(CASFenceTerminal, ADifferentLiveWriterEpochIsTerminalForTheOldOne) /// treated as terminal -- mirrors `claimMount`'s own refusal to trust a bare timestamp comparison. TEST(CASFenceTerminal, ExpiredButSameEpochAndUncertifiedIsNotTerminal) { - InMemoryBackend b; + Ops ops(std::make_shared()); Layout l{"p"}; /// A lease whose stamped expiry is already far in the past, same epoch throughout. - ASSERT_EQ(claimMount(b, l, "r", UInt128(1), /*our_epoch=*/7, /*now_ms=*/0, /*ttl_ms=*/1).kind, + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*our_epoch=*/7, /*now_ms=*/0, /*ttl_ms=*/1).kind, MountClaimResult::Claimed); - EXPECT_FALSE(isCreatorFenceTerminal(b, l, "r", 7)) + EXPECT_FALSE(isCreatorFenceTerminal(ops.op, l, "r", 7)) << "expiry alone is never a certificate of death, exactly like claimMount's own discipline"; } + +/// The absent-epoch path's post-conflict recheck is a CHECK, not a blanket refusal, and both halves +/// have to hold: work that became visible across the conflict must block the allocation, and an +/// unchanged, still-empty subtree must let it proceed from the winner's own epoch state. Dropping the +/// recheck breaks the first arm; turning it into an unconditional refusal breaks the second. +TEST(CASServerRoot, AllocateWriterEpochKeepsThePostConflictCorruptionCheck) +{ + const Layout layout("p"); + { + auto backend = std::make_shared(/*reveal_owned_work=*/true); + Ops ops(backend); + EXPECT_THROW(allocateWriterEpoch( + ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), DB::Exception); + EXPECT_TRUE(backend->fired); + ASSERT_TRUE(backend->winner_installed); + } + { + auto backend = std::make_shared(/*reveal_owned_work=*/false); + Ops ops(backend); + EXPECT_EQ(allocateWriterEpoch( + ops.op, layout, "root/x", EpochMintPolicy::NormalMount, 0, emptyCatalogObservation()), 2u) + << "the second decision must allocate from the conflict winner's epoch, not refuse outright"; + EXPECT_TRUE(backend->fired); + ASSERT_TRUE(backend->winner_installed); + const auto epoch = ops.op.read(layout.epochKey("root/x"), Retry::standard()); + ASSERT_TRUE(epoch.has_value()); + EXPECT_EQ(decodeServerEpoch(epoch->bytes).next_writer_epoch, 3u); + } +} + +/// Adoption costs exactly two requests: one read that both decides the branch and supplies the +/// precondition, and one write. A presence probe ahead of the read, or a second read to recover a +/// precondition the first one already carried, shows up here as a third request. +TEST(CASMountLease, ClaimAdoptIsTwoRequests) +{ + auto backend = std::make_shared(); + Layout l("p"); + uint64_t now = 1000; + Ops ops(backend); + + /// The absent-slot mint. + backend->reads = backend->heads = backend->writes = 0; + MountLeaseKeeper minting(ops.mount, ops.farewell, l, "fresh", UInt128(1), 7, + std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }); + minting.start(); + EXPECT_EQ(backend->reads, 1u); + EXPECT_EQ(backend->writes, 1u); + EXPECT_EQ(backend->heads, 0u); + + /// The adoption of a slot `claimMount` already wrote. + ASSERT_EQ(claimMount(ops.op, l, "adopted", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, + MountClaimResult::Claimed); + backend->reads = backend->heads = backend->writes = 0; + MountLeaseKeeper adopting(ops.mount, ops.farewell, l, "adopted", UInt128(1), 7, + std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }); + adopting.start(); + EXPECT_EQ(backend->reads, 1u); + EXPECT_EQ(backend->writes, 1u); + EXPECT_EQ(backend->heads, 0u); +} + +/// A mount whose fence has dropped must still hand its slot back: the renewal is refused (it would be +/// writing under authority this node no longer holds), while the farewell runs on the open plane and +/// lands. Deliberately two keepers: `release` is admitted only from `Active`, so a keeper whose +/// renewal already went terminal never reaches its own farewell -- the ordering the two halves below +/// pin separately. +TEST(CASMountLease, FarewellRunsOnAnOpenFenceAfterTheMountFenceIsLost) +{ + auto backend = std::make_shared(); + Layout l("p"); + uint64_t now = 1000; + uint64_t boot = 0; + bool fence_lost = false; + + CasRequests mount_requests(backend, Fence{ + [] { return uint64_t{0}; }, + [&fence_lost](uint64_t, uint64_t) { return fence_lost ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [&fence_lost](uint64_t) + { + if (fence_lost) + throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "mount fence lost"); + }}); + mount_requests.setNowFnForTest([&boot] { return boot; }); + mount_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); + CasRequests open_requests = openRequestsForTest(backend); + open_requests.setNowFnForTest([&boot] { return boot; }); + open_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); + CasOperation seed = open_requests.admit(); + + ASSERT_EQ(claimMount(seed, l, "renewing", UInt128(1), 7, now, /*ttl*/ 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(seed, l, "departing", UInt128(1), 7, now, /*ttl*/ 1000).kind, MountClaimResult::Claimed); + + MountLeaseKeeper renewing(mount_requests, open_requests, l, "renewing", UInt128(1), 7, + std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, + {}, std::chrono::milliseconds(0), [&] { return boot; }); + MountLeaseKeeper departing(mount_requests, open_requests, l, "departing", UInt128(1), 7, + std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, + {}, std::chrono::milliseconds(0), [&] { return boot; }); + renewing.start(); + departing.start(); + + fence_lost = true; + + const MountRenewResult refused = renewing.renew(MountRenewOperationEnvironment{}); + EXPECT_EQ(refused.outcome, MountRenewOutcome::Terminal); + EXPECT_FALSE(refused.sent_any); + EXPECT_FALSE(renewing.canRelease()) << "a terminal renewal leaves no farewell to run"; + + EXPECT_NO_THROW(departing.release()); + const MountLease farewell = decodeMountLease(seed.read(l.mountKey("departing"), Retry::standard())->bytes); + EXPECT_EQ(farewell.min_active_build_sequence, std::numeric_limits::max()); +} + +/// The claim is admitted off the mount fence, and it has to be: a self-remount runs with the fence +/// already latched lost, so a claim gated on it could never reclaim the slot. What keeps the claim +/// safe is the conditional write it makes, not the fence. +TEST(CASMountLease, ClaimIsNotAdmittedUnderTheMountFence) +{ + auto backend = std::make_shared(); + Layout l("p"); + uint64_t now = 1000; + uint64_t boot = 0; + + CasRequests mount_requests(backend, Fence{ + [] { return uint64_t{0}; }, + [](uint64_t, uint64_t) { return Fence::Admit::LostOrRearmed; }, + [](uint64_t) { throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "mount fence lost"); }}); + mount_requests.setNowFnForTest([&boot] { return boot; }); + mount_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); + CasRequests open_requests = openRequestsForTest(backend); + open_requests.setNowFnForTest([&boot] { return boot; }); + open_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); + + MountLeaseKeeper keeper(mount_requests, open_requests, l, "r", UInt128(1), 7, + std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, + {}, std::chrono::milliseconds(0), [&] { return boot; }); + EXPECT_NO_THROW(keeper.start()); + + CasOperation reader = open_requests.admit(); + const MountLease claimed = decodeMountLease(reader.read(l.mountKey("r"), Retry::standard())->bytes); + EXPECT_EQ(claimed.writer_epoch, 7u); + EXPECT_EQ(claimed.seq, 1u); +} + +/// A lost owner-claim race is decided from the conflict's OWN resolve observation, so the two outcomes +/// have to be told apart from that alone: a racer that installed our uuid leaves nothing to do, a +/// foreign one fails closed. Reading the key again would answer a later question than the conflict +/// asked, and would cost a request per race. +TEST(CASServerRootClaim, OwnerLostToARacerIsDecidedFromTheConflictObservation) +{ + Layout l("p"); + { + auto backend = std::make_shared(UInt128(1)); + Ops ops(backend); + EXPECT_NO_THROW(claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation())); + EXPECT_TRUE(backend->fired); + /// The pre-claim read plus the create's own conflict-resolve read, and no third: a re-read + /// added back for the decision itself would raise this to 3. + EXPECT_EQ(backend->owner_reads, 2u); + } + { + auto backend = std::make_shared(UInt128(2)); + Ops ops(backend); + DB::Cas::tests::expectThrowsCodeWithMessage( + DB::ErrorCodes::CORRUPTED_DATA, + "claimed by a different server during our claim", + [&] { claimOwnerOrThrow(ops.op, l, "r", UInt128(1), emptyCatalogObservation()); }); + EXPECT_TRUE(backend->fired); + EXPECT_EQ(backend->owner_reads, 2u); + } +} + +/// A remount re-anchors its lease BEFORE it arms the fence for the new incarnation, so the fence is +/// still latched lost at that moment. The steady-state renewal is refused there — the sibling test +/// above pins that — and the remount's own renewal has to be admitted off the fence, or the pool could +/// never re-anchor and the remount attempt would fail on exactly the throttled store that caused it. +TEST(CASMountLease, RemountRenewalIsAdmittedOffTheMountFence) +{ + auto backend = std::make_shared(); + Layout l("p"); + uint64_t now = 1000; + uint64_t boot = 0; + + CasRequests mount_requests(backend, Fence{ + [] { return uint64_t{0}; }, + [](uint64_t, uint64_t) { return Fence::Admit::LostOrRearmed; }, + [](uint64_t) { throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "mount fence lost"); }}); + mount_requests.setNowFnForTest([&boot] { return boot; }); + mount_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); + CasRequests open_requests = openRequestsForTest(backend); + open_requests.setNowFnForTest([&boot] { return boot; }); + open_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); + + MountLeaseKeeper keeper(mount_requests, open_requests, l, "r", UInt128(1), 7, + std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, + {}, std::chrono::milliseconds(0), [&] { return boot; }); + keeper.start(); + + const MountRenewResult redo = keeper.renewForRemount(); + EXPECT_EQ(redo.outcome, MountRenewOutcome::Committed); + + CasOperation reader = open_requests.admit(); + EXPECT_EQ(decodeMountLease(reader.read(l.mountKey("r"), Retry::standard())->bytes).seq, 2u); +} diff --git a/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp b/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp index 42dae767213a..bb1b8bd3adc6 100644 --- a/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp +++ b/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp @@ -14,15 +14,18 @@ using DB::Cas::tests::expectThrowsCodeWithMessage; namespace { -/// One keeper for the mount slot of server-root "r", under (uuid=1, epoch=7) unless overridden. +/// One keeper for the mount slot of server-root "r", under (uuid=1, epoch=7) unless overridden. Both +/// of its planes are the same open-fence one: what these tests exercise is the mount protocol's own +/// exclusivity, not a fence's, and no test here renews, which is the only caller of the mount plane. MountLeaseKeeper makeKeeper( - const std::shared_ptr & backend, + CasRequests & requests, uint64_t & now, DB::UInt128 uuid = DB::UInt128(1), uint64_t epoch = 7) { return MountLeaseKeeper( - backend, + requests, + requests, Layout("p"), "r", uuid, @@ -32,54 +35,36 @@ MountLeaseKeeper makeKeeper( [] { return uint64_t{0}; }); } -void markMountGcFenced(MountSlotRaceBackend & backend, const Layout & layout, const String & server_root_id) +void markMountGcFenced(CasOperation & op, const Layout & layout, const String & server_root_id) { const String key = layout.mountKey(server_root_id); - const auto got = backend.get(key); + const auto got = op.read(key, Retry::standard()); ASSERT_TRUE(got); MountLease lease = decodeMountLease(got->bytes); lease.gc_fenced = true; - const PutResult result = backend.putOverwrite(key, encodeMountLease(lease), got->token); - ASSERT_EQ(result.outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + op.replace(key, encodeMountLease(lease), got->incarnation, Retry::standard()))); } } -TEST(CASMountClaimConflicts, SlotAppearedBetweenHeadAndPutIfAbsent) +TEST(CASMountClaimConflicts, SlotAppearedBetweenTheReadAndTheCreate) { auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; - /// Empty at `head`; another process mints it before our `putIfAbsent` lands. + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + /// Absent at the read; another process mints it before our create lands. backend->before_put_if_absent = [&] { - claimMount(*backend, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100); + CasOperation racer = requests.admit(); + claimMount(racer, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100); }; - auto keeper = makeKeeper(backend, now); + auto keeper = makeKeeper(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, - "appeared between head and putIfAbsent", - [&] { keeper.start(); }); -} - -TEST(CASMountClaimConflicts, SlotVanishedBetweenHeadAndGet) -{ - auto backend = std::make_shared(); - Layout layout("p"); - uint64_t now = 1000; - ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, - MountClaimResult::Claimed); - backend->before_get = [&] - { - const auto got = backend->get(layout.mountKey("r")); - ASSERT_TRUE(got); - backend->deleteExact(layout.mountKey("r"), got->token); - }; - auto keeper = makeKeeper(backend, now); - expectThrowsCodeWithMessage( - DB::ErrorCodes::ABORTED, - "vanished between head and get while claiming", + "appeared between the read and the create", [&] { keeper.start(); }); } @@ -88,10 +73,12 @@ TEST(CASMountClaimConflicts, SlotHeldByForeignServer) auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100).kind, + claimMount(op, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - auto keeper = makeKeeper(backend, now); + auto keeper = makeKeeper(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "held by a foreign server", @@ -103,10 +90,12 @@ TEST(CASMountClaimConflicts, SlotHeldByDifferentWriterEpoch) auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - auto keeper = makeKeeper(backend, now, DB::UInt128(1), /*epoch=*/8); + auto keeper = makeKeeper(requests, now, DB::UInt128(1), /*epoch=*/8); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "held by a different writer_epoch", @@ -118,15 +107,18 @@ TEST(CASMountClaimConflicts, SlotChangedInsideAdoptionWindow) auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - /// Rewrite the slot under a NEW token after our `get`, so our adoption `putOverwrite` conflicts. + /// Rewrite the slot under a NEW incarnation after our read, so our adoption write is refused. backend->before_put_overwrite = [&] { - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now + 1, /*ttl_ms=*/100); + CasOperation racer = requests.admit(); + claimMount(racer, layout, "r", DB::UInt128(1), 7, now + 1, /*ttl_ms=*/100); }; - auto keeper = makeKeeper(backend, now); + auto keeper = makeKeeper(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "changed while adopting our own mount slot", @@ -138,16 +130,17 @@ TEST(CASMountClaimConflicts, SlotVanishedInsideAdoptionWindow) auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); backend->before_put_overwrite = [&] { - const auto got = backend->get(layout.mountKey("r")); - ASSERT_TRUE(got); - backend->deleteExact(layout.mountKey("r"), got->token); + CasOperation racer = requests.admit(); + ASSERT_EQ(racer.removeCurrent(layout.mountKey("r"), Retry::standard()), Removal::Removed); }; - auto keeper = makeKeeper(backend, now); + auto keeper = makeKeeper(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "vanished while adopting our own mount slot", @@ -162,11 +155,13 @@ TEST(CASMountClaimConflicts, FencedBeforeAdoptionRaisesMountFenced) auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - markMountGcFenced(*backend, layout, "r"); - auto keeper = makeKeeper(backend, now); + markMountGcFenced(op, layout, "r"); + auto keeper = makeKeeper(requests, now); EXPECT_THROW(keeper.start(), MountFencedException); } @@ -175,12 +170,18 @@ TEST(CASMountClaimConflicts, FencedInsideAdoptionWindowRaisesMountFencedNotAbort auto backend = std::make_shared(); Layout layout("p"); uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ( - claimMount(*backend, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); /// The slot changes inside the adoption window AND the new body is fenced: the fenced branch must /// win over the "changed while adopting" one. - backend->before_put_overwrite = [&] { markMountGcFenced(*backend, layout, "r"); }; - auto keeper = makeKeeper(backend, now); + backend->before_put_overwrite = [&] + { + CasOperation racer = requests.admit(); + markMountGcFenced(racer, layout, "r"); + }; + auto keeper = makeKeeper(requests, now); EXPECT_THROW(keeper.start(), MountFencedException); } diff --git a/src/Disks/tests/gtest_cas_mount_runtime.cpp b/src/Disks/tests/gtest_cas_mount_runtime.cpp new file mode 100644 index 000000000000..697d486c171a --- /dev/null +++ b/src/Disks/tests/gtest_cas_mount_runtime.cpp @@ -0,0 +1,156 @@ +#include +#include +#include +#include +#include +#include + +#include + +using namespace DB::Cas; + +namespace +{ + +/// A `CasMountRuntime` with nothing running on it: no keeper, no workers, an injected boot clock and a +/// fence the test arms by hand. Enough to exercise admission, which reads only the fence's own state. +class RuntimeFixture +{ +public: + explicit RuntimeFixture(uint64_t lease_safety_margin_ms, uint64_t attempt_timeout_ms = 10) + : backend(std::make_shared()) + , mount(backend, Fence{ + [this] { return runtime.fenceGeneration(); }, + [this](uint64_t g, uint64_t needed) { return runtime.admit(g, needed); }, + [this](uint64_t g) { runtime.checkFenceOrThrow(g); }}) + , farewell(backend, Fence::open()) + , runtime( + backend, mount, farewell, layout, + MountConfig{.boot_ms_fn = [this] { return boot_ms; }}, + "test", sink, + CasRequestBudget{.attempt_timeout_ms = attempt_timeout_ms, + .lease_safety_margin_ms = lease_safety_margin_ms}, + [] { return false; }) + { + } + + CasMountRuntime * operator->() { return &runtime; } + + uint64_t boot_ms = 1'000; + +private: + std::shared_ptr backend; + Layout layout{"mount-runtime-admit"}; + CasEventSink sink; + CasRequests mount; + CasRequests farewell; + CasMountRuntime runtime; +}; + +/// Named verdicts, so a failing expectation reads as the answer rather than as a raw byte. +const char * admitName(Fence::Admit verdict) +{ + switch (verdict) + { + case Fence::Admit::Ok: return "Ok"; + case Fence::Admit::LostOrRearmed: return "LostOrRearmed"; + case Fence::Admit::NoBudget: return "NoBudget"; + } + return "unknown"; +} + +constexpr DB::UInt128 kUuid{7}; + +} + +/// The boundary is STRICT on both terms: a request that would only just finish as the lease runs out +/// is one that may land after this node's fence is already gone. +TEST(CASMountRuntime, AdmitRefusesAtTheExactBudgetBoundary) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/20); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'100); /// 100 ms of lease left + const uint64_t generation = f->fenceGeneration(); + + EXPECT_STREQ(admitName(f->admit(generation, 80)), "NoBudget") << "needed + margin == remaining must refuse"; + EXPECT_STREQ(admitName(f->admit(generation, 79)), "Ok") << "one millisecond of slack is enough"; + EXPECT_STREQ(admitName(f->admit(generation, 100)), "NoBudget") << "needed == remaining must refuse"; +} + +/// The subtraction in `admit` exists for this: `needed_ms + margin` would wrap and read as room. +TEST(CASMountRuntime, AdmitDoesNotWrapOnAnAbsurdNeed) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/20); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'100); + + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), std::numeric_limits::max())), "NoBudget"); +} + +TEST(CASMountRuntime, AdmitRefusesAnExpiredLease) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/0); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'100); + const uint64_t generation = f->fenceGeneration(); + + f.boot_ms = 1'099; + EXPECT_STREQ(admitName(f->admit(generation, 0)), "Ok"); + f.boot_ms = 1'100; + EXPECT_STREQ(admitName(f->admit(generation, 0)), "NoBudget") << "the deadline instant is already past"; + /// One millisecond further is what the `now >= deadline` guard actually earns: without it + /// `deadline - now` underflows to a huge remaining and the budget test reads it as room. + f.boot_ms = 1'101; + EXPECT_STREQ(admitName(f->admit(generation, 0)), "NoBudget") + << "a deadline already past must not underflow into room"; +} + +/// A re-arm is a fresh lease incarnation. A caller admitted under the previous one is stale even though +/// the fence is live again, which is the whole point of carrying a generation. +TEST(CASMountRuntime, AdmitRefusesAGenerationTheFenceMovedPast) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/0); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/100'000); + const uint64_t stale = f->fenceGeneration(); + f->armMountFence(kUuid, 2, /*deadline_boot_ms=*/100'000); + + EXPECT_STREQ(admitName(f->admit(stale, 0)), "LostOrRearmed"); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 0)), "Ok"); +} + +/// The latch, isolated from the generation bump that accompanies it: the generation presented here is +/// the one the trip itself produced, so only `lost` can be refusing. +TEST(CASMountRuntime, AdmitRefusesALostFenceWhateverTheBudget) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/0); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/100'000); + f->tripMountLost(); + + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 0)), "LostOrRearmed"); +} + +/// The unarmed default (no lease deadline yet) permits work: the bootstrap-control writes that claim a +/// lease run before there is one to be gated on. +TEST(CASMountRuntime, AdmitAllowsAnUnarmedFence) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/2'000); + f.boot_ms = 1'000; + + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 5'000)), "Ok"); +} + +/// `refAppendFenceOk` is `admit` at one attempt's worth of budget under the live generation. +TEST(CASMountRuntime, RefAppendFenceOkIsAdmitAtTheAttemptTimeout) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/20, /*attempt_timeout_ms=*/10); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'031); /// 31 ms left: one more than 10 + 20 + EXPECT_TRUE(f->refAppendFenceOk()); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 10)), "Ok"); + + f->setMountDeadline(1'030); /// exactly 10 + 20 left + EXPECT_FALSE(f->refAppendFenceOk()); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 10)), "NoBudget"); +} diff --git a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp index b8f803e514f8..45fca6d2f1b1 100644 --- a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp +++ b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp @@ -98,24 +98,25 @@ TEST(CASNamespaceFileRequestProfile, CreateThenRewrite) store->putNamespaceFile(life, kFile, "1\n"); EXPECT_EQ(backend->headCount(key), 1u); - EXPECT_EQ(backend->putCount(key), 1u); /// putIfAbsent -- the key was absent + EXPECT_EQ(backend->putCount(key), 1u); /// create-shaped -- the key was absent EXPECT_EQ(backend->putOverwriteCount(key), 0u); EXPECT_EQ(backend->getCount(key), 0u); EXPECT_EQ(backend->deleteCount(key), 0u); EXPECT_EQ(backend->listTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + /// And no write beyond the one accounted for above, anywhere. + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->touchedKeys(), std::vector{key}); backend->resetCounts(); store->putNamespaceFile(life, kFile, "2\n"); EXPECT_EQ(backend->headCount(key), 1u); - EXPECT_EQ(backend->putOverwriteCount(key), 1u); /// token-conditioned replacement -- it existed + EXPECT_EQ(backend->putOverwriteCount(key), 1u); /// replace-shaped -- it existed EXPECT_EQ(backend->putCount(key), 0u); EXPECT_EQ(backend->getCount(key), 0u); EXPECT_EQ(backend->deleteCount(key), 0u); EXPECT_EQ(backend->listTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->touchedKeys(), std::vector{key}); } @@ -132,8 +133,9 @@ TEST(CASNamespaceFileRequestProfile, Read) EXPECT_EQ(store->getNamespaceFile(life, kFile), String("1\n")); + /// One GET, and it is necessarily a whole-object one: `Backend::get` refuses a non-whole window + /// outright, so there is no partial read left for a separate counter to tell apart. EXPECT_EQ(backend->getCount(key), 1u); - EXPECT_EQ(backend->wholeGetCount(key), 1u); EXPECT_EQ(backend->headCount(key), 0u); EXPECT_EQ(backend->putCount(key), 0u); EXPECT_EQ(backend->putOverwriteCount(key), 0u); @@ -218,7 +220,7 @@ TEST(CASNamespaceFileRequestProfile, DedupLogRotation) EXPECT_EQ(backend->headCount(old_key), 1u); EXPECT_EQ(backend->deleteCount(old_key), 1u); EXPECT_EQ(backend->getTotal(), 0u); /// rotation reads no body - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 1u); /// and writes only the new segment /// Sorted, and the files prefix is a proper prefix of both segment keys, so it comes first. EXPECT_EQ(backend->touchedKeys(), (std::vector{prefix, old_key, new_key})); diff --git a/src/Disks/tests/gtest_cas_namespace_janitor.cpp b/src/Disks/tests/gtest_cas_namespace_janitor.cpp index 7a3455738bc4..76d2f4aa5be4 100644 --- a/src/Disks/tests/gtest_cas_namespace_janitor.cpp +++ b/src/Disks/tests/gtest_cas_namespace_janitor.cpp @@ -5,50 +5,83 @@ using namespace DB::Cas; using namespace DB::Cas::tests; +namespace DB::ErrorCodes +{ + extern const int NETWORK_ERROR; +} + namespace { +/// `readGcMaintenanceState` now takes an admitted `CasOperation`, which cannot bind to an rvalue: every +/// call site below goes through this helper rather than materializing its own throwaway operation. +GcMaintenanceReadResult readState(CasRequests & requests, const Layout & layout) +{ + auto op = requests.admit(); + return readGcMaintenanceState(op, layout); +} + class OrderedJanitorBackend : public CountingBackend { public: - using CountingBackend::get; std::vector events; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { if (prefix.ends_with("/cas/ns/")) events.push_back("list"); - return CountingBackend::list(prefix, cursor, limit); + return CountingBackend::list(prefix, cursor, limit, access); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (key.ends_with("/cas/ref_catalog")) events.push_back("catalog"); - return CountingBackend::get(key, range); + return CountingBackend::read(key, access); } }; class OmitFirstNamespacePageBackend : public CountingBackend { public: - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { if (omit && prefix.ends_with("/cas/ns/")) { omit = false; return {}; } - return CountingBackend::list(prefix, cursor, limit); + return CountingBackend::list(prefix, cursor, limit, access); } private: bool omit = true; }; +/// Flips `delete_done` right after its one REMOVE returns, so a liveness predicate closing over it stays +/// true through every request up to and including that delete -- whatever their number or order -- and +/// only refuses the very next one. `liveness` is sampled before every request now, so driving a fence +/// loss at an exact point robustly (rather than by counting samples, which would couple this test to how +/// many reads the catalog snapshot happens to take) means keying it to an observable EVENT instead. +class FlipAfterFirstDeleteBackend : public CountingBackend +{ +public: + DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, + DB::Cas::TransportAccess & access) override + { + DB::Cas::Backend::RawRemoval outcome = CountingBackend::remove(key, expected_value, access); + delete_done = true; + return outcome; + } + bool delete_done = false; +}; + class ReplaceBeforeJanitorDeleteBackend : public CountingBackend { public: - DeleteOutcome deleteExact(const String & key, const Token & token) override + DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, + DB::Cas::TransportAccess & access) override { if (!replaced) { @@ -57,7 +90,7 @@ class ReplaceBeforeJanitorDeleteBackend : public CountingBackend if (current) (void)InMemoryBackend::casPut(key, "winner", current->token); } - return CountingBackend::deleteExact(key, token); + return CountingBackend::remove(key, expected_value, access); } private: bool replaced = false; @@ -66,23 +99,28 @@ class ReplaceBeforeJanitorDeleteBackend : public CountingBackend class TokenlessListBackend : public CountingBackend { public: - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { - ListPage page = CountingBackend::list(prefix, cursor, limit); - for (ListedKey & key : page.keys) - key.token.reset(); + DB::Cas::Backend::RawListPage page = CountingBackend::list(prefix, cursor, limit, access); + for (auto & key : page.keys) + key.value.reset(); return page; } bool supportsListTokens() const override { return false; } - HeadResult head(const String & key) override + std::optional head(const String & key, DB::Cas::TransportAccess & access) override { - HeadResult result = CountingBackend::head(key); - if (!replaced && result.exists && key == replace_on_head) + std::optional result = CountingBackend::head(key, access); + if (!replaced && result && key == replace_on_head) { replaced = true; - (void)InMemoryBackend::casPut(key, "winner", result.token); + /// The qualified primitive `write` -- not the legacy `head`/`casPut` convenience pair -- so + /// this simulated concurrent actor neither re-enters the counted `head` override (the legacy + /// forwarder calls back through the virtual primitive) nor is itself counted as a write the + /// janitor made. + (void)InMemoryBackend::write(key, "winner", result->value, access); } return result; } @@ -96,9 +134,9 @@ class TokenlessListBackend : public CountingBackend class FenceLossDuringHeadBackend : public TokenlessListBackend { public: - HeadResult head(const String & key) override + std::optional head(const String & key, DB::Cas::TransportAccess & access) override { - HeadResult result = TokenlessListBackend::head(key); + std::optional result = TokenlessListBackend::head(key, access); fence_held = false; return result; } @@ -111,23 +149,26 @@ class CatalogAfterListBackend : public CountingBackend public: explicit CatalogAfterListBackend(NamespaceLifeId life_) : protected_life(std::move(life_)) {} - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { - ListPage page = CountingBackend::list(prefix, cursor, limit); + DB::Cas::Backend::RawListPage page = CountingBackend::list(prefix, cursor, limit, access); if (!published && prefix.ends_with("/cas/ns/")) { published = true; const String catalog_key = "p/cas/ref_catalog"; - /// This models a CONCURRENT actor's read, not the janitor's own -- counting it here would - /// make `PostListCatalogCutProtectsConcurrentCreationWithOneGet`'s "exactly one get" assertion - /// count this simulated actor's read as the janitor's, defeating the point of that assertion. - const auto current = InMemoryBackend::get(catalog_key, {}); // NOLINT(bugprone-parent-virtual-call) + /// This models a CONCURRENT actor's read, not the janitor's own. It must go through the + /// qualified PRIMITIVE, not the virtual `read`/`write` this class's base counts: the janitor's + /// own catalog read reaches the store through that same virtual dispatch, and a call routed + /// through it here would be indistinguishable from the janitor's -- doubling the count + /// `PostListCatalogCutProtectsConcurrentCreationWithOneGet` asserts is exactly one. + const auto current = InMemoryBackend::read(catalog_key, access); // NOLINT(bugprone-parent-virtual-call) if (current) { RefCatalog catalog; catalog.entries.push_back(CatalogEntry{.ns = protected_life.ns, .state = NsState::Live, .incarnation = protected_life.incarnation}); - (void)InMemoryBackend::casPut(catalog_key, encodeRefCatalog(catalog), current->token); + (void)InMemoryBackend::write(catalog_key, encodeRefCatalog(catalog), current->value, access); } } return page; @@ -140,27 +181,54 @@ class CatalogAfterListBackend : public CountingBackend class RejectCursorBackend : public CountingBackend { public: - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { if (prefix.ends_with("/cas/ns/") && !cursor.empty()) throw std::runtime_error("backend rejected cursor"); - return CountingBackend::list(prefix, cursor, limit); + return CountingBackend::list(prefix, cursor, limit, access); } }; class FailMaintenancePublicationBackend : public CountingBackend { public: - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (fail_publication && key.ends_with("/gc/maintenance_state")) throw std::runtime_error("maintenance publication failed"); - return CountingBackend::casPut(key, bytes, expected, meta); + return CountingBackend::write(key, bytes, expected_value, access); } bool fail_publication = false; }; +/// The catch-path reset in `NamespaceJanitor::runOnePage` fires on any LIST failure. Both faults here +/// throw a `DB::Exception` classified `NETWORK_ERROR`: a `std::runtime_error` is not a `Poco::Exception`, +/// so `CasOperation`'s engine treats it as an unmodeled local bug and surfaces it immediately on every +/// path (read or write) without ever reaching the ambiguity-resolving machinery this test needs -- a +/// `NETWORK_ERROR` is a genuine transient-looking store answer instead. The write additionally counts +/// its own attempts (`CountingBackend::writeTotal()` stays 0 here: this override throws before ever +/// delegating to the base `write`). +class ThrowingListAndAmbiguousWriteBackend : public CountingBackend +{ +public: + DB::Cas::Backend::RawListPage list(const String &, const String &, size_t, + DB::Cas::TransportAccess &) override + { + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "list failed"); + } + std::expected write(const String &, const String &, + const std::optional &, + DB::Cas::TransportAccess &) override + { + ++write_attempts; + throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "ambiguous write"); + } + uint64_t write_attempts = 0; +}; + void seedCatalog(CountingBackend & backend, const Layout & layout, RefCatalog catalog = {}) { ASSERT_EQ(backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(catalog)).outcome, PutOutcome::Done); @@ -176,32 +244,34 @@ NamespaceLifeId life(const char * name, uint64_t id) TEST(CASNamespaceJanitor, DeletesDeadFilesAndCheckpointFromOnePostListCatalogCut) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const auto dead = life("dead", 41); const String file = layout.namespaceFilesPrefix(dead) + "part/data.bin"; const String ckpt = layout.refCkptKey(dead); - ASSERT_EQ(backend.putIfAbsent(file, "file-bytes").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(ckpt, "ckpt-bytes").outcome, PutOutcome::Done); - backend.resetCounts(); + ASSERT_EQ(backend->putIfAbsent(file, "file-bytes").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(ckpt, "ckpt-bytes").outcome, PutOutcome::Done); + backend->resetCounts(); - NamespaceJanitor janitor(backend, layout, 100); + NamespaceJanitor janitor(requests, layout, 100); const NamespaceJanitorResult result = janitor.runOnePage(false, [] { return true; }); EXPECT_EQ(result.pages, 1u); EXPECT_EQ(result.keys, 2u); EXPECT_EQ(result.deleted, 2u); - EXPECT_FALSE(backend.get(file)); - EXPECT_FALSE(backend.get(ckpt)); - EXPECT_EQ(backend.listCount(layout.namespaceRootPrefix()), 1u); - EXPECT_EQ(backend.getCount(layout.refCatalogKey()), 1u); - EXPECT_EQ(readGcMaintenanceState(backend, layout).state, GcMaintenanceState{}); + EXPECT_FALSE(backend->get(file)); + EXPECT_FALSE(backend->get(ckpt)); + EXPECT_EQ(backend->listCount(layout.namespaceRootPrefix()), 1u); + EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); + EXPECT_EQ(readState(requests, layout).state, GcMaintenanceState{}); } TEST(CASNamespaceJanitor, RetainsEveryCurrentLifecycleAndSuppressesAmbiguousCut) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); RefCatalog catalog; CatalogEntry creating{.ns = RootNamespace{"creating"}, .state = NsState::Creating, .incarnation = UInt128{51}, @@ -210,207 +280,222 @@ TEST(CASNamespaceJanitor, RetainsEveryCurrentLifecycleAndSuppressesAmbiguousCut) CatalogEntry removing{.ns = RootNamespace{"removing"}, .state = NsState::Removing, .incarnation = UInt128{53}, .removal_started_round = 1}; catalog.entries = {creating, live, removing}; - seedCatalog(backend, layout, catalog); + seedCatalog(*backend, layout, catalog); for (const auto & entry : catalog.entries) - ASSERT_EQ(backend.putIfAbsent(layout.refCkptKey( + ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey( NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation)), "keep").outcome, PutOutcome::Done); - NamespaceJanitor janitor(backend, layout, 100); + NamespaceJanitor janitor(requests, layout, 100); const auto result = janitor.runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); - EXPECT_EQ(backend.deleteTotal(), 0u); + EXPECT_EQ(backend->deleteTotal(), 0u); } TEST(CASNamespaceJanitor, CatalogFirstCreatingRetainsEveryObjectOfTheNewLife) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const CatalogEntry creating{ .ns = RootNamespace{"catalog-first"}, .state = NsState::Creating, .incarnation = UInt128{54}, .creator = CreatorFence{.server_root_id = "srv", .writer_epoch = 2, .fence_generation = 3}}; - seedCatalog(backend, layout, RefCatalog{.entries = {creating}}); + seedCatalog(*backend, layout, RefCatalog{.entries = {creating}}); /// The production creation order is the point: the catalog row is durable before either object. const NamespaceLifeId creating_life = NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation); const String ckpt = layout.refCkptKey(creating_life); const String file = layout.namespaceFilesPrefix(creating_life) + "data"; - ASSERT_EQ(backend.putIfAbsent(ckpt, "checkpoint").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(file, "file").outcome, PutOutcome::Done); - backend.resetCounts(); + ASSERT_EQ(backend->putIfAbsent(ckpt, "checkpoint").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(file, "file").outcome, PutOutcome::Done); + backend->resetCounts(); const NamespaceJanitorResult result - = NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }); + = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); - EXPECT_EQ(backend.deleteTotal(), 0u); - EXPECT_EQ(backend.getCount(layout.refCatalogKey()), 1u); - EXPECT_TRUE(backend.get(ckpt)); - EXPECT_TRUE(backend.get(file)); + EXPECT_EQ(backend->deleteTotal(), 0u); + EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); + EXPECT_TRUE(backend->get(ckpt)); + EXPECT_TRUE(backend->get(file)); } TEST(CASNamespaceJanitor, CancelledCreatingCheckpointIsReclaimedThroughPublicLifecycle) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const CatalogEntry creating{ .ns = RootNamespace{"cancelled"}, .state = NsState::Creating, .incarnation = UInt128{55}, .creator = CreatorFence{.server_root_id = "dead-srv", .writer_epoch = 4, .fence_generation = 5}}; - seedCatalog(backend, layout, RefCatalog{.entries = {creating}}); + seedCatalog(*backend, layout, RefCatalog{.entries = {creating}}); const String ckpt = layout.refCkptKey( NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation)); - ASSERT_EQ(backend.putIfAbsent(ckpt, "cancelled-checkpoint").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(ckpt, "cancelled-checkpoint").outcome, PutOutcome::Done); + auto cancel_op = requests.admit(); ASSERT_EQ(CasRefCatalog::cancelStalledCreating( - backend, layout, creating, [](const CreatorFence &) { return true; }, - /*admitted_generation=*/7, [](uint64_t) {}), + cancel_op, layout, creating, [](const CreatorFence &) { return true; }), CasRefCatalog::StalledCreatingCancelOutcome::Cancelled); - EXPECT_TRUE(CasRefCatalog::read(backend, layout).catalog.entries.empty()); + auto read_op = requests.admit(); + EXPECT_TRUE(CasRefCatalog::read(read_op, layout).catalog.entries.empty()); const NamespaceJanitorResult result - = NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }); + = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); - EXPECT_FALSE(backend.get(ckpt)); + EXPECT_FALSE(backend->get(ckpt)); } TEST(CASNamespaceJanitor, SuppressionAndFenceLossDeleteNothing) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String first = layout.refCkptKey(life("dead-a", 61)); const String second = layout.refCkptKey(life("dead-b", 62)); - ASSERT_EQ(backend.putIfAbsent(first, "first").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(second, "second").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(first, "first").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(second, "second").outcome, PutOutcome::Done); + + /// The seeding above (the catalog + the two checkpoints) lands through the same write primitive + /// CountingBackend counts, so reset before measuring what the suppressed page itself does. + backend->resetCounts(); - NamespaceJanitor janitor(backend, layout, 1); + NamespaceJanitor janitor(requests, layout, 1); EXPECT_EQ(janitor.runOnePage(true, [] { return true; }).deleted, 0u); - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Absent) + EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "a globally suppressed page is undecided and must not mint cleanup progress"; - EXPECT_EQ(backend.putCount(layout.gcMaintenanceStateKey()), 0u); - EXPECT_EQ(backend.casPutCount(layout.gcMaintenanceStateKey()), 0u); - EXPECT_EQ(janitor.runOnePage(false, [] { return false; }).deleted, 0u); - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Absent) + EXPECT_EQ(backend->writeTotal(), 0u); + + /// `liveness` is sampled before every request the page makes, starting with the maintenance read + /// itself -- a sample false from the start therefore ends the call by exception rather than by a + /// quiet no-op result. + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, + [&] { (void)janitor.runOnePage(false, [] { return false; }); }); + EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "fence loss must not mint progress past a page whose deletion was not authorized"; - EXPECT_TRUE(backend.get(first)); - EXPECT_TRUE(backend.get(second)); - EXPECT_EQ(backend.deleteTotal(), 0u); + EXPECT_TRUE(backend->get(first)); + EXPECT_TRUE(backend->get(second)); + EXPECT_EQ(backend->deleteTotal(), 0u); } TEST(CASNamespaceJanitor, FenceLossOnRetainedOnlyPageDoesNotAdvanceCursor) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const CatalogEntry current{ .ns = RootNamespace{"current"}, .state = NsState::Live, .incarnation = UInt128{63}}; - seedCatalog(backend, layout, RefCatalog{.entries = {current}}); + seedCatalog(*backend, layout, RefCatalog{.entries = {current}}); const NamespaceLifeId current_life = NamespaceLifeId::fromCatalogEntry(current.ns, current.incarnation); const String ckpt = layout.refCkptKey(current_life); const String file = layout.namespaceFilesPrefix(current_life) + "data"; - ASSERT_EQ(backend.putIfAbsent(ckpt, "checkpoint").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(file, "file").outcome, PutOutcome::Done); - - const NamespaceJanitorResult result - = NamespaceJanitor(backend, layout, 1).runOnePage(false, [] { return false; }); - - EXPECT_EQ(result.deleted, 0u); - EXPECT_EQ(backend.deleteTotal(), 0u); - EXPECT_TRUE(backend.get(ckpt)); - EXPECT_TRUE(backend.get(file)); - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Absent) + ASSERT_EQ(backend->putIfAbsent(ckpt, "checkpoint").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(file, "file").outcome, PutOutcome::Done); + + /// A liveness sample false from the start is refused at the maintenance read, before the page ever + /// gets to examine an object -- retained-only or not; the page ends by exception. + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, + [&] { (void)NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return false; }); }); + + EXPECT_EQ(backend->deleteTotal(), 0u); + EXPECT_TRUE(backend->get(ckpt)); + EXPECT_TRUE(backend->get(file)); + EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "a tenure that observes fence loss cannot publish progress even when every object was retained"; } TEST(CASNamespaceJanitor, FenceLossAfterLastDeleteRetainsCursorWithoutRollingBackDelete) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead-after-delete", 64)); - ASSERT_EQ(backend.putIfAbsent(dead, "dead").outcome, PutOutcome::Done); - uint64_t fence_checks = 0; + ASSERT_EQ(backend->putIfAbsent(dead, "dead").outcome, PutOutcome::Done); - const NamespaceJanitorResult result - = NamespaceJanitor(backend, layout, 1).runOnePage(false, [&] { return fence_checks++ == 0; }); + const NamespaceJanitorResult result = NamespaceJanitor(requests, layout, 1).runOnePage( + false, [&] { return !backend->delete_done; }); EXPECT_EQ(result.deleted, 1u); - EXPECT_FALSE(backend.get(dead)) + EXPECT_FALSE(backend->get(dead)) << "the exact delete completed under the fence and is never rolled back"; - EXPECT_EQ(fence_checks, 2u) - << "the fence must be checked before deletion and again immediately before cursor publication"; - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Absent) + EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "losing the fence after the delete keeps this page selected for an idempotent retry"; } TEST(CASNamespaceJanitor, CursorResumesThenResetsAtEnd) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const auto dead = life("dead", 71); - ASSERT_EQ(backend.putIfAbsent(layout.namespaceFilesPrefix(dead) + "a", "a").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(layout.namespaceFilesPrefix(dead) + "b", "b").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(layout.namespaceFilesPrefix(dead) + "a", "a").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(layout.namespaceFilesPrefix(dead) + "b", "b").outcome, PutOutcome::Done); - NamespaceJanitor first_process(backend, layout, 1); + NamespaceJanitor first_process(requests, layout, 1); EXPECT_EQ(first_process.runOnePage(false, [] { return true; }).deleted, 1u); - const auto mid = readGcMaintenanceState(backend, layout); + const auto mid = readState(requests, layout); ASSERT_EQ(mid.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(mid.state); EXPECT_FALSE(mid.state->janitor_cursor.empty()); - NamespaceJanitor restarted_process(backend, layout, 1); + NamespaceJanitor restarted_process(requests, layout, 1); EXPECT_EQ(restarted_process.runOnePage(false, [] { return true; }).deleted, 1u); - EXPECT_TRUE(readGcMaintenanceState(backend, layout).state->janitor_cursor.empty()); + EXPECT_TRUE(readState(requests, layout).state->janitor_cursor.empty()); } TEST(CASNamespaceJanitor, TakesOneCatalogCutAfterListingAndContinuesPastMalformedKey) { - OrderedJanitorBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const auto dead = life("dead", 81); const String valid = layout.namespaceFilesPrefix(dead) + "data"; const String malformed = layout.namespaceStreamRootPrefix() + "not-a-life/_log/1-1.zst"; const String malformed_state = layout.namespaceStateRootPrefix() + "not-a-life/_ckpt"; - ASSERT_EQ(backend.putIfAbsent(valid, "v").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(malformed, "bad").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(malformed_state, "bad-state").outcome, PutOutcome::Done); - backend.resetCounts(); - backend.events.clear(); + ASSERT_EQ(backend->putIfAbsent(valid, "v").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(malformed, "bad").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(malformed_state, "bad-state").outcome, PutOutcome::Done); + backend->resetCounts(); + backend->events.clear(); - const auto result = NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }); + const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); EXPECT_FALSE(result.anomalies.empty()); - EXPECT_TRUE(backend.get(malformed)); - EXPECT_TRUE(backend.get(malformed_state)); - ASSERT_EQ(backend.events.size(), 2u); - EXPECT_EQ(backend.events[0], "list"); - EXPECT_EQ(backend.events[1], "catalog"); - EXPECT_EQ(backend.getCount(layout.refCatalogKey()), 1u); + EXPECT_TRUE(backend->get(malformed)); + EXPECT_TRUE(backend->get(malformed_state)); + ASSERT_EQ(backend->events.size(), 2u); + EXPECT_EQ(backend->events[0], "list"); + EXPECT_EQ(backend->events[1], "catalog"); + EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); } TEST(CASNamespaceJanitor, MalformedKeyIsFinalAndAdvancesCursor) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String first = layout.namespaceStreamRootPrefix() + "bad-a/_log/1-1.zst"; const String second = layout.namespaceStreamRootPrefix() + "bad-b/_log/1-1.zst"; - ASSERT_EQ(backend.putIfAbsent(first, "first").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(second, "second").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(first, "first").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(second, "second").outcome, PutOutcome::Done); const NamespaceJanitorResult result - = NamespaceJanitor(backend, layout, 1).runOnePage(false, [] { return true; }); + = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); EXPECT_FALSE(result.anomalies.empty()); - EXPECT_TRUE(backend.get(first)); - EXPECT_TRUE(backend.get(second)); - const GcMaintenanceReadResult progress = readGcMaintenanceState(backend, layout); + EXPECT_TRUE(backend->get(first)); + EXPECT_TRUE(backend->get(second)); + const GcMaintenanceReadResult progress = readState(requests, layout); ASSERT_EQ(progress.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(progress.state); EXPECT_FALSE(progress.state->janitor_cursor.empty()) @@ -419,58 +504,61 @@ TEST(CASNamespaceJanitor, MalformedKeyIsFinalAndAdvancesCursor) TEST(CASNamespaceJanitor, DuplicateCurrentLifeSuppressesWholePage) { - CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); RefCatalog catalog; catalog.entries = { CatalogEntry{.ns = RootNamespace{"a"}, .state = NsState::Live, .incarnation = UInt128{91}}, CatalogEntry{.ns = RootNamespace{"b"}, .state = NsState::Live, .incarnation = UInt128{91}}}; - seedCatalog(backend, layout, catalog); + seedCatalog(*backend, layout, catalog); const String dead_a = layout.refCkptKey(life("dead-a", 92)); const String dead_b = layout.refCkptKey(life("dead-b", 93)); - ASSERT_EQ(backend.putIfAbsent(dead_a, "a").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(dead_b, "b").outcome, PutOutcome::Done); - const auto result = NamespaceJanitor(backend, layout, 1).runOnePage(false, [] { return true; }); + ASSERT_EQ(backend->putIfAbsent(dead_a, "a").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(dead_b, "b").outcome, PutOutcome::Done); + const auto result = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); - EXPECT_EQ(backend.deleteTotal(), 0u); - EXPECT_TRUE(backend.get(dead_a)); - EXPECT_TRUE(backend.get(dead_b)); - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Absent) + EXPECT_EQ(backend->deleteTotal(), 0u); + EXPECT_TRUE(backend->get(dead_a)); + EXPECT_TRUE(backend->get(dead_b)); + EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "an ambiguous catalog cut leaves the selected page undecided for an authoritative retry"; } TEST(CASNamespaceJanitor, CorruptProgressResetsWithoutDeletingAndFilesOnlyOmittedCycleRetries) { - OmitFirstNamespacePageBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String dead = layout.namespaceFilesPrefix(life("dead", 101)) + "only-residue"; - ASSERT_EQ(backend.putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(layout.gcMaintenanceStateKey(), "corrupt").outcome, PutOutcome::Done); - EXPECT_EQ(NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }).deleted, 0u); - EXPECT_TRUE(backend.get(dead)); - EXPECT_EQ(readGcMaintenanceState(backend, layout).status, GcMaintenanceReadStatus::Valid); - EXPECT_EQ(NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }).deleted, 0u); - EXPECT_TRUE(backend.get(dead)); - EXPECT_EQ(NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }).deleted, 1u); - EXPECT_FALSE(backend.get(dead)); + ASSERT_EQ(backend->putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(layout.gcMaintenanceStateKey(), "corrupt").outcome, PutOutcome::Done); + EXPECT_EQ(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }).deleted, 0u); + EXPECT_TRUE(backend->get(dead)); + EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Valid); + EXPECT_EQ(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }).deleted, 0u); + EXPECT_TRUE(backend->get(dead)); + EXPECT_EQ(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }).deleted, 1u); + EXPECT_FALSE(backend->get(dead)); } TEST(CASNamespaceJanitor, ExactTokenMismatchRetainsConcurrentReplacement) { - ReplaceBeforeJanitorDeleteBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead-a", 111)); const String later = layout.refCkptKey(life("dead-b", 112)); - ASSERT_EQ(backend.putIfAbsent(dead, "old").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(later, "later").outcome, PutOutcome::Done); - const auto result = NamespaceJanitor(backend, layout, 1).runOnePage(false, [] { return true; }); + ASSERT_EQ(backend->putIfAbsent(dead, "old").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(later, "later").outcome, PutOutcome::Done); + const auto result = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); - ASSERT_TRUE(backend.get(dead)); - EXPECT_EQ(backend.get(dead)->bytes, "winner"); - EXPECT_TRUE(backend.get(later)); - const GcMaintenanceReadResult progress = readGcMaintenanceState(backend, layout); + ASSERT_TRUE(backend->get(dead)); + EXPECT_EQ(backend->get(dead)->bytes, "winner"); + EXPECT_TRUE(backend->get(later)); + const GcMaintenanceReadResult progress = readState(requests, layout); ASSERT_EQ(progress.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(progress.state); EXPECT_FALSE(progress.state->janitor_cursor.empty()) @@ -479,99 +567,125 @@ TEST(CASNamespaceJanitor, ExactTokenMismatchRetainsConcurrentReplacement) TEST(CASNamespaceJanitor, TokenlessListHeadsDeadKeysAndRetainsConcurrentReplacement) { - TokenlessListBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); const CatalogEntry current{ .ns = RootNamespace{"current"}, .state = NsState::Live, .incarnation = UInt128{161}}; - seedCatalog(backend, layout, RefCatalog{.entries = {current}}); + seedCatalog(*backend, layout, RefCatalog{.entries = {current}}); const String live_key = layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(current.ns, current.incarnation)); const String dead_key = layout.refCkptKey(life("dead", 162)); const String raced_key = layout.namespaceFilesPrefix(life("raced", 163)) + "data"; - ASSERT_EQ(backend.putIfAbsent(live_key, "live").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(dead_key, "dead").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(raced_key, "old").outcome, PutOutcome::Done); - backend.replace_on_head = raced_key; - backend.resetCounts(); + ASSERT_EQ(backend->putIfAbsent(live_key, "live").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(dead_key, "dead").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(raced_key, "old").outcome, PutOutcome::Done); + backend->replace_on_head = raced_key; + backend->resetCounts(); - const auto result = NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }); + const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); EXPECT_TRUE(result.anomalies.empty()); - EXPECT_TRUE(backend.get(live_key)); - EXPECT_FALSE(backend.get(dead_key)); - ASSERT_TRUE(backend.get(raced_key)); - EXPECT_EQ(backend.get(raced_key)->bytes, "winner"); - EXPECT_EQ(backend.headCount(live_key), 0u); - EXPECT_EQ(backend.headCount(dead_key), 1u); - EXPECT_EQ(backend.headCount(raced_key), 1u); - EXPECT_EQ(backend.deleteCount(dead_key), 1u); - EXPECT_EQ(backend.deleteCount(raced_key), 1u); + EXPECT_TRUE(backend->get(live_key)); + EXPECT_FALSE(backend->get(dead_key)); + ASSERT_TRUE(backend->get(raced_key)); + EXPECT_EQ(backend->get(raced_key)->bytes, "winner"); + EXPECT_EQ(backend->headCount(live_key), 0u); + EXPECT_EQ(backend->headCount(dead_key), 1u); + EXPECT_EQ(backend->headCount(raced_key), 1u); + EXPECT_EQ(backend->deleteCount(dead_key), 1u); + EXPECT_EQ(backend->deleteCount(raced_key), 1u); } TEST(CASNamespaceJanitor, TokenlessListRechecksFenceAfterHeadBeforeDelete) { - FenceLossDuringHeadBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String dead_key = layout.refCkptKey(life("dead", 164)); - ASSERT_EQ(backend.putIfAbsent(dead_key, "dead").outcome, PutOutcome::Done); - backend.resetCounts(); + ASSERT_EQ(backend->putIfAbsent(dead_key, "dead").outcome, PutOutcome::Done); + backend->resetCounts(); - const auto result = NamespaceJanitor(backend, layout, 100).runOnePage( - false, [&] { return backend.fence_held; }); + const auto result = NamespaceJanitor(requests, layout, 100).runOnePage( + false, [&] { return backend->fence_held; }); EXPECT_EQ(result.deleted, 0u); - EXPECT_EQ(backend.headCount(dead_key), 1u); - EXPECT_EQ(backend.deleteCount(dead_key), 0u); - EXPECT_TRUE(backend.get(dead_key)); + EXPECT_EQ(backend->headCount(dead_key), 1u); + EXPECT_EQ(backend->deleteCount(dead_key), 0u); + EXPECT_TRUE(backend->get(dead_key)); } TEST(CASNamespaceJanitor, PostListCatalogCutProtectsConcurrentCreationWithOneGet) { const auto created = life("created", 121); - CatalogAfterListBackend backend(created); + auto backend = std::make_shared(created); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String first = layout.refCkptKey(created); const String second = layout.namespaceFilesPrefix(created) + "data"; - ASSERT_EQ(backend.putIfAbsent(first, "ckpt").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(second, "file").outcome, PutOutcome::Done); - backend.resetCounts(); - const auto result = NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }); + ASSERT_EQ(backend->putIfAbsent(first, "ckpt").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(second, "file").outcome, PutOutcome::Done); + backend->resetCounts(); + const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); - EXPECT_EQ(backend.deleteTotal(), 0u); - EXPECT_EQ(backend.getCount(layout.refCatalogKey()), 1u); - EXPECT_TRUE(backend.get(first)); - EXPECT_TRUE(backend.get(second)); + EXPECT_EQ(backend->deleteTotal(), 0u); + EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); + EXPECT_TRUE(backend->get(first)); + EXPECT_TRUE(backend->get(second)); } TEST(CASNamespaceJanitor, BackendRejectedCursorResetsExactlyAndDeletesNothing) { - RejectCursorBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead", 131)); - ASSERT_EQ(backend.putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent(layout.gcMaintenanceStateKey(), + ASSERT_EQ(backend->putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); + ASSERT_EQ(backend->putIfAbsent(layout.gcMaintenanceStateKey(), encodeGcMaintenanceState({.janitor_cursor = "rejected"})).outcome, PutOutcome::Done); - EXPECT_THROW(NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }), std::runtime_error); - EXPECT_EQ(backend.deleteTotal(), 0u); - EXPECT_TRUE(backend.get(dead)); - EXPECT_TRUE(readGcMaintenanceState(backend, layout).state->janitor_cursor.empty()); + EXPECT_THROW(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }), std::runtime_error); + EXPECT_EQ(backend->deleteTotal(), 0u); + EXPECT_TRUE(backend->get(dead)); + EXPECT_TRUE(readState(requests, layout).state->janitor_cursor.empty()); } TEST(CASNamespaceJanitor, CursorPublicationFailureIsLeakOnly) { - FailMaintenancePublicationBackend backend; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); const Layout layout("p"); - seedCatalog(backend, layout); + seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead", 141)); - ASSERT_EQ(backend.putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); - backend.fail_publication = true; - const auto result = NamespaceJanitor(backend, layout, 100).runOnePage(false, [] { return true; }); + ASSERT_EQ(backend->putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); + backend->fail_publication = true; + const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); EXPECT_FALSE(result.anomalies.empty()); - EXPECT_FALSE(backend.get(dead)); + EXPECT_FALSE(backend->get(dead)); +} + +/// The write inside `catch (...)` (the reset after a LIST failure) is admitted `once`: an unmodeled, +/// unresolvable write attempt must give up after its one exact resolve read rather than looping through +/// `Retry::standard()`'s backoff. `write_attempts == 1` is the discriminator: under `standard`, the same +/// unresolvable write would keep reissuing until the ninety-second policy window (the LIST failure is +/// itself a genuine `NETWORK_ERROR`, which the read engine retries to its OWN deadline before this catch +/// path is even entered -- so a raw sleep count is not a usable signal here, it is dirtied by the LIST's +/// unrelated retries regardless of which policy the catch-path write uses; the injected clock exists only +/// to keep both retry loops instant rather than to prove anything by its own emptiness). +TEST(CASGcMaintenanceState, CatchPathWriteIsOnce) +{ + auto backend = std::make_shared(); + FakeClock clock; + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn()); + const Layout layout("p"); + + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, + [&] { (void)NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); }); + EXPECT_EQ(backend->write_attempts, 1u) + << "the catch-path reset settles by its one resolve read and gives up rather than reissuing"; } TEST(CASNamespaceJanitorIntegration, RegularGcRoundDeletesDeadNamespaceBytes) diff --git a/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp b/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp index 7acab161e63d..7fee0407e45b 100644 --- a/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp +++ b/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp @@ -30,18 +30,6 @@ namespace DB::ErrorCodes namespace { -/// A fence that never refuses, for tests whose subject is not the fence -- same helper, same intent, -/// as `gtest_cas_ref_ckpt.cpp`'s identically-named constant (not shared: each `_ckpt`/catalog test file -/// defines its own copy, matching that file's own precedent). -const std::function ALWAYS_ADMITTED = [](uint64_t) {}; - -/// A deadline far enough out that only the test's own contention decides the outcome -- mirrors -/// `gtest_cas_ref_ckpt.cpp`'s `generousDeadline`. -CkptDeadline generousDeadline() -{ - return CkptDeadline{[] { return uint64_t{1000}; }, 60000}; -} - CreatorFence creatorFence(const String & srid, uint64_t writer_epoch, uint64_t fence_generation = 1) { return CreatorFence{.server_root_id = srid, .writer_epoch = writer_epoch, .fence_generation = fence_generation}; @@ -63,16 +51,65 @@ const CatalogEntry * findEntryForTest(const RefCatalog & catalog, const RootName return nullptr; } -/// Raw lifecycle tests operate below `Pool::open`, so model an already-bootstrapped pool explicitly. -class InitializedCatalogBackend : public InMemoryBackend +/// Withdraws an operation's admission, and lets a test smuggle a real concurrent write, at one chosen +/// point of the creation sequence: once this namespace's `_ckpt` is durable (step 2 landed, step 3 has +/// not run), or inside step 3's own read-then-write window. The `_ckpt` key carries an incarnation a +/// test cannot know before the creation mints it, so that arm names the object kind rather than a key. +/// +/// The read arm fires BEFORE the store is consulted, so the body the caller's `decide` receives already +/// carries whatever the hook wrote. That is what lets a test make the observed body stale on BOTH axes +/// -- a changed entry and a withdrawn admission -- inside ONE `decide` invocation, which is the only +/// place the two can be told apart. +class CreationHookBackend : public InMemoryBackend { public: - InitializedCatalogBackend() + bool admitted = true; + /// Withdraw once any `_ckpt` key has been written. + bool withdraw_after_ckpt_write = false; + /// Fires once before this key is read, then `withdraw_on_read` is applied. + String hook_before_read_of; + std::function on_read; + bool withdraw_on_read = false; + + std::optional read(const String & key, TransportAccess & access) override + { + if (!hook_before_read_of.empty() && key == hook_before_read_of && !hook_fired) + { + /// Latched before running: the hook reads and writes through this same backend, and an + /// unguarded re-entry would run the test's concurrent actor again against its own result. + hook_fired = true; + if (on_read) + on_read(); + if (withdraw_on_read) + admitted = false; + } + return InMemoryBackend::read(key, access); + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - CasRefCatalog::initializeEmptyForNewPool(*this, Layout("p")); + auto result = InMemoryBackend::write(key, bytes, expected_value, access); + if (withdraw_after_ckpt_write && Layout{"p"}.parseRefCkptKey(key)) + admitted = false; + return result; } + +private: + bool hook_fired = false; }; +/// Raw lifecycle tests operate below `Pool::open`, so model an already-bootstrapped pool explicitly. +std::shared_ptr initializedCatalogBackend() +{ + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(op, Layout("p")); + return backend; +} + } /// --------------------------------------------------------------------------------------------- @@ -81,16 +118,18 @@ class InitializedCatalogBackend : public InMemoryBackend TEST(CASNsCreationLifecycle, HappyPathReachesLiveWithADurableCkptAndAStableIncarnation) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence creator = creatorFence("srv1", /*writer_epoch=*/5); const auto outcome = CasRefCatalog::createNamespace( - backend, layout, 1, ns, creator, /*admitted_generation=*/1, ALWAYS_ADMITTED, generousDeadline()); + op, layout, 1, ns, creator); EXPECT_EQ(outcome, CasRefCatalog::NamespaceCreationOutcome::Live); - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); const CatalogEntry * entry = findEntryForTest(snap.catalog, ns); ASSERT_NE(entry, nullptr); EXPECT_EQ(entry->state, NsState::Live); @@ -98,12 +137,12 @@ TEST(CASNsCreationLifecycle, HappyPathReachesLiveWithADurableCkptAndAStableIncar const UInt128 incarnation = entry->incarnation; EXPECT_NE(incarnation, UInt128(0)); - const std::optional ckpt = readCkpt(backend, layout, NamespaceLifeId::fromCatalogEntry(entry->ns, incarnation)); + const std::optional ckpt = readCkpt(op, layout, NamespaceLifeId::fromCatalogEntry(entry->ns, incarnation)); ASSERT_TRUE(ckpt.has_value()) << "step 2's _ckpt must be durable"; EXPECT_EQ(ckpt->ckpt.life_epoch, 5u) << "INV-4's genesis epoch is the creator's writer_epoch"; /// Re-reading the catalog again must show the SAME incarnation -- nothing mints a second one. - EXPECT_EQ(CasRefCatalog::read(backend, layout).catalog.entries.at(0).incarnation, incarnation); + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries.at(0).incarnation, incarnation); } /// --------------------------------------------------------------------------------------------- @@ -114,16 +153,18 @@ TEST(CASNsCreationLifecycle, HappyPathReachesLiveWithADurableCkptAndAStableIncar #ifndef DEBUG_OR_SANITIZER_BUILD TEST(CASNsCreationLifecycle, CreateNamespaceRejectsAnAlreadyExistingEntry) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence creator = creatorFence("srv1", 1); - ASSERT_EQ(CasRefCatalog::createNamespace(backend, layout, 1, ns, creator, 1, ALWAYS_ADMITTED, generousDeadline()), + ASSERT_EQ(CasRefCatalog::createNamespace(op, layout, 1, ns, creator), CasRefCatalog::NamespaceCreationOutcome::Live); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { - CasRefCatalog::createNamespace(backend, layout, 1, ns, creatorFence("srv2", 2), 1, ALWAYS_ADMITTED, generousDeadline()); + CasRefCatalog::createNamespace(op, layout, 1, ns, creatorFence("srv2", 2)); }); } #endif @@ -131,16 +172,18 @@ TEST(CASNsCreationLifecycle, CreateNamespaceRejectsAnAlreadyExistingEntry) #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASNsCreationLifecycleDeathTest, CreateNamespaceRejectsAnAlreadyExistingEntryAborts) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence creator = creatorFence("srv1", 1); - ASSERT_EQ(CasRefCatalog::createNamespace(backend, layout, 1, ns, creator, 1, ALWAYS_ADMITTED, generousDeadline()), + ASSERT_EQ(CasRefCatalog::createNamespace(op, layout, 1, ns, creator), CasRefCatalog::NamespaceCreationOutcome::Live); EXPECT_DEATH( { - CasRefCatalog::createNamespace(backend, layout, 1, ns, creatorFence("srv2", 2), 1, ALWAYS_ADMITTED, generousDeadline()); + CasRefCatalog::createNamespace(op, layout, 1, ns, creatorFence("srv2", 2)); }, "already carries a catalog entry"); } @@ -173,34 +216,26 @@ TEST(CASNsCreationLifecycle, LiveAndRemovingAndAbsentAllAdmitPublication) /// ZombieGoLive: fenced-out between the `_ckpt` publish and the `Creating -> Live` CAS /// --------------------------------------------------------------------------------------------- -/// A fence callback that admits its FIRST call (spent by step 2's `publishCkpt`) and refuses every -/// call after (spent by step 3's `mutate`) -- deterministically reproducing "fenced out between the -/// `_ckpt` create and the `Creating -> Live` CAS" without a second thread or fault injection. -namespace -{ -std::function admittedOnceThenFenced() -{ - auto calls = std::make_shared(0); - return [calls](uint64_t admitted) - { - if (++*calls > 1) - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "fence generation moved since admission ({})", admitted); - }; -} -} - TEST(CASNsCreationLifecycle, FencedOutBetweenTheCkptPublishAndGoLiveRefusesAndLeavesEntryCreating) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence creator = creatorFence("srv1", 5); - const auto outcome = CasRefCatalog::createNamespace( - backend, layout, 1, ns, creator, /*admitted_generation=*/1, admittedOnceThenFenced(), generousDeadline()); + /// Admission is withdrawn the instant step 2's `_ckpt` is durable, so step 3 never runs -- + /// "fenced out between the `_ckpt` create and the `Creating -> Live` write", without a second + /// thread. + backend->withdraw_after_ckpt_write = true; + CasOperation creating_op = requests.admit([&backend] { return backend->admitted; }); + const auto outcome = CasRefCatalog::createNamespace(creating_op, layout, 1, ns, creator); EXPECT_EQ(outcome, CasRefCatalog::NamespaceCreationOutcome::FencedOut); + backend->withdraw_after_ckpt_write = false; + backend->admitted = true; - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); const CatalogEntry * entry = findEntryForTest(snap.catalog, ns); ASSERT_NE(entry, nullptr); EXPECT_EQ(entry->state, NsState::Creating) << "step 3 never ran its CAS -- ZombieGoLive refuses before sending it"; @@ -210,7 +245,7 @@ TEST(CASNsCreationLifecycle, FencedOutBetweenTheCkptPublishAndGoLiveRefusesAndLe /// Step 2's _ckpt DID land (it is not what the fence check gates) -- CKPT-FAILED-BIRTH-DEBRIS is a /// different mechanism (the OLD `RefOpKind::NamespaceBirth` writer, `Pool/CasRefLedger.cpp`); this /// driver's own `_ckpt` is simply left in place for whichever actor next reconciles this entry. - EXPECT_TRUE(readCkpt(backend, layout, NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation)).has_value()); + EXPECT_TRUE(readCkpt(op, layout, NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation)).has_value()); } /// Regression (CI PR#2073, `03611_freeze_partition_parallel_verbose` under `amd_tsan, cas s3 storage`): @@ -220,28 +255,33 @@ TEST(CASNsCreationLifecycle, FencedOutBetweenTheCkptPublishAndGoLiveRefusesAndLe /// must send the loser back through the resume loop (`Superseded`), never abort the server. TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsStillCreatingEntryReportsSupersededNotAbort) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence winner = creatorFence("srv1", 1); /// Leaves the entry in `Creating` without reaching `Live` -- the same shape `resolveNamespaceLife` /// observes when a sibling thread's `casAdmitEntry` has landed but its `completeCreation` has not. - const auto winner_outcome = CasRefCatalog::createNamespace( - backend, layout, 1, ns, winner, /*admitted_generation=*/1, admittedOnceThenFenced(), generousDeadline()); + backend->withdraw_after_ckpt_write = true; + CasOperation winner_op = requests.admit([&backend] { return backend->admitted; }); + const auto winner_outcome = CasRefCatalog::createNamespace(winner_op, layout, 1, ns, winner); ASSERT_EQ(winner_outcome, CasRefCatalog::NamespaceCreationOutcome::FencedOut); - ASSERT_EQ(CasRefCatalog::read(backend, layout).catalog.entries.at(0).state, NsState::Creating); + backend->withdraw_after_ckpt_write = false; + backend->admitted = true; + ASSERT_EQ(CasRefCatalog::read(op, layout).catalog.entries.at(0).state, NsState::Creating); /// The loser: a second call, as if a sibling thread's own outer "no entry" read had raced ahead of /// this one. Same fence as the winner (sibling threads of one query share a mount's fence) -- /// exercising exactly the case `resolveNamespaceLife`'s "own fence -> completeCreation" branch is /// built to resume, never a `LOGICAL_ERROR` abort. const auto loser_outcome = CasRefCatalog::createNamespace( - backend, layout, 1, ns, winner, /*admitted_generation=*/1, ALWAYS_ADMITTED, generousDeadline()); + op, layout, 1, ns, winner); EXPECT_EQ(loser_outcome, CasRefCatalog::NamespaceCreationOutcome::Superseded); /// Nothing about the winner's own still-`Creating` entry was disturbed by the loser's refused call. - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); const CatalogEntry * entry = findEntryForTest(snap.catalog, ns); ASSERT_NE(entry, nullptr); EXPECT_EQ(entry->state, NsState::Creating); @@ -260,7 +300,9 @@ TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsStillCreatingEntryRep /// single upfront read) can catch it. TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsFullCreateBetweenPreCheckAndStep1ReportsSupersededNotAbort) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence winner = creatorFence("srv1", 1); @@ -275,17 +317,17 @@ TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsFullCreateBetweenPreC CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest([&] { const auto winner_outcome = CasRefCatalog::createNamespace( - backend, layout, 1, ns, winner, /*admitted_generation=*/1, ALWAYS_ADMITTED, generousDeadline()); + op, layout, 1, ns, winner); ASSERT_EQ(winner_outcome, CasRefCatalog::NamespaceCreationOutcome::Live); }); const auto loser_outcome = CasRefCatalog::createNamespace( - backend, layout, 1, ns, loser, /*admitted_generation=*/1, ALWAYS_ADMITTED, generousDeadline()); + op, layout, 1, ns, loser); EXPECT_EQ(loser_outcome, CasRefCatalog::NamespaceCreationOutcome::Superseded); /// Exactly one row for `ns`, owned by the winner, at `Live` -- the loser's refused admission left /// no trace and did not disturb it. - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); size_t rows_for_ns = 0; for (const CatalogEntry & e : snap.catalog.entries) if (e.ns.string() == ns.string()) @@ -298,12 +340,14 @@ TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsFullCreateBetweenPreC } /// --------------------------------------------------------------------------------------------- -/// Token-stale: the observed entry no longer matches at the `Creating -> Live` CAS +/// Entry-stale: the observed entry no longer matches at the `Creating -> Live` write /// --------------------------------------------------------------------------------------------- TEST(CASNsCreationLifecycle, EntryStolenByAConcurrentReconcilerRefusesGoLiveAndLeavesTheStolenEntryAlone) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence original_creator = creatorFence("srv1", 5); @@ -311,32 +355,30 @@ TEST(CASNsCreationLifecycle, EntryStolenByAConcurrentReconcilerRefusesGoLiveAndL /// Write 1 only -- models "crash after write 1": no _ckpt yet, entry still Creating. const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(42), .creator = original_creator}; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); - - /// `check_fence_or_throw` is the seam this driver calls on EVERY attempt -- once inside step 2's - /// `publishCkpt`, once more inside step 3's own `mutate` -- so smuggling a REAL concurrent write - /// into it (rather than faking the outcome) has to land on the SECOND call specifically, or the - /// steal itself would run twice (and the second run would see its own first result and refuse). - /// This reproduces "stolen between the creator's _ckpt publish and its Creating -> Live CAS" - /// without a second thread. The steal itself must succeed (asserted), so the mismatch - /// `completeCreation` sees below is the entry ACTUALLY changing, not a contrived stub. - auto calls = std::make_shared(0); - const std::function steal_before_the_go_live_cas = [&, calls](uint64_t) + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); + + /// A REAL concurrent write, smuggled in just before step 3's own catalog read -- the first one + /// `completeCreation` performs, since step 2 touches only the `_ckpt`. `decide` therefore receives + /// the POST-steal body and refuses it without sending anything, which is what the entry check is + /// for. The steal itself must succeed (asserted), so the mismatch is the entry ACTUALLY changing, + /// not a contrived stub. It runs on its own operation, because it is a different actor. + CasOperation thief_op = requests.admit(); + backend->hook_before_read_of = layout.refCatalogKey(); + backend->on_read = [&] { - if (++*calls == 2) - ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(backend, layout, entry, thief, fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED), - CasRefCatalog::ReconcileCreatorOutcome::Reconciled); + ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(thief_op, layout, entry, thief, fixedTerminality(true)), + CasRefCatalog::ReconcileCreatorOutcome::Reconciled); }; - const auto outcome = CasRefCatalog::completeCreation( - backend, layout, entry, /*admitted_generation=*/1, steal_before_the_go_live_cas, generousDeadline()); + const auto outcome = CasRefCatalog::completeCreation(op, layout, entry); EXPECT_EQ(outcome, CasRefCatalog::NamespaceCreationOutcome::Superseded); + backend->on_read = nullptr; /// `read`'s `Snapshot` is bound to a name here, not chained through a temporary: a `const /// CatalogEntry *` taken from `.catalog` of an unbound temporary dangles the instant the full /// expression ends, which every other site in this file (and the copy/paste that spread it) got /// wrong until ASan caught it. - const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(op, layout); const CatalogEntry * after = findEntryForTest(snap_after.catalog, ns); ASSERT_NE(after, nullptr); EXPECT_EQ(after->state, NsState::Creating) << "the ORIGINAL creator's attempt wrote nothing -- only the thief's CAS did"; @@ -351,37 +393,45 @@ TEST(CASNsCreationLifecycle, EntryStolenByAConcurrentReconcilerRefusesGoLiveAndL TEST(CASNsCreationLifecycle, BothFenceAndEntryStaleRefusesGoLiveViaTheFenceCheckWhichRunsFirst) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence original_creator = creatorFence("srv1", 5); const CreatorFence thief = creatorFence("srv2", 9); const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(42), .creator = original_creator}; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); - - /// Same steal as the test above, landing on the SECOND `check_fence_or_throw` call (step 3's own - /// `mutate`, not step 2's `publishCkpt`) -- but this one ALSO throws on that same second call, so - /// both axes go stale in the SAME `mutate` invocation. `completeCreation`'s fence check runs before - /// its entry check (documented ordering), so this is reported `FencedOut`; the assertions below - /// confirm the entry ALSO changed, so the test is not merely re-proving the fence-only case above. - auto calls = std::make_shared(0); - const std::function steal_and_fence_before_the_go_live_cas = [&, calls](uint64_t admitted) + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); + + /// The same steal as the test above and in the same window, but this one ALSO withdraws admission + /// there. Because the hook runs BEFORE the read, the single `decide` invocation that follows sees a + /// body that is stale on both axes at once -- and that is the only situation in which the two + /// checks are distinguishable. `completeCreation` consults admission before it compares the entry, + /// so the answer is `FencedOut`. + /// + /// This is what makes the test discriminate rather than merely pass: delete the `op.admitted()` + /// check from that `mutate` and the entry check answers `Superseded` instead, because `decide` + /// refuses the stale entry before any write is sent and the engine's own gate never speaks. The + /// assertions below confirm the entry really did change too. + CasOperation thief_op = requests.admit(); + CasOperation creator_op = requests.admit([&backend] { return backend->admitted; }); + backend->hook_before_read_of = layout.refCatalogKey(); + backend->withdraw_on_read = true; + backend->on_read = [&] { - if (++*calls == 2) - { - ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(backend, layout, entry, thief, fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED), - CasRefCatalog::ReconcileCreatorOutcome::Reconciled); - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "fence generation moved since admission ({})", admitted); - } + ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(thief_op, layout, entry, thief, fixedTerminality(true)), + CasRefCatalog::ReconcileCreatorOutcome::Reconciled); }; - const auto outcome = CasRefCatalog::completeCreation( - backend, layout, entry, /*admitted_generation=*/1, steal_and_fence_before_the_go_live_cas, generousDeadline()); + const auto outcome = CasRefCatalog::completeCreation(creator_op, layout, entry); EXPECT_EQ(outcome, CasRefCatalog::NamespaceCreationOutcome::FencedOut) - << "both checks would refuse; the fence check speaks first by this driver's fixed ordering"; + << "both would refuse; admission speaks first by the documented ordering"; + backend->on_read = nullptr; + backend->withdraw_on_read = false; + backend->admitted = true; - const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(op, layout); const CatalogEntry * after = findEntryForTest(snap_after.catalog, ns); ASSERT_NE(after, nullptr); ASSERT_TRUE(after->creator.has_value()); @@ -394,18 +444,20 @@ TEST(CASNsCreationLifecycle, BothFenceAndEntryStaleRefusesGoLiveViaTheFenceCheck TEST(CASNsCreationLifecycle, ReconcileRefusedWhileTheOriginalCreatorFenceIsStillLive) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(7), .creator = creatorFence("srv1", 5)}; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); const auto outcome = CasRefCatalog::reconcileStaleCreator( - backend, layout, entry, creatorFence("srv2", 9), fixedTerminality(false), /*admitted_generation=*/1, ALWAYS_ADMITTED); + op, layout, entry, creatorFence("srv2", 9), fixedTerminality(false)); EXPECT_EQ(outcome, CasRefCatalog::ReconcileCreatorOutcome::CreatorFenceStillLive); - const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(op, layout); const CatalogEntry * after = findEntryForTest(snap_after.catalog, ns); ASSERT_NE(after, nullptr); EXPECT_EQ(*after, entry) << "refused -- nothing written"; @@ -413,34 +465,36 @@ TEST(CASNsCreationLifecycle, ReconcileRefusedWhileTheOriginalCreatorFenceIsStill TEST(CASNsCreationLifecycle, ReconcileSucceedsTokenExactlyAfterTheOriginalCreatorFenceIsTerminalThenResumesToLive) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CreatorFence original_creator = creatorFence("srv1", 5); const CreatorFence new_creator = creatorFence("srv2", 9); const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(7), .creator = original_creator}; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); /// "crash after write 1" -- no _ckpt yet + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); /// "crash after write 1" -- no _ckpt yet - ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(backend, layout, entry, new_creator, fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED), + ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(op, layout, entry, new_creator, fixedTerminality(true)), CasRefCatalog::ReconcileCreatorOutcome::Reconciled); CatalogEntry taken_over = entry; taken_over.creator = new_creator; - const CasRefCatalog::Snapshot snap_mid = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap_mid = CasRefCatalog::read(op, layout); const CatalogEntry * mid = findEntryForTest(snap_mid.catalog, ns); ASSERT_NE(mid, nullptr); EXPECT_EQ(*mid, taken_over) << "creator moved to the new actor; state and incarnation unchanged"; const auto outcome = CasRefCatalog::completeCreation( - backend, layout, taken_over, /*admitted_generation=*/1, ALWAYS_ADMITTED, generousDeadline()); + op, layout, taken_over); EXPECT_EQ(outcome, CasRefCatalog::NamespaceCreationOutcome::Live); - const CasRefCatalog::Snapshot snap_final = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap_final = CasRefCatalog::read(op, layout); const CatalogEntry * final_entry = findEntryForTest(snap_final.catalog, ns); ASSERT_NE(final_entry, nullptr); EXPECT_EQ(final_entry->state, NsState::Live); EXPECT_EQ(final_entry->incarnation, entry.incarnation) << "the SAME incarnation throughout -- resumption, not rebirth"; - const std::optional ckpt = readCkpt(backend, layout, NamespaceLifeId::fromCatalogEntry(final_entry->ns, final_entry->incarnation)); + const std::optional ckpt = readCkpt(op, layout, NamespaceLifeId::fromCatalogEntry(final_entry->ns, final_entry->incarnation)); ASSERT_TRUE(ckpt.has_value()); EXPECT_EQ(ckpt->ckpt.life_epoch, new_creator.writer_epoch) << "the RESUMING actor's writer_epoch is the genesis epoch that actually landed"; @@ -450,26 +504,28 @@ TEST(CASNsCreationLifecycle, ReconcileSucceedsTokenExactlyAfterTheOriginalCreato /// the SAME stale `observed` before either writes. TEST(CASNsCreationLifecycle, ReconcileFailsClosedWhenTheEntryAlreadyChanged) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const RootNamespace ns{"a"}; const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(7), .creator = creatorFence("srv1", 5)}; - CasRefCatalog::casAdmitEntry(backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); const CreatorFence first_reconciler = creatorFence("srv2", 9); const CreatorFence second_reconciler = creatorFence("srv3", 11); - ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(backend, layout, entry, first_reconciler, fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED), + ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(op, layout, entry, first_reconciler, fixedTerminality(true)), CasRefCatalog::ReconcileCreatorOutcome::Reconciled); /// The second reconciler still holds the ORIGINAL `entry` it read before either of them wrote -- /// token-exactness must refuse it even though the terminality predicate would still say yes. const auto outcome = CasRefCatalog::reconcileStaleCreator( - backend, layout, entry, second_reconciler, fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED); + op, layout, entry, second_reconciler, fixedTerminality(true)); EXPECT_EQ(outcome, CasRefCatalog::ReconcileCreatorOutcome::EntryChanged); - const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(op, layout); const CatalogEntry * after = findEntryForTest(snap_after.catalog, ns); ASSERT_NE(after, nullptr); ASSERT_TRUE(after->creator.has_value()); @@ -484,23 +540,27 @@ TEST(CASNsCreationLifecycle, ReconcileFailsClosedWhenTheEntryAlreadyChanged) #ifndef DEBUG_OR_SANITIZER_BUILD TEST(CASNsCreationLifecycle, CompleteCreationRejectsANonCreatingEntry) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const CatalogEntry live{.ns = RootNamespace{"a"}, .state = NsState::Live, .incarnation = UInt128(1)}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { - CasRefCatalog::completeCreation(backend, layout, live, 1, ALWAYS_ADMITTED, generousDeadline()); + CasRefCatalog::completeCreation(op, layout, live); }); } TEST(CASNsCreationLifecycle, ReconcileStaleCreatorRejectsANonCreatingEntry) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const CatalogEntry live{.ns = RootNamespace{"a"}, .state = NsState::Live, .incarnation = UInt128(1)}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { - CasRefCatalog::reconcileStaleCreator(backend, layout, live, creatorFence("srv2", 2), fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED); + CasRefCatalog::reconcileStaleCreator(op, layout, live, creatorFence("srv2", 2), fixedTerminality(true)); }); } #endif @@ -508,22 +568,26 @@ TEST(CASNsCreationLifecycle, ReconcileStaleCreatorRejectsANonCreatingEntry) #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASNsCreationLifecycleDeathTest, CompleteCreationRejectsANonCreatingEntryAborts) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const CatalogEntry live{.ns = RootNamespace{"a"}, .state = NsState::Live, .incarnation = UInt128(1)}; EXPECT_DEATH( - { CasRefCatalog::completeCreation(backend, layout, live, 1, ALWAYS_ADMITTED, generousDeadline()); }, + { CasRefCatalog::completeCreation(op, layout, live); }, "not a Creating entry"); } TEST(CASNsCreationLifecycleDeathTest, ReconcileStaleCreatorRejectsANonCreatingEntryAborts) { - InitializedCatalogBackend backend; + auto backend = initializedCatalogBackend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const CatalogEntry live{.ns = RootNamespace{"a"}, .state = NsState::Live, .incarnation = UInt128(1)}; EXPECT_DEATH( { - CasRefCatalog::reconcileStaleCreator(backend, layout, live, creatorFence("srv2", 2), fixedTerminality(true), /*admitted_generation=*/1, ALWAYS_ADMITTED); + CasRefCatalog::reconcileStaleCreator(op, layout, live, creatorFence("srv2", 2), fixedTerminality(true)); }, "not a Creating entry"); } diff --git a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp index ba1b0925a20f..f97590ab3e12 100644 --- a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp @@ -78,16 +78,18 @@ TEST(CASNsFileIncarnation, ColdReaderUsesCatalogCutWhileOldFileSurvivesRemoval) ASSERT_TRUE(store->listNamespaceFiles(*old_life).empty()) << "precondition: enumeration omits the file, so no cleanup pass can ever find it"; const size_t holes_before_gc = backend->holesServed(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); store->dropNamespace(ns); - ASSERT_TRUE(CasRefCatalog::lifeIfCataloged(*backend, layout, ns)); + ASSERT_TRUE(CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns)); Gc gc(store, kGcId); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred) << "N: the production terminal must fold"; - ASSERT_TRUE(CasRefCatalog::lifeIfCataloged(*backend, layout, ns)) + ASSERT_TRUE(CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns)) << "the terminal fold alone must not erase its catalog row"; (void)runRegularRoundReclaiming(gc); - ASSERT_FALSE(CasRefCatalog::lifeIfCataloged(*backend, layout, ns)) + ASSERT_FALSE(CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns)) << "N+1: the pre-fold drain must erase the exact completed Removing row"; ASSERT_GT(backend->holesServed(), holes_before_gc) << "the GC janitor must observe the injected LIST hole after the explicit precondition LIST"; @@ -130,17 +132,19 @@ TEST(CASNsFileIncarnation, FreshReaderAssignsOnlyLiveCatalogLifeWithoutMutation) const RootNamespace live{"00/live@cas@"}; const RootNamespace removing{"00/removing@cas@"}; const RootNamespace absent{"00/absent@cas@"}; + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); - CasRefCatalog::casAdmitEntry(*backend, layout, 1, CatalogEntry{ + CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, CatalogEntry{ .ns = creating, .state = NsState::Creating, .incarnation = UInt128{31}, .creator = CreatorFence{.server_root_id = "foreign", .writer_epoch = 7, .fence_generation = 1}}); - CasRefCatalog::casAdmitEntry(*backend, layout, 1, CatalogEntry{ + CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, CatalogEntry{ .ns = live, .state = NsState::Live, .incarnation = UInt128{32}}); - CasRefCatalog::casAdmitEntry(*backend, layout, 1, CatalogEntry{ + CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, CatalogEntry{ .ns = removing, .state = NsState::Live, .incarnation = UInt128{33}}); - CasRefCatalog::casUpdate(*backend, layout, [&](const RefCatalog & current) + CasRefCatalog::casUpdate(catalog_op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find_if(next.entries.begin(), next.entries.end(), [&](const CatalogEntry & entry) @@ -177,7 +181,6 @@ TEST(CASNsFileIncarnation, FreshReaderAssignsOnlyLiveCatalogLifeWithoutMutation) EXPECT_EQ(store->refTableLifeForTest(live)->incarnation, UInt128{32}); EXPECT_EQ(backend->putTotal(), 0u); EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); } /// A real GC fold records terminal evidence for the previous life while its namespace-file debris @@ -198,7 +201,9 @@ TEST(CASNsFileIncarnation, RebirthDoesNotWaitForFilesToBeEmpty) remove_op.kind = RefOpKind::RemoveNamespace; appendRefLogSeed(*backend, layout, ns, {remove_op}); } - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, ns).value(); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns).value(); writeRecoverableCkptForRawFixture(*backend, layout, ns, RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, diff --git a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp index 77ff85d2b72d..5bd58de528d5 100644 --- a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp @@ -38,6 +38,11 @@ const String kFile = "format_version.txt"; const String kFilePath = kTablePath + "/" + kFile; const UInt128 kLife2Id = hexToU128("22222222222222222222222222222222"); +/// The erase entry point requires a liveness refresh because a real drain's liveness is a cached flag +/// its owner re-reads from the store. This fixture's operation carries no liveness at all, so there is +/// nothing cached for a refresh to update. +void noAuthorityRefresh() {} + struct DiskFixture { DB::ObjectStoragePtr object_storage; @@ -78,7 +83,9 @@ void deleteCatalogLife( { Backend & backend = storage.store()->backend(); const Layout & layout = storage.store()->layout(); - CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & current) + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find_if(next.entries.begin(), next.entries.end(), [&](const CatalogEntry & entry) @@ -93,7 +100,7 @@ void deleteCatalogLife( return next; }); - const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout); const auto it = std::find_if(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == life1.ns && entry.incarnation == life1.incarnation; @@ -106,9 +113,7 @@ void deleteCatalogLife( parent.ref_lives.emplace(life1.incarnation, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); - if (CasRefCatalog::deleteCompletedRemoving( - backend, layout, *it, parent, 1, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }) + if (CasRefCatalog::deleteCompletedRemoving(op, layout, *it, parent, noAuthorityRefresh) != CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted) throw DB::Exception( DB::ErrorCodes::LOGICAL_ERROR, "Failed to delete fixture catalog life '{}'", life1.ns.string()); @@ -121,8 +126,10 @@ NamespaceLifeId admitReplacementLife( if (life1.incarnation == kLife2Id) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Fixture life ids unexpectedly collide"); const NamespaceLifeId life2 = NamespaceLifeId::fromCatalogEntry(life1.ns, kLife2Id); + CasRequests requests = DB::Cas::tests::openRequestsForTest(storage.store()->backend()); + CasOperation op = requests.admit(); CasRefCatalog::casAdmitEntry( - storage.store()->backend(), storage.store()->layout(), storage.store()->poolConfig().gc_shards, CatalogEntry{ + op, storage.store()->layout(), storage.store()->poolConfig().gc_shards, CatalogEntry{ .ns = life2.ns, .state = NsState::Live, .incarnation = life2.incarnation}); return life2; } diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index d7430d321425..23ce4eba8341 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -50,21 +50,25 @@ class RenewalCounterBackend final : public InMemoryBackend LandThenThrow, }; - using InMemoryBackend::putOverwrite; - Fault fault = Fault::None; - PutResult putOverwrite( + /// The fault sits on the WRITE PRIMITIVE, and only on a CONDITIONAL one: the renewal issues + /// `op.replace`, which reaches the store here, and a create on the same key must not consume the + /// one-shot fault. + std::expected write( const String & key, const String & bytes, - const Token & expected, - const ObjectMeta & meta) override + const std::optional & expected_value, + TransportAccess & access) override { + if (!expected_value) + return InMemoryBackend::write(key, bytes, expected_value, access); + const Fault current = std::exchange(fault, Fault::None); if (current == Fault::ThrowBefore) throw Poco::TimeoutException("injected renewal timeout before commit"); - PutResult result = InMemoryBackend::putOverwrite(key, bytes, expected, meta); + auto result = InMemoryBackend::write(key, bytes, expected_value, access); if (current == Fault::LandThenThrow) throw Poco::TimeoutException("injected renewal response loss after commit"); return result; @@ -180,9 +184,11 @@ TEST(CASObservability, ExternalLeaseDeadlineCountsOnceWithoutReconstructingAttem .boot_ms_fn = [&] { return boot_ms; }, }); - /// The confirmed external safety deadline is 1080. At 1071 a ten-millisecond physical attempt - /// no longer fits, so the logical renewal ends without reconstructing a sent attempt. - boot_ms = 1071; + /// The fence deadline is 1100 and the safety margin 20, so admission refuses once fewer than + /// twenty milliseconds of lease remain. At 1090 nothing can be started and the logical renewal ends + /// without reconstructing a sent attempt. (Not 1071: the engine reserves the backend's own attempt + /// timeout, which is zero for an in-memory backend, so 29 ms of remaining lease is still room.) + boot_ms = 1090; const RenewalCounterSnapshot before = renewalCounters(); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); const RenewalCounterSnapshot after = renewalCounters(); @@ -354,10 +360,14 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) std::copy_if(seen.begin(), seen.end(), std::back_inserter(replaced_events), [&](const CasEvent & e){ return is_this_blob(e) && e.type == CasEventType::BlobRetireReplaced; }); ASSERT_EQ(replaced_events.size(), 1u) << "exactly one blob_retire_replaced for the supersede"; - EXPECT_EQ(replaced_events[0].token, hB.token.value) << "the event's own token is the fresh CURRENT token B"; + /// The event's token text is dialect-qualified ("emulated:", matching `Incarnation::render` + /// and `PersistedIncarnation`'s wire word) -- `Token::value` alone (from the legacy `head()` this + /// test reads hA/hB through) is only the bare value. + EXPECT_EQ(replaced_events[0].token, "emulated:" + hB.token.value) + << "the event's own token is the fresh CURRENT token B"; ASSERT_TRUE(replaced_events[0].detail.count("superseded_token")); EXPECT_FALSE(replaced_events[0].detail.at("superseded_token").empty()); - EXPECT_EQ(replaced_events[0].detail.at("superseded_token"), hA.token.value) + EXPECT_EQ(replaced_events[0].detail.at("superseded_token"), "emulated:" + hA.token.value) << "superseded_token must name the stale token (A) that republication replaced"; EXPECT_EQ(replaced_after - replaced_before, 1u) << "CASGCRetireReplaced increments exactly once"; diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index e73a37fb4030..8b2e8d4478ac 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -44,7 +44,9 @@ void seedConsumedSealCursor(InMemoryBackend & backend, const Layout & layout, co /// creation would have, rather than relying on the retired sentinel fallback. void seedEmptyRecoveryAuthority(InMemoryBackend & backend, const Layout & layout, const RootNamespace & ns) { - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(op, layout); const auto entry = std::find_if(catalog.catalog.entries.begin(), catalog.catalog.entries.end(), [&](const CatalogEntry & candidate) { return candidate.ns == ns; }); ASSERT_NE(entry, catalog.catalog.entries.end()); @@ -63,8 +65,6 @@ void seedEmptyRecoveryAuthority(InMemoryBackend & backend, const Layout & layout class CatalogChangingOnSecondReadBackend : public InMemoryBackend { public: - using Backend::get; - void arm(const Layout & layout, CatalogEntry predecessor_, CatalogEntry successor_) { catalog_key = layout.refCatalogKey(); @@ -76,11 +76,11 @@ class CatalogChangingOnSecondReadBackend : public InMemoryBackend bool didSwitch() const { return did_switch; } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (armed && key == catalog_key && ++catalog_reads == 2) { - const auto current = InMemoryBackend::get(key, range); + const auto current = InMemoryBackend::read(key, access); if (!current) throw std::runtime_error("test catalog disappeared"); RefCatalog next = decodeRefCatalog(current->bytes); @@ -88,11 +88,11 @@ class CatalogChangingOnSecondReadBackend : public InMemoryBackend if (it == next.entries.end()) throw std::runtime_error("test predecessor catalog row disappeared"); *it = successor; - if (InMemoryBackend::casPut(key, encodeRefCatalog(next), current->token).outcome != CasOutcome::Committed) + if (!InMemoryBackend::write(key, encodeRefCatalog(next), current->value, access).has_value()) throw std::runtime_error("test catalog replacement conflicted"); did_switch = true; } - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } private: @@ -110,10 +110,6 @@ class CatalogChangingOnSecondReadBackend : public InMemoryBackend class ReplacingManifestAfterObservationBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. - using InMemoryBackend::list; - using Backend::get; - void arm(const Layout & layout, String manifest_key_) { catalog_key = layout.refCatalogKey(); @@ -126,23 +122,23 @@ class ReplacingManifestAfterObservationBackend : public InMemoryBackend bool didReplace() const { return replaced_manifest; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - const ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); if (armed && prefix == manifests_prefix) listed_page = true; return page; } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - const auto result = InMemoryBackend::get(key, range); + auto result = InMemoryBackend::read(key, access); if (armed && listed_page && !replaced_manifest && key == catalog_key) { - const auto current = InMemoryBackend::get(manifest_key); + const auto current = InMemoryBackend::read(manifest_key, access); if (!current) throw std::runtime_error("test manifest disappeared before replacement"); - if (InMemoryBackend::casPut(manifest_key, current->bytes, current->token).outcome != CasOutcome::Committed) + if (!InMemoryBackend::write(manifest_key, current->bytes, current->value, access).has_value()) throw std::runtime_error("test manifest replacement conflicted"); replaced_manifest = true; } @@ -186,7 +182,9 @@ TEST(CASOrphanManifestSweep, CheckpointSnapshotAtOlderEpochSealSkipsDeletion) const Layout & layout = store->layout(); const RootNamespace ns{"00/sweep-checkpoint-base-seal@cas@"}; fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const RefLogTxn birth{ .ns = ns.string(), .txn_id = RefTxnId{1, 1}, .ops = {namespaceBirthOp()}, @@ -423,15 +421,17 @@ TEST(CASOrphanManifestSweep, CursorPageRefusesAmbiguousCatalogLifeIndex) seedConsumedSealCursor(*backend, store->layout(), ns); seedEmptyRecoveryAuthority(*backend, store->layout(), ns); - const CasRefCatalog::Snapshot before = CasRefCatalog::read(*backend, store->layout()); + CasOperation op = store->gcRequests().admit(); + const CasRefCatalog::Snapshot before = CasRefCatalog::read(op, store->layout()); RefCatalog damaged = before.catalog; CatalogEntry duplicate = damaged.entries.front(); duplicate.ns = RootNamespace{"00/ambiguous-life-twin@cas@"}; damaged.entries.push_back(duplicate); std::sort(damaged.entries.begin(), damaged.entries.end(), [](const CatalogEntry & lhs, const CatalogEntry & rhs) { return lhs.ns.string() < rhs.ns.string(); }); - ASSERT_EQ(backend->casPut(store->layout().refCatalogKey(), encodeRefCatalog(damaged), before.token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(before.incarnation.has_value()); + ASSERT_TRUE(std::holds_alternative( + op.replace(store->layout().refCatalogKey(), encodeRefCatalog(damaged), *before.incarnation, Retry::standard()))); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); }); @@ -467,8 +467,9 @@ TEST(CASOrphanManifestSweep, MissingRequiredCheckpointSuppressesDestructiveDecis setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, 6); seedConsumedSealCursor(*backend, store->layout(), ns); - const CatalogEntry entry = CasRefCatalog::read(*backend, store->layout()).catalog.entries.front(); - ASSERT_FALSE(readCkpt(*backend, store->layout(), NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation))); + CasOperation op = store->gcRequests().admit(); + const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); + ASSERT_FALSE(readCkpt(op, store->layout(), NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation))); sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}); @@ -488,7 +489,8 @@ TEST(CASOrphanManifestSweep, EpochSealFoldCursorCrossesTailByExactDecodedSuccess auto store = openPoolForTest(backend); const RootNamespace ns{"00/seal-cursor-tail@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - const CatalogEntry entry = CasRefCatalog::read(*backend, store->layout()).catalog.entries.front(); + CasOperation op = store->gcRequests().admit(); + const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); const ManifestRef removed{.writer_epoch = 1, .build_sequence = 5, .manifest_ordinal = 1}; @@ -536,7 +538,8 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip auto store = openPoolForTest(backend); const RootNamespace ns{"00/missing-next-epoch@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - const CatalogEntry entry = CasRefCatalog::read(*backend, store->layout()).catalog.entries.front(); + CasOperation op = store->gcRequests().admit(); + const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); const RefTxnId cursor{2, 2}; @@ -596,7 +599,8 @@ TEST(CASOrphanManifestSweep, CleanedCursorCrossesOnlyThroughExactImmediateEpochH auto store = openPoolForTest(backend); const RootNamespace ns{"00/exact-next-epoch@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - const CatalogEntry entry = CasRefCatalog::read(*backend, store->layout()).catalog.entries.front(); + CasOperation op = store->gcRequests().admit(); + const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); const RefTxnId cursor{2, 2}; @@ -642,7 +646,8 @@ TEST(CASOrphanManifestSweep, LaterCatalogCutCannotSpliceOwnershipAuthority) auto store = openPoolForTest(backend); const RootNamespace ns{"00/frozen-catalog-cut@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - const CatalogEntry predecessor = CasRefCatalog::read(*backend, store->layout()).catalog.entries.front(); + CasOperation op = store->gcRequests().admit(); + const CatalogEntry predecessor = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const ManifestRef r = ref(5, 0xB1); writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index 6285f171e3f3..664b9763b650 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -32,19 +32,19 @@ bool manifestExists(Backend & backend, const Layout & layout, const ManifestId & return backend.head(layout.manifestKey(id)).exists; } -bool activeSourceExists(Backend & backend, const Layout & layout, const UInt128 & source_id) +bool activeSourceExists(CasOperation & op, const Layout & layout, const UInt128 & source_id) { - const auto state_got = backend.get(layout.gcStateKey()); + const auto state_got = op.read(layout.gcStateKey(), Retry::standard()); if (!state_got) return false; const GcState state = decodeGcState(state_got->bytes); - const auto seal_got = backend.get(layout.foldSealKey(state.snap_generation, state.snap_attempt)); + const auto seal_got = op.read(layout.foldSealKey(state.snap_generation, state.snap_attempt), Retry::standard()); if (!seal_got) return false; const CasFoldSeal seal = decodeFoldSeal(seal_got->bytes); for (const RunRef & run : seal.blob_target_runs) { - SourceEdgeRunView view = openSourceEdgeRun(backend, run.key); + SourceEdgeRunView view = openSourceEdgeRun(op, run.key); String key; String payload; while (view.next(key, payload)) @@ -62,15 +62,15 @@ bool activeSourceExists(Backend & backend, const Layout & layout, const UInt128 return false; } -size_t condemnedCount(Backend & backend, const Layout & layout) +size_t condemnedCount(CasOperation & op, const Layout & layout) { size_t count = 0; - const GcState state = decodeGcState(backend.get(layout.gcStateKey())->bytes); + const GcState state = decodeGcState(op.read(layout.gcStateKey(), Retry::standard())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend.get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + op.read(layout.foldSealKey(state.snap_generation, state.snap_attempt), Retry::standard())->bytes); for (const RunRef & run : seal.blob_target_runs) { - SourceEdgeRunView view = openSourceEdgeRun(backend, run.key); + SourceEdgeRunView view = openSourceEdgeRun(op, run.key); String key; String payload; while (view.next(key, payload)) @@ -83,23 +83,24 @@ size_t condemnedCount(Backend & backend, const Layout & layout) class NominationBackend : public InMemoryBackend { public: - using Backend::deleteExact; - using Backend::get; - using Backend::putOverwrite; - - DeleteOutcome deleteExact(const String & key, const Token & token) override + /// The sweep's exact-token delete reaches the store through the keyed removal, so the fault is + /// armed there. It runs before the base call takes the backend's lock, so probing this same + /// backend from inside it is safe. + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { if (key == watched_manifest_key) { - source_absent_when_delete_started = !activeSourceExists(*this, layout, watched_source_id); + CasRequests probe_requests = openRequestsForTest(*this); + CasOperation probe = probe_requests.admit(); + source_absent_when_delete_started = !activeSourceExists(probe, layout, watched_source_id); if (replace_manifest_before_delete) { - const auto got = get(key); + const auto got = InMemoryBackend::read(key, access); if (got) - putOverwrite(key, got->bytes, got->token); + static_cast(InMemoryBackend::write(key, got->bytes, got->value, access)); } } - return InMemoryBackend::deleteExact(key, token); + return InMemoryBackend::remove(key, expected_value, access); } Layout layout{"p"}; @@ -187,8 +188,9 @@ ReadyFixture makeReadyFixture() const uint64_t new_attempt = state.snap_attempt + 1000; std::vector runs; RetiredMergeResult retired; + CasOperation seed_op = f.store->gcRequests().admit(); foldDeltasIntoGeneration( - *f.backend, f.store->layout(), seal.blob_target_runs, + seed_op, f.store->layout(), seal.blob_target_runs, new_generation, new_attempt, /*shard=*/0, std::move(seeded_edges), runs, /*current_round=*/state.round, /*condemn_round=*/state.round, {}, {}, {}, &retired, /*suppress_destructive=*/false, nullptr); @@ -197,7 +199,7 @@ ReadyFixture makeReadyFixture() seal.blob_target_runs = std::move(runs); seal.condemned_summary[0] = CondemnedSummary{}; putDeterministicArtifact( - *f.backend, f.store->layout().foldSealKey(new_generation, new_attempt), encodeFoldSeal(seal)); + seed_op, f.store->layout().foldSealKey(new_generation, new_attempt), encodeFoldSeal(seal)); state.snap_generation = new_generation; state.snap_attempt = new_attempt; f.backend->putOverwrite(f.store->layout().gcStateKey(), encodeGcState(state), state_got->token); @@ -227,14 +229,15 @@ TEST(CASOrphanNomination, RetiresExactManifestSourcesBeforeDelete) EXPECT_FALSE(manifestExists(*f.backend, f.store->layout(), f.candidate)); EXPECT_TRUE(f.backend->source_absent_when_delete_started) << "the adopted in-degree run must retire the manifest source before exact deletion begins"; + CasOperation op = f.store->gcRequests().admit(); for (size_t i = 0; i < f.blobs.size(); ++i) { EXPECT_FALSE(activeSourceExists( - *f.backend, f.store->layout(), sourceEdgeId(f.candidate, "blob-" + std::to_string(i)))); + op, f.store->layout(), sourceEdgeId(f.candidate, "blob-" + std::to_string(i)))); EXPECT_EQ(inDegreeInRuns(*f.backend, runsForShard(*f.backend, f.store->layout(), 0), f.blobs[i]), i < 4 ? 1 : 0); } - EXPECT_EQ(condemnedCount(*f.backend, f.store->layout()), 2u); + EXPECT_EQ(condemnedCount(op, f.store->layout()), 2u); ASSERT_TRUE(fold_reduce.has_value()); EXPECT_EQ(fold_reduce->metrics.at("unmatched_removes"), 0u) @@ -303,19 +306,21 @@ TEST(CASOrphanNomination, SuppressedRoundNominatesNothing) TEST(CASOrphanNomination, SourceRetirementIsAccountingNeutral) { InMemoryBackend backend; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const BlobRef blob = legacyMetaTestRef(UInt128(0xA001)); const UInt128 source = UInt128(0xA002); std::vector parent_runs; foldDeltasIntoGeneration( - backend, layout, {}, /*new_generation=*/1, /*attempt=*/1, /*shard=*/0, + op, layout, {}, /*new_generation=*/1, /*attempt=*/1, /*shard=*/0, {BlobDelta{.ref = blob, .source_id = source, .remove = false}}, parent_runs); std::vector next_runs; RetiredMergeResult retired; std::vector applied{0x5A}; foldDeltasIntoGeneration( - backend, layout, parent_runs, /*new_generation=*/2, /*attempt=*/2, /*shard=*/0, + op, layout, parent_runs, /*new_generation=*/2, /*attempt=*/2, /*shard=*/0, {}, next_runs, /*current_round=*/1, /*condemn_round=*/1, {}, {}, {}, &retired, /*suppress_destructive=*/false, &applied, {BlobSourceRetirement{.ref = blob, .source_id = source}, diff --git a/src/Disks/tests/gtest_cas_part_folder_access.cpp b/src/Disks/tests/gtest_cas_part_folder_access.cpp index dc7928080b4b..03d7ef33186b 100644 --- a/src/Disks/tests/gtest_cas_part_folder_access.cpp +++ b/src/Disks/tests/gtest_cas_part_folder_access.cpp @@ -66,33 +66,26 @@ Cas::CachedPartFolderAccess::CacheParams cacheOn() /// Every mutating backend op throws once armed — models a correlated backend outage during the /// transaction's compensating rollback (dropRef must append a removal, which mutates the backend). +/// While armed, the store is unreachable for every mutation: a transport-class failure, so the request +/// engine settles it by a read (which fails too) and reissues until the call's own retry window closes. +/// A test arming it therefore drives the engine's clock, or pays that window in real time. class RollbackFaultBackend final : public Cas::InMemoryBackend { public: std::atomic armed{false}; - Cas::PutResult putIfAbsent(const String & k, const String & b, const Cas::ObjectMeta & m) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + Cas::TransportAccess & access) override { failIfArmed(); - return InMemoryBackend::putIfAbsent(k, b, m); + return InMemoryBackend::write(key, bytes, expected_value, access); } - Cas::PutResult putOverwrite(const String & k, const String & b, const Cas::Token & e, const Cas::ObjectMeta & m) override + RawRemoval remove(const String & key, const String & expected_value, Cas::TransportAccess & access) override { failIfArmed(); - return InMemoryBackend::putOverwrite(k, b, e, m); - } - - Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const Cas::ObjectMeta & m) override - { - failIfArmed(); - return InMemoryBackend::casPut(k, b, e, m); - } - - Cas::DeleteOutcome deleteExact(const String & k, const Cas::Token & t) override - { - failIfArmed(); - return InMemoryBackend::deleteExact(k, t); + return InMemoryBackend::remove(key, expected_value, access); } private: @@ -115,6 +108,9 @@ class RollbackFaultBackend final : public Cas::InMemoryBackend class PromoteConflictOnceBackend final : public Cas::InMemoryBackend { public: + /// Unhide the legacy `putIfAbsent` overloads the primitive override below would otherwise hide. + using InMemoryBackend::putIfAbsent; + String fault_key_substr; int skip = 0; int fault_count = 0; @@ -122,9 +118,13 @@ class PromoteConflictOnceBackend final : public Cas::InMemoryBackend /// cleanup path ran its ref-log append at all, on a table where that append can no longer succeed. int matching_put_attempts = 0; - Cas::PutResult putIfAbsent(const String & key, const String & bytes, const Cas::ObjectMeta & meta) override + /// Sabotages the PRIMITIVE, which every legacy forwarder (including `putIfAbsent`) reaches too, so + /// the fault fires whichever surface issued the create. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + Cas::TransportAccess & access) override { - if (!fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) + if (!expected_value && !fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) { ++matching_put_attempts; if (skip > 0) @@ -132,13 +132,13 @@ class PromoteConflictOnceBackend final : public Cas::InMemoryBackend else if (fault_count > 0) { --fault_count; - /// The 3-arg qualified call bypasses virtual dispatch entirely (unlike a 2-arg - /// convenience overload, which would re-enter this very override through the vtable). - InMemoryBackend::putIfAbsent(key, bytes + String("\x01_FOREIGN_DIFFERENT"), meta); + /// The qualified call bypasses virtual dispatch entirely (unlike a re-entrant call + /// through the vtable), landing a foreign object at the key before the response is lost. + InMemoryBackend::write(key, bytes + String("\x01_FOREIGN_DIFFERENT"), expected_value, access); throw Poco::TimeoutException("PromoteConflictOnceBackend: a foreign different object landed; response lost"); } } - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } }; @@ -152,14 +152,20 @@ class PromoteConflictOnceBackend final : public Cas::InMemoryBackend class PromoteDefiniteFailureBackend final : public Cas::InMemoryBackend { public: + /// Unhide the legacy `putIfAbsent` overloads the primitive override below would otherwise hide. + using InMemoryBackend::putIfAbsent; + String fault_key_substr; int skip = 0; int fault_count = 0; int matching_put_attempts = 0; - Cas::PutResult putIfAbsent(const String & key, const String & bytes, const Cas::ObjectMeta & meta) override + /// Sabotages the PRIMITIVE, which every legacy forwarder (including `putIfAbsent`) reaches too. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + Cas::TransportAccess & access) override { - if (!fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) + if (!expected_value && !fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) { ++matching_put_attempts; if (skip > 0) @@ -171,7 +177,7 @@ class PromoteDefiniteFailureBackend final : public Cas::InMemoryBackend Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); } } - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } }; @@ -1011,6 +1017,9 @@ TEST(CASPartFolderAccess, BestEffortRollbackDropCountsAndSurvivesABackendOutage) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); + /// Both drops below give up only when their own retry window closes, so the engine's inter-attempt + /// sleeps are paid in virtual time rather than by sleeping out the operation deadline for real. + auto clock = Cas::tests::VirtualRetryClock::installOn(store); Cas::CachedPartFolderAccess access(store, cacheOn()); const Cas::RootNamespace ns_a{"srv/ta"}; @@ -1028,31 +1037,12 @@ TEST(CASPartFolderAccess, BestEffortRollbackDropCountsAndSurvivesABackendOutage) access.dropRefBestEffort(Cas::PartRefKey{ns_b, "part_b"}); const auto after = global_counters[ProfileEvents::CASRefRollbackBestEffortDropFailed].load(); EXPECT_EQ(after, before + 1); + EXPECT_GT(clock->pauseCount(), 0u) + << "the give-up must be the call's own retry window, reached through the injected sleep"; backend->armed = false; /// let store teardown release its lease cleanly } -namespace -{ - -/// A pool whose ref lane makes ONE attempt per append. That is what turns a single lost-response fault -/// into a conclusive `Unresolved`: with retries allowed the controller's resolve-before-reissue would -/// settle the ambiguity inside the same attempt and the lane would never wedge. Same budget shape, and -/// the same reason, as `gtest_cas_ref_install_safety.cpp`'s `openPoolSingleAttempt`. -Cas::PoolPtr openPoolSingleAttempt(const std::shared_ptr & backend) -{ - Cas::PoolConfig cfg{.pool_prefix = "p", .server_root_id = "test"}; - Cas::CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) - budget.lease_safety_margin_ms = 100; - cfg.cas_request_budget = budget; - return Cas::Pool::open(backend, cfg); -} - -} - /// Part B review, MAJOR 3a: a promote whose ref-log append did not resolve MUST NOT be reported as /// "nothing was committed". /// @@ -1068,8 +1058,9 @@ Cas::PoolPtr openPoolSingleAttempt(const std::shared_ptr & /// whose append never resolved. TEST(CASPartFolderAccess, AnUnresolvedPromoteIsNotReportedAsDefinitelyNotCommitted) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolForTest(backend); + auto clock = Cas::tests::VirtualRetryClock::installOn(store); const Cas::RootNamespace ns{"srv/t1"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); Cas::CachedPartFolderAccess access(store, cacheOn()); @@ -1080,18 +1071,29 @@ TEST(CASPartFolderAccess, AnUnresolvedPromoteIsNotReportedAsDefinitelyNotCommitt /// The promotion's own ref-log object lands; only the acknowledgement, and the controller's /// verifying read, are lost. Scoped to this namespace's ref log so nothing else consumes the fault. + /// A COUNTED fault cannot produce an unresolved outcome: the request engine settles the ambiguity + /// by an exact read that would find the landed object and report `Committed` inside the very same + /// call. Both legs therefore stay LATCHED for the whole call, and `VirtualRetryClock` pays the + /// retry window in virtual time instead of real wall-clock. backend->fault_substr = store->layout().namespaceStreamPrefix(fixture::fixtureLife(ns)) + "_log/"; backend->mode = Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; - backend->fault_count = 1; + backend->latched = true; expectThrowsCode(ErrorCodes::NETWORK_ERROR, [&] { prepared.promote(); }); + ASSERT_GT(clock->pauseCount(), 1u) + << "one attempt cannot exhaust the retry window: the fault must have outlasted every reissue"; EXPECT_TRUE(prepared.commitIsUnresolved()) << "a promote whose append may have landed must not be classified as a mechanism failure -- the " "receiver would fetch the bytes and publish the same part a second time"; /// The hazard itself, stated as an assertion: the promote DID commit. Any further append into this - /// table resolves the wedge first, which is what makes the committed row visible. + /// table resolves the wedge first, which is what makes the committed row visible. Disarmed + /// COMPLETELY, because that flush must reach the store normally: a still-armed lost read would + /// fault the wedge's own settling read, and nothing would resolve. + backend->latched = false; backend->mode = Cas::tests::ChunkFaultBackend::Mode::None; + backend->fault_count = 0; + backend->fail_read_once_key.clear(); access.prepareEntries({ns, "flush_driver"}, {inlineEntry("f", "two")}, Cas::ProvenanceOp::Insert).abort(); EXPECT_TRUE(access.existsRef(key, Cas::Freshness::ForceFresh)) << "the promotion object landed, so 'the promote failed' says nothing about the ref"; diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 4ccd5cb0ba96..570407814093 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -69,6 +69,39 @@ PoolPtr openPool(const std::shared_ptr & b) return Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); } +/// Run the mount plane's request engine on a virtual clock that its own reissue sleep advances -- the +/// plane every write below is admitted on. A policy that has to be EXHAUSTED then closes in +/// microseconds of wall clock instead of ninety real seconds, and a policy that merely reissues costs +/// no sleep at all, so these tests pin retry semantics and never a schedule. +/// The clock a test installed on the mount plane, handed back so the test can assert that the pacing +/// really ran on it. It does not make a LOST injection fast -- each fault double's attempt cap does +/// that -- but it does say that the reissues this test claims to drive were the injected clock's and +/// not the wall clock's. +struct VirtualRequestClock +{ + std::shared_ptr> now; + std::shared_ptr> sleeps; +}; + +/// `step_ms` is a FLOOR on how far each sleep moves the clock. A test that must see a policy exhaust +/// raises it so the window closes after a countable handful of attempts rather than after however many +/// near-zero jitter draws fit into ninety seconds. +VirtualRequestClock useVirtualMountRequestClock(const PoolPtr & store, uint64_t step_ms = 0) +{ + const VirtualRequestClock clock{std::make_shared>(0), + std::make_shared>(0)}; + store->mountRequests().setNowFnForTest([now = clock.now] { return now->load(); }); + /// `+ 1` because a jittered draw may be zero, and a clock that can stand still never closes the + /// window. + store->mountRequests().setSleepFnForTest( + [now = clock.now, sleeps = clock.sleeps, step_ms](uint64_t ms) + { + sleeps->fetch_add(1); + now->fetch_add(std::max(ms, step_ms) + 1); + }); + return clock; +} + /// Start a build whose owning manifest namespace + final ref name are `ns`/`ref` (promote/stageManifest /// derive the manifest namespace by splitting PartWriteInfo::intended_ref on the LAST '/'). PartWriteTxnPtr startBuildFor(const PoolPtr & s, const RootNamespace & ns, const String & ref) @@ -141,27 +174,17 @@ ManifestId publishOneBlobPart( } /// A one-shot backend hook (mirrors the WriteCountingBackend delegation pattern in gtest_cas_pool.cpp): -/// it delegates every op to a wrapped Backend, but the FIRST time head(target_key) is called it fires a -/// deleteExact(target_key, condemned_token) AFTER computing the (present) HEAD result and BEFORE returning -/// it — simulating GC's exact-token content delete landing in the writer's HEAD->GET window (B136). +/// it delegates every op to a wrapped Backend, but the FIRST time the target key is HEADed it fires an +/// exact-incarnation delete of that key AFTER computing the (present) HEAD result and BEFORE returning +/// it — GC emptying the key underneath the writer's observation, so the writer decides from a HEAD +/// whose object no longer exists. `fired` is public because a test that does not assert it cannot tell +/// a plumbed fault from an unplumbed one. class HeadThenDeleteOnceBackend final : public DB::Cas::Backend { public: HeadThenDeleteOnceBackend(BackendPtr inner_, String target_key_, DB::Cas::Token condemned_) : inner(std::move(inner_)), target_key(std::move(target_key_)), condemned(condemned_) {} - DB::Cas::HeadResult head(const String & k) override - { - const DB::Cas::HeadResult hr = inner->head(k); - if (k == target_key && !fired) - { - fired = true; - /// GC's single content-delete site, landing in the HEAD->GET window. - inner->deleteExact(target_key, condemned); - } - return hr; - } - std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } @@ -175,10 +198,19 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The transport primitives forward to `inner`; the legacy overrides above are what this - /// double injects through. Declared because `Backend` declares them pure. + /// The fault sits on the HEAD primitive, which is the only path a writer's mandatory HEAD takes. std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } - std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } + std::optional head(const String & key, TransportAccess & access) override + { + const auto observed = inner->head(key, access); + if (key == target_key && !fired) + { + fired = true; + /// GC's single content-delete site, landing in the HEAD->GET window. + inner->remove(target_key, condemned.value, access); + } + return observed; + } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } std::expected write(const String & key, const String & bytes, @@ -190,11 +222,12 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend void publish(const BlobPublishRequest & request, TransportAccess & access) override { inner->publish(request, access); } Dialect dialect() const override { return inner->dialect(); } + bool fired = false; + private: BackendPtr inner; String target_key; DB::Cas::Token condemned; - bool fired = false; }; /// A delegating backend that counts head()/get() calls per key. Lets a test assert the promote gate @@ -255,7 +288,7 @@ class KeyCountingBackend final : public DB::Cas::Backend class RacingBlobPublicationBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy overload the primitive override below would otherwise hide. using InMemoryBackend::head; void watch(String key_) { @@ -265,12 +298,14 @@ class RacingBlobPublicationBackend final : public InMemoryBackend publish_calls = 0; } - HeadResult head(const String & requested_key) override + /// Both faults sit on the transport primitives: a writer's mandatory HEAD and its publication both + /// reach the store through them. + std::optional head(const String & requested_key, TransportAccess & access) override { if (requested_key != key) - return InMemoryBackend::head(requested_key); + return InMemoryBackend::head(requested_key, access); - const HeadResult observed = InMemoryBackend::head(requested_key); + const std::optional observed = InMemoryBackend::head(requested_key, access); std::unique_lock lock(mutex); ++head_calls; cv.notify_all(); @@ -278,14 +313,14 @@ class RacingBlobPublicationBackend final : public InMemoryBackend return observed; } - void publishBlob(const BlobPublishRequest & request) override + void publish(const BlobPublishRequest & request, TransportAccess & access) override { if (request.destination_key == key) { std::lock_guard lock(mutex); ++publish_calls; } - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); } String key; @@ -791,8 +826,8 @@ TEST(CASPartWriteTxn, PutBlobRepublishesVanishedBodyFromHeldSource) /// meta; t0 stays as the body token the delete-hook below fires with. condemnMeta(*b, layout, u128Of("payload-X"), /*condemn_round*/ 1); - /// 3. Wrap the backend so the NEXT head(blob_key) returns the (present) result and THEN fires - /// deleteExact(blob_key, t0) exactly once — GC's delete in the HEAD->GET window. Open a FRESH + /// 3. Wrap the backend so the NEXT head(blob_key) returns the (present) result and THEN deletes that + /// exact incarnation once — GC emptying the key underneath the writer's observation. Open a FRESH /// Pool over the hook so its retire view (refreshed at open) sees the condemnation. auto hook = std::make_shared(b, blob_key, t0); auto s = Pool::open(hook, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); @@ -805,6 +840,11 @@ TEST(CASPartWriteTxn, PutBlobRepublishesVanishedBodyFromHeldSource) auto ref = build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); EXPECT_EQ(ref.ref, id); + /// The republication below reads the same whether or not the key was emptied mid-observation, so + /// this is what says the injected delete actually ran: a fault plumbed onto a method the writer no + /// longer calls leaves it false. It proves the seam fires, not that the end state depended on it. + EXPECT_TRUE(hook->fired) << "the injected delete must have run inside the mandatory HEAD"; + /// 5. The blob is present again under a FRESH token, with the same payload; and the condemned token /// never returns (INV-NO-RETURN). const HeadResult hr = b->head(blob_key); @@ -829,21 +869,69 @@ TEST(CASPartWriteTxn, PutBlobRepublishesVanishedBodyFromHeldSource) << "a fresh re-upload over a stale Condemned marker must reconcile it back to Clean"; } -/// A persistently-failing freshness-meta write (every attempt of every outer reload-retry) must -/// surface as a controlled retry-later signal, not silently succeed with the marker left stale -/// (S22 RCA). The blob body PUT -/// itself is unaffected (MetaWriteFaultBackend only faults `.meta` keys) -- only the meta write -/// exhausts, and that exhaustion must reach putBlob's caller as NETWORK_ERROR. +namespace +{ + +/// Every `.meta` write is ambiguous, forever: the store's answer is lost on each attempt, so the +/// engine settles each one by a read and reissues, and only the policy's own window ends the call. +class AmbiguousMetaWriteBackend final : public InMemoryBackend +{ +public: + /// Past this many faulted attempts the double stops faulting and raises a deterministic local + /// failure instead, which every loop here surfaces unchanged. A caller whose retries are no longer + /// bounded therefore FAILS on the wrong error code rather than running until the suite times out. + int attempt_cap = 100; + int attempts = 0; + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override + { + if (!key.ends_with(".meta")) + return InMemoryBackend::write(key, bytes, expected_value, access); + if (++attempts > attempt_cap) + throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, + "AmbiguousMetaWriteBackend: {} attempts exceeded the cap of {}; the caller's retries " + "are no longer bounded", attempts, attempt_cap); + throw Poco::TimeoutException("AmbiguousMetaWriteBackend: blob meta write response lost"); + } +}; + +} + +/// A give-up reports the attempts it actually SENT, not merely whether it sent any. An operator +/// counting a write's retries has to count one that gave up as well as one that committed, and +/// `sent_any` cannot say how many. +TEST(CASPartWriteTxn, AGiveUpReportsHowManyAttemptsItSent) +{ + auto b = std::make_shared(); + auto s = openPool(b); + const VirtualRequestClock clock = useVirtualMountRequestClock(s, /*step_ms=*/10'000); + b->attempts = 0; + + const BlobRef ref = idOf("give-up-attempt-count"); + CasOperation op = s->mountRequests().admit(); + const WriteResult result = putMetaIfAbsent(op, s->layout(), ref, BlobMeta{.state = MetaState::Clean, .size = 7}); + + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr) << "every attempt was ambiguous and none landed"; + EXPECT_TRUE(gave_up->sent_any); + EXPECT_GT(gave_up->attempts_sent, 1u) << "the write reissued before it gave up"; + EXPECT_EQ(gave_up->attempts_sent, static_cast(b->attempts)) + << "every attempt the store saw must be in the count the give-up reports"; + EXPECT_GT(clock.sleeps->load(), 0u) + << "the reissues were paced on the injected clock, so this exhaustion cost no wall time"; +} + +/// A persistently-failing freshness-meta write must surface as a controlled retry-later signal, not +/// silently succeed with the marker left stale. The blob body publication itself is unaffected (only +/// `.meta` keys are faulted) -- only the meta reconciliation exhausts, and that exhaustion must reach +/// putBlob's caller as NETWORK_ERROR. TEST(CASPartWriteTxn, PutBlobFreshMetaExhaustionThrowsRetryLater) { - /// Short budget + zero backoff: keep the test fast. Each of the metadata reconciliation loop's 8 outer - /// attempts calls putMetaIfAbsent, which itself retries up to max_attempts times internally — - /// with max_attempts=1 the controller gives up on the first faulted attempt each time. - CasRequestBudget budget; - budget.max_attempts = 1; - budget.retry_initial_backoff_ms = 0; - auto b = std::make_shared(); - auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); + auto b = std::make_shared(); + auto s = openPool(b); + const VirtualRequestClock clock = useVirtualMountRequestClock(s, /*step_ms=*/10'000); const String payload = "fresh-meta-exhaustion-payload"; auto build = precommittedBuildForPayload( @@ -853,6 +941,9 @@ TEST(CASPartWriteTxn, PutBlobFreshMetaExhaustionThrowsRetryLater) build->putBlob(idOf(payload), BlobSource::fromString(payload)); }); + EXPECT_GT(clock.sleeps->load(), 0u) + << "the reissues were paced on the injected clock, so this exhaustion cost no wall time"; + /// The body itself landed (only .meta writes are faulted) -- confirming the failure is /// specifically the freshness marker, not the blob body. const HeadResult hr = b->head(s->layout().blobKey(idOf(payload))); @@ -916,25 +1007,29 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) auto b = std::make_shared(); - /// 1. Upload blob Y via a throwaway build; capture the token t0. + /// 1. Upload blob Y via a throwaway build; capture the incarnation t0 the way GC persists it, and + /// let GC's exact-incarnation delete land before the writer's dedup hit. BlobRef id; - Token t0; + PersistedIncarnation t0; { auto s0 = openPool(b); auto build0 = precommittedBuildForPayload( s0, RootNamespace{"srv1/condemned-absent-seed"}, "part", "payload-Y"); id = build0->putBlob(idOf("payload-Y"), BlobSource::fromString("payload-Y")).ref; - t0 = b->head(s0->layout().blobKey(id)).token; + CasOperation op0 = s0->mountRequests().admit(); + const String seed_key = s0->layout().blobKey(id); + const auto seeded = op0.head(seed_key, Retry::standard()); + ASSERT_TRUE(seeded.has_value()); + t0 = PersistedIncarnation::capture(seeded->incarnation); + ASSERT_EQ(op0.remove(seed_key, seeded->incarnation, Retry::standard()), Removal::Removed); build0->abandon(); } - /// 2. Condemn (Blob, hash(Y), t0) in the retire view, then GC-delete the object so it is absent - /// (simulates GC completing the delete before the writer's dedup hit). + /// 2. Condemn (Blob, hash(Y), t0) in the retire view; the object is already absent. DB::Cas::Layout layout("p"); const String blob_key = layout.blobKey(id); injectRetire(*b, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-Y"))}, .token = t0, .size = 9}}); - b->deleteExact(blob_key, t0); ASSERT_FALSE(b->head(blob_key).exists); /// 3. Open a fresh Pool over a GET-counting wrapper; the retire view sees the condemnation at open. @@ -950,9 +1045,13 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) EXPECT_EQ(ref.ref, id); EXPECT_EQ(counting->get_count, 0u) << "INV-1: putBlob must not GET the dying object to revive it"; - const HeadResult hr = b->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_NE(hr.token, t0) << "a fresh incarnation must have a new token"; + /// The republished body must NOT read back as the incarnation GC condemned -- that compare is the + /// one GC's redelete makes, so it is the one this asserts. + CasRequests probe(b, Fence::open()); + CasOperation probe_op = probe.admit(); + const auto after = probe_op.head(blob_key, Retry::standard()); + ASSERT_TRUE(after.has_value()); + EXPECT_FALSE(t0.matches(after->incarnation)) << "a fresh publication must have a fresh incarnation"; const auto raw = b->get(blob_key); ASSERT_TRUE(raw.has_value()); const auto hdr = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); @@ -1673,12 +1772,15 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) /// 1. PartWriteTxn A creates H ("shared-content"), publishes a part referencing it, then drops the ref. /// Capture H's first incarnation token so we can condemn exactly it. BlobRef h; - Token h_token0; + PersistedIncarnation h_token0; { auto s0 = Pool::open(b, cfg); publishOneBlobPart(s0, ns, "part_1", "f", content); h = idOf(content); - h_token0 = b->head(s0->layout().blobKey(h)).token; + CasOperation op0 = s0->mountRequests().admit(); + const auto seeded = op0.head(s0->layout().blobKey(h), Retry::standard()); + ASSERT_TRUE(seeded.has_value()); + h_token0 = PersistedIncarnation::capture(seeded->incarnation); s0->dropRef(ns, "part_1"); } @@ -1710,9 +1812,10 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) const auto ref_b = build_b->putBlob(h, BlobSource::fromString(content)); ASSERT_EQ(ref_b.ref, h); - const HeadResult after_reupload = b->head(blob_key); - ASSERT_TRUE(after_reupload.exists); - EXPECT_NE(after_reupload.token, h_token0); /// a genuinely fresh incarnation + CasOperation reupload_op = s->mountRequests().admit(); + const auto after_reupload = reupload_op.head(blob_key, Retry::standard()); + ASSERT_TRUE(after_reupload.has_value()); + EXPECT_FALSE(h_token0.matches(after_reupload->incarnation)); /// a genuinely fresh incarnation /// 4. THE ADVERSARIAL LOOP. A real, productive GC keeps trying to reclaim. It reclaims the now- /// unreferenced part_1 manifest (build A's, UNprotected) but H stays pinned by B's PRECOMMIT edge @@ -2030,17 +2133,20 @@ class RefLogConflictOnceBackend final : public InMemoryBackend String corrupt_key_substr; int corrupt_count = 0; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + /// The fault sits on the write primitive, which every conditional write reaches the store through. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { if (corrupt_count > 0 && !corrupt_key_substr.empty() && key.find(corrupt_key_substr) != String::npos) { --corrupt_count; - /// The 3-arg qualified call bypasses virtual dispatch entirely (unlike the 2-arg - /// convenience overload, which would re-enter this very override through the vtable). - InMemoryBackend::putIfAbsent(key, bytes + String("\x01_FOREIGN_DIFFERENT"), meta); + /// The qualified call bypasses virtual dispatch entirely, so the foreign write does not + /// re-enter this very override. + (void)InMemoryBackend::write(key, bytes + String("\x01_FOREIGN_DIFFERENT"), expected_value, access); throw Poco::TimeoutException("RefLogConflictOnceBackend: a foreign different object landed; response lost"); } - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } }; @@ -2327,47 +2433,56 @@ class ManifestPutFaultBackend final : public InMemoryBackend bool land_despite_fault = false; /// the faulted attempt's own write actually lands (response lost) String plant_different_on_fault; /// a FOREIGN different body lands at the key before the fault int put_attempts = 0; /// matching body-PUT attempts observed + /// Past this many attempts the double stops faulting and raises a deterministic local failure + /// instead, which every loop here surfaces unchanged. A caller whose retries are no longer bounded + /// therefore FAILS on the wrong error code rather than running until the suite times out. 0 = off. + int attempt_cap = 0; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + /// The fault sits on the write primitive: a staged part-manifest body is a create, which is a + /// `write` with no precondition. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { if (!isManifestBodyKey(key)) - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); ++put_attempts; - maybeFault(key, bytes); - return InMemoryBackend::putIfAbsent(key, bytes, meta); + if (attempt_cap > 0 && put_attempts > attempt_cap) + throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, + "ManifestPutFaultBackend: {} attempts exceeded the cap of {}; the caller's retries are " + "no longer bounded", put_attempts, attempt_cap); + maybeFault(key, bytes, access); + return InMemoryBackend::write(key, bytes, expected_value, access); } private: static bool isManifestBodyKey(const String & key) { return key.find("/cas/manifests/") != String::npos; } /// One fault: apply the configured server-side effect, then lose the response. - void maybeFault(const String & key, const String & bytes) + void maybeFault(const String & key, const String & bytes, TransportAccess & access) { if (fault_count <= 0) return; --fault_count; if (!plant_different_on_fault.empty()) - InMemoryBackend::putIfAbsent(key, plant_different_on_fault, {}); + (void)InMemoryBackend::write(key, plant_different_on_fault, std::nullopt, access); else if (land_despite_fault) - InMemoryBackend::putIfAbsent(key, bytes, {}); + (void)InMemoryBackend::write(key, bytes, std::nullopt, access); throw Poco::TimeoutException("ManifestPutFaultBackend: simulated ambiguous result (response lost)"); } }; } -/// The Task B core: two consecutive ambiguous timeouts on the part-manifest body PUT (each resolved -/// to "absent" by the controller's exact-GET), then a clean third attempt. The old single-attempt +/// The core ride: two consecutive ambiguous timeouts on the part-manifest body PUT (each resolved +/// to "absent" by the engine's exact read), then a clean third attempt. The old single-attempt /// path fails the whole stage on the FIRST timeout (the observed 19s-pause INSERT kill); the -/// controller path must ride its attempt budget and succeed. +/// engine's write loop must ride its policy and succeed. TEST(CASPartWriteTxnStageManifestRetry, AmbiguousTimeoutsThenCommitSucceedsWithinBudget) { - /// Zero backoff: the retry semantics are under test here, not the (controller-level-tested) - /// inter-attempt sleep schedule — keep the suite free of real sleeps. - CasRequestBudget budget; - budget.retry_initial_backoff_ms = 0; auto b = std::make_shared(); - auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); + auto s = openPool(b); + useVirtualMountRequestClock(s); const RootNamespace ns{"srv/tbl"}; auto build = startBuildFor(s, ns, "part_retry"); @@ -2406,8 +2521,12 @@ TEST(CASPartWriteTxnStageManifestRetry, AmbiguousLandedWriteResolvesToCommittedW const auto ev = std::find_if(events.begin(), events.end(), [](const CasEvent & e) { return e.type == CasEventType::ManifestPut; }); ASSERT_NE(ev, events.end()) << "the stage must still emit its ManifestPut audit event"; - EXPECT_EQ(ev->token, b->head(key).token.value) - << "the audit token must be the landed incarnation's token"; + CasRequests probe(b, Fence::open()); + CasOperation probe_op = probe.admit(); + const auto landed = probe_op.head(key, Retry::standard()); + ASSERT_TRUE(landed.has_value()); + EXPECT_EQ(ev->token, landed->incarnation.render()) + << "the audit token must be the landed incarnation, rendered"; } /// A DIFFERENT object at the exact staged key (a foreign body ahead of our ambiguous attempt) is a @@ -2429,26 +2548,38 @@ TEST(CASPartWriteTxnStageManifestRetry, DifferentObjectAtKeyStaysLoudConflict) EXPECT_EQ(b->put_attempts, 1) << "a proven conflict is never retried"; } -/// Budget exhaustion: EVERY attempt is ambiguous and nothing ever lands. The controller reports -/// Unresolved after `max_attempts` and stageManifest maps it to NETWORK_ERROR (fix #37 phase 2) — -/// the same retryable abort class the ref-log lane's exhausted budget maps to. Nothing was durably -/// named: the caller re-stages with a fresh ManifestId. +/// Policy exhaustion: EVERY attempt is ambiguous and nothing ever lands. The write gives up, and +/// stageManifest maps that to NETWORK_ERROR — the same retryable abort class the ref-log lane's +/// exhausted budget maps to. Nothing was durably named: the caller re-stages with a fresh ManifestId. +/// +/// Two things make the BOUND itself observable rather than assumed. The injected clock is advanced by +/// each reissue's own sleep, so the policy's window closes after a handful of attempts instead of after +/// ninety real seconds; and the double refuses deterministically past a cap far above that handful, so +/// a stage whose retries stopped being bounded fails on the wrong error code instead of running until +/// the suite times out. The exact attempt COUNT belongs to the request policy and is pinned where that +/// policy lives. TEST(CASPartWriteTxnStageManifestRetry, BudgetExhaustionMapsToNetworkError) { - CasRequestBudget budget; - budget.max_attempts = 3; - budget.retry_initial_backoff_ms = 0; /// no real sleeps; the backoff schedule has its own tests auto b = std::make_shared(); - auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); + auto s = openPool(b); + const VirtualRequestClock clock = useVirtualMountRequestClock(s, /*step_ms=*/10'000); const RootNamespace ns{"srv/tbl"}; auto build = startBuildFor(s, ns, "part_exhausted"); - b->fault_count = 1000; + b->fault_count = 1000000; + b->attempt_cap = 100; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { build->stageManifest({blobManifestEntry("a.bin", "a")}); }); - EXPECT_EQ(b->put_attempts, 3) << "attempts must be bounded by the configured budget"; + EXPECT_GT(b->put_attempts, 1) << "the ambiguous attempt must be reissued before the write gives up"; + EXPECT_LE(b->put_attempts, b->attempt_cap) << "the policy's window, not the double's cap, ended it"; + EXPECT_GT(clock.sleeps->load(), 0u) + << "the reissues were paced on the injected clock, so this exhaustion cost no wall time"; + /// Over the namespace's whole manifest prefix rather than one computed key: the ordinal the build + /// would have used is arithmetic this assertion should not have to reproduce to stay true. + EXPECT_TRUE(b->list(s->layout().manifestNamespacePrefix(ns), "", 10).keys.empty()) + << "an exhausted stage names nothing durable"; } /// ===================================================================================== @@ -2464,38 +2595,49 @@ namespace class BlobPutFaultBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy overload the primitive override below would otherwise hide. using InMemoryBackend::head; int fault_count = 0; /// remaining ambiguous faults on matching create attempts bool land_despite_fault = false; /// the faulted attempt's own write actually lands (response lost) int publish_stream_attempts = 0; /// unconditional streaming publications observed int publish_copy_attempts = 0; /// unconditional native-copy publications observed int blob_head_attempts = 0; /// transaction-level blob observations + std::function on_publish; /// runs before each publication, for a test that spends time in one + /// The envelope of every streaming publication the store was asked to make, in order. A test reads + /// it to prove two physical publications were two DIFFERENT bodies. + std::vector published_envelopes; - HeadResult head(const String & key) override + /// Both seams sit on the transport primitives: the writer's mandatory HEAD and its publication + /// both reach the store through them. + std::optional head(const String & key, TransportAccess & access) override { if (isBlobBodyKey(key)) ++blob_head_attempts; - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } - void publishBlob(const BlobPublishRequest & request) override + void publish(const BlobPublishRequest & request, TransportAccess & access) override { - if (std::holds_alternative(request.publication)) + if (const auto * streaming = std::get_if(&request.publication)) + { ++publish_stream_attempts; + published_envelopes.push_back(streaming->fresh_envelope); + } else ++publish_copy_attempts; + if (on_publish) + on_publish(); if (fault_count > 0) { --fault_count; if (land_despite_fault) - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); else if (const auto * streaming = std::get_if(&request.publication)) (void)streaming->open_payload(); throw Poco::TimeoutException("BlobPutFaultBackend: simulated ambiguous publication (response lost)"); } - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); } private: @@ -2506,14 +2648,16 @@ class BlobPutFaultBackend final : public InMemoryBackend }; -/// Zero-backoff store over a BlobPutFaultBackend: the sleep schedule has its own controller-level -/// tests; these Pool-level tests pin the retry/resolve/abort semantics without real sleeps. -PoolPtr openBlobFaultPool(const std::shared_ptr & b, uint32_t max_attempts = CasRequestBudget{}.max_attempts) +/// A store over a BlobPutFaultBackend. The publication loop's own bound is what these tests pin, so +/// nothing here configures a request policy: a faulted publication is one physical attempt, and the +/// loop's next iteration is what reissues it. +PoolPtr openBlobFaultPool(const std::shared_ptr & b) { - CasRequestBudget budget; - budget.max_attempts = max_attempts; - budget.retry_initial_backoff_ms = 0; - return Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); + PoolPtr store = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + /// The publication loop paces its iterations on the engine's clock, so drive that clock virtually: + /// eight iterations of real jittered backoff would be seconds of wall time per test. + useVirtualMountRequestClock(store); + return store; } /// A replayable BlobSource that COUNTS its own re-streams — pins INV-1's "retry = fresh re-stream @@ -2592,12 +2736,139 @@ TEST(CASPartWrite, AmbiguousLandedWriteAdoptsOccupantWithoutReupload) const auto adopt = std::find_if(events.begin(), events.end(), [](const CasEvent & e) { return e.type == CasEventType::BlobReuseAdopt; }); ASSERT_NE(adopt, events.end()) << "the landed occupant must be ADOPTED (the standard dedup leg)"; - EXPECT_EQ(adopt->token, b->head(key).token.value) << "the adopted token must be the landed incarnation's"; + CasRequests probe(b, Fence::open()); + CasOperation probe_op = probe.admit(); + const auto landed = probe_op.head(key, Retry::standard()); + ASSERT_TRUE(landed.has_value()); + EXPECT_EQ(adopt->token, landed->incarnation.render()) + << "the adopted token must be the landed incarnation, rendered"; EXPECT_EQ(std::count_if(events.begin(), events.end(), [](const CasEvent & e) { return e.type == CasEventType::BlobPut; }), 0) << "no fresh-upload event: the body was never re-uploaded"; } +/// Every physical publication mints its own envelope, and the request engine never reissues one. Two +/// attempts sharing an `incarnation_tag` would send byte-identical bodies, so on a +/// content-derived-ETag dialect the republished body would read back as the incarnation GC condemned, +/// and GC's exact-incarnation delete would then remove a live body. +TEST(CASPartWrite, EveryPhysicalPublicationMintsAFreshIncarnationTag) +{ + auto b = std::make_shared(); + auto s = openBlobFaultPool(b); + const RootNamespace ns{"srv/tbl"}; + const String payload = "blob-payload-fresh-tag"; + + auto build = startBuildFor(s, ns, "part_blob_fresh_tag"); + const ManifestId id = build->stageManifest({blobManifestEntry("a.bin", payload)}); + build->precommitAdd(ns, "part_blob_fresh_tag", id); + + int payload_streams = 0; + b->fault_count = 1; + const PutBlobResult res = build->putBlob(idOf(payload), countingSource(payload, payload_streams)); + EXPECT_EQ(res.size, payload.size()); + + ASSERT_EQ(b->published_envelopes.size(), 2u) << "one ambiguous publication and the committing second"; + EXPECT_NE(b->published_envelopes[0], b->published_envelopes[1]) + << "the second physical publication re-sent the first one's envelope"; + + const uint64_t object_size = s->poolMeta().blob_header_len + payload.size(); + const auto first = decodeEnvelopeHeader(b->published_envelopes[0], object_size, ObjectKind::Blob); + const auto second = decodeEnvelopeHeader(b->published_envelopes[1], object_size, ObjectKind::Blob); + EXPECT_TRUE(first.incarnation_tag != second.incarnation_tag) + << "a repeated incarnation_tag is a repeated incarnation: GC's condemn would name the live body"; +} + +/// "One `Retry::standard()`" is one policy VALUE, not one shared deadline: each verb of the +/// publication loop binds its own window from it, so a loop that has already spent far more than one +/// window still publishes. Here each ambiguous publication burns forty seconds of the injected clock, +/// so three of them exceed the standard ninety-second window; a loop that carried a single bound +/// across its iterations would refuse the fourth iteration's HEAD instead of committing. +TEST(CASPartWrite, EnsureBlobPresentSharesOneRetryAcrossItsLoop) +{ + auto b = std::make_shared(); + auto s = openBlobFaultPool(b); + auto now = std::make_shared>(0); + s->mountRequests().setNowFnForTest([now] { return now->load(); }); + s->mountRequests().setSleepFnForTest([now](uint64_t ms) { now->fetch_add(ms + 1); }); + b->on_publish = [now] { now->fetch_add(40'000); }; + + const RootNamespace ns{"srv/tbl"}; + const String payload = "blob-payload-long-loop"; + auto build = startBuildFor(s, ns, "part_blob_long_loop"); + const ManifestId id = build->stageManifest({blobManifestEntry("a.bin", payload)}); + build->precommitAdd(ns, "part_blob_long_loop", id); + + int payload_streams = 0; + b->fault_count = 3; + const PutBlobResult res = build->putBlob(idOf(payload), countingSource(payload, payload_streams)); + EXPECT_EQ(res.size, payload.size()); + + EXPECT_EQ(b->publish_stream_attempts, 4) << "three ambiguous publications + the committing fourth"; + EXPECT_GT(now->load(), 90'000u) << "the loop outlived one standard window, which is the point"; +} + +namespace +{ + +/// Fires an injected side effect once, immediately after the watched key's body read returns -- the +/// point at which `ensureBlobPresent` has everything it needs and is about to render its verdict. +class RearmAfterMetaReadBackend final : public InMemoryBackend +{ +public: + /// Unhide the legacy overload the primitive override below would otherwise hide. + using InMemoryBackend::get; + + String watched_key; + std::function trigger; + + std::optional read(const String & key, TransportAccess & access) override + { + auto observed = InMemoryBackend::read(key, access); + if (key == watched_key && trigger) + std::exchange(trigger, {})(); + return observed; + } +}; + +} + +/// A trip-and-rearm hidden inside the observation leaves the mount writable again, but NOT under the +/// generation this materialization was admitted under. The verdict points read that admission, so the +/// dependency proof is refused rather than handed back from an incarnation that has been superseded -- +/// which is what would let a fenced-out build commit a manifest naming blobs it never legally observed. +TEST(CASPartWrite, DependencyProofIsRefusedAfterARearm) +{ + auto b = std::make_shared(); + auto s = openPool(b); + const String payload = "dependency-proof-after-rearm"; + const UInt128 hash = u128Of(payload); + const BlobRef ref = idOf(payload); + + /// Pre-seed a present body and a Clean marker so the observation takes the ADOPT leg -- the leg + /// whose only durable output is the dependency proof itself. + const uint64_t header_len = s->poolMeta().blob_header_len; + String raw_body(header_len, '\0'); + raw_body += payload; + writeRawBlobBody(*b, s->layout(), hash, raw_body); + writeMetaClean(*b, s->layout(), hash, payload.size()); + + auto build = precommittedBuildForPayload(s, RootNamespace{"srv1/rearm-proof"}, "part", payload); + b->watched_key = s->layout().blobMetaKey(ref); + b->trigger = [&] + { + s->tripMountLost(); + DB::Cas::tests::rearmMountFenceAfterAnomalyForTest(s); + }; + + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + { + build->putBlob(ref, BlobSource::fromString(payload)); + }); + EXPECT_TRUE(s->mayMutate()) + << "the mount is writable again: the refusal is about the generation the build was admitted " + "under, not about the mount being closed"; +} + /// Budget exhaustion: EVERY attempt is ambiguous and nothing ever lands. The controller reports the /// uncertainty and `ensureBlobPresent` maps it to `NETWORK_ERROR` -- the same retryable /// abort class stageManifest and the ref-log lane map their exhausted budgets to. Unlike the OLD @@ -2607,7 +2878,7 @@ TEST(CASPartWrite, AmbiguousLandedWriteAdoptsOccupantWithoutReupload) TEST(CASPartWrite, AmbiguousNonLandingPublicationStopsAtOuterBound) { auto b = std::make_shared(); - auto s = openBlobFaultPool(b, /*max_attempts=*/3); + auto s = openBlobFaultPool(b); const RootNamespace ns{"srv/tbl"}; const String payload = "blob-payload-C"; diff --git a/src/Disks/tests/gtest_cas_plain_objects.cpp b/src/Disks/tests/gtest_cas_plain_objects.cpp new file mode 100644 index 000000000000..411d8cefd800 --- /dev/null +++ b/src/Disks/tests/gtest_cas_plain_objects.cpp @@ -0,0 +1,88 @@ +#include + +#include +#include "cas_test_helpers.h" + +#include +#include + +using namespace DB::Cas; + +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::FakeClock; + +namespace +{ + +/// Every test drives `CasRequests` on an injected clock (mirrors `gtest_cas_requests.cpp`'s +/// `makeRequests`), so a policy's whole deadline is exercised in no wall-clock time. +CasRequests makeRequests(BackendPtr backend, FakeClock & clock, Fence fence = Fence::open()) +{ + return CasRequests(std::move(backend), std::move(fence), clock.nowFn(), clock.sleepFn()); +} + +/// Refuses the FIRST removal attempt of every key with `Mismatch`, then delegates -- models a +/// concurrent replacement observed between `removeCurrent`'s internal HEAD and its DELETE. +struct MismatchOnceOnRemoveBackend : InMemoryBackend +{ + using InMemoryBackend::head; + + size_t heads = 0; + bool refuse_next_remove = true; + + std::optional head(const String & key, TransportAccess & access) override + { + ++heads; + return InMemoryBackend::head(key, access); + } + + Backend::RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override + { + if (std::exchange(refuse_next_remove, false)) + return Backend::RawRemoval::Mismatch; + return InMemoryBackend::remove(key, expected_value, access); + } +}; + +} + +TEST(CASPlainObjects, CasPutObjectIssuesHeadsOnly) +{ + FakeClock clock; + auto backend = std::make_shared(); + Layout layout("pool"); + auto requests = makeRequests(backend, clock); + CasPlainObjects objects(requests, layout); + + const RootNamespace ns("t"); + const auto life = DB::Cas::tests::fixture::fixtureLife(ns); + const String key = layout.namespaceFileKey(life, "f"); + + /// The create. + objects.putNamespaceFile(life, "f", "hello"); + EXPECT_GT(backend->headCount(key), 0u); + EXPECT_EQ(backend->getCount(key), 0u); + + /// A replace over an existing object follows the same protocol: HEAD only, never a body GET. + objects.putNamespaceFile(life, "f", "world"); + EXPECT_EQ(backend->getCount(key), 0u); + EXPECT_EQ(objects.getNamespaceFile(life, "f"), "world"); +} + +TEST(CASPlainObjects, CasRemoveObjectReheadsOnMismatch) +{ + FakeClock clock; + auto backend = std::make_shared(); + Layout layout("pool"); + auto requests = makeRequests(backend, clock); + CasPlainObjects objects(requests, layout); + + objects.putMountpointObject("f", "v"); + backend->heads = 0; + + /// The injected `Mismatch` on the first attempt must not surface as a failure: `removeCurrent` + /// re-heads the key and retries against what it now observes. + objects.removeMountpointObject("f"); + EXPECT_GE(backend->heads, 2u); + EXPECT_FALSE(objects.mountpointObjectExists("f")); +} diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index 6c7eb1d59667..32dbe2a0ad94 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -139,12 +139,13 @@ TEST(CASPluggableHash, CreateOrValidateRecordsConfigAlgoOnFreshPool) { auto backend = std::make_shared(); const Layout layout("p"); + OperationForTest meta_op(*backend); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, /*blob_header_len*/ 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::XXH3_128)})); /// Reopening with the SAME algo is a no-op reopen: the recorded value comes back unchanged. - const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128); + const PoolMeta reopened = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::XXH3_128); EXPECT_EQ(reopened.algos_used, (std::vector{static_cast(BlobHashAlgo::XXH3_128)})); EXPECT_EQ(reopened.pool_id, pm.pool_id); } @@ -153,8 +154,9 @@ TEST(CASPluggableHash, CreateOrValidateDefaultsToCityHash128) { auto backend = std::make_shared(); const Layout layout("p"); + OperationForTest meta_op(*backend); - const PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + const PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(pm.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); } @@ -166,20 +168,21 @@ TEST(CASPluggableHash, CreateOrValidateFailsClosedOnAlgoMismatchWithoutFlag) { auto backend = std::make_shared(); const Layout layout("p"); + OperationForTest meta_op(*backend); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); expectThrowsCodeWithMessage( DB::ErrorCodes::BAD_ARGUMENTS, "1", [&] { - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false); + PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::XXH3_128, /*allow_new*/ false); }); /// The pool is untouched by the refused reopen: a subsequent open with the ORIGINAL algo still /// succeeds and returns the same pool_id. - const PoolMeta reopened = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128); + const PoolMeta reopened = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128); EXPECT_EQ(reopened.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); } @@ -189,18 +192,19 @@ TEST(CASPluggableHash, AdmissionIsFlagGated) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); /// without the flag: refuse, pool untouched expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, false); }); + { PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::Sha256, false); }); /// with the flag: admitted - const PoolMeta admitted = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, true); + const PoolMeta admitted = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::Sha256, true); EXPECT_EQ(admitted.algos_used, (std::vector{1, 3})); /// steady state: admitted algo reopens WITHOUT the flag - const PoolMeta steady = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, false); + const PoolMeta steady = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::Sha256, false); EXPECT_EQ(steady.algos_used, (std::vector{1, 3})); } @@ -208,10 +212,11 @@ TEST(CASPluggableHash, ConcurrentAdmissionUnions) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, false, /*allow_mint*/ true); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::XXH3_128, true); - PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::Sha256, true); - const PoolMeta final_pm = PoolMeta::createOrValidate(*backend, layout, 256, BlobHashAlgo::CityHash128, false); + OperationForTest meta_op(*backend); + PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128, false, /*allow_mint*/ true); + PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::XXH3_128, true); + PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::Sha256, true); + const PoolMeta final_pm = PoolMeta::createOrValidate(*meta_op, layout, 256, BlobHashAlgo::CityHash128, false); EXPECT_EQ(final_pm.algos_used, (std::vector{1, 2, 3})); /// union, sorted, nothing lost } @@ -700,7 +705,8 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); pm.min_reader_generation = G_BUILD + 1; ASSERT_TRUE(backend->casPut(layout.poolMetaKey(), encodePoolMeta(pm), backend->get(layout.poolMetaKey())->token).outcome == CasOutcome::Committed); @@ -714,7 +720,8 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) { auto backend = std::make_shared(); const Layout layout("p"); - PoolMeta pm = PoolMeta::createOrValidate(*backend, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); + OperationForTest meta_op(*backend); + PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); const String fresh_bytes = encodePoolMeta(pm); const String from = "\"v\":" + std::to_string(G_BUILD); diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index c529e2f5126b..58c5636358a3 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -349,6 +349,19 @@ class ThrowingSingleAttemptBackend final : public ForwardingBackend } }; +/// A backend whose store-level preconditions refuse the pool outright — a stand-in for a versioning or +/// dialect combination `ObjectStorageBackend::checkPoolPreconditions` rejects. +class ThrowingPoolPreconditionsBackend final : public ForwardingBackend +{ +public: + using ForwardingBackend::ForwardingBackend; + + void checkPoolPreconditions() override + { + throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "test: pool preconditions refused"); + } +}; + /// A backend that forbids skipping the access-check battery — a stand-in for the writable /// generation-dialect (GCS) backend (see ObjectStorageBackend::checkSkipAccessCheckSupport). class ThrowingSkipAccessCheckBackend final : public ForwardingBackend @@ -423,6 +436,71 @@ TEST(CASPool, BackendForbiddingSkipAccessCheckStillOpensWhenTheBatteryRuns) ASSERT_NE(store, nullptr); } +/// The ORDINARY writable mount -- the one that runs the battery -- must still be refused by the two +/// store-level gates. They used to be the capability probe's own first two steps; they are the caller's +/// now, and nothing else in the open path would notice if the caller stopped asking. The write counter is +/// what makes each of these a fence rather than a bare `EXPECT_THROW`: `Pool::open` refuses for many +/// reasons, but only a refusal BEFORE the battery leaves the store unwritten. +TEST(CASPool, WritableOpenRunsThePoolPreconditionGateBeforeTheBattery) +{ + auto counting = std::make_shared(std::make_shared()); + auto backend = std::make_shared(counting); + + DB::Cas::PoolConfig cfg = writablePoolConfigForTest(); + ASSERT_FALSE(cfg.skip_access_check) << "this test is about the branch that RUNS the battery"; + + try + { + DB::Cas::Pool::open(backend, cfg); + FAIL() << "expected the pool-precondition gate to refuse the mount"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find("pool preconditions refused"), std::string::npos) + << "actual message: " << e.message(); + } + EXPECT_EQ(counting->writes, 0u) << "the gate must refuse before the battery writes anything"; +} + +TEST(CASPool, WritableOpenRunsTheSingleAttemptGateBeforeTheBattery) +{ + auto counting = std::make_shared(std::make_shared()); + auto backend = std::make_shared(counting); + + DB::Cas::PoolConfig cfg = writablePoolConfigForTest(); + ASSERT_FALSE(cfg.skip_access_check) << "this test is about the branch that RUNS the battery"; + + try + { + DB::Cas::Pool::open(backend, cfg); + FAIL() << "expected the single-attempt gate to refuse the mount"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find("no single-attempt client"), std::string::npos) + << "actual message: " << e.message(); + } + EXPECT_EQ(counting->writes, 0u) << "the gate must refuse before the battery writes anything"; +} + +/// The positive control for the two above: with no gate refusing, the same open DOES write. Without it +/// `writes == 0` would be satisfied by an open that refused for any earlier reason, and both fences would +/// pass while the gates were gone. +TEST(CASPool, WritableOpenWithoutAGateRefusalDoesReachTheBattery) +{ + auto counting = std::make_shared(std::make_shared()); + + DB::Cas::PoolConfig cfg = writablePoolConfigForTest(); + cfg.background_watermark = false; + ASSERT_FALSE(cfg.skip_access_check); + + auto store = DB::Cas::Pool::open(counting, cfg); + ASSERT_NE(store, nullptr); + EXPECT_GT(counting->writes, 0u); +} + TEST(CASPool, MinActiveTracksInFlightBuilds) { auto backend = std::make_shared(); @@ -504,10 +582,10 @@ TEST(CASPoolMeta, CreateThenReopen) { auto b = std::make_shared(); Layout layout("p"); - PoolMeta created = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 256, + PoolMeta created = PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_NE(created.pool_id, UInt128{}); - PoolMeta reopened = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512); + PoolMeta reopened = PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, /*blob_header_len*/ 512); EXPECT_EQ(reopened.pool_id, created.pool_id); /// pool is authoritative — config ignored on reopen EXPECT_EQ(reopened.blob_header_len, 256u); } @@ -521,7 +599,7 @@ TEST(CASPoolMeta, FailClosed) auto b2 = std::make_shared(); b2->putIfAbsent(layout.poolMetaKey(), "garbage"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { PoolMeta::createOrValidate(*b2, layout, 256); }); + [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b2), layout, 256); }); } TEST(CASPoolMeta, RoundTripAndReadability) @@ -551,17 +629,17 @@ TEST(CASPoolMeta, RejectsBadConstantsAtCreation) /// not 8-aligned (above the floor, so it is the alignment rule that rejects it) expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, - [&] { PoolMeta::createOrValidate(*b, layout, 250); }); + [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 250); }); /// below the v3 envelope floor (240) but 8-aligned: rejected by the floor, not the alignment rule. /// Without the raised floor this pool would pass creation and LOGICAL_ERROR on the first blob write. expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, - [&] { PoolMeta::createOrValidate(*b, layout, 128); }); + [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 128); }); /// well below the floor expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, - [&] { PoolMeta::createOrValidate(*b, layout, 64); }); + [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 64); }); /// above the 16 KiB ceiling expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, - [&] { PoolMeta::createOrValidate(*b, layout, 17 * 1024); }); + [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 17 * 1024); }); /// A creation that fails config validation must not have written anything. EXPECT_FALSE(b->get(layout.poolMetaKey()).has_value()); @@ -578,7 +656,7 @@ TEST(CASPoolMeta, RejectsBadConstantsOnDecode) bad_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(bad_pm)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { PoolMeta::createOrValidate(*b, layout, 256); }); + [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 256); }); } TEST(CASPoolMeta, DecodeGarbageFails) @@ -603,7 +681,7 @@ TEST(CASPoolMeta, ConcurrentCreateRace) foreign_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(foreign_pm)); - PoolMeta result = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512); + PoolMeta result = PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, /*blob_header_len*/ 512); EXPECT_EQ(result.pool_id, foreign); EXPECT_EQ(result.blob_header_len, 256u); /// the foreign pool's constants win } @@ -613,7 +691,7 @@ TEST(CASPoolMeta, CasConflictReReadsWinner) /// The subtlest branch: the initial GET sees ABSENT, so createOrValidate proceeds to the /// create-if-absent casPut — and loses, because a racing creator committed in between. The loser /// must then re-read and return the WINNER's pool identity, not LOGICAL_ERROR. A single-threaded - /// `failNextCasPut` alone cannot exercise this: it returns Conflict without leaving the object + /// `refuseNextWrite` alone cannot exercise this: it returns Conflict without leaving the object /// readable, so the re-read would fire the LOGICAL_ERROR guard. We model the real interleaving /// with a backend whose casPut commits the winner's object (via the public putIfAbsent) and THEN /// reports Conflict — exactly what the loser observes. @@ -621,17 +699,18 @@ TEST(CASPoolMeta, CasConflictReReadsWinner) { public: String winner_bytes; - CasResult casPut(const String & key, const String & bytes, - const std::optional & expected, const ObjectMeta & meta) override + /// The fault sits on the WRITE PRIMITIVE: the create-if-absent this models is issued there. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - if (!winner_committed) + if (!winner_committed && !expected_value) { winner_committed = true; /// The winner lands first; our create-if-absent now necessarily conflicts. - putIfAbsent(key, winner_bytes); - return {CasOutcome::Conflict, {}}; + (void)InMemoryBackend::write(key, winner_bytes, std::nullopt, access); + return std::unexpected(RawConflict{}); } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } private: bool winner_committed = false; @@ -648,7 +727,7 @@ TEST(CASPoolMeta, CasConflictReReadsWinner) Layout layout("p"); /// Our config (512) is what we WOULD have minted, but we lose the race and inherit the winner. - PoolMeta result = PoolMeta::createOrValidate(*b, layout, /*blob_header_len*/ 512, + PoolMeta result = PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, /*blob_header_len*/ 512, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); EXPECT_EQ(result.pool_id, winner); EXPECT_EQ(result.blob_header_len, 256u); @@ -1406,22 +1485,7 @@ class FenceInAdoptWindowBackend final : public DB::Cas::Backend explicit FenceInAdoptWindowBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} String fence_key; /// empty = fault disarmed; set to the mount key to arm the one-shot fence - std::optional get(const String & k, DB::Cas::Range r) override - { - auto got = inner->get(k, r); - if (!fence_key.empty() && k == fence_key && got.has_value()) - { - /// One-shot: fence the slot in place exactly as `computeHeartbeatFloor` does (preserve the - /// body, gc_fenced = true, seq + 1, token-guarded against the value we just read), then - /// disarm so the retry can adopt cleanly. - DB::Cas::MountLease fenced = DB::Cas::decodeMountLease(got->bytes); - fenced.gc_fenced = true; - fenced.seq += 1; - inner->putOverwrite(k, DB::Cas::encodeMountLease(fenced), got->token); - fence_key.clear(); - } - return got; - } + std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } @@ -1435,9 +1499,23 @@ class FenceInAdoptWindowBackend final : public DB::Cas::Backend DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The transport primitives forward to `inner`; the legacy overrides above are what this - /// double injects through. Declared because `Backend` declares them pure. - std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } + /// The fault sits on the READ PRIMITIVE: the keeper's adopt reads the mount slot through it. + std::optional read(const String & key, TransportAccess & access) override + { + auto got = inner->read(key, access); + if (!fence_key.empty() && key == fence_key && got.has_value()) + { + /// One-shot: fence the slot in place exactly as `computeHeartbeatFloor` does (preserve the + /// body, gc_fenced = true, seq + 1, guarded against the incarnation we just read), then + /// disarm so the retry can adopt cleanly. + DB::Cas::MountLease fenced = DB::Cas::decodeMountLease(got->bytes); + fenced.gc_fenced = true; + fenced.seq += 1; + (void)inner->write(key, DB::Cas::encodeMountLease(fenced), got->value, access); + fence_key.clear(); + } + return got; + } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } @@ -1741,25 +1819,33 @@ TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) namespace { -/// Forces the FIRST `putIfAbsent` whose key contains `fault_key_substr` to throw an ambiguous -/// (Unresolved-classified) exception, `fault_count` times -- the minimal one-shot subset of -/// `RefWriterTestBackend`'s fault injection (gtest_cas_ref_writer.cpp) this file's shutdown test needs -/// to drive a ref-log append into the `Unresolved`/wedge outcome, with `max_attempts = 1` in the budget -/// so the single failed attempt exhausts the retry budget immediately. +/// Makes every write whose key contains `fault_key_substr` throw an ambiguous exception -- the minimal +/// subset of `RefWriterTestBackend`'s fault injection (gtest_cas_ref_writer.cpp) this file's shutdown +/// and remount tests need to drive a ref-log append into the wedge outcome. It stays armed: one +/// ambiguous attempt is not a wedge, because the engine resolves it by reading and reissues -- the lane +/// wedges only once a bound refuses with an attempt already sent, so the tests injecting it also give +/// the pool a clock they can advance. class UnresolvedPutBackend final : public DB::Cas::tests::CountingBackend { public: String fault_key_substr; int fault_count = 0; - DB::Cas::PutResult putIfAbsent(const String & key, const String & bytes, const DB::Cas::ObjectMeta & meta) override + /// The fault sits on the WRITE PRIMITIVE: the ref-log append it models is issued there. Nothing + /// reaches the store, so the engine's resolve read proves the key absent and every reissue is + /// ambiguous again -- which is what leaves the lane wedged once a bound refuses. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, DB::Cas::TransportAccess & access) override { - if (fault_count > 0 && !fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) + /// Only the create: a ref-log append is a create-if-absent, so a conditional write on the same + /// key must not consume the fault. + if (!expected_value && fault_count > 0 && !fault_key_substr.empty() + && key.find(fault_key_substr) != String::npos) { --fault_count; throw Poco::TimeoutException("UnresolvedPutBackend: simulated ambiguous result (response lost)"); } - return DB::Cas::tests::CountingBackend::putIfAbsent(key, bytes, meta); + return DB::Cas::tests::CountingBackend::write(key, bytes, expected_value, access); } }; @@ -1775,14 +1861,22 @@ class RuntimeRenewBackend final : public DB::Cas::tests::CountingBackend BlockThenThrow, }; - using DB::Cas::tests::CountingBackend::putOverwrite; - Fault fault = Fault::None; DB::Cas::tests::ManualBarrier * barrier = nullptr; std::function after_commit; - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override + /// Runs just before an armed fault throws. The engine draws its inter-attempt backoff randomly and + /// admits the reissue against that drawn duration, so a test that needs the ambiguity to be refused + /// rather than reissued has to move the injected clock here -- from inside the attempt, which is the + /// only point between admission and the resolve read a test can reach. + std::function before_throw; + + /// The fault sits on the WRITE PRIMITIVE, and only on a CONDITIONAL one: a lease renewal is a + /// replace, so a create on the same key must not consume the one-shot fault. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, DB::Cas::TransportAccess & access) override { + if (!expected_value) + return DB::Cas::tests::CountingBackend::write(key, bytes, expected_value, access); const Fault current = std::exchange(fault, Fault::None); if (current == Fault::BlockThenDelegate || current == Fault::BlockThenThrow) { @@ -1791,9 +1885,13 @@ class RuntimeRenewBackend final : public DB::Cas::tests::CountingBackend barrier->arriveAndWait(); } if (current == Fault::ThrowBefore || current == Fault::BlockThenThrow) + { + if (before_throw) + before_throw(); throw Poco::TimeoutException("injected runtime renewal ambiguity before result"); + } - PutResult result = DB::Cas::tests::CountingBackend::putOverwrite(key, bytes, expected, meta); + auto result = DB::Cas::tests::CountingBackend::write(key, bytes, expected_value, access); if (after_commit) after_commit(); if (current == Fault::LandThenThrow) @@ -1804,6 +1902,52 @@ class RuntimeRenewBackend final : public DB::Cas::tests::CountingBackend CasRequestBudget runtimeRenewBudget(uint32_t max_attempts); +/// A directly-constructed `CasMountRuntime` plus the two request planes it needs. `Pool` builds those +/// from its own members; a test has no `Pool`, so the mount plane's fence reaches the runtime through +/// this holder -- the closures run only once the runtime is issuing requests, well after construction. +class RuntimeUnderTest +{ +public: + template + RuntimeUnderTest(DB::Cas::BackendPtr backend, Args &&... args) + : mount(backend, DB::Cas::Fence{ + [this] { return runtime.fenceGeneration(); }, + [this](uint64_t g, uint64_t needed) { return runtime.admit(g, needed); }, + [this](uint64_t g) { runtime.checkFenceOrThrow(g); }}) + , farewell(backend, DB::Cas::Fence::open()) + , runtime(backend, mount, farewell, std::forward(args)...) + { + /// The runtime arms its lease deadline on ITS boot clock, and the engine measures that deadline + /// against the clock it reads. Production runs both on `CLOCK_BOOTTIME`, so they agree; a test + /// that injects one MUST inject the other, or `Retry::untilLeaseSafe` compares a synthetic + /// deadline against real boottime, finds it long past, and refuses every request unsent. + mount.setNowFnForTest([this] { return runtime.bootMsNow(); }); + farewell.setNowFnForTest([this] { return runtime.bootMsNow(); }); + } + + /// The workers are joined HERE, not only by the tests that assert on teardown: `CasMountRuntime` + /// aborts the process when it is destroyed with a worker still joinable, so an exception on any + /// path out of a test body -- a barrier that timed out, an assertion that threw -- would take the + /// whole binary down and hide every test after it. + ~RuntimeUnderTest() + { + try + { + runtime.stopBackgroundWorkers(); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + } + } + + CasMountRuntime & operator*() { return runtime; } + +private: + DB::Cas::CasRequests mount; + DB::Cas::CasRequests farewell; + CasMountRuntime runtime; +}; + enum class ForeignConflictSinkBehavior : uint8_t { ReenterSameRuntime, @@ -1822,7 +1966,7 @@ void verifyForeignConflictSinkIsNonInterfering(ForeignConflictSinkBehavior behav uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, server_root_id, uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, server_root_id, uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); std::vector events; bool reentered = false; @@ -1850,7 +1994,7 @@ void verifyForeignConflictSinkIsNonInterfering(ForeignConflictSinkBehavior behav throw std::runtime_error("injected mount diagnostic sink failure"); } }; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ @@ -1861,6 +2005,7 @@ void verifyForeignConflictSinkIsNonInterfering(ForeignConflictSinkBehavior behav sink, runtimeRenewBudget(1), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); @@ -1950,14 +2095,14 @@ TEST(CASPoolRemount, ThrowingForeignConflictSinkCannotReplaceTerminalOutcome) class RemountStepBackend final : public DB::Cas::tests::CountingBackend { public: - void failNextGet(String key) + void failNextRead(String key) { failed_key = std::move(key); } - /// The fault sits on the PRIMITIVE, not on legacy `get`: the lifecycle gate reads `_pool_meta` - /// through `probeSentinelRaw`, which speaks the primitives. A legacy caller reaches this anyway, - /// through the forwarder, so arming it here covers both surfaces rather than only one. + /// The fault sits on the READ PRIMITIVE: the lifecycle gate reads `_pool_meta` through + /// `probeSentinelRaw`, which speaks the primitives. A legacy caller reaches it anyway, through the + /// forwarder, so arming it here covers both surfaces rather than only one. std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (!failed_key.empty() && key == failed_key) @@ -2107,8 +2252,14 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); + uint64_t fake_boot = 1'000'000; auto store = DB::Cas::Pool::open(backend, DB::Cas::PoolConfig{ - .pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); + .pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget, + .boot_ms_fn = [&fake_boot] { return fake_boot; }, + .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }}); + /// The engine's own inter-attempt sleep advances the same clock its deadlines are read from, so the + /// retry bound is reached in test time rather than in ninety real seconds. + store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); /// By value: `layout` is used after `store.reset()` below, a reference would dangle. const Layout layout = store->layout(); const RootNamespace ns{"srv/wedge_shutdown"}; @@ -2118,10 +2269,11 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) DB::Cas::tests::casAdmitRecoverableEntry(*backend, layout, ns, store->liveWriterEpoch()); publishPart(store, ns.string(), "x", "payload"); - /// Force the ref-log append the drop below performs into the Unresolved/wedge outcome (as in the - /// wedge tests in gtest_cas_ref_writer.cpp): the single attempt the budget allows fails ambiguously. + /// Force the ref-log append the drop below performs into the wedge outcome (as in the wedge tests + /// in gtest_cas_ref_writer.cpp): every attempt is ambiguous, so the lane is still unresolved when + /// the retry bound refuses. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; + backend->fault_count = std::numeric_limits::max(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); @@ -2138,7 +2290,7 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) /// A successor claimMount on this body must return LiveDoubleStart (unclean path): no certificate of /// death (not fenced, not the clean farewell marker, no proven-dead observation) justifies a /// same-uuid, different-epoch reclaim. - const MountClaimResult claim = claimMount(*backend, layout, "test", lease.server_uuid, + const MountClaimResult claim = claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", lease.server_uuid, lease.writer_epoch + 1, /*now_ms=*/1, /*ttl_ms=*/30000); EXPECT_EQ(claim.kind, MountClaimResult::LiveDoubleStart); } @@ -2161,7 +2313,7 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) /// Predecessor: claim epoch 7, no farewell (simulate crash: just drop the keeper) -- a bare /// `claimMount` plants the lease directly, with no clean-farewell `min_active_build_sequence` marker and no /// `gc_fenced`, so the successor below has no certificate of death until it observes one itself. - ASSERT_EQ(claimMount(*b, l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(b), l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, MountClaimResult::Claimed); /// A real predecessor at epoch 7 durably minted it first (`allocateWriterEpoch` always runs /// before the mount claim); seed that durable epoch object here too, or the successor's own @@ -2229,7 +2381,7 @@ TEST(CASMountOpenWaits, FencedPriorReclaimsWithoutAnyWait) auto b = std::make_shared(); Layout l{"p"}; DB::Cas::tests::seedPoolMetaForRestart(*b); - ASSERT_EQ(claimMount(*b, l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(b), l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, MountClaimResult::Claimed); /// A real predecessor at epoch 7 durably minted it first (`allocateWriterEpoch` always runs /// before the mount claim); seed that durable epoch object here too, or the successor's own @@ -2279,10 +2431,14 @@ class StalledMountClaimBackend final : public DB::Cas::InMemoryBackend std::atomic mount_writes{0}; std::atomic mount_writes_after_stall{0}; - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, - const DB::Cas::ObjectMeta & m) override + /// The hook sits on the WRITE PRIMITIVE: both the reclaim and the keeper's adopt reach the mount + /// slot through it. It counts only CONDITIONAL overwrites, which is what every production mount-slot + /// write is -- the legacy `putIfAbsent` that seeds the predecessor lease forwards through this same + /// virtual, and counting it would shift the stall onto the reclaim instead of the adopt. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, DB::Cas::TransportAccess & access) override { - if (k == mount_key) + if (key == mount_key && expected_value) { const int n = ++mount_writes; if (n == 2 && on_second_mount_write) @@ -2290,7 +2446,7 @@ class StalledMountClaimBackend final : public DB::Cas::InMemoryBackend else if (n > 2) ++mount_writes_after_stall; } - return InMemoryBackend::putOverwrite(k, b, e, m); + return InMemoryBackend::write(key, bytes, expected_value, access); } }; } @@ -2375,6 +2531,7 @@ TEST(CASRemountWaits, DrainedRemountPaysNoWait) }); ASSERT_TRUE(store); EXPECT_TRUE(waits.empty()) << "a fresh mount (no predecessor) pays no wait at open"; + store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); /// Trip the fence: advance the local boot clock past the deadline (as in `WriteFenceUsesInjectedBootClock` /// above) and mark the durable lease `gc_fenced` (the certificate `claimMountAwaitingExpiry` reclaims @@ -2409,6 +2566,12 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) }); ASSERT_TRUE(store); EXPECT_TRUE(waits.empty()) << "a fresh mount (no predecessor) pays no wait at open"; + /// `dropRef` below drives the fault through `ensureRefTableRecovered`'s own recovery-retry loop, + /// which sleeps via `recovery_retry_sleep_fn` (a REAL 200ms-slice sleep by default) while measuring + /// elapsed time against `boot_ms_now_fn` -- the frozen `fake_boot` this fixture already injects. + /// Without also virtualizing the sleep, that elapsed check never advances and the loop spins for + /// real until the harness times the test out. + store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); const Layout & layout = store->layout(); const RootNamespace ns{"srv/remount_wedge"}; @@ -2420,7 +2583,7 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) /// `CASPoolShutdown.UnresolvedWedgeSkipsFarewell`): the single attempt the budget allows fails /// ambiguously. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; + backend->fault_count = std::numeric_limits::max(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); @@ -2469,6 +2632,7 @@ TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPrede .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; }, }); ASSERT_TRUE(store); + store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); const Layout & layout = store->layout(); const RootNamespace ns1{"srv/table_a"}; @@ -2488,7 +2652,7 @@ TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPrede /// Force ns1's ref-log append into the Unresolved/wedge outcome (mirrors /// `UnresolvedWedgeRemountPaysNoWaitEither` above). backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns1)) + "_log/"; - backend->fault_count = 1; + backend->fault_count = std::numeric_limits::max(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns1, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns1)); @@ -2715,12 +2879,13 @@ TEST(CASPoolRemount, DirectRenewCannotRaceWorkerStartOrKeeperReplacement) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .boot_ms_fn = [&] { return boot_ms; }}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -2745,11 +2910,11 @@ TEST(CASPoolRemount, DueWorkerAdmissionIsReservedBeforeParkRequest) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier admitted; DB::Cas::tests::ManualBarrier remount; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), @@ -2761,6 +2926,7 @@ TEST(CASPoolRemount, DueWorkerAdmissionIsReservedBeforeParkRequest) remount.arriveAndWait(); return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -2784,10 +2950,10 @@ TEST(CASPoolRemount, DueWorkerAdmissionIsReservedBeforeStop) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier admitted; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), @@ -2795,6 +2961,7 @@ TEST(CASPoolRemount, DueWorkerAdmissionIsReservedBeforeStop) .boot_ms_fn = [&] { return boot_ms; }, .renewal_admitted_hook_for_test = [&] { admitted.arriveAndWait(); }}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -2815,13 +2982,14 @@ TEST(CASPoolRemount, DirectRenewIsRefusedForBackgroundConfiguredRuntimeAfterStop uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -2840,12 +3008,12 @@ TEST(CASPoolRemount, RemountWaitsForRenewalParkedBeforeReplacement) uint64_t wall_ms = 1000; uint64_t boot_ms = 10'000; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier renewal_barrier; DB::Cas::tests::ManualBarrier remount_barrier; std::atomic remount_calls{0}; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, @@ -2855,6 +3023,7 @@ TEST(CASPoolRemount, RemountWaitsForRenewalParkedBeforeReplacement) remount_barrier.arriveAndWait(); return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -2881,7 +3050,7 @@ TEST(CASPoolRemount, TeardownJoinsBothWorkersBeforeRelease) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); std::atomic worker_exits{0}; RuntimeWorkerFactory factory = [&](std::function worker_body) { @@ -2892,11 +3061,12 @@ TEST(CASPoolRemount, TeardownJoinsBothWorkersBeforeRelease) }); }; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }, .worker_factory = factory}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -2919,7 +3089,7 @@ TEST(CASPoolRemount, NaturalTerminalTransitionMakesBothPersistentWorkersSelfExit uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); WorkerExitLatch exits; DB::Cas::tests::ManualBarrier transitioned; RuntimeWorkerFactory factory = [&](std::function worker_body) @@ -2932,7 +3102,7 @@ TEST(CASPoolRemount, NaturalTerminalTransitionMakesBothPersistentWorkersSelfExit }; CasMountRuntime * runtime_ptr = nullptr; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), @@ -2948,6 +3118,7 @@ TEST(CASPoolRemount, NaturalTerminalTransitionMakesBothPersistentWorkersSelfExit transitioned.arriveAndWait(); return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); @@ -2976,7 +3147,7 @@ TEST(CASPoolRemount, ParkedRenewalCannotMissNaturalTerminalPublication) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); WorkerExitLatch exits; std::latch renewal_before_driver_lock{1}; std::latch release_renewal{1}; @@ -3002,7 +3173,7 @@ TEST(CASPoolRemount, ParkedRenewalCannotMissNaturalTerminalPublication) }; CasMountRuntime * runtime_ptr = nullptr; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), @@ -3056,6 +3227,7 @@ TEST(CASPoolRemount, ParkedRenewalCannotMissNaturalTerminalPublication) release_parked(); return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); @@ -3084,7 +3256,7 @@ TEST(CASPoolRemount, VanishedReasonPreparationFailureLeavesTerminalTransitionRet uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); WorkerExitLatch exits; RuntimeWorkerFactory factory = [&](std::function worker_body) { @@ -3096,7 +3268,7 @@ TEST(CASPoolRemount, VanishedReasonPreparationFailureLeavesTerminalTransitionRet }; std::atomic preparation_calls{0}; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), @@ -3109,6 +3281,7 @@ TEST(CASPoolRemount, VanishedReasonPreparationFailureLeavesTerminalTransitionRet throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected vanished-reason preparation failure"); }}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -3153,13 +3326,14 @@ TEST(CASPoolRemount, WorkerConstructionRollbackFailsOpenClosed) return ThreadFromGlobalPool(std::move(fn)); }; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }, .worker_factory = factory}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -3177,13 +3351,16 @@ TEST(CASPoolRemount, ExternalLossDuringRenewalUsesOneRecoveryGeneration) uint64_t wall_ms = 1000; uint64_t boot_ms = 100'000; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier renewal_barrier; DB::Cas::tests::ManualBarrier remount_barrier; std::atomic remount_calls{0}; std::atomic fresh_epochs{0}; CasEventSink sink; - CasMountRuntime runtime( + /// The remount callback reaches the runtime it is installed on, so it goes through a pointer the + /// line after construction fills in -- the callback runs only once the workers are started. + CasMountRuntime * runtime_ptr = nullptr; + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, @@ -3192,19 +3369,21 @@ TEST(CASPoolRemount, ExternalLossDuringRenewalUsesOneRecoveryGeneration) ++remount_calls; ++fresh_epochs; fenceOutMount(*backend, layout.mountKey("test")); - const MountClaimResult fresh = claimMount(*backend, layout, "test", uuid, 2, wall_ms, 1000); + const MountClaimResult fresh = claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 2, wall_ms, 1000); EXPECT_EQ(fresh.kind, MountClaimResult::Claimed); if (fresh.kind != MountClaimResult::Claimed) return false; - runtime.installKeeper(uuid, 2, [&] { return wall_ms; }); - const uint64_t fresh_anchor = runtime.startKeeper(); - runtime.setProcessEpoch(2, std::memory_order_release); - runtime.setLiveWriterEpoch(2); - runtime.armMountFence(uuid, 2, fresh_anchor + 1000); - runtime.noteRemounted(); + runtime_ptr->installKeeper(uuid, 2, [&] { return wall_ms; }); + const uint64_t fresh_anchor = runtime_ptr->startKeeper(); + runtime_ptr->setProcessEpoch(2, std::memory_order_release); + runtime_ptr->setLiveWriterEpoch(2); + runtime_ptr->armMountFence(uuid, 2, fresh_anchor + 1000); + runtime_ptr->noteRemounted(); remount_barrier.arriveAndWait(); return true; }); + CasMountRuntime & runtime = *runtime_holder; + runtime_ptr = &runtime; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -3233,13 +3412,13 @@ TEST(CASPoolRemount, TerminalDepositionDoesNotTouchKeeperAfterReplacement) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier terminal_deposited; DB::Cas::tests::ManualBarrier remount; std::atomic replaced{false}; CasMountRuntime * runtime_ptr = nullptr; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), @@ -3258,11 +3437,17 @@ TEST(CASPoolRemount, TerminalDepositionDoesNotTouchKeeperAfterReplacement) remount.arriveAndWait(); return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; + /// Expire the lease from inside the attempt. The fault alone no longer ends a renewal: the engine + /// settles the ambiguity by reading and then reissues, and the reissue commits. With the clock past + /// the deadline the renewal was admitted under, neither the settling read nor the reissue is + /// admitted, so the renewal ends terminal -- which is what this test deposits. + backend->before_throw = [&, deadline = anchor + 1000] { boot_ms = deadline; }; runtime.startBackgroundWorkers(std::chrono::milliseconds(0)); terminal_deposited.waitUntilArrived(); EXPECT_TRUE(replaced.load(std::memory_order_acquire)); @@ -3280,12 +3465,12 @@ TEST(CASPoolRemount, ConcurrentRemountRequestIsProcessedAfterActiveGeneration) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier first; DB::Cas::tests::ManualBarrier second; std::atomic calls{0}; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, @@ -3295,6 +3480,7 @@ TEST(CASPoolRemount, ConcurrentRemountRequestIsProcessedAfterActiveGeneration) (call == 1 ? first : second).arriveAndWait(); return true; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -3319,12 +3505,15 @@ TEST(CASPoolRemount, ImmediatePostRemountRenewalFailureIsNotDropped) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 10'000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 10'000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier first; DB::Cas::tests::ManualBarrier second; std::atomic calls{0}; CasEventSink sink; - CasMountRuntime runtime( + /// The remount callback reaches the runtime it is installed on, so it goes through a pointer the + /// line after construction fills in -- the callback runs only once the workers are started. + CasMountRuntime * runtime_ptr = nullptr; + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(10'000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, @@ -3334,22 +3523,28 @@ TEST(CASPoolRemount, ImmediatePostRemountRenewalFailureIsNotDropped) if (call == 1) { fenceOutMount(*backend, layout.mountKey("test")); - const MountClaimResult fresh = claimMount(*backend, layout, "test", uuid, 2, wall_ms, 10'000); + const MountClaimResult fresh = claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 2, wall_ms, 10'000); EXPECT_EQ(fresh.kind, MountClaimResult::Claimed); if (fresh.kind != MountClaimResult::Claimed) return false; - runtime.installKeeper(uuid, 2, [&] { return wall_ms; }); - const uint64_t fresh_anchor = runtime.startKeeper(); - runtime.armMountFence(uuid, 2, fresh_anchor + 10'000); - runtime.noteRemounted(); + runtime_ptr->installKeeper(uuid, 2, [&] { return wall_ms; }); + const uint64_t fresh_anchor = runtime_ptr->startKeeper(); + runtime_ptr->armMountFence(uuid, 2, fresh_anchor + 10'000); + runtime_ptr->noteRemounted(); boot_ms = 2'000; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; + /// Expire the fresh lease from inside the attempt, so the ambiguity can be neither + /// settled by a read nor reissued: otherwise the engine reissues and the renewal + /// commits, and there is no dropped failure to catch up on. + backend->before_throw = [&, deadline = fresh_anchor + 10'000] { boot_ms = deadline; }; first.arriveAndWait(); return true; } second.arriveAndWait(); return false; }); + CasMountRuntime & runtime = *runtime_holder; + runtime_ptr = &runtime; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 10'000); @@ -3421,12 +3616,15 @@ TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) result_observed.set_value(); }, .mount_lease_ttl_ms = std::chrono::milliseconds(1000), - .mount_renew_period = std::chrono::milliseconds(100), - .cas_request_budget = runtimeRenewBudget(2), + /// 500 with a 700 ms quiescence, so the redo's window (period + attempt timeout = 510) does not + /// fit the 280 ms of safe lease left -- and the reissue the ambiguity needs still does, whatever + /// the engine's jittered backoff draws from its first-reissue range of at most 200 ms. + .mount_renew_period = std::chrono::milliseconds(500), + .cas_request_budget = runtimeRenewBudget(), .boot_ms_fn = [&] { return fake_boot; }, .remount_quiesce_hook_for_test = [&] { - fake_boot += 900; + fake_boot += 700; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; }, }; @@ -3442,10 +3640,6 @@ TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) std::lock_guard lock(events_mutex); observed = events; } - const auto retrying = std::find_if(observed.begin(), observed.end(), [](const CasEvent & event) - { - return event.type == CasEventType::WatermarkRenew && event.outcome == "retrying"; - }); const auto recovered = std::find_if(observed.begin(), observed.end(), [](const CasEvent & event) { return event.type == CasEventType::WatermarkRenew && event.outcome == "recovered"; @@ -3454,14 +3648,13 @@ TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) { return event.type == CasEventType::MountRemount && event.outcome == "ok"; }); - ASSERT_NE(retrying, observed.end()); ASSERT_NE(recovered, observed.end()); ASSERT_NE(remounted, observed.end()); - EXPECT_LT(std::distance(observed.begin(), retrying), std::distance(observed.begin(), recovered)); EXPECT_LT(std::distance(observed.begin(), recovered), std::distance(observed.begin(), remounted)); - EXPECT_EQ(retrying->detail.at("remount_attempt_no"), remounted->detail.at("attempt_no")); EXPECT_EQ(recovered->detail.at("remount_attempt_no"), remounted->detail.at("attempt_no")); EXPECT_EQ(recovered->detail.at("classification"), "committed_after_retry"); + /// The physical retry itself: the ambiguous attempt and the reissue that committed. + EXPECT_EQ(recovered->detail.at("attempts_sent"), "2"); EXPECT_NE(renewal_logs.captured().find("CAS mount renewal 'test' recovered"), String::npos); /// `~Pool` stops and joins both persistent runtime workers. Make that quiescence boundary part of @@ -3495,12 +3688,15 @@ TEST(CASPoolRemount, ParkedRedoFailureObservabilityPrecedesRemountResult) }, .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .mount_renew_period = std::chrono::milliseconds(100), - .cas_request_budget = runtimeRenewBudget(1), + .cas_request_budget = runtimeRenewBudget(), .boot_ms_fn = [&] { return fake_boot; }, .remount_quiesce_hook_for_test = [&] { fake_boot += 900; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; + /// The attempt is admitted 80 ms before its lease-safe bound; spending 90 inside it puts the + /// resolve read past that bound, so the ambiguity is refused instead of reissued. + backend->before_throw = [&] { fake_boot += 90; }; }, }; auto store = Pool::open(backend, config); @@ -3528,7 +3724,7 @@ TEST(CASPoolRemount, ParkedRedoFailureObservabilityPrecedesRemountResult) EXPECT_LT(std::distance(observed.begin(), failed_renew), std::distance(observed.begin(), failed_remount)); EXPECT_EQ(failed_renew->detail.at("remount_attempt_no"), failed_remount->detail.at("attempt_no")); EXPECT_EQ(failed_renew->detail.at("attempts_sent"), "1"); - EXPECT_EQ(failed_renew->detail.at("classification"), "attempts_exhausted"); + EXPECT_EQ(failed_renew->detail.at("classification"), "external_lease_deadline"); EXPECT_NE(renewal_logs.captured().find("CAS mount renewal 'test' fenced"), String::npos); /// A ready final-result future proves publication order; destruction additionally proves the @@ -3569,17 +3765,18 @@ TEST(CASPoolShutdown, PreSendCancellationAllowsFarewellButAmbiguityDoesNot) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - const MountClaimResult claim = claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000); + const MountClaimResult claim = claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000); EXPECT_EQ(claim.kind, MountClaimResult::Claimed); if (claim.kind != MountClaimResult::Claimed) return uint64_t{0}; DB::Cas::tests::ManualBarrier barrier; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); @@ -3608,38 +3805,34 @@ TEST(CASPoolShutdown, PreSendCancellationAllowsFarewellButAmbiguityDoesNot) TEST(CASPool, DirectAndStartupTerminalFailuresRethrowTypedExceptions) { - enum class Refusal : uint8_t { PreAttemptDeadline, CancelledAfterSend, FenceLostAfterSend }; + enum class Refusal : uint8_t { PreAttemptDeadline, RefusedAfterSend }; const auto run = [](bool startup, Refusal refusal) { auto backend = std::make_shared(); const Layout layout(startup ? "typed-startup" : "typed-direct"); uint64_t wall_ms = 1000; uint64_t boot_ms = 100; - std::atomic stop_cause{CasOverwriteStopCause::Continue}; + std::atomic renewal_live{true}; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{ .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .boot_ms_fn = [&] { return boot_ms; }, - .renewal_stop_cause_for_test = [&] { return stop_cause.load(std::memory_order_acquire); }}, + .renewal_live_for_test = [&] { return renewal_live.load(std::memory_order_acquire); }}, "test", sink, runtimeRenewBudget(), [] { return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); if (refusal == Refusal::PreAttemptDeadline) - boot_ms = 1071; + /// Past the point where the lease has more room left than the safety margin (deadline + /// `anchor + 1000` == 1100, margin 20), so admission refuses before anything is sent. + boot_ms = 1090; else - backend->after_commit = [&] - { - stop_cause.store( - refusal == Refusal::CancelledAfterSend - ? CasOverwriteStopCause::Cancelled - : CasOverwriteStopCause::FenceOrLifecycleLost, - std::memory_order_release); - }; + backend->after_commit = [&] { renewal_live.store(false, std::memory_order_release); }; try { if (startup) @@ -3655,7 +3848,7 @@ TEST(CASPool, DirectAndStartupTerminalFailuresRethrowTypedExceptions) runtime.finishTeardown(false); }; for (bool startup : {true, false}) - for (Refusal refusal : {Refusal::PreAttemptDeadline, Refusal::CancelledAfterSend, Refusal::FenceLostAfterSend}) + for (Refusal refusal : {Refusal::PreAttemptDeadline, Refusal::RefusedAfterSend}) run(startup, refusal); } @@ -3671,9 +3864,9 @@ TEST(CASPool, BackgroundCadenceMustFitLeaseBeforeWritablePublication) .cas_request_budget = runtimeRenewBudget(), }; EXPECT_THROW((void)Pool::open(backend, config), DB::Exception); - EXPECT_EQ(backend->putTotal(), 0u); - EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + /// One assertion over every write shape: the counters now sit on the write primitive, which both + /// the create- and the replace-shaped verbs reach. + EXPECT_EQ(backend->writeTotal(), 0u); } TEST(CASPool, DecommissionCadenceValidationPrecedesAuthorityWrites) @@ -3694,9 +3887,9 @@ TEST(CASPool, DecommissionCadenceValidationPrecedesAuthorityWrites) { (void)Pool::openForDecommission(backend, config, "victim"); }); - EXPECT_EQ(backend->putTotal(), 0u); - EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + /// One assertion over every write shape: the counters now sit on the write primitive, which both + /// the create- and the replace-shaped verbs reach. + EXPECT_EQ(backend->writeTotal(), 0u); } TEST(CASPool, DisabledBackgroundDoesNotReserveRenewalCadence) @@ -3725,10 +3918,10 @@ TEST(CASPool, DeterministicWorkerFailureFencesWithoutWaitingForCadence) uint64_t wall_ms = 1000; uint64_t boot_ms = 100; const UInt128 uuid{1}; - ASSERT_EQ(claimMount(*backend, layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(backend), layout, "test", uuid, 1, wall_ms, 1000).kind, MountClaimResult::Claimed); DB::Cas::tests::ManualBarrier remount_entered; CasEventSink sink; - CasMountRuntime runtime( + RuntimeUnderTest runtime_holder( backend, layout, MountConfig{.mount_lease_ttl_ms = std::chrono::milliseconds(1000), .background_watermark = true, .boot_ms_fn = [&] { return boot_ms; }}, @@ -3737,10 +3930,14 @@ TEST(CASPool, DeterministicWorkerFailureFencesWithoutWaitingForCadence) remount_entered.arriveAndWait(); return false; }); + CasMountRuntime & runtime = *runtime_holder; runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); const uint64_t anchor = runtime.startKeeper(); runtime.armMountFence(uuid, 1, anchor + 1000); backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; + /// Expire the lease from inside the attempt, so the ambiguity can be neither settled by a read nor + /// reissued: without that the engine reissues and the renewal commits, and this worker never fences. + backend->before_throw = [&, deadline = anchor + 1000] { boot_ms = deadline; }; runtime.startBackgroundWorkers(std::chrono::milliseconds(0)); remount_entered.waitUntilArrived(); EXPECT_FALSE(runtime.mayMutate()); @@ -3768,6 +3965,10 @@ TEST(CASPool, RenewWatermarkOnceRefreshesFenceAndDepositsOneFailure) EXPECT_TRUE(store->mayMutate()) << "direct success must refresh the local fence from attempt start"; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; + /// The renewal that succeeded at 500 anchored the lease for its 1000 ms TTL, so it expires at 1500. + /// Expire it from inside the attempt: the fault alone no longer ends a renewal, because the engine + /// settles the ambiguity by reading and reissues, and the reissue commits. + backend->before_throw = [&] { fake_boot = 1500; }; const uint64_t schedules_before = store->scheduleRemountCallCountForTest(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->renewWatermarkOnce(); }); EXPECT_FALSE(store->mayMutate()); @@ -3786,7 +3987,7 @@ TEST(CASPoolRemount, WholeChainResultsAreNumberedAndStepLabelled) ScopedRemountLogCapture logs; store->tripMountLost(); - backend->failNextGet(store->layout().poolMetaKey()); + backend->failNextRead(store->layout().poolMetaKey()); const uint64_t attempts_before = ProfileEvents::global_counters[ProfileEvents::CASRemountAttempts].load(); const uint64_t succeeded_before = ProfileEvents::global_counters[ProfileEvents::CASRemountSucceeded].load(); const uint64_t failed_before = ProfileEvents::global_counters[ProfileEvents::CASRemountFailed].load(); @@ -3815,6 +4016,60 @@ TEST(CASPoolRemount, WholeChainResultsAreNumberedAndStepLabelled) EXPECT_EQ(countRemountFinalLogs(logs.captured()), 2u) << logs.captured(); } +/// The remount's `keeper_redo` step re-anchors the lease BEFORE `armMountFence`, so it runs with the +/// fence still latched lost. Admitted on the mount plane it could only ever give up, and every remount +/// that reached the step would fail -- so it renews on the keeper's open plane instead. +/// +/// Driven the way production reaches the step, which is the only way it CAN be reached: the persistent +/// renewal worker runs, `scheduleRemount` parks it, and the redo is the parked driver's one call. A +/// remount driven directly with no workers leaves that driver dormant, and the step's admission refuses +/// a dormant driver rather than renewing. +/// +/// The step is reached only when quiescence has eaten most of the new lease: with a fresh anchor the +/// renewal window fits and the step is skipped entirely. So the quiesce hook advances the injected boot +/// clock to just inside the safety margin, and the paired run with no quiesce cost is the control that +/// proves the step was reached rather than skipped. +TEST(CASPoolRemount, TheKeeperRedoRenewsOnTheOpenPlane) +{ + /// One successful self-remount whose quiescence costs `quiesce_ms`; returns the conditional + /// mount-slot writes it issued. Counted while the remount worker is still held inside the event + /// sink that reported the result, so the renewal worker it un-parks cannot add one. + const auto remountConditionalMountWrites = [](uint64_t quiesce_ms) -> uint64_t + { + auto backend = std::make_shared(); + uint64_t fake_boot = 1'000'000; + DB::Cas::tests::ManualBarrier committed; + auto store = Pool::open(backend, PoolConfig{ + .pool_prefix = "remount-keeper-redo", + .server_root_id = "test", + .background_watermark = true, + .event_sink = [&committed](const CasEvent & event) + { + if (event.type == CasEventType::MountRemount && event.outcome == "ok") + committed.arriveAndWait(); + }, + .boot_ms_fn = [&fake_boot] { return fake_boot; }, + .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }, + .remount_quiesce_hook_for_test = [&fake_boot, quiesce_ms] { fake_boot += quiesce_ms; }, + }); + const String mount_key = store->layout().mountKey("test"); + + fenceOutMount(*backend, mount_key); + const uint64_t before = backend->putOverwriteCount(mount_key); + EXPECT_TRUE(store->scheduleRemountForTest()) + << "the remount must be latched with quiesce_ms=" << quiesce_ms; + committed.waitUntilArrived(); + const uint64_t writes = backend->putOverwriteCount(mount_key) - before; + committed.release(); + return writes; + }; + + /// 27 s of a 30 s lease, against a 2 s safety margin and a window of one renewal period plus one + /// attempt (15 s, since the renewal worker runs here): the window no longer fits. + EXPECT_GT(remountConditionalMountWrites(27'000), remountConditionalMountWrites(0)) + << "a quiescence that consumed the lease must cost one extra lease write -- the redo"; +} + TEST(CASPoolRemount, LeaseLossHasOneOperationalOwner) { auto backend = std::make_shared(); @@ -3826,7 +4081,7 @@ TEST(CASPoolRemount, LeaseLossHasOneOperationalOwner) store->tripMountLost(); store->tripMountLost(); - backend->failNextGet(store->layout().poolMetaKey()); + backend->failNextRead(store->layout().poolMetaKey()); EXPECT_FALSE(store->tryRemountOnce()); store->beginShutdownForTest(); store->tripMountLost(); diff --git a/src/Disks/tests/gtest_cas_pool_meta.cpp b/src/Disks/tests/gtest_cas_pool_meta.cpp new file mode 100644 index 000000000000..fc2a668e1620 --- /dev/null +++ b/src/Disks/tests/gtest_cas_pool_meta.cpp @@ -0,0 +1,76 @@ +#include + +#include +#include +#include "cas_test_helpers.h" + +#include +#include + +namespace DB +{ +namespace ErrorCodes +{ + extern const int NETWORK_ERROR; +} +} + +using namespace DB::Cas; + +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::FakeClock; +using DB::Cas::tests::expectThrowsCode; + +namespace +{ + +CasRequests makeRequests(BackendPtr backend, FakeClock & clock, Fence fence = Fence::open()) +{ + return CasRequests(std::move(backend), std::move(fence), clock.nowFn(), clock.sleepFn()); +} + +} + +TEST(CASPoolMeta, AdmitOrValidateEndsAtTheDeadlineUnderPerpetualConflict) +{ + FakeClock clock; + auto backend = std::make_shared(); + Layout layout("pool"); + auto requests = makeRequests(backend, clock); + + auto create_op = requests.admit(); + const PoolMeta created = PoolMeta::createOrValidate( + create_op, layout, /*blob_header_len=*/256, /*gc_shards=*/1, + BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/true); + EXPECT_EQ(created.algos_used, (std::vector{static_cast(BlobHashAlgo::CityHash128)})); + + /// Rig the key permanently hot: every write attempt races a concurrent rewrite of the SAME + /// content, so the object's incarnation moves under every attempt and admission of a new algo + /// never lands. `putOverwrite` mints a fresh incarnation even though the bytes are unchanged. + const String key = layout.poolMetaKey(); + EXPECT_TRUE(clock.sleeps.empty()); /// nothing paced yet -- the trailing check below is about THIS call + bool inside_hook = false; + backend->onBeforeWrite(key, [&] + { + if (inside_hook) + return; + inside_hook = true; + if (auto cur = backend->get(key)) + (void)backend->putOverwrite(key, cur->bytes, cur->token); + inside_hook = false; + }); + + /// `orThrow`'s `GaveUp{Deadline}` arm throws exactly `NETWORK_ERROR` (`throwCasWriteRetryLater`), + /// pinning the deadline outcome apart from the two failures a wrong migration could also throw as + /// a `DB::Exception` here: `LOGICAL_ERROR` (the absence branch) or `BAD_ARGUMENTS` (`allow_new` + /// plumbing regressed). + auto admit_op = requests.admit(); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + { + (void)PoolMeta::createOrValidate( + admit_op, layout, 256, /*gc_shards=*/1, + BlobHashAlgo::XXH3_128, /*allow_new=*/true, /*allow_mint=*/false); + }); + /// Bounded by the deadline, not a live-lock: it paced its retries rather than spinning. + EXPECT_FALSE(clock.sleeps.empty()); +} diff --git a/src/Disks/tests/gtest_cas_probe.cpp b/src/Disks/tests/gtest_cas_probe.cpp index 1aa2467611f1..4d87da4543f2 100644 --- a/src/Disks/tests/gtest_cas_probe.cpp +++ b/src/Disks/tests/gtest_cas_probe.cpp @@ -1,74 +1,63 @@ #include +#include #include -#include #include #include +#include #include #include -#include -#include #include -namespace DB +using namespace DB::Cas; + +namespace { -namespace ErrorCodes + +/// Every test here constructs one backend and runs the battery against it once; a non-owning +/// `BackendPtr` over the test's stack- or shared_ptr-held backend keeps that construction pattern +/// rather than forcing a second allocation. The open fence never trips: `runCapabilityProbe` runs +/// during pool bootstrap, before any mount fence exists to enforce. +CasRequests makeRequests(Backend & backend) { - extern const int NOT_IMPLEMENTED; -} + return CasRequests(BackendPtr(&backend, [](Backend *) {}), Fence::open()); } -using namespace DB::Cas; +} TEST(CASProbe, PassesOnEnforcingBackend) { auto b = std::make_shared(); - EXPECT_NO_THROW(runCapabilityProbe(*b, "p/.cas_probe")); + auto requests = makeRequests(*b); + auto op = requests.admit(); + EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")); EXPECT_TRUE(b->list("p/.cas_probe", "", 10).keys.empty()); // probe cleans up after itself } -/// AWS S3 answers 400 InvalidArgument to a conditional DELETE with an EMPTY If-Match, and the -/// probe's exit cleanup used to issue exactly that (deleteExact with the absent HeadResult's empty -/// token) after step 8 had already deleted the probe keys — two scary AWSClient log lines -/// on every real-S3 mount. The cleanup must HEAD-gate the delete instead of firing blindly. -class EmptyTokenDeleteRecorder : public InMemoryBackend -{ -public: - size_t empty_token_deletes = 0; - - DeleteOutcome deleteExact(const String & key, const Token & token) override - { - if (token.empty()) - ++empty_token_deletes; - return InMemoryBackend::deleteExact(key, token); - } -}; - -TEST(CASProbe, CleanupNeverDeletesWithEmptyToken) -{ - auto b = std::make_shared(); - EXPECT_NO_THROW(runCapabilityProbe(*b, "p/.cas_probe")); - EXPECT_EQ(b->empty_token_deletes, 0u); -} - TEST(CASProbe, FailsClosedOnNonEnforcingDelete) { auto b = std::make_shared(); b->setEnforceTokens(false); // the MinIO-OSS failure mode - EXPECT_THROW(runCapabilityProbe(*b, "p/.cas_probe"), DB::Exception); + auto requests = makeRequests(*b); + auto op = requests.admit(); + EXPECT_THROW(runCapabilityProbe(op, "p/.cas_probe"), DB::Exception); } TEST(CASProbe, FailsClosedOnDeleteMarkers) { auto b = std::make_shared(); b->setSimulateDeleteMarkers(true); // versioning enabled on the prefix - EXPECT_THROW(runCapabilityProbe(*b, "p/.cas_probe"), DB::Exception); + auto requests = makeRequests(*b); + auto op = requests.admit(); + EXPECT_THROW(runCapabilityProbe(op, "p/.cas_probe"), DB::Exception); } TEST(CASProbe, PassesOnEmulatedLocal) { auto b = std::make_shared( DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); - EXPECT_NO_THROW(runCapabilityProbe(*b, "p/.cas_probe")); + auto requests = makeRequests(*b); + auto op = requests.admit(); + EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")); } /// B135: two servers mounting the SAME shared CA pool concurrently must not race on the probe keys. @@ -78,6 +67,9 @@ TEST(CASProbe, PassesOnEmulatedLocal) /// throws NOT_IMPLEMENTED ("putIfAbsent on a fresh key returned PreconditionFailed"). With the /// per-mount unique probe prefix `/_probe//token`, the seeded key does not collide and /// the open succeeds — exactly the concurrent-shared-pool-mount behaviour we need. +/// +/// Goes through `Pool::open` (owned elsewhere), so it exercises `runCapabilityProbe` only indirectly +/// and needs no signature change here. TEST(CASProbe, ConcurrentMountsDoNotCollide) { auto b = std::make_shared(); @@ -96,43 +88,11 @@ TEST(CASProbe, ConcurrentMountsDoNotCollide) EXPECT_TRUE(b->get("p/_probe/token").has_value()); } -/// The probe must consult the backend's store-preconditions hook BEFORE the op battery: a -/// generation-dialect store on a VERSIONED bucket passes every conditional-op check, but its -/// token-exact DELETEs archive noncurrent generations instead of reclaiming storage — only the -/// hook can see that, so a throwing hook must fail the probe closed. -class PreconditionRefusingBackend : public InMemoryBackend -{ -public: - void checkPoolPreconditions() override - { - throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, - "test: store precondition violated (e.g. bucket versioning enabled)"); - } -}; - -TEST(CASProbe, FailsClosedOnPoolPreconditions) -{ - auto b = std::make_shared(); - EXPECT_THROW(runCapabilityProbe(*b, "p/.cas_probe"), DB::Exception); - /// The hook fires FIRST: no probe keys may have been written. - EXPECT_TRUE(b->list("p/.cas_probe", "", 10).keys.empty()); -} - -/// `Pool::open` wraps the pool backend in `InstrumentedBackend` BEFORE calling `runCapabilityProbe` -/// (see CasPool.cpp), so the hook must actually fire THROUGH the wrapper on the real mount path — -/// not just on a raw backend, which `FailsClosedOnPoolPreconditions` above already covers. -TEST(CASProbe, PoolPreconditionsFireThroughInstrumentedWrapper) -{ - auto inner = std::make_shared(); - InstrumentedBackend wrapped(inner); - EXPECT_THROW(runCapabilityProbe(wrapped, "p/.cas_probe"), DB::Exception); - /// The hook fires FIRST: no probe keys may have been written to the inner backend. - EXPECT_TRUE(inner->list("p/.cas_probe", "", 10).keys.empty()); -} - /// RFC cas-s3-timeout-retry-control: a Native-mode mount over an object storage that does not support /// the SingleAttempt retry profile must never silently proceed under the disk's default (~500-attempt) -/// transparent retry policy — see Backend::checkConditionalWriteSingleAttemptSupport. +/// transparent retry policy — see Backend::checkConditionalWriteSingleAttemptSupport. This calls the +/// hook directly on the backend (not through `runCapabilityProbe`, which no longer runs it — see +/// CasProbe.h), so it is unaffected by the request-contract migration. /// LocalObjectStorage never supports the profile (IObjectStorage::supportsRetryProfile's default /// implementation only answers true for Default), so Native mode over it is exactly the case this must /// refuse. EmulatedSingleProcess is exempt: it never claims single-attempt S3 semantics in the first @@ -148,68 +108,34 @@ TEST(CASProbe, FailsClosedOnUnsupportedSingleAttemptProfile) EXPECT_NO_THROW(emulated->checkConditionalWriteSingleAttemptSupport()); } -/// The same fail-closed refusal through the actual capability probe (Step 0b) — the real gate a -/// writable Pool::open goes through, not just the hook in isolation above. -TEST(CASProbe, MissingSingleAttemptClientFailsCapabilityProbe) -{ - auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); - /// Native mode passes the key to the object storage verbatim, so the probe prefix must be anchored - /// under this storage's own root: a bare prefix lands beside the test process, where an object left - /// by another run answers the LIST below and an unrooted LIST answers "no keys" for free. - const String probe_prefix = DB::Cas::tests::nativeKeyUnder(storage, "p/.cas_probe"); - - auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); - EXPECT_THROW(runCapabilityProbe(*b, probe_prefix), DB::Exception); - /// The hook fires before the op battery: no probe keys may have been written. - EXPECT_TRUE(b->list(probe_prefix, "", 10).keys.empty()); - - /// The same LIST can see a key that IS under the prefix — otherwise the emptiness above would be - /// indistinguishable from a prefix this backend can never enumerate. Placed through the object - /// storage: a Native write over a local storage has no response incarnation to attribute itself - /// to, and Native passes the key verbatim, so this lands exactly where the LIST looks. - { - auto out = storage->writeObject(DB::StoredObject(probe_prefix + "/token"), DB::WriteMode::Rewrite); - DB::writeString(String("probe-v1"), *out); - out->finalize(); - } - EXPECT_FALSE(b->list(probe_prefix, "", 10).keys.empty()); -} - -/// Mirrors PoolPreconditionsFireThroughInstrumentedWrapper: the real mount path wraps the backend in -/// InstrumentedBackend BEFORE calling runCapabilityProbe, so this check must fire through it too. -TEST(CASProbe, MissingSingleAttemptClientFiresThroughInstrumentedWrapper) -{ - auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); - auto inner = std::make_shared(storage, ObjectStorageBackend::Mode::Native); - InstrumentedBackend wrapped(inner); - EXPECT_THROW(runCapabilityProbe(wrapped, DB::Cas::tests::nativeKeyUnder(storage, "p/.cas_probe")), DB::Exception); -} - namespace { -/// Honors every conditional WRITE but ignores the token on a token-exact DELETE. This is what a GCS -/// delete degenerates to when its numeric generation leaves as a raw `If-Match` — no -/// `x-goog-if-generation-match` — and the service ignores the header it does not recognise. +/// Honors every conditional WRITE but ignores the precondition on a conditional REMOVE. This is what a +/// GCS delete degenerates to when its numeric generation leaves as a raw `If-Match` — no +/// `x-goog-if-generation-match` — and the service ignores the header it does not recognise. Gated on +/// the PRIMITIVE (`Backend::remove`), which is what `CasOperation::remove` actually calls; a fault +/// injected on the legacy `deleteExact` forwarder would no longer intercept anything. class IgnoresDeleteTokenBackend : public InMemoryBackend { public: - DeleteOutcome deleteExact(const String & key, const Token &) override + RawRemoval remove(const String & key, const String & /*expected_value*/, TransportAccess & access) override { - return InMemoryBackend::deleteExact(key, head(key).token); + const auto meta = InMemoryBackend::head(key, access); + if (!meta) + return RawRemoval::Gone; + return InMemoryBackend::remove(key, meta->value, access); } }; /// The other half of that degeneracy: the service refuses the unrecognised header outright, so even -/// the correct token never removes anything. +/// the correct incarnation never removes anything. class RejectsDeleteTokenBackend : public InMemoryBackend { public: - DeleteOutcome deleteExact(const String &, const Token &) override + RawRemoval remove(const String &, const String &, TransportAccess &) override { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::TokenMismatch; - return d; + return RawRemoval::Mismatch; } }; @@ -217,175 +143,87 @@ class RejectsDeleteTokenBackend : public InMemoryBackend /// A GCS mount whose exact deletes lost their generation semantics can fail in either direction, and /// the probe's delete battery must reject the mount both times. Both backends enforce every -/// conditional write, so every step before the battery passes and only step 6's wrong-token -/// preservation check and step 8's correct-token deletion check can be what fires — -/// `PassesOnEnforcingBackend` above is the control showing the same probe succeeds when only -/// `deleteExact` is left alone. +/// conditional write, so every step before the battery's delete checks passes and only the +/// stale-incarnation-preserved check or the correct-incarnation-removed check can be what fires — +/// `PassesOnEnforcingBackend` above is the control showing the same probe succeeds when only `remove` +/// is left alone. /// /// This is about the battery, not about the marking: that the `NativeConditional` mode actually /// reaches the production request object is proven where the request is built, not here. TEST(CASProbe, ExactDeleteBatteryDetectsMissingGenerationMode) { IgnoresDeleteTokenBackend ignores; - EXPECT_THROW(runCapabilityProbe(ignores, "p/.cas_probe"), DB::Exception); + auto ignores_requests = makeRequests(ignores); + auto ignores_op = ignores_requests.admit(); + EXPECT_THROW(runCapabilityProbe(ignores_op, "p/.cas_probe"), DB::Exception); RejectsDeleteTokenBackend rejects; - EXPECT_THROW(runCapabilityProbe(rejects, "p/.cas_probe"), DB::Exception); + auto rejects_requests = makeRequests(rejects); + auto rejects_op = rejects_requests.admit(); + EXPECT_THROW(runCapabilityProbe(rejects_op, "p/.cas_probe"), DB::Exception); } namespace { -/// Models the exact shape of the trust-flip this suite must catch a regression of -/// (codex-review-triage §3.18, Critical): like the production `ObjectStorageBackend` in Native mode, -/// this backend mints and expects tokens under a dialect (`TokenType::ETag`) OTHER than -/// `TokenType::Emulated`, and rejects a foreign-dialect `expected`/`token` argument LOCALLY -- -/// before the value it carries ever reaches the real conditional-compare beneath the gate (`inner`, -/// a genuinely enforcing `InMemoryBackend`, standing in for "the wire"). Every gated method counts -/// how many times it actually delegated to `inner`, so a test can tell "rejected by the dialect -/// gate" apart from "rejected by the real enforcement" -- the exact distinction `Cas::Probe` exists -/// to prove, and the one the №19 hardening risked collapsing (see CasProbe.cpp step 3/5c/6). -class DialectGatedCountingBackend final : public Backend +/// Rejects, LOCALLY and without touching the store, any conditional write/remove whose precondition +/// value is not grammar-valid under the claimed dialect — the production shape of the retired +/// `DialectGatedCountingBackend`, ported to the primitive interface (`write`/`remove` carry a raw +/// precondition VALUE now, not a typed token, so the gate is `isIncarnationValue` rather than a type-tag +/// compare). `write_reached`/`remove_reached` count only the calls that got PAST the gate, so a +/// regression that reintroduces a synthesized (grammar-invalid-somewhere) precondition drops one of +/// these counts instead of passing silently. +class DialectOverrideBackend : public InMemoryBackend { public: - std::optional get(const String & key, Range range) override { return inner.get(key, range); } - - std::optional getStream(const String & key, Range range) override { return inner.getStream(key, range); } - - HeadResult head(const String & key) override - { - HeadResult r = inner.head(key); - if (r.exists) - r.token.type = TokenType::ETag; - return r; - } - - bool supportsListTokens() const override { return inner.supportsListTokens(); } - - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - /// No `expected` token to gate -- matches production (ObjectStorageBackend::putIfAbsent has - /// no dialect check either). - PutResult r = inner.putIfAbsent(key, bytes, meta); - if (r.outcome == PutOutcome::Done) - r.token.type = TokenType::ETag; - return r; - } - - void publishBlob(const BlobPublishRequest & request) override - { - inner.publishBlob(request); - } - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override - { - if (expected.type != TokenType::ETag) - return {PutOutcome::PreconditionFailed, {}}; /// dialect-gated: never reaches `inner` - ++overwrite_reached; - PutResult r = inner.putOverwrite(key, bytes, Token{expected.value, TokenType::Emulated}, meta); - if (r.outcome == PutOutcome::Done) - r.token.type = TokenType::ETag; - return r; - } + explicit DialectOverrideBackend(Dialect claimed_dialect_) : claimed_dialect(claimed_dialect_) {} + Dialect dialect() const override { return claimed_dialect; } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override { - if (expected.has_value() && expected->type != TokenType::ETag) - return {CasOutcome::Conflict, {}}; /// dialect-gated: never reaches `inner` - ++casput_reached; - std::optional retyped; - if (expected.has_value()) - retyped = Token{expected->value, TokenType::Emulated}; - CasResult r = inner.casPut(key, bytes, retyped, meta); - if (r.outcome == CasOutcome::Committed) - r.token.type = TokenType::ETag; - return r; + if (expected_value && !isIncarnationValue(claimed_dialect, *expected_value)) + return std::unexpected(RawConflict{}); /// dialect-gated: never reaches the real store + ++write_reached; + return InMemoryBackend::write(key, bytes, expected_value, access); } - DeleteOutcome deleteExact(const String & key, const Token & token) override + RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { - if (token.type != TokenType::ETag) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::TokenMismatch; /// dialect-gated: never reaches `inner` - return d; - } - ++delete_reached; - return inner.deleteExact(key, Token{token.value, TokenType::Emulated}); + if (!isIncarnationValue(claimed_dialect, expected_value)) + return RawRemoval::Mismatch; /// dialect-gated: never reaches the real store + ++remove_reached; + return InMemoryBackend::remove(key, expected_value, access); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override - { - ListPage p = inner.list(prefix, cursor, limit); - for (auto & k : p.keys) - if (k.token) - k.token->type = TokenType::ETag; - return p; - } - - /// The transport primitives forward verbatim. Nothing in this suite calls them -- the capability - /// probe speaks the legacy Token-typed surface this double gates -- so they exist to make the - /// class concrete; the dialect gate above is what the tests exercise. - std::optional read(const String & key, TransportAccess & a) override { return inner.read(key, a); } - std::optional head(const String & key, TransportAccess & a) override { return inner.head(key, a); } - RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & a) override - { - return inner.list(prefix, cursor, limit, a); - } - RawRemoval remove(const String & key, const String & expected_value, TransportAccess & a) override - { - return inner.remove(key, expected_value, a); - } - std::expected write(const String & key, const String & bytes, - const std::optional & expected_value, TransportAccess & a) override - { - return inner.write(key, bytes, expected_value, a); - } - std::unique_ptr stream(const String & key, TransportAccess & a) override { return inner.stream(key, a); } - void publish(const BlobPublishRequest & request, TransportAccess & a) override { inner.publish(request, a); } - /// The dialect its legacy surface stamps every token with. - Dialect dialect() const override { return TokenType::ETag; } - - /// Number of times putOverwrite/casPut(with expected)/deleteExact actually delegated to `inner` - /// (i.e. reached the real enforcement) rather than being short-circuited by the dialect gate. - int overwrite_reached = 0; - int casput_reached = 0; - int delete_reached = 0; + int write_reached = 0; + int remove_reached = 0; private: - InMemoryBackend inner; + Dialect claimed_dialect; }; } -/// codex-review-triage §3.18, Critical: `runCapabilityProbe`'s three wrong-token sites (step 3 -/// putOverwrite, step 5c casPut, step 6 deleteExact) must send a token in the LIVE dialect this -/// backend mints (t1.type / ct1.type / t2.type), not a hardcoded `TokenType::Emulated`. A backend -/// whose native dialect differs from Emulated -- exactly what `ObjectStorageBackend` mints in Native -/// mode -- would otherwise reject the old hardcoded tokens LOCALLY via a dialect gate, never -/// exercising the real conditional enforcement those three steps exist to validate; the probe would -/// still report success (the outcome enums match either way), so a regression here is invisible -/// unless something counts whether the real enforcement was ever reached. `DialectGatedCountingBackend` -/// enforces real (correct) conditional semantics AND gates on dialect exactly like the production -/// risk, so `runCapabilityProbe` runs to completion (unlike a real Native-mode ObjectStorageBackend -/// over LocalObjectStorage, which cannot even reach this point -- see -/// MissingSingleAttemptClientFailsCapabilityProbe and the fact that LocalObjectStorage does not honor -/// WriteSettings conditions at all); the exact reached-counts below pin down that every wrong-token -/// site got past the gate: a probe that regressed to the hardcoded-Emulated construction would still -/// pass (no throw) but under-count here by exactly one at each of the three sites, since the dialect -/// gate would swallow that one call before `inner` ever saw it. -TEST(CASProbe, WrongTokenAttemptsReachTheBackendPastTheDialectGate) +/// The probe's reordered "wrong incarnation" steps always reuse a REAL, backend-minted `Incarnation` +/// from the same key rather than a synthesized value — see CasProbe.cpp's step comments — and every +/// such value is grammar-valid under every dialect by construction (`InMemoryBackend`'s minted values are +/// a monotonically increasing decimal starting at "1": non-empty and comma/`*`-free for ETag, a canonical +/// positive decimal for Generation, merely non-empty for Emulated). So under EVERY dialect the battery's +/// four conditional writes (steps 1-4) and two conditional removes (steps 5, 7) must all reach the real +/// store — asserting the exact counts is what makes this test able to fail: a regression that +/// reintroduces a synthesized, foreign-dialect precondition would get gated locally on at least one +/// dialect, dropping one of these counts below the total instead of merely changing an outcome enum. +TEST(CASProbe, ReorderedProbePassesOnAllThreeDialects) { - DialectGatedCountingBackend b; - EXPECT_NO_THROW(runCapabilityProbe(b, "p/.cas_probe")); - - /// putOverwrite: step 3 (wrong token) + step 4 (correct token) -- both live-dialect, both gated - /// through to `inner`. - EXPECT_EQ(b.overwrite_reached, 2); - /// casPut: 5a (create), 5b (conflict-on-exists, no expected token to gate), 5c (wrong token, - /// live-dialect), 5d (correct token) -- all four reach `inner`. - EXPECT_EQ(b.casput_reached, 4); - /// deleteExact: step 6 (wrong token, live-dialect) + step 8 (correct token) + step 9 cleanup - /// (correct token for cas_key) -- all three reach `inner`. - EXPECT_EQ(b.delete_reached, 3); + for (const Dialect dialect : {Dialect::ETag, Dialect::Generation, Dialect::Emulated}) + { + DialectOverrideBackend b(dialect); + auto requests = makeRequests(b); + auto op = requests.admit(); + EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")) << "dialect " << static_cast(dialect); + EXPECT_TRUE(b.list("p/.cas_probe", "", 10).keys.empty()) << "dialect " << static_cast(dialect); + EXPECT_EQ(b.write_reached, 4) << "dialect " << static_cast(dialect); + EXPECT_EQ(b.remove_reached, 2) << "dialect " << static_cast(dialect); + } } diff --git a/src/Disks/tests/gtest_cas_protocol_scenarios.cpp b/src/Disks/tests/gtest_cas_protocol_scenarios.cpp index 924732bfc65d..422cc934c67d 100644 --- a/src/Disks/tests/gtest_cas_protocol_scenarios.cpp +++ b/src/Disks/tests/gtest_cas_protocol_scenarios.cpp @@ -35,6 +35,7 @@ namespace DB::ErrorCodes { extern const int ABORTED; +extern const int CORRUPTED_DATA; extern const int FILE_DOESNT_EXIST; extern const int LOGICAL_ERROR; } @@ -58,6 +59,25 @@ PoolPtr openPool(const std::shared_ptr & b) return Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); } +/// The object's incarnation as the store reports it now -- what these scenarios compare when they +/// assert an object was, or was not, displaced. +Incarnation currentIncarnation(Backend & b, const String & key) +{ + DB::Cas::tests::OperationForTest operation(b); + const std::optional meta = (*operation).head(key, Retry::standard()); + if (!meta) + throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "object {} is absent", key); + return meta->incarnation; +} + +/// An exact-incarnation delete attempt, for the scenarios whose discriminator is that a displaced +/// incarnation can never be current again. +Removal removeAtIncarnation(Backend & b, const String & key, const Incarnation & seen) +{ + DB::Cas::tests::OperationForTest operation(b); + return (*operation).remove(key, seen, Retry::standard()); +} + /// A single-blob manifest entry naming `payload` at `path` (the entry the part's manifest carries). ManifestEntry blobEntry(const String & path, const String & payload) { @@ -145,11 +165,11 @@ TEST(CASProtocol, FenceConflictCondemnedTokenedBlobCommitsWithTokenUnchanged) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); /// GC condemns X at t0 in round 1 and fences the namespace to round 1. injectRetire(*b, s->layout(), /*round*/ 1, /*shard*/ 0, - {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = t0, .size = 9}}); + {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedIncarnation::capture(t0), .size = 9}}); /// promote: mutateShard refreshes the view (fence_round 1 > view round 0), but the materialized leaf is /// edge-protected — skipped, not re-validated ⇒ commit, token unchanged. @@ -157,7 +177,7 @@ TEST(CASProtocol, FenceConflictCondemnedTokenedBlobCommitsWithTokenUnchanged) /// The ref is committed and reads back; the blob still rides t0 (no re-upload). assertPartReads(b, s, ns, "part_1", "data.bin", "payload-X"); - EXPECT_EQ(b->head(blob_key).token, t0); + EXPECT_EQ(currentIncarnation(*b, blob_key), t0); } TEST(CASProtocol, RevalidateReObservesStaleTokenKeepsWhenUnchanged) @@ -172,7 +192,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenKeepsWhenUnchanged) /// X pre-exists out-of-band; the build dedup-adopts it via putBlob (records the current token t0). writeBlobRaw(*b, s->layout(), "payload-X", s->poolMeta().blob_header_len, s->poolMeta().pool_id); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); /// Wiring order: stage + precommit (durable edge) BEFORE the adopting putBlob. auto build = startBuildFor(s, ns, "part_1"); @@ -189,7 +209,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenKeepsWhenUnchanged) assertPartReads(b, s, ns, "part_1", "data.bin", "payload-X"); /// No rewrite happened — the materialized leaf was never touched, so its object token stays at t0. - EXPECT_EQ(b->head(blob_key).token, t0); + EXPECT_EQ(currentIncarnation(*b, blob_key), t0); } TEST(CASProtocol, RevalidateReObservesStaleTokenAdoptsWhenDisplaced) @@ -204,7 +224,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenAdoptsWhenDisplaced) writeBlobRaw(*b, s->layout(), "payload-X", s->poolMeta().blob_header_len, s->poolMeta().pool_id); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); auto build = startBuildFor(s, ns, "part_1"); /// Wiring order (EDGE-BEFORE-OBSERVE): stageManifest -> precommitAdd -> putBlob. @@ -213,7 +233,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenAdoptsWhenDisplaced) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); /// dedup → adopts t0 /// Another writer displaces X out-of-band ⇒ a new current token t1 (same payload, fresh tag). - const Token t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); + const Incarnation t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); EXPECT_NE(t1, t0); /// GC advanced to round 1 with an EMPTY retired set; fence to 1. @@ -222,17 +242,17 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenAdoptsWhenDisplaced) /// promote refreshes ⇒ revalidate X ⇒ HEAD current t1 not condemned ⇒ commit. The dep rides t1. build->promote(ns, "part_1", build->buildId(), id); assertPartReads(b, s, ns, "part_1", "data.bin", "payload-X"); - EXPECT_EQ(b->head(blob_key).token, t1); + EXPECT_EQ(currentIncarnation(*b, blob_key), t1); /// Black-box proof the part reads the t1 incarnation: re-publish the same blob into a SECOND /// namespace with NO new GC injection. The blob is already present at t1; nothing is re-uploaded. publishBlobPart(s, RootNamespace{"srv1/tbl/copy"}, "part_2", "data.bin", "payload-X"); - EXPECT_EQ(b->head(blob_key).token, t1); + EXPECT_EQ(currentIncarnation(*b, blob_key), t1); assertPartReads(b, s, RootNamespace{"srv1/tbl/copy"}, "part_2", "data.bin", "payload-X"); /// Independent discriminator that the blob rides t1, not the stale t0: t0 is DEAD. A deleteExact /// against t0 must TokenMismatch (INV-NO-RETURN — t0 was displaced and can never be current again). - EXPECT_EQ(b->deleteExact(blob_key, t0).kind, DeleteOutcome::Kind::TokenMismatch); + EXPECT_EQ(removeAtIncarnation(*b, blob_key, t0), Removal::Mismatch); } TEST(CASProtocol, RevalidateAdoptsLiveTokenWhenOnlyPhantomCondemnedAtDifferentToken) @@ -250,9 +270,9 @@ TEST(CASProtocol, RevalidateAdoptsLiveTokenWhenOnlyPhantomCondemnedAtDifferentTo writeBlobRaw(*b, s0->layout(), "payload-X", s0->poolMeta().blob_header_len, s0->poolMeta().pool_id); } const String blob_key = layout.blobKey(idOf("payload-X")); - const Token t0 = b->head(blob_key).token; - const Token t_other{"emulated-phantom", DB::Cas::TokenType::Emulated}; - ASSERT_NE(t_other, t0); + const Incarnation t0 = currentIncarnation(*b, blob_key); + const PersistedIncarnation t_other{"emulated", "emulated-phantom"}; + ASSERT_FALSE(t_other.matches(t0)) << "the phantom must name a DIFFERENT incarnation than the live one"; injectRetire(*b, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = t_other, .size = 9}}); @@ -272,7 +292,7 @@ TEST(CASProtocol, RevalidateAdoptsLiveTokenWhenOnlyPhantomCondemnedAtDifferentTo assertPartReads(b, s, ns, "part_1", "data.bin", "payload-X"); /// The object was NOT displaced — it STAYS at t0 (no re-upload, only re-validated). - EXPECT_EQ(b->head(blob_key).token, t0); + EXPECT_EQ(currentIncarnation(*b, blob_key), t0); } /// (DELETED, Phase A) RevalidateAbsentTokenedBlobResurrectsFromSource — see the file-header note: a @@ -298,7 +318,7 @@ TEST(CASProtocol, EvidenceHitCondemnedPresentBlobCopiesForwardInClosure) const BlobRef seeded_ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128(hex))}; seedBlobWithDurablePrecommit(s, seeded_ref, "payload-X"); const String blob_key = s->layout().blobKey(seeded_ref); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); auto build = startBuildFor(s, ns, "part_1"); ManifestEntry entry = blobEntry("data.bin", "payload-X"); @@ -315,7 +335,7 @@ TEST(CASProtocol, EvidenceHitCondemnedPresentBlobCopiesForwardInClosure) /// The ref stands; X rides its ORIGINAL token t0 (trust never displaces a trusted leaf). EXPECT_TRUE(s->resolveRef(ns, "part_1").has_value()); - EXPECT_EQ(b->head(blob_key).token, t0) << "trust must not displace the adopted blob"; + EXPECT_EQ(currentIncarnation(*b, blob_key), t0) << "trust must not displace the adopted blob"; /// The meta is untouched — still Condemned (the gate never reads or flips it under trust). const auto lm_after = loadMetaForTest(*b, s->layout(), hexToU128(hex)); @@ -343,16 +363,16 @@ TEST(CASProtocol, WedgedHeartbeatCondemnedTokenedBlobCommitsWithTokenUnchanged) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); /// Full GC condemned the build's OWN upload. injectRetire(*b, s->layout(), /*round*/ 1, /*shard*/ 0, - {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = t0, .size = 9}}); + {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedIncarnation::capture(t0), .size = 9}}); /// promote: the materialized leaf is edge-protected — skipped, not revalidated ⇒ commit, token unchanged. build->promote(ns, "part_1", build->buildId(), id); assertPartReads(b, s, ns, "part_1", "data.bin", "payload-X"); - EXPECT_EQ(b->head(blob_key).token, t0); + EXPECT_EQ(currentIncarnation(*b, blob_key), t0); } TEST(CASProtocol, AbandonLeavesDebrisAndDisables) @@ -398,7 +418,7 @@ TEST(CASProtocol, DropReattachThroughDetachedNamespace) publishBlobPart(s, ns, "part_1", "data.bin", "payload-X"); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Token blob_tok = b->head(blob_key).token; + const Incarnation blob_tok = currentIncarnation(*b, blob_key); EXPECT_TRUE(s->listRefs(ns).contains("part_1")); EXPECT_TRUE(s->listRefs(detached).empty()); @@ -420,7 +440,7 @@ TEST(CASProtocol, DropReattachThroughDetachedNamespace) EXPECT_TRUE(s->listRefs(detached).empty()); /// The blob was never re-uploaded (token stable throughout — every publish dedup-adopted it). - EXPECT_EQ(b->head(blob_key).token, blob_tok); + EXPECT_EQ(currentIncarnation(*b, blob_key), blob_tok); } TEST(CASProtocol, FreezeIntoShadowNamespace) @@ -457,7 +477,7 @@ TEST(CASProtocol, DisplacedToLiveTokenCommitsAtCurrentIncarnation) writeBlobRaw(*b, s->layout(), "payload-X", s->poolMeta().blob_header_len, s->poolMeta().pool_id); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); auto build = startBuildFor(s, ns, "part_1"); /// Wiring order (EDGE-BEFORE-OBSERVE): stageManifest -> precommitAdd -> putBlob. @@ -466,23 +486,23 @@ TEST(CASProtocol, DisplacedToLiveTokenCommitsAtCurrentIncarnation) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); /// dedup → adopts t0 /// Another writer displaces X to t1 (uncondemned) before our gate runs. - const Token t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); + const Incarnation t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); ASSERT_NE(t1, t0); /// The view still condemns the OLD t0 at round 1, fenced. injectRetire(*b, s->layout(), /*round*/ 1, /*shard*/ 0, - {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = t0, .size = 9}}); + {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedIncarnation::capture(t0), .size = 9}}); /// promote: revalidate X ⇒ HEAD current t1 (NOT condemned; only the defunct t0 is) ⇒ commit. build->promote(ns, "part_1", build->buildId(), id); /// The blob lives at t1 (the displacing writer's incarnation) and the part reads. - EXPECT_EQ(b->head(blob_key).token, t1); + EXPECT_EQ(currentIncarnation(*b, blob_key), t1); assertPartReads(b, s, ns, "part_1", "data.bin", "payload-X"); /// NO-LOSS / NO-RETURN: t0 is dead — a deleteExact against it TokenMismatches (the GC delete of the /// condemned t0 spares the live t1). - EXPECT_EQ(b->deleteExact(blob_key, t0).kind, DeleteOutcome::Kind::TokenMismatch); + EXPECT_EQ(removeAtIncarnation(*b, blob_key, t0), Removal::Mismatch); } TEST(CASProtocol, NewNamespacePublishGatedByShardFenceFloor) @@ -562,7 +582,7 @@ TEST(CASProtocol, FreshEvidenceDepWithViewHitIsResolvedByGate) "payload-fresh-ev"); } const String blob_key = layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128(hex))}); - const Token t0 = b->head(blob_key).token; + const Incarnation t0 = currentIncarnation(*b, blob_key); condemnMeta(*b, layout, hexToU128(hex), /*condemn_round*/ 1); auto s = openPool(b); @@ -579,7 +599,7 @@ TEST(CASProtocol, FreshEvidenceDepWithViewHitIsResolvedByGate) /// promote trusts the adopted leaf ⇒ commit, no probe, no displacement. EXPECT_NO_THROW(build->promote(ns, "part_1", build->buildId(), id)); - EXPECT_EQ(b->head(blob_key).token, t0) << "trust must not displace the adopted blob"; + EXPECT_EQ(currentIncarnation(*b, blob_key), t0) << "trust must not displace the adopted blob"; EXPECT_TRUE(s->resolveRef(ns, "part_1").has_value()); } diff --git a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp index 07a9591e6fc6..5973f35d0c48 100644 --- a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp +++ b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp @@ -55,8 +55,6 @@ const RootNamespace kNsB{"00/zz@cas@"}; class CatalogChangesOnSecondReadBackend : public CountingBackend { public: - using Backend::get; - void armCatalogMutation(const String & key) { catalog_key = key; @@ -66,9 +64,9 @@ class CatalogChangesOnSecondReadBackend : public CountingBackend size_t catalogReads() const { return catalog_reads; } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - auto got = CountingBackend::get(key, range); + auto got = CountingBackend::read(key, access); if (!armed || key != catalog_key) return got; @@ -78,11 +76,9 @@ class CatalogChangesOnSecondReadBackend : public CountingBackend if (!got) throw std::runtime_error("catalog mutation fixture: second catalog read found absence"); - const PutResult put = CountingBackend::putOverwrite( - key, encodeRefCatalog(RefCatalog{}), got->token, {}); - if (put.outcome != PutOutcome::Done) + if (!CountingBackend::write(key, encodeRefCatalog(RefCatalog{}), got->value, access)) throw std::runtime_error("catalog mutation fixture: catalog rewrite conflicted"); - return CountingBackend::get(key, range); + return CountingBackend::read(key, access); } private: @@ -111,9 +107,10 @@ bool condemnedInSealedRuns(Backend & backend, const Layout & layout, const DB::U if (!sealed) return false; const CasFoldSeal seal = decodeFoldSeal(sealed->bytes); + DB::Cas::tests::OperationForTest operation(backend); for (const RunRef & r : seal.blob_target_runs) { - auto reader = openSourceEdgeRun(backend, r.key); + auto reader = openSourceEdgeRun(*operation, r.key); String k; String p; while (reader.next(k, p)) diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 355ec21a2126..25c4a36a465d 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -37,7 +37,7 @@ SourceEdgeRecord zero(const BlobRef & ref) return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Zero}; } -SourceEdgeRecord condemned(const BlobRef & ref, const Token & token, uint64_t size, uint64_t round, bool pend) +SourceEdgeRecord condemned(const BlobRef & ref, const PersistedIncarnation & token, uint64_t size, uint64_t round, bool pend) { return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = pend, .token = token, .size = size, .condemn_round = round}; @@ -105,7 +105,7 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) /// an edge; c has a zero marker. Blobs ascend a < b < c, so the sequence is already non-decreasing. std::vector recs = { edge(a, 10), - condemned(b, Token{"e-1", TokenType::ETag}, 4242, 7, /*pend*/ true), + condemned(b, PersistedIncarnation{"etag", "e-1"}, 4242, 7, /*pend*/ true), zero(c), }; const String bytes = encodeRun(recs); @@ -120,7 +120,8 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) EXPECT_EQ(back[1].source_id, UInt128(0)); EXPECT_EQ(back[1].marker, RunMarker::Condemned); EXPECT_TRUE(back[1].delete_pending); - EXPECT_EQ(back[1].token, (Token{"e-1", TokenType::ETag})); + EXPECT_EQ(back[1].token.dialect, "etag"); + EXPECT_EQ(back[1].token.value, "e-1"); EXPECT_EQ(back[1].size, 4242u); EXPECT_EQ(back[1].condemn_round, 7u); @@ -145,7 +146,7 @@ TEST(CASRecordStream, ClosedSetPinsRunMarkerWords) /// which is a different retention decision than the writer recorded. TEST(CASRecordStream, CondemnedRowMissingOneOfItsSixFieldsFailsClosed) { - const String good = encodeRun({condemned(chRef(2), Token{"e-1", TokenType::ETag}, 4242, 7, /*pend*/ true)}); + const String good = encodeRun({condemned(chRef(2), PersistedIncarnation{"etag", "e-1"}, 4242, 7, /*pend*/ true)}); for (const std::string_view field : {R"(,"pending":true)", R"(,"token_type":"etag")", R"(,"token":"e-1")", R"(,"size":4242)", R"(,"condemn_round":"7")", R"(,"confirmed":false)"}) { @@ -196,7 +197,7 @@ TEST(CASRecordStream, WriterIsByteDeterministic) std::vector recs = { edge(chRef(1), 5), edge(chRef(1), 9), - condemned(chRef(2), Token{"t/with/slashes", TokenType::ETag}, 1, 2, false), + condemned(chRef(2), PersistedIncarnation{"etag", "t/with/slashes"}, 1, 2, false), }; EXPECT_EQ(encodeRun(recs), encodeRun(recs)); /// pure function of the sorted record set } diff --git a/src/Disks/tests/gtest_cas_recovery_grounding.cpp b/src/Disks/tests/gtest_cas_recovery_grounding.cpp index d908038a153f..ddf267df7e22 100644 --- a/src/Disks/tests/gtest_cas_recovery_grounding.cpp +++ b/src/Disks/tests/gtest_cas_recovery_grounding.cpp @@ -47,18 +47,23 @@ class RecoveryListingBackend : public CountingBackend size_t list_calls = 0; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + /// On the transport primitive, not the legacy verb: every enumeration a `CasOperation` makes + /// reaches the store through this, so a distortion left on the verb would never fire and the + /// `list_calls` assertions would read zero whatever recovery did. + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { ++list_calls; - ListPage page = CountingBackend::list(prefix, cursor, limit); + DB::Cas::Backend::RawListPage page = CountingBackend::list(prefix, cursor, limit, access); if (mode == ListingMode::Empty) page.keys.clear(); else if (mode == ListingMode::Partial) { - page.keys.erase(std::remove_if(page.keys.begin(), page.keys.end(), [](const ListedKey & key) - { - return key.key.find("/_log/") != String::npos; - }), page.keys.end()); + page.keys.erase(std::remove_if(page.keys.begin(), page.keys.end(), + [](const DB::Cas::Backend::RawListedKey & key) + { + return key.key.find("/_log/") != String::npos; + }), page.keys.end()); } else if (mode == ListingMode::Reordered) std::reverse(page.keys.begin(), page.keys.end()); @@ -112,14 +117,16 @@ void seedAuthoritativeStream(Backend & backend, const Layout & layout, const Roo applyRefLogTxn(snapshot_state, first_txn); writeRefSnapshotRaw(backend, layout, snapshotOf(snapshot_state, ns.string())); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(backend, layout, ns); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const RefCkpt authority{ .life_epoch = 1, .committed_through = committed_through, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = committed_through.writer_epoch > 1 ? std::optional{RefTxnId{1, 2}} : std::nullopt}; - backend.putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(authority)); + (void)op.create(layout.refCkptKey(life), encodeRefCkpt(authority), Retry::standard()); } /// This is deliberately caller-side plumbing, not a convenience overload in `CasRefProtocol`: production @@ -127,7 +134,9 @@ void seedAuthoritativeStream(Backend & backend, const Layout & layout, const Roo /// The API under test receives those exact values and performs no catalog or checkpoint resolution itself. RecoveredRefTable recoverFromCurrentCatalogCut(Backend & backend, const Layout & layout, const RootNamespace & ns) { - const CasRefCatalog::Snapshot cut = CasRefCatalog::read(backend, layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot cut = CasRefCatalog::read(op, layout); std::optional entry; for (const CatalogEntry & candidate : cut.catalog.entries) { @@ -141,10 +150,10 @@ RecoveredRefTable recoverFromCurrentCatalogCut(Backend & backend, const Layout & if (entry) { const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation); - if (const std::optional sample = readCkpt(backend, layout, life)) + if (const std::optional sample = readCkpt(op, layout, life)) checkpoint = sample->ckpt; } - return recoverRefTableDetailedFromAuthority(backend, layout, entry, checkpoint); + return recoverRefTableDetailedFromAuthority(op, layout, entry, checkpoint); } CatalogEntry catalog(NsState state) @@ -295,11 +304,13 @@ TEST(CASRecoveryGrounding, RecoveryIsEquivalentUnderFullEmptyPartialAndReordered for (const ListingMode mode : {ListingMode::Full, ListingMode::Empty, ListingMode::Partial, ListingMode::Reordered}) { auto backend = std::make_shared(mode); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/list_equivalence"}; const RefTxnId frontier{2, 1}; seedAuthoritativeStream(*backend, layout, ns, frontier); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); backend->resetCounts(); backend->list_calls = 0; @@ -344,19 +355,25 @@ TEST(CASRecoveryGrounding, CatalogLifecycleAndCheckpointAreMandatoryForReadOnlyR { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, txn(ns, {1, 1}, {namespaceBirthOp()})); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); } { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); CasRefCatalog::casAdmitEntry( - *backend, layout, 1, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = 8}); + catalog_op, layout, 1, CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = 8}); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); } { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const CatalogEntry live{.ns = ns, .state = NsState::Live, .incarnation = 9}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, live); + CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, live); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(live.ns, live.incarnation); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), "not a sealed checkpoint").outcome, PutOutcome::Done); @@ -364,9 +381,11 @@ TEST(CASRecoveryGrounding, CatalogLifecycleAndCheckpointAreMandatoryForReadOnlyR } { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); CatalogEntry creating{.ns = ns, .state = NsState::Creating, .incarnation = 7, .creator = CreatorFence{"srv1", 1, 1}}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, creating); + CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, creating); backend->putIfAbsent(layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation)), encodeRefCkpt(RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})); @@ -374,6 +393,8 @@ TEST(CASRecoveryGrounding, CatalogLifecycleAndCheckpointAreMandatoryForReadOnlyR } { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); backend->putIfAbsent(layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)), encodeRefCkpt(RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})); @@ -393,30 +414,36 @@ TEST(CASRecoveryGrounding, NonrecoverableAuthorityPerformsNoBackendRecoveryIo) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); backend->resetCounts(); const CatalogEntry creating{ .ns = ns, .state = NsState::Creating, .incarnation = 1, .creator = CreatorFence{"srv1", 1, 1}}; expectCode( - [&] { (void)recoverRefTableDetailedFromAuthority(*backend, layout, creating, valid_ckpt); }, + [&] { (void)recoverRefTableDetailedFromAuthority(catalog_op, layout, creating, valid_ckpt); }, DB::ErrorCodes::INVALID_STATE); EXPECT_EQ(backend->list_calls, 0u); EXPECT_EQ(backend->getTotal(), 0u); } { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); backend->resetCounts(); const CatalogEntry live{.ns = ns, .state = NsState::Live, .incarnation = 2}; expectCode( - [&] { (void)recoverRefTableDetailedFromAuthority(*backend, layout, live, std::nullopt); }, + [&] { (void)recoverRefTableDetailedFromAuthority(catalog_op, layout, live, std::nullopt); }, DB::ErrorCodes::CORRUPTED_DATA); EXPECT_EQ(backend->list_calls, 0u); EXPECT_EQ(backend->getTotal(), 0u); } { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); backend->resetCounts(); expectCode( - [&] { (void)recoverRefTableDetailedFromAuthority(*backend, layout, std::nullopt, valid_ckpt); }, + [&] { (void)recoverRefTableDetailedFromAuthority(catalog_op, layout, std::nullopt, valid_ckpt); }, DB::ErrorCodes::INVALID_STATE); EXPECT_EQ(backend->list_calls, 0u); EXPECT_EQ(backend->getTotal(), 0u); @@ -426,6 +453,8 @@ TEST(CASRecoveryGrounding, NonrecoverableAuthorityPerformsNoBackendRecoveryIo) TEST(CASRecoveryGrounding, ReadOnlyRecoveryNeverAdoptsFPlusOne) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/read_only_excludes_f_plus_one"}; seedAuthoritativeStream(*backend, layout, ns, RefTxnId{1, 1}, /*include_f_plus_one=*/true); @@ -444,10 +473,12 @@ TEST(CASRecoveryGrounding, ForgedWellFormedListedSnapshotIsUnobservedAndRecovery for (const ListingMode mode : {ListingMode::Full, ListingMode::Empty, ListingMode::Partial, ListingMode::Reordered}) { auto backend = std::make_shared(mode); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/forged_listed_snapshot"}; seedAuthoritativeStream(*backend, layout, ns, RefTxnId{1, 1}, /*include_f_plus_one=*/true); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); RefTableState forged_state; std::vector birth{namespaceBirthOp()}; @@ -475,6 +506,8 @@ TEST(CASRecoveryGrounding, ForgedWellFormedListedSnapshotIsUnobservedAndRecovery TEST(CASRecoveryGrounding, SemanticallyMalformedCheckpointSnapshotIsCorruptionAfterExactRead) { auto backend = std::make_shared(ListingMode::Empty); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/semantically_malformed_checkpoint"}; const ManifestRef manifest{1, 1, 1}; @@ -488,7 +521,7 @@ TEST(CASRecoveryGrounding, SemanticallyMalformedCheckpointSnapshotIsCorruptionAf malformed.precommits.push_back(RefOwnerBinding{RefOwnerKind::Precommit, "precommit", manifest}); writeRefSnapshotRaw(*backend, layout, malformed); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); const String snapshot_key = layout.refSnapshotKey(life, {1, 1}); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, @@ -514,11 +547,13 @@ TEST(CASRecoveryGrounding, SemanticallyMalformedCheckpointSnapshotIsCorruptionAf TEST(CASRecoveryGrounding, CheckpointSnapshotEqualToLastEpochSealIsRejectedBeforeReadingItsLog) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/checkpoint_base_seal"}; /// The checkpoint directly contradicts itself: its sole snapshot base names its terminal seal. seedAuthoritativeStream(*backend, layout, ns, RefTxnId{1, 2}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); RefTableState through_seal; std::vector birth{namespaceBirthOp()}; @@ -530,14 +565,14 @@ TEST(CASRecoveryGrounding, CheckpointSnapshotEqualToLastEpochSealIsRejectedBefor applyRefLogTxn(through_seal, txn(ns, {1, 2}, {std::move(seal)})); writeRefSnapshotRaw(*backend, layout, snapshotOf(through_seal, ns.string())); - const CkptSample before = *readCkpt(*backend, layout, life); + const CkptSample before = *readCkpt(catalog_op, layout, life); const RefCkpt with_sealed_base{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = RefTxnId{1, 2}, .last_epoch_seal = RefTxnId{1, 2}}; - ASSERT_EQ(backend->casPut(layout.refCkptKey(life), encodeRefCkpt(with_sealed_base), before.token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(catalog_op.replace( + layout.refCkptKey(life), encodeRefCkpt(with_sealed_base), before.incarnation, Retry::standard()))); backend->resetCounts(); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -554,6 +589,8 @@ TEST(CASRecoveryGrounding, CheckpointSnapshotEqualToLastEpochSealIsRejectedBefor TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/frontier_after_seal"}; @@ -562,7 +599,7 @@ TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) seal.kind = RefOpKind::EpochSeal; DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, txn(ns, {1, 2}, {std::move(seal)})); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); String malformed_ckpt = encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, @@ -579,10 +616,12 @@ TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) TEST(CASRecoveryGrounding, OlderCheckpointSnapshotAtSealIsCorruption) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/older_checkpoint_base_seal"}; seedAuthoritativeStream(*backend, layout, ns, RefTxnId{2, 1}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); RefOp second_seal; second_seal.kind = RefOpKind::EpochSeal; @@ -600,14 +639,14 @@ TEST(CASRecoveryGrounding, OlderCheckpointSnapshotAtSealIsCorruption) applyRefLogTxn(through_first_seal, txn(ns, {1, 2}, {std::move(first_seal)})); writeRefSnapshotRaw(*backend, layout, snapshotOf(through_first_seal, ns.string())); - const CkptSample before = *readCkpt(*backend, layout, life); + const CkptSample before = *readCkpt(catalog_op, layout, life); const RefCkpt with_old_sealed_base{ .life_epoch = 1, .committed_through = RefTxnId{3, 1}, .checkpoint_snapshot_id = RefTxnId{1, 2}, .last_epoch_seal = RefTxnId{2, 2}}; - ASSERT_EQ(backend->casPut(layout.refCkptKey(life), encodeRefCkpt(with_old_sealed_base), before.token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(catalog_op.replace( + layout.refCkptKey(life), encodeRefCkpt(with_old_sealed_base), before.incarnation, Retry::standard()))); backend->resetCounts(); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -620,6 +659,8 @@ TEST(CASRecoveryGrounding, OlderCheckpointSnapshotAtSealIsCorruption) TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RootNamespace ns{"srv1/terminal_gap"}; const RefLogTxn birth = txn(ns, {1, 1}, {namespaceBirthOp()}); @@ -629,7 +670,7 @@ TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, txn(ns, {1, 2}, {std::move(remove)})); DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, txn(ns, {2, 1}, {namespaceBirthOp()})); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); String malformed_ckpt = encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, @@ -646,6 +687,8 @@ TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) TEST(CASRecoveryGrounding, LaterEpochCheckpointBaseRequiresItsContextualBacklink) { auto backend = std::make_shared(ListingMode::Full); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); const Layout layout("p"); const RefTxnId seal_id{1, 2}; const RefTxnId base_id{2, 1}; @@ -659,7 +702,7 @@ TEST(CASRecoveryGrounding, LaterEpochCheckpointBaseRequiresItsContextualBacklink DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, txn(ns, base_id, {}, backlink)); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base_id)); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = base_id, @@ -682,7 +725,7 @@ TEST(CASRecoveryGrounding, LaterEpochCheckpointBaseRequiresItsContextualBacklink DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, txn(ns, base_id, {}, seal_id)); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base_id)); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = base_id, diff --git a/src/Disks/tests/gtest_cas_recovery_streaming.cpp b/src/Disks/tests/gtest_cas_recovery_streaming.cpp index ff83de9240e7..085baaf7e231 100644 --- a/src/Disks/tests/gtest_cas_recovery_streaming.cpp +++ b/src/Disks/tests/gtest_cas_recovery_streaming.cpp @@ -143,9 +143,8 @@ bool pollUntil(Pred pred) class VanishMidTailOnceBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the primitive overload that the override below would otherwise hide. using InMemoryBackend::list; - using InMemoryBackend::get; /// keep the one-arg convenience overload visible past our override String target_log_key; String refs_prefix; @@ -153,18 +152,18 @@ class VanishMidTailOnceBackend : public InMemoryBackend std::atomic vanished{false}; std::atomic fresh_list_count{0}; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (armed.load() && key == target_log_key && !vanished.exchange(true)) return std::nullopt; /// selected object gone between LIST and GET; recovery must re-LIST - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (armed.load() && prefix == refs_prefix && cursor.empty()) fresh_list_count.fetch_add(1, std::memory_order_relaxed); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } }; @@ -174,9 +173,8 @@ class VanishMidTailOnceBackend : public InMemoryBackend class CorruptLogOnGetBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the primitive overload that the override below would otherwise hide. using InMemoryBackend::list; - using InMemoryBackend::get; /// keep the one-arg convenience overload visible past our override String target_log_key; String corrupt_bytes; @@ -184,19 +182,19 @@ class CorruptLogOnGetBackend : public InMemoryBackend std::atomic armed{false}; std::atomic refs_list_count{0}; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { - auto got = InMemoryBackend::get(key, range); + auto got = InMemoryBackend::read(key, access); if (armed.load() && got && key == target_log_key) got->bytes = corrupt_bytes; return got; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (armed.load() && prefix == refs_prefix && cursor.empty()) refs_list_count.fetch_add(1, std::memory_order_relaxed); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } }; @@ -206,9 +204,8 @@ class CorruptLogOnGetBackend : public InMemoryBackend class BlockingFirstLogGetBackend : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the primitive overload that the override below would otherwise hide. using InMemoryBackend::list; - using InMemoryBackend::get; String refs_prefix; String target_log_key; @@ -217,18 +214,18 @@ class BlockingFirstLogGetBackend : public InMemoryBackend std::atomic list_calls{0}; std::function on_first_target_get; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (armed.load() && key == target_log_key && !blocked.exchange(true)) on_first_target_get(); - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (armed.load() && prefix == refs_prefix && cursor.empty()) list_calls.fetch_add(1, std::memory_order_relaxed); - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } }; @@ -261,7 +258,9 @@ TEST(CASRecoveryStreaming, LongTailReplaysUnderMemoryBound) setRecoveryReplayMemoryProbeForTest(tracker.probe()); SCOPE_EXIT({ setRecoveryReplayMemoryProbeForTest({}); }); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*backend, layout); + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(catalog_op, layout); const RefTableState state = recoverRefTableDetailedAtCatalogCutForTest(*backend, layout, catalog_cut, ns).state; EXPECT_EQ(state.getPrecommits().size(), kTxns * kOpsPerTxn) << "the whole tail must have replayed"; EXPECT_LE(tracker.peak(), static_cast(bound)) @@ -477,7 +476,9 @@ TEST(CASRecoveryStreaming, OrphanSweepAndFsckSameBound) PeakTracker tracker; setRecoveryReplayMemoryProbeForTest(tracker.probe()); SCOPE_EXIT({ setRecoveryReplayMemoryProbeForTest({}); }); - const CasRefCatalog::Snapshot sweep_catalog_cut = CasRefCatalog::read(*backend, layout); + CasRequests sweep_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation sweep_op = sweep_requests.admit(); + const CasRefCatalog::Snapshot sweep_catalog_cut = CasRefCatalog::read(sweep_op, layout); const RecoveredRefTable recovered = recoverRefTableDetailedAtCatalogCutForTest(*backend, layout, sweep_catalog_cut, ns_sweep); EXPECT_EQ(recovered.state.getPrecommits().size(), kTxns * kOpsPerTxn); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index f2a3ec1d6570..de93bd54c817 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -11,10 +11,16 @@ #include #include #include +#include #include +#include +#include +#include +#include #include #include #include +#include #include @@ -36,7 +42,8 @@ class GcRoundPlanSignatureAccess public: using FoldSignature = decltype(&Gc::fold); using ExpectedFoldSignature = Gc::FoldResult (Gc::*)( - GcState &, Token &, RoundReport &, uint64_t, const RefPlan &, UniversePolicy, GcRoundWorkBudget &); + GcState &, std::optional &, RoundReport &, uint64_t, const RefPlan &, UniversePolicy, + GcRoundWorkBudget &); using BuilderSignature = decltype(&buildRefWalkPlan); using ExpectedBuilderSignature = RefPlan (*)(RoundInput &&); @@ -53,6 +60,7 @@ namespace DB::ErrorCodes extern const int LIMIT_EXCEEDED; extern const int NETWORK_ERROR; extern const int BAD_ARGUMENTS; + extern const int S3_ERROR; } namespace @@ -104,41 +112,71 @@ CatalogEntry entryInState(const String & ns, NsState state, uint64_t inc) return entry; } -class EraseWinnerBackend final : public DB::Cas::tests::CountingBackend +/// Per-key counts of the WRITE primitive. `CountingBackend` counts reads, heads and lists per key but +/// only totals for writes, and its legacy per-verb counters never see a caller that speaks the +/// primitives -- which every catalog writer below does. +class WriteCountingBackend : public DB::Cas::tests::CountingBackend { public: - using CountingBackend::casPut; - using CountingBackend::get; + uint64_t writes(const String & key) const + { + std::lock_guard lock(write_count_mutex); + const auto it = write_counts.find(key); + return it == write_counts.end() ? 0 : it->second; + } - void replaceOnNextCatalogCas(const String & key, std::optional replacement_) + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(write_count_mutex); + ++write_counts[key]; + } + return CountingBackend::write(key, bytes, expected_value, access); + } + +private: + mutable std::mutex write_count_mutex; + std::map write_counts; +}; + +/// Lands a competing catalog body under the erase's own attempt and withdraws this actor's admission +/// with it -- the concurrent winner an erase has to be resolved against, driven deterministically and +/// without a second thread. +class EraseWinnerBackend final : public WriteCountingBackend +{ +public: + void replaceOnNextCatalogWrite(const String & key, std::optional replacement_) { catalog_key = key; replacement = std::move(replacement_); armed = true; } - bool fenceMoved() const { return fence_moved; } + bool admitted() const { return !fence_moved; } - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (armed && key == catalog_key) { armed = false; - const auto current = CountingBackend::get(key); - if (!current) - throw std::runtime_error("test fixture lost mandatory catalog"); RefCatalog winner_catalog; if (replacement) winner_catalog.entries.push_back(*replacement); - const CasResult winner = CountingBackend::casPut( - key, encodeRefCatalog(winner_catalog), current->token, meta); - if (winner.outcome != CasOutcome::Committed) + /// Qualified, so the winner's own write is not counted as an attempt of the call under test. + const auto current = WriteCountingBackend::read(key, access); + if (!current) + throw std::runtime_error("test fixture lost mandatory catalog"); + const auto winner = CountingBackend::write( + key, encodeRefCatalog(winner_catalog), std::optional{current->value}, access); + if (!winner.has_value()) throw std::runtime_error("test fixture winner failed to replace catalog"); fence_moved = true; } - return CountingBackend::casPut(key, bytes, expected, meta); + return WriteCountingBackend::write(key, bytes, expected_value, access); } private: @@ -148,34 +186,17 @@ class EraseWinnerBackend final : public DB::Cas::tests::CountingBackend bool fence_moved = false; }; -class CasPutThrowsOnceBackend final : public DB::Cas::tests::CountingBackend -{ -public: - using CountingBackend::casPut; - - void armCasPutThrow(const String & key) - { - throw_key = key; - armed = true; - } - - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override - { - if (armed && key == throw_key) - { - armed = false; - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, - "injected casPut failure during completed-removal erase"); - } - return CountingBackend::casPut(key, bytes, expected, meta); - } +/// The erase entry points require a liveness refresh because a real drain's liveness is a cached flag +/// its owner re-reads from the store. These fixtures' operations carry either no liveness or a direct +/// read of the fixture, so there is nothing cached for a refresh to update. Named rather than repeated +/// inline, so a site that DOES need a refresh cannot hide among the ones that do not. +void noAuthorityRefresh() {} -private: - String throw_key; - bool armed = false; -}; +/// Seeds one object, failing the current test rather than returning a value nobody checks. +void seedObject(CasOperation & op, const String & key, const String & bytes) +{ + ASSERT_TRUE(std::holds_alternative(op.create(key, bytes, Retry::standard()))); +} class ScopedCasGcLogCapture { @@ -322,7 +343,9 @@ TEST(CASRefCatalogLifeIndex, DuplicatePhysicalIdsAreAmbiguousWithoutPoisoningUni /// candidate can be written. An unrelated unique point lookup remains available from the same cut. TEST(CASRefCatalogLifeIndex, AmbiguityStopsCatalogMutationButNotUnrelatedPointLookup) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); RefCatalog catalog; catalog.entries = { @@ -330,17 +353,17 @@ TEST(CASRefCatalogLifeIndex, AmbiguityStopsCatalogMutationButNotUnrelatedPointLo entryInState("b", NsState::Removing, 7), entryInState("c", NsState::Live, 9), }; - ASSERT_EQ(backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(catalog)).outcome, PutOutcome::Done); - const auto before = backend.get(layout.refCatalogKey()); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(catalog)); + const auto before = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(before); - EXPECT_THROW(CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & current) { return current; }), DB::Exception); - const auto after = backend.get(layout.refCatalogKey()); + EXPECT_THROW(CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { return current; }), DB::Exception); + const auto after = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(after); - EXPECT_EQ(after->token, before->token); + EXPECT_EQ(after->incarnation, before->incarnation); EXPECT_EQ(after->bytes, before->bytes); - const auto unique = CasRefCatalog::lifeIfCataloged(backend, layout, RootNamespace{"c"}); + const auto unique = CasRefCatalog::lifeIfCataloged(op, layout, RootNamespace{"c"}); ASSERT_TRUE(unique); EXPECT_EQ(unique->incarnation, UInt128{9}); } @@ -778,13 +801,17 @@ TEST(CASRefCatalogAdmission, RemovalNeverRefusedEvenAtCapacity) for (uint64_t i = 0; i < max_entries; ++i) full.entries.push_back(liveEntry(fmt::format("ns{:012}", i), i + 1)); - InMemoryBackend backend; - backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(full)); + auto backend = std::make_shared(); + + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + + CasOperation op = requests.admit(); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(full)); /// The removal transition (Live -> Removing) on one entry goes through the PLAIN update path /// (`casUpdate`, which runs no admission check at all) and succeeds even though the catalog is /// already at the point where ANY growth would be refused. - const RefCatalog after = CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & cur) + const RefCatalog after = CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & cur) { RefCatalog next = cur; next.entries[0].state = NsState::Removing; @@ -795,36 +822,42 @@ TEST(CASRefCatalogAdmission, RemovalNeverRefusedEvenAtCapacity) EXPECT_EQ(after.entries[0].state, NsState::Removing); } -/// ---------- Pool/CasRefCatalog: token-CAS read / create / update / conflict-retry ---------- +/// ---------- Pool/CasRefCatalog: read / create / update / conflict-retry ---------- TEST(CASRefCatalog, ReadAbsentFailsClosed) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)CasRefCatalog::read(backend, layout); }); + DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)CasRefCatalog::read(op, layout); }); } TEST(CASRefCatalog, CasUpdateRefusesWhenAbsent) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & cur) { return cur; }); + CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & cur) { return cur; }); }); - EXPECT_FALSE(backend.head(layout.refCatalogKey()).exists); + EXPECT_FALSE(op.head(layout.refCatalogKey(), Retry::standard()).has_value()); } TEST(CASRefCatalog, CasUpdateAppliesOnTopOfExistingState) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); - const RefCatalog updated = CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & cur) + const RefCatalog updated = CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & cur) { RefCatalog next = cur; next.entries[0].state = NsState::Removing; @@ -846,22 +879,26 @@ TEST(CASRefCatalog, GenericCasUpdateCannotDeleteOrReplaceCatalogIdentity) { const Layout layout("p"); { - InMemoryBackend backend; - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { - (void)CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog &) { return RefCatalog{}; }); + (void)CasRefCatalog::casUpdate(op, layout, [](const RefCatalog &) { return RefCatalog{}; }); }); } { - InMemoryBackend backend; - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { - (void)CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & current) + (void)CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { RefCatalog next = current; next.entries[0] = liveEntry("b", 2); @@ -877,21 +914,25 @@ TEST(CASRefCatalogDeathTest, GenericCasUpdateCannotDeleteOrReplaceCatalogIdentit { const Layout layout("p"); { - InMemoryBackend backend; - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); EXPECT_DEATH( - { (void)CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog &) { return RefCatalog{}; }); }, + { (void)CasRefCatalog::casUpdate(op, layout, [](const RefCatalog &) { return RefCatalog{}; }); }, "cannot add or delete catalog entries"); } { - InMemoryBackend backend; - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); EXPECT_DEATH( { - (void)CasRefCatalog::casUpdate(backend, layout, [](const RefCatalog & current) + (void)CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { RefCatalog next = current; next.entries[0] = liveEntry("b", 2); @@ -905,15 +946,17 @@ TEST(CASRefCatalogDeathTest, GenericCasUpdateCannotDeleteOrReplaceCatalogIdentit TEST(CASRefCatalog, CasUpdateRetriesOnConflictAgainstFreshState) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); - backend.failNextCasPut(layout.refCatalogKey()); /// one-shot artificial Conflict on the next write + backend->refuseNextWrite(layout.refCatalogKey()); /// one-shot artificial Conflict on the next write int mutate_calls = 0; - const RefCatalog result = CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & cur) + const RefCatalog result = CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & cur) { ++mutate_calls; RefCatalog next = cur; @@ -926,66 +969,64 @@ TEST(CASRefCatalog, CasUpdateRetriesOnConflictAgainstFreshState) ASSERT_EQ(result.entries.size(), 1u); EXPECT_EQ(result.entries[0].state, NsState::Removing); - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); EXPECT_EQ(snap.catalog, result); } -TEST(CASRefCatalog, BeginRemovingRechecksFenceAfterCatalogCasConflict) +TEST(CASRefCatalog, BeginRemovingRechecksAdmissionAfterACatalogConflict) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation reader = requests.admit(); const Layout layout("p"); const CatalogEntry observed = liveEntry("a", 1); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, observed); - - uint64_t current_fence_generation = 7; - size_t fence_checks = 0; - const auto outcome = CasRefCatalog::beginRemoving( - backend, layout, observed, /*removal_started_round*/ 13, /*admitted_generation*/ 7, - [&](uint64_t admitted_generation) - { - ++fence_checks; - if (admitted_generation != current_fence_generation) - throw std::runtime_error("stale catalog mutation fence"); - if (fence_checks == 1) - { - /// Move the caller fence after the first admission check and force that attempt's - /// catalog CAS to conflict. The next attempt must check the fence again before writing. - current_fence_generation = 8; - backend.failNextCasPut(layout.refCatalogKey()); - } - }); + CasRefCatalog::initializeEmptyForNewPool(reader, layout); + CasRefCatalog::casAdmitEntry(reader, layout, 1, observed); + const uint64_t writes_before = backend->writes(layout.refCatalogKey()); + + /// The transition's one attempt is refused, and this actor's admission is withdrawn as it is sent. + /// The refusal must end the call rather than start another attempt, and it must be reported as an + /// outcome rather than thrown. + bool admitted = true; + backend->refuseNextWrite(layout.refCatalogKey()); + backend->onBeforeWrite(layout.refCatalogKey(), [&admitted] { admitted = false; }); + CasOperation op = requests.admit([&admitted] { return admitted; }); + + const auto outcome = CasRefCatalog::beginRemoving(op, layout, observed, /*removal_started_round*/ 13); EXPECT_EQ(outcome, CasRefCatalog::BeginRemovingOutcome::FencedOut); - EXPECT_EQ(fence_checks, 2u); - const CasRefCatalog::Snapshot after = CasRefCatalog::read(backend, layout); - EXPECT_EQ(after.catalog.entries, std::vector{observed}); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_before + 1) + << "the refused attempt must not be followed by another"; + const CasRefCatalog::Snapshot after = CasRefCatalog::read(reader, layout); + EXPECT_EQ(after.catalog.entries, std::vector{observed}) + << "nothing may be written after the admission is gone"; } /// A re-read that finds the catalog genuinely ABSENT after it was previously observed present is a /// real concurrent delete, not a bootstrap -- `casUpdate` must refuse rather than silently create a /// fresh catalog containing only this one mutation's entry (which would drop every other namespace). /// Reproduced with a REAL delete (no fault injection needed): `mutate`'s first invocation deletes the -/// seeded object using the token `casUpdate`'s own initial read observed, so the loop's own `casPut` -/// against that now-stale token gets a genuine `Conflict`, and the follow-up re-read genuinely finds -/// the key absent. +/// seeded object at the incarnation the update's own initial read observed, so its conditional write +/// is refused and the read that settles the refusal genuinely finds the key absent. /// Missing mandatory authority raises `CORRUPTED_DATA`; the split remains only because the debug /// variant historically lived in the death-test suite. #ifndef DEBUG_OR_SANITIZER_BUILD TEST(CASRefCatalog, CasUpdateThrowsOnVanishMidRetryInsteadOfReplacingTheCatalog) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); - const CasRefCatalog::Snapshot seeded = CasRefCatalog::read(backend, layout); - ASSERT_TRUE(seeded.token.has_value()); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); + const CasRefCatalog::Snapshot seeded = CasRefCatalog::read(op, layout); + ASSERT_TRUE(seeded.incarnation.has_value()); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & cur) + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & cur) { - backend.deleteExact(layout.refCatalogKey(), *seeded.token); + EXPECT_EQ(op.remove(layout.refCatalogKey(), *seeded.incarnation, Retry::standard()), Removal::Removed); RefCatalog next = cur; next.entries[0].state = NsState::Removing; next.entries[0].removal_started_round = 1; @@ -995,25 +1036,27 @@ TEST(CASRefCatalog, CasUpdateThrowsOnVanishMidRetryInsteadOfReplacingTheCatalog) /// Nothing was written by the failed attempt: the object is exactly as the delete left it /// (absent), never a fresh single-entry catalog. - EXPECT_FALSE(backend.head(layout.refCatalogKey()).exists); + EXPECT_FALSE(op.head(layout.refCatalogKey(), Retry::standard()).has_value()); } #endif #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASRefCatalogDeathTest, CasUpdateThrowsOnVanishMidRetryInsteadOfReplacingTheCatalogAborts) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); - const CasRefCatalog::Snapshot seeded = CasRefCatalog::read(backend, layout); - ASSERT_TRUE(seeded.token.has_value()); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); + const CasRefCatalog::Snapshot seeded = CasRefCatalog::read(op, layout); + ASSERT_TRUE(seeded.incarnation.has_value()); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & cur) + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & cur) { - backend.deleteExact(layout.refCatalogKey(), *seeded.token); + EXPECT_EQ(op.remove(layout.refCatalogKey(), *seeded.incarnation, Retry::standard()), Removal::Removed); RefCatalog next = cur; next.entries[0].state = NsState::Removing; next.entries[0].removal_started_round = 1; @@ -1023,36 +1066,49 @@ TEST(CASRefCatalogDeathTest, CasUpdateThrowsOnVanishMidRetryInsteadOfReplacingTh } #endif -/// The retry loop is bounded (the same live-lock brake `publishCkpt`/`allocateWriterEpoch` use on -/// their own contended token-CAS singletons) and ends in the typed retryable error, not an infinite -/// spin. `mutate` re-arms the one-shot conflict injection on every call, so every attempt fails. -TEST(CASRefCatalog, CasUpdateGivesUpAfterBoundedAttemptsWithRetryLaterError) +/// Persistent contention ends at the write policy's DEADLINE, with the typed retryable error -- not +/// after a fixed number of unslept iterations, and not in an infinite spin. `mutate` re-arms the +/// one-shot conflict injection on every call, so every attempt is refused; the injected clock reaches +/// the deadline without the test sleeping at all. +TEST(CASRefCatalog, CasUpdateEndsAtTheDeadlineNotAfterAHundredUnsleptIterations) { - InMemoryBackend backend; + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + const uint64_t started_at = clock.now; int mutate_calls = 0; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { - CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & cur) + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & cur) { ++mutate_calls; - backend.failNextCasPut(layout.refCatalogKey()); + backend->refuseNextWrite(layout.refCatalogKey()); RefCatalog next = cur; return next; }); }); EXPECT_GT(mutate_calls, 1); /// genuinely retried, not a single-shot failure + EXPECT_FALSE(clock.sleeps.empty()) << "every retry must back off; an unslept loop would burn the " + "deadline on requests instead of waiting out the contention"; + EXPECT_GE(clock.now - started_at, Retry::standard().window_ms - 5000) + << "the loop ended at the policy's own deadline, not at an iteration count"; } TEST(CASRefCatalog, CasAdmitEntryAcceptsAnOrdinaryCreation) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); - const RefCatalog created = CasRefCatalog::casAdmitEntry(backend, layout, 1, + const RefCatalog created = CasRefCatalog::casAdmitEntry(op, layout, 1, CatalogEntry{.ns = RootNamespace{"a"}, .state = NsState::Creating, .incarnation = UInt128(1), .creator = CreatorFence{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}}); ASSERT_EQ(created.entries.size(), 1u); @@ -1061,12 +1117,14 @@ TEST(CASRefCatalog, CasAdmitEntryAcceptsAnOrdinaryCreation) TEST(CASRefCatalog, CasAdmitEntryInsertsAtCanonicalPosition) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("b", 1)); - const RefCatalog after = CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 2)); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("b", 1)); + const RefCatalog after = CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 2)); ASSERT_EQ(after.entries.size(), 2u); EXPECT_EQ(after.entries[0].ns.string(), "a"); /// inserted BEFORE "b", not appended EXPECT_EQ(after.entries[1].ns.string(), "b"); @@ -1078,29 +1136,35 @@ TEST(CASRefCatalog, CasAdmitEntryInsertsAtCanonicalPosition) #ifndef DEBUG_OR_SANITIZER_BUILD TEST(CASRefCatalog, CasAdmitEntryRejectsADuplicateNamespace) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, - [&] { CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 2)); }); + [&] { CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 2)); }); } #endif #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASRefCatalogDeathTest, CasAdmitEntryRejectsADuplicateNamespaceAborts) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 1)); - EXPECT_DEATH({ CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("a", 2)); }, "not canonically ordered"); + CasRefCatalog::initializeEmptyForNewPool(op, layout); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); + EXPECT_DEATH({ CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 2)); }, "not canonically ordered"); } #endif TEST(CASRefCatalog, CasAdmitEntryRefusesOverCapacity) { - InMemoryBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); Layout layout("p"); const uint64_t cap = foldSealCaps().object_cap; @@ -1115,30 +1179,31 @@ TEST(CASRefCatalog, CasAdmitEntryRefusesOverCapacity) full.entries.reserve(max_entries); for (uint64_t i = 0; i < max_entries; ++i) full.entries.push_back(liveEntry(fmt::format("ns{:012}", i), i + 1)); - backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(full)); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(full)); /// Admitting ONE more namespace is refused -- the additive predicate is checked BEFORE the write, /// so the backend object is untouched. DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LIMIT_EXCEEDED, [&] { - CasRefCatalog::casAdmitEntry(backend, layout, 1, liveEntry("zzz", 999999999)); + CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("zzz", 999999999)); }); - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); EXPECT_EQ(snap.catalog.entries.size(), max_entries); } -TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLeaderFence) +TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndAdmission) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, .incarnation = UInt128{7}, .removal_started_round = 13}; - ASSERT_EQ(backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})).outcome, - PutOutcome::Done); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); CasFoldSeal held_parent; held_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ @@ -1147,18 +1212,14 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe .last_folded_ref_id = RefTxnId{1, 2}, .hold = RefHold{.offending_position = RefTxnId{1, 3}}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, held_parent, 5, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }), + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, held_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::ProofRefused); CasFoldSeal mismatched_parent; mismatched_parent.ref_lives.emplace(UInt128{8}, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, mismatched_parent, 5, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }), + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, mismatched_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::ProofRefused); CasFoldSeal ready_parent; @@ -1169,42 +1230,39 @@ TEST(CASRefCatalogRemoval, DeleteCompletedRemovingRequiresExactAdoptedProofAndLe CatalogEntry live = removing; live.state = NsState::Live; live.removal_started_round.reset(); - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, live, ready_parent, 5, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }), + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, live, ready_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::ProofRefused); CatalogEntry creating = live; creating.state = NsState::Creating; creating.creator = CreatorFence{.server_root_id = "server", .writer_epoch = 3, .fence_generation = 4}; - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, creating, ready_parent, 5, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }), + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, creating, ready_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::ProofRefused); - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Moved; }), + /// An operation whose admission is gone erases nothing and sends nothing. It is driven at a cut an + /// ADMITTED operation took, because a withdrawn one cannot issue the mandatory read that would + /// take one. + CasOperation withdrawn = requests.admit([] { return false; }); + EXPECT_EQ(CasRefCatalog::deleteCompletedRemovingAtSnapshot( + withdrawn, layout, CasRefCatalog::read(op, layout), removing, ready_parent, + noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::FencedOut); - EXPECT_EQ(backend.casPutCount(layout.refCatalogKey()), 0); + const uint64_t writes_before_erase = backend->writes(layout.refCatalogKey()); - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, [](uint64_t generation) - { - EXPECT_EQ(generation, 5); - return CasRefCatalog::LeaderFenceStatus::Held; - }), + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); - EXPECT_TRUE(CasRefCatalog::read(backend, layout).catalog.entries.empty()); - EXPECT_EQ(backend.casPutCount(layout.refCatalogKey()), 1); - EXPECT_EQ(backend.listTotal(), 0); - EXPECT_EQ(backend.deleteTotal(), 0); + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_before_erase + 1); + EXPECT_EQ(backend->listTotal(), 0u); + EXPECT_EQ(backend->deleteTotal(), 0u); } TEST(CASRefCatalogRemoval, ExactDeletionRefusesChangedEntryAndAdmissionCannotCarryRemoval) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, @@ -1212,7 +1270,7 @@ TEST(CASRefCatalogRemoval, ExactDeletionRefusesChangedEntryAndAdmissionCannotCar .removal_started_round = 13}; #ifndef DEBUG_OR_SANITIZER_BUILD DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, - [&] { (void)CasRefCatalog::casAdmitEntry(backend, layout, 1, removing); }); + [&] { (void)CasRefCatalog::casAdmitEntry(op, layout, 1, removing); }); #endif const CatalogEntry current{ @@ -1220,27 +1278,28 @@ TEST(CASRefCatalogRemoval, ExactDeletionRefusesChangedEntryAndAdmissionCannotCar .state = NsState::Removing, .incarnation = UInt128{7}, .removal_started_round = 14}; - ASSERT_EQ(backend.putIfAbsent("unrelated", "sentinel").outcome, PutOutcome::Done); - ASSERT_EQ(backend.casPut(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {current}}), - CasRefCatalog::read(backend, layout).token).outcome, CasOutcome::Committed); + seedObject(op, "unrelated", "sentinel"); + ASSERT_TRUE(std::holds_alternative(op.replace( + layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {current}}), + *CasRefCatalog::read(op, layout).incarnation, Retry::standard()))); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); - EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }), + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::EntryChanged); - EXPECT_EQ(CasRefCatalog::read(backend, layout).catalog.entries, std::vector{current}); + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{current}); } #if defined(DEBUG_OR_SANITIZER_BUILD) TEST(CASRefCatalogRemovalDeathTest, AdmissionCannotCarryRemovalAborts) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); - CasRefCatalog::initializeEmptyForNewPool(backend, layout); + CasRefCatalog::initializeEmptyForNewPool(op, layout); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, @@ -1248,28 +1307,35 @@ TEST(CASRefCatalogRemovalDeathTest, AdmissionCannotCarryRemovalAborts) .removal_started_round = 13}; EXPECT_DEATH( - { (void)CasRefCatalog::casAdmitEntry(backend, layout, 1, removing); }, + { (void)CasRefCatalog::casAdmitEntry(op, layout, 1, removing); }, "cannot admit namespace.*directly as Removing"); } #endif -/// Mutation caught: deriving the control outcome from the resolution snapshot would turn a stale -/// leader's `FencedOut` into `Deleted` or `EntryChanged`. Resolution may prove the old life dead and -/// carry its invalidation, but it cannot restore the caller's authority to continue the GC round. +/// Mutation caught: deriving the control outcome from the resolution read would turn a stale leader's +/// `FencedOut` into `Deleted` or `EntryChanged`. A winner that replaced the catalog under this erase +/// cannot restore the caller's authority to continue the GC round, and the refusal stays a returned +/// outcome rather than an exception -- the caller distinguishes "I lost the round" from "I could not +/// talk to the store" by exactly that. +/// +/// An operation whose admission is gone cannot issue the resolution read either, so the result carries +/// the cut this call was GIVEN rather than a fresh one: the erase's own effect is deliberately left +/// unreported, because there is no admitted request left with which to learn it. TEST(CASRefCatalogRemoval, FenceLossRemainsControlOutcomeWhenWinnerRemovesOrReplacesLife) { for (const bool replace : {false, true}) { - EraseWinnerBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation reader = requests.admit(); + CasOperation op = requests.admit([&backend] { return backend->admitted(); }); const Layout layout(replace ? "replacement" : "absence"); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, .incarnation = UInt128{7}, .removal_started_round = 13}; - ASSERT_EQ(backend.putIfAbsent( - layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})).outcome, - PutOutcome::Done); + seedObject(reader, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ @@ -1281,22 +1347,15 @@ TEST(CASRefCatalogRemoval, FenceLossRemainsControlOutcomeWhenWinnerRemovesOrRepl .ns = removing.ns, .state = NsState::Live, .incarnation = UInt128{8}}; - backend.replaceOnNextCatalogCas(layout.refCatalogKey(), replacement); + backend->replaceOnNextCatalogWrite(layout.refCatalogKey(), replacement); const CasRefCatalog::CompletedRemovingDeleteResult result - = CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, [&](uint64_t) - { - if (backend.fenceMoved()) - return CasRefCatalog::LeaderFenceStatus::Moved; - return CasRefCatalog::LeaderFenceStatus::Held; - }); + = CasRefCatalog::deleteCompletedRemovingAtSnapshot( + op, layout, CasRefCatalog::read(reader, layout), removing, ready_parent, + noAuthorityRefresh); EXPECT_EQ(result.outcome, CasRefCatalog::CompletedRemovingDeleteOutcome::FencedOut); - ASSERT_TRUE(result.invalidated_life); - EXPECT_EQ(*result.invalidated_life, - NamespaceLifeId::fromCatalogEntry(removing.ns, removing.incarnation)); - const RefCatalog current = CasRefCatalog::read(backend, layout).catalog; + const RefCatalog current = CasRefCatalog::read(reader, layout).catalog; if (replace) EXPECT_EQ(current.entries, std::vector{*replacement}); else @@ -1304,141 +1363,333 @@ TEST(CASRefCatalogRemoval, FenceLossRemainsControlOutcomeWhenWinnerRemovesOrRepl } } -/// Mutation caught: treating every authority-check exception as a moved fence hides corruption and -/// backend/decode failures. Before any CAS, inability to evaluate authority must propagate unchanged. -TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesBeforeEraseCas) +/// A transient failure of the erase attempt is settled by the mandatory resolution read and reissued, +/// never concluded from. Treating it as ordinary non-convergence would hide a real backend fault behind +/// `ProofRefused`/`EntryChanged`; treating it as a landed erase would report a deletion nobody proved. +TEST(CASRefCatalogRemoval, ATransientEraseFailureIsResolvedByAReadAndReissued) { - DB::Cas::tests::CountingBackend backend; - const Layout layout("pre-cas-authority-error"); + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + const Layout layout("cas-put-throw"); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, .incarnation = UInt128{7}, .removal_started_round = 13}; - ASSERT_EQ(backend.putIfAbsent( - layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})).outcome, - PutOutcome::Done); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] + const uint64_t reads_before = backend->getCount(layout.refCatalogKey()); + backend->failNextWriteWith(layout.refCatalogKey(), std::make_exception_ptr( + Poco::TimeoutException("injected erase failure whose outcome never reached the caller"))); + + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh), + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), 3u) + << "the seed, the attempt whose outcome was lost, and the reissue that landed"; + EXPECT_GT(backend->getCount(layout.refCatalogKey()), reads_before + 1) + << "the lost attempt was settled by an exact read before anything was concluded from it"; +} + +/// A refused precondition is the only thing the erase loop retries, and it PACES that retry on the +/// engine's own clock. Without the pause a contended catalog would spend its whole conflict budget in +/// back-to-back requests, which is the shape that turns one hot key into a request storm. +/// +/// Nothing else on this path sleeps -- the write returns a refused precondition without reissuing, and +/// no read fails -- so every wait the clock records is the loop's own. +TEST(CASRefCatalogRemoval, AConflictingEraseBacksOffBeforeItsRetry) +{ + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + const Layout layout("erase-backoff"); + const CatalogEntry removing{ + .ns = RootNamespace{"a"}, + .state = NsState::Removing, + .incarnation = UInt128{7}, + .removal_started_round = 13}; + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + + backend->refuseNextWrite(layout.refCatalogKey()); + + EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh), + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + ASSERT_EQ(clock.sleeps.size(), 1u) << "one refusal, so exactly one paced retry"; + EXPECT_LE(clock.sleeps.front(), 200u) << "the first reissue's full-jitter ceiling"; + EXPECT_EQ(backend->writes(layout.refCatalogKey()), 3u) + << "the seed, the refused erase, and the retry that landed"; +} + +/// A caller's liveness need not be a fact this loop can read. The GC drain's is a cached +/// leader-authority flag its owner refreshes by re-reading `gc/state`, so a leader deposed while an +/// erase is in flight leaves that flag stale -- and on an unrecoverable delete path one reading taken +/// before the first erase must not authorise the rest. Hence the refresh hook, run at the top of every +/// attempt. +/// +/// The winner here writes the SAME row back: the incarnation moves, so the erase is refused, and the +/// row survives for the retry the loop would otherwise send. The cached flag still answers "admitted" +/// at the post-write probe, deliberately -- that probe is not the subject, and leaving it stale is what +/// makes this test about the refresh and nothing else. +TEST(CASRefCatalogRemoval, TheEraseLoopRefreshesItsLivenessBeforeEveryAttempt) +{ + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation reader = requests.admit(); + bool cached_authority = true; + CasOperation op = requests.admit([&cached_authority] { return cached_authority; }); + const Layout layout("erase-refresh"); + const CatalogEntry removing{ + .ns = RootNamespace{"a"}, + .state = NsState::Removing, + .incarnation = UInt128{7}, + .removal_started_round = 13}; + seedObject(reader, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + + backend->replaceOnNextCatalogWrite(layout.refCatalogKey(), removing); + const uint64_t writes_after_seed = backend->writes(layout.refCatalogKey()); + size_t refreshes = 0; + + const CasRefCatalog::CompletedRemovingDeleteResult result + = CasRefCatalog::deleteCompletedRemovingAtSnapshot( + op, layout, CasRefCatalog::read(reader, layout), removing, ready_parent, + [&] { ++refreshes; cached_authority = backend->admitted(); }); + + EXPECT_EQ(result.outcome, CasRefCatalog::CompletedRemovingDeleteOutcome::FencedOut); + EXPECT_EQ(refreshes, 2u) << "once before the attempt it sent, once before the one it did not"; + EXPECT_EQ(backend->writes(layout.refCatalogKey()) - writes_after_seed, 1u) + << "the one refused erase; the paced retry was abandoned before it reached the store"; + EXPECT_EQ(CasRefCatalog::read(reader, layout).catalog.entries, std::vector{removing}) + << "the row a deposed leader must not erase is still there"; +} + +/// The loop iterates on ONE alternative and one only: a refused precondition. Every other non-committed +/// answer is terminal for the call and leaves through the same throw, so a future retry added for any +/// of them would be retrying a write whose fate the store already settled. `Refused` is the alternative +/// a test can construct exactly; `GaveUp{FenceLost}` cannot reach this arm at all, because the +/// admission probe immediately after the write returns `FencedOut` first. +#if USE_AWS_S3 +TEST(CASRefCatalogRemoval, AStoreRefusalEndsTheEraseLoopInsteadOfRetryingIt) +{ + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + const Layout layout("erase-refused"); + const CatalogEntry removing{ + .ns = RootNamespace{"a"}, + .state = NsState::Removing, + .incarnation = UInt128{7}, + .removal_started_round = 13}; + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const uint64_t writes_after_seed = backend->writes(layout.refCatalogKey()); + + /// A malformed request is one of the answers that prove the write never applied, so the engine + /// reports it as a refusal rather than settling it by a read. + backend->failNextWriteWith(layout.refCatalogKey(), std::make_exception_ptr( + DB::S3Exception("injected malformed erase request", Aws::S3::S3Errors::UNKNOWN, "MalformedXML"))); + + /// The store's own code, not a class of this module's choosing: `orThrow` re-raises a refusal + /// under the code the store gave, so an operator sees what the store actually said. + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { - (void)CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, [](uint64_t) -> CasRefCatalog::LeaderFenceStatus - { - throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, - "injected authority read failure before erase CAS"); - }); + (void)CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh); }); - EXPECT_EQ(backend.casPutCount(layout.refCatalogKey()), 0u); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_after_seed + 1) + << "a refusal is terminal: the loop must not send a second erase"; + EXPECT_TRUE(clock.sleeps.empty()) << "and must not pace a retry it is not going to make"; + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{removing}) + << "the refused erase changed nothing"; } +#endif -/// The post-CAS authority check is distinct: the erase may already be durable and its mandatory -/// resolution complete, but inability to evaluate authority is still the original error, not -/// `FencedOut`. -TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesAfterEraseResolution) -{ - DB::Cas::tests::CountingBackend backend; - const Layout layout("post-cas-authority-error"); +/// The response to a conditional erase is not authority for what became durable, and this is the case +/// that makes that concrete: the attempt commits, a concurrent writer puts the row back, and the +/// mandatory resolution read contradicts the commit. Believing the response would report a namespace +/// deleted while its row is still cataloged, so the call fails retry-later instead. +TEST(CASRefCatalogRemoval, ACommitTheResolutionReadContradictsFailsRetryLaterInsteadOfReportingDeleted) +{ + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + CasOperation restorer = requests.admit(); + const Layout layout("erase-contradicted"); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, .incarnation = UInt128{7}, .removal_started_round = 13}; - ASSERT_EQ(backend.putIfAbsent( - layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})).outcome, - PutOutcome::Done); + const String seeded_catalog = encodeRefCatalog(RefCatalog{.entries = {removing}}); + seedObject(op, layout.refCatalogKey(), seeded_catalog); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); - size_t authority_checks = 0; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] + /// Fires once the erase is durable and before its answer is returned, so the call really does see + /// `Committed` and really does find the row back when it resolves. + bool restored = false; + backend->onWriteCommitted(layout.refCatalogKey(), [&] { - (void)CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, [&](uint64_t) - { - if (++authority_checks == 2) - throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, - "injected authority read failure after erase resolution"); - return CasRefCatalog::LeaderFenceStatus::Held; - }); + if (restored) + return; + restored = true; + (void)restorer.readModifyWrite(layout.refCatalogKey(), + [&](const std::optional &) -> std::optional { return seeded_catalog; }, + Retry::standard()); }); - EXPECT_EQ(authority_checks, 2u); - EXPECT_TRUE(CasRefCatalog::read(backend, layout).catalog.entries.empty()); + + String message; + try + { + (void)CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh); + ADD_FAILURE() << "a commit the resolution read contradicts must not be reported as a deletion"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::NETWORK_ERROR); + message = e.message(); + } + EXPECT_TRUE(restored) << "the concurrent restore never ran, so nothing was contradicted"; + EXPECT_NE(message.find("reported committed"), String::npos) << message; + EXPECT_NE(message.find(u128ToHex(removing.incarnation)), String::npos) + << "the message must name the life still observed: " << message; } -/// Mutation caught: swallowing a synchronous `casPut` exception raised during the erase attempt -/// itself (as opposed to the authority/fence check) and treating it as ordinary non-convergence -/// would hide a real backend fault behind ProofRefused/EntryChanged, and would skip the mandatory -/// resolution read that this branch's siblings above already prove runs before any conclusion. -TEST(CASRefCatalogRemoval, CasPutExceptionPropagatesAfterMandatoryResolution) +/// After the migration this cap is the ONLY bound the hand-written loop has of its own, so it is worth +/// proving it ends the call rather than letting a permanently contended catalog spin. Every erase is +/// refused, the injected clock absorbs every paced retry, and the loop stops on its attempt count -- +/// which the message says, and which is what tells it apart from a deadline. +TEST(CASRefCatalogRemoval, PerpetualConflictEndsAtTheAttemptCapAndSaysSo) { - CasPutThrowsOnceBackend backend; - const Layout layout("cas-put-throw"); + class AlwaysRefusesCatalogWrites final : public WriteCountingBackend + { + public: + String refused_key; + + uint64_t refusedAttempts() const { return refused_attempts; } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + if (key == refused_key) + { + /// A genuine refused precondition never applies -- unlike `WriteCountingBackend::write`, + /// which both counts AND delegates to the real, LANDING write. Counting this attempt + /// through that base would apply it to storage and then lie about the outcome, which + /// made the erase loop's very first attempt land for real and see `Deleted` instead of + /// the perpetual conflict this backend's name promises. Count it here instead, and + /// return only the refusal. + ++refused_attempts; + return std::unexpected(DB::Cas::Backend::RawConflict{}); + } + return WriteCountingBackend::write(key, bytes, expected_value, access); + } + + private: + uint64_t refused_attempts = 0; + }; + + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + const Layout layout("erase-cap"); const CatalogEntry removing{ .ns = RootNamespace{"a"}, .state = NsState::Removing, .incarnation = UInt128{7}, .removal_started_round = 13}; - ASSERT_EQ(backend.putIfAbsent( - layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})).outcome, - PutOutcome::Done); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + backend->refused_key = layout.refCatalogKey(); - backend.armCasPutThrow(layout.refCatalogKey()); - - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + String message; + try { - (void)CasRefCatalog::deleteCompletedRemoving( - backend, layout, removing, ready_parent, 5, [](uint64_t) - { - return CasRefCatalog::LeaderFenceStatus::Held; - }); - }); - /// The mandatory resolution read ran before the rethrow: the exact old row is still present, - /// unchanged by the failed attempt. - const RefCatalog current = CasRefCatalog::read(backend, layout).catalog; - EXPECT_EQ(current.entries, std::vector{removing}); + (void)CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh); + ADD_FAILURE() << "a permanently refused erase must not return an outcome"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::NETWORK_ERROR); + message = e.message(); + } + EXPECT_NE(message.find("did not converge"), String::npos) + << "the cap, not a deadline, is what ended this call: " << message; + /// One erase per iteration and one pause after each, so these agree exactly -- and both being + /// greater than one is what proves the loop iterated rather than failing on its first attempt. + EXPECT_EQ(backend->refusedAttempts(), clock.sleeps.size()); + EXPECT_GT(clock.sleeps.size(), 1u); + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{removing}); } TEST(CASRefCatalogRemoval, CancelStalledCreatingRequiresExactRowAndTerminalCreatorFence) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); const CatalogEntry creating{ .ns = RootNamespace{"a"}, .state = NsState::Creating, .incarnation = UInt128{7}, .creator = CreatorFence{.server_root_id = "server", .writer_epoch = 3, .fence_generation = 4}}; - ASSERT_EQ(backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {creating}})).outcome, - PutOutcome::Done); + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {creating}})); + const uint64_t writes_after_seed = backend->writes(layout.refCatalogKey()); EXPECT_EQ(CasRefCatalog::cancelStalledCreating( - backend, layout, creating, [](const CreatorFence &) { return false; }, 5, [](uint64_t) {}), + op, layout, creating, [](const CreatorFence &) { return false; }), CasRefCatalog::StalledCreatingCancelOutcome::CreatorFenceStillLive); - EXPECT_EQ(backend.casPutCount(layout.refCatalogKey()), 0); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_after_seed); CatalogEntry stale = creating; stale.creator->writer_epoch = 2; EXPECT_EQ(CasRefCatalog::cancelStalledCreating( - backend, layout, stale, [](const CreatorFence &) { return true; }, 5, [](uint64_t) {}), + op, layout, stale, [](const CreatorFence &) { return true; }), CasRefCatalog::StalledCreatingCancelOutcome::EntryChanged); - EXPECT_EQ(backend.casPutCount(layout.refCatalogKey()), 0); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_after_seed); EXPECT_EQ(CasRefCatalog::cancelStalledCreating( - backend, layout, creating, [](const CreatorFence &) { return true; }, 5, [](uint64_t) {}), + op, layout, creating, [](const CreatorFence &) { return true; }), CasRefCatalog::StalledCreatingCancelOutcome::Cancelled); - EXPECT_TRUE(CasRefCatalog::read(backend, layout).catalog.entries.empty()); - EXPECT_EQ(backend.casPutCount(layout.refCatalogKey()), 1); - EXPECT_EQ(backend.listTotal(), 0); - EXPECT_EQ(backend.deleteTotal(), 0); + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_after_seed + 1); + EXPECT_EQ(backend->listTotal(), 0u); + EXPECT_EQ(backend->deleteTotal(), 0u); } TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuildInputs) @@ -1458,7 +1709,7 @@ TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuil .removal_started_round = 8}, }; const CasRefCatalog::Snapshot cut{ - .catalog = catalog, .token = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary ordinary_scan; ordinary_scan.parent_ref_lives.emplace(UInt128{1}, RefLifeFoldState{ @@ -1587,7 +1838,9 @@ TEST(CASGCStuckRemoval, BoundaryAndAbsentVersusUnreadableMessagesAreExact) TEST(CASGCStuckRemoval, DiagnosticDoesNotAppendOrMutateBackend) { - DB::Cas::tests::CountingBackend backend; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout("p"); const RefWalkPlanRow row{ .life = NamespaceLifeId::fromCatalogEntry(RootNamespace{"removing"}, UInt128{7}), @@ -1597,21 +1850,21 @@ TEST(CASGCStuckRemoval, DiagnosticDoesNotAppendOrMutateBackend) .listed_hint = false, .checkpoint_observation = std::nullopt, .tail_observation = std::nullopt}; - const uint64_t puts_before = backend.putTotal(); - const uint64_t cas_before = backend.casPutTotal(); + const uint64_t writes_before = backend->writeTotal(); EXPECT_TRUE(stuckRemovalWarning(row, 11, 10, layout)); - EXPECT_EQ(backend.putTotal(), puts_before); - EXPECT_EQ(backend.casPutTotal(), cas_before); - EXPECT_EQ(backend.deleteTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), writes_before); + EXPECT_EQ(backend->deleteTotal(), 0u); } TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .gc_stuck_removal_rounds = 10}); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout & layout = store->layout(); const UInt128 gc_id{99}; const UInt128 life_id{7}; @@ -1621,11 +1874,11 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) .state = NsState::Removing, .incarnation = life_id, .removal_started_round = 1}; - const auto catalog = backend->get(layout.refCatalogKey()); + const auto catalog = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog); - ASSERT_EQ(backend->casPut( - layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}}), catalog->token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(op.replace( + layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}}), + catalog->incarnation, Retry::standard()))); CasFoldSeal seal; seal.generation = 1; @@ -1638,7 +1891,7 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) .retry_count = 0, .next_retry_round = 12}}}); seal.condemned_summary[0] = CondemnedSummary{}; - ASSERT_EQ(backend->putIfAbsent(layout.foldSealKey(1, 1), encodeFoldSeal(seal)).outcome, PutOutcome::Done); + seedObject(op, layout.foldSealKey(1, 1), encodeFoldSeal(seal)); GcState state; state.lease = GcLease{.owner = gc_id, .seq = 1}; @@ -1646,13 +1899,13 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) state.gc_shards = 1; state.snap_generation = 1; state.snap_attempt = 1; - ASSERT_EQ(backend->putIfAbsent(layout.gcStateKey(), encodeGcState(state)).outcome, PutOutcome::Done); + seedObject(op, layout.gcStateKey(), encodeGcState(state)); const uint64_t signals_before = ProfileEvents::global_counters[ProfileEvents::CASGCStuckRemovals].load(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(removing.ns, life_id); const String unreadable_ref_log_key = layout.refLogKey(life, RefTxnId{5, 6}); - const uint64_t append_puts_before = backend->putCount(unreadable_ref_log_key); + const uint64_t append_writes_before = backend->writes(unreadable_ref_log_key); ScopedCasGcLogCapture log_capture; Gc first_process(store, gc_id); EXPECT_TRUE(first_process.runRegularRound().acquired_lease); @@ -1660,7 +1913,7 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) EXPECT_TRUE(restarted_process.runRegularRound().acquired_lease); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASGCStuckRemovals].load() - signals_before, 2u); - EXPECT_EQ(backend->putCount(unreadable_ref_log_key), append_puts_before) + EXPECT_EQ(backend->writes(unreadable_ref_log_key), append_writes_before) << "the diagnostic cannot append the unreadable ref log"; const String captured = log_capture.captured(); EXPECT_EQ(std::count(captured.begin(), captured.end(), '\n'), 2u); @@ -1686,7 +1939,7 @@ TEST(CASGCRefWalkPlan, UnmatchedAdoptedParentLifeIsObservedWithoutEnteringThePla hexToU128("fedcba98765432100123456789abcdef"); RefCatalog catalog{.entries = {liveEntry("live", 2)}}; const CasRefCatalog::Snapshot cut{ - .catalog = catalog, .token = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary scan; scan.parent_ref_lives.emplace(current_life, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{2, 3}}}); @@ -1722,7 +1975,7 @@ TEST(CASGCRefPlan, RoundInputOwnsObservationsAndSuccessorStateCannotChangePlan) RefCatalog catalog; catalog.entries = {liveEntry("live", 2)}; CasRefCatalog::Snapshot cut{ - .catalog = catalog, .token = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary observations; observations.max_log_by_life.emplace(UInt128{2}, RefTxnId{2, 7}); diff --git a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp index 5954f9930f5a..f25a303371d8 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp @@ -9,6 +9,7 @@ #include #include +#include #include namespace DB::ErrorCodes @@ -46,44 +47,87 @@ namespace /// models a definite write failure, distinct from the acknowledgement-loss shape below: a retry must /// still be allowed to prove a new pool, and the failed first attempt must not have published /// `_pool_meta` without the catalog it makes mandatory. -class CatalogBootstrapPutFailsOnceBackend final : public CountingBackend +/// Per-key counts of the WRITE primitive. `CountingBackend` counts reads, heads and lists per key but +/// only totals for writes, and its legacy per-verb counters never see a caller that speaks the +/// primitives -- which every writer below does. +class WriteCountingBackend : public CountingBackend { public: - using CountingBackend::putIfAbsent; + uint64_t writes(const String & key) const + { + std::lock_guard lock(write_count_mutex); + const auto it = write_counts.find(key); + return it == write_counts.end() ? 0 : it->second; + } + + void resetWriteCounts() + { + std::lock_guard lock(write_count_mutex); + write_counts.clear(); + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(write_count_mutex); + ++write_counts[key]; + } + return CountingBackend::write(key, bytes, expected_value, access); + } + +private: + mutable std::mutex write_count_mutex; + std::map write_counts; +}; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override +/// Faults the mandatory catalog's very first bootstrap write before it reaches durable storage. This +/// models a definite write failure, distinct from the acknowledgement-loss shape below: a retry must +/// still be allowed to prove a new pool, and the failed first attempt must not have published +/// `_pool_meta` without the catalog it makes mandatory. +/// +/// A plain `std::runtime_error`, not a `Poco::Exception`: the engine's write loop treats any +/// `Poco`/transport exception as an ambiguity it settles itself with one resolve read, and a one-shot +/// fault of that class is retried and silently succeeds within the SAME `Pool::open` call -- it never +/// reaches the caller at all. A non-`Poco` `std::exception` is the engine's own signal for "this could +/// not have landed" and propagates unresolved, which is what "before it reaches durable storage" means. +class CatalogBootstrapWriteFailsOnceBackend final : public WriteCountingBackend +{ +public: + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (fail_once && key == Layout{"p"}.refCatalogKey()) { fail_once = false; - throw Poco::TimeoutException("CatalogBootstrapPutFailsOnceBackend: catalog PUT did not land"); + throw std::runtime_error("CatalogBootstrapWriteFailsOnceBackend: catalog write did not land"); } - return CountingBackend::putIfAbsent(key, bytes, meta); + return WriteCountingBackend::write(key, bytes, expected_value, access); } private: bool fail_once = true; }; -class CatalogCancellationRaceBackend final : public CountingBackend +class CatalogCancellationRaceBackend final : public WriteCountingBackend { public: - using CountingBackend::casPut; - - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (race_armed && key == Layout{"p"}.refCatalogKey()) { race_armed = false; - on_catalog_cas(); + on_catalog_write(); } - return CountingBackend::casPut(key, bytes, expected, meta); + return WriteCountingBackend::write(key, bytes, expected_value, access); } bool race_armed = false; - std::function on_catalog_cas; + std::function on_catalog_write; }; PoolPtr openPoolForBirthTest(const BackendPtr & backend, const String & server_root_id = "test") @@ -124,13 +168,15 @@ RefTxnId publishBirth(const PoolPtr & store, const RootNamespace & ns, const Str TEST(CASRefCatalogBirthWiring, FirstOpenMintsALiveCatalogEntryAndKeysTheBirthAtIt) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const RootNamespace ns{"srv1/birth_wiring"}; const RefTxnId id = publishBirth(store, ns, "a"); EXPECT_EQ(id, (RefTxnId{store->writerEpoch(), 1})); - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(*backend, store->layout()); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, store->layout()); const CatalogEntry * entry = findEntry(snap.catalog, ns); ASSERT_NE(entry, nullptr) << "the first open must mint a catalog entry"; EXPECT_EQ(entry->state, NsState::Live); @@ -138,103 +184,117 @@ TEST(CASRefCatalogBirthWiring, FirstOpenMintsALiveCatalogEntryAndKeysTheBirthAtI EXPECT_EQ(entry->creator, std::nullopt) << "creator is forbidden outside Creating (strict grammar)"; const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation); - EXPECT_TRUE(backend->head(store->layout().refLogKey(life, id)).exists) + EXPECT_TRUE(op.head(store->layout().refLogKey(life, id), Retry::standard()).has_value()) << "the birth transaction must be keyed at the REAL minted incarnation, not the Stage-A sentinel"; - EXPECT_FALSE(backend->head(store->layout().refLogKey(fixture::fixtureLife(ns), id)).exists) + EXPECT_FALSE(op.head(store->layout().refLogKey(fixture::fixtureLife(ns), id), Retry::standard()).has_value()) << "and must NOT be keyed at the sentinel any more"; } TEST(CASRefCatalogBirthWiring, CatalogLossAfterMountCannotRecreateAOneRowAuthority) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); publishBirth(store, RootNamespace{"srv1/existing"}, "old"); - const auto catalog = backend->get(layout.refCatalogKey()); + const auto catalog = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog); - ASSERT_EQ(backend->deleteExact(layout.refCatalogKey(), catalog->token).kind, - DeleteOutcome::Kind::Deleted); + ASSERT_EQ(op.remove(layout.refCatalogKey(), catalog->incarnation, Retry::standard()), Removal::Removed); backend->resetCounts(); + backend->resetWriteCounts(); EXPECT_THROW(publishBirth(store, RootNamespace{"srv1/new"}, "new"), DB::Exception); - EXPECT_FALSE(backend->head(layout.refCatalogKey()).exists) + EXPECT_FALSE(op.head(layout.refCatalogKey(), Retry::standard()).has_value()) << "runtime loss must not be repaired with a one-row replacement authority"; - EXPECT_EQ(backend->casPutTotal(), 0u); - EXPECT_EQ(backend->putTotal(), 0u) - << "the failed birth must not publish a checkpoint or ref-log body"; - EXPECT_EQ(backend->putOverwriteTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u) + << "the failed birth must not publish a catalog, a checkpoint or a ref-log body"; } TEST(CASRefCatalogBirthWiring, FailedCatalogBootstrapDoesNotPublishPoolMetaAndRetryConverges) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; EXPECT_ANY_THROW(Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"})); - EXPECT_FALSE(backend->head(layout.poolMetaKey()).exists) + EXPECT_FALSE(op.head(layout.poolMetaKey(), Retry::standard()).has_value()) << "a failed mandatory catalog bootstrap must leave no authoritative pool meta behind"; - EXPECT_FALSE(backend->head(layout.refCatalogKey()).exists); + EXPECT_FALSE(op.head(layout.refCatalogKey(), Retry::standard()).has_value()); PoolPtr retry; ASSERT_NO_THROW(retry = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"})); - EXPECT_TRUE(backend->head(layout.poolMetaKey()).exists); - EXPECT_TRUE(backend->head(layout.refCatalogKey()).exists); + EXPECT_TRUE(op.head(layout.poolMetaKey(), Retry::standard()).has_value()); + EXPECT_TRUE(op.head(layout.refCatalogKey(), Retry::standard()).has_value()); } -TEST(CASRefCatalogBirthWiring, LostCatalogBootstrapAcknowledgementLeavesOnlyRetryableCatalogResidue) +TEST(CASRefCatalogBirthWiring, LostCatalogBootstrapAcknowledgementResolvesToCommittedWithoutARetryOrADuplicate) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; backend->key_substr = layout.refCatalogKey(); - EXPECT_ANY_THROW(Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"})); - EXPECT_FALSE(backend->head(layout.poolMetaKey()).exists); - EXPECT_TRUE(backend->head(layout.refCatalogKey()).exists) - << "the injected write must land before its acknowledgement is lost"; - - PoolPtr retry; - ASSERT_NO_THROW(retry = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"})); - EXPECT_TRUE(backend->head(layout.poolMetaKey()).exists); + /// The catalog create's own write landed; only its response was lost. `LandedButAckLostOnceBackend` + /// is documented to model exactly that -- a caller that resolves the ambiguity meets its OWN + /// earlier write as the occupant -- so the engine's one resolve read proves this attempt committed. + /// The whole bootstrap therefore converges in this SINGLE `Pool::open` call: no throw, no second + /// catalog write, and no second `Pool::open` needed. + PoolPtr store; + ASSERT_NO_THROW(store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"})); + EXPECT_TRUE(op.head(layout.poolMetaKey(), Retry::standard()).has_value()); + EXPECT_TRUE(op.head(layout.refCatalogKey(), Retry::standard()).has_value()); + EXPECT_EQ(backend->putCount(layout.refCatalogKey()), 1u) + << "a landed write whose ack is lost must be proven by a read, never repeated"; } TEST(CASRefCatalogBirthWiring, BootstrapConflictExactReadsTheCanonicalEmptyCatalog) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const String canonical_empty = encodeRefCatalog(RefCatalog{}); - ASSERT_EQ(backend->putIfAbsent(layout.refCatalogKey(), canonical_empty).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCatalogKey(), canonical_empty, Retry::standard()))); backend->resetCounts(); + backend->resetWriteCounts(); - const CasRefCatalog::Snapshot snap = CasRefCatalog::initializeEmptyForNewPool(*backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::initializeEmptyForNewPool(op, layout); EXPECT_TRUE(snap.catalog.entries.empty()); - EXPECT_EQ(backend->putCount(layout.refCatalogKey()), 1u); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), 1u); EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u) << "a concurrent bootstrap winner must be exact-read before acceptance"; } TEST(CASRefCatalogBirthWiring, BootstrapConflictRefusesANonemptyCatalog) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const RefCatalog nonempty{.entries = {CatalogEntry{ .ns = RootNamespace{"test/nonempty"}, .state = NsState::Live, .incarnation = UInt128{1}, .creator = std::nullopt}}}; - ASSERT_EQ(backend->putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(nonempty)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCatalogKey(), encodeRefCatalog(nonempty), Retry::standard()))); backend->resetCounts(); + backend->resetWriteCounts(); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { CasRefCatalog::initializeEmptyForNewPool(*backend, layout); }); + [&] { CasRefCatalog::initializeEmptyForNewPool(op, layout); }); EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); } TEST(CASRefCatalogBirthWiring, ExistingPoolMetaWithMissingCatalogStillFailsClosed) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); PoolPtr first = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); - const auto catalog = backend->get(first->layout().refCatalogKey()); + const auto catalog = op.read(first->layout().refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog); - ASSERT_EQ(backend->deleteExact(first->layout().refCatalogKey(), catalog->token).kind, - DeleteOutcome::Kind::Deleted); + ASSERT_EQ(op.remove(first->layout().refCatalogKey(), catalog->incarnation, Retry::standard()), Removal::Removed); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); }); @@ -242,24 +302,27 @@ TEST(CASRefCatalogBirthWiring, ExistingPoolMetaWithMissingCatalogStillFailsClose TEST(CASRefCatalogBirthWiring, RestartFixturePreservesItsExistingNonemptyCatalog) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; seedPoolMetaForRestart(*backend); - const auto empty = backend->get(layout.refCatalogKey()); + const auto empty = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(empty); const RefCatalog nonempty{.entries = {CatalogEntry{ .ns = RootNamespace{"test/preserved"}, .state = NsState::Live, .incarnation = UInt128{1}, .creator = std::nullopt}}}; const String bytes = encodeRefCatalog(nonempty); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), bytes, empty->token).outcome, PutOutcome::Done); - const auto before = backend->get(layout.refCatalogKey()); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.refCatalogKey(), bytes, empty->incarnation, Retry::standard()))); + const auto before = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(before); seedPoolMetaForRestart(*backend); - const auto after = backend->get(layout.refCatalogKey()); + const auto after = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(after); EXPECT_EQ(after->bytes, before->bytes); - EXPECT_EQ(after->token, before->token); + EXPECT_EQ(after->incarnation, before->incarnation); } /// A namespace whose catalog entry is ALREADY `Live` (e.g. admitted by an earlier mount that this @@ -269,13 +332,15 @@ TEST(CASRefCatalogBirthWiring, RestartFixturePreservesItsExistingNonemptyCatalog TEST(CASRefCatalogBirthWiring, AnExistingLiveEntryIsAdoptedRatherThanReminted) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/adopt_live"}; const CatalogEntry entry{.ns = ns, .state = NsState::Live, .incarnation = UInt128(0xcafe), .creator = std::nullopt}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); DB::Cas::tests::writeRecoverableCkptForRawFixture(*backend, layout, ns, RefCkpt{ .life_epoch = store->writerEpoch(), .committed_through = std::nullopt, @@ -287,14 +352,14 @@ TEST(CASRefCatalogBirthWiring, AnExistingLiveEntryIsAdoptedRatherThanReminted) EXPECT_EQ(id, (RefTxnId{store->writerEpoch(), 1})); /// The read result must outlive the returned pointer -- findEntry points into its entries. - const auto after_cut = CasRefCatalog::read(*backend, layout); + const auto after_cut = CasRefCatalog::read(op, layout); const CatalogEntry * after = findEntry(after_cut.catalog, ns); ASSERT_NE(after, nullptr); EXPECT_EQ(after->incarnation, UInt128(0xcafe)) << "adopted, not re-minted"; EXPECT_EQ(after->state, NsState::Live); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(ns, UInt128(0xcafe)); - EXPECT_TRUE(backend->head(layout.refLogKey(life, id)).exists); + EXPECT_TRUE(op.head(layout.refLogKey(life, id), Retry::standard()).has_value()); } /// OBLIGATION 3, pinned through the PRODUCTION path: a `Creating` entry left by a DIFFERENT, still-live @@ -302,7 +367,9 @@ TEST(CASRefCatalogBirthWiring, AnExistingLiveEntryIsAdoptedRatherThanReminted) /// `resolveNamespaceLife`/`reconcileStaleCreator`, just an ordinary `appendRefOps`. TEST(CASRefCatalogBirthWiring, ANamespaceStuckCreatingUnderALiveForeignFenceRefusesProductionPublicationByConstruction) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/stuck_creating"}; @@ -313,20 +380,19 @@ TEST(CASRefCatalogBirthWiring, ANamespaceStuckCreatingUnderALiveForeignFenceRefu const CreatorFence foreign_creator{.server_root_id = "ghost-server", .writer_epoch = 9, .fence_generation = 1}; const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(0xdead), .creator = foreign_creator}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); backend->resetCounts(); + backend->resetWriteCounts(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishBirth(store, ns, "a"); }); /// Nothing was written: the entry is exactly as observed, still Creating, still the foreign fence. /// The read result must outlive the returned pointer -- findEntry points into its entries. - const auto still_cut = CasRefCatalog::read(*backend, layout); + const auto still_cut = CasRefCatalog::read(op, layout); const CatalogEntry * still = findEntry(still_cut.catalog, ns); ASSERT_NE(still, nullptr); EXPECT_EQ(*still, entry) << "a refused resolution must write nothing"; - EXPECT_EQ(backend->putTotal(), 0u); - EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); } /// The mirror image, and Task 3's own deferred obligation ("wire `reconcileStaleCreator` and pin it @@ -336,6 +402,8 @@ TEST(CASRefCatalogBirthWiring, ANamespaceStuckCreatingUnderALiveForeignFenceRefu TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsReconciledThroughTheProductionPath) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend, "this-server"); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/reconciled"}; @@ -345,7 +413,7 @@ TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsR const CreatorFence dead_creator{.server_root_id = "dead-server", .writer_epoch = 3, .fence_generation = 1}; const CatalogEntry entry{.ns = ns, .state = NsState::Creating, .incarnation = UInt128(0xbeef), .creator = dead_creator}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, entry); + CasRefCatalog::casAdmitEntry(op, layout, 1, entry); setWatermarkMinActive(*backend, layout, "dead-server", /*writer_epoch=*/3, /*min_active_build_sequence=*/std::numeric_limits::max()); @@ -355,7 +423,7 @@ TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsR EXPECT_EQ(id, (RefTxnId{store->writerEpoch(), 1})); /// The read result must outlive the returned pointer -- findEntry points into its entries. - const auto live_cut = CasRefCatalog::read(*backend, layout); + const auto live_cut = CasRefCatalog::read(op, layout); const CatalogEntry * live = findEntry(live_cut.catalog, ns); ASSERT_NE(live, nullptr); EXPECT_EQ(live->state, NsState::Live); @@ -363,12 +431,14 @@ TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsR EXPECT_EQ(live->creator, std::nullopt); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(ns, UInt128(0xbeef)); - EXPECT_TRUE(backend->head(layout.refLogKey(life, id)).exists); + EXPECT_TRUE(op.head(layout.refLogKey(life, id), Retry::standard()).has_value()); } TEST(CASRefCatalogBirthWiring, DropRefusesLiveCreatingFenceWithZeroCatalogMutation) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"drop_live_creator"}; @@ -377,20 +447,21 @@ TEST(CASRefCatalogBirthWiring, DropRefusesLiveCreatingFenceWithZeroCatalogMutati .state = NsState::Creating, .incarnation = UInt128{0xd001}, .creator = CreatorFence{.server_root_id = "unproven-live", .writer_epoch = 7, .fence_generation = 1}}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, creating); + CasRefCatalog::casAdmitEntry(op, layout, 1, creating); backend->resetCounts(); + backend->resetWriteCounts(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); - EXPECT_EQ(backend->putTotal(), 0u); - EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_EQ(CasRefCatalog::read(*backend, layout).catalog.entries, std::vector{creating}); + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{creating}); } TEST(CASRefCatalogBirthWiring, DropDeletesTerminalCreatingExactlyAndLeavesCkptForJanitor) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"drop_terminal_creator"}; @@ -399,19 +470,19 @@ TEST(CASRefCatalogBirthWiring, DropDeletesTerminalCreatingExactlyAndLeavesCkptFo .state = NsState::Creating, .incarnation = UInt128{0xd002}, .creator = CreatorFence{.server_root_id = "dead-creator", .writer_epoch = 8, .fence_generation = 1}}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, creating); + CasRefCatalog::casAdmitEntry(op, layout, 1, creating); setWatermarkMinActive(*backend, layout, "dead-creator", 8, std::numeric_limits::max()); const NamespaceLifeId old_life = NamespaceLifeId::fromCatalogEntry(ns, creating.incarnation); const String ckpt_key = layout.refCkptKey(old_life); - ASSERT_EQ(backend->putIfAbsent(ckpt_key, "stalled-ckpt").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(ckpt_key, "stalled-ckpt", Retry::standard()))); backend->resetCounts(); + backend->resetWriteCounts(); store->dropNamespace(ns); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 1u); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), 1u); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_EQ(backend->deleteCount(ckpt_key), 0u); - EXPECT_TRUE(backend->head(ckpt_key).exists); - EXPECT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()); + EXPECT_TRUE(op.head(ckpt_key, Retry::standard()).has_value()); + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); const NamespaceLifeId reborn = store->namespaceLife(ns); EXPECT_NE(reborn.incarnation, old_life.incarnation); @@ -420,6 +491,8 @@ TEST(CASRefCatalogBirthWiring, DropDeletesTerminalCreatingExactlyAndLeavesCkptFo TEST(CASRefCatalogBirthWiring, DropLosesExactCreatingRaceToReconciliationWithoutDeletingCkpt) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"drop_reconcile_race"}; @@ -427,29 +500,29 @@ TEST(CASRefCatalogBirthWiring, DropLosesExactCreatingRaceToReconciliationWithout .server_root_id = "dead-racing-creator", .writer_epoch = 9, .fence_generation = 1}; const CatalogEntry creating{ .ns = ns, .state = NsState::Creating, .incarnation = UInt128{0xd003}, .creator = old_creator}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, creating); + CasRefCatalog::casAdmitEntry(op, layout, 1, creating); setWatermarkMinActive( *backend, layout, old_creator.server_root_id, old_creator.writer_epoch, std::numeric_limits::max()); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(ns, creating.incarnation); const String ckpt_key = layout.refCkptKey(life); - ASSERT_EQ(backend->putIfAbsent(ckpt_key, "stalled-ckpt").outcome, PutOutcome::Done); - backend->on_catalog_cas = [&] + ASSERT_TRUE(std::holds_alternative(op.create(ckpt_key, "stalled-ckpt", Retry::standard()))); + backend->on_catalog_write = [&] { EXPECT_EQ(CasRefCatalog::reconcileStaleCreator( - *backend, layout, creating, + op, layout, creating, CreatorFence{.server_root_id = "replacement", .writer_epoch = 10, .fence_generation = 1}, - [](const CreatorFence &) { return true; }, store->fenceGeneration(), - [](uint64_t) {}), CasRefCatalog::ReconcileCreatorOutcome::Reconciled); + [](const CreatorFence &) { return true; }), + CasRefCatalog::ReconcileCreatorOutcome::Reconciled); }; backend->race_armed = true; backend->resetCounts(); + backend->resetWriteCounts(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_EQ(backend->deleteCount(ckpt_key), 0u); - EXPECT_TRUE(backend->head(ckpt_key).exists); - const CasRefCatalog::Snapshot after = CasRefCatalog::read(*backend, layout); + EXPECT_TRUE(op.head(ckpt_key, Retry::standard()).has_value()); + const CasRefCatalog::Snapshot after = CasRefCatalog::read(op, layout); ASSERT_EQ(after.catalog.entries.size(), 1u); ASSERT_TRUE(after.catalog.entries.front().creator); EXPECT_EQ(after.catalog.entries.front().creator->server_root_id, "replacement"); @@ -458,6 +531,8 @@ TEST(CASRefCatalogBirthWiring, DropLosesExactCreatingRaceToReconciliationWithout TEST(CASRefCatalogBirthWiring, FencedDropCannotCancelTerminalCreating) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"fenced_drop_terminal_creator"}; @@ -466,29 +541,32 @@ TEST(CASRefCatalogBirthWiring, FencedDropCannotCancelTerminalCreating) .state = NsState::Creating, .incarnation = UInt128{0xd004}, .creator = CreatorFence{.server_root_id = "dead-fenced-creator", .writer_epoch = 11, .fence_generation = 1}}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, creating); + CasRefCatalog::casAdmitEntry(op, layout, 1, creating); setWatermarkMinActive(*backend, layout, "dead-fenced-creator", 11, std::numeric_limits::max()); backend->resetCounts(); + backend->resetWriteCounts(); /// The first cancellation attempt passes its fence check, then loses its catalog CAS while the /// local mount is re-armed at a new fence generation. The retry must re-check the caller fence and /// refuse before another catalog mutation attempt. - backend->on_catalog_cas = [&] + backend->on_catalog_write = [&] { rearmMountFenceAfterAnomalyForTest(store); - backend->failNextCasPut(layout.refCatalogKey()); + backend->refuseNextWrite(layout.refCatalogKey()); }; backend->race_armed = true; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 1u); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), 1u); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_EQ(CasRefCatalog::read(*backend, layout).catalog.entries, std::vector{creating}); + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{creating}); } TEST(CASRefCatalogBirthWiring, ExactOldLifeCannotCancelReplacementTerminalCreating) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); auto store = openPoolForBirthTest(backend); const Layout & layout = store->layout(); const RootNamespace ns{"exact_old_life_terminal_creator"}; @@ -498,15 +576,15 @@ TEST(CASRefCatalogBirthWiring, ExactOldLifeCannotCancelReplacementTerminalCreati .state = NsState::Creating, .incarnation = UInt128{0xd006}, .creator = CreatorFence{.server_root_id = "dead-successor-creator", .writer_epoch = 12, .fence_generation = 1}}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, successor); + CasRefCatalog::casAdmitEntry(op, layout, 1, successor); setWatermarkMinActive(*backend, layout, "dead-successor-creator", 12, std::numeric_limits::max()); const String ckpt_key = layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(ns, successor.incarnation)); - ASSERT_EQ(backend->putIfAbsent(ckpt_key, "successor-ckpt").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(ckpt_key, "successor-ckpt", Retry::standard()))); backend->resetCounts(); + backend->resetWriteCounts(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(predecessor); }); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_EQ(backend->deleteCount(ckpt_key), 0u); - EXPECT_EQ(CasRefCatalog::read(*backend, layout).catalog.entries, std::vector{successor}); + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{successor}); } diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index 1099764d5500..bcb6861cba31 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -431,7 +431,9 @@ TEST(CASRefWriterChunkedFlush, DropNamespaceOverOpCapSucceeds) .checkpoint_snapshot_id = RefTxnId{epoch, 1}, .last_epoch_seal = std::nullopt, }); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, layout, ns).value(); + CasRequests catalog_requests(backend, Fence::open()); + CasOperation catalog_op = catalog_requests.admit(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns).value(); backend->resetCounts(); ASSERT_EQ(store->listRefs(ns).size(), kTotalRefs); EXPECT_EQ(backend->getCount(layout.refLogKey(life, RefTxnId{epoch, 1})), 1u); @@ -440,7 +442,7 @@ TEST(CASRefWriterChunkedFlush, DropNamespaceOverOpCapSucceeds) DropNamespaceStats stats; EXPECT_NO_THROW(stats = store->dropNamespace(ns)); EXPECT_EQ(stats.committed_refs, kTotalRefs); - EXPECT_EQ(CasRefCatalog::read(*backend, layout).catalog.entries.front().state, NsState::Removing); + EXPECT_EQ(CasRefCatalog::read(catalog_op, layout).catalog.entries.front().state, NsState::Removing); } /// Test 11, second leg: `WholeShard` scope ALONE is not the removal-class discriminator -- the @@ -712,28 +714,117 @@ namespace /// (the leader's own item), chunk 2 = {item_b}. `mode` faults ONLY chunk 2's `_log/` PUT (skip chunk 1). /// In every variant chunk 1 commits and the leader's own call returns chunk 1's real id, while chunk 2's /// caller fails. Returns the two callers' results plus chunk 1's id for the per-variant assertions. +/// The engine reissues an unresolved write until its OWN retry window closes, and that window is +/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine +/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline +/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling +/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the +/// closures own the clock, so it outlives everything that can still read it. +class VirtualRetryClock +{ +public: + static std::shared_ptr installOn(const PoolPtr & store) + { + auto clock = std::make_shared(); + store->setCasRequestNowFnForTest([clock] { return clock->nowMs(); }); + store->setCasRetrySleepForTest([clock](uint64_t ms) { clock->advance(ms); }); + return clock; + } + + uint64_t nowMs() const + { + std::lock_guard lock(mutex); + return now_ms; + } + size_t pauseCount() const + { + std::lock_guard lock(mutex); + return pauses; + } + uint64_t longestPause() const + { + std::lock_guard lock(mutex); + return longest_pause; + } + + void advance(uint64_t ms) + { + std::lock_guard lock(mutex); + /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move + /// would leave the loop reissuing for ever against a fault that never clears. + now_ms += ms + 1; + ++pauses; + longest_pause = std::max(longest_pause, ms); + } + +private: + mutable std::mutex mutex; + uint64_t now_ms = 0; + size_t pauses = 0; + uint64_t longest_pause = 0; +}; + +/// `ChunkFaultBackend` COUNTS its faults, and a count can no longer make one conclusive: the write +/// engine settles every ambiguity by an exact read and then REISSUES, so a fault that runs out +/// mid-call is answered by the next attempt instead of by the call's own deadline. This keeps it armed +/// until the latch is cleared, on both legs -- the write's, and the lost read `Mode::LandedThenLost` +/// arms, which the read engine would otherwise simply reissue past. +class LatchedChunkFaultBackend : public ChunkFaultBackend +{ +public: + bool latched = false; + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + if (latched && !fail_read_once_key.empty() && key == fail_read_once_key) + throw Poco::TimeoutException("LatchedChunkFaultBackend: the lost read stays lost"); + return ChunkFaultBackend::read(key, access); + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + if (latched && mode != Mode::None && fault_skip == 0 && !expected_value && !fault_substr.empty() + && key.find(fault_substr) != String::npos) + fault_count = 1; + return ChunkFaultBackend::write(key, bytes, expected_value, access); + } + + void disarm() + { + latched = false; + mode = Mode::None; + fault_count = 0; + fault_skip = 0; + fail_read_once_key.clear(); + } +}; + struct ChunkFailureOutcome { AppendResult leader; /// item_a, chunk 1 AppendResult follower; /// item_b, chunk 2 RefTxnId chunk1_id{}; - std::shared_ptr backend; + std::shared_ptr backend; PoolPtr store; + std::shared_ptr clock; }; ChunkFailureOutcome runChunkFailureCase(const String & ns_suffix, ChunkFaultBackend::Mode mode) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); PoolConfig cfg; - /// Single-attempt budget: one ambiguous PUT is a conclusive Unresolved (wedge) / DefiniteFailure, - /// with no inter-attempt sleep to serve. + /// The budget bounds the mount lease's own admission arithmetic and nothing else -- a write's + /// attempt count is the `Retry` policy's. What makes the injected fault conclusive is that it + /// stays armed for the whole call while the injected clock carries the call to its own deadline. CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) + budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; auto store = openPoolWith(backend, cfg); + auto clock = VirtualRetryClock::installOn(store); const DB::Cas::Layout & layout = store->layout(); const RootNamespace ns{String("srv1/") + ns_suffix}; publishEmptyPart(store, ns, "seed"); @@ -744,6 +835,7 @@ ChunkFailureOutcome runChunkFailureCase(const String & ns_suffix, ChunkFaultBack backend->mode = mode; backend->fault_skip = 1; backend->fault_count = 1; + backend->latched = true; auto sync = std::make_shared(); armPreCarveBlock(store, ns, sync, 2); @@ -762,19 +854,23 @@ ChunkFailureOutcome runChunkFailureCase(const String & ns_suffix, ChunkFaultBack out.follower = b.fut.get(); a.t.join(); b.t.join(); + /// Disarmed before anything else touches the store: a topped-up count would fault the pool's own + /// teardown writes too. + backend->disarm(); store->setRefPreCarveHookForTest(nullptr); out.chunk1_id = out.leader.id; out.backend = backend; out.store = store; + out.clock = clock; return out; } } -/// Test 9 (chunk-failure variant a -- definite failure): chunk 2's PUT is conclusively rejected -/// (`CasWriteOutcome::DefiniteFailure`). Chunk 1's caller (the leader's own item) observes SUCCESS with -/// chunk 1's real id; chunk 2's caller fails; the lane does NOT wedge (a definite rejection is a safe -/// gap, not an uncertain PUT). +/// Test 9 (chunk-failure variant a -- definite failure): chunk 2's create is conclusively rejected by +/// the store. Chunk 1's caller (the leader's own item) observes SUCCESS with chunk 1's real id; chunk +/// 2's caller fails; the lane does NOT wedge (a proven refusal is a safe gap, not an uncertain +/// write). TEST(CASRefWriterChunkedFlush, ChunkFailureDefinite) { #if !USE_AWS_S3 @@ -803,6 +899,11 @@ TEST(CASRefWriterChunkedFlush, ChunkFailureWedge) ChunkFailureOutcome out = runChunkFailureCase("chunk_fail_wedge", ChunkFaultBackend::Mode::Unresolved); ASSERT_TRUE(out.leader.err == nullptr) << "chunk-1 caller must observe success even though chunk 2 wedged"; ASSERT_TRUE(out.follower.err != nullptr) << "chunk-2 caller must observe the append failure"; + /// The give-up was chunk 2's OWN retry window: the fault outlasted several reissues, and every one + /// of them paced through the injected sleep rather than a real one. + EXPECT_GT(out.clock->pauseCount(), 1u); + EXPECT_LE(out.clock->longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; + EXPECT_GE(out.clock->nowMs(), 60000u); EXPECT_TRUE(out.store->refLaneWedgedForTest(ns)) << "chunk 2's unresolved PUT must wedge the lane"; RefTxnId chunk2_id = out.chunk1_id; diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index 7582d9670355..60a219eb5e00 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -20,8 +20,11 @@ #include #include +#include #include +#include #include +#include #include #include #include @@ -86,23 +89,13 @@ RefTxnId publishRef(const PoolPtr & store, const RootNamespace & ns, const Strin RootMutationOrigin::Writer, RootMutationKind::Publish); } -/// A fence that never refuses, for the tests whose subject is not the fence. -const std::function ALWAYS_ADMITTED = [](uint64_t) {}; - -/// A deadline far enough out that only the test's own contention decides the outcome. The clock is -/// frozen (a constant `now`), which is what makes every non-exhaustion test independent of wall time. -CkptDeadline generousDeadline() -{ - return CkptDeadline{[] { return uint64_t{1000}; }, 60000}; -} - /// Reads `life`'s `_ckpt` and returns its body, or a default-constructed one after failing the /// current test when the object is absent. Every assertion below goes through this rather than /// dereferencing the optional directly: a bare `->` on a disengaged optional ABORTS the whole test /// binary, so one regression would take every later suite's result with it instead of failing a test. -RefCkpt readCkptOrFail(Backend & backend, const Layout & layout, const NamespaceLifeId & life) +RefCkpt readCkptOrFail(CasOperation & op, const Layout & layout, const NamespaceLifeId & life) { - const std::optional sample = readCkpt(backend, layout, life); + const std::optional sample = readCkpt(op, layout, life); if (!sample) { ADD_FAILURE() << "expected a _ckpt for namespace '" << life.ns.string() << "', found none"; @@ -118,9 +111,9 @@ RefCkpt readCkptOrFail(Backend & backend, const Layout & layout, const Namespace /// incarnation it minted rather than assume the sentinel. Fails the current test (rather than /// dereferencing a disengaged optional) if the catalog carries no entry for `ns` -- e.g. called before /// the namespace's first append. -NamespaceLifeId liveLifeOrFail(Backend & backend, const Layout & layout, const RootNamespace & ns) +NamespaceLifeId liveLifeOrFail(CasOperation & op, const Layout & layout, const RootNamespace & ns) { - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); for (const CatalogEntry & entry : snap.catalog.entries) if (entry.ns.string() == ns.string()) return NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); @@ -130,47 +123,53 @@ NamespaceLifeId liveLifeOrFail(Backend & backend, const Layout & layout, const R /// Replaces the whole body of one key, minting a new incarnation -- how a test installs a deliberately /// malformed or concurrently-advanced object. -void overwriteObject(Backend & backend, const String & key, const String & bytes) +void overwriteObject(CasOperation & op, const String & key, const String & bytes) { - const HeadResult h = backend.head(key); - ASSERT_TRUE(h.exists) << "overwriteObject expects " << key << " to exist"; - ASSERT_EQ(backend.putOverwrite(key, bytes, h.token).outcome, PutOutcome::Done); + const WriteResult result = op.readModifyWrite(key, + [&bytes](const std::optional & current) -> std::optional + { + EXPECT_TRUE(current.has_value()) << "overwriteObject expects the key to exist"; + return bytes; + }, + Retry::standard()); + ASSERT_TRUE(std::holds_alternative(result)); } -/// Runs `on_get` right after every `get` of `watched_key` -- the deterministic way to act inside -/// another component's read-then-write window without a sleep or a second thread. The hook is a public -/// member rather than a constructor argument so it can be installed AFTER the backend exists (every -/// interesting hook writes through that same backend) and only once the test's setup writes are done. -class GetHookBackend : public CountingBackend +/// Per-key counts of the WRITE primitive. `CountingBackend` counts reads, heads and lists per key but +/// only totals for writes, and its legacy per-verb counters never see a caller that speaks the +/// primitives -- which every writer below does. +class WriteCountingBackend : public CountingBackend { public: - using CountingBackend::get; - - explicit GetHookBackend(String watched_key_) : watched_key(std::move(watched_key_)) {} - - /// Stage B (Task 4-C): a test that must watch a namespace's `_ckpt` key can no longer compute it - /// before the pool exists -- the real incarnation is minted only once the namespace's first open - /// resolves it, which requires the pool (and so this backend) to already be constructed. Lets a - /// test retarget the watch once it has learned the real key, strictly before arming `on_get`. - void setWatchedKey(String watched_key_) { watched_key = std::move(watched_key_); } - - std::function on_get; + uint64_t writes(const String & key) const + { + std::lock_guard lock(write_count_mutex); + const auto it = write_counts.find(key); + return it == write_counts.end() ? 0 : it->second; + } - std::optional get(const String & key, Range range) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - auto result = CountingBackend::get(key, range); - if (key == watched_key && on_get) - on_get(); - return result; + { + std::lock_guard lock(write_count_mutex); + ++write_counts[key]; + } + return CountingBackend::write(key, bytes, expected_value, access); } private: - String watched_key; + mutable std::mutex write_count_mutex; + std::map write_counts; }; -/// Records the exact `_ckpt` recovery protocol and injects an ambiguous CAS response. The fault is -/// armed only after fixture setup, so the journal contains solely the operation under test. -class AmbiguousCkptBackend : public CountingBackend +/// The two PRIMITIVES `publishCkpt` speaks, instrumented: the exact request sequence against one +/// watched key, and the two shapes an ambiguous response has -- the store applied the write and then +/// lost the answer, or it never applied it. Hooks are public members rather than constructor arguments +/// so they can be installed AFTER the backend exists (every interesting hook writes through that same +/// backend) and only once the test's setup writes are done. +class CkptProbeBackend : public WriteCountingBackend { public: enum class Fault : uint8_t @@ -181,72 +180,76 @@ class AmbiguousCkptBackend : public CountingBackend AlwaysThrowWithoutCommit, }; - using CountingBackend::casPut; - using CountingBackend::get; - String watched_key; Fault fault = Fault::None; + /// Written over this call's own committed attempt, so the resolve read finds a WINNER rather than + /// the bytes the attempt sent. String dominating_bytes; - bool fail_resolution_get = false; - std::function after_ambiguous_cas; - std::function before_resolution_get; - std::function after_resolution_get; + bool fail_reads_after_the_first = false; + std::function after_write; + std::function after_read; std::vector journal; + /// A test that must watch a namespace's `_ckpt` key cannot compute it before the pool exists -- + /// the real incarnation is minted only once the namespace's first open resolves it. So the watch + /// is retargeted once the test has learned the real key, strictly before arming any hook. + void watch(String key) + { + watched_key = std::move(key); + watched_reads = 0; + journal.clear(); + } + void arm(const String & key, Fault fault_) { - watched_key = key; + watch(key); fault = fault_; - watched_get_count = 0; - journal.clear(); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (key != watched_key) - return CountingBackend::get(key, range); - - journal.push_back("GET"); - ++watched_get_count; - if (watched_get_count >= 2 && before_resolution_get) - before_resolution_get(); - if (watched_get_count == 2 && fail_resolution_get) - throw Poco::TimeoutException("AmbiguousCkptBackend: exact-read response lost"); - auto result = CountingBackend::get(key, range); - if (watched_get_count >= 2 && after_resolution_get) - after_resolution_get(); + return WriteCountingBackend::read(key, access); + + journal.push_back("READ"); + ++watched_reads; + if (watched_reads >= 2 && fail_reads_after_the_first) + throw Poco::TimeoutException("CkptProbeBackend: read response lost"); + auto result = WriteCountingBackend::read(key, access); + if (after_read) + after_read(); return result; } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (key != watched_key) - return CountingBackend::casPut(key, bytes, expected, meta); + return WriteCountingBackend::write(key, bytes, expected_value, access); - journal.push_back("CAS"); - if (fault == Fault::None) - return CountingBackend::casPut(key, bytes, expected, meta); + journal.push_back("WRITE"); const Fault this_fault = fault; if (fault != Fault::AlwaysThrowWithoutCommit) fault = Fault::None; + if (this_fault == Fault::None) + return WriteCountingBackend::write(key, bytes, expected_value, access); + if (this_fault == Fault::CommitThenThrow) { - const CasResult result = CountingBackend::casPut(key, bytes, expected, meta); - if (result.outcome == CasOutcome::Committed && !dominating_bytes.empty()) - { - const HeadResult head_result = CountingBackend::head(key); - EXPECT_EQ(CountingBackend::putOverwrite(key, dominating_bytes, head_result.token).outcome, - PutOutcome::Done); - } + const auto committed = WriteCountingBackend::write(key, bytes, expected_value, access); + /// The winner's replacement is not journalled: it is not an attempt of the call under test. + if (committed.has_value() && !dominating_bytes.empty()) + EXPECT_TRUE(WriteCountingBackend::write(key, dominating_bytes, + std::optional{*committed}, access).has_value()); } - if (after_ambiguous_cas) - after_ambiguous_cas(); - throw Poco::TimeoutException("AmbiguousCkptBackend: CAS response lost"); + if (after_write) + after_write(); + throw Poco::TimeoutException("CkptProbeBackend: write response lost"); } private: - size_t watched_get_count = 0; + size_t watched_reads = 0; }; } @@ -562,14 +565,15 @@ TEST(CASRefCheckpoint, MergeTakesThePerFieldSemanticMaximum) TEST(CASRefCheckpoint, CreatesTheObjectWhenItIsAbsent) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const RootNamespace ns{"srv1/ckpt_create"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(ns); const RefCkpt birth{.life_epoch = std::optional{5}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - EXPECT_EQ(publishCkpt(*backend, layout, life, birth, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const auto sample = readCkpt(*backend, layout, life); + EXPECT_EQ(publishCkpt(op, layout, life, birth), CkptPublishOutcome::Published); + const auto sample = readCkpt(op, layout, life); ASSERT_TRUE(sample.has_value()); EXPECT_EQ(sample->ckpt, birth); } @@ -582,23 +586,23 @@ TEST(CASRefCheckpoint, CreatesTheObjectWhenItIsAbsent) TEST(CASRefCheckpoint, EachWriterCreatesWithOnlyWhatItKnowsAndTheOtherFieldsMergeInLater) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const RootNamespace ns{"srv1/ckpt_partial_create"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(ns); const RefCkpt publisher{.life_epoch = std::nullopt, .committed_through = ID_1_1, .checkpoint_snapshot_id = ID_1_1, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(publishCkpt(*backend, layout, life, publisher, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const auto created = readCkpt(*backend, layout, life); + ASSERT_EQ(publishCkpt(op, layout, life, publisher), CkptPublishOutcome::Published); + const auto created = readCkpt(op, layout, life); ASSERT_TRUE(created.has_value()); EXPECT_EQ(created->ckpt.checkpoint_snapshot_id, ID_1_1); EXPECT_FALSE(created->ckpt.life_epoch.has_value()) << "the publisher must not invent a genesis epoch"; const RefCkpt birth{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(publishCkpt(*backend, layout, life, birth, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const auto completed = readCkpt(*backend, layout, life); + ASSERT_EQ(publishCkpt(op, layout, life, birth), CkptPublishOutcome::Published); + const auto completed = readCkpt(op, layout, life); ASSERT_TRUE(completed.has_value()); EXPECT_EQ(completed->ckpt.life_epoch, 1u); EXPECT_EQ(completed->ckpt.checkpoint_snapshot_id, ID_1_1) << "and must not lose the checkpoint on the way in"; @@ -606,8 +610,8 @@ TEST(CASRefCheckpoint, EachWriterCreatesWithOnlyWhatItKnowsAndTheOtherFieldsMerg /// The conflict path is the whole reason the algorithm re-READS instead of retrying its bytes: the /// winner's field must survive the loser's retry. Here a concurrent writer advances the seal between -/// our read and our CAS; our retry must merge onto the new body, not overwrite it. -TEST(CASRefCheckpoint, TokenConflictRereadsAndMergesOntoTheWinner) +/// our read and our write; our retry must merge onto the new body, not overwrite it. +TEST(CASRefCheckpoint, AConflictRereadsAndMergesOntoTheWinner) { const Layout layout{"p"}; const RootNamespace ns{"srv1/ckpt_conflict"}; @@ -615,96 +619,112 @@ TEST(CASRefCheckpoint, TokenConflictRereadsAndMergesOntoTheWinner) const String key = layout.refCkptKey(life); const RefCkpt base{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - auto backend = std::make_shared(key); - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - - /// The concurrent sealer lands exactly ONCE, immediately after our first read -- so our first CAS - /// carries a token that is no longer current, and our retry has to merge onto its body. + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + CasOperation sealer_op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(sealer_op.create(key, encodeRefCkpt(base), Retry::standard()))); + backend->watch(key); + + /// The concurrent sealer lands exactly ONCE, immediately after our first read -- so our first + /// write carries a precondition that is no longer current, and our retry has to merge onto its + /// body. It writes on its OWN operation: the interference is a different actor, not a reentrant + /// call of the one under test. bool interfered = false; - backend->on_get = [&] + backend->after_read = [&] { if (interfered) return; interfered = true; const RefCkpt sealer{.life_epoch = std::optional{1}, .committed_through = ID_2_1, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = ID_2_1}; - const HeadResult h = backend->head(key); - ASSERT_EQ(backend->putOverwrite(key, encodeRefCkpt(mergeCkpt(base, sealer)), h.token).outcome, - PutOutcome::Done); + overwriteObject(sealer_op, key, encodeRefCkpt(mergeCkpt(base, sealer))); }; const RefCkpt publisher{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = std::nullopt}; - EXPECT_EQ(publishCkpt(*backend, layout, life, publisher, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); + EXPECT_EQ(publishCkpt(op, layout, life, publisher), CkptPublishOutcome::Published); - const auto sample = readCkpt(*backend, layout, life); + backend->after_read = nullptr; + const auto sample = readCkpt(op, layout, life); ASSERT_TRUE(sample.has_value()); EXPECT_EQ(sample->ckpt.checkpoint_snapshot_id, ID_1_2) << "our own contribution must land"; EXPECT_EQ(sample->ckpt.last_epoch_seal, ID_2_1) << "the concurrent writer's seal must survive our retry -- a retry that reused the body read " "before the conflict would silently drop it (TLC `_sab_sealclobbersbase`)"; EXPECT_EQ(sample->ckpt.life_epoch, 1u); - EXPECT_GE(backend->casPutCount(key), 2u) << "the first CAS must have been rejected, not skipped"; + EXPECT_GE(std::count(backend->journal.begin(), backend->journal.end(), String{"WRITE"}), 2) + << "the first write must have been refused, not skipped"; } /// A contribution that adds nothing issues NO write. This is a correctness property, not a saving: -/// both writers publish on every snapshot and every seal, and a no-op write would mint a fresh token -/// each time, turning every other writer's in-flight CAS into a conflict for identical bytes. +/// both writers publish on every snapshot and every seal, and a no-op write would mint a fresh +/// incarnation each time, turning every other writer's in-flight write into a conflict for identical +/// bytes. TEST(CASRefCheckpoint, AnIdenticalMergedBodyIssuesNoWrite) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const RootNamespace ns{"srv1/ckpt_noop"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(ns); const String key = layout.refCkptKey(life); const RefCkpt full{.life_epoch = std::optional{1}, .committed_through = ID_2_1, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = ID_2_1}; - ASSERT_EQ(publishCkpt(*backend, layout, life, full, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const uint64_t writes_after_create = backend->casPutCount(key); - const Token token_after_create = backend->head(key).token; + ASSERT_EQ(publishCkpt(op, layout, life, full), CkptPublishOutcome::Published); + const uint64_t writes_after_create = backend->writes(key); + const auto meta_after_create = op.head(key, Retry::standard()); + ASSERT_TRUE(meta_after_create.has_value()); /// The same contribution again, and a strictly OLDER one: neither adds anything. - EXPECT_EQ(publishCkpt(*backend, layout, life, full, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::IdenticalSkip); + EXPECT_EQ(publishCkpt(op, layout, life, full), CkptPublishOutcome::IdenticalSkip); const RefCkpt older{.life_epoch = std::optional{1}, .committed_through = ID_1_1, .checkpoint_snapshot_id = ID_1_1, .last_epoch_seal = std::nullopt}; - EXPECT_EQ(publishCkpt(*backend, layout, life, older, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::IdenticalSkip); + EXPECT_EQ(publishCkpt(op, layout, life, older), CkptPublishOutcome::IdenticalSkip); - EXPECT_EQ(backend->casPutCount(key), writes_after_create) << "a skip must issue no CAS at all"; - EXPECT_EQ(backend->head(key).token, token_after_create) << "and must not mint a new incarnation"; + EXPECT_EQ(backend->writes(key), writes_after_create) << "a skip must issue no write at all"; + const auto meta_after_skips = op.head(key, Retry::standard()); + ASSERT_TRUE(meta_after_skips.has_value()); + EXPECT_EQ(meta_after_skips->incarnation, meta_after_create->incarnation) + << "and must not mint a new incarnation"; } -/// The fence is re-checked AFTER the read and BEFORE the write, on every attempt. A generation that -/// moved means this writer's lease incarnation is gone, so its merged body is stale even if the fence -/// is live again under a fresh incarnation. -TEST(CASRefCheckpoint, AFenceBumpBetweenTheReadAndTheCasWritesNothing) +/// Admission is re-checked AFTER the read and BEFORE the write. A writer whose admission was lost in +/// that window has a stale merged body, so its write must never be sent. +TEST(CASRefCheckpoint, AnAdmissionLossBetweenTheReadAndTheWriteWritesNothing) { - auto backend = std::make_shared(); const Layout layout{"p"}; const RootNamespace ns{"srv1/ckpt_fenced"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(ns); const String key = layout.refCkptKey(life); + + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + bool admitted = true; + CasOperation op = requests.admit([&admitted] { return admitted; }); + CasOperation reader = requests.admit(); + const RefCkpt base{.life_epoch = std::optional{1}, .committed_through = ID_1_1, .checkpoint_snapshot_id = ID_1_1, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(publishCkpt(*backend, layout, life, base, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const Token token_before = backend->head(key).token; - const uint64_t writes_before = backend->casPutCount(key); + ASSERT_EQ(publishCkpt(op, layout, life, base), CkptPublishOutcome::Published); + const auto meta_before = reader.head(key, Retry::standard()); + ASSERT_TRUE(meta_before.has_value()); - /// The callback the pool wires from `CasMountRuntime::checkFenceOrThrow`: it throws when the - /// generation moved since admission. Mirrors the real site's class (the transient, upstream-retryable - /// one) so the stub cannot drift into testing a shape production never produces. - const auto moved_fence = [](uint64_t admitted) - { - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, - "fence generation moved since admission ({})", admitted); - }; + /// Armed only now, so the loss lands inside the publish's own read-then-write window rather than + /// before it began. + backend->watch(key); + backend->after_read = [&admitted] { admitted = false; }; + const uint64_t writes_before = backend->writes(key); const RefCkpt advance{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = std::nullopt}; - EXPECT_EQ(publishCkpt(*backend, layout, life, advance, 1, moved_fence, generousDeadline()), - CkptPublishOutcome::FencedOut); - EXPECT_EQ(backend->casPutCount(key), writes_before) << "the check precedes the CAS, so nothing is sent"; - EXPECT_EQ(backend->head(key).token, token_before); - EXPECT_EQ(readCkptOrFail(*backend, layout, life), base); + EXPECT_EQ(publishCkpt(op, layout, life, advance), CkptPublishOutcome::FencedOut); + backend->after_read = nullptr; + + EXPECT_EQ(backend->writes(key), writes_before) << "the check precedes the write, so nothing is sent"; + const auto meta_after = reader.head(key, Retry::standard()); + ASSERT_TRUE(meta_after.has_value()); + EXPECT_EQ(meta_after->incarnation, meta_before->incarnation); + EXPECT_EQ(readCkptOrFail(reader, layout, life), base); } /// Persistent contention fails CLOSED and says so. There is no partial state to clean up -- every @@ -718,32 +738,48 @@ TEST(CASRefCheckpoint, AnExhaustedDeadlineUnderPersistentConflictThrowsRetryLate const String key = layout.refCkptKey(life); const RefCkpt base{.life_epoch = std::optional{1}, .committed_through = ID_1_1, .checkpoint_snapshot_id = ID_1_1, .last_epoch_seal = std::nullopt}; - auto backend = std::make_shared(key); - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - - /// Every read is followed by a rewrite of the SAME body under a fresh incarnation, so the token this - /// call holds is always stale and every CAS it issues conflicts. The clock advances one step per - /// read, so the DEADLINE is what ends the loop -- deterministically, with no sleeping and well - /// before the live-lock brake. - uint64_t now = 0; - backend->on_get = [&] + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + CasOperation setup = requests.admit(); + ASSERT_TRUE(std::holds_alternative(setup.create(key, encodeRefCkpt(base), Retry::standard()))); + backend->watch(key); + + /// Every read is followed by a rewrite of the SAME body under a fresh incarnation, so the + /// precondition this call holds is always stale and every write it issues is refused. Only the + /// policy's deadline can end the loop, and the injected clock reaches it without sleeping. + /// + /// `overwriteObject` itself reads `key` (its own `readModifyWrite`'s precondition read), which + /// would re-enter this very hook -- unlike the concurrent-actor fixtures elsewhere in this file, + /// this rewrite is not a one-shot: it must keep firing on every OUTER read, so a plain one-shot + /// latch would silently stop the persistent conflict after the first attempt. Guard only the + /// reentrant call instead. + bool rewriting = false; + backend->after_read = [&] { - ++now; - const HeadResult h = backend->head(key); - if (h.exists) - backend->putOverwrite(key, encodeRefCkpt(base), h.token); + if (rewriting) + return; + rewriting = true; + overwriteObject(setup, key, encodeRefCkpt(base)); + rewriting = false; }; const RefCkpt advance{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = std::nullopt}; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, - [&] { publishCkpt(*backend, layout, life, advance, 1, ALWAYS_ADMITTED, CkptDeadline{[&] { return now; }, 5}); }); - EXPECT_EQ(readCkptOrFail(*backend, layout, life), base) << "no partial state: every attempt either " - "committed the complete merged body or wrote nothing"; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishCkpt(op, layout, life, advance); }); + backend->after_read = nullptr; + EXPECT_FALSE(clock.sleeps.empty()) << "the loop must back off between attempts, not spin"; + EXPECT_EQ(readCkptOrFail(setup, layout, life), base) << "no partial state: every attempt either " + "committed the complete merged body or wrote nothing"; } -TEST(CASRefCheckpoint, AmbiguousCommittedCasIsResolvedByOneExactReadWithoutBlindRetry) +TEST(CASRefCheckpoint, AnAmbiguousCommittedWriteIsResolvedByOneExactReadWithoutBlindRetry) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(RootNamespace{"srv1/ckpt_ambiguous_committed"}); const String key = layout.refCkptKey(life); @@ -751,18 +787,23 @@ TEST(CASRefCheckpoint, AmbiguousCommittedCasIsResolvedByOneExactReadWithoutBlind .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; const RefCkpt contribution{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - backend->arm(key, AmbiguousCkptBackend::Fault::CommitThenThrow); + ASSERT_TRUE(std::holds_alternative(op.create(key, encodeRefCkpt(base), Retry::standard()))); + backend->arm(key, CkptProbeBackend::Fault::CommitThenThrow); - EXPECT_EQ(publishCkpt(*backend, layout, life, contribution, 7, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - EXPECT_EQ(backend->journal, (std::vector{"GET", "CAS", "GET"})); - EXPECT_EQ(readCkptOrFail(*backend, layout, life).committed_through, ID_1_2); + EXPECT_EQ(publishCkpt(op, layout, life, contribution), CkptPublishOutcome::Published); + EXPECT_EQ(backend->journal, (std::vector{"READ", "WRITE", "READ"})); + backend->watched_key.clear(); + EXPECT_EQ(readCkptOrFail(op, layout, life).committed_through, ID_1_2); } -TEST(CASRefCheckpoint, AmbiguousUncommittedCasRetriesAgainstTheExactReadToken) +TEST(CASRefCheckpoint, AnAmbiguousUncommittedWriteRetriesAgainstTheExactRead) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); const Layout layout{"p"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(RootNamespace{"srv1/ckpt_ambiguous_retry"}); const String key = layout.refCkptKey(life); @@ -770,18 +811,26 @@ TEST(CASRefCheckpoint, AmbiguousUncommittedCasRetriesAgainstTheExactReadToken) .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; const RefCkpt contribution{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - backend->arm(key, AmbiguousCkptBackend::Fault::ThrowWithoutCommit); + ASSERT_TRUE(std::holds_alternative(op.create(key, encodeRefCkpt(base), Retry::standard()))); + backend->arm(key, CkptProbeBackend::Fault::ThrowWithoutCommit); - EXPECT_EQ(publishCkpt(*backend, layout, life, contribution, 7, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - EXPECT_EQ(backend->journal, (std::vector{"GET", "CAS", "GET", "CAS"})); - EXPECT_EQ(readCkptOrFail(*backend, layout, life).committed_through, ID_1_2); + EXPECT_EQ(publishCkpt(op, layout, life, contribution), CkptPublishOutcome::Published); + EXPECT_EQ(backend->journal, (std::vector{"READ", "WRITE", "READ", "WRITE"})); + backend->watched_key.clear(); + EXPECT_EQ(readCkptOrFail(op, layout, life).committed_through, ID_1_2); } -TEST(CASRefCheckpoint, AmbiguousCasAcceptsAValidDominatingDurableFrontier) +/// The durable body a winner left behind already dominates this contribution, so nothing more is owed. +/// The verdict is `Published` rather than `IdenticalSkip` because an attempt of THIS call was sent: +/// `IdenticalSkip` promises no write was issued, and that promise has to stay true. +TEST(CASRefCheckpoint, AnAmbiguousWriteAcceptsAValidDominatingDurableFrontier) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); const Layout layout{"p"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(RootNamespace{"srv1/ckpt_ambiguous_dominating"}); const String key = layout.refCkptKey(life); @@ -791,122 +840,153 @@ TEST(CASRefCheckpoint, AmbiguousCasAcceptsAValidDominatingDurableFrontier) .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; const RefCkpt dominating{.life_epoch = 1, .committed_through = ID_2_1, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = ID_2_1}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(op.create(key, encodeRefCkpt(base), Retry::standard()))); backend->dominating_bytes = encodeRefCkpt(dominating); - backend->arm(key, AmbiguousCkptBackend::Fault::CommitThenThrow); + backend->arm(key, CkptProbeBackend::Fault::CommitThenThrow); - EXPECT_EQ(publishCkpt(*backend, layout, life, contribution, 7, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - EXPECT_EQ(backend->journal, (std::vector{"GET", "CAS", "GET"})); - EXPECT_EQ(readCkptOrFail(*backend, layout, life), dominating); + EXPECT_EQ(publishCkpt(op, layout, life, contribution), CkptPublishOutcome::Published); + EXPECT_EQ(backend->journal, (std::vector{"READ", "WRITE", "READ"})); + backend->watched_key.clear(); + EXPECT_EQ(readCkptOrFail(op, layout, life), dominating); } -TEST(CASRefCheckpoint, FailedExactReadAfterAmbiguousCasFailsClosedWithoutAnotherCas) +/// A resolve read that never answers leaves the attempt unproven, and the call must neither report it +/// committed nor send a second attempt on top of it. The engine reissues -- that is its contract -- but +/// every reissue is preceded by its own exact read. +TEST(CASRefCheckpoint, AFailedResolveReadNeverReportsACommitAndNeverSkipsTheRead) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + CasOperation reader = requests.admit(); const Layout layout{"p"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(RootNamespace{"srv1/ckpt_ambiguous_read_failed"}); const String key = layout.refCkptKey(life); const RefCkpt base{.life_epoch = 1, .committed_through = ID_1_1, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - backend->fail_resolution_get = true; - backend->arm(key, AmbiguousCkptBackend::Fault::ThrowWithoutCommit); + ASSERT_TRUE(std::holds_alternative(op.create(key, encodeRefCkpt(base), Retry::standard()))); + backend->arm(key, CkptProbeBackend::Fault::AlwaysThrowWithoutCommit); + backend->fail_reads_after_the_first = true; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { - publishCkpt(*backend, layout, life, RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}, 7, - ALWAYS_ADMITTED, generousDeadline()); + publishCkpt(op, layout, life, RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); }); - EXPECT_EQ(backend->journal, (std::vector{"GET", "CAS", "GET"})); - EXPECT_EQ(readCkptOrFail(*backend, layout, life), base); -} - -TEST(CASRefCheckpoint, FenceMovementAroundAmbiguityResolutionMakesTheExactReadInert) -{ - for (const bool move_before_read : {true, false}) - { - auto backend = std::make_shared(); - const Layout layout{"p"}; - const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife( - RootNamespace{move_before_read ? "srv1/ckpt_fence_before_resolution" : "srv1/ckpt_fence_after_resolution"}); - const String key = layout.refCkptKey(life); - const RefCkpt base{.life_epoch = 1, .committed_through = ID_1_1, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - bool admitted = true; - const auto move_fence = [&] { admitted = false; }; - if (move_before_read) - backend->before_resolution_get = move_fence; - else - backend->after_resolution_get = move_fence; - backend->arm(key, AmbiguousCkptBackend::Fault::CommitThenThrow); - - const auto check_admission = [&](uint64_t) - { - if (!admitted) - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "fence moved"); - }; - EXPECT_EQ(publishCkpt(*backend, layout, life, - RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}, 7, - check_admission, generousDeadline()), CkptPublishOutcome::FencedOut); - EXPECT_EQ(backend->journal, (std::vector{"GET", "CAS", "GET"})); - } + for (size_t i = 1; i < backend->journal.size(); ++i) + EXPECT_FALSE(backend->journal[i] == "WRITE" && backend->journal[i - 1] == "WRITE") + << "two attempts with no exact observation between them, at journal position " << i; + EXPECT_GE(std::count(backend->journal.begin(), backend->journal.end(), String{"WRITE"}), 1); + backend->watched_key.clear(); + backend->fail_reads_after_the_first = false; + EXPECT_EQ(readCkptOrFail(reader, layout, life), base); } -TEST(CASRefCheckpoint, AdmissionLostWithTheAmbiguousCasPreventsItsResolutionGet) +/// Admission lost while the ambiguous write was in flight: the exact read that would settle it is +/// refused before it starts, so the call reports `FencedOut` and claims nothing about the object. +TEST(CASRefCheckpoint, AdmissionLostWithTheAmbiguousWritePreventsItsResolveRead) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + bool admitted = true; + CasOperation op = requests.admit([&admitted] { return admitted; }); + CasOperation reader = requests.admit(); const Layout layout{"p"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife( RootNamespace{"srv1/ckpt_admission_lost_before_resolution"}); const String key = layout.refCkptKey(life); const RefCkpt base{.life_epoch = 1, .committed_through = ID_1_1, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(reader.create(key, encodeRefCkpt(base), Retry::standard()))); - bool admitted = true; - backend->after_ambiguous_cas = [&] { admitted = false; }; - backend->arm(key, AmbiguousCkptBackend::Fault::CommitThenThrow); - const auto admit_request = [&] - { - if (!admitted) - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "admission withdrawn"); - }; + backend->after_write = [&admitted] { admitted = false; }; + backend->arm(key, CkptProbeBackend::Fault::CommitThenThrow); - EXPECT_EQ(publishCkpt(*backend, layout, life, + EXPECT_EQ(publishCkpt(op, layout, life, RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}, - 7, ALWAYS_ADMITTED, generousDeadline(), admit_request), CkptPublishOutcome::FencedOut); - EXPECT_EQ(backend->journal, (std::vector{"GET", "CAS"})) - << "publishCkpt started its ambiguity-resolution GET after admission was withdrawn"; + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}), + CkptPublishOutcome::FencedOut); + EXPECT_EQ(backend->journal, (std::vector{"READ", "WRITE"})) + << "the resolve read started after admission was withdrawn"; } -TEST(CASRefCheckpoint, ContinuedAmbiguityStopsAtTheDeadlineAndNeverIssuesConsecutiveCasAttempts) +/// Admission lost AFTER the resolve read proved the attempt durable: the object may well carry this +/// contribution, but a call whose admission is gone must never claim it. +TEST(CASRefCheckpoint, AdmissionLostAfterTheResolveReadStillRefusesToClaimTheCommit) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + bool admitted = true; + CasOperation op = requests.admit([&admitted] { return admitted; }); + CasOperation reader = requests.admit(); const Layout layout{"p"}; - const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(RootNamespace{"srv1/ckpt_ambiguity_deadline"}); + const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(RootNamespace{"srv1/ckpt_fence_after_resolution"}); const String key = layout.refCkptKey(life); const RefCkpt base{.life_epoch = 1, .committed_through = ID_1_1, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(backend->casPut(key, encodeRefCkpt(base), std::nullopt).outcome, CasOutcome::Committed); - uint64_t now = 0; - backend->after_resolution_get = [&] { ++now; }; - backend->arm(key, AmbiguousCkptBackend::Fault::AlwaysThrowWithoutCommit); + ASSERT_TRUE(std::holds_alternative(reader.create(key, encodeRefCkpt(base), Retry::standard()))); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + backend->arm(key, CkptProbeBackend::Fault::CommitThenThrow); + /// The SECOND read is the resolve read; withdrawing after the first would refuse the write instead + /// and never reach the point this test is about. + size_t reads = 0; + backend->after_read = [&] { - publishCkpt(*backend, layout, life, - RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}, - 7, ALWAYS_ADMITTED, CkptDeadline{[&] { return now; }, 3}); - }); - EXPECT_EQ(backend->journal, - (std::vector{"GET", "CAS", "GET", "CAS", "GET", "CAS", "GET"})); - EXPECT_EQ(readCkptOrFail(*backend, layout, life), base); + if (++reads == 2) + admitted = false; + }; + + EXPECT_EQ(publishCkpt(op, layout, life, + RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}), + CkptPublishOutcome::FencedOut); + EXPECT_EQ(backend->journal, (std::vector{"READ", "WRITE", "READ"})); +} + +/// Both verdicts `publishCkpt` reaches WITHOUT writing consult admission before they speak, and both +/// answer `FencedOut` when it is gone. A writer the fence is about to refuse landed nothing anywhere: +/// telling it `IdenticalSkip` would claim its contribution is already durable, and telling it +/// `CORRUPTED_DATA` would turn a transient control signal into a permanent verdict on the namespace. +TEST(CASRefCheckpoint, DeclineTimeVerdictsReadAdmitted) +{ + const Layout layout{"p"}; + for (const bool decreasing : {false, true}) + { + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + bool admitted = true; + CasOperation op = requests.admit([&admitted] { return admitted; }); + CasOperation reader = requests.admit(); + const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife( + RootNamespace{decreasing ? "srv1/ckpt_decline_decrease" : "srv1/ckpt_decline_identical"}); + const String key = layout.refCkptKey(life); + /// A genesis epoch of 2 with the frontier in that same epoch: `checkRefCkptInvariants` refuses + /// a `committed_through` preceding `life_epoch`, so the durable body a decrease is measured + /// against has to be one the format would actually store. + const RefCkpt durable{.life_epoch = std::optional{2}, .committed_through = ID_2_1, + .checkpoint_snapshot_id = ID_2_1, .last_epoch_seal = std::nullopt}; + ASSERT_EQ(publishCkpt(reader, layout, life, durable), CkptPublishOutcome::Published); + const uint64_t writes_before = backend->writes(key); + + /// Admission survives the read and is gone by the time the verdict is reached -- the exact + /// window in which the answer must be `FencedOut` and nothing else. + backend->watch(key); + backend->after_read = [&admitted] { admitted = false; }; + const RefCkpt contribution = decreasing + ? RefCkpt{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, + .last_epoch_seal = std::nullopt} + : durable; + EXPECT_EQ(publishCkpt(op, layout, life, contribution), CkptPublishOutcome::FencedOut) + << (decreasing ? "a superseded epoch from an unadmitted writer is not corruption" + : "an identical body from an unadmitted writer is not a skip"); + backend->after_read = nullptr; + + EXPECT_EQ(backend->writes(key), writes_before) << "neither verdict may write"; + EXPECT_EQ(readCkptOrFail(reader, layout, life), durable); + } } /// A `_ckpt` that does not decode is NEVER overwritten. It is the only record of recovery's base and @@ -915,33 +995,50 @@ TEST(CASRefCheckpoint, ContinuedAmbiguityStopsAtTheDeadlineAndNeverIssuesConsecu TEST(CASRefCheckpoint, ACorruptCheckpointIsNeverOverwritten) { auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const Layout layout{"p"}; const RootNamespace ns{"srv1/ckpt_corrupt"}; const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(ns); const String key = layout.refCkptKey(life); - ASSERT_EQ(publishCkpt(*backend, layout, life, - RefCkpt{.life_epoch = std::optional{1}, .committed_through = ID_1_2, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = std::nullopt}, - 1, ALWAYS_ADMITTED, generousDeadline()), CkptPublishOutcome::Published); + ASSERT_EQ(publishCkpt(op, layout, life, + RefCkpt{.life_epoch = std::optional{1}, .committed_through = ID_1_2, .checkpoint_snapshot_id = ID_1_2, .last_epoch_seal = std::nullopt}), + CkptPublishOutcome::Published); const String garbage = "not a cas object\n"; - overwriteObject(*backend, key, garbage); + overwriteObject(op, key, garbage); const RefCkpt birth{.life_epoch = std::optional{5}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { publishCkpt(*backend, layout, life, birth, 1, ALWAYS_ADMITTED, generousDeadline()); }); - EXPECT_EQ(backend->get(key)->bytes, garbage) << "corruption must be surfaced, never laundered into a " - "well-formed object"; + [&] { publishCkpt(op, layout, life, birth); }); + const auto still_there = op.read(key, Retry::standard()); + ASSERT_TRUE(still_there.has_value()); + EXPECT_EQ(still_there->bytes, garbage) << "corruption must be surfaced, never laundered into a " + "well-formed object"; } /// --------------------------------------------------------------------------------------------- /// The reader-side rules Task 6 and the cleanup call sites consume /// --------------------------------------------------------------------------------------------- -/// INV-4's three-way revalidation of a base that turned out to be missing. -TEST(CASRefCheckpoint, AMissingSampledBaseRestartsOnAnAdvancedTokenAndIsCorruptionOnAnUnchangedOne) +/// INV-4's three-way revalidation of a base that turned out to be missing. The two incarnations come +/// from real reads of the same key across a rewrite, because an incarnation exists only as something a +/// request observed. +TEST(CASRefCheckpoint, AMissingSampledBaseRestartsOnAnAdvancedIncarnationAndIsCorruptionOnAnUnchangedOne) { - const Token sampled{"t1", TokenType::Emulated}; - const Token advanced{"t2", TokenType::Emulated}; + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + const String key = "p/ckpt_incarnations"; + ASSERT_TRUE(std::holds_alternative(op.create(key, "first", Retry::standard()))); + const auto first = op.read(key, Retry::standard()); + ASSERT_TRUE(first.has_value()); + overwriteObject(op, key, "second"); + const auto second = op.read(key, Retry::standard()); + ASSERT_TRUE(second.has_value()); + const Incarnation sampled = first->incarnation; + const Incarnation advanced = second->incarnation; + ASSERT_FALSE(sampled == advanced) << "the rewrite must mint a different incarnation"; EXPECT_EQ(classifyMissingSampledBase(sampled, advanced), MissingBaseVerdict::RestartRecovery) << "cleanup legitimately moved the checkpoint while we read; restart from the newer base"; @@ -974,18 +1071,20 @@ TEST(CASRefCheckpoint, NamespaceBirthCreatesTheCheckpointCarryingItsLifeEpoch) { auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"srv1/ckpt_birth"}; /// Stage B (Task 4-C): the catalog carries no entry for `ns` before its first open, and the /// namespace's real incarnation does not exist to name a key with yet -- the pre-birth analog of /// "nothing exists" is "nothing is even NAMED", checked at the catalog rather than at a key this /// test cannot yet compute. - EXPECT_TRUE(CasRefCatalog::read(*backend, store->layout()).catalog.entries.empty()) + EXPECT_TRUE(CasRefCatalog::read(op, store->layout()).catalog.entries.empty()) << "nothing exists before the birth"; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{store->writerEpoch(), 1})); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); - const auto sample = readCkpt(*backend, store->layout(), life); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); + const auto sample = readCkpt(op, store->layout(), life); ASSERT_TRUE(sample.has_value()) << "spec §3 creates the _ckpt before the namespace becomes Live"; EXPECT_EQ(sample->ckpt.life_epoch, store->writerEpoch()); EXPECT_FALSE(sample->ckpt.checkpoint_snapshot_id.has_value()) << "a newborn namespace has no base yet"; @@ -997,25 +1096,27 @@ TEST(CASRefCheckpoint, ACommittedSnapshotPublishAdvancesTheCheckpoint) { auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const uint64_t epoch = store->writerEpoch(); const RootNamespace ns{"srv1/ckpt_publish"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{epoch, 1})); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); - ASSERT_FALSE(readCkptOrFail(*backend, store->layout(), life).checkpoint_snapshot_id.has_value()); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); + ASSERT_FALSE(readCkptOrFail(op, store->layout(), life).checkpoint_snapshot_id.has_value()); ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); const auto published = store->newestPublishedSnapshotIdForTest(ns); ASSERT_TRUE(published.has_value()); - const auto sample = readCkpt(*backend, store->layout(), life); + const auto sample = readCkpt(op, store->layout(), life); ASSERT_TRUE(sample.has_value()); EXPECT_EQ(sample->ckpt.checkpoint_snapshot_id, published); EXPECT_EQ(sample->ckpt.life_epoch, epoch) << "the publisher contributes nothing about life_epoch, so " "the merge must preserve what the birth wrote"; /// And the snapshot body it names really is there -- the checkpoint may never point at a key that /// does not exist, which is the premise the missing-base rule reasons from. - EXPECT_TRUE(backend->head(store->layout().refSnapshotKey(life, *published)).exists); + EXPECT_TRUE(op.head(store->layout().refSnapshotKey(life, *published), Retry::standard()).has_value()); } /// The body-PUT/cleanup/`_ckpt` race, decided by the ORDER of the two writes: cleanup planned in the @@ -1025,18 +1126,20 @@ TEST(CASRefCheckpoint, CleanupPlannedBetweenTheBodyPutAndTheCkptCasCannotDeleteT { auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const uint64_t epoch = store->writerEpoch(); const RootNamespace ns{"srv1/ckpt_race"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{epoch, 1})); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); const RefTxnId first_snapshot = *store->newestPublishedSnapshotIdForTest(ns); ASSERT_EQ(publishRef(store, ns, "ref_2", 2), (RefTxnId{epoch, 2})); /// The checkpoint a cleanup pass sampled BEFORE the second publication -- the stale reading the /// race hands it. - const std::optional stale_checkpoint = readCkptOrFail(*backend, store->layout(), life).checkpoint_snapshot_id; + const std::optional stale_checkpoint = readCkptOrFail(op, store->layout(), life).checkpoint_snapshot_id; ASSERT_EQ(stale_checkpoint, first_snapshot); ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); @@ -1048,7 +1151,7 @@ TEST(CASRefCheckpoint, CleanupPlannedBetweenTheBodyPutAndTheCkptCasCannotDeleteT EXPECT_FALSE(snapshotDeletableUnderCkpt(second_snapshot, stale_checkpoint)); EXPECT_FALSE(snapshotDeletableUnderCkpt(first_snapshot, stale_checkpoint)); /// Once the checkpoint is re-read, the older snapshot becomes reclaimable and the base does not. - const std::optional fresh_checkpoint = readCkptOrFail(*backend, store->layout(), life).checkpoint_snapshot_id; + const std::optional fresh_checkpoint = readCkptOrFail(op, store->layout(), life).checkpoint_snapshot_id; EXPECT_TRUE(snapshotDeletableUnderCkpt(first_snapshot, fresh_checkpoint)); EXPECT_FALSE(snapshotDeletableUnderCkpt(second_snapshot, fresh_checkpoint)); } @@ -1057,35 +1160,39 @@ TEST(CASRefCheckpoint, CleanupPlannedBetweenTheBodyPutAndTheCkptCasCannotDeleteT /// published, and a publisher with nothing above its newest snapshot touches it at all. TEST(CASRefCheckpoint, TheCheckpointIsWrittenOncePerPublicationAndNotOnIdleAttempts) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const uint64_t epoch = store->writerEpoch(); const RootNamespace ns{"srv1/ckpt_republish"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{epoch, 1})); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); const String key = store->layout().refCkptKey(life); - const uint64_t writes_after_birth = backend->casPutCount(key); + const uint64_t writes_after_birth = backend->writes(key); EXPECT_EQ(writes_after_birth, 2u) << "birth publishes `life_epoch` before its log, then the durable log's committed frontier"; ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); - EXPECT_EQ(backend->casPutCount(key), writes_after_birth + 1) << "one publication, one checkpoint CAS"; - const uint64_t writes_after_publish = backend->casPutCount(key); - const auto after_publish = readCkpt(*backend, store->layout(), life); + EXPECT_EQ(backend->writes(key), writes_after_birth + 1) << "one publication, one checkpoint write"; + const uint64_t writes_after_publish = backend->writes(key); + const auto after_publish = readCkpt(op, store->layout(), life); ASSERT_TRUE(after_publish.has_value()); /// Nothing was appended since, so there is nothing above the newest snapshot: the publisher declines /// before it reaches the checkpoint at all, and repeating the attempt changes nothing. EXPECT_FALSE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); EXPECT_FALSE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); - EXPECT_EQ(backend->casPutCount(key), writes_after_publish); - EXPECT_EQ(readCkptOrFail(*backend, store->layout(), life), after_publish->ckpt); + EXPECT_EQ(backend->writes(key), writes_after_publish); + EXPECT_EQ(readCkptOrFail(op, store->layout(), life), after_publish->ckpt); } TEST(CASRefCheckpoint, SnapshotPublisherRefusesEpochSealCandidateWithoutAnyWrite) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"srv1/no_snapshot_at_seal"}; uint64_t predecessor_epoch = 0; { @@ -1099,19 +1206,19 @@ TEST(CASRefCheckpoint, SnapshotPublisherRefusesEpochSealCandidateWithoutAnyWrite ASSERT_EQ(store->listRefs(ns).size(), 1u) << "recovery must close the predecessor epoch before publishing"; const RefTxnId seal_id{predecessor_epoch, 2}; - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); const String snapshot_key = store->layout().refSnapshotKey(life, seal_id); const String ckpt_key = store->layout().refCkptKey(life); ASSERT_EQ(store->lastEpochSealForTest(ns), std::make_optional(seal_id)); - ASSERT_EQ(readCkptOrFail(*backend, store->layout(), life).committed_through, std::make_optional(seal_id)); - const uint64_t snapshot_puts_before = backend->putCount(snapshot_key); - const uint64_t ckpt_cas_before = backend->casPutCount(ckpt_key); + ASSERT_EQ(readCkptOrFail(op, store->layout(), life).committed_through, std::make_optional(seal_id)); + const uint64_t snapshot_writes_before = backend->writes(snapshot_key); + const uint64_t ckpt_writes_before = backend->writes(ckpt_key); /// Recovery installed the epoch seal as the runtime's greatest applied transaction. The publisher /// must decline it without reaching either durable write. EXPECT_FALSE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); - EXPECT_EQ(backend->putCount(snapshot_key), snapshot_puts_before); - EXPECT_EQ(backend->casPutCount(ckpt_key), ckpt_cas_before); + EXPECT_EQ(backend->writes(snapshot_key), snapshot_writes_before); + EXPECT_EQ(backend->writes(ckpt_key), ckpt_writes_before); /// Once an ordinary transaction advances the candidate beyond the seal, normal publication resumes. ASSERT_EQ(publishRef(store, ns, "ref_2", 2), (RefTxnId{store->writerEpoch(), 1})); @@ -1121,17 +1228,19 @@ TEST(CASRefCheckpoint, SnapshotPublisherRefusesEpochSealCandidateWithoutAnyWrite /// Publication replays a `NeedsRecovery` lane before it captures a snapshot and advances `_ckpt`. TEST(CASRefCheckpoint, NeedsRecoveryReplaysBeforeCheckpointAdvance) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"srv1/ckpt_poisoned"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{store->writerEpoch(), 1})); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); const String key = store->layout().refCkptKey(life); - const auto before = readCkpt(*backend, store->layout(), life); + const auto before = readCkpt(op, store->layout(), life); ASSERT_TRUE(before.has_value()); ASSERT_FALSE(before->ckpt.checkpoint_snapshot_id.has_value()); - const uint64_t writes_before = backend->casPutCount(key); + const uint64_t writes_before = backend->writes(key); /// Enter `NeedsRecovery`: an install throws after its transaction is /// durable, leaving this cached table missing a transaction the log contains. @@ -1154,9 +1263,9 @@ TEST(CASRefCheckpoint, NeedsRecoveryReplaysBeforeCheckpointAdvance) EXPECT_TRUE(store->resolveRef(ns, "ref_2", /*allow_stale=*/false).has_value()) << "the stranded transaction is durable; the re-derivation must have applied it"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); - EXPECT_GT(backend->casPutCount(key), writes_before) + EXPECT_GT(backend->writes(key), writes_before) << "and the checkpoint advances -- truthfully, over a snapshot that is not missing anything"; - EXPECT_TRUE(readCkptOrFail(*backend, store->layout(), life).checkpoint_snapshot_id.has_value()); + EXPECT_TRUE(readCkptOrFail(op, store->layout(), life).checkpoint_snapshot_id.has_value()); } @@ -1169,26 +1278,27 @@ TEST(CASRefCheckpoint, APublishFencedOutMidAttemptDoesNotAdvanceTheCheckpoint) /// The watched key cannot be computed yet -- the real incarnation is minted only once the pool /// exists and this namespace's first open resolves it (`setWatchedKey` below, once it has). - auto backend = std::make_shared(""); + auto backend = std::make_shared(); DB::Cas::tests::seedPoolMetaForRestart(*backend); PoolPtr store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{store->writerEpoch(), 1})); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); const String ckpt_key = store->layout().refCkptKey(life); - backend->setWatchedKey(ckpt_key); - - const auto before = readCkpt(*backend, store->layout(), life); + const auto before = readCkpt(op, store->layout(), life); ASSERT_TRUE(before.has_value()); ASSERT_FALSE(before->ckpt.checkpoint_snapshot_id.has_value()); - const uint64_t writes_before = backend->casPutCount(ckpt_key); + const uint64_t writes_before = backend->writes(ckpt_key); + backend->watch(ckpt_key); /// Arm only after the precondition read above. The next watched `_ckpt` read is therefore the one /// inside this publish's read-then-CAS window, after the attempt captured its immutable runtime /// generation. Arming before `readCkpt` would stale the runtime before the operation began and test /// entry admission instead of the intended mid-attempt recheck. bool hook_fired = false; - backend->on_get = [&] + backend->after_read = [&] { if (hook_fired) return; @@ -1198,10 +1308,10 @@ TEST(CASRefCheckpoint, APublishFencedOutMidAttemptDoesNotAdvanceTheCheckpoint) EXPECT_FALSE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)) << "a publish whose checkpoint could not be advanced must not report success"; - EXPECT_TRUE(hook_fired) << "the checkpoint read-then-CAS seam was never exercised"; - backend->on_get = nullptr; - EXPECT_EQ(backend->casPutCount(ckpt_key), writes_before) << "nothing may be sent after the fence moved"; - EXPECT_FALSE(readCkptOrFail(*backend, store->layout(), life).checkpoint_snapshot_id.has_value()); + EXPECT_TRUE(hook_fired) << "the checkpoint read-then-write seam was never exercised"; + backend->after_read = nullptr; + EXPECT_EQ(backend->writes(ckpt_key), writes_before) << "nothing may be sent after the fence moved"; + EXPECT_FALSE(readCkptOrFail(op, store->layout(), life).checkpoint_snapshot_id.has_value()); EXPECT_FALSE(store->newestPublishedSnapshotIdForTest(ns).has_value()) << "the snapshot must not be adopted as the newest while its checkpoint is unpublished"; } @@ -1224,6 +1334,8 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) { auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"test/golden@cas@"}; const RefTxnId id = publishRef(store, ns, "gold_ref", 7); @@ -1235,7 +1347,7 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) /// a REAL, randomly minted catalog value rather than the Stage-A sentinel, so it is learned back /// from the catalog (`liveLifeOrFail`) rather than pasted as a literal -- the shape assertion below /// is unaffected, since it names every OTHER segment literally and renders this one dynamically. - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); const String key = store->layout().refLogKey(life, id); EXPECT_EQ(key, "p/cas/ns/stream/" + renderIncarnation(life.incarnation) + "/_log/0000000000000001-0000000000000001.zst") @@ -1245,7 +1357,7 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) /// but any change that survives both is a 128-bit collision at a fixed length, which is the trade for /// keeping the assertion readable. It is a function of `{format generation, ns, id, ops, /// chain_link}` only -- no incarnation reaches it. - const auto got = backend->get(key); + const auto got = op.read(key, Retry::standard()); ASSERT_TRUE(got.has_value()) << "the birth chunk must be durable at its canonical key"; const String plaintext = openObject(FormatId::RefLog, got->bytes); EXPECT_EQ(plaintext, R"({"type":"cas_ref_log","v":1} @@ -1274,24 +1386,26 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) /// need a sequence-recording backend to pin. TEST(CASRefCheckpoint, AppendRequestCountUnchangedByExtraction) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"test/req@cas@"}; const RefTxnId id = publishRef(store, ns, "req_ref", 1); - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); const String log_key = store->layout().refLogKey(life, id); const String ckpt_key = store->layout().refCkptKey(life); - EXPECT_EQ(backend->putCount(log_key), 1u) << "exactly one write-once PUT per committed chunk"; + EXPECT_EQ(backend->writes(log_key), 1u) << "exactly one write-once request per committed chunk"; /// ONE GET, not zero, since Stage B (Task 4-C): `resolveNamespaceLife`'s `completeCreation` call /// publishes this life's `_ckpt.life_epoch` BEFORE the birth chunk is prepared, so this table's /// OWN recovery walk (also inside this `appendRefOps`, ahead of the commit) grounds itself at the /// genesis position `_ckpt` now names and confirms it absent by exact key -- which is `log_key` /// itself, the position the birth chunk is about to occupy. That GET precedes the Committed PUT; /// the PUT itself still owes no read-back. - EXPECT_EQ(backend->getCount(log_key), 1u) << "one grounding probe from recovery, before the birth PUT"; - EXPECT_EQ(backend->casPutCount(ckpt_key), 2u) + EXPECT_EQ(backend->getCount(log_key), 1u) << "one grounding probe from recovery, before the birth write"; + EXPECT_EQ(backend->writes(ckpt_key), 2u) << "the birth contributes `life_epoch` before its log and `committed_through` after the durable " "log; these are two different ordering obligations, not a duplicate publication"; } @@ -1326,6 +1440,8 @@ TEST(CASRefCheckpoint, PostDurableInstallRegionStillEnteredAfterExtraction) { auto backend = std::make_shared(); auto store = openPool(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const RootNamespace ns{"test/region@cas@"}; unsigned probe_hits = 0; @@ -1336,6 +1452,6 @@ TEST(CASRefCheckpoint, PostDurableInstallRegionStillEnteredAfterExtraction) EXPECT_GT(probe_hits, 0u) << "no probe-instrumented post-durable install region was entered on a committing append -- " "the `Committed` install arm was not reached at all"; - const NamespaceLifeId life = liveLifeOrFail(*backend, store->layout(), ns); - EXPECT_TRUE(backend->get(store->layout().refLogKey(life, id)).has_value()); + const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); + EXPECT_TRUE(op.read(store->layout().refLogKey(life, id), Retry::standard()).has_value()); } diff --git a/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp b/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp index 0ae7eac58160..9e0d6918b3e0 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt_join.cpp @@ -14,10 +14,13 @@ #include #include +#include #include +#include #include #include #include +#include #include /// The `_ckpt` JOIN law and its `O(1)` SIZE invariant. @@ -93,29 +96,42 @@ String refName(size_t i) return fmt::format("r{:08}", i); } -/// A fence that never refuses, and a deadline far enough out that only the test's own contention -/// decides the outcome -- each `_ckpt`/catalog test file defines its own copy, matching the precedent -/// `gtest_cas_ns_creation_lifecycle.cpp` states explicitly. -const std::function ALWAYS_ADMITTED = [](uint64_t) {}; - -CkptDeadline generousDeadline() +/// Withdraws an operation's admission at a chosen point inside another component's read-then-write +/// window: deterministically, with no sleep and no second thread. A test arms exactly one of the two +/// points and reads `admitted` from its operation's liveness predicate. +class AdmissionHookBackend : public CountingBackend { - return CkptDeadline{[] { return uint64_t{1000}; }, 60000}; -} +public: + bool admitted = true; + /// Withdraw once this exact key has been read. + String withdraw_after_read_of; + /// Withdraw once any `_ckpt` key has been written. The key carries an incarnation the test cannot + /// know before the creation mints it, so the arm names the object kind rather than the key. + bool withdraw_after_ckpt_write = false; -/// Admits the FIRST call (spent by `completeCreation`'s step-2 `publishCkpt`) and refuses every call -/// after (step 3's own `mutate`): "fenced out between the `_ckpt` create and the `Creating -> Live` -/// CAS", deterministically and without a second thread. That is the durable shape a stalled creator -/// leaves behind, and the starting state the resumption test needs. -std::function admittedOnceThenFenced() -{ - auto calls = std::make_shared(0); - return [calls](uint64_t admitted) + explicit AdmissionHookBackend(Layout layout_) : layout(std::move(layout_)) {} + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - if (++*calls > 1) - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "fence generation moved since admission ({})", admitted); - }; -} + auto result = CountingBackend::read(key, access); + if (!withdraw_after_read_of.empty() && key == withdraw_after_read_of) + admitted = false; + return result; + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + auto result = CountingBackend::write(key, bytes, expected_value, access); + if (withdraw_after_ckpt_write && layout.parseRefCkptKey(key)) + admitted = false; + return result; + } + +private: + Layout layout; +}; CreatorFence creatorFence(const String & srid, uint64_t writer_epoch, uint64_t fence_generation = 1) { @@ -139,9 +155,9 @@ const CatalogEntry * findEntryForTest(const RefCatalog & catalog, const RootName /// `life`'s durable `life_epoch`, failing the current test rather than dereferencing a disengaged /// optional -- a bare `->` on one aborts the whole binary and takes every later suite's result with it. -uint64_t lifeEpochOrFail(Backend & backend, const Layout & layout, const NamespaceLifeId & life) +uint64_t lifeEpochOrFail(CasOperation & op, const Layout & layout, const NamespaceLifeId & life) { - const std::optional sample = readCkpt(backend, layout, life); + const std::optional sample = readCkpt(op, layout, life); if (!sample || !sample->ckpt.life_epoch) { ADD_FAILURE() << "expected a _ckpt carrying a life_epoch for namespace '" << life.ns.string() << "'"; @@ -166,9 +182,9 @@ PoolPtr openPool(const BackendPtr & backend, std::function boot_ms_f /// The incarnation the production birth wiring minted for `ns`, learned back from the catalog the way a /// real reader does. Fails the current test rather than dereferencing a disengaged optional, so one /// regression cannot abort the binary and take every later suite's result with it. -NamespaceLifeId liveLifeOrFail(Backend & backend, const Layout & layout, const RootNamespace & ns) +NamespaceLifeId liveLifeOrFail(CasOperation & op, const Layout & layout, const RootNamespace & ns) { - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); for (const CatalogEntry & entry : snap.catalog.entries) if (entry.ns.string() == ns.string()) return NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); @@ -182,7 +198,7 @@ NamespaceLifeId liveLifeOrFail(Backend & backend, const Layout & layout, const R /// One transaction also holds every OTHER dimension fixed while `ref_count` varies: two namespaces /// built this way end at the same transaction id, so a difference in their `_ckpt` bodies can only be /// the refs. `ref_count` must therefore stay within the append lane's per-item operation cap. -String encodedCkptOfNamespaceWithRefs(const PoolPtr & store, Backend & backend, const Layout & layout, +String encodedCkptOfNamespaceWithRefs(const PoolPtr & store, CasOperation & op, const Layout & layout, const RootNamespace & ns, size_t ref_count) { store->appendRefOps(ns, MutationScope::wholeShard(), @@ -192,14 +208,14 @@ String encodedCkptOfNamespaceWithRefs(const PoolPtr & store, Backend & backend, if (state.getLifecycle() != RefLifecycle::Live) ops.push_back(namespaceBirthOp()); for (size_t i = 0; i < ref_count; ++i) - for (const RefOp & op : publishCommittedOps(refName(i), ManifestRef{1, i + 1, 1})) - ops.push_back(op); + for (const RefOp & ref_op : publishCommittedOps(refName(i), ManifestRef{1, i + 1, 1})) + ops.push_back(ref_op); return ops; }, RootMutationOrigin::Writer, RootMutationKind::Publish); - const NamespaceLifeId life = liveLifeOrFail(backend, layout, ns); - const std::optional sample = readCkpt(backend, layout, life); + const NamespaceLifeId life = liveLifeOrFail(op, layout, ns); + const std::optional sample = readCkpt(op, layout, life); if (!sample) { ADD_FAILURE() << "expected a _ckpt for namespace '" << ns.string() << "' after its birth transaction"; @@ -299,36 +315,41 @@ TEST(CASRefCheckpointJoin, CrossEpochFrontierRequiresAnImmediatelyAdjacentSeal) /// only ever rise", so the fixture must not quietly model two roots. TEST(CASRefCheckpointJoin, ResumedCreationRaisesLifeEpochWithoutRefusal) { - InMemoryBackend backend; Layout layout("p"); - DB::Cas::tests::seedPoolMetaForRestart(backend); + auto backend = std::make_shared(layout); + DB::Cas::tests::seedPoolMetaForRestart(*backend); const RootNamespace ns{"a"}; - - ASSERT_EQ(CasRefCatalog::createNamespace(backend, layout, 1, ns, creatorFence("srv1", 5), - /*admitted_generation=*/1, admittedOnceThenFenced(), generousDeadline()), + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation reader = requests.admit(); + + /// The creator loses admission the instant its step-2 `_ckpt` is durable, so step 3 never sends + /// its `Creating -> Live` write. That is the durable shape a stalled creator leaves behind, and + /// the starting state this test needs. + backend->withdraw_after_ckpt_write = true; + CasOperation creator = requests.admit([&backend] { return backend->admitted; }); + ASSERT_EQ(CasRefCatalog::createNamespace(creator, layout, 1, ns, creatorFence("srv1", 5)), CasRefCatalog::NamespaceCreationOutcome::FencedOut); + backend->withdraw_after_ckpt_write = false; /// Bound to a name, never chained through a temporary: a `const CatalogEntry *` taken from an /// unbound `Snapshot` dangles the instant the full expression ends. - const CasRefCatalog::Snapshot stalled = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot stalled = CasRefCatalog::read(reader, layout); const CatalogEntry * entry = findEntryForTest(stalled.catalog, ns); ASSERT_NE(entry, nullptr); ASSERT_EQ(entry->state, NsState::Creating); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation); - EXPECT_EQ(lifeEpochOrFail(backend, layout, life), 5u) << "step 2 landed before the creator stalled"; + EXPECT_EQ(lifeEpochOrFail(reader, layout, life), 5u) << "step 2 landed before the creator stalled"; const CreatorFence resumer = creatorFence("srv1", 9); - ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(backend, layout, *entry, resumer, fixedTerminality(true), - /*admitted_generation=*/1, ALWAYS_ADMITTED), + ASSERT_EQ(CasRefCatalog::reconcileStaleCreator(reader, layout, *entry, resumer, fixedTerminality(true)), CasRefCatalog::ReconcileCreatorOutcome::Reconciled); CatalogEntry resumed = *entry; resumed.creator = resumer; - EXPECT_EQ(CasRefCatalog::completeCreation(backend, layout, resumed, /*admitted_generation=*/1, - ALWAYS_ADMITTED, generousDeadline()), + EXPECT_EQ(CasRefCatalog::completeCreation(reader, layout, resumed), CasRefCatalog::NamespaceCreationOutcome::Live) << "the resumption must not be refused by the join"; - EXPECT_EQ(lifeEpochOrFail(backend, layout, life), 9u) + EXPECT_EQ(lifeEpochOrFail(reader, layout, life), 9u) << "the genesis epoch that actually landed is the resuming actor's, and the join must let it rise"; } @@ -339,19 +360,19 @@ TEST(CASRefCheckpointJoin, ResumedCreationRaisesLifeEpochWithoutRefusal) /// chunk contributes the `NamespaceBirth` record's epoch. CREATE TABLE, restart, INSERT. TEST(CASRefCheckpointJoin, RestartBetweenCreationAndFirstWriteRaisesLifeEpochWithoutRefusal) { - InMemoryBackend backend; + auto backend = std::make_shared(); Layout layout("p"); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(RootNamespace{"a"}, UInt128(42)); const RefCkpt from_creation{.life_epoch = 4, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(publishCkpt(backend, layout, life, from_creation, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); + ASSERT_EQ(publishCkpt(op, layout, life, from_creation), CkptPublishOutcome::Published); const RefCkpt from_birth_chunk{.life_epoch = 7, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - EXPECT_EQ(publishCkpt(backend, layout, life, from_birth_chunk, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published) + EXPECT_EQ(publishCkpt(op, layout, life, from_birth_chunk), CkptPublishOutcome::Published) << "the birth chunk's later epoch must be publishable, not refused as a conflict"; - EXPECT_EQ(lifeEpochOrFail(backend, layout, life), 7u); + EXPECT_EQ(lifeEpochOrFail(op, layout, life), 7u); } /// THE REFUSAL, and the state it constructs IS UNREACHABLE ON ANY HONEST PATH -- that is the point of @@ -373,21 +394,23 @@ TEST(CASRefCheckpointJoin, RestartBetweenCreationAndFirstWriteRaisesLifeEpochWit /// of its arguments is durable. There is deliberately no merge-level counterpart to this test. TEST(CASRefCheckpointJoin, JoinDecreasingLifeEpochIsCorruptionAndPublishesNothing) { - CountingBackend backend; + auto backend = std::make_shared(); Layout layout("p"); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(RootNamespace{"a"}, UInt128(42)); const String key = layout.refCkptKey(life); const RefCkpt durable{.life_epoch = 9, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(publishCkpt(backend, layout, life, durable, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const uint64_t cas_puts_before = backend.casPutCount(key); + ASSERT_EQ(publishCkpt(op, layout, life, durable), CkptPublishOutcome::Published); + /// This suite writes one key only, so the backend's own total is that key's count. + const uint64_t writes_before = backend->writeTotal(); const RefCkpt superseded{.life_epoch = 3, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; String message; try { - publishCkpt(backend, layout, life, superseded, 1, ALWAYS_ADMITTED, generousDeadline()); + publishCkpt(op, layout, life, superseded); ADD_FAILURE() << "a contribution below the durable life_epoch must not be published"; } catch (const DB::Exception & e) @@ -410,8 +433,8 @@ TEST(CASRefCheckpointJoin, JoinDecreasingLifeEpochIsCorruptionAndPublishesNothin /// And nothing was written. The refusal is decided before the body is built, so the durable object /// is untouched and no write was even attempted. - EXPECT_EQ(backend.casPutCount(key), cas_puts_before) << "the publisher must not CAS on a refused publish"; - EXPECT_EQ(lifeEpochOrFail(backend, layout, life), 9u) << "the durable value is unchanged"; + EXPECT_EQ(backend->writeTotal(), writes_before) << "the publisher must not write on a refused publish"; + EXPECT_EQ(lifeEpochOrFail(op, layout, life), 9u) << "the durable value is unchanged"; } /// The other half of the refusal, and the reason it consults the fence before classifying: the SAME @@ -422,26 +445,28 @@ TEST(CASRefCheckpointJoin, JoinDecreasingLifeEpochIsCorruptionAndPublishesNothin /// violation is a STILL-ADMITTED writer contributing a superseded epoch. TEST(CASRefCheckpointJoin, ADecreasingLifeEpochFromAFencedOutWriterIsReportedFencedOutNotCorruption) { - CountingBackend backend; Layout layout("p"); + auto backend = std::make_shared(layout); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation reader = requests.admit(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(RootNamespace{"a"}, UInt128(42)); const String key = layout.refCkptKey(life); const RefCkpt durable{.life_epoch = 9, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - ASSERT_EQ(publishCkpt(backend, layout, life, durable, 1, ALWAYS_ADMITTED, generousDeadline()), - CkptPublishOutcome::Published); - const uint64_t cas_puts_before = backend.casPutCount(key); - - const std::function always_fenced = [](uint64_t admitted) - { - throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "fence generation moved since admission ({})", admitted); - }; + ASSERT_EQ(publishCkpt(reader, layout, life, durable), CkptPublishOutcome::Published); + /// This suite writes one key only, so the backend's own total is that key's count. + const uint64_t writes_before = backend->writeTotal(); + + /// Admission survives the read and is gone by the time the decrease is classified -- the exact + /// window in which the refusal must be reported as a control signal rather than as corruption. + backend->withdraw_after_read_of = key; + CasOperation superseded_writer = requests.admit([&backend] { return backend->admitted; }); const RefCkpt superseded{.life_epoch = 3, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; - EXPECT_EQ(publishCkpt(backend, layout, life, superseded, 1, always_fenced, generousDeadline()), - CkptPublishOutcome::FencedOut); + EXPECT_EQ(publishCkpt(superseded_writer, layout, life, superseded), CkptPublishOutcome::FencedOut); + backend->withdraw_after_read_of.clear(); - EXPECT_EQ(backend.casPutCount(key), cas_puts_before); - EXPECT_EQ(lifeEpochOrFail(backend, layout, life), 9u); + EXPECT_EQ(backend->writeTotal(), writes_before); + EXPECT_EQ(lifeEpochOrFail(reader, layout, life), 9u); } /// `checkpoint_snapshot_id` and `last_epoch_seal` continue to merge by SEMANTIC MAXIMUM. Unlike @@ -491,9 +516,11 @@ TEST(CASRefCheckpointJoin, EncodedCkptSizeIsIndependentOfCardinality) /// instead of racing it (see `openPool`'s doc comment). auto store = openPool(backend, [] { return uint64_t{0}; }); Layout layout("p"); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); - const String one = encodedCkptOfNamespaceWithRefs(store, *backend, layout, RootNamespace{"srv1/one"}, 1); - const String many = encodedCkptOfNamespaceWithRefs(store, *backend, layout, RootNamespace{"srv1/many"}, MANY_REFS); + const String one = encodedCkptOfNamespaceWithRefs(store, op, layout, RootNamespace{"srv1/one"}, 1); + const String many = encodedCkptOfNamespaceWithRefs(store, op, layout, RootNamespace{"srv1/many"}, MANY_REFS); ASSERT_FALSE(one.empty()); ASSERT_FALSE(many.empty()); diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index a81339967eaa..b9cc21481d3c 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -69,27 +69,38 @@ PoolPtr openPool(const BackendPtr & backend) } /// The fence-controlled pool of `gtest_cas_ref_install_safety.cpp`, for the pre-attempt refusal: the -/// boot clock is frozen so `setMountDeadline` alone decides both fence predicates, renewal is parked an -/// hour out so nothing re-arms the deadline underneath the test, and the single-attempt budget makes -/// `attempt_timeout_ms + lease_safety_margin_ms` (200 ms) the window between "the flush is admitted" -/// and "an attempt may start". -PoolPtr openPoolFenceControlled(const BackendPtr & backend) +/// boot clock is frozen so `setMountDeadline` alone decides every fence predicate, renewal is parked an +/// hour out so nothing re-arms the deadline underneath the test, and the backend reports the budget's +/// own `attempt_timeout_ms` because that -- not the budget field -- is what the request engine reserves +/// per attempt, exactly as `ContentAddressedMetadataStorage` pairs the two in production. No fault is +/// injected here at all -- the refusal comes from the lease having no room to start a write -- so +/// nothing in this fixture depends on an attempt count. +PoolPtr openPoolFenceControlled(const std::shared_ptr & backend) { DB::Cas::tests::seedPoolMetaForRestart(*backend); PoolConfig cfg{.pool_prefix = "p", .server_root_id = "test"}; cfg.boot_ms_fn = [] { return uint64_t{0}; }; cfg.mount_renew_period = std::chrono::milliseconds{3600000}; CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) + budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); return Pool::open(backend, cfg); } constexpr uint64_t FENCE_DEADLINE_HEALTHY_MS = 30000; -constexpr uint64_t FENCE_DEADLINE_REFUSES_ATTEMPT_MS = 100; +/// Between "the flush is admitted" and "an attempt may start" sit three gates with different +/// appetites, all measured against the lease's remaining time (the frozen clock at 0 makes the +/// deadline BE the remaining time), and each refuses until its own reservation plus +/// `lease_safety_margin_ms` (100) is STRICTLY cleared: +/// a `CasOperation::admitted` guard reserves nothing -- clears above 100; +/// a read reserves one attempt envelope -- clears above 200; +/// a write reserves TWO, the attempt and the read that settles it -- clears above 300. +/// This test wants the guards and the reads on the way in to pass while the append's own first +/// request is refused, so it sits strictly between the second and the third. +constexpr uint64_t FENCE_DEADLINE_REFUSES_ATTEMPT_MS = 250; /// A bare `Pool::open` with no `_pool_meta` seeded: the path an operator's pool RECREATION takes, and /// the only one that runs the bootstrap residual + quiesce gates (`seedPoolMetaForRestart` mints the @@ -290,18 +301,20 @@ TEST(CASRefContiguousAlloc, NonSuccessorIdIsRejectedOnApply) TEST(CASPoolMeta, GcShardsIsPersistedAndOverridesMismatchedReopenConfig) { - InMemoryBackend backend; + auto backend = std::make_shared(); const Layout layout("p"); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); const PoolMeta created = PoolMeta::createOrValidate( - backend, layout, /*blob_header_len=*/256, /*gc_shards=*/4, + op, layout, /*blob_header_len=*/256, /*gc_shards=*/4, BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/true); EXPECT_EQ(created.gc_shards, 4u); const PoolMeta reopened = PoolMeta::createOrValidate( - backend, layout, /*blob_header_len=*/256, /*gc_shards=*/1, + op, layout, /*blob_header_len=*/256, /*gc_shards=*/1, BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/false); EXPECT_EQ(reopened.gc_shards, 4u); - EXPECT_EQ(decodePoolMeta(backend.get(layout.poolMetaKey())->bytes).gc_shards, 4u); + EXPECT_EQ(decodePoolMeta(op.read(layout.poolMetaKey(), Retry::standard())->bytes).gc_shards, 4u); } /// The one path where "an attempt that provably sent nothing consumes nothing" does not hold, and the @@ -347,7 +360,9 @@ TEST(CASRefContiguousAlloc, NeedsRecoveryReplaysBeforeAllocatingTheNextId) /// The durable stream itself is dense: `1`, `2`, `3` all exist as objects. `ns` was born through /// the REAL append lane (Stage B Task 4-C), so its objects sit at a real catalog-minted incarnation, /// not the Stage-A sentinel -- resolve it the same way production discovery does. - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns).value(); + CasRequests catalog_requests(backend, Fence::open()); + CasOperation catalog_op = catalog_requests.admit(); + const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(catalog_op, store->layout(), ns).value(); for (uint64_t seq = 1; seq <= 3; ++seq) EXPECT_TRUE(backend->head(store->layout().refLogKey(life, RefTxnId{epoch, seq})).exists) << "log object " << epoch << "-" << seq << " must exist: the durable stream has no hole"; diff --git a/src/Disks/tests/gtest_cas_ref_gc.cpp b/src/Disks/tests/gtest_cas_ref_gc.cpp index 97d637982af6..0d34129d2d81 100644 --- a/src/Disks/tests/gtest_cas_ref_gc.cpp +++ b/src/Disks/tests/gtest_cas_ref_gc.cpp @@ -91,8 +91,12 @@ bool blobPresent(Backend & b, const Layout & layout, const UInt128 & hash) class DeposeRoundCommitBackend : public InMemoryBackend { public: - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + /// The fault sits on the WRITE PRIMITIVE, not the legacy `casPut` verb: `Gc::runRegularRound` + /// speaks the primitive directly, and `casPut`'s forwarding is one-way -- overriding it here would + /// intercept nothing. + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (arm && key == "p/gc/state") { @@ -105,7 +109,7 @@ class DeposeRoundCommitBackend : public InMemoryBackend "test-injected: round-commit gc/state CAS denied (losing leader deposed mid-round)"); } } - return InMemoryBackend::casPut(key, bytes, expected, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } bool arm = false; }; @@ -139,17 +143,23 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend armed = true; } - HeadResult head(const String & key) override + /// The primitive `head` override below hides every base overload of that name. + using CountingBackend::head; + + /// Both seams hang off the transport primitives, so they fire whichever surface the cleanup pass + /// reaches the store through. + std::optional head(const String & key, DB::Cas::TransportAccess & access) override { - HeadResult result = CountingBackend::head(key); + auto result = CountingBackend::head(key, access); if (armed && timing == Timing::BeforeFirstDelete && key == first_cleanup_key) moveAuthority(); return result; } - DeleteOutcome deleteExact(const String & key, const Token & token) override + DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, + DB::Cas::TransportAccess & access) override { - DeleteOutcome result = CountingBackend::deleteExact(key, token); + auto result = CountingBackend::remove(key, expected_value, access); if (armed && timing == Timing::AfterFirstDelete && key == first_cleanup_key) moveAuthority(); return result; @@ -551,13 +561,15 @@ TEST(CASRefGc, RefObjectCleanupRetainsCheckpointPredecessorSealProof) EXPECT_TRUE(backend->head(layout.refLogKey(life, seal_id)).exists) << "cleanup must retain the predecessor seal that proves the checkpoint base's epoch transition"; - const CasRefCatalog::Snapshot cut = CasRefCatalog::read(*backend, layout); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot cut = CasRefCatalog::read(op, layout); const auto entry = std::find_if(cut.catalog.entries.begin(), cut.catalog.entries.end(), [&](const CatalogEntry & candidate) { return candidate.ns == ns; }); ASSERT_NE(entry, cut.catalog.entries.end()); - const std::optional checkpoint = readCkpt(*backend, layout, life); + const std::optional checkpoint = readCkpt(op, layout, life); ASSERT_TRUE(checkpoint); - EXPECT_NO_THROW((void)recoverRefTableDetailedFromAuthority(*backend, layout, *entry, checkpoint->ckpt)); + EXPECT_NO_THROW((void)recoverRefTableDetailedFromAuthority(op, layout, *entry, checkpoint->ckpt)); } TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBeforeFirstDeleteRefusesEveryRefObjectDelete) @@ -713,16 +725,18 @@ TEST(CASRefGc, RefSnaplogLifecycleE2E) /// The writer's compaction: a snapshot of ns_a covering its greatest log (va2), the same /// deterministic bytes the oracle recomputes. - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*backend, layout); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); const RefTableState sa = recoverRefTableDetailedAtCatalogCutForTest(*backend, layout, catalog_cut, ns_a).state; writeRefSnapshotRaw(*backend, layout, snapshotOf(sa, ns_a.string())); const NamespaceLifeId life_a = store->namespaceLife(ns_a); - const CkptSample before_snapshot_publish = *readCkpt(*backend, layout, life_a); + const CkptSample before_snapshot_publish = *readCkpt(op, layout, life_a); RefCkpt after_snapshot_publish = before_snapshot_publish.ckpt; after_snapshot_publish.checkpoint_snapshot_id = RefTxnId{1, va2}; - ASSERT_EQ(backend->casPut( - layout.refCkptKey(life_a), encodeRefCkpt(after_snapshot_publish), before_snapshot_publish.token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative(op.replace( + layout.refCkptKey(life_a), encodeRefCkpt(after_snapshot_publish), + before_snapshot_publish.incarnation, Retry::standard()))); Gc gc(store, kGc); runToFixpoint(store, gc); @@ -973,7 +987,9 @@ TEST(CASRefGc, CatalogAdmittedFreshLifeWithoutParentSeedsSuccessorSeal) const RootNamespace ns{"00/aa@cas@"}; fixture::admitLive(*backend, layout, ns); - const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(*backend, layout); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); ASSERT_EQ(catalog_cut.catalog.entries.size(), 1u); const UInt128 life_id = catalog_cut.catalog.entries.front().incarnation; diff --git a/src/Disks/tests/gtest_cas_ref_install_safety.cpp b/src/Disks/tests/gtest_cas_ref_install_safety.cpp index b44ce523465c..fc215264f398 100644 --- a/src/Disks/tests/gtest_cas_ref_install_safety.cpp +++ b/src/Disks/tests/gtest_cas_ref_install_safety.cpp @@ -8,16 +8,22 @@ #include #include #include +#include #include #include #include +#include + +#include #include #include #include #include #include #include +#include +#include #include /// Task 3 (spec §A1, site 1): the region of `CasRefLedger::commitRefChunk` between "this chunk's @@ -60,57 +66,91 @@ PoolPtr openPool(const BackendPtr & backend) return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); } -/// As `openPool`, but with a SINGLE-attempt request budget, which is what makes one ambiguous `PUT` -/// conclusive: with retries allowed the controller's resolve-before-reissue would either re-`PUT` (the -/// object never landed) or prove the object durable (it did) and report `Committed`, and neither of the -/// wedge arms under test would ever be reached. Same budget shape as -/// `gtest_cas_ref_chunked_flush.cpp`'s `runChunkFailureCase`, including the short timeouts so there is -/// no inter-attempt sleep to serve. -PoolPtr openPoolSingleAttempt(const BackendPtr & backend) +/// As `openPool`, but with the budget that bounds the mount lease's own admission arithmetic: +/// `attempt_timeout_ms` is what one attempt reserves and `lease_safety_margin_ms` the room kept past +/// it, which together decide the two fence predicates the pre-attempt tests below drive. No budget +/// field bounds a write's ATTEMPT COUNT -- that is the `Retry` policy's, and the ref lane's is +/// `standard` -- so what makes an injected fault conclusive in these tests is that it stays armed for +/// the whole call (`LatchedChunkFaultBackend`) while `VirtualRetryClock` carries the call to its own +/// deadline. +PoolPtr openPoolWedgeBudget(const BackendPtr & backend) { DB::Cas::tests::seedPoolMetaForRestart(*backend); PoolConfig cfg{.pool_prefix = "p", .server_root_id = "test"}; CasRequestBudget budget; - /// ONE attempt is the whole mechanism these tests need: it is what turns an injected lost - /// acknowledgement into `Unresolved` instead of a transparent retry, and it does so independently of - /// how fast the machine is. - /// - /// The operation deadline must therefore NOT sit at `attempt_timeout_ms`, which is where it used to. - /// The controller's pre-send gate (`putIfAbsentControlled`: `now + attempt_timeout > deadline` - /// returns `Unresolved` WITHOUT sending) is then a zero-width race that passes only if no - /// millisecond tick elapses between the deadline capture and the gate. Under parallel-build load it - /// loses: the gate fires first, nothing is sent, the injected fault is never reached, and the flush - /// fails CLEAN -- so the product correctly does NOT wedge the lane and the wedge expectations flip. - /// `UncertainPrecommitKeepsItsCleanupOwnerAndItsBody` was observed failing exactly that way (Task 9, - /// `refLaneWedgedForTest` false at the wedge assertion), and every test on this fixture carries the - /// same razor. Same root cause and same fix as `8f9e63c7a19` for the sweep-interruption test. - /// - /// A WIDE deadline keeps the request always actually sent, so the injected fault decides the outcome - /// rather than the scheduler. Tests that want the pre-send REFUSAL instead use - /// `openPoolFenceControlled`, where a frozen clock makes that refusal deterministic rather than raced. - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; + budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; return Pool::open(backend, cfg); } +/// The engine reissues an unresolved write until its OWN retry window closes, and that window is +/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine +/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline +/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling +/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the +/// closures own the clock, so it outlives everything that can still read it. +class VirtualRetryClock +{ +public: + static std::shared_ptr installOn(const PoolPtr & store) + { + auto clock = std::make_shared(); + store->setCasRequestNowFnForTest([clock] { return clock->nowMs(); }); + store->setCasRetrySleepForTest([clock](uint64_t ms) { clock->advance(ms); }); + return clock; + } + + uint64_t nowMs() const + { + std::lock_guard lock(mutex); + return now_ms; + } + size_t pauseCount() const + { + std::lock_guard lock(mutex); + return pauses; + } + uint64_t longestPause() const + { + std::lock_guard lock(mutex); + return longest_pause; + } + + void advance(uint64_t ms) + { + std::lock_guard lock(mutex); + /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move + /// would leave the loop reissuing for ever against a fault that never clears. + now_ms += ms + 1; + ++pauses; + longest_pause = std::max(longest_pause, ms); + } + +private: + mutable std::mutex mutex; + uint64_t now_ms = 0; + size_t pauses = 0; + uint64_t longest_pause = 0; +}; + +using DB::Cas::tests::LatchedChunkFaultBackend; + /// The mount-fence deadlines the pre-attempt tests drive, in the FROZEN boot clock of /// `openPoolFenceControlled` (which is pinned at 0, so these are also the remaining lease budgets). /// -/// `CasMountRuntime` has TWO fence predicates and they are deliberately not the same: -/// `mayMutate` -- `now < deadline`; the top-of-flush gate in `flushRefBatch`. -/// `refAppendFenceOk` -- additionally `attempt_timeout_ms + lease_safety_margin_ms < deadline - now`, -/// i.e. "there is room for one whole controlled attempt"; the `fence_ok` -/// `commitRefChunk` hands to `putIfAbsentControlled`. -/// With `openPoolFenceControlled`'s budget below that margin is 100 + 100 = 200 ms, so a 100 ms -/// remaining lease sits BETWEEN them: the flush is admitted and then its very first pre-attempt gate -/// refuses. That is -/// exactly the production shape of a lease too short to start a write, not a lost -/// one), and it needs no fault injection at all -- which is the point: nothing is sent. +/// `mayMutate` -- `now < deadline` -- is the top-of-flush gate in `flushRefBatch`. Behind it the fence +/// is asked again by everything the flush issues, and each of those refuses until its own reservation +/// plus `lease_safety_margin_ms` (100) is STRICTLY cleared: +/// a `CasOperation::admitted` guard (e.g. `namespaceLife`'s "resident namespace life" one) reserves +/// nothing -- clears above 100; +/// a read reserves one attempt envelope -- clears above 200; +/// a write reserves TWO, the attempt and the read that settles it -- clears above 300. +/// A "pre-attempt refusal" test wants the flush admitted and everything on the way in to pass while +/// the append's own first request is refused, so it sits strictly between the second and the third. constexpr uint64_t FENCE_DEADLINE_HEALTHY_MS = 30000; -constexpr uint64_t FENCE_DEADLINE_REFUSES_ATTEMPT_MS = 100; +constexpr uint64_t FENCE_DEADLINE_REFUSES_ATTEMPT_MS = 250; /// A legal blob-free part: stage an empty manifest, precommit, promote -- enough to drive real /// ref-log transactions through the append lane. @@ -125,7 +165,7 @@ void publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const String build->promote(ns, ref, build->buildId(), id); } -/// As `openPoolSingleAttempt`, but with the mount fence under the TEST's control instead of the wall +/// As `openPoolWedgeBudget`, but with the mount fence under the TEST's control instead of the wall /// clock's: /// - the boot clock is FROZEN at 0, so `setMountDeadline` alone decides both fence predicates and no /// elapsed real time can flip one of them mid-test (the same load-bearing injection, for the same @@ -133,25 +173,28 @@ void publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const String /// - lease renewal is parked an hour out, so the runtime-owned renewal worker cannot re-arm the deadline /// underneath a test that just shortened it. Ten seconds (the default) would be enough in practice /// and flaky in principle; this removes the race rather than betting on it. -PoolPtr openPoolFenceControlled(const BackendPtr & backend) +PoolPtr openPoolFenceControlled(const std::shared_ptr & backend) { DB::Cas::tests::seedPoolMetaForRestart(*backend); PoolConfig cfg{.pool_prefix = "p", .server_root_id = "test"}; cfg.boot_ms_fn = [] { return uint64_t{0}; }; cfg.mount_renew_period = std::chrono::milliseconds{3600000}; CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) + budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field; production pairs the two in `ContentAddressedMetadataStorage`, and a fixture that sets + /// only the budget leaves the engine reserving nothing and no pre-attempt gate to refuse. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); return Pool::open(backend, cfg); } /// Runs `f`, requires it to throw the ref lane's retry-later condition, and returns the message so a -/// caller can assert WHICH condition it was. The message is the only place the `CasUnresolvedReason` -/// surfaces -- there is no accessor for it, by design (it is a diagnostic, not state) -- so this is how -/// a test proves the reason actually reached the decision site instead of defaulting. +/// caller can assert WHICH condition it was. The message is the only place the lane's own reading of a +/// give-up surfaces -- there is no accessor for it, by design (it is a diagnostic, not state) -- so +/// this is how a test proves the verdict reached the decision site instead of defaulting. String retryLaterMessageOf(const std::function & f) { try @@ -167,6 +210,41 @@ String retryLaterMessageOf(const std::function & f) return {}; } +/// `retryLaterMessageOf` with the fault held armed for the whole call, and with the give-up proven to +/// be the call's OWN retry window: the write engine settles each ambiguity by an exact read and then +/// reissues, so a bounded fault would be outlived and the write would commit. The pacing assertions +/// are what make a fixture whose sleep seam is not wired fail rather than sleep the window out for +/// real. +String wedgingRetryLaterMessageOf(VirtualRetryClock & clock, LatchedChunkFaultBackend & backend, + const std::function & f) +{ + const size_t pauses_before = clock.pauseCount(); + const uint64_t clock_before = clock.nowMs(); + backend.latched = true; + const String message = retryLaterMessageOf(f); + /// Disarmed COMPLETELY, not just unlatched: what every caller does next is a flush that must reach + /// the store normally -- the wedge resolution, or an abandon. A topped-up count or a still-armed + /// lost read would fault that one too, and a wedge resolution whose settling read fails does not + /// resolve anything. + backend.latched = false; + backend.mode = LatchedChunkFaultBackend::Mode::None; + backend.fault_count = 0; + backend.fault_skip = 0; + backend.fail_read_once_key.clear(); + EXPECT_GT(clock.pauseCount(), pauses_before + 1) + << "the reissues must pace through the injected sleep, never a real one"; + EXPECT_LE(clock.longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; + EXPECT_GE(clock.nowMs() - clock_before, 60000u) + << "the give-up must be the call's own retry window, not a pre-attempt refusal"; + return message; +} + +void driveToTheWedge(VirtualRetryClock & clock, LatchedChunkFaultBackend & backend, + const std::function & f) +{ + (void)wedgingRetryLaterMessageOf(clock, backend, f); +} + /// Installs a ONE-SHOT throwing probe into the post-durable install regions (spec §A2): the next region /// entered throws, every later one runs normally -- which is what lets a terminality test drive a /// successful flush after the recovery transition. @@ -243,8 +321,9 @@ TEST(CASRefInstallSafety, PostDurableInstallIsAllocationFree) /// counter must NOT advance -- an unproven transaction is not a recorded one. TEST(CASRefInstallSafety, UnresolvedAlwaysRecordsTheWedge) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/unresolved_wedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -257,7 +336,8 @@ TEST(CASRefInstallSafety, UnresolvedAlwaysRecordsTheWedge) backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - const String message = retryLaterMessageOf([&] { publishEmptyPart(store, ns, "part_a"); }); + const String message = wedgingRetryLaterMessageOf(*clock, *backend, + [&] { publishEmptyPart(store, ns, "part_a"); }); EXPECT_TRUE(store->refLaneWedgedForTest(ns)) << "an Unresolved PUT must always leave a wedge"; const String wedged_key = store->wedgedKeyForTest(ns); @@ -272,9 +352,9 @@ TEST(CASRefInstallSafety, UnresolvedAlwaysRecordsTheWedge) /// backend's `putIfAbsent`, so the request reached it), the single-attempt budget is then spent, and /// the lane wedges. The message must say so -- and must NOT say "no attempt was sent", which is the /// only shape allowed to skip the wedge. - EXPECT_NE(message.find("attempt budget was exhausted"), String::npos) + EXPECT_NE(message.find("is UNCERTAIN"), String::npos) << "the reason must reach the wedge message rather than defaulting: " << message; - EXPECT_EQ(message.find("no attempt was sent"), String::npos) + EXPECT_EQ(message.find("BEFORE any request was sent"), String::npos) << "an ambiguous PUT is not a pre-attempt refusal: " << message; } @@ -314,7 +394,7 @@ TEST(CASRefInstallSafety, PreAttemptRefusalDoesNotWedgeTheLane) const String message = retryLaterMessageOf([&] { store->dropRef(ns, "part_a"); }); - EXPECT_NE(message.find("no attempt was sent"), String::npos) + EXPECT_NE(message.find("refused BEFORE any request was sent"), String::npos) << "the caller must be told WHY, and this is the reason the no-wedge decision rests on: " << message; EXPECT_FALSE(store->refLaneWedgedForTest(ns)) << "nothing was sent, so nothing can be durable: there is no ambiguity for a wedge to resolve"; @@ -352,8 +432,9 @@ TEST(CASRefInstallSafety, PreAttemptRefusalDoesNotWedgeTheLane) /// that never can. TEST(CASRefInstallSafety, PreAttemptRefusalAfterAWedgeResolutionLeavesTheLaneClean) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPoolFenceControlled(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/pre_attempt_after_unwedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -364,12 +445,12 @@ TEST(CASRefInstallSafety, PreAttemptRefusalAfterAWedgeResolutionLeavesTheLaneCle publishEmptyPart(store, ns, "y"); const size_t tail_after_seed = store->tailSinceSnapshotCountForTest(ns); - /// Wedge over an object that IS durable: the write lands, its acknowledgement is lost, and the - /// controller's own verifying read is lost too (the only mode that reaches the resolution install). + /// Wedge over an object that IS durable: the write lands, its acknowledgement is lost, and every + /// settling read of the key is lost too (the only mode that reaches the resolution install). backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_after_seed); @@ -383,7 +464,7 @@ TEST(CASRefInstallSafety, PreAttemptRefusalAfterAWedgeResolutionLeavesTheLaneCle const String message = retryLaterMessageOf([&] { store->dropRef(ns, "y"); }); store->setRefPreCarveHookForTest(nullptr); - EXPECT_NE(message.find("no attempt was sent"), String::npos) << message; + EXPECT_NE(message.find("refused BEFORE any request was sent"), String::npos) << message; EXPECT_FALSE(store->refLaneWedgedForTest(ns)) << "the wedge that existed was RESOLVED, and the chunk that followed it was never sent -- the " "lane must be left clean, not re-wedged over an id that can never resolve"; @@ -407,8 +488,9 @@ TEST(CASRefInstallSafety, PreAttemptRefusalAfterAWedgeResolutionLeavesTheLaneCle /// lane must wedge again, now over the NEW transaction. TEST(CASRefInstallSafety, AmbiguousChunkAfterAWedgeResolutionRewedgesTheLane) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openPoolFenceControlled(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/ambiguous_after_unwedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -422,19 +504,19 @@ TEST(CASRefInstallSafety, AmbiguousChunkAfterAWedgeResolutionRewedgesTheLane) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); const String first_wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(first_wedged_key.empty()); /// The resolution is a conditional CREATE at the wedged key now, and that key already holds our - /// own landed object, so it conflicts and the follow-up read adopts it (`LandedThenLost`'s one-shot - /// lost read was consumed inside the previous attempt, so this read succeeds). `fault_skip` lets - /// that create through and puts the fault on this flush's OWN chunk PUT, which is the subject. + /// own landed object, so it conflicts and the settling read adopts it -- the lost-read leg is + /// disarmed above, so that read succeeds. `fault_skip` lets that create through and puts the fault + /// on this flush's OWN chunk PUT, which is the subject. backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_skip = 1; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "y"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "y"); }); EXPECT_TRUE(store->refLaneWedgedForTest(ns)) << "an attempt was sent for the new chunk, so its object may be durable: the lane must wedge"; @@ -446,39 +528,88 @@ TEST(CASRefInstallSafety, AmbiguousChunkAfterAWedgeResolutionRewedgesTheLane) << "a wedged lane may hold a durable transaction the runtime has not recorded"; } -/// Task 18's regression guard, asserted on the mapping itself rather than through six pieces of fault -/// choreography. `unresolvedProvesNothingWasSent` is the whole decision: the ledger wedges unless it -/// answers true, so this table IS the protocol. +/// The whole wedge decision, as one table. `GaveUp::sent_any` is what the ledger branches on -- it +/// returns the attempt to `Ready` when nothing was sent and wedges otherwise -- so this table IS the +/// protocol. Each row is DRIVEN rather than asserted about a mapping: the point of the old enum-shaped +/// version was that a value could be listed without any way to reach it. /// -/// What protects a future contributor who adds a `CasUnresolvedReason` member and forgets this file: -/// the predicate is a switch with NO `default`, so the addition is a `-Wswitch` build error (a forced -/// decision, not a silent one), and its trailing `return false` makes the runtime answer "wedge" even -/// if that diagnostic is ever suppressed. Both directions fail closed; neither can widen the allow-list -/// by omission. The `static_assert`s make the mapping a compile-time fact, and the `EXPECT`s repeat it -/// so a break names the offending value in the test report. -TEST(CASRefInstallSafety, OnlyNoAttemptSentMaySkipTheWedge) -{ - static_assert(unresolvedProvesNothingWasSent(CasUnresolvedReason::NoAttemptSent)); - static_assert(!unresolvedProvesNothingWasSent(CasUnresolvedReason::NotUnresolved)); - static_assert(!unresolvedProvesNothingWasSent(CasUnresolvedReason::FenceLostMidWay)); - static_assert(!unresolvedProvesNothingWasSent(CasUnresolvedReason::DeadlineMidWay)); - static_assert(!unresolvedProvesNothingWasSent(CasUnresolvedReason::FenceLostPostWrite)); - static_assert(!unresolvedProvesNothingWasSent(CasUnresolvedReason::AttemptsExhausted)); - - EXPECT_TRUE(unresolvedProvesNothingWasSent(CasUnresolvedReason::NoAttemptSent)) - << "the pre-attempt gates rejected before the first request: the key is provably unwritten"; - /// `NotUnresolved` is reachable at the decision site if any path ever returns `Unresolved` without - /// recording a reason, so it is listed here as a real case, not as enum hygiene. - EXPECT_FALSE(unresolvedProvesNothingWasSent(CasUnresolvedReason::NotUnresolved)) - << "an unrecorded reason proves nothing and must keep wedging"; - EXPECT_FALSE(unresolvedProvesNothingWasSent(CasUnresolvedReason::FenceLostMidWay)) - << "an attempt was already sent: its object may be durable"; - EXPECT_FALSE(unresolvedProvesNothingWasSent(CasUnresolvedReason::DeadlineMidWay)) - << "an attempt was already sent: its object may be durable"; - EXPECT_FALSE(unresolvedProvesNothingWasSent(CasUnresolvedReason::FenceLostPostWrite)) - << "the attempt COMMITTED and only the fence was lost afterwards -- the most durable case of all"; - EXPECT_FALSE(unresolvedProvesNothingWasSent(CasUnresolvedReason::AttemptsExhausted)) - << "every attempt is a candidate for having landed"; +/// `sent_any` is set on the line before the attempt goes out, so exactly one shape can report it false: +/// every gate refused before the first request. The four rows below it are the ways a call can end +/// AFTER something reached the network, and each leaves an object that may be durable. +TEST(CASRefInstallSafety, OnlySendingNothingMaySkipTheWedge) +{ + /// Row 1: the caller's own facts refuse before the attempt. Nothing reaches the store. + { + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit([] { return false; }); + const WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_FALSE(gave_up->sent_any) + << "the pre-attempt gates rejected before the first request: the key is provably unwritten"; + EXPECT_EQ(backend->writeTotal(), 0u); + } + + /// Row 2: the OTHER pre-attempt refusal. Same verdict, a different bound. + { + auto backend = std::make_shared(); + uint64_t clock = 0; + CasRequests requests(backend, Fence::open(), + [&clock]() -> uint64_t { const uint64_t t = clock; clock += 1000; return t; }); + CasOperation op = requests.admit(); + const WriteResult result = op.create("k", "v", Retry::within(500)); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(backend->writeTotal(), 0u); + } + + /// Row 3: an attempt was sent and its outcome never settled. + { + auto backend = std::make_shared(); + backend->injectAmbiguousWrite("k"); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + const WriteResult result = op.create("k", "v", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any) << "an attempt was already sent: its object may be durable"; + } + + /// Row 4: the attempt COMMITTED and only the admission was lost afterwards -- the most durable case + /// of all, and the one a caller is most tempted to report as success. + { + auto backend = std::make_shared(); + bool live = true; + backend->onWriteCommitted("k", [&live] { live = false; }); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit([&live] { return live; }); + const WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_EQ(backend->writeTotal(), 1u) << "and the object IS there"; + } + + /// Row 5: the deadline arrives AFTER an attempt rather than before one. The clock is moved by a + /// hook that runs inside the write, so the first attempt is admitted and the settling read is not. + { + auto backend = std::make_shared(); + uint64_t clock = 0; + backend->onBeforeWrite("k", [&clock] { clock = 100'000; }); + backend->injectAmbiguousWrite("k"); + CasRequests requests(backend, Fence::open(), [&clock]() -> uint64_t { return clock; }); + CasOperation op = requests.admit(); + const WriteResult result = op.create("k", "v", Retry::within(1000)); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_TRUE(gave_up->sent_any) << "an attempt was already sent: its object may be durable"; + } } /// Task 5 (spec §A1, site 2). Resolving a wedge is a post-durable install too: the resolving GET PROVES @@ -495,8 +626,9 @@ TEST(CASRefInstallSafety, OnlyNoAttemptSentMaySkipTheWedge) /// bumped once per install, so a re-applied transaction shows up as one extra. TEST(CASRefInstallSafety, WedgeResolutionInstallsExactlyOnce) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_resolution"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -509,13 +641,13 @@ TEST(CASRefInstallSafety, WedgeResolutionInstallsExactlyOnce) /// whole test is a handful of tiny transactions, so every delta below is exact. const size_t tail_after_seed = store->tailSinceSnapshotCountForTest(ns); - /// Drop "x" through a PUT that LANDS and then loses its response, plus the one-shot lost read that - /// keeps the controller's own resolve-before-reissue from settling it inside the same attempt. One - /// attempt, so the lane wedges over an object that is genuinely durable -- the only way in. + /// Drop "x" through a PUT that LANDS and then loses its response, plus the lost settling read that + /// keeps the engine from proving the commit inside the same call. Both stay armed for the whole + /// call, so the lane wedges over an object that is genuinely durable -- the only way in. backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)) << "the lost-response drop must wedge the lane"; ASSERT_FALSE(store->wedgedKeyForTest(ns).empty()); @@ -631,8 +763,9 @@ TEST(CASRefInstallSafety, WritingOwnsTheAttemptUntilInstall) /// Ambiguity transfers the same exact attempt from `Writing` to `Wedged`. TEST(CASRefInstallSafety, UnresolvedTransfersWritingToWedged) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/apply_state_wedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -643,7 +776,7 @@ TEST(CASRefInstallSafety, UnresolvedTransfersWritingToWedged) backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishEmptyPart(store, ns, "part_a"); }); + driveToTheWedge(*clock, *backend, [&] { publishEmptyPart(store, ns, "part_a"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged) @@ -653,8 +786,9 @@ TEST(CASRefInstallSafety, UnresolvedTransfersWritingToWedged) /// Durable resolution installs the attempt and returns the lane to `Ready`. TEST(CASRefInstallSafety, WedgeResolutionReturnsReady) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/apply_state_unwedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -665,11 +799,11 @@ TEST(CASRefInstallSafety, WedgeResolutionReturnsReady) publishEmptyPart(store, ns, "y"); /// The one mode that wedges over a GENUINELY durable object (see `ChunkFaultBackend`): the write - /// lands, its acknowledgement is lost, and the controller's own verifying read is lost too. + /// lands, its acknowledgement is lost, and every settling read of the key is lost too. backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged); @@ -684,8 +818,8 @@ TEST(CASRefInstallSafety, WedgeResolutionReturnsReady) /// A foreign occupant is a terminal `Faulted` verdict. TEST(CASRefInstallSafety, ConclusiveForeignConflictFaultsTheLane) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); const RootNamespace ns{"srv1/apply_state_conflict"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -711,8 +845,8 @@ TEST(CASRefInstallSafety, DefiniteFailureReturnsReady) #if !USE_AWS_S3 GTEST_SKIP() << "DefiniteFailure classification requires S3 error types (USE_AWS_S3 off)"; #else - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); const RootNamespace ns{"srv1/apply_state_definite"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -743,8 +877,9 @@ TEST(CASRefInstallSafety, DefiniteFailureReturnsReady) /// `CASAnomalyPolicy.ForeignBytesAtWedgeKeyTripFenceAndRemount`'s. TEST(CASRefInstallSafety, WedgeResolutionProvenForeignFaultsTheLane) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/apply_state_foreign_wedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -757,7 +892,7 @@ TEST(CASRefInstallSafety, WedgeResolutionProvenForeignFaultsTheLane) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged); /// Out of band, a foreign writer lands DIFFERENT bytes at the exact wedged key. The fault mode is @@ -804,8 +939,9 @@ TEST(CASRefInstallSafety, PostDurableInstallFailureRequiresRecovery) /// would have cleared it is in the same region that threw. TEST(CASRefInstallSafety, WedgeResolutionInstallFailureRequiresRecovery) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/apply_state_poison_unwedge"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -818,7 +954,7 @@ TEST(CASRefInstallSafety, WedgeResolutionInstallFailureRequiresRecovery) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::None; @@ -886,8 +1022,9 @@ TEST(CASRefInstallSafety, NeedsRecoveryReplaysBeforeALaterFlush) /// still live once the wedge resolves. TEST(CASRefInstallSafety, UncertainPrecommitKeepsItsCleanupOwnerAndItsBody) { - auto backend = std::make_shared(); - auto store = openPoolSingleAttempt(backend); + auto backend = std::make_shared(); + auto store = openPoolWedgeBudget(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/uncertain_precommit"}; /// Stage B (Task 4-C): pin `ns` to the Stage-A sentinel BEFORE its first real touch, so /// the fault injected below (computed from that same sentinel) lands on the key production @@ -903,12 +1040,12 @@ TEST(CASRefInstallSafety, UncertainPrecommitKeepsItsCleanupOwnerAndItsBody) ASSERT_TRUE(backend->head(manifest_key).exists) << "the staged body must exist before the precommit"; /// Scoped to THIS namespace's ref log so the manifest body's own PUT cannot consume the fault. The - /// object LANDS and only its acknowledgement is lost, which with the single-attempt budget wedges - /// the lane over a genuinely durable precommit -- the exact shape the old code mishandled. + /// object LANDS and only its acknowledgement is lost, and no settling read can prove otherwise, so + /// the lane wedges over a genuinely durable precommit -- the exact shape the old code mishandled. backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { build->precommitAdd(ns, "part_a", id); }); + driveToTheWedge(*clock, *backend, [&] { build->precommitAdd(ns, "part_a", id); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)) << "the lost-response precommit must wedge the lane"; EXPECT_EQ(build->precommitState(), PartWriteTxn::PrecommitState::Uncertain) << "an append that may have landed is neither 'never precommitted' nor 'durably precommitted'"; diff --git a/src/Disks/tests/gtest_cas_ref_protocol.cpp b/src/Disks/tests/gtest_cas_ref_protocol.cpp new file mode 100644 index 000000000000..93c71358cd0c --- /dev/null +++ b/src/Disks/tests/gtest_cas_ref_protocol.cpp @@ -0,0 +1,51 @@ +#include + +#include +#include "cas_test_helpers.h" + +#include + +using namespace DB::Cas; + +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::FakeClock; + +namespace +{ + +CasRequests makeRequests(BackendPtr backend, FakeClock & clock, Fence fence = Fence::open()) +{ + return CasRequests(std::move(backend), std::move(fence), clock.nowFn(), clock.sleepFn()); +} + +} + +TEST(CASRefProtocol, CrossEpochFromSealShortCircuitsWithoutAnyRequest) +{ + FakeClock clock; + auto backend = std::make_shared(); + Layout layout("pool"); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + const RootNamespace ns("t"); + const auto life = DB::Cas::tests::fixture::fixtureLife(ns); + + /// `from_seal == RefTxnId{}`: nothing consumed yet, so there is no seal to cross from -- proved + /// without reading anything. + { + const EpochCrossResult r = crossEpochFromSeal( + op, layout, ns, RefTxnId{}, std::nullopt, RefTxnId{2, 1}, life); + EXPECT_EQ(r.outcome, EpochCrossOutcome::NothingConsumed); + } + + /// The caller already decoded the record at `from_seal` and knows it is not an `EpochSeal` -- + /// also proved without any read here. + { + const EpochCrossResult r = crossEpochFromSeal( + op, layout, ns, RefTxnId{1, 5}, /*seal_proven=*/false, RefTxnId{2, 1}, life); + EXPECT_EQ(r.outcome, EpochCrossOutcome::NotASeal); + } + + EXPECT_EQ(backend->getTotal(), 0u); +} diff --git a/src/Disks/tests/gtest_cas_ref_read_contract.cpp b/src/Disks/tests/gtest_cas_ref_read_contract.cpp index 7917d59231e7..5fa0b77db52c 100644 --- a/src/Disks/tests/gtest_cas_ref_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ref_read_contract.cpp @@ -47,13 +47,22 @@ ManifestId publishRefThroughPool(const PoolPtr & store, const RootNamespace & ns return id; } +/// The `refresh_authority` hook `deleteCompletedRemoving` requires every caller to state explicitly. +/// This fixture's operation carries a direct liveness (`op.admitted()` re-checks the fence itself on +/// every call), not a cached flag, so there is nothing for a refresh to re-read between attempts. +void noAuthorityRefresh() +{ +} + /// Delete the current catalog life through the production exact-removal authority (`casUpdate` to /// `Removing`, then `deleteCompletedRemoving` under a held fence), retaining every old physical byte /// and any already-resident runtime. Mirrors `gtest_cas_ns_file_read_contract.cpp`'s /// `deleteCatalogLife` -- lifecycle-real, not a raw sentinel overwrite. -void deleteCatalogLife(Backend & backend, const Layout & layout, const NamespaceLifeId & life) +void deleteCatalogLife(const BackendPtr & backend, const Layout & layout, const NamespaceLifeId & life) { - CasRefCatalog::casUpdate(backend, layout, [&](const RefCatalog & current) + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find_if(next.entries.begin(), next.entries.end(), [&](const CatalogEntry & entry) @@ -67,7 +76,7 @@ void deleteCatalogLife(Backend & backend, const Layout & layout, const Namespace return next; }); - const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout); const auto it = std::find_if(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == life.ns && entry.incarnation == life.incarnation; @@ -79,9 +88,7 @@ void deleteCatalogLife(Backend & backend, const Layout & layout, const Namespace parent.ref_lives.emplace(life.incarnation, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 1}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); - if (CasRefCatalog::deleteCompletedRemoving( - backend, layout, *it, parent, 1, - [](uint64_t) { return CasRefCatalog::LeaderFenceStatus::Held; }) + if (CasRefCatalog::deleteCompletedRemoving(op, layout, *it, parent, noAuthorityRefresh).outcome != CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Failed to delete fixture catalog life '{}'", life.ns.string()); } @@ -90,13 +97,15 @@ void deleteCatalogLife(Backend & backend, const Layout & layout, const Namespace /// rebirth every test below drives. Mirrors `gtest_cas_ns_file_read_contract.cpp`'s /// `admitReplacementLife`. NamespaceLifeId admitReplacementLife( - Backend & backend, const Layout & layout, uint64_t gc_shards, + const BackendPtr & backend, const Layout & layout, uint64_t gc_shards, const NamespaceLifeId & predecessor, UInt128 successor_incarnation) { if (predecessor.incarnation == successor_incarnation) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Fixture life ids unexpectedly collide"); const NamespaceLifeId successor = NamespaceLifeId::fromCatalogEntry(predecessor.ns, successor_incarnation); - CasRefCatalog::casAdmitEntry(backend, layout, gc_shards, CatalogEntry{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + CasRefCatalog::casAdmitEntry(op, layout, gc_shards, CatalogEntry{ .ns = successor.ns, .state = NsState::Live, .incarnation = successor.incarnation}); return successor; } @@ -131,8 +140,8 @@ TEST(CASRefReadContract, HeldRuntimeAfterSameNameRebirthReadsStaleOrNotFoundNeve /// Drop and re-admit under the SAME logical name, bypassing this store's own ledger entirely -- /// exactly as an independent actor's drop/rebirth would look from this reader's point of view. - deleteCatalogLife(*backend, layout, life1); - const NamespaceLifeId life2 = admitReplacementLife(*backend, layout, store->poolConfig().gc_shards, life1, UInt128{0xabc123}); + deleteCatalogLife(backend, layout, life1); + const NamespaceLifeId life2 = admitReplacementLife(backend, layout, store->poolConfig().gc_shards, life1, UInt128{0xabc123}); ASSERT_NE(life1.incarnation, life2.incarnation); const ManifestRef life2_ref{/*writer_epoch*/ 1, /*build_sequence*/ 777, /*manifest_ordinal*/ 1}; @@ -184,7 +193,7 @@ TEST(CASRefReadContract, HotRefReadsThroughHeldRuntimeIssueZeroCatalogRequests) /// recorder that never saw anything. EXPECT_GT( backend->headCount(layout.refCatalogKey()) + backend->getCount(layout.refCatalogKey()) - + backend->casPutCount(layout.refCatalogKey()), + + backend->putOverwriteCount(layout.refCatalogKey()), 0u) << "the cold admission above must have reached the catalog at least once"; EXPECT_GT(backend->getCount(layout.refCkptKey(life)), 0u) << "the cold recovery above must have read this namespace's own checkpoint at least once"; @@ -197,7 +206,7 @@ TEST(CASRefReadContract, HotRefReadsThroughHeldRuntimeIssueZeroCatalogRequests) EXPECT_EQ(backend->headCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 0u); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 0u); + EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->putCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); /// Stronger than the catalog-only clauses above: a warm ref read is a pure map lookup over the @@ -222,8 +231,8 @@ TEST(CASRefReadContract, StaleLifeDropRefusesAfterRebirthAndNeverTouchesSuccesso ASSERT_TRUE(store->refTableLifeForTest(ns).has_value()); const NamespaceLifeId life1 = *store->refTableLifeForTest(ns); - deleteCatalogLife(*backend, layout, life1); - const NamespaceLifeId life2 = admitReplacementLife(*backend, layout, store->poolConfig().gc_shards, life1, UInt128{0xabc456}); + deleteCatalogLife(backend, layout, life1); + const NamespaceLifeId life2 = admitReplacementLife(backend, layout, store->poolConfig().gc_shards, life1, UInt128{0xabc456}); const ManifestRef life2_ref{/*writer_epoch*/ 1, /*build_sequence*/ 999, /*manifest_ordinal*/ 1}; publishCommittedTransition(*backend, layout, ns, ref_name, std::nullopt, life2_ref); diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index 2b75ae13ada0..d0c98a62bbaf 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -66,6 +66,7 @@ extern const Event CASRefCheckpointPublished; } using namespace DB::Cas; +using DB::Cas::tests::VirtualRetryClock; using DB::Cas::tests::committedRow; using DB::Cas::tests::CountingBackend; using DB::Cas::tests::expectThrowsCode; @@ -83,6 +84,32 @@ ManifestRef manifestRef(uint64_t epoch, uint64_t build_sequence, uint32_t ordina return ManifestRef{epoch, build_sequence, ordinal}; } +/// Fixture observations of durable state run on an OPEN fence: they are not writes a mount admitted, +/// and each owns the `CasRequests` its operation borrows, so none of these hands one back. +std::optional readCkptForTest(const BackendPtr & backend, const Layout & layout, + const NamespaceLifeId & life) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return readCkpt(op, layout, life); +} + +CasRefCatalog::Snapshot readCatalogForTest(const BackendPtr & backend, const Layout & layout) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return CasRefCatalog::read(op, layout); +} + +/// A fixture's own conditional replace, for the races these tests stage by hand. +bool replaceForTest(const BackendPtr & backend, const String & key, const String & bytes, + const Incarnation & expected) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return std::holds_alternative(op.replace(key, bytes, expected, Retry::standard())); +} + /// Make the durable mount immediately reclaimable so a test that deliberately moved the local fence /// generation can drive the production remount boundary without paying a live-lease expiry wait. void fenceOutMountForRemount(Backend & backend, const String & mount_key) @@ -111,35 +138,33 @@ class HidingListBackend : public CountingBackend DB::Cas::tests::seedPoolMetaForRestart(*this); } - using CountingBackend::get; - using CountingBackend::list; - using CountingBackend::putIfAbsent; - using CountingBackend::casPut; - std::set hidden_keys; std::set phantom_list_keys; - /// Every `putIfAbsent` of a key containing this substring throws a PLAIN (non-`DB::Exception`) - /// error, which `classifyConditionalWriteResult` can only ever classify `Unresolved` -- never - /// `DefiniteFailure`. Persistent rather than one-shot on purpose: the subject is what recovery does - /// when the store KEEPS refusing to say whether the write landed. + /// Every CREATING write of a key containing this substring throws a PLAIN (non-`DB::Exception`) + /// error, which is ambiguous by construction -- never a proven refusal. Persistent rather than + /// one-shot on purpose: the subject is what recovery does when the store KEEPS refusing to say + /// whether the write landed. String ambiguous_put_substr; - /// Persistent thrown response for a matching mutable checkpoint CAS. The ref-log PUT has already - /// completed when tests arm this, producing the exact one-successor recovery window. + /// Persistent thrown response for a matching CONDITIONAL replace of the mutable checkpoint. The + /// ref-log create has already completed when tests arm this, producing the exact one-successor + /// recovery window. String ambiguous_cas_substr; int ambiguous_cas_count = 0; - /// Runs after a checkpoint publisher read its expected token but before that publisher presents - /// its CAS. This is the exact window in which another admitted writer can advance the frontier. - std::function &)> before_cas_put; + /// Runs after a checkpoint publisher read the incarnation it expects but before that publisher + /// presents its conditional write. This is the exact window in which another admitted writer can + /// advance the frontier. + std::function &)> before_cas_put; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { - ListPage page = CountingBackend::list(prefix, cursor, limit); - std::vector kept; + DB::Cas::Backend::RawListPage page = CountingBackend::list(prefix, cursor, limit, access); + std::vector kept; kept.reserve(page.keys.size()); - for (ListedKey & lk : page.keys) + for (DB::Cas::Backend::RawListedKey & lk : page.keys) if (!hidden_keys.contains(lk.key)) kept.push_back(std::move(lk)); if (cursor.empty()) @@ -147,32 +172,35 @@ class HidingListBackend : public CountingBackend for (const String & key : phantom_list_keys) { if (key.starts_with(prefix)) - kept.push_back(ListedKey{.key = key, .size = 0, .token = std::nullopt}); + kept.push_back(DB::Cas::Backend::RawListedKey{.key = key, .size = 0, .value = std::nullopt}); } } page.keys = std::move(kept); return page; } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - if (!ambiguous_put_substr.empty() && key.find(ambiguous_put_substr) != String::npos) - throw std::runtime_error("injected ambiguous putIfAbsent"); - return CountingBackend::putIfAbsent(key, bytes, meta); - } - - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + /// Both write faults hang off the ONE keyed primitive every caller now reaches the store through; + /// which of them applies is decided by whether the write carries a precondition, which is exactly + /// what used to separate `putIfAbsent` from `casPut`. + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { + if (!expected_value) + { + if (!ambiguous_put_substr.empty() && key.find(ambiguous_put_substr) != String::npos) + throw std::runtime_error("injected ambiguous create"); + return CountingBackend::write(key, bytes, expected_value, access); + } if (before_cas_put) - before_cas_put(key, bytes, expected); + before_cas_put(key, bytes, expected_value); if (ambiguous_cas_count > 0 && !ambiguous_cas_substr.empty() && key.find(ambiguous_cas_substr) != String::npos) { --ambiguous_cas_count; - throw Poco::TimeoutException("HidingListBackend: simulated ambiguous checkpoint CAS"); + throw Poco::TimeoutException("HidingListBackend: simulated ambiguous checkpoint replace"); } - return CountingBackend::casPut(key, bytes, expected, meta); + return CountingBackend::write(key, bytes, expected_value, access); } }; @@ -182,28 +210,18 @@ class HidingListBackend : public CountingBackend class PutHookBackend : public HidingListBackend { public: - using HidingListBackend::putIfAbsent; - - using HidingListBackend::casPut; - String watched_substr; uint64_t skip = 0; std::function on_key; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - PutResult result = HidingListBackend::putIfAbsent(key, bytes, meta); - fireIfWatched(key); - return result; - } - - /// The `_ckpt` advance is a token-CAS, not a create, whenever the object already exists -- which is - /// the normal case, since the namespace birth creates it. Hooking only `putIfAbsent` would silently - /// never fire for it. - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + /// One override covers both write shapes: the `_ckpt` advance is a conditional replace, not a + /// create, whenever the object already exists -- which is the normal case, since the namespace + /// birth creates it -- so hooking only creates would silently never fire for it. + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - CasResult result = HidingListBackend::casPut(key, bytes, expected, meta); + auto result = HidingListBackend::write(key, bytes, expected_value, access); fireIfWatched(key); return result; } @@ -235,17 +253,15 @@ class PutHookBackend : public HidingListBackend class LateMaterializeBackend : public HidingListBackend { public: - using HidingListBackend::get; - String late_key; String late_bytes; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - std::optional result = HidingListBackend::get(key, range); + std::optional result = HidingListBackend::read(key, access); if (!result && !late_key.empty() && key == late_key) { - CountingBackend::putIfAbsent(late_key, late_bytes); + (void)CountingBackend::write(late_key, late_bytes, std::nullopt, access); late_key.clear(); /// one-shot: the walk must see it present from here on } return result; @@ -258,8 +274,6 @@ class LateMaterializeBackend : public HidingListBackend class GetSeamBackend : public HidingListBackend { public: - using HidingListBackend::get; - String watched_substr; /// Assigned from the test thread and read from whatever thread the recovery runs on, so the @@ -270,7 +284,7 @@ class GetSeamBackend : public HidingListBackend std::mutex hook_mutex; std::function on_key; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { std::unique_lock hook_lock(hook_mutex); if (on_key && !watched_substr.empty() && key.find(watched_substr) != String::npos) @@ -288,7 +302,7 @@ class GetSeamBackend : public HidingListBackend hook_lock.unlock(); hook(key); } - return HidingListBackend::get(key, range); + return HidingListBackend::read(key, access); } }; @@ -298,14 +312,12 @@ class GetSeamBackend : public HidingListBackend class AfterGetHookBackend : public HidingListBackend { public: - using HidingListBackend::get; - String watched_key; std::function after_get; - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - std::optional result = HidingListBackend::get(key, range); + std::optional result = HidingListBackend::read(key, access); if (after_get && key == watched_key) { auto hook = std::move(after_get); @@ -316,10 +328,17 @@ class AfterGetHookBackend : public HidingListBackend } }; + +/// More injected failures than the engine's own retry window can make attempts, on a clock that +/// advances at least a millisecond per pause: a call meeting this fault must end at its DEADLINE, +/// never by outliving the fault. A bounded count would be spent by ONE call's own reissues, because +/// the engine settles each ambiguity by an exact read and then reissues. +constexpr int kFaultsBeyondTheRetryWindow = 100'000; + CasRequestBudget tinyBudget() { return CasRequestBudget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = 500, .max_attempts = 1, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .operation_deadline_ms = 500, .lease_safety_margin_ms = 50}; } PoolConfig walkTestConfig() @@ -347,10 +366,12 @@ PoolPtr openWalkPool(const BackendPtr & backend, PoolConfig config = walkTestCon /// minted, never reclaimed (`CasPool.cpp`'s allocator), so this is exactly what a pool that has been /// mounted `n` times looks like -- including the burned epochs in which nothing was ever written, which /// the seal chain must cross. -void burnEpochsUpTo(Backend & backend, const Layout & layout, uint64_t target_live_epoch) +void burnEpochsUpTo(const BackendPtr & backend, const Layout & layout, uint64_t target_live_epoch) { + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); for (uint64_t e = 1; e < target_live_epoch; ++e) - allocateWriterEpoch(backend, layout, "test", EpochMintPolicy::NormalMount, 0, [] { return RefCatalog{}; }); + allocateWriterEpoch(op, layout, "test", EpochMintPolicy::NormalMount, 0, [] { return RefCatalog{}; }); } /// One ordinary transaction at `id`, publishing `ref` (prepending the birth op when `birth`). @@ -428,9 +449,9 @@ uint64_t counterOf(ProfileEvents::Event event) return ProfileEvents::global_counters[event].load(); } -NamespaceLifeId catalogLife(Backend & backend, const Layout & layout, const RootNamespace & ns) +NamespaceLifeId catalogLife(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns) { - const CasRefCatalog::Snapshot catalog = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot catalog = readCatalogForTest(backend, layout); for (const CatalogEntry & entry : catalog.catalog.entries) if (entry.ns == ns) return NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); @@ -438,7 +459,8 @@ NamespaceLifeId catalogLife(Backend & backend, const Layout & layout, const Root } NamespaceLifeId strandOneUnfrontieredSuccessor( - HidingListBackend & backend, const PoolPtr & store, const Layout & layout, const RootNamespace & ns) + const std::shared_ptr & backend, const PoolPtr & store, const Layout & layout, + const RootNamespace & ns) { store->appendRefOps(ns, MutationScope::ref("a"), [](const RefTableState & state) @@ -451,30 +473,39 @@ NamespaceLifeId strandOneUnfrontieredSuccessor( return ops; }, RootMutationOrigin::Writer, RootMutationKind::Publish); + /// The frontier publication reissues an ambiguous replace until its own retry window closes, so + /// the fault has to outlast the call and the clock the window is read from has to move for the call + /// to end at all. + auto clock = VirtualRetryClock::installOn(store); const NamespaceLifeId life = catalogLife(backend, layout, ns); - backend.ambiguous_cas_substr = layout.refCkptKey(life); - backend.ambiguous_cas_count = 200; + backend->ambiguous_cas_substr = layout.refCkptKey(life); + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->appendRefOps(ns, MutationScope::ref("b"), [](const RefTableState &) { return publishCommittedOps("b", manifestRef(1, 2, 1)); }, RootMutationOrigin::Writer, RootMutationKind::Publish); }); - backend.ambiguous_cas_count = 0; + backend->ambiguous_cas_count = 0; + EXPECT_GT(clock->pauseCount(), 1u) + << "the reissues must pace through the injected sleep, never a real one"; + EXPECT_LE(clock->longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; return life; } CatalogEntry replaceCatalogLifeForTest( - Backend & backend, const Layout & layout, const CatalogEntry & predecessor, UInt128 successor_incarnation) + const BackendPtr & backend, const Layout & layout, const CatalogEntry & predecessor, + UInt128 successor_incarnation) { - const CasRefCatalog::Snapshot before_delete = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot before_delete = readCatalogForTest(backend, layout); RefCatalog without_predecessor = before_delete.catalog; std::erase_if(without_predecessor.entries, [&](const CatalogEntry & entry) { return entry.ns == predecessor.ns && entry.incarnation == predecessor.incarnation; }); - if (backend.casPut(layout.refCatalogKey(), encodeRefCatalog(without_predecessor), before_delete.token).outcome - != CasOutcome::Committed) + if (!before_delete.incarnation + || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(without_predecessor), + *before_delete.incarnation)) throw std::runtime_error("test failed to retire exact predecessor catalog life"); CatalogEntry successor{ @@ -482,11 +513,11 @@ CatalogEntry replaceCatalogLifeForTest( .state = NsState::Live, .incarnation = successor_incarnation, .creator = std::nullopt}; - const CasRefCatalog::Snapshot after_delete = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot after_delete = readCatalogForTest(backend, layout); RefCatalog reborn = after_delete.catalog; reborn.entries.push_back(successor); - if (backend.casPut(layout.refCatalogKey(), encodeRefCatalog(reborn), after_delete.token).outcome - != CasOutcome::Committed) + if (!after_delete.incarnation + || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.incarnation)) throw std::runtime_error("test failed to publish successor catalog life"); return successor; } @@ -585,22 +616,22 @@ TEST(CASRefRecoveryCasWalk, MissingExactIdAtOrBelowCommittedFrontierIsCorruption const RefTxnId frontier{1, 2}; DB::Cas::tests::fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = std::optional{1}, .committed_through = frontier, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); - const auto ckpt_before = readCkpt(*backend, layout, life); + const auto ckpt_before = readCkptForTest(backend, layout, life); ASSERT_TRUE(ckpt_before); auto store = openWalkPool(backend); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)store->listRefs(ns); }); - const auto ckpt_after = readCkpt(*backend, layout, life); + const auto ckpt_after = readCkptForTest(backend, layout, life); ASSERT_TRUE(ckpt_after); - EXPECT_EQ(ckpt_after->token, ckpt_before->token) + EXPECT_EQ(ckpt_after->incarnation, ckpt_before->incarnation) << "an unchanged checkpoint makes the missing committed id corruption, not a shorter stream"; } @@ -613,7 +644,7 @@ TEST(CASRefRecoveryCasWalk, UncommittedSnapshotIsUnobservedWithoutStreamList) const RefTxnId uncommitted_snapshot_id{1, 2}; DB::Cas::tests::fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); seedTxn(*backend, layout, ns, frontier, "committed", /*birth=*/true); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), uncommitted_snapshot_id, @@ -652,18 +683,20 @@ TEST(CASRefRecoveryCasWalk, ListingShapeDoesNotAffectCheckpointRecovery) .committed_through = frontier, .checkpoint_snapshot_id = base, .last_epoch_seal = std::nullopt}); - const NamespaceLifeId life = catalogLife(*seed, layout, ns); + const NamespaceLifeId life = catalogLife(seed, layout, ns); const auto clone_seed = [&]() -> std::shared_ptr { /// A clone starts empty: constructing the normal fixture would pre-seed independent pool-meta /// bytes before this loop could copy the source's identical durable image. auto backend = std::make_shared(/*seed_pool_meta=*/false); + CasRequests seed_requests(seed, Fence::open()); + CasOperation seed_op = seed_requests.admit(); String cursor; do { - const ListPage page = seed->list("", cursor, 1000); - for (const ListedKey & listed : page.keys) + const KeyPage page = seed_op.list("", cursor, 1000, Retry::standard()); + for (const KeyEntry & listed : page.keys) { const auto object = seed->get(listed.key); if (!object) @@ -724,7 +757,7 @@ TEST(CASRefRecoveryCasWalk, PhantomListedSnapshotIsUnobserved) .committed_through = frontier, .checkpoint_snapshot_id = checkpoint_base, .last_epoch_seal = std::nullopt}); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); backend->phantom_list_keys.insert(layout.refSnapshotKey(life, frontier)); auto store = openWalkPool(backend); @@ -762,10 +795,10 @@ TEST(CASRefRecoveryCasWalk, DuplicateCatalogLifeIsCorruptionBeforeColdRuntimeAdm const Layout layout("p"); const RootNamespace ns{"srv1/ambiguous_life_a"}; auto store = openWalkPool(backend); - const NamespaceLifeId life = strandOneUnfrontieredSuccessor(*backend, store, layout, ns); + const NamespaceLifeId life = strandOneUnfrontieredSuccessor(backend, store, layout, ns); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); - const CasRefCatalog::Snapshot sampled = CasRefCatalog::read(*backend, layout); + const CasRefCatalog::Snapshot sampled = readCatalogForTest(backend, layout); ASSERT_EQ(sampled.catalog.entries.size(), 1u); RefCatalog ambiguous = sampled.catalog; ambiguous.entries.push_back(CatalogEntry{ @@ -774,8 +807,8 @@ TEST(CASRefRecoveryCasWalk, DuplicateCatalogLifeIsCorruptionBeforeColdRuntimeAdm .incarnation = life.incarnation}); std::sort(ambiguous.entries.begin(), ambiguous.entries.end(), [](const CatalogEntry & lhs, const CatalogEntry & rhs) { return lhs.ns.string() < rhs.ns.string(); }); - ASSERT_EQ(backend->casPut(layout.refCatalogKey(), encodeRefCatalog(ambiguous), sampled.token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(sampled.incarnation); + ASSERT_TRUE(replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(ambiguous), *sampled.incarnation)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { @@ -794,17 +827,16 @@ TEST(CASRefRecoveryCasWalk, CheckpointAdvanceAfterLastLogProbeRestartsBeforeInst seedTxn(*backend, layout, ns, initial_frontier, "a", /*birth=*/true); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, initial_frontier)); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); backend->watched_key = layout.refLogKey(life, concurrent_frontier); backend->after_get = [&] { seedTxn(*backend, layout, ns, concurrent_frontier, "b", /*birth=*/false); - const auto sampled = readCkpt(*backend, layout, life); + const auto sampled = readCkptForTest(backend, layout, life); ASSERT_TRUE(sampled); RefCkpt advanced = sampled->ckpt; advanced.committed_through = concurrent_frontier; - ASSERT_EQ(backend->casPut(layout.refCkptKey(life), encodeRefCkpt(advanced), sampled->token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(replaceForTest(backend, layout.refCkptKey(life), encodeRefCkpt(advanced), sampled->incarnation)); }; auto store = openWalkPool(backend); @@ -825,9 +857,9 @@ TEST(CASRefRecoveryCasWalk, LiveCatalogLifeWithoutReadableCheckpointIsCorruption const RootNamespace ns{"srv1/live_without_ckpt"}; DB::Cas::tests::fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); seedTxn(*backend, layout, ns, RefTxnId{7, 1}, "hint-must-not-be-genesis", /*birth=*/true); - ASSERT_FALSE(readCkpt(*backend, layout, life)); + ASSERT_FALSE(readCkptForTest(backend, layout, life)); auto store = openWalkPool(backend); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)store->listRefs(ns); }); @@ -868,7 +900,7 @@ TEST(CASRefRecoveryCasWalk, DeadEpochIsClosedByOurOwnSealAtTPlusOne) const Layout layout("p"); const RootNamespace ns{"srv1/seal_created"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -898,7 +930,7 @@ TEST(CASRefRecoveryCasWalk, ConcurrentRecoverersSealIsAdoptedNotContested) const Layout layout("p"); const RootNamespace ns{"srv1/seal_adopt"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); /// The peer's seal lands between our read of {1,2} and our create of it, so we meet it as an @@ -926,7 +958,7 @@ TEST(CASRefRecoveryCasWalk, StragglerAtTPlusOneIsAdoptedAndResealedAtTheNewTPlus const Layout layout("p"); const RootNamespace ns{"srv1/straggler"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); /// The dying epoch's last append materializes between our read of {1,2} and our create of it -- the @@ -973,10 +1005,10 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEveryOccupiedObjectBeforeAdvancingP const RootNamespace ns{"srv1/occupied_frontier_" + test_case.suffix}; const RefTxnId initial_frontier{1, 1}; - burnEpochsUpTo(*backend, layout, test_case.live_epoch); + burnEpochsUpTo(backend, layout, test_case.live_epoch); seedTxn(*backend, layout, ns, initial_frontier, "a", /*birth=*/true); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, initial_frontier)); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); backend->late_key = layout.refLogKey(life, test_case.occupant); const RefLogTxn occupant = test_case.occupant_is_seal ? makeSealTxn(ns, test_case.occupant) @@ -993,13 +1025,13 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEveryOccupiedObjectBeforeAdvancingP store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); backend->ambiguous_cas_substr = layout.refCkptKey(life); - backend->ambiguous_cas_count = 100'000; + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->listRefs(ns); }); EXPECT_TRUE(backend->get(layout.refLogKey(life, test_case.occupant))); EXPECT_FALSE(backend->get(layout.refLogKey(life, test_case.forbidden_successor))) << "recovery advanced before exact _ckpt certified the occupied object"; - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, initial_frontier); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, initial_frontier); EXPECT_FALSE(store->refTableRecoveredForTest(ns)); } } @@ -1015,7 +1047,7 @@ TEST(CASRefRecoveryCasWalk, TwoBurnedEmptyEpochsProduceTwoChainedSequenceOneSeal const Layout layout("p"); const RootNamespace ns{"srv1/burned"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/4); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/4); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1054,10 +1086,10 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEachCreatedSealBeforeCreatingTheNex const RefTxnId second_seal{2, 1}; const RefTxnId cold_remount_frontier{3, 1}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/3); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/3); seedTxn(*backend, layout, ns, initial_frontier, "a", /*birth=*/true); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, initial_frontier)); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); uint64_t fake_now = 1'000'000; PoolConfig config = walkTestConfig(); @@ -1070,14 +1102,14 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEachCreatedSealBeforeCreatingTheNex store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); backend->ambiguous_cas_substr = layout.refCkptKey(life); - backend->ambiguous_cas_count = 100'000; + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->listRefs(ns); }); EXPECT_TRUE(backend->get(layout.refLogKey(life, first_seal))) << "the first recovery seal became durable before its frontier attempt"; EXPECT_FALSE(backend->get(layout.refLogKey(life, second_seal))) << "recovery may not create a second object while the first is still above exact _ckpt"; - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, initial_frontier); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, initial_frontier); EXPECT_FALSE(store->refTableRecoveredForTest(ns)); /// Restart cold, without the failed mount's `NeedsRecovery` attempt. The first seal is durable but @@ -1089,7 +1121,7 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEachCreatedSealBeforeCreatingTheNex ASSERT_EQ(cold_store->listRefs(ns).size(), 1u); EXPECT_TRUE(backend->get(layout.refLogKey(life, second_seal))); EXPECT_TRUE(backend->get(layout.refLogKey(life, cold_remount_frontier))); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, cold_remount_frontier); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, cold_remount_frontier); } /// A straggler is not an exception to the recovered-successor rule. When it materializes in the seal @@ -1104,10 +1136,10 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesAnAdoptedStragglerBeforeCreatingIts const RefTxnId straggler{1, 2}; const RefTxnId following_seal{1, 3}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedTxn(*backend, layout, ns, initial_frontier, "a", /*birth=*/true); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, initial_frontier)); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); backend->late_key = layout.refLogKey(life, straggler); backend->late_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(makeOrdinaryTxn(ns, straggler, "late", /*birth=*/false))); @@ -1122,20 +1154,20 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesAnAdoptedStragglerBeforeCreatingIts store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); backend->ambiguous_cas_substr = layout.refCkptKey(life); - backend->ambiguous_cas_count = 100'000; + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->listRefs(ns); }); EXPECT_TRUE(backend->get(layout.refLogKey(life, straggler))) << "the straggler occupied the recovery seal slot"; EXPECT_FALSE(backend->get(layout.refLogKey(life, following_seal))) << "recovery may not create a seal after an adopted straggler above exact _ckpt"; - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, initial_frontier); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, initial_frontier); EXPECT_FALSE(store->refTableRecoveredForTest(ns)); backend->ambiguous_cas_count = 0; ASSERT_EQ(store->listRefs(ns).size(), 2u); EXPECT_TRUE(backend->get(layout.refLogKey(life, following_seal))); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, following_seal); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, following_seal); } /// GENESIS. A namespace born at epoch 5 has no epochs 1-4 of its own: they are not "empty epochs it @@ -1149,7 +1181,7 @@ TEST(CASRefRecoveryCasWalk, GenesisAtEpochFiveWritesNoPhantomSealsBelowLifeEpoch const Layout layout("p"); const RootNamespace ns{"srv1/genesis5"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/5); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/5); seedCkpt(*backend, layout, ns, lifeEpochCkpt(5, RefTxnId{5, 1})); seedTxn(*backend, layout, ns, RefTxnId{5, 1}, "a", /*birth=*/true); @@ -1210,10 +1242,10 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo const Layout layout("p"); const RootNamespace ns{"srv1/recovery-retired-mid-io"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "predecessor", /*birth=*/true); - const CatalogEntry predecessor = CasRefCatalog::read(*backend, layout).catalog.entries.front(); + const CatalogEntry predecessor = readCatalogForTest(backend, layout).catalog.entries.front(); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(predecessor.ns, predecessor.incarnation); const auto predecessor_ckpt_before = backend->get(layout.refCkptKey(predecessor_life)); @@ -1253,7 +1285,7 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo cv.wait(lock, [&] { return paused; }); } - const CatalogEntry successor = replaceCatalogLifeForTest(*backend, layout, predecessor, UInt128{0x5152}); + const CatalogEntry successor = replaceCatalogLifeForTest(backend, layout, predecessor, UInt128{0x5152}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(successor_life), encodeRefCkpt(lifeEpochCkpt(2))).outcome, @@ -1273,7 +1305,7 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo EXPECT_TRUE(recovery_error) << "the predecessor recovery must be refused, not exposed"; EXPECT_EQ(backend->putCount(layout.refLogKey(predecessor_life, RefTxnId{1, 2})), 0u) << "no predecessor seal retry may be sent after exact retirement"; - EXPECT_EQ(backend->casPutCount(layout.refCkptKey(predecessor_life)), 0u) + EXPECT_EQ(backend->writeCount(layout.refCkptKey(predecessor_life)), 0u) << "no predecessor checkpoint CAS may be sent after exact retirement"; const auto predecessor_ckpt_after = backend->get(layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_after); @@ -1301,14 +1333,14 @@ TEST(CASRefRecoveryCasWalk, FenceBumpedAfterSlotOccupyBeforeCkptCasAdvancesNoChe const Layout layout("p"); const RootNamespace ns{"srv1/bump_after_seal"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); auto store = openWalkPool(backend); ASSERT_TRUE(store); - const auto ckpt_before = readCkpt(*backend, layout, DB::Cas::tests::fixture::fixtureLife(ns)); + const auto ckpt_before = readCkptForTest(backend, layout, DB::Cas::tests::fixture::fixtureLife(ns)); ASSERT_TRUE(ckpt_before.has_value()); backend->watched_substr = "_log/"; @@ -1316,11 +1348,11 @@ TEST(CASRefRecoveryCasWalk, FenceBumpedAfterSlotOccupyBeforeCkptCasAdvancesNoChe EXPECT_ANY_THROW(store->listRefs(ns)); - const auto ckpt_after = readCkpt(*backend, layout, DB::Cas::tests::fixture::fixtureLife(ns)); + const auto ckpt_after = readCkptForTest(backend, layout, DB::Cas::tests::fixture::fixtureLife(ns)); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->ckpt.last_epoch_seal, std::nullopt) << "the seal is durable but the checkpoint must not record it under a generation that moved"; - EXPECT_EQ(ckpt_after->token, ckpt_before->token) << "no CAS was sent at all"; + EXPECT_EQ(ckpt_after->incarnation, ckpt_before->incarnation) << "no CAS was sent at all"; } /// Bump point 2: AFTER the `_ckpt` CAS, BEFORE the install. The checkpoint advance is harmless (the @@ -1333,7 +1365,7 @@ TEST(CASRefRecoveryCasWalk, FenceBumpedAfterCkptCasBeforeInstallPublishesNoState const Layout layout("p"); const RootNamespace ns{"srv1/bump_after_ckpt"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1347,7 +1379,7 @@ TEST(CASRefRecoveryCasWalk, FenceBumpedAfterCkptCasBeforeInstallPublishesNoState EXPECT_ANY_THROW(store->listRefs(ns)) << "the install recheck must refuse a result from a moved generation"; - const auto ckpt_after = readCkpt(*backend, layout, DB::Cas::tests::fixture::fixtureLife(ns)); + const auto ckpt_after = readCkptForTest(backend, layout, DB::Cas::tests::fixture::fixtureLife(ns)); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->ckpt.last_epoch_seal, std::optional(RefTxnId{1, 2})) << "the checkpoint advance already landed and is harmless -- the merge is a semantic maximum"; @@ -1378,7 +1410,7 @@ TEST(CASRefRecoveryCasWalk, RemountBarrierBlocksUntilAPausedRecoveryAcknowledges const Layout layout("p"); const RootNamespace ns{"srv1/remount_barrier"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1445,6 +1477,82 @@ TEST(CASRefRecoveryCasWalk, RemountBarrierBlocksUntilAPausedRecoveryAcknowledges EXPECT_FALSE(store->refTableRecoveredForTest(ns)) << "and ZERO installs"; } +/// The cancellation reaches the walk's REQUESTS, not just its own polls. `readCheckpointSnapshotBase` +/// issues several reads back to back -- the base log, then the snapshot body -- and the walk's poll runs +/// only before the call, so a cancellation landing between those two reads used to be invisible until +/// the whole call returned. The walk's operation now carries the cancellation in its liveness, so the +/// next request is the one that refuses. +/// +/// Parked on the base-log read, which is the FIRST request that call makes, so the snapshot body read +/// is the one that must never happen. +TEST(CASRefRecoveryCasWalk, CancellationStopsTheWalkBetweenTwoReadsOfOneCall) +{ + auto backend = std::make_shared(); + const Layout layout("p"); + const RootNamespace ns{"srv1/cancel_between_reads"}; + const RefTxnId base{1, 1}; + const RefTxnId frontier{1, 2}; + + DB::Cas::tests::fixture::admitLive(*backend, layout, ns); + seedTxn(*backend, layout, ns, base, "a", /*birth=*/true); + writeRefSnapshotRaw(*backend, layout, + minimalLiveSnapshot(ns.string(), base, {committedRow("a", manifestRef(1, 1, 1))})); + seedTxn(*backend, layout, ns, frontier, "b", /*birth=*/false); + seedCkpt(*backend, layout, ns, RefCkpt{ + .life_epoch = std::optional{1}, + .committed_through = frontier, + .checkpoint_snapshot_id = base, + .last_epoch_seal = std::nullopt}); + const NamespaceLifeId life = catalogLife(backend, layout, ns); + + auto store = openWalkPool(backend); + ASSERT_TRUE(store); + + std::mutex m; + std::condition_variable cv; + bool recovery_parked = false; + bool release_recovery = false; + + backend->watched_substr = "_log/"; + /// `GetSeamBackend` moves the hook out before calling it, so this parks exactly once. + backend->on_key = [&](const String &) + { + std::unique_lock lock(m); + recovery_parked = true; + cv.notify_all(); + cv.wait(lock, [&] { return release_recovery; }); + }; + + std::thread recovery([&] { try { store->listRefs(ns); } catch (...) {} }); // NOLINT(bugprone-empty-catch): the outcome is asserted below through the request counts + + { + std::unique_lock lock(m); + cv.wait(lock, [&] { return recovery_parked; }); + } + + std::thread barrier([&] { store->cancelRefRecoveriesAndAwaitQuiescence(); }); + /// Wait for the REQUEST to be visible before releasing: releasing any earlier would race the walk + /// past a flag set a moment too late, and the test would read an ordinary completion as a + /// cancellation that never happened. + while (!store->refRecoveryCancelRequestedForTest(ns)) + std::this_thread::yield(); + + const uint64_t snapshot_reads_before = backend->getCount(layout.refSnapshotKey(life, base)); + ASSERT_EQ(snapshot_reads_before, 0u) << "the parked read is the base LOG read, before the body read"; + + { + std::lock_guard lock(m); + release_recovery = true; + } + cv.notify_all(); + barrier.join(); + recovery.join(); + + EXPECT_EQ(backend->getCount(layout.refSnapshotKey(life, base)), 0u) + << "the cancellation must refuse the very next request of the same call, not be noticed after it"; + EXPECT_FALSE(store->refTableRecoveredForTest(ns)) << "and nothing is installed"; +} + /// A `NeedsRecovery` lane replays the known-durable transaction before returning to `Ready`. TEST(CASRefRecoveryCasWalk, NeedsRecoveryReplaysTheStrandedTxn) { @@ -1495,7 +1603,7 @@ TEST(CASRefRecoveryCasWalk, NeedsRecoveryReplaysTheStrandedTxn) /// file's OTHER tests use -- so its ref-layer objects sit at a REAL, catalog-minted incarnation, /// not the Stage-A sentinel `readLogTxn` assumes. Resolved here rather than through `readLogTxn`. { - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(*backend, layout); + const CasRefCatalog::Snapshot snap = readCatalogForTest(backend, layout); const CatalogEntry * entry = nullptr; for (const CatalogEntry & e : snap.catalog.entries) if (e.ns.string() == ns.string()) @@ -1532,11 +1640,12 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsOneExactUnfrontieredSuccessorAnd return ops; }, RootMutationOrigin::Writer, RootMutationKind::Publish)); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); const String ckpt_key = layout.refCkptKey(life); - ASSERT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); + ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); + auto clock = VirtualRetryClock::installOn(store); backend->ambiguous_cas_substr = ckpt_key; - backend->ambiguous_cas_count = 200; + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { @@ -1544,17 +1653,19 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsOneExactUnfrontieredSuccessorAnd [](const RefTableState &) { return publishCommittedOps("b", manifestRef(1, 2, 1)); }, RootMutationOrigin::Writer, RootMutationKind::Publish); }); + EXPECT_GT(clock->pauseCount(), 1u) + << "the reissues must pace through the injected sleep, never a real one"; ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); ASSERT_TRUE(backend->get(layout.refLogKey(life, RefTxnId{1, 2}))) << "the sole deterministic successor must be durable before recovery"; - ASSERT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); + ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); backend->ambiguous_cas_count = 0; const auto refs = store->listRefs(ns); EXPECT_TRUE(refs.contains("b")); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 2})) + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 2})) << "the successor is not installable until the current admitted fence publishes its frontier"; } @@ -1564,11 +1675,11 @@ TEST(CASRefRecoveryCasWalk, ColdWriterRecoveryPublishesOneExactUnfrontieredSucce const Layout layout("p"); const RootNamespace ns{"srv1/recovery_cold_successor"}; auto store = openWalkPool(backend); - const NamespaceLifeId life = strandOneUnfrontieredSuccessor(*backend, store, layout, ns); + const NamespaceLifeId life = strandOneUnfrontieredSuccessor(backend, store, layout, ns); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); store.reset(); std::vector checkpoint_cas_bodies; - backend->before_cas_put = [&](const String & key, const String & bytes, const std::optional &) + backend->before_cas_put = [&](const String & key, const String & bytes, const std::optional &) { if (key == layout.refCkptKey(life)) checkpoint_cas_bodies.push_back(decodeRefCkpt(bytes)); @@ -1584,7 +1695,7 @@ TEST(CASRefRecoveryCasWalk, ColdWriterRecoveryPublishesOneExactUnfrontieredSucce EXPECT_TRUE(std::any_of(checkpoint_cas_bodies.begin(), checkpoint_cas_bodies.end(), [](const RefCkpt & ckpt) { return ckpt.committed_through == std::make_optional(RefTxnId{1, 2}); })) << "the exact F+1 frontier must publish before the remount seals its dead epoch"; - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 3})); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 3})); } TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsFirstCommittedTxnAboveLifeEpochOnlyCheckpoint) @@ -1598,14 +1709,15 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsFirstCommittedTxnAboveLifeEpochO /// production birth before its first log. This makes `{1,1}` the first durable transaction above a /// readable checkpoint whose `committed_through` is absent. DB::Cas::tests::fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(lifeEpochCkpt(1))).outcome, PutOutcome::Done); - ASSERT_TRUE(readCkpt(*backend, layout, life)->ckpt.life_epoch); - ASSERT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, std::nullopt); + ASSERT_TRUE(readCkptForTest(backend, layout, life)->ckpt.life_epoch); + ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, std::nullopt); + auto clock = VirtualRetryClock::installOn(store); backend->ambiguous_cas_substr = layout.refCkptKey(life); - backend->ambiguous_cas_count = 200; + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->appendRefOps(ns, MutationScope::ref("a"), @@ -1621,7 +1733,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsFirstCommittedTxnAboveLifeEpochO }); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); ASSERT_TRUE(backend->get(layout.refLogKey(life, RefTxnId{1, 1}))); - ASSERT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, std::nullopt); + ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, std::nullopt); ASSERT_FALSE(backend->get(layout.refSnapshotKey(life, RefTxnId{1, 1}))) << "the grounding test must exercise the exact log successor, not a hinted snapshot"; @@ -1630,7 +1742,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsFirstCommittedTxnAboveLifeEpochO EXPECT_TRUE(refs.contains("a")); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); } TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPrivateCandidate) @@ -1639,7 +1751,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPriv const Layout layout("p"); const RootNamespace ns{"srv1/recovery_checkpoint_moves"}; auto store = openWalkPool(backend); - const NamespaceLifeId life = strandOneUnfrontieredSuccessor(*backend, store, layout, ns); + const NamespaceLifeId life = strandOneUnfrontieredSuccessor(backend, store, layout, ns); const String ckpt_key = layout.refCkptKey(life); const RefLogTxn later = makeOrdinaryTxn(ns, RefTxnId{1, 3}, "c", /*birth=*/false); bool injected = false; @@ -1648,7 +1760,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPriv /// `{1,3}` and publish its frontier before recovery's own checkpoint CAS. The stale private /// candidate contains only `b`; it must restart and replay `c`, not accept an `IdenticalSkip` and /// install below the exact checkpoint it just observed. - backend->before_cas_put = [&](const String & key, const String &, const std::optional & expected) + backend->before_cas_put = [&](const String & key, const String &, const std::optional & expected) { if (injected || key != ckpt_key) return; @@ -1659,7 +1771,9 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPriv PutOutcome::Done); const auto current = backend->get(key); ASSERT_TRUE(current); - ASSERT_EQ(current->token, *expected); + /// `expected` is the transport value the publisher is presenting; the legacy read hands back + /// the same observation wrapped in a `Token`, so its `value` is what compares. + ASSERT_EQ(current->token.value, *expected); const RefCkpt advanced = mergeCkpt( decodeRefCkpt(current->bytes), RefCkpt{.life_epoch = std::nullopt, @@ -1675,7 +1789,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPriv EXPECT_TRUE(refs.contains("b")); EXPECT_TRUE(refs.contains("c")) << "recovery must restart from the newer exact frontier"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, later.txn_id); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, later.txn_id); } TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsTwoUnfrontieredSuccessorsAfterExactCheckpointReread) @@ -1696,16 +1810,19 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsTwoUnfrontieredSuccessorsAfterE return ops; }, RootMutationOrigin::Writer, RootMutationKind::Publish)); - const NamespaceLifeId life = catalogLife(*backend, layout, ns); + const NamespaceLifeId life = catalogLife(backend, layout, ns); const String ckpt_key = layout.refCkptKey(life); + auto clock = VirtualRetryClock::installOn(store); backend->ambiguous_cas_substr = ckpt_key; - backend->ambiguous_cas_count = 200; + backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->appendRefOps(ns, MutationScope::ref("b"), [](const RefTableState &) { return publishCommittedOps("b", manifestRef(1, 2, 1)); }, RootMutationOrigin::Writer, RootMutationKind::Publish); }); + EXPECT_GT(clock->pauseCount(), 1u) + << "the reissues must pace through the injected sleep, never a real one"; ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); const RefLogTxn second_successor = makeOrdinaryTxn(ns, RefTxnId{1, 3}, "c", /*birth=*/false); @@ -1716,7 +1833,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsTwoUnfrontieredSuccessorsAfterE expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)store->listRefs(ns); }); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})) + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})) << "corruption must not launder either successor into the frontier"; } @@ -1726,7 +1843,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsDifferentOrdinaryBytesAtTheReta const Layout layout("p"); const RootNamespace ns{"srv1/recovery_different_successor"}; auto store = openWalkPool(backend); - const NamespaceLifeId life = strandOneUnfrontieredSuccessor(*backend, store, layout, ns); + const NamespaceLifeId life = strandOneUnfrontieredSuccessor(backend, store, layout, ns); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); const String successor_key = layout.refLogKey(life, RefTxnId{1, 2}); @@ -1740,7 +1857,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsDifferentOrdinaryBytesAtTheReta expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)store->listRefs(ns); }); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); } TEST(CASRefRecoveryCasWalk, RetainedOldWriterAttemptLosesConclusiveToASuccessorSeal) @@ -1749,7 +1866,7 @@ TEST(CASRefRecoveryCasWalk, RetainedOldWriterAttemptLosesConclusiveToASuccessorS const Layout layout("p"); const RootNamespace ns{"srv1/recovery_successor_seal"}; auto store = openWalkPool(backend); - const NamespaceLifeId life = strandOneUnfrontieredSuccessor(*backend, store, layout, ns); + const NamespaceLifeId life = strandOneUnfrontieredSuccessor(backend, store, layout, ns); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); const RefTxnId successor_id{1, 2}; @@ -1766,8 +1883,8 @@ TEST(CASRefRecoveryCasWalk, RetainedOldWriterAttemptLosesConclusiveToASuccessorS EXPECT_FALSE(refs.contains("b")) << "the old writer's retained ordinary bytes lost at the sealed slot"; EXPECT_EQ(store->lastEpochSealForTest(ns), std::make_optional(successor_id)); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.committed_through, successor_id); - EXPECT_EQ(readCkpt(*backend, layout, life)->ckpt.last_epoch_seal, successor_id); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, successor_id); + EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.last_epoch_seal, successor_id); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); } @@ -1785,7 +1902,7 @@ TEST(CASRefRecoveryCasWalk, UnresolvedSealSlotFailsClosedWithoutInstalling) const Layout layout("p"); const RootNamespace ns{"srv1/unresolved"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1824,7 +1941,7 @@ TEST(CASRefRecoveryCasWalk, ALatePredecessorPutAtTheSealedSlotIsRefusedByTheStor const Layout layout("p"); const RootNamespace ns{"srv1/ghost"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1857,7 +1974,7 @@ TEST(CASRefRecoveryCasWalk, UndecodableOccupantAtTheSealSlotFailsClosedAndLeaves const Layout layout("p"); const RootNamespace ns{"srv1/foreign_slot"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); backend->late_key = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), RefTxnId{1, 2}); @@ -1885,7 +2002,7 @@ TEST(CASRefRecoveryCasWalk, ASecondCallerWaitsForTheWalkInsteadOfRacingIt) const Layout layout("p"); const RootNamespace ns{"srv1/serialized"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/2); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/2); seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, RefTxnId{1, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1952,7 +2069,7 @@ TEST(CASRefRecoveryCasWalk, RecoveryStartsAtRecreatedLifeGenesisAndLeavesPredece const Layout layout("p"); const RootNamespace ns{"srv1/removed_then_reborn"}; - burnEpochsUpTo(*backend, layout, /*target_live_epoch=*/3); + burnEpochsUpTo(backend, layout, /*target_live_epoch=*/3); seedCkpt(*backend, layout, ns, lifeEpochCkpt(2, RefTxnId{2, 1})); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); @@ -1992,8 +2109,14 @@ TEST(CASRefRecoveryCasWalk, PutHookBackendComposesHidingListBackendCasPutFaultIn { auto backend = std::make_shared(); + /// `HidingListBackend::write` only runs `before_cas_put` on the CONDITIONAL branch (an `expected` + /// token present) -- a bare create-shaped `casPut(..., std::nullopt)` takes the other branch and + /// can never reach it. Seed the key first so the probed call below is a genuine replace. + const auto seeded = backend->putIfAbsent("p/probe", "seed"); + ASSERT_EQ(seeded.outcome, PutOutcome::Done); + bool before_cas_put_fired = false; - backend->before_cas_put = [&](const String &, const String &, const std::optional &) + backend->before_cas_put = [&](const String &, const String &, const std::optional &) { before_cas_put_fired = true; }; @@ -2002,7 +2125,7 @@ TEST(CASRefRecoveryCasWalk, PutHookBackendComposesHidingListBackendCasPutFaultIn bool on_key_fired = false; backend->on_key = [&] { on_key_fired = true; }; - ASSERT_EQ(backend->casPut("p/probe", "x", std::nullopt).outcome, CasOutcome::Committed); + ASSERT_EQ(backend->casPut("p/probe", "x", seeded.token).outcome, CasOutcome::Committed); EXPECT_TRUE(before_cas_put_fired) << "HidingListBackend's before_cas_put hook must still fire for a PutHookBackend instance"; diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp index 341484e8f84c..0ac0b9e93644 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp @@ -14,6 +14,7 @@ #include +#include #include #include #include @@ -55,6 +56,63 @@ using DB::Cas::tests::publishCommittedOps; namespace { +/// The engine reissues an unresolved write until its OWN retry window closes, and that window is +/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine +/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline +/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling +/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the +/// closures own the clock, so it outlives everything that can still read it. +class VirtualRetryClock +{ +public: + static std::shared_ptr installOn(const PoolPtr & store) + { + auto clock = std::make_shared(); + store->setCasRequestNowFnForTest([clock] { return clock->nowMs(); }); + store->setCasRetrySleepForTest([clock](uint64_t ms) { clock->advance(ms); }); + return clock; + } + + uint64_t nowMs() const + { + std::lock_guard lock(mutex); + return now_ms; + } + size_t pauseCount() const + { + std::lock_guard lock(mutex); + return pauses; + } + uint64_t longestPause() const + { + std::lock_guard lock(mutex); + return longest_pause; + } + + void advance(uint64_t ms) + { + std::lock_guard lock(mutex); + /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move + /// would leave the loop reissuing for ever against a fault that never clears. + now_ms += ms + 1; + ++pauses; + longest_pause = std::max(longest_pause, ms); + } + +private: + mutable std::mutex mutex; + uint64_t now_ms = 0; + size_t pauses = 0; + uint64_t longest_pause = 0; +}; + +/// More injected failures than the engine's own retry window can make attempts, on a clock that +/// advances at least a millisecond per pause: a call meeting this fault must end at its DEADLINE, never +/// by outliving the fault. A bounded count would be spent by ONE call's own reissues -- the engine +/// settles each ambiguity by an exact read and then reissues -- and the fixture would then be measuring +/// attempts where it means to measure dispatches. +constexpr int kFaultsBeyondTheRetryWindow = 100'000; + PoolPtr openPool(const std::shared_ptr & backend, PoolConfig config = {}) { config.pool_prefix = "p"; @@ -118,7 +176,7 @@ TEST(CASRefSnapshotPublishOrdering, SnapshotBodyIsDurableBeforeCheckpointAdvance /// nothing about the publisher's own ordering. const size_t offset = backend->journalSize(); const uint64_t put_before = backend->putCount(snapshot_key); - const uint64_t cas_before = backend->casPutCount(ckpt_key); + const uint64_t cas_before = backend->putOverwriteCount(ckpt_key); ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)) << "a healthy Ready-lane table with an uncovered tail must publish"; @@ -126,10 +184,10 @@ TEST(CASRefSnapshotPublishOrdering, SnapshotBodyIsDurableBeforeCheckpointAdvance /// Positive control: this attempt touched each key exactly once (no retry, no redundant write) -- /// which is what makes the index comparison below meaningful rather than an artifact of a busy log. EXPECT_EQ(backend->putCount(snapshot_key) - put_before, 1u); - EXPECT_EQ(backend->casPutCount(ckpt_key) - cas_before, 1u); + EXPECT_EQ(backend->putOverwriteCount(ckpt_key) - cas_before, 1u); - const auto body_index = backend->firstIndexFrom(OrderedFaultBackend::Op::Put, snapshot_key, offset); - const auto ckpt_index = backend->firstIndexFrom(OrderedFaultBackend::Op::Cas, ckpt_key, offset); + const auto body_index = backend->firstIndexFrom(snapshot_key, offset); + const auto ckpt_index = backend->firstIndexFrom(ckpt_key, offset); ASSERT_TRUE(body_index.has_value()) << "the snapshot body must have been PUT"; ASSERT_TRUE(ckpt_index.has_value()) << "the checkpoint must have been CAS-advanced"; EXPECT_LT(*body_index, *ckpt_index) @@ -145,6 +203,7 @@ TEST(CASRefSnapshotPublishOrdering, AdoptionHappensLastAndOnlyAfterBothDurableEf { auto backend = std::make_shared(); auto store = openPool(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/order_adoption_after_both"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{store->writerEpoch(), 1})); @@ -152,26 +211,30 @@ TEST(CASRefSnapshotPublishOrdering, AdoptionHappensLastAndOnlyAfterBothDurableEf const String snapshot_key = store->layout().refSnapshotKey(life, RefTxnId{store->writerEpoch(), 1}); const String ckpt_key = store->layout().refCkptKey(life); - /// Fail every one of the (attempt-bounded) 100 `_ckpt` CAS attempts `publishCkpt` will make: the - /// body PUT still commits (dedup: an identical, already-durable body resolves as `Committed` without - /// re-sending), but the checkpoint never advances within this call. - backend->armCasConflict(ckpt_key, 100); + /// Refuse the `_ckpt` CAS for as long as `publishCkpt` keeps reissuing, so it ends at its own retry + /// window: the body create still commits, but the checkpoint never advances within this call. + backend->armWriteConflict(ckpt_key, kFaultsBeyondTheRetryWindow); EXPECT_FALSE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)) << "a persistently conflicting checkpoint CAS must not be reported as a successful publish"; + backend->armWriteConflict(ckpt_key, 0); + EXPECT_GT(clock->pauseCount(), 1u) + << "the reissues must pace through the injected sleep, never a real one"; + EXPECT_LE(clock->longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; EXPECT_EQ(backend->putCount(snapshot_key), 1u) << "the body is durable regardless of the ckpt outcome"; EXPECT_FALSE(store->newestPublishedSnapshotIdForTest(ns).has_value()) << "in-memory adoption must NOT happen while the checkpoint has not advanced"; - /// Disarm the fault and retry (the one retry unit): the retry issues its OWN `putIfAbsent` attempt at + /// Retry with the fault disarmed (the one retry unit): the retry issues its OWN create attempt at /// the same content-addressed key with the same bytes (so `putCount`, a call counter, becomes 2 -- - /// not a "no write happened" 1), but the backend resolves it as `Committed` against the already-durable - /// object rather than sending a distinct object, and the checkpoint CAS now succeeds. + /// not a "no write happened" 1). That attempt meets its own identical bytes as a conflict, which the + /// publisher's occupant compare accepts rather than writing a second object, and the checkpoint CAS + /// now succeeds. ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)) << "the retry, with the fault cleared, must publish"; EXPECT_EQ(backend->putCount(snapshot_key), 2u) - << "the retry's body PUT is its own attempt, resolved via dedup against identical, " - "already-durable bytes rather than writing a second object"; + << "the retry's body create is its own attempt, accepted against identical, already-durable " + "bytes rather than writing a second object"; EXPECT_EQ(store->newestPublishedSnapshotIdForTest(ns), std::make_optional(RefTxnId{store->writerEpoch(), 1})) << "adoption happens exactly once, after both effects are durable"; } @@ -211,6 +274,7 @@ TEST(CASRefSnapshotPublishOrdering, NeedsRecoveryLaneRecoversBeforeAnySnapshotPu { auto backend = std::make_shared(); auto store = openPool(backend); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/order_poisoned_refuses"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{store->writerEpoch(), 1})); @@ -224,12 +288,14 @@ TEST(CASRefSnapshotPublishOrdering, NeedsRecoveryLaneRecoversBeforeAnySnapshotPu /// for `missing_durable_txn` commits durably, but its checkpoint never advances within this call, and /// the lane is left `NeedsRecovery` rather than installing an uncertain result -- so the cached view /// still reflects `ref_1` present, while the durable log already reflects it removed. - backend->armCasConflict(ckpt_key, 100); + backend->armWriteConflict(ckpt_key, kFaultsBeyondTheRetryWindow); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "ref_1"); }); + EXPECT_GT(clock->pauseCount(), 1u) + << "the frontier publication's reissues must pace through the injected sleep, never a real one"; ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); const uint64_t recovery_installs_before = store->recoveryInstallCountForTest(); - backend->armCasConflict(ckpt_key, 0); /// clear the fault so re-recovery's OWN catch-up CAN succeed + backend->armWriteConflict(ckpt_key, 0); /// clear the fault so re-recovery's OWN catch-up CAN succeed const size_t offset = backend->journalSize(); EXPECT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)) @@ -247,8 +313,8 @@ TEST(CASRefSnapshotPublishOrdering, NeedsRecoveryLaneRecoversBeforeAnySnapshotPu /// ORDER, not a global zero: recovery's OWN checkpoint catch-up CAS is the boundary marker. NO /// snapshot-publish effect (the new snapshot's body PUT, nor the publisher's own checkpoint-advance /// CAS) may appear at or before it. - const auto ckpt_cas_indices = backend->indicesFrom(OrderedFaultBackend::Op::Cas, ckpt_key, offset); - const auto snap_put_indices = backend->indicesFrom(OrderedFaultBackend::Op::Put, next_snapshot_key, offset); + const auto ckpt_cas_indices = backend->indicesFrom(ckpt_key, offset); + const auto snap_put_indices = backend->indicesFrom(next_snapshot_key, offset); ASSERT_GE(ckpt_cas_indices.size(), 2u) << "expected one checkpoint CAS from recovery's catch-up and one from the snapshot publisher"; const size_t recovery_catchup_index = ckpt_cas_indices.front(); @@ -291,12 +357,11 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) using ProfileEvents::global_counters; auto backend = std::make_shared(); - /// A single-attempt request budget, exactly as `gtest_cas_ref_writer.cpp`'s - /// `C4BackoffDefersThenRetriesAndPublishes` uses: with `max_attempts = 1` a faulted PUT resolves to a - /// definite, non-`Committed` outcome on its own attempt, with no internal retry loop and so no - /// wall-clock wait. + /// The budget bounds the mount lease's own admission arithmetic and nothing else. What makes each + /// dispatch below fail as ONE dispatch is that the injected fault outlasts the whole call while the + /// injected clock carries it to its own retry window; the fake boot clock this test drives the + /// backoff decisions on is a DIFFERENT clock, and stays frozen between steps. CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; @@ -311,6 +376,7 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) config.boot_ms_fn = [&fake_now] { return fake_now; }; config.cas_request_budget = budget; auto store = openPool(backend, config); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/order_backoff"}; ASSERT_EQ(publishRef(store, ns, "ref_1", 1), (RefTxnId{store->writerEpoch(), 1})); @@ -322,10 +388,11 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) /// Fault the snapshot BODY put (never the `_ckpt` CAS -- an append-commit's OWN checkpoint write /// shares that key, and faulting it would drive the append lane into `NeedsRecovery` instead of - /// exercising the snapshot-publish backoff this test targets). Exactly 3 failures: the next 3 - /// automatic dispatch attempts fail (arming, then doubling, then re-doubling the backoff); the 4th - /// finds the fault disarmed and succeeds. - backend->armPutFailure("_snap/", 3); + /// exercising the snapshot-publish backoff this test targets). Armed for as long as any dispatch + /// keeps reissuing, so each dispatch ends at its own retry window and counts as ONE dispatch: the + /// next 3 fail (arming, then doubling, then re-doubling the backoff) and the test disarms the fault + /// before the 4th. + backend->armWriteFailure("_snap/", kFaultsBeyondTheRetryWindow); const auto dispatchCount = [&] { return global_counters[ProfileEvents::CASRefSnapshotPublishDispatched].load(); }; @@ -371,15 +438,16 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) EXPECT_EQ(dispatchCount(), d1 + 2) << "2000ms past the doubled deadline is still short of the capped 4000ms backoff"; - /// Cross the (capped) 4000ms deadline: the retry's fault budget is exhausted, so this attempt - /// succeeds, and `resetPublishBackoff` clears the cooldown -- proved by the NEXT trigger dispatching - /// with no wait at all. + /// Cross the (capped) 4000ms deadline with the fault disarmed, so this attempt succeeds and + /// `resetPublishBackoff` clears the cooldown -- proved by the NEXT trigger dispatching with no wait + /// at all. + backend->armWriteFailure("_snap/", 0); fake_now += 2000; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d1 + 3) << "past the second (capped) deadline, the retry dispatches and succeeds"; EXPECT_NE(store->newestPublishedSnapshotIdForTest(ns), snapshot_after_birth) - << "the fault budget is exhausted, so this attempt actually advances the published snapshot"; + << "the fault is disarmed, so this attempt actually advances the published snapshot"; ASSERT_EQ(publishRef(store, ns, "ref_3", 3), (RefTxnId{store->writerEpoch(), 3})); store->waitForSnapshotPublishSettleForTest(ns); @@ -393,7 +461,7 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) /// the INITIAL 1000ms interval rather than continuing from the 4000ms cap -- refused short of /// 1000ms, admitted at 1000ms -- which a no-op reset cannot produce (it would refuse both probes, /// since the stale deadline is still far in the future). - backend->armPutFailure("_snap/", 1); + backend->armWriteFailure("_snap/", kFaultsBeyondTheRetryWindow); ASSERT_EQ(publishRef(store, ns, "ref_4", 4), (RefTxnId{store->writerEpoch(), 4})); store->waitForSnapshotPublishSettleForTest(ns); const uint64_t d2 = dispatchCount(); @@ -407,6 +475,9 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) EXPECT_EQ(dispatchCount(), d2 + 1) << "resetPublishBackoff must have restarted the schedule at the INITIAL 1000ms interval, not " "left it continuing from the 4000ms cap"; + backend->armWriteFailure("_snap/", 0); + EXPECT_GT(clock->pauseCount(), 1u) + << "every failed dispatch above must have paced through the injected sleep, never a real one"; } TEST(CASRefSnapshotPublishOrdering, NotReadyRefusalBacksOffAndResetsAfterDurablePublish) @@ -414,8 +485,9 @@ TEST(CASRefSnapshotPublishOrdering, NotReadyRefusalBacksOffAndResetsAfterDurable using ProfileEvents::global_counters; auto backend = std::make_shared(); + /// No write fault anywhere in this test: every refusal below comes from the lane not being Ready, + /// so the budget only has to keep the mount lease admitting. CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; diff --git a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp index c696323b3d80..60df4db7809c 100644 --- a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp +++ b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp @@ -25,6 +25,7 @@ #include #include #include +#include /// ================================================================================================ /// Task 4 (2026-07-28 CAS ref-chain Stage A streams, spec INV-1's every-attempt rule + INV-2's seal): @@ -69,8 +70,8 @@ extern const int NETWORK_ERROR; } using namespace DB::Cas; +using DB::Cas::tests::VirtualRetryClock; using DB::Cas::tests::CountingBackend; -using DB::Cas::tests::LandedButAckLostOnceBackend; using DB::Cas::tests::expectThrowsCode; namespace @@ -82,33 +83,20 @@ PoolPtr openPool(const BackendPtr & backend, CasRequestBudget budget = {}) return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); } -/// The budget every wedge test uses: ONE attempt, so a single injected ambiguity is the whole -/// operation and the lane wedges deterministically instead of retrying its way out. -CasRequestBudget singleAttemptBudget() +/// The budget every wedge test uses. It bounds the mount lease's own admission arithmetic +/// (`attempt_timeout_ms` is what one attempt reserves, `lease_safety_margin_ms` the room kept past it) +/// and nothing else: a write's ATTEMPT COUNT is the `Retry` policy's, so no budget field can make an +/// injected fault conclusive. What makes a fault conclusive here is that it stays armed for the whole +/// call while `VirtualRetryClock` carries the call to its own deadline. +CasRequestBudget wedgeTestBudget() { CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) + budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; return budget; } -/// TWO attempts of one logical operation, with the inter-attempt backoff disabled. Everything about the -/// call-level verdict rule lives BETWEEN two attempts of a single call, so it cannot be reached with the -/// one-attempt budget the tests above use; the backoff is switched off because the schedule is -/// `gtest_cas_request_control.cpp`'s subject and a real sleep here would only slow the suite. -/// -/// `[[maybe_unused]]`: both of its callers need a real S3-classified rejection to script their second -/// attempt, so they compile away entirely in a build without S3. -[[maybe_unused]] CasRequestBudget twoAttemptBudget() -{ - CasRequestBudget budget = singleAttemptBudget(); - budget.max_attempts = 2; - budget.retry_initial_backoff_ms = 0; - budget.retry_max_backoff_ms = 0; - return budget; -} PartWriteTxnPtr startBuildFor(const PoolPtr & s, const RootNamespace & ns, const String & ref) { @@ -132,24 +120,21 @@ class WedgeTestBackend : public CountingBackend { public: using CountingBackend::putIfAbsent; - using CountingBackend::get; /// One-shot ambiguity that writes NOTHING: the response is lost and the key stays absent, which is - /// the input that makes a later `slotOccupy` report `Created`. + /// the input that makes a later bounded create commit. String ambiguous_substr; int ambiguous_count = 0; /// One-shot DETERMINISTIC LOCAL failure (`BAD_ARGUMENTS`, in `isDeterministicLocalFailure`'s set), - /// which `slotOccupy` rethrows unchanged -- a definite refusal of THIS attempt. Portable stand-in - /// for the S3 `DefiniteFailure` shape, which needs `USE_AWS_S3`; both are "proven never applied". + /// which every loop surfaces unchanged -- an exception out of the write, not an outcome. String definite_substr; int definite_count = 0; - /// One-shot WHITELISTED SYNCHRONOUS REJECTION: the ONLY shape `classifyConditionalWriteResult` - /// answers `DefiniteFailure` for, and therefore the only way to drive the append lane's definite - /// arm. Distinct from `definite_substr` above on purpose -- that one is a deterministic LOCAL - /// failure, which `slotOccupy` rethrows but `putIfAbsentControlled` (no such special case) merely - /// classifies Unresolved, so it cannot script this arm at all. + /// One-shot WHITELISTED SYNCHRONOUS REJECTION: the shape `isDefinitelyRefusedWrite` answers TRUE + /// for, and therefore the only way to drive the append lane's `Refused` arm. Distinct from + /// `definite_substr` above on purpose -- that one is a local bug the engine rethrows, while this one + /// is the store's own answer and comes back as a value. String s3_definite_substr; int s3_definite_count = 0; @@ -160,39 +145,85 @@ class WedgeTestBackend : public CountingBackend int conflict_count = 0; String conflict_bytes; - /// Fail GETs of matching keys after skipping the first `fail_get_skip` of them -- the resolve read - /// that PROVES the conflict must succeed, so only the adjudication read that follows it is faulted. + /// Lose every GET of a matching key. Armed by `fail_get_latched` below, because a read fault that + /// clears mid-call is simply reissued: the read engine settles a transient read failure by trying + /// again, so only a fault that outlasts the read's whole window is conclusive. String fail_get_substr; - int fail_get_skip = 0; - int fail_get_count = 0; String fail_cas_substr; int fail_cas_count = 0; - std::optional get(const String & key, Range range) override + /// LATCHED variants of the four seams above. A COUNT cannot make an injected fault conclusive: the + /// write engine settles every ambiguity by an exact read and then REISSUES, so a fault that runs out + /// mid-call is answered by the next attempt instead of by the call's deadline. A latch stays armed + /// until the test clears it, which is what makes "every attempt of this call was unresolved" the + /// input the wedge rule is about. + bool ambiguous_latched = false; + bool s3_definite_latched = false; + bool fail_get_latched = false; + bool fail_cas_latched = false; + + /// Our OWN exact bytes land and only the response is lost. Paired with `fail_get_latched` on the + /// same key it is the only way to wedge over an object that IS durable: the settling read is what + /// would otherwise prove the commit inside the same call and report it committed. + String landed_ack_lost_substr; + bool landed_ack_lost_latched = false; + + /// The store's own PRECONDITION REFUSAL -- a value, not an exception, so the call carries no + /// ambiguity at all. It is the only shape that can report a conflict naming NO occupant: over an + /// absent key the settling read proves absence, and with `refuse_read_after_precondition` it is + /// refused outright. Latched by nature, because a substring match is either armed or it is not. + String refuse_precondition_substr; + bool refuse_read_after_precondition = false; + + /// Straight past every seam below, for the writes a test makes on its own behalf. `putIfAbsent` + /// reaches the store through the VIRTUAL `write`, so a qualified call cannot bypass this override -- + /// only this flag can. + std::atomic bypass_seams{false}; + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - if (fail_get_count > 0 && !fail_get_substr.empty() && key.find(fail_get_substr) != String::npos) - { - if (fail_get_skip > 0) - --fail_get_skip; - else - { - --fail_get_count; - throw Poco::TimeoutException("WedgeTestBackend: simulated lost GET (read response never arrived)"); - } - } - return CountingBackend::get(key, range); + if (!bypass_seams.load(std::memory_order_acquire) && refuse_read_after_precondition + && !refuse_precondition_substr.empty() && key.find(refuse_precondition_substr) != String::npos) + throwDefiniteStoreRefusal("WedgeTestBackend: the settling read is definitively refused"); + if (fail_get_latched && !fail_get_substr.empty() && key.find(fail_get_substr) != String::npos) + throw Poco::TimeoutException("WedgeTestBackend: simulated lost read (response never arrived)"); + return CountingBackend::read(key, access); } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + /// The store's own definitive answer, which every loop here surfaces unchanged rather than + /// reissuing. Only the S3 classification recognises it, so a build without S3 raises the + /// deterministic-local class instead -- also never reissued, but a different arm, which is why the + /// fixtures that need this shape are guarded. + [[noreturn]] static void throwDefiniteStoreRefusal(const String & what) { - if (fail_cas_count > 0 && !fail_cas_substr.empty() && key.find(fail_cas_substr) != String::npos) +#if USE_AWS_S3 + throw DB::S3Exception(what, Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); +#else + throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "{} (requires S3 error classification)", what); +#endif + } + + /// Every write fault hangs off the ONE keyed primitive; which of them applies is decided by whether + /// the write carries a precondition, which is what used to separate a replace from a create. + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + if (bypass_seams.load(std::memory_order_acquire)) + return CountingBackend::write(key, bytes, expected_value, access); + if (expected_value) { - --fail_cas_count; - throw Poco::TimeoutException("WedgeTestBackend: simulated ambiguous checkpoint CAS"); + if ((fail_cas_latched || fail_cas_count > 0) && !fail_cas_substr.empty() + && key.find(fail_cas_substr) != String::npos) + { + if (!fail_cas_latched) + --fail_cas_count; + throw Poco::TimeoutException("WedgeTestBackend: simulated ambiguous checkpoint replace"); + } + return CountingBackend::write(key, bytes, expected_value, access); } - return CountingBackend::casPut(key, bytes, expected, meta); + return createForTest(key, bytes, access); } /// Park a matching PUT until `releaseBlock()`, notifying `awaitBlockEntered()` on arrival, so a @@ -219,21 +250,35 @@ class WedgeTestBackend : public CountingBackend } /// Write straight through, bypassing every fault and block seam above -- how a test models what a - /// SUCCESSOR (another process entirely) put at a key. Using the faulting entry point instead would - /// park the test's own write on the very gate it is trying to drive a scenario through. The - /// qualification must name the THREE-argument overload: `Backend`'s two-argument convenience - /// forwards to the VIRTUAL one, so `CountingBackend::putIfAbsent(key, bytes)` would dispatch right - /// back into the override above and deadlock the test against its own block. + /// SUCCESSOR (another process entirely) put at a key. Routing it through the seams instead would + /// park the test's own write on the very gate it is trying to drive a scenario through, or spend + /// the fault meant for the lane's attempt. Qualifying the call does NOT achieve that: + /// `Backend::putIfAbsent` reaches the store through the VIRTUAL `write`, so the override above runs + /// either way -- `bypass_seams` is what it reads to step aside. PutResult putAsSuccessor(const String & key, const String & bytes) { - return CountingBackend::putIfAbsent(key, bytes, ObjectMeta{}); + bypass_seams.store(true, std::memory_order_release); + const PutResult result = CountingBackend::putIfAbsent(key, bytes, ObjectMeta{}); + bypass_seams.store(false, std::memory_order_release); + return result; } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected createForTest( + const String & key, const String & bytes, DB::Cas::TransportAccess & access) { - if (ambiguous_count > 0 && !ambiguous_substr.empty() && key.find(ambiguous_substr) != String::npos) + if (!refuse_precondition_substr.empty() && key.find(refuse_precondition_substr) != String::npos) + return std::unexpected(DB::Cas::Backend::RawConflict{}); + if (landed_ack_lost_latched && !landed_ack_lost_substr.empty() + && key.find(landed_ack_lost_substr) != String::npos) + { + (void)CountingBackend::write(key, bytes, std::nullopt, access); /// the write LANDS + throw Poco::TimeoutException("WedgeTestBackend: our own bytes landed and the response was lost"); + } + if ((ambiguous_latched || ambiguous_count > 0) && !ambiguous_substr.empty() + && key.find(ambiguous_substr) != String::npos) { - --ambiguous_count; + if (!ambiguous_latched) + --ambiguous_count; throw Poco::TimeoutException("WedgeTestBackend: simulated ambiguous PUT (response lost, nothing landed)"); } if (definite_count > 0 && !definite_substr.empty() && key.find(definite_substr) != String::npos) @@ -243,9 +288,11 @@ class WedgeTestBackend : public CountingBackend } /// AFTER the ambiguity seam, so arming both scripts one call's attempts in order: the first /// attempt goes ambiguous, the reissue is definitively refused. - if (s3_definite_count > 0 && !s3_definite_substr.empty() && key.find(s3_definite_substr) != String::npos) + if ((s3_definite_latched || s3_definite_count > 0) && !s3_definite_substr.empty() + && key.find(s3_definite_substr) != String::npos) { - --s3_definite_count; + if (!s3_definite_latched) + --s3_definite_count; #if USE_AWS_S3 throw DB::S3Exception("WedgeTestBackend: simulated malformed request", Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); @@ -257,7 +304,7 @@ class WedgeTestBackend : public CountingBackend if (conflict_count > 0 && !conflict_substr.empty() && key.find(conflict_substr) != String::npos) { --conflict_count; - CountingBackend::putIfAbsent(key, conflict_bytes, meta); + (void)CountingBackend::write(key, conflict_bytes, std::nullopt, access); throw Poco::TimeoutException("WedgeTestBackend: a successor's object landed; our response was lost"); } { @@ -270,7 +317,7 @@ class WedgeTestBackend : public CountingBackend block_cv.wait_for(lk, std::chrono::seconds(20), [&] { return !block_armed; }); } } - return CountingBackend::putIfAbsent(key, bytes, meta); + return CountingBackend::write(key, bytes, std::nullopt, access); } private: @@ -287,9 +334,11 @@ String logPrefix(const PoolPtr & store, const RootNamespace & ns) return store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; } -CatalogEntry catalogEntryOrThrow(Backend & backend, const Layout & layout, const RootNamespace & ns) +CatalogEntry catalogEntryOrThrow(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns) { - const RefCatalog catalog = CasRefCatalog::read(backend, layout).catalog; + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + const RefCatalog catalog = CasRefCatalog::read(op, layout).catalog; const auto it = std::find_if(catalog.entries.begin(), catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; @@ -302,34 +351,38 @@ CatalogEntry catalogEntryOrThrow(Backend & backend, const Layout & layout, const /// Wedge tests address raw ref-log keys at Stage A's deterministic sentinel identity, but their /// catalog fixture must still use production's `Creating -> _ckpt -> Live` birth order. A fixed /// creator identity makes the durable genesis checkpoint deterministic too. -void admitProperlyBornEntry(Backend & backend, const Layout & layout, const RootNamespace & ns) +void admitProperlyBornEntry(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns) { + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); const CatalogEntry creating{ .ns = ns, .state = NsState::Creating, .incarnation = DB::Cas::tests::fixture::fixtureLife(ns).incarnation, .creator = CreatorFence{.server_root_id = "test", .writer_epoch = 1, .fence_generation = 1}, }; - CasRefCatalog::casAdmitEntry(backend, layout, /*gc_shards=*/1, creating); + CasRefCatalog::casAdmitEntry(op, layout, /*gc_shards=*/1, creating); - const CkptDeadline deadline{.now_ms = [] { return uint64_t{1000}; }, .deadline_ms = 60000}; - ASSERT_EQ( - CasRefCatalog::completeCreation( - backend, layout, creating, /*admitted_generation=*/1, [](uint64_t) {}, deadline), - CasRefCatalog::NamespaceCreationOutcome::Live); + ASSERT_EQ(CasRefCatalog::completeCreation(op, layout, creating), + CasRefCatalog::NamespaceCreationOutcome::Live); } CatalogEntry replaceCatalogLifeForWedgeRace( - Backend & backend, const Layout & layout, const CatalogEntry & predecessor, UInt128 successor_incarnation) + const BackendPtr & backend, const Layout & layout, const CatalogEntry & predecessor, + UInt128 successor_incarnation) { - const CasRefCatalog::Snapshot before_delete = CasRefCatalog::read(backend, layout); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + const CasRefCatalog::Snapshot before_delete = CasRefCatalog::read(op, layout); RefCatalog without_predecessor = before_delete.catalog; std::erase_if(without_predecessor.entries, [&](const CatalogEntry & entry) { return entry.ns == predecessor.ns && entry.incarnation == predecessor.incarnation; }); - if (backend.casPut(layout.refCatalogKey(), encodeRefCatalog(without_predecessor), before_delete.token).outcome - != CasOutcome::Committed) + if (!before_delete.incarnation + || !std::holds_alternative(op.replace( + layout.refCatalogKey(), encodeRefCatalog(without_predecessor), *before_delete.incarnation, + Retry::standard()))) throw std::runtime_error("test failed to retire exact predecessor catalog life"); CatalogEntry successor{ @@ -337,11 +390,12 @@ CatalogEntry replaceCatalogLifeForWedgeRace( .state = NsState::Live, .incarnation = successor_incarnation, .creator = std::nullopt}; - const CasRefCatalog::Snapshot after_delete = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot after_delete = CasRefCatalog::read(op, layout); RefCatalog reborn = after_delete.catalog; reborn.entries.push_back(successor); - if (backend.casPut(layout.refCatalogKey(), encodeRefCatalog(reborn), after_delete.token).outcome - != CasOutcome::Committed) + if (!after_delete.incarnation + || !std::holds_alternative(op.replace( + layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.incarnation, Retry::standard()))) throw std::runtime_error("test failed to publish successor catalog life"); return successor; } @@ -397,6 +451,65 @@ void armOneShotInstallFailure(const PoolPtr & store) }); } +/// Wedge `ns`'s append lane the way the engine actually reaches that state: EVERY attempt of the +/// ref-log create is unresolved (the response is lost and nothing lands), the settling read proves the +/// key still absent, and the call gives up at its own retry window having sent something -- which is +/// the `sent_any` half of the wedge rule. A one-shot fault cannot produce it: the reissue would settle +/// the key and commit. So the fault stays armed for the whole call and is cleared here. +/// +/// The pacing assertions are what make a fixture whose sleep seam is not wired FAIL rather than sleep +/// the whole window out for real. +/// Wedge `ns`'s lane over an object that IS durable: our own bytes land, the response is lost, and +/// every settling read of the key is lost too, so the call gives up at its own window without ever +/// learning that it committed. BOTH legs are required -- a readable key proves the commit inside the +/// same call and reports it committed, and a read fault that clears mid-call is simply reissued. +void wedgeLaneOverADurableObject(VirtualRetryClock & clock, WedgeTestBackend & backend, + const PoolPtr & store, const RootNamespace & ns, const String & ref) +{ + const size_t pauses_before = clock.pauseCount(); + const uint64_t clock_before = clock.nowMs(); + + backend.landed_ack_lost_substr = logPrefix(store, ns); + backend.landed_ack_lost_latched = true; + backend.fail_get_substr = logPrefix(store, ns); + backend.fail_get_latched = true; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, ref); }); + backend.landed_ack_lost_latched = false; + backend.landed_ack_lost_substr.clear(); + backend.fail_get_latched = false; + backend.fail_get_substr.clear(); + + ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + ASSERT_GT(clock.pauseCount(), pauses_before + 1) + << "the settling read's reissues must pace through the injected sleep, never a real one"; + ASSERT_GE(clock.nowMs() - clock_before, 60000u) + << "the give-up must be the read's own retry window, not a single failed read"; +} + +void wedgeLaneOnUnresolvedAppend(VirtualRetryClock & clock, WedgeTestBackend & backend, + const PoolPtr & store, const RootNamespace & ns, + const std::function & drive) +{ + const size_t pauses_before = clock.pauseCount(); + const uint64_t clock_before = clock.nowMs(); + + backend.ambiguous_substr = logPrefix(store, ns); + backend.ambiguous_latched = true; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, drive); + backend.ambiguous_latched = false; + backend.ambiguous_substr.clear(); + + ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + /// `writeTotal` cannot prove "more than one attempt": the ambiguous fault throws before ever + /// reaching the counted primitive, so a latched reissue never moves it. The pacing below is what + /// proves multiple reissues happened. + ASSERT_GT(clock.pauseCount(), pauses_before + 1) + << "the reissues must pace through the injected sleep, never a real one"; + ASSERT_LE(clock.longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; + ASSERT_GE(clock.nowMs() - clock_before, 60000u) + << "the give-up must be the call's own retry window, not a pre-attempt refusal"; +} + } /// =================================================================================== @@ -410,20 +523,17 @@ void armOneShotInstallFailure(const PoolPtr & store) TEST(CASRefWedgeEveryAttempt, AmbiguousPutWedgesTheLaneAndTheNextFlushsCreateAdoptsItExactlyOnce) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_created"}; /// Stage B (Task 4-C): `logPrefix` below computes its fault-injection match at the sentinel; /// pinning `ns` there BEFORE the first real touch keeps the real production birth landing on the /// same key the fault targets. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->resolveRef(ns, "x").has_value()) << "a wedged transaction is not applied"; const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(backend->get(wedged_key).has_value()) << "the ambiguous attempt wrote nothing"; @@ -445,16 +555,14 @@ TEST(CASRefWedgeEveryAttempt, AmbiguousPutWedgesTheLaneAndTheNextFlushsCreateAdo TEST(CASRefWedgeEveryAttempt, DurableCreatedWedgeNeedsRecoveryWhenItsFrontierCannotBePublished) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_created_frontier_failed"}; - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(backend->get(wedged_key)); const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); @@ -462,11 +570,18 @@ TEST(CASRefWedgeEveryAttempt, DurableCreatedWedgeNeedsRecoveryWhenItsFrontierCan const NamespaceLifeId life = *store->refTableLifeForTest(ns); const String ckpt_key = store->layout().refCkptKey(life); const RefCkpt ckpt_before = decodeRefCkpt(backend->get(ckpt_key)->bytes); + /// Latched, not counted: the frontier publish reissues an ambiguous replace until ITS window + /// closes, so a bounded fault would simply be outlived and the publication would succeed. backend->fail_cas_substr = ckpt_key; - backend->fail_cas_count = 200; + backend->fail_cas_latched = true; + const size_t pauses_before = clock->pauseCount(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "y"); }); + backend->fail_cas_latched = false; + backend->fail_cas_substr.clear(); + EXPECT_GT(clock->pauseCount(), pauses_before + 1) + << "the publication's reissues must pace through the injected sleep, never a real one"; EXPECT_TRUE(backend->get(wedged_key)) << "the exact wedged log was proven durable"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery) << "a durable log without a confirmed frontier must not return to Ready"; @@ -478,19 +593,17 @@ TEST(CASRefWedgeEveryAttempt, DurableCreatedWedgeNeedsRecoveryWhenItsFrontierCan TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdoption) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge-retired-before-retry"}; - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, store->layout(), ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, store->layout(), ns); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(predecessor.ns, predecessor.incarnation); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(backend->get(wedged_key)); @@ -524,7 +637,7 @@ TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdop } const CatalogEntry successor - = replaceCatalogLifeForWedgeRace(*backend, store->layout(), predecessor, UInt128{0x71f2}); + = replaceCatalogLifeForWedgeRace(backend, store->layout(), predecessor, UInt128{0x71f2}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); ASSERT_EQ(backend->putIfAbsent(store->layout().refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ @@ -557,24 +670,17 @@ TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdop /// wedge's, and the transaction is adopted -- ONCE, not once per attempt. TEST(CASRefWedgeEveryAttempt, OwnLandedAttemptIsAdoptedFromOccupiedWithoutDoubleApply) { - auto backend = std::make_shared(); - /// Disarmed while the fixture is built: the one-shot fault matches ANY key until a substring is - /// set, and the pool's own bootstrap PUT would otherwise consume it. - backend->fired = true; - auto store = openPool(backend, singleAttemptBudget()); + auto backend = std::make_shared(); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_occupied_mine"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->key_substr = logPrefix(store, ns); - backend->lose_resolve_read = true; - backend->fired = false; /// armed: the next `_log/` PUT lands and loses its ack - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOverADurableObject(*clock, *backend, store, ns, "x"); const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_TRUE(backend->get(wedged_key).has_value()) << "this fault LANDS the write; only the ack was lost"; ASSERT_TRUE(store->resolveRef(ns, "x").has_value()) << "durable, but not applied while wedged"; @@ -597,16 +703,14 @@ TEST(CASRefWedgeEveryAttempt, OwnLandedAttemptIsAdoptedFromOccupiedWithoutDouble TEST(CASRefWedgeEveryAttempt, DefiniteRefusalOfARetryAttemptKeepsTheLaneWedged) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_ambiguous_then_definite"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); const RefTxnId wedged_id = store->layout().parseRefObjectKey(wedged_key)->txn_id; @@ -631,53 +735,58 @@ TEST(CASRefWedgeEveryAttempt, DefiniteRefusalOfARetryAttemptKeepsTheLaneWedged) } /// The SAME rule one level down, and the level where it was actually broken. The test above splits the -/// two attempts across two CALLS, which the wedge already handles. Inside ONE call the controller used -/// to report the LAST attempt's outcome: an ambiguous attempt followed by a definitively refused reissue -/// came back `DefiniteFailure` -- the verdict that means "the key is provably unwritten". It is not. The -/// refusal proves only that the SECOND request never applied; the first may still be in flight and may -/// still land, and `unresolvedProvesNothingWasSent` is false for exactly that reason. So the CALL is -/// unresolved, and a definite verdict is only ever the whole call's. +/// two attempts across two CALLS, which the wedge already handles. Inside ONE call the verdict used to +/// be the LAST attempt's: an ambiguous attempt followed by a definitively refused reissue came back as +/// a proven refusal -- the verdict that means "the key is provably unwritten". It is not. The refusal +/// proves only that the SECOND request never applied; the first may still be in flight and may still +/// land. So the CALL gives up, and a refusal is only ever reported when NO attempt of the call was +/// ambiguous. /// -/// The new reason lands on the fail-close side of the predicate the ledger acts on. Asserted at compile -/// time, beside the behaviour, because a member added to the enum without classifying it is precisely -/// how the wedge would silently stop happening. -static_assert(!unresolvedProvesNothingWasSent(CasUnresolvedReason::DefiniteFailureAfterAmbiguity)); - +/// Driven on an injected clock: the reissue schedule is what makes the two attempts happen, and a real +/// one would make this test sleep. TEST(CASRefWedgeEveryAttempt, ADefiniteRefusalCannotSpeakForAnEarlierAmbiguousAttemptOfTheSameCall) { #if !USE_AWS_S3 - GTEST_SKIP() << "DefiniteFailure classification requires S3 error types (USE_AWS_S3 off)"; + GTEST_SKIP() << "the store-refusal classification requires S3 error types (USE_AWS_S3 off)"; #else auto backend = std::make_shared(); - CasRequestController controller(backend, twoAttemptBudget()); - const std::function fence_ok = [] { return true; }; - - /// One call, two attempts: ambiguous, then definitively refused. + uint64_t clock = 0; + size_t pauses = 0; + /// The sleep advances the same clock the policy window is read from, plus a millisecond, because + /// full jitter can draw a zero pause and a clock that does not move never reaches the deadline. + CasRequests requests(backend, Fence::open(), + [&clock]() -> uint64_t { return clock; }, + [&clock, &pauses](uint64_t ms) { ++pauses; clock += ms + 1; }); + CasOperation op = requests.admit(); + + /// One call: the first attempt ambiguous, and every attempt after it refused by the store. The + /// refusal has to stay armed, because the engine does not stop at it -- a refusal that follows an + /// ambiguous attempt of the same call proves nothing about that attempt, so the call keeps + /// reissuing until its own window closes. backend->ambiguous_substr = "key/"; backend->ambiguous_count = 1; backend->s3_definite_substr = "key/"; - backend->s3_definite_count = 1; - - CasUnresolvedReason reason = CasUnresolvedReason::NotUnresolved; - const CasWriteOutcome outcome = - controller.putIfAbsentControlled("key/haunted", "bytes", fence_ok, /*out_token=*/nullptr, &reason); + backend->s3_definite_latched = true; - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved) - << "a definite refusal of the SECOND attempt cannot retire the first attempt's ambiguity"; - EXPECT_EQ(reason, CasUnresolvedReason::DefiniteFailureAfterAmbiguity); - EXPECT_FALSE(unresolvedProvesNothingWasSent(reason)) + const WriteResult haunted = op.create("key/haunted", "bytes", Retry::within(30'000)); + const auto * gave_up = std::get_if(&haunted); + ASSERT_TRUE(gave_up != nullptr) + << "a store refusal of a LATER attempt cannot retire the first attempt's ambiguity"; + EXPECT_TRUE(gave_up->sent_any) << "the caller must keep protecting itself: an earlier attempt was sent and may yet land"; - EXPECT_FALSE(backend->get("key/haunted").has_value()) + EXPECT_GT(pauses, 1u) << "the reissues must pace through the injected sleep, never a real one"; + EXPECT_GE(clock, 20'000u) << "and the call must end at its own 30 s window"; + backend->s3_definite_latched = false; + CasOperation reader = requests.admit(); + EXPECT_FALSE(reader.head("key/haunted", Retry::within(30'000)).has_value()) << "and the key is still empty -- which is exactly why an absent read settles nothing"; - /// THE CONTROL. Aggregation must not soften a definite refusal that speaks for the whole call: with - /// no ambiguous predecessor, the first attempt's whitelisted rejection is still `DefiniteFailure`, - /// and the ledger may still free the id on it. + /// THE CONTROL. Aggregation must not soften a refusal that speaks for the whole call: with no + /// ambiguous predecessor, the first attempt's rejection is still `Refused`, and the ledger may + /// still free the id on it. backend->s3_definite_count = 1; - CasUnresolvedReason clean_reason = CasUnresolvedReason::NotUnresolved; - EXPECT_EQ(controller.putIfAbsentControlled("key/clean", "bytes", fence_ok, /*out_token=*/nullptr, &clean_reason), - CasWriteOutcome::DefiniteFailure); - EXPECT_EQ(clean_reason, CasUnresolvedReason::NotUnresolved); + CasOperation clean = requests.admit(); + EXPECT_TRUE(std::holds_alternative(clean.create("key/clean", "bytes", Retry::within(30'000)))); #endif } @@ -692,24 +801,34 @@ TEST(CASRefWedgeEveryAttempt, ADefiniteRefusalAfterAnAmbiguousAttemptOfTheSameCa GTEST_SKIP() << "DefiniteFailure classification requires S3 error types (USE_AWS_S3 off)"; #else auto backend = std::make_shared(); - auto store = openPool(backend, twoAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_one_call_ambiguous_then_definite"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); + /// The first attempt goes ambiguous; the store then refuses every attempt after it, for as long as + /// the call keeps making them. A one-shot refusal would not reproduce the sequence this test names: + /// the engine reissues past it, and the third attempt would simply commit. backend->ambiguous_substr = logPrefix(store, ns); backend->ambiguous_count = 1; backend->s3_definite_substr = logPrefix(store, ns); - backend->s3_definite_count = 1; + backend->s3_definite_latched = true; const uint64_t wedged_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(); const uint64_t definite_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendDefiniteFailure].load(); + const size_t pauses_before = clock->pauseCount(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + backend->s3_definite_latched = false; + backend->s3_definite_substr.clear(); + backend->ambiguous_substr.clear(); + EXPECT_GT(clock->pauseCount(), pauses_before + 1) + << "the reissues must pace through the injected sleep, never a real one"; EXPECT_TRUE(store->refLaneWedgedForTest(ns)) << "one call whose first attempt is unresolved leaves an object that may become durable"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged) @@ -742,18 +861,16 @@ TEST(CASRefWedgeEveryAttempt, ADefiniteRefusalAfterAnAmbiguousAttemptOfTheSameCa TEST(CASRefWedgeEveryAttempt, SuccessorSealAtTheWedgedKeyRejectsConclusivelyAndSourcesPrevEpochSeal) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/wedge_sealed"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); const uint64_t epoch = store->liveWriterEpoch(); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); const RefTxnId seal_id = layout.parseRefObjectKey(wedged_key)->txn_id; ASSERT_EQ(seal_id.writer_epoch, epoch); @@ -812,7 +929,7 @@ TEST(CASRefWedgeEveryAttempt, OrdinaryFirstAppendAfterASealedTransitionCarriesTh const RootNamespace ns{"srv1/prev_epoch_seal_roundtrip"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `readRefLogTxn` above /// reads that exact key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); const uint64_t epoch = store->liveWriterEpoch(); publishEmptyPart(store, ns, "x"); @@ -846,7 +963,7 @@ TEST(CASRefWedgeEveryAttempt, GenesisBirthAtAHighEpochCarriesNoPrevEpochSeal) const RootNamespace ns{"srv1/genesis_at_five"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `readRefLogTxn` above /// reads that exact key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); bumpFenceGeneration(store, 5); ASSERT_EQ(store->liveWriterEpoch(), 5u); @@ -870,18 +987,16 @@ TEST(CASRefWedgeEveryAttempt, GenesisBirthAtAHighEpochCarriesNoPrevEpochSeal) TEST(CASRefWedgeEveryAttempt, ForeignNonSealOccupantIsCorruptedDataAndSchedulesARemount) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_foreign"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); const uint64_t remounts_before = store->scheduleRemountCallCountForTest(); @@ -906,7 +1021,7 @@ TEST(CASRefWedgeEveryAttempt, AppendSiteProvenDifferentObjectAlsoSchedulesARemou auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/append_site_foreign"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); /// Occupy the id the next append will derive with a foreign object, so its create conflicts and @@ -933,19 +1048,17 @@ TEST(CASRefWedgeEveryAttempt, AppendSiteProvenDifferentObjectAlsoSchedulesARemou TEST(CASRefWedgeEveryAttempt, RetryUnderAnOlderAdmissionGenerationSendsNothing) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_old_generation"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); const uint64_t epoch = store->liveWriterEpoch(); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_EQ(store->wedgedAdmittedGenerationForTest(ns), store->fenceGeneration()) << "the wedge records the generation it was admitted under"; @@ -971,20 +1084,18 @@ TEST(CASRefWedgeEveryAttempt, RetryUnderAnOlderAdmissionGenerationSendsNothing) TEST(CASRefWedgeEveryAttempt, ResultReleasedAfterAFenceBumpAndSuccessorSealIsInert) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/wedge_blocked_io"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); const uint64_t epoch = store->liveWriterEpoch(); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); const RefTxnId seal_id = layout.parseRefObjectKey(wedged_key)->txn_id; const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); @@ -1024,18 +1135,16 @@ TEST(CASRefWedgeEveryAttempt, ResultReleasedAfterAFenceBumpAndSuccessorSealIsIne TEST(CASRefWedgeEveryAttempt, ResultReleasedAfterTheWedgeIdentityChangedIsInert) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_identity_changed"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); const RefTxnId wedged_id = store->layout().parseRefObjectKey(wedged_key)->txn_id; const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); @@ -1065,21 +1174,17 @@ TEST(CASRefWedgeEveryAttempt, ResultReleasedAfterTheWedgeIdentityChangedIsInert) /// `NeedsRecovery`. It drops the attempt and forbids another write until replay catches the cache up. TEST(CASRefWedgeEveryAttempt, KnownDurableInstallFailureMovesDirectlyToRecovery) { - auto backend = std::make_shared(); - backend->fired = true; /// disarmed while the fixture is built (see the adoption test above) - auto store = openPool(backend, singleAttemptBudget()); + auto backend = std::make_shared(); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/wedge_floor"}; /// Stage B (Task 4-C): pin to the sentinel before the first real touch -- `logPrefix` below matches /// its fault at that key. - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - backend->key_substr = logPrefix(store, ns); - backend->lose_resolve_read = true; - backend->fired = false; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); - ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + wedgeLaneOverADurableObject(*clock, *backend, store, ns, "x"); const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_TRUE(backend->get(wedged_key).has_value()) << "the wedged transaction is durable"; /// The adoption reaches its install region and the install throws. @@ -1116,7 +1221,7 @@ TEST(CASRefWedgeEveryAttempt, AppendSiteMeetingASuccessorSealIsAConclusiveReject auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/append_site_seal"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); const uint64_t epoch = store->liveWriterEpoch(); @@ -1166,7 +1271,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesAConclusiveFirstRefLogRejectio auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/birth_ckpt_debris"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch const RefTxnId genesis{store->liveWriterEpoch(), 1}; const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); @@ -1201,7 +1306,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesALaterConclusiveRejection) auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/birth_ckpt_survives_live"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch /// ONE `publishEmptyPart` reaches sequence 2 (the precommit-add chunk at seq 1 carries the first /// `NamespaceBirth`, the promote chunk lands at seq 2), so `next` /// below is the SAME `{epoch, 3}` the sibling `AppendSiteMeetingASuccessorSealIsAConclusiveRejectionNotInterference` @@ -1238,67 +1343,53 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesALaterConclusiveRejection) TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthIsAmbiguous) { auto backend = std::make_shared(); - auto store = openPool(backend, singleAttemptBudget()); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/birth_ckpt_ambiguous"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); const auto ckpt_before = backend->get(ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation checkpoint must exist before the first ref-log attempt"; - backend->ambiguous_substr = logPrefix(store, ns); - backend->ambiguous_count = 1; - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishEmptyPart(store, ns, "x"); }); - - ASSERT_TRUE(store->refLaneWedgedForTest(ns)) << "an ambiguous outcome must WEDGE the lane, not " - "resolve into one of the conclusive branches"; + /// The wedge is the assertion: an ambiguous outcome must not resolve into one of the conclusive + /// branches, so `wedgeLaneOnUnresolvedAppend` insisting on it is what this row needs. + wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { publishEmptyPart(store, ns, "x"); }); const auto ckpt_after = backend->get(ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) << "the creation checkpoint must survive an ambiguous first ref-log outcome unchanged"; } -/// Final review F5: the two other removed call sites, given their own first-`NamespaceBirth` survival rows. -/// The reversed test above pins the `SuccessorSeal` branch; the sibling below it pins the ambiguous -/// branch (never called it in the first place). The remaining two -- occupant-unreadable -/// (`CORRUPTED_DATA` from a failed adjudication read) and genuine foreign interference -- had no -/// first-`NamespaceBirth` row at all: `AppendSiteFaultsWhenTheOccupantCannotBeRead`, -/// `ForeignNonSealOccupantIsCorruptedDataAndSchedulesARemount`, and `WellFormedNonSealOccupantIsStillForeign` +/// The other two former cleanup call sites, given their own first-`NamespaceBirth` survival rows: an +/// unnameable occupant, and genuine foreign interference. Neither had such a row, because +/// `AppendSiteWedgesWhenTheSettlingReadNamesNoOccupant`, +/// `ForeignNonSealOccupantIsCorruptedDataAndSchedulesARemount` and `WellFormedNonSealOccupantIsStillForeign` /// all `publishEmptyPart` FIRST, so none of them ever carries a `birth_contribution` -- a reinstated -/// GUARDED cleanup at either of these two sites would pass the whole suite with no first-transaction case to -/// catch it. Mirrors `WellFormedNonSealOccupantIsStillForeign`'s occupant shape (a decodable, well-formed -/// NON-seal transaction at the derived key), moved to sequence 1 of a namespace with no prior ref-log -/// transaction, so this attempt's own PUT is its first. -TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthOccupantCannotBeRead) +/// guarded cleanup at either site would pass the whole suite with no first-transaction case to catch it. +TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthNamesNoOccupant) { auto backend = std::make_shared(); auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/birth_ckpt_occupant_unreadable"}; - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); const RefTxnId genesis{store->liveWriterEpoch(), 1}; const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); const auto ckpt_before = backend->get(ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation checkpoint must exist before the first ref-log attempt"; - backend->conflict_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), genesis); - backend->conflict_bytes = epochSealBytes(ns, genesis); - backend->conflict_count = 1; - /// Proper birth makes recovery first probe this absent log key. Skip that probe and the resolve - /// read that PROVES the conflict; fail only the adjudication read after it, so the occupant's - /// identity (seal vs. breach) cannot be determined. - backend->fail_get_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), genesis); - backend->fail_get_skip = 2; - backend->fail_get_count = 1; + /// The store refuses the birth create's precondition while the key is in fact ABSENT, so the + /// settling read proves absence and the conflict comes back naming no occupant at all. + backend->refuse_precondition_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), genesis); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { publishEmptyPart(store, ns, "x"); }); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishEmptyPart(store, ns, "x"); }); const auto ckpt_after = backend->get(ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) - << "the creation checkpoint must survive an occupant-unreadable first ref-log outcome unchanged"; + << "the creation checkpoint must survive an unnameable first ref-log occupant unchanged"; } /// The other former call site: a genuine breach of write-exclusivity at the first `NamespaceBirth` @@ -1309,7 +1400,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesFirstNamespaceBirthForeignInte auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/birth_ckpt_foreign_interference"}; - admitProperlyBornEntry(*backend, store->layout(), ns); + admitProperlyBornEntry(backend, store->layout(), ns); const RefTxnId genesis{store->liveWriterEpoch(), 1}; const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); @@ -1333,44 +1424,96 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesFirstNamespaceBirthForeignInte << "the creation checkpoint must survive a foreign-interference first ref-log outcome unchanged"; } -/// The same conflict, but the read that would tell a seal from a breach fails. We must then decide -/// NEITHER: fencing the mount would be a guess, and reporting a conclusive rejection would acknowledge -/// a deposition nobody observed. The id is not consumed, so the next attempt re-derives it and -/// classifies again — deferring costs one round trip and decides nothing wrongly. -TEST(CASRefWedgeEveryAttempt, AppendSiteFaultsWhenTheOccupantCannotBeRead) +/// A conflict that names NO occupant. The store refused this create's precondition, and the settling +/// read then found the key gone -- so nothing can be adjudicated: fencing the mount would be a guess, +/// and reporting a conclusive rejection would acknowledge a deposition nobody observed. We decide +/// NEITHER and WEDGE, which is what the wedge-resolution site does with the identical observation. The +/// lane must therefore stay recoverable: the next flush re-creates at the same key and adjudicates +/// whatever it finds. A terminal `Faulted` here would cost the table its writes until a remount over a +/// read that the very next attempt may complete. +TEST(CASRefWedgeEveryAttempt, AppendSiteWedgesWhenTheSettlingReadNamesNoOccupant) { auto backend = std::make_shared(); auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/append_site_unreadable"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); const RefTxnId next{store->liveWriterEpoch(), 3}; const uint64_t remounts_before = store->scheduleRemountCallCountForTest(); - backend->conflict_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), next); - backend->conflict_bytes = epochSealBytes(ns, next); - backend->conflict_count = 1; - /// Skip the resolve read that PROVES the conflict; fail only the adjudication read after it. - backend->fail_get_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), next); - backend->fail_get_skip = 1; - backend->fail_get_count = 1; + /// A precondition refusal is a VALUE, not an exception, so this call carries no ambiguity -- which + /// is what lets the settling read's answer be the whole verdict. The key is absent, so that read + /// proves absence and names nobody. + backend->refuse_precondition_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), next); const uint64_t deferred_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { store->dropRef(ns, "x"); }); + const uint64_t wedged_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(), deferred_before + 1) << "the deferral is the one quiet arm here -- it must be counted or a starved loud path is invisible"; - EXPECT_TRUE(store->mayMutate()) << "the table faults without guessing that the whole mount is corrupt"; - EXPECT_EQ(store->scheduleRemountCallCountForTest(), remounts_before) << "nor schedule a remount"; + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(), wedged_before + 1) + << "and the lane it leaves behind is a wedge, so the wedge counter must say so"; + EXPECT_TRUE(store->mayMutate()) << "the table defers without guessing that the whole mount is corrupt"; + EXPECT_EQ(store->scheduleRemountCallCountForTest(), remounts_before) << "nor schedules a remount"; EXPECT_EQ(store->lastEpochSealForTest(ns), std::nullopt) - << "nor record a deposition that was never actually observed"; - EXPECT_FALSE(store->refLaneWedgedForTest(ns)) << "nothing of ours became durable, so nothing is wedged"; - EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Faulted); - expectThrowsCode(DB::ErrorCodes::INVALID_STATE, [&] { store->dropRef(ns, "x"); }); + << "nor records a deposition that was never actually observed"; + EXPECT_TRUE(store->refLaneWedgedForTest(ns)) + << "the key holds something this call could not name, which is exactly what a wedge is for"; + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged); + + /// RECOVERABLE, and this is the half a terminal `Faulted` forecloses: with the store answering + /// normally again, the next flush's bounded create lands the wedged transaction and adopts it. + /// Triggered by a DIFFERENT ref (not another drop of "x"): the wedge's own adopted drop already + /// removes "x", so a second "drop x" from this same call would find it already gone. + backend->refuse_precondition_substr.clear(); + EXPECT_NO_THROW(publishEmptyPart(store, ns, "y")); + EXPECT_FALSE(store->refLaneWedgedForTest(ns)); + EXPECT_FALSE(store->resolveRef(ns, "x").has_value()) << "the adopted wedge applied its drop"; + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); } +/// The OTHER observation that names no occupant, and the one the arm is really about: the settling read +/// does not merely find the key gone, it FAILS -- definitively, so the read engine surfaces it rather +/// than reissuing. Guarded to `USE_AWS_S3` builds, where alone a store refusal is recognised as one; +/// without that classification the same fault is a deterministic local failure, a different arm. +#if USE_AWS_S3 +TEST(CASRefWedgeEveryAttempt, AppendSiteWedgesWhenTheSettlingReadItselfIsRefused) +{ + auto backend = std::make_shared(); + auto store = openPool(backend); + const Layout & layout = store->layout(); + const RootNamespace ns{"srv1/append_site_read_refused"}; + admitProperlyBornEntry(backend, store->layout(), ns); + publishEmptyPart(store, ns, "x"); + + const RefTxnId next{store->liveWriterEpoch(), 3}; + const uint64_t remounts_before = store->scheduleRemountCallCountForTest(); + + backend->refuse_precondition_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), next); + backend->refuse_read_after_precondition = true; + + const uint64_t deferred_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendOccupantUnreadable].load(), deferred_before + 1); + EXPECT_TRUE(store->mayMutate()); + EXPECT_EQ(store->scheduleRemountCallCountForTest(), remounts_before); + EXPECT_TRUE(store->refLaneWedgedForTest(ns)); + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged); + + /// Triggered by a DIFFERENT ref (not another drop of "x"): the wedge's own adopted drop already + /// removes "x", so a second "drop x" from this same call would find it already gone. + backend->refuse_read_after_precondition = false; + backend->refuse_precondition_substr.clear(); + EXPECT_NO_THROW(publishEmptyPart(store, ns, "y")); + EXPECT_FALSE(store->resolveRef(ns, "x").has_value()) << "the adopted wedge applied its drop"; + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); +} +#endif + /// A WELL-FORMED ref-log transaction of this namespace at this id, which simply is not a seal, must be /// adjudicated `Foreign` on CONTENT — not because it failed to decode. The sibling test above reaches /// the same verdict through an undecodable body, so without this one the classifier could be deciding @@ -1381,7 +1524,7 @@ TEST(CASRefWedgeEveryAttempt, WellFormedNonSealOccupantIsStillForeign) auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/append_site_wellformed_foreign"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); const RefTxnId next{store->liveWriterEpoch(), 3}; @@ -1414,7 +1557,7 @@ TEST(CASRefWedgeEveryAttempt, ALiveEpochSealIsNeverStampedAsItsOwnPrevEpochSeal) auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/live_epoch_seal"}; - admitProperlyBornEntry(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishEmptyPart(store, ns, "x"); const uint64_t epoch = store->liveWriterEpoch(); @@ -1474,3 +1617,126 @@ TEST(CASRefWedgeEveryAttempt, ALiveEpochSealIsNeverStampedAsItsOwnPrevEpochSeal) /// re-claim storm. EXPECT_EQ(store->scheduleRemountCallCountForTest(), remounts_before); } + +/// ================================================================================================ +/// The ref lane's four write arms, after the append moved onto an admitted operation. Each of these +/// pins one arm the old outcome enum could not express, and each is reachable only through the whole +/// pool, because what the arm decides is a LANE TRANSITION, not a return value. +/// ================================================================================================ + +/// The store's own proven refusal returns the exact attempt to `Ready`. It is the one non-commit that +/// must NOT wedge: the request never applied, so there is nothing at the key for a wedge to resolve, +/// and the txn id stays underived. Guarded to `USE_AWS_S3` builds, where alone the refusal class is +/// recognised -- without it the same fault is an ambiguity, which is a different arm. +#if USE_AWS_S3 +TEST(CASRefLane, RefusedReturnsTheAttemptToReadyAndDoesNotWedge) +{ + auto backend = std::make_shared(); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); + const RootNamespace ns{"srv1/ref_lane_refused"}; + admitProperlyBornEntry(backend, store->layout(), ns); + publishEmptyPart(store, ns, "x"); + + backend->s3_definite_substr = logPrefix(store, ns); + backend->s3_definite_count = 1; + + const uint64_t wedged_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(); + const uint64_t definite_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendDefiniteFailure].load(); + + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + + EXPECT_FALSE(store->refLaneWedgedForTest(ns)) + << "a proven refusal wrote nothing, so there is nothing for a wedge to resolve"; + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendDefiniteFailure].load(), definite_before + 1); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(), wedged_before); + EXPECT_TRUE(store->resolveRef(ns, "x").has_value()) << "the refused drop applied nothing"; + + /// And the id was never consumed: the next caller re-derives it and lands the same transaction. + EXPECT_NO_THROW(store->dropRef(ns, "x")); + EXPECT_FALSE(store->resolveRef(ns, "x").has_value()); +} +#endif + +/// A ref-log create that COMMITS and only then loses its admission is reported unresolved, and the +/// lane wedges over an object that is in fact durable. That is the conservative half of the contract: +/// the call may not claim a commit it can no longer stand behind, and the wedge is what makes the next +/// flush settle the key rather than write around it. +TEST(CASRefLane, PostCommitFenceLossWedges) +{ + auto backend = std::make_shared(); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); + const RootNamespace ns{"srv1/ref_lane_post_commit_fence_loss"}; + admitProperlyBornEntry(backend, store->layout(), ns); + publishEmptyPart(store, ns, "x"); + ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); + + const uint64_t wedged_before = ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(); + + /// The fence is lost INSIDE the write window, so the object lands and the call may not claim it. + backend->armBlock(logPrefix(store, ns)); + std::exception_ptr caller_error; + std::thread writer([&] + { + try { store->dropRef(ns, "x"); } + catch (...) { caller_error = std::current_exception(); } + }); + backend->awaitBlockEntered(); + store->tripMountLost(); + backend->releaseBlock(); + writer.join(); + + ASSERT_TRUE(caller_error != nullptr) << "no acknowledgement: the caller must not be told this succeeded"; + EXPECT_TRUE(store->refLaneWedgedForTest(ns)) + << "the object is durable, so the lane must not be returned to Ready"; + EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(), wedged_before + 1); + EXPECT_TRUE(backend->get(store->wedgedKeyForTest(ns)).has_value()) + << "the write landed -- what was refused is the CLAIM, not the object"; +} + +/// The liveness the ledger hands its operations carries the runtime's own facts and NOT a generation +/// term -- the generation is the operation's, presented once at admission. This pins the half that is +/// easy to lose in that split: a runtime retired while the fence generation never MOVES must still end +/// the operation, and it must end it as an unresolved write rather than an installed commit. +TEST(CASRefLane, LivenessPredicateWithoutGenerationTermStillRefusesARetiredRuntime) +{ + auto backend = std::make_shared(); + auto store = openPool(backend, wedgeTestBudget()); + auto clock = VirtualRetryClock::installOn(store); + const RootNamespace ns{"srv1/ref_lane_retired_runtime"}; + admitProperlyBornEntry(backend, store->layout(), ns); + publishEmptyPart(store, ns, "x"); + ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); + + const NamespaceLifeId life = DB::Cas::tests::fixture::fixtureLife(ns); + const uint64_t generation_before = store->fenceGeneration(); + const uint64_t writes_before = backend->writeTotal(); + + backend->armBlock(logPrefix(store, ns)); + std::exception_ptr caller_error; + std::thread writer([&] + { + try { store->dropRef(ns, "x"); } + catch (...) { caller_error = std::current_exception(); } + }); + /// The append's own create is parked, which is what makes the retirement below land INSIDE the + /// write window rather than before the lane ever armed an attempt. + backend->awaitBlockEntered(); + /// The mount fence is untouched throughout, so the runtime term is the only thing that can end this + /// operation. Retirement also detaches the cache slot, so the lane state is no longer observable -- + /// what this pins is that the CALLER is refused, which is the property the term exists for. + store->invalidateRemovedCatalogLife(life); + backend->releaseBlock(); + writer.join(); + + EXPECT_EQ(store->fenceGeneration(), generation_before) + << "the fence never moved -- a generation term could not have produced this refusal"; + EXPECT_GT(backend->writeTotal(), writes_before) << "the parked create did reach the store"; + ASSERT_TRUE(caller_error != nullptr) << "a retired runtime must not be told its append succeeded"; + /// The retry-safe class, not a hard failure: a retirement racing a write is an ordinary fact about + /// the world, and the caller retries against a fresh observation. + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { std::rethrow_exception(caller_error); }); +} diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index c4e173fb96c5..d3fefe8109b4 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -53,7 +53,7 @@ extern const Event CASRefSnapshotPutBytes; extern const Event CASRefSnapshotTailLogs; extern const Event CASRefSnapshotPublishDispatched; extern const Event CASRefSnapshotPublishBackoff; -extern const Event CASConditionalWriteFenceLostPostWrite; +extern const Event CASRequestFenceLostPostWrite; extern const Event CASRefRecoveryEpochSealed; extern const Event CASRefRecoveryRetries; } @@ -72,31 +72,75 @@ using DB::Cas::tests::writeSealAt; namespace { -/// The operation deadline every SINGLE-ATTEMPT fixture in this file uses, and the reason it is -/// deliberately NOT `attempt_timeout_ms`. -/// -/// Those fixtures exist to make one injected ambiguous response conclusive, and `max_attempts = 1` -/// alone achieves that: with retries allowed the controller would resolve-before-reissue and report a -/// definite outcome instead. The deadline contributes nothing to that -- but setting it EQUAL to the -/// attempt timeout collapses the controller's pre-send gate into a race. The deadline is captured as -/// `now + operation_deadline_ms` and the gate asks `now + attempt_timeout_ms > deadline` -/// (`CasRequestControl.cpp`), so equal values reduce it to `now_2 > now_1`: ONE elapsed millisecond -/// between the two clock reads refuses the operation with NOTHING SENT, the injected fault is never -/// reached, and the product correctly does not wedge -- flipping every wedge expectation downstream. -/// -/// That is not hypothetical. It took down -/// `CASRefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts` on 5 of 6 sanitizer -/// CI runs (fixed in `8f9e63c7a19`), `CASRefInstallSafety.UncertainPrecommitKeepsItsCleanupOwnerAndItsBody` -/// under parallel-build load, and `CASRefWriterAppendLane.WedgedLaneBlocksSameTableWhileOtherTableProceeds` -/// in a full-binary ASan run -- the last one with the mechanism named verbatim in the thrown message -/// ("refused BEFORE any request was sent ... the operation deadline rejected before the first request"). -/// -/// A wide deadline keeps the request always actually sent, so what the test observes is the fault it -/// injected rather than the machine it ran on. A fixture that genuinely wants the pre-send REFUSAL -/// must drive it deterministically with a frozen clock (see `gtest_cas_ref_install_safety.cpp`'s -/// `openPoolFenceControlled`), never by racing the wall clock. +/// The operation deadline every wedge fixture in this file uses, distinct from `attempt_timeout_ms` +/// because `validateCasRequestBudget` requires the two to differ strictly: `Pool::open` refuses a +/// budget where `attempt_timeout_ms >= operation_deadline_ms` with `BAD_ARGUMENTS`, so an equal or +/// smaller value would refuse to open the pool at all rather than exercise any fixture below. constexpr uint64_t kSingleAttemptDeadlineMs = 5000; +/// The budget every wedge fixture here uses. It bounds the mount lease's own admission arithmetic and +/// nothing else: a write's ATTEMPT COUNT is the `Retry` policy's, and the ref lane's is `standard`, so +/// no budget field can make an injected fault conclusive. What does is `driveToTheWedge` below -- +/// the fault stays armed for the whole call while the injected clock carries it to its own deadline. +CasRequestBudget wedgeTestBudget() +{ + CasRequestBudget budget; + budget.attempt_timeout_ms = 100; + budget.operation_deadline_ms = kSingleAttemptDeadlineMs; + budget.lease_safety_margin_ms = 100; + return budget; +} + +/// The engine reissues an unresolved write until its OWN retry window closes, and that window is +/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine +/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline +/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling +/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the +/// closures own the clock, so it outlives everything that can still read it. +class VirtualRetryClock +{ +public: + static std::shared_ptr installOn(const PoolPtr & store) + { + auto clock = std::make_shared(); + store->setCasRequestNowFnForTest([clock] { return clock->nowMs(); }); + store->setCasRetrySleepForTest([clock](uint64_t ms) { clock->advance(ms); }); + return clock; + } + + uint64_t nowMs() const + { + std::lock_guard lock(mutex); + return now_ms; + } + size_t pauseCount() const + { + std::lock_guard lock(mutex); + return pauses; + } + uint64_t longestPause() const + { + std::lock_guard lock(mutex); + return longest_pause; + } + + void advance(uint64_t ms) + { + std::lock_guard lock(mutex); + /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move + /// would leave the loop reissuing for ever against a fault that never clears. + now_ms += ms + 1; + ++pauses; + longest_pause = std::max(longest_pause, ms); + } + +private: + mutable std::mutex mutex; + uint64_t now_ms = 0; + size_t pauses = 0; + uint64_t longest_pause = 0; +}; + /// A `CasEvent` sink safe to hand to `Pool::setEventSink`: the emit runs on whatever thread the pool's /// background syncer happens to be on, and the test reads the accumulated events afterward from the /// main test thread with no other ordering between the two -- a bare `std::vector` there is a real data @@ -181,9 +225,68 @@ void publishWithProductionBirth(const PoolPtr & store, const RootNamespace & ns, build->promote(ns, ref, build->buildId(), id); } -CatalogEntry catalogEntryOrThrow(Backend & backend, const Layout & layout, const RootNamespace & ns) +/// Fixture observations of durable state run on an OPEN fence: they are not writes a mount admitted, +/// and each owns the `CasRequests` its operation borrows, so none of these hands one back. +std::optional readCkptForTest(const BackendPtr & backend, const Layout & layout, + const NamespaceLifeId & life) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return readCkpt(op, layout, life); +} + +CasRefCatalog::Snapshot readCatalogForTest(const BackendPtr & backend, const Layout & layout) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return CasRefCatalog::read(op, layout); +} + +/// A fixture's own conditional replace, for the races these tests stage by hand. +bool replaceForTest(const BackendPtr & backend, const String & key, const String & bytes, + const Incarnation & expected) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return std::holds_alternative(op.replace(key, bytes, expected, Retry::standard())); +} + +void casAdmitEntryForTest(const BackendPtr & backend, const Layout & layout, uint64_t gc_shards, + const CatalogEntry & entry) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + CasRefCatalog::casAdmitEntry(op, layout, gc_shards, entry); +} + +std::optional lifeIfCatalogedForTest(const BackendPtr & backend, const Layout & layout, + const RootNamespace & ns) { - const RefCatalog catalog = CasRefCatalog::read(backend, layout).catalog; + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return CasRefCatalog::lifeIfCataloged(op, layout, ns); +} + +uint64_t allocateWriterEpochForTest(const BackendPtr & backend, const Layout & layout, const String & server_root_id) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return allocateWriterEpoch(op, layout, server_root_id, EpochMintPolicy::NormalMount, 0, + [] { return RefCatalog{}; }); +} + +/// A fixture enumeration on an open fence: the primitive `list` override in this file's test backend +/// hides the legacy name, and a test walking a prefix should ride the same engine production does. +KeyPage listForTest(const BackendPtr & backend, const String & prefix, const String & cursor, size_t limit) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return op.list(prefix, cursor, limit, Retry::standard()); +} + +CatalogEntry catalogEntryOrThrow(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns) +{ + const RefCatalog catalog = readCatalogForTest(backend, layout).catalog; const auto it = std::find_if(catalog.entries.begin(), catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; @@ -194,16 +297,18 @@ CatalogEntry catalogEntryOrThrow(Backend & backend, const Layout & layout, const } CatalogEntry replaceCatalogLifeForRuntimeRace( - Backend & backend, const Layout & layout, const CatalogEntry & predecessor, UInt128 successor_incarnation) + const BackendPtr & backend, const Layout & layout, const CatalogEntry & predecessor, + UInt128 successor_incarnation) { - const CasRefCatalog::Snapshot before_delete = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot before_delete = readCatalogForTest(backend, layout); RefCatalog without_predecessor = before_delete.catalog; std::erase_if(without_predecessor.entries, [&](const CatalogEntry & entry) { return entry.ns == predecessor.ns && entry.incarnation == predecessor.incarnation; }); - if (backend.casPut(layout.refCatalogKey(), encodeRefCatalog(without_predecessor), before_delete.token).outcome - != CasOutcome::Committed) + if (!before_delete.incarnation + || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(without_predecessor), + *before_delete.incarnation)) throw std::runtime_error("test failed to retire exact predecessor catalog life"); CatalogEntry successor{ @@ -211,11 +316,11 @@ CatalogEntry replaceCatalogLifeForRuntimeRace( .state = NsState::Live, .incarnation = successor_incarnation, .creator = std::nullopt}; - const CasRefCatalog::Snapshot after_delete = CasRefCatalog::read(backend, layout); + const CasRefCatalog::Snapshot after_delete = readCatalogForTest(backend, layout); RefCatalog reborn = after_delete.catalog; reborn.entries.push_back(successor); - if (backend.casPut(layout.refCatalogKey(), encodeRefCatalog(reborn), after_delete.token).outcome - != CasOutcome::Committed) + if (!after_delete.incarnation + || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.incarnation)) throw std::runtime_error("test failed to publish successor catalog life"); return successor; } @@ -252,7 +357,7 @@ struct CompletedRemovingFixture }; CompletedRemovingFixture prepareResidentRemovalForDrain( - const PoolPtr & store, Backend & backend, const RootNamespace & ns, Gc & gc) + const PoolPtr & store, const BackendPtr & backend, const RootNamespace & ns, Gc & gc) { publishWithProductionBirth(store, ns, "predecessor"); const CatalogEntry predecessor = catalogEntryOrThrow(backend, store->layout(), ns); @@ -268,9 +373,9 @@ CompletedRemovingFixture prepareResidentRemovalForDrain( if (runRegularRoundReclaiming(gc).deferred) throw std::runtime_error("fixture terminal fold unexpectedly deferred"); - const GcState state = decodeGcState(backend.get(store->layout().gcStateKey())->bytes); + const GcState state = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend.get(store->layout().foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + backend->get(store->layout().foldSealKey(state.snap_generation, state.snap_attempt))->bytes); const auto row = seal.ref_lives.find(predecessor.incarnation); if (row == seal.ref_lives.end() || !row->second.cleanup_evidence) throw std::runtime_error("fixture terminal fold produced no cleanup evidence"); @@ -355,11 +460,7 @@ class RefWriterTestBackend : public CountingBackend DB::Cas::tests::seedPoolMetaForRestart(*this); } - using CountingBackend::get; using CountingBackend::getStream; - using CountingBackend::putIfAbsent; - using CountingBackend::putOverwrite; - using CountingBackend::casPut; void clearRequestJournal() { @@ -400,6 +501,13 @@ class RefWriterTestBackend : public CountingBackend String fault_key_substr; int fault_count = 0; + /// LATCHED: a COUNT can no longer make an injected fault conclusive, because the write engine + /// settles each ambiguity by an exact read and then REISSUES -- a fault that runs out mid-call is + /// answered by the next attempt instead of by the call's own deadline, which is the difference + /// between a wedge and a commit. While this is set the count is topped up before every matching + /// create, so the fault outlasts the whole call. + bool fault_latched = false; + bool ckpt_conflict_latched = false; /// Let the first `fault_skip` matching PUTs through untouched before `fault_count` starts faulting. /// Needed now that recovery's in-band epoch seal (INV-2) shares the `_log/` prefix with every other /// write under a namespace: a test that wants to fault something LATER in the same prefix (e.g. the @@ -428,17 +536,18 @@ class RefWriterTestBackend : public CountingBackend /// times. Recovery must not consume this injection; callers that intentionally enumerate still do. int list_fault_count = 0; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + DB::Cas::Backend::RawListPage list(const String & prefix, const String & cursor, size_t limit, + DB::Cas::TransportAccess & access) override { if (list_fault_count > 0) { --list_fault_count; throw DB::Exception(DB::ErrorCodes::S3_ERROR, "RefWriterTestBackend: simulated transient LIST failure"); } - return CountingBackend::list(prefix, cursor, limit); + return CountingBackend::list(prefix, cursor, limit, access); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { recordRequestJournalEvent("GET " + key); if (key == ckpt_get_hook_key && ckpt_get_hook) @@ -472,18 +581,52 @@ class RefWriterTestBackend : public CountingBackend } return std::nullopt; } - return CountingBackend::get(key, range); + return CountingBackend::read(key, access); + } + + /// Every write fault hangs off the ONE keyed primitive. Which of them applies is decided by whether + /// the write carries a precondition, which is exactly what used to separate a conditional replace + /// from a create. + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + topUpLatchedFaults(key, expected_value); + if (expected_value) + return conditionalReplaceForTest(key, bytes, *expected_value, access); + return createForTest(key, bytes, access); } - CasResult casPut( - const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override + /// Both latches are re-armed HERE rather than inside each seam, so a latched fault reads exactly + /// like the counted one it replaces. + void topUpLatchedFaults(const String & key, const std::optional & expected_value) + { + if (ckpt_conflict_latched && key == ckpt_conflict_key) + ckpt_conflict_count = 1; + if (fault_latched && !expected_value && fault_skip == 0 && !fault_key_substr.empty() + && key.find(fault_key_substr) != String::npos) + fault_count = 1; + } + + void disarmFaults() + { + fault_latched = false; + ckpt_conflict_latched = false; + fault_count = 0; + fault_skip = 0; + ckpt_conflict_count = 0; + corrupt_count = 0; + } + + std::expected conditionalReplaceForTest( + const String & key, const String & bytes, const String & expected_value, + DB::Cas::TransportAccess & access) { recordRequestJournalEvent("CAS " + key); if (key == ckpt_conflict_key && ckpt_conflict_count > 0) { --ckpt_conflict_count; - return {CasOutcome::Conflict, {}}; + return std::unexpected(DB::Cas::Backend::RawConflict{}); } if (key == catalog_fault_key && catalog_cas_fault != CatalogCasFault::None) { @@ -491,31 +634,32 @@ class RefWriterTestBackend : public CountingBackend catalog_cas_fault_fired = true; if (fault == CatalogCasFault::CommitThenThrow) { - const CasResult result = CountingBackend::casPut(key, bytes, expected, meta); - if (result.outcome != CasOutcome::Committed) + auto result = CountingBackend::write(key, bytes, expected_value, access); + if (!result.has_value()) return result; throw Poco::TimeoutException( "RefWriterTestBackend: catalog CAS committed but its response was lost"); } - CasResult replacement = CountingBackend::casPut( - key, catalog_replacement_bytes, expected, meta); - if (replacement.outcome != CasOutcome::Committed) + auto replacement = CountingBackend::write(key, catalog_replacement_bytes, expected_value, access); + if (!replacement.has_value()) return replacement; - return {CasOutcome::Conflict, {}}; + return std::unexpected(DB::Cas::Backend::RawConflict{}); } - return CountingBackend::casPut(key, bytes, expected, meta); + return CountingBackend::write(key, bytes, expected_value, access); } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected createForTest( + const String & key, const String & bytes, DB::Cas::TransportAccess & access) { recordRequestJournalEvent("PUT " + key); if (corrupt_count > 0 && !corrupt_key_substr.empty() && key.find(corrupt_key_substr) != String::npos) { --corrupt_count; /// A foreign writer lands a DIFFERENT object at this exact key; then our own response is lost. - CountingBackend::putIfAbsent( - key, corrupt_foreign_bytes.empty() ? bytes + String("\x01_FOREIGN_DIFFERENT") : corrupt_foreign_bytes); + (void)CountingBackend::write( + key, corrupt_foreign_bytes.empty() ? bytes + String("\x01_FOREIGN_DIFFERENT") : corrupt_foreign_bytes, + std::nullopt, access); throw Poco::TimeoutException("RefWriterTestBackend: a foreign different object landed; response lost"); } if (!fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) @@ -561,16 +705,16 @@ class RefWriterTestBackend : public CountingBackend block_entered = true; block_cv.notify_all(); block_cv.wait(lk, [&] { return !block_armed; }); - /// fix-round F3-1a (CRITICAL, unlock-throw race harness): on release, behave like - /// `corrupt_key_substr` above instead of proceeding normally -- a foreign writer landed - /// DIFFERENT bytes at this exact key while we were parked, so our own attempt is a - /// PROVEN conflict once `putIfAbsentControlled`'s resolve-before-reissue GETs it. Lets a - /// test make the recovery seal's PUT throw CORRUPTED_DATA from INSIDE the unlocked - /// window, deterministically, instead of merely returning a non-Committed outcome. + /// On release, behave like `corrupt_key_substr` above instead of proceeding normally -- + /// a foreign writer landed DIFFERENT bytes at this exact key while we were parked, so + /// our own attempt is a PROVEN conflict once the write engine's own settling read + /// observes it. Lets a test make the recovery seal's write throw CORRUPTED_DATA from + /// INSIDE the unlocked window, deterministically, instead of merely returning a + /// non-Committed outcome. if (block_throw_corrupted_on_release) { lk.unlock(); - CountingBackend::putIfAbsent(key, bytes + String("\x01_FOREIGN_DIFFERENT")); + (void)CountingBackend::write(key, bytes + String("\x01_FOREIGN_DIFFERENT"), std::nullopt, access); { std::lock_guard g(block_mutex); block_call_completed = true; @@ -581,7 +725,7 @@ class RefWriterTestBackend : public CountingBackend } } } - const PutResult r = CountingBackend::putIfAbsent(key, bytes, meta); + auto r = CountingBackend::write(key, bytes, std::nullopt, access); { std::lock_guard g(block_mutex); block_call_completed = true; @@ -599,7 +743,7 @@ class RefWriterTestBackend : public CountingBackend { if (pending_delayed_write) { - CountingBackend::putIfAbsent(pending_delayed_write->first, pending_delayed_write->second); + (void)putIfAbsent(pending_delayed_write->first, pending_delayed_write->second); pending_delayed_write.reset(); } } @@ -708,6 +852,31 @@ class RefWriterTestBackend : public CountingBackend std::set independent_released_keys; }; +/// Wedge the lane the way the engine reaches that state: EVERY attempt of the ref-log create is +/// unresolved, the settling read proves the key still absent, and the call gives up at its own retry +/// window having sent something -- which is the `sent_any` half of the wedge rule. A one-shot fault +/// cannot produce it: the reissue would settle the key and commit. So the fault is held armed for the +/// whole call and fully disarmed afterwards, because what every caller does next is a flush that must +/// reach the store normally. +/// +/// The pacing assertions are what make a fixture whose sleep seam is not wired FAIL rather than sleep +/// the whole window out for real. +void driveToTheWedge(VirtualRetryClock & clock, RefWriterTestBackend & backend, + const std::function & drive) +{ + const size_t pauses_before = clock.pauseCount(); + const uint64_t clock_before = clock.nowMs(); + backend.fault_latched = true; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, drive); + backend.disarmFaults(); + EXPECT_GT(clock.pauseCount(), pauses_before + 1) + << "the reissues must pace through the injected sleep, never a real one"; + EXPECT_LE(clock.longestPause(), 5000u) << "each pause is the engine's own capped full jitter"; + EXPECT_GE(clock.nowMs() - clock_before, 60000u) + << "the give-up must be the call's own retry window, not a pre-attempt refusal"; +} + + } /// =================================================================================== @@ -739,7 +908,7 @@ TEST(CASRefWriterNonMinting, ListRefsOnAbsentNamespaceDoesNotMutateCatalog) EXPECT_EQ(backend->putCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 0u); + EXPECT_EQ(backend->writeCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->deleteCount(layout.refCatalogKey()), 0u); const auto catalog_after = backend->get(layout.refCatalogKey()); ASSERT_TRUE(catalog_after); @@ -754,7 +923,7 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsCatalogLifeReplacedWithoutLocal const Layout & layout = store->layout(); const RootNamespace ns{"srv1/cold-read-catalog-aba"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, layout, ns, store->liveWriterEpoch()); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); ASSERT_EQ(store->refTableRuntimeIdentityForTest(ns), 0u); std::mutex mutex; @@ -787,7 +956,7 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsCatalogLifeReplacedWithoutLocal } const CatalogEntry successor - = replaceCatalogLifeForRuntimeRace(*backend, layout, predecessor, UInt128{0xabc002}); + = replaceCatalogLifeForRuntimeRace(backend, layout, predecessor, UInt128{0xabc002}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ @@ -818,14 +987,14 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsReplacementByExternalPoolActor) const Layout & layout = store->layout(); const RootNamespace ns{"srv1/external-catalog-runtime-publication"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, layout, ns, store->liveWriterEpoch()); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); ASSERT_EQ(store->refTableRuntimeIdentityForTest(ns), 0u); CatalogEntry successor; store->setReadableCatalogAfterObservationHookForTest([&] { successor = replaceCatalogLifeForRuntimeRace( - external_store->backend(), external_store->layout(), predecessor, UInt128{0xabc003}); + external_store->poolBackendPtr(), external_store->layout(), predecessor, UInt128{0xabc003}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); if (external_store->backend().putIfAbsent( @@ -885,7 +1054,7 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsAliasingIncarnationAdmittedBetw const RootNamespace ns{"srv1/aliasing-incarnation-target"}; const RootNamespace alias{"srv1/aliasing-incarnation-thief"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, layout, ns, store->liveWriterEpoch()); - const CatalogEntry target_row = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry target_row = catalogEntryOrThrow(backend, layout, ns); ASSERT_EQ(store->refTableRuntimeIdentityForTest(ns), 0u); store->setReadableCatalogAfterObservationHookForTest([&] @@ -896,7 +1065,7 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsAliasingIncarnationAdmittedBetw thief.incarnation = target_row.incarnation; thief.creator = CreatorFence{ .server_root_id = "srv1", .writer_epoch = store->liveWriterEpoch(), .fence_generation = 1}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, thief); + casAdmitEntryForTest(backend, layout, 1, thief); }); EXPECT_THROW((void)store->listRefs(ns), DB::Exception); @@ -937,7 +1106,7 @@ TEST(CASRefWriterNonMinting, ResolveRefOnAbsentNamespaceDoesNotMutateCatalog) EXPECT_EQ(backend->putCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 0u); + EXPECT_EQ(backend->writeCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->deleteCount(layout.refCatalogKey()), 0u); const auto catalog_after = backend->get(layout.refCatalogKey()); ASSERT_TRUE(catalog_after); @@ -959,7 +1128,7 @@ TEST(CASRefWriterNonMinting, DropNamespaceOnAbsentNamespaceDoesNotMutateCatalog) EXPECT_EQ(backend->putCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); - EXPECT_EQ(backend->casPutCount(layout.refCatalogKey()), 0u); + EXPECT_EQ(backend->writeCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->deleteCount(layout.refCatalogKey()), 0u); const auto catalog_after = backend->get(layout.refCatalogKey()); ASSERT_TRUE(catalog_after); @@ -976,7 +1145,7 @@ TEST(CASRefWriterRecovery, BirthOnlyLogNoSnapshotRecoversToEmptyLiveTable) const RootNamespace ns{"srv1/birth_only"}; DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ns.string(), RefTxnId{1, 1}, {namespaceBirthOp()}, std::nullopt}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, @@ -1000,7 +1169,7 @@ TEST(CASRefWriterRecovery, BirthPlusPrecommitPromoteAcrossTwoLogsNoSnapshot) {namespaceBirthOp(), publishCommittedOps("part_1", m1)[0]}, std::nullopt}); DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ns.string(), RefTxnId{1, 2}, {publishCommittedOps("part_1", m1)[1]}, std::nullopt}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, @@ -1039,7 +1208,7 @@ TEST(CASRefWriterRecovery, TerminalGapBelowCheckpointFrontierIsCorruptionNotSame .last_epoch_seal = RefTxnId{1, 2}, }); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); const String next_log_key = layout.refLogKey(life, RefTxnId{2, 2}); auto store = openPool(backend); const uint64_t installs_before = store->recoveryInstallCountForTest(); @@ -1082,7 +1251,7 @@ TEST(CASRefWriterRecovery, SnapshotPlusTailRecovery) tail_ops.push_back(publishCommittedOps("b", mb)[0]); tail_ops.push_back(publishCommittedOps("b", mb)[1]); DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ns.string(), RefTxnId{1, 6}, tail_ops, std::nullopt}); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 6}, @@ -1118,7 +1287,7 @@ TEST(CASRefWriterRecovery, RestartOnVanishConvergesOnNewerSnapshot) /// UNADMITTED namespace mints a fresh RANDOM incarnation rather than adopting the sentinel the raw /// fixture writes at. DB::Cas::tests::fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); const RefTxnId snap_x{1, 10}; DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = ns.string(), @@ -1181,7 +1350,7 @@ TEST(CASRefWriterRecovery, DifferentBytesAtSelectedSnapshotIsCorruptionNotRestar /// further down is a real production read that would otherwise mint a fresh RANDOM incarnation /// for this unadmitted namespace instead of adopting the sentinel the raw fixture writes at. DB::Cas::tests::fixture::admitLive(*backend, layout, ns); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, layout, ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); const RootNamespace other_ns{"srv1/other"}; DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ .ns = ns.string(), @@ -1224,10 +1393,10 @@ TEST(CASRefWriterAppendLane, CommittedChunkPublishesFrontierBeforeInstallAndAck) ASSERT_TRUE(store->resolveRef(ns, "part_1").has_value()); ASSERT_TRUE(store->resolveRef(ns, "part_2").has_value()); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns).value(); + const NamespaceLifeId life = lifeIfCatalogedForTest(backend, store->layout(), ns).value(); const String log_prefix = store->layout().namespaceStreamPrefix(life) + "_log/"; const String ckpt_key = store->layout().refCkptKey(life); - const auto ckpt_before = readCkpt(*backend, store->layout(), life); + const auto ckpt_before = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(ckpt_before); ASSERT_TRUE(ckpt_before->ckpt.committed_through); const RefTxnId expected_frontier{ @@ -1237,7 +1406,7 @@ TEST(CASRefWriterAppendLane, CommittedChunkPublishesFrontierBeforeInstallAndAck) const uint64_t list_before = backend->listTotal(); const uint64_t put_before = backend->putTotal(); const uint64_t ckpt_get_before = backend->getCount(ckpt_key); - const uint64_t ckpt_cas_before = backend->casPutCount(ckpt_key); + const uint64_t ckpt_cas_before = backend->writeCount(ckpt_key); std::mutex mutex; std::condition_variable cv; @@ -1346,7 +1515,7 @@ TEST(CASRefWriterAppendLane, CommittedChunkPublishesFrontierBeforeInstallAndAck) EXPECT_EQ(backend->putTotal(), put_before + 1) << "exactly one body PUT with create-if-absent"; EXPECT_EQ(backend->getCount(ckpt_key), ckpt_get_before + 1) << "one committed chunk pays exactly one checkpoint GET"; - EXPECT_EQ(backend->casPutCount(ckpt_key), ckpt_cas_before + 1) + EXPECT_EQ(backend->writeCount(ckpt_key), ckpt_cas_before + 1) << "one committed chunk pays exactly one checkpoint CAS"; const std::vector journal = backend->requestJournal(); @@ -1357,7 +1526,7 @@ TEST(CASRefWriterAppendLane, CommittedChunkPublishesFrontierBeforeInstallAndAck) EXPECT_EQ(journal[3], "INSTALL"); EXPECT_EQ(journal[4], "FOLLOWER ACK"); - const auto durable_ckpt = readCkpt(*backend, store->layout(), life); + const auto durable_ckpt = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(durable_ckpt); EXPECT_EQ(durable_ckpt->ckpt.committed_through, expected_frontier); } @@ -1368,23 +1537,29 @@ TEST(CASRefWriterAppendLane, CheckpointConflictAfterLogCommitRequiresRecoveryWit auto store = openPool(backend); const RootNamespace ns{"srv1/frontier-conflict"}; publishEmptyPart(store, ns, "x"); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns).value(); + const NamespaceLifeId life = lifeIfCatalogedForTest(backend, store->layout(), ns).value(); const String ckpt_key = store->layout().refCkptKey(life); - const auto before = readCkpt(*backend, store->layout(), life); + const auto before = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(before); ASSERT_TRUE(before->ckpt.committed_through); const RefTxnId candidate{before->ckpt.committed_through->writer_epoch, before->ckpt.committed_through->ref_sequence + 1}; const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); + /// Latched, not counted: the frontier publication re-reads and reissues on every refusal until ITS + /// window closes, so a bounded refusal would be outlived and the checkpoint would advance. + auto clock = VirtualRetryClock::installOn(store); backend->ckpt_conflict_key = ckpt_key; - backend->ckpt_conflict_count = 100; + backend->ckpt_conflict_latched = true; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + backend->disarmFaults(); + EXPECT_GT(clock->pauseCount(), 1u) + << "the publication's reissues must pace through the injected sleep, never a real one"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_before) << "the durable log was not installed or acknowledged"; - const auto after = readCkpt(*backend, store->layout(), life); + const auto after = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(after); EXPECT_EQ(after->ckpt.committed_through, before->ckpt.committed_through); EXPECT_TRUE(backend->get(store->layout().refLogKey(life, candidate))) @@ -1400,9 +1575,9 @@ TEST(CASRefWriterAppendLane, FenceMovementAtCheckpointPublicationRequiresRecover auto store = openPool(backend); const RootNamespace ns{"srv1/frontier-fenced"}; publishEmptyPart(store, ns, "x"); - const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*backend, store->layout(), ns).value(); + const NamespaceLifeId life = lifeIfCatalogedForTest(backend, store->layout(), ns).value(); const String ckpt_key = store->layout().refCkptKey(life); - const auto before = readCkpt(*backend, store->layout(), life); + const auto before = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(before); ASSERT_TRUE(before->ckpt.committed_through); const RefTxnId candidate{before->ckpt.committed_through->writer_epoch, @@ -1416,7 +1591,7 @@ TEST(CASRefWriterAppendLane, FenceMovementAtCheckpointPublicationRequiresRecover EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_before) << "the fenced frontier attempt must not install or acknowledge the durable log"; - const auto after = readCkpt(*backend, store->layout(), life); + const auto after = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(after); EXPECT_EQ(after->ckpt.committed_through, before->ckpt.committed_through); EXPECT_TRUE(backend->get(store->layout().refLogKey(life, candidate))); @@ -1539,14 +1714,11 @@ TEST(CASRefWriterAppendLane, InvalidBatchEntryGetsOwnExceptionBatchSurvives) TEST(CASRefWriterAppendLane, WedgedLaneBlocksSameTableWhileOtherTableProceeds) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns_a{"srv1/wedge_a"}; const RootNamespace ns_b{"srv1/wedge_b"}; @@ -1555,9 +1727,7 @@ TEST(CASRefWriterAppendLane, WedgedLaneBlocksSameTableWhileOtherTableProceeds) publishEmptyPart(store, ns_b, "y"); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns_a)) + "_log/"; - backend->fault_count = 1; - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns_a, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns_a, "x"); }); EXPECT_TRUE(store->refLaneWedgedForTest(ns_a)); /// A different table proceeds normally while ns_a stays wedged. @@ -1577,23 +1747,18 @@ TEST(CASRefWriterAppendLane, WedgedLaneBlocksSameTableWhileOtherTableProceeds) TEST(CASRefWriterAppendLane, WedgedAppendObservedDurableAppliesBeforeNextId) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/wedge_unwedge"}; publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_TRUE(store->resolveRef(ns, "x").has_value()) << "not yet applied while wedged"; @@ -1617,14 +1782,11 @@ TEST(CASRefWriterAppendLane, WedgedAppendObservedDurableAppliesBeforeNextId) /// tail counter is a stable running count. TEST(CASRefWriterAppendLane, WedgeResolutionJoinsTailCountersAndFoldsOverlay) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/wedge_tail"}; publishEmptyPart(store, ns, "x"); @@ -1632,10 +1794,10 @@ TEST(CASRefWriterAppendLane, WedgeResolutionJoinsTailCountersAndFoldsOverlay) const size_t tail_after_setup = store->tailSinceSnapshotCountForTest(ns); - /// Wedge the lane: the single-attempt budget turns the ambiguous log PUT into an Unresolved outcome. + /// Wedge the lane: every attempt of the log create is unresolved, so the call gives up at its own + /// retry window having sent something. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_TRUE(store->resolveRef(ns, "x").has_value()) << "not applied while merely wedged"; EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_after_setup) @@ -1661,14 +1823,11 @@ TEST(CASRefWriterAppendLane, WedgeResolutionJoinsTailCountersAndFoldsOverlay) /// it, and it must track the wedge's full lifecycle (0 -> 1 -> 0), not just a one-shot snapshot. TEST(CASRefWriterAppendLane, WedgedRefLaneCountTracksExactlyTheWedgedTableThroughItsLifecycle) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns_a{"srv1/wedge_count_a"}; const RootNamespace ns_b{"srv1/wedge_count_b"}; @@ -1678,9 +1837,7 @@ TEST(CASRefWriterAppendLane, WedgedRefLaneCountTracksExactlyTheWedgedTableThroug ASSERT_EQ(store->wedgedRefLaneCount(), 0u) << "both tables cached and healthy before the fault"; backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns_a)) + "_log/"; - backend->fault_count = 1; - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns_a, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns_a, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns_a)); EXPECT_EQ(store->wedgedRefLaneCount(), 1u); @@ -1708,7 +1865,7 @@ TEST(CASRefWriterAppendLane, WedgedRefLaneCountTracksExactlyTheWedgedTableThroug /// bookkeeping is restored, proven by a bounded wait on both a same-table and an independent-table /// append. /// -/// The reaction is now the mount's, not the table's [review I5]: a foreign object at a key that +/// The reaction is now the mount's, not the table's: a foreign object at a key that /// mount-lease exclusivity says is exclusively ours contradicts the exclusivity itself, so the append /// site routes through `reportImpossibleInterference` exactly as the wedge-resolve site does -- fence /// closed, remount scheduled. The fence is released only after remount, so there are two separate scopes to keep straight, and this test @@ -1783,23 +1940,19 @@ TEST(CASRefWriterAppendLane, I1AppendCorruptionSurfacesAndFencesTheMountForRemou /// occupant for what it is -- corruption -- and one test covers every build. TEST(CASRefWriterAppendLane, I1WedgeResolveCorruptionSurfacesAndFaultsLane) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/i1_wedge"}; publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - /// Wedge the lane with an ambiguous PUT that never landed. + /// Wedge the lane: every attempt of the log create is unresolved and nothing lands. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); /// A foreign writer lands a DIFFERENT object at the exact wedged key; the next append's wedge resolve @@ -1849,15 +2002,12 @@ TEST(CASRefWriterAppendLane, I1WedgeResolveCorruptionSurfacesAndFaultsLane) /// used to stand in for debug/sanitizer builds went with it. TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); SynchronizedEventLog seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/anomaly_wedge"}; publishEmptyPart(store, ns, "x"); @@ -1865,10 +2015,9 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) store->setEventSink([&](const CasEvent & e) { seen.add(e); }); - /// Wedge the lane with an ambiguous PUT that never landed. + /// Wedge the lane: every attempt of the log create is unresolved and nothing lands. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_TRUE(store->mayMutate()) << "the fence must not be tripped yet -- only an ordinary Unresolved wedge so far"; ASSERT_EQ(store->scheduleRemountCallCountForTest(), 0u) << "no remount must have been scheduled yet by the ordinary wedge alone"; @@ -1929,8 +2078,8 @@ TEST(CASAnomalyPolicy, NonReadyAtNewIdAllocationFaultsAndFailsClosed) String cursor; for (;;) { - const ListPage page = backend->list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); - for (const ListedKey & lk : page.keys) + const KeyPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + for (const KeyEntry & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (parsed && parsed->life_id == DB::Cas::tests::fixture::fixtureLife(ns).incarnation && parsed->kind == RefObjectKind::Log) @@ -1990,48 +2139,58 @@ TEST(CASAnomalyPolicy, DiagnosticDispatchLoggingCannotReplaceFailClosedException expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { store->dropRef(ns, "x"); }); } -/// I3: a conditional write whose attempt classified Committed but whose FINAL post-write fence check -/// failed (the mount fence was lost after the write may have landed) is counted separately, not folded -/// into the generic Unresolved classifier (spec §Late Predecessor PUT best-effort diagnostic). -TEST(CASRequestControllerFenceLoss, I3PostWriteFenceLossIsCounted) +/// A write whose attempt landed but whose post-commit admission check failed is counted separately, +/// not folded into the generic unresolved give-up: the object may exist, and only the caller's own +/// resolution of the key may say so. +TEST(CASRefWriteContract, PostWriteFenceLossIsCounted) { using ProfileEvents::global_counters; auto backend = std::make_shared(); - CasRequestBudget budget; - budget.max_attempts = 3; - CasRequestController ctrl(backend, budget, [] { return static_cast(0); }); // fixed clock - - /// `fence_ok` holds for the pre-attempt check, then is lost by the post-write check. - int calls = 0; - auto fence_ok = [&calls] { return ++calls <= 1; }; - - const auto before = global_counters[ProfileEvents::CASConditionalWriteFenceLostPostWrite].load(); - const CasWriteOutcome outcome = ctrl.putIfAbsentControlled("k", "v", fence_ok); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved) << "a post-write fence loss must never be reported as Committed"; - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteFenceLostPostWrite].load(), before + 1); + bool live = true; + backend->onWriteCommitted("k", [&live] { live = false; }); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit([&live] { return live; }); + + const auto before = global_counters[ProfileEvents::CASRequestFenceLostPostWrite].load(); + const WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr) << "a post-write fence loss must never be reported as committed"; + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_EQ(global_counters[ProfileEvents::CASRequestFenceLostPostWrite].load(), before + 1); } -/// Task B (stageManifest rides the controller): a Committed return surfaces the committed -/// incarnation's token — from the attempt's own PutResult, and equally from a resolve that proves an -/// earlier ambiguous attempt landed — so audit emitters (`PartWriteTxn::stageManifest`'s `ManifestPut` -/// event) keep their token without a follow-up HEAD. -TEST(CASRequestController, CommittedSurfacesTokenFromPutAndFromResolve) +/// A commit surfaces the incarnation it created -- from the attempt's own response, and equally from a +/// settling read that proves an earlier ambiguous attempt of the SAME call landed -- so an audit +/// emitter keeps the incarnation without a follow-up head. +TEST(CASRefWriteContract, CommittedSurfacesTheIncarnationFromTheWriteAndFromTheSettlingRead) { auto backend = std::make_shared(); - CasRequestController ctrl(backend, CasRequestBudget{}, [] { return static_cast(0); }); - const auto fence_ok = [] { return true; }; - - Token direct_token; - ASSERT_EQ(ctrl.putIfAbsentControlled("k1", "v1", fence_ok, &direct_token), CasWriteOutcome::Committed); - EXPECT_EQ(direct_token, backend->head("k1").token) << "the direct-commit token is the PutResult's"; - - /// k2 already holds the IDENTICAL bytes (an earlier ambiguous attempt that landed): the attempt's - /// PreconditionFailed collapses to Unresolved and the resolve GET proves Committed — the token must - /// be the observed incarnation's, and no second incarnation is ever created. - const Token pre_existing = backend->putIfAbsent("k2", "v2").token; - Token resolved_token; - ASSERT_EQ(ctrl.putIfAbsentControlled("k2", "v2", fence_ok, &resolved_token), CasWriteOutcome::Committed); - EXPECT_EQ(resolved_token, pre_existing) << "the resolve-commit token is the observed incarnation's"; + CasRequests requests(backend, Fence::open()); + + CasOperation direct = requests.admit(); + const WriteResult first = direct.create("k1", "v1", Retry::standard()); + const auto * direct_committed = std::get_if(&first); + ASSERT_TRUE(direct_committed != nullptr); + EXPECT_FALSE(direct_committed->resolved_by_read); + CasOperation reader = requests.admit(); + const auto observed = reader.head("k1", Retry::standard()); + ASSERT_TRUE(observed.has_value()); + EXPECT_EQ(direct_committed->incarnation, observed->incarnation) + << "the direct-commit incarnation is the write's own response"; + + /// The write of `k2` lands and its response is lost: the settling read proves the commit, and the + /// incarnation reported is the one that is actually there. + backend->injectAmbiguousLandedWrite("k2"); + CasOperation resolving = requests.admit(); + const WriteResult second = resolving.create("k2", "v2", Retry::standard()); + const auto * resolved_committed = std::get_if(&second); + ASSERT_TRUE(resolved_committed != nullptr); + EXPECT_TRUE(resolved_committed->resolved_by_read); + CasOperation second_reader = requests.admit(); + const auto second_observed = second_reader.head("k2", Retry::standard()); + ASSERT_TRUE(second_observed.has_value()); + EXPECT_EQ(resolved_committed->incarnation, second_observed->incarnation) + << "the resolve-commit incarnation is the observed one"; } /// =================================================================================== @@ -2086,24 +2245,21 @@ TEST(CASRefTableCacheEviction, ZeroBudgetDisablesEviction) /// re-recovery must not be allowed to drop and re-materialize it (which could re-allocate an id). TEST(CASRefTableCacheEviction, WedgedTableIsNeverEvicted) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPoolWithConfig(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget, .ref_table_cache_bytes = 1}); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns_w{"srv1/wedged"}; publishEmptyPart(store, ns_w, "x"); - /// Wedge ns_w's append lane with one ambiguous (Unresolved) PUT that exhausts the single-attempt budget. + /// Wedge ns_w's append lane: every attempt of its log create is unresolved, so the call gives up at + /// its own retry window having sent something. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns_w)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns_w, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns_w, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns_w)); /// Pressure the cache with other tables. ns_w is idle and over the 1-byte budget, but its wedged lane @@ -2169,7 +2325,7 @@ TEST(CASRefWriterSnapshotPublish, CapturedPredecessorCannotPublishAfterSameNameR const RootNamespace ns{"srv1/publisher-predecessor-rebirth"}; publishWithProductionBirth(store, ns, "predecessor"); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(ns, predecessor.incarnation); const auto predecessor_snapshot @@ -2210,13 +2366,13 @@ TEST(CASRefWriterSnapshotPublish, CapturedPredecessorCannotPublishAfterSameNameR EXPECT_NO_THROW(store->dropNamespace(ns)); EXPECT_FALSE(runRegularRoundReclaiming(gc).deferred); EXPECT_TRUE(runRegularRoundReclaiming(gc).deferred); - const RefCatalog after_removal = CasRefCatalog::read(*backend, layout).catalog; + const RefCatalog after_removal = readCatalogForTest(backend, layout).catalog; EXPECT_TRUE(std::none_of(after_removal.entries.begin(), after_removal.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; })); publishWithProductionBirth(store, ns, "successor"); const uint64_t successor_runtime = store->refTableRuntimeIdentityForTest(ns); - const CatalogEntry successor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, predecessor.incarnation); const auto predecessor_ckpt_before_resume = backend->get(layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_before_resume) @@ -2259,7 +2415,7 @@ TEST(CASRefWriterSnapshotPublish, RetiredPredecessorCannotAdvanceCkptAfterSnapsh const RootNamespace ns{"srv1/publisher-predecessor-ckpt-race"}; publishWithProductionBirth(store, ns, "predecessor"); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(ns, predecessor.incarnation); const auto candidate_id = listGreatestLogIdForLifeForTest(*backend, layout, predecessor_life); @@ -2301,13 +2457,13 @@ TEST(CASRefWriterSnapshotPublish, RetiredPredecessorCannotAdvanceCkptAfterSnapsh EXPECT_NO_THROW(store->dropNamespace(ns)); EXPECT_FALSE(runRegularRoundReclaiming(gc).deferred); EXPECT_TRUE(runRegularRoundReclaiming(gc).deferred); - const RefCatalog after_removal = CasRefCatalog::read(*backend, layout).catalog; + const RefCatalog after_removal = readCatalogForTest(backend, layout).catalog; EXPECT_TRUE(std::none_of(after_removal.entries.begin(), after_removal.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; })); publishWithProductionBirth(store, ns, "successor"); const uint64_t successor_runtime = store->refTableRuntimeIdentityForTest(ns); - const CatalogEntry successor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, predecessor.incarnation); const auto predecessor_ckpt_before_resume = backend->get(layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_before_resume); @@ -2346,7 +2502,7 @@ TEST(CASRefWriterRuntimeIdentity, CapturedReaderCannotRetargetSameNameSuccessor) publishWithProductionBirth(store, ns, "shared"); ASSERT_TRUE(store->resolveRef(ns, "shared")); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); Gc gc(store, UInt128{107}); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); @@ -2383,7 +2539,7 @@ TEST(CASRefWriterRuntimeIdentity, CapturedReaderCannotRetargetSameNameSuccessor) EXPECT_FALSE(runRegularRoundReclaiming(gc).deferred); EXPECT_TRUE(runRegularRoundReclaiming(gc).deferred); publishWithProductionBirth(store, ns, "shared"); - const CatalogEntry successor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, predecessor.incarnation); const uint64_t successor_runtime = store->refTableRuntimeIdentityForTest(ns); const auto successor_ref = store->resolveRef(ns, "shared"); @@ -2420,7 +2576,7 @@ TEST(CASRefWriterRuntimeIdentity, CapturedAppendCannotEnqueueIntoSameNameSuccess const RootNamespace ns{"srv1/captured-append-rebirth"}; publishWithProductionBirth(store, ns, "shared"); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); Gc gc(store, UInt128{108}); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); @@ -2457,7 +2613,7 @@ TEST(CASRefWriterRuntimeIdentity, CapturedAppendCannotEnqueueIntoSameNameSuccess EXPECT_FALSE(runRegularRoundReclaiming(gc).deferred); EXPECT_TRUE(runRegularRoundReclaiming(gc).deferred); publishWithProductionBirth(store, ns, "shared"); - const CatalogEntry successor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, predecessor.incarnation); const uint64_t successor_runtime = store->refTableRuntimeIdentityForTest(ns); const auto successor_ref = store->resolveRef(ns, "shared"); @@ -2491,7 +2647,7 @@ TEST(CASRefWriterRuntimeIdentity, LatePredecessorInvalidationLeavesSuccessorAtta const RootNamespace ns{"srv1/late-predecessor-invalidation"}; publishWithProductionBirth(store, ns, "predecessor"); - const CatalogEntry predecessor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry predecessor = catalogEntryOrThrow(backend, layout, ns); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(ns, predecessor.incarnation); Gc gc(store, UInt128{109}); @@ -2501,7 +2657,7 @@ TEST(CASRefWriterRuntimeIdentity, LatePredecessorInvalidationLeavesSuccessorAtta ASSERT_TRUE(runRegularRoundReclaiming(gc).deferred); publishWithProductionBirth(store, ns, "successor"); - const CatalogEntry successor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); ASSERT_NE(successor.incarnation, predecessor.incarnation); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(ns, successor.incarnation); @@ -2903,11 +3059,7 @@ TEST(CASRefWriterSnapshotPublish, C4LatchBoundedUnderSustainedNonCommittedPublis auto backend = std::make_shared(); const RootNamespace ns{"srv1/c4_latch"}; - CasRequestBudget budget; /// one attempt per publish so a failure is a single PUT - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); uint64_t fake_now = 1'000'000; PoolConfig config; @@ -2918,10 +3070,15 @@ TEST(CASRefWriterSnapshotPublish, C4LatchBoundedUnderSustainedNonCommittedPublis config.boot_ms_fn = [&fake_now] { return fake_now; }; config.cas_request_budget = budget; auto store = openPoolWithConfig(backend, config); + /// The boot clock above is frozen (it is what keeps the publish backoff armed), so the REQUEST + /// engine needs its own advancing clock or a saturated publish never reaches its retry window and + /// reissues for ever. + VirtualRetryClock::installOn(store); - /// Every `_snap` PUT throws Unresolved (backend saturated), from the very first publish attempt. + /// Every `_snap` create is unresolved (backend saturated) and stays that way for the whole call, so + /// the publish gives up at its own window -- which is one dispatch, which is what this test counts. backend->fault_key_substr = "_snap/"; - backend->fault_count = 100000; + backend->fault_latched = true; publishEmptyPart(store, ns, "a"); /// crosses the threshold -> one dispatch -> fails -> backoff armed store->waitForSnapshotPublishSettleForTest(ns); @@ -2952,7 +3109,7 @@ TEST(CASRefWriterSnapshotPublish, RecoveredSealAboveThresholdDoesNotRedispatchUn predecessor_config.snapshot_log_bytes_threshold = 1ULL << 40; auto predecessor = openPoolWithConfig(backend, predecessor_config); DB::Cas::tests::fixture::admitLive(*backend, predecessor->layout(), ns); - const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(*backend, predecessor->layout(), ns); + const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, predecessor->layout(), ns); ASSERT_EQ(backend->putIfAbsent(predecessor->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = predecessor->liveWriterEpoch(), .committed_through = std::nullopt, @@ -3035,11 +3192,7 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) const Layout layout("p"); const RootNamespace ns{"srv1/c4_backoff"}; - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); uint64_t fake_now = 1'000'000; PoolConfig config; @@ -3050,10 +3203,13 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) config.boot_ms_fn = [&fake_now] { return fake_now; }; config.cas_request_budget = budget; auto store = openPoolWithConfig(backend, config); + /// As above: the frozen boot clock drives the backoff decisions, so the request engine gets its own. + VirtualRetryClock::installOn(store); - /// Fail ONLY the first `_snap` PUT (arms the backoff); later PUTs succeed. + /// Fail the FIRST dispatch's `_snap` create for the whole call, so it gives up at its own retry + /// window and arms the backoff; the fault is cleared before the retry below. backend->fault_key_substr = "_snap/"; - backend->fault_count = 1; + backend->fault_latched = true; publishEmptyPart(store, ns, "a"); /// dispatch -> publish fails -> backoff armed store->waitForSnapshotPublishSettleForTest(ns); @@ -3067,7 +3223,8 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) << "a read within the backoff window must not re-dispatch"; EXPECT_FALSE(listGreatestSnapshotIdForTest(*backend, layout, ns).has_value()); - /// Advance past the backoff: exactly one retry is dispatched and it publishes. + /// Advance past the backoff, with the fault cleared: exactly one retry is dispatched and it publishes. + backend->disarmFaults(); fake_now += 2000; store->resolveRef(ns, "a"); store->waitForSnapshotPublishSettleForTest(ns); @@ -3222,32 +3379,29 @@ TEST(CASRefWriterStalePrecommitSweep, BoundedBatchesAndInterruptionResumeAcrossM .last_epoch_seal = std::nullopt, }); - /// The successor: a tight retry budget so ONE simulated ambiguous response wedges rather than - /// transparently retries away. `8f9e63c7a19` widened `kSingleAttemptDeadlineMs` off a zero-width - /// race (equal attempt/operation deadlines), but it still measures the capture-to-gate window -- - /// encoding the removal chunk (up to `ref_txn_max_ops` ops) -- against the REAL wall clock, so it - /// recurred (3 of 3 sanitizer lanes) once that encode step got slow enough on its own, independent - /// of scheduler contention: msan in particular. `ref_request_controller` reads its clock through - /// the same injectable seam as the mount fence (`CasRefLedger`'s `controller_boot_ms_fn` is the - /// pool's `boot_ms_fn`), so freeze it here instead of racing it -- the fault-injecting PUT below - /// still reaches the backend synchronously; only the deadline arithmetic stops moving. - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + /// The successor: `fault_latched` below makes ONE simulated ambiguous response wedge the lane + /// rather than being resolved by the engine's own reissue. `boot_ms_fn` seeds every plane's own + /// clock at construction and drives the mount lease's deadline math, so freezing it here keeps the + /// fence alive across the whole retry window; the request engine gets its own separately advancing + /// clock below, so its reissues still pace forward. + const CasRequestBudget budget = wedgeTestBudget(); PoolConfig config; config.cas_request_budget = budget; config.boot_ms_fn = [] { return uint64_t{0}; }; auto successor = openPoolWithConfig(backend, config); + /// The boot clock above is frozen, so the request engine needs its own advancing one: an armed + /// fault otherwise reissues for ever instead of ending the call at its retry window. + auto clock = VirtualRetryClock::installOn(successor); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; /// The successor's own recovery runs first and mints one in-band seal for the dead predecessor /// epoch `e1` (its durable ids are `{e1,1}` and `{e1,2}`, so the seal lands at `{e1,3}`) -- that PUT /// shares this same `_log/` prefix, so it would eat the fault before the sweep ever gets a chance. - /// Skip it and land the fault on the sweep's FIRST removal chunk's PUT, as intended. + /// Skip it and land the fault on the sweep's FIRST removal chunk's PUT, as intended. Latched past + /// the skip, because the write engine reissues within a call: a single-shot fault would be answered + /// by the next attempt and the chunk would commit. backend->fault_skip = 1; - backend->fault_count = 1; /// hits exactly the sweep's FIRST removal chunk's PUT + backend->fault_latched = true; /// The sweep is piggybacked on this mount's very first touch; its (uncertain) failure is INSULATED /// from the read (resolveRef/listRefs call `sweepStalePrecommitsForRead`, not @@ -3258,6 +3412,9 @@ TEST(CASRefWriterStalePrecommitSweep, BoundedBatchesAndInterruptionResumeAcrossM EXPECT_EQ(deferred_after, deferred_before + 1) << "the read-only caller must observe (and count) the deferred sweep failure, not throw"; EXPECT_TRUE(successor->refLaneWedgedForTest(ns)); + backend->disarmFaults(); + EXPECT_GT(clock->pauseCount(), 1u) + << "the reissues must pace through the injected sleep, never a real one"; /// The first chunk's request actually landed server-side; the caller just never saw the ack. backend->materializePendingDelayedWrite(); @@ -3293,8 +3450,8 @@ TEST(CASRefWriterStalePrecommitSweep, BoundedBatchesAndInterruptionResumeAcrossM String cursor; for (;;) { - const ListPage page = backend->list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); - for (const ListedKey & lk : page.keys) + const KeyPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + for (const KeyEntry & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (parsed && parsed->life_id == DB::Cas::tests::fixture::fixtureLife(ns).incarnation @@ -3348,11 +3505,7 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) /// The successor: a tight retry budget so ONE simulated ambiguous response wedges rather than /// transparently retries away (mirrors the wedge-semantics tests in this file exactly). - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); PoolConfig config; config.cas_request_budget = budget; config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); @@ -3368,14 +3521,18 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) << "the unclean predecessor must exercise the injected mount-observation wait"; successor->setEventSink([&](const CasEvent & e) { seen.add(e); }); + /// The boot clock this test drives the sweep backoff on is its own; the request engine gets a + /// separate advancing clock, or an armed fault reissues for ever instead of ending its call. + auto clock = VirtualRetryClock::installOn(successor); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; /// The successor's own recovery runs first and mints one in-band seal for the predecessor's now-dead /// epoch (its three precommits are its only durable ids, so the seal takes the very next slot) -- /// that PUT shares this same `_log/` prefix, so it would eat the fault before the sweep gets a turn. - /// Skip it and land the fault on the sweep's FIRST removal chunk's PUT, as intended. + /// Skip it and land the fault on the sweep's FIRST removal chunk's PUT, as intended. Latched past + /// the skip, because the write engine reissues within a call. backend->fault_skip = 1; - backend->fault_count = 1; /// hits exactly the sweep's FIRST removal chunk's PUT + backend->fault_latched = true; /// FIRST trigger (read path): the sweep's removal PUT is uncertain -> the lane wedges; the read /// itself still succeeds and counts the deferral (existing contract) -- but the shot must NOT be @@ -3389,6 +3546,9 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) EXPECT_TRUE(successor->needsStalePrecommitSweepForTest(ns)) << "a failed sweep must re-arm needs_stale_precommit_sweep, not consume the once-per-mount shot"; EXPECT_EQ(global_counters[ProfileEvents::CASRefSweepRearmed].load(), rearmed_before + 1); + backend->disarmFaults(); + EXPECT_GT(clock->pauseCount(), 1u) + << "the reissues must pace through the injected sleep, never a real one"; /// Within the backoff window (the injected clock has not advanced) a read must NOT re-attempt -- /// the bounded-backoff storm latch: no new deferral, flag still armed. @@ -3507,25 +3667,26 @@ std::optional listGreatestLogIdForTest(Backend & backend, const Layout /// exactly what recovery must refuse. The link names the id the recovering pool's own CAS-walk will mint /// for the dead epoch: one past that epoch's greatest durable id, which is what `seal_of_previous_epoch` /// derives by listing rather than hard-coding, so the fixture cannot drift from the walk's arithmetic. -uint64_t seedTwinDrop(Backend & backend, const Layout & layout, const RootNamespace & ns, +uint64_t seedTwinDrop(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns, const String & ref_name, const ManifestRef & old_ref) { uint64_t greatest_in_previous_epoch = 0; uint64_t previous_epoch = 0; - forEachListedKey(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), [&](const ListedKey & lk) + for (const KeyEntry & lk : listForTest( + backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), "", 1000).keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (!parsed || parsed->kind != RefObjectKind::Log) - return; + continue; if (parsed->txn_id.writer_epoch > previous_epoch || (parsed->txn_id.writer_epoch == previous_epoch && parsed->txn_id.ref_sequence > greatest_in_previous_epoch)) { previous_epoch = parsed->txn_id.writer_epoch; greatest_in_previous_epoch = parsed->txn_id.ref_sequence; } - }, 1000); + } - const uint64_t twin_epoch = allocateWriterEpoch(backend, layout, "test", EpochMintPolicy::NormalMount, 0, [] { return RefCatalog{}; }); + const uint64_t twin_epoch = allocateWriterEpochForTest(backend, layout, "test"); RefLogTxn twin; twin.ns = ns.string(); twin.txn_id = RefTxnId{twin_epoch, 1}; @@ -3534,7 +3695,7 @@ uint64_t seedTwinDrop(Backend & backend, const Layout & layout, const RootNamesp drop.kind = RefOpKind::OwnerTransition; drop.old_binding = RefOwnerBinding{RefOwnerKind::Committed, ref_name, old_ref}; twin.ops = {drop}; - DB::Cas::tests::fixture::writeRefLogRaw(backend, layout, twin); + DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, twin); return twin_epoch; } @@ -3604,7 +3765,7 @@ TEST(CASRefWriterRemount, ReRecoversStaleCacheToTwinDrop) /// A same-uuid twin bumped the durable epoch and durably dropped "a"; this Pool's warm cache never /// observed it. - const uint64_t twin_epoch = seedTwinDrop(*backend, layout, ns, "a", a_id.ref); + const uint64_t twin_epoch = seedTwinDrop(backend, layout, ns, "a", a_id.ref); ASSERT_GT(twin_epoch, e1); ASSERT_TRUE(store->resolveRef(ns, "a").has_value()) << "precondition: the warm cache is stale"; @@ -3635,7 +3796,7 @@ TEST(CASRefWriterRemount, PostRemountAppendCarriesLiveEpochSortingAboveTwinLogs) const ManifestId a_id = publishEmptyPart(store, ns, "a"); const uint64_t e1 = store->liveWriterEpoch(); - const uint64_t twin_epoch = seedTwinDrop(*backend, layout, ns, "a", a_id.ref); + const uint64_t twin_epoch = seedTwinDrop(backend, layout, ns, "a", a_id.ref); ASSERT_GT(twin_epoch, e1); fenceOutRefMount(*backend, layout.mountKey("test")); @@ -3659,11 +3820,7 @@ TEST(CASRefWriterRemount, PostRemountAppendCarriesLiveEpochSortingAboveTwinLogs) /// its slot -- see `quiesceRefTablesForRemount`'s doc comment (`CasPool.h`). TEST(CASRefWriterRemount, DiscardsWedgeAndLaneRemainsUsable) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); /// The self-remount below blocks on nothing (see @@ -3680,10 +3837,10 @@ TEST(CASRefWriterRemount, DiscardsWedgeAndLaneRemainsUsable) publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - /// Wedge the lane with an ambiguous PUT that never landed server-side. + /// Wedge the lane: every attempt of the log create is unresolved and nothing lands server-side. + auto clock = VirtualRetryClock::installOn(store); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); + driveToTheWedge(*clock, *backend, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); fenceOutRefMount(*backend, layout.mountKey("test")); @@ -3786,7 +3943,7 @@ TEST(CASRefWriterNamespaceRemoval, CachedPositiveWriterCannotAppendAfterRemoving const RootNamespace ns{"srv1/removing_blocks_cached_writer"}; publishEmptyPart(store, ns, "existing"); - const CasRefCatalog::Snapshot before = CasRefCatalog::read(*backend, layout); + const CasRefCatalog::Snapshot before = readCatalogForTest(backend, layout); const auto observed = std::find_if(before.catalog.entries.begin(), before.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); ASSERT_NE(observed, before.catalog.entries.end()); @@ -3830,7 +3987,9 @@ TEST(CASRefWriterNamespaceRemoval, CachedPositiveWriterCannotAppendAfterRemoving } if (writer_parked) { - CasRefCatalog::casUpdate(*backend, layout, [&](const RefCatalog & current) + CasRequests catalog_requests(backend, Fence::open()); + CasOperation catalog_op = catalog_requests.admit(); + CasRefCatalog::casUpdate(catalog_op, layout, [&](const RefCatalog & current) { RefCatalog next = current; const auto it = std::find(next.entries.begin(), next.entries.end(), exact_live); @@ -3881,8 +4040,8 @@ TEST(CASRefWriterNamespaceRemoval, TxnNamesEveryOwnerThenRemoveNamespace) String cursor; for (;;) { - const ListPage page = backend->list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); - for (const ListedKey & lk : page.keys) + const KeyPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + for (const KeyEntry & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (parsed && parsed->life_id == DB::Cas::tests::fixture::fixtureLife(ns).incarnation && parsed->kind == RefObjectKind::Log @@ -3932,7 +4091,7 @@ TEST(CASRefWriterNamespaceRemoval, RemovalPublishesTerminalLogWithoutTerminalSna << "the terminal transaction remains ordinary immutable stream work until GC folds it"; size_t terminal_logs = 0; - for (const ListedKey & listed : backend->list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), "", 1000).keys) + for (const KeyEntry & listed : listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), "", 1000).keys) { const auto parsed = layout.parseRefObjectKey(listed.key); if (!parsed || parsed->kind != RefObjectKind::Log) @@ -3994,7 +4153,7 @@ TEST(CASRefWriterNamespaceRemoval, GenericAppendCannotWriteTerminalWhileCatalogI const Layout & layout = store->layout(); const RootNamespace ns{"srv1/unauthorized_terminal"}; publishEmptyPart(store, ns, "owned"); - const CatalogEntry live = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry live = catalogEntryOrThrow(backend, layout, ns); ASSERT_EQ(live.state, NsState::Live); const auto greatest_before = listGreatestLogIdForLifeForTest( *backend, layout, NamespaceLifeId::fromCatalogEntry(ns, live.incarnation)); @@ -4023,7 +4182,7 @@ TEST(CASRefWriterNamespaceRemoval, GenericAppendCannotWriteTerminalWhileCatalogI /*skip_stale_precommit_sweep=*/true); }); - EXPECT_EQ(catalogEntryOrThrow(*backend, layout, ns), live); + EXPECT_EQ(catalogEntryOrThrow(backend, layout, ns), live); EXPECT_EQ(listGreatestLogIdForLifeForTest( *backend, layout, NamespaceLifeId::fromCatalogEntry(ns, live.incarnation)), greatest_before) << "an unauthorized terminal must allocate no id and create no ref-log object"; @@ -4038,7 +4197,7 @@ TEST(CASRefWriterNamespaceRemoval, GenericTerminalOnAbsentNamePerformsZeroDurabl auto backend = std::make_shared(); auto store = openPool(backend); const RootNamespace ns{"srv1/absent_unauthorized_terminal"}; - const CasRefCatalog::Snapshot catalog_before = CasRefCatalog::read(*backend, store->layout()); + const CasRefCatalog::Snapshot catalog_before = readCatalogForTest(backend, store->layout()); backend->resetCounts(); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] @@ -4056,10 +4215,10 @@ TEST(CASRefWriterNamespaceRemoval, GenericTerminalOnAbsentNamePerformsZeroDurabl EXPECT_EQ(backend->putTotal(), 0u); EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); EXPECT_EQ(backend->deleteTotal(), 0u); - const CasRefCatalog::Snapshot catalog_after = CasRefCatalog::read(*backend, store->layout()); - EXPECT_EQ(catalog_after.token, catalog_before.token); + const CasRefCatalog::Snapshot catalog_after = readCatalogForTest(backend, store->layout()); + EXPECT_EQ(catalog_after.incarnation, catalog_before.incarnation); EXPECT_EQ(catalog_after.catalog, catalog_before.catalog); EXPECT_FALSE(store->refTableLifeForTest(ns)); } @@ -4077,13 +4236,13 @@ TEST(CASRefWriterNamespaceRemoval, CatalogedNamespaceFilesOnlyLifeCompletesRemov const RootNamespace ns{"srv1/files_only"}; const NamespaceLifeId life = store->namespaceLife(ns); store->putNamespaceFile(life, "format_version.txt", "1\n"); - ASSERT_TRUE(backend->list(layout.namespaceStreamPrefix(life), "", 100).keys.empty()); - ASSERT_EQ(catalogEntryOrThrow(*backend, layout, ns).state, NsState::Live); + ASSERT_TRUE(listForTest(backend, layout.namespaceStreamPrefix(life), "", 100).keys.empty()); + ASSERT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Live); EXPECT_NO_THROW(store->dropNamespace(ns)); - ASSERT_EQ(catalogEntryOrThrow(*backend, layout, ns).state, NsState::Removing); + ASSERT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Removing); - const ListPage terminal_page = backend->list(layout.namespaceStreamPrefix(life), "", 100); + const KeyPage terminal_page = listForTest(backend, layout.namespaceStreamPrefix(life), "", 100); ASSERT_EQ(terminal_page.keys.size(), 1u); const auto parsed = layout.parseRefObjectKey(terminal_page.keys.front().key); ASSERT_TRUE(parsed); @@ -4098,7 +4257,7 @@ TEST(CASRefWriterNamespaceRemoval, CatalogedNamespaceFilesOnlyLifeCompletesRemov Gc gc(store, UInt128{181}); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); (void)runRegularRoundReclaiming(gc); - const RefCatalog after = CasRefCatalog::read(*backend, layout).catalog; + const RefCatalog after = readCatalogForTest(backend, layout).catalog; EXPECT_TRUE(std::none_of(after.entries.begin(), after.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; @@ -4114,19 +4273,19 @@ TEST(CASRefWriterNamespaceRemoval, PredurableCatalogReadFailureReopensExactLiveL const Layout & layout = store->layout(); const RootNamespace ns{"srv1/predurable_read_failure"}; publishEmptyPart(store, ns, "owned"); - const CatalogEntry live = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry live = catalogEntryOrThrow(backend, layout, ns); backend->catalog_fault_key = layout.refCatalogKey(); backend->catalog_gets_before_fault = 1; /// initial discovery succeeds; post-close observation fails backend->catalog_get_fault_count = 1; EXPECT_THROW(store->dropNamespace(ns), std::runtime_error); - EXPECT_EQ(catalogEntryOrThrow(*backend, layout, ns), live); + EXPECT_EQ(catalogEntryOrThrow(backend, layout, ns), live); EXPECT_NO_THROW(store->updateRefPublishedAt(ns, "owned", [](RefPublishedAtUpdate & update) { update.published_at_ms = 17; })) << "a fresh exact Live observation must reopen the lane after a pre-durable failure"; - EXPECT_EQ(catalogEntryOrThrow(*backend, layout, ns).state, NsState::Live); + EXPECT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Live); } /// spec §Namespace Removal (writer, line 666): "After the transaction is durable, it applies the same @@ -4167,14 +4326,11 @@ TEST(CASRefWriterNamespaceRemoval, DropNamespaceCancelsInFlightBuildAndNextOpThr /// remains `Removing`, positive ownership is refused, and a retry of the same removal resolves the wedge. TEST(CASRefWriterNamespaceRemoval, RemovalAppendFailureLeavesRemovingAndRetryCompletes) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/remove_fault_keeps_build"}; @@ -4185,11 +4341,9 @@ TEST(CASRefWriterNamespaceRemoval, RemovalAppendFailureLeavesRemovingAndRetryCom build->precommitAdd(ns, "inflight", id); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); + driveToTheWedge(*clock, *backend, [&] { store->dropNamespace(ns); }); - EXPECT_EQ(catalogEntryOrThrow(*backend, layout, ns).state, NsState::Removing); + EXPECT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Removing); EXPECT_FALSE(store->resolveRef(ns, "committed")) << "a fresh name lookup must not expose a catalog-Removing life"; /// The build was NOT cancelled: a non-append operation (`stageManifest` -- it never touches the now @@ -4210,14 +4364,11 @@ TEST(CASRefWriterNamespaceRemoval, RemovalAppendFailureLeavesRemovingAndRetryCom /// `RemovalAppendFailureLeavesRemovingAndRetryCompletes`. TEST(CASRefWriterNamespaceRemoval, PresenceProbeStaysTrueThroughRemovingUntilTerminalRetrySucceeds) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/presence_removing_no_terminal"}; @@ -4225,10 +4376,9 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeStaysTrueThroughRemovingUntilTer EXPECT_TRUE(store->namespaceStillLogicallyPresent(ns)); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); + driveToTheWedge(*clock, *backend, [&] { store->dropNamespace(ns); }); - ASSERT_EQ(catalogEntryOrThrow(*backend, layout, ns).state, NsState::Removing); + ASSERT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Removing); EXPECT_TRUE(store->namespaceStillLogicallyPresent(ns)) << "the catalog transitioned but the terminal append never landed -- cleanup is unproven"; @@ -4252,7 +4402,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeCreatingIsPresentAndRemovalWaits entry.state = NsState::Creating; entry.incarnation = UInt128(99); entry.creator = CreatorFence{.server_root_id = "srv1", .writer_epoch = store->liveWriterEpoch(), .fence_generation = 1}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, entry); + casAdmitEntryForTest(backend, layout, 1, entry); EXPECT_TRUE(store->namespaceStillLogicallyPresent(ns)); @@ -4260,7 +4410,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeCreatingIsPresentAndRemovalWaits /// dead (absence proves nothing), so removal fails closed rather than cancelling a `Creating` row a /// live writer might still publish into. expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); - EXPECT_EQ(catalogEntryOrThrow(*backend, layout, ns).state, NsState::Creating); + EXPECT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Creating); EXPECT_TRUE(store->namespaceStillLogicallyPresent(ns)); /// Publish a GC-fenced lease for the SAME server root -- one of `isCreatorFenceTerminal`'s accepted @@ -4324,7 +4474,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeNoRowObservationRevalidatesRathe born.state = NsState::Creating; born.incarnation = UInt128(1234); born.creator = CreatorFence{.server_root_id = "srv1", .writer_epoch = store->liveWriterEpoch(), .fence_generation = 1}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, born); + casAdmitEntryForTest(backend, layout, 1, born); { std::lock_guard lock(mutex); @@ -4389,7 +4539,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeIgnoresUnrelatedCatalogChurnBetw born.state = NsState::Creating; born.incarnation = UInt128(5678); born.creator = CreatorFence{.server_root_id = "srv1", .writer_epoch = store->liveWriterEpoch(), .fence_generation = 1}; - CasRefCatalog::casAdmitEntry(*backend, layout, 1, born); + casAdmitEntryForTest(backend, layout, 1, born); { std::lock_guard lock(mutex); @@ -4444,14 +4594,11 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeFenceLossPropagatesRatherThanAns /// (absent, immediately, no GC). TEST(CASRefWriterNamespaceRemoval, PresenceProbeFacadeConsistencyAcrossRemovalLifecycle) { - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; - budget.lease_safety_margin_ms = 100; + const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); auto store = openPool(backend, budget); + auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/presence_facade_consistency"}; @@ -4460,8 +4607,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeFacadeConsistencyAcrossRemovalLi EXPECT_FALSE(store->listRefs(ns).empty()); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(ns); }); + driveToTheWedge(*clock, *backend, [&] { store->dropNamespace(ns); }); EXPECT_TRUE(store->namespaceStillLogicallyPresent(ns)) << "present for cleanup, even though content below is about to prove unreadable"; @@ -4495,7 +4641,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeRevalidatesAfterTerminalProvenRa const auto catalog_entry = [&]() -> std::optional { - const RefCatalog catalog = CasRefCatalog::read(*backend, layout).catalog; + const RefCatalog catalog = readCatalogForTest(backend, layout).catalog; const auto it = std::find_if(catalog.entries.begin(), catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; @@ -4678,7 +4824,7 @@ TEST(CASRefWriterNamespaceRemoval, CreateAgainstRemovingRetriesWithoutMutation) expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->namespaceLife(ns); }); EXPECT_EQ(backend->putTotal(), 0u); EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); } auto fresh_store = openPool(backend); @@ -4686,7 +4832,7 @@ TEST(CASRefWriterNamespaceRemoval, CreateAgainstRemovingRetriesWithoutMutation) expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)fresh_store->namespaceLife(ns); }); EXPECT_EQ(backend->putTotal(), 0u); EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); } /// Same-name rebirth must not inherit the predecessor's physical life or folded cursor even when the @@ -4715,7 +4861,7 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi }; const auto catalog_entry = [&]() -> std::optional { - const RefCatalog catalog = CasRefCatalog::read(*backend, layout).catalog; + const RefCatalog catalog = readCatalogForTest(backend, layout).catalog; const auto it = std::find_if(catalog.entries.begin(), catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; @@ -4743,7 +4889,7 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi ASSERT_GT(backend->putTotal(), puts_before_drop) << "control: the real removal call returned after durably writing its terminal artifacts"; std::optional terminal_id; - for (const ListedKey & listed : backend->list( + for (const KeyEntry & listed : listForTest(backend, layout.namespaceStreamPrefix(NamespaceLifeId::fromCatalogEntry(ns, predecessor.incarnation)), "", 1000).keys) { @@ -4790,7 +4936,7 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi ASSERT_EQ(store->refTableLifeForTest(ns)->incarnation, successor.incarnation); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(ns, successor.incarnation); - const ListPage successor_stream = backend->list(layout.namespaceStreamPrefix(successor_life), "", 1000); + const KeyPage successor_stream = listForTest(backend, layout.namespaceStreamPrefix(successor_life), "", 1000); ASSERT_FALSE(successor_stream.keys.empty()) << "the real successor writer produced foldable stream work"; std::vector successor_phases; gc.setPhaseSink([&](const GcPhaseRecord & phase) { successor_phases.push_back(phase); }); @@ -4829,13 +4975,13 @@ TEST(CASRefWriterNamespaceRemoval, CommitThenThrowEraseResolvesAndRebindsResiden const Layout & layout = store->layout(); const RootNamespace ns{"srv1/removal-erase-lost-response"}; Gc gc(store, UInt128{101}); - const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, *backend, ns, gc); + const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, backend, ns, gc); backend->catalog_fault_key = layout.refCatalogKey(); backend->catalog_cas_fault = RefWriterTestBackend::CatalogCasFault::CommitThenThrow; EXPECT_NO_THROW((void)runRegularRoundReclaiming(gc)); - const RefCatalog after_erase = CasRefCatalog::read(*backend, layout).catalog; + const RefCatalog after_erase = readCatalogForTest(backend, layout).catalog; EXPECT_TRUE(std::none_of(after_erase.entries.begin(), after_erase.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns && entry.incarnation == ready.predecessor.incarnation; @@ -4843,7 +4989,7 @@ TEST(CASRefWriterNamespaceRemoval, CommitThenThrowEraseResolvesAndRebindsResiden EXPECT_EQ(store->refTableRuntimeIdentityForTest(ns), 0u); EXPECT_NO_THROW(publishWithProductionBirth(store, ns, "successor")); - const CatalogEntry successor = catalogEntryOrThrow(*backend, layout, ns); + const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, ready.predecessor.incarnation); EXPECT_EQ(store->liveWriterEpoch(), ready.writer_epoch); EXPECT_NE(store->refTableRuntimeIdentityForTest(ns), ready.runtime_identity); @@ -4870,7 +5016,7 @@ TEST(CASRefWriterNamespaceRemoval, OtherWinnerReplacementInvalidatesExactPredece const Layout & layout = store->layout(); const RootNamespace ns{"srv1/removal-other-winner-replacement"}; Gc gc(store, UInt128{102}); - const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, *backend, ns, gc); + const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, backend, ns, gc); const CatalogEntry replacement{ .ns = ns, @@ -4904,13 +5050,13 @@ TEST(CASRefWriterNamespaceRemoval, LaterNameLookupReconcilesAfterEraseResolution const Layout & layout = store->layout(); const RootNamespace ns{"srv1/removal-resolution-read-failure-lookup"}; Gc gc(store, UInt128{103}); - const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, *backend, ns, gc); + const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, backend, ns, gc); backend->catalog_fault_key = layout.refCatalogKey(); backend->catalog_cas_fault = RefWriterTestBackend::CatalogCasFault::CommitThenThrow; backend->catalog_resolution_get_fault_count = 1; EXPECT_THROW((void)runRegularRoundReclaiming(gc), std::runtime_error); - EXPECT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()); + EXPECT_TRUE(readCatalogForTest(backend, layout).catalog.entries.empty()); std::optional successor; EXPECT_NO_THROW(successor = store->namespaceLife(ns)); @@ -4931,17 +5077,17 @@ TEST(CASRefWriterNamespaceRemoval, PostListCatalogCutReconcilesMissedEraseInvali const Layout & layout = store->layout(); const RootNamespace ns{"srv1/removal-resolution-read-failure-post-list"}; Gc gc(store, UInt128{104}); - const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, *backend, ns, gc); + const CompletedRemovingFixture ready = prepareResidentRemovalForDrain(store, backend, ns, gc); backend->catalog_fault_key = layout.refCatalogKey(); backend->catalog_cas_fault = RefWriterTestBackend::CatalogCasFault::CommitThenThrow; backend->catalog_resolution_get_fault_count = 1; EXPECT_THROW((void)runRegularRoundReclaiming(gc), std::runtime_error); - EXPECT_TRUE(CasRefCatalog::read(*backend, layout).catalog.entries.empty()); + EXPECT_TRUE(readCatalogForTest(backend, layout).catalog.entries.empty()); EXPECT_NO_THROW((void)runRegularRoundReclaiming(gc)); EXPECT_NO_THROW(publishWithProductionBirth(store, ns, "successor")); - EXPECT_NE(catalogEntryOrThrow(*backend, layout, ns).incarnation, ready.predecessor.incarnation); + EXPECT_NE(catalogEntryOrThrow(backend, layout, ns).incarnation, ready.predecessor.incarnation); EXPECT_NE(store->refTableRuntimeIdentityForTest(ns), ready.runtime_identity); } @@ -4956,7 +5102,7 @@ TEST(CASRefWriterNamespaceBirth, ExistingLiveCatalogRowPinsExactLifeWithoutMutat auto backend = std::make_shared(); auto store = openPool(backend); const RootNamespace ns{"srv1/existing-live-assignment"}; - CasRefCatalog::casAdmitEntry(*backend, store->layout(), 1, CatalogEntry{ + casAdmitEntryForTest(backend, store->layout(), 1, CatalogEntry{ .ns = ns, .state = NsState::Live, .incarnation = UInt128{41}}); DB::Cas::tests::writeRecoverableCkptForRawFixture(*backend, store->layout(), ns, RefCkpt{ .life_epoch = store->liveWriterEpoch(), @@ -4972,7 +5118,7 @@ TEST(CASRefWriterNamespaceBirth, ExistingLiveCatalogRowPinsExactLifeWithoutMutat EXPECT_EQ(store->refTableLifeForTest(ns)->incarnation, UInt128{41}); EXPECT_EQ(backend->putTotal(), 0u); EXPECT_EQ(backend->putOverwriteTotal(), 0u); - EXPECT_EQ(backend->casPutTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); } /// A read of a never-born name may observe the catalog, but it must not allocate the local name slot @@ -5065,17 +5211,17 @@ namespace /// under `ns` -- epoch 1 births ref "a", epoch 2 adds ref "b" -- with no snapshot, and burns the /// durable epoch counter to exactly 2 so a subsequent `Pool::open` allocates epoch 3 (both dead /// epochs land strictly below the fresh writer's own, as `dead_region_nonempty` requires). -void seedSealFixtureDeadEpochs(Backend & backend, const Layout & layout, const RootNamespace & ns) +void seedSealFixtureDeadEpochs(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns) { - allocateWriterEpoch(backend, layout, "test", EpochMintPolicy::NormalMount, 0, [] { return RefCatalog{}; }); /// burns epoch 1 - allocateWriterEpoch(backend, layout, "test", EpochMintPolicy::NormalMount, 0, [] { return RefCatalog{}; }); /// burns epoch 2 + allocateWriterEpochForTest(backend, layout, "test"); /// burns epoch 1 + allocateWriterEpochForTest(backend, layout, "test"); /// burns epoch 2 RefLogTxn birth; birth.ns = ns.string(); birth.txn_id = RefTxnId{1, 1}; birth.ops = {namespaceBirthOp(), publishCommittedOps("a", manifestRef(1, 1, 1))[0], publishCommittedOps("a", manifestRef(1, 1, 1))[1]}; - DB::Cas::tests::fixture::writeRefLogRaw(backend, layout, birth); + DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, birth); RefLogTxn mut; mut.ns = ns.string(); @@ -5086,8 +5232,8 @@ void seedSealFixtureDeadEpochs(Backend & backend, const Layout & layout, const R mut.prev_epoch_seal = RefTxnId{1, 2}; mut.ops = {publishCommittedOps("b", manifestRef(2, 1, 1))[0], publishCommittedOps("b", manifestRef(2, 1, 1))[1]}; - DB::Cas::tests::fixture::writeRefLogRaw(backend, layout, mut); - DB::Cas::tests::writeRecoverableCkptForRawFixture(backend, layout, ns, RefCkpt{ + DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, mut); + DB::Cas::tests::writeRecoverableCkptForRawFixture(*backend, layout, ns, RefCkpt{ /// The namespace was born in epoch 1 and only `{1,1}` is fronted initially. Recovery must mint /// the missing required seal `{1,2}` before it may adopt the already durable `{2,1}` successor. .life_epoch = 1, @@ -5102,10 +5248,12 @@ void seedSealFixtureDeadEpochs(Backend & backend, const Layout & layout, const R /// prior is an immediate certificate of death (`claimMountAwaitingExpiry` reclaims it on its FIRST /// attempt, no observation polling), so a fake-clocked successor `Pool::open` above it becomes /// unclean deterministically, without any real sleep. -void seedUncleanPredecessorMount(Backend & backend, const Layout & layout, uint64_t epoch) +void seedUncleanPredecessorMount(const BackendPtr & backend, const Layout & layout, uint64_t epoch) { - claimMount(backend, layout, "test", UInt128(1), epoch, /*now_ms=*/1000, /*ttl_ms=*/500); - fenceOutRefMount(backend, layout.mountKey("test")); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + claimMount(op, layout, "test", UInt128(1), epoch, /*now_ms=*/1000, /*ttl_ms=*/500); + fenceOutRefMount(*backend, layout.mountKey("test")); } /// The budget every seal test's successor `Pool::open` uses: a 500ms lease TTL needs a scaled-down @@ -5114,7 +5262,7 @@ void seedUncleanPredecessorMount(Backend & backend, const Layout & layout, uint6 CasRequestBudget sealTestTinyBudget() { return CasRequestBudget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = kSingleAttemptDeadlineMs, .max_attempts = 1, + .attempt_timeout_ms = 50, .operation_deadline_ms = kSingleAttemptDeadlineMs, .lease_safety_margin_ms = 50}; } @@ -5153,8 +5301,8 @@ TEST(CASRefWriterRecoveryRetry, TransientSealFailureIsRetriedThenSucceeds) const Layout layout("p"); const RootNamespace ns{"srv1/retry_ok"}; - seedSealFixtureDeadEpochs(*backend, layout, ns); - seedUncleanPredecessorMount(*backend, layout, /*epoch=*/2); + seedSealFixtureDeadEpochs(backend, layout, ns); + seedUncleanPredecessorMount(backend, layout, /*epoch=*/2); uint64_t fake_now = 1'000'000; @@ -5171,27 +5319,31 @@ TEST(CASRefWriterRecoveryRetry, TransientSealFailureIsRetriedThenSucceeds) ASSERT_TRUE(store); ASSERT_EQ(store->liveWriterEpoch(), 3u); - /// No-op backoff and a frozen clock: retries run until the transient faults are exhausted, and the - /// frozen clock keeps the mount fence alive across them (advancing it past the tiny lease TTL would - /// drop the fence and abort recovery -- exercising the fence path, which is the budget test's job). - store->setCasRetrySleepForTest([](uint64_t) {}); - - /// Fail the epoch seal's conditional create twice with a transient (timeout) error; the third - /// attempt lands. The seal is a LOG transaction at `{2,2}` -- the slot after the dead epoch's last - /// durable id -- because INV-2 closes an epoch in-band, at the key a straggler would have taken. + /// The engine's own clock advances (its sleep is what moves it), while `fake_now` -- the FENCE's + /// clock -- stays frozen, which is what keeps the mount alive across a retry: advancing it past the + /// tiny lease TTL would drop the fence and abort recovery, exercising the fence path instead. + VirtualRetryClock::installOn(store); + + /// Fail the epoch seal's conditional create for the whole of ONE recovery attempt, then clear the + /// fault from the recovery retry seam -- the only point between two recovery attempts a test can + /// reach. A bounded fault count cannot express this: the write engine reissues within a call, so a + /// count of two is spent by that one call's own reissues and no recovery retry ever happens. The + /// seal is a LOG transaction at `{2,2}` -- the slot after the dead epoch's last durable id -- + /// because INV-2 closes an epoch in-band, at the key a straggler would have taken. const RefTxnId seal_id{2, 2}; backend->fault_key_substr = layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), seal_id); - backend->fault_count = 2; + backend->fault_latched = true; + store->setRefRecoveryRetrySleepForTest([&backend](uint64_t, const auto &) { backend->disarmFaults(); }); using ProfileEvents::global_counters; const auto retries_before = global_counters[ProfileEvents::CASRefRecoveryRetries].load(); const auto sealed_before = global_counters[ProfileEvents::CASRefRecoveryEpochSealed].load(); - EXPECT_EQ(store->listRefs(ns).size(), 2u) << "recovery must succeed after retrying past the faults"; + EXPECT_EQ(store->listRefs(ns).size(), 2u) << "recovery must succeed after retrying past the fault"; - EXPECT_EQ(global_counters[ProfileEvents::CASRefRecoveryRetries].load(), retries_before + 2); + EXPECT_EQ(global_counters[ProfileEvents::CASRefRecoveryRetries].load(), retries_before + 1); /// TWO dead epochs (1 and 2) are closed by this walk, and a whole attempt is re-driven per transient - /// failure -- so the seals of the epochs a failed attempt already closed are ADOPTED on the retry + /// failure -- so the seals of the epochs the failed attempt already closed are ADOPTED on the retry /// rather than minted again. Exactly two are minted in total. EXPECT_EQ(global_counters[ProfileEvents::CASRefRecoveryEpochSealed].load(), sealed_before + 2); } @@ -5204,8 +5356,8 @@ TEST(CASRefWriterRecoveryRetry, RecoveryDoesNotEnumerateItsStream) const Layout layout("p"); const RootNamespace ns{"srv1/retry_list"}; - seedSealFixtureDeadEpochs(*backend, layout, ns); - seedUncleanPredecessorMount(*backend, layout, /*epoch=*/2); + seedSealFixtureDeadEpochs(backend, layout, ns); + seedUncleanPredecessorMount(backend, layout, /*epoch=*/2); uint64_t fake_now = 1'000'000; @@ -5244,8 +5396,8 @@ TEST(CASRefWriterRecoveryRetry, TransientFailureLongerThanBudgetPropagates) const Layout layout("p"); const RootNamespace ns{"srv1/retry_budget"}; - seedSealFixtureDeadEpochs(*backend, layout, ns); - seedUncleanPredecessorMount(*backend, layout, /*epoch=*/2); + seedSealFixtureDeadEpochs(backend, layout, ns); + seedUncleanPredecessorMount(backend, layout, /*epoch=*/2); uint64_t fake_now = 1'000'000; @@ -5279,8 +5431,8 @@ TEST(CASRefWriterRecoveryRetry, NonNetworkErrorIsNotRetried) const Layout layout("p"); const RootNamespace ns{"srv1/retry_fatal"}; - seedSealFixtureDeadEpochs(*backend, layout, ns); - seedUncleanPredecessorMount(*backend, layout, /*epoch=*/2); + seedSealFixtureDeadEpochs(backend, layout, ns); + seedUncleanPredecessorMount(backend, layout, /*epoch=*/2); PoolConfig config; config.server_id = UInt128(1); @@ -5362,8 +5514,8 @@ TEST(CASRefWriterRecoveryRetry, ThrowingBackoffSleepDoesNotWedgeRecovery) const Layout layout("p"); const RootNamespace ns{"srv1/retry_sleep_throw"}; - seedSealFixtureDeadEpochs(*backend, layout, ns); - seedUncleanPredecessorMount(*backend, layout, /*epoch=*/2); + seedSealFixtureDeadEpochs(backend, layout, ns); + seedUncleanPredecessorMount(backend, layout, /*epoch=*/2); PoolConfig config; config.server_id = UInt128(1); @@ -5374,7 +5526,9 @@ TEST(CASRefWriterRecoveryRetry, ThrowingBackoffSleepDoesNotWedgeRecovery) auto store = openPoolWithConfig(backend, config); ASSERT_TRUE(store); - /// First touch: the seal PUT fails transiently -> the loop enters backoff -> the sleep THROWS. + /// First touch: the seal create fails transiently and the next thing either loop does is sleep on + /// this one seam -- the write engine's reissue pause is simply the first to reach it -- so the throw + /// lands while `recovery_in_progress` is set, which is the state this test is about. bool sleep_should_throw = true; store->setCasRetrySleepForTest([&sleep_should_throw](uint64_t) { diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index 2d583f6a144a..9f696c2d2397 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -24,6 +24,7 @@ #include #include #include +#include namespace DB::ErrorCodes { @@ -39,6 +40,7 @@ using namespace DB::Cas; using DB::Cas::tests::CountingBackend; using DB::Cas::tests::FakeClock; +using DB::Cas::tests::expectBytes; using DB::Cas::tests::expectThrowsCode; namespace @@ -134,9 +136,19 @@ TEST(CASWriteResult, OrThrowMapsEveryAlternative) expectThrowsCode(DB::ErrorCodes::ABORTED, [&] { orThrow(WriteResult{Conflict{ProvenAbsent{}}}, "t"); }); expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { orThrow(WriteResult{Refused{DB::ErrorCodes::S3_ERROR, "denied"}}, "t"); }); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{GaveUp{GaveUp::Why::Deadline, GaveUp::Source::Policy, true, NotObserved{}}}, "t"); }); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{GaveUp{GaveUp::Why::Unresolved, GaveUp::Source::Policy, true, ProvenAbsent{}}}, "t"); }); - expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{GaveUp{GaveUp::Why::FenceLost, GaveUp::Source::Lease, false, NotObserved{}}}, "t"); }); + /// Designated rather than positional: `GaveUp` grows fields at its end, and a positional list is + /// the form a field inserted anywhere else would silently re-interpret. + const GaveUp deadline{ + .why = GaveUp::Why::Deadline, .deadline_source = GaveUp::Source::Policy, + .sent_any = true, .last_seen = NotObserved{}}; + const GaveUp unresolved{ + .why = GaveUp::Why::Unresolved, .deadline_source = GaveUp::Source::Policy, + .sent_any = true, .last_seen = ProvenAbsent{}}; + const GaveUp fence_lost{ + .why = GaveUp::Why::FenceLost, .deadline_source = GaveUp::Source::Lease, + .sent_any = false, .last_seen = NotObserved{}}; + for (const GaveUp & gave_up : {deadline, unresolved, fence_lost}) + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { orThrow(WriteResult{gave_up}, "t"); }); } TEST(CASFence, OpenFenceAdmitsEverythingAndNeverMoves) @@ -151,61 +163,60 @@ TEST(CASFence, OpenFenceAdmitsEverythingAndNeverMoves) /// The backend's keyed string primitives /// ================================================================================================ -/// The door the primitives are reachable through until `CasRequests` lands: `Backend` is a friend of -/// `TransportAccess`, so a `Backend` subclass can hand out the migration key. Both go at the lock. -struct RawDoor : DB::Cas::Backend -{ - static DB::Cas::TransportAccess key() { return migrationAccess(); } -}; - -TEST(CASBackendPrimitives, InMemoryWriteReadRemoveRoundTripInStrings) +TEST(CASBackendPrimitives, InMemoryWriteReadRemoveRoundTripThroughOneOperation) { + FakeClock clock; auto b = std::make_shared(); - auto key = RawDoor::key(); - auto w1 = b->write("k", "v1", std::nullopt, key); - ASSERT_TRUE(w1.has_value()); - auto r = b->read("k", key); + auto requests = makeRequests(b, clock); + auto op = requests.admit(); + + const std::optional w1 = orThrow(op.create("k", "v1", Retry::once()), "create"); + ASSERT_TRUE(w1); + const std::optional r = op.read("k", Retry::once()); ASSERT_TRUE(r); EXPECT_EQ(r->bytes, "v1"); - EXPECT_EQ(r->value, *w1); + EXPECT_EQ(r->incarnation, *w1); - auto h = b->head("k", key); + const std::optional h = op.head("k", Retry::once()); ASSERT_TRUE(h); EXPECT_EQ(h->size, 2u); - EXPECT_EQ(h->value, *w1); + EXPECT_EQ(h->incarnation, *w1); - auto w2 = b->write("k", "v2", std::nullopt, key); /// must be absent → refused - EXPECT_FALSE(w2.has_value()); - auto w3 = b->write("k", "v2", *w1, key); - ASSERT_TRUE(w3.has_value()); - EXPECT_NE(*w3, *w1); /// values never repeat + EXPECT_TRUE(std::holds_alternative(op.create("k", "v2", Retry::once()))); /// must be absent + const std::optional w3 = orThrow(op.replace("k", "v2", *w1, Retry::once()), "replace"); + ASSERT_TRUE(w3); + EXPECT_NE(*w3, *w1); /// incarnations never repeat - EXPECT_EQ(b->remove("k", *w1, key), Backend::RawRemoval::Mismatch); - EXPECT_EQ(b->remove("k", *w3, key), Backend::RawRemoval::Removed); - EXPECT_EQ(b->remove("k", *w3, key), Backend::RawRemoval::Gone); - EXPECT_FALSE(b->read("k", key).has_value()); + EXPECT_EQ(op.remove("k", *w1, Retry::once()), Removal::Mismatch); + EXPECT_EQ(op.remove("k", *w3, Retry::once()), Removal::Removed); + EXPECT_EQ(op.remove("k", *w3, Retry::once()), Removal::Gone); + EXPECT_FALSE(op.read("k", Retry::once()).has_value()); } -TEST(CASBackendPrimitives, ListSurfacesTheIncarnationValueAndPaginates) +TEST(CASBackendPrimitives, ListSurfacesTheIncarnationAndPaginates) { + FakeClock clock; auto b = std::make_shared(); - auto key = RawDoor::key(); - const String a = *b->write("p/a", "0123456789", std::nullopt, key); - b->write("p/b", "xy", std::nullopt, key); - b->write("q/c", "z", std::nullopt, key); + auto requests = makeRequests(b, clock); + auto op = requests.admit(); + + const std::optional a = orThrow(op.create("p/a", "0123456789", Retry::once()), "create"); + ASSERT_TRUE(a); + orThrow(op.create("p/b", "xy", Retry::once()), "create"); + orThrow(op.create("q/c", "z", Retry::once()), "create"); - const auto page = b->list("p/", "", 10, key); + const KeyPage page = op.list("p/", "", 10, Retry::once()); ASSERT_EQ(page.keys.size(), 2u); /// sorted, prefix-scoped EXPECT_EQ(page.keys[0].key, "p/a"); EXPECT_EQ(page.keys[0].size, 10u); - ASSERT_TRUE(page.keys[0].value.has_value()); - EXPECT_EQ(*page.keys[0].value, a); + ASSERT_TRUE(page.keys[0].incarnation.has_value()); + EXPECT_EQ(*page.keys[0].incarnation, *a); EXPECT_TRUE(page.next_cursor.empty()); - const auto first = b->list("p/", "", 1, key); + const KeyPage first = op.list("p/", "", 1, Retry::once()); ASSERT_EQ(first.keys.size(), 1u); EXPECT_EQ(first.next_cursor, "p/a"); - const auto second = b->list("p/", first.next_cursor, 1, key); + const KeyPage second = op.list("p/", first.next_cursor, 1, Retry::once()); ASSERT_EQ(second.keys.size(), 1u); EXPECT_EQ(second.keys[0].key, "p/b"); } @@ -237,53 +248,56 @@ struct WriteCountingBackend : InMemoryBackend } -TEST(CASBackendPrimitives, ALegacyCallReachesAnOverrideOfThePrimitiveItForwardsTo) +TEST(CASBackendPrimitives, EveryLegacyVerbReachesAnOverrideOfThePrimitiveItForwardsTo) { /// The migration rule: the new methods are the primitives, and a fault injection written against a - /// NEW signature intercepts a legacy caller too, because the legacy verb forwards through the - /// virtual. `putIfAbsent` and `casPut` are this backend's two documented exceptions -- they route - /// around the primitive to keep their knobs' verb identity -- so the rule is asserted on a verb - /// that does forward. + /// NEW signature intercepts a legacy caller too, because every legacy verb forwards through the + /// virtual. No verb is exempt -- an exemption would leave a double blind to whichever surface its + /// subject happens to use. auto b = std::make_shared(); - auto door = RawDoor::key(); - const String first = *b->write("k", "v", std::nullopt, door); + FakeClock clock; + auto requests = makeRequests(b, clock); + auto op = requests.admit(); + const std::optional first = orThrow(op.create("k", "v", Retry::once()), "create"); + ASSERT_TRUE(first); b->writes = 0; - b->putOverwrite("k", "w", Token{first, Dialect::Emulated}); /// legacy call - EXPECT_EQ(b->writes, 1u); - - /// And the negative half of the same ruling: the two exceptions really do route around the - /// primitive. Both writes LAND, so this cannot pass by the calls having done nothing. - b->writes = 0; EXPECT_EQ(b->putIfAbsent("k2", "v").outcome, PutOutcome::Done); EXPECT_EQ(b->casPut("k3", "v", std::nullopt).outcome, CasOutcome::Committed); - EXPECT_EQ(b->writes, 0u); + EXPECT_EQ(b->putOverwrite("k", "w", b->head("k").token).outcome, PutOutcome::Done); + EXPECT_EQ(b->writes, 3u); } -TEST(CASBackendPrimitives, EachWriteKnobFiresOnlyForTheVerbItNames) +TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotWhicheverSurfaceConsumesIt) { - /// A knob is armed against a VERB. The keyed `write` cannot see which verb its caller used, so - /// consuming both knobs there is right and consuming the other one from a legacy verb is not: a - /// test that arms an ambiguity for `putIfAbsent` must not have it fire on a `casPut`. + /// A knob names a KEY, not a verb: the keyed `write` every surface reaches cannot see which verb + /// its caller used, so a knob scoped to one verb would fire or not fire on where the caller + /// happened to enter rather than on what it did. auto b = std::make_shared(); - auto door = RawDoor::key(); + FakeClock clock; + auto requests = makeRequests(b, clock); + auto op = requests.admit(); + + b->refuseNextWrite("k"); + EXPECT_EQ(b->putIfAbsent("k", "v").outcome, PutOutcome::PreconditionFailed); /// consumed here + EXPECT_EQ(b->putIfAbsent("k", "v").outcome, PutOutcome::Done); /// and only once + expectBytes(b, "k", "v"); - b->failNextCasPut("k"); - EXPECT_EQ(b->putIfAbsent("k", "v").outcome, PutOutcome::Done); /// not casPut's knob to consume - const Token present = b->head("k").token; - EXPECT_EQ(b->casPut("k", "w", present).outcome, CasOutcome::Conflict); - EXPECT_EQ(b->get("k")->bytes, "v"); /// the refusal changed nothing + b->refuseNextWrite("k2"); + EXPECT_TRUE(std::holds_alternative(op.create("k2", "v", Retry::once()))); + EXPECT_TRUE(std::holds_alternative(op.create("k2", "v", Retry::once()))); - b->injectAmbiguousPutIfAbsent("k2"); - EXPECT_EQ(b->casPut("k2", "v", std::nullopt).outcome, CasOutcome::Committed); /// not casPut's knob either - EXPECT_THROW(b->putIfAbsent("k2", "w"), std::runtime_error); + b->injectAmbiguousWrite("k3"); + EXPECT_THROW(b->casPut("k3", "v", std::nullopt), Poco::TimeoutException); + EXPECT_FALSE(b->get("k3").has_value()) << "an ambiguous write leaves the store untouched"; + EXPECT_EQ(b->casPut("k3", "v", std::nullopt).outcome, CasOutcome::Committed); - /// The keyed primitive is the one caller every knob is armed against, and each is still one-shot. - b->injectAmbiguousPutIfAbsent("k3"); - b->failNextCasPut("k3"); - EXPECT_THROW((void)b->write("k3", "v", std::nullopt, door), std::runtime_error); - EXPECT_FALSE(b->write("k3", "v", std::nullopt, door).has_value()); - EXPECT_TRUE(b->write("k3", "v", std::nullopt, door).has_value()); + /// Both knobs on one key, each consumed by the next write in turn. + b->injectAmbiguousWrite("k4"); + b->refuseNextWrite("k4"); + EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); + EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); + EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); } TEST(CASBackendPrimitives, LegacyGetRefusesAValueThatIsNotAnIncarnation) @@ -326,7 +340,7 @@ TEST(CASBackendPrimitives, InstrumentedBackendPassesALegacyCallThroughAsLegacy) InstrumentedBackend instrumented(inner); EXPECT_EQ(instrumented.casPut("k", "v", std::nullopt).outcome, CasOutcome::Committed); EXPECT_TRUE(inner->legacy_cas_put_ran); - EXPECT_EQ(inner->casPutCount("k"), 1u); + EXPECT_EQ(inner->writeCount("k"), 1u); } TEST(CASBackendPrimitives, RefreshCredentialsIsOffUntilAskedFor) @@ -339,36 +353,35 @@ TEST(CASBackendPrimitives, RefreshCredentialsIsOffUntilAskedFor) #if USE_AWS_S3 -TEST(CASThrottlingBackend, FirstPerKeyRefusesOnceThenForwards) +TEST(CASThrottlingBackend, FirstPerKeyRefusesOnceAndTheCallStillSucceeds) { + FakeClock clock; auto inner = std::make_shared(); auto t = std::make_shared(inner, ThrottlingBackend::Mode::FirstPerKey, 0, 429); - auto key = RawDoor::key(); - EXPECT_THROW(t->read("k", key), DB::S3Exception); - EXPECT_NO_THROW(t->read("k", key)); - EXPECT_EQ(t->refusals("k"), 1u); - EXPECT_THROW(t->write("k2", "v", std::nullopt, key), DB::S3Exception); - EXPECT_TRUE(t->write("k2", "v", std::nullopt, key).has_value()); + auto requests = makeRequests(t, clock); + auto op = requests.admit(); + + orThrow(op.create("k2", "v", Retry::standard()), "create"); + EXPECT_EQ(t->refusals("k2"), 1u); + EXPECT_TRUE(op.read("k2", Retry::standard()).has_value()); + EXPECT_EQ(t->refusals("k2"), 1u) << "only the FIRST request naming a key is refused"; } TEST(CASThrottlingBackend, RefusalsAreRetryableUnderBothStatuses) { - auto key = RawDoor::key(); + /// The property the seam exists for: a refusal must reach the engine as an AMBIGUOUS attempt, not + /// a definite failure. What proves it is that the engine REISSUES -- a definite failure would + /// surface unchanged, with the refusal still the only request the store ever saw. for (const int status : {429, 503}) { + FakeClock clock; auto t = std::make_shared( std::make_shared(), ThrottlingBackend::Mode::FirstPerKey, 0, status); - try - { - t->head("k", key); - FAIL() << "expected a refusal for status " << status; - } - catch (const DB::S3Exception & e) - { - /// The property the seam exists for: the engine must see an AMBIGUOUS attempt, which is - /// what a retryable store error means, not a definite failure. - EXPECT_TRUE(e.isRetryableError()) << "status " << status; - } + auto requests = makeRequests(t, clock); + auto op = requests.admit(); + + EXPECT_FALSE(op.head("k", Retry::standard()).has_value()) << "status " << status; + EXPECT_EQ(t->refusals("k"), 1u) << "status " << status; } } @@ -379,20 +392,24 @@ TEST(CASThrottlingBackend, PassesALegacyCallThroughAsLegacy) auto t = std::make_shared(inner, ThrottlingBackend::Mode::EveryNth, 1000, 503); EXPECT_EQ(t->casPut("k", "v", std::nullopt).outcome, CasOutcome::Committed); EXPECT_TRUE(inner->legacy_cas_put_ran); - EXPECT_EQ(inner->casPutCount("k"), 1u); + EXPECT_EQ(inner->writeCount("k"), 1u); } TEST(CASThrottlingBackend, EveryNthRefusesOnThePeriodAcrossKeys) { + FakeClock clock; auto inner = std::make_shared(); auto t = std::make_shared(inner, ThrottlingBackend::Mode::EveryNth, 3, 503); - auto key = RawDoor::key(); - EXPECT_NO_THROW(t->read("a", key)); - EXPECT_NO_THROW(t->read("b", key)); - EXPECT_THROW(t->read("c", key), DB::S3Exception); /// the third request, whatever it names + auto requests = makeRequests(t, clock); + auto op = requests.admit(); + + EXPECT_FALSE(op.read("a", Retry::standard()).has_value()); + EXPECT_FALSE(op.read("b", Retry::standard()).has_value()); + /// The THIRD request is refused whatever it names; the engine reissues it as the fourth. + EXPECT_FALSE(op.read("c", Retry::standard()).has_value()); EXPECT_EQ(t->refusals("c"), 1u); EXPECT_EQ(t->refusals("a"), 0u); - EXPECT_NO_THROW(t->read("c", key)); + EXPECT_EQ(t->refusals("b"), 0u); } #endif @@ -572,7 +589,7 @@ TEST(CASRequests, KeyBindingThrowsBeforeAnyRequest) backend->resetCounts(); expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { (void)op.replace("b", "w", of_a, Retry::standard()); }); - EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); EXPECT_TRUE(clock.sleeps.empty()); } #else @@ -605,8 +622,8 @@ TEST(CASRequests, EveryConflictIsSettledByOneReadAndCarriesTheOccupant) EXPECT_EQ(occupant->bytes, "theirs"); /// The refused precondition says only that the key is taken; ONE exact read says by whom. - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); } TEST(CASRequests, AmbiguousCreateThatLandedIsCommittedByTheResolveRead) @@ -626,8 +643,8 @@ TEST(CASRequests, AmbiguousCreateThatLandedIsCommittedByTheResolveRead) EXPECT_EQ(committed->attempts_sent, 1u); /// Settled by reading, never by writing again: a second create would have conflicted with the /// first one's own object. - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -635,7 +652,7 @@ TEST(CASRequests, AmbiguousCreateThatNeverLandedIsReissued) { FakeClock clock; auto backend = std::make_shared(); - backend->injectAmbiguousPutIfAbsent("k"); /// the attempt's outcome is lost and the store is untouched + backend->injectAmbiguousWrite("k"); /// the attempt's outcome is lost and the store is untouched auto requests = makeRequests(backend, clock); auto op = requests.admit(); @@ -644,7 +661,7 @@ TEST(CASRequests, AmbiguousCreateThatNeverLandedIsReissued) ASSERT_NE(committed, nullptr); EXPECT_FALSE(committed->resolved_by_read); EXPECT_EQ(committed->attempts_sent, 2u); - EXPECT_EQ(backend->readRequests(), 1u); /// the resolve proved absence, and only then did a reissue follow + EXPECT_EQ(backend->getTotal(), 1u); /// the resolve proved absence, and only then did a reissue follow EXPECT_EQ(clock.sleeps.size(), 1u); } @@ -664,8 +681,8 @@ TEST(CASRequests, OnceSendsOneWriteAndAtMostOneResolveRead) EXPECT_TRUE(gave_up->sent_any); EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); /// One attempt is one attempt, but the read that would have settled it is still owed and sent. - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -680,8 +697,8 @@ TEST(CASRequests, DecideMayThrowAndTheExceptionPropagatesUnchanged) op.readModifyWrite("k", [](const std::optional &) -> std::optional { throw DecideMarker{}; }, Retry::standard()), DecideMarker); - EXPECT_EQ(backend->writeRequests(), 0u); - EXPECT_EQ(backend->readRequests(), 1u); /// the key was read, and nothing was decided about it + EXPECT_EQ(backend->writeTotal(), 0u); + EXPECT_EQ(backend->getTotal(), 1u); /// the key was read, and nothing was decided about it } TEST(CASRequests, OnPresenceIssuesHeadsAndNoGet) @@ -698,15 +715,17 @@ TEST(CASRequests, OnPresenceIssuesHeadsAndNoGet) }, Retry::standard()); ASSERT_TRUE(std::holds_alternative(result)); - EXPECT_EQ(backend->readRequests(), 0u); - EXPECT_GE(backend->headRequests(), 1u); + EXPECT_EQ(backend->getTotal(), 0u); + /// One HEAD decided it and one write landed it: the loop issues no request it does not need. + EXPECT_EQ(backend->headTotal(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); } TEST(CASRequests, OnPresenceSettlesARefusedPreconditionWithAHead) { FakeClock clock; auto backend = std::make_shared(); - backend->failNextCasPut("k"); /// the store refuses the precondition, writing nothing + backend->refuseNextWrite("k"); /// the store refuses the precondition, writing nothing auto requests = makeRequests(backend, clock); auto op = requests.admit(); @@ -718,9 +737,9 @@ TEST(CASRequests, OnPresenceSettlesARefusedPreconditionWithAHead) Retry::standard()); ASSERT_TRUE(std::holds_alternative(result)); /// A refused precondition needs only to know WHAT is at the key, so this loop never fetches a body. - EXPECT_EQ(backend->readRequests(), 0u); - EXPECT_GE(backend->headRequests(), 2u); - EXPECT_EQ(backend->writeRequests(), 2u); + EXPECT_EQ(backend->getTotal(), 0u); + EXPECT_EQ(backend->headTotal(), 2u); + EXPECT_EQ(backend->writeTotal(), 2u); } TEST(CASRequests, ForEachListedKeyStopsEarlyAndBudgetsPerPage) @@ -740,7 +759,7 @@ TEST(CASRequests, ForEachListedKeyStopsEarlyAndBudgetsPerPage) EXPECT_EQ(seen, 3u); /// The walk stops where the caller stops it: the remaining two pages are never fetched. EXPECT_EQ(pages, 1u); - EXPECT_EQ(backend->listRequests(), 1u); + EXPECT_EQ(backend->listTotal(), 1u); } TEST(CASRequests, DeleteMarkerIsANamedException) @@ -822,7 +841,6 @@ TEST(CASRequests, AdmissionIsCheckedAtThreePoints) { FakeClock clock; auto backend = std::make_shared(); - auto door = RawDoor::key(); uint64_t generation = 1; bool lost = false; Fence fence{ @@ -833,6 +851,10 @@ TEST(CASRequests, AdmissionIsCheckedAtThreePoints) }, [&](uint64_t) {}}; auto requests = makeRequests(backend, clock, fence); + /// The store is observed through an OPEN fence: these checks run while the subject's own fence is + /// closed, and a fenced read would report the fence rather than the store. + auto observer_requests = makeRequests(backend, clock); + auto observer = observer_requests.admit(); /// (1) before the first attempt, on a handle resumed under a generation the fence has moved past { @@ -842,7 +864,7 @@ TEST(CASRequests, AdmissionIsCheckedAtThreePoints) ASSERT_NE(gave_up, nullptr); EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); EXPECT_FALSE(gave_up->sent_any); - EXPECT_FALSE(backend->read("k", door).has_value()); + EXPECT_FALSE(observer.read("k", Retry::once()).has_value()); } /// (2) before the next verb of an admitted handle, after a re-arm between two verbs { @@ -854,7 +876,7 @@ TEST(CASRequests, AdmissionIsCheckedAtThreePoints) ASSERT_NE(gave_up, nullptr); EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); EXPECT_FALSE(gave_up->sent_any); - EXPECT_FALSE(backend->read("k", door).has_value()); + EXPECT_FALSE(observer.read("k", Retry::once()).has_value()); } /// (3) after a proven commit: the write landed, then the fence tripped before the call returned { @@ -866,7 +888,7 @@ TEST(CASRequests, AdmissionIsCheckedAtThreePoints) EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); EXPECT_TRUE(gave_up->sent_any); /// The object IS durable. This call refuses to CLAIM it; it does not undo it. - EXPECT_TRUE(backend->read("k2", door).has_value()); + EXPECT_TRUE(observer.read("k2", Retry::once()).has_value()); } } @@ -876,7 +898,7 @@ TEST(CASRequests, TheGateBeforeTheSleepEndsTheCallWithoutASecondWrite) auto backend = std::make_shared(); bool alive = true; backend->on_read = [&] { alive = false; }; - backend->injectAmbiguousPutIfAbsent("k"); + backend->injectAmbiguousWrite("k"); auto requests = makeRequests(backend, clock); auto op = requests.admit([&] { return alive; }); @@ -888,8 +910,8 @@ TEST(CASRequests, TheGateBeforeTheSleepEndsTheCallWithoutASecondWrite) /// The ambiguous attempt was resolved, and the pause before the reissue was refused rather than /// served: no sleep, and no second attempt after it. EXPECT_TRUE(clock.sleeps.empty()); - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); } TEST(CASRequests, AResolveReadRefusedForLeaseBudgetIsReportedAsTheLeaseDeadline) @@ -908,7 +930,7 @@ TEST(CASRequests, AResolveReadRefusedForLeaseBudgetIsReportedAsTheLeaseDeadline) /// The store refuses the precondition, and the lease budget is gone by the time the read that /// would say WHO holds the key is due. The call learned nothing about the key, so what it reports /// is the bound that stopped it -- not a conflict it never observed. - backend->failNextCasPut("k"); + backend->refuseNextWrite("k"); backend->onBeforeWrite("k", [&] { lease_spent = true; }); const uint64_t lease_deadline = clock.now + 10'000; WriteResult result = op.replace("k", "w", seen, Retry::untilLeaseSafe(lease_deadline, 2'000)); @@ -919,8 +941,8 @@ TEST(CASRequests, AResolveReadRefusedForLeaseBudgetIsReportedAsTheLeaseDeadline) EXPECT_TRUE(gave_up->sent_any); EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); EXPECT_TRUE(clock.sleeps.empty()); - EXPECT_EQ(backend->writeRequests(), 2u); /// the create and the one refused replace - EXPECT_EQ(backend->readRequests(), 0u); /// the resolve read never started + EXPECT_EQ(backend->writeTotal(), 2u); /// the create and the one refused replace + EXPECT_EQ(backend->getTotal(), 0u); /// the resolve read never started } TEST(CASRequests, AFenceWithNoBudgetForTheRequestSendsNothingAndNamesTheLease) @@ -946,8 +968,8 @@ TEST(CASRequests, AFenceWithNoBudgetForTheRequestSendsNothingAndNamesTheLease) EXPECT_FALSE(gave_up->sent_any); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.read("k", Retry::standard()); }); - EXPECT_EQ(backend->writeRequests(), 0u); - EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); + EXPECT_EQ(backend->getTotal(), 0u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -968,7 +990,7 @@ TEST(CASRequests, AnRmwWhoseFirstReadFailsGivesUpUnresolvedWithoutWriting) EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); EXPECT_FALSE(gave_up->sent_any); EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); - EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); } TEST(CASRequests, AnOnPresenceRmwWhoseFirstHeadFailsGivesUpUnresolvedWithoutWriting) @@ -985,8 +1007,8 @@ TEST(CASRequests, AnOnPresenceRmwWhoseFirstHeadFailsGivesUpUnresolvedWithoutWrit ASSERT_NE(gave_up, nullptr); EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); EXPECT_FALSE(gave_up->sent_any); - EXPECT_EQ(backend->writeRequests(), 0u); - EXPECT_EQ(backend->readRequests(), 0u); /// the presence loop does not fall back to a body read + EXPECT_EQ(backend->writeTotal(), 0u); + EXPECT_EQ(backend->getTotal(), 0u); /// the presence loop does not fall back to a body read } TEST(CASRequests, AConflictWhoseResolveReadFailsIsReportedWithNothingObserved) @@ -1014,7 +1036,7 @@ TEST(CASRequests, AFenceLostDuringTheResolveReadIsAFenceLossNotAConflict) /// The fence trips while the ambiguous attempt is in flight: the write's own hook runs before the /// store is touched, so the resolve read is the first request to meet the closed gate. backend->onBeforeWrite("k", [&] { alive = false; }); - backend->injectAmbiguousPutIfAbsent("k"); + backend->injectAmbiguousWrite("k"); auto requests = makeRequests(backend, clock); auto op = requests.admit([&] { return alive; }); @@ -1025,8 +1047,8 @@ TEST(CASRequests, AFenceLostDuringTheResolveReadIsAFenceLossNotAConflict) /// somebody else holds the key, when what happened is that this node stopped being allowed to ask. EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); EXPECT_TRUE(gave_up->sent_any); - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 0u); /// refused before the resolve read was issued + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 0u); /// refused before the resolve read was issued EXPECT_TRUE(clock.sleeps.empty()); } @@ -1049,18 +1071,30 @@ TEST(CASRequests, OnPresenceFetchesTheBodyToProveAnAmbiguousAttemptLanded) EXPECT_TRUE(committed->resolved_by_read); /// Presence-only is what this loop REPORTS, not a promise about what it may read: only the bytes /// can prove the ambiguous attempt was this call's own. - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); } TEST(CASRequests, OnPresenceReportsMetaEvenWhenItHadToFetchTheBody) { FakeClock clock; auto backend = std::make_shared(); + auto competitor = makeRequests(backend, clock); + auto rival = competitor.admit(); + + /// A competitor takes the key while our own create is in flight, and that create's own fate is + /// lost. The ambiguity is armed from inside the hook so the competitor's write cannot consume it. + bool staged = false; + backend->onBeforeWrite("k", [&] + { + if (staged) + return; + staged = true; + (void)rival.create("k", "theirs", Retry::once()); + backend->injectAmbiguousWrite("k"); + }); + auto requests = makeRequests(backend, clock); auto op = requests.admit(); - orThrow(op.create("k", "theirs", Retry::standard()), "create"); - - backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); WriteResult result = op.readModifyWriteOnPresence("k", [](const std::optional &) -> std::optional { return String("mine"); }, Retry::once()); const auto * conflict = std::get_if(&result); @@ -1069,7 +1103,177 @@ TEST(CASRequests, OnPresenceReportsMetaEvenWhenItHadToFetchTheBody) /// presence loop can never come to depend on bytes the loop does not promise. EXPECT_TRUE(std::holds_alternative(conflict->seen)); EXPECT_FALSE(std::holds_alternative(conflict->seen)); - EXPECT_GE(backend->readRequests(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); +} + +TEST(CASRequests, AmbiguousReplaceWhoseResolveShowsThePreconditionUnchangedIsReissued) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation seen = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + backend->resetCounts(); + + /// The attempt's fate is lost and the store is untouched. The incarnation it named is still the + /// current one -- which proves nothing landed, and leaves a precondition a reissue can still meet. + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + WriteResult result = op.replace("k", "v2", seen, Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_EQ(backend->writeTotal(), 2u); + EXPECT_EQ(backend->getTotal(), 1u); /// exactly one resolve read, and it settled the ambiguity + EXPECT_EQ(clock.sleeps.size(), 1u); +} + +TEST(CASRequests, AmbiguousReplaceOfIdenticalBytesIsReissuedNotClaimedByByteEquality) +{ + /// The key already holds exactly the bytes we are about to write, so byte equality alone can never + /// say whether the ambiguous attempt landed. The incarnation can: an attempt that applied would + /// have moved it. Under a policy with a reissue that means re-sending; under `once` it means saying + /// the write is unresolved rather than claiming somebody else's identical object. + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation seen = *orThrow(op.create("k", "B", Retry::standard()), "create"); + backend->resetCounts(); + + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + WriteResult result = op.replace("k", "B", seen, Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + /// Claiming the resolve read's object would have reported one attempt and a commit this call + /// never made; the reissue is what actually put these bytes there under a new incarnation. + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_NE(committed->incarnation, seen); + EXPECT_EQ(backend->writeTotal(), 2u); + EXPECT_EQ(backend->getTotal(), 1u); + EXPECT_EQ(clock.sleeps.size(), 1u); + } + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation seen = *orThrow(op.create("k", "B", Retry::standard()), "create"); + backend->resetCounts(); + + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + WriteResult result = op.replace("k", "B", seen, Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); + } +} + +TEST(CASRequests, AmbiguousReplaceWhoseResolveShowsAnotherIncarnationIsAConflict) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Incarnation stale = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + orThrow(op.replace("k", "theirs", stale, Retry::standard()), "the competitor's replace"); + backend->resetCounts(); + + /// The attempt's fate is lost, and the key has moved past the incarnation it named: no reissue of + /// it could ever apply, so the ambiguity is settled and the occupant is the answer. + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + WriteResult result = op.replace("k", "mine", stale, Retry::standard()); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + const auto * occupant = std::get_if(&conflict->seen); + ASSERT_NE(occupant, nullptr); + EXPECT_EQ(occupant->bytes, "theirs"); + EXPECT_NE(occupant->incarnation, stale); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); + /// The count the conflict reports is the count the transport saw, not a constant that happens to + /// match here: a caller totalling attempts across endings has to be able to add this one. + EXPECT_EQ(conflict->attempts_sent, backend->writeTotal()); + EXPECT_TRUE(clock.sleeps.empty()); +} + +TEST(CASRequests, ReadModifyWriteDoesNotClaimACompetitorsIdenticalBytesAfterAnEarlierAmbiguity) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto competitor = makeRequests(backend, clock); + auto rival = competitor.admit(); + + /// The competitor moves the key once before each of our two attempts, and its own writes re-enter + /// this hook. The ambiguity is armed here rather than up front so the competitor's create cannot + /// consume the arming meant for ours. + bool inside = false; + int staged = 0; + backend->onBeforeWrite("k", [&] + { + if (inside) + return; + inside = true; + if (staged == 0) + { + (void)rival.create("k", "X", Retry::once()); + backend->injectAmbiguousWrite("k"); + } + else if (staged == 1) + { + /// The bytes we are about to send, under an incarnation that is not ours. + if (const auto current = rival.read("k", Retry::once())) + (void)rival.replace("k", "B", current->incarnation, Retry::once()); + } + ++staged; + inside = false; + }); + + std::vector decided_on; + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + WriteResult result = op.readModifyWrite("k", + [&](const std::optional & current) -> std::optional + { + decided_on.push_back(current ? current->bytes : String("")); + if (!current) + return String("A"); + if (current->bytes == "X") + return String("B"); + return std::nullopt; + }, + Retry::standard()); + + /// The only ambiguity this call had belonged to "A", and the competitor's "X" already proved it + /// dead. "B" at the key is the competitor's, so the loop re-decides on it instead of claiming it. + const auto * declined = std::get_if(&result); + ASSERT_NE(declined, nullptr); + const auto * seen = std::get_if(&declined->seen); + ASSERT_NE(seen, nullptr); + EXPECT_EQ(seen->bytes, "B"); + EXPECT_EQ(decided_on, (std::vector{"", "X", "B"})); +} + +TEST(CASRequests, AnUnmodeledLocalExceptionOnAWritePropagatesUnchanged) +{ + FakeClock clock; + auto backend = std::make_shared(); + /// Not a `Poco::Exception`, so it did not come from the transport and cannot have landed anything. + /// Settling it by a read would report a store answer the store never gave. + backend->failNextWriteWith("k", std::make_exception_ptr(std::logic_error("a local bug"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + EXPECT_THROW((void)op.create("k", "v", Retry::standard()), std::logic_error); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 0u); + EXPECT_TRUE(clock.sleeps.empty()); } TEST(CASRequests, ForEachListedKeyGivesEachPageItsOwnPolicyWindow) @@ -1141,11 +1345,11 @@ TEST(CASRequests, LivenessPredicateEndsTheOperationLikeAFenceLoss) ASSERT_NE(gave_up, nullptr); EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); EXPECT_FALSE(gave_up->sent_any); - EXPECT_EQ(backend->writeRequests(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); /// The read surface reports the same refusal the only way it can: by exception. expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.read("k", Retry::standard()); }); - EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->getTotal(), 0u); } TEST(CASRequests, ReadModifyWriteLosesNoIncrementUnderContentionAndBoundsAHotKey) @@ -1182,15 +1386,16 @@ TEST(CASRequests, ReadModifyWriteLosesNoIncrementUnderContentionAndBoundsAHotKey /// A key rewritten under EVERY attempt is bounded by the deadline instead of looping forever. FakeClock clock; auto hot = makeRequests(backend, clock); + auto competitor = makeRequests(backend, clock); + auto rival = competitor.admit(); bool inside_hook = false; backend->onBeforeWrite("ctr", [&] { if (inside_hook) /// the hook's own write re-enters this callback return; inside_hook = true; - auto door = RawDoor::key(); - if (auto raw = backend->read("ctr", door)) - (void)backend->write("ctr", "999", raw->value, door); + if (const auto current = rival.read("ctr", Retry::once())) + (void)rival.replace("ctr", "999", current->incarnation, Retry::once()); inside_hook = false; }); @@ -1214,7 +1419,7 @@ TEST(CASRequests, ADeterministicLocalFailureSurfacesUnchangedWithoutAReissue) /// Reissuing would replay the same bug and bury it behind a retryable exception at the deadline. expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)op.read("k", Retry::standard()); }); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -1232,7 +1437,7 @@ TEST(CASRequests, ATransportTimeoutIsReissuedAndALocalFailureIsNot) const auto seen = op.read("k", Retry::standard()); ASSERT_TRUE(seen.has_value()); EXPECT_EQ(seen->bytes, "v"); - EXPECT_EQ(backend->readRequests(), 2u); + EXPECT_EQ(backend->getTotal(), 2u); EXPECT_EQ(clock.sleeps.size(), 1u); } { @@ -1244,7 +1449,7 @@ TEST(CASRequests, ATransportTimeoutIsReissuedAndALocalFailureIsNot) /// whole deadline replaying a local bug. backend->failNextReadWith("k", std::make_exception_ptr(std::logic_error("a local bug"))); EXPECT_THROW((void)op.read("k", Retry::standard()), std::logic_error); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } } @@ -1320,8 +1525,8 @@ TEST(CASRequests, AMalformedRequestIsRefusedWithoutAReissue) EXPECT_EQ(refused->store_error, DB::ErrorCodes::S3_ERROR); /// The store's own answer proves the request never applied: nothing to resolve, nothing to reissue, /// and no credential the refusal could be about. - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 0u); EXPECT_EQ(backend->refreshCredentialsCalls(), 0u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -1339,7 +1544,7 @@ TEST(CASRequests, AnAccessDenialNoRefreshCanFixIsRefusedOnTheFirstAttempt) ASSERT_TRUE(std::holds_alternative(result)); /// A refresh is asked for once and installs nothing, and THAT is what makes the denial terminal. EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); - EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -1356,28 +1561,36 @@ TEST(CASRequests, ASecondCredentialAnswerAfterTheOneRefreshIsRefused) WriteResult result = op.create("k", "v", Retry::standard()); /// The store answers a denial BEFORE it applies anything, so neither attempt landed and no read /// has anything to settle. A call gets one refresh, so the denial that survives it is the answer. - ASSERT_TRUE(std::holds_alternative(result)); + const auto * refused = std::get_if(&result); + ASSERT_NE(refused, nullptr); EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); - EXPECT_EQ(backend->writeRequests(), 2u); - EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->writeTotal(), 2u); + EXPECT_EQ(backend->getTotal(), 0u); + /// BOTH attempts are counted, not just the one that produced the answer -- which is why this is + /// asserted on the refusal that took two rather than on one of the single-attempt refusals. + EXPECT_EQ(refused->attempts_sent, backend->writeTotal()); EXPECT_EQ(clock.sleeps.size(), 1u); /// the one paced re-send under the credentials it installed } -TEST(CASRequests, UnderOnceTheStoreAnswerStandsEvenWhenTheRefreshSucceeded) +TEST(CASRequests, UnderOnceACredentialAnswerIsRefusedWithoutARefresh) { FakeClock clock; auto backend = std::make_shared(); + /// A refresh that WOULD have installed credentials, so the zero below is the gate and not the + /// storage refusing to hand any back. backend->setRefreshCredentialsResult(true); backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - /// Fresh credentials only help a reissue, and `once` has none to sign. Reporting the attempt as - /// unresolved instead would turn a policy that sends one request into one that sleeps. + /// Fresh credentials only help a reissue, and `once` has none to sign, so none are asked for -- + /// which is what keeps `Refused` meaning "no refresh installed credentials and no earlier + /// ambiguity" rather than "a refresh helped and the answer stood anyway". WriteResult result = op.create("k", "v", Retry::once()); ASSERT_TRUE(std::holds_alternative(result)); - EXPECT_EQ(backend->writeRequests(), 1u); - EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 0u); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 0u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -1398,7 +1611,7 @@ TEST(CASRequests, ACredentialAnswerAfterAnAmbiguousAttemptStillOwesTheResolveRea EXPECT_EQ(committed->attempts_sent, 3u); /// The refresh does not license a direct re-send here: the OTHER attempt is still unresolved, so /// the read that settles it is still owed. - EXPECT_EQ(backend->readRequests(), 2u); + EXPECT_EQ(backend->getTotal(), 2u); EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); } @@ -1418,7 +1631,7 @@ TEST(CASRequests, AnExpiredTokenARefreshFixesIsResentWithoutAResolveRead) EXPECT_FALSE(committed->resolved_by_read); /// The credential answer proves its OWN attempt never applied, and no earlier attempt of this call /// is unresolved, so the re-send under the fresh credentials owes no read. - EXPECT_EQ(backend->readRequests(), 0u); + EXPECT_EQ(backend->getTotal(), 0u); EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); EXPECT_EQ(clock.sleeps.size(), 1u); } @@ -1436,7 +1649,7 @@ TEST(CASRequests, AnExpiredTokenNoRefreshCanFixIsRefusedRatherThanRiddenToTheDea /// would otherwise spend the whole deadline being reissued. ASSERT_TRUE(std::holds_alternative(result)); EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); - EXPECT_EQ(backend->writeRequests(), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } @@ -1455,10 +1668,49 @@ TEST(CASRequests, ANameOnlyAccessDenialOnAReadPropagatesWhenNoRefreshIsAvailable /// propagates instead of spending its policy on a request that cannot start succeeding. expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { (void)op.read("k", Retry::standard()); }); EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); - EXPECT_EQ(backend->readRequests(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); EXPECT_TRUE(clock.sleeps.empty()); } +TEST(CASRequests, ReadModifyWriteWhoseResolveAndFreshObservationBothFailGivesUpUnresolved) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v0", Retry::standard()), "create"); + backend->resetCounts(); + + /// The store refuses the precondition, and both reads that would settle what happened answer with + /// a store refusal the read loop surfaces at once rather than reissuing. They are armed from inside + /// the write so the loop's OWN first read still succeeds and `decide` sees the object. + backend->refuseNextWrite("k"); + bool armed = false; + backend->onBeforeWrite("k", [&] + { + if (armed) + return; + armed = true; + backend->failNextReadWith("k", s3Error(Aws::S3::S3Errors::UNKNOWN, "MalformedXML")); + backend->failNextReadWith("k", s3Error(Aws::S3::S3Errors::UNKNOWN, "MalformedXML")); + }); + + WriteResult result = op.readModifyWrite("k", + [](const std::optional &) -> std::optional { return String("v1"); }, Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + /// No BOUND refused either read -- the reads themselves failed -- so naming a deadline the clock + /// never reached would send its reader to widen the wrong thing, and nothing was observed to + /// report as a conflict. + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); + /// The write count is unchanged after the first attempt: nothing ever said another one was safe. + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 3u); + EXPECT_EQ(clock.sleeps.size(), 1u); +} + TEST(CASRequests, AnUnmodeledStoreErrorOnAReadIsReissuedNotSurfaced) { FakeClock clock; @@ -1474,8 +1726,33 @@ TEST(CASRequests, AnUnmodeledStoreErrorOnAReadIsReissuedNotSurfaced) const auto seen = op.read("k", Retry::standard()); ASSERT_TRUE(seen.has_value()); EXPECT_EQ(seen->bytes, "v"); - EXPECT_EQ(backend->readRequests(), 2u); + EXPECT_EQ(backend->getTotal(), 2u); EXPECT_EQ(clock.sleeps.size(), 1u); } #endif + +/// A write reserves TWO request envelopes, not one: the attempt, and the read that settles it if the +/// attempt comes back ambiguous. At exactly one reservation of surplus before lease minus margin there +/// is room for the attempt alone, and an engine that reserved only the attempt would start one it +/// could not settle inside the bound. Nothing may be sent. +TEST(CASRequests, AWriteReservesTwoEnvelopesSoOneOfSurplusStartsNothing) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + requests.setAttemptReservationForTest(1'000); + + const uint64_t lease_deadline = clock.now + 2'000 + 1'000; + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::untilLeaseSafe(lease_deadline, 2'000)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Lease); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(backend->writeTotal(), 0u); + EXPECT_EQ(backend->getTotal(), 0u); + EXPECT_TRUE(clock.sleeps.empty()); +} + diff --git a/src/Disks/tests/gtest_cas_retirement_sweep.cpp b/src/Disks/tests/gtest_cas_retirement_sweep.cpp index 6deb241ec7a4..2c312c323772 100644 --- a/src/Disks/tests/gtest_cas_retirement_sweep.cpp +++ b/src/Disks/tests/gtest_cas_retirement_sweep.cpp @@ -82,14 +82,14 @@ class HoleyListBackend : public InMemoryBackend return served; } - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { - ListPage page = InMemoryBackend::list(prefix, cursor, limit); + RawListPage page = InMemoryBackend::list(prefix, cursor, limit, access); std::lock_guard lock(m); if (omitted.empty()) return page; auto it = std::find_if(page.keys.begin(), page.keys.end(), - [&](const ListedKey & k) { return k.key == omitted; }); + [&](const RawListedKey & k) { return k.key == omitted; }); if (it == page.keys.end()) return page; /// not a qualifying call -- do not count it if (seen_calls++ != target_call) @@ -123,38 +123,32 @@ class RefPrefixListCountingBackend : public InMemoryBackend std::atomic ref_prefix_lists{0}; std::atomic janitor_prefix_lists{0}; - ListPage list(const String & prefix, const String & cursor, size_t limit) override + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { if (!refs_prefix.empty() && prefix == refs_prefix) ++ref_prefix_lists; if (!janitor_prefix.empty() && prefix == janitor_prefix) ++janitor_prefix_lists; - return InMemoryBackend::list(prefix, cursor, limit); + return InMemoryBackend::list(prefix, cursor, limit, access); } }; -/// Forces the FIRST `putIfAbsent` whose key contains `fault_key_substr` to throw an ambiguous -/// (Unresolved-classified) exception, `fault_count` times -- the minimal fault injection needed to drive -/// a ref-log append into the `Unresolved`/wedge outcome, with `max_attempts = 1` in the budget so the -/// single failed attempt exhausts the retry budget immediately. (Same shape as `gtest_cas_pool.cpp`'s -/// file-local backend of the same name; both are three lines of `throw` over `InMemoryBackend`, and -/// hoisting a shared one would couple two suites' fault models for no gain.) +/// Every write of a matching key is a lost response, for as long as `fault_key_substr` names one. It +/// has to be every one: the request engine settles an ambiguity by an exact read and then reissues, so +/// a counted fault is outlived by the reissues and the write commits -- the difference between the +/// wedge this fixture needs and a clean commit. Clearing `fault_key_substr` disarms it. class UnresolvedPutBackend final : public InMemoryBackend { public: - using Backend::putIfAbsent; - String fault_key_substr; - int fault_count = 0; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + TransportAccess & access) override { - if (fault_count > 0 && !fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) - { - --fault_count; + if (!fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) throw Poco::TimeoutException("UnresolvedPutBackend: simulated ambiguous result (response lost)"); - } - return InMemoryBackend::putIfAbsent(key, bytes, meta); + return InMemoryBackend::write(key, bytes, expected_value, access); } }; @@ -337,7 +331,6 @@ TEST(CASRetirementSweep, TheRoundEnumeratesTheRefPrefixExactlyOnce) TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoverySeal) { CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; @@ -368,11 +361,14 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS publishOneBlobPart(store, ns, "x", "straggler-payload"); ASSERT_EQ(store->liveWriterEpoch(), 1u); - /// Drive the next ref-log append into the Unresolved/wedge outcome: the single attempt the budget - /// allows fails ambiguously, so this process can never learn whether its conditional PUT landed. - /// That undecidability is the whole reason the resolution is a conditional CREATE and not a GET. + /// Drive the next ref-log append into the Unresolved/wedge outcome: every attempt it makes fails + /// ambiguously, so this process can never learn whether its conditional PUT landed. That + /// undecidability is the whole reason the resolution is a conditional CREATE and not a GET. The + /// give-up is the append's own retry window, and the engine's inter-attempt sleeps pay it on the + /// same injected boot clock the fence and the deadline are measured against, so it costs no real + /// time and no lease. + store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms + 1; }); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; - backend->fault_count = 1; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 6beb6a08dabd..6d0e9f2d8335 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -156,13 +156,13 @@ class RecordingStagingBackend : public DB::Cas::InMemoryBackend std::vector copy_calls; - void publishBlob(const DB::Cas::BlobPublishRequest & request) override + void publish(const DB::Cas::BlobPublishRequest & request, DB::Cas::TransportAccess & access) override { if (const auto * copy = std::get_if(&request.publication)) copy_calls.push_back({copy->object_key, request.destination_key, true}); else copy_calls.push_back({String{}, request.destination_key, false}); - DB::Cas::InMemoryBackend::publishBlob(request); + DB::Cas::InMemoryBackend::publish(request, access); } /// Every key read as a stream, with a count. Republishing opens its source with `getStream`, so @@ -202,36 +202,40 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend explicit EtagFaithfulPublicationBackend(FaultScript script_) : script(script_) {} - /// Unhide the transport primitive that shares this name; the legacy override below is what the - /// production sites this double instruments still call. + /// Unhide the legacy convenience overloads that the primitive overrides below would otherwise + /// hide: `head(key)` and `deleteExact(key, token)` are inherited UNCHANGED (Backend's own default + /// implementations), so a caller through either legacy name still reaches the ETag-faithful + /// primitives below by virtual dispatch -- which is what the production sites this double + /// instruments actually call now. using DB::Cas::Backend::head; - DB::Cas::HeadResult head(const String & key) override + std::optional head(const String & key, DB::Cas::TransportAccess & access) override { - DB::Cas::HeadResult result = DB::Cas::InMemoryBackend::head(key); - if (result.exists && isBlobBodyKey(key)) + std::optional result = DB::Cas::InMemoryBackend::head(key, access); + if (result && isBlobBodyKey(key)) { - const auto body = DB::Cas::InMemoryBackend::get(key); + const auto body = DB::Cas::InMemoryBackend::read(key, access); chassert(body.has_value()); - result.token = DB::Cas::Token{sipHash128String(body->bytes), DB::Cas::TokenType::ETag}; + result->value = sipHash128String(body->bytes); } return result; } - DB::Cas::DeleteOutcome deleteExact(const String & key, const DB::Cas::Token & token) override + DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, + DB::Cas::TransportAccess & access) override { if (!isBlobBodyKey(key)) - return DB::Cas::InMemoryBackend::deleteExact(key, token); - - const DB::Cas::HeadResult current = head(key); - if (!current.exists) - return DB::Cas::DeleteOutcome{.kind = DB::Cas::DeleteOutcome::Kind::NotFound}; - if (current.token != token) - return DB::Cas::DeleteOutcome{.kind = DB::Cas::DeleteOutcome::Kind::TokenMismatch}; - return DB::Cas::InMemoryBackend::deleteExact(key, DB::Cas::InMemoryBackend::head(key).token); + return DB::Cas::InMemoryBackend::remove(key, expected_value, access); + + const auto current = head(key, access); + if (!current) + return DB::Cas::Backend::RawRemoval::Gone; + if (current->value != expected_value) + return DB::Cas::Backend::RawRemoval::Mismatch; + return DB::Cas::InMemoryBackend::remove(key, DB::Cas::InMemoryBackend::head(key, access)->value, access); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override + void publish(const DB::Cas::BlobPublishRequest & request, DB::Cas::TransportAccess & access) override { const bool is_copy = std::holds_alternative(request.publication); if (is_copy) @@ -245,7 +249,7 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend || (script == FaultScript::FirstCondemnedStreamLandsThenDeleted && !is_copy))) { fault_fired = true; - DB::Cas::InMemoryBackend::publishBlob(request); + DB::Cas::InMemoryBackend::publish(request, access); queued_delete_token = head(request.destination_key).token; if (script != FaultScript::CopyLandsThenCondemned) @@ -254,7 +258,7 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend throw Poco::TimeoutException("ETag-faithful staged publication response lost"); } - DB::Cas::InMemoryBackend::publishBlob(request); + DB::Cas::InMemoryBackend::publish(request, access); } FaultScript script; diff --git a/src/Disks/tests/gtest_cas_sentinel_probe.cpp b/src/Disks/tests/gtest_cas_sentinel_probe.cpp index 85896cf43d05..380343aa195f 100644 --- a/src/Disks/tests/gtest_cas_sentinel_probe.cpp +++ b/src/Disks/tests/gtest_cas_sentinel_probe.cpp @@ -3,9 +3,11 @@ #include "config.h" #include +#include #include #include #include +#include #include #include #include @@ -34,6 +36,20 @@ namespace using DB::Cas::tests::nativeKeyUnder; +/// Every test here constructs one backend and probes it once or a few times; a non-owning `BackendPtr` +/// over the test's stack-allocated backend keeps that construction pattern rather than forcing every +/// fixture in this file onto `std::make_shared`. The open fence never trips, matching every prior call +/// here having had no fence to enforce. `clock`, when given, drives `probeSentinel`'s reissue-on- +/// `Indeterminate` loop off an injected clock instead of a real sleep — needed by the one test whose +/// fault never resolves, so the loop runs its whole policy window without taking real wall-clock time. +CasRequests makeRequests(Backend & backend, DB::Cas::tests::FakeClock * clock = nullptr) +{ + BackendPtr ptr(&backend, [](Backend *) {}); + if (clock) + return CasRequests(std::move(ptr), Fence::open(), clock->nowFn(), clock->sleepFn()); + return CasRequests(std::move(ptr), Fence::open()); +} + /// A Backend decorator whose read/head/list all throw an untyped runtime error when armed — modelling /// a backend with no sharper evidence than "something went wrong" (a network timeout, a 5xx, an /// unclassifiable failure). The fault is injected on the PRIMITIVES, which is what @@ -78,7 +94,9 @@ TEST(CASSentinelProbe, PresentKeyReturnsPresentWithBody) InMemoryBackend backend; ASSERT_EQ(backend.putIfAbsent("k", "hello").outcome, PutOutcome::Done); - const auto result = probeSentinel(backend, "k"); + auto requests = makeRequests(backend); + auto op = requests.admit(); + const auto result = probeSentinel(op, "k", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::Present); ASSERT_TRUE(result.body.has_value()); EXPECT_EQ(*result.body, "hello"); @@ -90,7 +108,9 @@ TEST(CASSentinelProbe, AbsentKeyWithContainerAliveReturnsKeyAbsent) InMemoryBackend backend; ASSERT_EQ(backend.putIfAbsent("other", "x").outcome, PutOutcome::Done); // proves the backend is alive - const auto result = probeSentinel(backend, "missing"); + auto requests = makeRequests(backend); + auto op = requests.admit(); + const auto result = probeSentinel(op, "missing", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::KeyAbsent); EXPECT_FALSE(result.body.has_value()); } @@ -108,13 +128,16 @@ TEST(CASSentinelProbe, ContainerDirectoryRemovedReturnsContainerAbsent) ASSERT_EQ(backend.putIfAbsent("k", "hello").outcome, PutOutcome::Done); + auto requests = makeRequests(backend); + auto op = requests.admit(); + /// Sanity, container alive: Present vs. KeyAbsent are genuinely distinct before we remove anything. - EXPECT_EQ(probeSentinel(backend, "k").outcome, ProbeOutcome::Present); - EXPECT_EQ(probeSentinel(backend, "missing").outcome, ProbeOutcome::KeyAbsent); + EXPECT_EQ(probeSentinel(op, "k", Retry::standard()).outcome, ProbeOutcome::Present); + EXPECT_EQ(probeSentinel(op, "missing", Retry::standard()).outcome, ProbeOutcome::KeyAbsent); std::filesystem::remove_all(storage->getCommonKeyPrefix()); - const auto result = probeSentinel(backend, "k"); + const auto result = probeSentinel(op, "k", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::ContainerAbsent); EXPECT_FALSE(result.body.has_value()); } @@ -138,7 +161,9 @@ TEST(CASSentinelProbe, NativePresentKeyReturnsPresentWithBody) out->finalize(); } - const auto result = probeSentinel(backend, key); + auto requests = makeRequests(backend); + auto op = requests.admit(); + const auto result = probeSentinel(op, key, Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::Present); ASSERT_TRUE(result.body.has_value()); EXPECT_EQ(*result.body, "native body"); @@ -149,7 +174,12 @@ TEST(CASSentinelProbe, NativePresentKeyReturnsPresentWithBody) TEST(CASSentinelProbe, TransportErrorNeverClassifiesAsAbsent) { TransportFaultBackend backend; - const auto result = probeSentinel(backend, "k"); + /// The fault never resolves, so `probeSentinel`'s reissue-on-`Indeterminate` loop runs to its whole + /// policy window before giving up; an injected clock keeps that instantaneous instead of real time. + DB::Cas::tests::FakeClock clock; + auto requests = makeRequests(backend, &clock); + auto op = requests.admit(); + const auto result = probeSentinel(op, "k", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::Indeterminate); EXPECT_FALSE(result.body.has_value()); } @@ -220,7 +250,9 @@ TEST(CASSentinelProbe, NativeClassifiesNoSuchKeyAsKeyAbsent) storage->throwOnObjectAccess(Aws::S3::S3Errors::NO_SUCH_KEY); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::KeyAbsent); + auto requests = makeRequests(backend); + auto op = requests.admit(); + EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::KeyAbsent); } /// A real S3 HEAD's 404 has no response body, so the SDK cannot parse a `NoSuchKey` `` and @@ -233,7 +265,9 @@ TEST(CASSentinelProbe, NativeClassifiesResourceNotFoundAsKeyAbsent) storage->throwOnObjectAccess(Aws::S3::S3Errors::RESOURCE_NOT_FOUND); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::KeyAbsent); + auto requests = makeRequests(backend); + auto op = requests.admit(); + EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::KeyAbsent); } TEST(CASSentinelProbe, NativeClassifiesNoSuchBucketAsContainerAbsent) @@ -242,7 +276,9 @@ TEST(CASSentinelProbe, NativeClassifiesNoSuchBucketAsContainerAbsent) storage->throwOnObjectAccess(Aws::S3::S3Errors::NO_SUCH_BUCKET); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::ContainerAbsent); + auto requests = makeRequests(backend); + auto op = requests.admit(); + EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::ContainerAbsent); } TEST(CASSentinelProbe, NativeClassifiesAccessDeniedAsAccessDenied) @@ -251,7 +287,9 @@ TEST(CASSentinelProbe, NativeClassifiesAccessDeniedAsAccessDenied) storage->throwOnObjectAccess(Aws::S3::S3Errors::ACCESS_DENIED); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::AccessDenied); + auto requests = makeRequests(backend); + auto op = requests.admit(); + EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::AccessDenied); } TEST(CASSentinelProbe, NativeClassifiesUnmodeledErrorAsIndeterminate) @@ -260,7 +298,12 @@ TEST(CASSentinelProbe, NativeClassifiesUnmodeledErrorAsIndeterminate) storage->throwOnObjectAccess(Aws::S3::S3Errors::SERVICE_UNAVAILABLE); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - EXPECT_EQ(probeSentinel(backend, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::Indeterminate); + /// Every attempt classifies Indeterminate here too, so the reissue loop runs its whole policy + /// window; an injected clock keeps that instantaneous instead of real time. + DB::Cas::tests::FakeClock clock; + auto requests = makeRequests(backend, &clock); + auto op = requests.admit(); + EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::Indeterminate); } /// Production wiring (`Pool::open`) ALWAYS wraps the real backend in `InstrumentedBackend` before @@ -278,7 +321,9 @@ TEST(CASSentinelProbe, InstrumentedBackendForwardsToInnerClassification) auto inner = std::make_shared(storage, ObjectStorageBackend::Mode::Native); InstrumentedBackend instrumented(inner); - EXPECT_EQ(probeSentinel(instrumented, nativeKeyUnder(storage, "some/key")).outcome, ProbeOutcome::ContainerAbsent); + auto requests = makeRequests(instrumented); + auto op = requests.admit(); + EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::ContainerAbsent); } #endif diff --git a/src/Disks/tests/gtest_cas_slot_occupy.cpp b/src/Disks/tests/gtest_cas_slot_occupy.cpp index d248509c2d31..901b155df99c 100644 --- a/src/Disks/tests/gtest_cas_slot_occupy.cpp +++ b/src/Disks/tests/gtest_cas_slot_occupy.cpp @@ -2,16 +2,16 @@ #include "config.h" -#include #include +#include #include #include #include +#include + using namespace DB::Cas; using DB::Cas::tests::CountingBackend; -using DB::Cas::tests::ChunkFaultBackend; -using DB::Cas::tests::LandedButAckLostOnceBackend; namespace DB::ErrorCodes { @@ -19,244 +19,254 @@ namespace DB::ErrorCodes } /// ================================================================================================ -/// Task 2 (2026-07-28 CAS ref-chain Stage A streams, spec INV-2): CasRequestController::slotOccupy -- -/// the dedicated RAW slot-occupy primitive every seal writer and wedge retry uses. ONE conditional -/// create; on conflict, ONE raw exact GET of the occupant -- NEVER retries internally, NEVER lists, -/// and NEVER composes putIfAbsentControlled (which retries the same (key, bytes) internally) or -/// resolveByExactGet (which compares against an expected body and throws CORRUPTED_DATA on a -/// mismatch) [codex finding 3]. Adjudicating whether an Occupied occupant is "mine" is entirely the -/// CALLER's job (Task 4/6, the CaCasMountCore `mine` contract) -- these tests only pin the -/// primitive's own three-way outcome and its op-count contract (Created=1, Occupied=2, -/// Unresolved<=2 backend ops). +/// The ref lane's slot occupy: ONE conditional create of a write-once ref-log key, `Retry::once()`, +/// on an operation the caller resumed under the generation its transaction was admitted at. It is +/// what every epoch-seal writer and every wedge retry issues, so these tests pin the shape those two +/// callers depend on -- the four alternatives and the request count behind each -- rather than the +/// engine's general write contract, which `gtest_cas_requests.cpp` owns. +/// +/// Adjudicating whether a conflicting occupant is "mine" is entirely the CALLER's job (the +/// `CaCasMountCore` `mine` contract: byte equality, never a shape or generation match); nothing here +/// compares bytes for meaning. /// ================================================================================================ namespace { /// Deletes the key the INSTANT its own conditional create conflicts, modelling "the occupant that -/// caused the conflict vanished before slotOccupy's single resolve GET" -- a race a real backend can -/// produce (e.g. GC reclaiming an already-condemned object) that the primitive must survive by -/// reporting Unresolved, NEVER a fabricated Created. +/// caused the conflict vanished before the settling read" -- a race a real backend can produce (e.g. +/// GC reclaiming an already-condemned object) that the call must survive by reporting what it saw, +/// never a fabricated commit. class VanishOnConflictBackend : public CountingBackend { public: - using CountingBackend::putIfAbsent; - - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { - PutResult result = CountingBackend::putIfAbsent(key, bytes, meta); - if (result.outcome == PutOutcome::PreconditionFailed) + auto result = CountingBackend::write(key, bytes, expected_value, access); + if (!result.has_value()) { - const HeadResult h = head(key); - if (h.exists) - deleteExact(key, h.token); + if (const auto meta = CountingBackend::head(key, access)) + CountingBackend::remove(key, meta->value, access); } return result; } }; -/// Throws a deterministic LOCAL failure (BAD_ARGUMENTS, in isDeterministicLocalFailure's set) on the -/// first putIfAbsent -- models a backend-level programming bug, distinct from ChunkFaultBackend's -/// Mode::Definite below, which is a whitelisted SYNCHRONOUS REJECTION -/// (classifyConditionalWriteResult's DefiniteFailure). slotOccupy must rethrow both, unchanged, never -/// folding either into Unresolved (SlotOccupyResult::Kind has no DefiniteFailure member to carry it). +/// Throws a deterministic LOCAL failure (`BAD_ARGUMENTS`, in `isDeterministicLocalFailure`'s set) on +/// the first write -- a backend-level programming bug, distinct from a whitelisted synchronous +/// rejection, which the store gives as an answer and the engine reports as `Refused`. class LocalFailureOnceBackend : public CountingBackend { public: - using CountingBackend::putIfAbsent; bool fail_once = true; - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override { if (fail_once) { fail_once = false; throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, "scripted deterministic local failure"); } - return CountingBackend::putIfAbsent(key, bytes, meta); + return CountingBackend::write(key, bytes, expected_value, access); } }; -/// (`LandedButAckLostOnceBackend` -- "the write LANDS, then the ack is lost" -- was lifted into -/// `cas_test_helpers.h` for Task 4, whose wedge-adoption tests need the identical seam through a whole -/// Pool. Its `key_substr` defaults to empty, which is exactly this file's original behaviour: fault the -/// first `putIfAbsent` of any key.) -/// Delegates the FIRST putIfAbsent for a key to CountingBackend -- so the write actually LANDS -- and -/// only THEN throws an ambiguous exception, modelling "our own PUT committed but its response was lost" -/// (the Task-4 adoption input: plan's "Occupied + bytes == wedge.bytes -> an earlier attempt landed -> -/// adopt"). Distinct from InMemoryBackend::injectAmbiguousPutIfAbsent, which never touches the store at -/// all -- that hook models an attempt that did NOT land; this one models an attempt that DID. -/// One-shot per backend instance: review finding I2 asked specifically for a ~10-line local backend rather than -/// reusing ChunkFaultBackend::Mode::LandedThenLost, which also arms a one-shot lost-GET fault that would -/// obscure whether slotOccupy's OWN immediate resolve (not just a later caller's retry) is correct too. +/// Withdraws the caller's liveness the instant a conditional create conflicts, so the settling read +/// is the first request the operation is no longer admitted for. +class WithdrawAdmissionOnConflictBackend : public CountingBackend +{ +public: + bool live = true; + + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + auto result = CountingBackend::write(key, bytes, expected_value, access); + if (!result.has_value()) + live = false; + return result; + } +}; +/// The occupant a conflict names, or null when the result is not a conflict that observed one. +const Object * conflictObject(const WriteResult & result) +{ + const auto * conflict = std::get_if(&result); + return conflict ? std::get_if(&conflict->seen) : nullptr; } -/// ---- Step 1 required scenarios ---- +} -TEST(CASSlotOccupy, AbsentKeyCreatesWithOneOp) +TEST(CASSlotOccupy, AbsentKeyCommitsWithOneRequest) { auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); - const auto result = controller.slotOccupy("k", "payload", [] { return true; }); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Created); - EXPECT_TRUE(result.occupant_bytes.empty()); - EXPECT_TRUE(result.occupant_token.empty()) << "occupant_token is Occupied-only; must stay default on Created"; - EXPECT_EQ(result.unresolved_reason, CasUnresolvedReason::NotUnresolved); + const WriteResult result = op.create("k", "payload", Retry::once()); + const auto * committed = std::get_if(&result); + ASSERT_TRUE(committed != nullptr); + EXPECT_EQ(committed->attempts_sent, 1u); + EXPECT_FALSE(committed->resolved_by_read) << "an unambiguous create is proven by its own response"; - EXPECT_EQ(backend->putCount("k"), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 0u); EXPECT_EQ(backend->headCount("k"), 0u); - const auto landed = backend->get("k"); + CasOperation reader = requests.admit(); + const auto landed = reader.read("k", Retry::once()); ASSERT_TRUE(landed.has_value()); EXPECT_EQ(landed->bytes, "payload"); } -TEST(CASSlotOccupy, PreExistingKeyOccupiedWithExactBytesAndTokenTwoOps) +TEST(CASSlotOccupy, PreExistingKeyConflictsWithExactBytesAndIncarnationInTwoRequests) { auto backend = std::make_shared(); - const PutResult seeded = backend->putIfAbsent("k", "occupant-bytes"); - ASSERT_EQ(seeded.outcome, PutOutcome::Done); + CasRequests requests(backend, Fence::open()); + + CasOperation seeder = requests.admit(); + const WriteResult seeded = seeder.create("k", "occupant-bytes", Retry::once()); + const auto * seeded_committed = std::get_if(&seeded); + ASSERT_TRUE(seeded_committed != nullptr); + const Incarnation seeded_incarnation = seeded_committed->incarnation; backend->resetCounts(); - CasRequestController controller(backend, CasRequestBudget{}); - const auto result = controller.slotOccupy("k", "my-attempt-bytes", [] { return true; }); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Occupied); - EXPECT_EQ(result.occupant_bytes, "occupant-bytes"); - EXPECT_EQ(result.occupant_token, seeded.token); + CasOperation op = requests.admit(); + const WriteResult result = op.create("k", "my-attempt-bytes", Retry::once()); + const Object * occupant = conflictObject(result); + ASSERT_TRUE(occupant != nullptr) << "the settling read must have named the occupant"; + EXPECT_EQ(occupant->bytes, "occupant-bytes"); + EXPECT_EQ(occupant->incarnation, seeded_incarnation); - EXPECT_EQ(backend->putCount("k"), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 1u); - EXPECT_EQ(backend->headCount("k"), 0u) << "exactly PUT+GET -- a HEAD-then-GET implementation must fail this"; + EXPECT_EQ(backend->headCount("k"), 0u) + << "exactly one write and one settling read -- a HEAD-then-read implementation must fail this"; - /// A conflict never overwrites or appends -- the pre-existing object is untouched. - const auto current = backend->get("k"); + CasOperation reader = requests.admit(); + const auto current = reader.read("k", Retry::once()); ASSERT_TRUE(current.has_value()); - EXPECT_EQ(current->bytes, "occupant-bytes"); + EXPECT_EQ(current->bytes, "occupant-bytes") << "a conflict never overwrites or appends"; } -TEST(CASSlotOccupy, InjectedAmbiguousPutResolvesUnresolvedWhenGetFindsNothing) +TEST(CASSlotOccupy, AmbiguousWriteThatLandedNothingGivesUpHavingSentOne) { auto backend = std::make_shared(); - backend->injectAmbiguousPutIfAbsent("k"); - - CasRequestController controller(backend, CasRequestBudget{}); - const auto result = controller.slotOccupy("k", "payload", [] { return true; }); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Unresolved); - /// An attempt WAS sent (the ambiguous PUT itself) -- this is never the pre-attempt NoAttemptSent - /// case. Of the existing CasUnresolvedReason values, AttemptsExhausted is the one documented as - /// "the genuine case the 'retry budget exhausted' wording describes" -- exactly this call's single - /// (and only) attempt having nothing left to give once its resolve GET came up empty. - EXPECT_EQ(result.unresolved_reason, CasUnresolvedReason::AttemptsExhausted); - EXPECT_FALSE(unresolvedProvesNothingWasSent(result.unresolved_reason)); - - EXPECT_EQ(backend->putCount("k"), 1u); + backend->injectAmbiguousWrite("k"); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + + const WriteResult result = op.create("k", "payload", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any) << "the ambiguous attempt itself was sent -- this is never the pre-attempt case"; + EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); + + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 1u); - EXPECT_FALSE(backend->head("k").exists) << "the injected fault must not actually create anything"; + CasOperation reader = requests.admit(); + EXPECT_FALSE(reader.head("k", Retry::once()).has_value()) << "the injected fault must not create anything"; } -TEST(CASSlotOccupy, ConflictThenVanishResolvesUnresolved) +TEST(CASSlotOccupy, ConflictThenVanishGivesUpRatherThanFabricatingACommit) { auto backend = std::make_shared(); - const auto seeded = backend->putIfAbsent("k", "occupant-bytes"); - ASSERT_EQ(seeded.outcome, PutOutcome::Done); + CasRequests requests(backend, Fence::open()); + + CasOperation seeder = requests.admit(); + ASSERT_TRUE(std::holds_alternative(seeder.create("k", "occupant-bytes", Retry::once()))); backend->resetCounts(); - CasRequestController controller(backend, CasRequestBudget{}); - const auto result = controller.slotOccupy("k", "my-attempt-bytes", [] { return true; }); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Unresolved); - EXPECT_EQ(result.unresolved_reason, CasUnresolvedReason::AttemptsExhausted); + CasOperation op = requests.admit(); + const WriteResult result = op.create("k", "my-attempt-bytes", Retry::once()); + /// Nothing of ours was ever ambiguous, so an absence settles the call as a conflict against an + /// occupant that is no longer there -- never as a commit. + const auto * conflict = std::get_if(&result); + ASSERT_TRUE(conflict != nullptr); + EXPECT_TRUE(std::holds_alternative(conflict->seen)); - EXPECT_EQ(backend->putCount("k"), 1u); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 1u); - /// No headCount assertion here (unlike the sibling Occupied test above): VanishOnConflictBackend's - /// OWN fixture issues a HEAD internally (to fetch the token before deleteExact) -- that HEAD belongs - /// to the test's vanish mechanism, not to slotOccupy, so asserting headCount==0 would be wrong, not - /// stronger. slotOccupy itself never calls head(); only put+get are its own ops. - EXPECT_FALSE(backend->head("k").exists) << "the occupant vanished between the conflict and the resolve GET"; } -TEST(CASSlotOccupy, FenceFlipMidCallRefusesPreAttemptNeverLiesCreated) +TEST(CASSlotOccupy, LivenessRefusalBeforeTheAttemptSendsNothing) { auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit([] { return false; }); - const auto result = controller.slotOccupy("k", "payload", [] { return false; }); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Unresolved); - /// The pre-attempt reason: fence_ok refused before anything was sent to the backend. - EXPECT_EQ(result.unresolved_reason, CasUnresolvedReason::NoAttemptSent); - EXPECT_TRUE(unresolvedProvesNothingWasSent(result.unresolved_reason)); + const WriteResult result = op.create("k", "payload", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any) << "the whole point: the key is provably unwritten"; - EXPECT_EQ(backend->putTotal(), 0u); + EXPECT_EQ(backend->writeTotal(), 0u); EXPECT_EQ(backend->getTotal(), 0u); - EXPECT_FALSE(backend->head("k").exists) << "never a lie of Created -- the key must be untouched"; + CasRequests open_requests(backend, Fence::open()); + CasOperation reader = open_requests.admit(); + EXPECT_FALSE(reader.head("k", Retry::once()).has_value()) << "never a lie of committed -- the key must be untouched"; } -/// ---- Bonus coverage: the deadline pre-gate (the OTHER half of "fence/deadline-gated"), and the two -/// rethrow paths this primitive shares with its sibling controlled ops. ---- - -/// The deadline gate is the SAME pre-attempt refusal as the fence gate above -- a fake clock proves it -/// fires from elapsed time alone, with a fence that always says yes. -TEST(CASSlotOccupy, OperationDeadlineExhaustedRefusesPreAttempt) +/// The deadline is the OTHER pre-attempt refusal: a fake clock proves it fires from elapsed time +/// alone, under a fence that always says yes. +TEST(CASSlotOccupy, ExhaustedPolicyDeadlineRefusesBeforeTheAttempt) { auto backend = std::make_shared(); uint64_t clock = 0; - auto now_ms = [&clock]() -> uint64_t { const uint64_t t = clock; clock += 1000; return t; }; - - CasRequestBudget budget; - budget.attempt_timeout_ms = 50; - budget.operation_deadline_ms = 500; /// entry now_ms()==0 -> deadline_ms=500; the gate's OWN - /// now_ms() call then returns 1000 -> 1000+50 > 500 -> refuse - CasRequestController controller(backend, budget, now_ms); - - const auto result = controller.slotOccupy("k", "payload", [] { return true; }); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Unresolved); - EXPECT_EQ(result.unresolved_reason, CasUnresolvedReason::NoAttemptSent); - EXPECT_EQ(backend->putTotal(), 0u); - EXPECT_EQ(backend->getTotal(), 0u) << "zero ops total -- the deadline gate must refuse before any I/O, same as the fence gate"; + CasRequests requests(backend, Fence::open(), + [&clock]() -> uint64_t { const uint64_t t = clock; clock += 1000; return t; }); + requests.setAttemptReservationForTest(50); + CasOperation op = requests.admit(); + + /// Entry `now_ms()` is 0, so the bound is 500; the loop's own `now_ms()` then reads 1000. + const WriteResult result = op.create("k", "payload", Retry::within(500)); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(backend->writeTotal(), 0u); + EXPECT_EQ(backend->getTotal(), 0u) + << "zero requests -- the deadline refuses before any I/O, exactly as the liveness gate does"; } -/// A whitelisted synchronous rejection (classifyConditionalWriteResult's DefiniteFailure) PROVES the -/// request was never applied -- slotOccupy must surface it unchanged rather than resolving or folding -/// it into Unresolved. Guarded to USE_AWS_S3 builds ONLY [review M6]: DefiniteFailure classification is -/// structurally unreachable without it (classifyConditionalWriteResult's whitelist is entirely inside -/// its own `#if USE_AWS_S3`), so on a no-S3 build ChunkFaultBackend::Mode::Definite instead throws a -/// plain CORRUPTED_DATA DB::Exception -- which is in isDeterministicLocalFailure's set, meaning this -/// test would silently exercise the SAME slotOccupy branch as DeterministicLocalFailurePropagatesWithoutResolve -/// below rather than the DefiniteFailure branch it claims to cover. Better a visibly-absent test on that -/// config than a passing one that isn't testing what its name says. +/// A whitelisted synchronous rejection PROVES the request was never applied, so the engine reports it +/// as a value and settles nothing by reading. Guarded to `USE_AWS_S3` builds ONLY: the classification +/// lives entirely inside `isDefinitelyRefusedWrite`'s own `#if USE_AWS_S3`, so without it this would +/// silently exercise the ambiguity path instead of the refusal it names. #if USE_AWS_S3 -TEST(CASSlotOccupy, DefiniteFailurePropagatesWithoutResolve) +TEST(CASSlotOccupy, DefiniteStoreRefusalIsAValueAndSettlesNothing) { - auto backend = std::make_shared(); - backend->fault_substr = "k"; - backend->mode = ChunkFaultBackend::Mode::Definite; - backend->fault_count = 1; - - CasRequestController controller(backend, CasRequestBudget{}); - EXPECT_THROW(controller.slotOccupy("k", "payload", [] { return true; }), DB::Exception); - /// ChunkFaultBackend's fault check throws BEFORE delegating to CountingBackend::putIfAbsent, so - /// putCount stays 0 on this path -- fault_count reaching 0 is this backend's own proof the (one) - /// attempt was made and consumed the fault. - EXPECT_EQ(backend->fault_count, 0); - EXPECT_EQ(backend->getCount("k"), 0u) << "a whitelisted definite rejection must never trigger a resolve GET"; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", std::make_exception_ptr( + DB::S3Exception("simulated malformed request", Aws::S3::S3Errors::UNKNOWN, "MalformedXML"))); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + + const WriteResult result = op.create("k", "payload", Retry::once()); + EXPECT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->getCount("k"), 0u) << "a proven refusal must never trigger a settling read"; } #endif -/// A deterministic LOCAL failure (isDeterministicLocalFailure's set) is the OTHER rethrow path -- -/// distinct from DefiniteFailure above, and checked first in the implementation, so it needs its own -/// backend-level fault to prove both branches are wired, not just one masking the other. -TEST(CASSlotOccupy, DeterministicLocalFailurePropagatesWithoutResolve) +/// A deterministic LOCAL failure is the one thing the write surface still reports by exception: +/// reissuing only replays it, and folding it into an outcome would bury the root cause. +TEST(CASSlotOccupy, DeterministicLocalFailurePropagatesWithoutSettling) { auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); bool threw = false; try { - controller.slotOccupy("k", "payload", [] { return true; }); + op.create("k", "payload", Retry::once()); } catch (const DB::Exception & e) { @@ -264,95 +274,83 @@ TEST(CASSlotOccupy, DeterministicLocalFailurePropagatesWithoutResolve) EXPECT_EQ(e.code(), DB::ErrorCodes::BAD_ARGUMENTS) << "the ORIGINAL exception must propagate unchanged"; } EXPECT_TRUE(threw) << "a deterministic local failure must propagate, never return an outcome"; - /// LocalFailureOnceBackend throws BEFORE delegating to CountingBackend::putIfAbsent (same shape as - /// ChunkFaultBackend above), so putCount stays 0 here too -- fail_once flipping is this backend's - /// own proof the attempt was made. EXPECT_FALSE(backend->fail_once); EXPECT_EQ(backend->getCount("k"), 0u); } -/// ---- Fix round 1 (review findings I1, I2): the two gaps the reviewer required landed before Task 4 -/// consumes this primitive. Both guard the design decisions the review approved -- see -/// task-2-review.md concern (a) and finding I2's Task-4-adoption note. ---- - -/// I1: pins the single-`fence_ok`-call `Created` design (concern (a)) so a future contributor cannot -/// silently "fix the inconsistency" by re-adding the sibling ops' post-write fence recheck. That change -/// would break Task 4's old-generation-retry semantics (resolveWedgeOnce deliberately calls slotOccupy under -/// the wedge's ORIGINAL admitted_fence_generation, and relies on ITS OWN post-I/O checkFenceOrThrow, -/// not a second internal check here, to decide whether the result is still relevant). A counting -/// fence_ok that only answers true on its FIRST call: if slotOccupy ever called it again after the -/// write landed, this test would see Unresolved instead of Created, OR (if the outcome happened to -/// still read Created some other way) the call-count assertion below would catch the extra invocation -/// either way. -TEST(CASSlotOccupy, CreatedNeverRechecksFenceAfterTheWrite) +/// A commit whose admission was withdrawn while it was in flight is reported as unresolved, never as +/// committed: the object may well exist, and the caller has to resolve the key rather than act on a +/// claim made under an incarnation it no longer holds. This is the OPPOSITE of the retired +/// slot-occupy primitive's single-pre-attempt-check contract, and it is what lets the ref lane's +/// wedge stay wedged instead of installing against a fence it has already lost. +TEST(CASSlotOccupy, AdmissionLostAfterTheWriteIsNeverReportedCommitted) { auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}); - - int fence_calls = 0; - const auto fence_ok = [&fence_calls] - { - ++fence_calls; - return fence_calls == 1; - }; - - const auto result = controller.slotOccupy("k", "payload", fence_ok); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Created); - EXPECT_EQ(fence_calls, 1) << "slotOccupy must call fence_ok() exactly ONCE (pre-attempt only) -- " - "a post-write recheck would falsely report Unresolved here (fence_calls's " - "SECOND answer is false) and would break Task 4's old-generation-retry design"; + bool live = true; + backend->onWriteCommitted("k", [&live] { live = false; }); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit([&live] { return live; }); + + const WriteResult result = op.create("k", "payload", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any) << "the write was sent, and it landed -- the caller must resolve the key"; + + /// The object IS durable; only the claim about it is refused. + CasRequests open_requests(backend, Fence::open()); + CasOperation reader = open_requests.admit(); + const auto landed = reader.read("k", Retry::once()); + ASSERT_TRUE(landed.has_value()); + EXPECT_EQ(landed->bytes, "payload"); } -/// A conflict needs a second backend request to resolve its occupant. Admission may disappear while -/// the conditional create is in flight; in that case the resolver must fail closed before starting -/// the `GET`, while preserving the one-check `Created` contract above. -TEST(CASSlotOccupy, AdmissionLostAfterConflictPreventsTheResolveGet) +/// A conflict needs a second request to name its occupant. Admission may disappear while the +/// conditional create is in flight; the settling read must then not start at all. +TEST(CASSlotOccupy, AdmissionLostAfterAConflictPreventsTheSettlingRead) { - auto backend = std::make_shared(); - ASSERT_EQ(backend->putIfAbsent("k", "existing").outcome, PutOutcome::Done); - CasRequestController controller(backend, CasRequestBudget{}); + auto backend = std::make_shared(); + CasRequests seed_requests(backend, Fence::open()); + CasOperation seeder = seed_requests.admit(); + ASSERT_TRUE(std::holds_alternative(seeder.create("k", "existing", Retry::once()))); + backend->resetCounts(); - int admission_checks = 0; - const auto admitted = [&admission_checks] - { - ++admission_checks; - return admission_checks == 1; - }; - - const auto result = controller.slotOccupy("k", "attempt", admitted); - EXPECT_EQ(result.kind, SlotOccupyResult::Kind::Unresolved); - EXPECT_EQ(admission_checks, 2); - EXPECT_EQ(backend->putCount("k"), 2u); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit([&backend] { return backend->live; }); + const WriteResult result = op.create("k", "attempt", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_TRUE(gave_up != nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 0u) - << "slotOccupy started its ambiguity-resolution GET after admission was withdrawn"; + << "the settling read started after admission was withdrawn"; } -/// I2: proves Occupied is reachable for an occupant that is OUR OWN earlier ambiguous write, not only -/// for a foreign pre-seeded one (PreExistingKeyOccupiedWithExactBytesAndTokenTwoOps above always seeds -/// via a plain, unambiguous putIfAbsent). This is the exact input shape Task 4's resolveWedgeOnce -/// adjudicates: "Occupied + bytes == wedge.bytes -> an earlier attempt landed -> adopt" (plan :329). -TEST(CASSlotOccupy, OwnLandedAmbiguousWriteObservedAsOccupiedOnRetry) +/// The wedge-adoption input shape: an earlier ambiguous attempt of the SAME key and bytes landed, and +/// a later flush issues a fresh create for it. The first call settles it by reading its own bytes; the +/// second sees them as an ordinary occupant, which is what the lane's `mine` adjudication consumes. +TEST(CASSlotOccupy, OwnLandedAmbiguousWriteIsObservedOnTheNextAttempt) { - auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}); - - /// Call 1 -- the original attempt: the PUT's own response is lost, but the write DID commit, and - /// THIS call's own resolve GET (unfaulted) observes it immediately -- Occupied with OUR bytes, - /// proving the same-call resolve path works for a landed ambiguous write, not only a foreign one. - const auto first = controller.slotOccupy("k", "my-bytes", [] { return true; }); - EXPECT_EQ(first.kind, SlotOccupyResult::Kind::Occupied); - EXPECT_EQ(first.occupant_bytes, "my-bytes"); - EXPECT_EQ(backend->putCount("k"), 1u); + auto backend = std::make_shared(); + backend->injectAmbiguousLandedWrite("k"); + CasRequests requests(backend, Fence::open()); + + CasOperation first_op = requests.admit(); + const WriteResult first = first_op.create("k", "my-bytes", Retry::once()); + const auto * committed = std::get_if(&first); + ASSERT_TRUE(committed != nullptr) << "the write landed; the settling read proves it"; + EXPECT_TRUE(committed->resolved_by_read); + const Incarnation landed_incarnation = committed->incarnation; + EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 1u); - /// Call 2 -- Task 4's resolveWedgeOnce pattern: a LATER caller's flush resolving the SAME logical - /// attempt via a FRESH slotOccupy call. The fault is already consumed (one-shot), so this PUT - /// conflicts cleanly (PreconditionFailed) and the resolve GET observes OUR OWN earlier bytes again -- - /// the exact adoption input Task 4 is built on, and the SAME incarnation both calls saw. - const auto second = controller.slotOccupy("k", "my-bytes", [] { return true; }); - EXPECT_EQ(second.kind, SlotOccupyResult::Kind::Occupied); - EXPECT_EQ(second.occupant_bytes, "my-bytes"); - EXPECT_EQ(second.occupant_token, first.occupant_token) << "both calls must observe the SAME landed incarnation"; - EXPECT_EQ(backend->putCount("k"), 2u); + CasOperation second_op = requests.admit(); + const WriteResult second = second_op.create("k", "my-bytes", Retry::once()); + const Object * occupant = conflictObject(second); + ASSERT_TRUE(occupant != nullptr); + EXPECT_EQ(occupant->bytes, "my-bytes"); + EXPECT_EQ(occupant->incarnation, landed_incarnation) << "both calls must observe the SAME landed incarnation"; + EXPECT_EQ(backend->writeTotal(), 2u); EXPECT_EQ(backend->getCount("k"), 2u); } diff --git a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp index 2f15a9d9df95..fefed9949677 100644 --- a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp +++ b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp @@ -276,7 +276,9 @@ TEST(CASTruncateReclaim, DropNamespaceLeavesSharedBlobDebrisForPerpetualSweep) << "an emptied pool must drain instead of standing still; the sweep owned these blobs and " "reclaimed them within " << rounds << " GC rounds"; EXPECT_EQ(after.reachable, 0u); - EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(s->backend(), s->layout(), ns)) + DB::Cas::CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(s->backend()); + DB::Cas::CasOperation catalog_op = catalog_requests.admit(); + EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(catalog_op, s->layout(), ns)) << "physical debris must not keep the logical namespace life cataloged"; } } diff --git a/src/Disks/tests/gtest_cas_txn_apply_ledger.cpp b/src/Disks/tests/gtest_cas_txn_apply_ledger.cpp index 80d262854555..2cfe3fd2a8b5 100644 --- a/src/Disks/tests/gtest_cas_txn_apply_ledger.cpp +++ b/src/Disks/tests/gtest_cas_txn_apply_ledger.cpp @@ -4,6 +4,7 @@ #include #include #include +#include using namespace DB::Cas; @@ -94,6 +95,7 @@ TEST(CASTxnApplyLedger, OnlyTheTransactionWhoseDeltasVanishedIsReported) TEST(CASTxnApplyLedger, ReducerMarksTheOrdinalOfEveryDeltaItConsumes) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest operation(backend); Layout layout{"pool"}; TxnApplyLedger ledger; @@ -107,7 +109,7 @@ TEST(CASTxnApplyLedger, ReducerMarksTheOrdinalOfEveryDeltaItConsumes) {bh(2), s(1), /*remove*/false, routed}, }; std::vector runs; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, + foldDeltasIntoGeneration(*operation, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, /*shard*/0, deltas, runs, /*current_round*/0, /*condemn_round*/0, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, /*out_retired*/nullptr, @@ -135,6 +137,7 @@ TEST(CASTxnApplyLedger, ReducerMarksTheOrdinalOfEveryDeltaItConsumes) TEST(CASTxnApplyLedger, ReducerMarksAnUnmatchedRemovalDelta) { InMemoryBackend backend; + DB::Cas::tests::OperationForTest operation(backend); Layout layout{"pool"}; TxnApplyLedger ledger; @@ -146,7 +149,7 @@ TEST(CASTxnApplyLedger, ReducerMarksAnUnmatchedRemovalDelta) std::vector deltas{{bh(1), s(1), /*remove*/true, removal}}; std::vector runs; RetiredMergeResult merged; - foldDeltasIntoGeneration(backend, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, + foldDeltasIntoGeneration(*operation, layout, /*prior_runs*/{}, /*new_generation*/1, /*attempt*/0, /*shard*/0, deltas, runs, /*current_round*/0, /*condemn_round*/0, /*head_blob*/{}, /*peek_head*/{}, /*confirm_condemned_marker*/{}, &merged, diff --git a/src/Disks/tests/gtest_cas_upload_detached.cpp b/src/Disks/tests/gtest_cas_upload_detached.cpp index 0f158475f269..999bab3f400e 100644 --- a/src/Disks/tests/gtest_cas_upload_detached.cpp +++ b/src/Disks/tests/gtest_cas_upload_detached.cpp @@ -106,7 +106,7 @@ std::optional metaStateAt(InMemoryBackend & b, const Layout & layout, class ProtocolRecordingBackend final : public InMemoryBackend { public: - /// Unhide the primitive overload that the legacy override below would otherwise hide. + /// Unhide the legacy overload that the primitive override below would otherwise hide. using InMemoryBackend::head; void watch(String blob_key_, String meta_key_) { @@ -119,27 +119,27 @@ class ProtocolRecordingBackend final : public InMemoryBackend meta_gets_before_first_publish.reset(); } - HeadResult head(const String & key) override + std::optional head(const String & key, TransportAccess & access) override { if (key == blob_key) { ++blob_heads; operations.emplace_back("head"); } - return InMemoryBackend::head(key); + return InMemoryBackend::head(key, access); } - std::optional get(const String & key, Range range) override + std::optional read(const String & key, TransportAccess & access) override { if (key == meta_key) { ++meta_gets; operations.emplace_back("meta-get"); } - return InMemoryBackend::get(key, range); + return InMemoryBackend::read(key, access); } - void publishBlob(const BlobPublishRequest & request) override + void publish(const BlobPublishRequest & request, TransportAccess & access) override { if (request.destination_key == blob_key) { @@ -148,7 +148,7 @@ class ProtocolRecordingBackend final : public InMemoryBackend if (!meta_gets_before_first_publish) meta_gets_before_first_publish = meta_gets; } - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); } String blob_key; diff --git a/src/Disks/tests/gtest_cas_upload_fanout.cpp b/src/Disks/tests/gtest_cas_upload_fanout.cpp index 1371d8a70fc7..d83bec5bdf96 100644 --- a/src/Disks/tests/gtest_cas_upload_fanout.cpp +++ b/src/Disks/tests/gtest_cas_upload_fanout.cpp @@ -224,7 +224,7 @@ struct ConcurrencyProbe class RejectFirstStagedCopyBackend final : public InMemoryBackend { public: - void publishBlob(const BlobPublishRequest & request) override + void publish(const BlobPublishRequest & request, TransportAccess & access) override { if (std::holds_alternative(request.publication)) { @@ -239,7 +239,7 @@ class RejectFirstStagedCopyBackend final : public InMemoryBackend { ++streaming_publications; } - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); } bool reject_copy = true; diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index 93385f38177a..a375389d3c7b 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -1,12 +1,17 @@ #include #include +#include +#include #include +#include #include #include #include #include +#include #include +#include using namespace DB::Cas; @@ -57,7 +62,7 @@ TEST(CASWireVocab, EnumTablesPinTheCurrentWords) TEST(CASWireVocab, ClosedSetsRoundTripEveryEnumeratorExhaustively) { for (const auto t : magic_enum::enum_values()) - EXPECT_EQ(tokenTypeFromWord(tokenTypeToWord(t), "t"), t); + EXPECT_EQ(kTokenTypeWords.fromWord(kTokenTypeWords.toWord(t, "t"), "t"), t); for (const auto k : magic_enum::enum_values()) EXPECT_EQ(objectKindFromWord(objectKindToWord(k), "k"), k); for (const auto a : magic_enum::enum_values()) @@ -69,11 +74,11 @@ TEST(CASWireVocab, ClosedSetsRoundTripEveryEnumeratorExhaustively) TEST(CASWireVocab, EnumWordsRoundTrip) { for (TokenType t : {TokenType::ETag, TokenType::Generation, TokenType::Emulated}) - EXPECT_EQ(tokenTypeFromWord(tokenTypeToWord(t), "t"), t); + EXPECT_EQ(kTokenTypeWords.fromWord(kTokenTypeWords.toWord(t, "t"), "t"), t); for (BlobHashAlgo a : {BlobHashAlgo::CityHash128, BlobHashAlgo::XXH3_128, BlobHashAlgo::Sha256}) EXPECT_EQ(blobHashAlgoFromWord(blobHashAlgoName(a), "a"), a); EXPECT_EQ(objectKindFromWord(objectKindToWord(ObjectKind::Blob), "k"), ObjectKind::Blob); - expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { tokenTypeFromWord("nope", "t"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectWordFromString("nope", "t"); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { blobHashAlgoFromWord("nope", "a"); }); } @@ -81,7 +86,7 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) { CasJsonWriter out; bool first = true; - writeTokenFields(out, first, Token{"etag-abc\"x", TokenType::ETag}); + writeTokenFields(out, first, PersistedIncarnation{"etag", "etag-abc\"x"}); const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}; writeBlobRefFields(out, first, ref); closeObject(out, first); @@ -95,16 +100,16 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) String tv; String ha; String h; - TokenType tt{}; + String tt; while (r.nextKey(key)) { - if (key == "token_type") tt = tokenTypeFromWord(r.readString(), "t"); + if (key == "token_type") tt = String(dialectWordFromString(r.readString(), "t")); else if (key == "token") tv = r.readString(); else if (key == "algo") ha = r.readString(); else if (key == "digest") h = r.readString(); else r.skipUnknown(key); } - EXPECT_EQ(tt, TokenType::ETag); + EXPECT_EQ(tt, "etag"); EXPECT_EQ(tv, "etag-abc\"x"); const BlobRef back{blobHashAlgoFromWord(ha, "a"), codecFor(blobHashAlgoFromWord(ha, "a")).fromHex(h)}; EXPECT_EQ(back, ref); @@ -224,7 +229,9 @@ TEST(CASWireVocab, TokenFieldsBuildsInAnyKeyOrderAndRequiresBothFields) continue; r.skipUnknown(key); } - EXPECT_EQ(fields.build("t"), (Token{"abc", TokenType::ETag})); + const PersistedIncarnation built = fields.build("t"); + EXPECT_EQ(built.dialect, "etag"); + EXPECT_EQ(built.value, "abc"); TokenFields only_type; only_type.type_word = "etag"; @@ -256,3 +263,80 @@ TEST(CASWireVocab, OldManifestEpochKeyDoesNotAliasTheSemanticKey) EXPECT_EQ(e.message(), "CAS RefTableSnapshot: committed manifest_ref missing epoch/build/ord"); } } + +/// A `PersistedIncarnation` survives every encoding a durable CAS record uses for one, and the type +/// system refuses the reverse direction: a persisted value must never be trusted to mint a live +/// `Incarnation`, which only an admitted request may produce. +static_assert(!std::is_constructible_v); + +TEST(CASPersistedIncarnation, RoundTripsThroughEveryFormatAndNeverBecomesAnIncarnation) +{ + const PersistedIncarnation recorded{"generation", R"(17"3)"}; /// a quote the JSON encodings must escape + + /// 1. The shared `token_type`/`token` JSON pair. + { + CasJsonWriter out; + bool first = true; + writeTokenFields(out, first, recorded); + closeObject(out, first); + const String rendered = std::move(out).take(); + DB::ReadBufferFromMemory in(rendered.data(), rendered.size()); + JsonObjectReader r(in, KeyStrictness::Strict, "t"); + TokenFields fields; + String key; + while (r.nextKey(key)) + ASSERT_TRUE(matchTokenFields(key, r, fields)) << "unexpected key " << key; + const PersistedIncarnation back = fields.build("t"); + EXPECT_EQ(back.dialect, recorded.dialect); + EXPECT_EQ(back.value, recorded.value); + } + + /// 2. The `cas_run` condemned row's NDJSON form. + { + const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(9))}; + DB::WriteBufferFromOwnString out; + SourceEdgeRunWriter writer(out); + writer.append(SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Condemned, + .delete_pending = true, .token = recorded, .size = 64, + .condemn_round = 3, .marker_confirmed = true}); + writer.finish(); + out.finalize(); + const String bytes = out.str(); + DB::ReadBufferFromMemory in(bytes.data(), bytes.size()); + SourceEdgeRunReader reader(in); + SourceEdgeRecord back; + ASSERT_TRUE(reader.next(back)); + EXPECT_EQ(back.token.dialect, recorded.dialect); + EXPECT_EQ(back.token.value, recorded.value); + EXPECT_FALSE(reader.next(back)); + } + + /// 3. The GC outcome log. + { + OutcomeLog log; + log.entries.push_back(OutcomeEntry{ObjectKind::Blob, + BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(UInt128(4))}, recorded, OutcomeKind::Deleted}); + const OutcomeLog back = decodeOutcomeLog(encodeOutcomeLog(log)); + ASSERT_EQ(back.entries.size(), 1u); + EXPECT_EQ(back.entries[0].token.dialect, recorded.dialect); + EXPECT_EQ(back.entries[0].token.value, recorded.value); + } + + /// 4. The condemned row's packed byte form, whose dialect rides one byte rather than a word. + { + const CondemnedRow row{.delete_pending = false, .token = recorded, .size = 5, + .condemn_round = 11, .marker_confirmed = true}; + EXPECT_EQ(decodeCondemnedRow(encodeCondemnedRow(row)), row); + } +} + +/// Both directions of the dialect vocabulary fail closed, so neither encoding can carry a value the +/// other cannot name. +TEST(CASPersistedIncarnation, UnknownDialectWordAndByteAreBothRefused) +{ + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectWordFromString("etags", "t"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectByteFromWord("etags", "t"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectWordFromByte(0, "t"); }); + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectWordFromByte(4, "t"); }); + EXPECT_EQ(dialectWordFromByte(dialectByteFromWord("generation", "t"), "t"), "generation"); +} diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index 5539d58dda80..fc3b22b96951 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -7,6 +7,8 @@ #include #include +#include + #include #include #include @@ -88,6 +90,56 @@ uint64_t leaveRejectedCleanupDuty(const PoolPtr & store, const RootNamespace & n return rejected_seq; } +/// `ChunkFaultBackend` counts its faults, and a count can no longer wedge one logical write: the write +/// engine settles every ambiguity by an exact read and reissues, so a bounded fault is outlived and the +/// call commits on a later attempt instead of exhausting its own retry window. Latching keeps the fault +/// (and, for `LandedThenLost`, the paired lost resolve-read) armed on every reissue, so the duty tests +/// below can drive a call all the way to a genuine give-up. +class LatchedChunkFaultBackend : public DB::Cas::tests::ChunkFaultBackend +{ +public: + bool latched = false; + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + if (latched && !fail_read_once_key.empty() && key == fail_read_once_key) + throw Poco::TimeoutException("LatchedChunkFaultBackend: the lost read stays lost"); + return ChunkFaultBackend::read(key, access); + } + + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, + DB::Cas::TransportAccess & access) override + { + if (latched && mode != Mode::None && fault_skip == 0 && !expected_value && !fault_substr.empty() + && key.find(fault_substr) != String::npos) + fault_count = 1; + return ChunkFaultBackend::write(key, bytes, expected_value, access); + } + + /// Disarms completely (not just unlatches): what a caller does right after driving a call to its + /// give-up is a further mutation that must reach the store normally. + void disarm() + { + latched = false; + mode = Mode::None; + fault_count = 0; + fault_skip = 0; + fail_read_once_key.clear(); + } +}; + +/// Latches `backend` and drives `f` to a NETWORK_ERROR give-up, then disarms the fault completely so a +/// caller's next mutation reaches the store normally. The caller must have installed a +/// `VirtualRetryClock` on the store first, or the give-up paces through a real sleep instead of a +/// virtual one. +void driveToNetworkErrorGiveUp(LatchedChunkFaultBackend & backend, const std::function & f) +{ + backend.latched = true; + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, f); + backend.disarm(); +} + } /// Removing the deferred-cleanup transfer from `~PartWriteTxn` makes this test fail at the first @@ -96,8 +148,9 @@ uint64_t leaveRejectedCleanupDuty(const PoolPtr & store, const RootNamespace & n /// next mutation resolves the durable wedge, removes the exact old precommit, and only then retires it. TEST(CASWriterDuties, UncertainAdoptedGrantStaysActiveUntilTheNextMutationRemovesIt) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openSingleAttemptPool(backend); + DB::Cas::tests::VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/writer_duty_adopt"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); @@ -109,9 +162,7 @@ TEST(CASWriterDuties, UncertainAdoptedGrantStaysActiveUntilTheNextMutationRemove backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::NETWORK_ERROR, - [&] { abandoned->precommitAdd(ns, "abandoned", abandoned_id); }); + driveToNetworkErrorGiveUp(*backend, [&] { abandoned->precommitAdd(ns, "abandoned", abandoned_id); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(abandoned->precommitState(), PartWriteTxn::PrecommitState::Uncertain); @@ -190,8 +241,9 @@ TEST(CASWriterDuties, ProvenAbsentGrantDrainsAsNoOpBeforeTheNextMutation) /// past the rejected build exactly as the no-wedge reject arm does. TEST(CASWriterDuties, WedgeResolvedAsRejectDrainsTheDutyAsNoOp) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openSingleAttemptPool(backend); + DB::Cas::tests::VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/writer_duty_wedge_reject"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); @@ -202,9 +254,7 @@ TEST(CASWriterDuties, WedgeResolvedAsRejectDrainsTheDutyAsNoOp) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::NETWORK_ERROR, - [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); + driveToNetworkErrorGiveUp(*backend, [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(rejected->precommitState(), PartWriteTxn::PrecommitState::Uncertain); @@ -396,7 +446,7 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem /// build is REJECTED rather than adopted) and then runs real GC rounds until the body is gone. TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); DB::Cas::tests::seedPoolMetaForRestart(*backend); const CasRequestBudget budget{ .attempt_timeout_ms = 50, @@ -423,6 +473,7 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = budget, }); + DB::Cas::tests::VirtualRetryClock::installOn(predecessor); /// A real, fully-promoted ref through the ordinary production write path (no seeded catalog/ckpt) /// gives the namespace genuine epoch-1 content, so the successor's recovery below has something @@ -444,9 +495,7 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) backend->fault_substr = predecessor->layout().namespaceStreamPrefix(predecessor->namespaceLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::NETWORK_ERROR, - [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); + driveToNetworkErrorGiveUp(*backend, [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); ASSERT_TRUE(predecessor->refLaneWedgedForTest(ns)); ASSERT_EQ(rejected->precommitState(), PartWriteTxn::PrecommitState::Uncertain); const uint64_t predecessor_epoch = predecessor->writerEpoch(); @@ -506,8 +555,9 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) /// very next drain -- once the fault clears -- settles the duty and lets that mutation proceed. TEST(CASWriterDuties, DutySurvivesSettlementFailureForRetry) { - auto backend = std::make_shared(); + auto backend = std::make_shared(); auto store = openFrozenSingleAttemptPool(backend); + DB::Cas::tests::VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/writer_duty_settlement_retry"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); publishEmptyRef(store, ns, "target"); @@ -525,9 +575,7 @@ TEST(CASWriterDuties, DutySurvivesSettlementFailureForRetry) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::NETWORK_ERROR, - [&] { store->dropRef(ns, "target"); }); + driveToNetworkErrorGiveUp(*backend, [&] { store->dropRef(ns, "target"); }); EXPECT_TRUE(store->writerCleanupDutiesPendingForTest()) << "a settlement that throws must retain the duty for retry, never lose it"; @@ -535,7 +583,6 @@ TEST(CASWriterDuties, DutySurvivesSettlementFailureForRetry) << "the settlement's failure must abort the mutation it was blocking too, not just its own append"; EXPECT_EQ(store->minActive(), durable_seq); - backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::None; store->dropRef(ns, "target"); EXPECT_FALSE(store->writerCleanupDutiesPendingForTest()); diff --git a/src/Storages/System/StorageSystemContentAddressedMounts.cpp b/src/Storages/System/StorageSystemContentAddressedMounts.cpp index e856acad2286..3aaa5a10af22 100644 --- a/src/Storages/System/StorageSystemContentAddressedMounts.cpp +++ b/src/Storages/System/StorageSystemContentAddressedMounts.cpp @@ -160,7 +160,11 @@ Pipe StorageSystemContentAddressedMounts::read( bool list_ok = true; try { - mounts = Cas::listMounts(store->backend(), store->layout(), now_ms, skew_margin_ms); + /// Introspection reads on the open fence: a row describing this disk's mount slots must + /// still be produced when the local mount fence has already run down -- that is exactly + /// the state an operator opens this table to look at. + Cas::CasOperation op = store->gcRequests().admit(); + mounts = Cas::listMounts(op, store->layout(), now_ms, skew_margin_ms); } catch (...) { From c3f7b20f8abbe53a71c3f2d625d8f784b29f686f Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:03:42 +0200 Subject: [PATCH 16/81] cas: rename Incarnation to Etag/Dialect, migrate the gtest suite onto the engine, delete the old controller MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two closing passes over the request-engine migration. First, a naming decision: `Incarnation` becomes `Etag` (`PersistedIncarnation` -> `PersistedEtag`, `CasIncarnation.{h,cpp}` -> `CasEtag.{h,cpp}`) and `TokenType` folds into a `Dialect` alias, because the class was carrying its wire field's name rather than its actual role — an ETag on S3-compatible stores, a generation on GCS's JSON dialect, a minted sequence value on the emulated backends. This does not touch the blob envelope's `incarnation_tag` or the catalog's incarnation namespace, which are unrelated concepts the rename exists to stop colliding with. Second, the entire gtest suite (~120 files) moves off the legacy backend overrides and onto `CasOperation`/`CasRequests`, naming the etag and the listed key the way production code now does. With every test migrated, the old controller and its 1500-line test file (`gtest_cas_request_control.cpp`) are deleted outright — this was the last thing keeping it alive. `MountLeaseKeeper` is renamed to `MountLeaseRenewer` in the same pass (it renews a lease; it is not ClickHouse's Keeper, and the old name kept reading as if it were). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../cas/architecture/mounts-and-leases.md | 20 +- docs/en/antalya/cas/configuration.md | 2 + docs/en/antalya/cas/operations/debugging.md | 38 +- docs/en/antalya/cas/operations/monitoring.md | 9 +- .../antalya/cas/operations/troubleshooting.md | 34 +- docs/en/operations/system-tables/cas_log.md | 6 +- programs/disks/CommandCaInspect.cpp | 5 +- src/Common/setThreadName.h | 2 +- .../ContentAddressed/Backend/CasBackend.h | 397 +---- .../{CasIncarnation.cpp => CasEtag.cpp} | 2 +- .../Backend/{CasIncarnation.h => CasEtag.h} | 40 +- .../Backend/CasInMemoryBackend.cpp | 115 +- .../Backend/CasInMemoryBackend.h | 34 +- .../Backend/CasInstrumentedBackend.cpp | 6 - .../Backend/CasInstrumentedBackend.h | 110 +- .../Backend/CasObjectStorageBackend.cpp | 285 ++-- .../Backend/CasObjectStorageBackend.h | 95 +- .../ContentAddressed/Backend/CasProbe.cpp | 20 +- .../Backend/CasRequestBudget.cpp | 38 +- .../Backend/CasRequestBudget.h | 64 +- .../Backend/CasRequestControl.cpp | 860 ---------- .../Backend/CasRequestControl.h | 553 ------ .../ContentAddressed/Backend/CasRequests.cpp | 94 +- .../ContentAddressed/Backend/CasRequests.h | 87 +- .../Backend/CasSentinelProbe.cpp | 2 +- .../Backend/CasThrottlingBackend.h | 78 - .../Backend/CasTransportAccess.h | 8 +- .../ContentAddressed/Backend/CasWriteResult.h | 33 +- .../ContentAddressedMetadataStorage.cpp | 12 +- .../ContentAddressedMetadataStorage.h | 11 +- .../ContentAddressedSettings.cpp | 2 + .../Formats/CasGcOutcomesFormat.h | 4 +- .../Formats/CasRecordStreamFormat.h | 6 +- .../ContentAddressed/Formats/CasWireVocab.cpp | 8 +- .../ContentAddressed/Formats/CasWireVocab.h | 14 +- .../ContentAddressed/Formats/README.md | 7 +- .../ContentAddressed/Gc/CasBlobInDegree.cpp | 6 +- .../ContentAddressed/Gc/CasBlobInDegree.h | 8 +- .../ContentAddressed/Gc/CasGc.cpp | 120 +- .../ContentAddressed/Gc/CasGc.h | 12 +- .../Gc/CasGcMaintenanceState.cpp | 8 +- .../Gc/CasGcMaintenanceState.h | 4 +- .../ContentAddressed/Gc/CasGcMetaWriter.cpp | 26 +- .../ContentAddressed/Gc/CasGcMetaWriter.h | 18 +- .../Gc/CasNamespaceJanitor.cpp | 18 +- .../Gc/CasOrphanManifestSweep.cpp | 22 +- .../Gc/CasOrphanManifestSweep.h | 2 +- .../Gc/CatalogLifecycleReconciler.cpp | 2 +- .../ContentAddressed/Pool/CasBlobMeta.cpp | 6 +- .../ContentAddressed/Pool/CasBlobMeta.h | 6 +- .../ContentAddressed/Pool/CasMountRuntime.cpp | 110 +- .../ContentAddressed/Pool/CasMountRuntime.h | 56 +- .../ContentAddressed/Pool/CasPartWriteTxn.cpp | 19 +- .../ContentAddressed/Pool/CasPlainObjects.cpp | 2 +- .../ContentAddressed/Pool/CasPool.cpp | 76 +- .../ContentAddressed/Pool/CasPool.h | 26 +- .../ContentAddressed/Pool/CasRefCatalog.cpp | 16 +- .../ContentAddressed/Pool/CasRefCatalog.h | 2 +- .../ContentAddressed/Pool/CasRefCkpt.cpp | 6 +- .../ContentAddressed/Pool/CasRefCkpt.h | 4 +- .../ContentAddressed/Pool/CasRefLedger.cpp | 25 +- .../ContentAddressed/Pool/CasRefLedger.h | 7 - .../ContentAddressed/Pool/CasServerRoot.cpp | 150 +- .../ContentAddressed/Pool/CasServerRoot.h | 42 +- .../ContentAddressed/Primitives/CasTypes.h | 15 +- .../ContentAddressed/README.md | 2 +- .../Tools/CasDecommission.cpp | 24 +- .../ContentAddressed/Tools/CasFsck.cpp | 10 +- .../ContentAddressed/Tools/CasInspect.cpp | 4 +- .../benchmarks/benchmark_cas_ref_protocol.cpp | 2 +- src/Disks/tests/cas_sweep_test_support.h | 6 +- src/Disks/tests/cas_test_helpers.h | 239 +-- src/Disks/tests/gtest_ca_wiring.cpp | 85 +- src/Disks/tests/gtest_cas_b140_dangle.cpp | 7 +- src/Disks/tests/gtest_cas_backend.cpp | 952 +++++------ .../tests/gtest_cas_backend_contract.cpp | 201 ++- .../tests/gtest_cas_backend_generation.cpp | 176 +- src/Disks/tests/gtest_cas_backend_listing.cpp | 35 +- src/Disks/tests/gtest_cas_blob_indegree.cpp | 45 +- src/Disks/tests/gtest_cas_blob_meta.cpp | 12 +- .../tests/gtest_cas_bootstrap_ordering.cpp | 74 +- .../tests/gtest_cas_confirm_exact_ref.cpp | 5 +- src/Disks/tests/gtest_cas_decommission.cpp | 305 ++-- .../gtest_cas_decommission_catalog_duties.cpp | 25 +- src/Disks/tests/gtest_cas_detached_work.cpp | 4 - src/Disks/tests/gtest_cas_empty_proof.cpp | 2 +- src/Disks/tests/gtest_cas_encoding_pins.cpp | 6 +- src/Disks/tests/gtest_cas_event_log.cpp | 53 +- .../tests/gtest_cas_fence_generation.cpp | 32 +- src/Disks/tests/gtest_cas_forget.cpp | 31 +- src/Disks/tests/gtest_cas_fsck.cpp | 121 +- src/Disks/tests/gtest_cas_gc_ack_floor.cpp | 140 +- .../tests/gtest_cas_gc_arithmetic_intake.cpp | 8 +- src/Disks/tests/gtest_cas_gc_attempt.cpp | 16 +- src/Disks/tests/gtest_cas_gc_bounded_walk.cpp | 8 +- src/Disks/tests/gtest_cas_gc_fold.cpp | 69 +- .../tests/gtest_cas_gc_frontier_gate.cpp | 310 ++-- src/Disks/tests/gtest_cas_gc_hold_grammar.cpp | 148 +- src/Disks/tests/gtest_cas_gc_leak.cpp | 37 +- .../gtest_cas_gc_maintenance_state_format.cpp | 44 +- src/Disks/tests/gtest_cas_gc_meta_writer.cpp | 4 +- .../tests/gtest_cas_gc_outcomes_format.cpp | 12 +- src/Disks/tests/gtest_cas_gc_rebuild.cpp | 107 +- src/Disks/tests/gtest_cas_gc_resume.cpp | 14 +- src/Disks/tests/gtest_cas_gc_round.cpp | 156 +- src/Disks/tests/gtest_cas_gc_round_defer.cpp | 68 +- .../tests/gtest_cas_gc_shard_incarnation.cpp | 89 +- src/Disks/tests/gtest_cas_gc_shard_plan.cpp | 11 +- src/Disks/tests/gtest_cas_gc_stop_start.cpp | 7 +- .../tests/gtest_cas_gc_undercount_repro.cpp | 3 +- src/Disks/tests/gtest_cas_heartbeat.cpp | 256 +-- .../tests/gtest_cas_holey_list_detector.cpp | 18 +- src/Disks/tests/gtest_cas_ids.cpp | 15 +- src/Disks/tests/gtest_cas_inspect.cpp | 6 +- .../tests/gtest_cas_lifecycle_condition.cpp | 26 +- .../tests/gtest_cas_lifecycle_snapshot.cpp | 5 +- .../tests/gtest_cas_list_liar_end_to_end.cpp | 20 +- src/Disks/tests/gtest_cas_mount.cpp | 131 +- .../tests/gtest_cas_mount_claim_conflicts.cpp | 36 +- src/Disks/tests/gtest_cas_mount_runtime.cpp | 2 +- ...est_cas_namespace_file_request_profile.cpp | 13 +- .../tests/gtest_cas_namespace_janitor.cpp | 165 +- .../tests/gtest_cas_ns_file_incarnation.cpp | 25 +- .../tests/gtest_cas_ns_file_read_contract.cpp | 24 +- src/Disks/tests/gtest_cas_observability.cpp | 30 +- src/Disks/tests/gtest_cas_operation_gate.cpp | 9 +- .../tests/gtest_cas_orphan_manifest_sweep.cpp | 101 +- .../tests/gtest_cas_orphan_nomination.cpp | 30 +- .../tests/gtest_cas_part_folder_access.cpp | 23 +- src/Disks/tests/gtest_cas_part_write.cpp | 323 ++-- .../gtest_cas_part_write_root_dangle.cpp | 17 +- src/Disks/tests/gtest_cas_pluggable_hash.cpp | 80 +- src/Disks/tests/gtest_cas_pool.cpp | 384 ++--- src/Disks/tests/gtest_cas_pool_meta.cpp | 6 +- src/Disks/tests/gtest_cas_probe.cpp | 13 +- .../tests/gtest_cas_protocol_scenarios.cpp | 51 +- .../gtest_cas_rebuild_condemn_nothing.cpp | 74 +- .../tests/gtest_cas_record_stream_format.cpp | 8 +- .../tests/gtest_cas_recovery_grounding.cpp | 31 +- .../tests/gtest_cas_recovery_streaming.cpp | 10 +- src/Disks/tests/gtest_cas_ref_carve.cpp | 12 +- src/Disks/tests/gtest_cas_ref_catalog.cpp | 22 +- .../gtest_cas_ref_catalog_birth_wiring.cpp | 8 +- .../tests/gtest_cas_ref_chunked_flush.cpp | 8 +- src/Disks/tests/gtest_cas_ref_ckpt.cpp | 8 +- .../tests/gtest_cas_ref_contiguous_alloc.cpp | 25 +- src/Disks/tests/gtest_cas_ref_gc.cpp | 100 +- .../tests/gtest_cas_ref_install_safety.cpp | 18 +- .../tests/gtest_cas_ref_read_contract.cpp | 15 +- .../tests/gtest_cas_ref_recovery_cas_walk.cpp | 177 +- ...test_cas_ref_snapshot_publish_ordering.cpp | 2 - .../gtest_cas_ref_wedge_every_attempt.cpp | 115 +- src/Disks/tests/gtest_cas_ref_writer.cpp | 284 ++-- src/Disks/tests/gtest_cas_request_control.cpp | 1498 ----------------- src/Disks/tests/gtest_cas_requests.cpp | 195 +-- .../tests/gtest_cas_retirement_sweep.cpp | 29 +- src/Disks/tests/gtest_cas_s3_staging.cpp | 164 +- src/Disks/tests/gtest_cas_sentinel_probe.cpp | 15 +- .../tests/gtest_cas_shutdown_context.cpp | 3 +- src/Disks/tests/gtest_cas_slot_occupy.cpp | 8 +- .../gtest_cas_sweep_deletion_premise.cpp | 32 +- .../tests/gtest_cas_truncate_reclaim.cpp | 4 +- src/Disks/tests/gtest_cas_upload_detached.cpp | 74 +- src/Disks/tests/gtest_cas_upload_fanout.cpp | 76 +- src/Disks/tests/gtest_cas_wire_vocab.cpp | 30 +- src/Disks/tests/gtest_cas_writer_duties.cpp | 28 +- .../tests/gtest_gcs_conditional_dialect.cpp | 4 +- .../StorageSystemContentAddressedMounts.cpp | 2 +- 168 files changed, 4712 insertions(+), 7800 deletions(-) rename src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/{CasIncarnation.cpp => CasEtag.cpp} (96%) rename src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/{CasIncarnation.h => CasEtag.h} (62%) delete mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp delete mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h delete mode 100644 src/Disks/tests/gtest_cas_request_control.cpp diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index 66e540a5464f..685248ce61c0 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -9,7 +9,7 @@ doc_type: 'reference' Page 4 of 4 in the CAS architecture set. Covers server identity, the mount lease that fences writers, and the server-scoped control-plane objects. No external coordinator is involved: there -is no ZooKeeper/Keeper client anywhere in this protocol — `MountLeaseKeeper` is a local lease +is no ZooKeeper/Keeper client anywhere in this protocol — `MountLeaseRenewer` is a local lease *renewer*, not a Keeper client. ## `cas_server_root_id` — the identity {#server-root-id} @@ -75,7 +75,7 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve write_attempt_id)` tuple before I/O. Every physical retry repeats it byte-for-byte; a later GC fence preserves the observed ID, while reclaim and successor bodies mint new IDs. - **Resolve before retry.** A transient or ambiguous conditional `PUT` is followed by one exact - `GET`. The keeper adopts the result only when the complete body, including `write_attempt_id`, + `GET`. The renewer adopts the result only when the complete body, including `write_attempt_id`, equals its immutable request. If the predecessor token is still current, another identical `PUT` may follow bounded backoff. A same-pair twin, GC-fenced body, successor, foreign holder, or absent body is never treated as this renewal. @@ -97,7 +97,7 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve `attempt_timeout + safety_margin` fits inside the remaining lease, rejecting with `BAD_ARGUMENTS` at request-admission time rather than mid-flight. -**Losing the lease is neither read-only mode nor a process abort.** `MountLeaseKeeper` is a +**Losing the lease is neither read-only mode nor a process abort.** `MountLeaseRenewer` is a synchronous durable-slot state machine. A committed result advances its token, sequence, confirmed BOOTTIME deadline, and cadence anchor. Any admitted deterministic failure, confirmed conflict, or ambiguity left at the deadline/attempt limit moves it to `RenewalTerminal`; it cannot mint another @@ -105,7 +105,7 @@ body or publish a clean farewell. Owner cancellation before any request is the o `NotAttempted` result and leaves clean release possible. Cancellation after a request was sent is terminal because that request may still land. -After the keeper call returns, `CasMountRuntime` consumes the result. A terminal result trips the +After the renewer call returns, `CasMountRuntime` consumes the result. A terminal result trips the local fence (latches `lost`, bumps the fence generation, moves the in-process runtime to `TransientNotLive`) and latches one self-remount generation. A confirmed foreign/successor or same-pair conflict remains a typed fail-closed error; it is never adopted. A real fence still costs @@ -166,7 +166,7 @@ the claim outcomes above and is shown here as behavior, not as a type in the cod stateDiagram-v2 [*] --> Absent Absent --> Live: claimMount putIfAbsent, seq=1 - Live --> Live: keeper beat, putOverwrite seq+1 + Live --> Live: renewer beat, putOverwrite seq+1 Live --> Fenced: GC observes a stable token past threshold, gc_fenced=1, body preserved Live --> Terminated: certified drain, terminal farewell (expires_at=now, min_active_build_sequence=MAX) Fenced --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim @@ -202,7 +202,7 @@ under a live mount is an operator-level event. **Writable open** runs in a strict order: bootstrap-residual proof, capability probe under a random per-mount prefix, pool-meta create-or-validate, `validateServerRootId`, owner claim, -`allocateWriterEpoch`, mount claim and synchronous keeper start, materialization grace if the +`allocateWriterEpoch`, mount claim and synchronous renewer start, materialization grace if the predecessor was unclean (default 30 s), arm the fence, then create and release the runtime-owned renewal and remount workers before the writable pool becomes externally visible. If the grace period consumed the TTL, one fresh synchronous renewal re-anchors the deadline before the fence is armed. @@ -211,15 +211,15 @@ open. No incident path constructs a thread. The renewal and remount workers are separate and long-lived under one stable `CasMountRuntime`. `scheduleRemount` increments a requested-generation latch and wakes the persistent remount worker, -including while an older generation is active. Before keeper replacement, remount requests -`ParkRequested` and waits for the renewal driver to report `Parked`, which proves that no keeper call +including while an older generation is active. Before renewer replacement, remount requests +`ParkRequested` and waits for the renewal driver to report `Parked`, which proves that no renewer call is in flight. A successful remount handles only its snapshotted generation; a newer request is processed before renewal resumes. **Clean unmount:** request stop and join both persistent workers, drain the ref lanes, and only if -the drain *certified* quiescence call `MountLeaseKeeper::release` on an `Active` keeper to write the +the drain *certified* quiescence call `MountLeaseRenewer::release` on an `Active` renewer to write the terminal farewell (`expires_at_ms` already expired, `min_active_build_sequence = UINT64_MAX`). That sentinel is what -lets a successor reclaim instantly. A `RenewalTerminal` keeper, an unresolved ref write, or a sent +lets a successor reclaim instantly. A `RenewalTerminal` renewer, an unresolved ref write, or a sent renewal ambiguity writes no farewell — an unearned farewell would let a successor start mutating while a stale conditional request from the predecessor is still in flight. diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 2e9c6403b91b..7939db3034bf 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -100,6 +100,8 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_part_folder_cache_max_entry_bytes` | 16 MiB | Oversized part-folder views bypass retention above this size | | `cas_manifest_decode_cache_bytes` | 128 MiB | Manifest decode cache byte budget (`0` disables) | | `cas_gc_meta_pool_size` | `16` | Bounded pool size for GC per-hash freshness-meta writes | +| `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove) | +| `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: `cas_attempt_timeout_ms + cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, or the disk refuses to open writable | | `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount | ## Advanced GC pacing settings {#advanced-gc-pacing-settings} diff --git a/docs/en/antalya/cas/operations/debugging.md b/docs/en/antalya/cas/operations/debugging.md index cbc8b0c693f0..cb2e48f4f3b3 100644 --- a/docs/en/antalya/cas/operations/debugging.md +++ b/docs/en/antalya/cas/operations/debugging.md @@ -112,8 +112,6 @@ SELECT event_time_microseconds, event_type, outcome, reason, detail['write_attempt_id'] AS write_attempt_id, detail['attempts_sent'] AS attempts_sent, detail['classification'] AS classification, - detail['deadline_source'] AS deadline_source, - detail['stop_cause'] AS stop_cause, detail['attempt_no'] AS remount_attempt, detail['step'] AS remount_step, detail['error'] AS error @@ -123,15 +121,33 @@ WHERE disk_name = 'cas' ORDER BY event_time_microseconds; ``` -Interpret the sequence as follows: - -- `retrying -> recovered` with the same `write_attempt_id` means an in-budget blip recovered in the - existing epoch; `classification = 'committed_by_get'` means exact `GET` proved a landed request, - while `committed_after_retry` means a later identical physical `PUT` completed. -- A `failed` renewal carries the decisive `unresolved_reason`, `deadline_source`, `stop_cause`, and - `classification`. `external_lease_deadline`, `cancelled`, `conflict`, - `fence_or_lifecycle_lost`, and `attempts_exhausted` are different operator diagnoses; do not - collapse them into a generic timeout. +A `watermark_renew` row now carries only two detail keys beyond the identifying ones: +`attempts_sent` (the number of physical HTTP attempts the whole logical renewal made) and +`classification`. There is no per-attempt `retrying` row any more — a renewal that recovers after +one or more physical attempts produces exactly one `recovered` row when it settles, not a `retrying` +row followed by a `recovered` one — and the older `unresolved_reason`, `deadline_source`, and +`stop_cause` keys are gone; everything they used to distinguish is now named directly by +`classification`. Interpret the sequence as follows: + +- `outcome = 'recovered'` means an in-budget renewal landed, in the same epoch. `classification` + says how: `committed_by_read` means an exact `GET` proved a landed request; `committed_after_retry` + means a later identical physical `PUT` completed and the response itself proved it. +- `outcome = 'failed'` carries the decisive `classification`: `external_lease_deadline` (the + confirmed lease's own safety margin, not the request policy, ran out first — check object-store + latency or `BOOTTIME` advancement before anything else), `request_deadline` (the ninety-second + request policy exhausted first), `unresolved` (every attempt was ambiguous and never settled by + the time the operation gave up), `conflict` (an exact resolve read found another body — a + same-pair twin, a GC-fenced body, a successor epoch, or a foreign holder), `cancelled` (a + renewal in flight was cancelled by shutdown or a remount park request; expected during graceful + shutdown), `fence_or_lifecycle_lost` (another local fence loss or a terminal lifecycle transition + closed admission while the operation was active), `deterministic_failure` (the store's own + answer proved the write never applied), and `vanished` (an exact resolve read proved the mount + slot absent — the pool directory was removed or renamed out of band, or a decommission raced the + renewal). `terminal_unclassified` means the renewal terminated through a path that assigned no + classification; that is a defect to report together with the surrounding rows, not an operator + condition. Do not collapse these into a generic timeout — the action + differs by classification, and only `external_lease_deadline` and `request_deadline` are about a + deadline at all. - A following `mount_remount` row names the whole-chain `attempt_no` and final `step`. An `ok` row restored `Live` under the reported fresh `writer_epoch`; a `failed` row's `step` and optional `error` identify where that whole-chain attempt stopped. diff --git a/docs/en/antalya/cas/operations/monitoring.md b/docs/en/antalya/cas/operations/monitoring.md index 58600e8e058c..3261d91891bf 100644 --- a/docs/en/antalya/cas/operations/monitoring.md +++ b/docs/en/antalya/cas/operations/monitoring.md @@ -82,9 +82,12 @@ SETTINGS system_events_show_zero_values = 1; ``` `system.cas_log` records only nontrivial logical renewals. A `watermark_renew` row has outcome -`retrying`, `recovered`, or `failed`, with detail keys `server_root_id`, `writer_epoch`, `seq`, a -shortened `write_attempt_id`, `attempts_sent`, `elapsed_ms`, `remaining_confirmed_budget_ms`, -`unresolved_reason`, `deadline_source`, `stop_cause`, and `classification`. Ordinary first-attempt +`recovered` or `failed` — there is no per-attempt `retrying` row; the terminal event is the whole +story — with detail keys `server_root_id`, `writer_epoch`, `seq`, a shortened `write_attempt_id`, +`attempts_sent`, `elapsed_ms`, `remaining_confirmed_budget_ms`, and `classification`. The older +`unresolved_reason`, `deadline_source`, and `stop_cause` keys no longer exist; `classification` +carries what they used to say between them (see [debugging](/antalya/cas/operations/debugging#trace-renewal-remount) +for the full value list). Ordinary first-attempt success produces no row. Every `mount_remount` attempt produces one final row with outcome `ok` or `failed` and details `attempt_no`, `step`, `server_root_id`, optional `writer_epoch`, and optional `error`. diff --git a/docs/en/antalya/cas/operations/troubleshooting.md b/docs/en/antalya/cas/operations/troubleshooting.md index fe81f56b64ea..1113ec1855dd 100644 --- a/docs/en/antalya/cas/operations/troubleshooting.md +++ b/docs/en/antalya/cas/operations/troubleshooting.md @@ -16,15 +16,15 @@ tools. | Symptom | Diagnosis | Action | |---|---|---| -| A server keeps losing its mount lease and self-remounting | Check `system.cas_mounts` for the server's own `state`/`expires_at`, then correlate `watermark_renew` and `mount_remount` in `system.cas_log`; losing the lease trips a local fence and latches a remount generation | Read `classification`, `deadline_source`, and `stop_cause` before changing anything. Look for object-store latency consuming the confirmed lease or BOOTTIME advancement; see [the decision flow](#mount-renewal-remount-flow) and [the mount lease](/antalya/cas/architecture/mounts-and-leases#mount-lease) | -| Writes slow down or stall under load, with no exception reaching the client | S3 `SlowDown`/`ServiceUnavailable`/`RequestTimeout`/`InternalError` (5xx) responses are not on `CasRequestController`'s definite-failure whitelist (only malformed-request, entity-too-large, and access-denied are), so they classify as `Unresolved` and are retried automatically. Confirm with `sum(ProfileEvents['CASConditionalWriteUnresolved'])` rising alongside `sum(ProfileEvents['CASConditionalWriteAttempts'])` over `system.query_log` for the affected window (or `ProfileEvent_CASConditionalWriteUnresolved` in `system.metric_log` for a cumulative view across queries), and check `system.blob_storage_log` for `disk_name = ''` rows with a nonzero `error_code` around the same window | Nothing to configure per-request: the controller retries the same `(key, bytes)` with capped-exponential backoff (200ms initial, capped at 5s) for up to 16 attempts inside a 90-second operation deadline, and the mount-lease renewer keeps extending the fence across the disruption — this is the "blips, throttling, partial outages" case the write path is built to survive. Confirm the mount lease itself is still renewing (`system.cas_mounts.expires_at` moving forward, `last_success_age_seconds` not climbing) — if it is, this is expected and self-resolving. If `SlowDown` responses are sustained rather than transient, check the bucket's request-rate limits against the pool's actual PUT/GET rate (see [bucket requirements](/antalya/cas/bucket-requirements)) and consider lowering `cas_blob_upload_pool_size` to reduce concurrent upload traffic; a write only surfaces a client-visible `NETWORK_ERROR` if the 90-second deadline is exhausted before the store recovers, and that error is retried by the ordinary merge/insert backoff, not silently dropped | +| A server keeps losing its mount lease and self-remounting | Check `system.cas_mounts` for the server's own `state`/`expires_at`, then correlate `watermark_renew` and `mount_remount` in `system.cas_log`; losing the lease trips a local fence and latches a remount generation | Read the failed renewal's `classification` before changing anything — it alone now says why (see [the decision flow](#mount-renewal-remount-flow)). Look for object-store latency consuming the confirmed lease or BOOTTIME advancement; see [the mount lease](/antalya/cas/architecture/mounts-and-leases#mount-lease) | +| Writes slow down or stall under load, with no exception reaching the client | S3 `SlowDown`/`ServiceUnavailable`/`RequestTimeout`/`InternalError` (5xx) responses are not on the request engine's `isDefinitelyRefusedWrite` definite-failure list (only malformed-request, entity-too-large, and access-denied that no credential refresh can fix are), so they classify as ambiguous and are retried automatically. Confirm with `sum(ProfileEvents['CASConditionalWriteUnresolved'])` rising alongside `sum(ProfileEvents['CASConditionalWriteAttempts'])` over `system.query_log` for the affected window (or `ProfileEvent_CASConditionalWriteUnresolved` in `system.metric_log` for a cumulative view across queries), and check `system.blob_storage_log` for `disk_name = ''` rows with a nonzero `error_code` around the same window | Nothing to configure per-request: the request engine retries the same `(key, bytes)` with capped-exponential backoff (200ms initial, capped at 5s, full jitter) until the 90-second operation deadline — there is no separate attempts ceiling, only the deadline — and the mount-lease renewer keeps extending the fence across the disruption — this is the "blips, throttling, partial outages" case the write path is built to survive. Confirm the mount lease itself is still renewing (`system.cas_mounts.expires_at` moving forward, `last_success_age_seconds` not climbing) — if it is, this is expected and self-resolving. If `SlowDown` responses are sustained rather than transient, check the bucket's request-rate limits against the pool's actual PUT/GET rate (see [bucket requirements](/antalya/cas/bucket-requirements)) and consider lowering `cas_blob_upload_pool_size` to reduce concurrent upload traffic; a write only surfaces a client-visible `NETWORK_ERROR` if the 90-second deadline is exhausted before the store recovers, and that error is retried by the ordinary merge/insert backoff, not silently dropped | | `GC` never seems to reclaim space after tables are dropped | `SELECT * FROM system.cas_gc_log WHERE event_type='Finish' ORDER BY event_time DESC LIMIT 5` — check `outcome`; also `SELECT is_leader FROM system.cas_mounts` on this node | If `outcome != 'Success'`/`'Deferred'`, see [reading GC health](/antalya/cas/operations/monitoring#gc-health); if this node is not the leader (`is_leader = 0`), it never reclaims for this disk — check the peer holding leadership. Reclamation also needs at least two full rounds past condemnation by design (the grace period is rounds, not acks) — a single manual `SYSTEM CAS GC RUN` will not finish it | | A dangling-access exception or `CORRUPTED_DATA` on read | Run `clickhouse-disks cas-fsck --detail` and check `dangling` specifically — it is the one class that means data loss, distinct from `unreachable`/`awaiting-gc`, which are just waiting for graduation | A nonzero `dangling` count is a real incident: collect the `--detail` output (see [what to collect before filing a bug](/antalya/cas/operations/debugging#filing-a-bug)) before taking any destructive action | | `SYSTEM CAS FSCK` or `clickhouse-disks cas-fsck` times out on a large pool | The scan is bounded by `--timeout` (default 600s / the `SYSTEM` form has no override); a large `roots/` prefix can make the scan slow | Retry with `--partial` to see the counts accumulated so far instead of aborting empty-handed, or `--namespace ` to scope the scan to a subset of namespaces | | `SYSTEM CAS DROP POOL MEMBER` returns a non-empty `warnings` column | A per-object drain step could not confirm emptiness; the mount slot is left terminated but not fully drained, as a resume anchor | Rerun the same command — it is resumable and skips namespaces already marked removed, reporting them under `namespaces_already_removed` | | Writes or `ALTER`s on a `CAS` disk fail with a `READONLY`-class error | The disk's metadata storage rejects every mutating entry point; this is deliberate for a disk opened with `true`, used by every offline `clickhouse-disks` tool | Confirm whether the disk was intentionally configured read-only (offline inspection, `cas-fsck`, `cas-gc-dryrun`, `cas-gc-rebuild`, `cas-drop-member` all require it); a production disk serving writes must not carry `true` | | A table stays unavailable after a transient network error during startup | `AsyncLoader` has no retry/requeue path for a failed table load job: a transient S3 `NETWORK_ERROR` during `CAS` ref-table startup recovery can leave the job permanently `FAILED` | Restart the server, or issue a fresh load for the table; this is a one-shot job design, not a `CAS`-specific bug | -| A mounted pool directory was removed or renamed out of band | Renewal observes an absent, foreign, successor, or otherwise conflicting mount body and terminates the keeper with a typed fail-closed exception; the runtime closes the local write fence and requests remount rather than adopting the body | Never remove or rename a live pool's storage path. To retire a member permanently use [`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission) instead of raw filesystem operations; collect the `watermark_renew` classification and subsequent `mount_remount` step | +| A mounted pool directory was removed or renamed out of band | Renewal observes an absent, foreign, successor, or otherwise conflicting mount body and terminates the keeper with a typed fail-closed exception; the runtime closes the local write fence and requests remount rather than adopting the body | Never remove or rename a live pool's storage path. To retire a member permanently use [`SYSTEM CAS DROP POOL MEMBER`](/antalya/cas/operations/migration#decommission) instead of raw filesystem operations; collect the `watermark_renew` classification (`vanished` for an absent body, `conflict` for a foreign, successor, or otherwise conflicting one) and the subsequent `mount_remount` step | | Stale-looking part metadata after an out-of-band change to the pool | The part-folder view cache may be serving a retained (not re-validated) view | Set the disk-level `cas_part_folder_cache_bytes = 0` to disable view retention and `cas_manifest_decode_cache_bytes = 0` to make every manifest read fetch the body, then run `fsck`; both are diagnostic kill switches, not steady-state settings | | A wide merge (many thousands of columns) fails with a port-exhaustion error from the network layer | Each column in a wide part can cost a separate object-store operation in one merge, and a very wide part can issue on the order of the column count in requests, exhausting local ephemeral TCP ports under load | Reduce concurrent merge parallelism on that table, or increase the host's ephemeral port range; this is a general high-fan-out-merge limit, not specific to content addressing | @@ -33,28 +33,30 @@ tools. Start with the `watermark_renew` timeline described in [debugging](/antalya/cas/operations/debugging#trace-renewal-remount), then follow the matching case: -1. **Recovered blip.** `retrying` is followed by `recovered` for the same shortened - `write_attempt_id`; `CASMountRenewalRecovered` rises while `CASMountLeaseLost` and all remount - counters stay flat. No intervention is needed unless the rate is sustained; investigate backend - throttling/latency before the blips consume the lease budget. +1. **Recovered blip.** A single `outcome = 'recovered'` row (there is no separate `retrying` row to + look for) with `classification` of `committed_by_read` or `committed_after_retry`; + `CASMountRenewalRecovered` rises while `CASMountLeaseLost` and all remount counters stay flat. No + intervention is needed unless the rate is sustained; investigate backend throttling/latency before + the blips consume the lease budget. 2. **External lease-safety exhaustion.** The failed row has - `classification = 'external_lease_deadline'` and - `deadline_source = 'external_lease_safety'`; `CASMountRenewalDeadlineExceeded` and + `classification = 'external_lease_deadline'`; `CASMountRenewalDeadlineExceeded` and `CASMountLeaseLost` rise. The runtime correctly refused to manufacture authority beyond the last confirmed lease. Check object-store latency and BOOTTIME/suspend history, then follow the ensuing - remount. -3. **Cancellation.** `stop_cause = 'cancelled'` after a sent request is terminal and suppresses a - clean farewell because the request may still land. Cancellation before any request is - `NotAttempted`, remains `Active`, and emits no failed aggregate row; during graceful shutdown that - is the expected clean-release path. + remount. `classification = 'request_deadline'` is the sibling case: the ninety-second request + policy exhausted first rather than the lease's own safety margin. +3. **Cancellation.** `classification = 'cancelled'` after a sent request is terminal and suppresses a + clean farewell because the request may still land. Cancellation before any request remains + `Active` and emits no failed aggregate row; during graceful shutdown that is the expected + clean-release path. 4. **Confirmed conflict.** `classification = 'conflict'` means exact resolution found another body; inspect `server_root_id`, `writer_epoch`, `seq`, and `write_attempt_id`. Same-pair twins, GC-fenced bodies, successor epochs, and foreign holders all remain fail closed. Do not delete or rewrite the mount key by hand. -5. **Fence or lifecycle loss.** `stop_cause = 'fence_or_lifecycle_lost'` means another local loss, +5. **Fence or lifecycle loss.** `classification = 'fence_or_lifecycle_lost'` means another local loss, remount park request, or terminal lifecycle closed admission while the operation was active. A parked result reuses the already-requested recovery generation and must not double-count - `CASMountLeaseLost`. + `CASMountLeaseLost`. `classification = 'unresolved'` is a related but distinct case: every attempt + stayed ambiguous and the operation gave up without ever settling one way or the other. 6. **Whole-chain remount failure.** Read the following `mount_remount` row. Its `attempt_no`, `step`, and optional `error` identify the failed owner/catalog/epoch/claim/install/quiescence/fence step. The current protocol retries the whole chain with bounded backoff; it does not preserve per-step diff --git a/docs/en/operations/system-tables/cas_log.md b/docs/en/operations/system-tables/cas_log.md index 17616cd9b329..d4742cda279e 100644 --- a/docs/en/operations/system-tables/cas_log.md +++ b/docs/en/operations/system-tables/cas_log.md @@ -25,13 +25,13 @@ specified (it is enabled by default in the shipped `config.xml`). - `event_date` ([Date](/sql-reference/data-types/date)) — Event date. - `event_time` ([DateTime](/sql-reference/data-types/datetime)) — Event time. - `event_time_microseconds` ([DateTime64(6)](/sql-reference/data-types/datetime64)) — Event time with microseconds precision. -- `event_type` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — The CAS decision/event, e.g. `blob_put`, `blob_reuse_adopt`, `root_remove`, `indegree_zero`, `gc_retire_decision`, `gc_recheck_verdict`, `blob_delete`, `dangling_access`, `corrupt_dangle`. +- `event_type` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — The CAS decision/event, e.g. `blob_put`, `blob_reuse_adopt`, `root_remove`, `indegree_zero`, `gc_retire_decision`, `gc_recheck_verdict`, `blob_delete`, `dangling_access`, `corrupt_dangle`, `watermark_renew`, `mount_remount`. - `disk_name` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — The content-addressed disk / pool the event belongs to. - `namespace` ([String](/sql-reference/data-types/string)) — `roots/` (server/table); empty if not applicable. - `ref_name` ([String](/sql-reference/data-types/string)) — Part name / ref the event concerns; empty if not applicable. - `object_kind` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — One of `none`, `blob`, `manifest`, `root`, `snapshot`. - `object_hash` ([String](/sql-reference/data-types/string)) — Content hash (lowercase hex) of the object; empty if not applicable. -- `token` ([String](/sql-reference/data-types/string)) — Incarnation token (`ETag`) involved; empty if not applicable. +- `token` ([String](/sql-reference/data-types/string)) — On events about a stored object, the incarnation involved, rendered uniformly as `:` (e.g. `etag:"a1b2c3"` on S3-compatible stores, `generation:1234` on GCS); the part-build lifecycle events reuse the column for the 128-bit build id in hex; empty if not applicable. - `round` ([UInt64](/sql-reference/data-types/int-uint)) — GC round (`0` if not applicable). - `generation` ([UInt64](/sql-reference/data-types/int-uint)) — GC snapshot generation (`0` if not applicable). - `at_version` ([UInt64](/sql-reference/data-types/int-uint)) — Manifest `shard_version` of the driving journal record (`0` if not applicable). @@ -39,7 +39,7 @@ specified (it is enabled by default in the shipped `config.xml`). - `reason` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — Human-readable rationale for the decision. Templated across rows, so it is `LowCardinality`. - `thread_id` ([UInt64](/sql-reference/data-types/int-uint)) — OS thread that emitted the event. - `query_id` ([String](/sql-reference/data-types/string)) — Query id for correlation with [`system.query_log`](/operations/system-tables/query_log); empty if not applicable. -- `detail` ([Map(LowCardinality(String), String)](/sql-reference/data-types/map)) — Structured event-specific facts, e.g. `condemn_round`, `superseded_token`, `code`, `site`. +- `detail` ([Map(LowCardinality(String), String)](/sql-reference/data-types/map)) — Structured event-specific facts, e.g. `condemn_round`, `superseded_token`, `code`, `site`, or — on `watermark_renew` — `attempts_sent` and `classification`; see [debugging](/antalya/cas/operations/debugging#trace-renewal-remount) for the mount-renewal detail keys. ## Example {#example} diff --git a/programs/disks/CommandCaInspect.cpp b/programs/disks/CommandCaInspect.cpp index b4a3a24dbb89..80f80324f62e 100644 --- a/programs/disks/CommandCaInspect.cpp +++ b/programs/disks/CommandCaInspect.cpp @@ -48,7 +48,8 @@ class CommandCaInspect final : public ICommand if (!ca->isReadOnly()) throw Exception(ErrorCodes::BAD_ARGUMENTS, "cas-inspect: open the CA disk read-only"); - const auto got = ca->store()->backend().get(key); + Cas::CasOperation op = ca->store()->openRequests().admit(); + const auto got = op.read(key, Cas::Retry::standard()); if (!got) throw Exception(ErrorCodes::BAD_ARGUMENTS, "cas-inspect: key '{}' does not exist", key); @@ -61,7 +62,7 @@ class CommandCaInspect final : public ICommand life_id = *parsed_ckpt; if (life_id) { - const Cas::CasRefCatalog::Snapshot cut = Cas::CasRefCatalog::read(ca->store()->backend(), layout); + const Cas::CasRefCatalog::Snapshot cut = Cas::CasRefCatalog::read(op, layout); resolved_life = cut.life_index.resolve(*life_id); } diff --git a/src/Common/setThreadName.h b/src/Common/setThreadName.h index 4959fb535647..bfeb53141384 100644 --- a/src/Common/setThreadName.h +++ b/src/Common/setThreadName.h @@ -35,7 +35,7 @@ namespace DB M(CAS_ANOMALY_DIAG, "CasAnomalyDiag") \ M(CAS_GC_HEARTBEAT, "CasGcHeartbeat") \ M(CAS_GC_SCHEDULER, "CasGcSched") \ - M(CAS_LEASE_KEEPER, "CasLeaseKeeper") \ + M(CAS_LEASE_RENEWER, "CasLeaseRenewer") \ M(CAS_REF_SNAPSHOT_PUBLISH, "CasRefSnapPub") \ M(CAS_REMOUNT, "CasRemount") \ M(CGROUP_MEMORY_OBSERVER, "CgrpMemUsgObsr") \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index 758faa33daa5..9187c15aa430 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -12,7 +12,6 @@ #include #include #include -#include #include #include #include @@ -20,124 +19,9 @@ #include #include -namespace DB::ErrorCodes -{ - extern const int CAS_WRITE_UNATTRIBUTED; - extern const int CORRUPTED_DATA; - extern const int LOGICAL_ERROR; - extern const int NOT_IMPLEMENTED; -} - namespace DB::Cas { -/// User metadata carried alongside an object (S3 x-amz-meta-*). RETIRED: the transport neither -/// writes nor reads attributes, and this alias survives only in the legacy signatures below. -using ObjectMeta = std::map; - -/// A byte window requested from an object. An absent length means that the window extends to EOF. -/// RETIRED for materialized reads: `get` accepts only a whole-object window. It survives on -/// `getStream`, which is not a forwarder. The offset is exact, while a backend may expose an advisory -/// end when its underlying read buffer cannot enforce one. -struct Range -{ - uint64_t offset = 0; - std::optional length; /// nullopt => to the end - bool whole() const { return offset == 0 && !length; } -}; - -/// Materialized object bytes together with the incarnation and user metadata observed by the read. -/// The token identifies the exact object version whose bytes are in `bytes`; callers may use it to -/// validate a subsequent token-conditional mutation. -struct GetResult -{ - String bytes; - Token token; /// token of the incarnation the bytes came from - ObjectMeta attributes; -}; - -/// A forward-only read of a WRITE-ONCE object (runs, seals): nothing is materialized by the seam. -/// MUTABLE objects (root shards, gc/state, mounts) MUST keep using `get` — their bytes may change -/// under an open stream. `token` identifies the incarnation the stream reads, same as `get`. -struct GetStreamResult -{ - std::unique_ptr stream; - Token token; -}; - -/// Metadata returned by `Backend::head`. For an absent key, `exists` is false and the other fields -/// retain their defaults; for a present key, `size`, `token`, and `attributes` describe one current -/// incarnation as observed by the backend. -struct HeadResult -{ - bool exists = false; - uint64_t size = 0; - Token token; - ObjectMeta attributes; -}; - -/// Outcome of a write-once create or a token-conditional overwrite. A precondition failure means -/// that the backend preserved the existing object; it is an expected result, not an exception. -enum class PutOutcome : uint8_t -{ - Done, /// object written; the returned PutResult.token is the new incarnation's token - PreconditionFailed, /// If-None-Match hit an existing key / If-Match mismatched — nothing changed -}; - -/// Outcome of a compare-and-set write. `Conflict` means that the expected token (or expected -/// absence) did not match and that the backend left the object unchanged. -enum class CasOutcome : uint8_t -{ - Committed, - Conflict, /// expected token (or absence) did not match — nothing changed -}; - -/// Result of a backend write: the outcome plus the resulting object token (previously a `Token * out_token` -/// out-parameter). `token` is set ONLY when the write actually landed an incarnation (a `Done`/`Committed` -/// outcome); on `PreconditionFailed`/`Conflict` nothing was written and `token` is left default-constructed, -/// exactly mirroring the old contract where callers only read `*out_token` on success. -template -struct WriteResultT -{ - Outcome outcome; - Token token; -}; - -using PutResult = WriteResultT; -using CasResult = WriteResultT; - -/// Result of deleting one exact incarnation. `TokenMismatch` and `NotFound` are deliberately -/// distinct: the former proves that another incarnation is now current, while the latter means -/// there is no object to remove. `created_delete_marker` exposes a storage-versioning behavior -/// that is incompatible with current-object reclamation. -struct DeleteOutcome -{ - enum class Kind : uint8_t { Deleted, TokenMismatch, NotFound } kind = Kind::NotFound; - /// TRUE if the backend reported a delete marker was created because versioning is enabled. The - /// capability probe rejects this for the current-object storage model: exact deletion must reclaim - /// the current object rather than archive a noncurrent version. - bool created_delete_marker = false; -}; - -/// A key returned by `Backend::list`. The `token` field is populated ONLY when the backend -/// returns TRUE from `supportsListTokens` — it identifies the key's current incarnation, matching -/// what `head` would return for the same key at that instant. Callers that do not need the token -/// (e.g. GC fence sweep, orphan sweep) ignore the field; GC discover uses it to skip unchanged -/// root shards. -struct ListedKey -{ - String key; - uint64_t size = 0; - std::optional token; /// present iff supportsListTokens() == true -}; -/// One page returned by `Backend::list`. `keys` contains only the requested prefix and the cursor -/// resumes strictly after the last returned key; an empty cursor marks the end of the enumeration. -struct ListPage -{ - std::vector keys; - String next_cursor; /// Last returned key; empty => no more pages. -}; - /// Typed erasure evidence for one key or one prefix. `head`/ /// `get` deliberately flatten every kind of miss (a clean absence, a missing bucket/container, a /// permission failure, a transport fault) into one "not found" result, which is exactly right for @@ -231,29 +115,30 @@ inline BlobPayloadCopyResult copyBlobPayloadBounded(ReadBuffer & from, WriteBuff } -/// Token-aware storage seam used by the content-addressed pool. TOKEN SEMANTICS ARE THE CONTRACT: -/// - every present key has exactly one current incarnation identified by an opaque Token; -/// - putOverwrite/casPut succeed only against the expected current token (or expected absence); -/// - deleteExact removes ONLY the incarnation whose token matches — wrong token MUST be a -/// TokenMismatch with the object untouched (backends that silently ignore the condition are -/// rejected by `Cas::Probe`); -/// - conditional PUTs are protocol hygiene; casPut and deleteExact are SAFETY-critical. +/// Etag-aware storage seam used by the content-addressed pool. INCARNATION SEMANTICS ARE THE +/// CONTRACT: +/// - every present key has exactly one current incarnation identified by an opaque backend value; +/// - `write` with an `expected_value` succeeds only against that exact current value (or expected +/// absence); +/// - `remove` removes ONLY the incarnation whose value matches — a mismatch MUST report +/// `RawRemoval::Mismatch` with the object untouched (backends that silently ignore the condition +/// are rejected by `Cas::Probe`). /// -/// TOKEN ⟹ CONTENT PRECONDITION (read-path caches depend on this): a token must uniquely identify -/// the byte-content of the incarnation it labels — i.e. `head(k).token == prior get(k).token` MUST -/// imply the bytes are unchanged. The protocol's SAFETY only needs the contrapositive (changed -/// bytes ⟹ a new token, so a stale CAS/delete is rejected), but `Cas::Pool`'s read-path decode -/// cache (`readShardDecoded`) skips a re-`get`+decode on a token match, so a backend whose token -/// could REPEAT across different content would make it serve stale manifests (wrong results). Holds -/// for every backend in use: S3 ETag is content-derived; the emulated/in-memory backends mint a -/// strictly-monotonic sequence that is never reused. A backend with a weak/recycled token must NOT -/// be used as a Cas pool. The capability probe currently verifies conditional-operation behavior but -/// does not test token non-reuse across different contents, so this invariant remains a requirement -/// of every backend implementation. +/// VALUE ⟹ CONTENT PRECONDITION (read-path caches depend on this): a value must uniquely identify the +/// byte-content of the incarnation it labels — i.e. `head(k)`'s value equalling a prior `read(k)`'s +/// value MUST imply the bytes are unchanged. The protocol's SAFETY only needs the contrapositive +/// (changed bytes ⟹ a new value, so a stale conditional write/delete is rejected), but `Cas::Pool`'s +/// read-path decode cache (`readShardDecoded`) skips a re-`read`+decode on a value match, so a backend +/// whose value could REPEAT across different content would make it serve stale manifests (wrong +/// results). Holds for every backend in use: S3 ETag is content-derived; the emulated/in-memory +/// backends mint a strictly-monotonic sequence that is never reused. A backend with a weak/recycled +/// value must NOT be used as a Cas pool. The capability probe currently verifies conditional-operation +/// behavior but does not test value non-reuse across different contents, so this invariant remains a +/// requirement of every backend implementation. /// /// Most ops take/return whole `String` bodies — sufficient for manifests, trees, and probe/GC -/// objects. Large content blobs use the transport-only `publishBlob` seam; reads stay String-based -/// because blob payload reads go through the wiring's read stack, not this seam. +/// objects. Large content blobs use the transport-only `publish` seam; reads stay String-based because +/// blob payload reads go through the wiring's read stack, not this seam. class Backend { public: @@ -383,184 +268,6 @@ class Backend /// are not gated here — see ObjectStorageBackend's override for the one backend that is). virtual void checkConditionalWriteSingleAttemptSupport() {} - /// ---- The legacy Token-typed surface ---- - /// - /// Every one of these obtains the migration key and calls the primitive above, so a fault - /// injection written against a PRIMITIVE intercepts a legacy caller too. They stay virtual while - /// the migration runs, so a test double that overrides one of THESE keeps working until the site - /// it instruments moves; the whole block, and `migrationAccess` with it, is deleted at the lock. - /// `Range` and `ObjectMeta` are already retired: a non-whole window is refused, and object - /// attributes are neither written nor returned. - virtual std::optional get(const String & key, Range range) - { - if (!range.whole()) - throw Exception(ErrorCodes::NOT_IMPLEMENTED, "CAS backend: a ranged get is retired; read the object whole"); - auto access = migrationAccess(); - auto raw = read(key, access); - if (!raw) - return std::nullopt; - return GetResult{std::move(raw->bytes), legacyMintObserved(key, std::move(raw->value)), {}}; - } - std::optional get(const String & key) { return get(key, {}); } - - /// Forward-only stream over the object's `range` (default: whole object) for WRITE-ONCE objects - /// (runs, seals). Not a forwarder: `stream` returns no incarnation, and a forwarder would have to - /// fill `GetStreamResult::token` with a default-constructed `Token` that names nothing. Each - /// backend keeps its own implementation until the last caller moves onto `stream`. - /// CAVEAT: the window END is advisory on storages where `setReadUntilPosition` is a hint - /// (LocalObjectStorage) — the stream may yield bytes past the window; consumers MUST bound their - /// own consumption (RunFileReader bounds to its data_end). The window START is always exact. - virtual std::optional getStream(const String & key, Range range) = 0; - std::optional getStream(const String & key) { return getStream(key, {}); } - - virtual HeadResult head(const String & key) - { - auto access = migrationAccess(); - auto raw = head(key, access); - if (!raw) - return {}; - return HeadResult{true, raw->size, legacyMintObserved(key, std::move(raw->value)), {}}; - } - - virtual PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & /*meta*/) - { - auto access = migrationAccess(); - auto r = write(key, bytes, std::nullopt, access); - if (!r) - return PutResult{PutOutcome::PreconditionFailed, {}}; - return PutResult{PutOutcome::Done, legacyMintWritten(key, std::move(*r))}; - } - PutResult putIfAbsent(const String & key, const String & bytes) { return putIfAbsent(key, bytes, {}); } - - virtual PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, - const ObjectMeta & /*meta*/) - { - if (legacyTokenIsForeign(key, expected)) - return PutResult{PutOutcome::PreconditionFailed, {}}; - auto access = migrationAccess(); - auto r = write(key, bytes, expected.value, access); - if (!r) - return PutResult{PutOutcome::PreconditionFailed, {}}; - return PutResult{PutOutcome::Done, legacyMintWritten(key, std::move(*r))}; - } - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected) - { - return putOverwrite(key, bytes, expected, {}); - } - - virtual CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & /*meta*/) - { - if (expected && legacyTokenIsForeign(key, *expected)) - return CasResult{CasOutcome::Conflict, {}}; - auto access = migrationAccess(); - auto r = write(key, bytes, expected ? std::optional(expected->value) : std::nullopt, access); - if (!r) - return CasResult{CasOutcome::Conflict, {}}; - return CasResult{CasOutcome::Committed, legacyMintWritten(key, std::move(*r))}; - } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected) - { - return casPut(key, bytes, expected, {}); - } - - virtual DeleteOutcome deleteExact(const String & key, const Token & token) - { - if (legacyTokenIsForeign(key, token)) - return DeleteOutcome{DeleteOutcome::Kind::TokenMismatch, false}; - auto access = migrationAccess(); - switch (remove(key, token.value, access)) - { - case RawRemoval::Removed: return {DeleteOutcome::Kind::Deleted, false}; - case RawRemoval::Gone: return {DeleteOutcome::Kind::NotFound, false}; - case RawRemoval::Mismatch: return {DeleteOutcome::Kind::TokenMismatch, false}; - case RawRemoval::DeleteMarker: return {DeleteOutcome::Kind::Deleted, true}; - } - UNREACHABLE(); - } - - virtual ListPage list(const String & prefix, const String & cursor, size_t limit) - { - auto access = migrationAccess(); - auto raw = list(prefix, cursor, limit, access); - ListPage page; - page.next_cursor = std::move(raw.next_cursor); - page.keys.reserve(raw.keys.size()); - for (auto & k : raw.keys) - { - std::optional token; - if (k.value) - token = legacyMintObserved(k.key, std::move(*k.value)); - page.keys.push_back(ListedKey{std::move(k.key), k.size, std::move(token)}); - } - return page; - } - - virtual void publishBlob(const BlobPublishRequest & request) - { - auto access = migrationAccess(); - publish(request, access); - } - - virtual SentinelProbeResult probeSentinelRaw(const String & key) - { - auto access = migrationAccess(); - return probeSentinelRaw(key, access); - } - -protected: - /// The migration key. Every legacy forwarder above obtains one; nothing else may, and the whole - /// mechanism is deleted with the forwarders at the lock. - static TransportAccess migrationAccess() { return TransportAccess{}; } - - /// ---- Where a raw response becomes a legacy `Token` ---- - /// - /// The primitives return the store's value as it arrived: judging it belongs to the caller that - /// can act on the judgement. A legacy caller cannot -- it puts the `Token` straight into its next - /// request -- so these two are where a legacy response is judged, and the ONLY places a legacy - /// `Token` is minted. A backend that overrides a legacy method for the migration window mints - /// through them too. - - /// An OBSERVED value (read, head, list) that is not an incarnation means the response fell - /// through unmapped: nothing was changed by it, and the caller must not act on it. - Token legacyMintObserved(const String & key, String value) - { - if (!isIncarnationValue(dialect(), value)) - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CAS backend: the store answered for '{}' with a value '{}' that is not a valid incarnation", - key, value); - return Token{std::move(value), dialect()}; - } - - /// A value from a SUCCESSFUL write that is not an incarnation is the ambiguous case, not the - /// corrupt one: the write may well have landed, and only reading the key back can say. Hence - /// `CAS_WRITE_UNATTRIBUTED`, which names that duty, and never `CORRUPTED_DATA`, which a caller - /// may treat as a deterministic failure and stop. - Token legacyMintWritten(const String & key, String value) - { - if (!isIncarnationValue(dialect(), value)) - throw Exception(ErrorCodes::CAS_WRITE_UNATTRIBUTED, - "CAS backend: the store accepted a write of '{}' but answered with '{}', which is not an " - "incarnation; the write may have committed and must be resolved by reading back", - key, value); - return Token{std::move(value), dialect()}; - } - - /// The dialect half of the legacy token check: the primitives take a bare VALUE and cannot see - /// the dialect a `Token` declares, so a foreign-dialect token is answered here as an ordinary - /// non-match rather than being forwarded to a wire (or a value space) that was never designed to - /// discriminate it. A MALFORMED value is refused FIRST, under its own declared dialect, so a - /// token that is both malformed and foreign is still reported as the caller bug it is. - bool legacyTokenIsForeign(const String & key, const Token & token) - { - if (!isIncarnationValue(token.type, token.value)) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CAS backend: refusing a conditional mutation of '{}' with a malformed token '{}' (dialect {}): " - "an empty, wildcard or list token would turn the precondition into an unconditional write", - key, token.value, static_cast(token.type)); - return token.type != dialect(); - } - private: uint64_t backend_id; }; @@ -575,64 +282,4 @@ inline Backend::Backend() using BackendPtr = std::shared_ptr; -/// Walk every key under `prefix` exactly once, resuming by the backend's explicit last-returned-key -/// cursor (`ListPage::next_cursor`, empty => done). This centralizes the pagination contract shared by -/// GC, fsck, and cleanup sweeps: each returned key is delivered once, and the backend's cursor is the -/// only state used to request the next page. -/// -/// `on_page_fetched`, if set, fires exactly once per physical `backend.list` call (including an -/// empty/undersized final page) — a GC-owned caller's hook for a page-level ProfileEvents counter, -/// without misattributing a non-GC caller (e.g. fsck) that leaves it unset. Trails `page_limit` -/// (rather than sitting before it) so the two existing callers that override `page_limit` -/// (`Gc::fold`, `CasFsck.cpp`'s `listAll`) can override `page_limit` without changing callback order. -inline void forEachListedKey(Backend & backend, const String & prefix, - const std::function & cb, - size_t page_limit = 1000, - const std::function & on_page_fetched = {}) -{ - String cursor; - for (;;) - { - const ListPage page = backend.list(prefix, cursor, page_limit); - if (on_page_fetched) - on_page_fetched(); - for (const ListedKey & k : page.keys) - cb(k); - if (page.next_cursor.empty()) - break; - cursor = page.next_cursor; - } -} - -/// The normalized verdict of a token-exact delete, unifying the DeleteOutcome::Kind three-way that GC -/// (blob + manifest delete) and the orphan-manifest sweep each mapped by hand. -enum class DeleteClass : uint8_t { Deleted, Absent, Replaced }; - -/// Converts a backend-specific delete outcome into the three states used by cleanup callers. The -/// default branch is fail-safe: an unknown value is treated as `Replaced`, so cleanup never reports -/// an unverified deletion as successful. -inline DeleteClass classifyDeleteOutcome(const DeleteOutcome & d) -{ - switch (d.kind) - { - case DeleteOutcome::Kind::Deleted: return DeleteClass::Deleted; - case DeleteOutcome::Kind::NotFound: return DeleteClass::Absent; - case DeleteOutcome::Kind::TokenMismatch: return DeleteClass::Replaced; - } - return DeleteClass::Replaced; /// unreachable; fail-safe toward "leave it" (never a false Deleted) -} - -/// Returns the stable lowercase label used when reporting a normalized delete result. Unknown enum -/// values are labeled `replaced`, matching `classifyDeleteOutcome`'s fail-safe behavior. -inline std::string_view deleteClassName(DeleteClass c) -{ - switch (c) - { - case DeleteClass::Deleted: return "deleted"; - case DeleteClass::Absent: return "absent"; - case DeleteClass::Replaced: return "replaced"; - } - return "replaced"; -} - } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasEtag.cpp similarity index 96% rename from src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp rename to src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasEtag.cpp index f01a45d95fc0..f029480e6fe8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasEtag.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasEtag.h similarity index 62% rename from src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h rename to src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasEtag.h index 1f8edf1889d6..cce53b29fc21 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasIncarnation.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasEtag.h @@ -7,23 +7,25 @@ namespace DB::Cas { -using Dialect = TokenType; - /// The per-dialect grammar a response value must meet to be an incarnation. Generation: canonical /// positive decimal (no leading zero, not "0" -- zero is the dialect's absence sentinel). ETag: /// non-empty, not "*" after trimming whitespace, no comma (a list matches any member). Emulated: /// non-empty. `ObjectStorageBackend::isValidTokenValue` forwards here. bool isIncarnationValue(Dialect dialect, const String & value); -/// One backend-observed incarnation of an object: the transport's own token value together with the -/// backend and key it was observed against. Not default-constructible, not constructible from a bare -/// `String`, and minted ONLY by `CasRequests` -- a caller can hold one only by way of an admitted -/// read or write, so an `Incarnation` is always traceable to the request that produced it. -class Incarnation +/// One backend-observed incarnation of an object: the transport's own value naming that incarnation, +/// together with the backend and key it was observed against. "Etag" names the ROLE this value plays +/// -- whatever the backend hands back as the object's identity for a conditional write -- not the wire +/// field: it is a literal ETag on S3-compatible stores, a generation number on GCS's JSON dialect, and +/// a minted sequence value on the emulated/in-memory backends. Not default-constructible, not +/// constructible from a bare `String`, and minted ONLY by `CasRequests` -- a caller can hold one only +/// by way of an admitted read or write, so an `Etag` is always traceable to the request that produced +/// it. +class Etag { public: - Incarnation() = delete; - bool operator==(const Incarnation &) const = default; + Etag() = delete; + bool operator==(const Etag &) const = default; /// "etag:" | "generation:" | "emulated:" String render() const; @@ -34,7 +36,7 @@ class Incarnation private: friend class CasRequests; - Incarnation(uint64_t backend_id, String key, Dialect dialect, String value) + Etag(uint64_t backend_id, String key, Dialect dialect, String value) : backend_id_(backend_id), key_(std::move(key)), dialect_(dialect), value_(std::move(value)) { } @@ -48,7 +50,7 @@ class Incarnation String value_; }; -inline String Incarnation::render() const +inline String Etag::render() const { switch (dialect_) { @@ -60,27 +62,27 @@ inline String Incarnation::render() const } /// An incarnation as recorded in a persisted manifest/ref: the dialect word and value, without any -/// live backend to check them against. Forward-only: a `PersistedIncarnation` is captured FROM a live -/// `Incarnation`, never the reverse -- a persisted record must never be trusted to mint a live one. +/// live backend to check them against. Forward-only: a `PersistedEtag` is captured FROM a live +/// `Etag`, never the reverse -- a persisted record must never be trusted to mint a live one. /// `matches` re-derives the same rendering the live incarnation would produce and compares it /// textually, so the two representations can never drift apart. -struct PersistedIncarnation +struct PersistedEtag { String dialect; /// "etag" | "generation" | "emulated" String value; - static PersistedIncarnation capture(const Incarnation & live); - bool matches(const Incarnation & live) const; + static PersistedEtag capture(const Etag & live); + bool matches(const Etag & live) const; }; -inline PersistedIncarnation PersistedIncarnation::capture(const Incarnation & live) +inline PersistedEtag PersistedEtag::capture(const Etag & live) { const String rendered = live.render(); const auto colon = rendered.find(':'); - return PersistedIncarnation{rendered.substr(0, colon), rendered.substr(colon + 1)}; + return PersistedEtag{rendered.substr(0, colon), rendered.substr(colon + 1)}; } -inline bool PersistedIncarnation::matches(const Incarnation & live) const +inline bool PersistedEtag::matches(const Etag & live) const { return live.render() == dialect + ":" + value; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp index f7f51b5b45d8..6ebdae19d037 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp @@ -22,18 +22,6 @@ namespace DB::Cas namespace { -/// The windowed slice of `data` for `range`, with the clamping `getStream` documents: an offset at or -/// past EOF yields an empty result; an open-ended length runs to EOF. -String sliceWindow(const String & data, Range range) -{ - const size_t offset = static_cast(range.offset); - if (offset >= data.size()) - return {}; - if (range.length.has_value()) - return data.substr(offset, static_cast(*range.length)); - return data.substr(offset); -} - /// A CALLER bug, refused before it ever reaches the store: an empty, wildcard or list value would /// turn a conditional mutation into an unconditional one. Stricter than /// `isIncarnationValue(Dialect::Emulated, ...)` (non-empty only): this backend is a test double @@ -60,12 +48,9 @@ void checkExpectedValue(const String & key, const String & value) } -Token InMemoryBackend::mintToken() +String InMemoryBackend::mintValue() { - Token t; - t.value = std::to_string(++token_seq_); - t.type = TokenType::Emulated; - return t; + return std::to_string(++token_seq_); } std::exception_ptr InMemoryBackend::takeArmedFailure(ArmedFailures & armed, const String & key) @@ -96,30 +81,19 @@ std::optional InMemoryBackend::read(const String & key, TransportA if (it == store_.end()) return std::nullopt; - return Raw{it->second.bytes, it->second.token.value}; + return Raw{it->second.bytes, it->second.value}; } -std::optional InMemoryBackend::getStream(const String & key, Range range) +std::unique_ptr InMemoryBackend::stream(const String & key, TransportAccess &) { std::lock_guard lock(mutex_); auto it = store_.find(key); if (it == store_.end()) - return std::nullopt; - - /// Copy the windowed bytes into an owning buffer — the in-memory backend has no separate storage - /// to stream from, so the "stream" reads from a private copy of exactly the requested window. - GetStreamResult sr; - sr.stream = std::make_unique(sliceWindow(it->second.bytes, range)); - sr.token = it->second.token; - return sr; -} - -std::unique_ptr InMemoryBackend::stream(const String & key, TransportAccess &) -{ - auto sr = getStream(key, Range{}); - if (!sr) return nullptr; - return std::move(sr->stream); + + /// Copies the bytes into an owning buffer — the in-memory backend has no separate storage to + /// stream from, so the "stream" reads from a private copy taken while the lock is held. + return std::make_unique(it->second.bytes); } std::optional InMemoryBackend::head(const String & key, TransportAccess &) @@ -132,7 +106,7 @@ std::optional InMemoryBackend::head(const String & key, Transp if (it == store_.end()) return std::nullopt; - return RawMeta{static_cast(it->second.bytes.size()), it->second.token.value}; + return RawMeta{static_cast(it->second.bytes.size()), it->second.value}; } std::expected InMemoryBackend::write( @@ -197,24 +171,24 @@ std::expected InMemoryBackend::writeUnderLock( if (store_.contains(key)) return std::unexpected(RawConflict{}); - Token t = mintToken(); + String v = mintValue(); Object obj; obj.bytes = bytes; - obj.token = t; + obj.value = v; store_[key] = std::move(obj); - return t.value; + return v; } auto it = store_.find(key); if (it == store_.end()) return std::unexpected(RawConflict{}); - if (enforce_tokens_ && it->second.token.value != *expected_value) + if (enforce_tokens_ && it->second.value != *expected_value) return std::unexpected(RawConflict{}); - Token t = mintToken(); + String v = mintValue(); it->second.bytes = bytes; - it->second.token = t; - return t.value; + it->second.value = v; + return v; } void InMemoryBackend::publish(const BlobPublishRequest & request, TransportAccess &) @@ -253,7 +227,7 @@ void InMemoryBackend::publish(const BlobPublishRequest & request, TransportAcces std::lock_guard lock(mutex_); Object object; object.bytes = std::move(body); - object.token = mintToken(); + object.value = mintValue(); store_[request.destination_key] = std::move(object); return; } @@ -269,33 +243,22 @@ void InMemoryBackend::publish(const BlobPublishRequest & request, TransportAcces Object object; object.bytes = source->second.bytes; - object.token = mintToken(); + object.value = mintValue(); store_[request.destination_key] = std::move(object); } -DeleteOutcome InMemoryBackend::applyDelete(const String & key, const Token & token) +Backend::RawRemoval InMemoryBackend::applyDelete(const String & key, const String & expected_value) { // Caller holds the mutex. auto it = store_.find(key); if (it == store_.end()) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::NotFound; - return d; - } + return RawRemoval::Gone; - if (enforce_tokens_ && it->second.token != token) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::TokenMismatch; - return d; - } + if (enforce_tokens_ && it->second.value != expected_value) + return RawRemoval::Mismatch; store_.erase(it); - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::Deleted; - d.created_delete_marker = simulate_delete_markers_; - return d; + return simulate_delete_markers_ ? RawRemoval::DeleteMarker : RawRemoval::Removed; } Backend::RawRemoval InMemoryBackend::remove(const String & key, const String & expected_value, TransportAccess &) @@ -305,8 +268,6 @@ Backend::RawRemoval InMemoryBackend::remove(const String & key, const String & e /// PendingDelete can never carry a malformed value either. checkExpectedValue(key, expected_value); - const Token expected{expected_value, TokenType::Emulated}; - std::lock_guard lock(mutex_); if (hold_deletes_) @@ -316,26 +277,16 @@ Backend::RawRemoval InMemoryBackend::remove(const String & key, const String & e auto it = store_.find(key); if (it == store_.end()) return RawRemoval::Gone; - if (enforce_tokens_ && it->second.token != expected) + if (enforce_tokens_ && it->second.value != expected_value) return RawRemoval::Mismatch; PendingDelete pd; pd.key = key; - pd.token = expected; + pd.value = expected_value; pending_deletes_.push_back(std::move(pd)); return simulate_delete_markers_ ? RawRemoval::DeleteMarker : RawRemoval::Removed; } - const DeleteOutcome d = applyDelete(key, expected); - switch (d.kind) - { - case DeleteOutcome::Kind::Deleted: - return d.created_delete_marker ? RawRemoval::DeleteMarker : RawRemoval::Removed; - case DeleteOutcome::Kind::NotFound: - return RawRemoval::Gone; - case DeleteOutcome::Kind::TokenMismatch: - return RawRemoval::Mismatch; - } - UNREACHABLE(); + return applyDelete(key, expected_value); } Backend::RawListPage InMemoryBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess &) @@ -358,7 +309,7 @@ Backend::RawListPage InMemoryBackend::list(const String & prefix, const String & RawListedKey lk; lk.key = it->first; lk.size = static_cast(it->second.bytes.size()); - lk.value = it->second.token.value; /// in-memory backend always surfaces it (supportsListTokens == true) + lk.value = it->second.value; /// in-memory backend always surfaces it (supportsListTokens == true) page.keys.push_back(std::move(lk)); ++count; ++it; @@ -426,21 +377,17 @@ size_t InMemoryBackend::pendingDeletes() const return pending_deletes_.size(); } -DeleteOutcome InMemoryBackend::landPendingDelete(size_t i) +Backend::RawRemoval InMemoryBackend::landPendingDelete(size_t i) { std::lock_guard lock(mutex_); if (i >= pending_deletes_.size()) - { - DeleteOutcome d; - d.kind = DeleteOutcome::Kind::NotFound; - return d; - } + return RawRemoval::Gone; PendingDelete pd = pending_deletes_[i]; pending_deletes_.erase(pending_deletes_.begin() + static_cast(i)); - // Apply the token check at LAND time — the object may have been modified since the delete was enqueued. - return applyDelete(pd.key, pd.token); + // Apply the value check at LAND time — the object may have been modified since the delete was enqueued. + return applyDelete(pd.key, pd.value); } void InMemoryBackend::refuseNextWrite(const String & key) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h index e4e5bbe5bb03..a41d0dd4ab3d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h @@ -10,10 +10,11 @@ namespace DB::Cas { -/// Thread-safe, token-enforcing in-memory `Backend` implementation used by CAS tests. +/// Thread-safe, value-enforcing in-memory `Backend` implementation used by CAS tests. Mints its +/// values in the `Dialect::Emulated` dialect. /// -/// All successful writes mint a monotonically increasing token (`TokenType::Emulated`). -/// Tokens NEVER repeat across the lifetime of a backend instance. +/// All successful writes mint a monotonically increasing value. Values NEVER repeat across the +/// lifetime of a backend instance. /// /// The backend also exposes fault-injection controls for probe tests and CAS correctness tests: /// - `setHoldDeletes` / `landPendingDelete`: simulate async/delayed conditional deletes @@ -28,12 +29,6 @@ class InMemoryBackend : public Backend public: InMemoryBackend() = default; - /// Unhide the base overloads this class's own declarations would otherwise shadow: the legacy - /// `head`/`list`/`getStream` names, and the omitted-`Range` convenience. - using Backend::getStream; - using Backend::head; - using Backend::list; - // ---- Backend interface ---- /// Returns the stored bytes and the key's current incarnation value, or `nullopt` when absent. @@ -81,11 +76,6 @@ class InMemoryBackend : public Backend /// not opted in models a backend with no refresh mechanism. bool refreshCredentials() override; - /// Returns a forward-only stream over the requested byte window, or `nullopt` when the key is absent. - /// The in-memory implementation copies the window into an owning read buffer while holding the - /// backend lock, so the returned stream remains independent of later backend mutations. - std::optional getStream(const String & key, Range range) override; - // ---- Fault-injection controls ---- /// When true, `remove` validates and enqueues deletes rather than applying them immediately. @@ -99,7 +89,7 @@ class InMemoryBackend : public Backend /// Applies and removes the held delete at index `i`. The expected value is evaluated against the /// current object at land time; the queue entry is removed whether the result is `Mismatch` or /// `Removed`. An invalid index returns `NotFound`. - DeleteOutcome landPendingDelete(size_t i); + RawRemoval landPendingDelete(size_t i); /// Refuses the next write of `key` once, as a clean precondition failure that leaves the store /// unchanged -- whatever the write's shape and whichever surface issued it. @@ -154,11 +144,11 @@ class InMemoryBackend : public Backend private: /// Complete in-memory incarnation state for one key. All fields are read or modified while - /// `mutex_` is held; replacing `token` marks a new incarnation even when the bytes are unchanged. + /// `mutex_` is held; replacing `value` marks a new incarnation even when the bytes are unchanged. struct Object { String bytes; - Token token; + String value; }; /// Value captured when a held delete is queued. It is intentionally checked again at land time so @@ -166,16 +156,16 @@ class InMemoryBackend : public Backend struct PendingDelete { String key; - Token token; + String value; }; - /// Mints the next process-local token. Tokens are strictly increasing and never reused by this - /// backend instance, which also makes token equality a safe content-cache identity check in tests. - Token mintToken(); + /// Mints the next process-local incarnation value. Strictly increasing and never reused by this + /// backend instance, which also makes value equality a safe content-cache identity check in tests. + String mintValue(); /// Applies an exact-value delete while `mutex_` is already held. Used by immediate deletes and by /// `landPendingDelete` after its queue entry has been removed. - DeleteOutcome applyDelete(const String & key, const Token & token); + RawRemoval applyDelete(const String & key, const String & expected_value); using ArmedFailures = std::map>; using Hooks = std::map>; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp index 5c6a892f8b2b..7f4f97cd1bb6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp @@ -144,10 +144,4 @@ void InstrumentedBackend::publish(const BlobPublishRequest & request, TransportA incrementCasEvent(classifyCasNs(request.destination_key), CasOp::Put); } -void InstrumentedBackend::publishBlob(const BlobPublishRequest & request) -{ - inner->publishBlob(request); - incrementCasEvent(classifyCasNs(request.destination_key), CasOp::Put); -} - } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h index 25d2407a57c4..c321bfa27fd5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h @@ -35,28 +35,19 @@ enum class CasNs : uint8_t }; static constexpr size_t CAS_NS_COUNT = 6; -/// Operation + outcome class, mapped from the `Backend` method and its result. -/// -/// The decorator counts BOTH surfaces for the migration window. It must: it forwards a legacy call -/// to the inner backend AS a legacy call, so that a double wrapped by a `Pool` still intercepts it, -/// and the conversion to a primitive happens one level down, inside that double. Each request is -/// therefore counted exactly once, on whichever surface its caller used. -/// putIfAbsent → Done ⇒ Put ; PreconditionFailed ⇒ PutDeduplicated -/// putOverwrite → Done ⇒ Overwrite ; PreconditionFailed ⇒ CasConflict -/// casPut → Committed ⇒ Cas ; Conflict ⇒ CasConflict +/// Operation + outcome class, mapped from the `Backend` primitive and its result. /// write, no expected value → a value ⇒ Put ; RawConflict ⇒ PutDeduplicated /// write, an expected value → a value ⇒ Overwrite ; RawConflict ⇒ CasConflict -/// head, head(key) → present ⇒ Head ; absent ⇒ HeadMiss (the 404 signal) -/// read, get → Read (all calls, hit or miss) -/// getStream → GetStream -/// remove, deleteExact → Delete (all outcomes) +/// head → present ⇒ Head ; absent ⇒ HeadMiss (the 404 signal) +/// read → Read (all calls, hit or miss) +/// stream → GetStream +/// remove → Delete (all outcomes) /// list → List -/// publish, publishBlob → Put +/// publish → Put /// -/// `Cas` therefore counts only what a caller sent as a `casPut`: the primitive cannot tell a -/// compare-and-set from any other replacement, so a migrated replace counts as `Overwrite`. That -/// distinction, and the `CAS*CompareSwap`/`CAS*GetStream` events it feeds, go when the legacy -/// surface does. +/// `Cas` has no current producer: the primitive `write` cannot tell a compare-and-set from any other +/// conditional replacement, so every conditional replace counts as `Overwrite`/`CasConflict`. Kept +/// for the `CAS*CompareSwap` events it still backs. enum class CasOp : uint8_t { Put = 0, @@ -87,17 +78,6 @@ void incrementCasEvent(CasNs ns, CasOp op); class InstrumentedBackend final : public Backend { public: - /// Unhide the base overloads this class's own declarations would otherwise shadow: the - /// convenience forms that omit Range/ObjectMeta/expected-token. - using Backend::get; - using Backend::getStream; - using Backend::head; - using Backend::list; - using Backend::probeSentinelRaw; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - explicit InstrumentedBackend(BackendPtr inner_) : inner(std::move(inner_)) {} /// Capability checks are deliberately uninstrumented: they do not represent storage operations. @@ -106,17 +86,11 @@ class InstrumentedBackend final : public Backend void checkConditionalWriteSingleAttemptSupport() override { inner->checkConditionalWriteSingleAttemptSupport(); } /// The typed sentinel probe is a diagnostic/authoritative read, not a routine storage operation — - /// deliberately uninstrumented (no ProfileEvent), like the capability checks above. MUST still be - /// forwarded explicitly: `Backend::probeSentinelRaw`'s generic default derives its classification from - /// THIS object's own `head`/`read` (virtual dispatch would otherwise resolve back to - /// `InstrumentedBackend`'s plain, non-typed overrides above), silently discarding whatever sharper - /// container/permission evidence the wrapped `inner` backend (e.g. `ObjectStorageBackend`'s S3/Local - /// classification) is able to provide. + /// deliberately uninstrumented (no ProfileEvent), like the capability checks above. SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override { return inner->probeSentinelRaw(key, access); } - SentinelProbeResult probeSentinelRaw(const String & key) override { return inner->probeSentinelRaw(key); } /// Delegate the read and count it after the inner call succeeds or returns absent. Exceptions /// propagate unchanged and therefore do not produce a separate outcome event. @@ -171,70 +145,6 @@ class InstrumentedBackend final : public Backend incrementCasEvent(classifyCasNs(key), CasOp::GetStream); return result; } - std::optional getStream(const String & key, Range range) override - { - auto result = inner->getStream(key, range); - incrementCasEvent(classifyCasNs(key), CasOp::GetStream); - return result; - } - - /// ---- The legacy surface, forwarded AS legacy ---- - /// - /// Not inherited from `Backend`: its forwarder would call the primitive on THIS object, so the - /// inner backend would receive a primitive and any legacy override it carries -- which is how - /// almost every fault injection in the test suite is written -- would never run. - std::optional get(const String & key, Range range) override - { - auto result = inner->get(key, range); - incrementCasEvent(classifyCasNs(key), CasOp::Read); - return result; - } - - HeadResult head(const String & key) override - { - HeadResult result = inner->head(key); - incrementCasEvent(classifyCasNs(key), result.exists ? CasOp::Head : CasOp::HeadMiss); - return result; - } - - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - PutResult result = inner->putIfAbsent(key, bytes, meta); - incrementCasEvent(classifyCasNs(key), result.outcome == PutOutcome::Done ? CasOp::Put : CasOp::PutDeduplicated); - return result; - } - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, - const ObjectMeta & meta) override - { - PutResult result = inner->putOverwrite(key, bytes, expected, meta); - incrementCasEvent(classifyCasNs(key), result.outcome == PutOutcome::Done ? CasOp::Overwrite : CasOp::CasConflict); - return result; - } - - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override - { - CasResult result = inner->casPut(key, bytes, expected, meta); - incrementCasEvent(classifyCasNs(key), result.outcome == CasOutcome::Committed ? CasOp::Cas : CasOp::CasConflict); - return result; - } - - DeleteOutcome deleteExact(const String & key, const Token & token) override - { - DeleteOutcome outcome = inner->deleteExact(key, token); - incrementCasEvent(classifyCasNs(key), CasOp::Delete); - return outcome; - } - - ListPage list(const String & prefix, const String & cursor, size_t limit) override - { - ListPage page = inner->list(prefix, cursor, limit); - incrementCasEvent(classifyCasNs(prefix), CasOp::List); - return page; - } - - void publishBlob(const BlobPublishRequest & request) override; /// Count one successful physical blob publication after delegating exactly once. The backend has /// no lifecycle reason to classify here; decision diagnostics remain with the writer. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index 25575c709a08..a1f08aa2ec17 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -1,7 +1,6 @@ #include -#include -#include +#include #include #include #include @@ -18,6 +17,7 @@ #include #include +#include #include #include "config.h" @@ -30,6 +30,14 @@ #include #include +namespace ProfileEvents +{ + extern const Event CASConditionalWriteAttempts; + extern const Event CASConditionalWriteCommitted; + extern const Event CASConditionalWriteDefiniteFailure; + extern const Event CASConditionalWriteUnresolved; +} + namespace DB { namespace ErrorCodes @@ -45,6 +53,76 @@ namespace ErrorCodes namespace DB::Cas { +namespace +{ + +/// Outcome of ONE HTTP attempt at a CAS conditional write, for the ProfileEvents below only -- +/// distinct from `detail::ConditionalWriteOutcome`, which the caller (`nativeConditionalPut`) acts on. +/// - Committed: the attempt's own request completed successfully (2xx). +/// - DefiniteFailure: a synchronous rejection that PROVES the request was never applied server-side +/// -- a WHITELISTED malformed-request / entity-too-large / access-denied error ONLY. +/// - Unresolved: everything else -- a lost precondition, a client-side timeout, a connection loss, a +/// 5xx, or any error this classifier does not recognize. +enum class CasWriteOutcome : uint8_t +{ + Committed, + DefiniteFailure, + Unresolved, +}; + +/// The exception path: classify what `buf.finalize()` threw for ONE CAS conditional-write HTTP +/// attempt. Never rethrows, never touches counters. +CasWriteOutcome classifyConditionalWriteResult([[maybe_unused]] const std::exception & e) +{ +#if USE_AWS_S3 + /// `PreconditionFailed`/`NoSuchKey` (a lost If-None-Match/If-Match), any 5xx + /// (InternalError/ServiceUnavailable/SlowDown/RequestTimeout), and any S3 error this function does + /// not recognize all fall through to the fail-safe default below: Unresolved. Only the WHITELIST + /// below proves the request was never applied. + if (const auto * s3e = dynamic_cast(&e)) + { + if (S3::isMalformedRequestError(*s3e) || S3::isEntityTooLargeError(*s3e) || S3::isAccessDeniedError(*s3e)) + return CasWriteOutcome::DefiniteFailure; + } +#endif + /// Poco::Net::NetException (connection loss) / Poco::TimeoutException (client-side timeout) and + /// every other error type: the request's fate is unproven -- fail toward "resolve before + /// reissuing", never toward a false DefiniteFailure. + return CasWriteOutcome::Unresolved; +} + +/// The success path: `buf.finalize()` returned without throwing. Always Committed -- kept as a named, +/// counted entry point so both paths of a classify-then-record call site read the same way. +constexpr CasWriteOutcome classifyConditionalWriteResult() +{ + return CasWriteOutcome::Committed; +} + +/// Records the start of one HTTP attempt for a CAS conditional write (the attempts counter). +void recordConditionalWriteAttemptStarted() +{ + ProfileEvents::increment(ProfileEvents::CASConditionalWriteAttempts); +} + +/// Records one attempt's terminal outcome (the per-class outcome counters). +void recordConditionalWriteOutcome(CasWriteOutcome outcome) +{ + switch (outcome) + { + case CasWriteOutcome::Committed: + ProfileEvents::increment(ProfileEvents::CASConditionalWriteCommitted); + return; + case CasWriteOutcome::DefiniteFailure: + ProfileEvents::increment(ProfileEvents::CASConditionalWriteDefiniteFailure); + return; + case CasWriteOutcome::Unresolved: + ProfileEvents::increment(ProfileEvents::CASConditionalWriteUnresolved); + return; + } +} + +} + ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_, bool single_attempt_control_plane_, uint64_t attempt_timeout_ms_) : object_storage(std::move(object_storage_)) @@ -54,7 +132,7 @@ ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mod , emu_root(object_storage->getCommonKeyPrefix()) { if (mode == Mode::Native && object_storage->conditionalOpsUseGenerationTokens()) - native_token_type = TokenType::Generation; + native_token_type = Dialect::Generation; } /// See Backend::checkPoolPreconditions. Only the Native, generation-dialect (GCS) combination has @@ -66,7 +144,7 @@ ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mod /// mount proceeds with a warning that names what was not verified and how the operator can verify it. void ObjectStorageBackend::checkPoolPreconditions() { - if (mode != Mode::Native || native_token_type != TokenType::Generation) + if (mode != Mode::Native || native_token_type != Dialect::Generation) return; const auto versioned = object_storage->isBucketVersioningEnabled(); @@ -97,7 +175,7 @@ void ObjectStorageBackend::checkPoolPreconditions() /// DELETE, and nothing else in the mount path proves it. void ObjectStorageBackend::checkSkipAccessCheckSupport() { - if (mode != Mode::Native || native_token_type != TokenType::Generation) + if (mode != Mode::Native || native_token_type != Dialect::Generation) return; throw Exception(ErrorCodes::NOT_IMPLEMENTED, @@ -137,7 +215,7 @@ void ObjectStorageBackend::checkConditionalWriteSingleAttemptSupport() /// Native helpers /// ========================================================================================= -bool ObjectStorageBackend::isValidTokenValue(TokenType type, const String & value) +bool ObjectStorageBackend::isValidTokenValue(Dialect type, const String & value) { return isIncarnationValue(type, value); } @@ -172,7 +250,7 @@ std::optional ObjectStorageBackend::nativeHead( /// coverage. Unit tests cover the emulated semantics, the typed exception path, and this classifier /// through the test-only `detail` declaration. #if USE_AWS_S3 -PutOutcome detail::finalizeConditionalWrite(WriteBuffer & buf) +detail::ConditionalWriteOutcome detail::finalizeConditionalWrite(WriteBuffer & buf) { try { @@ -183,38 +261,38 @@ PutOutcome detail::finalizeConditionalWrite(WriteBuffer & buf) if (e.isPreconditionFailed() || e.getExceptionName() == "NoSuchKey" || e.getS3ErrorCode() == Aws::S3::S3Errors::NO_SUCH_KEY) - return PutOutcome::PreconditionFailed; + return ConditionalWriteOutcome::PreconditionLost; throw; } - return PutOutcome::Done; + return ConditionalWriteOutcome::Applied; } #endif /// Build-dispatching shim for the write paths below: without the AWS SDK there is no S3Exception /// to classify, so the errors of finalize simply propagate. -static PutOutcome finalizeConditionalWrite(WriteBuffer & buf) +static detail::ConditionalWriteOutcome finalizeConditionalWrite(WriteBuffer & buf) { #if USE_AWS_S3 return detail::finalizeConditionalWrite(buf); #else buf.finalize(); - return PutOutcome::Done; + return detail::ConditionalWriteOutcome::Applied; #endif } /// Instrument the same single `finalize` call used by both Native write paths without changing their -/// `Done`/`PreconditionFailed`-or-rethrow contract. A classified precondition loss is `Unresolved`, -/// not `Committed` or a definite exception, because the response does not prove who created or -/// replaced the object; the higher-level request controller may then resolve it with exact-key state. -static PutOutcome finalizeConditionalWriteInstrumented(WriteBuffer & buf) +/// Applied/PreconditionLost-or-rethrow contract. A classified precondition loss is `Unresolved`, not +/// `Committed` or a definite exception, because the response does not prove who created or replaced +/// the object -- the caller's own retry loop resolves it with exact-key state. +static detail::ConditionalWriteOutcome finalizeConditionalWriteInstrumented(WriteBuffer & buf) { recordConditionalWriteAttemptStarted(); try { - const PutOutcome legacy = finalizeConditionalWrite(buf); + const detail::ConditionalWriteOutcome outcome = finalizeConditionalWrite(buf); recordConditionalWriteOutcome( - legacy == PutOutcome::Done ? classifyConditionalWriteResult() : CasWriteOutcome::Unresolved); - return legacy; + outcome == detail::ConditionalWriteOutcome::Applied ? classifyConditionalWriteResult() : CasWriteOutcome::Unresolved); + return outcome; } catch (const std::exception & e) { @@ -232,7 +310,7 @@ std::expected ObjectStorageBackend::nativeConditio auto buf = object_storage->writeObject( StoredObject(key), WriteMode::Rewrite, /*attributes=*/std::nullopt, DBMS_DEFAULT_BUFFER_SIZE, ws); buf->write(bytes.data(), bytes.size()); - if (finalizeConditionalWriteInstrumented(*buf) == PutOutcome::PreconditionFailed) + if (finalizeConditionalWriteInstrumented(*buf) == detail::ConditionalWriteOutcome::PreconditionLost) return std::unexpected(RawConflict{}); /// The response's own value for what it just wrote, normalized and otherwise untouched. An S3 @@ -242,11 +320,6 @@ std::expected ObjectStorageBackend::nativeConditio return normalizeTokenValue(buf->getResultObjectETag().value_or(String{})); } -namespace -{ - -} - /// True when an exception from a read means "the KEY is simply not there". /// Two surfaces: /// 1. S3/RustFS: `S3Exception` with `S3Errors::NO_SUCH_KEY` (the modeled enum — the primary @@ -285,34 +358,6 @@ static String readWholeObject(IObjectStorage & object_storage, const String & pa return content; } -/// Open a forward-only stream over `range` of the object at `path`, positioned at the window's first -/// byte and bounded to its last. Nothing is materialized whole: the caller reads at its own pace. An -/// offset at or past EOF yields an empty stream rather than an error. -static std::unique_ptr openObjectRangedStream(IObjectStorage & object_storage, const String & path, Range range, - uint64_t known_size = 0) -{ - auto buf = object_storage.readObject( - StoredObject(path), casSizedReadSettings(getReadSettings(), known_size), /*read_hint=*/std::nullopt); - if (range.whole()) - return buf; - - /// `seek` past the object size may throw depending on the storage, so fail-close against the known - /// size before touching the buffer position. A caller-supplied size avoids another metadata round - /// trip; zero means that the size is unknown and must be fetched. - const uint64_t object_size = known_size != 0 ? known_size - : object_storage.getObjectMetadata(path, /*with_tags=*/false).size_bytes; - if (range.offset >= object_size) - return std::make_unique(std::string_view{}); - - /// `setReadUntilPosition` is only a hint (LocalObjectStorage does not honor it), but for a returned - /// stream it is the only bound available — the caller drains to EOF, so a storage that DOES honor - /// the hint stops at the window end, and one that does not over-reads only the trailing bytes. - if (range.length.has_value()) - buf->setReadUntilPosition(range.offset + *range.length); - buf->seek(static_cast(range.offset), SEEK_SET); - return buf; -} - ReadSettings casSizedReadSettings(const ReadSettings & base, uint64_t known_size) { if (known_size == 0) @@ -428,7 +473,7 @@ String ObjectStorageBackend::emuRead(const String & key) const return readWholeObject(*object_storage, emuPath(key)); } -Token ObjectStorageBackend::emuWrite(const String & key, const String & bytes) +String ObjectStorageBackend::emuWrite(const String & key, const String & bytes) { auto buf = object_storage->writeObject(StoredObject(emuPath(key)), WriteMode::Rewrite); buf->write(bytes.data(), bytes.size()); @@ -505,13 +550,13 @@ void ObjectStorageBackend::emuPublishBlobAtomically(const String & key, const St ++existing_token_state->second.second; } -Token ObjectStorageBackend::emuObserveToken(const String & key) +String ObjectStorageBackend::emuObserveToken(const String & key) { const auto metadata = object_storage->tryGetObjectMetadata(emuPath(key), /*with_tags=*/false); return emuMintToken(key, metadata ? metadata->etag : String{}, /*just_wrote=*/false); } -Token ObjectStorageBackend::emuMintToken(const String & key, const String & etag, bool just_wrote) +String ObjectStorageBackend::emuMintToken(const String & key, const String & etag, bool just_wrote) { emuPruneTokenState(emuNowNs()); @@ -540,14 +585,13 @@ Token ObjectStorageBackend::emuMintToken(const String & key, const String & etag /// tokens, so bump a small per-key disambiguator (mtime-quantum guard, triage §3.18 19c step 4). if (just_wrote) ++it->second.second; - const String value = it->second.second == 0 ? etag : etag + "#" + std::to_string(it->second.second); - return Token{value, TokenType::Emulated}; + return it->second.second == 0 ? etag : etag + "#" + std::to_string(it->second.second); } /// The etag advanced (or this key is seen for the first time): the bare etag is the token, and any /// previous disambiguator is dropped — a genuinely new incarnation starts clean. emu_token_state[key] = {etag, 0}; - return Token{etag, TokenType::Emulated}; + return etag; } /// ========================================================================================= @@ -614,57 +658,10 @@ std::optional ObjectStorageBackend::readUnder( return std::nullopt; throw; } - raw.value = emuObserveToken(key).value; + raw.value = emuObserveToken(key); return raw; } -std::optional ObjectStorageBackend::getStream(const String & key, Range range) -{ - if (mode == Mode::Native) - { - auto hr = nativeHead(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); - if (!hr) - return std::nullopt; - - /// The object may be deleted between the HEAD above and the stream open below. Honor the - /// `optional` contract on a not-found signal; any other error (network, auth, corruption) - /// propagates unchanged — fail-closed by construction. - GetStreamResult sr; - try - { - sr.stream = openObjectRangedStream(*object_storage, key, range, hr->size); - } - catch (const std::exception & e) - { - if (isObjectNotFound(e)) - return std::nullopt; - throw; - } - sr.token = legacyMintObserved(key, hr->value); - return sr; - } - - std::lock_guard lock(emu_mutex); - if (!emuExists(key)) - return std::nullopt; - - /// The emulated path holds emu_mutex across the exists-check and the stream open, matching `read`. - /// External deletion still converts to nullopt rather than escaping as an unexplained exception. - GetStreamResult sr; - try - { - sr.stream = openObjectRangedStream(*object_storage, emuPath(key), range); - } - catch (const std::exception & e) - { - if (isObjectNotFound(e)) - return std::nullopt; - throw; - } - sr.token = emuObserveToken(key); - return sr; -} - std::unique_ptr ObjectStorageBackend::stream(const String & key, TransportAccess &) { /// No HEAD: this is ONE request, and the caller reserved one. Opening an object-storage buffer @@ -723,7 +720,7 @@ std::optional ObjectStorageBackend::headUnder( /// traversed by system.remote_data_paths) as a file and a later body read throws EISDIR. if (!metadata) return std::nullopt; - return RawMeta{metadata->size_bytes, emuObserveToken(key).value}; + return RawMeta{metadata->size_bytes, emuObserveToken(key)}; } /// See Backend::probeSentinelRaw / CasBackend.h's ProbeOutcome for the semantics this classifies. @@ -732,11 +729,6 @@ SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key, T return probeSentinelUnder(key, controlPlaneProfile(), attempt_timeout_ms); } -SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key) -{ - return probeSentinelUnder(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); -} - SentinelProbeResult ObjectStorageBackend::probeSentinelUnder( const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) { @@ -807,7 +799,7 @@ WriteSettings ObjectStorageBackend::conditionalWriteSettings() const { WriteSettings ws; ws.object_storage_request_mode = ObjectStorageRequestMode::NativeConditional; - if (native_token_type == TokenType::Generation) + if (native_token_type == Dialect::Generation) ws.s3_force_single_part_upload = true; ws.s3_check_objects_after_upload_override = false; /// Exactly one attempt at the WriteBufferFromS3 layer too: makeSinglepartUpload/ @@ -857,11 +849,11 @@ std::expected ObjectStorageBackend::write( { if (!exists) return std::unexpected(RawConflict{}); - if (!tokenMatches(emuObserveToken(key), Token{*expected_value, TokenType::Emulated})) + if (emuObserveToken(key) != *expected_value) return std::unexpected(RawConflict{}); } - return emuWrite(key, bytes).value; + return emuWrite(key, bytes); } void ObjectStorageBackend::publish(const BlobPublishRequest & request, TransportAccess &) @@ -976,7 +968,7 @@ Backend::RawRemoval ObjectStorageBackend::removeUnder( std::lock_guard lock(emu_mutex); if (!emuExists(key)) return RawRemoval::Gone; - if (!tokenMatches(emuObserveToken(key), Token{expected_value, TokenType::Emulated})) + if (emuObserveToken(key) != expected_value) return RawRemoval::Mismatch; object_storage->removeObjectIfExists(StoredObject(emuPath(key))); @@ -1034,7 +1026,7 @@ Backend::RawListPage ObjectStorageBackend::listUnder( lk.key = child->relative_path.substr(strip.size()); lk.size = child->metadata ? child->metadata->size_bytes : 0; if (child->metadata) - lk.value = emuMintToken(lk.key, child->metadata->etag, /*just_wrote=*/false).value; + lk.value = emuMintToken(lk.key, child->metadata->etag, /*just_wrote=*/false); all.push_back(std::move(lk)); } std::sort(all.begin(), all.end(), [](const RawListedKey & a, const RawListedKey & b) { return a.key < b.key; }); @@ -1077,8 +1069,7 @@ Backend::RawListPage ObjectStorageBackend::listUnder( /// empty-etag gate lives in tokenForList; whether the value it passes IS an incarnation is /// judged where the answer can be acted on, not here. if (child->metadata) - if (const auto token = tokenForList(child->metadata->etag)) - lk.value = token->value; + lk.value = tokenForList(child->metadata->etag); if (page.keys.size() == limit) { @@ -1091,61 +1082,5 @@ Backend::RawListPage ObjectStorageBackend::listUnder( return page; } -/// ========================================================================================= -/// The legacy surface — see the declarations for why these are not the base's forwarders. -/// Every one of them issues its request under the storage's own retry profile, and mints through -/// the base's legacy mint so a malformed response is judged in exactly one place. -/// ========================================================================================= - -std::optional ObjectStorageBackend::get(const String & key, Range range) -{ - if (!range.whole()) - throw Exception(ErrorCodes::NOT_IMPLEMENTED, "CAS backend: a ranged get is retired; read the object whole"); - - auto raw = readUnder(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); - if (!raw) - return std::nullopt; - return GetResult{std::move(raw->bytes), legacyMintObserved(key, std::move(raw->value)), {}}; -} - -HeadResult ObjectStorageBackend::head(const String & key) -{ - auto raw = headUnder(key, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); - if (!raw) - return {}; - return HeadResult{true, raw->size, legacyMintObserved(key, std::move(raw->value)), {}}; -} - -ListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit) -{ - auto raw = listUnder(prefix, cursor, limit, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0); - - ListPage page; - page.next_cursor = std::move(raw.next_cursor); - page.keys.reserve(raw.keys.size()); - for (auto & k : raw.keys) - { - std::optional token; - if (k.value) - token = legacyMintObserved(k.key, std::move(*k.value)); - page.keys.push_back(ListedKey{std::move(k.key), k.size, std::move(token)}); - } - return page; -} - -DeleteOutcome ObjectStorageBackend::deleteExact(const String & key, const Token & token) -{ - if (legacyTokenIsForeign(key, token)) - return DeleteOutcome{DeleteOutcome::Kind::TokenMismatch, false}; - - switch (removeUnder(key, token.value, ObjectStorageRetryProfile::Default, /*timeout_ms=*/0)) - { - case RawRemoval::Removed: return {DeleteOutcome::Kind::Deleted, false}; - case RawRemoval::Gone: return {DeleteOutcome::Kind::NotFound, false}; - case RawRemoval::Mismatch: return {DeleteOutcome::Kind::TokenMismatch, false}; - case RawRemoval::DeleteMarker: return {DeleteOutcome::Kind::Deleted, true}; - } - UNREACHABLE(); -} } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index 1c491058802d..e0a4d95f971f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -18,16 +18,22 @@ namespace DB::Cas constexpr uint64_t CAS_FOLD_READ_SLACK_BYTES = 4096; ReadSettings casSizedReadSettings(const ReadSettings & base, uint64_t known_size); -#if USE_AWS_S3 namespace detail { +/// Outcome of ONE conditional-write attempt at the transport level: did the store apply it, or did it +/// refuse the precondition (If-None-Match/If-Match)? Distinct from `Backend::RawConflict` because a +/// caller inside this TU needs to know WHICH of "committed" or "precondition lost" happened before it +/// decides what to return, whereas `RawConflict` only ever means the latter. +enum class ConditionalWriteOutcome : uint8_t { Applied, PreconditionLost }; + +#if USE_AWS_S3 /// Finalize a conditional write (the condition rode on the buffer's WriteSettings) and map a /// precondition loss to an OUTCOME — anything else propagates. This is the classifier for the /// typed `S3Exception` signal; exposed here for unit tests only — production callers go through /// `ObjectStorageBackend`. See the definition for the exact matching rules. -PutOutcome finalizeConditionalWrite(WriteBuffer & buf); -} +ConditionalWriteOutcome finalizeConditionalWrite(WriteBuffer & buf); #endif +} /// Production Backend over IObjectStorage. /// @@ -48,14 +54,6 @@ PutOutcome finalizeConditionalWrite(WriteBuffer & buf); class ObjectStorageBackend final : public Backend { public: - /// Unhide the base overloads this class's own declarations would otherwise shadow: the - /// convenience forms that omit `Range`, and the keyed primitives that share a legacy name. - using Backend::get; - using Backend::getStream; - using Backend::head; - using Backend::list; - using Backend::probeSentinelRaw; - enum class Mode { Native, EmulatedSingleProcess }; /// Construct a backend over `object_storage`. Native mode uses the storage's conditional @@ -104,23 +102,6 @@ class ObjectStorageBackend final : public Backend /// Ask the storage to re-acquire credentials through its refresh callback. bool refreshCredentials() override { return object_storage->tryRefreshCredentialsViaCallback(); } - /// ---- The legacy surface, kept for the migration window ---- - /// - /// Not inherited from `Backend`: its forwarders would issue these requests under the profile the - /// keyed primitives use, which on a writable Native mount is SingleAttempt. That is right for a - /// primitive -- `CasRequests` is the retry loop around it -- and wrong for a legacy caller, which - /// has no loop at all and would lose the storage's own retries on the first blip. These keep the - /// storage's default profile until their callers move onto the engine, and are deleted at the - /// lock. Each mints its `Token` through the base's legacy mint, so a malformed response is - /// judged in exactly one place. - std::optional get(const String & key, Range range) override; - HeadResult head(const String & key) override; - ListPage list(const String & prefix, const String & cursor, size_t limit) override; - DeleteOutcome deleteExact(const String & key, const Token & token) override; - SentinelProbeResult probeSentinelRaw(const String & key) override; - /// Open a forward-only ranged stream for a write-once object. See `Backend::getStream` for why it - /// is not a forwarder. - std::optional getStream(const String & key, Range range) override; /// S3 ETags are content-derived and surfaced in list responses — TRUE for ETag-token Native /// and EmulatedSingleProcess modes. FALSE on a generation-token store (GCS): the XML LIST /// surfaces MD5-style ETags in the response BODY, which the header-level response adaptation @@ -128,7 +109,7 @@ class ObjectStorageBackend final : public Backend /// `If-Match` token; generation stores deliberately omit it and make GC re-read each shard. /// Consumers already treat absent list tokens as Read/fail-closed (GC discover re-reads every /// shard — a cost, not a correctness change). - bool supportsListTokens() const override { return native_token_type != TokenType::Generation; } + bool supportsListTokens() const override { return native_token_type != Dialect::Generation; } /// Pool-level precondition: on a Native, generation-dialect (GCS) backend, reject the pool when /// object versioning is verified ENABLED; warn and continue when the probe cannot answer — see @@ -154,13 +135,13 @@ class ObjectStorageBackend final : public Backend /// `ContainerAbsent` if it is gone -- then the key. SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override; - /// The token kind this backend's object storage mints: TokenType::ETag for AWS-compatible - /// stores, TokenType::Generation when the storage mints GCS generations (the + /// The token kind this backend's object storage mints: Dialect::ETag for AWS-compatible + /// stores, Dialect::Generation when the storage mints GCS generations (the /// generation rides the ETag plumbing; the VALUE stays opaque either way). - TokenType nativeTokenType() const { return native_token_type; } - void setNativeTokenTypeForTest(TokenType t) { native_token_type = t; } + Dialect nativeTokenType() const { return native_token_type; } + void setNativeTokenTypeForTest(Dialect t) { native_token_type = t; } - /// ---- Token policy (single source of truth; see the .cpp) ---- + /// ---- Etag value policy (single source of truth; see the .cpp) ---- /// A GCS generation reaches this layer through the AWS SDK's ETag field, which the HTTP boundary /// fills with an ETag-shaped — that is, quoted — value. A generation is a number, and quotes are /// transport syntax that must not enter CAS protocol state, where token values are compared for @@ -172,43 +153,29 @@ class ObjectStorageBackend final : public Backend /// corrupt the AWS-compatible path. String normalizeTokenValue(const String & etag) const { - if (native_token_type != TokenType::Generation) + if (native_token_type != Dialect::Generation) return etag; if (etag.size() >= 2 && etag.front() == '"' && etag.back() == '"') return etag.substr(1, etag.size() - 2); return etag; } - /// The `Token` form of an observed ETag/generation: normalized, and stamped with this backend's - /// native dialect. The transport itself deals in bare values; this is the normalize-and-stamp - /// step on its own, with `tokenForList` as its LIST-side sibling. - Token tokenForHead(const String & etag) const - { - return Token{normalizeTokenValue(etag), native_token_type}; - } - - /// The token to surface for a LISTED key: present iff this backend surfaces per-key list tokens - /// (supportsListTokens — FALSE on a generation store, where a list-derived token is a poisoned - /// If-Match) AND the listing carried a non-empty etag. Matches what tokenForHead would return. - std::optional tokenForList(const String & etag) const + /// The normalized incarnation value to surface for a LISTED key: present iff this backend surfaces + /// per-key list values (supportsListTokens — FALSE on a generation store, where a list-derived + /// value is a poisoned If-Match) AND the listing carried a non-empty etag. Matches what + /// `normalizeTokenValue` would return for the same etag. + std::optional tokenForList(const String & etag) const { if (!supportsListTokens() || etag.empty()) return std::nullopt; - return Token{etag, native_token_type}; - } - - /// Whether an observed incarnation token satisfies an expected one: exact identity (value AND - /// type). Every conditional compare in this backend goes through here. - static bool tokenMatches(const Token & observed, const Token & expected) - { - return observed == expected; + return normalizeTokenValue(etag); } /// The per-dialect grammar a response value must meet to be an incarnation. Generation: canonical /// positive decimal AFTER the SDK ETag-field quote strip (no leading zero, not "0" — zero is the /// dialect's absence sentinel). ETag: non-empty, not "*" after trimming whitespace, no comma (a /// list matches any member). Emulated: non-empty. - static bool isValidTokenValue(TokenType type, const String & value); + static bool isValidTokenValue(Dialect type, const String & value); /// Settings for a Native COMPARE/CREATE write (create-if-absent, compare-and-set): mark the request /// conditional, make exactly one attempt at every retry layer, skip the racy post-upload @@ -224,7 +191,7 @@ class ObjectStorageBackend final : public Backend private: const ObjectStoragePtr object_storage; const Mode mode; - TokenType native_token_type = TokenType::ETag; + Dialect native_token_type = Dialect::ETag; /// See the constructor: what the READ-class requests (read, head, list, remove) carry. const bool single_attempt_control_plane; const uint64_t attempt_timeout_ms; @@ -274,14 +241,6 @@ class ObjectStorageBackend final : public Backend /// caller's to resolve, not this seam's to refuse. std::expected nativeConditionalPut(const String & key, const String & bytes, const WriteSettings & ws); - /// §3.18 №19 hardening: whether `t` is the dialect this backend itself mints (native_token_type - /// for Native mode, always TokenType::Emulated for EmulatedSingleProcess). Every conditional - /// mutation checks this BEFORE touching the wire (Native forwards only Token::value as the - /// If-Match/removeObjectIfTokenMatches argument, blind to Token::type) or comparing values - /// (Emulated) — a foreign-dialect token is rejected locally rather than trusted to the remote - /// backend, or to a value-space that was never designed to discriminate it. - bool mintingTypeMatches(TokenType t) const { return t == (mode == Mode::Native ? native_token_type : TokenType::Emulated); } - /// ---- Emulated helpers (caller holds emu_mutex) ---- /// /// EmulatedSingleProcess resolves logical keys under the object storage's common key prefix (its @@ -296,7 +255,7 @@ class ObjectStorageBackend final : public Backend String emuRead(const String & key) const; /// Write a body as the new incarnation of `key` and return its freshly minted token (the /// object's own post-write etag — see emuMintToken). - Token emuWrite(const String & key, const String & bytes); + String emuWrite(const String & key, const String & bytes); /// Write a complete blob body to a sibling temporary local object, then atomically replace `key` /// and advance any existing same-ETag disambiguator. A failure before the rename leaves the old /// destination and its token state untouched and cleans the temporary. @@ -307,7 +266,7 @@ class ObjectStorageBackend final : public Backend void emuPublishBlobAtomically(const String & key, const String & envelope, ReadBuffer & payload, uint64_t payload_size); /// Return the current emulated token for a key we just read/HEAD'd, reflecting its on-disk etag — /// does NOT advance the same-etag disambiguator (that only applies to a just-completed write). - Token emuObserveToken(const String & key); + String emuObserveToken(const String & key); uint64_t emuNowNs() const; /// Examine a fixed number of oldest deleted-state records, expiring only an exact current match. void emuPruneTokenState(uint64_t now_ns); @@ -317,7 +276,7 @@ class ObjectStorageBackend final : public Backend /// declaration). An empty `etag` means the storage could not identify the object at all, and /// there is nothing to invent from: a just-completed write cannot be attributed /// (`CAS_WRITE_UNATTRIBUTED`) and an observation has no incarnation to report (`CORRUPTED_DATA`). - Token emuMintToken(const String & key, const String & etag, bool just_wrote); + String emuMintToken(const String & key, const String & etag, bool just_wrote); }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp index 091b3ae39754..625c04af8b41 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp @@ -21,7 +21,7 @@ namespace /// the probe has always thrown for it — everything else propagates unchanged. Every `remove` the battery /// issues against a live incarnation goes through this: a store that ignores the delete precondition AND /// mints delete markers must still be reported with this message, not the engine's terse one. -Removal removeOrReportDeleteMarker(CasOperation & op, const String & key, const Incarnation & seen) +Removal removeOrReportDeleteMarker(CasOperation & op, const String & key, const Etag & seen) { try { @@ -54,14 +54,14 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) auto cleanup = [&]() noexcept { // Skip the remove when HEAD says the key is already gone (the happy path: the battery's own - // delete already ran). `Incarnation` can only be minted from an actual HEAD/read observation, so + // delete already ran). `Etag` can only be minted from an actual HEAD/read observation, so // an unconditional "delete with whatever precondition" this backend never saw is not // constructible here — the gate below is the only way to reach `remove` at all. try { const auto h = op.head(key, Retry::standard()); if (h) - op.remove(key, h->incarnation, Retry::standard()); + op.remove(key, h->etag, Retry::standard()); } catch (...) {} /// NOLINT(bugprone-empty-catch) }; @@ -69,13 +69,13 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) try { // ---- Step 1: create fresh -> Committed; read-after-write returns the bytes. ---- - Incarnation t1 = [&] + Etag t1 = [&] { WriteResult r = op.create(key, "probe-v1", Retry::standard()); if (!std::holds_alternative(r)) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "CasProbe: create on a fresh key did not commit — backend is unexpectedly occupied or broken"); - return std::get(r).incarnation; + return std::get(r).etag; }(); { const auto g = op.read(key, Retry::standard()); @@ -101,16 +101,16 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) // ---- Step 3: replace against the CURRENT incarnation (t1) -> Committed; incarnation changed; // bytes replaced. Every "wrong incarnation" step below reuses THIS key's own prior - // incarnations (never a synthetic value) — an `Incarnation` is minted only from an + // incarnations (never a synthetic value) — an `Etag` is minted only from an // actual backend observation, so there is no other way to name one that is // guaranteed wrong yet dialect-valid. ---- - Incarnation t2 = [&] + Etag t2 = [&] { WriteResult r = op.replace(key, "probe-v2", t1, Retry::standard()); if (!std::holds_alternative(r)) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "CasProbe: replace with the correct incarnation was rejected — backend does not accept a valid overwrite"); - Incarnation next = std::get(r).incarnation; + Etag next = std::get(r).etag; if (next == t1) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "CasProbe: replace succeeded but did not mint a new incarnation — an incarnation must " @@ -158,7 +158,7 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) // ---- Step 6: list(probe_prefix) contains the probe key (list-after-write). ---- { bool found = false; - op.forEachListedKey(probe_prefix, [&](const KeyEntry & listed) -> bool + op.forEachListedKey(probe_prefix, [&](const ListedKey & listed) -> bool { if (listed.key != key) return true; @@ -183,7 +183,7 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "CasProbe: remove succeeded (Removed) but the object is still readable — backend delete is not effective"); bool still_listed = false; - op.forEachListedKey(probe_prefix, [&](const KeyEntry & listed) -> bool + op.forEachListedKey(probe_prefix, [&](const ListedKey & listed) -> bool { if (listed.key != key) return true; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp index 536518abc7c1..4f22c85364f5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp @@ -16,12 +16,6 @@ namespace DB::Cas void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms) { - if (budget.max_attempts < 1) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: max_attempts must be at least 1 (got {}) — zero would let " - "putIfAbsentControlled return Unresolved without ever sending an attempt.", - budget.max_attempts); - /// Overflow-safe: `attempt_timeout_ms + lease_safety_margin_ms` could wrap uint64 for absurd config /// values, which would make the sum spuriously small and the inequality below pass when it should /// fail closed. Compare via subtraction against the (unsigned, so already non-negative) TTL instead @@ -33,37 +27,11 @@ void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_le "strictly less than the mount lease TTL ({} ms). A writable mount refuses to open with " "this budget.", budget.attempt_timeout_ms, budget.lease_safety_margin_ms, mount_lease_ttl_ms); - /// STRICTLY less, and the strictness is the load-bearing half. `attempt_timeout_ms > - /// operation_deadline_ms` is the obvious error — a single attempt cannot outlast the logical - /// operation it belongs to. EQUALITY is the subtle one, and it is worse than useless: the deadline - /// is captured as `now + operation_deadline_ms` and every pre-send gate below asks - /// `now + attempt_timeout_ms > deadline_ms`, so equal values collapse that to `now_2 > now_1` and - /// ONE elapsed millisecond between the two clock reads refuses the operation having sent NOTHING. - /// The resulting behaviour is "mostly works, occasionally refuses with nothing sent", decided by - /// the scheduler rather than by the budget — exactly the flakiness this validation exists to catch, - /// and observed three times in tests before it was forbidden. A caller that wants one attempt says - /// `max_attempts = 1`; the equality adds only the race. - if (!(budget.attempt_timeout_ms < budget.operation_deadline_ms)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: attempt_timeout_ms ({}) must be strictly less than " - "operation_deadline_ms ({}) — equality turns the pre-send gate into a wall-clock race that " - "refuses after a single elapsed tick, having sent nothing. Use max_attempts to bound the " - "number of attempts.", - budget.attempt_timeout_ms, budget.operation_deadline_ms); - if (!(budget.retry_initial_backoff_ms <= budget.retry_max_backoff_ms)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: retry_initial_backoff_ms ({}) must not exceed " - "retry_max_backoff_ms ({}) — the capped-exponential backoff cap cannot sit below its own " - "starting value. Set both to 0 to disable inter-attempt backoff.", - budget.retry_initial_backoff_ms, budget.retry_max_backoff_ms); - LOG_INFO(getLogger("CasRequestControl"), - "CAS request budget in effect: attempt_timeout_ms={} operation_deadline_ms={} max_attempts={} " - "lease_safety_margin_ms={} retry_initial_backoff_ms={} retry_max_backoff_ms={} " + LOG_INFO(getLogger("CasRequestBudget"), + "CAS request budget in effect: attempt_timeout_ms={} lease_safety_margin_ms={} " "(mount_lease_ttl_ms={} mount_renew_period_ms={})", - budget.attempt_timeout_ms, budget.operation_deadline_ms, budget.max_attempts, - budget.lease_safety_margin_ms, budget.retry_initial_backoff_ms, budget.retry_max_backoff_ms, - mount_lease_ttl_ms, mount_renew_period_ms); + budget.attempt_timeout_ms, budget.lease_safety_margin_ms, mount_lease_ttl_ms, mount_renew_period_ms); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h index 84c251b5c2af..ab6f6300666f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h @@ -8,62 +8,25 @@ namespace DB::Cas /// are what `CasMountRuntime::admit` measures a request against, and the three `recovery_retry_*` fields /// bound a whole ref-table recovery; see `validateCasRequestBudget` for the relationship a writable /// mount enforces at startup. -/// -/// The four fields below them -- `operation_deadline_ms`, `max_attempts` and the two inter-attempt -/// backoff bounds -- belong to the retiring `CasRequestController` and have no consumer in the request -/// contract, which expresses the same bounds as a `Retry` policy per call. They are kept only so the -/// controller and its tests still compile, and they go when it does; their doc comments still describe -/// the controller's own loop. struct CasRequestBudget { - /// Maximum client wait budgeted for one HTTP attempt. `CasRequestController` uses this ONLY as a - /// per-attempt scheduling check (an attempt is not started unless it could still finish inside the - /// operation deadline) — the actual socket-level wait is configured on the object storage's client - /// (the object storage backend's single-attempt client), not by this struct. + /// Maximum client wait budgeted for one HTTP attempt. The request contract reserves this before + /// every attempt it starts; the actual socket-level wait is configured on the object storage's + /// client (the object storage backend's single-attempt client), not by this struct. uint64_t attempt_timeout_ms = 5000; - /// Maximum wall-clock time for the COMPLETE logical operation — every attempt, every exact-key - /// resolution, and every inter-attempt backoff sleep — counted from the first call to - /// `putIfAbsentControlled`. A DURATION, not an absolute deadline: each call establishes its own - /// `now + operation_deadline_ms` bound. - /// - /// This deadline is the authoritative bound on how long a CAS conditional write keeps riding an S3 - /// disruption server-side before the caller sees an abort. 90s absorbs a ~60s object-store outage - /// with margin (see the arithmetic on `max_attempts` below) — PROVIDED the mount fence stays alive. - /// The fence, not this deadline, is - /// what binds under a TOTAL outage: lease renewals are conditional writes against the same store, - /// so when everything is unreachable the fence deadline freezes at `last_renew + mount_lease_ttl` - /// and `fence_ok` stops the loop ≈ TTL−attempt_timeout−margin (~23s) after the last successful - /// renewal — the required fail-closed behavior (never an attempt past the lease), not a - /// budget limitation. While renewals DO land (blips, throttling, partial outages — the runtime-owned - /// renewal worker keeps extending the fence deadline), the op is NOT bounded by - /// the lease TTL and rides the full deadline here. - uint64_t operation_deadline_ms = 90000; - /// Maximum number of controlled attempts for one logical operation (the first attempt counts as 1). - /// Sized so the operation deadline above — never this count — is what binds under the observed - /// failure shape (~3s adaptive first-attempt PUT timeout per failed attempt + capped-exponential - /// backoff): 16 attempts × ~3s + Σ backoff (0.2+0.4+0.8+1.6+3.2 + 10×5 = 56.2s) ≈ 104s > 90s. - uint32_t max_attempts = 16; /// Startup-only margin folded into `validateCasRequestBudget`'s inequality against the mount lease - /// TTL. Not consulted at runtime by the controller itself — the caller's `fence_ok` callback (backed - /// by the local write fence's own deadline) is what actually gates lease-relative timing per attempt. + /// TTL. Not consulted at runtime by the engine itself -- the caller's fence (backed by the local + /// write fence's own deadline) is what actually gates lease-relative timing per attempt. uint64_t lease_safety_margin_ms = 2000; - /// Inter-attempt backoff (`cas_s3_retry_initial_backoff_ms` / - /// `cas_s3_retry_max_backoff_ms`): the sleep before reissuing - /// after an ambiguous attempt whose resolve observed the key absent, capped exponential — - /// `initial · 2^(reissues-1)`, never above `retry_max_backoff_ms`. 0 disables backoff (immediate - /// reissue — the pre-backoff behavior, and what most exhaustion-path unit tests configure). The - /// controller checks the fence BEFORE every sleep and never sleeps past the operation deadline. - uint64_t retry_initial_backoff_ms = 200; - uint64_t retry_max_backoff_ms = 5000; /// Recovery-level retry (`CasRefLedger::ensureRefTableRecovered`): a whole ref-table recovery /// attempt (LIST + snapshot/log GETs + seal PUT) that fails with a transient NETWORK_ERROR is /// retried, with capped-exponential backoff, until this total wall-clock budget is spent — then the /// error propagates and the table's load fails for this touch (the `lazy_load_tables` database - /// setting makes the NEXT touch retry). This sits ON TOP of the per-request `operation_deadline_ms` - /// envelope above: one recovery attempt may itself burn ~90s inside a single seal PUT. Independent - /// of the mount-lease invariants validated in `validateCasRequestBudget` — not part of that - /// inequality set. + /// setting makes the NEXT touch retry). This sits ON TOP of each write's own `Retry` policy window + /// (`Retry::standard()`'s 90s): one recovery attempt may itself burn ~90s inside a single seal PUT. + /// Independent of the mount-lease invariants validated in `validateCasRequestBudget` — not part of + /// that inequality set. uint64_t recovery_retry_budget_ms = 120000; uint64_t recovery_retry_initial_backoff_ms = 1000; uint64_t recovery_retry_max_backoff_ms = 30000; @@ -71,16 +34,9 @@ struct CasRequestBudget /// Startup validation: a writable mount refuses to open with an inconsistent budget rather than /// silently falling back to an unbounded or unsafe retry policy. Throws -/// `BAD_ARGUMENTS` unless ALL hold: +/// `BAD_ARGUMENTS` unless: /// attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms -/// attempt_timeout_ms < operation_deadline_ms (STRICTLY — see below) -/// retry_initial_backoff_ms <= retry_max_backoff_ms /// -/// The middle one is strict on purpose. Equality does not mean "one attempt's worth of budget": the -/// deadline is captured as `now + operation_deadline_ms` and each pre-send gate asks -/// `now + attempt_timeout_ms > deadline_ms`, so equal values reduce it to `now_2 > now_1` and a single -/// elapsed millisecond refuses the operation having sent NOTHING. Bound the attempt COUNT with -/// `max_attempts`, never by starving the deadline. /// `mount_renew_period_ms` takes no part in the inequality (the renewer keeps the fence deadline /// refreshed well ahead of the TTL by construction) — it is accepted only so the effective-values log /// line records the full picture in one place. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp deleted file mode 100644 index 30f6a9f06475..000000000000 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.cpp +++ /dev/null @@ -1,860 +0,0 @@ -#include - -#include -#include -#include -#include - -#include "config.h" - -#if USE_AWS_S3 -#include -#endif - -#include -#include -#include -#include -#include - -namespace ProfileEvents -{ - extern const Event CASConditionalWriteAttempts; - extern const Event CASConditionalWriteCommitted; - extern const Event CASConditionalWriteDefiniteFailure; - extern const Event CASConditionalWriteUnresolved; - extern const Event CASConditionalWriteFenceLostPostWrite; -} - -namespace DB -{ -namespace ErrorCodes -{ - extern const int BAD_ARGUMENTS; - extern const int CORRUPTED_DATA; - extern const int LOGICAL_ERROR; - extern const int NETWORK_ERROR; - extern const int NOT_IMPLEMENTED; -} -} - -namespace DB::Cas -{ - -CasWriteOutcome classifyConditionalWriteResult([[maybe_unused]] const std::exception & e) -{ -#if USE_AWS_S3 - /// `PreconditionFailed`/`NoSuchKey` (a lost If-None-Match/If-Match — see - /// ObjectStorageBackend::finalizeConditionalWrite for the exact matching), any 5xx - /// (InternalError/ServiceUnavailable/SlowDown/RequestTimeout), and any S3 error this function does - /// not recognize all fall through to the fail-safe default below: Unresolved. Only the WHITELIST - /// below proves the request was never applied. - if (const auto * s3e = dynamic_cast(&e)) - { - if (S3::isMalformedRequestError(*s3e) || S3::isEntityTooLargeError(*s3e) || S3::isAccessDeniedError(*s3e)) - return CasWriteOutcome::DefiniteFailure; - } -#endif - /// Poco::Net::NetException (connection loss) / Poco::TimeoutException (client-side timeout) and - /// every other error type: the request's fate is unproven — fail toward "resolve before - /// reissuing, never toward a false - /// DefiniteFailure. - return CasWriteOutcome::Unresolved; -} - -void recordConditionalWriteAttemptStarted() -{ - ProfileEvents::increment(ProfileEvents::CASConditionalWriteAttempts); -} - -void recordConditionalWriteOutcome(CasWriteOutcome outcome) -{ - switch (outcome) - { - case CasWriteOutcome::Committed: - ProfileEvents::increment(ProfileEvents::CASConditionalWriteCommitted); - return; - case CasWriteOutcome::DefiniteFailure: - ProfileEvents::increment(ProfileEvents::CASConditionalWriteDefiniteFailure); - return; - case CasWriteOutcome::Unresolved: - ProfileEvents::increment(ProfileEvents::CASConditionalWriteUnresolved); - return; - } -} - -namespace -{ - -uint64_t steadyClockNowMs() -{ - return static_cast(std::chrono::duration_cast( - std::chrono::steady_clock::now().time_since_epoch()).count()); -} - -/// The default inter-attempt backoff sleep. NOT a race-fix sleep: it is deliberate, bounded, -/// fence-gated pacing of reissues toward a recovering object store, and it is injectable so tests -/// never wait on it. -void threadSleepMs(uint64_t ms) -{ - std::this_thread::sleep_for(std::chrono::milliseconds(ms)); -} - -/// Deterministic caller/local bugs a mutable conditional retry loop must surface immediately: -/// reissuing only replays the same failure and buries the root cause behind a retryable exception. -/// The set: -/// LOGICAL_ERROR — a local invariant violation -/// NOT_IMPLEMENTED — a deterministic mode or capability guard -/// BAD_ARGUMENTS — a deterministic encode/argument rejection (e.g. BAD_ARGUMENTS escaping -/// buildHeader's second, intended_ref-less encode) -/// CORRUPTED_DATA — integrity failure; retrying re-reads/re-streams the same bad bytes (the same -/// fail-fast rule the driver-side correctness markers enforce) -/// Fail-safe either way: a propagated exception is never a false Committed. -bool isDeterministicLocalFailure(int code) -{ - return code == ErrorCodes::LOGICAL_ERROR || code == ErrorCodes::NOT_IMPLEMENTED - || code == ErrorCodes::BAD_ARGUMENTS || code == ErrorCodes::CORRUPTED_DATA; -} - -enum class OverwriteGateClosure : uint8_t -{ - Open, - FenceOrLifecycleLost, - Cancelled, - Deadline, -}; - -struct OverwriteGateSample -{ - OverwriteGateClosure closure; - CasOverwriteStopCause stop_cause; -}; - -/// Sample the operation stop cause and clock exactly once. The returned closure has already applied -/// the protocol precedence; callers only map its position to `CasUnresolvedReason`. -OverwriteGateSample sampleOverwriteGate( - const CasOverwriteOperationContext & context, - const std::function & now_ms, - uint64_t required_time_ms) -{ - const CasOverwriteStopCause stop_cause = context.stop_cause(); - const uint64_t now = now_ms(); - const bool deadline_closed - = now >= context.absolute_deadline_ms || required_time_ms > context.absolute_deadline_ms - now; - - if (stop_cause == CasOverwriteStopCause::FenceOrLifecycleLost) - return {OverwriteGateClosure::FenceOrLifecycleLost, stop_cause}; - if (stop_cause == CasOverwriteStopCause::Cancelled) - return {OverwriteGateClosure::Cancelled, stop_cause}; - if (deadline_closed) - return {OverwriteGateClosure::Deadline, stop_cause}; - return {OverwriteGateClosure::Open, stop_cause}; -} - -uint64_t saturatingAdd(uint64_t lhs, uint64_t rhs) -{ - if (rhs > std::numeric_limits::max() - lhs) - return std::numeric_limits::max(); - return lhs + rhs; -} - -} - -namespace -{ -/// Shared by both public entry points below so the log line and the exception's message text can -/// never drift apart. Rate-limited (not per-distinct-`why` -- `LogSeriesLimiter` keys on the LOGGER -/// NAME only, so under a sustained outage where `why` keeps changing slightly, only the first message -/// in each window prints; this is the intended throttle, not a bug). Warning-level visibility is -/// intentional: this condition is expected to self-heal -/// (the caller retries), but an operator watching CAS logs directly should see it without having to -/// know to look at system.replication_queue. -void logCasWriteRetryLater(const String & why) -{ - LogSeriesLimiter log(getLogger("CasWriteRetryLater"), /*allowed_count=*/1, /*interval_s=*/30); - LOG_WARNING(log, "CAS write could not be committed ({}); retrying later", why); -} -} - -[[noreturn]] void throwCasWriteRetryLater(const String & why) -{ - logCasWriteRetryLater(why); - throw Exception(ErrorCodes::NETWORK_ERROR, "CAS write could not be committed ({}); retrying later", why); -} - -std::exception_ptr makeCasWriteRetryLaterExceptionPtr(const String & why) -{ - logCasWriteRetryLater(why); - return std::make_exception_ptr( - Exception(ErrorCodes::NETWORK_ERROR, "CAS write could not be committed ({}); retrying later", why)); -} - -[[noreturn]] void throwCasTransientUnavailable(const String & subject, const String & condition) -{ - /// The code is coarse (it shares a `system.errors` row with socket failures), so the MESSAGE must - /// carry the whole truth: which CA condition refused, and that the refusal is a state rather than - /// damage. Consumers key on the code; operators read this line. - /// - /// The shared suffix carries ONLY the classification, because that is the one claim true at every - /// site: retry-later is right even where the condition may turn out terminal, since the next attempt - /// re-decides against fresh state. Any promise about HOW the condition clears belongs in `condition`, - /// where the site that can actually prove it makes it -- `checkFenceOrThrow` provably cannot. - throw Exception(ErrorCodes::NETWORK_ERROR, - "{} -- {}; TRANSIENT unavailability, not damage", subject, condition); -} - -CasRequestController::CasRequestController(BackendPtr backend_, CasRequestBudget budget_, std::function now_ms_, - std::function sleep_ms_) - : backend(std::move(backend_)) - , budget(budget_) - , now_ms(now_ms_ ? std::move(now_ms_) : std::function(steadyClockNowMs)) - , sleep_ms(sleep_ms_ ? std::move(sleep_ms_) : std::function(threadSleepMs)) -{ -} - -void CasRequestController::setSleepFnForTest(std::function sleep_ms_) -{ - sleep_ms = sleep_ms_ ? std::move(sleep_ms_) : std::function(threadSleepMs); -} - -uint64_t CasRequestController::backoffBeforeAttempt(uint32_t next_attempt) const -{ - const uint64_t initial = budget.retry_initial_backoff_ms; - const uint64_t cap = budget.retry_max_backoff_ms; - if (initial == 0 || next_attempt < 2) - return 0; - /// Saturating `initial << doublings`: `initial > cap >> doublings` implies the unshifted product - /// already exceeds the cap, so return the cap without ever computing an overflowing shift. - const uint32_t doublings = next_attempt - 2; - if (doublings >= 63 || initial > (cap >> doublings)) - return cap; - return std::min(initial << doublings, cap); -} - -bool CasRequestController::pauseBeforeReissue(uint32_t completed_attempt, uint64_t deadline_ms, - const std::function & fence_ok, CasUnresolvedReason * out_reason) -{ - /// Fence BEFORE the sleep (the pre-attempt fence rule applies to the whole loop, not just the - /// attempt): a fence lost mid-backoff aborts the operation instantly — sleeping first would keep a - /// fenced writer alive for up to a full backoff cap after it lost its right to write. - if (!fence_ok()) - { - if (out_reason) - *out_reason = CasUnresolvedReason::FenceLostMidWay; - return false; - } - const uint64_t backoff = backoffBeforeAttempt(completed_attempt + 1); - if (backoff == 0) - return true; - /// Never serve a sleep the operation cannot afford: if the backoff plus one more attempt would - /// cross the operation deadline, give up NOW instead of sleeping into a guaranteed Unresolved. - if (now_ms() + backoff + budget.attempt_timeout_ms > deadline_ms) - { - if (out_reason) - *out_reason = CasUnresolvedReason::DeadlineMidWay; - return false; - } - sleep_ms(backoff); - return true; -} - -CasWriteOutcome CasRequestController::resolveByExactGet(std::string_view key, std::string_view expected_bytes, - Token * out_token) -{ - const String key_s{key}; - std::optional got; - try - { - got = backend->get(key_s); - } - catch (const std::exception &) - { - /// The GET itself failed (network, auth, ...): the object's identity cannot be proven either - /// way — an unresolved read leaves this Unresolved, exactly like an absent read. - return CasWriteOutcome::Unresolved; - } - - if (!got) - return CasWriteOutcome::Unresolved; /// absent -> another attempt may still be legal - - if (got->bytes == expected_bytes) - { - if (out_token) - *out_token = got->token; - return CasWriteOutcome::Committed; /// identical deterministic bytes -> the earlier attempt DID commit - } - - /// A DIFFERENT valid object at the exact key this create intended: a real conflict, not a retryable - /// ambiguity. Fail closed rather than silently treating it as Unresolved/DefiniteFailure. - throw Exception(ErrorCodes::CORRUPTED_DATA, - "CasRequestController: exact-key resolution at '{}' observed a DIFFERENT object than the one " - "this attempt intended to create — a real conflict, not a retryable ambiguity", key_s); -} - -CasWriteOutcome CasRequestController::putIfAbsentControlled( - std::string_view key, std::string_view bytes, const std::function & fence_ok, Token * out_token, - CasUnresolvedReason * out_reason) -{ - const String key_s{key}; - const String bytes_s{bytes}; - const uint64_t deadline_ms = now_ms() + budget.operation_deadline_ms; - /// Diagnostic bookkeeping only -- nothing below branches on it (finding #37 defect 3). - uint32_t attempts_sent = 0; - /// Does an EARLIER attempt of THIS call remain unresolved -- sent, and neither proven applied nor - /// proven refused? Set on the one path that produces exactly that state: an attempt whose outcome - /// was ambiguous and whose exact-key resolve came back absent or unreadable. The request may have - /// been received; an absent read now proves nothing about what materializes later, which is the - /// whole reason the reissue loop exists. This is NOT diagnostic: it decides the CALL's verdict at - /// the DefiniteFailure arm below. A pre-attempt gate refusal never sets it -- those return without - /// sending, so they leave nothing that could land. - bool earlier_attempt_unresolved = false; - const auto unresolved = [&](CasUnresolvedReason reason) - { - if (out_reason) - *out_reason = reason; - return CasWriteOutcome::Unresolved; - }; - if (out_reason) - *out_reason = CasUnresolvedReason::NotUnresolved; - - for (uint32_t attempt = 1; attempt <= budget.max_attempts; ++attempt) - { - /// Gate BEFORE every attempt: the - /// local mount fence must still hold, and there must be enough of the operation's own deadline - /// left for one more attempt to plausibly complete. Neither check sends anything to the backend. - if (!fence_ok()) - return unresolved(attempts_sent == 0 ? CasUnresolvedReason::NoAttemptSent - : CasUnresolvedReason::FenceLostMidWay); - if (now_ms() + budget.attempt_timeout_ms > deadline_ms) - return unresolved(attempts_sent == 0 ? CasUnresolvedReason::NoAttemptSent - : CasUnresolvedReason::DeadlineMidWay); - ++attempts_sent; - - /// The committed incarnation's token, filled by whichever leg proves Committed below. - Token committed_token; - CasWriteOutcome attempt_outcome{}; - try - { - const PutResult put = backend->putIfAbsent(key_s, bytes_s); - /// PreconditionFailed here means only "the key already exists" — it does NOT prove who - /// created it (possibly OUR earlier unresolved attempt). Collapse it onto Unresolved so it - /// goes through the SAME resolve-before-reissue path as an ambiguous exception, never a - /// false DefiniteFailure/Committed. - attempt_outcome = put.outcome == PutOutcome::Done ? CasWriteOutcome::Committed : CasWriteOutcome::Unresolved; - if (put.outcome == PutOutcome::Done) - committed_token = put.token; - } - catch (const std::exception & e) - { - attempt_outcome = classifyConditionalWriteResult(e); - } - - if (attempt_outcome == CasWriteOutcome::DefiniteFailure) - { - /// THIS attempt is proven never applied — but the verdict belongs to the CALL. An earlier - /// attempt that is still unresolved may yet materialize at the key, and a caller reading - /// `DefiniteFailure` acts on "the key is unwritten": `CasRefLedger::commitRefChunk` clears - /// its apply-pending marker and reports the txn id never used, so the next append re-derives - /// that id and a late-landing predecessor becomes an acked-then-lost transaction. Ambiguity - /// dominates a definite refusal that came after it; the caller wedges and resolves the key - /// instead. No resolve and no retry either way — this attempt has nothing left to settle. - if (earlier_attempt_unresolved) - return unresolved(CasUnresolvedReason::DefiniteFailureAfterAmbiguity); - return CasWriteOutcome::DefiniteFailure; /// every attempt of this call was proven never applied - } - - if (attempt_outcome == CasWriteOutcome::Unresolved) - { - /// Resolve-before-reissue. May throw CORRUPTED_DATA (a real - /// conflict) straight out of this call — that is never a retry signal. - attempt_outcome = resolveByExactGet(key_s, bytes_s, &committed_token); - if (attempt_outcome == CasWriteOutcome::Unresolved) - { - /// This attempt is now one that may still land: it was sent, and the resolve settled - /// nothing. Recorded BEFORE the exhaustion checks below so it is set no matter which of - /// them ends the loop, and read by the DefiniteFailure arm of every later attempt. - earlier_attempt_unresolved = true; - /// Absent/unreadable: another attempt of the SAME (key, bytes) may be legal — after the - /// fence-gated capped-exponential backoff (pauseBeforeReissue). No pause after the LAST - /// attempt: the budget is spent, sleeping would only delay the Unresolved verdict. - /// - /// Both refusals report through `unresolved`, never a bare `return Unresolved`: this is - /// the ordinary way a busy lane exhausts itself, so leaving `out_reason` at its initial - /// `NotUnresolved` here made the ref lane's wedge message read "is UNCERTAIN (not - /// unresolved)" for the single most common wedge there is. - if (attempt == budget.max_attempts) - return unresolved(CasUnresolvedReason::AttemptsExhausted); - CasUnresolvedReason pause_reason = CasUnresolvedReason::AttemptsExhausted; - if (!pauseBeforeReissue(attempt, deadline_ms, fence_ok, &pause_reason)) - return unresolved(pause_reason); - continue; - } - } - - /// attempt_outcome == Committed here (either the attempt's own 2xx, or resolution found - /// identical bytes). Final fence check before reporting success: a fence lost here means the - /// write may have landed but this call must never claim it did. Count this "response observed - /// after the local fence" leg separately from the generic Unresolved classifier so a cross-epoch - /// fence loss is visible rather than folded into ordinary retry-budget exhaustion. - if (!fence_ok()) - { - ProfileEvents::increment(ProfileEvents::CASConditionalWriteFenceLostPostWrite); - return unresolved(CasUnresolvedReason::FenceLostPostWrite); - } - if (out_token) - *out_token = committed_token; - return CasWriteOutcome::Committed; - } - - return unresolved(CasUnresolvedReason::AttemptsExhausted); /// budget exhausted, no definite outcome -} - -CasOverwriteResult CasRequestController::putOverwriteControlled( - std::string_view key, std::string_view bytes, const Token & expected, const std::function & fence_ok) -{ - const uint64_t operation_start_ms = now_ms(); - const uint64_t deadline_ms = saturatingAdd(operation_start_ms, budget.operation_deadline_ms); - const CasOverwriteOperationContext context{ - .absolute_deadline_ms = deadline_ms, - .deadline_source = CasOverwriteDeadlineSource::RequestBudget, - .stop_cause = [&fence_ok] - { - return fence_ok() ? CasOverwriteStopCause::Continue : CasOverwriteStopCause::FenceOrLifecycleLost; - }, - .wait_before_retry = [this](uint64_t wait_ms) - { - sleep_ms(wait_ms); - return true; - }, - .observe = [](const CasOverwriteProgress &) {}, - }; - return putOverwriteControlledImpl(key, bytes, expected, context, /*preserve_legacy_gates=*/true); -} - -CasOverwriteResult CasRequestController::putOverwriteControlled( - std::string_view key, - std::string_view bytes, - const Token & expected, - const CasOverwriteOperationContext & context) -{ - return putOverwriteControlledImpl(key, bytes, expected, context, /*preserve_legacy_gates=*/false); -} - -CasOverwriteResult CasRequestController::putOverwriteControlledImpl( - std::string_view key, - std::string_view bytes, - const Token & expected, - const CasOverwriteOperationContext & context, - bool preserve_legacy_gates) -{ - const String key_s{key}; - const String bytes_s{bytes}; - CasOverwriteDiagnostics diagnostics; - diagnostics.deadline_source = context.deadline_source; - bool ambiguity_observed = false; - bool earlier_attempt_unresolved = false; - bool observer_failure_reported = false; - - const auto observe = [&](CasOverwriteProgressKind kind, uint32_t attempt_no) noexcept - { - try - { - context.observe(CasOverwriteProgress{kind, attempt_no}); - } - catch (...) - { - if (!observer_failure_reported) - { - observer_failure_reported = true; - try - { - LOG_DEBUG(getLogger("CasRequestControl"), - "CAS overwrite progress observer threw; suppressing this and further observer exceptions"); - } - catch (...) - { - } - } - } - }; - - const auto unresolved = [&](CasUnresolvedReason reason, CasOverwriteStopCause stop_cause) - { - diagnostics.unresolved_reason = reason; - diagnostics.stop_cause = stop_cause; - return CasOverwriteResult{CasOverwriteOutcome::Unresolved, {}, diagnostics}; - }; - - const auto resultForGate = [&](const OverwriteGateSample & sample, bool commit_proved) - -> std::optional - { - if (sample.closure == OverwriteGateClosure::Open) - return std::nullopt; - - if (sample.closure == OverwriteGateClosure::Deadline) - { - return unresolved( - diagnostics.attempts_sent == 0 ? CasUnresolvedReason::NoAttemptSent : CasUnresolvedReason::DeadlineMidWay, - CasOverwriteStopCause::Continue); - } - - const CasUnresolvedReason reason = diagnostics.attempts_sent == 0 - ? CasUnresolvedReason::NoAttemptSent - : (commit_proved ? CasUnresolvedReason::FenceLostPostWrite : CasUnresolvedReason::FenceLostMidWay); - if (commit_proved) - ProfileEvents::increment(ProfileEvents::CASConditionalWriteFenceLostPostWrite); - return unresolved(reason, sample.stop_cause); - }; - - const auto gate = [&](uint64_t required_time_ms, bool commit_proved = false) - -> std::optional - { - return resultForGate(sampleOverwriteGate(context, now_ms, required_time_ms), commit_proved); - }; - - /// The existing overload historically checked its fence before each `PUT`/sleep and after a - /// proven commit, but did not add clock/fence samples around the resolving `GET`. Keep that exact - /// schedule while adapting its callbacks into the context representation; otherwise an injected - /// clock that advances per sample loses a physical attempt solely because the adapter observed it. - const auto legacyGate = [&](uint64_t required_time_ms, bool commit_proved, bool check_deadline) - -> std::optional - { - const CasOverwriteStopCause stop_cause = context.stop_cause(); - if (stop_cause != CasOverwriteStopCause::Continue) - { - const OverwriteGateClosure closure = stop_cause == CasOverwriteStopCause::FenceOrLifecycleLost - ? OverwriteGateClosure::FenceOrLifecycleLost - : OverwriteGateClosure::Cancelled; - return resultForGate({closure, stop_cause}, commit_proved); - } - if (!check_deadline) - return std::nullopt; - - const uint64_t now = now_ms(); - if (now > context.absolute_deadline_ms || required_time_ms > context.absolute_deadline_ms - now) - return resultForGate({OverwriteGateClosure::Deadline, CasOverwriteStopCause::Continue}, commit_proved); - return std::nullopt; - }; - - while (true) - { - const auto pre_put_refusal = preserve_legacy_gates - ? legacyGate(budget.attempt_timeout_ms, /*commit_proved=*/false, /*check_deadline=*/true) - : gate(budget.attempt_timeout_ms); - if (pre_put_refusal) - return *pre_put_refusal; - if (diagnostics.attempts_sent >= budget.max_attempts) - return unresolved(CasUnresolvedReason::AttemptsExhausted, CasOverwriteStopCause::Continue); - - const uint32_t attempt_no = diagnostics.attempts_sent + 1; - if (attempt_no > 1) - observe(CasOverwriteProgressKind::RetryStarted, attempt_no); - ++diagnostics.attempts_sent; - observe(CasOverwriteProgressKind::PutStarted, attempt_no); - - std::optional put; - bool attempt_may_still_land = false; - try - { - put = backend->putOverwrite(key_s, bytes_s, expected); - } - catch (const std::exception & e) - { - /// A deterministic local bug or a whitelisted synchronous rejection proves no retry can - /// help. It surfaces unchanged unless an earlier request from this logical operation is - /// still ambiguous; that earlier request dominates the call-wide result because it may - /// still land after this later attempt was refused. - const auto * db_e = dynamic_cast(&e); - const bool definite_failure = (db_e && isDeterministicLocalFailure(db_e->code())) - || classifyConditionalWriteResult(e) == CasWriteOutcome::DefiniteFailure; - if (definite_failure) - { - if (!preserve_legacy_gates) - { - if (auto refusal = gate(/*required_time_ms=*/0)) - return *refusal; - if (earlier_attempt_unresolved) - return unresolved( - CasUnresolvedReason::DefiniteFailureAfterAmbiguity, - CasOverwriteStopCause::Continue); - } - throw; - } - attempt_may_still_land = true; - /// Else ambiguous -- fall through to resolve below. - } - - if (put && put->outcome == PutOutcome::Done) - { - const auto post_write_refusal = preserve_legacy_gates - ? legacyGate(/*required_time_ms=*/0, /*commit_proved=*/true, /*check_deadline=*/false) - : gate(/*required_time_ms=*/0, /*commit_proved=*/true); - if (post_write_refusal) - return *post_write_refusal; - return {CasOverwriteOutcome::Committed, put->token, diagnostics}; - } - - /// Ambiguous: either a caught transient exception, or PreconditionFailed (which alone does - /// NOT prove a real conflict -- it may be our own earlier attempt's write landing under a - /// concurrent resolve). Resolve with one GET. - if (!ambiguity_observed) - { - ambiguity_observed = true; - observe(CasOverwriteProgressKind::BecameAmbiguous, attempt_no); - } - if (!preserve_legacy_gates) - { - if (auto refusal = gate(budget.attempt_timeout_ms)) - return *refusal; - } - - observe(CasOverwriteProgressKind::ResolveStarted, attempt_no); - std::optional got; - try - { - got = backend->get(key_s); - diagnostics.resolve_observation_completed = true; - diagnostics.observed_bytes = got ? std::optional{got->bytes} : std::nullopt; - } - catch (const std::exception &) - { - diagnostics.resolve_observation_completed = false; - diagnostics.observed_bytes.reset(); - got.reset(); /// GET failed: still ambiguous, fall through to retry below. - } - - if (got && got->token != expected && got->bytes == bytes_s) - { - diagnostics.resolved_by_get = true; - observe(CasOverwriteProgressKind::ResolvedByGet, attempt_no); - const auto post_write_refusal = preserve_legacy_gates - ? legacyGate(/*required_time_ms=*/0, /*commit_proved=*/true, /*check_deadline=*/false) - : gate(/*required_time_ms=*/0, /*commit_proved=*/true); - if (post_write_refusal) - return *post_write_refusal; - return {CasOverwriteOutcome::Committed, got->token, diagnostics}; - } - - /// Apply stop/deadline precedence to the completed resolve before accepting a conflict, - /// waiting, or letting attempt exhaustion decide whether another `PUT` is legal. - if (!preserve_legacy_gates) - { - if (auto refusal = gate(/*required_time_ms=*/0)) - return *refusal; - } - - if (got && got->token == expected) - { - /// The token we CAS'd against is STILL current: our attempt never applied. Fall through - /// to the pause-and-reissue gate below (same key, bytes, expected). - } - else if (got) - { - /// A DIFFERENT token AND different bytes: a genuine competing write. Real conflict -- - /// never collapsed into Unresolved/DefiniteFailure, never thrown. - return {CasOverwriteOutcome::Conflict, {}, diagnostics}; - } - /// else: the GET itself failed or the key vanished -- still ambiguous, fall through to retry. - - if (attempt_may_still_land) - earlier_attempt_unresolved = true; - - /// Attempt exhaustion participates only when another physical `PUT` would be sent. It is - /// deliberately evaluated after the final attempt's resolving `GET` and after the gate above. - if (diagnostics.attempts_sent >= budget.max_attempts) - { - if (!preserve_legacy_gates) - { - if (auto refusal = gate(budget.attempt_timeout_ms)) - return *refusal; - } - return unresolved(CasUnresolvedReason::AttemptsExhausted, CasOverwriteStopCause::Continue); - } - - const uint64_t backoff_ms = backoffBeforeAttempt(attempt_no + 1); - if (backoff_ms == 0) - continue; - - const uint64_t retry_reservation_ms = saturatingAdd(backoff_ms, budget.attempt_timeout_ms); - const auto pre_wait_refusal = preserve_legacy_gates - ? legacyGate(retry_reservation_ms, /*commit_proved=*/false, /*check_deadline=*/true) - : gate(retry_reservation_ms); - if (pre_wait_refusal) - return *pre_wait_refusal; - - const bool wait_completed = context.wait_before_retry(backoff_ms); - if (preserve_legacy_gates) - continue; - - const OverwriteGateSample after_wait = sampleOverwriteGate(context, now_ms, budget.attempt_timeout_ms); - if (!wait_completed && after_wait.stop_cause == CasOverwriteStopCause::Continue) - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CasRequestController: wait_before_retry returned false while stop_cause remained Continue"); - if (auto refusal = resultForGate(after_wait, /*commit_proved=*/false)) - return *refusal; - } -} - -CasOverwriteResult CasRequestController::putIfAbsentControlledMutable( - std::string_view key, std::string_view bytes, const std::function & fence_ok) -{ - const String key_s{key}; - const String bytes_s{bytes}; - const uint64_t deadline_ms = now_ms() + budget.operation_deadline_ms; - - for (uint32_t attempt_no = 1; attempt_no <= budget.max_attempts; ++attempt_no) - { - if (!fence_ok()) - return {CasOverwriteOutcome::Unresolved, {}, {}}; - if (now_ms() + budget.attempt_timeout_ms > deadline_ms) - return {CasOverwriteOutcome::Unresolved, {}, {}}; - - std::optional put; - try - { - put = backend->putIfAbsent(key_s, bytes_s); - } - catch (const std::exception & e) - { - /// Same rethrow convention as `putOverwriteControlled`. - if (const auto * db_e = dynamic_cast(&e); db_e && isDeterministicLocalFailure(db_e->code())) - throw; - if (classifyConditionalWriteResult(e) == CasWriteOutcome::DefiniteFailure) - throw; - /// Else ambiguous -- fall through to resolve below. - } - - if (put && put->outcome == PutOutcome::Done) - { - if (!fence_ok()) - { - ProfileEvents::increment(ProfileEvents::CASConditionalWriteFenceLostPostWrite); - return {CasOverwriteOutcome::Unresolved, {}, {}}; - } - return {CasOverwriteOutcome::Committed, put->token, {}}; - } - - /// Ambiguous: either a caught transient exception, or PreconditionFailed (which alone does - /// NOT prove a real conflict -- it may be our own earlier attempt's write landing under a - /// concurrent resolve, or a racing writer creating the identical value). Resolve with one GET. - std::optional got; - try - { - got = backend->get(key_s); - } - catch (const std::exception &) - { - got.reset(); /// GET failed: still ambiguous, fall through to retry below. - } - - if (!got) - { - /// Still absent: our attempt never applied. Fall through to the pause-and-reissue gate - /// below (same key, bytes). - } - else if (got->bytes == bytes_s) - { - if (!fence_ok()) - { - ProfileEvents::increment(ProfileEvents::CASConditionalWriteFenceLostPostWrite); - return {CasOverwriteOutcome::Unresolved, {}, {}}; - } - return {CasOverwriteOutcome::Committed, got->token, {}}; - } - else - { - /// Present with DIFFERENT bytes: something else already occupies the key with a - /// different value. For a MUTABLE marker this is a normal outcome, not corruption -- - /// return it as a value, never thrown. - return {CasOverwriteOutcome::Conflict, {}, {}}; - } - - if (attempt_no == budget.max_attempts || !pauseBeforeReissue(attempt_no, deadline_ms, fence_ok)) - return {CasOverwriteOutcome::Unresolved, {}, {}}; - } - - return {CasOverwriteOutcome::Unresolved, {}, {}}; /// attempt budget exhausted without a definite outcome -} - -SlotOccupyResult CasRequestController::slotOccupy( - std::string_view key, std::string_view bytes, const std::function & fence_ok) -{ - const String key_s{key}; - const String bytes_s{bytes}; - const uint64_t deadline_ms = now_ms() + budget.operation_deadline_ms; - - /// Pre-attempt gate -- the same two checks every controlled op runs before its first (here, only) - /// attempt: the mount fence must still hold, and there must be enough of the operation's own - /// deadline left for one attempt to plausibly complete. Neither check sends anything to the - /// backend, so a refusal here PROVES the key is untouched by this call. UNLIKE every sibling - /// controlled op, there is no post-write result recheck below: the stronger post-I/O consistency - /// check (fence generation together with wedge/txn identity) is the CALLER's contract (Task 4/6's - /// re-acquire-lock-and-checkFenceOrThrow step), not this raw primitive's. The admission predicate is - /// checked again only if this attempt needs a second backend request to resolve its result; a - /// `Created` result still performs no post-write recheck. - if (!fence_ok() || now_ms() + budget.attempt_timeout_ms > deadline_ms) - return {.kind = SlotOccupyResult::Kind::Unresolved, .occupant_bytes = {}, .occupant_token = {}, - .unresolved_reason = CasUnresolvedReason::NoAttemptSent}; - - std::optional put; - try - { - put = backend->putIfAbsent(key_s, bytes_s); - } - catch (const std::exception & e) - { - /// Same rethrow convention as putOverwriteControlled/putIfAbsentControlledMutable: a - /// deterministic local bug, or a whitelisted synchronous rejection that PROVES the request was - /// never applied, surfaces unchanged -- SlotOccupyResult::Kind has no DefiniteFailure member to - /// carry either one. Anything else is ambiguous: fall through to the raw resolve GET below, - /// exactly like a clean PreconditionFailed -- this primitive cannot and does not distinguish - /// the two. - if (const auto * db_e = dynamic_cast(&e); db_e && isDeterministicLocalFailure(db_e->code())) - throw; - if (classifyConditionalWriteResult(e) == CasWriteOutcome::DefiniteFailure) - throw; - } - - if (put && put->outcome == PutOutcome::Done) - return {.kind = SlotOccupyResult::Kind::Created, .occupant_bytes = {}, .occupant_token = {}, - .unresolved_reason = CasUnresolvedReason::NotUnresolved}; - - /// Ambiguous attempt or a clean conflict: resolve with exactly ONE raw exact GET -- no byte-compare, - /// no throw on a different occupant [codex finding 3: this is a DEDICATED slot operation, not - /// putIfAbsentControlled (which retries the same (key, bytes) internally) or resolveByExactGet - /// (which compares against an expected body and throws CORRUPTED_DATA on a mismatch) composed - /// together]. Adjudicating whether the occupant is "mine" is entirely the CALLER's job (the - /// CaCasMountCore `mine` contract), never this primitive's. - /// - /// Whole-object resolution is safe here because `slotOccupy` is scoped by its callers (Task 4/6, - /// spec INV-2) to small, write-once control slots -- ref-log transactions and epoch seals -- whose - /// size is bounded by their own format's registry cap (the strict-grammar object caps - /// CasRefLogFormat/CasRefCkptFormat enforce on decode), never a data blob. slotOccupy itself stays - /// format-agnostic (it takes a raw key/bytes pair, per the "Interface handed to Stage B" contract in - /// the plan) and does not encode any format's cap here -- the size bound is a property of what - /// callers are allowed to pass it, enforced where the returned bytes are decoded, not by this seam. - if (!fence_ok()) - return {.kind = SlotOccupyResult::Kind::Unresolved, .occupant_bytes = {}, .occupant_token = {}, - .unresolved_reason = CasUnresolvedReason::AttemptsExhausted}; - - std::optional got; - try - { - got = backend->get(key_s); - } - catch (const std::exception &) - { - got.reset(); /// the GET itself failed: still unresolved -- a one-shot primitive never retries - } - - if (!got) - /// The occupant that caused the conflict vanished before this GET (or the GET itself failed): - /// the outcome is unknowable right now -- NEVER a fabricated Created. - return {.kind = SlotOccupyResult::Kind::Unresolved, .occupant_bytes = {}, .occupant_token = {}, - .unresolved_reason = CasUnresolvedReason::AttemptsExhausted}; - - return {.kind = SlotOccupyResult::Kind::Occupied, .occupant_bytes = std::move(got->bytes), - .occupant_token = got->token, .unresolved_reason = CasUnresolvedReason::NotUnresolved}; -} - -} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h deleted file mode 100644 index 643da6f46246..000000000000 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestControl.h +++ /dev/null @@ -1,553 +0,0 @@ -#pragma once -#include -#include -#include -#include -#include -#include -#include -#include - -namespace DB::Cas -{ - -/// Outcome of ONE HTTP attempt for a CAS conditional write (`If-None-Match`/`If-Match`), issued with -/// the generic S3 client's transparent retries disabled for that attempt. This is the seam the -/// `CasRequestController` is built on: it decides whether another attempt is legal and how an -/// uncertain result is resolved. -/// - Committed: the attempt's own request completed successfully (2xx) — the object is durable. -/// - DefiniteFailure: a synchronous rejection that PROVES the request was never applied server-side -/// — a WHITELISTED malformed-request / entity-too-large / access-denied error ONLY. Never -/// `PreconditionFailed`: a lost precondition means the key exists, not that the request failed. -/// - Unresolved: everything else — `PreconditionFailed`/`NoSuchKey`, a client-side timeout, a -/// connection loss, a 5xx, or any error this classifier does not recognize. The caller resolves -/// the exact key before deciding whether another attempt is legal; ambiguity always -/// resolves toward Unresolved, never toward a false DefiniteFailure or a false Committed. -enum class CasWriteOutcome : uint8_t -{ - Committed, - DefiniteFailure, - Unresolved, -}; - -/// WHY a controlled write came back `Unresolved`. It exists because `Unresolved` covers two materially -/// different states, and telling them apart is the difference between a five-minute triage and an hour -/// of it — and, since finding #37 defect 3, between a table that keeps its write availability and one -/// that loses it until remount. -/// -/// `NoAttemptSent` is the one that carries real information: both pre-attempt gates (the mount fence -/// and the operation deadline) reject BEFORE anything reaches the backend, so on the very first -/// iteration the key is PROVABLY unwritten — there is no ambiguity to resolve, only a lost right to -/// write. Every other reason leaves an object that may or may not be durable, which is what the -/// wedge/resolve machinery exists for. -/// -/// NOT purely diagnostic any more: `unresolvedProvesNothingWasSent` below turns this into the fact the -/// ref append lane acts on (`CasRefLedger::commitRefChunk`'s `Unresolved` arm), so ADDING A MEMBER HERE -/// IS A PROTOCOL DECISION — read that predicate before you do. -enum class CasUnresolvedReason : uint8_t -{ - NotUnresolved, /// the call did not return Unresolved - NoAttemptSent, /// a pre-attempt gate rejected on the FIRST iteration: nothing was ever sent - FenceLostMidWay, /// >= 1 attempt was sent, then the mount fence dropped - DeadlineMidWay, /// >= 1 attempt was sent, then the operation deadline left no room for another - FenceLostPostWrite,/// an attempt COMMITTED but the fence had dropped by the time it returned - AttemptsExhausted, /// the genuine case the "retry budget exhausted" wording describes - /// A LATER attempt was definitively refused while an EARLIER one of the same call is still - /// unresolved. The refusal proves only its own attempt never applied; the earlier one may still - /// materialize at the key, so the CALL cannot report `DefiniteFailure` (see - /// `putIfAbsentControlled`). Reported instead of the definite verdict, never alongside it. - DefiniteFailureAfterAmbiguity, -}; - -/// Does this `Unresolved` PROVE that no attempt ever reached the network — i.e. that the key is -/// unwritten and there is nothing for an exact-key resolution to settle? -/// -/// True for exactly ONE value, and that is the whole design: `NoAttemptSent` is reported only when a -/// pre-attempt gate rejected while `attempts_sent == 0`, so `backend->putIfAbsent` was never called -/// (see `putIfAbsentControlled`). Every other value — including `NotUnresolved`, which a caller can -/// still observe if some path returns `Unresolved` without recording a reason — leaves an object that -/// MAY be durable, and callers that protect themselves against that (the ref lane's append wedge) must -/// keep doing so. -/// -/// Written as an allow-list: a switch with no `default` and a trailing `return false`, so a member -/// added to `CasUnresolvedReason` later fails BOTH ways safely. The missing case is a `-Wswitch` build -/// error, which forces the contributor to classify it deliberately; and if that diagnostic is ever -/// silenced, the runtime answer for the unclassified member is "no, this does not prove anything", -/// which is the conservative side. Never turn this into a deny-list — a new reason must not be able to -/// claim "nothing was sent" by omission. -constexpr bool unresolvedProvesNothingWasSent(CasUnresolvedReason reason) -{ - switch (reason) - { - case CasUnresolvedReason::NoAttemptSent: - return true; - case CasUnresolvedReason::NotUnresolved: - case CasUnresolvedReason::FenceLostMidWay: - case CasUnresolvedReason::DeadlineMidWay: - case CasUnresolvedReason::FenceLostPostWrite: - case CasUnresolvedReason::AttemptsExhausted: - /// The whole point of this value is that an earlier attempt WAS sent and may still land. - case CasUnresolvedReason::DefiniteFailureAfterAmbiguity: - return false; - } - return false; -} - -/// Human-readable tail for an exception or log line, so the two states above stop reading alike. -constexpr std::string_view describeUnresolvedReason(CasUnresolvedReason reason) -{ - switch (reason) - { - case CasUnresolvedReason::NotUnresolved: return "not unresolved"; - case CasUnresolvedReason::NoAttemptSent: return "no attempt was sent (the mount fence or the " - "operation deadline rejected before the first " - "request) — the key is provably unwritten"; - case CasUnresolvedReason::FenceLostMidWay: return "the mount fence dropped after at least one " - "attempt had been sent"; - case CasUnresolvedReason::DeadlineMidWay: return "the operation deadline ran out after at least " - "one attempt had been sent"; - case CasUnresolvedReason::FenceLostPostWrite: return "an attempt committed but the mount fence had " - "dropped before it returned"; - case CasUnresolvedReason::AttemptsExhausted: return "the attempt budget was exhausted without a " - "definite outcome"; - case CasUnresolvedReason::DefiniteFailureAfterAmbiguity: - return "a later attempt was definitively refused, but " - "an earlier attempt of the same call is still " - "unresolved and may yet land"; - } - return "unspecified"; -} - -/// The success path: `buf.finalize()` returned without throwing. Always Committed — kept as a named, -/// counted entry point so both paths of a classify-then-record call site read the same way (see the -/// exception overload below). -constexpr CasWriteOutcome classifyConditionalWriteResult() -{ - return CasWriteOutcome::Committed; -} - -/// The exception path: classify what `buf.finalize()` threw for ONE CAS conditional-write HTTP -/// attempt, according to the CAS conditional-write operation classes. Pure — never rethrows, never -/// touches counters; see recordConditionalWriteOutcome for the counters hookup. -CasWriteOutcome classifyConditionalWriteResult(const std::exception & e); - -/// Records the start of one HTTP attempt for a CAS conditional write (the attempts counter). -void recordConditionalWriteAttemptStarted(); - -/// Records one attempt's terminal outcome (the per-class outcome counters). Callers pass the result of -/// whichever classifyConditionalWriteResult overload applies, or an outcome already known by -/// construction (e.g. the legacy `PutOutcome::PreconditionFailed` path, which today resolves without -/// throwing — see ObjectStorageBackend::nativeConditionalPut). -void recordConditionalWriteOutcome(CasWriteOutcome outcome); - -/// Throw the recoverable "CAS write could not be committed, retry later" condition. -/// -/// WHY NETWORK_ERROR (this replaces an earlier ABORTED throw): -/// A content-addressed write can fail for a reason that is neither the caller's fault -/// nor permanent: the mount-lease / write fence was lost (e.g. a renewal PUT timed out -/// against a slow or throttling object store), or a conditional PUT exhausted its retry -/// budget mid-outage. The right response is "abandon this attempt, try again later" -- -/// which is precisely what a transient error means. -/// -/// It previously threw `ABORTED`, which was actively harmful to background merges: -/// `ReplicatedMergeMutateTaskBase` treats `ABORTED` as "merge deliberately cancelled -/// (shutdown / `DROP` / merges-blocker), not an error", so it neither records -/// `last_exception_time_ms` nor lets `ReplicatedMergeTreeQueue`'s exponential backoff -/// engage. Under a sustained store outage the queue re-executed the merge roughly every -/// 2 seconds, recomputing the whole (possibly multi-GiB) output part every time for the -/// entire outage -- hundreds of full recomputes, and invisible in system.replication_queue. -/// -/// `NETWORK_ERROR` is the best-fitting EXISTING code: -/// - it is NOT in the merge "retry silently, no backoff" exemption set (only `ABORTED` -/// and `PART_IS_TEMPORARILY_LOCKED` are), so the existing backoff -- capped by -/// `max_postpone_time_for_failed_replicated_merges_ms` -- engages automatically; -/// - it is already in ClickHouse's transient/retryable taxonomy -/// (`checkDataPart::isRetryableException` lists it beside `ABORTED`), so a part under -/// verification is not misread as corrupted; -/// - nothing on the merge / insert / replication commit path special-cases it in a way -/// that would misfire (ZooKeeper retriability keys on `Coordination::Exception`, a -/// different type), and it is not caught specially on the CAS write path. -/// -/// Honest caveat: `NETWORK_ERROR` is coarser than the true condition. For the -/// throttled-store / timed-out / lost-lease cases it is accurate; for a purely logical -/// fence loss (e.g. the namespace is being dropped) it slightly overstates "network". -/// The precise cause is always in the exception MESSAGE, never inferred from the code. -/// -/// If that imprecision ever matters -- operator confusion, or a future upstream change -/// that attaches merge-path handling to `NETWORK_ERROR` and reintroduces a collision -- -/// switch to a dedicated code (e.g. CAS_WRITE_RETRY_LATER) by changing the -/// single throw below. A dedicated code is honest and collision-proof by construction -/// (backoff still engages, since only `ABORTED` / `PART_IS_TEMPORARILY_LOCKED` are exempt); -/// the only extra work is one appended line in `ErrorCodes.cpp` and, optionally, adding it -/// to `checkDataPart::isRetryableException` and an HTTP-status mapping for the foreground -/// `INSERT` client. We deliberately kept `NETWORK_ERROR` for now to add zero new coupling to -/// generic ClickHouse code, consistent with the rest of the CAS layer. -/// -/// SCOPE: only the ESCAPING retry-later throws route here (fence lost or a controlled write outcome -/// remaining uncertain). Startup/decommission and generic live-lock-brake `ABORTED` values keep -/// their meaning and are not rerouted here. -[[noreturn]] void throwCasWriteRetryLater(const String & why); - -/// Same classification as `throwCasWriteRetryLater`, but returns the exception as a -/// `std::exception_ptr` for call sites that fail a pending future/promise (`CasRefLedger`'s -/// `complete_error`) rather than throw directly. Both entry points route through the SAME -/// construction internally, so the error code / message shape has exactly one place that decides it. -std::exception_ptr makeCasWriteRetryLaterExceptionPtr(const String & why); - -/// Throw the recoverable "this content-addressed disk cannot serve the request right now" condition. -/// Sibling of `throwCasWriteRetryLater`, same class for the same reasons (see the long rationale above), -/// differing only in what it describes: that one names a WRITE whose commit did not land, this one names -/// a DISK STATE that refused the request before it started -- on either plane. -/// -/// The class is load-bearing beyond CAS. `ReplicatedMergeTreePartCheckThread::checkPartImpl` rethrows -/// (leaving the part queued for a later check) exactly when `checkDataPart::isRetryableException` -/// recognises the error, and otherwise declares the part broken -- detach and re-fetch. -/// `INVALID_STATE` is absent from that classifier, so a lease blip used to read as part corruption -/// (BACKLOG `{#lease-blip-part-check-collapse}`). Re-coding the CA transients -/// was chosen over widening the upstream classifier because `INVALID_STATE` is broad: widening it would -/// also reclassify 18 unrelated TERMINAL sites, CA and non-CA alike. -/// -/// SCOPE, narrow by design: a refusal routes here when it either names an AUTO-RECOVERING disk condition, -/// or CANNOT ESTABLISH that its condition is terminal. `checkFenceOrThrow` is the second kind -- one guard -/// trips for a lease blip and for a FORGET decommission alike and it cannot tell them apart -- and it is in -/// scope for write-plane uniformity: its 32 sibling write-transient sites already mint this class, and an -/// unproven condition must be retried rather than consumed as damage. What is NEVER in scope is a refusal -/// whose condition is PROVEN terminal: `IdentityLost`, both `Vanished` flavours, a storage that is not -/// started, an unbootstrappable prefix, a proven-absent pool identity, a closed writer epoch -- all keep -/// `INVALID_STATE`. A proven-terminal state that read as retryable would make every consumer retry forever -/// against a disk that is never coming back. -/// -/// `subject` names the refusing disk or pool (e.g. "content-addressed disk 'ca'") and `condition` states -/// the CA condition truthfully, INCLUDING any promise about how it clears -- only the site knows whether -/// it can make one. What is appended HERE is the classification alone, so it cannot drift between call -/// sites. Unlike `throwCasWriteRetryLater` this deliberately does not log: these sites fire once per -/// refused operation (tens of thousands within a single observed lease gap) and every caller already -/// reports the exception it receives. -[[noreturn]] void throwCasTransientUnavailable(const String & subject, const String & condition); - -/// Outcome of a controlled MUTABLE conditional overwrite (`putOverwriteControlled`) -- an If-Match -/// replace whose caller can, unlike a content-addressed create, supply the intended bytes for -/// GET-based resolution, because the payload here is deterministic (a pure function of the -/// caller's record), not freshly minted per attempt. -/// - Committed: an attempt's own request completed (2xx) and the final fence check held, or -/// resolution proved the intended bytes are already what's currently stored -- `token` names -/// that incarnation. -/// - Conflict: resolution proved the key's CURRENT token AND bytes both differ from what this -/// call intended -- a genuine competing write. Returned as a value, never thrown, never -/// collapsed into Unresolved/DefiniteFailure -- mirrors the existing uncontrolled -/// casMeta/CasResult contract (a conflict lets the caller reload and decide). -/// - Unresolved: budget exhausted, fence lost, or the current token still equals `expected` (the -/// attempt provably never applied) with the resolve unable to prove either outcome yet -- -/// caller must not ACK. -enum class CasOverwriteOutcome : uint8_t -{ - Committed, - Conflict, - Unresolved, -}; - -enum class CasOverwriteDeadlineSource : uint8_t -{ - RequestBudget, - ExternalLeaseSafety, -}; - -enum class CasOverwriteStopCause : uint8_t -{ - Continue, - Cancelled, - FenceOrLifecycleLost, -}; - -enum class CasOverwriteProgressKind : uint8_t -{ - PutStarted, - BecameAmbiguous, - ResolveStarted, - RetryStarted, - ResolvedByGet, -}; - -struct CasOverwriteProgress -{ - CasOverwriteProgressKind kind; - uint32_t attempt_no; -}; - -/// Per-operation gates for a controlled mutable overwrite. `absolute_deadline_ms` uses the same -/// clock as the controller's injected `now_ms`; it is fixed by the caller before controller entry -/// and therefore cannot be re-anchored after preemption. `wait_before_retry` is interruptible and -/// must return false only after publishing a non-`Continue` stop cause. `observe` is diagnostic only: -/// an exception from it is contained and cannot affect the protocol result. -struct CasOverwriteOperationContext -{ - uint64_t absolute_deadline_ms; - CasOverwriteDeadlineSource deadline_source; - std::function stop_cause; - std::function wait_before_retry; - std::function observe; -}; - -struct CasOverwriteDiagnostics -{ - uint32_t attempts_sent = 0; - bool resolved_by_get = false; - CasUnresolvedReason unresolved_reason = CasUnresolvedReason::NotUnresolved; - CasOverwriteDeadlineSource deadline_source = CasOverwriteDeadlineSource::RequestBudget; - CasOverwriteStopCause stop_cause = CasOverwriteStopCause::Continue; - /// The last exact resolving GET completed by this controller. `resolve_observation_completed` - /// distinguishes a confirmed absence (`observed_bytes == nullopt`) from a failed/not-run read. - /// Terminal protocol owners use this snapshot instead of starting diagnostic I/O after the - /// controller has closed its deadline/cancellation gate. - bool resolve_observation_completed = false; - std::optional observed_bytes; -}; - -/// Result of one `CasRequestController::putOverwriteControlled` operation. `token` is meaningful -/// only when `outcome` is `Committed`. -struct CasOverwriteResult -{ - CasOverwriteOutcome outcome = CasOverwriteOutcome::Unresolved; - Token token; /// set ONLY on Committed - CasOverwriteDiagnostics diagnostics; -}; - -/// Result of one `CasRequestController::slotOccupy` operation — a WRITE-ONCE conditional create whose -/// body is content-addressed or otherwise not byte-comparable across separate CALLS the way -/// `putOverwriteControlled`'s deterministic marker is (each caller of `slotOccupy` — an epoch seal, a -/// wedge retry — mints its own attempt and decides for itself, from `Occupied`'s bytes, whether the -/// occupant is its own earlier write or something else entirely; see the adjudication note below). -/// - Created: this call's OWN conditional create committed — the key held nothing before it. -/// - Occupied: the key already holds an object, observed by ONE raw exact `GET` after the create -/// conflicted — `occupant_bytes`/`occupant_token` name exactly what is there NOW. The primitive -/// never compares these bytes against what this call attempted and never throws on a mismatch: -/// unlike `resolveByExactGet` (whose caller supplies ONE expected body across every retry of the -/// SAME logical attempt), `slotOccupy` never retries, so there is no "our earlier attempt" to -/// distinguish from a genuine foreign occupant — that adjudication (the `CaCasMountCore` `mine` -/// contract: an occupant is this caller's write only if the BYTES match, never a generation/shape -/// match alone) is entirely the CALLER's job. -/// - Unresolved: the outcome is unknowable right now — a pre-attempt gate refused (fence lost / -/// deadline exhausted, `unresolved_reason == NoAttemptSent`, nothing was sent — `unresolvedProvesNothingWasSent` -/// is TRUE only for this case), admission was lost after the create but before its resolution GET, -/// or the conditional create was itself ambiguous (a transient exception) and the follow-up resolve -/// GET found nothing (the occupant that caused the conflict vanished before the GET, or the GET -/// itself failed). Both post-create cases report `unresolved_reason == AttemptsExhausted`, for which -/// `unresolvedProvesNothingWasSent` is FALSE — NEVER fabricated into -/// a false `Created`. CALLERS: do not log a bare `describeUnresolvedReason(AttemptsExhausted)` for -/// this case — it reads "the attempt budget was exhausted", which is misleading for a primitive -/// with no retry budget, and it silently folds admission loss together with "the resolve GET found -/// nothing" and "the occupant that caused the conflict was DELETED under a live epoch" (a -/// GC-invariant alarm, not routine contention) into the same generic wording. `SlotOccupyResult` -/// carries no discriminator between these sub-cases; the day a caller NEEDS the split is the trigger -/// for adding a dedicated `CasUnresolvedReason` value (a gated protocol decision, not a drive-by). -struct SlotOccupyResult -{ - enum class Kind : uint8_t { Created, Occupied, Unresolved }; - Kind kind = Kind::Unresolved; - /// Occupied only: the occupant, fetched by exact GET after the conditional create conflicted. - String occupant_bytes; - Token occupant_token; - /// Unresolved only: why the attempt outcome is unknowable right now. - CasUnresolvedReason unresolved_reason{}; -}; - -/// CAS-owned retry controller: the only place that decides whether a conditional-write attempt may be -/// reissued. It does not touch a writer cache or return ACK. Callers update their cache and acknowledge -/// the operation only after this controller has resolved the outcome and performed its final fence -/// check, using the returned `CasWriteOutcome`. -class CasRequestController -{ -public: - /// `now_ms_`: monotonic-ish clock, defaulting to `std::chrono::steady_clock`; tests inject a fake - /// one to drive deadline behavior deterministically (no sleeps). - /// `sleep_ms_`: the inter-attempt backoff sleep, defaulting to a real `std::this_thread::sleep_for`; - /// tests inject a recorder/no-op to assert the backoff schedule without wall-clock waits. The - /// controller only ever sleeps BETWEEN attempts of one logical operation, on the calling thread, - /// with no Pool mutex held (every call site — the ref append lane's leader, `stageManifest`, and - /// snapshot publishes — invokes the controller outside its locks; the append lane's - /// LEADERSHIP is deliberately held across the sleep: same-table appends must queue behind an - /// unresolved predecessor PUT anyway, preserving the writer's per-table ordering. - CasRequestController(BackendPtr backend_, CasRequestBudget budget_, std::function now_ms_ = {}, - std::function sleep_ms_ = {}); - - /// Controlled `putIfAbsent` with resolve-before-reissue. Performs at - /// most `budget.max_attempts` attempts of the exact SAME (key, bytes) — never a different key, never - /// a different body — bounded by `budget.operation_deadline_ms` measured from this call's own start, - /// with capped-exponential inter-attempt backoff (`retry_initial_backoff_ms`/`retry_max_backoff_ms`). - /// `fence_ok` is consulted before EVERY attempt (a false answer sends no further attempt), before - /// EVERY backoff sleep (a fence lost mid-loop aborts instantly, never after a pointless sleep), and - /// once more before a `Committed` return (a false answer there means the write may have landed but - /// this call reports `Unresolved`, never a false `Committed`). A sleep is - /// never entered when it (plus one more attempt) could not fit the operation deadline. An uncertain - /// attempt is resolved via `resolveByExactGet` before deciding whether to reissue. - /// Throws `CORRUPTED_DATA` if resolution ever observes DIFFERENT valid bytes at `key` — a real - /// conflict, never collapsed into `Unresolved`/`DefiniteFailure`. Returns `Unresolved` (never - /// throws) when the fence is lost or the budget is exhausted before a definite outcome is reached. - /// - /// THE VERDICT IS THE CALL'S, NOT THE LAST ATTEMPT'S. `DefiniteFailure` is returned only when EVERY - /// attempt this call sent was itself proven never applied. One attempt's whitelisted rejection - /// proves nothing about an EARLIER attempt of the same call that went ambiguous: that request may - /// have been received and may still materialize at `key` (an absent resolve GET is not evidence — - /// `unresolvedProvesNothingWasSent`). Any such attempt therefore dominates the result, which becomes - /// `Unresolved`/`DefiniteFailureAfterAmbiguity` — the wedge path — because a caller acting on - /// `DefiniteFailure` declares the key unwritten and reuses the id (`CasRefLedger::commitRefChunk`), - /// which an ambiguous predecessor can turn into an acked-then-lost transaction. Attempts a - /// pre-attempt gate refused never reach the backend, so they never make this call ambiguous. - /// `out_token` (optional): set ONLY on a `Committed` return, to the committed incarnation's token — - /// the attempt's own `PutResult` token, or the token the resolve GET observed when it proved an - /// earlier ambiguous attempt landed. Lets audit emitters (e.g. `PartWriteTxn::stageManifest`'s - /// `ManifestPut` event) keep the token without a follow-up HEAD. Untouched on any other return. - /// `out_reason` (optional): WHY an `Unresolved` was returned. Diagnostic only — the returned - /// outcome is unchanged, so no caller's decision depends on it. It exists because `Unresolved` - /// currently conflates two very different situations, and the resulting message - /// ("retry budget exhausted") is printed even where NOTHING was ever sent: finding #37 defect 3, - /// whose own note records that the opacity "plausibly fed 3 prior wrong analyses" — and it did so - /// again on 2026-07-24, when a sanitizer-slow unit test fenced itself and the text sent the CI - /// triage looking for a retry problem that did not exist. `NoAttemptSent` is the load-bearing - /// distinction: it means the key was provably never written, whereas the other reasons leave a - /// possibly-durable object behind. - CasWriteOutcome putIfAbsentControlled(std::string_view key, std::string_view bytes, - const std::function & fence_ok, Token * out_token = nullptr, - CasUnresolvedReason * out_reason = nullptr); - - /// One-shot exact-key resolution of an uncertain immutable create: - /// - identical bytes observed at `key` -> Committed (the earlier attempt DID commit) - /// - DIFFERENT bytes observed at `key` -> throws CORRUPTED_DATA (a real conflict, not a retry - /// signal — never silently treated as ambiguous) - /// - absent, or the GET itself fails -> Unresolved (another attempt may still be legal) - /// NEVER returns DefiniteFailure: an absent or unreadable key proves nothing about whether the - /// original request will eventually be provably non-applied, so resolution alone can never produce - /// that verdict. `out_token` (optional): set ONLY on `Committed`, to the observed incarnation's token. - CasWriteOutcome resolveByExactGet(std::string_view key, std::string_view expected_bytes, - Token * out_token = nullptr); - - /// Controlled If-Match overwrite with resolve-before-reissue, for a MUTABLE marker whose bytes - /// are deterministic so GET-based resolution can compare them (unlike a content-addressed - /// create's freshly-minted-per-attempt body). Performs at most `budget.max_attempts` attempts of - /// the exact SAME (key, bytes, expected token), bounded by `budget.operation_deadline_ms`, with - /// the same fence/backoff/deadline gates as `putIfAbsentControlled`. An ambiguous attempt - /// (`PreconditionFailed`, or a transient exception classified `Unresolved`) is resolved with ONE - /// GET at `key`: - /// - the current token still equals `expected` -> the attempt provably never applied; another - /// attempt of the SAME (key, bytes, expected) is legal (fence/backoff/deadline-gated) - /// - the current bytes equal `bytes` -> Committed (an earlier ambiguous attempt of - /// THIS call already landed); `token` is the observed incarnation - /// - neither -> Conflict: a genuine competing write - /// landed; returned as a value, never thrown - /// - the GET itself fails -> still ambiguous; reissue is safe - /// A whitelisted `DefiniteFailure` classification, or a deterministic local failure - /// (`isDeterministicLocalFailure`), rethrows the original exception rather than collapsing it - /// into an outcome. - CasOverwriteResult putOverwriteControlled(std::string_view key, std::string_view bytes, - const Token & expected, const std::function & fence_ok); - - /// Controlled overwrite with a caller-owned absolute deadline, cancellation/lifecycle cause, - /// interruptible retry wait, and contained diagnostic observer. Stop and deadline gates run - /// before every backend request, before and after every wait, and before accepting a proven - /// commit. The physical-attempt limit is considered only when another `PUT` would be sent, so the - /// exact resolving `GET` for the final sent attempt is never suppressed. - CasOverwriteResult putOverwriteControlled( - std::string_view key, - std::string_view bytes, - const Token & expected, - const CasOverwriteOperationContext & context); - - /// Controlled put-if-absent for a MUTABLE marker whose bytes are deterministic, where an - /// EXISTING DIFFERENT value at the key is a normal outcome (Conflict), not corruption. This is - /// the create-side sibling of `putOverwriteControlled` and deliberately does NOT reuse - /// `putIfAbsentControlled`: that method's resolve (`resolveByExactGet`) throws `CORRUPTED_DATA` - /// on any different bytes at the key, which is correct for the ref-log lane's immutable, - /// content-addressed keys (a different value there truly is impossible-by-construction) but - /// wrong for a mutable state marker (e.g. a blob's freshness-meta sidecar), where a - /// pre-existing DIFFERENT value is an expected, non-corrupt state a racing writer or GC pass - /// left behind. Performs at most `budget.max_attempts` attempts of the exact SAME (key, bytes), - /// bounded by `budget.operation_deadline_ms`, with the same fence/backoff/deadline gates as - /// `putIfAbsentControlled`. An ambiguous attempt (`PreconditionFailed`, or a transient exception - /// classified `Unresolved`) is resolved with ONE GET at `key`: - /// - absent -> the attempt provably never applied; another attempt of - /// the SAME (key, bytes) is legal (fence/backoff/deadline-gated) - /// - present, bytes equal `bytes` -> Committed (an earlier ambiguous attempt of THIS call, - /// or a racing writer creating the identical value, already landed); `token` is the observed - /// incarnation - /// - present, bytes differ -> Conflict: something else already occupies the key with - /// a different value; returned as a value, never thrown - /// - the GET itself fails -> still ambiguous; reissue is safe - /// Same DefiniteFailure/deterministic-local-failure rethrow convention as `putOverwriteControlled`. - CasOverwriteResult putIfAbsentControlledMutable(std::string_view key, std::string_view bytes, - const std::function & fence_ok); - - /// A DEDICATED RAW slot-occupy primitive [codex finding 3]: exactly ONE fence/deadline-gated - /// conditional create of `bytes` at `key`; on conflict, exactly ONE raw exact `GET` of the - /// occupant. NEVER retries internally, NEVER lists, and NEVER composes `putIfAbsentControlled` - /// (which retries the SAME (key, bytes) internally) or `resolveByExactGet` (which compares against - /// an expected body and throws `CORRUPTED_DATA` on a mismatch) — both contradict "one conditional - /// create" and `Occupied(bytes, token)` respectively. This is the primitive every seal writer and - /// wedge retry uses (spec INV-2): each CALL is one bounded attempt, and a caller that wants to keep - /// trying calls this again later, under its OWN fence/deadline/backoff discipline. - /// - /// `fence_ok` and the operation deadline are checked before the (only) create attempt — a refusal - /// there sends nothing and reports `Unresolved`/`NoAttemptSent`, exactly like every other - /// controlled op's first iteration. If the create conflicts or is ambiguous, `fence_ok` is checked - /// once more immediately before its resolution GET; refusal starts no GET and reports - /// `Unresolved`/`AttemptsExhausted`, because the create was already sent. There is deliberately no - /// post-I/O fence recheck after either request: verifying that a `Created`/`Occupied` result is still - /// relevant remains the caller's contract (Task 4/6's recheck under its own state lock). - /// - /// CONSEQUENCE, stated bluntly because it is the OPPOSITE of every sibling op's behavior: a - /// `Created` or `Occupied` returned here may come from a call whose fence was lost WHILE the PUT or - /// GET was in flight — this primitive does not know and does not check. Acting on either result - /// (adopting, acknowledging, installing) without the caller's OWN post-I/O - /// `checkFenceOrThrow(admitted_generation)` under its own lock is a correctness bug, not a missed - /// diagnostic — see Task 4's `resolveWedgeOnce` and Task 6's recovery CAS-walk in the plan for the - /// exact recheck shape. - /// - /// A whitelisted synchronous rejection (`classifyConditionalWriteResult`'s `DefiniteFailure`) or a - /// deterministic local failure (`isDeterministicLocalFailure`) RETHROWS the original exception - /// unchanged — the same convention as `putOverwriteControlled`/ - /// `putIfAbsentControlledMutable` (`SlotOccupyResult::Kind` has no `DefiniteFailure` member to carry - /// it). Any other exception, or a clean `PreconditionFailed`, is ambiguous and falls through to the - /// resolve GET identically — this primitive cannot and does not distinguish the two. - /// - /// Op-count contract (asserted by every `gtest_cas_slot_occupy.cpp` test): `Created` costs exactly - /// one backend op (the create); `Occupied` costs exactly two (the create, then the resolve GET); - /// `Unresolved` costs at most two (zero when a pre-attempt gate refuses, one when admission is lost - /// before resolution, otherwise the create plus a resolve GET that came up empty or failed). - SlotOccupyResult slotOccupy(std::string_view key, std::string_view bytes, - const std::function & fence_ok); - - /// Test-only: replace the inter-attempt backoff sleep (e.g. with a no-op) on an already-constructed - /// controller — for tests that reach the controller only through a fully-wired Pool/disk and cannot - /// pass the ctor parameter (see `Pool::setCasRetrySleepForTest`). Passing an empty function restores - /// the real sleep. Not thread-safe: call before driving any traffic through the controller. - void setSleepFnForTest(std::function sleep_ms_); - -private: - CasOverwriteResult putOverwriteControlledImpl( - std::string_view key, - std::string_view bytes, - const Token & expected, - const CasOverwriteOperationContext & context, - bool preserve_legacy_gates); - - /// The gate between a completed ambiguous attempt and its reissue: fence check FIRST (a fence lost - /// mid-loop must abort before any sleep), then the capped-exponential backoff sleep — skipped - /// entirely (returning false, no sleep served) when the sleep plus one more attempt could not fit - /// the operation deadline. Returns true when the loop may proceed to the next attempt; the loop - /// top's own pre-attempt fence/deadline checks re-run AFTER the sleep. - /// - /// `out_reason` (optional) receives WHICH of the two refusals returned false, so a caller reporting - /// an `Unresolved` from here does not have to guess between them. Both are mid-way by construction: - /// this gate is only reached once an attempt has been sent. - bool pauseBeforeReissue(uint32_t completed_attempt, uint64_t deadline_ms, const std::function & fence_ok, - CasUnresolvedReason * out_reason = nullptr); - /// The backoff scheduled before attempt `next_attempt` (attempt 2 sleeps `retry_initial_backoff_ms`, - /// doubling per reissue), saturating at `retry_max_backoff_ms`. 0 when backoff is disabled. - uint64_t backoffBeforeAttempt(uint32_t next_attempt) const; - - BackendPtr backend; - CasRequestBudget budget; - std::function now_ms; - std::function sleep_ms; -}; - -} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 3fdf7e417611..aa13743e2f77 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -1,9 +1,10 @@ #include -#include #include +#include #include #include +#include #include #include "config.h" @@ -36,6 +37,7 @@ namespace DB::ErrorCodes extern const int CAS_DELETE_MARKER; extern const int CORRUPTED_DATA; extern const int LOGICAL_ERROR; + extern const int NETWORK_ERROR; extern const int NOT_IMPLEMENTED; } @@ -57,6 +59,48 @@ void recordReissue() } +namespace +{ +/// Shared by the two entry points below so the log line and the exception's message text can never +/// drift apart. Rate-limited (not per-distinct-`why` -- `LogSeriesLimiter` keys on the LOGGER NAME +/// only, so under a sustained outage where `why` keeps changing slightly, only the first message in +/// each window prints; this is the intended throttle, not a bug). Warning-level visibility is +/// intentional: this condition is expected to self-heal (the caller retries), but an operator watching +/// CAS logs directly should see it without having to know to look at system.replication_queue. +void logCasWriteRetryLater(const String & why) +{ + LogSeriesLimiter log(getLogger("CasWriteRetryLater"), /*allowed_count=*/1, /*interval_s=*/30); + LOG_WARNING(log, "CAS write could not be committed ({}); retrying later", why); +} +} + +[[noreturn]] void throwCasWriteRetryLater(const String & why) +{ + logCasWriteRetryLater(why); + throw Exception(ErrorCodes::NETWORK_ERROR, "CAS write could not be committed ({}); retrying later", why); +} + +std::exception_ptr makeCasWriteRetryLaterExceptionPtr(const String & why) +{ + logCasWriteRetryLater(why); + return std::make_exception_ptr( + Exception(ErrorCodes::NETWORK_ERROR, "CAS write could not be committed ({}); retrying later", why)); +} + +[[noreturn]] void throwCasTransientUnavailable(const String & subject, const String & condition) +{ + /// The code is coarse (it shares a `system.errors` row with socket failures), so the MESSAGE must + /// carry the whole truth: which CA condition refused, and that the refusal is a state rather than + /// damage. Consumers key on the code; operators read this line. + /// + /// The shared suffix carries ONLY the classification, because that is the one claim true at every + /// site: retry-later is right even where the condition may turn out terminal, since the next attempt + /// re-decides against fresh state. Any promise about HOW the condition clears belongs in `condition`, + /// where the site that can actually prove it makes it -- `checkFenceOrThrow` provably cannot. + throw Exception(ErrorCodes::NETWORK_ERROR, + "{} -- {}; TRANSIENT unavailability, not damage", subject, condition); +} + namespace { @@ -121,15 +165,15 @@ GaveUp::Source sourceFor(const Retry::Bound & bound) /// Could the precondition this write was built with still be met by what the resolve read saw? A /// create needs the key absent; a replace needs the incarnation it named to still be current. -bool preconditionStillSatisfiable(const Observation & seen, const std::optional & expected) +bool preconditionStillSatisfiable(const Observation & seen, const std::optional & expected) { return std::visit(detail::Overload{ /// The read itself failed, so it proved nothing either way and an ambiguous attempt may still /// be alive. Reporting a conflict on it would name an occupant nobody observed. [](const NotObserved &) { return true; }, [&](const ProvenAbsent &) { return !expected.has_value(); }, - [&](const Meta & m) { return expected.has_value() && m.incarnation == *expected; }, - [&](const Object & o) { return expected.has_value() && o.incarnation == *expected; }}, + [&](const Meta & m) { return expected.has_value() && m.etag == *expected; }, + [&](const Object & o) { return expected.has_value() && o.etag == *expected; }}, seen); } @@ -139,7 +183,7 @@ bool preconditionStillSatisfiable(const Observation & seen, const std::optional< Observation withoutBody(Observation seen) { if (const auto * obj = std::get_if(&seen)) - return Meta{obj->bytes.size(), obj->incarnation}; + return Meta{obj->bytes.size(), obj->etag}; return seen; } @@ -206,14 +250,14 @@ CasOperation CasRequests::resume(uint64_t admitted_generation, Liveness liveness return CasOperation(*this, admitted_generation, std::move(liveness)); } -std::optional CasRequests::tryMint(const String & key, String value) const +std::optional CasRequests::tryMint(const String & key, String value) const { if (!isIncarnationValue(backend->dialect(), value)) return std::nullopt; - return Incarnation(backend->backendId(), key, backend->dialect(), std::move(value)); + return Etag(backend->backendId(), key, backend->dialect(), std::move(value)); } -Incarnation CasRequests::mint(const String & key, String value) const +Etag CasRequests::mint(const String & key, String value) const { if (auto minted = tryMint(key, value)) return std::move(*minted); @@ -221,7 +265,7 @@ Incarnation CasRequests::mint(const String & key, String value) const "CAS: the store answered for '{}' with a value '{}' that is not a valid incarnation", key, value); } -const String & CasRequests::valueFor(const String & key, const Incarnation & inc) const +const String & CasRequests::valueFor(const String & key, const Etag & inc) const { if (inc.key() != key || inc.backendId() != backend->backendId()) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -333,20 +377,20 @@ std::optional CasOperation::headUnder(const String & key, const Retry & po }); } -KeyPage CasOperation::listUnder(const String & prefix, const String & cursor, size_t limit, +ListPage CasOperation::listUnder(const String & prefix, const String & cursor, size_t limit, const Retry & policy, const Retry::Bound & bound) { return readLoop("list", prefix, policy, bound, [&](auto & access) { Backend::RawListPage raw = owner.backend->list(prefix, cursor, limit, access); - KeyPage page; + ListPage page; page.next_cursor = std::move(raw.next_cursor); page.keys.reserve(raw.keys.size()); for (auto & listed : raw.keys) { - KeyEntry entry{std::move(listed.key), listed.size, std::nullopt}; + ListedKey entry{std::move(listed.key), listed.size, std::nullopt}; if (listed.value) - entry.incarnation = owner.mint(entry.key, std::move(*listed.value)); + entry.etag = owner.mint(entry.key, std::move(*listed.value)); page.keys.push_back(std::move(entry)); } return page; @@ -384,21 +428,21 @@ std::optional CasOperation::head(const String & key, const Retry & policy) return headUnder(key, policy, policy.bind(owner.now_ms())); } -KeyPage CasOperation::list(const String & prefix, const String & cursor, size_t limit, const Retry & policy) +ListPage CasOperation::list(const String & prefix, const String & cursor, size_t limit, const Retry & policy) { return listUnder(prefix, cursor, limit, policy, policy.bind(owner.now_ms())); } -void CasOperation::forEachListedKey(const String & prefix, const KeyEntryFn & fn, const Retry & per_page, +void CasOperation::forEachListedKey(const String & prefix, const ListedKeyFn & fn, const Retry & per_page, size_t page_limit, const std::function & on_page_fetched) { String cursor; for (;;) { - KeyPage page = list(prefix, cursor, page_limit, per_page); + ListPage page = list(prefix, cursor, page_limit, per_page); if (on_page_fetched) on_page_fetched(); - for (const KeyEntry & entry : page.keys) + for (const ListedKey & entry : page.keys) if (!fn(entry)) return; if (page.next_cursor.empty()) @@ -407,7 +451,7 @@ void CasOperation::forEachListedKey(const String & prefix, const KeyEntryFn & fn } } -Removal CasOperation::remove(const String & key, const Incarnation & seen, const Retry & policy) +Removal CasOperation::remove(const String & key, const Etag & seen, const Retry & policy) { return removeUnder(key, owner.valueFor(key, seen), policy, policy.bind(owner.now_ms())); } @@ -420,7 +464,7 @@ Removal CasOperation::removeCurrent(const String & key, const Retry & policy) const std::optional seen = headUnder(key, policy, bound); if (!seen) return Removal::Gone; - const Removal removed = removeUnder(key, owner.valueFor(key, seen->incarnation), policy, bound); + const Removal removed = removeUnder(key, owner.valueFor(key, seen->etag), policy, bound); if (removed != Removal::Mismatch) return removed; @@ -610,7 +654,7 @@ WriteResult CasOperation::gaveUpAfterFailedObservation(std::optional s return gaveUp(GaveUp::Why::Unresolved, sourceFor(bound), state); } -WriteResult CasOperation::postCommit(Incarnation inc, bool resolved_by_read, WriteState & state, const Retry::Bound & bound) +WriteResult CasOperation::postCommit(Etag inc, bool resolved_by_read, WriteState & state, const Retry::Bound & bound) { /// Admission once more, now that the write is proven durable: a fence lost here means the object /// may well exist, but this call must never claim it -- the caller has to resolve the key instead. @@ -645,7 +689,7 @@ std::optional CasOperation::pauseAndReissue(WriteState & state, con return std::nullopt; } -WriteResult CasOperation::writeLoop(const String & key, const String & bytes, const std::optional & expected, +WriteResult CasOperation::writeLoop(const String & key, const String & bytes, const std::optional & expected, const Retry & policy, const Retry::Bound & bound, WriteState & state, ResolveWith resolve_refusal_with) { @@ -774,7 +818,7 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// did not move our own bytes cannot be told from the bytes already there. A create reaches /// here for every object it sees -- an occupied key never satisfies its precondition. if (const auto * obj = std::get_if(&state.last_seen); obj && obj->bytes == bytes) - return postCommit(obj->incarnation, /*resolved_by_read=*/true, state, bound); + return postCommit(obj->etag, /*resolved_by_read=*/true, state, bound); /// Nothing of this inner write's is at the key, and a reissue would be refused too. return Conflict{state.last_seen, state.attempts_sent}; } @@ -796,7 +840,7 @@ WriteResult CasOperation::create(const String & key, const String & bytes, const return writeLoop(key, bytes, std::nullopt, policy, policy.bind(owner.now_ms()), state, ResolveWith::Body); } -WriteResult CasOperation::replace(const String & key, const String & bytes, const Incarnation & seen, const Retry & policy) +WriteResult CasOperation::replace(const String & key, const String & bytes, const Etag & seen, const Retry & policy) { WriteState state; return writeLoop(key, bytes, seen, policy, policy.bind(owner.now_ms()), state, ResolveWith::Body); @@ -824,7 +868,7 @@ WriteResult CasOperation::readModifyWrite(const String & key, const DecideOnObje return Declined{state.last_seen}; WriteResult result = writeLoop(key, *next, - current ? std::optional(current->incarnation) : std::nullopt, policy, bound, state, + current ? std::optional(current->etag) : std::nullopt, policy, bound, state, ResolveWith::Body); if (!std::holds_alternative(result)) return result; @@ -877,7 +921,7 @@ WriteResult CasOperation::readModifyWriteOnPresence(const String & key, const De return Declined{state.last_seen}; WriteResult result = writeLoop(key, *next, - current ? std::optional(current->incarnation) : std::nullopt, policy, bound, state, + current ? std::optional(current->etag) : std::nullopt, policy, bound, state, ResolveWith::Presence); /// A refused precondition was settled by a HEAD, but proving an ambiguous attempt landed needs /// the bytes; this loop is presence-only by contract, so that body stops here. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index 75aa3fee87c6..de61d8be400a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -1,7 +1,7 @@ #pragma once #include #include -#include +#include #include #include #include @@ -34,6 +34,57 @@ bool isDefinitelyRefusedWrite(const std::exception & e); /// `NOT_IMPLEMENTED`, `BAD_ARGUMENTS` and `CORRUPTED_DATA`. bool isDeterministicLocalFailure(int code); +/// Throw the recoverable "CAS write could not be committed, retry later" condition. +/// +/// WHY NETWORK_ERROR (this replaces an earlier ABORTED throw): +/// A content-addressed write can fail for a reason that is neither the caller's fault +/// nor permanent: the mount-lease / write fence was lost (e.g. a renewal PUT timed out +/// against a slow or throttling object store), or a conditional PUT exhausted its retry +/// budget mid-outage. The right response is "abandon this attempt, try again later" -- +/// which is precisely what a transient error means. +/// +/// It previously threw `ABORTED`, which was actively harmful to background merges: +/// `ReplicatedMergeMutateTaskBase` treats `ABORTED` as "merge deliberately cancelled +/// (shutdown / `DROP` / merges-blocker), not an error", so it neither records +/// `last_exception_time_ms` nor lets `ReplicatedMergeTreeQueue`'s exponential backoff +/// engage. Under a sustained store outage the queue re-executed the merge roughly every +/// 2 seconds, recomputing the whole (possibly multi-GiB) output part every time for the +/// entire outage -- hundreds of full recomputes, and invisible in system.replication_queue. +/// +/// `NETWORK_ERROR` is the best-fitting EXISTING code: +/// - it is NOT in the merge "retry silently, no backoff" exemption set (only `ABORTED` +/// and `PART_IS_TEMPORARILY_LOCKED` are), so the existing backoff -- capped by +/// `max_postpone_time_for_failed_replicated_merges_ms` -- engages automatically; +/// - it is already in ClickHouse's transient/retryable taxonomy +/// (`checkDataPart::isRetryableException` lists it beside `ABORTED`), so a part under +/// verification is not misread as corrupted; +/// - nothing on the merge / insert / replication commit path special-cases it in a way +/// that would misfire (ZooKeeper retriability keys on `Coordination::Exception`, a +/// different type), and it is not caught specially on the CAS write path. +/// +/// SCOPE: only the ESCAPING retry-later throws route here (fence lost or a controlled write outcome +/// remaining uncertain). Startup/decommission and generic live-lock-brake `ABORTED` values keep +/// their meaning and are not rerouted here. +[[noreturn]] void throwCasWriteRetryLater(const String & why); + +/// Same classification as `throwCasWriteRetryLater`, but returns the exception as a +/// `std::exception_ptr` for call sites that fail a pending future/promise rather than throw directly. +/// Both entry points route through the SAME construction internally, so the error code / message shape +/// has exactly one place that decides it. +std::exception_ptr makeCasWriteRetryLaterExceptionPtr(const String & why); + +/// Throw the recoverable "this content-addressed disk cannot serve the request right now" condition. +/// Sibling of `throwCasWriteRetryLater`, same class for the same reasons, differing only in what it +/// describes: that one names a WRITE whose commit did not land, this one names a DISK STATE that +/// refused the request before it started -- on either plane. +/// +/// `subject` names the refusing disk or pool (e.g. "content-addressed disk 'ca'") and `condition` states +/// the CA condition truthfully, INCLUDING any promise about how it clears -- only the site knows whether +/// it can make one. What is appended HERE is the classification alone, so it cannot drift between call +/// sites. Unlike `throwCasWriteRetryLater` this deliberately does not log: these sites can fire +/// repeatedly per refused operation and every caller already reports the exception it receives. +[[noreturn]] void throwCasTransientUnavailable(const String & subject, const String & condition); + /// The failure class a FRESH CREDENTIAL could fix -- a subset of `isDefinitelyRefusedWrite`, exposed /// because a caller whose own loop makes the next physical attempt must not treat it as terminal: the /// engine refreshes once before it gives the answer, and the caller's next attempt signs with what the @@ -48,23 +99,23 @@ using Liveness = std::function; using DecideOnObject = std::function(const std::optional &)>; using DecideOnMeta = std::function(const std::optional &)>; -/// One key returned by `list`. `incarnation` is present only on a backend that surfaces per-key +/// One key returned by `list`. `etag` is present only on a backend that surfaces per-key /// incarnations through LIST -- see `Backend::supportsListTokens`. -struct KeyEntry +struct ListedKey { String key; uint64_t size; - std::optional incarnation; + std::optional etag; }; /// One page of an enumeration. `next_cursor` resumes strictly after the last returned key; empty /// marks the end. -struct KeyPage +struct ListPage { - std::vector keys; + std::vector keys; String next_cursor; }; /// The walk's callback: FALSE stops the walk. -using KeyEntryFn = std::function; +using ListedKeyFn = std::function; namespace detail { @@ -77,7 +128,7 @@ class CasOperation; /// The only caller of `Backend`. It owns the three things a physical request must be measured /// against -- the transport, the mount fence, and the clock -- and it is the sole minter of -/// `Incarnation`, so a caller can hold one only by way of a request this class admitted. +/// `Etag`, so a caller can hold one only by way of a request this class admitted. /// /// It is constructed with a fence because a fence is a property of whoever holds the lease, not of a /// call: the mount plane passes the mount fence, the GC plane and the offline tools an open one. A @@ -129,14 +180,14 @@ class CasRequests /// The store's answer for `key`, as an incarnation. Throws `CORRUPTED_DATA` naming the key when /// the value fails this backend's dialect grammar. - Incarnation mint(const String & key, String value) const; + Etag mint(const String & key, String value) const; /// `mint` without the verdict, for the one caller that must treat a malformed value as an /// ambiguity to settle by reading rather than as corruption: a write's own 2xx response. - std::optional tryMint(const String & key, String value) const; + std::optional tryMint(const String & key, String value) const; /// The transport value to send as a precondition. Throws `LOGICAL_ERROR` when the incarnation /// names another key or another backend -- a precondition built from it would silently mean /// something else. - const String & valueFor(const String & key, const Incarnation & inc) const; + const String & valueFor(const String & key, const Etag & inc) const; BackendPtr backend; Fence fence; @@ -170,14 +221,14 @@ class CasOperation std::optional read(const String & key, const Retry & policy); std::optional head(const String & key, const Retry & policy); - KeyPage list(const String & prefix, const String & cursor, size_t limit, const Retry & policy); + ListPage list(const String & prefix, const String & cursor, size_t limit, const Retry & policy); /// Walks every key under `prefix` exactly once. The policy governs EACH PAGE, not the walk: a walk /// is an unbounded number of requests, and a silently truncated enumeration is the error a /// coverage record exists to prevent. `on_page_fetched` fires once per page DELIVERED; a page that /// took several reissues still fires once, and `CASRequestAttempt` is the physical count. - void forEachListedKey(const String & prefix, const KeyEntryFn & fn, const Retry & per_page, + void forEachListedKey(const String & prefix, const ListedKeyFn & fn, const Retry & per_page, size_t page_limit = 1000, const std::function & on_page_fetched = {}); - Removal remove(const String & key, const Incarnation & seen, const Retry & policy); + Removal remove(const String & key, const Etag & seen, const Retry & policy); /// `head` then `remove` of what it saw, repeating on `Mismatch`. `Gone` when the key is already /// absent; never returns `Mismatch` -- under `once`, where there is no reissue to resolve one, a /// `Mismatch` is the retry-later throw the read verbs use when their policy is exhausted. @@ -193,7 +244,7 @@ class CasOperation void publish(const BlobPublishRequest & request, const Retry & policy); WriteResult create(const String & key, const String & bytes, const Retry & policy); - WriteResult replace(const String & key, const String & bytes, const Incarnation & seen, const Retry & policy); + WriteResult replace(const String & key, const String & bytes, const Etag & seen, const Retry & policy); /// Read, decide, write, and re-decide on conflict against what the write's own resolve read /// already observed. `decide` returning nullopt is `Declined`. WriteResult readModifyWrite(const String & key, const DecideOnObject & decide, const Retry & policy); @@ -257,7 +308,7 @@ class CasOperation std::optional readUnder(const String & key, const Retry & policy, const Retry::Bound & bound); std::optional headUnder(const String & key, const Retry & policy, const Retry::Bound & bound); - KeyPage listUnder(const String & prefix, const String & cursor, size_t limit, + ListPage listUnder(const String & prefix, const String & cursor, size_t limit, const Retry & policy, const Retry::Bound & bound); Removal removeUnder(const String & key, const String & expected_value, const Retry & policy, const Retry::Bound & bound); @@ -269,7 +320,7 @@ class CasOperation /// The write engine: one call, any policy. Settles every refused precondition and every ambiguity /// by an exact read before it reports anything. - WriteResult writeLoop(const String & key, const String & bytes, const std::optional & expected, + WriteResult writeLoop(const String & key, const String & bytes, const std::optional & expected, const Retry & policy, const Retry::Bound & bound, WriteState & state, ResolveWith resolve_refusal_with); /// The resolve read: an exact read under the same policy and deadline, reporting what it saw and, @@ -278,7 +329,7 @@ class CasOperation /// The presence-only sibling, for the one loop that must not fetch a body. Resolved observePresence(const String & key, const Retry & policy, const Retry::Bound & bound); - WriteResult postCommit(Incarnation inc, bool resolved_by_read, WriteState & state, const Retry::Bound & bound); + WriteResult postCommit(Etag inc, bool resolved_by_read, WriteState & state, const Retry::Bound & bound); WriteResult gaveUp(GaveUp::Why why, GaveUp::Source source, WriteState & state) const; /// The bound that refused the resolve read, reported as the outcome it actually is. WriteResult gaveUpForReadStop(ReadStop stop, WriteState & state, const Retry::Bound & bound) const; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp index 0f17ef3c4be1..229847ff1d76 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp @@ -45,7 +45,7 @@ BootstrapResidual probePoolBootstrapResidual(CasOperation & op, const Layout & l bool has_catalog = false; try { - op.forEachListedKey(prefix, [&](const KeyEntry & listed) -> bool + op.forEachListedKey(prefix, [&](const ListedKey & listed) -> bool { if (listed.key == pool_meta_key) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h index 96e9545832b8..4e00208ac57c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h @@ -103,84 +103,6 @@ class ThrottlingBackend final : public Backend return inner->probeSentinelRaw(key, access); } - /// ---- The legacy surface, refused and forwarded AS legacy ---- - /// - /// Not inherited from `Backend`: its forwarder would call the primitive on THIS object, so the - /// inner backend would receive a primitive and any legacy override it carries would never run. - /// Each request is refused (or not) exactly once, on whichever surface its caller used. - std::optional getStream(const String & key, Range range) override - { - refuseOrPass(key); - return inner->getStream(key, range); - } - - std::optional get(const String & key, Range range) override - { - refuseOrPass(key); - return inner->get(key, range); - } - - HeadResult head(const String & key) override - { - refuseOrPass(key); - return inner->head(key); - } - - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - refuseOrPass(key); - return inner->putIfAbsent(key, bytes, meta); - } - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, - const ObjectMeta & meta) override - { - refuseOrPass(key); - return inner->putOverwrite(key, bytes, expected, meta); - } - - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override - { - refuseOrPass(key); - return inner->casPut(key, bytes, expected, meta); - } - - DeleteOutcome deleteExact(const String & key, const Token & token) override - { - refuseOrPass(key); - return inner->deleteExact(key, token); - } - - ListPage list(const String & prefix, const String & cursor, size_t limit) override - { - refuseOrPass(prefix); - return inner->list(prefix, cursor, limit); - } - - void publishBlob(const BlobPublishRequest & request) override - { - refuseOrPass(request.destination_key); - inner->publishBlob(request); - } - - SentinelProbeResult probeSentinelRaw(const String & key) override - { - refuseOrPass(key); - return inner->probeSentinelRaw(key); - } - - /// Unhide the base overloads this class's own declarations would otherwise shadow: the - /// convenience forms that omit Range/ObjectMeta/expected-token. - using Backend::get; - using Backend::getStream; - using Backend::head; - using Backend::list; - using Backend::probeSentinelRaw; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - /// Facts about the wrapped backend, not requests to refuse. Dialect dialect() const override { return inner->dialect(); } bool supportsListTokens() const override { return inner->supportsListTokens(); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h index 4b54449a147f..02ecdbc873cb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h @@ -3,14 +3,12 @@ namespace DB::Cas { -/// A capability token: holding one proves the holder is `CasRequests` (or, during the migration, -/// `Backend` -- deleted at the lock, Task 20). Not copyable, not constructible outside those two -/// friends, and carries no data -- its only job is to gate access at compile time to the backend -/// entry points that must not be called except through the contract. +/// A capability token: holding one proves the holder is `CasRequests`. Not copyable, not +/// constructible outside that one friend, and carries no data -- its only job is to gate access at +/// compile time to the backend entry points that must not be called except through the contract. class TransportAccess { friend class CasRequests; - friend class Backend; /// migration only; deleted at the lock (Task 20) TransportAccess() = default; public: diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h index d6a8a172007e..4312c9d9da0f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h @@ -1,8 +1,5 @@ #pragma once -#include -/// `throwCasWriteRetryLater` / `throwCasTransientUnavailable` are declared here today; the lock moves -/// them into `CasRequests.h` alongside the rest of this contract. -#include +#include #include #include @@ -22,8 +19,14 @@ namespace DB::ErrorCodes namespace DB::Cas { -struct Object { String bytes; Incarnation incarnation; }; -struct Meta { uint64_t size; Incarnation incarnation; }; +/// Declared (and defined) in `CasRequests.h`/`.cpp`. Forward-declared narrowly here, rather than +/// including that header, because `CasRequests.h` itself includes this one for `WriteResult` -- +/// including it back would be circular. +[[noreturn]] void throwCasWriteRetryLater(const String & why); +[[noreturn]] void throwCasTransientUnavailable(const String & subject, const String & condition); + +struct Object { String bytes; Etag etag; }; +struct Meta { uint64_t size; Etag etag; }; enum class Removal : uint8_t { Removed, Gone, Mismatch }; /// What a write attempt observed of the key's current state before giving up, so a caller (or the @@ -32,11 +35,11 @@ struct NotObserved {}; struct ProvenAbsent {}; using Observation = std::variant; -/// A durable write landed: `incarnation` names the incarnation it created (or, for a retried write +/// A durable write landed: `etag` names the incarnation it created (or, for a retried write /// resolved by a read, the incarnation already present), `attempts_sent` counts the HTTP attempts this /// call made, and `resolved_by_read` is true when the commit was proven by a read rather than by the /// attempt's own response. -struct Committed { Incarnation incarnation; uint32_t attempts_sent; bool resolved_by_read; }; +struct Committed { Etag etag; uint32_t attempts_sent; bool resolved_by_read; }; /// The write was never attempted or never needed -- e.g. `putIfAbsent` finding the key already /// present under the caller's intended content. `seen` is whatever the resolve read observed. struct Declined { Observation seen; }; @@ -81,7 +84,7 @@ inline String renderObservation(const Observation & seen) [](const NotObserved &) -> String { return "nothing observed"; }, [](const ProvenAbsent &) -> String { return "absent"; }, [](const Meta &) -> String { return "present (meta)"; }, - [](const Object & o) -> String { return "present (" + o.incarnation.render() + ")"; }}, seen); + [](const Object & o) -> String { return "present (" + o.etag.render() + ")"; }}, seen); } } @@ -90,22 +93,22 @@ inline String renderObservation(const Observation & seen) /// (nothing changed, nothing to report), the committed incarnation otherwise -- or throw, mapping /// every non-success alternative to the error class its meaning already implies. `what` names the /// call for the thrown message. -inline std::optional orThrow(WriteResult && result, std::string_view what) +inline std::optional orThrow(WriteResult && result, std::string_view what) { using detail::Overload; using detail::renderObservation; return std::visit(Overload{ - [](Committed & c) -> std::optional { return std::move(c.incarnation); }, - [](Declined &) -> std::optional { return std::nullopt; }, - [&](Conflict & c) -> std::optional + [](Committed & c) -> std::optional { return std::move(c.etag); }, + [](Declined &) -> std::optional { return std::nullopt; }, + [&](Conflict & c) -> std::optional { throw Exception(ErrorCodes::ABORTED, "{}: conflict, observed {}", what, renderObservation(c.seen)); }, - [&](Refused & r) -> std::optional + [&](Refused & r) -> std::optional { throw Exception(r.store_error, "{}: the store refused the write: {}", what, r.message); }, - [&](GaveUp & g) -> std::optional + [&](GaveUp & g) -> std::optional { switch (g.why) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index fedde91b1c9a..7e2d88658fe0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -5,7 +5,7 @@ #include #include #include -#include +#include #include #include #include @@ -81,6 +81,8 @@ namespace ContentAddressedSetting extern const ContentAddressedSettingsUInt64 part_folder_cache_max_entry_bytes; extern const ContentAddressedSettingsUInt64 manifest_decode_cache_bytes; extern const ContentAddressedSettingsUInt64 gc_meta_pool_size; + extern const ContentAddressedSettingsUInt64 attempt_timeout_ms; + extern const ContentAddressedSettingsUInt64 lease_safety_margin_ms; extern const ContentAddressedSettingsBool blob_hash_allow_new; } @@ -298,6 +300,8 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage( , cas_part_folder_cache_max_entry_bytes(settings_[ContentAddressedSetting::part_folder_cache_max_entry_bytes].value) , manifest_decode_cache_bytes(settings_[ContentAddressedSetting::manifest_decode_cache_bytes].value) , gc_meta_pool_size(settings_[ContentAddressedSetting::gc_meta_pool_size].value) + , cas_attempt_timeout_ms(settings_[ContentAddressedSetting::attempt_timeout_ms].value) + , cas_lease_safety_margin_ms(settings_[ContentAddressedSetting::lease_safety_margin_ms].value) , staging_backend(settings_.stagingBackend()) , blob_hash_algo(settings_.blobHashAlgo()) , blob_hash_allow_new(settings_[ContentAddressedSetting::blob_hash_allow_new].value) @@ -761,6 +765,8 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.gc_round_handoff_prefix_wholesale_budget = gc_round_handoff_prefix_wholesale_budget; pool_config.gc_round_outcome_entry_budget = gc_round_outcome_entry_budget; pool_config.gc_meta_pool_size = gc_meta_pool_size; + pool_config.cas_request_budget.attempt_timeout_ms = cas_attempt_timeout_ms; + pool_config.cas_request_budget.lease_safety_margin_ms = cas_lease_safety_margin_ms; pool_config.event_sink = makeCasEventSink(); /// Built here rather than above so it carries the budget the pool was configured with. Only a @@ -884,7 +890,7 @@ void ContentAddressedMetadataStorage::startup() /// that swapped the client under that state would leave persisted tokens uncomparable. The refusal /// has to happen in the object storage: only there is the effective `http_client` known, merged from /// the storage's current settings, any endpoint-level block and the disk's own section. - object_storage->pinConditionalOpsGenerationDialect(native_token_type == Cas::TokenType::Generation); + object_storage->pinConditionalOpsGenerationDialect(native_token_type == Cas::Dialect::Generation); } void ContentAddressedMetadataStorage::shutdown() @@ -1237,7 +1243,7 @@ void ContentAddressedMetadataStorage::confirmPoolIdentityForEmptyEnumeration(con ++empty_proof_probe_count_for_test; /// The open plane: this probe is what authorizes an empty answer, and a mount whose lease has /// blipped must still be able to ask it. - Cas::CasOperation probe_op = pool->gcRequests().admit(); + Cas::CasOperation probe_op = pool->openRequests().admit(); const Cas::SentinelProbeResult probe = empty_proof_probe_override_for_test ? empty_proof_probe_override_for_test() /// `once`: this probe is the gate that authorises an empty answer, and an inconclusive one is a diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 15f73b32117a..183b470367c9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -610,6 +610,13 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC const uint64_t manifest_decode_cache_bytes; /// Bounded pool size for GC's per-hash freshness-metadata writes. const uint64_t gc_meta_pool_size; + /// The budget for one HTTP attempt of a writable Native mount's control-plane requests; feeds + /// `Cas::PoolConfig::cas_request_budget.attempt_timeout_ms` and the backend's own + /// `attemptTimeoutMs()`. + const uint64_t cas_attempt_timeout_ms; + /// Startup-only margin validated against the mount lease TTL; feeds + /// `Cas::PoolConfig::cas_request_budget.lease_safety_margin_ms`. + const uint64_t cas_lease_safety_margin_ms; /// Configured staging backend; `Local` preserves the existing write path. const Cas::StagingBackend staging_backend; /// Blob content-hash function passed to `Cas::PoolConfig`. @@ -642,7 +649,7 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// native_token_type`) -- immutable afterwards. `startup` also hands it to the object storage as a /// pin, which is what refuses a reload that would flip the dialect under this live pool; the check /// belongs there because only the object storage knows the effective `http_client`. - Cas::TokenType native_token_type = Cas::TokenType::ETag; + Cas::Dialect native_token_type = Cas::Dialect::ETag; /// shared_ptr so `runGarbageCollectionRoundNow`/`runOneGcRoundForTest` can take a snapshot under /// `pointer_mutex`, release it, and run the (long) round via the snapshot -- never holding /// `pointer_mutex` itself for the round's duration, so `gcHealth`/`store`/`partAccess` never @@ -739,7 +746,7 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// captured while the concrete backend is still in scope. Reading it back through `pool` /// would mean unwrapping the instrumentation decorator `Pool::open` adds, so it is returned /// here instead. - Cas::TokenType native_token_type = Cas::TokenType::ETag; + Cas::Dialect native_token_type = Cas::Dialect::ETag; }; /// Builds the backend + `Cas::PoolConfig` and opens a pool exactly as `startup()` does. A read-only diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index 11a45c61bb8c..d1fc07746645 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -76,6 +76,8 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, part_folder_cache_max_entry_bytes, 16ULL << 20, "Oversized part-folder views bypass retention above this size", 0) \ DECLARE(UInt64, manifest_decode_cache_bytes, 128ULL << 20, "Manifest DECODE cache byte budget (0 disables)", 0) \ DECLARE(UInt64, gc_meta_pool_size, 16, "Bounded pool size for GC per-hash freshness-meta writes", 0) \ + DECLARE(UInt64, attempt_timeout_ms, 5000, "Budget for one HTTP attempt of a writable Native mount's control-plane requests", 0) \ + DECLARE(UInt64, lease_safety_margin_ms, 2000, "Startup-only margin validated against the mount lease TTL (attempt_timeout_ms + this must be strictly less than the lease TTL)", 0) \ DECLARE(String, staging_backend, "local", "Blob staging backend (local | s3); s3 is opt-in", 0) \ DECLARE_SETTINGS_TRAITS(ContentAddressedSettingsTraits, LIST_OF_CONTENT_ADDRESSED_SETTINGS, CONTENT_ADDRESSED_SETTINGS_SUPPORTED_TYPES) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h index c8fd6d8919c9..adc7bad57305 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -41,7 +41,7 @@ struct OutcomeEntry { ObjectKind kind = ObjectKind::Blob; BlobRef ref{}; - PersistedIncarnation token; + PersistedEtag token; OutcomeKind outcome = OutcomeKind::Spared; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h index e18da0869b0a..5741a61fc9b7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -58,7 +58,7 @@ inline RunMarker runMarkerFromByte(char byte, std::string_view what) /// compression) + `Strict` (byte-deterministic for `putDeterministicArtifact` adoption). /// /// This file is backend-free: it accepts caller-owned `ReadBuffer`/`WriteBuffer` objects and reaches -/// no backend or GC machinery -- `PersistedIncarnation` is a value type with no live backend behind +/// no backend or GC machinery -- `PersistedEtag` is a value type with no live backend behind /// it, which is exactly why a persisted row may hold one. The GC layer owns the stream lifetime and the bridge to /// packed keys and condemned rows; this codec owns only the durable text representation and its /// identifier-layer types. Keeping that boundary physical prevents storage or GC dependencies from @@ -87,7 +87,7 @@ struct SourceEdgeRecord UInt128 source_id{}; RunMarker marker = RunMarker::Edge; bool delete_pending = false; - PersistedIncarnation token{}; + PersistedEtag token{}; uint64_t size = 0; uint64_t condemn_round = 0; bool marker_confirmed = false; /// durable Condemned meta confirmed for this entry (graduation gate) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp index 93bda6fd1b91..ab5fe8344082 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.cpp @@ -17,7 +17,7 @@ namespace ErrorCodes namespace DB::Cas { -static_assert(casEnumTableCoversEnum()); +static_assert(casEnumTableCoversEnum()); static_assert(casEnumTableCoversEnum()); std::string_view dialectWordFromString(std::string_view w, std::string_view what) @@ -56,7 +56,7 @@ ObjectKind objectKindFromWord(std::string_view w, std::string_view what) return kObjectKindWords.fromWord(w, what); } -void writeTokenFields(CasJsonWriter & out, bool & first, const PersistedIncarnation & inc) +void writeTokenFields(CasJsonWriter & out, bool & first, const PersistedEtag & inc) { writeStringField(out, SharedWire::token_type, dialectWordFromString(inc.dialect, "wire: dialect"), first); writeStringField(out, SharedWire::token, inc.value, first); @@ -118,11 +118,11 @@ BlobRef BlobRefFields::build(std::string_view what) const return BlobRef{algo, codecFor(algo).fromHex(*digest_hex)}; } -PersistedIncarnation TokenFields::build(std::string_view what) const +PersistedEtag TokenFields::build(std::string_view what) const { if (!type_word || !value) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS {}: token missing token_type/token", what); - return PersistedIncarnation{String(dialectWordFromString(*type_word, what)), *value}; + return PersistedEtag{String(dialectWordFromString(*type_word, what)), *value}; } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h index c370e60db464..9741b258e785 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasWireVocab.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -21,10 +21,10 @@ namespace DB::Cas /// The incarnation-dialect wire vocabulary; coverage is proven in `CasWireVocab.cpp`. It has two /// persisted encodings -- the word, used by every JSON codec here, and the one byte the condemned-row /// payload stores -- and both go through this one table so they can never name different sets. -inline constexpr EnumWireTable kTokenTypeWords{{{ - {TokenType::ETag, "etag"}, - {TokenType::Generation, "generation"}, - {TokenType::Emulated, "emulated"}, +inline constexpr EnumWireTable kTokenTypeWords{{{ + {Dialect::ETag, "etag"}, + {Dialect::Generation, "generation"}, + {Dialect::Emulated, "emulated"}, }}}; /// The `ObjectKind` wire vocabulary; coverage is proven in `CasWireVocab.cpp`. @@ -58,7 +58,7 @@ ObjectKind objectKindFromWord(std::string_view w, std::string_view what); /// Append the sibling fields `token_type` and `token` to an in-progress JSON object. The caller owns `first`, /// which must describe the fields already written to that object; the value is JSON-escaped. The /// dialect is validated on the way out, so a record can never persist a word its reader would reject. -void writeTokenFields(CasJsonWriter & out, bool & first, const PersistedIncarnation & inc); +void writeTokenFields(CasJsonWriter & out, bool & first, const PersistedEtag & inc); /// Append the sibling fields `algo` and `digest` to an in-progress JSON object. The algorithm word and /// lowercase digest are canonical, and the digest is rendered at the width required by `r.algo`. @@ -160,7 +160,7 @@ struct TokenFields /// Requires both fields and validates the dialect word. `what` identifies the enclosing codec /// in `CORRUPTED_DATA` exceptions. - PersistedIncarnation build(std::string_view what) const; + PersistedEtag build(std::string_view what) const; }; /// Each `match*Fields` helper tests `key` against the one or two field names it owns, consumes the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md index 52cd954377fd..50fe520b649e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/README.md @@ -61,9 +61,10 @@ algo-width hex (two chars per digest byte), with their algo name (`sha256:ab12 hex would be ambiguous; unbounded u64 = decimal strings; bounded counts/lengths/ms-timestamps = numbers; units documented here per object as codecs land. -`CasWireVocab.{h,cpp}` owns repeated value fields: `BlobRef` uses `algo`/`digest`, `Token` uses -the jointly required `token_type`/`token`, `ManifestRef` uses `epoch`/`build`/`ord`, and owner-transition bindings use -the corresponding `old_*` and `new_*` key bundles. +`CasWireVocab.{h,cpp}` owns repeated value fields: `BlobRef` uses `algo`/`digest`, a persisted `Etag` +uses the jointly required `token_type`/`token` (the wire key spellings predate, and are independent +of, the C++ type's own name), `ManifestRef` uses `epoch`/`build`/`ord`, and owner-transition bindings +use the corresponding `old_*` and `new_*` key bundles. ## Evolution rules (one screen) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp index 9343906d7202..3170b197968d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.cpp @@ -541,12 +541,12 @@ void foldDeltasIntoGeneration(CasOperation & op, const Layout & layout, if (cur_edges == 0 && cur_touched && peek_head) { if (const auto hr = peek_head(cur_blob); - hr && !stale.token.matches(hr->incarnation)) + hr && !stale.token.matches(hr->etag)) { RetiredEntry fresh; fresh.kind = ObjectKind::Blob; fresh.ref = cur_blob; - fresh.token = PersistedIncarnation::capture(hr->incarnation); + fresh.token = PersistedEtag::capture(hr->etag); fresh.size = hr->size; fresh.condemn_round = condemn_round; ReplacedEntry re; @@ -569,7 +569,7 @@ void foldDeltasIntoGeneration(CasOperation & op, const Layout & layout, RetiredEntry fresh; fresh.kind = ObjectKind::Blob; fresh.ref = cur_blob; - fresh.token = PersistedIncarnation::capture(hr->incarnation); + fresh.token = PersistedEtag::capture(hr->etag); fresh.size = hr->size; fresh.condemn_round = condemn_round; rmr.still_retired.push_back(std::move(fresh)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h index 485d4ec3e6fa..db6f66b52f92 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasBlobInDegree.h @@ -26,7 +26,7 @@ struct RetiredEntry { ObjectKind kind = ObjectKind::Blob; BlobRef ref{}; - PersistedIncarnation token; /// the exact incarnation GC observed; the delete re-heads and compares it + PersistedEtag token; /// the exact incarnation GC observed; the delete re-heads and compares it uint64_t size = 0; uint64_t condemn_round = 0; /// the GC round that condemned this incarnation (round-paced /// graduation: an entry graduates only once condemn_round < the @@ -79,11 +79,11 @@ void assertValidSourceEdgeId(const UInt128 & source_id); struct CondemnedRow { bool delete_pending = false; - PersistedIncarnation token; // the incarnation the exact-token delete must re-observe + PersistedEtag token; // the incarnation the exact-token delete must re-observe uint64_t size = 0; uint64_t condemn_round = 0; bool marker_confirmed = false; // durable Condemned meta confirmed (graduation gate) - /// Spelled out rather than defaulted because `PersistedIncarnation` carries no equality of its + /// Spelled out rather than defaulted because `PersistedEtag` carries no equality of its /// own. A field added above belongs here too. bool operator==(const CondemnedRow & o) const { @@ -215,7 +215,7 @@ struct BlobCandidate struct ReplacedEntry { RetiredEntry fresh; /// the freshly condemned CURRENT incarnation (also pushed into still_retired byte-identically) - PersistedIncarnation old_token; /// the superseded (stale) entry's — what republication replaced + PersistedEtag old_token; /// the superseded (stale) entry's — what republication replaced }; /// One example of an unmatched-remove delta, kept for the caller's single once-per-round WARNING diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 283805bd892e..f7501805a165 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -88,7 +88,7 @@ uint64_t deletePrefixWholesale(CasOperation & op, const String & prefix, uint64_ /// The text an incarnation carries into the event log. Persisted and live incarnations render the /// same way, so the column speaks one vocabulary whichever half of the pipeline wrote the row. -String renderIncarnation(const PersistedIncarnation & token) +String renderIncarnation(const PersistedEtag & token) { return token.dialect + ":" + token.value; } @@ -342,7 +342,7 @@ void Gc::runNamespaceJanitorPage( NamespaceJanitorResult janitor_result; try { - CasRequests & requests = store->gcRequests(); + CasRequests & requests = store->openRequests(); const Layout & layout = store->layout(); NamespaceJanitor janitor(requests, layout, 1000); /// ONE authority read per page, made here rather than from the predicate: the janitor's @@ -371,7 +371,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al RoundReport & report = progress ? *progress : local_report; report = RoundReport{}; GcState state; - std::optional state_incarnation; + std::optional state_etag; /// Every exit path waits for this round's meta jobs. The throwing `meta_pool_wait` phase below is /// a protocol barrier -- this round's condemns must be durable no later than the ledger they are @@ -386,7 +386,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// are correlated by `round_id` and not by the round number a follower never learns. { GcPhaseTimer t(phase_sink, "lease"); - report.acquired_lease = acquireOrRenewLease(state, state_incarnation, allow_steal); + report.acquired_lease = acquireOrRenewLease(state, state_etag, allow_steal); t.metric("acquired", report.acquired_lease ? 1 : 0); t.metric("steal_allowed", allow_steal ? 1 : 0); } @@ -412,7 +412,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// attempt is idempotent. const Layout & layout = store->layout(); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const uint64_t new_round = state.round + 1; /// ONE budget instance for the WHOLE round, threaded into every destructive-or-observability-write @@ -446,7 +446,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al t.metric("deleted", drain_result.deleted); } - /// Incarnation-guarded fence-out of dead mounts (liveness only — graduation itself paces on GC + /// Etag-guarded fence-out of dead mounts (liveness only — graduation itself paces on GC /// rounds via `new_round`, not on heartbeat acks). Fencing no longer trusts a predecessor's stamped /// `expires_at_ms` against our wall clock — it /// fences ONLY once `mount_obs` has watched the mount's write-token hold unchanged for the full @@ -626,7 +626,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// The pass performs discovery, windowing, and the three-cursor merge (spare / graduate / condemn). /// It emits phases 5..10 of its own. - FoldResult folded = fold(state, state_incarnation, report, new_round, *walk_plan, policy, round_work_budget); + FoldResult folded = fold(state, state_etag, report, new_round, *walk_plan, policy, round_work_budget); /// THE ROUND'S DESTRUCTIVE GATE, read once, here, and consulted at EVERY destructive site below. /// It is available this early because `fold` computes it (see `FoldResult::suppress_destructive`), @@ -692,8 +692,8 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al const std::optional observed = op.head(blob_key, Retry::standard()); Removal del = Removal::Gone; if (observed) - del = entry.token.matches(observed->incarnation) - ? op.remove(blob_key, observed->incarnation, Retry::standard()) + del = entry.token.matches(observed->etag) + ? op.remove(blob_key, observed->etag, Retry::standard()) : Removal::Mismatch; const OutcomeKind outcome_kind = del == Removal::Removed ? OutcomeKind::Deleted @@ -946,12 +946,12 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al round_commit_timer->metric("generations_visited", next.snap_pruned_through - pruned_through_before); round_commit_timer->metric("pruned_through", next.snap_pruned_through); round_commit_timer->metric("generations_referenced", referenced_generations.size()); - WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_incarnation, + WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_etag, Retry::standard()); if (std::holds_alternative(commit)) throw Exception(ErrorCodes::ABORTED, "CAS gc round: gc/state moved during the round (another leader advanced it); retry next round"); - state_incarnation = orThrow(std::move(commit), "CAS gc round commit"); + state_etag = orThrow(std::move(commit), "CAS gc round commit"); state = std::move(next); report.round = state.round; round_commit_timer->metric("round", report.round); @@ -1057,8 +1057,8 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// sealed AND taken on a round that could prove its frontier -- an unprovable round's `-1` may /// itself be the observation that is missing an owner elsewhere, so deleting the body on it is /// exactly the irreversible step the gate exists to withhold. - static const std::map kNoManifestCleanup; - const std::map & mf_cleanup_now = + static const std::map kNoManifestCleanup; + const std::map & mf_cleanup_now = suppress_destructive ? kNoManifestCleanup : folded.mf_cleanup; uint64_t attempted = 0; for (const auto & [id, incarnation] : mf_cleanup_now) @@ -1120,8 +1120,8 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al const std::optional observed = op.head(nomination.key, Retry::standard()); Removal outcome = Removal::Gone; if (observed) - outcome = nomination.token.matches(observed->incarnation) - ? op.remove(nomination.key, observed->incarnation, Retry::standard()) + outcome = nomination.token.matches(observed->etag) + ? op.remove(nomination.key, observed->etag, Retry::standard()) : Removal::Mismatch; EventEmitter{*store}.emit([&](CasEvent & e) { @@ -1181,7 +1181,7 @@ void Gc::reportStuckRemovals(const RefPlan & plan, uint64_t current_round) } bool Gc::foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, std::vector & deltas, - std::map & mf_cleanup, uint32_t txn_ordinal) + std::map & mf_cleanup, uint32_t txn_ordinal) { const Layout & layout = store->layout(); @@ -1258,7 +1258,7 @@ bool Gc::foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, s } if (sign < 0) - mf_cleanup.emplace(id, got->incarnation); /// owner removed: defer the exact body delete to recheck + mf_cleanup.emplace(id, got->etag); /// owner removed: defer the exact body delete to recheck return true; } @@ -1274,7 +1274,7 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::mapgcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); std::set witness_namespaces; @@ -1344,7 +1344,7 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::map> Gc::newestFoldSealRef() { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); const String gen_prefix = layout.gcGenPrefix(0); const String top = gen_prefix.substr(0, gen_prefix.size() - 2); /// ".../gc/gen/" @@ -1356,7 +1356,7 @@ std::optional> Gc::newestFoldSealRef() std::set listed_generations; bool listed_anything = false; std::optional> newest; - op.forEachListedKey(top, [&](const KeyEntry & k) + op.forEachListedKey(top, [&](const ListedKey & k) { listed_anything = true; const size_t from = top.size(); @@ -1476,11 +1476,11 @@ std::optional> Gc::newestFoldSealRef() Gc::GenerationSealProbe Gc::probeGenerationForSeal(uint64_t generation) { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); GenerationSealProbe probe; - op.forEachListedKey(layout.gcGenPrefix(generation), [&](const KeyEntry & k) + op.forEachListedKey(layout.gcGenPrefix(generation), [&](const ListedKey & k) { probe.generation_exists = true; /// ANY object proves this generation was minted /// Parse a candidate attempt out of the path and then PROVE it by rebuilding the key: only a @@ -1553,12 +1553,12 @@ void Gc::FoldResult::FrontierDeficit::count(FrontierUnproven reason) } } -Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_incarnation*/, +Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, RoundReport & report, uint64_t current_round, const RefPlan & walk_plan, UniversePolicy policy, GcRoundWorkBudget & work_budget) { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); FoldResult result; @@ -1601,13 +1601,13 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_in /// `cleanupRefObjects` and terminal-evidence attribution -- retains both the chosen incarnation and /// the lifecycle/absence distinction instead of re-reading or reducing the catalog independently. result.catalog_cut = catalog_snapshot; - /// THE POSITIVE EMPTY-UNIVERSE PROOF (see the destructive gate below). `incarnation` is guaranteed by + /// THE POSITIVE EMPTY-UNIVERSE PROOF (see the destructive gate below). `etag` is guaranteed by /// `CasRefCatalog::read` on every operational path -- absence there is `CORRUPTED_DATA`, never an /// empty snapshot -- but the check stays here so this fails closed if a bootstrap/test snapshot /// ever reaches this line. `entries` (not `live_incarnation`, which drops `Creating`) is the right /// source: a catalog holding only `Creating` rows must NOT read as an empty universe, and `entries` /// is the one view that still carries those rows. - result.catalog_cut_proved_empty = catalog_snapshot.incarnation.has_value() && catalog_snapshot.catalog.entries.empty(); + result.catalog_cut_proved_empty = catalog_snapshot.etag.has_value() && catalog_snapshot.catalog.entries.empty(); /// A malformed ref-object key or namespace aborts ref folding for the whole round: the /// round produces no ref delta, advances no cursor, and authorizes no destructive work -- recorded as @@ -1707,7 +1707,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_in e.type = CasEventType::GcRetireObserve; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(ref); - e.token = observed ? observed->incarnation.render() : ""; + e.token = observed ? observed->etag.render() : ""; e.round = condemn_round; e.gen = state.snap_generation + 1; e.outcome = observed ? "present" : "absent"; @@ -1723,7 +1723,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_in e.type = CasEventType::BlobRetire; e.object_kind = CasEventObjectKind::Blob; e.object_hash = blobIdOf(ref); - e.token = observed->incarnation.render(); + e.token = observed->etag.render(); e.round = condemn_round; e.gen = state.snap_generation + 1; e.outcome = "retired"; @@ -1737,7 +1737,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_in /// sees it; a successful write records the in-process (hash, token) confirmation the graduation /// gate consumes (`scheduleCondemnMarkerWrite` captures everything BY VALUE — never by reference /// to `cur_blob`, which the fold's tight streaming loop mutates while the job is queued). - meta_writer->scheduleCondemnMarkerWrite(ref, PersistedIncarnation::capture(observed->incarnation), + meta_writer->scheduleCondemnMarkerWrite(ref, PersistedEtag::capture(observed->etag), condemn_round, adjusted.size); return adjusted; }; @@ -2411,7 +2411,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_in /// CLAMP (barrier), never a round abort: keep the cursor below THIS log and re-read it next /// round. A removed precommit whose body never existed emitted no edge -- skip, no clamp. std::vector log_deltas; - std::map log_mf_cleanup; + std::map log_mf_cleanup; for (const RefManifestEdge & edge : edges) { ProfileEvents::increment(ProfileEvents::CASRefEmittedEdges); /// one manifest-edge event @@ -3265,7 +3265,7 @@ void Gc::cleanupRefObjects( if (suppress_destructive) return; - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); if (!folded.catalog_cut) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS GC ref cleanup: fold result carries no catalog cut"); @@ -3307,7 +3307,7 @@ void Gc::cleanupRefObjects( [](const CatalogEntry & entry, const RootNamespace & needle) { return entry.ns < needle; }); const std::optional current_life = current_catalog.life_index.resolve(life.incarnation); - if (current_catalog.incarnation != folded.catalog_cut->incarnation + if (current_catalog.etag != folded.catalog_cut->etag || current_entry_it == current_catalog.catalog.entries.end() || current_entry_it->ns != ns || *current_entry_it != observed_entry || !current_life || *current_life != life) @@ -3344,7 +3344,7 @@ void Gc::cleanupRefObjects( return false; } - op.remove(key, h->incarnation, Retry::standard()); + op.remove(key, h->etag, Retry::standard()); ProfileEvents::increment(ProfileEvents::CASRefCleanupObjectsDeleted); /// cleanup object deletion return true; }; @@ -3437,22 +3437,22 @@ uint64_t deletePrefixWholesale(CasOperation & op, const String & prefix, uint64_ String cursor; while (deleted < bounded_remaining) { - KeyPage page = op.list(prefix, cursor, kListPageLimit, Retry::standard()); + ListPage page = op.list(prefix, cursor, kListPageLimit, Retry::standard()); /// One page fetched, not one increment per listed key below. ProfileEvents::increment(ProfileEvents::CASGCEnumerationPages); for (const auto & listed : page.keys) { if (deleted >= bounded_remaining) return deleted; - if (listed.incarnation.has_value()) + if (listed.etag.has_value()) { /// `Gone` and `Mismatch` are both benign here (already gone / rewritten by a live /// attempt); do not throw. - op.remove(listed.key, *listed.incarnation, Retry::standard()); + op.remove(listed.key, *listed.etag, Retry::standard()); } else if (const auto head = op.head(listed.key, Retry::standard())) { - op.remove(listed.key, head->incarnation, Retry::standard()); + op.remove(listed.key, head->etag, Retry::standard()); } ++deleted; } @@ -3483,7 +3483,7 @@ void Gc::pruneSupersededGenerations(uint64_t adopted_generation, uint64_t attemp if (keep == 0) return; /// keep ALL (debug/forensics — replay GC's in-degree view as-of a past round) - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); static constexpr uint64_t kMaxPrunePerRound = 64; /// bound the per-round prune burst @@ -3567,7 +3567,7 @@ void Gc::pruneSupersededGenerations(uint64_t adopted_generation, uint64_t attemp std::optional Gc::readFoldSeal(uint64_t generation, uint64_t attempt) { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); if (const auto got = op.read(store->layout().foldSealKey(generation, attempt), Retry::standard())) return decodeFoldSeal( got->bytes, store->layout(), store->poolConfig().gc_shards, generation); @@ -3617,7 +3617,7 @@ std::vector Gc::discoverUniverse() /// The filter itself lives in `CasRefCatalog::liveUniverse` (review Important C) -- fsck's own /// reachability walk needed the identical catalog-authoritative set and is not this class, so the /// filter moved to where both can share it rather than grow a second copy that could disagree. - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); return CasRefCatalog::liveUniverse(op, store->layout()); } @@ -3665,12 +3665,12 @@ RefScanSummary Gc::enumerateRefPrefix() /// name is absorbed per key by `parseRefObjectKeyForEnumeration`, which is what keeps this /// enumeration -- which runs before the fold, outside its catch -- unable to wedge the round. const Layout & layout = store->layout(); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); RefScanSummary scan; static constexpr size_t kListPageLimit = 1000; size_t count_in_page = 0; - op.forEachListedKey(layout.casRefsPrefix(), [&](const KeyEntry & lk) + op.forEachListedKey(layout.casRefsPrefix(), [&](const ListedKey & lk) { scan.keys.push_back(lk.key); const auto parsed = parseRefObjectKeyForEnumeration(layout, lk.key); @@ -3706,7 +3706,7 @@ RoundInput Gc::listRefPrefix(const GcState & state) /// plan. A listed id absent from the later cut is dead, inert debris: it contributes no work and /// cannot force DEFER. RefScanSummary scan = enumerateRefPrefix(); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, store->layout()); /// TEST SEAM: see `setPostHotScanCatalogReadHookForTest`. Moved into a local before invoking (the /// same reason `create_namespace_step1_pre_read_hook_for_test` is swapped rather than called @@ -3745,7 +3745,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// Writes ONLY the GC plane; namespace streams/state, manifests, and blobs are read-only inputs; /// the rebuild never deletes them. RebuildReport rep; - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); /// Read bookkeeping health before the lease (the lease acquire on an absent state CREATES a @@ -3896,8 +3896,8 @@ RebuildReport Gc::rebuildBaseline(bool force) /// has_observation==false always takes the non-steal branch on its one and only call), pass it /// explicitly rather than rely on that invariant. GcState state; - std::optional state_incarnation; - if (!acquireOrRenewLease(state, state_incarnation, /*allow_steal=*/false)) + std::optional state_etag; + if (!acquireOrRenewLease(state, state_etag, /*allow_steal=*/false)) { rep.refusal = "another GC leader holds the lease"; return rep; @@ -3937,7 +3937,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { std::vector table_keys; op.forEachListedKey(layout.namespaceStreamPrefix(life), - [&](const KeyEntry & lk) { table_keys.push_back(lk.key); return true; }, + [&](const ListedKey & lk) { table_keys.push_back(lk.key); return true; }, Retry::standard(), 1000, onGcEnumerationPage); std::map grouped; try @@ -3974,7 +3974,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { const String gen_prefix = layout.gcGenPrefix(0); const String top = gen_prefix.substr(0, gen_prefix.size() - 2); /// ".../gc/gen/" - op.forEachListedKey(top, [&](const KeyEntry & k) + op.forEachListedKey(top, [&](const ListedKey & k) { const size_t from = top.size(); const size_t slash = k.key.find('/', from); @@ -4049,7 +4049,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// round once it is known, and nothing else is touched. std::set minted_hold_lives; uint64_t max_fence_round = 0; - std::map mf_cleanup_unused; + std::map mf_cleanup_unused; for (const NamespaceLifeId & life : rebuild_walk_universe) { @@ -4171,7 +4171,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { const RootNamespace ns{ns_str}; std::vector deltas; - op.forEachListedKey(layout.manifestNamespacePrefix(ns), [&](const KeyEntry & k) + op.forEachListedKey(layout.manifestNamespacePrefix(ns), [&](const ListedKey & k) { if (owned_manifest_keys.contains(k.key)) return true; @@ -4267,7 +4267,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// family independently of that, and the two reasons are stated apart on purpose — a future reader /// must not take this line as evidence that REBUILD still produces condemnations somewhere. next.manifest_sweep_cursor = ""; - WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_incarnation, + WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_etag, Retry::standard()); if (std::holds_alternative(commit)) { @@ -4303,7 +4303,7 @@ std::vector Gc::previewDeletes() { std::vector out; - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const auto state_bytes = op.read(store->layout().gcStateKey(), Retry::standard()); if (!state_bytes) return out; @@ -4389,7 +4389,7 @@ void Gc::refreshAuthority(uint64_t admitted_generation) authority_held = false; try { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const auto got = op.read(store->layout().gcStateKey(), Retry::standard()); if (!got) return; @@ -4406,7 +4406,7 @@ void Gc::refreshAuthority(uint64_t admitted_generation) void Gc::pulseHeartbeat(Pool & store, UInt128 gc_id) { - CasOperation op = store.gcRequests().admit(); + CasOperation op = store.openRequests().admit(); const String key = store.layout().gcHbKey(); const auto got = op.read(key, Retry::standard()); GcHeartbeat hb; @@ -4418,21 +4418,21 @@ void Gc::pulseHeartbeat(Pool & store, UInt128 gc_id) /// one on cadence, so a deposed leader must never spend a whole retry budget fighting for this key. const String body = encodeGcHeartbeat(hb); if (got) - op.replace(key, body, got->incarnation, Retry::once()); + op.replace(key, body, got->etag, Retry::once()); else op.create(key, body, Retry::once()); } -bool Gc::acquireOrRenewLease(GcState & state, std::optional & state_incarnation, bool allow_steal) +bool Gc::acquireOrRenewLease(GcState & state, std::optional & state_etag, bool allow_steal) { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const String key = store->layout().gcStateKey(); /// What the decision that actually landed wrote. `readModifyWrite` re-decides on every conflict /// against what the losing write's own resolve read observed, so the last decision is the committed /// one, and a competing leader's write makes the next decision see a moved lease tuple. GcState decided; - const std::optional committed = orThrow(op.readModifyWrite(key, + const std::optional committed = orThrow(op.readModifyWrite(key, [&](const std::optional & current_object) -> std::optional { if (!current_object) @@ -4518,7 +4518,7 @@ bool Gc::acquireOrRenewLease(GcState & state, std::optional & state rememberObservation(decided.lease); state = std::move(decided); - state_incarnation = committed; + state_etag = committed; return true; } @@ -4547,7 +4547,7 @@ CatalogLifecycleReconcileResult Gc::drainCompletedRemoving(const GcState & lease /// second -- the single reading taken here would otherwise authorise all of them. const uint64_t admitted_generation = leased_state.lease.seq; refreshAuthority(admitted_generation); - CasOperation op = store->gcRequests().admit([this] { return authority_held; }); + CasOperation op = store->openRequests().admit([this] { return authority_held; }); return CatalogLifecycleReconciler(op, store->layout(), *parent) .reconcile([this, admitted_generation] { refreshAuthority(admitted_generation); }); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index f6bf38fdbaea..4cbd9733c19f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -454,7 +454,7 @@ class Gc String key; uint64_t size = 0; String reason; /// "unreachable" | "delete_pending" | "awaiting_graduation" - PersistedIncarnation token; /// stored condemn-time incarnation (empty for "unreachable") + PersistedEtag token; /// stored condemn-time incarnation (empty for "unreachable") uint64_t condemn_round = 0; }; @@ -510,10 +510,10 @@ class Gc private: /// Lease acquire/renew/steal per the documented observation protocol. On success `state` holds the - /// committed gc/state (with our lease) and `state_incarnation` the incarnation that write created. + /// committed gc/state (with our lease) and `state_etag` the etag that write created. /// `allow_steal=false` suppresses only the steal (see runRegularRound's doc comment) — acquiring a /// free lease and renewing our own are unaffected. - bool acquireOrRenewLease(GcState & state, std::optional & state_incarnation, bool allow_steal); + bool acquireOrRenewLease(GcState & state, std::optional & state_etag, bool allow_steal); /// Catalog-only helping barrier run immediately after lease acquisition. It validates the adopted /// parent and delegates deterministic `Removing`-row settlement to `CatalogLifecycleReconciler`. @@ -538,7 +538,7 @@ class Gc { CasFoldSeal fold_seal; std::vector> root_shards; - std::map mf_cleanup; + std::map mf_cleanup; /// Bounded orphan candidates exact-read before reduce. Their source retirements ride this /// fold's runs; their manifest tokens become deletable only after the round CAS adopts them. ManifestSweepResult orphan_sweep; @@ -696,7 +696,7 @@ class Gc /// in-memory; the SINGLE round CAS commits them. /// `walk_plan` owns the round's one enumeration of `cas/ns/stream/` (see `RefScanSummary`) and /// its catalog cut; the fold regroups those keys strictly rather than listing the prefix again. - FoldResult fold(GcState & state, std::optional & state_incarnation, + FoldResult fold(GcState & state, std::optional & state_etag, RoundReport & report, uint64_t current_round, const RefPlan & walk_plan, UniversePolicy policy, /// One instance for the WHOLE round, owned by `runRegularRound` and threaded through @@ -786,7 +786,7 @@ class Gc /// `txn_ordinal` stamps every delta this call pushes with the round-local ordinal of the ref /// transaction that emitted it (probe B2 — see `TxnApplyLedger`). bool foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, std::vector & deltas, - std::map & mf_cleanup, uint32_t txn_ordinal); + std::map & mf_cleanup, uint32_t txn_ordinal); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp index 8fba83c8f027..b082c3820203 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.cpp @@ -13,23 +13,23 @@ GcMaintenanceReadResult readGcMaintenanceState(CasOperation & op, const Layout & { const auto got = op.read(layout.gcMaintenanceStateKey(), Retry::standard()); if (!got) - return {.status = GcMaintenanceReadStatus::Absent, .state = std::nullopt, .incarnation = std::nullopt, .diagnostic = {}}; + return {.status = GcMaintenanceReadStatus::Absent, .state = std::nullopt, .etag = std::nullopt, .diagnostic = {}}; try { return {.status = GcMaintenanceReadStatus::Valid, .state = decodeGcMaintenanceState(got->bytes), - .incarnation = got->incarnation, .diagnostic = {}}; + .etag = got->etag, .diagnostic = {}}; } catch (const DB::Exception & e) { if (e.code() != ErrorCodes::CORRUPTED_DATA) throw; return {.status = GcMaintenanceReadStatus::Corrupt, .state = std::nullopt, - .incarnation = got->incarnation, .diagnostic = e.message()}; + .etag = got->etag, .diagnostic = e.message()}; } } WriteResult casGcMaintenanceState( - CasOperation & op, const Layout & layout, const std::optional & expected, + CasOperation & op, const Layout & layout, const std::optional & expected, const GcMaintenanceState & next, const Retry & policy) { const String key = layout.gcMaintenanceStateKey(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h index 5839bbfc6914..5c12272fc03a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMaintenanceState.h @@ -12,7 +12,7 @@ struct GcMaintenanceReadResult { GcMaintenanceReadStatus status; std::optional state; - std::optional incarnation; + std::optional etag; String diagnostic; }; @@ -22,7 +22,7 @@ GcMaintenanceReadResult readGcMaintenanceState(CasOperation & op, const Layout & /// already-failed step (a reset after a failed enumeration) must send at most one attempt rather than /// spend the round's remaining time retrying a write nothing downstream is waiting on. WriteResult casGcMaintenanceState( - CasOperation & op, const Layout & layout, const std::optional & expected, + CasOperation & op, const Layout & layout, const std::optional & expected, const GcMaintenanceState & next, const Retry & policy); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp index b157fca7cba1..cd36d866f776 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.cpp @@ -27,8 +27,8 @@ namespace { /// The registry key for one condemned incarnation: the persisted pair rendered the way a live -/// `Incarnation` renders itself, so two dialects can never collide on a shared value. -String condemnMarkerKey(const PersistedIncarnation & token) +/// `Etag` renders itself, so two dialects can never collide on a shared value. +String condemnMarkerKey(const PersistedEtag & token) { return token.dialect + ":" + token.value; } @@ -75,7 +75,7 @@ bool writeCondemnedMeta(CasOperation & op, const Layout & layout, const BlobRef if (!lm) return std::holds_alternative(putMetaIfAbsent(op, layout, ref, desired)); if (lm->meta.state != MetaState::Condemned) - return std::holds_alternative(casMeta(op, layout, ref, lm->incarnation, desired)); + return std::holds_alternative(casMeta(op, layout, ref, lm->etag, desired)); return true; } @@ -88,7 +88,7 @@ void deleteConfirmedMeta(CasOperation & op, const Layout & layout, const BlobRef const auto lm = loadMeta(op, layout, ref); if (!lm) return; - deleteMetaExact(op, layout, ref, lm->incarnation); + deleteMetaExact(op, layout, ref, lm->etag); } } @@ -144,14 +144,14 @@ void GcMetaWriter::submit(std::function op) } } -void GcMetaWriter::scheduleCondemnMarkerWrite(const BlobRef & ref, const PersistedIncarnation & token, +void GcMetaWriter::scheduleCondemnMarkerWrite(const BlobRef & ref, const PersistedEtag & token, uint64_t condemn_round, uint64_t size) { /// The job admits its OWN operation: a `CasOperation` carries per-call state and belongs to one /// task, while several of these run concurrently on the pool. submit([st = state, ref, token, condemn_round, size]() { - CasOperation op = st->store->gcRequests().admit(); + CasOperation op = st->store->openRequests().admit(); if (writeCondemnedMeta(op, st->store->layout(), ref, condemn_round, size)) st->noteCondemnMarkerDurable(ref, token); }); @@ -161,7 +161,7 @@ void GcMetaWriter::scheduleConfirmedMetaDelete(const BlobRef & ref) { submit([st = state, ref]() { - CasOperation op = st->store->gcRequests().admit(); + CasOperation op = st->store->openRequests().admit(); deleteConfirmedMeta(op, st->store->layout(), ref); }); } @@ -201,35 +201,35 @@ uint64_t GcMetaWriter::completed() const return state->completed.load(std::memory_order_relaxed); } -void GcMetaWriter::State::noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token) +void GcMetaWriter::State::noteCondemnMarkerDurable(const BlobRef & ref, const PersistedEtag & token) { std::lock_guard lock(condemn_marker_mutex); condemn_markers_confirmed.emplace(ref, condemnMarkerKey(token)); } -bool GcMetaWriter::State::condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token) +bool GcMetaWriter::State::condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedEtag & token) { std::lock_guard lock(condemn_marker_mutex); return condemn_markers_confirmed.contains({ref, condemnMarkerKey(token)}); } -void GcMetaWriter::State::forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token) +void GcMetaWriter::State::forgetCondemnMarker(const BlobRef & ref, const PersistedEtag & token) { std::lock_guard lock(condemn_marker_mutex); condemn_markers_confirmed.erase({ref, condemnMarkerKey(token)}); } -void GcMetaWriter::noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token) +void GcMetaWriter::noteCondemnMarkerDurable(const BlobRef & ref, const PersistedEtag & token) { state->noteCondemnMarkerDurable(ref, token); } -bool GcMetaWriter::condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token) +bool GcMetaWriter::condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedEtag & token) { return state->condemnMarkerConfirmedInProcess(ref, token); } -void GcMetaWriter::forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token) +void GcMetaWriter::forgetCondemnMarker(const BlobRef & ref, const PersistedEtag & token) { state->forgetCondemnMarker(ref, token); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h index 95646762d61f..9392da61c9e6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcMetaWriter.h @@ -1,6 +1,6 @@ #pragma once -#include +#include #include #include #include @@ -33,7 +33,7 @@ class GcMetaWriter /// pair is recorded in the in-process confirmation registry, which the graduation gate reads. A /// refused write or a thrown error leaves the pair UNCONFIRMED: the gate then carries the entry and /// a later round retries the write. - void scheduleCondemnMarkerWrite(const BlobRef & ref, const PersistedIncarnation & token, + void scheduleCondemnMarkerWrite(const BlobRef & ref, const PersistedEtag & token, uint64_t condemn_round, uint64_t size); /// Drop the freshness meta of a blob whose body is confirmed deleted or absent. @@ -53,12 +53,12 @@ class GcMetaWriter /// The in-process condemn-marker confirmation registry, keyed (blob, rendered incarnation). It is /// keyed by the PERSISTED pair because every entry that consults it arrives from a durable - /// condemned row; a live observation enters through `PersistedIncarnation::capture`. Pool + /// condemned row; a live observation enters through `PersistedEtag::capture`. Pool /// completions insert concurrently with the round thread's reads, and the round thread also /// inserts directly when it re-checks a marker synchronously. - void noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token); - bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token); - void forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token); + void noteCondemnMarkerDurable(const BlobRef & ref, const PersistedEtag & token); + bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedEtag & token); + void forgetCondemnMarker(const BlobRef & ref, const PersistedEtag & token); private: /// Everything a job reaches. Held by `shared_ptr` and captured by value into every job. @@ -71,9 +71,9 @@ class GcMetaWriter std::mutex condemn_marker_mutex; std::set> condemn_markers_confirmed; - void noteCondemnMarkerDurable(const BlobRef & ref, const PersistedIncarnation & token); - bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedIncarnation & token); - void forgetCondemnMarker(const BlobRef & ref, const PersistedIncarnation & token); + void noteCondemnMarkerDurable(const BlobRef & ref, const PersistedEtag & token); + bool condemnMarkerConfirmedInProcess(const BlobRef & ref, const PersistedEtag & token); + void forgetCondemnMarker(const BlobRef & ref, const PersistedEtag & token); }; /// Catch each meta-operation exception, count the job, and put it on the pool -- running it diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp index 40839952a2b5..f8c153ed0f20 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasNamespaceJanitor.cpp @@ -30,20 +30,20 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage(bool suppress_deletes, Liven { result.anomalies.push_back(progress.diagnostic); throwOnRefusedOrGaveUp( - casGcMaintenanceState(op, layout, progress.incarnation, GcMaintenanceState{}, Retry::standard()), + casGcMaintenanceState(op, layout, progress.etag, GcMaintenanceState{}, Retry::standard()), "CAS namespace janitor: corrupt maintenance-state reset"); return result; } const String cursor = progress.state ? progress.state->janitor_cursor : String{}; - KeyPage page; + ListPage page; try { page = op.list(layout.namespaceRootPrefix(), cursor, page_budget, Retry::standard()); } catch (...) { - (void)casGcMaintenanceState(op, layout, progress.incarnation, GcMaintenanceState{}, Retry::once()); + (void)casGcMaintenanceState(op, layout, progress.etag, GcMaintenanceState{}, Retry::once()); throw; } result.pages = 1; @@ -71,7 +71,7 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage(bool suppress_deletes, Liven /// themselves prevent progress. bool page_decided = !ambiguous && !suppress_deletes; - for (const KeyEntry & listed : page.keys) + for (const ListedKey & listed : page.keys) { std::optional life_id; try @@ -103,15 +103,15 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage(bool suppress_deletes, Liven if (ambiguous || suppress_deletes || catalog_cut.life_index.resolve(*life_id)) continue; - std::optional incarnation = listed.incarnation; - if (!incarnation) + std::optional etag = listed.etag; + if (!etag) { try { const std::optional current = op.head(listed.key, Retry::standard()); if (!current) continue; - incarnation = current->incarnation; + etag = current->etag; } catch (const std::exception & e) { @@ -128,7 +128,7 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage(bool suppress_deletes, Liven } try { - if (op.remove(listed.key, *incarnation, Retry::standard()) == Removal::Removed) + if (op.remove(listed.key, *etag, Retry::standard()) == Removal::Removed) ++result.deleted; } catch (const std::exception & e) @@ -150,7 +150,7 @@ NamespaceJanitorResult NamespaceJanitor::runOnePage(bool suppress_deletes, Liven const GcMaintenanceState next{.janitor_cursor = page.next_cursor}; try { - const WriteResult published = casGcMaintenanceState(op, layout, progress.incarnation, next, Retry::standard()); + const WriteResult published = casGcMaintenanceState(op, layout, progress.etag, next, Retry::standard()); if (std::holds_alternative(published) || std::holds_alternative(published)) result.anomalies.push_back("cursor publication did not commit"); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp index e9c7cde963ef..5c3a88422846 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp @@ -350,7 +350,7 @@ NamespaceProtection activeManifestKeys( NamespaceFoldView namespaceFoldView(Pool & store, const RootNamespace & ns) { - CasOperation op = store.gcRequests().admit(); + CasOperation op = store.openRequests().admit(); NamespaceFoldView view; const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, store.layout()); catalog_cut.life_index.throwIfAmbiguous("CAS orphan manifest sweep"); @@ -498,7 +498,7 @@ bool prefixEligibleOn(CasOperation & op, const Layout & layout, const RootNamesp bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix) { - CasOperation op = store.gcRequests().admit(); + CasOperation op = store.openRequests().admit(); return prefixEligibleOn(op, store.layout(), ns, prefix); } @@ -506,7 +506,7 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi std::vector * warnings) { const Layout & layout = store.layout(); - CasOperation op = store.gcRequests().admit(); + CasOperation op = store.openRequests().admit(); if (!prefixEligibleOn(op, layout, ns, prefix)) return 0; /// not eligible by the durable watermark fact — delete nothing (controls #8/#9) @@ -562,7 +562,7 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi + renderRefTxnId(RefTxnId{prefix.writer_epoch, prefix.build_sequence}) + "/"; uint64_t deleted = 0; - op.forEachListedKey(prefix_key, [&](const KeyEntry & listed) + op.forEachListedKey(prefix_key, [&](const ListedKey & listed) { if (protection.active.contains(listed.key)) return true; /// owned by a committed or precommit owner — never sweep @@ -593,7 +593,7 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi const std::optional head = op.head(listed.key, Retry::standard()); if (!head) return true; - if (op.remove(listed.key, head->incarnation, Retry::standard()) == Removal::Removed) + if (op.remove(listed.key, head->etag, Retry::standard()) == Removal::Removed) ++deleted; } catch (...) @@ -622,8 +622,8 @@ ManifestSweepResult planManifestCursorPage( return result; const Layout & layout = store.layout(); - CasOperation op = store.gcRequests().admit(); - const KeyPage page = op.list(layout.casManifestsPrefix(), cursor, list_budget, Retry::standard()); + CasOperation op = store.openRequests().admit(); + const ListPage page = op.list(layout.casManifestsPrefix(), cursor, list_budget, Retry::standard()); /// This pass fetches exactly one page per round (the cursor advances across rounds, not within this /// call), so the metric increments once per call, not once per listed key. ProfileEvents::increment(ProfileEvents::CASGCEnumerationPages); @@ -644,7 +644,7 @@ ManifestSweepResult planManifestCursorPage( if (nomination_budget > 0) { uint64_t frozen = 0; - for (const KeyEntry & listed : page.keys) + for (const ListedKey & listed : page.keys) { if (frozen >= nomination_budget) break; @@ -668,12 +668,12 @@ ManifestSweepResult planManifestCursorPage( std::set errored_namespaces; /// protection view unavailable => skip, never delete /// The key of the last candidate this page actually DECIDED on. The cursor resumes strictly after - /// it (`KeyPage::next_cursor` is the last returned key), so a candidate the page never decided on + /// it (`ListPage::next_cursor` is the last returned key), so a candidate the page never decided on /// stays ahead of the cursor and is examined next pass. See the budget rule below. String decided_through; bool budget_exhausted = false; - for (const KeyEntry & listed : page.keys) + for (const ListedKey & listed : page.keys) { ++result.listed; @@ -916,7 +916,7 @@ ManifestSweepResult planManifestCursorPage( ManifestSweepResult::Nomination nomination{ .id = id, .key = parsed->key, - .token = PersistedIncarnation::capture(got->incarnation), + .token = PersistedEtag::capture(got->etag), .source_retirements = {}}; for (const ManifestEntry & entry : body->entries) if (entry.placement == EntryPlacement::Blob) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h index 1d5b9a0dbcb8..72f723431f8c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h @@ -138,7 +138,7 @@ struct ManifestSweepResult { ManifestId id; String key; - PersistedIncarnation token; + PersistedEtag token; std::vector source_retirements; }; std::vector nominations; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp index 321e6fdb467e..56244a5d39bf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CatalogLifecycleReconciler.cpp @@ -1,5 +1,5 @@ #include -#include +#include #include #include diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp index f483cc204f19..ae3f1c69c317 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp @@ -17,7 +17,7 @@ std::optional loadMeta(CasOperation & op, const Layout & layout, con auto got = op.read(layout.blobMetaKey(ref), Retry::standard()); if (!got) return std::nullopt; - return LoadedMeta{.meta = decodeBlobMeta(got->bytes), .incarnation = std::move(got->incarnation)}; + return LoadedMeta{.meta = decodeBlobMeta(got->bytes), .etag = std::move(got->etag)}; } WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const BlobRef & ref, @@ -28,14 +28,14 @@ WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const Blob } WriteResult casMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, - const Incarnation & expected, const BlobMeta & meta) + const Etag & expected, const BlobMeta & meta) { ProfileEvents::increment(ProfileEvents::CASMetaCompareSwap); return op.replace(layout.blobMetaKey(ref), encodeBlobMeta(meta), expected, Retry::standard()); } Removal deleteMetaExact(CasOperation & op, const Layout & layout, const BlobRef & ref, - const Incarnation & expected) + const Etag & expected) { ProfileEvents::increment(ProfileEvents::CASMetaDelete); return op.remove(layout.blobMetaKey(ref), expected, Retry::standard()); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h index b10364d6c445..401c115cccb7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h @@ -16,7 +16,7 @@ namespace DB::Cas struct LoadedMeta { BlobMeta meta; - Incarnation incarnation; + Etag etag; }; /// Shared lifecycle operations for the blob freshness marker used by the writer and GC. The key is @@ -47,12 +47,12 @@ WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const Blob /// A competing write is reported as `Conflict` carrying what the resolve read observed, never thrown, /// so the caller's own reload-and-retry reconciliation decides what to do about it. WriteResult casMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, - const Incarnation & expected, const BlobMeta & meta); + const Etag & expected, const BlobMeta & meta); /// Deletes only the marker incarnation named by `expected`. `Mismatch` leaves the current marker /// untouched and is distinct from `Gone`, so callers can tell absence from a raced replacement. A /// versioned bucket that archives instead of reclaiming raises `CAS_DELETE_MARKER`. Removal deleteMetaExact(CasOperation & op, const Layout & layout, const BlobRef & ref, - const Incarnation & expected); + const Etag & expected); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp index c5e61f3a1471..700fed37dafa 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp @@ -193,8 +193,8 @@ uint64_t CasMountRuntime::peekNextBuildSeq() void CasMountRuntime::renewWatermarkOnce() { - auto call = admitKeeperCall(RenewalDriverState::Dormant, RenewalDriverState::DirectCall); - (void)renewKeeperOnce( + auto call = admitRenewerCall(RenewalDriverState::Dormant, RenewalDriverState::DirectCall); + (void)renewRenewerOnce( std::move(call), RenewalDriverState::DirectCall, /*propagate_failure=*/true, @@ -271,8 +271,8 @@ CasMountRuntime::DriverLease::DriverLease(CasMountRuntime & runtime_, RenewalDri bool CasMountRuntime::renewalWorkerMayRenew() const { return renewal_driver_state == RenewalDriverState::WorkerIdle - && mount_keeper - && mount_keeper->state() == MountLeaseKeeperState::Active; + && mount_renewer + && mount_renewer->state() == MountLeaseRenewerState::Active; } CasMountRuntime::DriverLease::~DriverLease() @@ -333,7 +333,7 @@ RenewalDriverState CasMountRuntime::DriverLease::finish( return runtime.renewal_driver_state; } -CasMountRuntime::AdmittedKeeperCall CasMountRuntime::admitKeeperCall( +CasMountRuntime::AdmittedRenewerCall CasMountRuntime::admitRenewerCall( RenewalDriverState required, RenewalDriverState active) { @@ -346,23 +346,23 @@ CasMountRuntime::AdmittedKeeperCall CasMountRuntime::admitKeeperCall( throw Exception( ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: renewal driver is not admitted from the required state"); - if (!mount_keeper) - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: renewal without a keeper"); - if (mount_keeper->state() != MountLeaseKeeperState::Active) - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: renewal requires an Active keeper"); - MountLeaseKeeper * keeper = mount_keeper.get(); + if (!mount_renewer) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: renewal without a renewer"); + if (mount_renewer->state() != MountLeaseRenewerState::Active) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: renewal requires an Active renewer"); + MountLeaseRenewer * renewer = mount_renewer.get(); auto lease = std::make_unique(*this, active); renewal_driver_state = active; driver_cv.notify_all(); - return AdmittedKeeperCall{std::move(lease), keeper}; + return AdmittedRenewerCall{std::move(lease), renewer}; } -void CasMountRuntime::installKeeper( +void CasMountRuntime::installRenewer( UInt128 our_uuid, uint64_t writer_epoch, const std::function & now_ms) { - auto replacement = std::make_unique( + auto replacement = std::make_unique( mount_requests, farewell_requests, layout, server_root_id, our_uuid, writer_epoch, config.mount_lease_ttl_ms, now_ms, [this] { return minActive(); }, @@ -375,21 +375,21 @@ void CasMountRuntime::installKeeper( && renewal_driver_state != RenewalDriverState::Parked) throw Exception( ErrorCodes::LOGICAL_ERROR, - "CAS mount runtime: keeper replacement requires Dormant or Parked renewal ownership"); - mount_keeper = std::move(replacement); + "CAS mount runtime: renewer replacement requires Dormant or Parked renewal ownership"); + mount_renewer = std::move(replacement); } -uint64_t CasMountRuntime::startKeeper() +uint64_t CasMountRuntime::startRenewer() { RenewalDriverState active; RenewalDriverState destination; - MountLeaseKeeper * keeper; + MountLeaseRenewer * renewer; { std::lock_guard lock(driver_mutex); - if (!mount_keeper) - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: startKeeper without a keeper"); - if (mount_keeper->state() != MountLeaseKeeperState::New) - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: startKeeper requires a New keeper"); + if (!mount_renewer) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: startRenewer without a renewer"); + if (mount_renewer->state() != MountLeaseRenewerState::New) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: startRenewer requires a New renewer"); if (renewal_driver_state == RenewalDriverState::Dormant) { active = RenewalDriverState::StartupCall; @@ -402,15 +402,15 @@ uint64_t CasMountRuntime::startKeeper() } else { - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: startKeeper is not admitted in the current state"); + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: startRenewer is not admitted in the current state"); } - keeper = mount_keeper.get(); + renewer = mount_renewer.get(); renewal_driver_state = active; driver_cv.notify_all(); } DriverLease lease(*this, active); - const uint64_t anchor = keeper->start([this] { return !renewalCancelled(); }); + const uint64_t anchor = renewer->start([this] { return !renewalCancelled(); }); (void)lease.finish(destination); return anchor; } @@ -458,7 +458,7 @@ void CasMountRuntime::consumeRenewResult( bool propagate_failure) { /// Driver ownership has already been restored by `DriverLease::finish`; this is the single logical - /// consumption boundary and it runs without `driver_mutex` or keeper access. + /// consumption boundary and it runs without `driver_mutex` or renewer access. /// The physical counters come off the result rather than off a per-attempt callback, so they count /// the same on every ending: a renewal that gave up still sent what it sent. if (result.attempts_sent > 0) @@ -515,8 +515,8 @@ void CasMountRuntime::consumeRenewResult( std::rethrow_exception(result.failure); } -uint64_t CasMountRuntime::renewKeeperOnce( - AdmittedKeeperCall call, +uint64_t CasMountRuntime::renewRenewerOnce( + AdmittedRenewerCall call, RenewalDriverState active, bool propagate_failure, bool worker_call) @@ -526,12 +526,12 @@ uint64_t CasMountRuntime::renewKeeperOnce( configureMountRenewObservability( &server_root_id, &event_sink, active == RenewalDriverState::RemountCall); /// The remount redo re-anchors the lease BEFORE `armMountFence`, with the fence still latched lost, - /// so it renews on the keeper's open plane: admitted under the mount fence it could only ever give + /// so it renews on the renewer's open plane: admitted under the mount fence it could only ever give /// up, and every remount would fail at this step. `RemountCall` is reached from - /// `renewKeeperForRemountOnce` alone. + /// `renewRenewerForRemountOnce` alone. const MountRenewResult result = active == RenewalDriverState::RemountCall - ? call.keeper->renewForRemount(renewalEnvironment(worker_call)) - : call.keeper->renew(renewalEnvironment(worker_call)); + ? call.renewer->renewForRemount(renewalEnvironment(worker_call)) + : call.renewer->renew(renewalEnvironment(worker_call)); const RenewalDriverState destination = active == RenewalDriverState::WorkerCall ? RenewalDriverState::WorkerIdle : (active == RenewalDriverState::RemountCall ? RenewalDriverState::Parked : RenewalDriverState::Dormant); @@ -542,33 +542,33 @@ uint64_t CasMountRuntime::renewKeeperOnce( return result.attempt_start_boot_ms; } -uint64_t CasMountRuntime::renewKeeperForStartupOnce() +uint64_t CasMountRuntime::renewRenewerForStartupOnce() { - auto call = admitKeeperCall(RenewalDriverState::Dormant, RenewalDriverState::StartupCall); - return renewKeeperOnce( + auto call = admitRenewerCall(RenewalDriverState::Dormant, RenewalDriverState::StartupCall); + return renewRenewerOnce( std::move(call), RenewalDriverState::StartupCall, /*propagate_failure=*/true, /*worker_call=*/false); } -uint64_t CasMountRuntime::renewKeeperForRemountOnce() +uint64_t CasMountRuntime::renewRenewerForRemountOnce() { - auto call = admitKeeperCall(RenewalDriverState::Parked, RenewalDriverState::RemountCall); - return renewKeeperOnce( + auto call = admitRenewerCall(RenewalDriverState::Parked, RenewalDriverState::RemountCall); + return renewRenewerOnce( std::move(call), RenewalDriverState::RemountCall, /*propagate_failure=*/true, /*worker_call=*/false); } -void CasMountRuntime::keeperReset() +void CasMountRuntime::renewerReset() { std::lock_guard lock(driver_mutex); if (renewal_driver_state != RenewalDriverState::Dormant && renewal_driver_state != RenewalDriverState::Parked) - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: keeper reset while renewal is active"); - mount_keeper.reset(); + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: renewer reset while renewal is active"); + mount_renewer.reset(); } ThreadFromGlobalPool CasMountRuntime::makeWorker(std::function body) @@ -586,8 +586,8 @@ void CasMountRuntime::startBackgroundWorkers(std::chrono::milliseconds period) || workers_starting || workers_started || renewal_worker.joinable() || remount_worker.joinable()) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: background workers cannot start in the current state"); - if (!mount_keeper || mount_keeper->state() != MountLeaseKeeperState::Active) - throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: background workers require an Active keeper"); + if (!mount_renewer || mount_renewer->state() != MountLeaseRenewerState::Active) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount runtime: background workers require an Active renewer"); workers_starting = true; workers_stop_requested = false; worker_loops_released = false; @@ -638,7 +638,7 @@ void CasMountRuntime::startBackgroundWorkers(std::chrono::milliseconds period) void CasMountRuntime::renewalLoop() { - setThreadName(ThreadName::CAS_LEASE_KEEPER); + setThreadName(ThreadName::CAS_LEASE_RENEWER); { std::unique_lock lock(driver_mutex); driver_cv.wait(lock, [this] { return worker_loops_released; }); @@ -651,7 +651,7 @@ void CasMountRuntime::renewalLoop() if (config.renewal_before_driver_lock_hook_for_test) config.renewal_before_driver_lock_hook_for_test(); - AdmittedKeeperCall call; + AdmittedRenewerCall call; { std::unique_lock lock(driver_mutex); if (workers_stop_requested || remountTerminal()) @@ -678,7 +678,7 @@ void CasMountRuntime::renewalLoop() continue; } - const uint64_t last_anchor = mount_keeper->lastCommittedAttemptStartBootMs(); + const uint64_t last_anchor = mount_renewer->lastCommittedAttemptStartBootMs(); const uint64_t period_ms = static_cast(std::max(0, renewal_period.count())); const uint64_t due = last_anchor > std::numeric_limits::max() - period_ms ? std::numeric_limits::max() @@ -688,23 +688,23 @@ void CasMountRuntime::renewalLoop() { /// Every runtime notification can change the cadence decision: park/resume may happen /// entirely while this worker is idle, and a remount may publish an already-overdue - /// keeper anchor. Re-sample state and BOOTTIME after any wake instead of retaining the + /// renewer anchor. Re-sample state and BOOTTIME after any wake instead of retaining the /// old relative wait until its wall-clock timeout. driver_cv.wait_for(lock, std::chrono::milliseconds(due - now)); continue; } - MountLeaseKeeper * keeper = mount_keeper.get(); + MountLeaseRenewer * renewer = mount_renewer.get(); auto lease = std::make_unique(*this, RenewalDriverState::WorkerCall); renewal_driver_state = RenewalDriverState::WorkerCall; driver_cv.notify_all(); - call = AdmittedKeeperCall{std::move(lease), keeper}; + call = AdmittedRenewerCall{std::move(lease), renewer}; } if (config.renewal_admitted_hook_for_test) config.renewal_admitted_hook_for_test(); try { - (void)renewKeeperOnce( + (void)renewRenewerOnce( std::move(call), RenewalDriverState::WorkerCall, /*propagate_failure=*/false, @@ -811,8 +811,8 @@ void CasMountRuntime::remountLoop() if (remount_requested_generation > remount_handled_generation) continue; if (lifecycle() == PoolLifecycle::Live - && mount_keeper - && mount_keeper->state() == MountLeaseKeeperState::Active) + && mount_renewer + && mount_renewer->state() == MountLeaseRenewerState::Active) { renewal_driver_state = RenewalDriverState::WorkerIdle; driver_cv.notify_all(); @@ -935,7 +935,7 @@ void CasMountRuntime::enterIdentityLost() { /// `TransientNotLive -> IdentityLost`, one way. The compare-exchange FROM `TransientNotLive` gives /// the brief's "from TransientNotLive only" precondition, idempotency (a second call finds the state - /// already `IdentityLost` and its exchange fails), and safety against a concurrent keeper + /// already `IdentityLost` and its exchange fails), and safety against a concurrent renewer /// `noteLeaseLost` (which only ever moves `Live -> TransientNotLive`, never away from it). It does NOT /// set `vanished_intent` (that latch is reserved for the `Vanished*` idempotency/FORGET protocol); /// rev.8 makes `IdentityLost` a fail-loud TERMINAL state through `remountTerminal`, which folds it @@ -1120,13 +1120,13 @@ void CasMountRuntime::finishTeardown(bool drained) { stopBackgroundWorkers(); - if (!mount_keeper) + if (!mount_renewer) return; - if (drained && mount_keeper->state() == MountLeaseKeeperState::Active) + if (drained && mount_renewer->state() == MountLeaseRenewerState::Active) { try { - mount_keeper->release(); + mount_renewer->release(); } catch (...) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h index 3bb3a929a9ad..9264b33f0c5a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h @@ -81,9 +81,9 @@ struct MountConfig /// `driver_mutex` to inspect cadence or parking state. std::function renewal_before_driver_lock_hook_for_test = {}; /// Deterministic test interposition after a due worker has atomically reserved renewal ownership - /// and captured its keeper, but before keeper/backend I/O starts. + /// and captured its renewer, but before renewer/backend I/O starts. std::function renewal_admitted_hook_for_test = {}; - /// Deterministic test interposition after terminal ownership has been deposited and the keeper is + /// Deterministic test interposition after terminal ownership has been deposited and the renewer is /// no longer reachable by the completed call. std::function renewal_terminal_deposited_hook_for_test = {}; /// Deterministic test interposition after the parked renewal predicate has sampled terminal false, @@ -105,7 +105,7 @@ struct MountConfig }; /// Local, in-memory write fence. It is deliberately not checked by reading the object store for every -/// write: the `MountLeaseKeeper` is the sole lease reader/renewer. A successful renewal translates the +/// write: the `MountLeaseRenewer` is the sole lease reader/renewer. A successful renewal translates the /// durable `expires_at_ms` into `deadline_boot_ms`; a foreign owner, newer `writer_epoch`, or failed /// renewal latches `lost`. Mutable operations are allowed only while the latch is clear and the local /// deadline has not passed. The `writer_epoch` is the durable fencing token. @@ -127,7 +127,7 @@ struct MountFence }; /// Owns the live writer-incarnation mechanics shared by the pool's mount and recovery orchestration: -/// the `MountLeaseKeeper`, local `MountFence`, build watermark and in-flight build registry, +/// the `MountLeaseRenewer`, local `MountFence`, build watermark and in-flight build registry, /// `live_writer_epoch`, unclean-boundary marker, and both persistent workers. `Pool` retains the higher-level /// claim/recovery sequence and its `remount_mutex`; in particular, the runtime does not acquire or own /// the ref-ledger locks. The runtime receives its backend, layout, configuration, event sink, request @@ -138,7 +138,7 @@ class CasMountRuntime public: CasMountRuntime( BackendPtr backend_ptr_, - /// The two planes the `MountLeaseKeeper` runs on: renewals under the mount fence, the farewell + /// The two planes the `MountLeaseRenewer` runs on: renewals under the mount fence, the farewell /// on an open one. Owned by `Pool` and outliving this runtime. CasRequests & mount_requests_, CasRequests & farewell_requests_, @@ -162,7 +162,7 @@ class CasMountRuntime /// Test/assertion accessor for the next-to-allocate build_seq under the lock. uint64_t peekNextBuildSeq(); /// Renew the merged mount heartbeat once, including its build-watermark floor. A read-only runtime - /// has no keeper and fails with a logical exception rather than fabricating a heartbeat. + /// has no renewer and fails with a logical exception rather than fabricating a heartbeat. void renewWatermarkOnce(); /// ---- local write fence ---- @@ -215,7 +215,7 @@ class CasMountRuntime /// `enterVanished`, OR EARLY (spec §5 step 1) by FORGET's `publishVanishedIntent`, and NEVER by the /// non-absorbing `IdentityLost` ([C1]). This is the EARLIEST terminal signal: it can already be true /// while the state is still pre-terminal (mid-FORGET). Consulted alongside `isVanished()` by every - /// background worker that must self-exit the moment the pool is (being driven) terminal — the keeper + /// background worker that must self-exit the moment the pool is (being driven) terminal — the renewer /// callback (`scheduleRemount`), the remount loop, and the GC scheduler. bool vanishedIntentPublished() const { return vanished_intent.load(std::memory_order_acquire); } @@ -353,12 +353,12 @@ class CasMountRuntime /// Publish the live-incarnation `live_writer_epoch` with release ordering. void setLiveWriterEpoch(uint64_t v); - /// ---- mount-lease keeper and persistent workers ---- - void installKeeper(UInt128 our_uuid, uint64_t writer_epoch, const std::function & now_ms); - uint64_t startKeeper(); - uint64_t renewKeeperForStartupOnce(); - uint64_t renewKeeperForRemountOnce(); - void keeperReset(); + /// ---- mount-lease renewer and persistent workers ---- + void installRenewer(UInt128 our_uuid, uint64_t writer_epoch, const std::function & now_ms); + uint64_t startRenewer(); + uint64_t renewRenewerForStartupOnce(); + uint64_t renewRenewerForRemountOnce(); + void renewerReset(); void startBackgroundWorkers(std::chrono::milliseconds period); void stopBackgroundWorkers(); /// Latch a recovery generation. Persistent remount ownership means this never constructs a thread. @@ -366,7 +366,7 @@ class CasMountRuntime bool scheduleRemountForTest(); void beginShutdownForTest(); /// Return how many times `scheduleRemount` was entered, including calls refused by the background - /// setting. This is useful for testing the keeper's loss callback without starting a real recovery. + /// setting. This is useful for testing the renewer's loss callback without starting a real recovery. uint64_t scheduleRemountCallCountForTest() const { return schedule_remount_calls_for_test.load(std::memory_order_relaxed); @@ -377,14 +377,14 @@ class CasMountRuntime bool workersRunningForTest() const; uint64_t remountRequestedGenerationForTest() const; - /// Join both persistent workers before an `Active` keeper may write its clean farewell. + /// Join both persistent workers before an `Active` renewer may write its clean farewell. void finishTeardown(bool drained); /// Sleep through the injected test hook when present; otherwise use the production thread sleep. /// `Pool` claim observation and materialization grace waits share this seam so tests control both. void waitSleep(uint64_t ms) const; - /// Forward keeper events to the injected sink. The sink is held by reference so it observes the + /// Forward renewer events to the injected sink. The sink is held by reference so it observes the /// owning pool's current event routing for the runtime's entire lifetime. void emitEvent(CasEvent && e) const { if (event_sink) event_sink(std::move(e)); } @@ -402,22 +402,22 @@ class CasMountRuntime bool finished = false; }; - struct AdmittedKeeperCall + struct AdmittedRenewerCall { std::unique_ptr lease; - MountLeaseKeeper * keeper = nullptr; + MountLeaseRenewer * renewer = nullptr; }; - /// The renewal worker may drive a renewal only while it exclusively owns the driver and the keeper - /// is Active. Requires `driver_mutex`. `admitKeeperCall` enforces the same three conditions for every + /// The renewal worker may drive a renewal only while it exclusively owns the driver and the renewer + /// is Active. Requires `driver_mutex`. `admitRenewerCall` enforces the same three conditions for every /// other driver; the worker loop must park rather than throw when they do not hold, so it needs the /// predicate separately. Both the park test and the wake predicate use this one definition, so they /// cannot drift apart. bool renewalWorkerMayRenew() const; - AdmittedKeeperCall admitKeeperCall(RenewalDriverState required, RenewalDriverState active); - uint64_t renewKeeperOnce( - AdmittedKeeperCall call, + AdmittedRenewerCall admitRenewerCall(RenewalDriverState required, RenewalDriverState active); + uint64_t renewRenewerOnce( + AdmittedRenewerCall call, RenewalDriverState active, bool propagate_failure, bool worker_call); @@ -455,7 +455,7 @@ class CasMountRuntime /// epoch is from a dead incarnation), never for ordering. next_build_seq is a strictly-increasing /// per-process counter (monotonicity is load-bearing — a seq is never reused or lowered); /// active_build_seqs holds the seqs of in-flight builds, so `minActive` yields the GC floor. The floor - /// is published by the merged `mount_keeper` + /// is published by the merged `mount_renewer` /// beat (there is no standalone watermark object anymore). ATOMIC because a self-remount re-stamps it /// (kept equal to `live_writer_epoch`) from the runtime-owned remount worker while `epoch`/`writerEpoch` /// may observe it; the ref-lane hot readers were moved to `liveWriterEpoch`, so this now backs only @@ -472,15 +472,15 @@ class CasMountRuntime /// Synchronous mount-lease protocol state. Constructed and started on a writable open after the /// owner/epoch/mount startup protocol; the runtime-owned renewal worker is its sole background /// driver and publishes successful anchors or terminal loss into the local fence. After both - /// workers join, teardown releases an `Active` keeper so a same-server reopen can reclaim + /// workers join, teardown releases an `Active` renewer so a same-server reopen can reclaim /// immediately. Null on a read-only open. - std::unique_ptr mount_keeper; + std::unique_ptr mount_renewer; std::atomic live_writer_epoch{0}; /// One mutex/condition pair owns driver admission, worker lifecycle, cadence, and the remount /// generation latch, and terminal predicates paired with `driver_cv`. It is never held across - /// keeper/backend calls, remount callbacks, logging, or joins. + /// renewer/backend calls, remount callbacks, logging, or joins. mutable std::mutex driver_mutex; mutable std::condition_variable driver_cv; RenewalDriverState renewal_driver_state = RenewalDriverState::Dormant; @@ -497,7 +497,7 @@ class CasMountRuntime std::atomic schedule_remount_calls_for_test{0}; /// Local write fence. The unarmed default (`deadline_boot_ms = UINT64_MAX`, `lost = false`) permits - /// mutation until a keeper supplies a real lease deadline or reports that the lease was lost. This + /// mutation until a renewer supplies a real lease deadline or reports that the lease was lost. This /// is the gate at the ref-append mutation chokepoint. MountFence mount_fence; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp index 1abb2f991638..ea0082d6c64f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp @@ -4,7 +4,6 @@ #include #include #include -#include #include #include #include @@ -432,7 +431,7 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) event.type = CasEventType::BlobReuseAdopt; event.object_kind = CasEventObjectKind::Blob; event.object_hash = blobIdOf(ref); - event.token = present->incarnation.render(); + event.token = present->etag.render(); event.outcome = "observed"; event.reason = "a present non-condemned blob was observed after mandatory `HEAD`"; event.detail = {{"action", "observed"}, {"size", std::to_string(source.size)}}; @@ -634,25 +633,25 @@ ManifestId PartWriteTxn::stageManifest(std::vector entries) /// `encodePartManifest` is canonical/deterministic), so the engine's resolve read can prove whether /// an ambiguous attempt landed. Still NO preliminary HEAD. WriteResult staged = store->stagingPutIfAbsent(key, encoded); - const Incarnation manifest_incarnation = std::visit(detail::Overload{ - [](Committed & committed) -> Incarnation { return std::move(committed.incarnation); }, - [&](Conflict & conflict) -> Incarnation + const Etag manifest_incarnation = std::visit(detail::Overload{ + [](Committed & committed) -> Etag { return std::move(committed.etag); }, + [&](Conflict & conflict) -> Etag { /// Our own bytes under our own `ManifestId` name this same body, whoever wrote them; a /// DIFFERENT object under an id this build minted is a ManifestId collision, fail-closed /// before any owner transition can name it. if (const auto * object = std::get_if(&conflict.seen); object && object->bytes == encoded) - return object->incarnation; + return object->etag; throw Exception(ErrorCodes::CORRUPTED_DATA, "stageManifest: part-manifest key '{}' already holds {} that is not this manifest's body " "-- a ManifestId collision", key, detail::renderObservation(conflict.seen)); }, - [&](Declined &) -> Incarnation + [&](Declined &) -> Etag { throw Exception(ErrorCodes::LOGICAL_ERROR, "stageManifest: the part-manifest create at '{}' declined; a create has nothing to decline", key); }, - [&](Refused & refused) -> Incarnation + [&](Refused & refused) -> Etag { throwCasWriteRetryLater(fmt::format( "stageManifest: part-manifest PUT at '{}' definitively failed ({}); " @@ -661,7 +660,7 @@ ManifestId PartWriteTxn::stageManifest(std::vector entries) /// Unlike the ref-log lane there is nothing to wedge: this id was never named by any owner /// transition (`next_manifest_ordinal` is already past it, so no re-stage ever reuses the key), /// and a late-landing body is inert unreferenced debris for the orphan-manifest sweep. - [&](GaveUp &) -> Incarnation + [&](GaveUp &) -> Etag { throwCasWriteRetryLater(fmt::format( "stageManifest: part-manifest PUT at '{}' is UNCERTAIN (retry budget exhausted) — " @@ -1212,7 +1211,7 @@ void PartWriteTxn::cleanupStagedManifestDebrisBestEffort() { const String key = store->layout().manifestKey(id); if (const auto observed = op.head(key, Retry::standard())) - op.remove(key, observed->incarnation, Retry::standard()); + op.remove(key, observed->etag, Retry::standard()); } catch (...) // NOLINT(bugprone-empty-catch) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp index 4f1803c9bcbf..0c87b37758fb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPlainObjects.cpp @@ -56,7 +56,7 @@ std::vector CasPlainObjects::listNamespaceFiles(const NamespaceLifeId & const String prefix = layout.namespaceFilesPrefix(life); std::vector names; CasOperation op = requests.admit(); - op.forEachListedKey(prefix, [&](const KeyEntry & entry) + op.forEachListedKey(prefix, [&](const ListedKey & entry) { /// Strip the storage prefix so callers receive the bare flat file name. if (entry.key.starts_with(prefix)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index d719a9392c93..c7a61fbc76cc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -280,7 +280,7 @@ std::vector Pool::refreshAdmittedAlgos() return admitted_algos; } -/// ==== mount-runtime delegates ==== The mount lease keeper, the local write +/// ==== mount-runtime delegates ==== The mount lease renewer, the local write /// fence, the per-server build watermark, the live-incarnation epoch, and the self-remount recovery /// thread live in the `mount_runtime` member (Pool/CasMountRuntime.h); Pool keeps these thin public /// forwarders so the wiring, PartWriteTxn, Gc, the ref-ledger callbacks, and every test call site are unchanged. @@ -431,7 +431,7 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// relied on to catch a straggler afterwards. Clearing the prefix also destroys the /// durable writer-epoch counter, so a recreation by the SAME server uuid is handed the /// very `(uuid, epoch)` the survivor still holds -- and the two are then indistinguishable - /// to the lease protocol, which reads the survivor's renewal as its own keeper adopting a + /// to the lease protocol, which reads the survivor's renewal as its own renewer adopting a /// refreshed body. The fence only bites when the recreating mount is DISTINGUISHABLE (a /// different server uuid, or a surviving epoch counter): then the survivor's next renewal /// finds a slot it cannot hold and its local fence latches shut. @@ -660,7 +660,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// Mount-slot writer audit (the "foreign writer" instrument): route every mount-slot /// write/conflict event through the Pool's own sink. The factory installs the configured sink /// before this mount protocol starts, including before either runtime worker can emit. - /// `s` outlives the lambda: the runtime-owned keeper and workers are stopped before `Pool` + /// `s` outlives the lambda: the runtime-owned renewer and workers are stopped before `Pool` /// destruction reaches the event dispatcher. const auto emit_mount_event = [s = store.get()](CasEvent e) { s->emitEvent(std::move(e)); }; @@ -678,7 +678,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// (the lease expired mid-open — e.g. a slow first beat — and a GC round fenced it), that is a /// RECOVERABLE state, not a wedge: a fence costs an epoch, so allocate a fresh writer_epoch and /// re-claim. Bounded so a pathological fence storm still fails closed. The fence can surface two - /// ways: `claimMount` observes an already-fenced own slot (`FencedSelf`), or the keeper's adopt + /// ways: `claimMount` observes an already-fenced own slot (`FencedSelf`), or the renewer's adopt /// races a fence between its GET and CAS (`MountFencedException` from `start()`). /// which certificate of death (if any) justified the reclaim FINALLY adopted below /// (the last iteration's `claim` before `break` -- `claim` itself is loop-scoped). Read after the @@ -740,22 +740,22 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol } claimed_prior = claim.prior; - /// The mount object now holds OUR live (uuid, epoch) body. `installKeeper` constructs the keeper + /// The mount object now holds OUR live (uuid, epoch) body. `installRenewer` constructs the renewer /// -- which ADOPTS that very (uuid, epoch) slot rather than self-tripping the double-start guard -- /// AND wires its `minActive` build-watermark reader, its event sink, and the fence-coupling /// runtime-owned synchronous renewal driver, all captured on `mount_runtime`. - store->mount_runtime.installKeeper(our_uuid, writer_epoch, now_ms); + store->mount_runtime.installRenewer(our_uuid, writer_epoch, now_ms); try { - claim_anchor_boot_ms = store->mount_runtime.startKeeper(); + claim_anchor_boot_ms = store->mount_runtime.startRenewer(); } catch (const MountFencedException &) { - /// The GC fenced our fresh lease between the keeper's adopt GET and CAS. Recoverable: - /// drop this keeper, take a fresh epoch, and re-claim. + /// The GC fenced our fresh lease between the renewer's adopt GET and CAS. Recoverable: + /// drop this renewer, take a fresh epoch, and re-claim. if (fence_recovery >= max_fence_recoveries) throw; - store->mount_runtime.keeperReset(); + store->mount_runtime.renewerReset(); CasOperation refenced_epoch_op = store->gc_requests.admit(); writer_epoch = allocateWriterEpoch( refenced_epoch_op, store->pool_layout, srid, epoch_policy, now_ms(), observe_catalog); @@ -817,7 +817,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol : store->config.cas_request_budget.attempt_timeout_ms; /// Preserve one ordinary cadence followed by one physical renewal attempt inside the safe lease /// window. If that publication horizon was consumed, re-anchor synchronously before opening the - /// fence; the keeper independently retains its per-request deadline checks. + /// fence; the renewer independently retains its per-request deadline checks. const bool renewal_window_fits = now_boot_ms <= safe_deadline && renewal_window_ms <= safe_deadline - now_boot_ms; if (!renewal_window_fits) @@ -836,7 +836,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol LOG_WARNING(getLogger("CasPool"), "Content-addressed mount {}: the mount claim consumed the lease TTL ({} ms) before the write " "fence could be armed; re-writing the lease first", srid, ttl_ms_u); - claim_anchor_boot_ms = store->mount_runtime.renewKeeperForStartupOnce(); + claim_anchor_boot_ms = store->mount_runtime.renewRenewerForStartupOnce(); } store->mount_runtime.setLiveWriterEpoch(writer_epoch); store->armMountFence( @@ -848,7 +848,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol /// Gate the two persistent runtime workers with `background_watermark`: they run only in production /// (`background_watermark` = context != nullptr && !read_only), never in unit tests — which /// drive `renewWatermarkOnce` explicitly and rely on the armed sub-TTL deadline, never on a loop. - /// The synchronous keeper is still started above (it must adopt the mount and arm the fence on + /// The synchronous renewer is still started above (it must adopt the mount and arm the fence on /// every writable open); only the worker pair is conditional. The merged /// heartbeat renews at `mount_renew_period` — one beat now renews the lease and the floor. if (store->config.background_watermark) @@ -951,7 +951,7 @@ Pool::~Pool() } }; - /// 1. Stop and join both persistent mount-runtime workers before draining or releasing the keeper. + /// 1. Stop and join both persistent mount-runtime workers before draining or releasing the renewer. guarded([this] { if (config.teardown_phase1_throw_for_test) @@ -959,7 +959,7 @@ Pool::~Pool() mount_runtime.stopBackgroundWorkers(); }, "CAS pool teardown: stopping background workers"); - /// 2. The farewell marker the keeper's `release` writes is a certificate that no in-flight ref-log + /// 2. The farewell marker the renewer's `release` writes is a certificate that no in-flight ref-log /// conditional PUT from this incarnation can land after it. A successor treats it as proof of a /// clean death (`MountPriorState::Clean`, no observation wait needed). Writing it without an actual /// drain would be a protocol-safety bug: an uncertain PUT this incarnation is still resolving could @@ -1079,7 +1079,7 @@ void Pool::forgetDisk(const std::function & stop_and_join_gc, const Stri /// Idempotent: an already-terminal `Vanished` pool (a second FORGET, or a pool that naturally vanished /// as replaced) is already the terminal truth — nothing to force, and re-running the teardown - /// would double-retire the keeper. `IdentityLost`/`TransientNotLive`/`Live` all proceed (FORGET is + /// would double-retire the renewer. `IdentityLost`/`TransientNotLive`/`Live` all proceed (FORGET is /// their escape hatch). Reading `isVanished()` here without the lock is safe: only a terminal transition /// sets it, terminal states are absorbing, and a natural transition that wins concurrently below merely /// makes our own `enterVanished` a no-op (first terminal transition wins). @@ -1111,7 +1111,7 @@ void Pool::forgetDisk(const std::function & stop_and_join_gc, const Stri /// that window re-arms the local fence (`lost = false`). Now that the remount worker is JOINED and can /// never run again, re-latch the fence so the terminal `mayMutate() == false` holds regardless of any /// such raced reclaim. Idempotent; the durable mount lease the reclaim wrote is retired by the - /// `finishTeardown` below (it operates on whatever keeper is current — the reclaimed one). + /// `finishTeardown` below (it operates on whatever renewer is current — the reclaimed one). mount_runtime.tripMountLost(); /// (5b) Drain the ref lanes (bounded by one attempt's budget + safety margin) to learn whether a clean @@ -1126,10 +1126,10 @@ void Pool::forgetDisk(const std::function & stop_and_join_gc, const Stri mount_runtime.finishTeardown(drained); /// The pool object OUTLIVES this FORGET (it stays registered, `Vanished(forgotten)`, until DROP/restart), - /// so `~Pool` will re-run the same teardown. Drop the keeper now so that later teardown finds none and - /// skips it: `MountLeaseKeeper::release` is admitted only from `Active`, so a keeper already released - /// here must not be released again. `keeperReset` is safe now: both keeper-driving workers are joined. - mount_runtime.keeperReset(); + /// so `~Pool` will re-run the same teardown. Drop the renewer now so that later teardown finds none and + /// skips it: `MountLeaseRenewer::release` is admitted only from `Active`, so a renewer already released + /// here must not be released again. `renewerReset` is safe now: both renewer-driving workers are joined. + mount_runtime.renewerReset(); /// (6) Publish the terminal state + WARN, under remount serialization — matching the natural-transition /// contract. Every pool thread is already joined, so taking `remount_mutex` here cannot self-deadlock. @@ -1351,7 +1351,7 @@ bool Pool::tryRemountOnce() } /// The same startup protocol as Pool::open steps 2-4, as a FRESH incarnation (the old one is - /// dead by the fence-out contract and its keeper never re-mints). Open THROWS on any failure + /// dead by the fence-out contract and its renewer never re-mints). Open THROWS on any failure /// (startup is fail-closed); the remount RETURNS false instead — the recovery loop retries. try { @@ -1367,7 +1367,7 @@ bool Pool::tryRemountOnce() step = "ref_catalog_observe"; (void)observe_catalog(); step = "owner_claim"; - /// The open plane throughout: the mount fence is latched lost here (starting the keeper does + /// The open plane throughout: the mount fence is latched lost here (starting the renewer does /// not clear it), so an operation admitted under the fence could never make the claim that /// re-establishes it. CasOperation owner_op = gc_requests.admit(); @@ -1421,15 +1421,15 @@ bool Pool::tryRemountOnce() /// build's own tests the moment someone reuses an epoch across a remount. chassert(writer_epoch > mount_runtime.liveWriterEpoch()); - /// The persistent renewal worker is parked before this callback is entered, so keeper + /// The persistent renewal worker is parked before this callback is entered, so renewer /// replacement cannot race any synchronous lease operation. - step = "keeper_install"; - mount_runtime.installKeeper(our_uuid, writer_epoch, now_ms); - step = "keeper_start"; - uint64_t remount_anchor_boot_ms = mount_runtime.startKeeper(); + step = "renewer_install"; + mount_runtime.installRenewer(our_uuid, writer_epoch, now_ms); + step = "renewer_start"; + uint64_t remount_anchor_boot_ms = mount_runtime.startRenewer(); /// Re-establish the ref-protocol incarnation BEFORE re-arming the fence. Order is load-bearing: - /// Starting the keeper does NOT clear `lost`, so the fence stays closed here and no append/publish can race the + /// Starting the renewer does NOT clear `lost`, so the fence stays closed here and no append/publish can race the /// swap. /// 1. Bump the live epoch so every subsequent `allocateRefTxnId` sorts strictly above any older /// (dead-incarnation or twin) durable log. Do this BEFORE `armMountFence` so there is no window @@ -1470,11 +1470,11 @@ bool Pool::tryRemountOnce() && renewal_window_ms <= safe_deadline - now_boot_ms; if (!renewal_window_fits) { - step = "keeper_redo"; - remount_anchor_boot_ms = mount_runtime.renewKeeperForRemountOnce(); + step = "renewer_redo"; + remount_anchor_boot_ms = mount_runtime.renewRenewerForRemountOnce(); } - /// No-throw commit section: publish the fence and lifecycle only after epoch, keeper, recovery + /// No-throw commit section: publish the fence and lifecycle only after epoch, renewer, recovery /// cancellation, and ref-runtime quiescence are complete. step = "arm_fence"; mount_runtime.armMountFence( @@ -1851,7 +1851,7 @@ std::vector Pool::listMirroredChildren(const String & prefix) const String roots_full = pool_layout.rootsPrefix() + prefix; CasOperation roots_op = mount_requests.admit(); - roots_op.forEachListedKey(roots_full, [&](const KeyEntry & listed) + roots_op.forEachListedKey(roots_full, [&](const ListedKey & listed) { const String & key = listed.key; if (key.starts_with(roots_full)) @@ -2001,16 +2001,6 @@ WriteResult Pool::stagingPutIfAbsent(const String & key, const String & bytes) return ref_ledger.stagingPutIfAbsent(key, bytes); } -WriteResult Pool::stagingConditionalOverwrite(const String & key, const String & bytes, const Incarnation & expected) -{ - return ref_ledger.stagingConditionalOverwrite(key, bytes, expected); -} - -WriteResult Pool::stagingPutIfAbsentMutable(const String & key, const String & bytes) -{ - return ref_ledger.stagingPutIfAbsentMutable(key, bytes); -} - void Pool::cancelInflightBuildsForNamespace(const RootNamespace & ns) { /// Delegate to `mount_runtime`. Invoked by diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 76ca6ed063f8..180b7ba525a9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -441,7 +441,7 @@ class Pool : public std::enable_shared_from_this /// or foreign observation; the gated mutate chokepoints then fail closed. void tripMountLost(); /// Refresh the write-fence deadline (a CLOCK_BOOTTIME-milliseconds instant; release). - /// keeper renew calls this on success. + /// renewer renew calls this on success. void setMountDeadline(uint64_t deadline_boot_ms); /// Arm the fence at startup: set (uuid, epoch, deadline), clear `lost`. void armMountFence(UInt128 server_uuid, uint64_t writer_epoch, uint64_t deadline_boot_ms); @@ -508,7 +508,7 @@ class Pool : public std::enable_shared_from_this /// next step boundary, bounding the joins below); (2) trip the local fence (the deliberate /// decommission act, allowed on a live disk); (3+4) stop the GC scheduler via `stop_and_join_gc` — /// injected because the scheduler is owned above the Pool, a no-op in contexts that run none — and stop - /// + join both persistent workers; (5) drain the ref lanes (bounded) and retire the keeper WITHOUT an + /// + join both persistent workers; (5) drain the ref lanes (bounded) and retire the renewer WITHOUT an /// unearned clean farewell (the lease expires by observation unless the lanes provably drained); then /// (6) publish `Vanished(forgotten)` carrying `reason` (the [D5] message with the operator's decommission /// timestamp). Idempotent: an already-`Vanished` pool returns immediately (first terminal transition @@ -708,7 +708,6 @@ class Pool : public std::enable_shared_from_this const PoolConfig & poolConfig() const { return config; } const PoolMeta & poolMeta() const { return meta; } const Layout & layout() const { return pool_layout; } - Backend & backend() { return *pool_backend; } /// ---- the three request planes ---- /// The mount plane: the durable writes whose right to land IS this node's mount lease. An @@ -723,22 +722,17 @@ class Pool : public std::enable_shared_from_this /// claims. None of them hold a mount lease -- the claims are what ESTABLISHES one, so gating them /// on the fence would make a self-remount, which runs with the fence latched lost, unable ever to /// reclaim. - CasRequests & gcRequests() { return gc_requests; } + CasRequests & openRequests() { return gc_requests; } /// The owning `BackendPtr` itself (not just a reference into it): the decommission slot-retirement /// decommission step (`CasDecommission.cpp`) must keep the backend alive across `admin.reset()` -- the graceful /// close that stamps the mount's farewell -- to physically delete the control objects afterward. A /// bare `Backend &` from `backend()` would dangle the instant the owning `Pool` is destroyed. BackendPtr poolBackendPtr() const { return pool_backend; } - /// Staging write surface for `PartWriteTxn`: thin delegates onto the ref ledger, so a staging write + /// Staging write surface for `PartWriteTxn`: thin delegate onto the ref ledger, so a staging write /// is admitted on the same plane and under the same policy as a ref-lane write and `PartWriteTxn` - /// reaches neither directly. + /// reaches it neither directly. WriteResult stagingPutIfAbsent(const String & key, const String & bytes); - /// Same retry/fence policy as `stagingPutIfAbsent`, for a mutable exact-incarnation overwrite. - WriteResult stagingConditionalOverwrite(const String & key, const String & bytes, const Incarnation & expected); - /// Same retry/fence policy as `stagingPutIfAbsent`, for a mutable marker where an existing - /// DIFFERENT value at the key is a normal `Conflict`, not corruption. - WriteResult stagingPutIfAbsentMutable(const String & key, const String & bytes); /// CAS mixed-algo pools: /// the NODE-LOCAL algo this Pool mints NEW content with (`PoolConfig::blob_hash_algo` -- never @@ -773,10 +767,10 @@ class Pool : public std::enable_shared_from_this void setLiveWriterEpochForTest(uint64_t writer_epoch) { mount_runtime.setLiveWriterEpoch(writer_epoch); } /// Self-remount after a GC fence-out (liveness counterpart of the fence-out safety rule): the - /// OLD incarnation may never write again (the keeper never re-mints), but a FRESH incarnation — + /// OLD incarnation may never write again (the renewer never re-mints), but a FRESH incarnation — /// durable writer_epoch bump + mount reclaim + re-armed write fence — is exactly what a server /// restart would create, so a live server may create it in place. Runs the same claim machinery as - /// `Pool::open`. Orchestration stays here; the owned mount primitives it drives (keeper swap, + /// `Pool::open`. Orchestration stays here; the owned mount primitives it drives (renewer swap, /// epoch bump, fence re-arm) live on `mount_runtime`. Returns false (and changes nothing durable /// beyond the epoch bump) when the /// mount cannot be claimed (foreign owner / a genuinely live twin) — the caller retries. Safe to @@ -790,7 +784,7 @@ class Pool : public std::enable_shared_from_this /// Test seam: how many times `scheduleRemount` has been ENTERED, counted /// unconditionally as its very first statement. This increments even under the default /// `background_watermark = false` (no worker exists; a - /// test never pays for a real self-remount attempt racing this Pool's own still-live keeper, which + /// test never pays for a real self-remount attempt racing this Pool's own still-live renewer, which /// -- confirmed while building this seam -- reliably takes 30+ seconds per call and is not something /// a fast unit test should be driving). Positively pins that a production call site (e.g. /// `reportImpossibleInterference`) actually invoked `scheduleRemount`, as opposed to merely observing @@ -850,7 +844,7 @@ class Pool : public std::enable_shared_from_this }; /// The writable-mount startup tail shared by `open` and `openForDecommission`: owner claim → - /// writer_epoch → mount claim (+fence-recovery loop) → `MountLeaseKeeper` start → watermark + /// writer_epoch → mount claim (+fence-recovery loop) → `MountLeaseRenewer` start → watermark /// anchor. `our_uuid` is the identity to mount as -- `config.server_id` for a normal open, the /// victim's owner uuid for decommission (impersonation). `policy` changes only what happens when /// the mount claim does not resolve `Claimed`/`FencedSelf`: `WaitForExpiry` observes a stale- @@ -1191,7 +1185,7 @@ class Pool : public std::enable_shared_from_this /// `mount_runtime.finishTeardown` exactly as before. CasRefLedger ref_ledger; /// The mount / write-fence / build-watermark / self-remount runtime, extracted - /// from Pool. Owns the `MountLeaseKeeper`, the local `MountFence`, the per-server + /// from Pool. Owns the `MountLeaseRenewer`, the local `MountFence`, the per-server /// build watermark (`process_epoch` + the `builds_mutex`-guarded seq/registry) and its in-flight-build /// map, the live-incarnation `live_writer_epoch`, the unclean-epoch high-water-mark, and the /// persistent renewal and remount workers (with one driver mutex/condition pair). Injected with backend/layout diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp index e6f0637e94ec..84653aabc3a6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp @@ -1,5 +1,5 @@ #include -#include +#include #include #include #include @@ -50,11 +50,11 @@ CasRefCatalog::Snapshot readOptionalForBootstrap(CasOperation & op, const Layout { RefCatalog empty; return CasRefCatalog::Snapshot{ - .catalog = empty, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(empty)}; + .catalog = empty, .etag = std::nullopt, .life_index = CatalogLifeIndex(empty)}; } RefCatalog catalog = decodeRefCatalog(got->bytes); return CasRefCatalog::Snapshot{ - .catalog = catalog, .incarnation = got->incarnation, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .etag = got->etag, .life_index = CatalogLifeIndex(catalog)}; } } @@ -62,7 +62,7 @@ CasRefCatalog::Snapshot readOptionalForBootstrap(CasOperation & op, const Layout CasRefCatalog::Snapshot CasRefCatalog::read(CasOperation & op, const Layout & layout) { Snapshot snapshot = readOptionalForBootstrap(op, layout); - if (!snapshot.incarnation) + if (!snapshot.etag) throwMandatoryCatalogAbsent(layout.refCatalogKey()); return snapshot; } @@ -74,7 +74,7 @@ CasRefCatalog::Snapshot CasRefCatalog::initializeEmptyForNewPool(CasOperation & const String canonical_empty = encodeRefCatalog(empty); WriteResult result = op.create(key, canonical_empty, Retry::standard()); if (const auto * committed = std::get_if(&result)) - return Snapshot{.catalog = empty, .incarnation = committed->incarnation, .life_index = CatalogLifeIndex(empty)}; + return Snapshot{.catalog = empty, .etag = committed->etag, .life_index = CatalogLifeIndex(empty)}; /// A second opener can win after both proved the prefix empty. The refused precondition was /// settled by an exact read, so the winner's object is decoded from what that read observed; @@ -91,7 +91,7 @@ CasRefCatalog::Snapshot CasRefCatalog::initializeEmptyForNewPool(CasOperation & if (!catalog.entries.empty() || occupant->bytes != canonical_empty) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref catalog '{}' conflicts with bootstrap's required canonical empty catalog", key); - return Snapshot{.catalog = std::move(catalog), .incarnation = occupant->incarnation, + return Snapshot{.catalog = std::move(catalog), .etag = occupant->etag, .life_index = CatalogLifeIndex(empty)}; } @@ -460,7 +460,7 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov catalog_snapshot.life_index.throwIfAmbiguous("CAS completed-removal deletion"); /// A caller-supplied cut without an incarnation cannot state a precondition, and an erase that /// fell back to an unconditional write would delete whatever a concurrent writer had put there. - if (!catalog_snapshot.incarnation) + if (!catalog_snapshot.etag) throwMandatoryCatalogAbsent(layout.refCatalogKey()); const auto observed_it = findEntry(catalog_snapshot.catalog, observed.ns); if (observed_it == catalog_snapshot.catalog.entries.end() || *observed_it != observed) @@ -473,7 +473,7 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov RefCatalog candidate = catalog_snapshot.catalog; candidate.entries.erase(candidate.entries.begin() + (observed_it - catalog_snapshot.catalog.entries.begin())); WriteResult erase = op.replace(layout.refCatalogKey(), encodeRefCatalog(candidate), - *catalog_snapshot.incarnation, policy); + *catalog_snapshot.etag, policy); /// An operation whose admission is gone cannot issue the resolution read either, so the call /// ends HERE and reports the cut it was given rather than a fresh one. There is nothing further diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h index 40df568bef08..f65b1184834a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h @@ -26,7 +26,7 @@ class CasRefCatalog struct Snapshot { RefCatalog catalog; - std::optional incarnation; + std::optional etag; CatalogLifeIndex life_index; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp index 28923ce51b94..cea8f53619cd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp @@ -1,5 +1,5 @@ #include -#include +#include #include #include #include @@ -189,7 +189,7 @@ std::optional readCkpt(CasOperation & op, const Layout & layout, con return std::nullopt; /// Materialized read, then decode: the object is MUTABLE, so the body must be fixed before it is /// parsed, and the incarnation must be the one that labels exactly these bytes. - return CkptSample{decodeRefCkpt(got->bytes), got->incarnation}; + return CkptSample{decodeRefCkpt(got->bytes), got->etag}; } CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life, @@ -285,7 +285,7 @@ CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const N UNREACHABLE(); } -MissingBaseVerdict classifyMissingSampledBase(const Incarnation & sampled, const std::optional & current) +MissingBaseVerdict classifyMissingSampledBase(const Etag & sampled, const std::optional & current) { if (current && !(*current == sampled)) return MissingBaseVerdict::RestartRecovery; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h index 5730e9c98ccb..7defb5f975c4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h @@ -97,7 +97,7 @@ CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const N struct CkptSample { RefCkpt ckpt; - Incarnation incarnation; + Etag etag; }; /// Point-read of `life`'s `_ckpt`. `nullopt` means the object is absent (a namespace whose creation has @@ -128,7 +128,7 @@ enum class MissingBaseVerdict : uint8_t /// /// Pure, so it is decided the same way at every call site; the caller raises `CORRUPTED_DATA` on /// `Corrupted` with its own context. -MissingBaseVerdict classifyMissingSampledBase(const Incarnation & sampled, const std::optional & current); +MissingBaseVerdict classifyMissingSampledBase(const Etag & sampled, const std::optional & current); /// INV-4's snapshot-deletion gate: a snapshot is deletable only STRICTLY BELOW the checkpoint. Strict /// rather than at-or-below because the checkpoint names the snapshot a recovery is entitled to fetch diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp index 97e8e03c6577..7e6042679d35 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp @@ -1,6 +1,6 @@ #include #include -#include +#include #include #include #include @@ -255,19 +255,6 @@ WriteResult CasRefLedger::stagingPutIfAbsent(const String & key, const String & return op.create(key, bytes, Retry::standard()); } -WriteResult CasRefLedger::stagingConditionalOverwrite(const String & key, const String & bytes, - const Incarnation & expected) -{ - CasOperation op = mount_requests.admit(); - return op.replace(key, bytes, expected, Retry::standard()); -} - -WriteResult CasRefLedger::stagingPutIfAbsentMutable(const String & key, const String & bytes) -{ - CasOperation op = mount_requests.admit(); - return op.create(key, bytes, Retry::standard()); -} - void CasRefLedger::refuseUnlessAdmitted(const CasOperation & op, std::string_view what) const { if (op.admitted()) @@ -884,8 +871,8 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// fail-closed corruption it describes. checkRecoveryStillAdmitted(ns, rt, cancelled, token); const std::optional current = readCkpt(op, layout, life); - if (classifyMissingSampledBase(sampled_ckpt->incarnation, - current ? std::optional(current->incarnation) : std::nullopt) + if (classifyMissingSampledBase(sampled_ckpt->etag, + current ? std::optional(current->etag) : std::nullopt) == MissingBaseVerdict::RestartRecovery) return std::nullopt; throw; @@ -1024,7 +1011,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( { checkRecoveryStillAdmitted(ns, rt, cancelled, token); const std::optional current = readCkpt(op, layout, life); - if (!sampled_ckpt || !current || current->incarnation != sampled_ckpt->incarnation) + if (!sampled_ckpt || !current || current->etag != sampled_ckpt->etag) return std::nullopt; const String frontier_description = sampled_frontier ? fmt::format("{}-{}", sampled_frontier->writer_epoch, sampled_frontier->ref_sequence) @@ -1085,7 +1072,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( /// from durable-data loss under an unchanged authority token. checkRecoveryStillAdmitted(ns, rt, cancelled, token); const std::optional current = readCkpt(op, layout, life); - if (!current || current->incarnation != sampled_ckpt->incarnation) + if (!current || current->etag != sampled_ckpt->etag) return std::nullopt; throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS ref-table recovery for namespace '{}': committed log id {}-{} is absent while " @@ -1279,7 +1266,7 @@ std::optional CasRefLedger::runRecoveryWalkOnce( checkRecoveryStillAdmitted(ns, rt, cancelled, token); const std::optional final_ckpt = readCkpt(op, layout, life); if (!final_ckpt || !accepted_ckpt_sample - || final_ckpt->incarnation != accepted_ckpt_sample->incarnation + || final_ckpt->etag != accepted_ckpt_sample->etag || final_ckpt->ckpt != accepted_ckpt_sample->ckpt) return std::nullopt; checkRecoveryStillAdmitted(ns, rt, cancelled, token); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h index f8833b3f6d09..ae96ed9d1e3e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h @@ -294,13 +294,6 @@ class CasRefLedger /// corruption. WriteResult stagingPutIfAbsent(const String & key, const String & bytes); - /// The If-Match sibling of `stagingPutIfAbsent`, for a MUTABLE marker. - WriteResult stagingConditionalOverwrite(const String & key, const String & bytes, const Incarnation & expected); - - /// Create-if-absent for a MUTABLE marker, where a DIFFERENT value already at the key is an - /// ordinary `Conflict` for the caller to act on rather than corruption. - WriteResult stagingPutIfAbsentMutable(const String & key, const String & bytes); - /// Hooks required by `EventEmitter`: events are delivered to the injected sink when one is present. bool hasEventSink() const noexcept { return static_cast(event_sink); } void emitEvent(CasEvent && e) const { if (event_sink) event_sink(std::move(e)); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 6824312132c6..9f9d5eb8aade 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -132,7 +132,7 @@ struct MountRenewObservabilityConfiguration /// registered outer per-call snapshot stable without allocation, including while a parked redo holds /// `remount_mutex`. Overflow suppresses rich event/log delivery for the nested call rather than /// aliasing an outer call or changing protocol behavior; physical attempt truth is independently -/// retained by the stack-local observer in `MountLeaseKeeper::renew`. +/// retained by the stack-local observer in `MountLeaseRenewer::renew`. struct MountRenewObservabilityStack { static constexpr size_t capacity = 8; @@ -334,7 +334,7 @@ void deliverMountRenewObservability( try { LOG_DEBUG( - getLogger("CasMountLeaseKeeper"), + getLogger("CasMountLeaseRenewer"), "CAS mount renewal '{}' physical retry attempt {} (writer_epoch={}, seq={})", *context.server_root_id, attempt_no, @@ -364,7 +364,7 @@ void deliverMountRenewObservability( try { LOG_INFO( - getLogger("CasMountLeaseKeeper"), + getLogger("CasMountLeaseRenewer"), "CAS mount renewal '{}' recovered after {} physical attempts in {} ms " "(classification={}, confirmed_deadline_boot_ms={})", *context.server_root_id, @@ -391,7 +391,7 @@ void deliverMountRenewObservability( try { LOG_WARNING( - getLogger("CasMountLeaseKeeper"), + getLogger("CasMountLeaseRenewer"), "CAS mount renewal '{}' fenced after {} physical attempts in {} ms " "(classification={}, confirmed_deadline_boot_ms={})", *context.server_root_id, @@ -771,7 +771,7 @@ void emitMountEvent(const CasEventSink & sink, CasEventType type, const String & MountClaimResult claimMount( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, - uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation, + uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation, const CasEventSink & sink) { const String key = l.mountKey(srid); @@ -786,9 +786,9 @@ MountClaimResult claimMount( /// Raced with a concurrent writer between the read and the create. Treat as a live double /// start — fail closed; never overwrite a slot that appeared under us. The occupant was /// not decoded here, so no conflicting identity is known to attach to an event. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .etag = std::nullopt}; emitMountEvent(sink, CasEventType::MountClaim, srid, "mint", nullptr, "fresh mount slot minted"); - return {.kind = MountClaimResult::Claimed, .body = body, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::Claimed, .body = body, .etag = std::nullopt}; } const MountLease existing = decodeMountLease(got->bytes); @@ -799,7 +799,7 @@ MountClaimResult claimMount( { emitMountEvent(sink, CasEventType::MountConflict, srid, "foreign_owner", &existing, "mount slot is held by a foreign server_uuid — refusing to take over across identities"); - return {.kind = MountClaimResult::ForeignOwner, .body = existing, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::ForeignOwner, .body = existing, .etag = std::nullopt}; } /// Same uuid + same epoch: it is OUR OWN claim — but a FENCED body is terminal for this @@ -813,29 +813,29 @@ MountClaimResult claimMount( emitMountEvent(sink, CasEventType::MountConflict, srid, "fenced_by_gc", &existing, "own (uuid, epoch) mount slot is GC-fenced — terminal for this incarnation; " "recover with a fresh writer_epoch"); - return {.kind = MountClaimResult::FencedSelf, .body = existing, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::FencedSelf, .body = existing, .etag = std::nullopt}; } const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); - if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->incarnation, Retry::standard()), + if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->etag, Retry::standard()), fmt::format("CAS mount slot refresh of '{}'", key))) - /// The mount changed under us between the read and the write: `got->incarnation` is now + /// The mount changed under us between the read and the write: `got->etag` is now /// KNOWN STALE (that mismatch is exactly why the write was refused), not merely unknown -- - /// leaving `.incarnation` unset (rather than handing back one the caller would wrongly + /// leaving `.etag` unset (rather than handing back one the caller would wrongly /// treat as current) is deliberate, matching the identical race below. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .etag = std::nullopt}; emitMountEvent(sink, CasEventType::MountClaim, srid, "refresh", &existing, "own claim replayed — refreshed seq + expiry"); - return {.kind = MountClaimResult::Claimed, .body = body, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::Claimed, .body = body, .etag = std::nullopt}; } /// Same uuid, DIFFERENT epoch: reclaim ONLY on a certificate of death that needs no fresh /// wall-clock trust — never by comparing `expires_at_ms` against `now_ms`: - /// - `gc_fenced` → the fence-out is terminal for that incarnation by construction (its keeper's + /// - `gc_fenced` → the fence-out is terminal for that incarnation by construction (its renewer's /// every renewal fails the token guard forever, so it can never write again) — there is no /// liveness left to wait for. This is what makes self-remount (and a fast restart after a /// fence-out) instant instead of an observation wait. /// - the clean marker (`min_active_build_sequence == UINT64_MAX`) → the predecessor's OWN graceful farewell - /// (`MountLeaseKeeper::terminate`) — no observation needed either. + /// (`MountLeaseRenewer::terminate`) — no observation needed either. /// - `proven_dead_incarnation` matches the one we just read → the CALLER /// (`claimMountAwaitingExpiry`) already watched that exact incarnation hold stable for the full /// observation threshold on its own clock; re-deriving that here from a bare wall-clock @@ -845,16 +845,16 @@ MountClaimResult claimMount( /// clean-marked, not (yet) proven-dead lease may simply be a live twin, and `expires_at_ms` alone /// can never distinguish that from a dead predecessor across two different clocks. const bool clean_marker = existing.min_active_build_sequence == std::numeric_limits::max(); - const bool proven_dead = proven_dead_incarnation && *proven_dead_incarnation == got->incarnation; + const bool proven_dead = proven_dead_incarnation && *proven_dead_incarnation == got->etag; if (existing.gc_fenced || clean_marker || proven_dead) { const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); - if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->incarnation, Retry::standard()), + if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->etag, Retry::standard()), fmt::format("CAS mount slot reclaim of '{}'", key))) /// The mount changed under us between the read and the write — someone else is racing the - /// reclaim. Fail closed. `got->incarnation` is now KNOWN STALE (that mismatch is exactly why - /// the write was refused) -- leaving `.incarnation` unset is deliberate, not an oversight. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .incarnation = std::nullopt}; + /// reclaim. Fail closed. `got->etag` is now KNOWN STALE (that mismatch is exactly why + /// the write was refused) -- leaving `.etag` unset is deliberate, not an oversight. + return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .etag = std::nullopt}; const MountPriorState prior = existing.gc_fenced ? MountPriorState::Fenced : clean_marker ? MountPriorState::Clean : MountPriorState::UncleanObserved; @@ -863,16 +863,16 @@ MountClaimResult claimMount( : clean_marker ? "same server_uuid, different writer_epoch, clean farewell — reclaimed" : "same server_uuid, different writer_epoch, observed dead by " "incarnation stability — reclaimed"); - return {.kind = MountClaimResult::Claimed, .body = body, .prior = prior, .incarnation = std::nullopt}; + return {.kind = MountClaimResult::Claimed, .body = body, .prior = prior, .etag = std::nullopt}; } emitMountEvent(sink, CasEventType::MountConflict, srid, "live_double_start", &existing, "same server_uuid, different writer_epoch, not fenced/clean/proven-dead — no wall-clock trust; " "the caller must run the incarnation-stability observation wait before reclaiming"); - /// No write was attempted on this path -- `got->incarnation` is exactly the CURRENT body's - /// incarnation (what we just read is what's still there), so it is safe to hand back for the + /// No write was attempted on this path -- `got->etag` is exactly the CURRENT body's + /// etag (what we just read is what's still there), so it is safe to hand back for the /// caller's observation loop to compare across polls without a redundant re-read. - return {.kind = MountClaimResult::LiveDoubleStart, .body = existing, .incarnation = got->incarnation}; + return {.kind = MountClaimResult::LiveDoubleStart, .body = existing, .etag = got->etag}; } String mountDoubleStartMessage(const String & srid, const MountLease & existing) @@ -931,7 +931,7 @@ MountClaimResult claimMountAwaitingExpiry( /// identical. const uint64_t threshold_ms = mountObservationThresholdMs(ttl_ms, poll); - std::optional observed; + std::optional observed; uint64_t observed_since = 0; size_t restarts = 0; @@ -943,12 +943,12 @@ MountClaimResult claimMountAwaitingExpiry( if (r.kind != MountClaimResult::LiveDoubleStart) return r; - /// `claimMount` already read the current body. Reuse `r.incarnation` whenever `claimMount` + /// `claimMount` already read the current body. Reuse `r.etag` whenever `claimMount` /// set it (the common case: no write was attempted, so what it read is still current) instead of - /// re-reading the SAME key here. The rare stale-race branches deliberately leave `.incarnation` + /// re-reading the SAME key here. The rare stale-race branches deliberately leave `.etag` /// unset (see their own comments), so this still falls back to a fresh read exactly there. - std::optional current_incarnation = r.incarnation; - if (!current_incarnation) + std::optional current_etag = r.etag; + if (!current_etag) { const auto got = op.read(l.mountKey(srid), Retry::standard()); if (!got) @@ -966,16 +966,16 @@ MountClaimResult claimMountAwaitingExpiry( sleep_ms_fn(poll); continue; } - current_incarnation = got->incarnation; + current_etag = got->etag; } - if (!observed || *observed != *current_incarnation) + if (!observed || *observed != *current_etag) { if (observed && ++restarts > kMaxObservationRestarts) /// The incarnation kept changing across bounded restarts — the holder is genuinely alive /// (actively renewing), not a dead predecessor. Report it rather than waiting forever. return r; - observed = *current_incarnation; + observed = *current_etag; observed_since = mono_ms_fn(); if (on_wait_start) on_wait_start(r.body, threshold_ms); @@ -1005,7 +1005,7 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 std::set seen_srids; const String prefix = l.serverRootsPrefix(); - op.forEachListedKey(prefix, [&](const KeyEntry & listed) + op.forEachListedKey(prefix, [&](const ListedKey & listed) { /// `/owner` and `/epoch` objects share the subtree — only mount bodies gate the floor. static constexpr std::string_view mount_suffix = "/mount"; @@ -1054,13 +1054,13 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 /// raced against our own fence-out attempt) — (re)starts the observation window and /// counts as `live` this call. const auto it = obs.find(srid); - const bool stable = it != obs.end() && it->second.incarnation == observed->incarnation + const bool stable = it != obs.end() && it->second.etag == observed->etag && mono_now_ms - it->second.first_seen_mono_ms >= stable_threshold_ms; if (!stable) { - if (it == obs.end() || it->second.incarnation != observed->incarnation) - obs.insert_or_assign(srid, MountIncarnationObservation{observed->incarnation, mono_now_ms}); + if (it == obs.end() || it->second.etag != observed->etag) + obs.insert_or_assign(srid, MountIncarnationObservation{observed->etag, mono_now_ms}); ++floor.live; return std::nullopt; } @@ -1109,7 +1109,7 @@ std::vector probeNonTerminalMountSlots(CasOperation & op, /// `/mount` bodies -- but read-only and without any observation state: this answers "is anyone /// still entitled to write here", not "may I fence them out". const String prefix = l.serverRootsPrefix(); - op.forEachListedKey(prefix, [&](const KeyEntry & listed) + op.forEachListedKey(prefix, [&](const ListedKey & listed) { static constexpr std::string_view mount_suffix = "/mount"; if (!listed.key.ends_with(mount_suffix)) @@ -1155,7 +1155,7 @@ std::vector listMounts(CasOperation & op, const Layout & layout, uint { std::vector out; const String prefix = layout.serverRootsPrefix(); - op.forEachListedKey(prefix, [&](const KeyEntry & listed) + op.forEachListedKey(prefix, [&](const ListedKey & listed) { static constexpr std::string_view suffix = "/mount"; if (!listed.key.ends_with(suffix)) @@ -1264,7 +1264,7 @@ bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const Stri /// and a slot it fails to hand back is fenced out by the next GC round anyway. constexpr uint64_t kFarewellBudgetMs = 10'000; -MountLeaseKeeper::MountLeaseKeeper( +MountLeaseRenewer::MountLeaseRenewer( CasRequests & mount_requests_, CasRequests & open_requests_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, @@ -1287,7 +1287,7 @@ MountLeaseKeeper::MountLeaseKeeper( { } -String MountLeaseKeeper::encodeBody( +String MountLeaseRenewer::encodeBody( uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const { const uint64_t ttl_ms = static_cast(ttl.count()); @@ -1307,16 +1307,16 @@ String MountLeaseKeeper::encodeBody( }); } -const Incarnation & MountLeaseKeeper::precondition() const +const Etag & MountLeaseRenewer::precondition() const { - if (!last_incarnation) + if (!last_etag) throw Exception( ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: key '{}' has no incarnation to name as a write precondition", key); - return *last_incarnation; + return *last_etag; } -Incarnation MountLeaseKeeper::claim(CasOperation & op, const String & body) +Etag MountLeaseRenewer::claim(CasOperation & op, const String & body) { /// One read decides the branch AND supplies the precondition, so both the mint and the adoption /// are two requests: a separate presence probe would only re-ask what these bytes already answer. @@ -1328,12 +1328,12 @@ Incarnation MountLeaseKeeper::claim(CasOperation & op, const String & body) throw Exception( ErrorCodes::ABORTED, "CAS mount-lease: key '{}' appeared between the read and the create", key); - const std::optional incarnation + const std::optional etag = orThrow(std::move(minted), fmt::format("CAS mount-lease mint of key '{}'", key)); emitMountEvent( event_sink, CasEventType::MountClaim, srid, "mint", nullptr, - "mount slot absent -- keeper minted it directly"); - return *incarnation; + "mount slot absent -- renewer minted it directly"); + return *etag; } const MountLease observed = decodeMountLease(got->bytes); @@ -1361,13 +1361,13 @@ Incarnation MountLeaseKeeper::claim(CasOperation & op, const String & body) { emitMountEvent( event_sink, CasEventType::MountConflict, srid, "fenced_by_gc", &observed, - "own mount slot was fenced by GC before keeper adoption"); + "own mount slot was fenced by GC before renewer adoption"); throw MountFencedException(fmt::format( - "CAS mount-lease: key '{}' was fenced by GC before keeper adoption ({})", + "CAS mount-lease: key '{}' was fenced by GC before renewer adoption ({})", key, describeMountHolder(observed))); } - WriteResult adopted = op.replace(key, body, got->incarnation, Retry::standard()); + WriteResult adopted = op.replace(key, body, got->etag, Retry::standard()); if (const Conflict * conflict = std::get_if(&adopted)) { /// The write's own resolve read is the re-read: it observed what took the key from us. @@ -1388,18 +1388,18 @@ Incarnation MountLeaseKeeper::claim(CasOperation & op, const String & body) ErrorCodes::ABORTED, "CAS mount-lease: key '{}' vanished while adopting our own mount slot", key); } - const std::optional incarnation + const std::optional etag = orThrow(std::move(adopted), fmt::format("CAS mount-lease adoption of key '{}'", key)); emitMountEvent( event_sink, CasEventType::MountClaim, srid, "adopt", &observed, "adopted our own already-live mount slot"); - return *incarnation; + return *etag; } -uint64_t MountLeaseKeeper::start(Liveness liveness) +uint64_t MountLeaseRenewer::start(Liveness liveness) { - if (keeper_state != MountLeaseKeeperState::New) + if (renewer_state != MountLeaseRenewerState::New) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: start is allowed only in New state for key '{}'", key); const uint64_t wall_ms = now_ms_fn(); @@ -1409,20 +1409,20 @@ uint64_t MountLeaseKeeper::start(Liveness liveness) /// admitted under it would be refused on every request. What makes the claim safe is that every /// write below is conditional. CasOperation op = open_requests.admit(std::move(liveness)); - const Incarnation incarnation = claim(op, body); + const Etag etag = claim(op, body); seq = 1; - last_incarnation = incarnation; + last_etag = etag; last_committed_attempt_start_boot_ms = attempt_start_boot_ms; const uint64_t ttl_ms = static_cast(ttl.count()); confirmed_deadline_boot_ms = attempt_start_boot_ms > std::numeric_limits::max() - ttl_ms ? std::numeric_limits::max() : attempt_start_boot_ms + ttl_ms; - keeper_state = MountLeaseKeeperState::Active; + renewer_state = MountLeaseRenewerState::Active; return attempt_start_boot_ms; } -[[noreturn]] void MountLeaseKeeper::throwRenewConflict(const Observation & seen) const +[[noreturn]] void MountLeaseRenewer::throwRenewConflict(const Observation & seen) const { if (const Object * occupant = std::get_if(&seen)) { @@ -1460,7 +1460,7 @@ uint64_t MountLeaseKeeper::start(Liveness liveness) /// This decoded authoritative observation is the exact point at which this incarnation learns /// that a foreign successor owns the slot. Terminal teardown intentionally performs no release - /// I/O, so account the skipped farewell here, once, before the keeper enters its terminal state. + /// I/O, so account the skipped farewell here, once, before the renewer enters its terminal state. /// The renewal may be parked under `remount_mutex`; keep the increment trace-free. ProfileEvents::incrementNoTrace(ProfileEvents::CASMountReleaseSkippedForeignOccupant); emitMountEvent( @@ -1491,7 +1491,7 @@ uint64_t MountLeaseKeeper::start(Liveness liveness) "an occupant nor an absence", key)); } -MountRenewResult MountLeaseKeeper::terminalResult(MountRenewResult result) +MountRenewResult MountLeaseRenewer::terminalResult(MountRenewResult result) { if (!result.failure) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: terminal renewal requires a failure"); @@ -1507,38 +1507,38 @@ MountRenewResult MountLeaseKeeper::terminalResult(MountRenewResult result) catch (...) { } - if (keeper_state != MountLeaseKeeperState::Active) + if (renewer_state != MountLeaseRenewerState::Active) throw Exception( ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: terminal renewal outside Active state (observed state {})", - static_cast(keeper_state)); - keeper_state = MountLeaseKeeperState::RenewalTerminal; + static_cast(renewer_state)); + renewer_state = MountLeaseRenewerState::RenewalTerminal; result.outcome = MountRenewOutcome::Terminal; return result; } -MountRenewResult MountLeaseKeeper::renew(const MountRenewOperationEnvironment & environment) +MountRenewResult MountLeaseRenewer::renew(const MountRenewOperationEnvironment & environment) { return renewOn(mount_requests, environment); } -MountRenewResult MountLeaseKeeper::renewForRemount(const MountRenewOperationEnvironment & environment) +MountRenewResult MountLeaseRenewer::renewForRemount(const MountRenewOperationEnvironment & environment) { return renewOn(open_requests, environment); } -MountRenewResult MountLeaseKeeper::renewOn( +MountRenewResult MountLeaseRenewer::renewOn( CasRequests & plane, const MountRenewOperationEnvironment & environment) { const MountRenewObservabilityRegistration observability_registration = beginMountRenewObservabilityCall(); const MountRenewObservabilityCallGuard observability_guard(observability_registration); - if (keeper_state != MountLeaseKeeperState::Active) + if (renewer_state != MountLeaseRenewerState::Active) throw Exception( ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: renew is allowed only in Active state for key '{}' (observed state {})", key, - static_cast(keeper_state)); + static_cast(renewer_state)); const auto boot_clock = environment.boot_ms ? environment.boot_ms : boot_ms_fn; /// Sampled BEFORE the write. A refused admission is reported as "never attempted" only when this @@ -1585,7 +1585,7 @@ MountRenewResult MountLeaseKeeper::renewOn( if (Committed * committed = std::get_if(&*written)) { seq = next_seq; - last_incarnation = std::move(committed->incarnation); + last_etag = std::move(committed->etag); last_committed_attempt_start_boot_ms = attempt_start_boot_ms; const uint64_t ttl_ms = static_cast(ttl.count()); confirmed_deadline_boot_ms = attempt_start_boot_ms > std::numeric_limits::max() - ttl_ms @@ -1675,7 +1675,7 @@ MountRenewResult MountLeaseKeeper::renewOn( "CAS mount-lease: the renewal of key '{}' was declined, which a replace cannot report", key); } -void MountLeaseKeeper::terminate(CasOperation & op) +void MountLeaseRenewer::terminate(CasOperation & op) { const uint64_t wall_ms = now_ms_fn(); const String body = encodeMountLease(MountLease{ @@ -1694,7 +1694,7 @@ void MountLeaseKeeper::terminate(CasOperation & op) if (Committed * committed = std::get_if(&written)) { seq += 1; - last_incarnation = std::move(committed->incarnation); + last_etag = std::move(committed->etag); emitMountEvent( event_sink, CasEventType::MountRelease, srid, "farewell", nullptr, "graceful release -- lease stamped already-expired and watermark retired"); @@ -1722,11 +1722,11 @@ void MountLeaseKeeper::terminate(CasOperation & op) orThrow(std::move(written), fmt::format("CAS mount-lease release of key '{}'", key)); } -void MountLeaseKeeper::release() +void MountLeaseRenewer::release() { - if (keeper_state != MountLeaseKeeperState::Active) + if (renewer_state != MountLeaseRenewerState::Active) throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS mount-lease: release is allowed only in Active state for key '{}'", key); - keeper_state = MountLeaseKeeperState::Released; + renewer_state = MountLeaseRenewerState::Released; /// Off the mount fence, for the same reason the claim is: a departing mount whose lease has already /// run down still has to hand the slot back, and refusing the write there would leave the slot /// looking live until GC fences it out. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index 419f05ac37d2..5ab74f9fc971 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -31,7 +31,7 @@ namespace ErrorCodes namespace DB::Cas { -enum class MountLeaseKeeperState : uint8_t +enum class MountLeaseRenewerState : uint8_t { New, Active, @@ -201,7 +201,7 @@ enum class MountPriorState /// slot. Decision over `get(mountKey)`: /// - absent → write our body via `create` → `Claimed`; /// - same `server_uuid` AND same `writer_epoch` as (our_uuid, our_epoch) → it is OUR OWN claim -/// (a replay / the keeper adopting it): +/// (a replay / the renewer adopting it): /// - `gc_fenced` → terminal for THIS (uuid, epoch) — a fence costs an epoch, so refreshing it /// in place would reactivate a fenced incarnation → `FencedSelf` (no write); /// - otherwise → refresh (`replace` to bump seq + fresh `expires_at_ms`) → `Claimed`; @@ -209,7 +209,7 @@ enum class MountPriorState /// needs no fresh wall-clock trust (see /// `claimMountAwaitingExpiry` below for how a plain "looks expired" reading is turned into one): /// - `gc_fenced` (the GC leader already, itself, threshold-gated this incarnation dead; a fence -/// costs an epoch, so its keeper can never renew again) → reclaim, `prior = Fenced`; +/// costs an epoch, so its renewer can never renew again) → reclaim, `prior = Fenced`; /// - the clean marker (`min_active_build_sequence == UINT64_MAX`, the predecessor's own graceful farewell) → /// reclaim, `prior = Clean`; /// - `proven_dead_incarnation` matches the CURRENTLY OBSERVED incarnation (the caller itself @@ -244,7 +244,7 @@ struct MountClaimResult /// loop would otherwise re-read the mount key just to recover what `claimMount` had already read /// one line earlier and thrown away -- one wasted read per iteration. Empty for every other /// `Kind` (nothing to compare against). - std::optional incarnation; + std::optional etag; }; /// Thrown when a mount operation observes that OUR OWN (uuid, epoch) slot was `gc_fenced` by the GC @@ -267,7 +267,7 @@ class MountFencedException : public DB::Exception /// attempt with no such proof. MountClaimResult claimMount( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, - uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation = {}, + uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation = {}, const CasEventSink & sink = {}); /// Format the operator-actionable startup error shown when the mount lease is held by a genuinely @@ -321,7 +321,7 @@ MountClaimResult claimMountAwaitingExpiry( /// tight poll loop. struct MountIncarnationObservation { - Incarnation incarnation; + Etag etag; uint64_t first_seen_mono_ms = 0; }; @@ -336,7 +336,7 @@ using MountObservationMap = std::map; /// Classification per body: /// - `gc_fenced` already set → excluded (`already_fenced`); a fenced mount is terminal, no PUT; /// - terminated (`min_active_build_sequence == UINT64_MAX`, the farewell sentinel stamped by -/// `MountLeaseKeeper::terminate`) → excluded (`terminated`). `expires_at_ms` alone cannot +/// `MountLeaseRenewer::terminate`) → excluded (`terminated`). `expires_at_ms` alone cannot /// distinguish a graceful farewell from an unclean stop, so the sentinel — not the timestamps — is the /// terminated marker; /// - otherwise, observation-based liveness (the same @@ -391,7 +391,7 @@ struct NonTerminalMountSlot /// Read-only scan of every mount slot under the pool prefix, answering ONE question: is some writer /// still entitled to this prefix? A slot counts as terminal on exactly the two clock-free certificates /// the mount protocol already recognises (`computeHeartbeatFloor`'s own classification): `gc_fenced` -/// (the GC leader fenced that incarnation out, and a fence costs an epoch, so its keeper can never +/// (the GC leader fenced that incarnation out, and a fence costs an epoch, so its renewer can never /// renew again) and `min_active_build_sequence == UINT64_MAX` (the holder's own graceful farewell). Everything else is /// reported, INCLUDING a body this build cannot decode -- an unreadable lease of some other format /// generation is precisely the case that must block, not the one to wave through. @@ -445,7 +445,7 @@ std::vector listMounts(CasOperation & op, const Layout & layout, uint /// two clock-free certificates `probeNonTerminalMountSlots`/`computeHeartbeatFloor` already use for the /// identical question at pool-prefix and GC-heartbeat granularity — /// - `gc_fenced` (the GC leader already fenced this incarnation; a fence costs an epoch, so its -/// keeper can never renew again), +/// renewer can never renew again), /// - the clean-farewell sentinel `min_active_build_sequence == UINT64_MAX`, /// PLUS one more certificate available here that neither of those needs: a DIFFERENT `writer_epoch` /// currently live at that slot proves `writer_epoch`'s specific incarnation is superseded regardless of @@ -483,7 +483,7 @@ bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const Stri /// runtime, or performs a durable write from its destructor. /// /// ADOPT RULE (critical): the steady-state flow is `claimMount(...)` writes the live mount under -/// (our_uuid, our_epoch), THEN `keeper.start()`. So `start`'s `claim` hook must ADOPT a live mount +/// (our_uuid, our_epoch), THEN `renewer.start()`. So `start`'s `claim` hook must ADOPT a live mount /// that is ALREADY ours — same `server_uuid` AND same `writer_epoch` — instead of self-tripping the /// live-double-start guard. The discriminator is the (uuid, epoch) pair: /// - same uuid + same epoch → our own just-written claim (or a replay) → adopt: `replace` @@ -498,10 +498,10 @@ bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const Stri /// a farewell refused because the fence has run down would leave the slot looking live until GC /// fences it out. Neither is unguarded: a claim's safety is its own conditional write, and a caller /// that has shutdown facts hands them over as a `Liveness`. -class MountLeaseKeeper +class MountLeaseRenewer { public: - MountLeaseKeeper( + MountLeaseRenewer( CasRequests & mount_requests_, CasRequests & open_requests_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, uint64_t writer_epoch_, std::chrono::milliseconds ttl_, std::function now_ms_fn_, @@ -509,7 +509,7 @@ class MountLeaseKeeper CasEventSink event_sink_ = {}, std::chrono::milliseconds lease_safety_margin_ = std::chrono::milliseconds(2000), /// boot-domain clock for the on_renew_ok anchor; empty = real CLOCK_BOOTTIME. Injectable for - /// tests and wired by CasMountRuntime::installKeeper. + /// tests and wired by CasMountRuntime::installRenewer. std::function boot_ms_fn_ = {}); /// Adopt the already-claimed mount. Returns the exact pre-I/O BOOTTIME anchor. `liveness` carries @@ -520,23 +520,23 @@ class MountLeaseKeeper /// The remount's re-anchor, which is bootstrap control rather than steady state: a remount renews /// BEFORE it arms the fence for the new incarnation, so the fence is still latched lost and an /// operation admitted under it would be refused before its first attempt. It admits on this - /// keeper's own open plane, the one the claim and the farewell use, so there is no plane for a + /// renewer's own open plane, the one the claim and the farewell use, so there is no plane for a /// caller to get wrong. Same policy and same verdicts as `renew`. MountRenewResult renewForRemount(const MountRenewOperationEnvironment & environment = {}); void release(); - MountLeaseKeeperState state() const { return keeper_state; } - bool canRelease() const { return keeper_state == MountLeaseKeeperState::Active; } + MountLeaseRenewerState state() const { return renewer_state; } + bool canRelease() const { return renewer_state == MountLeaseRenewerState::Active; } uint64_t lastCommittedAttemptStartBootMs() const { return last_committed_attempt_start_boot_ms; } private: String encodeBody(uint64_t seq_, uint64_t wall_ms, uint64_t min_active_build_sequence, UInt128 write_attempt_id) const; /// The incarnation every guarded write of this slot names. Engaged for exactly the states that /// admit such a write: `start` establishes it and each committed renewal replaces it. - const Incarnation & precondition() const; + const Etag & precondition() const; /// One renewal admitted on `plane`; `renew` and `renewForRemount` differ only in which they pass. MountRenewResult renewOn(CasRequests & plane, const MountRenewOperationEnvironment & environment); - Incarnation claim(CasOperation & op, const String & body); + Etag claim(CasOperation & op, const String & body); [[noreturn]] void throwRenewConflict(const Observation & seen) const; MountRenewResult terminalResult(MountRenewResult result); void terminate(CasOperation & op); @@ -554,13 +554,13 @@ class MountLeaseKeeper CasEventSink event_sink; std::chrono::milliseconds lease_safety_margin; /// boot-domain clock for the on_renew_ok anchor; empty = real CLOCK_BOOTTIME. Injectable for - /// tests and wired by CasMountRuntime::installKeeper. + /// tests and wired by CasMountRuntime::installRenewer. std::function boot_ms_fn; - MountLeaseKeeperState keeper_state = MountLeaseKeeperState::New; + MountLeaseRenewerState renewer_state = MountLeaseRenewerState::New; uint64_t seq = 0; /// The incarnation our last landed write created; every renewal and the farewell name it as the /// precondition. Unset only before `start` has landed one. - std::optional last_incarnation; + std::optional last_etag; uint64_t confirmed_deadline_boot_ms = 0; uint64_t last_committed_attempt_start_boot_ms = 0; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasTypes.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasTypes.h index ff25c75d7dd5..50c56d280ddf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasTypes.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasTypes.h @@ -245,26 +245,13 @@ namespace DB::Cas { /// How a backend identifies one physical incarnation of an object key. -enum class TokenType : uint8_t +enum class Dialect : uint8_t { ETag = 1, /// S3-family / Azure Generation = 2, /// GCS (binding deferred; fail-closed until probed) Emulated = 3, /// test backends (in-memory fake, Local emulation) }; -/// A backend-native incarnation token. Opaque to every CAS caller and sent back to the backend -/// EXACTLY as held here. The backend owns the one conversion between its transport representation and -/// this value — see `ObjectStorageBackend::normalizeTokenValue`, which removes the quoting the GCS -/// generation picks up from riding the SDK's ETag field. -struct Token -{ - String value; - TokenType type = TokenType::ETag; - - bool empty() const { return value.empty(); } - bool operator==(const Token &) const = default; -}; - /// The ordered ref-transaction identifier. A successful writer mount establishes a strictly newer /// `writer_epoch`; within an epoch, one namespace's `ref_sequence` values are CONTIGUOUS from 1 -- /// derived per append from the table's own greatest applied id (`nextRefTxnId`), not drawn from any diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md index dbf5d104d79e..4f757f072e69 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md @@ -81,7 +81,7 @@ Primitives → Formats → Backend → Pool → Gc → Tools ≈ Parts → facad - **`Primitives/`** — the vocabulary, zero outward dependencies: `CasBlobDigest` (`BlobHashAlgo` + `BlobDigest` + `DigestCodec` + `BlobRef` — blob identity), - `CasTypes.h` (the other identity types: `RootNamespace`, `Token`, + `CasTypes.h` (the other identity types: `RootNamespace`, `Dialect`, `ManifestId`, `RefTxnId`), `CasNamespaceLifeId` (`NamespaceLifeId` — one LIFE of a namespace's ref layer, the pair every ref key is built from), `CasBlobHashingWriteBuffer` (streaming diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index 3cf9938cedf9..253f002fe1e3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -60,12 +60,12 @@ std::string_view removalName(Removal r) uint64_t deleteListedPrefix(CasOperation & op, const String & prefix, std::vector & warnings) { uint64_t deleted = 0; - op.forEachListedKey(prefix, [&](const KeyEntry & listed) + op.forEachListedKey(prefix, [&](const ListedKey & listed) { try { - std::optional incarnation = listed.incarnation; - if (!incarnation) + std::optional etag = listed.etag; + if (!etag) { const std::optional head = op.head(listed.key, Retry::standard()); if (!head) @@ -73,10 +73,10 @@ uint64_t deleteListedPrefix(CasOperation & op, const String & prefix, std::vecto warnings.push_back("decommission drain: " + listed.key + " vanished before delete"); return true; } - incarnation = head->incarnation; + etag = head->etag; } - const Removal outcome = op.remove(listed.key, *incarnation, Retry::standard()); + const Removal outcome = op.remove(listed.key, *etag, Retry::standard()); if (outcome == Removal::Removed) ++deleted; else @@ -96,11 +96,11 @@ uint64_t deleteListedPrefix(CasOperation & op, const String & prefix, std::vecto /// Delete one slot control object by an incarnation captured at the protocol-defined fence point. Slot /// retirement is fail-closed: unlike the debris drains above, any non-`Removed` outcome or exception /// stops the tail before it can touch the next control object. -bool deleteSlotObject(CasOperation & op, const String & key, const Incarnation & incarnation, std::vector & warnings) +bool deleteSlotObject(CasOperation & op, const String & key, const Etag & etag, std::vector & warnings) { try { - const Removal outcome = op.remove(key, incarnation, Retry::standard()); + const Removal outcome = op.remove(key, etag, Retry::standard()); if (outcome == Removal::Removed) return true; @@ -242,7 +242,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, { const String debris_prefix = admin->layout().casManifestsServerPrefix(victim_srid); std::set> groups; /// (namespace, writer epoch, build sequence) - op.forEachListedKey(debris_prefix, [&](const KeyEntry & listed) + op.forEachListedKey(debris_prefix, [&](const ListedKey & listed) { if (const auto parsed = admin->layout().parseManifestKey(listed.key)) groups.emplace(parsed->root_namespace.string(), parsed->ref.writer_epoch, parsed->ref.build_sequence); @@ -296,7 +296,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, { const CasRefCatalog::Snapshot fresh_retirement_catalog = CasRefCatalog::read(op, admin->layout()); if (!retirement_catalog_cut - || fresh_retirement_catalog.incarnation != retirement_catalog_cut->incarnation + || fresh_retirement_catalog.etag != retirement_catalog_cut->etag || fresh_retirement_catalog.catalog != retirement_catalog_cut->catalog) { report.warnings.push_back( @@ -391,8 +391,8 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// (finding #9) intentionally stopped short of making concurrent decommission-vs-recreate /// airtight to the microsecond, since that was explicitly not the priority for this fix. report.slot_removed = false; - if (captures_match && deleteSlotObject(op, mount_key, farewell_mount->incarnation, report.warnings) - && deleteSlotObject(op, epoch_key, claimed_epoch->incarnation, report.warnings)) + if (captures_match && deleteSlotObject(op, mount_key, farewell_mount->etag, report.warnings) + && deleteSlotObject(op, epoch_key, claimed_epoch->etag, report.warnings)) { std::optional current_mount; std::optional current_epoch; @@ -440,7 +440,7 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, /// `Refused` is the store's own definite answer (a denial, a malformed /// request, an expired credential) and carries its own code and message, /// which is worth more here than the generic retry advice below. - WriteResult result = op.replace(owner_key, encodeOwner(tombstoned), owner->incarnation, Retry::standard()); + WriteResult result = op.replace(owner_key, encodeOwner(tombstoned), owner->etag, Retry::standard()); if (std::holds_alternative(result)) report.slot_removed = true; else if (std::holds_alternative(result)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp index cc0cf437f281..8b67d00251a1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasFsck.cpp @@ -57,7 +57,7 @@ void listAll(CasOperation & op, const String & prefix, std::unordered_map current = readCkpt(op, layout, life); - if (!current || !checkpoint_sample || current->incarnation != checkpoint_sample->incarnation) + if (!current || !checkpoint_sample || current->etag != checkpoint_sample->etag) { verdicts.recordUnchecked(report, ns, key, note + "; checkpoint authority changed while validating its snapshot base"); @@ -425,7 +425,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co const String & namespace_prefix, FsckReport & report) { const Layout & layout = store.layout(); - CasOperation op = store.gcRequests().admit(); + CasOperation op = store.openRequests().admit(); /// Path-derived per-object algorithm parsing: every listed blob-tree key -- across every /// admitted algo, not just the pool's node-local write algo -- is classified via /// `Layout::parseBlobKey`, which derives the `BlobRef` from the key's OWN `` path segment @@ -526,7 +526,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co }; std::vector canonical_candidates; - op.forEachListedKey(layout.namespaceRootPrefix(), [&](const KeyEntry & listed) + op.forEachListedKey(layout.namespaceRootPrefix(), [&](const ListedKey & listed) { std::optional physical_id; try @@ -959,7 +959,7 @@ void runFsckImpl(Pool & store, bool detail, const FsckProgress & on_progress, co /// HEAD only for a hash the snapshot actually retired, so an unretired blob still costs no request. const std::optional retired_head = rit != retired_by_hash.end() ? op.head(bkey, Retry::standard()) : std::nullopt; - if (retired_head && rit->second.token.matches(retired_head->incarnation)) + if (retired_head && rit->second.token.matches(retired_head->etag)) { /// The PRESENT incarnation is the condemned one — deletion is scheduled. A mismatch /// means the listed entry belongs to a displaced older incarnation and says diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp index c6ff7c75f007..0d37a5e0ba9a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasInspect.cpp @@ -288,7 +288,7 @@ String renderGcState(const GcState & s) /// A recorded incarnation's value is an opaque backend-native string (e.g. an S3 ETag) — NOT a /// 128-bit hash — so it renders verbatim (escaped), not hex-converted; `type` is the dialect word, /// naming which backend family minted it. -String renderPersistedIncarnation(const PersistedIncarnation & inc) +String renderPersistedEtag(const PersistedEtag & inc) { return JsonObj() .add("value", jsonEscape(inc.value)) @@ -401,7 +401,7 @@ String renderCondemnedRow(const CondemnedRow & r) { return JsonObj() .add("delete_pending", jsonBool(r.delete_pending)) - .add("token", renderPersistedIncarnation(r.token)) + .add("token", renderPersistedEtag(r.token)) .add("size", jsonUInt(r.size)) .add("condemn_round", jsonUInt(r.condemn_round)) .add("marker_confirmed", jsonBool(r.marker_confirmed)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp index 5b425d33c974..34bde37517dd 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/benchmarks/benchmark_cas_ref_protocol.cpp @@ -635,7 +635,7 @@ std::vector makeSourceEdgeRecords(size_t n) rec.source_id = UInt128(0); rec.marker = RunMarker::Condemned; rec.delete_pending = (i % 200 == 0); - rec.token = PersistedIncarnation{"etag", "\"e1b2c3d4e5f6071829300a0b0c0d0e0f\""}; + rec.token = PersistedEtag{"etag", "\"e1b2c3d4e5f6071829300a0b0c0d0e0f\""}; rec.size = 64 * 1024; rec.condemn_round = 7; } diff --git a/src/Disks/tests/cas_sweep_test_support.h b/src/Disks/tests/cas_sweep_test_support.h index 37144b65f415..6326dd7c0546 100644 --- a/src/Disks/tests/cas_sweep_test_support.h +++ b/src/Disks/tests/cas_sweep_test_support.h @@ -22,15 +22,15 @@ inline ManifestSweepResult sweepManifestCursorPageForTest( { ManifestSweepResult result = planManifestCursorPage( store, cursor, list_budget, delete_budget, /*catalog_recovery_authoritative=*/true, work_budget); - CasOperation op = store.gcRequests().admit(); + CasOperation op = store.openRequests().admit(); for (const ManifestSweepResult::Nomination & nomination : result.nominations) { /// A nomination records the incarnation it was planned against, so the delete re-observes the /// key and refuses unless what is there now is still that one: a key a fresh owner has since /// replaced must survive. const std::optional seen = op.head(nomination.key, Retry::standard()); - if (seen && nomination.token.matches(seen->incarnation) - && op.remove(nomination.key, seen->incarnation, Retry::standard()) == Removal::Removed) + if (seen && nomination.token.matches(seen->etag) + && op.remove(nomination.key, seen->etag, Retry::standard()) == Removal::Removed) ++result.deleted; else ++result.skipped; diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 141e04c64a38..27a232219509 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -79,6 +79,48 @@ namespace DB::ErrorCodes namespace DB::Cas::tests { +/// A `CasRequests` over an always-open fence, for a fixture that has a backend but no mounted pool. +/// Every operation admitted from it holds a reference to it, so it must be named and outlive them. +inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::BackendPtr backend) +{ + return DB::Cas::CasRequests(std::move(backend), DB::Cas::Fence::open()); +} + +/// The same, for a fixture holding only a reference. The aliasing `shared_ptr` owns nothing, so the +/// caller keeps the backend alive for as long as the returned object and its operations live. +inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::Backend & backend) +{ + return openRequestsForTest(DB::Cas::BackendPtr(std::shared_ptr(), &backend)); +} + +/// An open-fence operation together with the `CasRequests` it refers to, for a fixture that holds a +/// backend and needs to call a production entry point taking a `CasOperation &`. Neither copyable nor +/// movable: the operation points at the member beside it. +class OperationForTest +{ +public: + explicit OperationForTest(DB::Cas::BackendPtr backend) + : requests(std::move(backend), DB::Cas::Fence::open()), operation(requests.admit()) + { + } + + /// For a fixture holding only a reference: the aliasing `shared_ptr` owns nothing, so the caller + /// keeps the backend alive for as long as this object. + explicit OperationForTest(DB::Cas::Backend & backend) + : OperationForTest(DB::Cas::BackendPtr(std::shared_ptr(), &backend)) + { + } + + OperationForTest(const OperationForTest &) = delete; + OperationForTest & operator=(const OperationForTest &) = delete; + + DB::Cas::CasOperation & operator*() { return operation; } + +private: + DB::Cas::CasRequests requests; + DB::Cas::CasOperation operation; +}; + /// Deterministic two-phase barrier for worker-lifecycle tests. The worker calls `arriveAndWait` at /// the exact operation boundary under test; the test waits for that arrival and later calls /// `release`. The bounded waits are only hang protection -- correctness never depends on elapsed @@ -187,7 +229,8 @@ void expectThrowsCode(int expected_code, F && fn) /// an empty optional and take the whole binary down instead of failing this one case. inline void expectBytes(DB::Cas::Backend & backend, const String & key, const String & expected) { - const auto got = backend.get(key); + OperationForTest op(backend); + const auto got = (*op).read(key, DB::Cas::Retry::standard()); ASSERT_TRUE(got.has_value()) << "object '" << key << "' is absent"; EXPECT_EQ(got->bytes, expected); } @@ -197,48 +240,6 @@ inline void expectBytes(const DB::Cas::BackendPtr & backend, const String & key, expectBytes(*backend, key, expected); } -/// A `CasRequests` over an always-open fence, for a fixture that has a backend but no mounted pool. -/// Every operation admitted from it holds a reference to it, so it must be named and outlive them. -inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::BackendPtr backend) -{ - return DB::Cas::CasRequests(std::move(backend), DB::Cas::Fence::open()); -} - -/// The same, for a fixture holding only a reference. The aliasing `shared_ptr` owns nothing, so the -/// caller keeps the backend alive for as long as the returned object and its operations live. -inline DB::Cas::CasRequests openRequestsForTest(DB::Cas::Backend & backend) -{ - return openRequestsForTest(DB::Cas::BackendPtr(std::shared_ptr(), &backend)); -} - -/// An open-fence operation together with the `CasRequests` it refers to, for a fixture that holds a -/// backend and needs to call a production entry point taking a `CasOperation &`. Neither copyable nor -/// movable: the operation points at the member beside it. -class OperationForTest -{ -public: - explicit OperationForTest(DB::Cas::BackendPtr backend) - : requests(std::move(backend), DB::Cas::Fence::open()), operation(requests.admit()) - { - } - - /// For a fixture holding only a reference: the aliasing `shared_ptr` owns nothing, so the caller - /// keeps the backend alive for as long as this object. - explicit OperationForTest(DB::Cas::Backend & backend) - : OperationForTest(DB::Cas::BackendPtr(std::shared_ptr(), &backend)) - { - } - - OperationForTest(const OperationForTest &) = delete; - OperationForTest & operator=(const OperationForTest &) = delete; - - DB::Cas::CasOperation & operator*() { return operation; } - -private: - DB::Cas::CasRequests requests; - DB::Cas::CasOperation operation; -}; - /// Build a `LocalObjectStorage` rooted at a fresh, unique temporary directory (one per call). /// /// Used by the unit tests that exercise the `Cas::Backend` seam against a real on-disk object storage @@ -333,7 +334,8 @@ inline DB::Cas::BlobRef writeBlobRaw( header.build_id = DB::UInt128(0x5678); const String head = DB::Cas::encodeEnvelopeHeader(header, static_cast(blob_header_len)); - backend.putIfAbsent(layout.blobKey(id), head + payload); + OperationForTest op(backend); + (*op).create(layout.blobKey(id), head + payload, DB::Cas::Retry::standard()); return id; } @@ -355,8 +357,9 @@ inline DB::Cas::ManifestId writeManifestRaw( body.root_namespace_id = ns; body.entries = entries; body.payload_digest = DB::Cas::computePayloadDigest(body); - backend.putIfAbsent(layout.manifestKey(id), - DB::Cas::sealObject(DB::Cas::FormatId::PartManifest, DB::Cas::encodePartManifest(body))); + OperationForTest op(backend); + (*op).create(layout.manifestKey(id), + DB::Cas::sealObject(DB::Cas::FormatId::PartManifest, DB::Cas::encodePartManifest(body)), DB::Cas::Retry::standard()); return id; } @@ -424,7 +427,7 @@ inline uint64_t appendRefLogSeed( String cursor; while (true) { - const DB::Cas::ListPage page = backend.list(prefix, cursor, /*limit=*/1000); + const DB::Cas::ListPage page = (*operation).list(prefix, cursor, /*limit=*/1000, DB::Cas::Retry::standard()); for (const DB::Cas::ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -534,9 +537,9 @@ inline void deleteManifestBody( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::ManifestId & id) { const String key = layout.manifestKey(id); - const DB::Cas::HeadResult h = backend.head(key); - if (h.exists) - backend.deleteExact(key, h.token); + OperationForTest op(backend); + if (const auto h = (*op).head(key, DB::Cas::Retry::standard())) + (*op).remove(key, h->etag, DB::Cas::Retry::standard()); } /// Formerly wrote the namespace into `gc/registry`. Real write helpers now admit the authoritative @@ -573,9 +576,9 @@ inline void injectRetire( { OperationForTest operation(backend); DB::Cas::GcState gc_state; - const DB::Cas::HeadResult head = backend.head(layout.gcStateKey()); - if (head.exists) - gc_state = DB::Cas::decodeGcState(backend.get(layout.gcStateKey())->bytes); + const auto existing_state = (*operation).read(layout.gcStateKey(), DB::Cas::Retry::standard()); + if (existing_state) + gc_state = DB::Cas::decodeGcState(existing_state->bytes); gc_state.round = round; if (!entries.empty()) @@ -636,17 +639,17 @@ inline void injectRetire( cs.pending_total = pending_total; cs.oldest_nonpending_condemn_round = oldest_nonpending; seal.condemned_summary[shard] = cs; - backend.putIfAbsent(layout.foldSealKey(generation, attempt), DB::Cas::encodeFoldSeal(seal)); + (*operation).create(layout.foldSealKey(generation, attempt), DB::Cas::encodeFoldSeal(seal), DB::Cas::Retry::standard()); gc_state.snap_generation = generation; gc_state.snap_attempt = attempt; } const String state = DB::Cas::encodeGcState(gc_state); - if (!head.exists) - backend.putIfAbsent(layout.gcStateKey(), state); + if (!existing_state) + (*operation).create(layout.gcStateKey(), state, DB::Cas::Retry::standard()); else - backend.putOverwrite(layout.gcStateKey(), state, head.token); + (*operation).replace(layout.gcStateKey(), state, existing_state->etag, DB::Cas::Retry::standard()); } /// Adopt a fold seal carrying a given per-gc-shard `condemned_summary` and point @@ -659,9 +662,10 @@ inline void injectCondemnedSummarySeal( uint64_t generation, uint64_t attempt, uint64_t gc_shards, const std::map & summary) { + OperationForTest operation(backend); const String seal_key = layout.foldSealKey(generation, attempt); DB::Cas::CasFoldSeal seal; - const auto existing = backend.get(seal_key); + const auto existing = (*operation).read(seal_key, DB::Cas::Retry::standard()); if (existing) seal = DB::Cas::decodeFoldSeal(existing->bytes); else @@ -670,28 +674,30 @@ inline void injectCondemnedSummarySeal( seal.condemned_summary = summary; const String seal_bytes = DB::Cas::encodeFoldSeal(seal); if (existing) - backend.putOverwrite(seal_key, seal_bytes, existing->token); + (*operation).replace(seal_key, seal_bytes, existing->etag, DB::Cas::Retry::standard()); else - backend.putIfAbsent(seal_key, seal_bytes); + (*operation).create(seal_key, seal_bytes, DB::Cas::Retry::standard()); DB::Cas::GcState gc_state; - const DB::Cas::HeadResult head = backend.head(layout.gcStateKey()); - if (head.exists) - gc_state = DB::Cas::decodeGcState(backend.get(layout.gcStateKey())->bytes); + const auto existing_state = (*operation).read(layout.gcStateKey(), DB::Cas::Retry::standard()); + if (existing_state) + gc_state = DB::Cas::decodeGcState(existing_state->bytes); gc_state.gc_shards = gc_shards; gc_state.snap_generation = generation; gc_state.snap_attempt = attempt; const String state = DB::Cas::encodeGcState(gc_state); - if (!head.exists) - backend.putIfAbsent(layout.gcStateKey(), state); + if (!existing_state) + (*operation).create(layout.gcStateKey(), state, DB::Cas::Retry::standard()); else - backend.putOverwrite(layout.gcStateKey(), state, head.token); + (*operation).replace(layout.gcStateKey(), state, existing_state->etag, DB::Cas::Retry::standard()); } /// Whether blob `hash` is absent from the backend (its exact-token content object is gone). inline bool blobAbsent(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::UInt128 & hash) { - return !backend.head(layout.blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)})).exists; + OperationForTest op(backend); + return !(*op).head(layout.blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)}), + DB::Cas::Retry::standard()).has_value(); } /// ONE round that is allowed to RECLAIM -- the name is the point, so that grepping for the tests whose @@ -736,13 +742,13 @@ inline std::vector currentRetiredSet( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, uint64_t shard) { OperationForTest operation(backend); - const auto st = backend.get(layout.gcStateKey()); + const auto st = (*operation).read(layout.gcStateKey(), DB::Cas::Retry::standard()); if (!st) return {}; const DB::Cas::GcState gc_state = DB::Cas::decodeGcState(st->bytes); if (gc_state.snap_generation == 0) return {}; - const auto seal_bytes = backend.get(layout.foldSealKey(gc_state.snap_generation, gc_state.snap_attempt)); + const auto seal_bytes = (*operation).read(layout.foldSealKey(gc_state.snap_generation, gc_state.snap_attempt), DB::Cas::Retry::standard()); if (!seal_bytes) return {}; const DB::Cas::CasFoldSeal seal = DB::Cas::decodeFoldSeal(seal_bytes->bytes); @@ -781,7 +787,8 @@ inline std::vector currentRetiredSet( inline bool anyCondemnedInSeal( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, uint64_t gc_shards = 0) { - const auto st = backend.get(layout.gcStateKey()); + OperationForTest operation(backend); + const auto st = (*operation).read(layout.gcStateKey(), DB::Cas::Retry::standard()); if (!st) return false; const DB::Cas::GcState gc_state = DB::Cas::decodeGcState(st->bytes); @@ -796,7 +803,7 @@ inline bool anyCondemnedInSeal( /// incarnation_tag in its envelope header (preserving header_len + payload), putOverwrite against the /// current token, and return the NEW token. Used to drive the W-REVALIDATE adopt branch (current token /// differs from the writer's stale observation). -inline DB::Cas::Incarnation displaceObjectToken( +inline DB::Cas::Etag displaceObjectToken( DB::Cas::Backend & backend, const String & key, DB::Cas::ObjectKind kind) { OperationForTest operation(backend); @@ -812,8 +819,8 @@ inline DB::Cas::Incarnation displaceObjectToken( const String new_head = DB::Cas::encodeEnvelopeHeader(header, header.header_len); const String body = new_head + got->bytes.substr(header.header_len); - const std::optional displaced = DB::Cas::orThrow( - (*operation).replace(key, body, got->incarnation, DB::Cas::Retry::standard()), + const std::optional displaced = DB::Cas::orThrow( + (*operation).replace(key, body, got->etag, DB::Cas::Retry::standard()), "displace the object at " + key); if (!displaced) throw DB::Exception(DB::ErrorCodes::BAD_ARGUMENTS, @@ -821,7 +828,7 @@ inline DB::Cas::Incarnation displaceObjectToken( return *displaced; } -inline DB::Cas::Incarnation displaceBlobToken( +inline DB::Cas::Etag displaceBlobToken( DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::Cas::BlobRef & id) { return displaceObjectToken(backend, layout.blobKey(id), DB::Cas::ObjectKind::Blob); @@ -866,7 +873,7 @@ inline void seedPoolMetaForRestart( DB::Cas::PoolMeta::createOrValidate( *operation, layout, /*blob_header_len=*/256, gc_shards, DB::Cas::BlobHashAlgo::CityHash128, /*allow_new=*/false, /*allow_mint=*/true); - if (!backend.get(layout.refCatalogKey())) + if (!(*operation).read(layout.refCatalogKey(), DB::Cas::Retry::standard())) DB::Cas::CasRefCatalog::initializeEmptyForNewPool(*operation, layout); else (void)DB::Cas::CasRefCatalog::read(*operation, layout); @@ -883,7 +890,9 @@ inline void writeBlobBody( header.incarnation_tag = DB::UInt128(0x1234); header.build_id = DB::UInt128(0x5678); const String head = DB::Cas::encodeEnvelopeHeader(header, static_cast(blob_header_len)); - backend.putIfAbsent(layout.blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)}), head + String("x")); + OperationForTest op(backend); + (*op).create(layout.blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)}), head + String("x"), + DB::Cas::Retry::standard()); } /// Write a raw blob body (payload written verbatim, no envelope) — the raw-body-refinement shape @@ -891,7 +900,9 @@ inline void writeBlobBody( inline void writeRawBlobBody(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const DB::UInt128 & hash, const String & payload) { - backend.casPut(layout.blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)}), payload, std::nullopt); + OperationForTest op(backend); + (*op).create(layout.blobKey(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)}), payload, + DB::Cas::Retry::standard()); } /// These `UInt128`-hash meta-op wrappers are the pre-mixed-algo 128-bit-only test convenience surface: @@ -909,8 +920,9 @@ inline void writeMetaClean(DB::Cas::Backend & backend, const DB::Cas::Layout & l const DB::UInt128 & hash, uint64_t size) { const DB::Cas::BlobRef ref = legacyMetaTestRef(hash); - backend.putIfAbsent(layout.blobMetaKey(ref), DB::Cas::encodeBlobMeta( - DB::Cas::BlobMeta{.state = DB::Cas::MetaState::Clean, .condemn_round = 0, .size = size})); + OperationForTest op(backend); + (*op).create(layout.blobMetaKey(ref), DB::Cas::encodeBlobMeta( + DB::Cas::BlobMeta{.state = DB::Cas::MetaState::Clean, .condemn_round = 0, .size = size}), DB::Cas::Retry::standard()); } /// Transition an existing meta descriptor to Condemned at `condemn_round`, via a read-modify-CAS on @@ -926,7 +938,7 @@ inline void condemnMeta(DB::Cas::Backend & backend, const DB::Cas::Layout & layo c.state = DB::Cas::MetaState::Condemned; c.condemn_round = condemn_round; ASSERT_TRUE(std::holds_alternative( - DB::Cas::casMeta(*operation, layout, ref, lm->incarnation, c))); + DB::Cas::casMeta(*operation, layout, ref, lm->etag, c))); } /// Load the meta descriptor for `hash` via the shared ops layer (nullopt = absent). @@ -940,7 +952,8 @@ inline std::optional loadMetaForTest(DB::Cas::Backend & bac /// The latest GC generation (snap_generation pointer in gc/state), or 0 when absent. inline uint64_t currentGenerationOf(DB::Cas::Backend & backend, const DB::Cas::Layout & layout) { - const auto got = backend.get(layout.gcStateKey()); + OperationForTest op(backend); + const auto got = (*op).read(layout.gcStateKey(), DB::Cas::Retry::standard()); if (!got) return 0; return DB::Cas::decodeGcState(got->bytes).snap_generation; @@ -949,7 +962,8 @@ inline uint64_t currentGenerationOf(DB::Cas::Backend & backend, const DB::Cas::L /// The adopted attempt (snap_attempt pointer in gc/state), or 0 when absent. inline uint64_t currentAttemptOf(DB::Cas::Backend & backend, const DB::Cas::Layout & layout) { - const auto got = backend.get(layout.gcStateKey()); + OperationForTest op(backend); + const auto got = (*op).read(layout.gcStateKey(), DB::Cas::Retry::standard()); if (!got) return 0; return DB::Cas::decodeGcState(got->bytes).snap_attempt; @@ -963,9 +977,10 @@ inline std::vector runsForShard( { const uint64_t gen = currentGenerationOf(backend, layout); const uint64_t attempt = currentAttemptOf(backend, layout); + OperationForTest op(backend); for (uint64_t g = gen; ; --g) { - if (const auto got = backend.get(layout.foldSealKey(g, attempt))) + if (const auto got = (*op).read(layout.foldSealKey(g, attempt), DB::Cas::Retry::standard())) { const DB::Cas::CasFoldSeal seal = DB::Cas::decodeFoldSeal(got->bytes); std::vector out; @@ -1103,7 +1118,7 @@ inline void seedFoldCursorForTest( const String seal_key = layout.foldSealKey(generation, attempt); DB::Cas::CasFoldSeal seal; - const auto existing = backend.get(seal_key); + const auto existing = (*operation).read(seal_key, DB::Cas::Retry::standard()); if (existing) seal = DB::Cas::decodeFoldSeal(existing->bytes); seal.generation = generation; @@ -1115,9 +1130,9 @@ inline void seedFoldCursorForTest( seal.ref_lives[life.incarnation].coverage = cov; DB::Cas::GcState gc_state; - const DB::Cas::HeadResult head = backend.head(layout.gcStateKey()); - if (head.exists) - gc_state = DB::Cas::decodeGcState(backend.get(layout.gcStateKey())->bytes); + const auto existing_state = (*operation).read(layout.gcStateKey(), DB::Cas::Retry::standard()); + if (existing_state) + gc_state = DB::Cas::decodeGcState(existing_state->bytes); /// Totality over `gc_shards` — see the doc comment's SHARP EDGE note for what throws without it. const uint64_t gc_shards = gc_state.gc_shards ? gc_state.gc_shards : 1; @@ -1126,17 +1141,17 @@ inline void seedFoldCursorForTest( const String seal_bytes = DB::Cas::encodeFoldSeal(seal); if (existing) - backend.putOverwrite(seal_key, seal_bytes, existing->token); + (*operation).replace(seal_key, seal_bytes, existing->etag, DB::Cas::Retry::standard()); else - backend.putIfAbsent(seal_key, seal_bytes); + (*operation).create(seal_key, seal_bytes, DB::Cas::Retry::standard()); gc_state.snap_generation = generation; gc_state.snap_attempt = attempt; const String state = DB::Cas::encodeGcState(gc_state); - if (!head.exists) - backend.putIfAbsent(layout.gcStateKey(), state); + if (!existing_state) + (*operation).create(layout.gcStateKey(), state, DB::Cas::Retry::standard()); else - backend.putOverwrite(layout.gcStateKey(), state, head.token); + (*operation).replace(layout.gcStateKey(), state, existing_state->etag, DB::Cas::Retry::standard()); } /// The folded cursor sealed for (ns, shard) by the latest fold seal, or 0 when absent. After a COMPLETE @@ -1156,7 +1171,7 @@ inline uint64_t foldCursorOf( const uint64_t attempt = currentAttemptOf(backend, layout); for (uint64_t g = gen; ; --g) { - if (const auto got = backend.get(layout.foldSealKey(g, attempt))) + if (const auto got = (*operation).read(layout.foldSealKey(g, attempt), DB::Cas::Retry::standard())) { const DB::Cas::CasFoldSeal seal = DB::Cas::decodeFoldSeal(got->bytes); const auto it = seal.ref_lives.find(life->incarnation); @@ -1184,11 +1199,12 @@ inline void setWatermarkMinActive( m.seq = 1; m.write_attempt_id = DB::UInt128{1}; const String key = layout.mountKey(server_root_id); - const DB::Cas::HeadResult h = backend.head(key); - if (h.exists) - backend.putOverwrite(key, DB::Cas::encodeMountLease(m), h.token); + OperationForTest op(backend); + const auto h = (*op).head(key, DB::Cas::Retry::standard()); + if (h) + (*op).replace(key, DB::Cas::encodeMountLease(m), h->etag, DB::Cas::Retry::standard()); else - backend.putIfAbsent(key, DB::Cas::encodeMountLease(m)); + (*op).create(key, DB::Cas::encodeMountLease(m), DB::Cas::Retry::standard()); } /// ---- Task 10 ref snapshot+log raw fixtures ---- @@ -1211,7 +1227,8 @@ inline void writeRefSnapshotRaw( const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*operation, layout, ns).value_or(fixture::fixtureLife(ns)); const String key = layout.refSnapshotKey(life, snapshot.snapshot_id); - backend.putIfAbsent(key, DB::Cas::sealObject(DB::Cas::FormatId::RefSnapshot, DB::Cas::encodeRefTableSnapshot(snapshot))); + (*operation).create(key, DB::Cas::sealObject(DB::Cas::FormatId::RefSnapshot, DB::Cas::encodeRefTableSnapshot(snapshot)), + DB::Cas::Retry::standard()); } /// Admits `ns` into the catalog as a `Live` entry, IDEMPOTENTLY (a no-op once `ns` already carries @@ -1311,7 +1328,7 @@ inline void advanceRecoverableCkptForRawFixture( RefCkpt advanced = sample->ckpt; advanced.committed_through = through; if (!std::holds_alternative( - (*operation).replace(layout.refCkptKey(life), encodeRefCkpt(advanced), sample->incarnation, + (*operation).replace(layout.refCkptKey(life), encodeRefCkpt(advanced), sample->etag, DB::Cas::Retry::standard()))) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' could not advance its checkpoint", ns.string()); @@ -1351,7 +1368,7 @@ inline void replaceRecoverableCkptForRawFixture( "raw recovery fixture for namespace '{}' cannot regress its checkpoint frontier", ns.string()); if (!std::holds_alternative( - (*operation).replace(layout.refCkptKey(life), encodeRefCkpt(next), existing->incarnation, + (*operation).replace(layout.refCkptKey(life), encodeRefCkpt(next), existing->etag, DB::Cas::Retry::standard()))) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "raw recovery fixture for namespace '{}' could not replace its checkpoint", ns.string()); @@ -1465,7 +1482,7 @@ inline void writeRefLogTxnRaw( const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(*operation, layout, ns).value_or(fixture::fixtureLife(ns)); const String key = layout.refLogKey(life, txn.txn_id); - backend.putIfAbsent(key, DB::Cas::sealObject(DB::Cas::FormatId::RefLog, DB::Cas::encodeRefLogTxn(txn))); + (*operation).create(key, DB::Cas::sealObject(DB::Cas::FormatId::RefLog, DB::Cas::encodeRefLogTxn(txn)), DB::Cas::Retry::standard()); } namespace fixture @@ -1627,12 +1644,6 @@ class ChunkedStreamForTest : public DB::ReadBuffer class CountingBackend : public DB::Cas::InMemoryBackend { public: - /// Unhide the legacy names the primitive overrides below would otherwise shadow, and the - /// omitted-`Range` convenience. - using DB::Cas::Backend::getStream; - using DB::Cas::Backend::head; - using DB::Cas::Backend::list; - /// ---- The counters live on the transport primitives, so a request is counted once ---- /// /// Whichever surface a caller used, its request passes through one of the primitives below: a @@ -1697,18 +1708,14 @@ class CountingBackend : public DB::Cas::InMemoryBackend return InMemoryBackend::remove(key, expected_value, access); } - /// `InMemoryBackend::stream` opens its reader through this, so both the primitive and the legacy - /// stream verb are counted here, once each. - std::optional getStream(const String & key, DB::Cas::Range range) override + std::unique_ptr stream(const String & key, DB::Cas::TransportAccess & access) override { tick(get_stream_counts, get_stream_total, key); - std::optional opened = InMemoryBackend::getStream(key, range); + std::unique_ptr opened = InMemoryBackend::stream(key, access); const size_t chunk = stream_chunk.load(); - if (!opened || !opened->stream || chunk == 0) + if (!opened || chunk == 0) return opened; - opened->stream = std::make_unique( - std::move(opened->stream), chunk, largestChunkSlot(key)); - return opened; + return std::make_unique(std::move(opened), chunk, largestChunkSlot(key)); } /// Serve every stream opened from now on in windows of at most `bytes`, as a network-backed store @@ -2467,7 +2474,7 @@ inline void awaitLatchEntered(MetaWriteLatchBackend & backend) } /// Runs a caller-supplied action ONCE, immediately before the named backend call, so a test can make -/// the mount slot change inside a window `MountLeaseKeeper::claim` holds open. Each hook clears +/// the mount slot change inside a window `MountLeaseRenewer::claim` holds open. Each hook clears /// itself after firing. class MountSlotRaceBackend : public DB::Cas::InMemoryBackend { diff --git a/src/Disks/tests/gtest_ca_wiring.cpp b/src/Disks/tests/gtest_ca_wiring.cpp index 2003b6374c99..8fe21886f780 100644 --- a/src/Disks/tests/gtest_ca_wiring.cpp +++ b/src/Disks/tests/gtest_ca_wiring.cpp @@ -400,6 +400,23 @@ std::shared_ptr openWiringStorage() return storage; } +/// The current meta of a present key, read through the request engine over an open fence — the +/// sanctioned way for a fixture to observe a `Pool`'s backend without owning it (`Pool::backend()` is +/// gone; `poolBackendPtr()` is the surviving accessor). +DB::Cas::Meta headMetaOf(const DB::Cas::PoolPtr & pool, const String & key) +{ + DB::Cas::tests::OperationForTest op(*pool->poolBackendPtr()); + const auto h = (*op).head(key, DB::Cas::Retry::standard()); + if (!h.has_value()) + throw std::runtime_error("headMetaOf: key '" + key + "' is absent"); + return *h; +} + +DB::Cas::Etag headIncarnationOf(const DB::Cas::PoolPtr & pool, const String & key) +{ + return headMetaOf(pool, key).etag; +} + /// One part with a content blob, a projection file, and the small per-part files (uuid.txt, /// metadata_version.txt — ordinary Inline entries now, all-tree-part-files Task 6/9), published /// through the real PartWriteTxn into `ns` under `ref`. @@ -601,9 +618,9 @@ TEST(CASWiringRead, DeletedBlobUnderStaleViewFailsTypedNotEmpty) auto pool = storage->store(); const String blob_key = pool->layout().blobKey(DB::Cas::tests::idOf("payload-A")); { - const DB::Cas::HeadResult h = pool->backend().head(blob_key); - ASSERT_TRUE(h.exists); - pool->backend().deleteExact(blob_key, h.token); + const DB::Cas::Etag incarnation = headIncarnationOf(pool, blob_key); + DB::Cas::tests::OperationForTest op(*pool->poolBackendPtr()); + ASSERT_EQ((*op).remove(blob_key, incarnation, DB::Cas::Retry::standard()), DB::Cas::Removal::Removed); } /// Planning still succeeds from the cached manifest and names the same object. @@ -1713,8 +1730,8 @@ TEST(CASWiringExchange, AdoptPartFromManifestPublishesFreshLocalManifest) /// the receiver never overwrites or re-creates the blobs — their head tokens are unchanged. const auto data_key = storage->store()->layout().blobKey(idOf("payload-A")); const auto proj_key = storage->store()->layout().blobKey(idOf("payload-B")); - const auto data_tok_before = storage->store()->backend().head(data_key).token; - const auto proj_tok_before = storage->store()->backend().head(proj_key).token; + const auto data_tok_before = headIncarnationOf(storage->store(), data_key); + const auto proj_tok_before = headIncarnationOf(storage->store(), proj_key); /// Adopt into a DIFFERENT table (a22a22a2-2222-4222-8222-222222222222). The transferred body's root_namespace_id is the sender's /// (a11a11a1-1111-4111-8111-111111111111) — the receiver must IGNORE it and use a22a22a2-2222-4222-8222-222222222222. @@ -1738,9 +1755,9 @@ TEST(CASWiringExchange, AdoptPartFromManifestPublishesFreshLocalManifest) EXPECT_FALSE(receiver_ns.string() == sender_ns.string()); /// NO blob body was uploaded: the shared blobs' incarnations are untouched by the adopt. - EXPECT_EQ(storage->store()->backend().head(data_key).token, data_tok_before) + EXPECT_EQ(headIncarnationOf(storage->store(), data_key), data_tok_before) << "adopt-from-manifest must not re-upload a blob already in the shared pool"; - EXPECT_EQ(storage->store()->backend().head(proj_key).token, proj_tok_before); + EXPECT_EQ(headIncarnationOf(storage->store(), proj_key), proj_tok_before); } /// B7 fail-closed: if a referenced blob is absent/condemned in the pool, adoptPartFromManifest must @@ -1768,10 +1785,11 @@ TEST(CASWiringExchange, AdoptFailsClosedAndFallsBackOnCondemnedBlob) /// Artificially delete a referenced pool blob — the live-sender invariant excludes this on the real /// path; §4 promote does not re-probe it, so adopt trusts the manifest edge and publishes. const auto data_key = storage->store()->layout().blobKey(idOf("payload-A")); - const auto h = storage->store()->backend().head(data_key); - ASSERT_TRUE(h.exists); - ASSERT_EQ(storage->store()->backend().deleteExact(data_key, h.token).kind, - DB::Cas::DeleteOutcome::Kind::Deleted); + { + const DB::Cas::Etag incarnation = headIncarnationOf(storage->store(), data_key); + DB::Cas::tests::OperationForTest op(*storage->store()->poolBackendPtr()); + ASSERT_EQ((*op).remove(data_key, incarnation, DB::Cas::Retry::standard()), DB::Cas::Removal::Removed); + } /// §4: promote trusts the adopted leaves — no re-probe — so adopt SUCCEEDS (returns true) and publishes. const bool ok = adoptPartFromManifestAndPromote(*exchange, kReceiverTmpFetchPath, bytes); @@ -2887,17 +2905,18 @@ DB::Cas::PoolPtr openResurrectStore(std::shared_ptr & /// meta, which is what the writer's condemned decision ACTUALLY point-reads (spec §meta-protocols v3). /// Bumps the round so the retirement is a fresh one; leaves the object itself in place (condemn, NOT delete). void seedCondemnBlobToken(DB::Cas::Pool & store, const DB::UInt128 & hash, - [[maybe_unused]] const DB::Cas::Token & token, [[maybe_unused]] uint64_t size) + [[maybe_unused]] const DB::Cas::Etag & token, [[maybe_unused]] uint64_t size) { using namespace DB::Cas; - Backend & b = store.backend(); + Backend & b = *store.poolBackendPtr(); const Layout & layout = store.layout(); + DB::Cas::tests::OperationForTest op(b); GcState state; - const HeadResult head = b.head(layout.gcStateKey()); - if (head.exists) + const auto head = (*op).head(layout.gcStateKey(), Retry::standard()); + if (head.has_value()) { - const auto got = b.get(layout.gcStateKey()); + const auto got = (*op).read(layout.gcStateKey(), Retry::standard()); state = decodeGcState(got->bytes); } state.round += 1; @@ -2906,10 +2925,10 @@ void seedCondemnBlobToken(DB::Cas::Pool & store, const DB::UInt128 & hash, /// GC snapshot runs, which this writer-side edge-protection test does not exercise. The writer's /// condemned decision point-reads the per-hash freshness meta (condemned below), so bumping the round /// and condemning the meta is enough. - if (head.exists) - b.putOverwrite(layout.gcStateKey(), encodeGcState(state), head.token); + if (head.has_value()) + (void)(*op).replace(layout.gcStateKey(), encodeGcState(state), head->etag, Retry::standard()); else - b.putIfAbsent(layout.gcStateKey(), encodeGcState(state)); + (void)(*op).create(layout.gcStateKey(), encodeGcState(state), Retry::standard()); /// The writer's fresh upload (putBlob) already wrote a Clean meta for `hash` (Task 3), so this is a /// plain Clean -> Condemned CAS — exactly what GC's real condemn path does. @@ -2942,12 +2961,11 @@ TEST(CASWiringResurrect, PromoteIgnoresCondemnedMaterializedBlobEdgeProtected) /// Condemn the freshly-uploaded blob's CURRENT token (GC condemning the not-yet-folded fresh incarnation). const String blob_key = store->layout().blobKey(idOf(P)); - const HeadResult h1 = store->backend().head(blob_key); - ASSERT_TRUE(h1.exists); - const Token t0 = h1.token; + const Meta h1 = headMetaOf(store, blob_key); + const Etag t0 = h1.etag; seedCondemnBlobToken(*store, u128Of(P), t0, h1.size); { - const auto lm = DB::Cas::tests::loadMetaForTest(store->backend(), store->layout(), u128Of(P)); + const auto lm = DB::Cas::tests::loadMetaForTest(*store->poolBackendPtr(), store->layout(), u128Of(P)); ASSERT_TRUE(lm.has_value() && lm->meta.state == MetaState::Condemned) << "precondition: the putBlob'd token must be condemned before promote"; } @@ -2958,9 +2976,7 @@ TEST(CASWiringResurrect, PromoteIgnoresCondemnedMaterializedBlobEdgeProtected) /// The ref is committed and the blob's token is unchanged — no replacement PUT ran (`Materialized` leaves are /// not re-validated: EDGE-BEFORE-OBSERVE guarantees the condemnation is doomed, not the blob). EXPECT_TRUE(store->resolveRef(ns, ref).has_value()) << "the ref must resolve after promote"; - const HeadResult h2 = store->backend().head(blob_key); - ASSERT_TRUE(h2.exists); - EXPECT_EQ(h2.token, t0) + EXPECT_EQ(headIncarnationOf(store, blob_key), t0) << "materialized leaf is edge-protected: promote must not re-upload it (token unchanged)"; } @@ -2998,18 +3014,17 @@ TEST(CASWiringResurrect, PromoteWithoutLivePrecommitAbortsWithoutResurrect) /// Physical publication through `putBlob` requires that durable edge, while this test deliberately /// needs the owner binding absent when `promote` runs. DB::Cas::tests::writeBlobRaw( - store->backend(), store->layout(), P, store->poolMeta().blob_header_len, store->poolMeta().pool_id); - DB::Cas::tests::writeMetaClean(store->backend(), store->layout(), u128Of(P), P.size()); + *store->poolBackendPtr(), store->layout(), P, store->poolMeta().blob_header_len, store->poolMeta().pool_id); + DB::Cas::tests::writeMetaClean(*store->poolBackendPtr(), store->layout(), u128Of(P), P.size()); const ManifestId id = build->stageManifest({wiringBlobEntry("data.bin", P)}); const String blob_key = store->layout().blobKey(idOf(P)); - const HeadResult h1 = store->backend().head(blob_key); - ASSERT_TRUE(h1.exists); + const Meta h1 = headMetaOf(store, blob_key); /// Condemn the leaf so that, were the blob gate reached, promote would republish it — proving the abort /// happens strictly BEFORE any blob work. - seedCondemnBlobToken(*store, u128Of(P), h1.token, h1.size); + seedCondemnBlobToken(*store, u128Of(P), h1.etag, h1.size); { - const auto lm = DB::Cas::tests::loadMetaForTest(store->backend(), store->layout(), u128Of(P)); + const auto lm = DB::Cas::tests::loadMetaForTest(*store->poolBackendPtr(), store->layout(), u128Of(P)); ASSERT_TRUE(lm.has_value() && lm->meta.state == MetaState::Condemned); } @@ -3026,11 +3041,9 @@ TEST(CASWiringResurrect, PromoteWithoutLivePrecommitAbortsWithoutResurrect) /// No blob work ran before the abort: the leaf's token is UNCHANGED (still the condemned one) and its /// metadata is still Condemned — the owner check aborts before any blob publication. - const HeadResult h2 = store->backend().head(blob_key); - ASSERT_TRUE(h2.exists); - EXPECT_EQ(h2.token, h1.token) + EXPECT_EQ(headIncarnationOf(store, blob_key), h1.etag) << "the aborting path must perform no PUT — the materialized leaf is untouched"; - const auto lm_after = DB::Cas::tests::loadMetaForTest(store->backend(), store->layout(), u128Of(P)); + const auto lm_after = DB::Cas::tests::loadMetaForTest(*store->poolBackendPtr(), store->layout(), u128Of(P)); EXPECT_TRUE(lm_after.has_value() && lm_after->meta.state == MetaState::Condemned) << "no republication before the owner check — the token is still the condemned one"; } diff --git a/src/Disks/tests/gtest_cas_b140_dangle.cpp b/src/Disks/tests/gtest_cas_b140_dangle.cpp index 05af3e47e1cf..9ec0cb9d6dcf 100644 --- a/src/Disks/tests/gtest_cas_b140_dangle.cpp +++ b/src/Disks/tests/gtest_cas_b140_dangle.cpp @@ -117,11 +117,14 @@ TEST(CASGCDangle, SharedBlobSurvivesDropOfOneOfTwoLiveRefs) const FsckReport rep = runFsck(*s, /*detail=*/true); + DB::Cas::tests::OperationForTest op(*b); + const bool b_present = (*op).head(s->layout().blobKey(idOf("B")), DB::Cas::Retry::once()).has_value(); + /// THE DANGLE ASSERTION: GC must NEVER delete a blob a live ref references. EXPECT_EQ(rep.dangling, 0u) << "B140-dangle: GC deleted shared blob B still referenced by the live ref rb_cur " << "after " << rounds << " rounds (dangling=" << rep.dangling << ", reachable=" << rep.reachable - << ", B_present=" << b->head(s->layout().blobKey(idOf("B"))).exists << ")."; - EXPECT_TRUE(b->head(s->layout().blobKey(idOf("B"))).exists) + << ", B_present=" << b_present << ")."; + EXPECT_TRUE(b_present) << "shared blob B must remain present while rb_cur references it"; } diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index aabb3f21f0c1..b40b24e56445 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -36,9 +36,12 @@ using namespace DB::Cas; using DB::Cas::tests::expectBytes; +using DB::Cas::tests::openRequestsForTest; +using DB::Cas::tests::OperationForTest; namespace DB::ErrorCodes { +extern const int CAS_DELETE_MARKER; extern const int CORRUPTED_DATA; extern const int NOT_IMPLEMENTED; extern const int LOGICAL_ERROR; @@ -113,10 +116,10 @@ BlobPublishRequest countedLongPublication( class PublishCountingInMemoryBackend final : public InMemoryBackend { public: - void publishBlob(const BlobPublishRequest & request) override + void publish(const BlobPublishRequest & request, TransportAccess & access) override { ++publish_calls; - InMemoryBackend::publishBlob(request); + InMemoryBackend::publish(request, access); } size_t publish_calls = 0; @@ -124,120 +127,14 @@ class PublishCountingInMemoryBackend final : public InMemoryBackend } -/// Minimal concrete implementation that overrides every pure virtual with trivial defaults. -/// Purpose: verify the interface compiles, is overridable, and result-type defaults are sane. -struct NullBackend final : Backend -{ - std::optional get(const String & /*key*/, Range /*range*/) override - { - return std::nullopt; - } - - std::optional getStream(const String & /*key*/, Range /*range*/) override - { - return std::nullopt; - } - - HeadResult head(const String & /*key*/) override - { - return HeadResult{}; - } - - PutResult putIfAbsent(const String & /*key*/, const String & /*bytes*/, const ObjectMeta & /*meta*/) override - { - return {PutOutcome::Done, {}}; - } - - void publishBlob(const BlobPublishRequest & /*request*/) override - { - } - - PutResult putOverwrite(const String & /*key*/, const String & /*bytes*/, const Token & /*expected*/, const ObjectMeta & /*meta*/) override - { - return {PutOutcome::PreconditionFailed, {}}; - } - - CasResult casPut(const String & /*key*/, const String & /*bytes*/, const std::optional & /*expected*/, const ObjectMeta & /*meta*/) override - { - return {CasOutcome::Conflict, {}}; - } - - DeleteOutcome deleteExact(const String & /*key*/, const Token & /*token*/) override - { - return DeleteOutcome{}; - } - - ListPage list(const String & /*prefix*/, const String & /*cursor*/, size_t /*limit*/) override - { - return ListPage{}; - } - - /// The primitives, equally trivial. The legacy overrides above still answer the legacy calls -- - /// that is what this double is for -- so these exist to make the class concrete and to pin that - /// implementing the primitive surface alone is enough. - std::optional read(const String & /*key*/, TransportAccess &) override { return std::nullopt; } - std::optional head(const String & /*key*/, TransportAccess &) override { return std::nullopt; } - RawListPage list(const String & /*prefix*/, const String & /*cursor*/, size_t /*limit*/, TransportAccess &) override - { - return RawListPage{}; - } - RawRemoval remove(const String & /*key*/, const String & /*expected_value*/, TransportAccess &) override - { - return RawRemoval::Gone; - } - std::expected write(const String & /*key*/, const String & /*bytes*/, - const std::optional &, TransportAccess &) override - { - return std::unexpected(RawConflict{}); - } - std::unique_ptr stream(const String & /*key*/, TransportAccess &) override { return nullptr; } - void publish(const BlobPublishRequest & /*request*/, TransportAccess &) override {} - Dialect dialect() const override { return Dialect::Emulated; } - - bool supportsListTokens() const override { return false; } -}; - -TEST(CASBackend, PublishBlobReturnsNoIncarnationToken) -{ - static_assert(std::is_same_v< - decltype(std::declval().publishBlob(std::declval())), - void>); -} - -TEST(CASBackend, NullBackendShapeAndDefaults) -{ - NullBackend b; - // Use the base-class reference so virtual dispatch uses base-class default args. - Backend & ref = b; - - // get returns absent - EXPECT_FALSE(ref.get("k").has_value()); - - // head returns non-existent - HeadResult h = b.head("k"); - EXPECT_FALSE(h.exists); - EXPECT_EQ(h.size, 0u); - EXPECT_TRUE(h.token.empty()); - - // putIfAbsent returns Done - EXPECT_EQ(ref.putIfAbsent("k", "v").outcome, PutOutcome::Done); - - // putOverwrite returns PreconditionFailed - EXPECT_EQ(ref.putOverwrite("k", "v", Token{}).outcome, PutOutcome::PreconditionFailed); - - // casPut returns Conflict - EXPECT_EQ(ref.casPut("k", "v", std::nullopt).outcome, CasOutcome::Conflict); - - // deleteExact default kind is NotFound - DeleteOutcome d = b.deleteExact("k", Token{}); - EXPECT_EQ(d.kind, DeleteOutcome::Kind::NotFound); - EXPECT_FALSE(d.created_delete_marker); - - // list returns empty page - ListPage page = b.list("p/", "", 10); - EXPECT_TRUE(page.keys.empty()); - EXPECT_TRUE(page.next_cursor.empty()); -} +/// `NullBackend` and its two tests (`PublishBlobReturnsNoIncarnationToken`, +/// `NullBackendShapeAndDefaults`) are deleted here: their entire subject was the shape and defaults of +/// the legacy Token-typed forwarders (get/head/putIfAbsent/putOverwrite/casPut/deleteExact/list), which +/// no longer exist -- `Backend` now declares only the primitives, all pure virtual, with no default +/// bodies to pin. The primitive surface's own shape is exercised by every concrete-backend test below +/// (`CASInMemory`, `CASObjectStorageBackend`) through `CasRequests`/`CasOperation`, and the request +/// engine's own default behaviour (a `create` finding the key occupied, a `replace` losing its +/// precondition, a `remove` of an absent key) is pinned in `gtest_cas_requests.cpp`. // ===================================================================== // Task 3: CasInMemoryBackend — enforcing token semantics @@ -246,89 +143,117 @@ TEST(CASBackend, NullBackendShapeAndDefaults) TEST(CASInMemory, PutIfAbsentAndGet) { InMemoryBackend b; - const auto put = b.putIfAbsent("k", "v1"); - const Token t1 = put.token; - EXPECT_EQ(put.outcome, PutOutcome::Done); - EXPECT_FALSE(t1.empty()); - EXPECT_EQ(b.putIfAbsent("k", "clobber").outcome, PutOutcome::PreconditionFailed); - auto g = b.get("k"); + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + + const WriteResult put = op.create("k", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); + const Etag t1 = std::get(put).etag; + + const WriteResult clobber = op.create("k", "clobber", Retry::once()); + EXPECT_TRUE(std::holds_alternative(clobber)); + + auto g = op.read("k", Retry::once()); ASSERT_TRUE(g.has_value()); EXPECT_EQ(g->bytes, "v1"); - EXPECT_EQ(g->token, t1); - EXPECT_FALSE(b.get("absent").has_value()); + EXPECT_EQ(g->etag, t1); + EXPECT_FALSE(op.read("absent", Retry::once()).has_value()); } TEST(CASInMemory, OverwriteIsTokenExactAndMintsFreshToken) { InMemoryBackend b; - const Token t1 = b.putIfAbsent("k", "v1").token; - EXPECT_EQ(b.putOverwrite("k", "v2", Token{"wrong", TokenType::Emulated}).outcome, PutOutcome::PreconditionFailed); - expectBytes(b, "k", "v1"); // untouched on mismatch - const auto overwrite = b.putOverwrite("k", "v2", t1); - EXPECT_EQ(overwrite.outcome, PutOutcome::Done); - EXPECT_NE(overwrite.token, t1); // tokens never repeat + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + + const Etag t1 = std::get(op.create("k", "v1", Retry::once())).etag; + /// A stale precondition for the SAME key: an Etag is bound to the key it was minted for, so a + /// cross-key Etag is a caller bug (LOGICAL_ERROR), not "the wrong token" any more -- a stale + /// same-key incarnation is the real-world shape a precondition mismatch has to cover instead. + const Etag t2 = std::get(op.replace("k", "v1.5", t1, Retry::once())).etag; + EXPECT_TRUE(std::holds_alternative(op.replace("k", "v2", t1, Retry::once()))); + expectBytes(b, "k", "v1.5"); // untouched on mismatch + + const WriteResult overwrite = op.replace("k", "v2", t2, Retry::once()); + ASSERT_TRUE(std::holds_alternative(overwrite)); + EXPECT_NE(std::get(overwrite).etag, t2); // tokens never repeat expectBytes(b, "k", "v2"); } TEST(CASInMemory, CasPutCreateAndSwap) { InMemoryBackend b; - const auto create = b.casPut("m", "s1", std::nullopt); - const Token t1 = create.token; - EXPECT_EQ(create.outcome, CasOutcome::Committed); // create-if-absent - EXPECT_EQ(b.casPut("m", "s1x", std::nullopt).outcome, CasOutcome::Conflict); // exists now - EXPECT_EQ(b.casPut("m", "s2", Token{"stale", TokenType::Emulated}).outcome, CasOutcome::Conflict); - EXPECT_EQ(b.get("m")->bytes, "s1"); - EXPECT_EQ(b.casPut("m", "s2", t1).outcome, CasOutcome::Committed); - EXPECT_EQ(b.get("m")->bytes, "s2"); + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + + const WriteResult create = op.create("m", "s1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(create)); // create-if-absent + const Etag t1 = std::get(create).etag; + EXPECT_TRUE(std::holds_alternative(op.create("m", "s1x", Retry::once()))); // exists now + + /// A stale, same-key precondition -- see OverwriteIsTokenExactAndMintsFreshToken for why a + /// cross-key Etag can no longer stand in for "the wrong token". + const Etag t2 = std::get(op.replace("m", "s1.5", t1, Retry::once())).etag; + EXPECT_TRUE(std::holds_alternative(op.replace("m", "s2", t1, Retry::once()))); + EXPECT_EQ(op.read("m", Retry::once())->bytes, "s1.5"); + EXPECT_TRUE(std::holds_alternative(op.replace("m", "s2", t2, Retry::once()))); + EXPECT_EQ(op.read("m", Retry::once())->bytes, "s2"); } TEST(CASInMemory, DeleteExactEnforced) { InMemoryBackend b; - const Token t1 = b.putIfAbsent("k", "v1").token; - auto d1 = b.deleteExact("k", Token{"wrong", TokenType::Emulated}); - EXPECT_EQ(d1.kind, DeleteOutcome::Kind::TokenMismatch); - EXPECT_TRUE(b.get("k").has_value()); // SURVIVES wrong-token delete - auto d2 = b.deleteExact("k", t1); - EXPECT_EQ(d2.kind, DeleteOutcome::Kind::Deleted); - EXPECT_FALSE(d2.created_delete_marker); - EXPECT_FALSE(b.get("k").has_value()); - EXPECT_EQ(b.deleteExact("k", t1).kind, DeleteOutcome::Kind::NotFound); + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + + const Etag t0 = std::get(op.create("k", "v1", Retry::once())).etag; + const Etag t1 = std::get(op.replace("k", "v1b", t0, Retry::once())).etag; + /// t0 is now stale for this SAME key -- see OverwriteIsTokenExactAndMintsFreshToken for why a + /// cross-key Etag can no longer stand in for "the wrong token". + EXPECT_EQ(op.remove("k", t0, Retry::once()), Removal::Mismatch); + EXPECT_TRUE(op.read("k", Retry::once()).has_value()); // SURVIVES wrong-token delete + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Removed); + EXPECT_FALSE(op.read("k", Retry::once()).has_value()); + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Gone); } TEST(CASInMemory, GetAndHeadAndList) { InMemoryBackend b; - b.putIfAbsent("p/a", "0123456789"); - b.putIfAbsent("p/b", "xy"); - b.putIfAbsent("q/c", "z"); - EXPECT_EQ(b.get("p/a")->bytes, "0123456789"); - auto h = b.head("p/a"); - EXPECT_TRUE(h.exists); - EXPECT_EQ(h.size, 10u); - auto page = b.list("p/", "", 10); + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + + op.create("p/a", "0123456789", Retry::once()); + op.create("p/b", "xy", Retry::once()); + op.create("q/c", "z", Retry::once()); + EXPECT_EQ(op.read("p/a", Retry::once())->bytes, "0123456789"); + auto h = op.head("p/a", Retry::once()); + ASSERT_TRUE(h.has_value()); + EXPECT_EQ(h->size, 10u); + auto page = op.list("p/", "", 10, Retry::once()); ASSERT_EQ(page.keys.size(), 2u); // sorted, prefix-scoped EXPECT_EQ(page.keys[0].key, "p/a"); EXPECT_EQ(page.keys[1].key, "p/b"); EXPECT_TRUE(page.next_cursor.empty()); - auto page1 = b.list("p/", "", 1); // pagination + auto page1 = op.list("p/", "", 1, Retry::once()); // pagination EXPECT_EQ(page1.keys.size(), 1u); EXPECT_EQ(page1.keys[0].key, "p/a"); EXPECT_EQ(page1.next_cursor, "p/a"); EXPECT_FALSE(page1.next_cursor.empty()); - auto page2 = b.list("p/", page1.next_cursor, 1); + auto page2 = op.list("p/", page1.next_cursor, 1, Retry::once()); EXPECT_EQ(page2.keys[0].key, "p/b"); } TEST(CASInMemory, PublishBlobStreamingWritesFreshEnvelopeAndExactPayload) { InMemoryBackend backend; + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const auto request = streamingPublication("blob", "fresh-envelope", "payload", 7); - backend.publishBlob(request); + op.publish(request, Retry::once()); - const auto result = backend.get("blob"); + const auto result = op.read("blob", Retry::once()); ASSERT_TRUE(result.has_value()); EXPECT_EQ(result->bytes, "fresh-envelopepayload"); } @@ -336,8 +261,10 @@ TEST(CASInMemory, PublishBlobStreamingWritesFreshEnvelopeAndExactPayload) TEST(CASInMemory, PublishBlobRejectsShortAndLongStreamingSourcesWithoutVisibility) { InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent("short", "old-short").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent("long", "old-long").outcome, PutOutcome::Done); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("short", "old-short", Retry::once()))); + ASSERT_TRUE(std::holds_alternative(op.create("long", "old-long", Retry::once()))); for (const auto & [key, payload, declared_size] : std::vector>{ {"short", "abc", 4}, @@ -346,7 +273,7 @@ TEST(CASInMemory, PublishBlobRejectsShortAndLongStreamingSourcesWithoutVisibilit const auto request = streamingPublication(key, "fresh", payload, declared_size); try { - backend.publishBlob(request); + op.publish(request, Retry::once()); FAIL() << "expected a source-size mismatch for " << key; } catch (const DB::Exception & e) @@ -355,19 +282,21 @@ TEST(CASInMemory, PublishBlobRejectsShortAndLongStreamingSourcesWithoutVisibilit } } - EXPECT_EQ(backend.get("short")->bytes, "old-short"); - EXPECT_EQ(backend.get("long")->bytes, "old-long"); + EXPECT_EQ(op.read("short", Retry::once())->bytes, "old-short"); + EXPECT_EQ(op.read("long", Retry::once())->bytes, "old-long"); } TEST(CASInMemory, PublishBlobLongSourceReadsOnlyDeclaredPayloadAndOneProbeByte) { InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent("long", "old-complete-body").outcome, PutOutcome::Done); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("long", "old-complete-body", Retry::once()))); auto state = std::make_shared(); try { - backend.publishBlob(countedLongPublication("long", "fresh", 3, 1024, state)); + op.publish(countedLongPublication("long", "fresh", 3, 1024, state), Retry::once()); FAIL() << "expected a long-source mismatch"; } catch (const DB::Exception & e) @@ -376,8 +305,9 @@ TEST(CASInMemory, PublishBlobLongSourceReadsOnlyDeclaredPayloadAndOneProbeByte) } EXPECT_EQ(state->bytes_exposed, 4u); - ASSERT_TRUE(backend.get("long").has_value()); - EXPECT_EQ(backend.get("long")->bytes, "old-complete-body"); + const auto still_present = op.read("long", Retry::once()); + ASSERT_TRUE(still_present.has_value()); + EXPECT_EQ(still_present->bytes, "old-complete-body"); } TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteBodyIsReady) @@ -385,7 +315,9 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB using namespace std::chrono_literals; InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent("blob", "old-complete-body").outcome, PutOutcome::Done); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("blob", "old-complete-body", Retry::once()))); std::promise source_opened; std::promise release_source; @@ -402,10 +334,10 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB return std::make_unique(String("payload")); }}}; - auto publication = std::async(std::launch::async, [&] { backend.publishBlob(request); }); + auto publication = std::async(std::launch::async, [&] { op.publish(request, Retry::once()); }); source_opened.get_future().wait(); - auto observation = std::async(std::launch::async, [&] { return backend.get("blob"); }); + auto observation = std::async(std::launch::async, [&] { return op.read("blob", Retry::once()); }); const auto observation_status = observation.wait_for(2s); EXPECT_EQ(observation_status, std::future_status::ready) << "publication must not hold the visibility lock while draining its source"; @@ -418,24 +350,28 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB release_source.set_value(); EXPECT_NO_THROW(publication.get()); - ASSERT_TRUE(backend.get("blob").has_value()); - EXPECT_EQ(backend.get("blob")->bytes, "fresh-envelopepayload"); + const auto after = op.read("blob", Retry::once()); + ASSERT_TRUE(after.has_value()); + EXPECT_EQ(after->bytes, "fresh-envelopepayload"); } TEST(CASInMemory, PublishBlobCopiesStagedObjectBytesVerbatim) { InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent("stage", "staged-envelopepayload").outcome, PutOutcome::Done); - ASSERT_EQ(backend.putIfAbsent("blob", "old-body").outcome, PutOutcome::Done); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("stage", "staged-envelopepayload", Retry::once()))); + ASSERT_TRUE(std::holds_alternative(op.create("blob", "old-body", Retry::once()))); - backend.publishBlob(BlobPublishRequest{ + op.publish(BlobPublishRequest{ .destination_key = "blob", .publication = VerbatimStagedBlobPublication{ .object_key = "stage", - .object_size = 22}}); + .object_size = 22}}, Retry::once()); - ASSERT_TRUE(backend.get("blob").has_value()); - EXPECT_EQ(backend.get("blob")->bytes, "staged-envelopepayload"); + const auto after = op.read("blob", Retry::once()); + ASSERT_TRUE(after.has_value()); + EXPECT_EQ(after->bytes, "staged-envelopepayload"); } // ===================================================================== @@ -445,62 +381,78 @@ TEST(CASInMemory, PublishBlobCopiesStagedObjectBytesVerbatim) TEST(CASInMemoryFaults, HeldDeleteLandsLater) { InMemoryBackend b; - const Token t1 = b.putIfAbsent("k", "v1").token; + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + const Etag t1 = std::get(op.create("k", "v1", Retry::once())).etag; b.setHoldDeletes(true); - auto d = b.deleteExact("k", t1); // message "sent", not landed - EXPECT_EQ(d.kind, DeleteOutcome::Kind::Deleted); // caller sees the send accepted - EXPECT_TRUE(b.get("k").has_value()); // ... but nothing landed yet + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Removed); // message "sent", not landed + EXPECT_TRUE(op.read("k", Retry::once()).has_value()); // ... but nothing landed yet ASSERT_EQ(b.pendingDeletes(), 1u); // the object is recreated before the zombie lands: - b.putOverwrite("k", "v1'", t1); + op.replace("k", "v1'", t1, Retry::once()); auto landed = b.landPendingDelete(0); // the zombie lands NOW - EXPECT_EQ(landed.kind, DeleteOutcome::Kind::TokenMismatch); // 412 — INV-NO-RETURN in miniature + EXPECT_EQ(landed, DB::Cas::Backend::RawRemoval::Mismatch); // 412 — INV-NO-RETURN in miniature expectBytes(b, "k", "v1'"); } TEST(CASInMemoryFaults, InjectedCasConflictFiresOnce) { InMemoryBackend b; - const Token t1 = b.casPut("m", "s1", std::nullopt).token; + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + const Etag t1 = std::get(op.create("m", "s1", Retry::once())).etag; b.refuseNextWrite("m"); - EXPECT_EQ(b.casPut("m", "s2", t1).outcome, CasOutcome::Conflict); // injected - EXPECT_EQ(b.get("m")->bytes, "s1"); - EXPECT_EQ(b.casPut("m", "s2", t1).outcome, CasOutcome::Committed); // next attempt is real + EXPECT_TRUE(std::holds_alternative(op.replace("m", "s2", t1, Retry::once()))); // injected + EXPECT_EQ(op.read("m", Retry::once())->bytes, "s1"); + EXPECT_TRUE(std::holds_alternative(op.replace("m", "s2", t1, Retry::once()))); // next attempt is real } TEST(CASInMemoryFaults, NonEnforcingModeMimicsBadBackend) { InMemoryBackend b; + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); b.setEnforceTokens(false); // MinIO-OSS-shaped backend - b.putIfAbsent("k", "v1"); - auto d = b.deleteExact("k", Token{"totally-wrong", TokenType::Emulated}); - EXPECT_EQ(d.kind, DeleteOutcome::Kind::Deleted); // silently deletes anyway — the dangerous behavior - EXPECT_FALSE(b.get("k").has_value()); + const Etag t0 = std::get(op.create("k", "v1", Retry::once())).etag; + ASSERT_TRUE(std::holds_alternative(op.replace("k", "v2", t0, Retry::once()))); // mints a later incarnation + EXPECT_EQ(op.remove("k", t0, Retry::once()), Removal::Removed); // stale-but-same-key precondition silently deletes anyway — the dangerous behavior + EXPECT_FALSE(op.read("k", Retry::once()).has_value()); } TEST(CASInMemoryFaults, VersioningMarkerMode) { InMemoryBackend b; b.setSimulateDeleteMarkers(true); - const Token t1 = b.putIfAbsent("k", "v1").token; - EXPECT_TRUE(b.deleteExact("k", t1).created_delete_marker); // probe must reject this pool + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + const Etag t1 = std::get(op.create("k", "v1", Retry::once())).etag; + /// A removal that only archives (never reclaims) is not an ordinary Removed: the engine reports it + /// as CAS_DELETE_MARKER so the capability probe can reject a versioned pool. + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CAS_DELETE_MARKER, [&] { op.remove("k", t1, Retry::once()); }); } // ===================================================================== -// getStream seam (forward-only reads of write-once objects) +// stream seam (forward-only reads of write-once objects) // ===================================================================== -TEST(CASBackendStream, StreamsBodyWindow) +/// The legacy getStream's byte-range window is retired along with it: the primitive `stream` takes no +/// Range, and every consumer (RunFileReader) already bounds its own consumption client-side rather than +/// relying on a server-side window. What survives here is presence: a present key opens a readable +/// stream, an absent one opens none. +TEST(CASBackendStream, StreamsWholeBodyOrNullWhenAbsent) { auto backend = std::make_shared(); - backend->putIfAbsent("k", "0123456789"); - auto got = backend->getStream("k", DB::Cas::Range{.offset = 2, .length = 5}); - ASSERT_TRUE(got.has_value()); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + op.create("k", "0123456789", Retry::once()); + + auto got = op.stream("k", Retry::once()); + ASSERT_TRUE(got != nullptr); String out; - DB::readStringUntilEOF(out, *got->stream); - EXPECT_EQ(out, "23456"); - EXPECT_FALSE(got->token.empty()); - EXPECT_FALSE(backend->getStream("absent").has_value()); + DB::readStringUntilEOF(out, *got); + EXPECT_EQ(out, "0123456789"); + + EXPECT_EQ(op.stream("absent", Retry::once()), nullptr); } // ===================================================================== @@ -513,7 +465,7 @@ extern const Event CASBlobPut; extern const Event CASBlobPutDeduplicated; extern const Event CASBlobHead; extern const Event CASBlobHeadMiss; -extern const Event CASGCCompareSwap; +extern const Event CASGCPut; } TEST(CASInstrumentedBackend, ClassifierAndPerNamespaceOpEvents) @@ -538,27 +490,29 @@ TEST(CASInstrumentedBackend, ClassifierAndPerNamespaceOpEvents) EXPECT_EQ(classifyCasNs("pool/cas/manifests/0/srv/store/d18/uuid@cas@/24/1/000001.proto"), CasNs::Manifest); auto inner = std::make_shared(); - InstrumentedBackend b(inner); + auto instrumented = std::make_shared(inner); + CasRequests requests = openRequestsForTest(instrumented); + CasOperation op = requests.admit(); using ProfileEvents::global_counters; const auto blob_put_before = global_counters[ProfileEvents::CASBlobPut].load(); const auto blob_dedup_before = global_counters[ProfileEvents::CASBlobPutDeduplicated].load(); const auto blob_head_before = global_counters[ProfileEvents::CASBlobHead].load(); const auto blob_miss_before = global_counters[ProfileEvents::CASBlobHeadMiss].load(); - const auto gc_cas_before = global_counters[ProfileEvents::CASGCCompareSwap].load(); + const auto gc_put_before = global_counters[ProfileEvents::CASGCPut].load(); const String blob_key = "pool/blobs/ab/abcdef0123456789"; - /// First put of a blob ⇒ Put. - EXPECT_EQ(b.putIfAbsent(blob_key, "payload").outcome, PutOutcome::Done); - /// Second put of the same key ⇒ PutDeduplicated (content already exists). - EXPECT_EQ(b.putIfAbsent(blob_key, "payload").outcome, PutOutcome::PreconditionFailed); + /// First create of a blob ⇒ Put. + EXPECT_TRUE(std::holds_alternative(op.create(blob_key, "payload", Retry::once()))); + /// Second create of the same key ⇒ PutDeduplicated (content already exists). + EXPECT_TRUE(std::holds_alternative(op.create(blob_key, "payload", Retry::once()))); /// head of an absent blob key ⇒ HeadMiss (the 404 signal). - EXPECT_FALSE(b.head("pool/blobs/zz/absent").exists); + EXPECT_FALSE(op.head("pool/blobs/zz/absent", Retry::once()).has_value()); /// head of the present blob key ⇒ Head. - EXPECT_TRUE(b.head(blob_key).exists); - /// casPut create on a gc key ⇒ Gc Cas. - EXPECT_EQ(b.casPut("pool/gc/state", "g1", std::nullopt).outcome, CasOutcome::Committed); + EXPECT_TRUE(op.head(blob_key, Retry::once()).has_value()); + /// create on a gc key ⇒ Gc Put. + EXPECT_TRUE(std::holds_alternative(op.create("pool/gc/state", "g1", Retry::once()))); /// Under coverage builds ProfileEvents propagate into a thread-local subtree that does not reach /// `global_counters`; deltas read 0 there only (see gtest_unique_key_index_cache). #if !WITH_COVERAGE @@ -566,27 +520,32 @@ TEST(CASInstrumentedBackend, ClassifierAndPerNamespaceOpEvents) EXPECT_EQ(global_counters[ProfileEvents::CASBlobPutDeduplicated].load() - blob_dedup_before, 1u); EXPECT_EQ(global_counters[ProfileEvents::CASBlobHead].load() - blob_head_before, 1u); EXPECT_EQ(global_counters[ProfileEvents::CASBlobHeadMiss].load() - blob_miss_before, 1u); - EXPECT_EQ(global_counters[ProfileEvents::CASGCCompareSwap].load() - gc_cas_before, 1u); + EXPECT_EQ(global_counters[ProfileEvents::CASGCPut].load() - gc_put_before, 1u); #else (void)blob_put_before; (void)blob_dedup_before; (void)blob_head_before; - (void)blob_miss_before; (void)gc_cas_before; + (void)blob_miss_before; (void)gc_put_before; #endif } TEST(CASInstrumentedBackend, PublishBlobDelegatesOnceAndRecordsOnePhysicalBlobWrite) { auto inner = std::make_shared(); - InstrumentedBackend backend(inner); + auto backend = std::make_shared(inner); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); using ProfileEvents::global_counters; const auto blob_put_before = global_counters[ProfileEvents::CASBlobPut].load(); const auto request = streamingPublication("pool/blobs/ab/published", "fresh", "payload", 7); - backend.publishBlob(request); + op.publish(request, Retry::once()); EXPECT_EQ(inner->publish_calls, 1u); - ASSERT_TRUE(inner->get("pool/blobs/ab/published").has_value()); - EXPECT_EQ(inner->get("pool/blobs/ab/published")->bytes, "freshpayload"); + CasRequests inner_requests = openRequestsForTest(inner); + CasOperation inner_op = inner_requests.admit(); + const auto published = inner_op.read("pool/blobs/ab/published", Retry::once()); + ASSERT_TRUE(published.has_value()); + EXPECT_EQ(published->bytes, "freshpayload"); #if !WITH_COVERAGE EXPECT_EQ(global_counters[ProfileEvents::CASBlobPut].load() - blob_put_before, 1u); #else @@ -602,17 +561,17 @@ TEST(CASInstrumentedBackend, PublishBlobDelegatesOnceAndRecordsOnePhysicalBlobWr TEST(CASBackendGrammar, GenerationDialectAcceptsOnlyCanonicalPositiveDecimal) { using DB::Cas::ObjectStorageBackend; - using DB::Cas::TokenType; - EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "123")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "0")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "00123")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "\"123\"")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Generation, "12a")); - EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(TokenType::ETag, "\"abc\"")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::ETag, " * ")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::ETag, "a,b")); - EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(TokenType::Emulated, "7")); - EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(TokenType::Emulated, "")); + using DB::Cas::Dialect; + EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(Dialect::Generation, "123")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::Generation, "0")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::Generation, "00123")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::Generation, "\"123\"")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::Generation, "12a")); + EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(Dialect::ETag, "\"abc\"")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::ETag, " * ")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::ETag, "a,b")); + EXPECT_TRUE(ObjectStorageBackend::isValidTokenValue(Dialect::Emulated, "7")); + EXPECT_FALSE(ObjectStorageBackend::isValidTokenValue(Dialect::Emulated, "")); } /// §1 (opt round-B): the fold/point GETs read tiny bodies but a default `ReadBufferFromS3` preallocates @@ -639,22 +598,24 @@ TEST(CASSizedReadSettings, CapsToKnownSizePlusSlackButNeverAboveBase) } /// The CountingBackend recorders the streaming-memory gates consume: per-key and total stream counts. -/// A window is no longer part of the shape -- a materialized read is always whole, so only `getStream` -/// still carries one, and it is not what the gates measure. +/// A window is no longer part of the shape -- a materialized read is always whole, so `stream` no +/// longer carries one either, and it is not what the gates measure. TEST(CASCountingBackendShape, RecordsStreamOpensPerKeyAndInTotal) { - DB::Cas::tests::CountingBackend backend; - backend.putIfAbsent("k", String(1000, 'x')); + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + op.create("k", String(1000, 'x'), Retry::once()); - backend.getStream("k", DB::Cas::Range{.offset = 2, .length = 5}); - backend.getStream("k"); - backend.getStream("absent"); - EXPECT_EQ(backend.getStreamCount("k"), 2u); - EXPECT_EQ(backend.getStreamTotal(), 3u); + op.stream("k", Retry::once()); + op.stream("k", Retry::once()); + op.stream("absent", Retry::once()); + EXPECT_EQ(backend->getStreamCount("k"), 2u); + EXPECT_EQ(backend->getStreamTotal(), 3u); - backend.resetCounts(); - EXPECT_EQ(backend.getStreamCount("k"), 0u); - EXPECT_EQ(backend.getStreamTotal(), 0u); + backend->resetCounts(); + EXPECT_EQ(backend->getStreamCount("k"), 0u); + EXPECT_EQ(backend->getStreamTotal(), 0u); } /// Armed chunking makes this backend serve a stream the way a network-backed store does, in bounded @@ -665,25 +626,27 @@ TEST(CASCountingBackendShape, AnArmedChunkBoundsTheWindowAStreamHandsOut) { const String body(10'000, 'x'); auto backend = std::make_shared(); - ASSERT_EQ(backend->putIfAbsent("run", body).outcome, PutOutcome::Done); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("run", body, Retry::once()))); /// Unarmed: the whole object arrives as one window, which is what this backend's materialization /// makes of any stream and exactly what the bound exists to remove. { - auto opened = backend->getStream("run"); - ASSERT_TRUE(opened); + auto opened = op.stream("run", Retry::once()); + ASSERT_TRUE(opened != nullptr); String drained; - readStringUntilEOF(drained, *opened->stream); + readStringUntilEOF(drained, *opened); EXPECT_EQ(drained, body); EXPECT_EQ(backend->largestStreamChunk("run"), 0u) << "nothing records a window while chunking is off"; } backend->setStreamChunkForTest(4096); { - auto opened = backend->getStream("run"); - ASSERT_TRUE(opened); + auto opened = op.stream("run", Retry::once()); + ASSERT_TRUE(opened != nullptr); String drained; - readStringUntilEOF(drained, *opened->stream); + readStringUntilEOF(drained, *opened); EXPECT_EQ(drained, body) << "chunking changes the window, never the bytes"; EXPECT_EQ(backend->largestStreamChunk("run"), 4096u); EXPECT_LT(backend->largestStreamChunk("run"), body.size()) @@ -693,47 +656,47 @@ TEST(CASCountingBackendShape, AnArmedChunkBoundsTheWindowAStreamHandsOut) /// The mode outlives a counter reset, and the recorded window does not. backend->resetCounts(); EXPECT_EQ(backend->largestStreamChunk("run"), 0u); - auto reopened = backend->getStream("run"); - ASSERT_TRUE(reopened); + auto reopened = op.stream("run", Retry::once()); + ASSERT_TRUE(reopened != nullptr); String again; - readStringUntilEOF(again, *reopened->stream); + readStringUntilEOF(again, *reopened); EXPECT_EQ(backend->largestStreamChunk("run"), 4096u); } -/// What makes every request-profile gate in this tree trustworthy: a counter names a PHYSICAL request, -/// so the same request counts once whichever surface issued it. Before the counters moved onto the -/// transport primitives a legacy call and a `CasOperation` call landed on different counters, and a -/// gate written against one was blind to the other. +/// What makes every request-profile gate in this tree trustworthy: a counter names a PHYSICAL request. +/// The transport primitives are the only surface left that can issue one, so this now pins that a +/// create/head/read/replace/remove issued through `CasOperation` counts exactly once each. TEST(CASCountingBackendShape, OneRequestIsCountedOnceWhicheverSurfaceIssuedIt) { auto backend = std::make_shared(); DB::Cas::CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); DB::Cas::CasOperation op = requests.admit(); - EXPECT_EQ(backend->putIfAbsent("k", "v").outcome, PutOutcome::Done); /// legacy create - EXPECT_TRUE(std::holds_alternative( - op.create("k2", "v", Retry::standard()))); /// the same request, admitted + EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::standard()))); + EXPECT_TRUE(std::holds_alternative(op.create("k2", "v", Retry::standard()))); EXPECT_EQ(backend->putCount("k"), 1u); EXPECT_EQ(backend->putCount("k2"), 1u); EXPECT_EQ(backend->writeTotal(), 2u); EXPECT_EQ(backend->putOverwriteTotal(), 0u) << "neither write carried a precondition"; - const Token seen = backend->head("k").token; /// legacy head - EXPECT_TRUE(op.head("k", Retry::standard())); /// admitted head - EXPECT_EQ(backend->headCount("k"), 2u); + const std::optional k_meta = op.head("k", Retry::standard()); + ASSERT_TRUE(k_meta); + EXPECT_EQ(backend->headCount("k"), 1u); - expectBytes(*backend, "k", "v"); /// legacy read - EXPECT_TRUE(op.read("k", Retry::standard())); /// admitted read + expectBytes(*backend, "k", "v"); + EXPECT_TRUE(op.read("k", Retry::standard())); EXPECT_EQ(backend->getCount("k"), 2u); - EXPECT_EQ(backend->putOverwrite("k", "w", seen).outcome, PutOutcome::Done); + EXPECT_TRUE(std::holds_alternative(op.replace("k", "w", k_meta->etag, Retry::standard()))); EXPECT_EQ(backend->putOverwriteCount("k"), 1u) << "a write with a precondition is the replace shape"; EXPECT_EQ(backend->writeCount("k"), 2u); const std::optional k2_meta = op.head("k2", Retry::standard()); ASSERT_TRUE(k2_meta); - EXPECT_EQ(op.remove("k2", k2_meta->incarnation, Retry::standard()), Removal::Removed); - EXPECT_EQ(backend->deleteExact("k", backend->head("k").token).kind, DeleteOutcome::Kind::Deleted); + EXPECT_EQ(op.remove("k2", k2_meta->etag, Retry::standard()), Removal::Removed); + const std::optional k_meta_after = op.head("k", Retry::standard()); + ASSERT_TRUE(k_meta_after); + EXPECT_EQ(op.remove("k", k_meta_after->etag, Retry::standard()), Removal::Removed); EXPECT_EQ(backend->deleteCount("k"), 1u); EXPECT_EQ(backend->deleteCount("k2"), 1u); EXPECT_EQ(backend->deleteTotal(), 2u); @@ -914,12 +877,14 @@ String readStorageObject(const DB::ObjectStoragePtr & storage, const String & ke TEST(CASObjectStorageBackend, PublishBlobStreamingUsesOrdinaryDefaultWriteTransport) { auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - backend.setNativeTokenTypeForTest(TokenType::Generation); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::Native); + backend->setNativeTokenTypeForTest(Dialect::Generation); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String destination = DB::Cas::tests::nativeKeyUnder(storage, "publish/streaming"); const auto request = streamingPublication(destination, "fresh-envelope", "payload", 7); - backend.publishBlob(request); + op.publish(request, Retry::once()); ASSERT_EQ(storage->write_calls, 1u); ASSERT_TRUE(storage->last_write_mode.has_value()); @@ -941,7 +906,9 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedKeepsDestinationCompleteUntilAt using namespace std::chrono_literals; auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String key = "publish/emulated-atomic"; const String physical_key = DB::Cas::tests::nativeKeyUnder(storage, key); @@ -958,7 +925,7 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedKeepsDestinationCompleteUntilAt auto opened = barrier->opened.get_future(); auto publication = std::async(std::launch::async, [&] { - backend.publishBlob(streamingPublication(key, "fresh-envelope", "payload", 7)); + op.publish(streamingPublication(key, "fresh-envelope", "payload", 7), Retry::once()); }); const auto opened_status = opened.wait_for(2s); @@ -975,7 +942,9 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedKeepsDestinationCompleteUntilAt TEST(CASObjectStorageBackend, PublishBlobEmulatedWriteFailurePreservesDestinationAndCleansTemporary) { auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String key = "publish/emulated-failure"; const String physical_key = DB::Cas::tests::nativeKeyUnder(storage, key); @@ -985,24 +954,26 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedWriteFailurePreservesDestinatio DB::writeString(String("old-complete-body"), *out); out->finalize(); } - const Token old_token = backend.head(key).token; + const Etag old_token = op.head(key, Retry::once())->etag; storage->throw_after_open = true; EXPECT_THROW( - backend.publishBlob(streamingPublication(key, "fresh-envelope", "payload", 7)), + op.publish(streamingPublication(key, "fresh-envelope", "payload", 7), Retry::once()), std::runtime_error); storage->throw_after_open = false; EXPECT_NE(storage->last_opened_key, physical_key); EXPECT_FALSE(storage->exists(DB::StoredObject(storage->last_opened_key))); EXPECT_EQ(readStorageObject(storage, physical_key), "old-complete-body"); - EXPECT_EQ(backend.head(key).token, old_token); + EXPECT_EQ(op.head(key, Retry::once())->etag, old_token); } TEST(CASObjectStorageBackend, PublishBlobCancelsShortAndLongStreamingSourcesBeforeVisibility) { auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::Native); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String destination = DB::Cas::tests::nativeKeyUnder(storage, "publish/mismatch"); { @@ -1016,7 +987,7 @@ TEST(CASObjectStorageBackend, PublishBlobCancelsShortAndLongStreamingSourcesBefo DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - backend.publishBlob(streamingPublication(destination, "fresh", "abc", 4)); + op.publish(streamingPublication(destination, "fresh", "abc", 4), Retry::once()); }); EXPECT_EQ(storage->cancel_calls, 1u); EXPECT_EQ(storage->finalize_calls, 0u); @@ -1026,7 +997,7 @@ TEST(CASObjectStorageBackend, PublishBlobCancelsShortAndLongStreamingSourcesBefo auto state = std::make_shared(); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - backend.publishBlob(countedLongPublication(destination, "fresh", 3, 1024, state)); + op.publish(countedLongPublication(destination, "fresh", 3, 1024, state), Retry::once()); }); EXPECT_EQ(state->bytes_exposed, 4u); EXPECT_EQ(storage->cancel_calls, 2u); @@ -1038,7 +1009,9 @@ TEST(CASObjectStorageBackend, PublishBlobCancelsShortAndLongStreamingSourcesBefo TEST(CASObjectStorageBackend, PublishBlobEmulatedLongSourceReadsOnlyDeclaredPayloadAndOneProbeByte) { auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String key = "publish/emulated-long"; const String physical_key = DB::Cas::tests::nativeKeyUnder(storage, key); @@ -1052,7 +1025,7 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedLongSourceReadsOnlyDeclaredPayl auto state = std::make_shared(); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { - backend.publishBlob(countedLongPublication(key, "fresh", 3, 1024, state)); + op.publish(countedLongPublication(key, "fresh", 3, 1024, state), Retry::once()); }); EXPECT_EQ(state->bytes_exposed, 4u); @@ -1062,7 +1035,9 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedLongSourceReadsOnlyDeclaredPayl TEST(CASObjectStorageBackend, PublishBlobCopiesStagedBytesWithNativeOnlyDefaultRequestMode) { auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::Native); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String staging = DB::Cas::tests::nativeKeyUnder(storage, "publish/staging"); const String destination = DB::Cas::tests::nativeKeyUnder(storage, "publish/copied"); @@ -1074,11 +1049,11 @@ TEST(CASObjectStorageBackend, PublishBlobCopiesStagedBytesWithNativeOnlyDefaultR } storage->resetRecording(); - backend.publishBlob(BlobPublishRequest{ + op.publish(BlobPublishRequest{ .destination_key = destination, .publication = VerbatimStagedBlobPublication{ .object_key = staging, - .object_size = 22}}); + .object_size = 22}}, Retry::once()); ASSERT_EQ(storage->copy_calls, 1u); ASSERT_TRUE(storage->last_copy_settings.has_value()); @@ -1093,7 +1068,9 @@ TEST(CASObjectStorageBackend, PublishBlobCopiesStagedBytesWithNativeOnlyDefaultR TEST(CASObjectStorageBackend, PublishBlobRefusesVerbatimCopyWithoutNativeTransport) { auto storage = makePublicationRecordingStorage(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::Native); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String staging = DB::Cas::tests::nativeKeyUnder(storage, "publish/unsupported-staging"); const String destination = DB::Cas::tests::nativeKeyUnder(storage, "publish/unsupported-copy"); @@ -1108,11 +1085,11 @@ TEST(CASObjectStorageBackend, PublishBlobRefusesVerbatimCopyWithoutNativeTranspo DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NOT_IMPLEMENTED, [&] { - backend.publishBlob(BlobPublishRequest{ + op.publish(BlobPublishRequest{ .destination_key = destination, .publication = VerbatimStagedBlobPublication{ .object_key = staging, - .object_size = 22}}); + .object_size = 22}}, Retry::once()); }); EXPECT_EQ(storage->copy_calls, 0u); @@ -1170,17 +1147,17 @@ TEST(CASS3Signal, FinalizeClassifierMapsPreconditionLossExactly) }; EXPECT_EQ(classify(DB::S3Exception("412", Aws::S3::S3Errors::UNKNOWN, "PreconditionFailed")), - PutOutcome::PreconditionFailed); + DB::Cas::detail::ConditionalWriteOutcome::PreconditionLost); EXPECT_EQ(classify(DB::S3Exception("404 gone under If-Match", Aws::S3::S3Errors::UNKNOWN, "NoSuchKey")), - PutOutcome::PreconditionFailed); + DB::Cas::detail::ConditionalWriteOutcome::PreconditionLost); EXPECT_EQ(classify(DB::S3Exception("retries exhausted, no name attached", Aws::S3::S3Errors::NO_SUCH_KEY)), - PutOutcome::PreconditionFailed); + DB::Cas::detail::ConditionalWriteOutcome::PreconditionLost); ThrowOnFinalizeBuffer unrelated(DB::S3Exception("503", Aws::S3::S3Errors::UNKNOWN, "SlowDown")); EXPECT_THROW(finalizeConditionalWrite(unrelated), DB::S3Exception); ThrowOnFinalizeBuffer clean; - EXPECT_EQ(finalizeConditionalWrite(clean), PutOutcome::Done); + EXPECT_EQ(finalizeConditionalWrite(clean), DB::Cas::detail::ConditionalWriteOutcome::Applied); } namespace @@ -1264,18 +1241,17 @@ TEST(CASObjectStorageBackend, NativeModeGetReturnsNulloptOnMidGetNoSuchKey) /// logical key IS the physical one the fixture wrote and armed. const auto fixture = makeThrowOnReadStorageForTest("pool/blobs/ab/abcdef0123456789abcdef0123456789"); - ObjectStorageBackend backend(fixture.storage, ObjectStorageBackend::Mode::Native); + auto backend = std::make_shared(fixture.storage, ObjectStorageBackend::Mode::Native); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); - /// `get` HEADs before it reads and answers nullopt for an absent key, so without this the nullopt - /// below would be satisfied by an object the fixture failed to place — the mid-GET race would go - /// untested and the case would still pass. - Backend & iface = backend; - ASSERT_TRUE(iface.head(fixture.key).exists); + /// `head` answers present for a key the fixture failed to place, so without this the nullopt below + /// would be satisfied vacuously — the mid-read race would go untested and the case would still pass. + ASSERT_TRUE(op.head(fixture.key, Retry::once()).has_value()); /// HEAD reports the key present; readObject then throws NO_SUCH_KEY. - /// Contract: get must return std::nullopt, not propagate the S3Exception. - /// Call through the base-class interface so the default `Range{}` arg is available. - const auto result = iface.get(fixture.key); + /// Contract: read must return std::nullopt, not propagate the S3Exception. + const auto result = op.read(fixture.key, Retry::once()); EXPECT_FALSE(result.has_value()); } @@ -1286,59 +1262,109 @@ TEST(CASObjectStorageBackend, NativeModeGetReturnsNulloptOnMidGetNoSuchKey) /// value that TEXTUALLY collides with a token persisted before the restart (e.g. a GC condemned-delete /// token queued for replay), even though the two values name completely different incarnations of the /// key. `deleteExact` must never let a stale, pre-restart token match a freshly recreated object. +#ifndef DEBUG_OR_SANITIZER_BUILD TEST(CASObjectStorageBackend, EmuTokenSurvivesProcessRestartAcrossRecreate) { auto storage = tests::makeLocalObjectStorageForTest(); auto backend1 = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests1 = openRequestsForTest(backend1); + CasOperation op1 = requests1.admit(); /// A throwaway prior mutation on a DIFFERENT key: with the old counter this advances backend1's /// process-wide op counter to 1, so "k/restart"'s own mint below lands on 2 — chosen so it collides /// with backend2's post-restart recreate mint further down (also its SECOND op; see there). - ASSERT_EQ(backend1->putIfAbsent("k/other", "junk").outcome, PutOutcome::Done); - ASSERT_EQ(backend1->putIfAbsent("k/restart", "v1").outcome, PutOutcome::Done); - const Token stale_token = backend1->head("k/restart").token; + ASSERT_TRUE(std::holds_alternative(op1.create("k/other", "junk", Retry::once()))); + const WriteResult restart_create = op1.create("k/restart", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(restart_create)); + const Etag stale_token = std::get(restart_create).etag; /// Simulate a process restart: a brand-new `ObjectStorageBackend` instance (fresh emu state) over /// the SAME underlying storage — exactly what happens when the CAS process restarts. auto backend2 = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests2 = openRequestsForTest(backend2); + CasOperation op2 = requests2.admit(); /// Delete and recreate the key through the NEW instance — a fresh incarnation with a fresh mtime. /// This is backend2's first-ever op (op 1) then a delete (no mint) then the recreate (op 2) — the /// same op-index as `stale_token` above under the old counter, so the two textually collide there. - const Token current = backend2->head("k/restart").token; - ASSERT_EQ(backend2->deleteExact("k/restart", current).kind, DeleteOutcome::Kind::Deleted); - ASSERT_EQ(backend2->putIfAbsent("k/restart", "v2-after-restart").outcome, PutOutcome::Done); - - /// The pre-restart token must NEVER match the post-restart incarnation, however coincidentally a - /// process-local counter would have re-minted the identical textual value. - const auto stale_delete = backend2->deleteExact("k/restart", stale_token); - EXPECT_EQ(stale_delete.kind, DeleteOutcome::Kind::TokenMismatch); + const auto current = op2.head("k/restart", Retry::once()); + ASSERT_TRUE(current.has_value()); + ASSERT_EQ(op2.remove("k/restart", current->etag, Retry::once()), Removal::Removed); + ASSERT_TRUE(std::holds_alternative(op2.create("k/restart", "v2-after-restart", Retry::once()))); + + /// The pre-restart incarnation must NEVER be usable as a precondition against the post-restart + /// backend instance, however coincidentally a process-local counter would have re-minted the + /// identical textual value: an `Etag` carries the identity of the backend that observed it, and the + /// engine refuses one minted elsewhere before it ever reaches the store (LOGICAL_ERROR), which is a + /// STRONGER guarantee than the old bare-value comparison this test used to pin. The underlying + /// same-instance mtime-quantum disambiguation this fixture was ALSO probing is covered directly by + /// `EmuTokenDisambiguatesSameEtagRewrite`, within one backend instance where the engine's own + /// cross-backend check cannot pre-empt it. A LOGICAL_ERROR aborts under + /// DEBUG_OR_SANITIZER_BUILD before it can ever be thrown and caught here; the debug/sanitizer arm + /// of this split (below) pins the same refusal via EXPECT_DEATH instead. + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] + { + op2.remove("k/restart", stale_token, Retry::once()); + }); /// The live (post-restart) incarnation must be untouched by the rejected stale delete. - EXPECT_TRUE(backend2->head("k/restart").exists); + EXPECT_TRUE(op2.head("k/restart", Retry::once()).has_value()); +} +#endif + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASObjectStorageBackendDeathTest, EmuTokenSurvivesProcessRestartAcrossRecreateAborts) +{ + auto storage = tests::makeLocalObjectStorageForTest(); + + auto backend1 = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests1 = openRequestsForTest(backend1); + CasOperation op1 = requests1.admit(); + ASSERT_TRUE(std::holds_alternative(op1.create("k/other", "junk", Retry::once()))); + const WriteResult restart_create = op1.create("k/restart", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(restart_create)); + const Etag stale_token = std::get(restart_create).etag; + + auto backend2 = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests2 = openRequestsForTest(backend2); + CasOperation op2 = requests2.admit(); + + const auto current = op2.head("k/restart", Retry::once()); + ASSERT_TRUE(current.has_value()); + ASSERT_EQ(op2.remove("k/restart", current->etag, Retry::once()), Removal::Removed); + ASSERT_TRUE(std::holds_alternative(op2.create("k/restart", "v2-after-restart", Retry::once()))); + + /// See EmuTokenSurvivesProcessRestartAcrossRecreate above for the property under test; a + /// LOGICAL_ERROR aborts the process under DEBUG_OR_SANITIZER_BUILD, so this arm pins the refusal + /// via EXPECT_DEATH instead of an exception. + EXPECT_DEATH( + { op2.remove("k/restart", stale_token, Retry::once()); }, + "cannot be the precondition for"); } +#endif -/// codex-review-triage §3.18, finding №18: `list`'s `EmulatedSingleProcess` branch minted its per-key -/// token via `tokenForList`, which always stamps `native_token_type` (ETag) REGARDLESS of `mode` -- -/// while `head`/`get` mint `TokenType::Emulated`. `Token::operator==` compares type AND value, so a -/// list-derived token could never satisfy an emulated `deleteExact`/`putOverwrite` expectation: a -/// fail-safe leak (never a wrong delete), but every consumer of listed tokens (GC namespace cleanup, -/// `deletePrefixWholesale`, orphan sweep, decommission drain) always saw `TokenMismatch` against a -/// LOCAL pool. `list` must surface the SAME (type, value) as `head` for the same key. +/// `list`'s `EmulatedSingleProcess` branch must surface the SAME incarnation value `head` would for the +/// same key. An earlier defect minted the listed value under the wrong dialect regardless of `mode`, +/// so a list-derived value could never satisfy an emulated `remove`/`replace` precondition: a +/// fail-safe leak (never a wrong delete), but every consumer of listed values (GC namespace cleanup, +/// `deletePrefixWholesale`, orphan sweep, decommission drain) always saw a mismatch against a LOCAL pool. TEST(CASObjectStorageBackend, EmulatedListTokenMatchesHeadToken) { auto backend = std::make_shared( tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); - ASSERT_EQ(backend->putIfAbsent("k/listed", "body").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create("k/listed", "body", Retry::once()))); - const Token head_token = backend->head("k/listed").token; - ASSERT_EQ(head_token.type, TokenType::Emulated); + const auto head = op.head("k/listed", Retry::once()); + ASSERT_TRUE(head.has_value()); + ASSERT_EQ(head->etag.dialect(), Dialect::Emulated); - const ListPage page = backend->list("k/", "", /*limit=*/10); + const ListPage page = op.list("k/", "", /*limit=*/10, Retry::once()); ASSERT_EQ(page.keys.size(), 1u); - ASSERT_TRUE(page.keys.front().token.has_value()); - EXPECT_EQ(*page.keys.front().token, head_token); + ASSERT_TRUE(page.keys.front().etag.has_value()); + EXPECT_EQ(*page.keys.front().etag, head->etag); } namespace @@ -1384,42 +1410,50 @@ DB::ObjectStoragePtr makeFixedEtagStorageForTest() /// the second. TEST(CASObjectStorageBackend, EmuTokenDisambiguatesSameEtagRewrite) { - ObjectStorageBackend backend(makeFixedEtagStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); + auto backend = std::make_shared(makeFixedEtagStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); - const auto put1 = backend.putIfAbsent("k/tick", "v1"); - ASSERT_EQ(put1.outcome, PutOutcome::Done); - const auto put2 = backend.putOverwrite("k/tick", "v2", put1.token); - ASSERT_EQ(put2.outcome, PutOutcome::Done); + const WriteResult put1 = op.create("k/tick", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put1)); + const Etag inc1 = std::get(put1).etag; + const WriteResult put2 = op.replace("k/tick", "v2", inc1, Retry::once()); + ASSERT_TRUE(std::holds_alternative(put2)); + const Etag inc2 = std::get(put2).etag; - EXPECT_NE(put1.token.value, put2.token.value); - EXPECT_EQ(put1.token.type, TokenType::Emulated); - EXPECT_EQ(put2.token.type, TokenType::Emulated); + EXPECT_NE(inc1, inc2); + EXPECT_EQ(inc1.dialect(), Dialect::Emulated); + EXPECT_EQ(inc2.dialect(), Dialect::Emulated); - /// A stale delete using the FIRST incarnation's token must not match the live (second) one. - EXPECT_EQ(backend.deleteExact("k/tick", put1.token).kind, DeleteOutcome::Kind::TokenMismatch); - EXPECT_TRUE(backend.head("k/tick").exists); + /// A stale delete using the FIRST incarnation must not match the live (second) one. + EXPECT_EQ(op.remove("k/tick", inc1, Retry::once()), Removal::Mismatch); + EXPECT_TRUE(op.head("k/tick", Retry::once()).has_value()); } TEST(CASObjectStorageBackend, PublishBlobEmulatedDisambiguatesSameEtagFromStaleDelete) { - ObjectStorageBackend backend(makeFixedEtagStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); + auto backend = std::make_shared(makeFixedEtagStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); const String key = "k/publish-tick"; - ASSERT_EQ(backend.putIfAbsent(key, "old-complete-body").outcome, PutOutcome::Done); - const Token stale_token = backend.head(key).token; + ASSERT_TRUE(std::holds_alternative(op.create(key, "old-complete-body", Retry::once()))); + const auto stale = op.head(key, Retry::once()); + ASSERT_TRUE(stale.has_value()); + const Etag stale_token = stale->etag; - backend.publishBlob(streamingPublication(key, "fresh-envelope", "payload", 7)); + op.publish(streamingPublication(key, "fresh-envelope", "payload", 7), Retry::once()); - const HeadResult published = backend.head(key); - ASSERT_TRUE(published.exists); - EXPECT_NE(published.token, stale_token); - EXPECT_EQ(published.token.type, TokenType::Emulated); - EXPECT_EQ(backend.deleteExact(key, stale_token).kind, DeleteOutcome::Kind::TokenMismatch); + const auto published = op.head(key, Retry::once()); + ASSERT_TRUE(published.has_value()); + EXPECT_NE(published->etag, stale_token); + EXPECT_EQ(published->etag.dialect(), Dialect::Emulated); + EXPECT_EQ(op.remove(key, stale_token, Retry::once()), Removal::Mismatch); - const auto live = backend.get(key); + const auto live = op.read(key, Retry::once()); ASSERT_TRUE(live.has_value()); EXPECT_EQ(live->bytes, "fresh-envelopepayload"); - EXPECT_EQ(live->token, published.token); + EXPECT_EQ(live->etag, published->etag); } namespace @@ -1511,17 +1545,20 @@ TEST(CASObjectStorageBackend, DeleteExactErasesEmuTokenStateOnlyWhenEtagIsComfor /// etag, no disambiguator) rather than a same-quantum tie with the just-consumed delete token. { const String old_etag = "1000000000000000000"; - ObjectStorageBackend backend(makeFixedNumericEtagStorageForTest(old_etag), ObjectStorageBackend::Mode::EmulatedSingleProcess); - - const auto put1 = backend.putIfAbsent("k/old", "v1"); - ASSERT_EQ(put1.outcome, PutOutcome::Done); - ASSERT_EQ(put1.token.value, old_etag); - ASSERT_EQ(backend.deleteExact("k/old", put1.token).kind, DeleteOutcome::Kind::Deleted); - - const auto put2 = backend.putIfAbsent("k/old", "v2"); - ASSERT_EQ(put2.outcome, PutOutcome::Done); - EXPECT_EQ(put2.token.value, old_etag) << "entry should have been erased on delete (etag comfortably old), " - "so the recreate mints the bare etag, not a disambiguated one"; + auto backend = std::make_shared(makeFixedNumericEtagStorageForTest(old_etag), ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + + const WriteResult put1 = op.create("k/old", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put1)); + ASSERT_EQ(PersistedEtag::capture(std::get(put1).etag).value, old_etag); + ASSERT_EQ(op.remove("k/old", std::get(put1).etag, Retry::once()), Removal::Removed); + + const WriteResult put2 = op.create("k/old", "v2", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put2)); + EXPECT_EQ(PersistedEtag::capture(std::get(put2).etag).value, old_etag) + << "entry should have been erased on delete (etag comfortably old), " + "so the recreate mints the bare etag, not a disambiguated one"; } /// An etag within the safety margin of "now": delete must RETAIN the entry, so the same @@ -1530,17 +1567,20 @@ TEST(CASObjectStorageBackend, DeleteExactErasesEmuTokenStateOnlyWhenEtagIsComfor const auto now_ns = std::chrono::duration_cast( std::chrono::system_clock::now().time_since_epoch()).count(); const String recent_etag = std::to_string(now_ns); - ObjectStorageBackend backend(makeFixedNumericEtagStorageForTest(recent_etag), ObjectStorageBackend::Mode::EmulatedSingleProcess); - - const auto put1 = backend.putIfAbsent("k/fresh", "v1"); - ASSERT_EQ(put1.outcome, PutOutcome::Done); - ASSERT_EQ(put1.token.value, recent_etag); - ASSERT_EQ(backend.deleteExact("k/fresh", put1.token).kind, DeleteOutcome::Kind::Deleted); - - const auto put2 = backend.putIfAbsent("k/fresh", "v2"); - ASSERT_EQ(put2.outcome, PutOutcome::Done); - EXPECT_EQ(put2.token.value, recent_etag + "#1") << "entry should have been RETAINED on delete (etag recent), " - "so the recreate is disambiguated against it"; + auto backend = std::make_shared(makeFixedNumericEtagStorageForTest(recent_etag), ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + + const WriteResult put1 = op.create("k/fresh", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put1)); + ASSERT_EQ(PersistedEtag::capture(std::get(put1).etag).value, recent_etag); + ASSERT_EQ(op.remove("k/fresh", std::get(put1).etag, Retry::once()), Removal::Removed); + + const WriteResult put2 = op.create("k/fresh", "v2", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put2)); + EXPECT_EQ(PersistedEtag::capture(std::get(put2).etag).value, recent_etag + "#1") + << "entry should have been RETAINED on delete (etag recent), " + "so the recreate is disambiguated against it"; } } @@ -1552,160 +1592,52 @@ TEST(CASObjectStorageBackend, EmuTokenStateEventuallyPrunesDistinctShortLivedKey constexpr size_t expected_recent_key_bound = 24; auto now_ns = std::make_shared>(start_ns); - ObjectStorageBackend backend(makeClockEtagStorageForTest(now_ns), ObjectStorageBackend::Mode::EmulatedSingleProcess); + auto backend = std::make_shared(makeClockEtagStorageForTest(now_ns), ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); for (size_t i = 0; i < key_count; ++i) { const uint64_t current_ns = start_ns + i * step_ns; now_ns->store(current_ns); - backend.setEmuNowNsForTest(current_ns); + backend->setEmuNowNsForTest(current_ns); const String key = "k/short-lived-" + std::to_string(i); - const auto put = backend.putIfAbsent(key, "body"); - ASSERT_EQ(put.outcome, PutOutcome::Done); - ASSERT_EQ(backend.deleteExact(key, put.token).kind, DeleteOutcome::Kind::Deleted); + const WriteResult put = op.create(key, "body", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); + ASSERT_EQ(op.remove(key, std::get(put).etag, Retry::once()), Removal::Removed); } const uint64_t sweep_ns = start_ns + key_count * step_ns + 2'000'000'000ULL; now_ns->store(sweep_ns); - backend.setEmuNowNsForTest(sweep_ns); - const auto trigger = backend.putIfAbsent("k/sweep-trigger", "body"); - ASSERT_EQ(trigger.outcome, PutOutcome::Done); - ASSERT_EQ(backend.deleteExact("k/sweep-trigger", trigger.token).kind, DeleteOutcome::Kind::Deleted); + backend->setEmuNowNsForTest(sweep_ns); + const WriteResult trigger = op.create("k/sweep-trigger", "body", Retry::once()); + ASSERT_TRUE(std::holds_alternative(trigger)); + ASSERT_EQ(op.remove("k/sweep-trigger", std::get(trigger).etag, Retry::once()), Removal::Removed); - EXPECT_LE(backend.emuTokenStateSizeForTest(), expected_recent_key_bound) + EXPECT_LE(backend->emuTokenStateSizeForTest(), expected_recent_key_bound) << "token state should track only the bounded recent-key window, not all " << key_count << " deleted keys"; } -namespace -{ - -/// A `LocalObjectStorage` that counts `writeObject`/`removeObjectIfTokenMatches` calls -- used to -/// prove that a wrong-dialect expected token is rejected LOCALLY, before anything reaches the wire. -class CallCountingObjectStorage final : public DB::LocalObjectStorage -{ -public: - using DB::LocalObjectStorage::LocalObjectStorage; - - std::unique_ptr writeObject( - const DB::StoredObject & object, - DB::WriteMode mode, - std::optional attributes, - size_t buf_size, - const DB::WriteSettings & write_settings) override - { - ++write_calls; - return DB::LocalObjectStorage::writeObject(object, mode, attributes, buf_size, write_settings); - } - - DB::ConditionalRemoveResult removeObjectIfTokenMatches(const DB::StoredObject & object, const std::string & etag) override - { - ++remove_if_matches_calls; - return DB::LocalObjectStorage::removeObjectIfTokenMatches(object, etag); - } - - std::atomic write_calls{0}; - std::atomic remove_if_matches_calls{0}; -}; - -DB::ObjectStoragePtr makeCallCountingStorageForTest() -{ - static std::atomic counter{0}; - const auto unique = std::to_string(::getpid()) + "_" + std::to_string(counter.fetch_add(1)); - const auto root = (std::filesystem::temp_directory_path() / ("cas_call_counting_unit_" + unique)).string(); - - std::error_code ec; - std::filesystem::remove_all(root, ec); - std::filesystem::create_directories(root, ec); - - DB::LocalObjectStorageSettings settings("test", root, /*read_only_=*/false); - return std::make_shared(std::move(settings)); -} - -} - -/// codex-review-triage §3.18, finding №19: Native-mode conditional mutations forward only -/// `Token::value` to the wire (`object_storage_write_if_match` / `removeObjectIfTokenMatches`), -/// blind to `Token::type`. A wrong-dialect token whose VALUE happens to equal the live incarnation's -/// must be rejected LOCALLY -- before any wire call is made -- never merely rely on the remote -/// backend to reject a foreign-dialect value it was never designed to compare. -TEST(CASObjectStorageBackend, NativeRejectsWrongDialectTokenBeforeTouchingTheWire) -{ - auto storage = std::static_pointer_cast(makeCallCountingStorageForTest()); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - - /// Placed through the object storage rather than through the backend: a Native write over a local - /// storage has no response incarnation to attribute itself to. Native passes the key to the - /// storage verbatim, so this is the same object the backend reads below -- anchored under the - /// storage's own root, since a bare relative key would resolve beside the test process. - const String key = DB::Cas::tests::nativeKeyUnder(storage, "k/dialect"); - { - auto out = storage->writeObject( - DB::StoredObject(key), DB::WriteMode::Rewrite, {}, DB::DBMS_DEFAULT_BUFFER_SIZE, DB::WriteSettings{}); - DB::writeString(String("v1"), *out); - out->finalize(); - } - const Token live = backend.head(key).token; - ASSERT_EQ(live.type, TokenType::ETag); - - storage->write_calls = 0; - storage->remove_if_matches_calls = 0; - - /// Same wire VALUE, wrong dialect TYPE (Emulated instead of this backend's native ETag dialect). - const Token wrong_type_token{live.value, TokenType::Emulated}; - - EXPECT_EQ(backend.putOverwrite(key, "v2", wrong_type_token).outcome, PutOutcome::PreconditionFailed); - EXPECT_EQ(backend.casPut(key, "v2", wrong_type_token).outcome, CasOutcome::Conflict); - EXPECT_EQ(backend.deleteExact(key, wrong_type_token).kind, DeleteOutcome::Kind::TokenMismatch); - - EXPECT_EQ(storage->write_calls.load(), 0); - EXPECT_EQ(storage->remove_if_matches_calls.load(), 0); - - /// The live incarnation must be untouched by all three rejected attempts. - EXPECT_EQ(backend.head(key).token, live); -} - -/// The incarnation grammar: an empty, wildcard or list token would turn a conditional mutation into -/// an unconditional one, so every mutation refuses it as a caller bug (LOGICAL_ERROR) rather than -/// forwarding it to the wire or the emu compare -- distinct from a WRONG-dialect token (see -/// NativeRejectsWrongDialectTokenBeforeTouchingTheWire above), which is a graceful non-match, not a -/// malformed value. Under a debug or sanitizer build, constructing a LOGICAL_ERROR exception ABORTS -/// at construction (Exception::handle_error_code), so the same table is asserted as a death -/// expectation there instead; the contract ("a malformed token is refused") is what both forms pin. -#ifndef DEBUG_OR_SANITIZER_BUILD -TEST(CASBackendGrammar, RejectsEmptyStarAndListTokensOnEveryMutation) -{ - auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - const DB::Cas::Token empty{"", DB::Cas::TokenType::ETag}; - const DB::Cas::Token star{"*", DB::Cas::TokenType::ETag}; - const DB::Cas::Token list{"\"a\", \"b\"", DB::Cas::TokenType::ETag}; - for (const auto & bad : {empty, star, list}) - { - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { backend.putOverwrite("k", "v", bad); }); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { backend.casPut("k", "v", bad); }); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { backend.deleteExact("k", bad); }); - } -} -#endif - -#if defined(DEBUG_OR_SANITIZER_BUILD) -TEST(CASBackendGrammarDeathTest, RejectsEmptyStarAndListTokensOnEveryMutation) -{ - auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); - ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - - const DB::Cas::Token empty{"", DB::Cas::TokenType::ETag}; - const DB::Cas::Token star{"*", DB::Cas::TokenType::ETag}; - const DB::Cas::Token list{"\"a\", \"b\"", DB::Cas::TokenType::ETag}; - for (const auto & bad : {empty, star, list}) - { - EXPECT_DEATH({ (void)backend.putOverwrite("k", "v", bad); }, ""); - EXPECT_DEATH({ (void)backend.casPut("k", "v", bad); }, ""); - EXPECT_DEATH({ (void)backend.deleteExact("k", bad); }, ""); - } -} -#endif +/// `NativeRejectsWrongDialectTokenBeforeTouchingTheWire` is deleted here: it built a `Token{value, +/// Dialect::Emulated}` holding a NATIVE backend's live wire value under the WRONG dialect tag, to prove +/// the mismatch was caught locally rather than forwarded to the wire. `Etag` no longer admits that +/// construction -- it is minted ONLY by `CasRequests::mint`/`tryMint`, always from `backend->dialect()`, +/// so a caller can never hold an `Etag` tagged with a dialect other than the backend that observed it. +/// The property this test pinned ("a value observed under one dialect can never be mistaken for another +/// backend's incarnation") is now enforced by the type itself rather than by a runtime comparison; see +/// `CasRequests::valueFor`'s backend-identity check (also exercised, from the other side, by +/// `EmuTokenSurvivesProcessRestartAcrossRecreate` above). +/// +/// `CASBackendGrammar.RejectsEmptyStarAndListTokensOnEveryMutation` and its +/// `CASBackendGrammarDeathTest` sibling are deleted for the same reason: they built literal +/// `Token{"", ...}` / `Token{"*", ...}` / `Token{"\"a\", \"b\"", ...}` values to drive `putOverwrite`/ +/// `casPut`/`deleteExact` into the primitive's `LOGICAL_ERROR` grammar guard. `Etag::mint`/`tryMint` +/// refuse to construct an `Etag` from a malformed value in the first place (`CORRUPTED_DATA`), so no +/// caller reaching the primitives through `CasOperation` can ever hold one -- the grammar guard inside +/// `ObjectStorageBackend::write`/`removeUnder` is unreachable from the public engine surface and stays +/// as defense in depth only. The grammar predicate itself remains directly pinned by +/// `CASBackendGrammar.GenerationDialectAcceptsOnlyCanonicalPositiveDecimal` above. #endif diff --git a/src/Disks/tests/gtest_cas_backend_contract.cpp b/src/Disks/tests/gtest_cas_backend_contract.cpp index d84757c67573..ee7a47fa1b89 100644 --- a/src/Disks/tests/gtest_cas_backend_contract.cpp +++ b/src/Disks/tests/gtest_cas_backend_contract.cpp @@ -2,22 +2,20 @@ #include #include #include +#include #include #include #include +#include using namespace DB::Cas; -namespace DB::ErrorCodes -{ -extern const int NOT_IMPLEMENTED; -} - using DB::Cas::tests::expectBytes; +using DB::Cas::tests::openRequestsForTest; -/// Parameterized contract suite: every case creates a fresh backend from the factory, -/// then exercises the Backend seam generically (no InMemoryBackend-specific calls). -/// Fault-injection-only features are excluded — those are InMemory-specific tests. +/// Parameterized contract suite: every case creates a fresh backend from the factory, then exercises +/// the seam generically through `CasRequests`/`CasOperation` over an open fence (no InMemoryBackend- +/// specific calls). Fault-injection-only features are excluded -- those are InMemory-specific tests. class CASBackendContract : public ::testing::TestWithParam> { }; @@ -25,138 +23,173 @@ class CASBackendContract : public ::testing::TestWithParamputIfAbsent("k", "v1"); - const Token t1 = put.token; - EXPECT_EQ(put.outcome, PutOutcome::Done); - EXPECT_FALSE(t1.empty()); - EXPECT_EQ(b->putIfAbsent("k", "clobber").outcome, PutOutcome::PreconditionFailed); - auto g = b->get("k"); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto put = op.create("k", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); + const Etag t1 = std::get(put).etag; + EXPECT_TRUE(std::holds_alternative(op.create("k", "clobber", Retry::once()))); + auto g = op.read("k", Retry::once()); ASSERT_TRUE(g.has_value()); EXPECT_EQ(g->bytes, "v1"); - EXPECT_EQ(g->token, t1); - EXPECT_FALSE(b->get("absent").has_value()); + EXPECT_EQ(g->etag, t1); + EXPECT_FALSE(op.read("absent", Retry::once()).has_value()); } +/// A wrong-but-REAL precondition, since `Etag` has no public constructor any more: overwriting the key +/// once legitimately mints a second incarnation, which makes the FIRST one genuinely stale for this +/// same key -- a value the engine accepts as a precondition (unlike a fabricated one) but refuses as +/// the wrong one, because the object has already moved past it. TEST_P(CASBackendContract, OverwriteIsTokenExactAndMintsFreshToken) { auto b = GetParam()(); - const Token t1 = b->putIfAbsent("k", "v1").token; - EXPECT_EQ(b->putOverwrite("k", "v2", Token{"wrong", TokenType::Emulated}).outcome, PutOutcome::PreconditionFailed); - expectBytes(b, "k", "v1"); // untouched on mismatch - const auto overwrite = b->putOverwrite("k", "v2", t1); - EXPECT_EQ(overwrite.outcome, PutOutcome::Done); - EXPECT_NE(overwrite.token, t1); // tokens never repeat + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto created = op.create("k", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(created)); + const Etag t1 = std::get(created).etag; + const auto warmup = op.replace("k", "v1b", t1, Retry::once()); // mints a second incarnation, so t1 goes stale + ASSERT_TRUE(std::holds_alternative(warmup)); + const Etag t2 = std::get(warmup).etag; + + EXPECT_TRUE(std::holds_alternative(op.replace("k", "v2", t1, Retry::once()))); + expectBytes(b, "k", "v1b"); // untouched on mismatch + + const auto overwrite = op.replace("k", "v2", t2, Retry::once()); + ASSERT_TRUE(std::holds_alternative(overwrite)); + EXPECT_NE(std::get(overwrite).etag, t2); // etags never repeat expectBytes(b, "k", "v2"); } TEST_P(CASBackendContract, CasPutCreateAndSwap) { auto b = GetParam()(); - const auto create = b->casPut("m", "s1", std::nullopt); - const Token t1 = create.token; - EXPECT_EQ(create.outcome, CasOutcome::Committed); // create-if-absent - EXPECT_EQ(b->casPut("m", "s1x", std::nullopt).outcome, CasOutcome::Conflict); // exists now - EXPECT_EQ(b->casPut("m", "s2", Token{"stale", TokenType::Emulated}).outcome, CasOutcome::Conflict); - expectBytes(b, "m", "s1"); - EXPECT_EQ(b->casPut("m", "s2", t1).outcome, CasOutcome::Committed); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto create = op.create("m", "s1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(create)); // create-if-absent + const Etag t1 = std::get(create).etag; + EXPECT_TRUE(std::holds_alternative(op.create("m", "s1x", Retry::once()))); // exists now + + /// Mint a second real incarnation so `t1` becomes a genuinely stale (never fabricated) wrong swap. + const auto warmup = op.replace("m", "s1y", t1, Retry::once()); + ASSERT_TRUE(std::holds_alternative(warmup)); + const Etag t2 = std::get(warmup).etag; + EXPECT_TRUE(std::holds_alternative(op.replace("m", "s2", t1, Retry::once()))); + expectBytes(b, "m", "s1y"); + + EXPECT_TRUE(std::holds_alternative(op.replace("m", "s2", t2, Retry::once()))); expectBytes(b, "m", "s2"); } TEST_P(CASBackendContract, DeleteExactnessAndSurvival) { auto b = GetParam()(); - const Token t1 = b->putIfAbsent("k", "v1").token; - auto d1 = b->deleteExact("k", Token{"wrong", TokenType::Emulated}); - EXPECT_EQ(d1.kind, DeleteOutcome::Kind::TokenMismatch); - EXPECT_TRUE(b->get("k").has_value()); // SURVIVES wrong-token delete - auto d2 = b->deleteExact("k", t1); - EXPECT_EQ(d2.kind, DeleteOutcome::Kind::Deleted); - EXPECT_FALSE(d2.created_delete_marker); - EXPECT_FALSE(b->get("k").has_value()); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto created = op.create("k", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(created)); + const Etag t1 = std::get(created).etag; + + /// A real but stale incarnation, minted by a legitimate overwrite (see the comment on + /// `OverwriteIsTokenExactAndMintsFreshToken`). + const auto warmup = op.replace("k", "v1b", t1, Retry::once()); + ASSERT_TRUE(std::holds_alternative(warmup)); + const Etag t2 = std::get(warmup).etag; + + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Mismatch); + EXPECT_TRUE(op.read("k", Retry::once()).has_value()); // SURVIVES wrong-incarnation delete + EXPECT_EQ(op.remove("k", t2, Retry::once()), Removal::Removed); + EXPECT_FALSE(op.read("k", Retry::once()).has_value()); } TEST_P(CASBackendContract, DeleteNotFound) { auto b = GetParam()(); - const Token t1 = b->putIfAbsent("k", "v1").token; - b->deleteExact("k", t1); - EXPECT_EQ(b->deleteExact("k", t1).kind, DeleteOutcome::Kind::NotFound); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto created = op.create("k", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(created)); + const Etag t1 = std::get(created).etag; + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Removed); + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Gone); } -/// `Range` is retired for materialized reads: a non-whole window is REFUSED rather than served, so -/// no caller can silently receive a partial body where it expected the object. -TEST_P(CASBackendContract, RangedGetIsRefusedAndTheWholeReadStillServes) -{ - auto b = GetParam()(); - b->putIfAbsent("k", "0123456789"); - Range r; - r.offset = 2; - r.length = 3u; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NOT_IMPLEMENTED, [&] { (void)b->get("k", r); }); - expectBytes(b, "k", "0123456789"); -} +/// `Range` had no primitive read counterpart even before this migration -- `op.read` takes no range +/// argument at all, so a non-whole window is refused by the TYPE, not by a runtime NOT_IMPLEMENTED +/// throw. The property (the whole read still serves) is what `ReadAfterWrite` below already pins. TEST_P(CASBackendContract, Head) { auto b = GetParam()(); - b->putIfAbsent("k", "hello"); - auto h = b->head("k"); - EXPECT_TRUE(h.exists); - EXPECT_EQ(h.size, 5u); - EXPECT_FALSE(h.token.empty()); - auto h2 = b->head("missing"); - EXPECT_FALSE(h2.exists); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("k", "hello", Retry::once()))); + auto h = op.head("k", Retry::once()); + ASSERT_TRUE(h.has_value()); + EXPECT_EQ(h->size, 5u); + auto h2 = op.head("missing", Retry::once()); + EXPECT_FALSE(h2.has_value()); } TEST_P(CASBackendContract, ListPagination) { auto b = GetParam()(); - b->putIfAbsent("p/a", "0123456789"); - b->putIfAbsent("p/b", "xy"); - b->putIfAbsent("q/c", "z"); - auto page = b->list("p/", "", 10); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("p/a", "0123456789", Retry::once()))); + ASSERT_TRUE(std::holds_alternative(op.create("p/b", "xy", Retry::once()))); + ASSERT_TRUE(std::holds_alternative(op.create("q/c", "z", Retry::once()))); + auto page = op.list("p/", "", 10, Retry::once()); ASSERT_EQ(page.keys.size(), 2u); // sorted, prefix-scoped EXPECT_EQ(page.keys[0].key, "p/a"); EXPECT_EQ(page.keys[1].key, "p/b"); EXPECT_TRUE(page.next_cursor.empty()); - auto page1 = b->list("p/", "", 1); // pagination + auto page1 = op.list("p/", "", 1, Retry::once()); // pagination EXPECT_EQ(page1.keys.size(), 1u); EXPECT_EQ(page1.keys[0].key, "p/a"); EXPECT_EQ(page1.next_cursor, "p/a"); EXPECT_FALSE(page1.next_cursor.empty()); - auto page2 = b->list("p/", page1.next_cursor, 1); + auto page2 = op.list("p/", page1.next_cursor, 1, Retry::once()); EXPECT_EQ(page2.keys[0].key, "p/b"); } TEST_P(CASBackendContract, ReadAfterWrite) { auto b = GetParam()(); - const Token t1 = b->putIfAbsent("rw", "payload").token; - auto g = b->get("rw"); + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto created = op.create("rw", "payload", Retry::once()); + ASSERT_TRUE(std::holds_alternative(created)); + const Etag t1 = std::get(created).etag; + auto g = op.read("rw", Retry::once()); ASSERT_TRUE(g.has_value()); EXPECT_EQ(g->bytes, "payload"); - EXPECT_EQ(g->token, t1); - auto h = b->head("rw"); - EXPECT_TRUE(h.exists); - EXPECT_EQ(h.token, t1); + EXPECT_EQ(g->etag, t1); + auto h = op.head("rw", Retry::once()); + ASSERT_TRUE(h.has_value()); + EXPECT_EQ(h->etag, t1); } -/// After an object is created then deleted (key absent again), BOTH conditional updates against a stale -/// token must be rejected with the object still absent — a token-conditional update can never recreate a -/// missing key. For the Native S3 adapter this pins the 404-on-If-Match -> PreconditionFailed/Conflict -/// mapping; for every backend it pins that absence is not a write opportunity for a stale token. +/// After an object is created then deleted (key absent again), a conditional update against the +/// incarnation it held while alive must be rejected with the object still absent -- an +/// incarnation-conditional update can never recreate a missing key. For the Native S3 adapter this +/// pins the 404-on-If-Match -> Conflict mapping; for every backend it pins that absence is not a write +/// opportunity for a since-deleted incarnation. The legacy `putOverwrite` and `casPut(expected)` +/// verbs this test used to drive separately both reach this SAME primitive (`replace`) now. TEST_P(CASBackendContract, OverwriteAndCasOnMissingKey) { auto b = GetParam()(); - const Token t1 = b->putIfAbsent("k", "v1").token; - EXPECT_EQ(b->deleteExact("k", t1).kind, DeleteOutcome::Kind::Deleted); - ASSERT_FALSE(b->get("k").has_value()); // key is absent - - EXPECT_EQ(b->putOverwrite("k", "v2", t1).outcome, PutOutcome::PreconditionFailed); - EXPECT_FALSE(b->get("k").has_value()); // still absent - - EXPECT_EQ(b->casPut("k", "v2", t1).outcome, CasOutcome::Conflict); - EXPECT_FALSE(b->get("k").has_value()); // still absent + auto requests = openRequestsForTest(b); + auto op = requests.admit(); + const auto created = op.create("k", "v1", Retry::once()); + ASSERT_TRUE(std::holds_alternative(created)); + const Etag t1 = std::get(created).etag; + EXPECT_EQ(op.remove("k", t1, Retry::once()), Removal::Removed); + ASSERT_FALSE(op.read("k", Retry::once()).has_value()); // key is absent + + EXPECT_TRUE(std::holds_alternative(op.replace("k", "v2", t1, Retry::once()))); + EXPECT_FALSE(op.read("k", Retry::once()).has_value()); // still absent } INSTANTIATE_TEST_SUITE_P(CASInMemory, CASBackendContract, diff --git a/src/Disks/tests/gtest_cas_backend_generation.cpp b/src/Disks/tests/gtest_cas_backend_generation.cpp index d0856c44f647..77c3d0a2d647 100644 --- a/src/Disks/tests/gtest_cas_backend_generation.cpp +++ b/src/Disks/tests/gtest_cas_backend_generation.cpp @@ -38,7 +38,6 @@ using namespace DB::Cas; namespace DB::ErrorCodes { extern const int NOT_IMPLEMENTED; - extern const int CAS_WRITE_UNATTRIBUTED; } #if USE_AWS_S3 @@ -202,20 +201,21 @@ TEST(CASBackendGeneration, NativeHeadUsesNativeTokenMetadataApi) storage->ordinary_calls = 0; storage->native_calls = 0; - const auto hr = b->head(key); - ASSERT_TRUE(hr.exists); + DB::Cas::tests::OperationForTest op(*b); + const auto hr = (*op).head(key, Retry::once()); + ASSERT_TRUE(hr.has_value()); EXPECT_EQ(storage->native_calls, 1); EXPECT_EQ(storage->ordinary_calls, 0); } -/// Every token the backend mints carries native_token_type rather than a hardcoded TokenType::ETag. +/// Every token the backend mints carries native_token_type rather than a hardcoded Dialect::ETag. /// The HEAD mint is the site exercised here; the write-response mint has its own tests over the fake /// S3 client below, which is the only place a Native write can produce a response incarnation. TEST(CASBackendGeneration, StampedTokenTypeFollowsNativeKind) { auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); /// A local file's etag is its mtime in nanoseconds, which is also a valid generation value. const String key = DB::Cas::tests::nativeKeyUnder(storage, "p/gen/tok"); @@ -225,10 +225,11 @@ TEST(CASBackendGeneration, StampedTokenTypeFollowsNativeKind) out->finalize(); } - const auto hr = b->head(key); - ASSERT_TRUE(hr.exists); - EXPECT_EQ(hr.token.type, TokenType::Generation); - EXPECT_EQ(b->dialect(), TokenType::Generation); + DB::Cas::tests::OperationForTest op(*b); + const auto hr = (*op).head(key, Retry::once()); + ASSERT_TRUE(hr.has_value()); + EXPECT_EQ(hr->etag.dialect(), Dialect::Generation); + EXPECT_EQ(b->dialect(), Dialect::Generation); } /// A generation-dialect (GCS) mount wants bucket versioning to be verifiably off: a token-exact @@ -241,7 +242,7 @@ TEST(CASBackendGeneration, CheckPoolPreconditionsWarnsAndContinuesOnUnverifiable { auto b = std::make_shared( makeVersioningObjectStorageForTest(std::nullopt), ObjectStorageBackend::Mode::Native); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); ScopedBackendLogCapture capture; EXPECT_NO_THROW(b->checkPoolPreconditions()); @@ -256,7 +257,7 @@ TEST(CASBackendGeneration, CheckPoolPreconditionsRejectsEnabledVersioning) { auto b = std::make_shared( makeVersioningObjectStorageForTest(true), ObjectStorageBackend::Mode::Native); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); expectThrowsNotImplementedSaying("VERSIONING enabled", [&] { b->checkPoolPreconditions(); }); } @@ -266,7 +267,7 @@ TEST(CASBackendGeneration, CheckPoolPreconditionsAcceptsVerifiedDisabledVersioni { auto b = std::make_shared( makeVersioningObjectStorageForTest(false), ObjectStorageBackend::Mode::Native); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); ScopedBackendLogCapture capture; EXPECT_NO_THROW(b->checkPoolPreconditions()); @@ -274,14 +275,14 @@ TEST(CASBackendGeneration, CheckPoolPreconditionsAcceptsVerifiedDisabledVersioni } /// The ETag-dialect (AWS-compatible) backend never consults bucket versioning at all — the check is -/// a silent no-op for any backend that is not Native + TokenType::Generation. Driven over a storage +/// a silent no-op for any backend that is not Native + Dialect::Generation. Driven over a storage /// whose probe is unverifiable, which is what a generation-dialect backend warns about: dropping the /// dialect guard from checkPoolPreconditions would fail the silence assertion. TEST(CASBackendGeneration, CheckPoolPreconditionsNoOpOnEtagDialect) { auto b = std::make_shared( makeVersioningObjectStorageForTest(std::nullopt), ObjectStorageBackend::Mode::Native); - ASSERT_EQ(b->nativeTokenType(), TokenType::ETag); + ASSERT_EQ(b->nativeTokenType(), Dialect::ETag); ScopedBackendLogCapture capture; EXPECT_NO_THROW(b->checkPoolPreconditions()); @@ -294,7 +295,7 @@ TEST(CASBackendGeneration, CheckSkipAccessCheckSupportRejectsGenerationDialect) { auto b = std::make_shared( DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); expectThrowsNotImplementedSaying("skip_access_check=true is not supported", [&] { b->checkSkipAccessCheckSupport(); }); } @@ -305,7 +306,7 @@ TEST(CASBackendGeneration, CheckSkipAccessCheckSupportAllowsEtagAndEmulatedBacke { auto etag = std::make_shared( DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - ASSERT_EQ(etag->nativeTokenType(), TokenType::ETag); + ASSERT_EQ(etag->nativeTokenType(), Dialect::ETag); EXPECT_NO_THROW(etag->checkSkipAccessCheckSupport()); auto emulated = std::make_shared( @@ -324,9 +325,9 @@ TEST(CASBackendGeneration, ListTokensDisabledOnGenerationStores) auto b = std::make_shared( DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); EXPECT_TRUE(b->supportsListTokens()); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); EXPECT_FALSE(b->supportsListTokens()); - b->setNativeTokenTypeForTest(TokenType::ETag); + b->setNativeTokenTypeForTest(Dialect::ETag); EXPECT_TRUE(b->supportsListTokens()); } @@ -334,7 +335,7 @@ TEST(CASBackendGeneration, ConditionalWriteSettingsForceSinglePutOnGenerationSto { auto b = std::make_shared( DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - b->setNativeTokenTypeForTest(TokenType::Generation); + b->setNativeTokenTypeForTest(Dialect::Generation); const auto ws = b->conditionalWriteSettingsForTest(); EXPECT_EQ(ws.object_storage_request_mode, DB::ObjectStorageRequestMode::NativeConditional); EXPECT_TRUE(ws.s3_force_single_part_upload); @@ -343,7 +344,7 @@ TEST(CASBackendGeneration, ConditionalWriteSettingsForceSinglePutOnGenerationSto ASSERT_TRUE(ws.s3_check_objects_after_upload_override.has_value()); EXPECT_FALSE(*ws.s3_check_objects_after_upload_override); - b->setNativeTokenTypeForTest(TokenType::ETag); + b->setNativeTokenTypeForTest(Dialect::ETag); const auto ws2 = b->conditionalWriteSettingsForTest(); EXPECT_EQ(ws2.object_storage_request_mode, DB::ObjectStorageRequestMode::NativeConditional); EXPECT_FALSE(ws2.s3_force_single_part_upload); @@ -353,32 +354,14 @@ TEST(CASBackendGeneration, ConditionalWriteSettingsForceSinglePutOnGenerationSto EXPECT_FALSE(*ws2.s3_check_objects_after_upload_override); } -/// C1: the three token-policy helpers are the single source of truth for how a Native-mode backend -/// mints a HEAD/PUT token, gates a LIST token, and compares tokens. Characterizes the behavior the -/// scattered call sites have today so the consolidation stays byte-for-byte behavior-preserving. -TEST(CASBackendGeneration, TokenPolicyHelpersAreConsistentWithDialect) -{ - auto b = std::make_shared( - DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - - /// ETag dialect: head/put tokens carry ETag; list surfaces the same-typed token for a non-empty etag. - ASSERT_EQ(b->nativeTokenType(), TokenType::ETag); - EXPECT_EQ(b->tokenForHead("abc").type, TokenType::ETag); - EXPECT_EQ(b->tokenForHead("abc"), (Token{"abc", TokenType::ETag})); - ASSERT_TRUE(b->tokenForList("abc").has_value()); - EXPECT_EQ(*b->tokenForList("abc"), b->tokenForHead("abc")); /// list token == head token (same etag) - EXPECT_FALSE(b->tokenForList("").has_value()); /// empty etag => no list token - - /// Generation dialect (GCS): head token flips to Generation; list tokens are disabled wholesale - /// (poisoned If-Match), so tokenForList is always nullopt regardless of the etag. - b->setNativeTokenTypeForTest(TokenType::Generation); - EXPECT_EQ(b->tokenForHead("g1").type, TokenType::Generation); - EXPECT_FALSE(b->tokenForList("g1").has_value()); - - /// tokenMatches is exact identity (value AND type) — a same-value/different-type token never matches. - EXPECT_TRUE(ObjectStorageBackend::tokenMatches(Token{"x", TokenType::ETag}, Token{"x", TokenType::ETag})); - EXPECT_FALSE(ObjectStorageBackend::tokenMatches(Token{"x", TokenType::ETag}, Token{"x", TokenType::Emulated})); -} +/// `tokenForHead` and `tokenMatches` were deleted with the `Token` type they built: minting an +/// incarnation and comparing it against a precondition are now `CasRequests::mint`/`tryMint` and +/// `CasRequests::valueFor`, always stamped with the observing backend's own dialect, so no caller can +/// construct or compare one by hand any more. The remaining piece, `tokenForList`'s ETag/Generation +/// gating, stays pinned by `CASBackendGeneration.ListTokensDisabledOnGenerationStores` above; +/// dialect-aware value validation is `CASBackendGrammar.GenerationDialectAcceptsOnlyCanonicalPositiveDecimal` +/// and the cross-backend precondition guard `CASObjectStorageBackend.EmuTokenSurvivesProcessRestartAcrossRecreate`, +/// both in gtest_cas_backend.cpp. #if USE_AWS_S3 @@ -619,7 +602,7 @@ class CASBackendGenerationS3 : public ::testing::Test /// A fresh backend, native token type forced to Generation unless overridden (the ETag dialect /// is needed to prove the generation-only quote handling does not touch it). - std::shared_ptr makeBackend(TokenType token_type = TokenType::Generation) + std::shared_ptr makeBackend(Dialect token_type = Dialect::Generation) { auto storage = makeGenerationS3ObjectStorageForTest(client); auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); @@ -635,10 +618,11 @@ TEST(CASBackendGeneration, PublishBlobAboveFormerGenerationCapUsesOrdinaryMultip auto storage = makeGenerationS3ObjectStorageForTest( client, /*force_multipart=*/true, /*conditional_put_cap=*/16); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - backend.setNativeTokenTypeForTest(TokenType::Generation); + backend.setNativeTokenTypeForTest(Dialect::Generation); const String payload(1024, 'x'); - backend.publishBlob(BlobPublishRequest{ + DB::Cas::tests::OperationForTest op(backend); + (*op).publish(BlobPublishRequest{ .destination_key = "p/gen/publish-multipart", .publication = StreamingBlobPublication{ .payload_size = payload.size(), @@ -646,7 +630,7 @@ TEST(CASBackendGeneration, PublishBlobAboveFormerGenerationCapUsesOrdinaryMultip .open_payload = [payload] { return std::make_unique(payload); - }}}); + }}}, Retry::once()); EXPECT_EQ(client->put_object_calls, 0u); EXPECT_EQ(client->create_multipart_calls, 1u); @@ -664,11 +648,12 @@ TEST(CASBackendGeneration, PublishBlobSucceedsWithoutResponseGeneration) auto storage = makeGenerationS3ObjectStorageForTest( client, /*force_multipart=*/false, /*conditional_put_cap=*/1); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - backend.setNativeTokenTypeForTest(TokenType::Generation); + backend.setNativeTokenTypeForTest(Dialect::Generation); client->put_returns_no_etag = true; const String payload = "payload"; - EXPECT_NO_THROW(backend.publishBlob(BlobPublishRequest{ + DB::Cas::tests::OperationForTest op(backend); + EXPECT_NO_THROW((*op).publish(BlobPublishRequest{ .destination_key = "p/gen/publish-no-generation", .publication = StreamingBlobPublication{ .payload_size = payload.size(), @@ -676,7 +661,7 @@ TEST(CASBackendGeneration, PublishBlobSucceedsWithoutResponseGeneration) .open_payload = [payload] { return std::make_unique(payload); - }}})); + }}}, Retry::once())); EXPECT_EQ(client->put_object_calls, 1u); EXPECT_EQ(client->head_object_calls, 0u); @@ -689,18 +674,25 @@ TEST(CASBackendGeneration, PublishBlobSucceedsWithoutResponseGeneration) /// ETag-dialect sibling of PublishBlobSucceedsWithoutResponseGeneration above: a publication has no /// incarnation to attribute in the first place, so it is unaffected by this guard. Default (ETag) /// dialect here, deliberately NOT stamped Generation, whose own two cases are covered by -/// CASBackendGenerationS3.WriteEmptyGenerationIsUnattributed and WriteNonNumericGenerationIsUnattributed. -TEST(CASBackendGrammar, NamelessWriteResponseThrowsWriteUnattributed) +/// CASBackendGenerationS3.WriteEmptyGenerationIsUnresolvedNotThrown and WriteNonNumericGenerationIsUnresolvedNotThrown. +TEST(CASBackendGrammar, NamelessWriteResponseIsUnresolvedNotThrown) { (void)getContext(); FakeGenerationS3Client * client = nullptr; auto storage = makeGenerationS3ObjectStorageForTest(client); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - ASSERT_EQ(backend.nativeTokenType(), TokenType::ETag); + ASSERT_EQ(backend.nativeTokenType(), Dialect::ETag); client->put_returns_no_etag = true; - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, [&] { backend.putIfAbsent("p/gen/nameless-write", "v"); }); + /// A 2xx write reply carrying no usable incarnation is an ambiguity the engine settles by a + /// resolve read (CasRequests::writeLoop), never an immediate corruption verdict; the mock's + /// resolve GET finds nothing at the key, so the create-if-absent precondition is still + /// satisfiable and a single-attempt policy reports GaveUp{Unresolved} rather than throwing. + DB::Cas::tests::OperationForTest op(backend); + const WriteResult result = (*op).create("p/gen/nameless-write", "v", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); EXPECT_EQ(client->put_object_calls, 1u); } @@ -714,10 +706,11 @@ TEST(CASBackendGeneration, ConditionalWriteHonoursTheObjectStorageConditionalPut auto storage = makeGenerationS3ObjectStorageForTest( client, /*force_multipart=*/false, /*conditional_put_cap=*/64); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::Native); - backend.setNativeTokenTypeForTest(TokenType::Generation); + backend.setNativeTokenTypeForTest(Dialect::Generation); const String small(32, 'a'); - EXPECT_NO_THROW(backend.casPut("p/gen/under-cap", small, std::nullopt, ObjectMeta{})); + DB::Cas::tests::OperationForTest op(backend); + EXPECT_NO_THROW((*op).create("p/gen/under-cap", small, Retry::once())); EXPECT_EQ(client->put_object_calls, 1u); EXPECT_EQ(client->create_multipart_calls, 0u); const auto single_attempt_client = storage->getSingleAttemptClient(/*request_timeout_ms=*/0); @@ -730,7 +723,7 @@ TEST(CASBackendGeneration, ConditionalWriteHonoursTheObjectStorageConditionalPut const String large(4096, 'b'); try { - backend.casPut("p/gen/over-cap", large, std::nullopt, ObjectMeta{}); + (*op).create("p/gen/over-cap", large, Retry::once()); FAIL() << "a conditional write above the cap must refuse, not go multipart"; } catch (const DB::Exception & e) @@ -749,26 +742,34 @@ TEST(CASBackendGeneration, ConditionalWriteHonoursTheObjectStorageConditionalPut /// the CAS layer receives in the shape production actually produces, which is why a mount that could /// never succeed passed every unit test. These three tests are that crossing. -TEST_F(CASBackendGenerationS3, WriteEmptyGenerationIsUnattributed) +TEST_F(CASBackendGenerationS3, WriteEmptyGenerationIsUnresolvedNotThrown) { backend = makeBackend(); client->next_put_etag = ""; /// The write may well have landed -- an empty response value says nothing about that -- so this is - /// the resolve-by-reading class, not the corrupt-response one. - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, - [&] { backend->putIfAbsent("p/gen/no-etag", "v"); }); + /// the resolve-by-reading class, not the corrupt-response one (CasRequests::writeLoop): a 2xx + /// carrying a value no grammar accepts is an ambiguity, never an immediate corruption verdict. The + /// mock's resolve GET finds nothing at the key either, so the create-if-absent precondition is + /// still satisfiable and a single-attempt policy reports GaveUp{Unresolved} rather than throwing. + DB::Cas::tests::OperationForTest op(*backend); + const WriteResult result = (*op).create("p/gen/no-etag", "v", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); } -TEST_F(CASBackendGenerationS3, WriteNonNumericGenerationIsUnattributed) +TEST_F(CASBackendGenerationS3, WriteNonNumericGenerationIsUnresolvedNotThrown) { backend = makeBackend(); /// An MD5-shaped ETag where a generation belongs: the store answered, but not with an incarnation - /// this dialect can use, and no follow-up read can attribute the write on its behalf. + /// this dialect can use; see WriteEmptyGenerationIsUnresolvedNotThrown for why this settles as an + /// ambiguity (GaveUp{Unresolved}) rather than a thrown exception. client->next_put_etag = "\"d41d8cd98f00b204e9800998ecf8427e\""; - DB::Cas::tests::expectThrowsCode( - DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, - [&] { backend->putIfAbsent("p/gen/bad-etag", "v"); }); + DB::Cas::tests::OperationForTest op(*backend); + const WriteResult result = (*op).create("p/gen/bad-etag", "v", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); } /// A mutable conditional write whose response generation arrives quoted -- exactly what @@ -778,9 +779,12 @@ TEST_F(CASBackendGenerationS3, WriteGenerationTokenStripsTransportQuoting) { backend = makeBackend(); client->next_put_etag = "\"1783078552147137\""; - const auto put = backend->putIfAbsent("p/gen/quoted-write", "v"); - ASSERT_EQ(put.outcome, PutOutcome::Done); - EXPECT_EQ(put.token, (Token{"1783078552147137", TokenType::Generation})); + DB::Cas::tests::OperationForTest op(*backend); + const WriteResult put = (*op).create("p/gen/quoted-write", "v", Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); + const Etag & minted = std::get(put).etag; + EXPECT_EQ(minted.dialect(), Dialect::Generation); + EXPECT_EQ(PersistedEtag::capture(minted).value, "1783078552147137"); } /// The same crossing on the read side: a marked HEAD whose ETag field carries a quoted generation @@ -792,9 +796,11 @@ TEST_F(CASBackendGenerationS3, HeadGenerationTokenStripsTransportQuoting) client->objects["p/gen/quoted-head"] = "body"; client->next_head_etag = "\"1783078552147137\""; - const auto hr = backend->head("p/gen/quoted-head"); - ASSERT_TRUE(hr.exists); - EXPECT_EQ(hr.token, (Token{"1783078552147137", TokenType::Generation})); + DB::Cas::tests::OperationForTest op(*backend); + const auto hr = (*op).head("p/gen/quoted-head", Retry::once()); + ASSERT_TRUE(hr.has_value()); + EXPECT_EQ(hr->etag.dialect(), Dialect::Generation); + EXPECT_EQ(PersistedEtag::capture(hr->etag).value, "1783078552147137"); } /// The bound on that stripping. An ETag-dialect token IS the quoted ETag, and the quotes are required @@ -802,13 +808,15 @@ TEST_F(CASBackendGenerationS3, HeadGenerationTokenStripsTransportQuoting) /// This is the test that fails if the quote handling is ever made unconditional. TEST_F(CASBackendGenerationS3, EtagDialectKeepsTransportQuotingVerbatim) { - backend = makeBackend(TokenType::ETag); + backend = makeBackend(Dialect::ETag); client->objects["p/etag/quoted-head"] = "body"; client->next_head_etag = "\"d41d8cd98f00b204e9800998ecf8427e\""; - const auto hr = backend->head("p/etag/quoted-head"); - ASSERT_TRUE(hr.exists); - EXPECT_EQ(hr.token, (Token{"\"d41d8cd98f00b204e9800998ecf8427e\"", TokenType::ETag})); + DB::Cas::tests::OperationForTest op(*backend); + const auto hr = (*op).head("p/etag/quoted-head", Retry::once()); + ASSERT_TRUE(hr.has_value()); + EXPECT_EQ(hr->etag.dialect(), Dialect::ETag); + EXPECT_EQ(PersistedEtag::capture(hr->etag).value, "\"d41d8cd98f00b204e9800998ecf8427e\""); } /// A successful HEAD on a generation-dialect backend whose response carries no ETag/generation at all must not mint a token @@ -819,8 +827,9 @@ TEST_F(CASBackendGenerationS3, HeadMissingGenerationThrows) client->objects["p/gen/no-generation-head"] = "body"; /// next_head_etag stays empty: SetETag is never called, so the response carries no ETag field. + DB::Cas::tests::OperationForTest op(*backend); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { backend->head("p/gen/no-generation-head"); }); + [&] { (*op).head("p/gen/no-generation-head", Retry::once()); }); } /// An ordinary AWS-style ETag reaching a generation-dialect backend through a successful HEAD (a proxy dropping @@ -831,8 +840,9 @@ TEST_F(CASBackendGenerationS3, HeadNonNumericGenerationThrows) client->objects["p/gen/bad-etag-head"] = "body"; client->next_head_etag = "\"d41d8cd98f00b204e9800998ecf8427e\""; + DB::Cas::tests::OperationForTest op(*backend); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, - [&] { backend->head("p/gen/bad-etag-head"); }); + [&] { (*op).head("p/gen/bad-etag-head", Retry::once()); }); } #endif diff --git a/src/Disks/tests/gtest_cas_backend_listing.cpp b/src/Disks/tests/gtest_cas_backend_listing.cpp index 591dfdba65db..31c0c3d1a6e8 100644 --- a/src/Disks/tests/gtest_cas_backend_listing.cpp +++ b/src/Disks/tests/gtest_cas_backend_listing.cpp @@ -2,21 +2,29 @@ #include #include +#include + +#include "cas_test_helpers.h" #include #include using namespace DB::Cas; +using DB::Cas::tests::openRequestsForTest; + TEST(CASBackendListing, ForEachWalksEveryPageOnce) { InMemoryBackend b; + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); for (int i = 0; i < 2500; ++i) - b.putIfAbsent("p/" + std::to_string(1000000 + i), "v"); - b.putIfAbsent("q/other", "v"); /// out of prefix — must not be visited + op.create("p/" + std::to_string(1000000 + i), "v", Retry::once()); + op.create("q/other", "v", Retry::once()); /// out of prefix — must not be visited std::vector seen; - forEachListedKey(b, "p/", [&](const ListedKey & k) { seen.push_back(k.key); }, /*page_limit=*/1000); + op.forEachListedKey("p/", [&](const ListedKey & k) { seen.push_back(k.key); return true; }, + Retry::standard(), /*page_limit=*/1000); EXPECT_EQ(seen.size(), 2500u); /// paged (3 pages), no key dropped/duplicated EXPECT_TRUE(std::is_sorted(seen.begin(), seen.end())); } @@ -24,20 +32,17 @@ TEST(CASBackendListing, ForEachWalksEveryPageOnce) TEST(CASBackendListing, ForEachEmptyPrefixVisitsNothing) { InMemoryBackend b; - b.putIfAbsent("q/other", "v"); + CasRequests requests = openRequestsForTest(b); + CasOperation op = requests.admit(); + op.create("q/other", "v", Retry::once()); size_t visits = 0; - forEachListedKey(b, "p/", [&](const ListedKey &) { ++visits; }); + op.forEachListedKey("p/", [&](const ListedKey &) { ++visits; return true; }, Retry::standard()); EXPECT_EQ(visits, 0u); } -TEST(CASBackendListing, ClassifyMapsEveryDeleteKind) -{ - EXPECT_EQ(classifyDeleteOutcome({DeleteOutcome::Kind::Deleted, false}), DeleteClass::Deleted); - EXPECT_EQ(classifyDeleteOutcome({DeleteOutcome::Kind::NotFound, false}), DeleteClass::Absent); - EXPECT_EQ(classifyDeleteOutcome({DeleteOutcome::Kind::TokenMismatch, false}), DeleteClass::Replaced); - - EXPECT_EQ(deleteClassName(DeleteClass::Deleted), "deleted"); - EXPECT_EQ(deleteClassName(DeleteClass::Absent), "absent"); - EXPECT_EQ(deleteClassName(DeleteClass::Replaced), "replaced"); -} +/// `ClassifyMapsEveryDeleteKind` is deleted here: its whole subject was `classifyDeleteOutcome` and +/// `deleteClassName`, free helpers that translated the legacy `DeleteOutcome::Kind` three-value shape +/// into a `DeleteClass`. Both the legacy shape and the helpers are gone -- `CasOperation::remove` +/// already reports its outcome as the four-value `Removal` enum directly, with no separate +/// classification step to pin. diff --git a/src/Disks/tests/gtest_cas_blob_indegree.cpp b/src/Disks/tests/gtest_cas_blob_indegree.cpp index 7df585a018a9..d1eb99b05cc2 100644 --- a/src/Disks/tests/gtest_cas_blob_indegree.cpp +++ b/src/Disks/tests/gtest_cas_blob_indegree.cpp @@ -90,8 +90,8 @@ TEST(CASBlobInDegree, RunsAreByteDeterministic) {{bh(3), s(1), false}, {bh(1), s(1), false}, {bh(2), s(1), false}}, ra); foldDeltasIntoGeneration(*b2_req, layout, /*prior_runs*/{}, 1, /*attempt*/0, 0, {{bh(1), s(1), false}, {bh(2), s(1), false}, {bh(3), s(1), false}}, rb); - const auto ga = a.get(layout.blobTargetRunKey(1, /*attempt*/0, 0, 0)); - const auto gb = b2.get(layout.blobTargetRunKey(1, /*attempt*/0, 0, 0)); + const auto ga = (*a_req).read(layout.blobTargetRunKey(1, /*attempt*/0, 0, 0), Retry::standard()); + const auto gb = (*b2_req).read(layout.blobTargetRunKey(1, /*attempt*/0, 0, 0), Retry::standard()); ASSERT_TRUE(ga.has_value()); ASSERT_TRUE(gb.has_value()); EXPECT_EQ(ga->bytes, gb->bytes); @@ -143,7 +143,7 @@ TEST(CASBlobInDegree, FoldDeltaDivergentBytesThrowsCorrupted) DB::Cas::tests::OperationForTest backend_req(backend); Layout layout{"pool"}; /// Pre-occupy the run key (attempt 7) with junk, then fold => divergent => CORRUPTED_DATA. - backend.putIfAbsent(layout.blobTargetRunKey(1, /*attempt*/7, /*shard*/0, /*seq*/0), "not-a-valid-run"); + (*backend_req).create(layout.blobTargetRunKey(1, /*attempt*/7, /*shard*/0, /*seq*/0), "not-a-valid-run", Retry::once()); std::vector deltas{{bh(1), s(1), false}}; std::vector runs; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, @@ -184,7 +184,7 @@ TEST(CASBlobInDegree, DeterministicArtifactWhoseResolveReadObservedNothingIsNotC /// Occupy the key so the create's precondition is refused, THEN arm the refusal, so the failure /// falls on the resolve read rather than on the setup. - ASSERT_EQ(backend.putIfAbsent(key, "someone else's bytes").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative((*backend_req).create(key, "someone else's bytes", Retry::once()))); backend.refuse_key = key; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::ABORTED, @@ -235,7 +235,7 @@ CondemnedRow condemnedRowFor(uint64_t condemn_round, const String & tok = "t", bool delete_pending = false, uint64_t size = 1) { return CondemnedRow{.delete_pending = delete_pending, - .token = PersistedIncarnation{"emulated", tok}, + .token = PersistedEtag{"emulated", tok}, .size = size, .condemn_round = condemn_round}; } @@ -273,7 +273,8 @@ RunRef writeSourceEdgeRun(InMemoryBackend & backend, const Layout & layout, const String bytes = out.str(); const String key = layout.blobTargetRunKey(gen, attempt, shard, 0); - backend.putIfAbsent(key, bytes); + DB::Cas::tests::OperationForTest op(backend); + (*op).create(key, bytes, Retry::once()); return RunRef{.key = key, .checksum = sourceEdgeRunChecksum(bytes), .shard = shard, .key_generation = gen}; } @@ -467,7 +468,7 @@ TEST(CASThreeCursorMerge, NewCandidateCondemned) EXPECT_EQ(rmr.still_retired[0].ref, bh(3)); const std::optional present = (*backend_req).head(layout.blobKey(bh(3)), Retry::standard()); ASSERT_TRUE(present.has_value()); - EXPECT_TRUE(rmr.still_retired[0].token.matches(present->incarnation)); + EXPECT_TRUE(rmr.still_retired[0].token.matches(present->etag)); EXPECT_EQ(rmr.still_retired[0].size, 42u); EXPECT_EQ(rmr.still_retired[0].condemn_round, 7u); EXPECT_TRUE(rmr.graduated.empty()); @@ -477,7 +478,7 @@ TEST(CASThreeCursorMerge, NewCandidateCondemned) const DecodedRun out = decodeRun(*backend_req, runs2[0]); ASSERT_EQ(out.condemned.size(), 1u); EXPECT_EQ(out.condemned[0].first, b(3)); - EXPECT_TRUE(out.condemned[0].second.token.matches(present->incarnation)); + EXPECT_TRUE(out.condemned[0].second.token.matches(present->etag)); EXPECT_TRUE(out.zero_markers.empty()); } @@ -586,7 +587,7 @@ TEST(CASTwoCursorMerge, CarriedSentinelIsNotATouch) EXPECT_EQ(g2.condemned[0].first, b(2)); const std::optional present = (*backend_req).head(layout.blobKey(bh(2)), Retry::standard()); ASSERT_TRUE(present.has_value()); - EXPECT_TRUE(g2.condemned[0].second.token.matches(present->incarnation)); + EXPECT_TRUE(g2.condemned[0].second.token.matches(present->etag)); EXPECT_EQ(g2.condemned[0].second.size, 7u); EXPECT_TRUE(g2.zero_markers.empty()); } @@ -607,7 +608,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) const String bytes = out.str(); const RunRef bad{.key = layout.blobTargetRunKey(1, 0, 0, 0), .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .key_generation = 1}; - backend.putIfAbsent(bad.key, bytes); + (*backend_req).create(bad.key, bytes, Retry::once()); std::vector runs2; EXPECT_THROW(foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{bad}, 2, 0, 0, {}, runs2), @@ -628,7 +629,7 @@ TEST(CASTwoCursorMerge, MalformedRunFailsClosed) const String bytes = out.str(); const RunRef bad{.key = layout.blobTargetRunKey(1, 0, 0, 0), .checksum = sourceEdgeRunChecksum(bytes), .shard = 0, .key_generation = 1}; - backend.putIfAbsent(bad.key, bytes); + (*backend_req).create(bad.key, bytes, Retry::once()); std::vector runs2; EXPECT_THROW(foldDeltasIntoGeneration(*backend_req, layout, /*prior_runs*/{bad}, 2, 0, 0, {}, runs2), @@ -667,7 +668,7 @@ TEST(CASBlobInDegree, FoldStreamsPriorRunWithoutReadingItWhole) foldDeltasIntoGeneration(*oracle_req, layout, /*prior_runs*/{}, 1, 0, 0, gen1, runs1_o); const String gen1_run_key = layout.blobTargetRunKey(1, 0, 0, 0); - const auto gen1_run = backend.get(gen1_run_key); + const auto gen1_run = (*backend_req).read(gen1_run_key, Retry::standard()); ASSERT_TRUE(gen1_run.has_value()); const String gen1_run_bytes = gen1_run->bytes; /// Sanity: the prior run is far larger than one read buffer, so "it was never read whole" is a @@ -688,8 +689,8 @@ TEST(CASBlobInDegree, FoldStreamsPriorRunWithoutReadingItWhole) /// Byte-reproducibility canary: streaming and materialized folds produce identical output bytes. const String gen2_run_key = layout.blobTargetRunKey(2, 0, 0, 0); - const auto gen2_c = backend.get(gen2_run_key); - const auto gen2_o = oracle.get(gen2_run_key); + const auto gen2_c = (*backend_req).read(gen2_run_key, Retry::standard()); + const auto gen2_o = (*oracle_req).read(gen2_run_key, Retry::standard()); ASSERT_TRUE(gen2_c.has_value()); ASSERT_TRUE(gen2_o.has_value()); EXPECT_EQ(gen2_c->bytes, gen2_o->bytes); @@ -744,7 +745,7 @@ TEST(CASBlobInDegree, ZeroInDegreeStreamsRunWithoutReadingItWhole) foldDeltasIntoGeneration(*oracle_req, layout, /*prior_runs*/runs1_o, 2, 0, 0, gen2, runs2_o); const String gen2_run_key = layout.blobTargetRunKey(2, 0, 0, 0); - const auto gen2_run = backend.get(gen2_run_key); + const auto gen2_run = (*backend_req).read(gen2_run_key, Retry::standard()); ASSERT_TRUE(gen2_run.has_value()); /// Sanity: the run is far larger than one read buffer, for the same reason as above. ASSERT_GT(gen2_run->bytes.size(), static_cast(kLegacyBlockSize) * 3); @@ -785,9 +786,9 @@ TEST(CASCondemnedRow, RoundTripAllTokenTypes) for (const auto & entry : DB::Cas::kTokenTypeWords.entries) { DB::Cas::CondemnedRow row; - row.delete_pending = (entry.value == DB::Cas::TokenType::Generation); - row.marker_confirmed = (entry.value == DB::Cas::TokenType::Emulated); - row.token = DB::Cas::PersistedIncarnation{String(entry.word), "etag-abc-123"}; + row.delete_pending = (entry.value == DB::Cas::Dialect::Generation); + row.marker_confirmed = (entry.value == DB::Cas::Dialect::Emulated); + row.token = DB::Cas::PersistedEtag{String(entry.word), "etag-abc-123"}; row.size = 4096; row.condemn_round = 7; const auto bytes = DB::Cas::encodeCondemnedRow(row); @@ -800,7 +801,7 @@ TEST(CASCondemnedRow, UnknownMarkerByteFailsClosedWithCorruptedData) { /// This pins the condemned-row decoder's own marker validation. DB::Cas::CondemnedRow row; - row.token = DB::Cas::PersistedIncarnation{"etag", "t"}; + row.token = DB::Cas::PersistedEtag{"etag", "t"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes[0] = 0x03; @@ -835,7 +836,7 @@ TEST(CASRecordStream, RunMarkerByteContractFailsClosed) TEST(CASCondemnedRow, UnknownFlagBitsFailClosed) { DB::Cas::CondemnedRow row; - row.token = DB::Cas::PersistedIncarnation{"etag", "t"}; + row.token = DB::Cas::PersistedEtag{"etag", "t"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes[1] = 4; // flags byte: only bits 0 (delete_pending) and 1 (marker_confirmed) are defined EXPECT_THROW(DB::Cas::decodeCondemnedRow(bytes), DB::Exception); @@ -844,7 +845,7 @@ TEST(CASCondemnedRow, UnknownFlagBitsFailClosed) TEST(CASCondemnedRow, UnknownTokenTypeFailsClosed) { DB::Cas::CondemnedRow row; - row.token = DB::Cas::PersistedIncarnation{"etag", "t"}; + row.token = DB::Cas::PersistedEtag{"etag", "t"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes[2] = 99; // token_type byte (offset: [0]=0x02 [1]=flags [2]=token_type) EXPECT_THROW(DB::Cas::decodeCondemnedRow(bytes), DB::Exception); @@ -853,7 +854,7 @@ TEST(CASCondemnedRow, UnknownTokenTypeFailsClosed) TEST(CASCondemnedRow, TruncatedPayloadFailsClosed) { DB::Cas::CondemnedRow row; - row.token = DB::Cas::PersistedIncarnation{"etag", "0123456789"}; + row.token = DB::Cas::PersistedEtag{"etag", "0123456789"}; auto bytes = DB::Cas::encodeCondemnedRow(row); bytes.resize(bytes.size() - 3); // token bytes shorter than declared token_len EXPECT_THROW(DB::Cas::decodeCondemnedRow(bytes), DB::Exception); diff --git a/src/Disks/tests/gtest_cas_blob_meta.cpp b/src/Disks/tests/gtest_cas_blob_meta.cpp index f38769fb6824..7d33a8614aa6 100644 --- a/src/Disks/tests/gtest_cas_blob_meta.cpp +++ b/src/Disks/tests/gtest_cas_blob_meta.cpp @@ -37,11 +37,11 @@ TEST(CASBlobMeta, PutIfAbsentThenCasTransitions) ASSERT_TRUE(lm.has_value()); EXPECT_EQ(lm->meta.state, MetaState::Clean); - EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, lm->incarnation, + EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, lm->etag, BlobMeta{.state = MetaState::Condemned, .condemn_round = 5, .size = 10}))); /// the stale incarnation loses - EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, lm->incarnation, + EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, lm->etag, BlobMeta{.state = MetaState::Clean}))); } @@ -54,7 +54,7 @@ TEST(CASBlobMeta, DeleteMetaExactMatchesTheObservedIncarnation) putMetaIfAbsent(op, store->layout(), ref, BlobMeta{.state = MetaState::Condemned}); const auto lm = loadMeta(op, store->layout(), ref); ASSERT_TRUE(lm.has_value()); - EXPECT_EQ(deleteMetaExact(op, store->layout(), ref, lm->incarnation), Removal::Removed); + EXPECT_EQ(deleteMetaExact(op, store->layout(), ref, lm->etag), Removal::Removed); EXPECT_FALSE(loadMeta(op, store->layout(), ref).has_value()); } @@ -87,13 +87,13 @@ TEST(CASBlobMeta, PutLoadCasDeleteRoundTripAtWidth32) EXPECT_EQ(lm->meta.state, MetaState::Clean); EXPECT_EQ(lm->meta.size, 555u); - ASSERT_TRUE(std::holds_alternative(casMeta(op, layout, ref, lm->incarnation, + ASSERT_TRUE(std::holds_alternative(casMeta(op, layout, ref, lm->etag, BlobMeta{.state = MetaState::Condemned, .condemn_round = 7, .size = 555}))); const auto lm2 = loadMeta(op, layout, ref); ASSERT_TRUE(lm2.has_value()); EXPECT_EQ(lm2->meta.state, MetaState::Condemned); - EXPECT_EQ(deleteMetaExact(op, layout, ref, lm2->incarnation), Removal::Removed); + EXPECT_EQ(deleteMetaExact(op, layout, ref, lm2->etag), Removal::Removed); EXPECT_FALSE(loadMeta(op, layout, ref).has_value()); } @@ -159,7 +159,7 @@ TEST(CASBlobMeta, AnAmbiguousMarkerWriteIsResolvedAndReissued) const auto clean = loadMeta(op, store->layout(), ref); ASSERT_TRUE(clean.has_value()); backend->throw_next_overwrite = true; - EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, clean->incarnation, + EXPECT_TRUE(std::holds_alternative(casMeta(op, store->layout(), ref, clean->etag, BlobMeta{.state = MetaState::Condemned, .condemn_round = 1, .size = 10}))); EXPECT_EQ(backend->overwrite_attempts, 2u); } diff --git a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp index d67802db6bcd..680217bcf3e1 100644 --- a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp +++ b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp @@ -137,6 +137,27 @@ PoolConfig makeConfig() return cfg; } +/// A one-shot `create` for seeding fixture bytes before `Pool::open` runs, asserting it committed. +void seedObject(Backend & backend, const String & key, const String & bytes) +{ + DB::Cas::tests::OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// Whether `key` has a value, through an exact read (mirrors the retired `backend->get(key).has_value()`). +bool readPresent(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).read(key, Retry::standard()).has_value(); +} + +/// Whether `key` has a value, through a HEAD (mirrors the retired `backend->head(key).exists`). +bool headPresent(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + template void expectThrowsCodeContaining(int expected_code, const String & needle, F && fn); @@ -144,9 +165,9 @@ void expectCatalogResidueRefusesWithoutPoolMeta(const String & bytes, const Stri { auto backend = std::make_shared(); const Layout layout{kPrefix}; - ASSERT_EQ(backend->putIfAbsent(layout.refCatalogKey(), bytes).outcome, PutOutcome::Done); + seedObject(*backend, layout.refCatalogKey(), bytes); if (!extra_key.empty()) - ASSERT_EQ(backend->putIfAbsent(extra_key, "residual").outcome, PutOutcome::Done); + seedObject(*backend, extra_key, "residual"); backend->clearLog(); try @@ -159,7 +180,7 @@ void expectCatalogResidueRefusesWithoutPoolMeta(const String & bytes, const Stri EXPECT_EQ(e.code(), DB::ErrorCodes::INVALID_STATE); } EXPECT_EQ(backend->writeCount(), 0u); - EXPECT_FALSE(backend->head(layout.poolMetaKey()).exists); + EXPECT_FALSE(headPresent(*backend, layout.poolMetaKey())); } /// Index of the first op matching `pred`, if any. @@ -201,7 +222,7 @@ TEST(CASBootstrapOrdering, EmptyPrefixOpensAndListsBeforeAnyWrite) PoolPtr store = Pool::open(backend, makeConfig()); ASSERT_EQ(store->lifecycle(), PoolLifecycle::Live); - EXPECT_TRUE(backend->get(kPoolMetaKey).has_value()) << "_pool_meta must be created on a fresh empty prefix"; + EXPECT_TRUE(readPresent(*backend, kPoolMetaKey)) << "_pool_meta must be created on a fresh empty prefix"; const auto log = backend->snapshot(); const auto residual_list = firstIndex(log, [](const RecordingBackend::Entry & e) @@ -232,15 +253,14 @@ TEST(CASBootstrapOrdering, ResidualWithoutMetaFailsTypedWithZeroWrites) { auto backend = std::make_shared(); /// Seed residue an incomplete erase would have left behind (a ref-log object), with no `_pool_meta`. - ASSERT_EQ(backend->putIfAbsent(residualRefLogKey(), "x").outcome, - PutOutcome::Done); + seedObject(*backend, residualRefLogKey(), "x"); backend->clearLog(); expectThrowsCodeContaining(DB::ErrorCodes::INVALID_STATE, "refusing to bootstrap over residual data", [&] { Pool::open(backend, makeConfig()); }); EXPECT_EQ(backend->writeCount(), 0u) << "the fail path must perform zero writes (battery never ran)"; - EXPECT_FALSE(backend->get(kPoolMetaKey).has_value()) << "a fresh _pool_meta must NOT have been minted"; + EXPECT_FALSE(readPresent(*backend, kPoolMetaKey)) << "a fresh _pool_meta must NOT have been minted"; } /// (c) A prefix containing ONLY stale, structurally-valid `_probe//…` debris (a crash-mid-battery @@ -249,27 +269,27 @@ TEST(CASBootstrapOrdering, ResidualWithoutMetaFailsTypedWithZeroWrites) TEST(CASBootstrapOrdering, StaleProbeDebrisOnlyIsTreatedAsEmpty) { auto backend = std::make_shared(); - ASSERT_EQ(backend->putIfAbsent("p/_probe/" + kProbeUid + "/token", "probe-v1").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent("p/_probe/" + kProbeUid + "/cas", "cas-s1").outcome, PutOutcome::Done); + seedObject(*backend, "p/_probe/" + kProbeUid + "/token", "probe-v1"); + seedObject(*backend, "p/_probe/" + kProbeUid + "/cas", "cas-s1"); backend->clearLog(); PoolPtr store; ASSERT_NO_THROW(store = Pool::open(backend, makeConfig())); EXPECT_EQ(store->lifecycle(), PoolLifecycle::Live); - EXPECT_TRUE(backend->get(kPoolMetaKey).has_value()) << "_pool_meta must be created over a probe-only prefix"; + EXPECT_TRUE(readPresent(*backend, kPoolMetaKey)) << "_pool_meta must be created over a probe-only prefix"; } TEST(CASBootstrapOrdering, CanonicalEmptyCatalogOnlyIsTheSoleRetryablePreMetaResidue) { auto backend = std::make_shared(); const Layout layout{kPrefix}; - ASSERT_EQ(backend->putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{})).outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(kPrefix + "/_probe/" + kProbeUid + "/token", "probe-v1").outcome, PutOutcome::Done); + seedObject(*backend, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{})); + seedObject(*backend, kPrefix + "/_probe/" + kProbeUid + "/token", "probe-v1"); backend->clearLog(); PoolPtr store; ASSERT_NO_THROW(store = Pool::open(backend, makeConfig())); - EXPECT_TRUE(backend->head(layout.poolMetaKey()).exists); + EXPECT_TRUE(headPresent(*backend, layout.poolMetaKey())); } TEST(CASBootstrapOrdering, MalformedCatalogOnlyResidueRefusesWithoutPoolMeta) @@ -309,11 +329,11 @@ TEST(CASBootstrapOrdering, ListedCatalogMissingAtExactGetRefusesWithoutPoolMeta) { auto backend = std::make_shared(); const Layout layout{kPrefix}; - ASSERT_EQ(backend->putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(RefCatalog{})).outcome, PutOutcome::Done); + seedObject(*backend, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{})); expectThrowsCodeContaining(DB::ErrorCodes::INVALID_STATE, "refusing to bootstrap over residual data", [&] { Pool::open(backend, makeConfig()); }); - EXPECT_FALSE(backend->head(layout.poolMetaKey()).exists); + EXPECT_FALSE(headPresent(*backend, layout.poolMetaKey())); } /// (d) An existing healthy pool (meta present + data) → reopen is unchanged: the pool identity is @@ -342,9 +362,9 @@ TEST(CASBootstrapOrdering, ConcurrentOpenerProbeDebrisIsAlsoSkipped) { auto backend = std::make_shared(); /// This mount's own crashed battery AND a concurrent opener's in-flight battery. - ASSERT_EQ(backend->putIfAbsent("p/_probe/" + kProbeUid + "/token", "probe-v1").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent("p/_probe/" + kProbeUid2 + "/token", "probe-v1").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent("p/_probe/" + kProbeUid2 + "/cas", "cas-s1").outcome, PutOutcome::Done); + seedObject(*backend, "p/_probe/" + kProbeUid + "/token", "probe-v1"); + seedObject(*backend, "p/_probe/" + kProbeUid2 + "/token", "probe-v1"); + seedObject(*backend, "p/_probe/" + kProbeUid2 + "/cas", "cas-s1"); backend->clearLog(); PoolPtr store; @@ -360,13 +380,13 @@ TEST(CASBootstrapOrdering, ConcurrentOpenerProbeDebrisIsAlsoSkipped) TEST(CASBootstrapOrdering, ProbeSiblingLookalikeIsResidualNotDebris) { auto backend = std::make_shared(); - ASSERT_EQ(backend->putIfAbsent("p/_probelike/token", "x").outcome, PutOutcome::Done); + seedObject(*backend, "p/_probelike/token", "x"); backend->clearLog(); expectThrowsCodeContaining(DB::ErrorCodes::INVALID_STATE, "refusing to bootstrap over residual data", [&] { Pool::open(backend, makeConfig()); }); EXPECT_EQ(backend->writeCount(), 0u); - EXPECT_FALSE(backend->get(kPoolMetaKey).has_value()); + EXPECT_FALSE(readPresent(*backend, kPoolMetaKey)); } /// (g) An OBSERVE / read-only open over a partially-erased pool (residual data, `_pool_meta` deleted) @@ -377,8 +397,7 @@ TEST(CASBootstrapOrdering, ProbeSiblingLookalikeIsResidualNotDebris) TEST(CASBootstrapOrdering, ReadOnlyOverResidualWithoutMetaFailsClosedNoMint) { auto backend = std::make_shared(); - ASSERT_EQ(backend->putIfAbsent(residualRefLogKey(), "x").outcome, - PutOutcome::Done); + seedObject(*backend, residualRefLogKey(), "x"); backend->clearLog(); PoolConfig cfg = makeConfig(); @@ -387,7 +406,7 @@ TEST(CASBootstrapOrdering, ReadOnlyOverResidualWithoutMetaFailsClosedNoMint) [&] { Pool::open(backend, cfg); }); EXPECT_EQ(backend->writeCount(), 0u) << "an observe open must never write (least of all mint _pool_meta)"; - EXPECT_FALSE(backend->get(kPoolMetaKey).has_value()); + EXPECT_FALSE(readPresent(*backend, kPoolMetaKey)); } /// (h) An observe / read-only open over a HEALTHY pool (meta present) is unchanged: it validates the @@ -421,9 +440,10 @@ TEST(CASBootstrapOrdering, DecommissionWithAbsentMetaFailsClosedNoMint) } /// Delete only `_pool_meta`, leaving the owner anchor (and other control objects) behind. { - const auto h = backend->head(kPoolMetaKey); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->deleteExact(kPoolMetaKey, h.token).kind, DeleteOutcome::Kind::Deleted); + DB::Cas::tests::OperationForTest op(*backend); + const auto h = (*op).head(kPoolMetaKey, Retry::standard()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ((*op).remove(kPoolMetaKey, h->etag, Retry::once()), Removal::Removed); } backend->clearLog(); @@ -431,5 +451,5 @@ TEST(CASBootstrapOrdering, DecommissionWithAbsentMetaFailsClosedNoMint) [&] { Pool::openForDecommission(backend, makeConfig(), kSrid); }); EXPECT_EQ(backend->writeCount(), 0u) << "decommission must not mint a fresh _pool_meta"; - EXPECT_FALSE(backend->get(kPoolMetaKey).has_value()); + EXPECT_FALSE(readPresent(*backend, kPoolMetaKey)); } diff --git a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp index b2c1542c6017..46f970ec64ed 100644 --- a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp +++ b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp @@ -12,7 +12,7 @@ #include #include #include -#include +#include #include #include #include @@ -166,8 +166,6 @@ class LatchedChunkFaultBackend : public DB::Cas::tests::ChunkFaultBackend class RecoveryLatchBackend : public CountingBackend { public: - using CountingBackend::getStream; - /// Set before the driving call; consumed by the first matching recovery read. String fail_get_once_key; @@ -756,7 +754,6 @@ TEST(CASConfirmExactRef, WedgedTransactionRefusesEveryRef) /// for the whole call while the injected clock below carries the call to its own deadline. CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; auto store = openPoolWithConfig(backend, cfg); diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index aa562738649a..3a3033e2bae7 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -86,8 +86,8 @@ class CatalogChangesAfterFirstReadBackend : public InMemoryBackend bool fired() const { return replacement_fired; } - /// Injects on the `read` PRIMITIVE rather than the legacy `get`: whichever caller reaches this - /// key -- through the legacy forwarder or through `CasOperation::read` -- funnels through here. + /// Injects on the `read` PRIMITIVE: every caller reaches this key through `CasOperation::read`, + /// which funnels through here. std::optional read(const String & key, TransportAccess & access) override { auto got = InMemoryBackend::read(key, access); @@ -116,20 +116,22 @@ class CatalogChangesAfterFirstReadBackend : public InMemoryBackend bool replacement_fired = false; }; -std::vector> snapshotPrefixObjects( +std::vector> snapshotPrefixObjects( InMemoryBackend & backend, const String & prefix) { - std::vector> objects; + std::vector> objects; + auto requests = openRequestsForTest(backend); + auto op = requests.admit(); String cursor; while (true) { - const ListPage page = backend.list(prefix, cursor, 1000); + const ListPage page = op.list(prefix, cursor, 1000, Retry::once()); for (const ListedKey & listed : page.keys) { - const auto got = backend.get(listed.key); + const auto got = op.read(listed.key, Retry::once()); if (!got) throw std::runtime_error("prefix snapshot fixture: listed object disappeared"); - objects.emplace_back(listed.key, got->bytes, got->token); + objects.emplace_back(listed.key, got->bytes, got->etag); } if (page.next_cursor.empty()) return objects; @@ -175,17 +177,17 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend } bool successorInjected() const { return successor_injected; } - const Token & successorMountToken() const { return successor_mount_token; } - const Token & successorEpochToken() const { return successor_epoch_token; } + const String & successorMountValue() const { return successor_mount_value; } + const String & successorEpochValue() const { return successor_epoch_value; } const String & successorMountBytes() const { return successor_mount_bytes; } const String & successorEpochBytes() const { return successor_epoch_bytes; } private: - /// Every request here is issued on a PRIMITIVE. A legacy verb would be re-dispatched through the - /// virtual primitive it forwards to -- `get` through `read` -- which is this very hook, so the - /// injection would re-enter itself with `successor_injected` still false. The two `Token`s the - /// tests compare against are minted through `Backend`'s own minter, the one the legacy forwarders - /// use, so they are the values a legacy caller would have received. + /// Every request here is issued on the PRIMITIVE `write`, using the `access` token the caller's + /// own in-flight request already holds -- no `CasRequests`/`CasOperation` exists inside this + /// reentrant hook to mint one. The captured `successor_*_value` fields are the raw wire values + /// `write` returned, which the tests compare against a real incarnation's rendered value + /// (`PersistedEtag::capture`) taken through their own `CasOperation`. void injectSuccessor(TransportAccess & access) { const auto epoch = InMemoryBackend::read(epoch_key, access); @@ -200,7 +202,7 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend const auto epoch_written = InMemoryBackend::write(epoch_key, successor_epoch_bytes, epoch->value, access); if (!epoch_written) throw std::runtime_error("successor-reclaim fixture: epoch bump conflicted"); - successor_epoch_token = legacyMintWritten(epoch_key, *epoch_written); + successor_epoch_value = *epoch_written; MountLease mount_value = decodeMountLease(mount->bytes); mount_value.writer_epoch = successor_writer_epoch; @@ -213,7 +215,7 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend const auto mount_written = InMemoryBackend::write(mount_key, successor_mount_bytes, mount->value, access); if (!mount_written) throw std::runtime_error("successor-reclaim fixture: mount reclaim conflicted"); - successor_mount_token = legacyMintWritten(mount_key, *mount_written); + successor_mount_value = *mount_written; successor_injected = true; } @@ -222,8 +224,8 @@ class SuccessorReclaimAfterFarewellBackend : public InMemoryBackend bool armed = false; bool farewell_seen = false; bool successor_injected = false; - Token successor_mount_token; - Token successor_epoch_token; + String successor_mount_value; + String successor_epoch_value; String successor_mount_bytes; String successor_epoch_bytes; }; @@ -248,8 +250,8 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend bool successorInjected() const { return successor_injected; } uint64_t ownerRewriteAttempts() const { return owner_rewrite_attempts; } - const Token & successorMountToken() const { return successor_mount_token; } - const Token & successorEpochToken() const { return successor_epoch_token; } + const String & successorMountValue() const { return successor_mount_value; } + const String & successorEpochValue() const { return successor_epoch_value; } const String & successorMountBytes() const { return successor_mount_bytes; } const String & successorEpochBytes() const { return successor_epoch_bytes; } @@ -268,16 +270,16 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend return InMemoryBackend::write(key, bytes, expected_value, access); } - /// On the `write` PRIMITIVE, for the reason the sibling fixture above states: a legacy verb is - /// re-dispatched through the virtual primitive it forwards to, so it would re-enter this class's - /// own overrides instead of reaching the store directly. + /// On the `write` PRIMITIVE, using the caller's own in-flight `access`: this hook is reentrant + /// (called from inside another primitive override on the same object), so it reaches the store + /// directly through `InMemoryBackend::write` rather than admitting a fresh request of its own. void injectSuccessor(TransportAccess & access) { successor_epoch_bytes = encodeServerEpoch(ServerEpoch{.next_writer_epoch = 102}); const auto epoch_written = InMemoryBackend::write(epoch_key, successor_epoch_bytes, std::nullopt, access); if (!epoch_written) throw std::runtime_error("late-successor fixture: epoch recreation conflicted"); - successor_epoch_token = legacyMintWritten(epoch_key, *epoch_written); + successor_epoch_value = *epoch_written; successor_mount_bytes = encodeMountLease(MountLease{ .server_uuid = UInt128(0x1234), @@ -292,7 +294,7 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend const auto mount_written = InMemoryBackend::write(mount_key, successor_mount_bytes, std::nullopt, access); if (!mount_written) throw std::runtime_error("late-successor fixture: mount recreation conflicted"); - successor_mount_token = legacyMintWritten(mount_key, *mount_written); + successor_mount_value = *mount_written; successor_injected = true; } @@ -302,8 +304,8 @@ class SuccessorReclaimAfterEpochDeleteBackend : public InMemoryBackend bool armed = false; bool successor_injected = false; uint64_t owner_rewrite_attempts = 0; - Token successor_mount_token; - Token successor_epoch_token; + String successor_mount_value; + String successor_epoch_value; String successor_mount_bytes; String successor_epoch_bytes; }; @@ -330,7 +332,7 @@ class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend const auto put = InMemoryBackend::write(owner_key, successor_owner_bytes, result->value, access); if (!put.has_value()) throw std::runtime_error("owner-successor fixture: owner rewrite conflicted"); - successor_owner_token = legacyMintWritten(owner_key, *put); + successor_owner_value = *put; successor_injected = true; } return result; @@ -347,7 +349,7 @@ class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend } bool successorInjected() const { return successor_injected; } - const Token & successorOwnerToken() const { return successor_owner_token; } + const String & successorOwnerValue() const { return successor_owner_value; } const String & successorOwnerBytes() const { return successor_owner_bytes; } private: @@ -356,7 +358,7 @@ class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend bool armed = false; bool epoch_deleted = false; bool successor_injected = false; - Token successor_owner_token; + String successor_owner_value; String successor_owner_bytes; }; @@ -371,7 +373,7 @@ class SuccessorOwnerRewriteBeforeTombstoneBackend : public InMemoryBackend void makeTableWithRefs(Pool & victim, const String & ns_str, uint64_t committed, uint64_t precommits) { const RootNamespace ns(ns_str); - Backend & backend = victim.backend(); + Backend & backend = *victim.poolBackendPtr(); const Layout & layout = victim.layout(); /// A throwaway open-fence operation, for the two `CasRefCatalog` calls below only: this fixture @@ -433,10 +435,11 @@ ManifestId seedOrphanManifestBody(Pool & victim, const String & ns_str) { const RootNamespace ns(ns_str); const ManifestRef ref{.writer_epoch = victim.writerEpoch(), .build_sequence = 99, .manifest_ordinal = 1}; - const ManifestId id = writeManifestRaw(victim.backend(), victim.layout(), ns, ref, {}); + const ManifestId id = writeManifestRaw(*victim.poolBackendPtr(), victim.layout(), ns, ref, {}); /// EXPECT, not ASSERT: this function returns a value now, and ASSERT_* expands to a bare `return;` /// -- invalid in a non-void function. - EXPECT_TRUE(victim.backend().head(victim.layout().manifestKey(id)).exists); + OperationForTest op(*victim.poolBackendPtr()); + EXPECT_TRUE((*op).head(victim.layout().manifestKey(id), Retry::once()).has_value()); return id; } @@ -531,31 +534,32 @@ TEST(CASDecommission, DuplicateLifeIdRefusesBeforeAnyNamespaceOrSlotMutation) .incarnation = UInt128{77}, .removal_started_round = 1}, }; - const auto empty_catalog = backend->get(layout.refCatalogKey()); + OperationForTest raw_op(*backend); + const auto empty_catalog = (*raw_op).read(layout.refCatalogKey(), Retry::once()); ASSERT_TRUE(empty_catalog); - ASSERT_EQ(backend->putOverwrite( - layout.refCatalogKey(), encodeRefCatalog(catalog), empty_catalog->token).outcome, PutOutcome::Done); - const auto owner_before = backend->get(layout.ownerKey("victim")); - const auto epoch_before = backend->get(layout.epochKey("victim")); - const auto mount_before = backend->get(layout.mountKey("victim")); + ASSERT_TRUE(std::holds_alternative((*raw_op).replace( + layout.refCatalogKey(), encodeRefCatalog(catalog), empty_catalog->etag, Retry::once()))); + const auto owner_before = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); + const auto epoch_before = (*raw_op).read(layout.epochKey("victim"), Retry::once()); + const auto mount_before = (*raw_op).read(layout.mountKey("victim"), Retry::once()); ASSERT_TRUE(owner_before); ASSERT_TRUE(epoch_before); ASSERT_TRUE(mount_before); EXPECT_THROW(decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"), DB::Exception); - const auto owner_after = backend->get(layout.ownerKey("victim")); - const auto epoch_after = backend->get(layout.epochKey("victim")); - const auto mount_after = backend->get(layout.mountKey("victim")); + const auto owner_after = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); + const auto epoch_after = (*raw_op).read(layout.epochKey("victim"), Retry::once()); + const auto mount_after = (*raw_op).read(layout.mountKey("victim"), Retry::once()); ASSERT_TRUE(owner_after); ASSERT_TRUE(epoch_after); ASSERT_TRUE(mount_after); EXPECT_EQ(owner_after->bytes, owner_before->bytes); - EXPECT_EQ(owner_after->token, owner_before->token); + EXPECT_EQ(owner_after->etag, owner_before->etag); EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); - EXPECT_EQ(epoch_after->token, epoch_before->token); + EXPECT_EQ(epoch_after->etag, epoch_before->etag); EXPECT_EQ(mount_after->bytes, mount_before->bytes); - EXPECT_EQ(mount_after->token, mount_before->token); + EXPECT_EQ(mount_after->etag, mount_before->etag); } TEST(CASDecommission, CatalogCutIsValidatedBeforeImpersonationAndReusedForSelection) @@ -574,9 +578,10 @@ TEST(CASDecommission, CatalogCutIsValidatedBeforeImpersonationAndReusedForSelect .removal_started_round = 1}, }; - const auto owner_before = backend->get(layout.ownerKey("victim")); - const auto epoch_before = backend->get(layout.epochKey("victim")); - const auto mount_before = backend->get(layout.mountKey("victim")); + OperationForTest raw_op(*backend); + const auto owner_before = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); + const auto epoch_before = (*raw_op).read(layout.epochKey("victim"), Retry::once()); + const auto mount_before = (*raw_op).read(layout.mountKey("victim"), Retry::once()); ASSERT_TRUE(owner_before); ASSERT_TRUE(epoch_before); ASSERT_TRUE(mount_before); @@ -586,18 +591,18 @@ TEST(CASDecommission, CatalogCutIsValidatedBeforeImpersonationAndReusedForSelect backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"), DB::Exception); ASSERT_TRUE(backend->fired()); - const auto owner_after = backend->get(layout.ownerKey("victim")); - const auto epoch_after = backend->get(layout.epochKey("victim")); - const auto mount_after = backend->get(layout.mountKey("victim")); + const auto owner_after = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); + const auto epoch_after = (*raw_op).read(layout.epochKey("victim"), Retry::once()); + const auto mount_after = (*raw_op).read(layout.mountKey("victim"), Retry::once()); ASSERT_TRUE(owner_after); ASSERT_TRUE(epoch_after); ASSERT_TRUE(mount_after); EXPECT_EQ(owner_after->bytes, owner_before->bytes); - EXPECT_EQ(owner_after->token, owner_before->token); + EXPECT_EQ(owner_after->etag, owner_before->etag); EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); - EXPECT_EQ(epoch_after->token, epoch_before->token); + EXPECT_EQ(epoch_after->etag, epoch_before->etag); EXPECT_EQ(mount_after->bytes, mount_before->bytes); - EXPECT_EQ(mount_after->token, mount_before->token); + EXPECT_EQ(mount_after->etag, mount_before->etag); } TEST(CASDecommission, NamespaceSelectionUsesThePreImpersonationCut) @@ -633,28 +638,26 @@ TEST(CASDecommission, SameNameRebirthAfterTheCutIsRefusedWithoutTouchingTheNewLi old_catalog.entries = { CatalogEntry{.ns = ns, .state = NsState::Live, .incarnation = old_life.incarnation}, }; - const auto empty_catalog = backend->get(layout.refCatalogKey()); + OperationForTest raw_op(*backend); + const auto empty_catalog = (*raw_op).read(layout.refCatalogKey(), Retry::once()); ASSERT_TRUE(empty_catalog); - ASSERT_EQ(backend->putOverwrite( - layout.refCatalogKey(), encodeRefCatalog(old_catalog), empty_catalog->token).outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative((*raw_op).replace( + layout.refCatalogKey(), encodeRefCatalog(old_catalog), empty_catalog->etag, Retry::once()))); RefLogTxn new_birth; new_birth.ns = ns.string(); new_birth.txn_id = RefTxnId{1, 1}; new_birth.ops = {namespaceBirthOp()}; - ASSERT_EQ(backend->putIfAbsent( + ASSERT_TRUE(std::holds_alternative((*raw_op).create( layout.refLogKey(new_life, new_birth.txn_id), - sealObject(FormatId::RefLog, encodeRefLogTxn(new_birth))).outcome, - PutOutcome::Done); + sealObject(FormatId::RefLog, encodeRefLogTxn(new_birth)), Retry::once()))); RefLogTxn new_seal; new_seal.ns = ns.string(); new_seal.txn_id = RefTxnId{1, 2}; new_seal.ops = {epochSealOp()}; - ASSERT_EQ(backend->putIfAbsent( + ASSERT_TRUE(std::holds_alternative((*raw_op).create( layout.refLogKey(new_life, new_seal.txn_id), - sealObject(FormatId::RefLog, encodeRefLogTxn(new_seal))).outcome, - PutOutcome::Done); + sealObject(FormatId::RefLog, encodeRefLogTxn(new_seal)), Retry::once()))); const auto new_life_before = snapshotPrefixObjects(*backend, layout.namespaceStreamPrefix(new_life)); RefCatalog replacement; @@ -761,8 +764,8 @@ TEST(CASDecommission, CountsRealisticEpochPrecommit) /// -- a REAL build's `ManifestRef` is unique per build, and a colliding one would trip the ref /// state machine's "manifest already has a conflicting owner" guard. const ManifestRef ref{.writer_epoch = victim_epoch, .build_sequence = 2, .manifest_ordinal = 1}; - writeManifestRaw(victim->backend(), victim->layout(), ns, ref, {}); - addPrecommitTransition(victim->backend(), victim->layout(), ns, UInt128(1), "precommit_0", std::nullopt, ref); + writeManifestRaw(*victim->poolBackendPtr(), victim->layout(), ns, ref, {}); + addPrecommitTransition(*victim->poolBackendPtr(), victim->layout(), ns, UInt128(1), "precommit_0", std::nullopt, ref); } const auto report = decommissionPoolMember( @@ -817,9 +820,12 @@ TEST(CASDecommission, DrainsDebrisStagingAndRoots) /// Foreign staging + mountpoint objects, written raw (no writer machinery needed): the victim's /// writers are fenced by the claim before decommission ever gets here, so these are ordinary debris, /// not a live in-flight write. - backend->putIfAbsent("p/staging/victim/upload1.tmp", "x"); - backend->putIfAbsent("p/staging/victim/upload2.tmp", "x"); - backend->putIfAbsent("p/roots/victim/clickhouse_access_check_abc", "x"); + { + OperationForTest seed_op(*backend); + (*seed_op).create("p/staging/victim/upload1.tmp", "x", Retry::once()); + (*seed_op).create("p/staging/victim/upload2.tmp", "x", Retry::once()); + (*seed_op).create("p/roots/victim/clickhouse_access_check_abc", "x", Retry::once()); + } const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); @@ -835,8 +841,9 @@ TEST(CASDecommission, DrainsDebrisStagingAndRoots) /// Nothing of the victim remains under staging/ or roots/ (scoped LISTs are empty). Those two phases /// run to completion even though the debris phase retained -- the drain is per-phase, not all-or-nothing. - EXPECT_TRUE(backend->list("p/staging/victim/", "", 10).keys.empty()); - EXPECT_TRUE(backend->list("p/roots/victim/", "", 10).keys.empty()); + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).list("p/staging/victim/", "", 10, Retry::once()).keys.empty()); + EXPECT_TRUE((*raw_op).list("p/roots/victim/", "", 10, Retry::once()).keys.empty()); } /// The §6 deletion premise applies to the decommission drain too, and this pins what that COSTS. With no @@ -866,7 +873,8 @@ TEST(CASDecommission, RetainsDebrisWhoseEpochSealIsUnconsumed) backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); EXPECT_EQ(report.manifest_debris_removed, 0u); - EXPECT_TRUE(backend->head(debris_key).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()) << "the body is retained untouched, not deleted and not corrupted"; ASSERT_FALSE(report.warnings.empty()) << "a retained manifest is a visible decision -- the operator must be able to see why the drain " @@ -892,9 +900,12 @@ TEST(CASDecommission, PerObjectFailureWarnsAndContinuesDrain) auto victim = openVictim(backend); makeTableWithRefs(*victim, "victim/db/t1", 1, 0); } - backend->putIfAbsent("p/staging/victim/upload_ok.tmp", "x"); - backend->putIfAbsent("p/staging/victim/upload_throws.tmp", "x"); - backend->putIfAbsent("p/roots/victim/clickhouse_access_check_abc", "x"); + { + OperationForTest seed_op(*backend); + (*seed_op).create("p/staging/victim/upload_ok.tmp", "x", Retry::once()); + (*seed_op).create("p/staging/victim/upload_throws.tmp", "x", Retry::once()); + (*seed_op).create("p/roots/victim/clickhouse_access_check_abc", "x", Retry::once()); + } backend->failWithThrow("p/staging/victim/upload_throws.tmp"); backend->failWithTokenMismatch("p/roots/victim/clickhouse_access_check_abc"); @@ -907,11 +918,12 @@ TEST(CASDecommission, PerObjectFailureWarnsAndContinuesDrain) EXPECT_EQ(report.warnings.size(), 2u) << "one warning for the thrown exception, one for the TokenMismatch outcome"; - EXPECT_FALSE(backend->head("p/staging/victim/upload_ok.tmp").exists) + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head("p/staging/victim/upload_ok.tmp", Retry::once()).has_value()) << "the healthy staging object was actually deleted, not merely skipped"; - EXPECT_TRUE(backend->head("p/staging/victim/upload_throws.tmp").exists) + EXPECT_TRUE((*raw_op).head("p/staging/victim/upload_throws.tmp", Retry::once()).has_value()) << "the failing object is left behind (untouched) so a re-run can retry it"; - EXPECT_TRUE(backend->head("p/roots/victim/clickhouse_access_check_abc").exists) + EXPECT_TRUE((*raw_op).head("p/roots/victim/clickhouse_access_check_abc", Retry::once()).has_value()) << "TokenMismatch means nothing was actually deleted -- the object survives"; } @@ -927,13 +939,15 @@ TEST(CASDecommission, LifelessPhysicalKeyCannotRedirectCatalogOwnedDecommission) /// Hand-built: no helper can mint the un-incarnated shape any more. lifeless = victim->layout().casRefsPrefix() + String("victim/db/t1/_log/") + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; - ASSERT_EQ(backend->putIfAbsent(lifeless, "garbage").outcome, PutOutcome::Done); + OperationForTest seed_op(*backend); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(lifeless, "garbage", Retry::once()))); } const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); EXPECT_EQ(report.namespaces_removed, 1u); - EXPECT_TRUE(backend->head(lifeless).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(lifeless, Retry::once()).has_value()) << "decommission must neither adopt nor delete an unowned physical life key"; } @@ -952,7 +966,10 @@ TEST(CASDecommission, ManifestDebrisDeleteFailureWarnsAndContinues) debris_key = victim->layout().manifestKey(debris_id); } backend->failWithThrow(debris_key); - backend->putIfAbsent("p/staging/victim/upload_ok.tmp", "x"); + { + OperationForTest seed_op(*backend); + (*seed_op).create("p/staging/victim/upload_ok.tmp", "x", Retry::once()); + } const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); @@ -966,7 +983,8 @@ TEST(CASDecommission, ManifestDebrisDeleteFailureWarnsAndContinues) << "the staging phase still ran to completion after the manifest-debris phase's failures -- " "the whole command did not abort"; - EXPECT_TRUE(backend->head(debris_key).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()) << "the failing object is left behind (untouched) so a re-run can retry it"; } @@ -988,11 +1006,12 @@ TEST(CASDecommission, RemovesMutableSlotAndRefusesTombstonedRerun) backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); EXPECT_TRUE(report.slot_removed); EXPECT_TRUE(report.warnings.empty()); - EXPECT_FALSE(backend->get("p/gc/server-roots/victim/mount").has_value()); - const auto owner = backend->get("p/gc/server-roots/victim/owner"); + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()); + const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); ASSERT_TRUE(owner.has_value()); EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); - EXPECT_FALSE(backend->get("p/gc/server-roots/victim/epoch").has_value()); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/epoch", Retry::once()).has_value()); expectThrowsCode(ErrorCodes::CORRUPTED_DATA, [&] { @@ -1021,16 +1040,17 @@ TEST(CASDecommission, SuccessorReclaimFencesSlotRetirementTail) EXPECT_NE(report.warnings.front().find("p/gc/server-roots/victim/mount"), String::npos); EXPECT_NE(report.warnings.front().find("mismatch"), String::npos); - const auto mount = backend->get("p/gc/server-roots/victim/mount"); + OperationForTest raw_op(*backend); + const auto mount = (*raw_op).read("p/gc/server-roots/victim/mount", Retry::once()); ASSERT_TRUE(mount.has_value()); - EXPECT_EQ(mount->token, backend->successorMountToken()); + EXPECT_EQ(PersistedEtag::capture(mount->etag).value, backend->successorMountValue()); EXPECT_EQ(mount->bytes, backend->successorMountBytes()); - const auto epoch = backend->get("p/gc/server-roots/victim/epoch"); + const auto epoch = (*raw_op).read("p/gc/server-roots/victim/epoch", Retry::once()); ASSERT_TRUE(epoch.has_value()); - EXPECT_EQ(epoch->token, backend->successorEpochToken()); + EXPECT_EQ(PersistedEtag::capture(epoch->etag).value, backend->successorEpochValue()); EXPECT_EQ(epoch->bytes, backend->successorEpochBytes()); - EXPECT_TRUE(backend->get("p/gc/server-roots/victim/owner").has_value()); + EXPECT_TRUE((*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()).has_value()); ASSERT_FALSE(seen.empty()); EXPECT_EQ(seen.back().outcome, "end"); @@ -1045,7 +1065,8 @@ TEST(CASDecommission, SuccessorReclaimAfterEpochDeleteKeepsOwnerAnchor) { auto victim = openVictim(backend); } const String owner_key = "p/gc/server-roots/victim/owner"; - const auto original_owner = backend->get(owner_key); + OperationForTest raw_op(*backend); + const auto original_owner = (*raw_op).read(owner_key, Retry::once()); ASSERT_TRUE(original_owner.has_value()); backend->armForSuccessorReclaim(); @@ -1057,19 +1078,19 @@ TEST(CASDecommission, SuccessorReclaimAfterEpochDeleteKeepsOwnerAnchor) EXPECT_FALSE(report.warnings.empty()); EXPECT_EQ(backend->ownerRewriteAttempts(), 0u); - const auto owner = backend->get(owner_key); + const auto owner = (*raw_op).read(owner_key, Retry::once()); ASSERT_TRUE(owner.has_value()); - EXPECT_EQ(owner->token, original_owner->token); + EXPECT_EQ(owner->etag, original_owner->etag); EXPECT_EQ(owner->bytes, original_owner->bytes); - const auto mount = backend->get("p/gc/server-roots/victim/mount"); + const auto mount = (*raw_op).read("p/gc/server-roots/victim/mount", Retry::once()); ASSERT_TRUE(mount.has_value()); - EXPECT_EQ(mount->token, backend->successorMountToken()); + EXPECT_EQ(PersistedEtag::capture(mount->etag).value, backend->successorMountValue()); EXPECT_EQ(mount->bytes, backend->successorMountBytes()); - const auto epoch = backend->get("p/gc/server-roots/victim/epoch"); + const auto epoch = (*raw_op).read("p/gc/server-roots/victim/epoch", Retry::once()); ASSERT_TRUE(epoch.has_value()); - EXPECT_EQ(epoch->token, backend->successorEpochToken()); + EXPECT_EQ(PersistedEtag::capture(epoch->etag).value, backend->successorEpochValue()); EXPECT_EQ(epoch->bytes, backend->successorEpochBytes()); } @@ -1085,9 +1106,10 @@ TEST(CASDecommission, FencedSlotRetirementTailRetiresUncontendedSlot) EXPECT_TRUE(report.warnings.empty()); EXPECT_TRUE(report.slot_removed); - EXPECT_FALSE(backend->get("p/gc/server-roots/victim/mount").has_value()); - EXPECT_FALSE(backend->get("p/gc/server-roots/victim/epoch").has_value()); - const auto owner = backend->get("p/gc/server-roots/victim/owner"); + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/epoch", Retry::once()).has_value()); + const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); ASSERT_TRUE(owner.has_value()); EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); } @@ -1119,9 +1141,10 @@ TEST(CASDecommission, RunsOnAnOpenFence) backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin2"}, "victim"); EXPECT_TRUE(report.warnings.empty()); EXPECT_TRUE(report.slot_removed); - EXPECT_FALSE(backend->get("p/gc/server-roots/victim/mount").has_value()); - EXPECT_FALSE(backend->get("p/gc/server-roots/victim/epoch").has_value()); - const auto owner = backend->get("p/gc/server-roots/victim/owner"); + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/epoch", Retry::once()).has_value()); + const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); ASSERT_TRUE(owner.has_value()); EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); } @@ -1132,7 +1155,8 @@ TEST(CASDecommission, SuccessfulDecommissionLeavesTombstonedOwnerAnchor) { auto victim = openVictim(backend); } const String owner_key = "p/gc/server-roots/victim/owner"; - const auto before = backend->get(owner_key); + OperationForTest raw_op(*backend); + const auto before = (*raw_op).read(owner_key, Retry::once()); ASSERT_TRUE(before.has_value()); EXPECT_FALSE(decodeOwner(before->bytes).retired_at_ms.has_value()); @@ -1141,9 +1165,9 @@ TEST(CASDecommission, SuccessfulDecommissionLeavesTombstonedOwnerAnchor) EXPECT_TRUE(report.warnings.empty()); EXPECT_TRUE(report.slot_removed); - const auto after = backend->get(owner_key); + const auto after = (*raw_op).read(owner_key, Retry::once()); ASSERT_TRUE(after.has_value()); - EXPECT_NE(after->token, before->token); + EXPECT_NE(after->etag, before->etag); EXPECT_EQ(decodeOwner(after->bytes).server_uuid, decodeOwner(before->bytes).server_uuid); EXPECT_TRUE(decodeOwner(after->bytes).retired_at_ms.has_value()); } @@ -1162,9 +1186,10 @@ TEST(CASDecommission, SuccessorOwnerRewriteWinsBeforeTombstone) ASSERT_EQ(report.warnings.size(), 1u); EXPECT_NE(report.warnings.front().find("successor reclaimed"), String::npos); - const auto owner = backend->get("p/gc/server-roots/victim/owner"); + OperationForTest raw_op(*backend); + const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); ASSERT_TRUE(owner.has_value()); - EXPECT_EQ(owner->token, backend->successorOwnerToken()); + EXPECT_EQ(PersistedEtag::capture(owner->etag).value, backend->successorOwnerValue()); EXPECT_EQ(owner->bytes, backend->successorOwnerBytes()); EXPECT_FALSE(decodeOwner(owner->bytes).retired_at_ms.has_value()); } @@ -1189,7 +1214,8 @@ TEST(CASDecommission, OwnerTombstoneAmbiguousSuccessResolvesToCommitted) EXPECT_TRUE(report.slot_removed) << "the ambiguous write actually landed and must resolve to Committed"; EXPECT_TRUE(report.warnings.empty()); - const auto owner = backend->get("p/gc/server-roots/victim/owner"); + OperationForTest raw_op(*backend); + const auto owner = (*raw_op).read("p/gc/server-roots/victim/owner", Retry::once()); ASSERT_TRUE(owner.has_value()); EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); } @@ -1203,12 +1229,6 @@ TEST(CASDecommission, OwnerTombstoneAmbiguousSuccessResolvesToCommitted) class FailDeletesUnderPrefixBackend : public Backend { public: - using Backend::get; - using Backend::getStream; - using Backend::putIfAbsent; - using Backend::putOverwrite; - using Backend::casPut; - FailDeletesUnderPrefixBackend(std::shared_ptr inner_, String fail_prefix_) : inner(std::move(inner_)), fail_prefix(std::move(fail_prefix_)) { @@ -1216,26 +1236,6 @@ class FailDeletesUnderPrefixBackend : public Backend void disarm() { armed = false; } - std::optional get(const String & key, Range range) override { return inner->get(key, range); } - std::optional getStream(const String & key, Range range) override { return inner->getStream(key, range); } - HeadResult head(const String & key) override { return inner->head(key); } - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - return inner->putIfAbsent(key, bytes, meta); - } - void publishBlob(const BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override - { - return inner->putOverwrite(key, bytes, expected, meta); - } - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, const ObjectMeta & meta) override - { - return inner->casPut(key, bytes, expected, meta); - } - ListPage list(const String & prefix, const String & cursor, size_t limit) override { return inner->list(prefix, cursor, limit); } bool supportsListTokens() const override { return inner->supportsListTokens(); } /// The transport primitives forward to `inner`, except `remove`, which is what this double @@ -1279,14 +1279,15 @@ TEST(CASDecommission, FailedDrainKeepsSlotThenResumes) auto victim = Pool::open(inner, PoolConfig{.pool_prefix = "p", .server_root_id = "victim"}); makeTableWithRefs(*victim, "victim/db/t1", 1, 0); } - inner->putIfAbsent("p/roots/victim/loose_file", "x"); + OperationForTest raw_op(*inner); + (*raw_op).create("p/roots/victim/loose_file", "x", Retry::once()); auto failing = std::make_shared(inner, "p/roots/victim/"); const auto first = decommissionPoolMember( failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim"); EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); - EXPECT_TRUE(inner->get("p/gc/server-roots/victim/mount").has_value()) + EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) << "slot kept -- resume anchor"; failing->disarm(); @@ -1325,9 +1326,10 @@ TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); EXPECT_EQ(first.manifest_debris_removed, 0u); - EXPECT_TRUE(backend->get("p/gc/server-roots/victim/mount").has_value()) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) << "slot kept -- resume anchor"; - EXPECT_TRUE(backend->head(debris_key).exists) + EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()) << "the failing object is left behind (untouched) so a re-run can retry it"; /// COVERAGE LOST HERE, DELIBERATELY NAMED. Before the §6 premise, clearing the injected failure let @@ -1343,8 +1345,8 @@ TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) EXPECT_EQ(second.namespaces_already_removed, 1u); EXPECT_EQ(second.manifest_debris_removed, 0u); EXPECT_FALSE(second.slot_removed); - EXPECT_TRUE(backend->head(debris_key).exists); - EXPECT_TRUE(backend->get("p/gc/server-roots/victim/mount").has_value()) + EXPECT_TRUE((*raw_op).head(debris_key, Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) << "the slot is still the resume anchor -- nothing was retired against unreclaimed debris"; } @@ -1362,7 +1364,7 @@ TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) /// therefore uses a victim with NO namespaces at all: identity persisted /// (mount/owner/epoch exist from a real graceful close), data subtree genuinely empty -- the exact /// precondition the fallback is designed for. Simulate the crash directly: claim the slot once (exactly -/// `decommissionPoolMember`'s own first step), let it close gracefully (the mount-lease keeper's +/// `decommissionPoolMember`'s own first step), let it close gracefully (the mount-lease renewer's /// farewell stamp, same as a real `admin.reset()`), then manually strike `epochKey`+`ownerKey`, leaving /// `mountKey`. A `decommissionPoolMember` re-run must resolve identity via the mount-lease fallback and /// finish retiring the slot; a further re-run then sees the tombstone and refuses to resume it. @@ -1380,15 +1382,16 @@ TEST(CASDecommission, MidRetirementCrashResumesViaMountLeaseFallback) } /// Manually strike epoch + owner, leaving the mount -- the legacy partial hand-cleanup shape. + OperationForTest raw_op(*backend); for (const String & key : {layout.epochKey("victim"), layout.ownerKey("victim")}) { - const auto head = backend->head(key); - ASSERT_TRUE(head.exists); - backend->deleteExact(key, head.token); + const auto head = (*raw_op).head(key, Retry::once()); + ASSERT_TRUE(head.has_value()); + ASSERT_EQ((*raw_op).remove(key, head->etag, Retry::once()), Removal::Removed); } - ASSERT_FALSE(backend->get(layout.epochKey("victim")).has_value()); - ASSERT_FALSE(backend->get(layout.ownerKey("victim")).has_value()); - ASSERT_TRUE(backend->get(layout.mountKey("victim")).has_value()) + ASSERT_FALSE((*raw_op).head(layout.epochKey("victim"), Retry::once()).has_value()); + ASSERT_FALSE((*raw_op).head(layout.ownerKey("victim"), Retry::once()).has_value()); + ASSERT_TRUE((*raw_op).head(layout.mountKey("victim"), Retry::once()).has_value()) << "the mount lease must survive -- it is the resume anchor the fallback reads"; const auto report = decommissionPoolMember( @@ -1397,11 +1400,11 @@ TEST(CASDecommission, MidRetirementCrashResumesViaMountLeaseFallback) EXPECT_TRUE(report.warnings.empty()); EXPECT_EQ(report.namespaces_removed, 0u); EXPECT_TRUE(report.slot_removed); - EXPECT_FALSE(backend->get(layout.epochKey("victim")).has_value()); - const auto owner = backend->get(layout.ownerKey("victim")); + EXPECT_FALSE((*raw_op).head(layout.epochKey("victim"), Retry::once()).has_value()); + const auto owner = (*raw_op).read(layout.ownerKey("victim"), Retry::once()); ASSERT_TRUE(owner.has_value()); EXPECT_TRUE(decodeOwner(owner->bytes).retired_at_ms.has_value()); - EXPECT_FALSE(backend->get(layout.mountKey("victim")).has_value()); + EXPECT_FALSE((*raw_op).head(layout.mountKey("victim"), Retry::once()).has_value()); expectThrowsCode(ErrorCodes::CORRUPTED_DATA, [&] { diff --git a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp index 5407d4b0f086..34cb8b7fad07 100644 --- a/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp +++ b/src/Disks/tests/gtest_cas_decommission_catalog_duties.cpp @@ -50,7 +50,8 @@ void makeRemoving(CasOperation & op, const Layout & layout, const CatalogEntry & bool slotObjectExists(Backend & backend, const String & leaf) { - return backend.head("p/gc/server-roots/victim/" + leaf).exists; + DB::Cas::tests::OperationForTest op(backend); + return (*op).head("p/gc/server-roots/victim/" + leaf, Retry::standard()).has_value(); } class AddVictimEntryDuringRootDrainBackend final : public InMemoryBackend @@ -181,8 +182,8 @@ TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndKeeps life = victim->namespaceLife(ns); const CatalogEntry live = catalogEntry(catalog_op, victim->layout(), ns); makeRemoving(catalog_op, victim->layout(), live); - ASSERT_TRUE(backend->head(victim->layout().refCkptKey(*life)).exists); - ASSERT_TRUE(backend->list(victim->layout().namespaceStreamPrefix(*life), "", 100).keys.empty()); + ASSERT_TRUE(catalog_op.head(victim->layout().refCkptKey(*life), Retry::standard()).has_value()); + ASSERT_TRUE(catalog_op.list(victim->layout().namespaceStreamPrefix(*life), "", 100, Retry::standard()).keys.empty()); } std::atomic wake_requests{0}; @@ -196,11 +197,11 @@ TEST(CASDecommissionCatalogDuties, RemovingWithCheckpointResumesTerminalAndKeeps EXPECT_FALSE(report.warnings.empty()); EXPECT_TRUE(slotObjectExists(*backend, "owner")); - const ListPage stream = backend->list(Layout("p").namespaceStreamPrefix(*life), "", 100); + const ListPage stream = catalog_op.list(Layout("p").namespaceStreamPrefix(*life), "", 100, Retry::standard()); ASSERT_EQ(stream.keys.size(), 1u); const auto parsed = Layout("p").parseRefObjectKey(stream.keys.front().key); ASSERT_TRUE(parsed); - const auto body = backend->get(stream.keys.front().key); + const auto body = catalog_op.read(stream.keys.front().key, Retry::standard()); ASSERT_TRUE(body); const RefLogTxn terminal = decodeRefLogTxn( openObject(FormatId::RefLog, body->bytes), ns.string(), parsed->txn_id); @@ -230,8 +231,8 @@ TEST(CASDecommissionCatalogDuties, PartialRemovalProgressStillWakesGcWhenLaterNa CasRefCatalog::casAdmitEntry( catalog_op, victim->layout(), victim->poolConfig().gc_shards, broken_live); makeRemoving(catalog_op, victim->layout(), broken_live); - ASSERT_FALSE(backend->head(victim->layout().refCkptKey( - NamespaceLifeId::fromCatalogEntry(broken_ns, broken_live.incarnation))).exists); + ASSERT_FALSE(catalog_op.head(victim->layout().refCkptKey( + NamespaceLifeId::fromCatalogEntry(broken_ns, broken_live.incarnation)), Retry::standard()).has_value()); } std::atomic wake_requests{0}; @@ -246,7 +247,7 @@ TEST(CASDecommissionCatalogDuties, PartialRemovalProgressStillWakesGcWhenLaterNa << "progress already made for an earlier life must wake GC even when a later life fails closed"; EXPECT_TRUE(slotObjectExists(*backend, "owner")); const ListPage progressed_stream - = backend->list(Layout("p").namespaceStreamPrefix(*progressed_life), "", 100); + = catalog_op.list(Layout("p").namespaceStreamPrefix(*progressed_life), "", 100, Retry::standard()); ASSERT_EQ(progressed_stream.keys.size(), 1u); } @@ -312,7 +313,7 @@ TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAn Gc gc(victim, UInt128{811}); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); ASSERT_EQ(catalogEntry(catalog_op, victim->layout(), ns).state, NsState::Removing); - for (const ListedKey & key : backend->list(victim->layout().namespaceStreamPrefix(*life), "", 100).keys) + for (const ListedKey & key : catalog_op.list(victim->layout().namespaceStreamPrefix(*life), "", 100, Retry::standard()).keys) stream_before.push_back(key.key); ASSERT_FALSE(stream_before.empty()); } @@ -327,7 +328,7 @@ TEST(CASDecommissionCatalogDuties, FoldedTerminalRemainsGcOwnedAndOnlyRequestsAn EXPECT_FALSE(report.slot_removed); EXPECT_EQ(catalogEntry(catalog_op, Layout("p"), ns).state, NsState::Removing); std::vector stream_after; - for (const ListedKey & key : backend->list(Layout("p").namespaceStreamPrefix(*life), "", 100).keys) + for (const ListedKey & key : catalog_op.list(Layout("p").namespaceStreamPrefix(*life), "", 100, Retry::standard()).keys) stream_after.push_back(key.key); EXPECT_EQ(stream_after, stream_before) << "decommission must not append a second terminal or become a catalog deletion driver"; @@ -343,14 +344,14 @@ TEST(CASDecommissionCatalogDuties, OpaqueLifeDebrisWithoutCatalogOwnershipDoesNo const NamespaceLifeId dead_life = NamespaceLifeId::fromCatalogEntry(RootNamespace("historical/name"), UInt128{709}); const String debris_key = layout.refCkptKey(dead_life); - ASSERT_EQ(backend->putIfAbsent(debris_key, "debris").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(catalog_op.create(debris_key, "debris", Retry::standard()))); const DecommissionReport report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); EXPECT_TRUE(report.warnings.empty()); EXPECT_TRUE(report.slot_removed); - EXPECT_TRUE(backend->head(debris_key).exists); + EXPECT_TRUE(catalog_op.head(debris_key, Retry::standard()).has_value()); } } diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index 4f0be05d79f0..c74181bd04d2 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -141,9 +141,7 @@ class FinalAuthorityBackend : public DB::Cas::tests::OrderedFaultBackend CasRequestBudget oneAttemptBudget() { CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; return budget; } @@ -286,9 +284,7 @@ PoolPtr openPublishingPool(const std::shared_ptr readForTest(const String & key) + { + DB::Cas::tests::OperationForTest op(*this); + return (*op).read(key, Retry::standard()); + } + + bool replaceForTest(const String & key, const String & bytes, const Etag & expected) + { + DB::Cas::tests::OperationForTest op(*this); + return std::holds_alternative((*op).replace(key, bytes, expected, Retry::standard())); + } + /// The engine settles an ambiguous write by reading the key back, so the observation belongs on the /// READ PRIMITIVE -- the resolve read never reaches the legacy `get`. std::optional read(const String & key, DB::Cas::TransportAccess & access) override @@ -111,11 +125,7 @@ CasRequestBudget renewalEventBudget() { return CasRequestBudget{ .attempt_timeout_ms = 10, - .operation_deadline_ms = 500, - .max_attempts = 2, .lease_safety_margin_ms = 20, - .retry_initial_backoff_ms = 0, - .retry_max_backoff_ms = 0, }; } @@ -234,7 +244,7 @@ TEST(CASEvent, WatermarkRenewEventsAreBoundedAndComplete) EXPECT_LT(renewals[0].detail.at("write_attempt_id").size(), 32u); /// Both attempts sent the same body, so the event names the id the lease actually landed with -- /// a reissue that minted a fresh id would leave the two disagreeing. - const MountLease landed = decodeMountLease(backend->get(store->layout().mountKey("test"))->bytes); + const MountLease landed = decodeMountLease(backend->readForTest(store->layout().mountKey("test"))->bytes); EXPECT_EQ(renewals[0].detail.at("write_attempt_id"), u128ToHex(landed.write_attempt_id).substr(0, 12)); for (const String & key : { @@ -288,7 +298,7 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) std::array, depth> backends; std::array, depth> layouts; std::array, depth> planes; - std::array, depth> keepers; + std::array, depth> renewers; std::array server_root_ids; std::array sinks; std::array renew_events{}; @@ -300,7 +310,7 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) renew_at = [&](size_t index) { configureMountRenewObservability(&server_root_ids[index], &sinks[index], /*deferred=*/false); - MountRenewResult result = keepers[index]->renew(MountRenewOperationEnvironment{}); + MountRenewResult result = renewers[index]->renew(MountRenewOperationEnvironment{}); reportMountRenewCompletion(result); return result; }; @@ -321,12 +331,12 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) deepest_result = std::move(child_result); } }; - /// One open-fence plane per keeper, on the same injected clock the keeper anchors its lease + /// One open-fence plane per renewer, on the same injected clock the renewer anchors its lease /// against, and with a sleep that advances it: the deepest renewal reissues, and no unit test /// may serve the engine's jittered backoff for real. planes[index] = std::make_unique( backends[index], Fence::open(), [&] { return boot_ms; }, [&](uint64_t ms) { boot_ms += ms; }); - keepers[index] = std::make_unique( + renewers[index] = std::make_unique( *planes[index], *planes[index], *layouts[index], @@ -339,18 +349,16 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) sinks[index], std::chrono::milliseconds(0), [&] { return boot_ms; }); - keepers[index]->start(); + renewers[index]->start(); if (index + 1 < depth) { const String key = layouts[index]->mountKey(server_root_ids[index]); - auto observed = backends[index]->get(key); + auto observed = backends[index]->readForTest(key); ASSERT_TRUE(observed.has_value()); MountLease foreign = decodeMountLease(observed->bytes); foreign.server_uuid = UInt128(100 + index); - ASSERT_EQ( - backends[index]->putOverwrite(key, encodeMountLease(foreign), observed->token).outcome, - PutOutcome::Done); + ASSERT_TRUE(backends[index]->replaceForTest(key, encodeMountLease(foreign), observed->etag)); } } /// The deepest slot is the only one nobody took, so its renewal can recover: the attempt is lost @@ -376,7 +384,7 @@ TEST(CASEvent, WatermarkRenewSinkFailureCannotChangeOutcome) uint64_t boot_ms = 100; auto store = openRenewalEventPool(backend, boot_ms); const String mount_key = store->layout().mountKey("test"); - const uint64_t seq_before = decodeMountLease(backend->get(mount_key)->bytes).seq; + const uint64_t seq_before = decodeMountLease(backend->readForTest(mount_key)->bytes).seq; store->setEventSink([](const CasEvent & event) { if (event.type == CasEventType::WatermarkRenew) @@ -385,7 +393,7 @@ TEST(CASEvent, WatermarkRenewSinkFailureCannotChangeOutcome) backend->throw_before_next_write = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); - EXPECT_EQ(decodeMountLease(backend->get(mount_key)->bytes).seq, seq_before + 1); + EXPECT_EQ(decodeMountLease(backend->readForTest(mount_key)->bytes).seq, seq_before + 1); EXPECT_TRUE(store->mayMutate()); } @@ -418,7 +426,7 @@ TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); const std::optional failed = one_failed_event(events); ASSERT_TRUE(failed.has_value()) << "the store's refusal must reach the event log"; - /// A deterministic failure reaches the keeper as the exception the engine refuses to reissue, + /// A deterministic failure reaches the renewer as the exception the engine refuses to reissue, /// and an exception carries no attempt count -- so the classification is all this ending states. EXPECT_EQ(failed->detail.at("classification"), "deterministic_failure"); } @@ -468,7 +476,7 @@ TEST(CASEvent, ReentrantRenewalSinkPreservesOuterObservationIdentity) EXPECT_EQ(events[0].outcome, "recovered"); EXPECT_EQ(events[0].detail.at("attempts_sent"), "2"); EXPECT_EQ(events[0].detail.at("seq"), "2"); - EXPECT_EQ(decodeMountLease(backend->get(store->layout().mountKey("test"))->bytes).seq, 3u) + EXPECT_EQ(decodeMountLease(backend->readForTest(store->layout().mountKey("test"))->bytes).seq, 3u) << "the nested first-attempt success must run without replacing the outer observation"; } @@ -566,7 +574,7 @@ bool anyRetiredPending(const PoolPtr & s) { /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. - return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); + return DB::Cas::tests::anyCondemnedInSeal(*s->poolBackendPtr(), s->layout()); } /// Drive regular GC to a fixpoint over the ACK-FLOOR round (renew the store's mount ack after each round; @@ -632,8 +640,11 @@ TEST(CASEvent, LifecycleReconstructionFromRows) runGcToFixpoint(s, gc); /// The blob must actually be gone (the delete fired). - ASSERT_FALSE(b->head(s->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of(payload))})).exists) - << "GC must have deleted the now-unreferenced blob"; + { + DB::Cas::tests::OperationForTest blob_op(b); + ASSERT_FALSE((*blob_op).head(s->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of(payload))}), Retry::standard()).has_value()) + << "GC must have deleted the now-unreferenced blob"; + } /// (a) the expected taxonomy was emitted across the lifecycle (manifest model: no standalone trees). EXPECT_TRUE(hasType(events, CasEventType::BlobPut)); diff --git a/src/Disks/tests/gtest_cas_fence_generation.cpp b/src/Disks/tests/gtest_cas_fence_generation.cpp index b298c6144029..f4f060d576f9 100644 --- a/src/Disks/tests/gtest_cas_fence_generation.cpp +++ b/src/Disks/tests/gtest_cas_fence_generation.cpp @@ -244,7 +244,10 @@ TEST(CASFenceGeneration, BlobPublicationFenceLossBeforeFinalCheckPublishesNothin }); EXPECT_EQ(backend->publish_calls, 0u); - EXPECT_FALSE(backend->head(backend->watched_key).exists); + { + DB::Cas::tests::OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(backend->watched_key, Retry::once()).has_value()); + } EXPECT_EQ(build->dependencyProof(ref), std::nullopt); } @@ -272,7 +275,10 @@ TEST(CASFenceGeneration, BlobPublicationHeadTripAndRearmCannotAdoptNewFenceGener }); EXPECT_EQ(backend->publish_calls, 0u); - EXPECT_FALSE(backend->head(backend->watched_key).exists); + { + DB::Cas::tests::OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(backend->watched_key, Retry::once()).has_value()); + } /// The mount is live again under a FRESH generation, so this read is admitted where the stale /// operation's writes were not. CasOperation probe = store->mountRequests().admit(); @@ -298,8 +304,11 @@ TEST(CASFenceGeneration, BlobPublicationFenceLossAfterLandingReturnsNoProof) }); EXPECT_EQ(backend->publish_calls, 1u); - EXPECT_TRUE(backend->head(backend->watched_key).exists) - << "a publication that landed before fence loss is safe unreferenced debris"; + { + DB::Cas::tests::OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(backend->watched_key, Retry::once()).has_value()) + << "a publication that landed before fence loss is safe unreferenced debris"; + } EXPECT_EQ(build->dependencyProof(ref), std::nullopt); } @@ -322,8 +331,9 @@ TEST(CASFenceGeneration, PlainObjectPutAbortsWhenFenceTripsBetweenAdmissionAndDu /// No durable write ever landed. Asserted through the RAW backend: every request the pool issues /// is admitted under the mount fence, which this test has just tripped, so a read through the pool /// would report that refusal rather than what the store holds. - EXPECT_TRUE(backend->list(store->layout().namespaceFilesPrefix( - DB::Cas::tests::fixture::fixtureLife(ns)), "", 100).keys.empty()); + DB::Cas::tests::OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).list(store->layout().namespaceFilesPrefix( + DB::Cas::tests::fixture::fixtureLife(ns)), "", 100, Retry::once()).keys.empty()); } /// `casRemoveObject`'s delete sibling, same shape: the fence trips between admission and the durable @@ -347,8 +357,9 @@ TEST(CASFenceGeneration, PlainObjectRemoveAbortsWhenFenceTripsBetweenAdmissionAn /// The durable delete never ran, so the object survives -- read raw, since a read through the pool /// is admitted under the fence this test has tripped and would report that refusal instead. - const auto still_there = backend->get(store->layout().namespaceFileKey( - DB::Cas::tests::fixture::fixtureLife(ns), "victim")); + DB::Cas::tests::OperationForTest raw_op(*backend); + const auto still_there = (*raw_op).read(store->layout().namespaceFileKey( + DB::Cas::tests::fixture::fixtureLife(ns), "victim"), Retry::once()); ASSERT_TRUE(still_there.has_value()); EXPECT_EQ(still_there->bytes, "still here"); } @@ -374,8 +385,9 @@ TEST(CASFenceGeneration, PlainObjectPutRechecksFenceOnEveryRetryIterationNotJust }); EXPECT_EQ(backend->head_calls, 2); - EXPECT_TRUE(backend->list(store->layout().namespaceFilesPrefix( - DB::Cas::tests::fixture::fixtureLife(ns)), "", 100).keys.empty()); + DB::Cas::tests::OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).list(store->layout().namespaceFilesPrefix( + DB::Cas::tests::fixture::fixtureLife(ns)), "", 100, Retry::once()).keys.empty()); } /// (b) The S3-native staging-buffer finalize: the fence trips AFTER the buffer is constructed diff --git a/src/Disks/tests/gtest_cas_forget.cpp b/src/Disks/tests/gtest_cas_forget.cpp index c73417fb15c1..094f46660eed 100644 --- a/src/Disks/tests/gtest_cas_forget.cpp +++ b/src/Disks/tests/gtest_cas_forget.cpp @@ -25,8 +25,8 @@ /// Task 10 (rev.7 spec §5): `SYSTEM CAS FORGET` — the operator force-Vanish. FORGET drives a /// content-addressed pool to `Vanished(forgotten)` with the fence-first protocol: (1) publish terminal -/// intent, (2) trip the local fence, (3+4) stop the GC scheduler, (5) join keeper/remount, drain, retire -/// the keeper WITHOUT an unearned clean farewell, (6) publish `Vanished(forgotten)` with the [D5] message +/// intent, (2) trip the local fence, (3+4) stop the GC scheduler, (5) join renewer/remount, drain, retire +/// the renewer WITHOUT an unearned clean farewell, (6) publish `Vanished(forgotten)` with the [D5] message /// carrying the decommission timestamp. These tests exercise the Pool-level protocol body (`Pool::forgetDisk`) /// and the end-to-end verb through a real `ContentAddressedMetadataStorage` (the six-class gate wired to the /// new state). Harness patterns follow gtest_cas_lifecycle_condition.cpp and gtest_cas_operation_gate.cpp. @@ -56,10 +56,11 @@ const String kForgetReason = /// gtest_cas_lifecycle_condition.cpp — used to drive a live pool into `IdentityLost`. void deleteKeyExact(DB::Cas::Backend & backend, const String & key) { - const auto got = backend.get(key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(key, DB::Cas::Retry::once()); ASSERT_TRUE(got.has_value()) << "expected '" << key << "' to exist before deletion"; if (got) - backend.deleteExact(key, got->token); + (*op).remove(key, got->etag, DB::Cas::Retry::once()); } /// GC's fence-out applied directly to the mount lease (preserve the body, set `gc_fenced`, bump `seq`) — @@ -67,13 +68,14 @@ void deleteKeyExact(DB::Cas::Backend & backend, const String & key) /// lease-expiry wait), reaching `armMountFence`. Mirrors gtest_cas_lifecycle_condition.cpp's helper. void fenceOutMount(DB::Cas::Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, DB::Cas::Retry::once()); ASSERT_TRUE(got.has_value()); DB::Cas::MountLease m = DB::Cas::decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, DB::Cas::encodeMountLease(m), got->token).outcome, - DB::Cas::PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(mount_key, DB::Cas::encodeMountLease(m), got->etag, DB::Cas::Retry::once()))); } /// A Backend decorator whose reads, heads and lists throw an untyped transport error while `fail` is @@ -324,12 +326,13 @@ TEST(CASForget, ForgetCleanFarewellGatedOnDrain) auto backend = std::make_shared(); auto store = DB::Cas::tests::openPoolForTest(backend); const String mount_key = store->layout().mountKey(kSrid); - ASSERT_NE(decodeMountLease(backend->get(mount_key)->bytes).min_active_build_sequence, kTerminated); /// baseline + DB::Cas::tests::OperationForTest op(*backend); + ASSERT_NE(decodeMountLease((*op).read(mount_key, DB::Cas::Retry::once())->bytes).min_active_build_sequence, kTerminated); /// baseline store->forgetDisk([] {}, kForgetReason); ASSERT_EQ(store->lifecycle(), PoolLifecycle::VanishedForgotten); - const auto got = backend->get(mount_key); + const auto got = (*op).read(mount_key, DB::Cas::Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_EQ(decodeMountLease(got->bytes).min_active_build_sequence, kTerminated) << "a drained FORGET earns the clean-release farewell"; @@ -348,7 +351,8 @@ TEST(CASForget, ForgetCleanFarewellGatedOnDrain) store->forgetDisk([] {}, kForgetReason); ASSERT_EQ(store->lifecycle(), PoolLifecycle::VanishedForgotten); - const auto got = backend->get(mount_key); + DB::Cas::tests::OperationForTest op(*backend); + const auto got = (*op).read(mount_key, DB::Cas::Retry::once()); ASSERT_TRUE(got.has_value()) << "the lease object must still be present (expiry by observation)"; EXPECT_NE(decodeMountLease(got->bytes).min_active_build_sequence, kTerminated) << "an unearned clean farewell must NOT be written when the ref lanes did not drain"; @@ -437,12 +441,13 @@ TEST(CASForget, ForgetIntentBlocksNaturalReplacedPromotion) /// Make the identity gate verdict `Replaced`: overwrite `_pool_meta` with a FOREIGN pool_id (present, /// mismatched identity) — exactly gtest_cas_lifecycle_condition.cpp scenario (b). const String meta_key = store->layout().poolMetaKey(); - const auto got = backend->get(meta_key); + DB::Cas::tests::OperationForTest op(*backend); + const auto got = (*op).read(meta_key, DB::Cas::Retry::once()); ASSERT_TRUE(got.has_value()); DB::Cas::PoolMeta foreign = DB::Cas::decodePoolMeta(got->bytes); foreign.pool_id = foreign.pool_id + DB::UInt128(1); - ASSERT_EQ(backend->putOverwrite(meta_key, DB::Cas::encodePoolMeta(foreign), got->token).outcome, - DB::Cas::PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(meta_key, DB::Cas::encodePoolMeta(foreign), got->etag, DB::Cas::Retry::once()))); /// The in-flight gate (run from the GC-stop callback) reaches the `Replaced` verdict but must BAIL on the /// already-published intent rather than settle `Vanished(replaced)`. diff --git a/src/Disks/tests/gtest_cas_fsck.cpp b/src/Disks/tests/gtest_cas_fsck.cpp index b1d1696ad0d5..abc1ff77e505 100644 --- a/src/Disks/tests/gtest_cas_fsck.cpp +++ b/src/Disks/tests/gtest_cas_fsck.cpp @@ -199,11 +199,11 @@ void writeFsckCheckpoint(Backend & backend, const Layout & layout, const RootNam .committed_through = committed_through, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); - const HeadResult current = backend.head(key); - const PutResult put = current.exists - ? backend.putOverwrite(key, body, current.token) - : backend.putIfAbsent(key, body); - ASSERT_EQ(put.outcome, PutOutcome::Done); + const auto current = op.head(key, Retry::once()); + const WriteResult put = current + ? op.replace(key, body, current->etag, Retry::once()) + : op.create(key, body, Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); } void writeFsckCheckpointWithBase( @@ -213,11 +213,11 @@ void writeFsckCheckpointWithBase( CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); CasOperation op = requests.admit(); const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); - ASSERT_EQ(backend.putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = base, .checkpoint_snapshot_id = base, - .last_epoch_seal = last_epoch_seal})).outcome, PutOutcome::Done); + .last_epoch_seal = last_epoch_seal}), Retry::once()))); } void expectCheckpointBaseVerdict( @@ -254,9 +254,9 @@ void replaceCatalogLife(Backend & backend, const Layout & layout, const RootName it->state = NsState::Live; it->creator.reset(); it->removal_started_round.reset(); - ASSERT_TRUE(current.incarnation.has_value()); + ASSERT_TRUE(current.etag.has_value()); ASSERT_TRUE(std::holds_alternative( - op.replace(layout.refCatalogKey(), encodeRefCatalog(current.catalog), *current.incarnation, Retry::standard()))); + op.replace(layout.refCatalogKey(), encodeRefCatalog(current.catalog), *current.etag, Retry::standard()))); } FsckReport runFsckWithListingMode(FsckListingMode mode, std::string_view suffix) @@ -346,10 +346,10 @@ FsckReport runCheckpointBaseFsckWithListingMode( if (corrupt_exact_base) { const String base_snapshot_key = layout.refSnapshotKey(life, base); - const HeadResult head = backend->head(base_snapshot_key); - EXPECT_TRUE(head.exists); - if (head.exists) - EXPECT_EQ(backend->deleteExact(base_snapshot_key, head.token).kind, DeleteOutcome::Kind::Deleted); + const auto head = op.head(base_snapshot_key, Retry::once()); + EXPECT_TRUE(head.has_value()); + if (head) + EXPECT_EQ(op.remove(base_snapshot_key, head->etag, Retry::once()), Removal::Removed); } else { @@ -440,7 +440,10 @@ TEST(CASFsck, LifelessKeyIsRecordedAndTheHealthyNamespaceIsStillReported) /// Hand-built: no helper can mint the un-incarnated shape any more. const String lifeless = store->layout().casRefsPrefix() + ns.string() + "/_log/" + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; - ASSERT_EQ(backend->putIfAbsent(lifeless, "garbage").outcome, PutOutcome::Done); + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).create(lifeless, "garbage", Retry::once()))); + } FsckReport rep; ASSERT_NO_THROW(rep = runFsck(*store, /*detail*/true)) @@ -483,16 +486,16 @@ TEST(CASFsck, CanonicalDeadLifeResidueIsJanitorPendingNotHardFinding) /// protocol this fixture is not driving), so inject the post-deletion catalog snapshot directly, /// mirroring `DuplicateLifeIdIsReportedWhileAnUnrelatedUniqueNamespaceStillProgresses` below. { - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, store->layout()); const auto it = std::find_if(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [&](const CatalogEntry & entry) { return entry.ns == ns; }); ASSERT_NE(it, snapshot.catalog.entries.end()); snapshot.catalog.entries.erase(it); - const auto catalog_head = backend->head(store->layout().refCatalogKey()); - ASSERT_TRUE(catalog_head.exists); - ASSERT_EQ(backend->putOverwrite(store->layout().refCatalogKey(), encodeRefCatalog(snapshot.catalog), - catalog_head.token).outcome, PutOutcome::Done); + const auto catalog_head = op.head(store->layout().refCatalogKey(), Retry::once()); + ASSERT_TRUE(catalog_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(store->layout().refCatalogKey(), encodeRefCatalog(snapshot.catalog), + catalog_head->etag, Retry::once()))); } FsckReport rep; @@ -554,8 +557,11 @@ TEST(CASFsck, LifeAdmittedBetweenNamespaceListingAndLaterCutIsNotResidue) /// The physical object exists before the listing runs, exactly as a legitimate late admission would /// leave it: written only after `casAdmitEntry` above, but here pre-seeded since the injected /// backend admits the CATALOG row, not the physical file, on the list callback. - ASSERT_EQ(backend->putIfAbsent(store->layout().namespaceFilesPrefix(life) + "format_version.txt", "1\n").outcome, - PutOutcome::Done); + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative( + (*op).create(store->layout().namespaceFilesPrefix(life) + "format_version.txt", "1\n", Retry::once()))); + } FsckReport rep; ASSERT_NO_THROW(rep = runFsck(*store, /*detail*/true)); @@ -582,7 +588,10 @@ TEST(CASFsck, MalformedNamespaceTreeShapesStayHardFindings) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); - ASSERT_EQ(backend->putIfAbsent(c.key, "garbage").outcome, PutOutcome::Done) << c.description; + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).create(c.key, "garbage", Retry::once()))) << c.description; + } FsckReport rep; ASSERT_NO_THROW(rep = runFsck(*store, /*detail*/true)) << c.description; @@ -619,10 +628,9 @@ TEST(CASFsck, DuplicateLifeIdIsReportedWhileAnUnrelatedUniqueNamespaceStillProgr .removal_started_round = 1}); std::sort(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [](const CatalogEntry & lhs, const CatalogEntry & rhs) { return lhs.ns.string() < rhs.ns.string(); }); - const auto catalog_head = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(catalog_head.exists); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), encodeRefCatalog(snapshot.catalog), catalog_head.token).outcome, - PutOutcome::Done); + const auto catalog_head = op.head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(catalog_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(layout.refCatalogKey(), encodeRefCatalog(snapshot.catalog), catalog_head->etag, Retry::once()))); FsckReport report; ASSERT_NO_THROW(report = runFsck(*store, /*detail=*/true)); @@ -651,11 +659,10 @@ TEST(CASFsck, AmbiguousLifeUnderAPhysicalKeyIsRecordedNotAborted) writeFsckCheckpoint(*backend, layout, unique_ns, RefTxnId{1, sequence}); const NamespaceLifeId duplicated_life = NamespaceLifeId::fromCatalogEntry(RootNamespace{"bad/a"}, UInt128{777}); - ASSERT_EQ(backend->putIfAbsent(layout.namespaceFilesPrefix(duplicated_life) + "format_version.txt", "1\n").outcome, - PutOutcome::Done); - CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative( + op.create(layout.namespaceFilesPrefix(duplicated_life) + "format_version.txt", "1\n", Retry::once()))); CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout); snapshot.catalog.entries.push_back(CatalogEntry{ .ns = RootNamespace{"bad/a"}, .state = NsState::Live, .incarnation = UInt128{777}}); @@ -666,10 +673,9 @@ TEST(CASFsck, AmbiguousLifeUnderAPhysicalKeyIsRecordedNotAborted) .removal_started_round = 1}); std::sort(snapshot.catalog.entries.begin(), snapshot.catalog.entries.end(), [](const CatalogEntry & lhs, const CatalogEntry & rhs) { return lhs.ns.string() < rhs.ns.string(); }); - const auto catalog_head = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(catalog_head.exists); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), encodeRefCatalog(snapshot.catalog), catalog_head.token).outcome, - PutOutcome::Done); + const auto catalog_head = op.head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(catalog_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(layout.refCatalogKey(), encodeRefCatalog(snapshot.catalog), catalog_head->etag, Retry::once()))); FsckReport report; ASSERT_NO_THROW(report = runFsck(*store, /*detail=*/true)) @@ -845,14 +851,14 @@ TEST(CASFsckAuthority, MissingBurnedEpochSealIsChainBroken) const auto old_epoch = skipped_bytes.find(old_epoch_token); ASSERT_NE(old_epoch, String::npos); skipped_bytes.replace(old_epoch, old_epoch_token.size(), R"("!prev_epoch":"1")"); - ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(life, RefTxnId{7, 1}), - sealObject(FormatId::RefLog, skipped_bytes)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.refLogKey(life, RefTxnId{7, 1}), + sealObject(FormatId::RefLog, skipped_bytes), Retry::once()))); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{7, 1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = RefTxnId{6, 1}})).outcome, PutOutcome::Done); + .last_epoch_seal = RefTxnId{6, 1}}), Retry::once()))); const FsckReport report = runFsck(*store, /*detail=*/true); EXPECT_EQ(report.chain_broken, 1u); @@ -944,11 +950,11 @@ TEST(CASFsckAuthority, CheckpointSnapshotAtOlderEpochSealIsChainBroken) CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); CasOperation op = requests.admit(); const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{2, 1}, .checkpoint_snapshot_id = RefTxnId{1, 2}, - .last_epoch_seal = RefTxnId{2, 1}})).outcome, PutOutcome::Done); + .last_epoch_seal = RefTxnId{2, 1}}), Retry::once()))); const FsckReport report = runFsck(*store, /*detail=*/true); EXPECT_EQ(report.ref_records_walked, 0u) @@ -1011,13 +1017,14 @@ TEST(CASFsckAuthority, CheckpointBaseVanishingAfterAuthorityAdvanceIsUnchecked) backend->armOnFirstGet(layout.refLogKey(life, old_base), [&] { const String ckpt_key = layout.refCkptKey(life); - const HeadResult head = backend->head(ckpt_key); - ASSERT_TRUE(head.exists); - ASSERT_EQ(backend->putOverwrite(ckpt_key, encodeRefCkpt(RefCkpt{ + OperationForTest nested_op(*backend); + const auto head = (*nested_op).head(ckpt_key, Retry::once()); + ASSERT_TRUE(head.has_value()); + ASSERT_TRUE(std::holds_alternative((*nested_op).replace(ckpt_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt}), head.token).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), head->etag, Retry::once()))); }); const FsckReport report = runFsck(*store, /*detail=*/true); @@ -1391,9 +1398,10 @@ TEST(CASFsck, PhantomDanglingFromRepublishedRefIsReresolvedAway) writeFsckCheckpoint(*backend, store->layout(), ns, RefTxnId{1, repoint_sequence}); const String old_key = store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h1)}); - const HeadResult head = backend->head(old_key); - ASSERT_TRUE(head.exists); - backend->deleteExact(old_key, head.token); /// legitimate GC delete of the now-unreferenced blob + OperationForTest nested_op(*backend); + const auto head = (*nested_op).head(old_key, Retry::once()); + ASSERT_TRUE(head.has_value()); + (*nested_op).remove(old_key, head->etag, Retry::once()); /// legitimate GC delete of the now-unreferenced blob }); const FsckReport rep = runFsck(*store, /*detail*/true); @@ -1424,9 +1432,10 @@ TEST(CASFsck, PhantomDanglingFromDroppedRefIsReresolvedAway) writeFsckCheckpoint(*backend, store->layout(), ns, RefTxnId{1, drop_sequence}); const String old_key = store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h1)}); - const HeadResult head = backend->head(old_key); - ASSERT_TRUE(head.exists); - backend->deleteExact(old_key, head.token); /// legitimate GC delete after the drop folds + OperationForTest nested_op(*backend); + const auto head = (*nested_op).head(old_key, Retry::once()); + ASSERT_TRUE(head.has_value()); + (*nested_op).remove(old_key, head->etag, Retry::once()); /// legitimate GC delete after the drop folds }); const FsckReport rep = runFsck(*store, /*detail*/true); @@ -1451,9 +1460,10 @@ TEST(CASFsck, RealDanglingStillCaughtAfterReresolve) writeFsckCheckpoint(*backend, store->layout(), ns, RefTxnId{1, sequence}); const String key = store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(h)}); - const HeadResult head = backend->head(key); - ASSERT_TRUE(head.exists); - backend->deleteExact(key, head.token); /// genuine loss — the ref is UNCHANGED, still names this blob + OperationForTest op(*backend); + const auto head = (*op).head(key, Retry::once()); + ASSERT_TRUE(head.has_value()); + (*op).remove(key, head->etag, Retry::once()); /// genuine loss — the ref is UNCHANGED, still names this blob const FsckReport rep = runFsck(*store, /*detail*/true); EXPECT_EQ(rep.dangling, 1u); @@ -1491,9 +1501,10 @@ TEST(CASFsck, PhantomDanglingManifestFromRepublishedRefIsReresolvedAway) *backend, store->layout(), ns, "tbl", r1, r2); /// re-publish writeFsckCheckpoint(*backend, store->layout(), ns, RefTxnId{1, repoint_sequence}); - const HeadResult head = backend->head(m1_key); - ASSERT_TRUE(head.exists); - backend->deleteExact(m1_key, head.token); /// legitimate GC delete of the superseded manifest + OperationForTest nested_op(*backend); + const auto head = (*nested_op).head(m1_key, Retry::once()); + ASSERT_TRUE(head.has_value()); + (*nested_op).remove(m1_key, head->etag, Retry::once()); /// legitimate GC delete of the superseded manifest }); const FsckReport rep = runFsck(*store, /*detail*/true); diff --git a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp index 2f8bc102efc8..8a4c04f6beb2 100644 --- a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp +++ b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp @@ -31,9 +31,30 @@ ManifestRef ref(const String &, uint64_t seq, uint64_t inst) { return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = static_cast(inst)}; } +/// A one-shot `create`, asserting it committed (mirrors the retired `backend.putIfAbsent(key, bytes)`). +void createObj(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// An exact read (mirrors the retired `backend.get(key)`). +std::optional readObj(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +/// A HEAD (mirrors the retired `backend.head(key)`). +std::optional headObj(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::standard()); +} + bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash) { - return b.head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + return headObj(b, layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).has_value(); } /// Publish one physical blob through the production durable-precommit ordering. The committed fixture @@ -149,7 +170,7 @@ TEST(CASSemanticRefFixture, WrapperAdvancesCheckpointWithoutDiscardingSnapshot) RefCkpt with_snapshot = before_drop->ckpt; with_snapshot.checkpoint_snapshot_id = publish_id; ASSERT_TRUE(std::holds_alternative(op.replace( - store->layout().refCkptKey(life), encodeRefCkpt(with_snapshot), before_drop->incarnation, + store->layout().refCkptKey(life), encodeRefCkpt(with_snapshot), before_drop->etag, Retry::once()))); const uint64_t drop_sequence = dropRefTransition(*backend, store->layout(), ns, "tbl", manifest); @@ -179,9 +200,9 @@ TEST(CASSemanticRefFixture, CheckpointAdvanceRejectsNonMonotoneAndInvalidState) fixture::admitLive(*backend, store->layout(), invalid_ns); const NamespaceLifeId invalid_life = *CasRefCatalog::lifeIfCataloged(op, store->layout(), invalid_ns); const String invalid_key = store->layout().refCkptKey(invalid_life); - ASSERT_EQ(backend->putIfAbsent(invalid_key, "not a checkpoint").outcome, PutOutcome::Done); + createObj(*backend, invalid_key, "not a checkpoint"); EXPECT_THROW(advanceRecoverableCkptForRawFixture(*backend, store->layout(), invalid_ns, id), DB::Exception); - EXPECT_EQ(backend->get(invalid_key)->bytes, "not a checkpoint"); + EXPECT_EQ(readObj(*backend, invalid_key)->bytes, "not a checkpoint"); } TEST(CASRawRefFixture, RawLogWriteDoesNotCreateCheckpoint) @@ -275,11 +296,11 @@ TEST(CASGCRetire, ManifestBodyDeletedAfterDecrementsSealed) publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, r); Gc gc(store, kGc); runRegularRoundReclaiming(gc); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_TRUE(headObj(*backend, store->layout().manifestKey(ManifestId{ns, r})).has_value()); dropRefTransition(*backend, store->layout(), ns, "tbl", r); runRegularRoundReclaiming(gc); - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_FALSE(headObj(*backend, store->layout().manifestKey(ManifestId{ns, r})).has_value()); } /// A publish racing the pass (in-degree restored) is SPARED, not deleted (#14). @@ -322,7 +343,7 @@ TEST(CASGCRecheck, UnreferencedBlobDeletedExactToken) dropRefTransition(*backend, store->layout(), ns, "tbl", r); // The drop's -1 condemns blob 1; the retired-cursor pipeline (condemn -> graduate -> delete) reclaims it. EXPECT_TRUE(runRoundsUntilAbsent(store, gc, *backend, store->layout(), DB::UInt128(1))); - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_FALSE(headObj(*backend, store->layout().manifestKey(ManifestId{ns, r})).has_value()); } /// Task 5 (spec 2026-07-09 §raw-body-refinement, v3): GC writes the writer's freshness meta ALONGSIDE @@ -396,7 +417,7 @@ TEST(CASGCRetire, SpareLeavesMetaCondemned) publishBlobWithDurablePrecommit(store, seed_ns, "seed", id, payload); store->dropRef(seed_ns, "seed"); store->renewWatermarkOnce(); - const Token t_seed = backend->head(store->layout().blobKey(id)).token; + const Etag t_seed = headObj(*backend, store->layout().blobKey(id))->etag; const ManifestRef r1 = ref("srv-a:1", 1, 0xA1); writeManifestRaw(*backend, store->layout(), ns, r1, {blobEntryFor("a", hash)}); @@ -423,7 +444,7 @@ TEST(CASGCRetire, SpareLeavesMetaCondemned) EXPECT_FALSE(currentEntryFor(*backend, store->layout(), hash).has_value()) << "the spared entry drops from the retired set"; EXPECT_TRUE(blobExists(*backend, store->layout(), hash)); - EXPECT_EQ(backend->head(store->layout().blobKey(id)).token, t_seed) + EXPECT_EQ(headObj(*backend, store->layout().blobKey(id))->etag, t_seed) << "spare does not touch the body — the incarnation token is unchanged"; /// ADD-ONLY: the spare must NOT clear the meta back to Clean (that is the deposed-leader hole). @@ -440,7 +461,7 @@ TEST(CASGCRetire, SpareLeavesMetaCondemned) const RootNamespace writer_ns{"00/spare-writer@cas@"}; auto ref_w = publishBlobWithDurablePrecommit(store, writer_ns, "writer", id, payload); EXPECT_EQ(ref_w.ref, id); - const Token t_resurrect = backend->head(store->layout().blobKey(id)).token; + const Etag t_resurrect = headObj(*backend, store->layout().blobKey(id))->etag; EXPECT_NE(t_resurrect, t_seed) << "republication displaces the body with a fresh incarnation token"; const auto lm_after = loadMetaForTest(*backend, store->layout(), hash); ASSERT_TRUE(lm_after.has_value()); @@ -492,10 +513,10 @@ TEST(CASGCRetire, StaleRedeleteAfterSpareDoesNotDeleteLiveReuse) CasOperation op = requests.admit(); const auto condemned_entry = currentEntryFor(*backend, store->layout(), hash); ASSERT_TRUE(condemned_entry.has_value()); - const PersistedIncarnation t1 = condemned_entry->token; + const PersistedEtag t1 = condemned_entry->token; const std::optional at_condemn = op.head(blob_key, Retry::once()); ASSERT_TRUE(at_condemn); - ASSERT_TRUE(t1.matches(at_condemn->incarnation)); + ASSERT_TRUE(t1.matches(at_condemn->etag)); /// A NEW leader L2 folds a +1 that recovered h's in-degree and adopts a SPARE for h. const ManifestRef r2 = ref("srv-a:1", 2, 0xA2); @@ -517,7 +538,7 @@ TEST(CASGCRetire, StaleRedeleteAfterSpareDoesNotDeleteLiveReuse) publishBlobWithDurablePrecommit(store, writer_ns, "writer", id, payload); const std::optional t2 = op.head(blob_key, Retry::once()); ASSERT_TRUE(t2); - EXPECT_FALSE(t1.matches(t2->incarnation)) + EXPECT_FALSE(t1.matches(t2->etag)) << "the writer resurrected to a fresh incarnation, not a reuse of t1"; /// L1 resumes and replays its stale pre-CAS redelete exactly as the round performs one: observe the @@ -527,17 +548,17 @@ TEST(CASGCRetire, StaleRedeleteAfterSpareDoesNotDeleteLiveReuse) const uint64_t removals_before = backend->deleteCount(blob_key); const std::optional stale = op.head(blob_key, Retry::once()); ASSERT_TRUE(stale); - EXPECT_FALSE(t1.matches(stale->incarnation)) + EXPECT_FALSE(t1.matches(stale->etag)) << "the stale redelete must miss the live reuse (add-only closes INV_NO_LOSS)"; - if (t1.matches(stale->incarnation)) - (void)op.remove(blob_key, stale->incarnation, Retry::once()); + if (t1.matches(stale->etag)) + (void)op.remove(blob_key, stale->etag, Retry::once()); EXPECT_EQ(backend->deleteCount(blob_key), removals_before) << "the comparison failed, so the redelete sent no removal at all against the live body"; /// The live body under t2 survives, stays reachable via the committed r2, and fsck sees no dangle. const std::optional survivor = op.head(blob_key, Retry::once()); ASSERT_TRUE(survivor); - EXPECT_EQ(survivor->incarnation, t2->incarnation); + EXPECT_EQ(survivor->etag, t2->etag); replaceRecoverableCkptForRawFixture( *backend, store->layout(), ns, RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 3}, @@ -569,9 +590,11 @@ TEST(CASGCRetire, CopyForwardedBlobSurvivesWhenRepublished) /// same verified bytes under a fresh token t1, then republish a part referencing the blob (the /// promoted dst ref of a republishRef move). const String blob_key = store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))}); - const Token t0 = backend->head(blob_key).token; - const auto res = backend->putOverwrite(blob_key, backend->get(blob_key)->bytes, t0); - ASSERT_EQ(res.outcome, PutOutcome::Done); + OperationForTest displace(*backend); + const Etag t0 = (*displace).head(blob_key, Retry::standard())->etag; + const WriteResult res = (*displace).replace(blob_key, readObj(*backend, blob_key)->bytes, t0, Retry::once()); + ASSERT_TRUE(std::holds_alternative(res)); + const Etag res_etag = std::get(res).etag; const ManifestRef r2 = ref("srv-a:1", 2, 0xA2); writeManifestRaw(*backend, store->layout(), ns, r2, {blobEntryFor("a", DB::UInt128(1))}); publishCommittedTransition(*backend, store->layout(), ns, "tbl_detached", std::nullopt, r2); @@ -580,9 +603,9 @@ TEST(CASGCRetire, CopyForwardedBlobSurvivesWhenRepublished) for (int i = 0; i < 4; ++i) gc.runRegularRound(); EXPECT_FALSE(currentEntryFor(*backend, store->layout(), DB::UInt128(1)).has_value()); - const HeadResult hr = backend->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_EQ(hr.token, res.token); + const auto hr = (*displace).head(blob_key, Retry::standard()); + ASSERT_TRUE(hr.has_value()); + EXPECT_EQ(hr->etag, res_etag); } /// Copy-forward aftermath, stale-entry arm: a listed (hash, t0) entry whose incarnation was @@ -609,9 +632,11 @@ TEST(CASGCRetire, AbandonedCopyForwardDropsEntryWithoutWrongTokenDelete) ASSERT_TRUE(currentEntryFor(*backend, store->layout(), DB::UInt128(1)).has_value()); const String blob_key = store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))}); - const Token t0 = backend->head(blob_key).token; - const auto res = backend->putOverwrite(blob_key, backend->get(blob_key)->bytes, t0); - ASSERT_EQ(res.outcome, PutOutcome::Done); + OperationForTest displace(*backend); + const Etag t0 = (*displace).head(blob_key, Retry::standard())->etag; + const WriteResult res = (*displace).replace(blob_key, readObj(*backend, blob_key)->bytes, t0, Retry::once()); + ASSERT_TRUE(std::holds_alternative(res)); + const Etag res_etag = std::get(res).etag; /// No events land at all (raw displacement). Drive rounds with the store's ack kept current so /// the (1, t0) entry graduates; its exact-token delete mismatches t1 and the entry drops. @@ -622,9 +647,9 @@ TEST(CASGCRetire, AbandonedCopyForwardDropsEntryWithoutWrongTokenDelete) } EXPECT_FALSE(currentEntryFor(*backend, store->layout(), DB::UInt128(1)).has_value()) << "the stale (hash, t0) entry must settle (mismatch redelete drops it), not wedge the list"; - const HeadResult hr = backend->head(blob_key); - ASSERT_TRUE(hr.exists) << "the fresh incarnation must never be deleted under the stale token"; - EXPECT_EQ(hr.token, res.token); + const auto hr = (*displace).head(blob_key, Retry::standard()); + ASSERT_TRUE(hr.has_value()) << "the fresh incarnation must never be deleted under the stale token"; + EXPECT_EQ(hr->etag, res_etag); } /// A completed round adopts the SAME attempt its fold minted (the round's single gc/state CAS commits the @@ -643,22 +668,22 @@ TEST(CASGCRecheck, CompletionInheritsFoldAttempt) Gc gc(store, kGc); gc.runRegularRound(); // round 1: one pass, single CAS commits (snap_generation, snap_attempt) - const auto after_round1 = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_round1 = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes); // The round adopted the attempt of THIS round's fold: snap_attempt == the lease.seq that folded it. EXPECT_EQ(after_round1.snap_attempt, after_round1.lease.seq); EXPECT_GT(after_round1.snap_generation, 0u); // The fold seal is durable under the adopted (snap_generation, snap_attempt) pair (no completion seal). - EXPECT_TRUE(backend->head(store->layout() - .foldSealKey(after_round1.snap_generation, after_round1.snap_attempt)).exists); + EXPECT_TRUE(headObj(*backend, store->layout() + .foldSealKey(after_round1.snap_generation, after_round1.snap_attempt)).has_value()); dropRefTransition(*backend, store->layout(), ns, "tbl", r); gc.runRegularRound(); // round 2: re-acquire (bump lease.seq) -> fresh attempt at its fold - const auto after_round2 = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_round2 = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes); EXPECT_EQ(after_round2.snap_attempt, after_round2.lease.seq); EXPECT_GT(after_round2.snap_attempt, after_round1.snap_attempt); // per-round monotone attempt EXPECT_GT(after_round2.snap_generation, after_round1.snap_generation); - EXPECT_TRUE(backend->head(store->layout() - .foldSealKey(after_round2.snap_generation, after_round2.snap_attempt)).exists); + EXPECT_TRUE(headObj(*backend, store->layout() + .foldSealKey(after_round2.snap_generation, after_round2.snap_attempt)).has_value()); } /// ---- round-paced graduation suite (spec 2026-07-02 + Task-9 amendment; re-keyed off acks in v3 Task 6) ---- @@ -680,9 +705,10 @@ TEST(CASGCAckFloor, NoOpRoundDoesNotMutateRefShards) { std::set keys; String cursor; + OperationForTest op(*backend); for (;;) { - const ListPage page = backend->list(store->layout().namespaceStreamPrefix(fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*op).list(store->layout().namespaceStreamPrefix(fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) keys.insert(lk.key); if (page.next_cursor.empty()) @@ -699,7 +725,7 @@ TEST(CASGCAckFloor, NoOpRoundDoesNotMutateRefShards) const std::set after = listRefKeys(); EXPECT_EQ(before, after) << "a no-op GC round must not mutate the table's ref objects"; // The registry object is gone (Task 4); the fence never existed to write it. - EXPECT_FALSE(backend->get("p/gc/registry").has_value()); + EXPECT_FALSE(readObj(*backend, "p/gc/registry").has_value()); } /// The canonical pipeline: a blob condemned at round K stays present after the condemning round; the @@ -878,18 +904,18 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) auto store = openPoolForTest(backend); const Layout & layout = store->layout(); - // srid2's keeper claims ONE lease via `start` and is never renewed again — tests never enable + // srid2's renewer claims ONE lease via `start` and is never renewed again — tests never enable // the runtime-owned renewal worker (`background_watermark` defaults to false), so this alone models a // crashed process: a body that is live-shaped (not terminated, not fenced) but whose write token // never changes again. const String srid2 = "stale-server"; - CasRequests keeper_requests = openRequestsForTest(backend); - MountLeaseKeeper srid2_keeper(keeper_requests, keeper_requests, layout, srid2, DB::UInt128(0x2222), + CasRequests renewer_requests = openRequestsForTest(backend); + MountLeaseRenewer srid2_renewer(renewer_requests, renewer_requests, layout, srid2, DB::UInt128(0x2222), /*writer_epoch=*/1, std::chrono::milliseconds(100), [] { return 1000u; }, [] { return 0u; }, {}, std::chrono::milliseconds(0), [] { return 0u; }); - srid2_keeper.start(); - ASSERT_FALSE(decodeMountLease(backend->get(layout.mountKey(srid2))->bytes).gc_fenced); + srid2_renewer.start(); + ASSERT_FALSE(decodeMountLease(readObj(*backend, layout.mountKey(srid2))->bytes).gc_fenced); // The fence-out threshold on the GC leader's OWN monotonic clock — mirrors the production formula // in `Gc::runRegularRound` (ttl + 5% drift allowance + one round's worth of renewal slack). @@ -924,7 +950,7 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) const RoundReport rep = gc.runRegularRound(); EXPECT_EQ(rep.fence_outs, 1u); // exactly one dead mount fenced-out this round - const MountLease fenced = decodeMountLease(backend->get(layout.mountKey(srid2))->bytes); + const MountLease fenced = decodeMountLease(readObj(*backend, layout.mountKey(srid2))->bytes); EXPECT_TRUE(fenced.gc_fenced); // Exactly one GcFenceOut audit row was emitted, naming srid2 in its detail. @@ -944,7 +970,7 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) // srid2's writer comes back and tries to renew: its held token was invalidated by the fence rewrite, // so synchronous renewal returns a terminal failure. (It renews on its own clock; liveness is irrelevant — the token guard // trips regardless.) - const MountRenewResult renewed = srid2_keeper.renew(MountRenewOperationEnvironment{}); + const MountRenewResult renewed = srid2_renewer.renew(MountRenewOperationEnvironment{}); ASSERT_EQ(renewed.outcome, MountRenewOutcome::Terminal); ASSERT_NE(renewed.failure, nullptr); EXPECT_THROW(std::rethrow_exception(renewed.failure), DB::Exception); @@ -973,12 +999,12 @@ TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) // A stale mount, exactly as `ExpiredMountFencedOutAndExcluded`: one claim, never renewed again. const String srid2 = "stale-server"; - CasRequests keeper_requests = openRequestsForTest(backend); - MountLeaseKeeper srid2_keeper(keeper_requests, keeper_requests, layout, srid2, DB::UInt128(0x2222), + CasRequests renewer_requests = openRequestsForTest(backend); + MountLeaseRenewer srid2_renewer(renewer_requests, renewer_requests, layout, srid2, DB::UInt128(0x2222), /*writer_epoch=*/1, std::chrono::milliseconds(100), [] { return 1000u; }, [&] { return fake_boot; }); - srid2_keeper.start(); - ASSERT_FALSE(decodeMountLease(backend->get(layout.mountKey(srid2))->bytes).gc_fenced); + srid2_renewer.start(); + ASSERT_FALSE(decodeMountLease(readObj(*backend, layout.mountKey(srid2))->bytes).gc_fenced); const uint64_t ttl_ms = static_cast(store->poolConfig().mount_lease_ttl_ms.count()); const uint64_t threshold_ms = ttl_ms + ttl_ms / 20 @@ -996,7 +1022,7 @@ TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) const RoundReport rep2 = gc.runRegularRound(); EXPECT_EQ(rep2.fence_outs, 1u) << "Gc's default mono_ms_fn must track the Pool's injected boot clock, not the real wall clock"; - EXPECT_TRUE(decodeMountLease(backend->get(layout.mountKey(srid2))->bytes).gc_fenced); + EXPECT_TRUE(decodeMountLease(readObj(*backend, layout.mountKey(srid2))->bytes).gc_fenced); } /// A redelete of a blob the writer RECREATED (fresh incarnation) between the pending publish and the @@ -1095,15 +1121,15 @@ TEST(CASGCAckFloor, ResumeAfterCrashBetweenRetiredPutAndStateCas) const String pending_key = store->layout().blobKey(blob_id); const std::optional doomed = op.head(pending_key, Retry::once()); ASSERT_TRUE(doomed); - ASSERT_TRUE(pending_entry.token.matches(doomed->incarnation)); - ASSERT_EQ(op.remove(pending_key, doomed->incarnation, Retry::once()), Removal::Removed); + ASSERT_TRUE(pending_entry.token.matches(doomed->etag)); + ASSERT_EQ(op.remove(pending_key, doomed->etag, Retry::once()), Removal::Removed); - const uint64_t round_before = decodeGcState(backend->get(store->layout().gcStateKey())->bytes).round; + const uint64_t round_before = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes).round; Gc gc2(store, kGc); const RoundReport rep = runRegularRoundReclaiming(gc2); store->renewWatermarkOnce(); EXPECT_EQ(rep.absent, 1u); // the replayed delete found the object already gone - const uint64_t round_after = decodeGcState(backend->get(store->layout().gcStateKey())->bytes).round; + const uint64_t round_after = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes).round; EXPECT_GT(round_after, round_before); // the round completed (no wedge) EXPECT_FALSE(currentEntryFor(*backend, store->layout(), blob).has_value()); } @@ -1160,8 +1186,8 @@ TEST(CASGCAckFloor, AbsentBlobSettlesAsAbsentWithoutASpeculativeConditionalRemov const String blob_key = store->layout().blobKey(blob_id); const std::optional doomed = op.head(blob_key, Retry::once()); ASSERT_TRUE(doomed); - ASSERT_TRUE(pending_entry.token.matches(doomed->incarnation)); - ASSERT_EQ(op.remove(blob_key, doomed->incarnation, Retry::once()), Removal::Removed); + ASSERT_TRUE(pending_entry.token.matches(doomed->etag)); + ASSERT_EQ(op.remove(blob_key, doomed->etag, Retry::once()), Removal::Removed); ASSERT_FALSE(op.head(blob_key, Retry::once())); backend->watch(blob_key); @@ -1211,8 +1237,8 @@ TEST(CASGCCondemnMarker, SwallowedMarkerWriteCarriesEntryInsteadOfDeleting) /// plane's lease-bound policies keep their real clock. std::atomic engine_now_ms{0}; std::atomic engine_sleeps{0}; - store->gcRequests().setNowFnForTest([&] { return engine_now_ms.load(); }); - store->gcRequests().setSleepFnForTest([&](uint64_t pause_ms) + store->openRequests().setNowFnForTest([&] { return engine_now_ms.load(); }); + store->openRequests().setSleepFnForTest([&](uint64_t pause_ms) { engine_sleeps.fetch_add(1); engine_now_ms.fetch_add(pause_ms + 1); diff --git a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp index ed913c2c1a2b..d583b31df256 100644 --- a/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp +++ b/src/Disks/tests/gtest_cas_gc_arithmetic_intake.cpp @@ -58,7 +58,8 @@ std::optional coverageOf(Backend & backend, const Layout & layout, const UInt128 life_id = catalogLifeIdForTest(backend, layout, ns); for (uint64_t g = gen; ; --g) { - if (const auto got = backend.get(layout.foldSealKey(g, attempt))) + OperationForTest op(backend); + if (const auto got = (*op).read(layout.foldSealKey(g, attempt), Retry::once())) { const CasFoldSeal seal = decodeFoldSeal(got->bytes); const auto it = seal.ref_lives.find(life_id); @@ -482,7 +483,10 @@ TEST(CASGCArithmeticIntake, CorruptBodyClampsOneNamespaceWhileAnotherFolds) const RootNamespace ns_b{"00/bb@cas@"}; publishAt(*backend, layout, ns_a, RefTxnId{1, 1}, "ref_1", 1, DB::UInt128(1), /*birth=*/true); - backend->putIfAbsent(layout.refLogKey(fixture::fixtureLife(ns_a), RefTxnId{1, 2}), "this is not a cas_ref_log object"); + { + OperationForTest op(*backend); + (*op).create(layout.refLogKey(fixture::fixtureLife(ns_a), RefTxnId{1, 2}), "this is not a cas_ref_log object", Retry::once()); + } writeRecoverableCkptForRawFixture(*backend, layout, ns_a, RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, diff --git a/src/Disks/tests/gtest_cas_gc_attempt.cpp b/src/Disks/tests/gtest_cas_gc_attempt.cpp index e4e5ff635290..546eda954a0e 100644 --- a/src/Disks/tests/gtest_cas_gc_attempt.cpp +++ b/src/Disks/tests/gtest_cas_gc_attempt.cpp @@ -47,7 +47,8 @@ ManifestRef ref(const String &, uint64_t seq, uint64_t inst) /// Whether a blob's body object is present in the backend (HEADs the object key directly). bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash) { - return b.head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + OperationForTest op(b); + return (*op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)}), Retry::once()).has_value(); } /// Whether the CURRENT retired list (any gc-shard) still holds an entry — the ack-floor deletion pipeline @@ -56,7 +57,7 @@ bool anyRetiredPending(const PoolPtr & s) { /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. - return anyCondemnedInSeal(s->backend(), s->layout()); + return anyCondemnedInSeal(*s->poolBackendPtr(), s->layout()); } /// Drive regular GC to a fixpoint over the ACK-FLOOR round (advancing the store's own mount ack after each @@ -135,12 +136,13 @@ TEST(CASGCAttempt, DeposedFoldAttemptDoesNotWedge) publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, r); Gc gc(store, kGcA); + OperationForTest raw_op(*backend); // Round 1 (honest): fold the +1 so the blob is pinned in the in-degree generation, and adopt the // first (snap_generation, snap_attempt). runRegularRoundReclaiming(gc); EXPECT_EQ(inDegreeOf(*backend, store->layout(), DB::UInt128(1)), 1) << "blob pinned by the committed ref"; - const auto after_fold = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_fold = decodeGcState((*raw_op).read(store->layout().gcStateKey(), Retry::once())->bytes); ASSERT_EQ(after_fold.snap_attempt, after_fold.lease.seq); ASSERT_GT(after_fold.snap_generation, 0u); @@ -155,7 +157,7 @@ TEST(CASGCAttempt, DeposedFoldAttemptDoesNotWedge) EXPECT_ANY_THROW(runRegularRoundReclaiming(gc)); // ABORTED: round-commit CAS denied backend->arm_interrupt = false; - const auto after_deposed = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_deposed = decodeGcState((*raw_op).read(store->layout().gcStateKey(), Retry::once())->bytes); EXPECT_EQ(after_deposed.snap_generation, after_fold.snap_generation) << "the denied round-commit CAS must NOT advance the adopted generation"; EXPECT_EQ(after_deposed.snap_attempt, after_fold.snap_attempt) @@ -170,9 +172,9 @@ TEST(CASGCAttempt, DeposedFoldAttemptDoesNotWedge) const uint64_t a1 = after_fold.lease.seq + 1; // round 2 renewed the lease => seq bumped once const uint64_t g_f = after_fold.snap_generation + 1; // the generation the deposed fold minted EXPECT_NE(a1, after_deposed.snap_attempt) << "the deposed attempt must differ from the adopted one"; - EXPECT_TRUE(backend->head(store->layout().foldSealKey(g_f, a1)).exists) + EXPECT_TRUE((*raw_op).head(store->layout().foldSealKey(g_f, a1), Retry::once()).has_value()) << "the deposed leader's fold seal is durable under its own (unadopted) attempt a1"; - EXPECT_FALSE(backend->head(store->layout().foldSealKey(g_f, after_deposed.snap_attempt)).exists) + EXPECT_FALSE((*raw_op).head(store->layout().foldSealKey(g_f, after_deposed.snap_attempt), Retry::once()).has_value()) << "no fold seal exists under the still-adopted attempt at the deposed fold generation (orphan is invisible)"; // An HONEST GC to a fixpoint (CAS now allowed). The KEY property: with attempt-scoping this SUCCEEDS — @@ -189,7 +191,7 @@ TEST(CASGCAttempt, DeposedFoldAttemptDoesNotWedge) // GC advanced past the deposed attempt: the adopted (snap_generation, snap_attempt) moved on, and the // adopted attempt is a fresh one (never the deposed a1). - const auto after_drain = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_drain = decodeGcState((*raw_op).read(store->layout().gcStateKey(), Retry::once())->bytes); EXPECT_GT(after_drain.snap_generation, after_fold.snap_generation) << "completion advanced the generation"; EXPECT_NE(after_drain.snap_attempt, a1) << "the drained round never adopted the deposed attempt a1"; } diff --git a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp index 7fd166eb7805..076487c7d88b 100644 --- a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp +++ b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp @@ -54,12 +54,13 @@ using CountingHintHoleBackend = DB::Cas::tests::HintHoleBackendOn coverageOf(Backend & backend, const Layout & layout, const RootNamespace & ns) { + DB::Cas::tests::OperationForTest op(backend); const uint64_t gen = currentGenerationOf(backend, layout); const uint64_t attempt = currentAttemptOf(backend, layout); const UInt128 life_id = catalogLifeIdForTest(backend, layout, ns); for (uint64_t g = gen; ; --g) { - if (const auto got = backend.get(layout.foldSealKey(g, attempt))) + if (const auto got = (*op).read(layout.foldSealKey(g, attempt), Retry::standard())) { const CasFoldSeal seal = decodeFoldSeal(got->bytes); const auto it = seal.ref_lives.find(life_id); @@ -392,7 +393,10 @@ TEST(CASGCBoundedWalk, ARawRecordBeyondTheCommittedFrontierCannotSuppressDestruc EXPECT_EQ(backend->deleteTotal(), 1u) << "the committed frontier permits the round's immediate manifest cleanup. Deleted:" << deletedKeysMessage(*backend); - EXPECT_TRUE(backend->head(layout.blobKey(legacyMetaTestRef(blob))).exists); + { + DB::Cas::tests::OperationForTest head_op(*backend); + EXPECT_TRUE((*head_op).head(layout.blobKey(legacyMetaTestRef(blob)), Retry::standard()).has_value()); + } /// The raw F+1 record remains outside the CTE; it cannot defer the normal destructive pipeline. backend->disarm(); diff --git a/src/Disks/tests/gtest_cas_gc_fold.cpp b/src/Disks/tests/gtest_cas_gc_fold.cpp index 5e85ac7de218..b8f68f3f2080 100644 --- a/src/Disks/tests/gtest_cas_gc_fold.cpp +++ b/src/Disks/tests/gtest_cas_gc_fold.cpp @@ -21,6 +21,18 @@ ManifestRef ref(const String &, uint64_t seq, uint64_t inst) { return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = static_cast(inst)}; } + +std::optional readOf(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +bool headExists(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} } /// Committed new_manifest => +1 per blob entry (BlobInDegreeMatchesActiveManifests). @@ -38,12 +50,12 @@ TEST(CASGCFold, FoldAdoptsAttemptEqualsLeaseSeq) Gc gc(store, kGc); gc.runRegularRound(); - const auto st = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); EXPECT_EQ(st.snap_attempt, st.lease.seq); EXPECT_GT(st.snap_generation, 0u); /// The one-pass round's fold seal is durable under (snap_generation, snap_attempt) — the adopted /// attempt locates it (a seal under any other attempt would be unadopted debris). - EXPECT_TRUE(backend->head(store->layout().foldSealKey(st.snap_generation, st.snap_attempt)).exists); + EXPECT_TRUE(headExists(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))); } TEST(CASGCFold, CommittedAddEmitsPlusOnePerBlob) @@ -142,7 +154,7 @@ TEST(CASGCFold, PromoteOfActivatedPrecommitEmitsNoDelta) promoteTransition(*backend, store->layout(), ns, DB::UInt128(7), "tbl", r); gc.runRegularRound(); EXPECT_EQ(inDegreeOf(*backend, store->layout(), DB::UInt128(1)), 1); // unchanged, still pinned - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); // not condemned + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); // not condemned } /// Committed add naming a MISSING body (404) => clamp + anomaly, never a guessed +1, never a throw. @@ -173,7 +185,10 @@ TEST(CASGCFold, RefMismatchFailsClosed) bad.root_namespace_id = ns; bad.entries = {blobEntryFor("a", DB::UInt128(1))}; bad.payload_digest = computePayloadDigest(bad); - backend->putIfAbsent(store->layout().manifestKey(ManifestId{ns, r}), encodePartManifest(bad)); + { + OperationForTest op(*backend); + (*op).create(store->layout().manifestKey(ManifestId{ns, r}), encodePartManifest(bad), Retry::standard()); + } publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, r); Gc gc(store, kGc); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&]{ gc.runRegularRound(); }); @@ -228,9 +243,9 @@ TEST(CASGCFold, EmptyDeltaShardCarriesParentRunRef) Gc gc(store, kGc); gc.runRegularRound(); // round 1: folds the +1, seals the gen-1 blob_target run - const auto st1 = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st1 = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); const auto parent_seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); ASSERT_EQ(parent_seal.blob_target_runs.size(), 1u); const RunRef parent_ref = parent_seal.blob_target_runs.front(); @@ -240,10 +255,10 @@ TEST(CASGCFold, EmptyDeltaShardCarriesParentRunRef) EXPECT_EQ(backend->ioCountForKeysContaining("/blob_target/"), 0u) << "idle round must not GET/getStream/PUT any blob_target run object"; - const auto st2 = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st2 = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); EXPECT_GT(st2.snap_generation, st1.snap_generation); const auto new_seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes); ASSERT_EQ(new_seal.blob_target_runs.size(), 1u); const RunRef carried = new_seal.blob_target_runs.front(); EXPECT_EQ(carried.key, parent_ref.key) << "carried ref points at the PARENT generation's run key"; @@ -306,13 +321,13 @@ TEST(CASGCFold, PreviewResolvesCarriedRef) Gc gc(store, kGc); gc.runRegularRound(); // gen 1: blob referenced, in-degree 1 - const auto st1 = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st1 = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); gc.runRegularRound(); // gen 2: no delta, no retired => pure ref-carry (ref points back at gen 1) - const auto st2 = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st2 = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); ASSERT_GT(st2.snap_generation, st1.snap_generation); const auto seal2 = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes); ASSERT_EQ(seal2.blob_target_runs.size(), 1u); ASSERT_EQ(seal2.blob_target_runs.front().key_generation, st1.snap_generation) << "the current seal's ref physically lives at the parent generation (carried, not reconstructed)"; @@ -335,13 +350,14 @@ namespace String corruptSealedRunChecksum(InMemoryBackend & backend, const Layout & layout, const GcState & st) { const String sk = layout.foldSealKey(st.snap_generation, st.snap_attempt); - const auto existing = backend.get(sk); + const auto existing = readOf(backend, sk); auto seal = decodeFoldSeal(existing->bytes); if (seal.blob_target_runs.empty()) return {}; const String run_key = seal.blob_target_runs.front().key; seal.blob_target_runs.front().checksum = seal.blob_target_runs.front().checksum + 1; - backend.putOverwrite(sk, encodeFoldSeal(seal), existing->token); + OperationForTest op(backend); + (*op).replace(sk, encodeFoldSeal(seal), existing->etag, Retry::standard()); return run_key; } } @@ -359,7 +375,7 @@ TEST(CASGCFold, PreviewDeletesSealChecksumMismatchFailsClosed) Gc gc(store, kGc); gc.runRegularRound(); // seals gen-1 with one blob_target run - const auto st = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); ASSERT_FALSE(corruptSealedRunChecksum(*backend, store->layout(), st).empty()); // A deletion preview must never be derived from an unverified run: fail closed. @@ -384,7 +400,7 @@ TEST(CASGCFold, FsckSealChecksumMismatchCataloguedAndAuditCompletes) Gc gc(store, kGc); gc.runRegularRound(); - const auto st = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st = decodeGcState(readOf(*backend, store->layout().gcStateKey())->bytes); /// A present-but-unreferenced blob (written AFTER the round so GC never touches it) is what makes /// fsck enter its GC-pipeline classification path (guarded by a non-empty unreferenced set), which @@ -439,7 +455,7 @@ TEST(CASGCFold, MidLogClampPreservesEarlierRemovalBodyAndRecovers) const RoundReport clamp_report = gc.runRegularRound(); EXPECT_TRUE(clamp_report.hasAnomaly(ns, /*shard*/0)) << "the missing B body must clamp this log"; EXPECT_LT(foldCursorOf(*backend, store->layout(), ns, 0), log_seq) << "the clamp halts the cursor below the log"; - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, a})).exists) + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, a}))) << "A's body must survive the clamp round: its `-1` was staged, not merged, so no post-CAS delete " "reclaimed it -- otherwise the re-fold would clamp on A's missing body forever"; EXPECT_EQ(inDegreeOf(*backend, store->layout(), DB::UInt128(1)), 1) << "A's `-1` was not adopted (clamp)"; @@ -518,7 +534,7 @@ TEST(CASGCFold, SingleAnomalySuppressesEveryDestructiveActionInTheRound) EXPECT_EQ(rep.deleted, 0u); EXPECT_EQ(rep.redeleted, 0u); EXPECT_EQ(rep.graduated, 0u); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, a})).exists); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, a}))); EXPECT_EQ(inDegreeOf(*backend, store->layout(), DB::UInt128(1)), 1); } @@ -564,7 +580,10 @@ TEST(CASGCFold, RoundSideAnomalySuppressesRefLogCleanupWhileRemovalDebrisStaysJa const String debris_key = layout.namespaceFilesPrefix(CasRefCatalog::lifeIfCataloged(op, layout, ns_removed).value()) + "leftover_verbatim_file"; - backend->putIfAbsent(debris_key, "debris"); + { + OperationForTest debris_op(*backend); + (*debris_op).create(debris_key, "debris", Retry::standard()); + } const ManifestRef removed_body = ref("srv-r:1", 1, 0xEE); writeManifestRaw(*backend, layout, ns_removed, removed_body, {blobEntryFor("r", DB::UInt128(9))}); const String debris_manifest_key = layout.manifestKey(ManifestId{ns_removed, removed_body}); @@ -587,7 +606,7 @@ TEST(CASGCFold, RoundSideAnomalySuppressesRefLogCleanupWhileRemovalDebrisStaysJa .last_epoch_seal = std::nullopt, }); const String covered_log_key = layout.refLogKey(fixture::fixtureLife(ns_covered), RefTxnId{1, cv1}); - ASSERT_TRUE(backend->head(covered_log_key).exists); + ASSERT_TRUE(headExists(*backend, covered_log_key)); /// Trigger the clamp in ns_clamp: drop committed A, add precommit B whose body is absent. writeManifestRaw(*backend, layout, ns_clamp, b, {blobEntryFor("b", DB::UInt128(2))}); @@ -604,13 +623,13 @@ TEST(CASGCFold, RoundSideAnomalySuppressesRefLogCleanupWhileRemovalDebrisStaysJa EXPECT_EQ(rep.graduated, 0u); /// Removal folding never performs lifecycle-specific physical cleanup, with or without a clamp. - EXPECT_TRUE(backend->head(debris_manifest_key).exists) + EXPECT_TRUE(headExists(*backend, debris_manifest_key)) << "removed manifest debris remains ordinary orphan-sweep work"; - EXPECT_TRUE(backend->head(debris_key).exists) + EXPECT_TRUE(headExists(*backend, debris_key)) << "removed verbatim-file debris remains ordinary janitor work"; /// `cleanupRefObjects` must not have deleted anything anywhere this round. - EXPECT_TRUE(backend->head(covered_log_key).exists) + EXPECT_TRUE(headExists(*backend, covered_log_key)) << "a clamp anywhere in the round must suppress ref-log cleanup pool-wide, even for an unrelated live table"; /// Heal the clamp and run a clean round. Ordinary ref-log cleanup resumes, while removal debris @@ -618,9 +637,9 @@ TEST(CASGCFold, RoundSideAnomalySuppressesRefLogCleanupWhileRemovalDebrisStaysJa writeManifestRaw(*backend, layout, ns_clamp, b, {blobEntryFor("b", DB::UInt128(2))}); const RoundReport clean_rep = runRegularRoundReclaiming(gc); EXPECT_FALSE(clean_rep.hasAnomaly(ns_clamp, /*shard*/0)); - EXPECT_TRUE(backend->head(debris_manifest_key).exists) + EXPECT_TRUE(headExists(*backend, debris_manifest_key)) << "a clamp-free fold still performs no lifecycle-specific manifest deletion"; - EXPECT_TRUE(backend->head(debris_key).exists) + EXPECT_TRUE(headExists(*backend, debris_key)) << "a clamp-free fold still performs no lifecycle-specific verbatim-file deletion"; - EXPECT_FALSE(backend->head(covered_log_key).exists) << "a clamp-free round cleans the covered ref-log"; + EXPECT_FALSE(headExists(*backend, covered_log_key)) << "a clamp-free round cleans the covered ref-log"; } diff --git a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp index 57839fd4a3fa..d8c90fa7dc1c 100644 --- a/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp +++ b/src/Disks/tests/gtest_cas_gc_frontier_gate.cpp @@ -270,11 +270,15 @@ class PostFoldUnreadableTerminalBackend final : public CountingBackend } /// The test's own look at the key the fault hides, taken through the same primitive with the fault - /// suspended -- there is no second door to the store. + /// suspended -- there is no second door to the store. A fresh open-fence CasRequests over `this` + /// (aliasing, owns nothing): every Backend call needs a CasRequests-minted TransportAccess, and this + /// method has no caller-supplied one to reuse. bool existsIgnoringFault(const String & key) { bypass_fault = true; - const bool present = CountingBackend::head(key).exists; + CasRequests requests(BackendPtr(std::shared_ptr(), this), Fence::open()); + CasOperation op = requests.admit(); + const bool present = op.head(key, Retry::once()).has_value(); bypass_fault = false; return present; } @@ -326,7 +330,7 @@ struct CompletedRemovingFixture }; CompletedRemovingFixture seedCompletedRemoving( - DrainRaceBackend & backend, CasOperation & op, const PoolPtr & store, const UInt128 & lease_owner) + CasOperation & op, const PoolPtr & store, const UInt128 & lease_owner) { const Layout & layout = store->layout(); CompletedRemovingFixture fixture{ @@ -344,7 +348,7 @@ CompletedRemovingFixture seedCompletedRemoving( .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, }); - backend.putIfAbsent(fixture.checkpoint_key, fixture.checkpoint_bytes); + op.create(fixture.checkpoint_key, fixture.checkpoint_bytes, Retry::once()); EXPECT_TRUE(store->namespaceFilesLifeIfReadable(fixture.ns)); CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { @@ -361,7 +365,7 @@ CompletedRemovingFixture seedCompletedRemoving( .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); - backend.putIfAbsent(layout.foldSealKey(1, 1), encodeFoldSeal(parent)); + op.create(layout.foldSealKey(1, 1), encodeFoldSeal(parent), Retry::once()); GcState state; state.round = 1; @@ -369,13 +373,13 @@ CompletedRemovingFixture seedCompletedRemoving( state.snap_generation = 1; state.snap_attempt = 1; state.lease = GcLease{.owner = lease_owner, .seq = 1}; - backend.putIfAbsent(layout.gcStateKey(), encodeGcState(state)); + op.create(layout.gcStateKey(), encodeGcState(state), Retry::once()); return fixture; } void seedCompletedRemovingBatch( - DrainRaceBackend & backend, CasOperation & op, const PoolPtr & store, const UInt128 & lease_owner, + CasOperation & op, const PoolPtr & store, const UInt128 & lease_owner, size_t count) { const Layout & layout = store->layout(); @@ -409,8 +413,7 @@ void seedCompletedRemovingBatch( .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); - ASSERT_EQ(backend.putIfAbsent(layout.foldSealKey(1, 1), encodeFoldSeal(parent)).outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.foldSealKey(1, 1), encodeFoldSeal(parent), Retry::once()))); GcState state; state.round = 1; @@ -418,7 +421,7 @@ void seedCompletedRemovingBatch( state.snap_generation = 1; state.snap_attempt = 1; state.lease = GcLease{.owner = lease_owner, .seq = 1}; - ASSERT_EQ(backend.putIfAbsent(layout.gcStateKey(), encodeGcState(state)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.gcStateKey(), encodeGcState(state), Retry::once()))); } enum class CompetingCatalogOutcome : uint8_t @@ -433,13 +436,15 @@ class CASGCCompletedRemovalFenceRace : public testing::TestWithParambytes); state.lease.owner = new_owner; ++state.lease.seq; - ASSERT_EQ(backend.casPut(layout.gcStateKey(), encodeGcState(state), got->token).outcome, - CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.gcStateKey(), encodeGcState(state), got->etag, Retry::once()))); } size_t findJournalAfter(const std::vector & journal, const String & entry, size_t after) @@ -649,7 +654,8 @@ TEST(CASGCFrontierGate, AHiddenEdgeIsFoundByTheExactKeyProbeAndSavesTheBlobOnACo } ASSERT_TRUE(verdict.saw_fold) << "no round folded, so none published a gate verdict"; - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "the blob a hidden namespace still owns must survive"; EXPECT_TRUE(verdict.frontier_complete) << "the exact-key probe reads at `cursor + 1` and a LIST hole cannot hide an exact key, so the " @@ -698,7 +704,8 @@ TEST(CASGCFrontierGate, TheSameBlobDrainsOnceHiddenGenuinelyProvesItsOwnFrontier drive(store, gc, /*rounds*/ 5, UniversePolicy::Authoritative); - EXPECT_FALSE(backend->head(blobKeyOf(layout, blob)).exists) + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "both namespaces genuinely proved their frontier and the blob is genuinely unreferenced -- " "the round must still be able to reclaim it"; } @@ -720,7 +727,8 @@ TEST(CASGCFrontierGate, AKnownNamespaceIsProbedByExactKeyAndItsHiddenEdgeSavesTh Gc gc(store, kGc); drive(store, gc, /*rounds*/ 5, UniversePolicy::Authoritative); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "the cursor kept the namespace in the universe, so its frontier was probed and its edge folded"; } @@ -823,7 +831,8 @@ TEST(CASGCFrontierGate, EveryInventoriedDestructiveSiteIsInertUnderSuppression) << "with no universe supplied the frontier can never be complete, whatever the probes proved"; EXPECT_TRUE(verdict.suppress_destructive); expectEveryDeleteFamilyInert(*backend, "no universe supplied"); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists); + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()); /// The control: the identical pool DOES reclaim at those sites on the production path, so the zeros /// above are the gate at work and not an empty work queue -- and it is also what makes the "on that @@ -831,7 +840,7 @@ TEST(CASGCFrontierGate, EveryInventoriedDestructiveSiteIsInertUnderSuppression) drive(store, gc, /*rounds*/ 4, UniversePolicy::kDefault); EXPECT_GT(backend->deleteTotal(), 0u) << "the work queue was real -- a round with a universe drains it"; - EXPECT_FALSE(backend->head(blobKeyOf(layout, blob)).exists); + EXPECT_FALSE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()); } /// (1) ONE ANOMALY. A namespace whose `_ckpt` is present but undecodable records the "no usable @@ -860,7 +869,7 @@ TEST(CASGCFrontierGate, AnUndecodableCheckpointAnomalySuppressesEveryDeleteFamil const std::optional damaged_ckpt = readCkpt(op, layout, *damaged_life); ASSERT_TRUE(damaged_ckpt.has_value()) << "the publish must have left a `_ckpt` to damage"; ASSERT_TRUE(std::holds_alternative(op.replace( - layout.refCkptKey(*damaged_life), "not a checkpoint", damaged_ckpt->incarnation, Retry::once()))); + layout.refCkptKey(*damaged_life), "not a checkpoint", damaged_ckpt->etag, Retry::once()))); backend->resetCounts(); std::vector anomaly_counts; @@ -874,7 +883,7 @@ TEST(CASGCFrontierGate, AnUndecodableCheckpointAnomalySuppressesEveryDeleteFamil << "the undecodable `_ckpt` must be RECORDED, not silently absorbed -- a silent exit would make " "this test pass for the wrong reason"; expectEveryDeleteFamilyInert(*backend, "one anomaly"); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists); + EXPECT_TRUE(op.head(blobKeyOf(layout, blob), Retry::once()).has_value()); } /// (2) ONE CARRIED HOLD. The gate's second term reads the SEAL, not this round's anomaly list, so the @@ -921,7 +930,8 @@ TEST(CASGCFrontierGate, ACarriedHoldSuppressesEveryDeleteFamily) store->renewWatermarkOnce(); } expectEveryDeleteFamilyInert(*backend, "one carried hold"); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists); + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()); } /// (3c) THE PROBE BUDGET. A namespace with a sealed cursor, no `_ckpt` and no listing left can be proven @@ -954,7 +964,7 @@ TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSuppressesEveryDeleteFamily) ASSERT_TRUE(quiet_life.has_value()); const std::optional quiet_ckpt = readCkpt(op, layout, *quiet_life); ASSERT_TRUE(quiet_ckpt.has_value()) << "there must be a `_ckpt` to remove"; - ASSERT_EQ(op.remove(layout.refCkptKey(*quiet_life), quiet_ckpt->incarnation, Retry::once()), + ASSERT_EQ(op.remove(layout.refCkptKey(*quiet_life), quiet_ckpt->etag, Retry::once()), Removal::Removed); backend->hidePrefix(layout.namespaceStreamPrefix(*quiet_life)); @@ -972,7 +982,7 @@ TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSuppressesEveryDeleteFamily) EXPECT_FALSE(verdict.frontier_complete); EXPECT_TRUE(verdict.suppress_destructive); expectEveryDeleteFamilyInert(*backend, "exhausted probe budget"); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists); + EXPECT_TRUE(op.head(blobKeyOf(layout, blob), Retry::once()).has_value()); } /// `frontier_proven == frontier_namespaces` is `0 == 0` -- vacuously TRUE -- on an empty universe, which @@ -998,7 +1008,7 @@ TEST(CASGCFrontierGate, ADecodedTokenBearingEmptyCatalogCompletesTheFrontierAndD const BlobRef blob_ref = legacyMetaTestRef(blob); const std::optional blob_observed = op.head(layout.blobKey(blob_ref), Retry::once()); ASSERT_TRUE(blob_observed) << "the seeded blob body must be present before it is condemned"; - const PersistedIncarnation blob_token = PersistedIncarnation::capture(blob_observed->incarnation); + const PersistedEtag blob_token = PersistedEtag::capture(blob_observed->etag); injectRetire(*backend, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = blob_ref, .token = blob_token, .size = 0}}); store->renewWatermarkOnce(); @@ -1023,7 +1033,7 @@ TEST(CASGCFrontierGate, ADecodedTokenBearingEmptyCatalogCompletesTheFrontierAndD /// Bound the drive at that plus one (5): enough slack for the fixture's own cadence to be measured /// without hand-counting rounds against this gate, but tight enough that a real regression in the /// confirm/retry cadence still fails loudly instead of silently absorbing into a generous loop. - ASSERT_TRUE(backend->head(layout.blobKey(blob_ref)).exists) + ASSERT_TRUE(op.head(layout.blobKey(blob_ref), Retry::once()).has_value()) << "the scenario starts with the condemned blob present, or the loop below measures nothing"; constexpr int kMaxRounds = 5; /// measured cadence (4) + 1; see the comment above @@ -1036,11 +1046,11 @@ TEST(CASGCFrontierGate, ADecodedTokenBearingEmptyCatalogCompletesTheFrontierAndD int round_blob_vanished = -1; /// the delete side GateVerdict last; int rounds_run = 0; - for (int i = 0; i < kMaxRounds && backend->head(layout.blobKey(blob_ref)).exists; ++i) + for (int i = 0; i < kMaxRounds && op.head(layout.blobKey(blob_ref), Retry::once()).has_value(); ++i) { last = runRoundCapturingGate(store, gc, UniversePolicy::Authoritative); ++rounds_run; - const bool still_present = backend->head(layout.blobKey(blob_ref)).exists; + const bool still_present = op.head(layout.blobKey(blob_ref), Retry::once()).has_value(); if (round_gate_opened_while_present < 0 && last.saw_fold && !last.suppress_destructive && still_present) round_gate_opened_while_present = rounds_run; if (round_blob_vanished < 0 && !still_present) @@ -1064,7 +1074,7 @@ TEST(CASGCFrontierGate, ADecodedTokenBearingEmptyCatalogCompletesTheFrontierAndD ASSERT_GT(round_blob_vanished, round_gate_opened_while_present) << "the delete must be a round STRICTLY LATER than the one that opened the gate, never the same " "round -- a round that both graduates and deletes in one step would hide the two-phase split"; - EXPECT_FALSE(backend->head(layout.blobKey(blob_ref)).exists) + EXPECT_FALSE(op.head(layout.blobKey(blob_ref), Retry::once()).has_value()) << "a proved-empty universe is a COMPLETE frontier, not a suppressed one -- the condemned blob " "must drain through the ordinary two-phase pipeline instead of leaking forever"; } @@ -1087,7 +1097,7 @@ TEST(CASGCFrontierGate, AZeroWalkableFrontierWithACreatingCatalogRowIsNotProvedE const BlobRef blob_ref = legacyMetaTestRef(blob); const std::optional blob_observed = op.head(layout.blobKey(blob_ref), Retry::once()); ASSERT_TRUE(blob_observed) << "the seeded blob body must be present before it is condemned"; - const PersistedIncarnation blob_token = PersistedIncarnation::capture(blob_observed->incarnation); + const PersistedEtag blob_token = PersistedEtag::capture(blob_observed->etag); injectRetire(*backend, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = blob_ref, .token = blob_token, .size = 0}}); store->renewWatermarkOnce(); @@ -1120,7 +1130,7 @@ TEST(CASGCFrontierGate, AZeroWalkableFrontierWithACreatingCatalogRowIsNotProvedE EXPECT_TRUE(v.suppress_destructive); } expectEveryDeleteFamilyInert(*backend, "Creating-only catalog"); - EXPECT_TRUE(backend->head(layout.blobKey(blob_ref)).exists); + EXPECT_TRUE(op.head(layout.blobKey(blob_ref), Retry::once()).has_value()); } /// The bootstrap-only absent-as-empty representation (`initializeEmptyForNewPool`) must never leak into @@ -1131,8 +1141,10 @@ TEST(CASGCFrontierGate, AnAbsentCatalogNeverReadsAsAnEmptyUniverse) auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); const Layout & layout = store->layout(); - const Token catalog_token = backend->head(layout.refCatalogKey()).token; - ASSERT_EQ(backend->deleteExact(layout.refCatalogKey(), catalog_token).kind, DeleteOutcome::Kind::Deleted); + OperationForTest raw_op(*backend); + const auto catalog_head = (*raw_op).head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(catalog_head.has_value()); + ASSERT_EQ((*raw_op).remove(layout.refCatalogKey(), catalog_head->etag, Retry::once()), Removal::Removed); Gc gc(store, kGc); backend->resetCounts(); @@ -1212,9 +1224,11 @@ TEST(CASGCFrontierGate, AMalformedCatalogNeverDecodesIntoAnEmptyProof) auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds=*/0); const Layout & layout = store->layout(); - const Token bootstrap_token = backend->head(layout.refCatalogKey()).token; - ASSERT_EQ(backend->casPut(layout.refCatalogKey(), c.bytes, bootstrap_token).outcome, - CasOutcome::Committed) << c.name; + OperationForTest raw_op(*backend); + const auto bootstrap_head = (*raw_op).head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(bootstrap_head.has_value()) << c.name; + ASSERT_TRUE(std::holds_alternative( + (*raw_op).replace(layout.refCatalogKey(), c.bytes, bootstrap_head->etag, Retry::once()))) << c.name; Gc gc(store, kGc); backend->resetCounts(); @@ -1244,7 +1258,7 @@ TEST(CASGCFrontierGate, AProvedEmptyCatalogUnderStageASuppressedStaysSuppressed) CasOperation op = requests.admit(); const std::optional blob_observed = op.head(layout.blobKey(blob_ref), Retry::once()); ASSERT_TRUE(blob_observed) << "the seeded blob body must be present before it is condemned"; - const PersistedIncarnation blob_token = PersistedIncarnation::capture(blob_observed->incarnation); + const PersistedEtag blob_token = PersistedEtag::capture(blob_observed->etag); injectRetire(*backend, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = blob_ref, .token = blob_token, .size = 0}}); store->renewWatermarkOnce(); @@ -1267,7 +1281,7 @@ TEST(CASGCFrontierGate, AProvedEmptyCatalogUnderStageASuppressedStaysSuppressed) EXPECT_TRUE(v.suppress_destructive); } expectEveryDeleteFamilyInert(*backend, "StageA_Suppressed over a proved-empty catalog"); - EXPECT_TRUE(backend->head(layout.blobKey(blob_ref)).exists); + EXPECT_TRUE(op.head(layout.blobKey(blob_ref), Retry::once()).has_value()); } /// THE BIRTH-AFTER-EMPTY-CUT BLOB RACE. The proved-empty exception's soundness rests on one hard fact: @@ -1310,7 +1324,9 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob dropRefTransition(*backend, layout, doomed, "ref_1", mref); runRegularRoundReclaiming(gc); /// condemns: durable Condemned meta store->renewWatermarkOnce(); - const Token condemned_token = backend->head(key).token; + const auto condemned_head = op.head(key, Retry::once()); + ASSERT_TRUE(condemned_head.has_value()); + const Etag condemned_token = condemned_head->etag; const auto condemned_meta = loadMetaForTest(*backend, layout, hash); ASSERT_TRUE(condemned_meta.has_value()); ASSERT_EQ(condemned_meta->meta.state, MetaState::Condemned) @@ -1343,7 +1359,7 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob /// until round R below, rather than letting it drain the ordinary way while `doomed` is still Live. gc.runRegularRound({}, /*allow_steal*/true, UniversePolicy::StageA_Suppressed); store->renewWatermarkOnce(); - EXPECT_TRUE(backend->head(key).exists) << "the pending delete must still be carried, not yet run"; + EXPECT_TRUE(op.head(key, Retry::once()).has_value()) << "the pending delete must still be carried, not yet run"; /// Round R: its pre-fold drain (`drainCompletedRemoving`) reads the round just above's /// `cleanup_evidence` and drops `doomed`'s catalog row BEFORE this round's own hot-scan `GET` -- @@ -1351,7 +1367,7 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob /// instant that cut is taken and races a real namespace birth into the window before round R's own /// pre-CAS delete phase runs. bool hook_fired = false; - Token fresh_token{}; + std::optional fresh_token; gc.setPostHotScanCatalogReadHookForTest([&]() { hook_fired = true; @@ -1365,8 +1381,10 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob build->precommitAdd(newborn, "ref_1", new_id); /// mints `newborn` via real createNamespace const PutBlobResult uploaded = build->putBlob(id, BlobSource::fromString(payload)); EXPECT_EQ(uploaded.ref, id); - fresh_token = backend->head(key).token; - EXPECT_NE(fresh_token, condemned_token) + const auto fresh_head = op.head(key, Retry::once()); + ASSERT_TRUE(fresh_head.has_value()); + fresh_token = fresh_head->etag; + EXPECT_NE(*fresh_token, condemned_token) << "the writer must have observed Condemned and resurrected -- a fresh token, not an adopt " "of the dying incarnation"; }); @@ -1378,10 +1396,12 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob EXPECT_TRUE(verdict.frontier_complete); EXPECT_FALSE(verdict.suppress_destructive); - EXPECT_TRUE(backend->head(key).exists) + const auto surviving_head = op.head(key, Retry::once()); + EXPECT_TRUE(surviving_head.has_value()) << "the resurrected incarnation must survive round R's delete"; - EXPECT_EQ(backend->head(key).token, fresh_token) << "and it is still the writer's incarnation"; - EXPECT_EQ(backend->deleteExact(key, condemned_token).kind, DeleteOutcome::Kind::TokenMismatch) + ASSERT_TRUE(fresh_token.has_value()); + EXPECT_EQ(surviving_head->etag, *fresh_token) << "and it is still the writer's incarnation"; + EXPECT_EQ(op.remove(key, condemned_token, Retry::once()), Removal::Mismatch) << "the condemned token can never remove the fresh object (INV_NO_LOSS)"; /// A later round's own fresh catalog cut names `newborn`, folds its `+1`, and the blob's frontier is @@ -1390,7 +1410,7 @@ TEST(CASGCFrontierGate, ANamespaceBornAfterTheEmptyCutResurrectsTheCondemnedBlob ASSERT_TRUE(later.saw_fold); EXPECT_EQ(later.frontier_namespaces, 1u); EXPECT_EQ(later.frontier_proven, 1u); - EXPECT_TRUE(backend->head(key).exists) << "the newly folded owner keeps the blob alive"; + EXPECT_TRUE(op.head(key, Retry::once()).has_value()) << "the newly folded owner keeps the blob alive"; } /// The generation prune's cursor must not move on a suppressed round either. It is a monotone @@ -1410,8 +1430,9 @@ TEST(CASGCFrontierGate, ASuppressedRoundDoesNotAdvanceTheGenerationPruneCursor) runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); } + OperationForTest raw_op(*backend); const uint64_t pruned_through_before = - decodeGcState(backend->get(layout.gcStateKey())->bytes).snap_pruned_through; + decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes).snap_pruned_through; for (uint64_t i = 7; i <= 10; ++i) { @@ -1420,7 +1441,7 @@ TEST(CASGCFrontierGate, ASuppressedRoundDoesNotAdvanceTheGenerationPruneCursor) store->renewWatermarkOnce(); } - EXPECT_EQ(decodeGcState(backend->get(layout.gcStateKey())->bytes).snap_pruned_through, + EXPECT_EQ(decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes).snap_pruned_through, pruned_through_before) << "the retention cursor is a high-water mark; it may not pass a generation nothing deleted"; } @@ -1447,9 +1468,10 @@ TEST(CASGCFrontierGate, TheHandOffReclaimIsInertUnderSuppression) Gc gc(store, kGc); runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); - const uint64_t old_gen = decodeGcState(backend->get(layout.gcStateKey())->bytes).snap_generation; + OperationForTest raw_op(*backend); + const uint64_t old_gen = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes).snap_generation; const String old_prefix = layout.gcGenPrefix(old_gen); - ASSERT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()); + ASSERT_FALSE((*raw_op).list(old_prefix, "", 1000, Retry::once()).keys.empty()); /// Idle-carry the ref until the retention cursor is strictly PAST its generation. Until then an /// ordinary prune could still reclaim it and the hand-off would not be the load-bearing path. @@ -1458,9 +1480,9 @@ TEST(CASGCFrontierGate, TheHandOffReclaimIsInertUnderSuppression) runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); } - ASSERT_GT(decodeGcState(backend->get(layout.gcStateKey())->bytes).snap_pruned_through, old_gen) + ASSERT_GT(decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes).snap_pruned_through, old_gen) << "the generation must be behind the retention cursor before the hand-off is exercised"; - ASSERT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) + ASSERT_FALSE((*raw_op).list(old_prefix, "", 1000, Retry::once()).keys.empty()) << "and still retained, because a live ref pins it"; /// A real delta moves the shard's run off the old generation. This is the round the hand-off would @@ -1475,7 +1497,7 @@ TEST(CASGCFrontierGate, TheHandOffReclaimIsInertUnderSuppression) EXPECT_EQ(backend->deleteCountForKeysContaining("/gc/gen/"), 0u) << "a suppressed round hands nothing off. Deleted:" << deletedKeysMessage(*backend); - EXPECT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) + EXPECT_FALSE((*raw_op).list(old_prefix, "", 1000, Retry::once()).keys.empty()) << "the superseded generation's prefix survives a suppressed round intact"; /// AND THE OPPORTUNITY IS CONSUMED, NOT DEFERRED -- the gate @@ -1492,7 +1514,7 @@ TEST(CASGCFrontierGate, TheHandOffReclaimIsInertUnderSuppression) /// The hand-off itself is not going untested: `CASGCRetention.HandOffDeletesSupersededRef` drives /// the same transition on an authoritative round and asserts the prefix IS reclaimed. runRegularRoundReclaiming(gc); - EXPECT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) + EXPECT_FALSE((*raw_op).list(old_prefix, "", 1000, Retry::once()).keys.empty()) << "the hand-off is a one-shot difference: the suppressed round consumed it, so the prefix is " "now fsck's problem rather than a later round's"; } @@ -1553,11 +1575,12 @@ TEST(CASGCFrontierGate, TheOrphanManifestSweepAndItsCursorAreInertUnderSuppressi store->renewWatermarkOnce(); } + OperationForTest raw_op(*backend); EXPECT_EQ(backend->deleteCountForKeysContaining("/cas/manifests/"), 0u) << "a suppressed round sweeps nothing. Deleted:" << deletedKeysMessage(*backend); - EXPECT_TRUE(backend->head(layout.manifestKey(ManifestId{ns, r1})).exists); - EXPECT_TRUE(backend->head(layout.manifestKey(ManifestId{ns, r2})).exists); - EXPECT_TRUE(decodeGcState(backend->get(layout.gcStateKey())->bytes).manifest_sweep_cursor.empty()) + EXPECT_TRUE((*raw_op).head(layout.manifestKey(ManifestId{ns, r1}), Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head(layout.manifestKey(ManifestId{ns, r2}), Retry::once()).has_value()); + EXPECT_TRUE(decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes).manifest_sweep_cursor.empty()) << "the sweep cursor must not advance over a range the round declined to sweep -- nothing " "revisits it"; @@ -1567,8 +1590,8 @@ TEST(CASGCFrontierGate, TheOrphanManifestSweepAndItsCursorAreInertUnderSuppressi runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); } - EXPECT_FALSE(backend->head(layout.manifestKey(ManifestId{ns, r1})).exists); - EXPECT_FALSE(backend->head(layout.manifestKey(ManifestId{ns, r2})).exists); + EXPECT_FALSE((*raw_op).head(layout.manifestKey(ManifestId{ns, r1}), Retry::once()).has_value()); + EXPECT_FALSE((*raw_op).head(layout.manifestKey(ManifestId{ns, r2}), Retry::once()).has_value()); } @@ -1713,14 +1736,14 @@ TEST(CASGCFrontierGate, CheckpointFrontierBehindAnInheritedCursorFailsClosed) const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String checkpoint_key = layout.refCkptKey(life); - const HeadResult checkpoint_head = backend->head(checkpoint_key); - ASSERT_TRUE(checkpoint_head.exists); - ASSERT_EQ(backend->putOverwrite(checkpoint_key, encodeRefCkpt(RefCkpt{ + const auto checkpoint_head = op.head(checkpoint_key, Retry::once()); + ASSERT_TRUE(checkpoint_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(checkpoint_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, - }), checkpoint_head.token).outcome, PutOutcome::Done); + }), checkpoint_head->etag, Retry::once()))); std::map intake; gc.setPhaseSink([&](const GcPhaseRecord & rec) @@ -1767,14 +1790,14 @@ TEST(CASGCFrontierGate, CheckpointFrontierCrossesAnInheritedEpochSeal) /*birth=*/false, /*prev_epoch_seal=*/RefTxnId{1, 2}); const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String checkpoint_key = layout.refCkptKey(life); - const HeadResult checkpoint_head = backend->head(checkpoint_key); - ASSERT_TRUE(checkpoint_head.exists); - ASSERT_EQ(backend->putOverwrite(checkpoint_key, encodeRefCkpt(RefCkpt{ + const auto checkpoint_head = op.head(checkpoint_key, Retry::once()); + ASSERT_TRUE(checkpoint_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(checkpoint_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{2, 1}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = RefTxnId{1, 2}, - }), checkpoint_head.token).outcome, PutOutcome::Done); + }), checkpoint_head->etag, Retry::once()))); std::map intake; gc.setPhaseSink([&](const GcPhaseRecord & rec) @@ -1857,14 +1880,14 @@ TEST(CASGCFrontierGate, AWronglyQuietNamespaceIsWalkedTheSameRound) publish(*backend, layout, quiet, "ref_2", 2, late_blob); const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, quiet); const String checkpoint_key = layout.refCkptKey(life); - const HeadResult checkpoint_head = backend->head(checkpoint_key); - ASSERT_TRUE(checkpoint_head.exists); - ASSERT_EQ(backend->putOverwrite(checkpoint_key, encodeRefCkpt(RefCkpt{ + const auto checkpoint_head = op.head(checkpoint_key, Retry::once()); + ASSERT_TRUE(checkpoint_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(checkpoint_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, - }), checkpoint_head.token).outcome, PutOutcome::Done); + }), checkpoint_head->etag, Retry::once()))); backend->hidePrefix(layout.namespaceStreamPrefix(fixture::fixtureLife(quiet))); runRegularRoundReclaiming(gc); @@ -2055,9 +2078,9 @@ TEST(CASGCFrontierGate, MissingCommittedCheckpointLogHoldsInsteadOfProvingTheFro const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String missing_key = layout.refLogKey(life, RefTxnId{1, 2}); - const HeadResult missing_head = backend->head(missing_key); - ASSERT_TRUE(missing_head.exists); - ASSERT_EQ(backend->deleteExact(missing_key, missing_head.token).kind, DeleteOutcome::Kind::Deleted); + const auto missing_head = op.head(missing_key, Retry::once()); + ASSERT_TRUE(missing_head.has_value()); + ASSERT_EQ(op.remove(missing_key, missing_head->etag, Retry::once()), Removal::Removed); std::map intake; Gc gc(store, kGc); @@ -2173,14 +2196,14 @@ TEST(CASGCFrontierGate, EmptyCheckpointFrontierRejectsAnInheritedCursor) const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(op, layout, ns); const String checkpoint_key = layout.refCkptKey(life); - const HeadResult checkpoint_head = backend->head(checkpoint_key); - ASSERT_TRUE(checkpoint_head.exists); - ASSERT_EQ(backend->putOverwrite(checkpoint_key, encodeRefCkpt(RefCkpt{ + const auto checkpoint_head = op.head(checkpoint_key, Retry::once()); + ASSERT_TRUE(checkpoint_head.has_value()); + ASSERT_TRUE(std::holds_alternative(op.replace(checkpoint_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, - }), checkpoint_head.token).outcome, PutOutcome::Done); + }), checkpoint_head->etag, Retry::once()))); std::map intake; gc.setPhaseSink([&](const GcPhaseRecord & rec) @@ -2274,7 +2297,7 @@ TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSealsCursorsAndDeletesNothing) EXPECT_GT(backend->deleteTotal(), 0u) << "the quiet life's checkpoint authority leaves unrelated deletion eligible"; - EXPECT_FALSE(backend->head(blobKeyOf(layout, blob)).exists) + EXPECT_FALSE(op.head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "the busy life's removal remains reclaimable despite the quiet LIST omission"; EXPECT_EQ(sealedCursorOf(*backend, layout, quiet), quiet_cursor) << "the unprobed namespace's cursor rides verbatim -- it is never dropped"; @@ -2283,7 +2306,7 @@ TEST(CASGCFrontierGate, AnExhaustedProbeBudgetSealsCursorsAndDeletesNothing) ASSERT_TRUE(quiet_checkpoint.has_value()); EXPECT_EQ(quiet_checkpoint->ckpt.committed_through, quiet_cursor) << "the quiet life's valid CTE is unaffected by LIST omission and a zero probe budget"; - EXPECT_GT(decodeGcState(backend->get(layout.gcStateKey())->bytes).round, 1u) + EXPECT_GT(decodeGcState(op.read(layout.gcStateKey(), Retry::once())->bytes).round, 1u) << "the round still commits; only its destructive half is withheld"; } @@ -2374,7 +2397,7 @@ TEST(CASGCFrontierGate, ACommittedGapIsRedetectedAndSuppressesEveryRound) EXPECT_EQ(backend->deleteTotal(), 0u) << "the re-detected committed gap suppresses each round's destructive work. " "Deleted:" << deletedKeysMessage(*backend); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists); + EXPECT_TRUE(op.head(blobKeyOf(layout, blob), Retry::once()).has_value()); EXPECT_EQ(sealedCursorOf(*backend, layout, held), (RefTxnId{1, 2})) << "the committed gap remains unresolved and the cursor cannot advance through it"; const auto final_checkpoint = readCkpt(op, layout, held_life); @@ -2407,7 +2430,8 @@ TEST(CASGCFrontierGate, ABlobCondemnedThisRoundIsNeverDeletedThisRound) backend->resetCounts(); runRegularRoundReclaiming(gc); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "the condemning round must not also delete"; EXPECT_EQ(backend->deleteCount(blobKeyOf(layout, blob)), 0u) << "not merely still present -- the delete was never attempted"; @@ -2447,7 +2471,8 @@ TEST(CASGCFrontierGate, ALateEdgeSparesADeletePendingBlobAtTheDeleteSite) runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "the delete-site in-degree re-read spares a blob a fresh edge re-referenced"; EXPECT_GT(inDegreeOf(*backend, layout, blob), 0); } @@ -2489,7 +2514,10 @@ TEST(CASGCFrontierGate, AResurrectedIncarnationSurvivesTheDelayedStaleTokenDelet runRegularRoundReclaiming(gc); /// graduate: publishes delete_pending against THIS token store->renewWatermarkOnce(); - const Token condemned_token = backend->head(key).token; + OperationForTest raw_op(*backend); + const auto condemned_head = (*raw_op).head(key, Retry::once()); + ASSERT_TRUE(condemned_head.has_value()); + const Etag condemned_token = condemned_head->etag; const auto condemned_meta = loadMetaForTest(*backend, layout, hash); ASSERT_TRUE(condemned_meta.has_value()); ASSERT_EQ(condemned_meta->meta.state, MetaState::Condemned) @@ -2507,16 +2535,19 @@ TEST(CASGCFrontierGate, AResurrectedIncarnationSurvivesTheDelayedStaleTokenDelet const PutBlobResult uploaded = build->putBlob(id, BlobSource::fromString(payload)); EXPECT_EQ(uploaded.ref, id); build->promote(ns, "republished", build->buildId(), republished_manifest); - const Token fresh_token = backend->head(key).token; + const auto fresh_head = (*raw_op).head(key, Retry::once()); + ASSERT_TRUE(fresh_head.has_value()); + const Etag fresh_token = fresh_head->etag; ASSERT_NE(fresh_token, condemned_token) << "republication must displace the condemned incarnation"; /// GC's delayed delete still names the OLD token. It cannot touch the new object. drive(store, gc, /*rounds*/ 2, UniversePolicy::Authoritative); - ASSERT_TRUE(backend->head(key).exists) + const auto surviving_head = (*raw_op).head(key, Retry::once()); + ASSERT_TRUE(surviving_head.has_value()) << "the resurrected incarnation survives the delete published against its predecessor"; - EXPECT_EQ(backend->head(key).token, fresh_token) << "and it is still the writer's incarnation"; - EXPECT_EQ(backend->deleteExact(key, condemned_token).kind, DeleteOutcome::Kind::TokenMismatch) + EXPECT_EQ(surviving_head->etag, fresh_token) << "and it is still the writer's incarnation"; + EXPECT_EQ((*raw_op).remove(key, condemned_token, Retry::once()), Removal::Mismatch) << "the condemned token can never remove the fresh object (INV-NO-RETURN)"; } @@ -2555,7 +2586,8 @@ TEST(CASGCFrontierGate, ATokenlessRelinkMakesTheReceiverEdgeDurableBeforeTheSour dropRefTransition(*backend, layout, source, "part_1", source_ref); drive(store, gc, /*rounds*/ 4, UniversePolicy::Authoritative); - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists) + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(blobKeyOf(layout, blob), Retry::once()).has_value()) << "the source released its edge only after the receiver's was durable, so nothing may collect it"; EXPECT_EQ(inDegreeOf(*backend, layout, blob), 1) << "the receiver is the sole remaining owner"; @@ -2730,12 +2762,12 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito return next; }); const String ckpt_key = layout.refCkptKey(life); - backend->putIfAbsent(ckpt_key, encodeRefCkpt(RefCkpt{ + op.create(ckpt_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, - })); + }), Retry::once()); /// The removal evidence must arise from a replay-valid terminal lifecycle, rather than merely /// from a raw terminal record that the recovery state machine refuses. @@ -2747,14 +2779,14 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito Gc gc(store, kGc); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState st = decodeGcState(op.read(layout.gcStateKey(), Retry::once())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend->get(layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + op.read(layout.foldSealKey(st.snap_generation, st.snap_attempt), Retry::once())->bytes); const auto row_it = seal.ref_lives.find(life.incarnation); ASSERT_NE(row_it, seal.ref_lives.end()); ASSERT_TRUE(row_it->second.cleanup_evidence.has_value()); EXPECT_EQ(row_it->second.cleanup_evidence->remove_txn_id, (RefTxnId{1, 2})); - EXPECT_TRUE(backend->head(ckpt_key).exists); + EXPECT_TRUE(op.head(ckpt_key, Retry::once()).has_value()); for (const String & key : backend->touchedKeys()) EXPECT_EQ(key.find("/_cleanup/"), String::npos) << key; @@ -2783,7 +2815,7 @@ TEST(CASGCFrontierGate, CleanupEvidenceLeavesRemovedNamespaceCheckpointForJanito EXPECT_GE(janitor_metrics.at("janitor_deleted"), 1u) << "the janitor's OWN counter must show the delete -- now that the proved-empty gate has " "opened, not because some other site happened to remove the key"; - EXPECT_FALSE(backend->head(ckpt_key).exists); + EXPECT_FALSE(op.head(ckpt_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(ckpt_key), 1); } @@ -2822,12 +2854,12 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro it->removal_started_round = 1; return next; }); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(removed_life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(removed_life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, - })).outcome, PutOutcome::Done); + }), Retry::once()))); const DB::UInt128 blob(0xfeed); const ManifestRef manifest = publish(*backend, layout, progressing, "victim", 1, blob); @@ -2835,9 +2867,9 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro Gc gc(store, kGc); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - const GcState folded_state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState folded_state = decodeGcState(op.read(layout.gcStateKey(), Retry::once())->bytes); const CasFoldSeal folded_seal = decodeFoldSeal( - backend->get(layout.foldSealKey(folded_state.snap_generation, folded_state.snap_attempt))->bytes); + op.read(layout.foldSealKey(folded_state.snap_generation, folded_state.snap_attempt), Retry::once())->bytes); const auto folded_row = folded_seal.ref_lives.find(removed_life.incarnation); ASSERT_NE(folded_row, folded_seal.ref_lives.end()); ASSERT_TRUE(folded_row->second.cleanup_evidence.has_value()); @@ -2845,8 +2877,8 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro dropRefTransition(*backend, layout, progressing, "victim", manifest); const String terminal_key = layout.refLogKey(removed_life, RefTxnId{1, 2}); const String later_dead_residue = layout.refLogKey(removed_life, RefTxnId{1, 3}); - ASSERT_EQ(backend->putIfAbsent(later_dead_residue, "dead residue after the folded terminal").outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + op.create(later_dead_residue, "dead residue after the folded terminal", Retry::once()))); backend->makeUnreadable(terminal_key); std::map namespace_cleanup; @@ -2867,7 +2899,7 @@ TEST(CASGCFrontierGate, PostFoldUnreadableTerminalIsCountedWithoutSuppressingPro EXPECT_TRUE(CasRefCatalog::lifeIfCataloged(op, layout, progressing)); EXPECT_EQ(report.manifests_deleted, 1u) << "the janitor leak cannot promote itself into pool-wide destructive suppression"; - EXPECT_FALSE(backend->head(layout.manifestKey(manifest_id)).exists); + EXPECT_FALSE(op.head(layout.manifestKey(manifest_id), Retry::once()).has_value()); EXPECT_TRUE(backend->existsIgnoringFault(terminal_key)); EXPECT_FALSE(backend->existsIgnoringFault(later_dead_residue)) << "one unreadable key cannot stop the perpetual janitor from deciding the rest of its page"; @@ -2892,21 +2924,21 @@ TEST(CASGCFrontierGate, UnmatchedAdoptedParentLifeDoesNotSuppressAuthoritativeDe const ManifestId manifest_id{ns, mref}; Gc gc(store, kGc); + OperationForTest raw_op(*backend); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - ASSERT_TRUE(backend->head(layout.manifestKey(manifest_id)).exists); + ASSERT_TRUE((*raw_op).head(layout.manifestKey(manifest_id), Retry::once()).has_value()); - const GcState before = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState before = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); const String parent_seal_key = layout.foldSealKey(before.snap_generation, before.snap_attempt); - const auto parent_object = backend->get(parent_seal_key); + const auto parent_object = (*raw_op).read(parent_seal_key, Retry::once()); ASSERT_TRUE(parent_object); CasFoldSeal parent = decodeFoldSeal(parent_object->bytes, before.snap_generation); const UInt128 unmatched_life = hexToU128("fedcba98765432100123456789abcdef"); ASSERT_FALSE(parent.ref_lives.contains(unmatched_life)); parent.ref_lives.emplace(unmatched_life, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{9, 9}}}); - ASSERT_EQ( - backend->putOverwrite(parent_seal_key, encodeFoldSeal(parent), parent_object->token).outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*raw_op).replace(parent_seal_key, encodeFoldSeal(parent), parent_object->etag, Retry::once()))); dropRefTransition(*backend, layout, ns, "victim", mref); const uint64_t events_before = @@ -2919,12 +2951,12 @@ TEST(CASGCFrontierGate, UnmatchedAdoptedParentLifeDoesNotSuppressAuthoritativeDe 1u); EXPECT_EQ(report.manifests_deleted, 1u) << "an unmatched adopted-parent row is observed and dropped, not promoted to pool-wide suppression"; - EXPECT_FALSE(backend->head(layout.manifestKey(manifest_id)).exists) + EXPECT_FALSE((*raw_op).head(layout.manifestKey(manifest_id), Retry::once()).has_value()) << "the valid manifest candidate must be physically deleted by the same authoritative round"; - const GcState after = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState after = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); const CasFoldSeal successor = decodeFoldSeal( - backend->get(layout.foldSealKey(after.snap_generation, after.snap_attempt))->bytes, + (*raw_op).read(layout.foldSealKey(after.snap_generation, after.snap_attempt), Retry::once())->bytes, after.snap_generation); EXPECT_FALSE(successor.ref_lives.contains(unmatched_life)); } @@ -2958,8 +2990,8 @@ TEST(CASCatalogLifecycleReconciler, DeletesEligibleRowsFromReturnedResolutionCut CasOperation op = requests.admit(); const Layout & layout = store->layout(); constexpr size_t deletes = 3; - seedCompletedRemovingBatch(*backend, op, store, kGc, deletes); - const auto parent_object = backend->get(layout.foldSealKey(1, 1)); + seedCompletedRemovingBatch(op, store, kGc, deletes); + const auto parent_object = op.read(layout.foldSealKey(1, 1), Retry::once()); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); backend->clearJournal(); @@ -2986,8 +3018,8 @@ TEST(CASCatalogLifecycleReconciler, ReturnsRetiredLifeWhenAuthorityMovesAfterRes CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); - const auto parent_object = backend->get(layout.foldSealKey(1, 1)); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); + const auto parent_object = op.read(layout.foldSealKey(1, 1), Retry::once()); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); @@ -3021,8 +3053,8 @@ TEST(CASCatalogLifecycleReconciler, InitialFenceLossReportsEligibleRowStillPrese CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); - const auto parent_object = backend->get(layout.foldSealKey(1, 1)); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); + const auto parent_object = op.read(layout.foldSealKey(1, 1), Retry::once()); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); backend->resetCounts(); @@ -3055,8 +3087,8 @@ TEST(CASCatalogLifecycleReconciler, RetriesFromTheMandatoryConflictResolutionCut CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - seedCompletedRemoving(*backend, op, store, kGc); - const auto parent_object = backend->get(layout.foldSealKey(1, 1)); + seedCompletedRemoving(op, store, kGc); + const auto parent_object = op.read(layout.foldSealKey(1, 1), Retry::once()); ASSERT_TRUE(parent_object); const CasFoldSeal parent = decodeFoldSeal(parent_object->bytes); backend->clearJournal(); @@ -3090,7 +3122,7 @@ TEST(CASGCFrontierGate, ADeposedLeaderErasesNoCatalogRow) CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); const uint64_t catalog_writes_before = backend->putOverwriteCount(layout.refCatalogKey()); /// Another leader steals `gc/state` after this round's lease renewal and before its drain. @@ -3102,7 +3134,7 @@ TEST(CASGCFrontierGate, ADeposedLeaderErasesNoCatalogRow) stolen.lease.owner = hexToU128("00000000000000000000000000000099"); ++stolen.lease.seq; ASSERT_TRUE(std::holds_alternative( - op.replace(layout.gcStateKey(), encodeGcState(stolen), got->incarnation, Retry::once()))); + op.replace(layout.gcStateKey(), encodeGcState(stolen), got->etag, Retry::once()))); }; Gc gc(store, kGc); @@ -3128,7 +3160,7 @@ TEST(CASGCFrontierGate, ALeaderDeposedBetweenTwoErasesStopsAfterTheFirst) CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - seedCompletedRemovingBatch(*backend, op, store, kGc, /*count=*/2); + seedCompletedRemovingBatch(op, store, kGc, /*count=*/2); const std::vector seeded{ RootNamespace{"00/drain-batch-0@cas@"}, RootNamespace{"00/drain-batch-1@cas@"}}; const uint64_t catalog_writes_before = backend->putOverwriteCount(layout.refCatalogKey()); @@ -3148,7 +3180,7 @@ TEST(CASGCFrontierGate, ALeaderDeposedBetweenTwoErasesStopsAfterTheFirst) stolen.lease.owner = hexToU128("00000000000000000000000000000099"); ++stolen.lease.seq; EXPECT_TRUE(std::holds_alternative( - op.replace(layout.gcStateKey(), encodeGcState(stolen), got->incarnation, Retry::once()))); + op.replace(layout.gcStateKey(), encodeGcState(stolen), got->etag, Retry::once()))); }); Gc gc(store, kGc); @@ -3172,7 +3204,7 @@ TEST(CASGCFrontierGate, HealthyRebuildUsesTheCatalogLifecycleReconciler) CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); Gc gc(store, kGc); @@ -3242,17 +3274,17 @@ TEST(CASGCFrontierGate, DeferredRoundDrainsCompletedRemovingBeforeReturning) .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 1}}}); for (uint64_t shard = 0; shard < store->poolConfig().gc_shards; ++shard) parent.condemned_summary.emplace(shard, CondemnedSummary{}); - ASSERT_EQ(backend->putIfAbsent(layout.foldSealKey(1, 1), encodeFoldSeal(parent)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.foldSealKey(1, 1), encodeFoldSeal(parent), Retry::once()))); GcState state; state.round = 1; state.gc_shards = store->poolConfig().gc_shards; state.snap_generation = 1; state.snap_attempt = 1; state.lease = GcLease{.owner = kGc, .seq = 1}; - ASSERT_EQ(backend->putIfAbsent(layout.gcStateKey(), encodeGcState(state)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(layout.gcStateKey(), encodeGcState(state), Retry::once()))); const String ckpt_key = layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(removed, life_id)); - ASSERT_EQ(backend->putIfAbsent(ckpt_key, "inert checkpoint debris").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(ckpt_key, "inert checkpoint debris", Retry::once()))); const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); Gc gc(store, kGc); @@ -3262,7 +3294,7 @@ TEST(CASGCFrontierGate, DeferredRoundDrainsCompletedRemovingBeforeReturning) EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(op, layout, removed)); EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), catalog_cas_before + 1); - EXPECT_TRUE(backend->head(ckpt_key).exists); + EXPECT_TRUE(op.head(ckpt_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(ckpt_key), 0); } @@ -3274,7 +3306,7 @@ TEST(CASGCFrontierGate, StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListi CasOperation op = requests.admit(); const Layout & layout = store->layout(); const UInt128 leader_b = hexToU128("00000000000000000000000000000002"); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); backend->clearJournal(); backend->blockNextCatalogCas(layout.refCatalogKey()); @@ -3339,7 +3371,7 @@ TEST(CASGCFrontierGate, StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListi const size_t fresh_catalog_cut = findJournalAfter( before_a_release, "get " + layout.refCatalogKey(), stream_list + 1); ASSERT_LT(fresh_catalog_cut, before_a_release.size()); - const GcState adopted = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState adopted = decodeGcState(op.read(layout.gcStateKey(), Retry::once())->bytes); const String successor_seal_key = layout.foldSealKey(adopted.snap_generation, adopted.snap_attempt); const size_t successor_seal_put = findJournalAfter( before_a_release, "put_end " + successor_seal_key, fresh_catalog_cut + 1); @@ -3378,7 +3410,7 @@ TEST(CASGCFrontierGate, StaleIssuedCatalogCasLosesAfterNewLeaderHelpsBeforeListi ASSERT_FALSE(janitor_metrics_b.empty()) << "the namespace_cleanup phase must have run this round"; EXPECT_GE(janitor_metrics_b.at("janitor_deleted"), 1u) << "the janitor's OWN counter must show the delete, now that the proved-empty gate has opened"; - EXPECT_FALSE(backend->get(fixture.checkpoint_key).has_value()); + EXPECT_FALSE(op.read(fixture.checkpoint_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(fixture.checkpoint_key), 1); } @@ -3389,7 +3421,7 @@ TEST(CASGCFrontierGate, LostCatalogCasResponseIsResolvedBeforeListing) CasRequests requests = openRequestsForTest(backend); CasOperation op = requests.admit(); const Layout & layout = store->layout(); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); backend->clearJournal(); backend->loseNextCatalogCasResponse(layout.refCatalogKey()); @@ -3435,7 +3467,7 @@ TEST(CASGCFrontierGate, LostCatalogCasResponseIsResolvedBeforeListing) ASSERT_FALSE(janitor_metrics.empty()) << "the namespace_cleanup phase must have run this round"; EXPECT_GE(janitor_metrics.at("janitor_deleted"), 1u) << "the janitor's OWN counter must show the delete, now that the proved-empty gate has opened"; - EXPECT_FALSE(backend->get(fixture.checkpoint_key).has_value()); + EXPECT_FALSE(op.read(fixture.checkpoint_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(fixture.checkpoint_key), 1); } @@ -3450,7 +3482,7 @@ TEST_P(CASGCCompletedRemovalFenceRace, FencedLeaderStopsAfterWinnerRemovesOrRepl CasOperation op = requests.admit(); const Layout & layout = store->layout(); const UInt128 leader_b = hexToU128("00000000000000000000000000000002"); - const CompletedRemovingFixture fixture = seedCompletedRemoving(*backend, op, store, kGc); + const CompletedRemovingFixture fixture = seedCompletedRemoving(op, store, kGc); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(fixture.ns, fixture.life_id); ASSERT_TRUE(store->refTableRecoveredForTest(fixture.ns)) @@ -3488,16 +3520,16 @@ TEST_P(CASGCCompletedRemovalFenceRace, FencedLeaderStopsAfterWinnerRemovesOrRepl /// Mirror production's publish-then-flip order: the successor life needs a readable `_ckpt` /// before its catalog row can read `Live`, or `chooseRecoveryGrounding` rejects it. const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(fixture.ns, UInt128{178}); - backend->putIfAbsent(layout.refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ + op.create(layout.refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt, - })); + }), Retry::once()); } - ASSERT_TRUE(observed.incarnation); + ASSERT_TRUE(observed.etag); ASSERT_TRUE(std::holds_alternative(op.replace( - layout.refCatalogKey(), encodeRefCatalog(winner_catalog), *observed.incarnation, Retry::once()))); + layout.refCatalogKey(), encodeRefCatalog(winner_catalog), *observed.etag, Retry::once()))); backend->clearJournal(); const uint64_t plans_before /// NOLINT(clang-analyzer-deadcode.DeadStores) @@ -3555,7 +3587,7 @@ TEST(CASGCFrontierGate, CompletedRemovalDrainUsesNPlusOneCatalogReads) CasOperation op = requests.admit(); const Layout & layout = store->layout(); constexpr size_t deletes = 3; - seedCompletedRemovingBatch(*backend, op, store, kGc, deletes); + seedCompletedRemovingBatch(op, store, kGc, deletes); backend->clearJournal(); backend->resetCounts(); diff --git a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp index f8089907fed0..3d67ac02f317 100644 --- a/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp +++ b/src/Disks/tests/gtest_cas_gc_hold_grammar.cpp @@ -141,9 +141,10 @@ std::optional newestSeal(Backend & backend, const Layout & layout) { const uint64_t gen = currentGenerationOf(backend, layout); const uint64_t attempt = currentAttemptOf(backend, layout); + OperationForTest op(backend); for (uint64_t g = gen; ; --g) { - if (const auto got = backend.get(layout.foldSealKey(g, attempt))) + if (const auto got = (*op).read(layout.foldSealKey(g, attempt), Retry::once())) return decodeFoldSeal(got->bytes); if (g == 0) return std::nullopt; @@ -325,6 +326,53 @@ std::vector> illFormedSealsTheEncoderMustRe return out; } +/// ---- Small raw-fixture request-engine wrappers shared by the tests below ---- +/// (`head`/`get`/`putOverwrite`/`putIfAbsent`/`deleteExact` are the legacy `Backend` verbs; every +/// caller now goes through an admitted `CasOperation`.) + +/// The durable object at `key`, or `nullopt`. +std::optional readAt(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::once()); +} + +/// True iff `key` exists. +bool existsAt(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::once()).has_value(); +} + +/// Unconditional create of a fresh key (the fixture's own corruption/injection setup, never a +/// real conflict). +void createAt(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + EXPECT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// Head, then unconditionally overwrite what was seen -- the raw-fixture corruption idiom this file's +/// tests use to replace an object's body in place. +void headThenReplace(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + const auto current = (*op).head(key, Retry::once()); + EXPECT_TRUE(current.has_value()) << "expected '" << key << "' to exist before overwrite"; + if (current) + EXPECT_TRUE(std::holds_alternative((*op).replace(key, bytes, current->etag, Retry::once()))); +} + +/// Head, then exact-delete what was seen -- the raw-fixture corruption idiom for removing an object +/// this test just observed present. +void headThenRemove(Backend & backend, const String & key) +{ + OperationForTest op(backend); + const auto current = (*op).head(key, Retry::once()); + ASSERT_TRUE(current.has_value()) << "expected '" << key << "' to exist before removal"; + ASSERT_EQ((*op).remove(key, current->etag, Retry::once()), Removal::Removed); +} + } /// ===================== THE SHARED BYTE ARITHMETIC ===================== @@ -724,7 +772,10 @@ TEST(CASGCHoldGrammar, UndecodableBodyNamesTheRecordItCouldNotRead) fixture::admitLive(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch publishAt(*backend, layout, ns, RefTxnId{1, 1}, "ref_1", 1, DB::UInt128(1), /*birth=*/true); - backend->putIfAbsent(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, 2}), "this is not a cas_ref_log object"); + { + OperationForTest op(*backend); + (*op).create(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, 2}), "this is not a cas_ref_log object", Retry::once()); + } writeCommittedCkptAt(*backend, layout, ns, RefTxnId{1, 2}); Gc gc(store, kGc); @@ -971,22 +1022,23 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointHoldsOnlyItsOwnNamespace) /// Corrupt EXACTLY ONE OBJECT: the first namespace's `_ckpt` body. Nothing else in the pool changes, /// so everything the next round does differently is attributable to this one object. const String bad_ckpt_key = layout.refCkptKey(fixture::fixtureLife(bad)); - const HeadResult ckpt_head = backend->head(bad_ckpt_key); - ASSERT_TRUE(ckpt_head.exists); - ASSERT_EQ(backend->putOverwrite(bad_ckpt_key, "this is not a cas_ref_ckpt", ckpt_head.token).outcome, - PutOutcome::Done); + OperationForTest corrupt_op(*backend); + const auto ckpt_head = (*corrupt_op).head(bad_ckpt_key, Retry::once()); + ASSERT_TRUE(ckpt_head.has_value()); + ASSERT_TRUE(std::holds_alternative( + (*corrupt_op).replace(bad_ckpt_key, "this is not a cas_ref_ckpt", ckpt_head->etag, Retry::once()))); /// Work only a round that COMPLETES can fold. publishAt(*backend, layout, good, RefTxnId{1, 2}, "ref_2", 2, DB::UInt128(12)); const String good_ckpt_key = layout.refCkptKey(fixture::fixtureLife(good)); - const HeadResult good_ckpt_head = backend->head(good_ckpt_key); - ASSERT_TRUE(good_ckpt_head.exists); - ASSERT_EQ(backend->putOverwrite(good_ckpt_key, encodeRefCkpt(RefCkpt{ + const auto good_ckpt_head = (*corrupt_op).head(good_ckpt_key, Retry::once()); + ASSERT_TRUE(good_ckpt_head.has_value()); + ASSERT_TRUE(std::holds_alternative((*corrupt_op).replace(good_ckpt_key, encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = RefTxnId{1, 2}, .last_epoch_seal = std::nullopt, - }), good_ckpt_head.token).outcome, PutOutcome::Done); + }), good_ckpt_head->etag, Retry::once()))); ASSERT_TRUE(gc.runRegularRound().acquired_lease); @@ -1009,7 +1061,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointHoldsOnlyItsOwnNamespace) /// its ref objects — including the ones a cleanup range computed WITHOUT the unreadable checkpoint /// would have widened onto — are all still there. for (const RefTxnId & id : {RefTxnId{1, 1}, RefTxnId{1, 2}}) - EXPECT_TRUE(backend->head(layout.refLogKey(fixture::fixtureLife(bad), id)).exists) + EXPECT_TRUE(existsAt(*backend, layout.refLogKey(fixture::fixtureLife(bad), id))) << "ref log " << renderRefTxnId(id) << " of the held namespace was deleted"; } @@ -1051,7 +1103,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointWithNoWalkPositionRecordsAnAnomaly fixture::admitLive(*backend, layout, phantom); /// A lone `_ckpt` with an undecodable body, and NOTHING else under that namespace. - backend->putIfAbsent(layout.refCkptKey(fixture::fixtureLife(phantom)), "this is not a cas_ref_ckpt"); + createAt(*backend, layout.refCkptKey(fixture::fixtureLife(phantom)), "this is not a cas_ref_ckpt"); publishAt(*backend, layout, good, RefTxnId{1, 1}, "ref_1", 1, DB::UInt128(11), /*birth=*/true); writeCommittedCkptAt(*backend, layout, good, RefTxnId{1, 1}); @@ -1087,7 +1139,7 @@ TEST(CASGCHoldGrammar, AnUndecodableCheckpointWithNoWalkPositionRecordsAnAnomaly /// The unreadable object itself is never deleted as debris — repairing it is the operator's move, /// and GC removing it would erase the only evidence of what stopped the namespace. - EXPECT_TRUE(backend->head(layout.refCkptKey(fixture::fixtureLife(phantom))).exists); + EXPECT_TRUE(existsAt(*backend, layout.refCkptKey(fixture::fixtureLife(phantom)))); } /// ===================== THE HOLD IS DURABLE ===================== @@ -1202,9 +1254,13 @@ void mutateSealAt(Backend & backend, const Layout & layout, uint64_t generation, const std::function & mutate) { const String key = layout.foldSealKey(generation, attempt); - CasFoldSeal seal = decodeFoldSeal(backend.get(key)->bytes); + OperationForTest op(backend); + CasFoldSeal seal = decodeFoldSeal((*op).read(key, Retry::once())->bytes); mutate(seal); - backend.putOverwrite(key, encodeFoldSeal(seal), backend.head(key).token); + const auto current = (*op).head(key, Retry::once()); + EXPECT_TRUE(current.has_value()); + if (current) + EXPECT_TRUE(std::holds_alternative((*op).replace(key, encodeFoldSeal(seal), current->etag, Retry::once()))); } /// Rewrite the adopted fold seal, applying `mutate` to it. Used to plant a hold that the rebuild must @@ -1212,11 +1268,15 @@ void mutateSealAt(Backend & backend, const Layout & layout, uint64_t generation, /// the carry, not about how the hold arose. void mutateAdoptedSeal(Backend & backend, const Layout & layout, const std::function & mutate) { - const GcState st = decodeGcState(backend.get(layout.gcStateKey())->bytes); + OperationForTest op(backend); + const GcState st = decodeGcState((*op).read(layout.gcStateKey(), Retry::once())->bytes); const String key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - CasFoldSeal seal = decodeFoldSeal(backend.get(key)->bytes); + CasFoldSeal seal = decodeFoldSeal((*op).read(key, Retry::once())->bytes); mutate(seal); - backend.putOverwrite(key, encodeFoldSeal(seal), backend.head(key).token); + const auto current = (*op).head(key, Retry::once()); + EXPECT_TRUE(current.has_value()); + if (current) + EXPECT_TRUE(std::holds_alternative((*op).replace(key, encodeFoldSeal(seal), current->etag, Retry::once()))); } RefHold plantedHold() @@ -1290,7 +1350,7 @@ TEST(CASGCHoldGrammar, RebuildStepsDownPastACrashedNewestGenerationToTheSealBelo writeCommittedCkptAt(*backend, layout, ns, RefTxnId{1, 1}); Gc gc(store, kGc); ASSERT_TRUE(gc.runRegularRound().acquired_lease); - const GcState after_first = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState after_first = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); const uint64_t older_generation = after_first.snap_generation; const uint64_t older_attempt = after_first.snap_attempt; const UInt128 life_id = catalogLifeIdForTest(*backend, layout, ns); @@ -1298,7 +1358,7 @@ TEST(CASGCHoldGrammar, RebuildStepsDownPastACrashedNewestGenerationToTheSealBelo publishAt(*backend, layout, ns, RefTxnId{1, 2}, "ref_2", 2, DB::UInt128(2)); advanceRecoverableCkptForRawFixture(*backend, layout, ns, RefTxnId{1, 2}); ASSERT_TRUE(gc.runRegularRound().acquired_lease); - const GcState after_second = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState after_second = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); ASSERT_GT(after_second.snap_generation, older_generation) << "the fixture needs two generations"; /// The older generation is the one holding the pool's durable hold. @@ -1312,13 +1372,13 @@ TEST(CASGCHoldGrammar, RebuildStepsDownPastACrashedNewestGenerationToTheSealBelo /// THE CRASH: the newest generation's run objects are there, its seal never got written. Then /// `gc/state` is lost, which is this path's whole premise. const String newest_seal = layout.foldSealKey(after_second.snap_generation, after_second.snap_attempt); - const HeadResult seal_head = backend->head(newest_seal); - ASSERT_TRUE(seal_head.exists); - ASSERT_EQ(backend->deleteExact(newest_seal, seal_head.token).kind, DeleteOutcome::Kind::Deleted); - ASSERT_FALSE(backend->list(layout.gcGenPrefix(after_second.snap_generation), "", 1).keys.empty()) - << "the crashed generation must still hold objects, or it is not the shape being modelled"; - const HeadResult sh = backend->head(layout.gcStateKey()); - ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); + headThenRemove(*backend, newest_seal); + { + OperationForTest op(*backend); + ASSERT_FALSE((*op).list(layout.gcGenPrefix(after_second.snap_generation), "", 1, Retry::once()).keys.empty()) + << "the crashed generation must still hold objects, or it is not the shape being modelled"; + } + headThenRemove(*backend, layout.gcStateKey()); Gc gc2(store, hexToU128("0000000000000000000000000000000c")); const RebuildReport rep = gc2.rebuildBaseline(/*force=*/false); @@ -1352,12 +1412,10 @@ TEST(CASGCHoldGrammar, RebuildRefusesWithAMissingPriorSeal) Gc gc(store, kGc); ASSERT_TRUE(gc.runRegularRound().acquired_lease); - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState st = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); ASSERT_GT(st.snap_generation, 0u); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - const HeadResult sh = backend->head(seal_key); - ASSERT_TRUE(sh.exists); - ASSERT_EQ(backend->deleteExact(seal_key, sh.token).kind, DeleteOutcome::Kind::Deleted); + headThenRemove(*backend, seal_key); /// FORCE does not buy past it either: force means "rebuild deliberately", never "drop the holds". for (const bool force : {false, true}) @@ -1366,7 +1424,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWithAMissingPriorSeal) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc.rebuildBaseline(force); }); } - const GcState after = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState after = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); EXPECT_EQ(after.snap_generation, st.snap_generation) << "a refused rebuild adopts nothing"; } @@ -1381,10 +1439,9 @@ TEST(CASGCHoldGrammar, RebuildRefusesWithAnUndecodablePriorSeal) Gc gc(store, kGc); ASSERT_TRUE(gc.runRegularRound().acquired_lease); - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState st = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n", - backend->head(seal_key).token); + headThenReplace(*backend, seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc.rebuildBaseline(/*force=*/true); }); } @@ -1423,9 +1480,7 @@ TEST(CASGCHoldGrammar, RebuildWithLostStateStillCarriesHoldsFromTheNewestSeal) }); /// The pointer vanishes; every seal object survives. - const HeadResult sh = backend->head(layout.gcStateKey()); - ASSERT_TRUE(sh.exists); - ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); + headThenRemove(*backend, layout.gcStateKey()); Gc gc2(store, hexToU128("00000000000000000000000000000009")); const RebuildReport rep = gc2.rebuildBaseline(/*force=*/false); @@ -1453,12 +1508,10 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenTheNewestSealIsUnreadableAndTheStateIsL Gc gc(store, kGc); ASSERT_TRUE(gc.runRegularRound().acquired_lease); - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState st = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); const String seal_key = layout.foldSealKey(st.snap_generation, st.snap_attempt); - backend->putOverwrite(seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n", - backend->head(seal_key).token); - const HeadResult sh = backend->head(layout.gcStateKey()); - ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); + headThenReplace(*backend, seal_key, "{\"type\":\"cas_fold_seal\",\"v\":1}\nthis is not a seal body\n"); + headThenRemove(*backend, layout.gcStateKey()); Gc gc2(store, hexToU128("0000000000000000000000000000000a")); for (const bool force : {false, true}) @@ -1520,7 +1573,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa publishAt(*backend, layout, ns, RefTxnId{1, 2}, "ref_2", 2, DB::UInt128(2)); ASSERT_TRUE(gc.runRegularRound().acquired_lease); - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState st = decodeGcState(readAt(*backend, layout.gcStateKey())->bytes); ASSERT_GT(st.snap_generation, 1u) << "the fixture needs a newer generation to hide"; const UInt128 life_id = catalogLifeIdForTest(*backend, layout, ns); mutateAdoptedSeal(*backend, layout, [&](CasFoldSeal & seal) @@ -1534,8 +1587,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa const String gen_prefix = layout.gcGenPrefix(0); backend->hide_under_prefix = gen_prefix.substr(0, gen_prefix.size() - 2); /// ".../gc/gen/" backend->hidden_key_infix = layout.gcGenPrefix(st.snap_generation); - const HeadResult sh = backend->head(layout.gcStateKey()); - ASSERT_EQ(backend->deleteExact(layout.gcStateKey(), sh.token).kind, DeleteOutcome::Kind::Deleted); + headThenRemove(*backend, layout.gcStateKey()); Gc gc2(store, hexToU128("0000000000000000000000000000000b")); for (const bool force : {false, true}) @@ -1546,7 +1598,7 @@ TEST(CASGCHoldGrammar, RebuildRefusesWhenANarrowProbeFindsASealAboveTheListingMa ASSERT_GT(backend->holes_served, 0u) << "the broad listing never actually lied"; /// Nothing was adopted: the refusal fires before the lease, so the pool is exactly as it was. - EXPECT_FALSE(backend->head(layout.gcStateKey()).exists) + EXPECT_FALSE(existsAt(*backend, layout.gcStateKey())) << "a refused rebuild must not mint a baseline, nor a bootstrap body"; } @@ -1566,7 +1618,7 @@ TEST(CASGCHoldGrammar, RebuildProceedsOnAPoolThatNeverSealedABaselineAndCountsTh /// No round has run, so there is no `gc/state` and no seal — only owner state to rebuild from. publishAt(*backend, layout, ns, RefTxnId{1, 1}, "ref_1", 1, DB::UInt128(1), /*birth=*/true); writeCommittedCkptAt(*backend, layout, ns, RefTxnId{1, 1}); - ASSERT_FALSE(backend->head(layout.gcStateKey()).exists); + ASSERT_FALSE(existsAt(*backend, layout.gcStateKey())); using ProfileEvents::global_counters; const auto virgin_before = global_counters[ProfileEvents::CASGCRebuildVirginByEnumeration].load(); diff --git a/src/Disks/tests/gtest_cas_gc_leak.cpp b/src/Disks/tests/gtest_cas_gc_leak.cpp index 06983bc52aab..e6353d962261 100644 --- a/src/Disks/tests/gtest_cas_gc_leak.cpp +++ b/src/Disks/tests/gtest_cas_gc_leak.cpp @@ -49,7 +49,7 @@ bool anyRetiredPending(const PoolPtr & s) { /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. - return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); + return DB::Cas::tests::anyCondemnedInSeal(*s->poolBackendPtr(), s->layout()); } /// Drive regular GC to a fixpoint. A condemned blob is not deleted in the round that folds its removal: @@ -126,13 +126,15 @@ ManifestId publishOneBlobPart( /// HEADs the object key, never the Pool's manifest decode cache). bool blobPresent(const std::shared_ptr & b, const Layout & layout, const String & payload) { - return b->head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of(payload))})).exists; + DB::Cas::tests::OperationForTest op(*b); + return (*op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of(payload))}), Retry::once()).has_value(); } /// Whether a manifest body object is present in the backend. bool manifestPresent(const std::shared_ptr & b, const Layout & layout, const ManifestId & id) { - return b->head(layout.manifestKey(id)).exists; + DB::Cas::tests::OperationForTest op(*b); + return (*op).head(layout.manifestKey(id), Retry::once()).has_value(); } /// Replace the existing ref with partB through the real durable-precommit writer sequence. The @@ -171,8 +173,11 @@ FsckReport displaceAndGc( /// Publish partB's full closure and atomically repoint the ref from partA to partB. const ManifestId part_b = publishPartBReplacement(s, ns, ref, "data-B", "mark-B"); - EXPECT_TRUE(b->head(s->layout().manifestKey(part_a)).exists) - << "partA manifest body must still be present so GC can read its -1 edges at removal-fold"; + { + DB::Cas::tests::OperationForTest op(*b); + EXPECT_TRUE((*op).head(s->layout().manifestKey(part_a), Retry::once()).has_value()) + << "partA manifest body must still be present so GC can read its -1 edges at removal-fold"; + } const auto resolved = s->resolveRef(ns, ref); EXPECT_TRUE(resolved.has_value()); @@ -314,8 +319,9 @@ TEST(CASGCLeak, ResurrectReplacedIncarnationReclaimed) /// 1. Publish ref r1 -> token A referenced; capture A. publishOneBlobPart(s, ns, "r1", P); - const HeadResult hA = b->head(s->layout().blobKey(idOf(P))); - ASSERT_TRUE(hA.exists); + DB::Cas::tests::OperationForTest op(*b); + const auto hA = (*op).head(s->layout().blobKey(idOf(P)), Retry::once()); + ASSERT_TRUE(hA.has_value()); /// 2. Drop r1 -> A dereferenced. s->dropRef(ns, "r1"); @@ -334,9 +340,9 @@ TEST(CASGCLeak, ResurrectReplacedIncarnationReclaimed) /// 4. RESURRECT: a fresh build dedup-hits P; putBlob sees A condemned -> re-uploads a DISTINCT /// incarnation B at the same content-addressed key (INV-1 revival-from-source). publishOneBlobPart(s, ns, "r2", P); - const HeadResult hB = b->head(s->layout().blobKey(idOf(P))); - ASSERT_TRUE(hB.exists); - ASSERT_NE(hB.token.value, hA.token.value) << "republication must mint a new incarnation token B"; + const auto hB = (*op).head(s->layout().blobKey(idOf(P)), Retry::once()); + ASSERT_TRUE(hB.has_value()); + ASSERT_NE(hB->etag, hA->etag) << "republication must mint a new incarnation token B"; /// 5. Drop r2 -> B dereferenced. s->dropRef(ns, "r2"); @@ -423,17 +429,18 @@ TEST(CASGCLeak, ResurrectReplacedTokenIsCondemnedInMeta) /// 1. Publish ref r1 -> token A referenced; capture A, then drop it and condemn via ONE GC round. publishOneBlobPart(s, ns, "r1", P); - const HeadResult hA = b->head(s->layout().blobKey(idOf(P))); - ASSERT_TRUE(hA.exists); + DB::Cas::tests::OperationForTest op(*b); + const auto hA = (*op).head(s->layout().blobKey(idOf(P)), Retry::once()); + ASSERT_TRUE(hA.has_value()); s->dropRef(ns, "r1"); s->renewWatermarkOnce(); /// advance the floor so A is not spared as in-flight gc.runRegularRound(); /// 2. RESURRECT: r2 dedup-hits P while A is condemned -> mints a fresh incarnation B. publishOneBlobPart(s, ns, "r2", P); - const HeadResult hB = b->head(s->layout().blobKey(idOf(P))); - ASSERT_TRUE(hB.exists); - ASSERT_NE(hB.token.value, hA.token.value) << "republication must mint a distinct incarnation"; + const auto hB = (*op).head(s->layout().blobKey(idOf(P)), Retry::once()); + ASSERT_TRUE(hB.has_value()); + ASSERT_NE(hB->etag, hA->etag) << "republication must mint a distinct incarnation"; s->dropRef(ns, "r2"); s->renewWatermarkOnce(); diff --git a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp index 4d4dd313dae7..309d20f8de89 100644 --- a/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_maintenance_state_format.cpp @@ -109,28 +109,28 @@ TEST(CASGCMaintenanceState, ReadsAndCasWithoutAdoptingConflicts) const GcMaintenanceReadResult absent = readGcMaintenanceState(op, layout); EXPECT_EQ(absent.status, GcMaintenanceReadStatus::Absent); EXPECT_FALSE(absent.state); - EXPECT_FALSE(absent.incarnation); + EXPECT_FALSE(absent.etag); const GcMaintenanceState first{.janitor_cursor = "cas/ns/first"}; ASSERT_TRUE(std::holds_alternative( casGcMaintenanceState(op, layout, std::nullopt, first, Retry::standard()))); const GcMaintenanceReadResult valid = readGcMaintenanceState(op, layout); ASSERT_EQ(valid.status, GcMaintenanceReadStatus::Valid); - ASSERT_TRUE(valid.incarnation); + ASSERT_TRUE(valid.etag); ASSERT_TRUE(valid.state); EXPECT_EQ(*valid.state, first); - const WriteResult advanced = casGcMaintenanceState(op, layout, valid.incarnation, + const WriteResult advanced = casGcMaintenanceState(op, layout, valid.etag, GcMaintenanceState{.janitor_cursor = "cas/ns/advanced"}, Retry::standard()); ASSERT_TRUE(std::holds_alternative(advanced)); - const Incarnation advanced_incarnation = std::get(advanced).incarnation; + const Etag advanced_etag = std::get(advanced).etag; ASSERT_TRUE(std::holds_alternative( - op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), advanced_incarnation, Retry::standard()))); - const WriteResult conflict = casGcMaintenanceState(op, layout, valid.incarnation, + op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), advanced_etag, Retry::standard()))); + const WriteResult conflict = casGcMaintenanceState(op, layout, valid.etag, GcMaintenanceState{.janitor_cursor = "loser"}, Retry::standard()); EXPECT_TRUE(std::holds_alternative(conflict)); - EXPECT_EQ(decodeGcMaintenanceState(backend->get(key)->bytes).janitor_cursor, "winner"); + EXPECT_EQ(decodeGcMaintenanceState(op.read(key, Retry::standard())->bytes).janitor_cursor, "winner"); } TEST(CASGCMaintenanceState, ClassifiesCorruptionAndResetsOnlyExactToken) @@ -141,15 +141,15 @@ TEST(CASGCMaintenanceState, ClassifiesCorruptionAndResetsOnlyExactToken) const String key = layout.gcMaintenanceStateKey(); auto op = requests.admit(); - ASSERT_EQ(backend->putIfAbsent(key, "malformed").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(key, "malformed", Retry::once()))); const GcMaintenanceReadResult corrupt = readGcMaintenanceState(op, layout); ASSERT_EQ(corrupt.status, GcMaintenanceReadStatus::Corrupt); - ASSERT_TRUE(corrupt.incarnation); + ASSERT_TRUE(corrupt.etag); EXPECT_FALSE(corrupt.state); EXPECT_FALSE(corrupt.diagnostic.empty()); ASSERT_TRUE(std::holds_alternative( - casGcMaintenanceState(op, layout, corrupt.incarnation, {}, Retry::standard()))); - EXPECT_EQ(decodeGcMaintenanceState(backend->get(key)->bytes), GcMaintenanceState{}); + casGcMaintenanceState(op, layout, corrupt.etag, {}, Retry::standard()))); + EXPECT_EQ(decodeGcMaintenanceState(op.read(key, Retry::standard())->bytes), GcMaintenanceState{}); } TEST(CASGCMaintenanceState, UsesExactlyOneReadOrCasAttempt) @@ -181,15 +181,15 @@ TEST(CASGCMaintenanceState, UsesExactlyOneReadOrCasAttempt) const std::optional current = op.read(key, Retry::standard()); ASSERT_TRUE(current); ASSERT_TRUE(std::holds_alternative( - op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), current->incarnation, Retry::standard()))); + op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), current->etag, Retry::standard()))); backend->resetCounts(); - const WriteResult stale_attempt = casGcMaintenanceState(op, layout, current->incarnation, + const WriteResult stale_attempt = casGcMaintenanceState(op, layout, current->etag, GcMaintenanceState{.janitor_cursor = "stale"}, Retry::standard()); ASSERT_TRUE(std::holds_alternative(stale_attempt)); EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount(key), 1u); - EXPECT_EQ(decodeGcMaintenanceState(backend->InMemoryBackend::get(key)->bytes).janitor_cursor, "winner"); + EXPECT_EQ(decodeGcMaintenanceState(op.read(key, Retry::standard())->bytes).janitor_cursor, "winner"); } TEST(CASGCMaintenanceState, FutureVersionPropagatesInsteadOfResetting) @@ -200,9 +200,9 @@ TEST(CASGCMaintenanceState, FutureVersionPropagatesInsteadOfResetting) const String key = layout.gcMaintenanceStateKey(); auto op = requests.admit(); - ASSERT_EQ(backend->putIfAbsent(key, fmt::format( - "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion() + 1)).outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(key, fmt::format( + "{{\"type\":\"cas_gc_maintenance_state\",\"v\":{}}}\n{{\"janitor_cursor\":\"\"}}\n", currentCompatibilityVersion() + 1), + Retry::once()))); /// The seed write above lands through the same `write` primitive `CountingBackend` counts, so /// reset before measuring what the read itself does. @@ -225,17 +225,17 @@ TEST(CASGCMaintenanceState, LosingCorruptResetPreservesConcurrentWinner) const String key = layout.gcMaintenanceStateKey(); auto op = requests.admit(); - ASSERT_EQ(backend->putIfAbsent(key, "corrupt").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(key, "corrupt", Retry::once()))); const GcMaintenanceReadResult corrupt = readGcMaintenanceState(op, layout); ASSERT_EQ(corrupt.status, GcMaintenanceReadStatus::Corrupt); - ASSERT_TRUE(corrupt.incarnation); + ASSERT_TRUE(corrupt.etag); ASSERT_TRUE(std::holds_alternative( - op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), *corrupt.incarnation, Retry::standard()))); + op.replace(key, encodeGcMaintenanceState({.janitor_cursor = "winner"}), *corrupt.etag, Retry::standard()))); backend->resetCounts(); - const WriteResult reset_attempt = casGcMaintenanceState(op, layout, corrupt.incarnation, {}, Retry::standard()); + const WriteResult reset_attempt = casGcMaintenanceState(op, layout, corrupt.etag, {}, Retry::standard()); ASSERT_TRUE(std::holds_alternative(reset_attempt)); EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount(key), 1u); - EXPECT_EQ(decodeGcMaintenanceState(backend->InMemoryBackend::get(key)->bytes).janitor_cursor, "winner"); + EXPECT_EQ(decodeGcMaintenanceState(op.read(key, Retry::standard())->bytes).janitor_cursor, "winner"); } diff --git a/src/Disks/tests/gtest_cas_gc_meta_writer.cpp b/src/Disks/tests/gtest_cas_gc_meta_writer.cpp index ada0deb892c1..c51b006be9bf 100644 --- a/src/Disks/tests/gtest_cas_gc_meta_writer.cpp +++ b/src/Disks/tests/gtest_cas_gc_meta_writer.cpp @@ -38,7 +38,7 @@ TEST(CASGcMetaWriter, RealCondemnMarkerJobCompletesAcrossOwnerDestruction) auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const BlobRef ref = DB::Cas::tests::idOf("1"); - const PersistedIncarnation token{"emulated", "tok-1"}; + const PersistedEtag token{"emulated", "tok-1"}; auto gc = std::make_unique(store, DB::Cas::tests::u128Of(kGcId)); backend->arm(); @@ -67,7 +67,7 @@ TEST(CASGcMetaWriter, CondemnMarkerConfirmationIsVisibleAfterDrain) auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const BlobRef ref = DB::Cas::tests::idOf("1"); - const PersistedIncarnation token{"emulated", "tok-1"}; + const PersistedEtag token{"emulated", "tok-1"}; Gc gc(store, DB::Cas::tests::u128Of(kGcId)); EXPECT_FALSE(gc.metaWriterForTest().condemnMarkerConfirmedInProcess(ref, token)); diff --git a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp index 197cda3c1eab..eda515afea23 100644 --- a/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp +++ b/src/Disks/tests/gtest_cas_gc_outcomes_format.cpp @@ -37,7 +37,7 @@ TEST(CASFormatBattery, GcOutcomes) OutcomeEntry e; e.kind = ObjectKind::Blob; e.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}; - e.token = PersistedIncarnation{"etag", "e-1"}; + e.token = PersistedEtag{"etag", "e-1"}; e.outcome = OutcomeKind::Deleted; log.entries.push_back(e); runFormatBattery({FormatId::GcOutcomes, @@ -57,13 +57,13 @@ TEST(CASGCOutcomesFormat, MultiEntryRoundTripAllOutcomes) { OutcomeLog log; log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("aa00000000000000000000000000000a"))}, - PersistedIncarnation{"etag", "etag-1"}, OutcomeKind::Deleted}); + PersistedEtag{"etag", "etag-1"}, OutcomeKind::Deleted}); log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("bb00000000000000000000000000000b"))}, - PersistedIncarnation{"emulated", "7"}, OutcomeKind::Spared}); + PersistedEtag{"emulated", "7"}, OutcomeKind::Spared}); log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("cc00000000000000000000000000000c"))}, - PersistedIncarnation{"emulated", "8"}, OutcomeKind::Replaced}); + PersistedEtag{"emulated", "8"}, OutcomeKind::Replaced}); log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("dd00000000000000000000000000000d"))}, - PersistedIncarnation{"emulated", "9"}, OutcomeKind::Absent}); + PersistedEtag{"emulated", "9"}, OutcomeKind::Absent}); const String text = encodeOutcomeLog(log); const OutcomeLog d = decodeOutcomeLog(text); ASSERT_EQ(d.entries.size(), 4u); @@ -97,7 +97,7 @@ TEST(CASGCOutcomesFormat, RecordRequiresCompleteBlobRefAndTokenGroups) OutcomeLog log; log.entries.push_back({ObjectKind::Blob, BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}, - PersistedIncarnation{"etag", "e-1"}, OutcomeKind::Deleted}); + PersistedEtag{"etag", "e-1"}, OutcomeKind::Deleted}); const String bytes = encodeOutcomeLog(log); for (const auto & [field, expected_message] : { diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index 8b6d74834d41..81d1505dc4cd 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -32,6 +32,35 @@ ManifestRef ref(uint64_t seq, uint64_t inst) { return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = static_cast(inst)}; } + +/// An exact read (mirrors the retired `backend->get(key)`). +std::optional readObj(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +/// A HEAD (mirrors the retired `backend->head(key)`). +std::optional headObj(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::standard()); +} + +/// A one-shot `create`, asserting it committed (mirrors the retired `backend->putIfAbsent(key, bytes)`). +void createObj(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// A delete-exact against `key`/`expected` (mirrors the retired `backend->deleteExact(key, token)`); +/// the caller decides whether to assert the outcome. +Removal removeExact(Backend & backend, const String & key, const Etag & expected) +{ + OperationForTest op(backend); + return (*op).remove(key, expected, Retry::once()); +} } /// (`CASGCBaselineGuard.FreshStateOverTrimmedJournalsFailsClosed` was removed with the snapshot+log ref @@ -72,12 +101,12 @@ TEST(CASGCBaselineGuard, AbsentAdoptedSealFailsClosed) gc.runRegularRound(); /// Corrupt (б): delete the adopted fold seal out from under a healthy gc/state. - const GcState st = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const GcState st = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes); ASSERT_GT(st.snap_generation, 0u); const String seal_key = store->layout().foldSealKey(st.snap_generation, st.snap_attempt); - const HeadResult sh = backend->head(seal_key); - ASSERT_TRUE(sh.exists); - ASSERT_EQ(backend->deleteExact(seal_key, sh.token).kind, DeleteOutcome::Kind::Deleted); + const auto sh = headObj(*backend, seal_key); + ASSERT_TRUE(sh.has_value()); + ASSERT_EQ(removeExact(*backend, seal_key, sh->etag), Removal::Removed); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc.runRegularRound(); }); } @@ -114,10 +143,10 @@ TEST(CASGCRebuild, RecoversLostStateAndConverges) store->renewWatermarkOnce(); /// renews the lease + build-watermark floor /// Capture the round reached before gc/state is destroyed (the rebuild must mint strictly above it). - const auto pre_rebuild_got = backend->get(store->layout().gcStateKey()); + const auto pre_rebuild_got = readObj(*backend, store->layout().gcStateKey()); ASSERT_TRUE(pre_rebuild_got.has_value()); const uint64_t pre_rebuild_round = decodeGcState(pre_rebuild_got->bytes).round; - ASSERT_EQ(backend->deleteExact(store->layout().gcStateKey(), pre_rebuild_got->token).kind, DeleteOutcome::Kind::Deleted); + ASSERT_EQ(removeExact(*backend, store->layout().gcStateKey(), pre_rebuild_got->etag), Removal::Removed); Gc gc2(store, hexToU128("00000000000000000000000000000003")); /// A fresh GC over the orphaned generation artifacts fails closed: re-folding from a fresh gc/state @@ -142,8 +171,8 @@ TEST(CASGCRebuild, RecoversLostStateAndConverges) runRegularRoundReclaiming(gc2); store->renewWatermarkOnce(); } - EXPECT_TRUE(backend->head(store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))})).exists); - EXPECT_TRUE(backend->head(store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(2))})).exists) + EXPECT_TRUE(headObj(*backend, store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))})).has_value()); + EXPECT_TRUE(headObj(*backend, store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(2))})).has_value()) << "a rebuild condemns nothing, so a pre-rebuild drop is retained — never reclaimed by a " "substitute pass, and never lost"; @@ -175,13 +204,13 @@ TEST(CASGCRebuild, RecoversLostGenerationArtifact) gc.runRegularRound(); /// Lose one snapshot run object out from under the healthy state. - const GcState st = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); - const auto seal = decodeFoldSeal(backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + const GcState st = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes); + const auto seal = decodeFoldSeal(readObj(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); ASSERT_FALSE(seal.blob_target_runs.empty()); const String run_key = seal.blob_target_runs.front().key; - const HeadResult rh = backend->head(run_key); - ASSERT_TRUE(rh.exists); - ASSERT_EQ(backend->deleteExact(run_key, rh.token).kind, DeleteOutcome::Kind::Deleted); + const auto rh = headObj(*backend, run_key); + ASSERT_TRUE(rh.has_value()); + ASSERT_EQ(removeExact(*backend, run_key, rh->etag), Removal::Removed); /// A pure ref-carry round would not read the lost run; land a REAL delta so the fold's /// three-cursor merge must stream the prior run — and fails closed on its absence. @@ -194,7 +223,7 @@ TEST(CASGCRebuild, RecoversLostGenerationArtifact) const RebuildReport rep = gc.rebuildBaseline(/*force*/ false); ASSERT_TRUE(rep.performed) << rep.refusal; EXPECT_NO_THROW(gc.runRegularRound()); - EXPECT_TRUE(backend->head(store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))})).exists); + EXPECT_TRUE(headObj(*backend, store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))})).has_value()); } /// FORCE: a healthy state refuses the plain rebuild; FORCE rebuilds; rounds run clean after. @@ -261,12 +290,12 @@ TEST(CASGCRebuild, FrozenCheckpointFrontierExcludesVisibleUnfrontieredTail) fixture::writeRefLogRaw(*backend, layout, RefLogTxn{.ns = ns.string(), .txn_id = RefTxnId{1, 2}, .ops = publishCommittedOps("unfrontiered", unfrontiered), .prev_epoch_seal = std::nullopt}); - ASSERT_TRUE(backend->head(layout.refLogKey(life, RefTxnId{1, 2})).exists); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(headObj(*backend, layout.refLogKey(life, RefTxnId{1, 2})).has_value()); + createObj(*backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt})); Gc gc(store, kGc); const RebuildReport report = gc.rebuildBaseline(/*force=*/false); @@ -302,7 +331,7 @@ TEST(CASGCRebuild, LiveCatalogLifeWithoutCheckpointFailsClosed) Gc gc(store, kGc); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)gc.rebuildBaseline(/*force=*/false); }); - const auto state = backend->get(layout.gcStateKey()); + const auto state = readObj(*backend, layout.gcStateKey()); ASSERT_TRUE(state); EXPECT_EQ(decodeGcState(state->bytes).snap_generation, 0u) << "a rejected recovery must not adopt a new baseline"; @@ -344,16 +373,16 @@ TEST(CASGCRebuild, CheckpointSnapshotAtOlderEpochSealFailsClosed) applyRefLogTxn(through_seal, seal_txn); writeRefSnapshotRaw(*backend, layout, snapshotOf(through_seal, ns.string())); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + createObj(*backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{2, 1}, .checkpoint_snapshot_id = RefTxnId{1, 2}, - .last_epoch_seal = RefTxnId{2, 1}})).outcome, PutOutcome::Done); + .last_epoch_seal = RefTxnId{2, 1}})); Gc gc(store, kGc); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)gc.rebuildBaseline(/*force=*/false); }); - const auto state = backend->get(layout.gcStateKey()); + const auto state = readObj(*backend, layout.gcStateKey()); ASSERT_TRUE(state); EXPECT_EQ(decodeGcState(state->bytes).snap_generation, 0u) << "a rejected checkpoint base must not publish a REBUILD baseline"; @@ -377,11 +406,11 @@ TEST(CASGCRebuild, DamagedGenerationZeroStatePerformsNoCatalogDrainMutation) next.entries[0].removal_started_round = 1; return next; }); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(ns, life_id)), encodeRefCkpt(RefCkpt{ + createObj(*backend, layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(ns, life_id)), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt})); const uint64_t catalog_cas_before = backend->putOverwriteCount(layout.refCatalogKey()); const uint64_t plans_before = ProfileEvents::global_counters[ProfileEvents::CASGCRefWalkPlansBuilt].load(); @@ -416,12 +445,13 @@ TEST(CASGCRebuild, MissingCommittedManifestRefuses) gc.runRegularRound(); /// trim /// Disaster pair: gc/state lost AND tbl_b's manifest body lost. - const HeadResult st = backend->head(store->layout().gcStateKey()); - backend->deleteExact(store->layout().gcStateKey(), st.token); + const auto st = headObj(*backend, store->layout().gcStateKey()); + ASSERT_TRUE(st.has_value()); + removeExact(*backend, store->layout().gcStateKey(), st->etag); const String mkey = store->layout().manifestKey(ManifestId{ns, b}); - const HeadResult mh = backend->head(mkey); - ASSERT_TRUE(mh.exists); - backend->deleteExact(mkey, mh.token); + const auto mh = headObj(*backend, mkey); + ASSERT_TRUE(mh.has_value()); + removeExact(*backend, mkey, mh->etag); Gc gc2(store, hexToU128("00000000000000000000000000000004")); const RebuildReport rep = gc2.rebuildBaseline(/*force*/ false); @@ -429,7 +459,7 @@ TEST(CASGCRebuild, MissingCommittedManifestRefuses) EXPECT_NE(rep.refusal.find("tbl_b"), String::npos) << rep.refusal; /// The lease acquire minted a gen-0 bootstrap body (that is the acquire's contract, not the /// rebuild's); the rebuild's own contract is that NO baseline was blessed by the refusal. - const auto post = backend->get(store->layout().gcStateKey()); + const auto post = readObj(*backend, store->layout().gcStateKey()); ASSERT_TRUE(post.has_value()); const GcState post_state = decodeGcState(post->bytes); EXPECT_EQ(post_state.snap_generation, 0u) << "a refused rebuild must not adopt a baseline"; @@ -463,7 +493,7 @@ TEST(CASGCRebuild, LivePrecommitEdgesIncluded) gc2.runRegularRound(); store->renewWatermarkOnce(); } - EXPECT_TRUE(backend->head(store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(9))})).exists); + EXPECT_TRUE(headObj(*backend, store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(9))})).has_value()); } /// O(budget) attempt iteration: a tiny edge budget forces multi-batch folding; the rebuilt @@ -496,8 +526,9 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) Gc gc(store, kGc); gc.runRegularRound(); gc.runRegularRound(); - const HeadResult st = backend->head(store->layout().gcStateKey()); - backend->deleteExact(store->layout().gcStateKey(), st.token); + const auto st = headObj(*backend, store->layout().gcStateKey()); + ASSERT_TRUE(st.has_value()); + removeExact(*backend, store->layout().gcStateKey(), st->etag); Gc gc2(store, hexToU128("00000000000000000000000000000006")); /// Every shard has `edge_budget + 1` live edges, so each independently crosses the flush budget; @@ -510,9 +541,9 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) /// Multiple rebuild flushes still converge to one authoritative row domain: no more than one /// canonical seq-0 `blob_run` per shard and exactly one `condemned` per shard. These are the cardinalities the /// catalog admission reservation over-covers independently of catalog-entry count. - const GcState rebuilt_state = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const GcState rebuilt_state = decodeGcState(readObj(*backend, store->layout().gcStateKey())->bytes); const CasFoldSeal rebuilt_seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey( + readObj(*backend, store->layout().foldSealKey( rebuilt_state.snap_generation, rebuilt_state.snap_attempt))->bytes, store->layout(), gc_shards); ASSERT_EQ(rebuilt_seal.condemned_summary.size(), gc_shards); @@ -538,7 +569,7 @@ TEST(CASGCRebuild, BatchedRebuildProtectsAllRefs) store->renewWatermarkOnce(); } for (const UInt128 blob : blobs) - EXPECT_TRUE(backend->head(store->layout().blobKey(legacyMetaTestRef(blob))).exists) + EXPECT_TRUE(headObj(*backend, store->layout().blobKey(legacyMetaTestRef(blob))).has_value()) << "blob " << u128ToHex(blob); } @@ -576,7 +607,7 @@ TEST(CASGCRebuild, UnownedAliveManifestOverProtected) gc.runRegularRound(); store->renewWatermarkOnce(); } - EXPECT_TRUE(backend->head(store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(9))})).exists); + EXPECT_TRUE(headObj(*backend, store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(9))})).has_value()); } /// Task 4 (SYSTEM CAS GC REBUILD): a rebuild refuses when ANOTHER Gc instance holds @@ -654,7 +685,7 @@ TEST(CASGCClampSuppression, LandedEdgeBehindClampNeverDeleted) { gc.runRegularRound(); store->renewWatermarkOnce(); - ASSERT_TRUE(backend->head(blob_key).exists) + ASSERT_TRUE(headObj(*backend, blob_key).has_value()) << "round " << i << ": X was deleted while its landed +1 sat unfolded behind the clamp"; } @@ -676,7 +707,7 @@ TEST(CASGCClampSuppression, LandedEdgeBehindClampNeverDeleted) gc.runRegularRound(); store->renewWatermarkOnce(); } - EXPECT_TRUE(backend->head(blob_key).exists); + EXPECT_TRUE(headObj(*backend, blob_key).has_value()); /// And the pipeline is unwedged: a genuinely-unreferenced blob still gets reclaimed. const ManifestRef m3 = ref(3, 0xC3); writeBlobBody(*backend, store->layout(), DB::UInt128(5)); diff --git a/src/Disks/tests/gtest_cas_gc_resume.cpp b/src/Disks/tests/gtest_cas_gc_resume.cpp index 7596ef2cc783..6433576401bc 100644 --- a/src/Disks/tests/gtest_cas_gc_resume.cpp +++ b/src/Disks/tests/gtest_cas_gc_resume.cpp @@ -22,7 +22,8 @@ ManifestRef ref(uint64_t seq, uint64_t inst) } bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash) { - return b.head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + OperationForTest op(b); + return (*op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)}), Retry::once()).has_value(); } /// Whether the CURRENT retired list (any gc-shard) still holds an entry. @@ -30,7 +31,7 @@ bool anyRetiredPending(const PoolPtr & s) { /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. - return anyCondemnedInSeal(s->backend(), s->layout()); + return anyCondemnedInSeal(*s->poolBackendPtr(), s->layout()); } /// Drive regular GC to a fixpoint over the ACK-FLOOR round (renew the store's mount ack after each round; @@ -145,7 +146,8 @@ TEST(CASGCReplay, DeposedRoundRerunsUnderFreshAttempt) Gc gc1(store, hexToU128("00000000000000000000000000000001")); runRegularRoundReclaiming(gc1); store->renewWatermarkOnce(); - const auto after_fold = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + OperationForTest op(*backend); + const auto after_fold = decodeGcState((*op).read(store->layout().gcStateKey(), Retry::once())->bytes); ASSERT_EQ(after_fold.snap_attempt, after_fold.lease.seq); ASSERT_GT(after_fold.snap_generation, 0u); @@ -157,7 +159,7 @@ TEST(CASGCReplay, DeposedRoundRerunsUnderFreshAttempt) EXPECT_THROW(runRegularRoundReclaiming(gc1), DB::Exception); backend->arm_interrupt = false; - const auto after_interrupt = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_interrupt = decodeGcState((*op).read(store->layout().gcStateKey(), Retry::once())->bytes); EXPECT_EQ(after_interrupt.snap_generation, after_fold.snap_generation) << "the denied round-commit CAS must NOT advance the adopted generation"; EXPECT_EQ(after_interrupt.snap_attempt, after_fold.snap_attempt) @@ -165,7 +167,7 @@ TEST(CASGCReplay, DeposedRoundRerunsUnderFreshAttempt) // The deposed round's fold seal is durable under its OWN (unadopted) attempt — unreferenced by gc/state. const uint64_t deposed_attempt = after_fold.lease.seq + 1; // round 2 renewed the lease once const uint64_t deposed_gen = after_fold.snap_generation + 1; - EXPECT_TRUE(backend->head(store->layout().foldSealKey(deposed_gen, deposed_attempt)).exists) + EXPECT_TRUE((*op).head(store->layout().foldSealKey(deposed_gen, deposed_attempt), Retry::once()).has_value()) << "the deposed round's fold seal is durable under its own unadopted attempt (harmless debris)"; // A DIFFERENT leader takes over. The lease steal protocol observes the stalled lease twice before @@ -180,7 +182,7 @@ TEST(CASGCReplay, DeposedRoundRerunsUnderFreshAttempt) EXPECT_NO_THROW(runGcToFixpoint(store, gc2)); EXPECT_FALSE(blobExists(*backend, store->layout(), DB::UInt128(1))); - const auto after_drain = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto after_drain = decodeGcState((*op).read(store->layout().gcStateKey(), Retry::once())->bytes); EXPECT_GT(after_drain.snap_generation, after_fold.snap_generation) << "the round completed under gc2"; EXPECT_NE(after_drain.snap_attempt, deposed_attempt) << "the drained round never adopted the deposed attempt"; diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index 31ac941282db..8139c3fa23ad 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -59,14 +59,38 @@ ManifestRef ref(uint64_t seq, uint64_t inst) return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = static_cast(inst)}; } +std::optional readOf(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +bool headExists(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + +ListPage listOf(Backend & backend, const String & prefix, const String & cursor, size_t limit) +{ + OperationForTest op(backend); + return (*op).list(prefix, cursor, limit, Retry::standard()); +} + +void createRaw(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + (*op).create(key, bytes, Retry::standard()); +} + bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash) { - return b.head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + return headExists(b, layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})); } bool manifestExists(InMemoryBackend & b, const Layout & layout, const ManifestId & id) { - return b.head(layout.manifestKey(id)).exists; + return headExists(b, layout.manifestKey(id)); } PoolPtr openTestPool(std::shared_ptr & out_backend) @@ -117,7 +141,7 @@ class GcStateCasFaultBackend : public InMemoryBackend GcState readState(InMemoryBackend & b, const Pool & s) { - const auto got = b.get(s.layout().gcStateKey()); + const auto got = readOf(b, s.layout().gcStateKey()); if (!got) { ADD_FAILURE() << "gc/state absent"; @@ -168,9 +192,9 @@ std::map snapshotKeyTokens(CasOperation & op) String cursor; while (true) { - const KeyPage page = op.list("", cursor, 100000, Retry::once()); - for (const KeyEntry & k : page.keys) - out[k.key] = k.incarnation ? k.incarnation->render() : String{}; + const ListPage page = op.list("", cursor, 100000, Retry::once()); + for (const ListedKey & k : page.keys) + out[k.key] = k.etag ? k.etag->render() : String{}; if (page.next_cursor.empty()) break; cursor = page.next_cursor; @@ -548,9 +572,10 @@ TEST(CASGCLease, VanishedStateAfterObservationFailsClosed) ASSERT_TRUE(gc1.runRegularRound().acquired_lease); EXPECT_FALSE(gc2.runRegularRound().acquired_lease); /// gc2 records an observation - const auto head = b->head(s->layout().gcStateKey()); /// out-of-model wipe (raw delete) - ASSERT_TRUE(head.exists); - ASSERT_EQ(b->deleteExact(s->layout().gcStateKey(), head.token).kind, DeleteOutcome::Kind::Deleted); + OperationForTest wipe_op(*b); /// out-of-model wipe (raw delete) + const auto meta = (*wipe_op).head(s->layout().gcStateKey(), Retry::standard()); + ASSERT_TRUE(meta.has_value()); + ASSERT_EQ((*wipe_op).remove(s->layout().gcStateKey(), meta->etag, Retry::standard()), Removal::Removed); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { gc2.runRegularRound(); }); } @@ -620,7 +645,7 @@ TEST(CASGCRound, CondemnRoundSealSummaryCountsCondemned) store->renewWatermarkOnce(); const GcState st = readState(*backend, *store); seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); for (const RetiredEntry & e : currentRetiredSet(*backend, store->layout(), /*shard*/0)) if (e.ref == DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(DB::UInt128(1))}) condemned = true; @@ -713,7 +738,7 @@ TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) gc.runRegularRound(); const GcState st1 = readState(*backend, *store); const CasFoldSeal seal1 = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes, + readOf(*backend, store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes, store->layout(), /*gc_shards=*/2); /// No state changes after the parent. The zero defer bound forces an actual fold rather than DEFER, @@ -721,7 +746,7 @@ TEST(CASGCRound, PureCarryRoundPreservesAuthoritativeShardRowsVerbatim) gc.runRegularRound(); const GcState st2 = readState(*backend, *store); const CasFoldSeal seal2 = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes, + readOf(*backend, store->layout().foldSealKey(st2.snap_generation, st2.snap_attempt))->bytes, store->layout(), /*gc_shards=*/2); /// TOTALITY: both seals carry a summary entry for every gc-shard. @@ -798,7 +823,7 @@ TEST(CASGCRound, NonAdoptedAttemptSealIgnored) /// Plant a decoy fold seal under a DIFFERENT attempt at the SAME generation (a deposed leader's /// unadopted artifact). It must be invisible to the adopted-attempt readers. - backend->putIfAbsent(store->layout().foldSealKey(st.snap_generation, st.snap_attempt + 999), + createRaw(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt + 999), "decoy-seal-bytes"); /// No reader resolves the non-adopted attempt: no throw, and the preview is unchanged by the decoy. @@ -1276,14 +1301,14 @@ TEST(CASGCSnapRetention, PrunesOldGenerationsKeepingLastThree) /// Every generation at or below the floor is fully gone (fold seal absent). for (uint64_t g = 1; g <= floor; ++g) { - EXPECT_FALSE(backend->head(store->layout().foldSealKey(g, st.snap_attempt)).exists) + EXPECT_FALSE(headExists(*backend, store->layout().foldSealKey(g, st.snap_attempt))) << "fold seal of pruned generation " << g << " must be gone"; - EXPECT_FALSE(backend->head(store->layout().blobTargetRunKey(g, st.snap_attempt, /*shard*/0, /*seq*/0)).exists) + EXPECT_FALSE(headExists(*backend, store->layout().blobTargetRunKey(g, st.snap_attempt, /*shard*/0, /*seq*/0))) << "blob-target run of pruned generation " << g << " must be gone"; } /// The fold seal at the current generation survives (the live in-degree view). - EXPECT_TRUE(backend->head(store->layout().foldSealKey(st.snap_generation, st.snap_attempt)).exists) + EXPECT_TRUE(headExists(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))) << "the current generation's seal must NOT be pruned"; /// No-loss: the live blob and owner body are intact throughout retention pruning. @@ -1325,9 +1350,9 @@ TEST(CASGCSnapRetention, WholesalePruneReclaimsAllAttemptsIncludingRetiredOutcom const String decoy_outcomes = store->layout().outcomesKey(old_gen, decoy_attempt, /*round*/0, /*shard*/0); const String decoy_seal = store->layout().foldSealKey(old_gen, decoy_attempt); const String decoy_run = store->layout().blobTargetRunKey(old_gen, decoy_attempt, /*shard*/0, /*seq*/0); - backend->putIfAbsent(decoy_outcomes, "decoy-outcomes"); - backend->putIfAbsent(decoy_seal, "decoy-seal"); - backend->putIfAbsent(decoy_run, "decoy-run"); + createRaw(*backend, decoy_outcomes, "decoy-outcomes"); + createRaw(*backend, decoy_seal, "decoy-seal"); + createRaw(*backend, decoy_run, "decoy-run"); /// Drop the ref so the next fold writes a FRESH run under a newer generation and the adopted seal's /// blob_target ref moves OFF `old_gen`. Under T0 reference-parent carry, a still-referenced generation @@ -1343,12 +1368,12 @@ TEST(CASGCSnapRetention, WholesalePruneReclaimsAllAttemptsIncludingRetiredOutcom ASSERT_GT(st.snap_generation, old_gen + 3) << "generation 1 must be below the retention floor"; /// The ENTIRE gc/gen// subtree — across ALL attempts — must be reclaimed. - EXPECT_FALSE(backend->head(decoy_outcomes).exists) << "non-adopted outcomes log leaked past retention"; - EXPECT_FALSE(backend->head(decoy_seal).exists) << "non-adopted fold seal leaked past retention"; - EXPECT_FALSE(backend->head(decoy_run).exists) << "non-adopted blob-target run leaked past retention"; + EXPECT_FALSE(headExists(*backend, decoy_outcomes)) << "non-adopted outcomes log leaked past retention"; + EXPECT_FALSE(headExists(*backend, decoy_seal)) << "non-adopted fold seal leaked past retention"; + EXPECT_FALSE(headExists(*backend, decoy_run)) << "non-adopted blob-target run leaked past retention"; /// Nothing remains under the old generation prefix at all. - const ListPage residue = backend->list(store->layout().gcGenPrefix(old_gen), "", 1000); + const ListPage residue = listOf(*backend, store->layout().gcGenPrefix(old_gen), "", 1000); EXPECT_TRUE(residue.keys.empty()) << "old generation prefix must be fully reclaimed; left " << residue.keys.size() << " objects"; @@ -1393,21 +1418,21 @@ TEST(CASGCSnapRetention, PruneRespectsPrefixWholesaleBudgetAndNeverStrandsAParti /// can wholesale-delete in a single pass, regardless of whatever real fold artifacts already live /// there. for (int i = 0; i < 10; ++i) - backend->putIfAbsent(store->layout().gcGenPrefix(old_gen) + "debris" + std::to_string(i), "x"); + createRaw(*backend, store->layout().gcGenPrefix(old_gen) + "debris" + std::to_string(i), "x"); /// Move the ref off `old_gen`'s run (as `WholesalePruneReclaimsAllAttemptsIncludingRetiredOutcomes` /// does) so the WHOLESALE RETENTION PRUNE -- not the one-shot post-CAS hand-off -- is what /// eventually processes this generation once the cursor reaches it. dropRefTransition(*backend, store->layout(), ns, "tbl", r); - size_t previous_residue = backend->list(store->layout().gcGenPrefix(old_gen), "", 1000).keys.size(); + size_t previous_residue = listOf(*backend, store->layout().gcGenPrefix(old_gen), "", 1000).keys.size(); std::optional drain_start_round; /// first round the residue count actually DROPS std::optional drain_done_round; /// first round the residue reaches zero for (int i = 0; i < 40 && !drain_done_round; ++i) { ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); const GcState st = readState(*backend, *store); - const ListPage residue = backend->list(store->layout().gcGenPrefix(old_gen), "", 1000); + const ListPage residue = listOf(*backend, store->layout().gcGenPrefix(old_gen), "", 1000); if (st.snap_pruned_through >= old_gen) EXPECT_TRUE(residue.keys.empty()) @@ -1465,13 +1490,13 @@ TEST(CASGCSnapRetention, ReclaimsNonAdoptedCurrentGenAttemptViaRetention) const uint64_t orphan_attempt = st.snap_attempt - 1; const String orphan_seal = store->layout().foldSealKey(orphan_gen, orphan_attempt); const String orphan_run = store->layout().blobTargetRunKey(orphan_gen, orphan_attempt, 0, 0); - backend->putIfAbsent(orphan_seal, "orphan-seal"); - backend->putIfAbsent(orphan_run, "orphan-run"); + createRaw(*backend, orphan_seal, "orphan-seal"); + createRaw(*backend, orphan_run, "orphan-run"); /// One more round folds into `orphan_gen` and completes. The orphan must SURVIVE this round — there /// is no current-generation sweep; retention has not yet reached `orphan_gen`. ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - EXPECT_TRUE(backend->head(orphan_seal).exists) + EXPECT_TRUE(headExists(*backend, orphan_seal)) << "orphan must survive its own round — there is no per-round current-gen sweep"; /// Age `orphan_gen` well past the retention floor (keep=3): several more quiescent rounds. The @@ -1483,9 +1508,9 @@ TEST(CASGCSnapRetention, ReclaimsNonAdoptedCurrentGenAttemptViaRetention) const GcState st_after = readState(*backend, *store); ASSERT_GT(st_after.snap_generation, orphan_gen + 3) << "orphan_gen must be below the retention floor"; - EXPECT_FALSE(backend->head(orphan_seal).exists) + EXPECT_FALSE(headExists(*backend, orphan_seal)) << "non-adopted attempt orphan must be reclaimed by wholesale retention once its generation ages out"; - EXPECT_FALSE(backend->head(orphan_run).exists) + EXPECT_FALSE(headExists(*backend, orphan_run)) << "the whole orphan subtree must be reclaimed by wholesale retention"; /// No-loss: the live data is intact throughout. @@ -1520,11 +1545,11 @@ TEST(CASGCRetention, PruneRetainsLiveReferencedRun) /// The gen-1 seal's ref names gen-1's physical run key — capture it so we can assert the OBJECT /// (not just the generation number) survives retention. const auto seal1 = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); ASSERT_EQ(seal1.blob_target_runs.size(), 1u); const String referenced_run_key = seal1.blob_target_runs.front().key; ASSERT_EQ(seal1.blob_target_runs.front().key_generation, ref_gen); - ASSERT_TRUE(backend->head(referenced_run_key).exists); + ASSERT_TRUE(headExists(*backend, referenced_run_key)); /// Several idle rounds: no delta, no retired => pure ref-carry. Each round advances the generation /// and, once adopted_generation > keep, drives the retention prune forward. gen-1 is referenced every @@ -1539,13 +1564,13 @@ TEST(CASGCRetention, PruneRetainsLiveReferencedRun) << "the retention cursor must have advanced past the still-referenced generation"; /// The referenced run object is STILL ALIVE despite the cursor passing its generation. - EXPECT_TRUE(backend->head(referenced_run_key).exists) + EXPECT_TRUE(headExists(*backend, referenced_run_key)) << "a run referenced by the live seal must be retained even after the cursor passes its generation"; /// The current seal still references that same physical gen-1 object (carried, not reconstructed), /// and in-degree resolution THROUGH the carried ref still works. const auto seal_now = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); ASSERT_EQ(seal_now.blob_target_runs.size(), 1u); EXPECT_EQ(seal_now.blob_target_runs.front().key, referenced_run_key); EXPECT_EQ(seal_now.blob_target_runs.front().key_generation, ref_gen); @@ -1576,7 +1601,7 @@ TEST(CASGCRetention, HandOffDeletesSupersededRef) const GcState st1 = readState(*backend, *store); const uint64_t old_gen = st1.snap_generation; const String old_prefix = store->layout().gcGenPrefix(old_gen); - ASSERT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) << "gen-1 prefix must be populated"; + ASSERT_FALSE(listOf(*backend, old_prefix, "", 1000).keys.empty()) << "gen-1 prefix must be populated"; /// Idle-carry the gen-1 ref until the retention cursor has advanced strictly PAST gen-1. Until it /// does, a normal prune could still reclaim gen-1 when the ref moves — the hand-off is only load- @@ -1586,7 +1611,7 @@ TEST(CASGCRetention, HandOffDeletesSupersededRef) ASSERT_GT(readState(*backend, *store).snap_pruned_through, old_gen) << "gen-1 must be behind the retention cursor before the hand-off is exercised"; /// gen-1 is retained (referenced) even though the cursor passed it. - ASSERT_FALSE(backend->list(old_prefix, "", 1000).keys.empty()) + ASSERT_FALSE(listOf(*backend, old_prefix, "", 1000).keys.empty()) << "the referenced gen-1 prefix must still exist before the ref moves off it"; /// A real delta: swap the ref to a new manifest naming a different blob. The next fold writes a FRESH @@ -1601,14 +1626,14 @@ TEST(CASGCRetention, HandOffDeletesSupersededRef) /// The seal no longer references gen-1 ... const GcState st_after = readState(*backend, *store); const auto seal_after = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st_after.snap_generation, st_after.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st_after.snap_generation, st_after.snap_attempt))->bytes); for (const RunRef & rr : seal_after.blob_target_runs) EXPECT_NE(rr.key_generation, old_gen) << "the live seal must have moved its ref off gen-1"; /// ... and the post-CAS hand-off delete reclaimed gen-1's WHOLE prefix (not just the single run /// object): seal, attempt subtree, run — all gone. The ordinary prune would have leaked it because its /// cursor is already past gen-1. - const ListPage residue = backend->list(old_prefix, "", 1000); + const ListPage residue = listOf(*backend, old_prefix, "", 1000); EXPECT_TRUE(residue.keys.empty()) << "the superseded gen-1 prefix must be hand-off deleted; left " << residue.keys.size() << " objects"; @@ -1661,13 +1686,13 @@ TEST(CASGCRetention, HandoffOwnBudgetSurvivesAPruneHeavyRound) ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); const uint64_t handoff_gen = readState(*backend, *store).snap_generation; const String handoff_prefix = store->layout().gcGenPrefix(handoff_gen); - ASSERT_FALSE(backend->list(handoff_prefix, "", 1000).keys.empty()); + ASSERT_FALSE(listOf(*backend, handoff_prefix, "", 1000).keys.empty()); for (int i = 0; i < 20; ++i) ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); ASSERT_GT(readState(*backend, *store).snap_pruned_through, handoff_gen) << "the hand-off generation must be behind the cursor before this test is meaningful"; - ASSERT_FALSE(backend->list(handoff_prefix, "", 1000).keys.empty()) + ASSERT_FALSE(listOf(*backend, handoff_prefix, "", 1000).keys.empty()) << "still referenced -- must survive despite the cursor having passed it"; /// The PRUNE-DEBRIS generation: a second table, on the OTHER shard, unreferenced from the start, @@ -1684,7 +1709,7 @@ TEST(CASGCRetention, HandoffOwnBudgetSurvivesAPruneHeavyRound) << "the debris generation must still be ahead of the cursor when its drop folds, or the hand-off " "(not the prune) would claim it"; for (int i = 0; i < 10; ++i) - backend->putIfAbsent(store->layout().gcGenPrefix(debris_gen) + "debris" + std::to_string(i), "x"); + createRaw(*backend, store->layout().gcGenPrefix(debris_gen) + "debris" + std::to_string(i), "x"); dropRefTransition(*backend, store->layout(), ns, "debris", r_debris); /// Drive rounds until the debris generation is MID-DRAIN (prune has started but not yet finished it -- @@ -1693,11 +1718,11 @@ TEST(CASGCRetention, HandoffOwnBudgetSurvivesAPruneHeavyRound) for (int i = 0; i < 20 && !mid_drain; ++i) { ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - const size_t residue = backend->list(store->layout().gcGenPrefix(debris_gen), "", 1000).keys.size(); + const size_t residue = listOf(*backend, store->layout().gcGenPrefix(debris_gen), "", 1000).keys.size(); mid_drain = residue > 0 && residue < 10; } ASSERT_TRUE(mid_drain) << "the debris generation never reached a partially-drained state to test against"; - ASSERT_FALSE(backend->list(handoff_prefix, "", 1000).keys.empty()) + ASSERT_FALSE(listOf(*backend, handoff_prefix, "", 1000).keys.empty()) << "the hand-off generation must still be intact (untouched) going into the contended round"; /// NOW, in a round where the prune is busy mid-drain on the debris generation (spending its entire @@ -1706,17 +1731,17 @@ TEST(CASGCRetention, HandoffOwnBudgetSurvivesAPruneHeavyRound) writeBlobBody(*backend, store->layout(), blob_keep_2); writeManifestRaw(*backend, store->layout(), ns, r_keep_2, {blobEntryFor("a", blob_keep_2)}); publishCommittedTransition(*backend, store->layout(), ns, "keep", r_keep_1, r_keep_2); - const size_t debris_residue_before = backend->list(store->layout().gcGenPrefix(debris_gen), "", 1000).keys.size(); + const size_t debris_residue_before = listOf(*backend, store->layout().gcGenPrefix(debris_gen), "", 1000).keys.size(); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); /// THE LOAD-BEARING ASSERTIONS: the prune spent its whole (separate) budget on the debris generation /// this very round (proving the two really contended for I/O in the same round) ... - const size_t debris_residue_after = backend->list(store->layout().gcGenPrefix(debris_gen), "", 1000).keys.size(); + const size_t debris_residue_after = listOf(*backend, store->layout().gcGenPrefix(debris_gen), "", 1000).keys.size(); EXPECT_EQ(debris_residue_before - debris_residue_after, 2u) << "the prune must have spent its entire per-round budget on the debris generation this round"; /// ... and the hand-off, drawing from its OWN reserve, still fully reclaimed the generation the ref /// just moved off -- zero, not starved to zero by the prune's consumption. - EXPECT_TRUE(backend->list(handoff_prefix, "", 1000).keys.empty()) + EXPECT_TRUE(listOf(*backend, handoff_prefix, "", 1000).keys.empty()) << "the hand-off must not be starved by a prune-heavy round that exhausted a SEPARATE budget"; } @@ -1747,11 +1772,11 @@ TEST(CASGCRetention, LosingRoundNeverDestroysParentSealGeneration) const GcState st1 = readState(*backend, *store); const uint64_t g_parent = st1.snap_generation; - const auto seal1 = decodeFoldSeal(backend->get(layout.foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); + const auto seal1 = decodeFoldSeal(readOf(*backend, layout.foldSealKey(st1.snap_generation, st1.snap_attempt))->bytes); ASSERT_EQ(seal1.blob_target_runs.size(), 1u); const String parent_run_key = seal1.blob_target_runs.front().key; const String parent_gen_prefix = layout.gcGenPrefix(g_parent); - ASSERT_FALSE(backend->list(parent_gen_prefix, "", 1000).keys.empty()); + ASSERT_FALSE(listOf(*backend, parent_gen_prefix, "", 1000).keys.empty()); /// A real delta: swap the ref to a new manifest naming a different blob. The next fold will move /// shard 0's run OFF `g_parent` onto a fresh generation. @@ -1783,9 +1808,9 @@ TEST(CASGCRetention, LosingRoundNeverDestroysParentSealGeneration) /// GREEN evidence: the losing round's pre-CAS prune must NOT have destroyed `g_parent` — it is still /// exactly what the (unreplaced, still-adopted) parent seal references. - EXPECT_FALSE(backend->list(parent_gen_prefix, "", 1000).keys.empty()) + EXPECT_FALSE(listOf(*backend, parent_gen_prefix, "", 1000).keys.empty()) << "a losing round must never destroy the generation the still-adopted parent seal references"; - EXPECT_TRUE(backend->head(parent_run_key).exists) + EXPECT_TRUE(headExists(*backend, parent_run_key)) << "the parent seal's exact run object must survive a losing round's pre-CAS prune"; /// GC is NOT wedged: gc/state is unchanged (the CAS never committed) and the original blob still @@ -1801,7 +1826,7 @@ TEST(CASGCRetention, LosingRoundNeverDestroysParentSealGeneration) const uint64_t g_after = readState(*backend, *store).snap_generation; ASSERT_GT(g_after, g_parent); for (uint64_t g = g_parent + 1; g < g_after; ++g) - EXPECT_TRUE(backend->list(layout.gcGenPrefix(g), "", 1000).keys.empty()) + EXPECT_TRUE(listOf(*backend, layout.gcGenPrefix(g), "", 1000).keys.empty()) << "generation " << g << " (the losing round's own abandoned attempt debris, referenced by " "neither the parent nor the new proposed seal) must still be reclaimed on a successful " "round — the fix must not disable pruning"; @@ -1837,7 +1862,7 @@ TEST(CASGCSnapRetention, KeepZeroPrunesNothing) { bool seal_present = false; for (uint64_t a = 0; a <= st.snap_attempt && !seal_present; ++a) - seal_present = backend->head(store->layout().foldSealKey(g, a)).exists; + seal_present = headExists(*backend, store->layout().foldSealKey(g, a)); EXPECT_TRUE(seal_present) << "keep==0: seal of generation " << g << " must remain"; } } @@ -1848,7 +1873,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) PoolConfig config; config.pool_prefix = "p"; /// The GC runner owns a different mount from the synthetic `test` watermark below. This keeps the - /// cursor-sweep assertions in the parent process without replacing its live keeper incarnation. + /// cursor-sweep assertions in the parent process without replacing its live renewer incarnation. config.server_root_id = "gc-runner"; config.manifest_sweep_list_budget_keys = 1; config.manifest_sweep_delete_budget_keys = 1; @@ -1911,14 +1936,17 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) const std::optional life = CasRefCatalog::lifeIfCataloged(op, store->layout(), ns); ASSERT_TRUE(life.has_value()); const String ckpt_key = store->layout().refCkptKey(*life); - const auto old_ckpt = backend->get(ckpt_key); + const auto old_ckpt = readOf(*backend, ckpt_key); ASSERT_TRUE(old_ckpt.has_value()); - ASSERT_EQ(backend->putOverwrite(ckpt_key, encodeRefCkpt(RefCkpt{ - .life_epoch = 1, - .committed_through = RefTxnId{2, 1}, - .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = RefTxnId{1, 2}, - }), old_ckpt->token).outcome, PutOutcome::Done); + { + OperationForTest ckpt_op(*backend); + ASSERT_TRUE(std::holds_alternative((*ckpt_op).replace(ckpt_key, encodeRefCkpt(RefCkpt{ + .life_epoch = 1, + .committed_through = RefTxnId{2, 1}, + .checkpoint_snapshot_id = std::nullopt, + .last_epoch_seal = RefTxnId{1, 2}, + }), old_ckpt->etag, Retry::standard()))); + } /// The list budget is one key per round, so reclaiming both debris bodies takes a circuit. for (int round = 0; round < 12; ++round) @@ -1937,7 +1965,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) { const GcState st = readState(*backend, *store); const CasFoldSeal seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + readOf(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); const auto it = seal.ref_lives.find(catalogLifeIdForTest(*backend, store->layout(), ns)); ASSERT_NE(it, seal.ref_lives.end()) << "the round must have sealed a coverage row"; EXPECT_FALSE(it->second.coverage.hold.has_value()) << "a held namespace can never reach the premise"; @@ -1958,7 +1986,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) auto invalid_store = openTestPoolWithConfig(foreign_backend, std::move(foreign_config)); const String foreign_mount_key = invalid_store->layout().mountKey("test"); setWatermarkMinActive(*foreign_backend, invalid_store->layout(), "test", r1.writer_epoch, /*min_active_build_sequence*/6); - const auto occupant_before = foreign_backend->get(foreign_mount_key); + const auto occupant_before = readOf(*foreign_backend, foreign_mount_key); ASSERT_TRUE(occupant_before.has_value()); const uint64_t violations_before = ProfileEvents::global_counters[ProfileEvents::CASMountExclusivityViolation].load(); @@ -1969,7 +1997,7 @@ TEST(CASGCRound, OrphanManifestCursorSweepDeletesAndPersistsCursor) violations_before + 1) << "a runtime that never observed a deposition must report the foreign occupant as a broken " "single-writer guarantee"; - const auto occupant_after = foreign_backend->get(foreign_mount_key); + const auto occupant_after = readOf(*foreign_backend, foreign_mount_key); ASSERT_TRUE(occupant_after.has_value()) << "the release must never delete another incarnation's lease"; EXPECT_EQ(occupant_after->bytes, occupant_before->bytes) << "the release must leave the slot byte-for-byte untouched, never stamp our farewell over it"; diff --git a/src/Disks/tests/gtest_cas_gc_round_defer.cpp b/src/Disks/tests/gtest_cas_gc_round_defer.cpp index 86983a35b1bc..43a6a909e86a 100644 --- a/src/Disks/tests/gtest_cas_gc_round_defer.cpp +++ b/src/Disks/tests/gtest_cas_gc_round_defer.cpp @@ -56,7 +56,8 @@ TEST(CASGCRoundDefer, GraduationDueDetectsDuePendingAndRoundCrossing) .oldest_nonpending_condemn_round = 2}}}); Gc gc(store, kGc); - const GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + OperationForTest raw_op(*backend); + const GcState state = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); EXPECT_FALSE(gc.graduationDueForTest(state, /*current_round=*/2)) << "oldest non-pending condemn round (2) is not < current_round (2); not yet due to graduate"; @@ -67,7 +68,7 @@ TEST(CASGCRoundDefer, GraduationDueDetectsDuePendingAndRoundCrossing) injectCondemnedSummarySeal(*backend, layout, /*generation*/1, /*attempt*/1, /*gc_shards*/1, {{0, CondemnedSummary{.condemned_total = 1, .pending_total = 1, .oldest_nonpending_condemn_round = std::numeric_limits::max()}}}); - const GcState state_pending = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState state_pending = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); EXPECT_TRUE(gc.graduationDueForTest(state_pending, /*current_round=*/0)) << "a delete_pending entry must force graduationDue true regardless of current_round"; @@ -84,13 +85,14 @@ TEST(CASGCRoundDefer, GraduationDueFailsClosedWhenSealMissing) injectCondemnedSummarySeal(*backend, layout, /*generation*/1, /*attempt*/1, /*gc_shards*/1, {{0, CondemnedSummary{}}}); - const GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + OperationForTest raw_op(*backend); + const GcState state = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); /// Delete the adopted seal object (corrupt destructive bookkeeping). const String seal_key = layout.foldSealKey(state.snap_generation, state.snap_attempt); - const HeadResult h = backend->head(seal_key); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->deleteExact(seal_key, h.token).kind, DeleteOutcome::Kind::Deleted); + const auto h = (*raw_op).head(seal_key, Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ((*raw_op).remove(seal_key, h->etag, Retry::once()), Removal::Removed); Gc gc(store, kGc); EXPECT_TRUE(gc.graduationDueForTest(state, /*current_round=*/5)) @@ -107,7 +109,8 @@ TEST(CASGCRoundDefer, GraduationDueFalseOnAllZeroSummary) injectCondemnedSummarySeal(*backend, layout, /*generation*/1, /*attempt*/1, /*gc_shards*/2, {{0, CondemnedSummary{}}, {1, CondemnedSummary{}}}); - const GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + OperationForTest raw_op(*backend); + const GcState state = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); Gc gc(store, kGc); EXPECT_FALSE(gc.graduationDueForTest(state, /*current_round=*/9)) @@ -116,7 +119,7 @@ TEST(CASGCRoundDefer, GraduationDueFalseOnAllZeroSummary) /// Fail-closed if the summary is NOT total over gc_shards (shard 1 missing). injectCondemnedSummarySeal(*backend, layout, /*generation*/1, /*attempt*/1, /*gc_shards*/2, {{0, CondemnedSummary{}}}); - const GcState partial = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState partial = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); EXPECT_TRUE(gc.graduationDueForTest(partial, /*current_round=*/9)) << "a summary not total over gc_shards is corrupt => fail-closed force-fold"; } @@ -143,7 +146,8 @@ TEST(CASGCRoundDefer, ChangedShardCountIsZeroWhenQuiescent) /// trim, so THIS round's fold seal finally /// captures the shard's actual current token. - const GcState quiescent_state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + OperationForTest raw_op(*backend); + const GcState quiescent_state = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); EXPECT_EQ(gc.listRefPrefixForTest(quiescent_state).changed_shards, 0u) << "a quiescent shard (listed token == sealed token) must not count as changed"; @@ -172,10 +176,13 @@ TEST(CASGCRoundDefer, HotEnumerationOffersLogsAndSnapshotsButNeverCheckpointOrFi const String snap_key = layout.refSnapshotKey(life, id); const String ckpt_key = layout.refCkptKey(life); const String file_key = layout.namespaceFileKey(life, "f"); - ASSERT_EQ(backend->putIfAbsent(log_key, "log").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(snap_key, "snap").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(ckpt_key, "ckpt").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(file_key, "file").outcome, PutOutcome::Done); + { + OperationForTest seed_op(*backend); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(log_key, "log", Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(snap_key, "snap", Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(ckpt_key, "ckpt", Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(file_key, "file", Retry::once()))); + } backend->resetCounts(); Gc gc(store, kGc); @@ -196,7 +203,10 @@ TEST(CASGCRoundDefer, ListedLifeAbsentFromThePostListCatalogCutIsInertDebris) const Layout & layout = store->layout(); const NamespaceLifeId unknown = NamespaceLifeId::fromCatalogEntry(RootNamespace{"cannot-authorize"}, UInt128{0x456}); const String log_key = layout.refLogKey(unknown, RefTxnId{1, 1}); - ASSERT_EQ(backend->putIfAbsent(log_key, "not-read-on-defer").outcome, PutOutcome::Done); + { + OperationForTest seed_op(*backend); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(log_key, "not-read-on-defer", Retry::once()))); + } backend->resetCounts(); Gc gc(store, kGc); @@ -222,7 +232,10 @@ TEST(CASGCRoundDefer, SnapshotLifeAbsentFromThePostListCatalogCutIsInertDebris) const Layout & layout = store->layout(); const NamespaceLifeId unknown = NamespaceLifeId::fromCatalogEntry(RootNamespace{"cannot-authorize"}, UInt128{0x457}); const String snapshot_key = layout.refSnapshotKey(unknown, RefTxnId{1, 1}); - ASSERT_EQ(backend->putIfAbsent(snapshot_key, "not-read-on-defer").outcome, PutOutcome::Done); + { + OperationForTest seed_op(*backend); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(snapshot_key, "not-read-on-defer", Retry::once()))); + } backend->resetCounts(); Gc gc(store, kGc); @@ -262,7 +275,8 @@ TEST(CASGCRoundDefer, IdleRoundDefersAndReadsNoGeneration) const uint64_t fold_round_gets = backend->getTotal(); EXPECT_GT(fold_round_gets, 0u) << "sanity: a real fold round performs some GETs"; - const auto st_before = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + OperationForTest raw_op(*backend); + const auto st_before = decodeGcState((*raw_op).read(store->layout().gcStateKey(), Retry::once())->bytes); backend->resetCounts(); const RoundReport rep = gc.runRegularRound(); /// round 2: genuinely quiesced now => must defer @@ -278,7 +292,7 @@ TEST(CASGCRoundDefer, IdleRoundDefersAndReadsNoGeneration) EXPECT_EQ(rep.round, fold_rep.round) << "a deferred round re-adopts the already-committed round, not a fabricated new one"; - const auto st_after = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const auto st_after = decodeGcState((*raw_op).read(store->layout().gcStateKey(), Retry::once())->bytes); EXPECT_EQ(st_after.snap_generation, st_before.snap_generation) << "a deferred round must not mint a new generation (snapshot rebuild elided)"; EXPECT_EQ(st_after.snap_attempt, st_before.snap_attempt); @@ -383,8 +397,8 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP = NamespaceLifeId::fromCatalogEntry(RootNamespace{"dead/b"}, UInt128{0xDB}); const String key_a = layout.refCkptKey(dead_a); const String key_b = layout.refCkptKey(dead_b); - ASSERT_EQ(backend->putIfAbsent(key_a, "dead-a").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(key_b, "dead-b").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create(key_a, "dead-a", Retry::once()))); + ASSERT_TRUE(std::holds_alternative(op.create(key_b, "dead-b", Retry::once()))); /// Establish real opaque backend progress rather than fabricating a cursor value. One key remains /// after this page and the durable cursor must be non-empty. @@ -396,19 +410,19 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP ASSERT_EQ(partial.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(partial.state); ASSERT_FALSE(partial.state->janitor_cursor.empty()); - ASSERT_EQ(static_cast(backend->head(key_a).exists) + static_cast(backend->head(key_b).exists), 1u); + ASSERT_EQ(static_cast(op.head(key_a, Retry::once()).has_value()) + static_cast(op.head(key_b, Retry::once()).has_value()), 1u); /// Give the forced fold a nonempty, fully proved authoritative universe. The R11 floor correctly /// refuses to open the destructive gate for an empty 0-of-0 universe even in the test-only policy. const RootNamespace live_namespace{"live/frontier@cas@"}; fixture::admitLive(*backend, layout, live_namespace); - ASSERT_EQ(backend->putIfAbsent( + ASSERT_TRUE(std::holds_alternative(op.create( layout.refCkptKey(fixture::fixtureLife(live_namespace)), encodeRefCkpt(RefCkpt{ .life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, - PutOutcome::Done); + .last_epoch_seal = std::nullopt}), + Retry::once()))); backend->resetCounts(); std::vector phases; @@ -442,14 +456,14 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP ASSERT_TRUE(deferred_progress.state); EXPECT_EQ(deferred_progress.state->janitor_cursor, partial.state->janitor_cursor) << "a suppressed DEFER page is undecided and must remain selected for the authoritative fold"; - EXPECT_EQ(static_cast(backend->head(key_a).exists) + static_cast(backend->head(key_b).exists), 1u); + EXPECT_EQ(static_cast(op.head(key_a, Retry::once()).has_value()) + static_cast(op.head(key_b, Retry::once()).has_value()), 1u); - const auto gc_state = backend->get(layout.gcStateKey()); + const auto gc_state = op.read(layout.gcStateKey(), Retry::once()); ASSERT_TRUE(gc_state); const GcState state = decodeGcState(gc_state->bytes); EXPECT_EQ(state.snap_generation, 0u); EXPECT_EQ(state.snap_attempt, 0u); - EXPECT_FALSE(backend->head(layout.foldSealKey(1, 1)).exists) + EXPECT_FALSE(op.head(layout.foldSealKey(1, 1), Retry::once()).has_value()) << "maintenance on DEFER must not publish a fold successor"; backend->resetCounts(); @@ -468,7 +482,7 @@ TEST(CASGCRoundDefer, DeferredRoundRetriesPartialJanitorPageAtForcedFoldWithoutP EXPECT_EQ(folded_cleanup->metrics.at("janitor_pages"), 1u); EXPECT_GE(folded_cleanup->metrics.at("janitor_keys"), 1u); EXPECT_EQ(folded_cleanup->metrics.at("janitor_deleted"), 1u); - EXPECT_EQ(static_cast(backend->head(key_a).exists) + static_cast(backend->head(key_b).exists), 0u) + EXPECT_EQ(static_cast(op.head(key_a, Retry::once()).has_value()) + static_cast(op.head(key_b, Retry::once()).has_value()), 0u) << "the fold must retry and delete the exact page that DEFER left undecided"; const GcMaintenanceReadResult completed = readGcMaintenanceState(op, layout); ASSERT_EQ(completed.status, GcMaintenanceReadStatus::Valid); diff --git a/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp b/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp index a510b818ee6c..8d21ca73da19 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_incarnation.cpp @@ -36,6 +36,22 @@ ManifestRef testRef(uint64_t seq) return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = 1}; } +/// ---- Small raw-fixture request-engine wrappers shared by the tests below ---- + +/// True iff `key` exists. +bool existsAt(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::once()).has_value(); +} + +/// Unconditional create of a fresh key (the fixture's own setup, never a real conflict). +void createAt(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + EXPECT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + } /// Review I5: `discoverUniverse` is catalog-authoritative (Task 4-C), and this test used to survive @@ -76,10 +92,10 @@ TEST(CASGCShardIncarnation, DiscoveryEqualsPresentShards) { CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); std::erase_if(snap.catalog.entries, [&](const CatalogEntry & e) { return e.ns.string() == ns_uncataloged.string(); }); - const HeadResult h = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h.token).outcome, - PutOutcome::Done); + const auto h = op.head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h->etag, Retry::once()))); } const auto universe = gc.discoverUniverseForTest(); @@ -123,10 +139,11 @@ TEST(CASGCShardIncarnation, DuplicateLifeIdStopsDestructiveRoundAndRebuild) .incarnation = UInt128{77}, .removal_started_round = 1}, }; - const auto empty_catalog = backend->get(layout.refCatalogKey()); + OperationForTest op(*backend); + const auto empty_catalog = (*op).read(layout.refCatalogKey(), Retry::once()); ASSERT_TRUE(empty_catalog); - ASSERT_EQ(backend->putOverwrite( - layout.refCatalogKey(), encodeRefCatalog(catalog), empty_catalog->token).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.refCatalogKey(), encodeRefCatalog(catalog), empty_catalog->etag, Retry::once()))); backend->resetCounts(); Gc gc(store, hexToU128("0000000000000000000000000000000a")); @@ -166,17 +183,17 @@ TEST(CASGCShardIncarnation, DeadLifeStreamIsOpaqueInertDebris) [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); ASSERT_NE(it, snap.catalog.entries.end()); it->incarnation = UInt128(22); // "recreated" -- same name, different (empty) key space - const HeadResult h = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h.token).outcome, - PutOutcome::Done); + const auto h = op.head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h->etag, Retry::once()))); } const NamespaceLifeId current_life = NamespaceLifeId::fromCatalogEntry(ns, UInt128(22)); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(current_life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(current_life), encodeRefCkpt(RefCkpt{ .life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), Retry::once()))); const ManifestRef current_ref = testRef(2); writeBlobBody(*backend, layout, UInt128(22)); writeManifestRaw(*backend, layout, ns, current_ref, {blobEntryFor("current", UInt128(22))}); @@ -223,18 +240,18 @@ TEST(CASGCShardIncarnation, CurrentLifeCheckpointIsReadByExactKeyOutsideHotList) [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); ASSERT_NE(it, snap.catalog.entries.end()); *it = after_rebirth; // "recreated" -- same name, new (current) incarnation 22 - const HeadResult h = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h.token).outcome, - PutOutcome::Done); + const auto h = op.head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h->etag, Retry::once()))); } /// The successor's own genesis `_ckpt`, published for the current physical life. Hiding it from /// LIST must be irrelevant because the walk obtains state only through exact GETs. const NamespaceLifeId current_life = NamespaceLifeId::fromCatalogEntry(ns, UInt128(22)); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(current_life), + createAt(*backend, layout.refCkptKey(current_life), encodeRefCkpt(RefCkpt{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt})); backend->hide(layout.refCkptKey(current_life)); backend->resetCounts(); std::vector phases; @@ -252,7 +269,7 @@ TEST(CASGCShardIncarnation, CurrentLifeCheckpointIsReadByExactKeyOutsideHotList) << "the only broader LIST is the separately paced janitor page"; EXPECT_EQ(backend->holesServed(), 1u) << "the hidden checkpoint is omitted only from the janitor's broad page, never from the hot stream LIST"; - EXPECT_TRUE(backend->head(layout.refCkptKey(current_life)).exists) + EXPECT_TRUE(existsAt(*backend, layout.refCkptKey(current_life))) << "the post-page catalog cut retains the current life even when LIST omitted its checkpoint"; const auto cleanup = std::find_if(phases.begin(), phases.end(), [](const GcPhaseRecord & phase) { @@ -288,15 +305,15 @@ TEST(CASGCShardIncarnation, UncatalogedStreamLifeDefersWithoutInventingNamespace { CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); std::erase_if(snap.catalog.entries, [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); - const HeadResult h = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->putOverwrite(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h.token).outcome, - PutOutcome::Done); + const auto h = op.head(layout.refCatalogKey(), Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_TRUE(std::holds_alternative( + op.replace(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), h->etag, Retry::once()))); } const RoundReport report = gc.runRegularRound({}, /*allow_steal=*/true, UniversePolicy::Authoritative); EXPECT_TRUE(report.anomalies.empty()); - EXPECT_FALSE(backend->list(layout.namespaceStreamPrefix(forgotten_life), "", 100).keys.empty()); + EXPECT_FALSE(op.list(layout.namespaceStreamPrefix(forgotten_life), "", 100, Retry::once()).keys.empty()); } /// State-tree objects are point-addressed only. A stalled creator's checkpoint and an unowned opaque @@ -321,15 +338,15 @@ TEST(CASGCShardIncarnation, StateCheckpointsOutsideCatalogAreInertToHotWalk) /// Creating entry names -- exactly what `completeCreation` durably leaves behind if the creator /// crashes between its own steps 2 and 3. const NamespaceLifeId creating_life = NamespaceLifeId::fromCatalogEntry(creating_ns, UInt128(33)); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(creating_life), + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(creating_life), encodeRefCkpt(RefCkpt{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), Retry::once()))); /// Opaque state debris with no corresponding catalog entry. const NamespaceLifeId gone_life = NamespaceLifeId::fromCatalogEntry(unrelated_gone_ns, UInt128(44)); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(gone_life), + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(gone_life), encodeRefCkpt(RefCkpt{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), Retry::once()))); /// Add one fully current stream so the round performs a fold rather than stopping at an empty /// walk. Catalog and checkpoint admission keep this traffic out of the janitor's dead-life set, @@ -337,9 +354,9 @@ TEST(CASGCShardIncarnation, StateCheckpointsOutsideCatalogAreInertToHotWalk) const RootNamespace ordinary_ns{"srv1/tblOrdinaryTraffic"}; fixture::admitLive(*backend, layout, ordinary_ns); const NamespaceLifeId ordinary_life = fixture::fixtureLife(ordinary_ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(ordinary_life), + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(ordinary_life), encodeRefCkpt(RefCkpt{.life_epoch = std::optional{1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), Retry::once()))); appendRefLogSeed(*backend, layout, ordinary_ns, {}); backend->resetCounts(); @@ -370,9 +387,9 @@ TEST(CASGCShardIncarnation, StateCheckpointsOutsideCatalogAreInertToHotWalk) << "Creating is retained by the janitor cut but excluded from hot checkpoint intake"; EXPECT_EQ(backend->getCount(layout.refCkptKey(gone_life)), 0u) << "uncataloged state debris is classified by the janitor page, never exact-read by the hot walk"; - EXPECT_TRUE(backend->head(layout.refCkptKey(creating_life)).exists); - EXPECT_TRUE(backend->head(layout.refCkptKey(ordinary_life)).exists); - EXPECT_FALSE(backend->head(layout.refCkptKey(gone_life)).exists) + EXPECT_TRUE(op.head(layout.refCkptKey(creating_life), Retry::once()).has_value()); + EXPECT_TRUE(op.head(layout.refCkptKey(ordinary_life), Retry::once()).has_value()); + EXPECT_FALSE(op.head(layout.refCkptKey(gone_life), Retry::once()).has_value()) << "catalog absence is inert to the hot walk but authorizes the later janitor exact-token delete"; const auto cleanup = std::find_if(phases.begin(), phases.end(), [](const GcPhaseRecord & phase) { @@ -468,7 +485,7 @@ TEST(CASGCShardIncarnation, NewbornPrecommitProtectsDedupBlobAgainstConcurrentDr const String b1_key = store->layout().blobKey(b1_ref); const std::optional b1_observed = op.head(b1_key, Retry::once()); ASSERT_TRUE(b1_observed) << "b1 body must be present after the seed putBlob"; - const PersistedIncarnation b1_token = PersistedIncarnation::capture(b1_observed->incarnation); + const PersistedEtag b1_token = PersistedEtag::capture(b1_observed->etag); /// --- Phase 2: Inject gc/state at round 1 with b1 CONDEMNED (body still present). --- /// This simulates GC having advanced to round 1 and retired b1 (condemned token recorded @@ -521,7 +538,7 @@ TEST(CASGCShardIncarnation, NewbornPrecommitProtectsDedupBlobAgainstConcurrentDr /// The condemned token is bound UNCHANGED — no displacement happens (and none is needed). const std::optional b1_after = op.head(b1_key, Retry::once()); ASSERT_TRUE(b1_after); - EXPECT_TRUE(b1_token.matches(b1_after->incarnation)) + EXPECT_TRUE(b1_token.matches(b1_after->etag)) << "gc_shards=" << gc_shards << ": no copy-forward under the Phase-A contract — the " "incarnation stays; the folded edge will spare it at the next fold (no round runs " "here to delete it)"; diff --git a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp index 62f12b94655c..81c3af12fa86 100644 --- a/src/Disks/tests/gtest_cas_gc_shard_plan.cpp +++ b/src/Disks/tests/gtest_cas_gc_shard_plan.cpp @@ -500,9 +500,9 @@ TEST(CASGCShardTwoReplica, DisjointShardsConcurrentPerShardRuns) /// The blob-target runs for both shards are durably present (the reducer's write-once `putIfAbsent`), /// at disjoint object keys. - EXPECT_TRUE(backend->head(layout.blobTargetRunKey(kNewGen, kAttempt, /*shard=*/0, /*seq=*/0)).exists) + EXPECT_TRUE(op.head(layout.blobTargetRunKey(kNewGen, kAttempt, /*shard=*/0, /*seq=*/0), Retry::once()).has_value()) << "shard-0 blob-target run must be durably written by r0.reduce"; - EXPECT_TRUE(backend->head(layout.blobTargetRunKey(kNewGen, kAttempt, /*shard=*/1, /*seq=*/0)).exists) + EXPECT_TRUE(op.head(layout.blobTargetRunKey(kNewGen, kAttempt, /*shard=*/1, /*seq=*/0), Retry::once()).has_value()) << "shard-1 blob-target run must be durably written by r1.reduce"; /// (c) MERGED IN-DEGREE — the merged in-degrees across both shards equal the expected edge multiset. @@ -559,14 +559,15 @@ TEST(CASGCShardRetireDrain, ReclaimsDroppableBlobOwnedByNonZeroShard) const ManifestId id0{ns, r0}; const ManifestId id1{ns, r1}; + OperationForTest verify_op(*backend); /// Local blobExists (the round-level helper is file-local to gtest_cas_gc_round.cpp). auto blobExists = [&](const UInt128 & hash) { - return backend->head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + return (*verify_op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)}), Retry::once()).has_value(); }; auto manifestExists = [&](const ManifestId & id) { - return backend->head(layout.manifestKey(id)).exists; + return (*verify_op).head(layout.manifestKey(id), Retry::once()).has_value(); }; /// Whether ANY gc-shard still holds an in-flight condemned entry (the ack-floor deletion pipeline is /// in flight while this is true). Condemned state is reconstructed from the adopted fold seal's @@ -608,7 +609,7 @@ TEST(CASGCShardRetireDrain, ReclaimsDroppableBlobOwnedByNonZeroShard) /// While both refs are live: each blob's in-degree is 1 in its OWNING shard's run, and nothing is /// collected (no-loss). Derive generation/attempt from gc/state — never hardcode. - const GcState live = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState live = decodeGcState((*verify_op).read(layout.gcStateKey(), Retry::once())->bytes); ASSERT_GT(live.snap_generation, 0u); ASSERT_EQ(live.gc_shards, kGcShards) << "the pool must be running with gc_shards=2"; EXPECT_EQ(inDegreeInRuns(*backend, runsForShard(*backend, layout, /*shard=*/0), BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(blob_shard0)}), 1) diff --git a/src/Disks/tests/gtest_cas_gc_stop_start.cpp b/src/Disks/tests/gtest_cas_gc_stop_start.cpp index fa4c864cc8de..6079a2b3436d 100644 --- a/src/Disks/tests/gtest_cas_gc_stop_start.cpp +++ b/src/Disks/tests/gtest_cas_gc_stop_start.cpp @@ -63,13 +63,14 @@ const std::string kSrid = "test"; /// gtest_cas_lifecycle_condition.cpp's helper — used by the operator-STOP-persistence test below. void fenceOutMount(DB::Cas::Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, DB::Cas::Retry::once()); ASSERT_TRUE(got.has_value()); DB::Cas::MountLease m = DB::Cas::decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, DB::Cas::encodeMountLease(m), got->token).outcome, - DB::Cas::PutOutcome::Done); + const auto put = (*op).replace(mount_key, DB::Cas::encodeMountLease(m), got->etag, DB::Cas::Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); } /// A real `ContentAddressedMetadataStorage` over a fresh, unique local object storage. `context == nullptr` diff --git a/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp b/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp index 74410c1794c1..994b526af636 100644 --- a/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp +++ b/src/Disks/tests/gtest_cas_gc_undercount_repro.cpp @@ -50,7 +50,8 @@ ManifestRef ref(uint64_t seq, uint64_t inst) bool blobExists(InMemoryBackend & b, const Layout & layout, const UInt128 & hash) { - return b.head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + DB::Cas::tests::OperationForTest op(b); + return (*op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)}), Retry::standard()).has_value(); } /// A committed `RefOwnerBinding` for a raw `owner_transition` op. The raw appender is now diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index cb852d928618..0810370cf343 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -23,22 +23,22 @@ namespace DB::ErrorCodes using namespace DB::Cas; -/// MountLeaseKeeper behavior: the per-server mount lease and the merged build-watermark floor ride the -/// SAME slot, renewed by one beat. The keeper anchors durably before return, adopts a slot already +/// MountLeaseRenewer behavior: the per-server mount lease and the merged build-watermark floor ride the +/// SAME slot, renewed by one beat. The renewer anchors durably before return, adopts a slot already /// written by `claimMount` (same uuid+epoch), re-reads the callback on each renew and bumps `seq`, /// stamps the farewell sentinel (`min_active_build_sequence = UINT64_MAX`, `expires_at_ms <= now`) on `release`, and /// returns typed terminal results on any foreign touch. namespace { -/// The two request planes this file's keepers run on. Both are open-fence -- the exclusivity these +/// The two request planes this file's renewers run on. Both are open-fence -- the exclusivity these /// tests exercise is the mount protocol's own, not a fence's -- on the same injected boot clock the -/// keeper's lease deadline is expressed on, so the two never disagree about how much budget is left. +/// renewer's lease deadline is expressed on, so the two never disagree about how much budget is left. /// `sleep_step_ms`, when set, makes one inter-attempt pause jump the clock past the lease bound: that /// is how a test asks for exactly one physical attempt without a per-call attempt cap. It depends on /// the engine checking the bound, sleeping, then checking again -- a reissue that slept first would /// send a second attempt. `tests::OperationForTest` covers a fixture needing one operation, but -/// neither the two planes a keeper takes nor this clock, which is why this stays local. +/// neither the two planes a renewer takes nor this clock, which is why this stays local. class Ops { public: @@ -71,7 +71,7 @@ void mustCommit(WriteResult && result, const String & what) throw DB::Exception(DB::ErrorCodes::ABORTED, "test fixture write '{}' did not commit", what); } -/// The normal steady-state flow: `claimMount` writes the live (uuid, epoch) mount, THEN the keeper +/// The normal steady-state flow: `claimMount` writes the live (uuid, epoch) mount, THEN the renewer /// adopts it. Seed that claim so `start` adopts instead of self-tripping the double-start guard. void seedOwnClaim(CasOperation & op, const Layout & l, const String & srid, UInt128 uuid, uint64_t epoch, uint64_t now_ms, uint64_t ttl_ms) @@ -181,18 +181,18 @@ DB::Exception terminalException(const MountRenewResult & result) } catch (...) { - ADD_FAILURE() << "terminal keeper failure was not a typed DB::Exception"; + ADD_FAILURE() << "terminal renewer failure was not a typed DB::Exception"; } return DB::Exception(DB::ErrorCodes::ABORTED, "missing terminal exception"); } -void renewKeeperOrThrow(MountLeaseKeeper & keeper) +void renewOrThrow(MountLeaseRenewer & renewer) { - const MountRenewResult result = keeper.renew(MountRenewOperationEnvironment{}); + const MountRenewResult result = renewer.renew(MountRenewOperationEnvironment{}); if (result.outcome == MountRenewOutcome::Terminal) std::rethrow_exception(result.failure); if (result.outcome != MountRenewOutcome::Committed) - throw DB::Exception(DB::ErrorCodes::ABORTED, "keeper renewal was not attempted"); + throw DB::Exception(DB::ErrorCodes::ABORTED, "renewer renewal was not attempted"); } } @@ -208,11 +208,11 @@ TEST(CASHeartbeat, AnchorCarriesFloor) Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); ASSERT_TRUE(ops.op.head(layout.mountKey(srid), Retry::standard()).has_value()); auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); @@ -234,16 +234,16 @@ TEST(CASHeartbeat, RenewRereadsCallbackAndBumpsSeq) Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [&] { return min_active_build_sequence_now; }, {}, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); /// The dynamic field moves; the renewal re-reads it off the callback and bumps seq. now_ms = 1500; min_active_build_sequence_now = 8; - renewKeeperOrThrow(keeper); + renewOrThrow(renewer); auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); EXPECT_EQ(m.min_active_build_sequence, 8u); @@ -262,14 +262,14 @@ TEST(CASHeartbeat, StopStampsExpiredAndFarewellSentinel) Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); now_ms = 2000; - keeper.release(); + renewer.release(); auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); /// Terminal body stamps the lease already-expired (so a same-server reopen reclaims immediately) @@ -293,11 +293,11 @@ TEST(CASHeartbeat, SameEpochUnfencedTouchIsUncertainNotFatal) Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); /// The slot advances past the incarnation we hold, under our own pair (the ambiguous-landed-renewal shape). const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); @@ -307,12 +307,12 @@ TEST(CASHeartbeat, SameEpochUnfencedTouchIsUncertainNotFatal) advanced.writer_epoch = 9; advanced.seq = 99; advanced.write_attempt_id = UInt128{99}; - mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(advanced), observed->incarnation, + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(advanced), observed->etag, Retry::standard()), "advanced slot"); try { - renewKeeperOrThrow(keeper); + renewOrThrow(renewer); FAIL() << "renew must return a terminal conflict"; } catch (const DB::Exception & e) @@ -340,11 +340,11 @@ TEST(CASHeartbeat, SupersededTouchIsFailClosedNotFatal) Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); ASSERT_TRUE(observed.has_value()); @@ -353,12 +353,12 @@ TEST(CASHeartbeat, SupersededTouchIsFailClosedNotFatal) successor.writer_epoch = 10; successor.seq = 1; successor.write_attempt_id = UInt128{1}; - mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(successor), observed->incarnation, + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(successor), observed->etag, Retry::standard()), "successor slot"); try { - renewKeeperOrThrow(keeper); + renewOrThrow(renewer); FAIL() << "renew must return a terminal conflict"; } catch (const DB::Exception & e) @@ -394,11 +394,11 @@ TEST(CASHeartbeat, ForeignUuidTouchFailsClosedWithoutAborting) Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); ASSERT_TRUE(observed.has_value()); @@ -407,7 +407,7 @@ TEST(CASHeartbeat, ForeignUuidTouchFailsClosedWithoutAborting) foreign.writer_epoch = 1; foreign.seq = 1; foreign.write_attempt_id = UInt128{1}; - mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(foreign), observed->incarnation, + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(foreign), observed->etag, Retry::standard()), "foreign slot"); /// Restored on every exit: this flag is process-global and every later test in this binary would @@ -420,7 +420,7 @@ TEST(CASHeartbeat, ForeignUuidTouchFailsClosedWithoutAborting) int code = 0; try { - renewKeeperOrThrow(keeper); + renewOrThrow(renewer); FAIL() << "a foreign holder must fail the renewal closed, not be silently taken over"; } catch (const DB::Exception & e) @@ -467,9 +467,9 @@ TEST(CASMountAudit, ClaimReleaseAndForeignConflictEmitEvents) EXPECT_NE(seen.back().detail.at("holder_uuid"), u128ToHex(UInt128{2})); } -/// The MountLeaseKeeper wiring: `start` adopting an already-claimed slot emits mount_claim, `stop` +/// The MountLeaseRenewer wiring: `start` adopting an already-claimed slot emits mount_claim, `stop` /// (the farewell write) emits mount_release. -TEST(CASMountAudit, KeeperAdoptEmitsClaimAndTerminateEmitsRelease) +TEST(CASMountAudit, RenewerAdoptEmitsClaimAndTerminateEmitsRelease) { auto backend = std::make_shared(); Layout layout("pool"); @@ -482,11 +482,11 @@ TEST(CASMountAudit, KeeperAdoptEmitsClaimAndTerminateEmitsRelease) std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); ASSERT_EQ(seen.size(), 1u); EXPECT_EQ(seen[0].type, CasEventType::MountClaim); @@ -494,18 +494,18 @@ TEST(CASMountAudit, KeeperAdoptEmitsClaimAndTerminateEmitsRelease) seen.clear(); now_ms = 2000; - keeper.release(); + renewer.release(); ASSERT_EQ(seen.size(), 1u); EXPECT_EQ(seen[0].type, CasEventType::MountRelease); EXPECT_EQ(seen[0].detail.at("branch"), "farewell"); } -/// Keeper-level foreign-conflict refusal: the mount slot is already held by a FOREIGN uuid (X) when -/// a keeper for a DIFFERENT uuid (Y) tries to claim it. This must fail closed and — since the +/// Renewer-level foreign-conflict refusal: the mount slot is already held by a FOREIGN uuid (X) when +/// a renewer for a DIFFERENT uuid (Y) tries to claim it. This must fail closed and — since the /// mount-audit sink is not yet installed at first-open — name X in the exception's message text /// (the only identity carrier in err.log at that point). MountConflict payload coverage is above. -TEST(CASMountAudit, KeeperForeignConflictRefusesAndNamesHolder) +TEST(CASMountAudit, RenewerForeignConflictRefusesAndNamesHolder) { auto backend = std::make_shared(); Layout layout("pool"); @@ -520,7 +520,7 @@ TEST(CASMountAudit, KeeperForeignConflictRefusesAndNamesHolder) ASSERT_EQ(claimMount(ops.op, layout, srid, uuid_x, /*our_epoch=*/1, now_ms, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid_y, /*writer_epoch=*/1, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid_y, /*writer_epoch=*/1, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(2000), [&] { return boot_ms; }); @@ -530,15 +530,15 @@ TEST(CASMountAudit, KeeperForeignConflictRefusesAndNamesHolder) DB::Cas::tests::expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, holder_uuid, - [&] { keeper.start(); }); + [&] { renewer.start(); }); } /// `Pool::open` can fail before/inside `doStart` (e.g. a foreign-conflict refusal, see -/// `KeeperForeignConflictRefusesAndNamesHolder` above) — the keeper is destroyed without ever having +/// `RenewerForeignConflictRefusesAndNamesHolder` above) — the renewer is destroyed without ever having /// claimed anything. Teardown must not throw "release before start"; there is nothing to release. A /// stop AFTER a successful start still performs the farewell (covered by /// `StopStampsExpiredAndFarewellSentinel` above); a genuinely-started DOUBLE terminate stays loud. -TEST(CASMountAudit, KeeperAdoptRefusesFencedSelfWithTypedError) +TEST(CASMountAudit, RenewerAdoptRefusesFencedSelfWithTypedError) { auto backend = std::make_shared(); Layout layout("pool"); @@ -555,14 +555,14 @@ TEST(CASMountAudit, KeeperAdoptRefusesFencedSelfWithTypedError) MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(fenced), got->incarnation, + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(fenced), got->etag, Retry::standard()), "fence-out"); } std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; - /// A keeper for the SAME (uuid, epoch) tries to adopt the now-fenced slot. - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + /// A renewer for the SAME (uuid, epoch) tries to adopt the now-fenced slot. + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(2000), [&] { return boot_ms; }); @@ -570,7 +570,7 @@ TEST(CASMountAudit, KeeperAdoptRefusesFencedSelfWithTypedError) bool threw = false; try { - keeper.start(); + renewer.start(); } catch (const MountFencedException & e) { @@ -587,7 +587,7 @@ TEST(CASMountAudit, KeeperAdoptRefusesFencedSelfWithTypedError) /// A renew mismatch is classified by BODY, not blamed on "a foreign writer" by default: the GC can /// fence our OWN (uuid, epoch) mount slot after our lease expires (a late renewal beat racing the -/// GC's fence-out). The keeper must re-read and recognize this as its OWN incarnation being fenced — +/// GC's fence-out). The renewer must re-read and recognize this as its OWN incarnation being fenced — /// a recoverable `MountFencedException`, not the generic single-writer-violation text. TEST(CASHeartbeat, RenewOverFencedOwnSlotIsClassifiedNotForeign) { @@ -602,11 +602,11 @@ TEST(CASHeartbeat, RenewOverFencedOwnSlotIsClassifiedNotForeign) std::vector seen; CasEventSink sink = [&](const CasEvent & e) { seen.push_back(e); }; - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, std::chrono::milliseconds(100), [&] { return now_ms; }, [] { return uint64_t{5}; }, sink, std::chrono::milliseconds(0), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); seen.clear(); /// Mid-run: the GC fences our own (uuid, epoch) mount slot in place (as `computeHeartbeatFloor` @@ -617,14 +617,14 @@ TEST(CASHeartbeat, RenewOverFencedOwnSlotIsClassifiedNotForeign) MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(fenced), got->incarnation, + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(fenced), got->etag, Retry::standard()), "fence-out"); } /// The renewal must classify the fence honestly — not "foreign writer": try { - renewKeeperOrThrow(keeper); + renewOrThrow(renewer); FAIL() << "renew over a fenced slot must be terminal"; } catch (const MountFencedException & e) @@ -639,12 +639,12 @@ TEST(CASHeartbeat, RenewOverFencedOwnSlotIsClassifiedNotForeign) EXPECT_EQ(seen.back().detail.at("holder_uuid"), u128ToHex(uuid)); } -TEST(CASHeartbeat, KeeperStateAllowsOnlyActiveReleaseOrTerminal) +TEST(CASHeartbeat, RenewerStateAllowsOnlyActiveReleaseOrTerminal) { #if defined(DEBUG_OR_SANITIZER_BUILD) -#define EXPECT_KEEPER_STATE_REJECTION(statement) EXPECT_DEATH({ statement; }, "allowed only in") +#define EXPECT_RENEWER_STATE_REJECTION(statement) EXPECT_DEATH({ statement; }, "allowed only in") #else -#define EXPECT_KEEPER_STATE_REJECTION(statement) EXPECT_THROW(statement, DB::Exception) +#define EXPECT_RENEWER_STATE_REJECTION(statement) EXPECT_THROW(statement, DB::Exception) #endif Layout layout("pool"); @@ -656,21 +656,21 @@ TEST(CASHeartbeat, KeeperStateAllowsOnlyActiveReleaseOrTerminal) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "released", uuid, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "released", uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::New); - EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalEnvironment(boot_ms))); - EXPECT_KEEPER_STATE_REJECTION(keeper.release()); - EXPECT_EQ(keeper.start(), 100u); - EXPECT_KEEPER_STATE_REJECTION(keeper.start()); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::Active); - keeper.release(); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::Released); - EXPECT_KEEPER_STATE_REJECTION(keeper.start()); - EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalEnvironment(boot_ms))); - EXPECT_KEEPER_STATE_REJECTION(keeper.release()); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::New); + EXPECT_RENEWER_STATE_REJECTION(renewer.renew(renewalEnvironment(boot_ms))); + EXPECT_RENEWER_STATE_REJECTION(renewer.release()); + EXPECT_EQ(renewer.start(), 100u); + EXPECT_RENEWER_STATE_REJECTION(renewer.start()); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::Active); + renewer.release(); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::Released); + EXPECT_RENEWER_STATE_REJECTION(renewer.start()); + EXPECT_RENEWER_STATE_REJECTION(renewer.renew(renewalEnvironment(boot_ms))); + EXPECT_RENEWER_STATE_REJECTION(renewer.release()); } { @@ -681,22 +681,22 @@ TEST(CASHeartbeat, KeeperStateAllowsOnlyActiveReleaseOrTerminal) /// one this renewal ever sends and its verdict is the terminal one under test. Ops ops(backend, &boot_ms, /*sleep_step_ms=*/10'000); seedOwnClaim(ops.op, layout, "terminal", uuid, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "terminal", uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->actions = {RenewalScriptBackend::Action::ThrowBefore}; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Terminal); EXPECT_NE(result.failure, nullptr); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); - EXPECT_KEEPER_STATE_REJECTION(keeper.start()); - EXPECT_KEEPER_STATE_REJECTION(keeper.renew(renewalEnvironment(boot_ms))); - EXPECT_KEEPER_STATE_REJECTION(keeper.release()); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::RenewalTerminal); + EXPECT_RENEWER_STATE_REJECTION(renewer.start()); + EXPECT_RENEWER_STATE_REJECTION(renewer.renew(renewalEnvironment(boot_ms))); + EXPECT_RENEWER_STATE_REJECTION(renewer.release()); } -#undef EXPECT_KEEPER_STATE_REJECTION +#undef EXPECT_RENEWER_STATE_REJECTION } TEST(CASHeartbeat, RenewalRetriesOneImmutableBodyAndAdoptsLostResponse) @@ -709,15 +709,15 @@ TEST(CASHeartbeat, RenewalRetriesOneImmutableBodyAndAdoptsLostResponse) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, srid, uuid, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, srid, uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->attempts.clear(); backend->actions = {RenewalScriptBackend::Action::ThrowBefore, RenewalScriptBackend::Action::Delegate}; - MountRenewResult retried = keeper.renew(renewalEnvironment(boot_ms)); + MountRenewResult retried = renewer.renew(renewalEnvironment(boot_ms)); ASSERT_EQ(retried.outcome, MountRenewOutcome::Committed); ASSERT_EQ(backend->attempts.size(), 2u); EXPECT_EQ(backend->attempts[0].key, backend->attempts[1].key); @@ -728,7 +728,7 @@ TEST(CASHeartbeat, RenewalRetriesOneImmutableBodyAndAdoptsLostResponse) backend->attempts.clear(); backend->actions = {RenewalScriptBackend::Action::LandThenThrow}; - MountRenewResult adopted = keeper.renew(renewalEnvironment(boot_ms)); + MountRenewResult adopted = renewer.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(adopted.outcome, MountRenewOutcome::Committed); EXPECT_TRUE(adopted.resolved_by_read); EXPECT_EQ(adopted.attempts_sent, 1u); @@ -744,15 +744,15 @@ TEST(CASHeartbeat, DeadlineBeforeSendTerminalizesWithTypedFailure) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 100); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(100), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->attempts.clear(); backend->read_calls = 0; boot_ms = 180; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); const DB::Exception failure = terminalException(result); EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); EXPECT_NE(failure.message().find("no attempt sent"), String::npos) << failure.message(); @@ -772,21 +772,21 @@ TEST(CASHeartbeat, CancellationBeforeSendIsNotAttemptedAndAllowsRelease) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->attempts.clear(); backend->read_calls = 0; - const MountRenewResult result = keeper.renew(renewalEnvironment( + const MountRenewResult result = renewer.renew(renewalEnvironment( boot_ms, /*live=*/[] { return false; }, /*cancelled=*/[] { return true; })); EXPECT_EQ(result.outcome, MountRenewOutcome::NotAttempted); EXPECT_EQ(result.failure, nullptr); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::Active); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::Active); EXPECT_TRUE(backend->attempts.empty()); - EXPECT_NO_THROW(keeper.release()); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::Released); + EXPECT_NO_THROW(renewer.release()); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::Released); } TEST(CASHeartbeat, CancellationAfterSendIsTerminalAndForbidsRelease) @@ -798,24 +798,24 @@ TEST(CASHeartbeat, CancellationAfterSendIsTerminalAndForbidsRelease) bool cancelled = false; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->attempts.clear(); backend->read_calls = 0; backend->cancel_after_write = [&] { cancelled = true; }; backend->actions = {RenewalScriptBackend::Action::ReturnThenCancel}; - const MountRenewResult result = keeper.renew( + const MountRenewResult result = renewer.renew( renewalEnvironment(boot_ms, /*live=*/[&] { return !cancelled; }, /*cancelled=*/[&] { return cancelled; })); const DB::Exception failure = terminalException(result); EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); EXPECT_TRUE(result.sent_any); EXPECT_EQ(backend->read_calls, 0u) << "post-write cancellation must not start a diagnostic read"; - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::RenewalTerminal); const String bytes_before = ops.op.read(layout.mountKey("test"), Retry::standard())->bytes; - EXPECT_FALSE(keeper.canRelease()); + EXPECT_FALSE(renewer.canRelease()); EXPECT_EQ(ops.op.read(layout.mountKey("test"), Retry::standard())->bytes, bytes_before); } @@ -827,18 +827,18 @@ TEST(CASHeartbeat, SlowResolvedSuccessKeepsAttemptStartAnchor) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); boot_ms = 150; backend->cancel_after_write = [&] { boot_ms = 400; }; backend->actions = {RenewalScriptBackend::Action::LandThenThrow}; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Committed); EXPECT_EQ(result.attempt_start_boot_ms, 150u); - EXPECT_EQ(keeper.lastCommittedAttemptStartBootMs(), 150u); + EXPECT_EQ(renewer.lastCommittedAttemptStartBootMs(), 150u); } TEST(CASHeartbeat, SamePairTwinAndForeignOrSuccessorStayTerminal) @@ -852,24 +852,24 @@ TEST(CASHeartbeat, SamePairTwinAndForeignOrSuccessorStayTerminal) const UInt128 uuid{1}; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", uuid, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", uuid, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); auto got = ops.op.read(layout.mountKey("test"), Retry::standard()); MountLease current = decodeMountLease(got->bytes); current.server_uuid = current_uuid; current.writer_epoch = current_epoch; current.write_attempt_id = current_attempt; ++current.seq; - mustCommit(ops.op.replace(layout.mountKey("test"), encodeMountLease(current), got->incarnation, + mustCommit(ops.op.replace(layout.mountKey("test"), encodeMountLease(current), got->etag, Retry::standard()), "competing slot"); backend->read_calls = 0; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); const DB::Exception failure = terminalException(result); EXPECT_NE(failure.code(), DB::ErrorCodes::LOGICAL_ERROR); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::RenewalTerminal); EXPECT_EQ(backend->read_calls, 1u) << "the write's own resolving read must be the only terminal read"; }; @@ -886,17 +886,17 @@ TEST(CASHeartbeat, ExpectedPredecessorThenLateLandingIsAdoptedExactly) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->attempts.clear(); backend->actions = { RenewalScriptBackend::Action::ThrowBeforeThenLandAfterResolve, RenewalScriptBackend::Action::Delegate, }; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Committed); EXPECT_TRUE(result.resolved_by_read); ASSERT_EQ(backend->attempts.size(), 2u); @@ -915,26 +915,26 @@ TEST(CASHeartbeat, GcFenceAndVanishedMountStayTerminal) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); const String key = layout.mountKey("test"); auto got = ops.op.read(key, Retry::standard()); if (vanish) - ASSERT_EQ(ops.op.remove(key, got->incarnation, Retry::standard()), Removal::Removed); + ASSERT_EQ(ops.op.remove(key, got->etag, Retry::standard()), Removal::Removed); else { MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; ++fenced.seq; - mustCommit(ops.op.replace(key, encodeMountLease(fenced), got->incarnation, Retry::standard()), + mustCommit(ops.op.replace(key, encodeMountLease(fenced), got->etag, Retry::standard()), "fence-out"); } - const DB::Exception failure = terminalException(keeper.renew(renewalEnvironment(boot_ms))); + const DB::Exception failure = terminalException(renewer.renew(renewalEnvironment(boot_ms))); EXPECT_NE(failure.code(), DB::ErrorCodes::LOGICAL_ERROR); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::RenewalTerminal); }; run_case(false); run_case(true); @@ -951,21 +951,21 @@ TEST(CASHeartbeat, LateDeliveryAfterTerminalCannotRearmOrOverwriteSuccessor) /// one this renewal sends and the renewal ends terminal with that attempt still in flight. Ops ops(backend, &boot_ms, /*sleep_step_ms=*/10'000); seedOwnClaim(ops.op, layout, "before-reclaim", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "before-reclaim", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, CasEventSink{}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); backend->actions = {RenewalScriptBackend::Action::ThrowBeforeThenLandAfterResolve}; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); EXPECT_EQ(result.outcome, MountRenewOutcome::Terminal); - /// The delayed write landed during the resolving read. It carries this keeper's own epoch, and - /// it does not put the keeper back in business. + /// The delayed write landed during the resolving read. It carries this renewer's own epoch, and + /// it does not put the renewer back in business. const MountLease landed = decodeMountLease( ops.op.read(layout.mountKey("before-reclaim"), Retry::standard())->bytes); EXPECT_EQ(landed.writer_epoch, 9u); - EXPECT_EQ(keeper.state(), MountLeaseKeeperState::RenewalTerminal); + EXPECT_EQ(renewer.state(), MountLeaseRenewerState::RenewalTerminal); } { auto backend = std::make_shared(); @@ -973,19 +973,19 @@ TEST(CASHeartbeat, LateDeliveryAfterTerminalCannotRearmOrOverwriteSuccessor) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms, /*sleep_step_ms=*/10'000); seedOwnClaim(ops.op, layout, "after-successor", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "after-successor", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); /// The incarnation the about-to-be-terminal renewal names as its precondition: a late delivery /// of that attempt can only ever be replayed against exactly this one. - const Incarnation delayed_precondition - = ops.op.read(layout.mountKey("after-successor"), Retry::standard())->incarnation; + const Etag delayed_precondition + = ops.op.read(layout.mountKey("after-successor"), Retry::standard())->etag; backend->actions = {RenewalScriptBackend::Action::ThrowBefore}; - const MountRenewResult result = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); ASSERT_EQ(result.outcome, MountRenewOutcome::Terminal); ASSERT_FALSE(backend->attempts.empty()); const auto delayed = backend->attempts.back(); @@ -995,11 +995,11 @@ TEST(CASHeartbeat, LateDeliveryAfterTerminalCannotRearmOrOverwriteSuccessor) MountLease fenced = decodeMountLease(current->bytes); fenced.gc_fenced = true; ++fenced.seq; - mustCommit(ops.op.replace(delayed.key, encodeMountLease(fenced), current->incarnation, Retry::standard()), + mustCommit(ops.op.replace(delayed.key, encodeMountLease(fenced), current->etag, Retry::standard()), "fence-out"); ASSERT_EQ(claimMount(ops.op, layout, "after-successor", UInt128{1}, 10, wall_ms, 1000).kind, MountClaimResult::Claimed); - MountLeaseKeeper successor( + MountLeaseRenewer successor( ops.mount, ops.farewell, layout, "after-successor", UInt128{1}, 10, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); @@ -1021,20 +1021,20 @@ TEST(CASHeartbeat, WallClockStepsAndBootSuspendCannotExtendAuthority) uint64_t boot_ms = 100; Ops ops(backend, &boot_ms); seedOwnClaim(ops.op, layout, "test", UInt128{1}, 9, wall_ms, 1000); - MountLeaseKeeper keeper( + MountLeaseRenewer renewer( ops.mount, ops.farewell, layout, "test", UInt128{1}, 9, std::chrono::milliseconds(1000), [&] { return wall_ms; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(20), [&] { return boot_ms; }); - keeper.start(); + renewer.start(); wall_ms = 9'000'000; - EXPECT_EQ(keeper.renew(renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); + EXPECT_EQ(renewer.renew(renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); wall_ms = 1; - EXPECT_EQ(keeper.renew(renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); + EXPECT_EQ(renewer.renew(renewalEnvironment(boot_ms)).outcome, MountRenewOutcome::Committed); backend->attempts.clear(); boot_ms += 10'000; - const MountRenewResult suspended = keeper.renew(renewalEnvironment(boot_ms)); + const MountRenewResult suspended = renewer.renew(renewalEnvironment(boot_ms)); const DB::Exception failure = terminalException(suspended); EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); EXPECT_TRUE(backend->attempts.empty()) << "suspend-sized BOOTTIME overshoot must close admission"; diff --git a/src/Disks/tests/gtest_cas_holey_list_detector.cpp b/src/Disks/tests/gtest_cas_holey_list_detector.cpp index 8841836ebffc..05da38ca5b61 100644 --- a/src/Disks/tests/gtest_cas_holey_list_detector.cpp +++ b/src/Disks/tests/gtest_cas_holey_list_detector.cpp @@ -137,8 +137,9 @@ ManifestId publishOneBlobPart(const PoolPtr & s, const RootNamespace & ns, const bool blobPresent(const std::shared_ptr & b, const Layout & layout, const String & payload) { - return b->head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, - BlobDigest::fromU128(u128Of(payload))})).exists; + DB::Cas::tests::OperationForTest op(*b); + return (*op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, + BlobDigest::fromU128(u128Of(payload))}), Retry::standard()).has_value(); } /// Every ref object key of one namespace. Used to identify WHICH objects a publish appended, rather @@ -153,7 +154,8 @@ std::set listRefKeys(Backend & b, const Layout & layout, const RootNames CasOperation op = requests.admit(); const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(op, layout, ns).value(); std::set keys; - forEachListedKey(b, layout.namespaceStreamPrefix(life), [&](const ListedKey & k) { keys.insert(k.key); }); + op.forEachListedKey(layout.namespaceStreamPrefix(life), + [&](const ListedKey & k) { keys.insert(k.key); return true; }, Retry::standard()); return keys; } @@ -178,13 +180,14 @@ String refLogKeyEmittingEdge(Backend & b, const Layout & layout, const RootNames const std::vector & candidates, const ManifestId & manifest_id, int change) { + DB::Cas::tests::OperationForTest probe(b); std::vector hits; for (const String & key : candidates) { const auto parsed = layout.parseRefObjectKey(key); if (!parsed || parsed->kind != RefObjectKind::Log) continue; - const auto got = b.get(key); + const auto got = (*probe).read(key, Retry::standard()); if (!got) continue; const RefLogTxn txn = @@ -279,8 +282,11 @@ TEST(CASHoleyListDetector, OmittedActivationNeverPermitsDeletingALiveBlob) Gc gc(s, hexToU128("00000000000000000000000000000001")); runRounds(s, gc, 2); ASSERT_TRUE(blobPresent(b, layout, payload)); - ASSERT_TRUE(b->head(layout.manifestKey(m1)).exists) - << "M1's body must still be present so its `-1` edges are readable at removal-fold"; + { + DB::Cas::tests::OperationForTest m1_probe(*b); + ASSERT_TRUE((*m1_probe).head(layout.manifestKey(m1), Retry::standard()).has_value()) + << "M1's body must still be present so its `-1` edges are readable at removal-fold"; + } /// M2 adopts the SAME deduplicated blob (`putBlob` of an identical payload dedups). Learn WHICH /// ref-log object carries M2's ACTIVATION by diffing the namespace's ref prefix around the publish diff --git a/src/Disks/tests/gtest_cas_ids.cpp b/src/Disks/tests/gtest_cas_ids.cpp index 046696124bd3..578b3c99b00c 100644 --- a/src/Disks/tests/gtest_cas_ids.cpp +++ b/src/Disks/tests/gtest_cas_ids.cpp @@ -29,13 +29,8 @@ TEST(CASIds, HexU128RoundTrip) EXPECT_THROW(hexToU128("0123"), DB::Exception); // wrong length } -TEST(CASToken, Basics) -{ - Token a{"etag-1", TokenType::ETag}; - Token b{"etag-1", TokenType::ETag}; - Token c{"etag-2", TokenType::ETag}; - EXPECT_EQ(a, b); - EXPECT_NE(a, c); - EXPECT_TRUE(Token{}.empty()); - EXPECT_FALSE(a.empty()); -} +/// `Token`'s free-standing equality/emptiness was deleted with the type itself: `Etag` has no public +/// constructor (minted only by `CasRequests::mint`/`tryMint`) and no `empty()`, so this test's subject +/// no longer exists to construct by hand. `Etag` equality and inequality are exercised by +/// `CASInMemory.PutIfAbsentAndGet` and `CASInMemory.OverwriteIsTokenExactAndMintsFreshToken` in +/// gtest_cas_backend.cpp, which compare an observed incarnation against the one a prior write returned. diff --git a/src/Disks/tests/gtest_cas_inspect.cpp b/src/Disks/tests/gtest_cas_inspect.cpp index 7d0ce0067c65..e031e59f6ae2 100644 --- a/src/Disks/tests/gtest_cas_inspect.cpp +++ b/src/Disks/tests/gtest_cas_inspect.cpp @@ -147,13 +147,13 @@ TEST(CASInspect, RendersTokenTypeWireWordsEtagAndGeneration) etag_rec.ref = bh(1); etag_rec.source_id = UInt128{0}; etag_rec.marker = RunMarker::Condemned; - etag_rec.token = PersistedIncarnation{"etag", "v-etag"}; + etag_rec.token = PersistedEtag{"etag", "v-etag"}; SourceEdgeRecord gen_rec; gen_rec.ref = bh(1); gen_rec.source_id = UInt128{1}; gen_rec.marker = RunMarker::Condemned; - gen_rec.token = PersistedIncarnation{"generation", "v-gen"}; + gen_rec.token = PersistedEtag{"generation", "v-gen"}; DB::WriteBufferFromOwnString out; SourceEdgeRunWriter writer(out); @@ -209,7 +209,7 @@ TEST(CASInspect, RendersBlobTargetRunEdgeAndCondemnedRows) condemned_rec.source_id = UInt128{0}; condemned_rec.marker = RunMarker::Condemned; condemned_rec.delete_pending = true; - condemned_rec.token = PersistedIncarnation{"emulated", "etag-1"}; + condemned_rec.token = PersistedEtag{"emulated", "etag-1"}; condemned_rec.size = 123; condemned_rec.condemn_round = 7; condemned_rec.marker_confirmed = true; diff --git a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp index db8ab08711bb..f4d4e65cf91d 100644 --- a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp +++ b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp @@ -36,11 +36,12 @@ const String kSrid = "test"; /// so a test can restore it verbatim later (scenario d). String deleteKeyReturningBody(Backend & backend, const String & key) { - const auto got = backend.get(key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(key, Retry::once()); EXPECT_TRUE(got.has_value()) << "expected '" << key << "' to exist before deletion"; if (!got) return {}; - backend.deleteExact(key, got->token); + (*op).remove(key, got->etag, Retry::once()); return got->bytes; } @@ -49,12 +50,14 @@ String deleteKeyReturningBody(Backend & backend, const String & key) /// fresh incarnation and returns true. Mirrors gtest_cas_pool.cpp's `fenceOutMount`. void fenceOutMount(Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, Retry::once()); ASSERT_TRUE(got.has_value()); MountLease m = decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, encodeMountLease(m), got->token).outcome, PutOutcome::Done); + const auto put = (*op).replace(mount_key, encodeMountLease(m), got->etag, Retry::once()); + ASSERT_TRUE(std::holds_alternative(put)); } /// A Backend decorator whose reads, heads and lists throw an untyped transport error while `fail` is @@ -166,11 +169,12 @@ TEST(CASLifecycleCondition, PoolMetaForeignPoolIdEntersVanishedReplacedImmediate /// Overwrite `_pool_meta` with a FOREIGN pool_id (identity replaced); the object stays present. const String meta_key = store->layout().poolMetaKey(); - const auto got = backend->get(meta_key); + DB::Cas::tests::OperationForTest op(*backend); + const auto got = (*op).read(meta_key, Retry::once()); ASSERT_TRUE(got.has_value()); PoolMeta foreign = decodePoolMeta(got->bytes); foreign.pool_id = foreign.pool_id + DB::UInt128(1); - ASSERT_EQ(backend->putOverwrite(meta_key, encodePoolMeta(foreign), got->token).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative((*op).replace(meta_key, encodePoolMeta(foreign), got->etag, Retry::once()))); EXPECT_FALSE(store->tryRemountOnce()); EXPECT_EQ(store->lifecycle(), PoolLifecycle::VanishedReplaced); @@ -187,7 +191,8 @@ TEST(CASLifecycleCondition, PoolMetaAlgosUsedDifferIsNotReplacementRecoveryProce auto store = DB::Cas::tests::openPoolForTest(backend); const String meta_key = store->layout().poolMetaKey(); - const auto got = backend->get(meta_key); + DB::Cas::tests::OperationForTest op(*backend); + const auto got = (*op).read(meta_key, Retry::once()); ASSERT_TRUE(got.has_value()); PoolMeta mutated = decodePoolMeta(got->bytes); /// pool_id + blob_header_len UNCHANGED; only `algos_used` gains a member (a mutable field, [B6]). @@ -195,7 +200,7 @@ TEST(CASLifecycleCondition, PoolMetaAlgosUsedDifferIsNotReplacementRecoveryProce ASSERT_FALSE(std::binary_search(mutated.algos_used.begin(), mutated.algos_used.end(), extra)); mutated.algos_used.push_back(extra); std::sort(mutated.algos_used.begin(), mutated.algos_used.end()); - ASSERT_EQ(backend->putOverwrite(meta_key, encodePoolMeta(mutated), got->token).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative((*op).replace(meta_key, encodePoolMeta(mutated), got->etag, Retry::once()))); /// Fence out the mount so the (correctly non-replacement) recovery cleanly reclaims a fresh incarnation. fenceOutMount(*backend, store->layout().mountKey(kSrid)); @@ -224,8 +229,9 @@ TEST(CASLifecycleCondition, IdentityLostDoesNotAutoReviveWhenSentinelsRestored) ASSERT_EQ(store->lifecycle(), PoolLifecycle::IdentityLost); /// Restore both sentinels verbatim (a backup restore with matching identity). - ASSERT_EQ(backend->putIfAbsent(meta_key, meta_body).outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(owner_key, owner_body).outcome, PutOutcome::Done); + DB::Cas::tests::OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).create(meta_key, meta_body, Retry::once()))); + ASSERT_TRUE(std::holds_alternative((*op).create(owner_key, owner_body, Retry::once()))); /// The gate now sees Present+match, but the state is `IdentityLost`, so it stays fail-loud. EXPECT_FALSE(store->tryRemountOnce()); diff --git a/src/Disks/tests/gtest_cas_lifecycle_snapshot.cpp b/src/Disks/tests/gtest_cas_lifecycle_snapshot.cpp index 154c365a2a6d..ea504c894641 100644 --- a/src/Disks/tests/gtest_cas_lifecycle_snapshot.cpp +++ b/src/Disks/tests/gtest_cas_lifecycle_snapshot.cpp @@ -66,10 +66,11 @@ void commitOnePart(ContentAddressedMetadataStorage & storage) /// into a NATURAL `IdentityLost`. Mirrors gtest_cas_forget.cpp / gtest_cas_lifecycle_condition.cpp. void deleteKeyExact(DB::Cas::Backend & backend, const String & key) { - const auto got = backend.get(key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(key, DB::Cas::Retry::standard()); ASSERT_TRUE(got.has_value()) << "expected '" << key << "' to exist before deletion"; if (got) - backend.deleteExact(key, got->token); + (*op).remove(key, got->etag, DB::Cas::Retry::standard()); } } diff --git a/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp b/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp index 1b48718663cb..d40d30fd334d 100644 --- a/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp +++ b/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp @@ -34,8 +34,8 @@ /// anyway -- while a genuinely absent expected id is a durable HOLD, never a silent skip. /// /// This file is that claim stated end to end, against a store that lies exactly the way the real one -/// did. `setListOmissions` names the omitted keys; `get`/`head`/`putIfAbsent`/`casPut`/`deleteExact` -/// keep serving them honestly. Each test below asserts the lie changed NOTHING -- not the folded +/// did. `setListOmissions` names the omitted keys; every other primitive keeps serving them honestly. +/// Each test below asserts the lie changed NOTHING -- not the folded /// edges, not the cursor, not the recovered table, not fsck's verdict -- and the arms that are about /// reclamation additionally assert that reclamation still happens, so "nothing was deleted" can never /// pass for "the lie was harmless". @@ -64,6 +64,12 @@ String blobKeyOf(const Layout & layout, const DB::UInt128 & hash) return layout.blobKey(legacyMetaTestRef(hash)); } +bool headExists(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + /// The sealed fold cursor for `ns` as a full `RefTxnId`. Every fixture here writes ids inside writer /// epoch 1, which is the assumption `foldCursorOf` (returning the sequence alone) already makes. RefTxnId sealedCursorOf(Backend & backend, const Layout & layout, const RootNamespace & ns) @@ -311,7 +317,7 @@ TEST(CASListLiarEndToEnd, AHiddenPlusOneKeepsItsBlobWhenAVisibleMinusOneLandsLat runRegularRoundReclaiming(gc); store->renewWatermarkOnce(); } - EXPECT_TRUE(backend->head(blobKeyOf(layout, shared)).exists) + EXPECT_TRUE(headExists(*backend, blobKeyOf(layout, shared))) << "a blob a live ref still names was DELETED -- the hidden `+1` was never folded"; EXPECT_EQ(backend->deleteCount(blobKeyOf(layout, shared)), 0u) << "not merely still present: the delete was never even attempted"; @@ -362,13 +368,13 @@ TEST(CASListLiarEndToEnd, AHiddenMinusOneIsStillFoldedSoTheBlobIsActuallyReclaim EXPECT_EQ(sealedCursorOf(*backend, layout, ns), (RefTxnId{1, 3})); EXPECT_EQ(inDegreeOf(*backend, layout, released), 0) << "the hidden `-1` must be folded: nothing owns this blob any more"; - EXPECT_TRUE(backend->head(blobKeyOf(layout, released)).exists) + EXPECT_TRUE(headExists(*backend, blobKeyOf(layout, released))) << "round pacing: the round that CONDEMNS never also deletes"; store->renewWatermarkOnce(); EXPECT_TRUE(runRoundsUntilAbsent(store, gc, *backend, layout, released)) << "the blob was never reclaimed -- the hidden `-1` left it pinned by a phantom owner"; - EXPECT_TRUE(backend->head(blobKeyOf(layout, unrelated)).exists) + EXPECT_TRUE(headExists(*backend, blobKeyOf(layout, unrelated))) << "and the still-owned blob is untouched"; } @@ -460,7 +466,7 @@ TEST(CASListLiarEndToEnd, AHiddenNamespacesBirthIsFoundByExactKeyAndSavesTheBlob ASSERT_TRUE(evidence.saw_fold) << "no round folded, so none published a gate verdict"; ASSERT_GT(backend->holesServed(), 0u) << "the omission was never actually served -- the test would pass vacuously"; - EXPECT_TRUE(backend->head(blobKeyOf(layout, blob)).exists) + EXPECT_TRUE(headExists(*backend, blobKeyOf(layout, blob))) << "the blob a hidden namespace still owns must survive"; EXPECT_EQ(backend->deleteCount(blobKeyOf(layout, blob)), 0u) << "not merely still present: the blob must never even be offered for deletion"; @@ -528,7 +534,7 @@ TEST(CASListLiarEndToEnd, TheSameBlobDrainsOnceHiddenGenuinelyProvesItsOwnFronti ASSERT_GT(backend->holesServed(), 0u) << "the omission was never actually served -- the test would pass vacuously"; - EXPECT_FALSE(backend->head(blobKeyOf(layout, blob)).exists) + EXPECT_FALSE(headExists(*backend, blobKeyOf(layout, blob))) << "both namespaces genuinely proved their frontier and the blob is genuinely unreferenced -- " "the round must still be able to reclaim it"; } diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 077312363d71..1989c8cd5937 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -48,15 +48,15 @@ RefCatalog catalogOwning(const String & ns, NsState state) return RefCatalog{.entries = {std::move(entry)}}; } -void renewKeeperOrThrow(MountLeaseKeeper & keeper) +void renewOrThrow(MountLeaseRenewer & renewer) { - const MountRenewResult result = keeper.renew(MountRenewOperationEnvironment{}); + const MountRenewResult result = renewer.renew(MountRenewOperationEnvironment{}); if (result.outcome == MountRenewOutcome::Terminal) std::rethrow_exception(result.failure); ASSERT_EQ(result.outcome, MountRenewOutcome::Committed); } -/// The two request planes a keeper in this file runs on, plus one operation for the protocol calls +/// The two request planes a renewer in this file runs on, plus one operation for the protocol calls /// driven directly. Both planes are open-fence: these fixtures hold no mount lease, so nothing here /// should be refused by a fence it does not have. The clock and the sleep are ALWAYS injected -- a /// fixture that drives a lease deadline passes its own so a slow machine cannot run the bound out @@ -93,12 +93,12 @@ class Ops }; /// The incarnation currently at `key`, for a fixture that has to name it as a precondition. -Incarnation currentIncarnation(CasOperation & op, const String & key) +Etag currentEtag(CasOperation & op, const String & key) { const auto got = op.read(key, Retry::standard()); if (!got) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "test fixture read of '{}' found nothing", key); - return got->incarnation; + return got->etag; } /// A fixture write that must land, so a mis-seeded fixture fails where it is written rather than in @@ -252,7 +252,7 @@ class ScopedRenewalLogCapture { public: explicit ScopedRenewalLogCapture(const String & level) - : logger(getLogger("CasMountLeaseKeeper")) + : logger(getLogger("CasMountLeaseRenewer")) , channel(new Poco::StreamChannel(stream)) , old_channel(logger->getChannel(), /*shared=*/true) , old_level(logger->getLevel()) @@ -286,15 +286,11 @@ size_t countRenewalLogText(const String & haystack, std::string_view needle) return count; } -CasRequestBudget renewalLogBudget(uint32_t max_attempts = 2) +CasRequestBudget renewalLogBudget() { return CasRequestBudget{ .attempt_timeout_ms = 10, - .operation_deadline_ms = 500, - .max_attempts = max_attempts, .lease_safety_margin_ms = 20, - .retry_initial_backoff_ms = 0, - .retry_max_backoff_ms = 0, }; } @@ -705,12 +701,12 @@ TEST(CASMountLease, AbsentClaimThenRenewBumpsSeq) Ops ops(b, &boot); auto r = claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100); EXPECT_EQ(r.kind, MountClaimResult::Claimed); - MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), + MountLeaseRenewer k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); k.start(); EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).seq, 1u); - renewKeeperOrThrow(k); + renewOrThrow(k); EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).seq, 2u); } @@ -725,11 +721,11 @@ TEST(CASMountLease, HolderBodiesMintFreshAttemptIdsAndFenceCopiesIt) const String key = layout.mountKey("r"); const MountLease claimed = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); - MountLeaseKeeper keeper(ops.mount, ops.farewell, layout, "r", UInt128{1}, 7, std::chrono::milliseconds(100), + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, "r", UInt128{1}, 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); - keeper.start(); - renewKeeperOrThrow(keeper); + renewer.start(); + renewOrThrow(renewer); const MountLease renewed = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); EXPECT_NE(claimed.write_attempt_id, UInt128{}); EXPECT_NE(renewed.write_attempt_id, UInt128{}); @@ -740,7 +736,7 @@ TEST(CASMountLease, HolderBodiesMintFreshAttemptIdsAndFenceCopiesIt) MountLease fenced = decodeMountLease(observed->bytes); fenced.gc_fenced = true; ++fenced.seq; - mustCommit(ops.op.replace(key, encodeMountLease(fenced), observed->incarnation, Retry::standard()), "fence-out"); + mustCommit(ops.op.replace(key, encodeMountLease(fenced), observed->etag, Retry::standard()), "fence-out"); EXPECT_EQ(decodeMountLease(ops.op.read(key, Retry::standard())->bytes).write_attempt_id, renewed.write_attempt_id); } @@ -758,7 +754,7 @@ TEST(CASMountLease, ReclaimAndSuccessorBodiesMintNewAttemptIds) MountLease fenced = decodeMountLease(observed->bytes); fenced.gc_fenced = true; ++fenced.seq; - mustCommit(ops.op.replace(key, encodeMountLease(fenced), observed->incarnation, Retry::standard()), "fence-out"); + mustCommit(ops.op.replace(key, encodeMountLease(fenced), observed->etag, Retry::standard()), "fence-out"); const MountLease fence = decodeMountLease(ops.op.read(key, Retry::standard())->bytes); EXPECT_EQ(fence.write_attempt_id, first.write_attempt_id); @@ -769,7 +765,7 @@ TEST(CASMountLease, ReclaimAndSuccessorBodiesMintNewAttemptIds) } /// STID 3982-3b48: `rm -rf` of the pool dir under a live mount deletes the mount slot object out from -/// under a running keeper. The next synchronous renewal must return terminal WITHOUT constructing a +/// under a running renewer. The next synchronous renewal must return terminal WITHOUT constructing a /// `LOGICAL_ERROR` -- that aborts debug/ASan builds at /// exception construction, and there is no foreign writer here to fail closed against, only an /// environmental condition. @@ -781,7 +777,7 @@ TEST(CASMountLease, VanishedBackingStoreStopsRenewalWithoutLogicalError) uint64_t boot = 0; Ops ops(b, &boot); ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); - MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), + MountLeaseRenewer k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); k.start(); @@ -789,13 +785,13 @@ TEST(CASMountLease, VanishedBackingStoreStopsRenewalWithoutLogicalError) const String mount_key = l.mountKey("r"); const auto lost_before = ProfileEvents::global_counters[ProfileEvents::CASMountLeaseLost].load(); /// NOLINT(clang-analyzer-deadcode.DeadStores) - /// Simulate `rm -rf` of the backing store: the mount slot object is gone, but the keeper still + /// Simulate `rm -rf` of the backing store: the mount slot object is gone, but the renewer still /// names a (now stale) incarnation as its precondition. ASSERT_EQ(ops.op.removeCurrent(mount_key, Retry::standard()), Removal::Removed); try { - renewKeeperOrThrow(k); + renewOrThrow(k); FAIL() << "renew against a vanished mount object must throw"; } catch (const DB::Exception & e) @@ -804,7 +800,7 @@ TEST(CASMountLease, VanishedBackingStoreStopsRenewalWithoutLogicalError) EXPECT_NE(e.code(), DB::ErrorCodes::LOGICAL_ERROR); } EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountLeaseLost].load(), lost_before) - << "keeper classification is metric-free; the runtime records operational loss"; + << "renewer classification is metric-free; the runtime records operational loss"; } /// STID 3982-3b48 (part 1b): the terminal/clean-release counterpart to the renewal fix above. When @@ -824,7 +820,7 @@ TEST(CASMountLease, TerminateAfterVanishedBackingStoreIsNoOpRelease) uint64_t now = 1000; Ops ops(b); ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); - MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), + MountLeaseRenewer k(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }); k.start(); @@ -970,7 +966,7 @@ namespace /// fix-round F5 harness: makes the mount key vanish to EVERY read, unconditionally, while the real /// underlying object stays put -- forcing `claimMount`'s own read to take the absent-slot race /// branch every call (its create then conflicts against the real, still-present object, returning -/// `LiveDoubleStart` with no incarnation -- that branch deliberately leaves `.incarnation` unset, +/// `LiveDoubleStart` with no incarnation -- that branch deliberately leaves `.etag` unset, /// since no re-read was done). That in turn forces `claimMountAwaitingExpiry`'s fallback re-read, /// which ALSO sees the slot as vanished -- deterministically reproducing "the slot vanished between /// claimMount's own read and ours" on EVERY loop iteration, not just a lucky one-shot race. @@ -980,7 +976,6 @@ class AlwaysVanishesBackend final : public DB::Cas::Backend explicit AlwaysVanishesBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} String watched_key; - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } bool supportsListTokens() const override { return inner->supportsListTokens(); } /// The fault is on the read primitive, which is the only way anything now reaches the store. @@ -1086,15 +1081,15 @@ TEST(CASMountAwaitExpiry, SkewedFarFutureExpiryHasNoEffectOnObservationThreshold EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 8u); // reclaimed } -TEST(CASMountLease, KeeperStartAdoptsOurOwnClaimNotDoubleStart) +TEST(CASMountLease, RenewerStartAdoptsOurOwnClaimNotDoubleStart) { auto b = std::make_shared(); Layout l("p"); uint64_t now = 1000; Ops ops(b); - // The normal flow: claimMount writes the live mount under (uuid=1, epoch=7), THEN keeper.start(). + // The normal flow: claimMount writes the live mount under (uuid=1, epoch=7), THEN renewer.start(). ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); - MountLeaseKeeper k(ops.mount, ops.farewell, l, "r", UInt128(1), /*epoch*/ 7, std::chrono::milliseconds(100), + MountLeaseRenewer k(ops.mount, ops.farewell, l, "r", UInt128(1), /*epoch*/ 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }); EXPECT_NO_THROW(k.start()); // adopts our own live (uuid=1,epoch=7) mount — NOT a double-start EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 7u); @@ -1138,7 +1133,7 @@ TEST(CASMountStartup, WriterEpochStrictlyIncreasesAcrossReopen) .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r"}); const uint64_t e1 = s1->writerEpoch(); - /// Simulate shutdown: the Pool dtor stops the keeper, whose terminate() retires the lease + /// Simulate shutdown: the Pool dtor stops the renewer, whose terminate() retires the lease /// (stamps it already-expired). The owner + the durable epoch object stay sticky. s1.reset(); @@ -1201,11 +1196,11 @@ TEST(CASMountStartup, ExistingPoolWithoutCatalogFailsBeforeSlotMutation) ASSERT_TRUE(epoch_after.has_value()); ASSERT_TRUE(mount_after.has_value()); EXPECT_EQ(owner_after->bytes, owner_before->bytes); - EXPECT_EQ(owner_after->incarnation, owner_before->incarnation); + EXPECT_EQ(owner_after->etag, owner_before->etag); EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); - EXPECT_EQ(epoch_after->incarnation, epoch_before->incarnation); + EXPECT_EQ(epoch_after->etag, epoch_before->etag); EXPECT_EQ(mount_after->bytes, mount_before->bytes); - EXPECT_EQ(mount_after->incarnation, mount_before->incarnation); + EXPECT_EQ(mount_after->etag, mount_before->etag); } TEST(CASMountReadOnly, ForeignOwnedPoolOpensWithoutMutation) @@ -1264,7 +1259,7 @@ TEST(CASMountStartup, RefusesWritableOpenWithInconsistentCasRequestBudget) /// attempt_timeout_ms + lease_safety_margin_ms == mount_lease_ttl_ms below (30000): not STRICTLY /// less, so this must be rejected. const CasRequestBudget bad_budget{ - .attempt_timeout_ms = 25000, .operation_deadline_ms = 30000, .max_attempts = 3, .lease_safety_margin_ms = 5000}; + .attempt_timeout_ms = 25000, .lease_safety_margin_ms = 5000}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] { Pool::open(b, PoolConfig{ @@ -1286,7 +1281,7 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) /// lease TTL), so it also scales down cas_request_budget to fit — the budget itself is not /// exercised here, only Pool::open's validateCasRequestBudget startup gate. const CasRequestBudget tiny_budget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = 500, .max_attempts = 1, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; auto a = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r", .mount_lease_ttl_ms = std::chrono::milliseconds(300), @@ -1305,7 +1300,7 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) a.reset(); const auto farewell = ops.op.read(mount_key, Retry::standard()); ASSERT_TRUE(farewell.has_value()); - mustCommit(ops.op.replace(mount_key, stale_mount->bytes, farewell->incarnation, Retry::standard()), + mustCommit(ops.op.replace(mount_key, stale_mount->bytes, farewell->etag, Retry::standard()), "replayed stale lease"); /// A restart of the SAME server (same uuid) must NOT abort: it waits out the stale lease (<= ~300ms) @@ -1408,7 +1403,7 @@ constexpr uint64_t kNowMs = 1'000'000; /// of any lease's stamped `expires_at_ms`. constexpr uint64_t kStableThresholdMs = 10'000; -/// Seed one mount body under mountKey(srid) via the on-storage codec — the same interface the keeper +/// Seed one mount body under mountKey(srid) via the on-storage codec — the same interface the renewer /// writes through. MountLease seedMount( CasOperation & op, const Layout & l, const String & srid, @@ -1429,7 +1424,7 @@ MountLease seedMount( return m; } -/// Simulate a keeper's real renewal between two `computeHeartbeatFloor` calls: a guarded write that +/// Simulate a renewer's real renewal between two `computeHeartbeatFloor` calls: a guarded write that /// bumps `seq` (and so mints a fresh incarnation), leaving everything else as-is. Models the one /// thing the observation-based fence cares about: the incarnation changed, so any in-progress /// observation of the OLD one must restart. @@ -1439,7 +1434,7 @@ void renewMount(CasOperation & op, const Layout & l, const String & srid) ASSERT_TRUE(got.has_value()); MountLease m = decodeMountLease(got->bytes); m.seq += 1; - mustCommit(op.replace(l.mountKey(srid), encodeMountLease(m), got->incarnation, Retry::standard()), + mustCommit(op.replace(l.mountKey(srid), encodeMountLease(m), got->etag, Retry::standard()), "renewed mount " + srid); } } @@ -1499,18 +1494,18 @@ TEST(CASHeartbeatFloor, RenewalBetweenRoundsRestartsObservation) MountObservationMap obs; computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ 0, kStableThresholdMs, obs); ASSERT_TRUE(obs.contains("s1")); - const Incarnation first_incarnation = obs.at("s1").incarnation; + const Etag first_etag = obs.at("s1").etag; renewMount(ops.op, l, "s1"); - const Incarnation renewed_incarnation = currentIncarnation(ops.op, l.mountKey("s1")); - EXPECT_NE(renewed_incarnation, first_incarnation); + const Etag renewed_etag = currentEtag(ops.op, l.mountKey("s1")); + EXPECT_NE(renewed_etag, first_etag); const HeartbeatFloor floor2 = computeHeartbeatFloor(ops.op, l, kNowMs, /*mono*/ kStableThresholdMs, kStableThresholdMs, obs); EXPECT_EQ(floor2.fenced_now, 0u); ASSERT_TRUE(obs.contains("s1")); - EXPECT_EQ(obs.at("s1").incarnation, renewed_incarnation); + EXPECT_EQ(obs.at("s1").etag, renewed_etag); EXPECT_EQ(obs.at("s1").first_seen_mono_ms, kStableThresholdMs); } @@ -1533,7 +1528,7 @@ TEST(CASHeartbeatFloor, UnseenSridPrunedFromObservationMap) ASSERT_TRUE(obs.contains("s2")); /// s2's `/mount` key is removed entirely -- e.g. `SYSTEM CAS DROP POOL MEMBER` -- so - /// no future LIST pass will ever visit it again. s1 renews (a live keeper would), so its OWN + /// no future LIST pass will ever visit it again. s1 renews (a live renewer would), so its OWN /// observation restarts and it stays `live` -- isolating this test to the pruning behavior alone, /// not confounding it with s1 also becoming fence-eligible (which would erase its `obs` entry too, /// for an unrelated reason). @@ -1573,7 +1568,7 @@ TEST(CASHeartbeatFloor, ClassifiesAndFencesOut) EXPECT_EQ(floor_before.fenced_now, 0u); EXPECT_EQ(floor_before.already_fenced, 1u); // s4 - /// s1 and s2 renew between rounds (as a live keeper would); s3 does not (it crashed). + /// s1 and s2 renew between rounds (as a live renewer would); s3 does not (it crashed). renewMount(ops.op, l, "s1"); renewMount(ops.op, l, "s2"); @@ -1771,7 +1766,7 @@ TEST(CASClaimMount, SameEpochFencedIsNotRefreshable) MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - mustCommit(ops.op.replace(layout.mountKey("a"), encodeMountLease(fenced), got->incarnation, Retry::standard()), + mustCommit(ops.op.replace(layout.mountKey("a"), encodeMountLease(fenced), got->etag, Retry::standard()), "fence-out"); } /// Same (uuid, epoch) re-claim must NOT refresh a fenced body — a fence costs an epoch: @@ -1833,18 +1828,18 @@ TEST(CASMountObservation, RenewalDuringObservationRestartsIt) { auto b = std::make_shared(); Layout l{"p"}; - uint64_t keeper_boot = 0; - Ops ops(b, &keeper_boot); + uint64_t renewer_boot = 0; + Ops ops(b, &renewer_boot); ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, 500, 500).kind, MountClaimResult::Claimed); - /// The real (still-alive) holder's keeper for epoch 7: `start()` adopts the slot `claimMount` just + /// The real (still-alive) holder's renewer for epoch 7: `start()` adopts the slot `claimMount` just /// wrote (no seq bump, per the ADOPT RULE), then a synchronous renewal mints a new incarnation /// mid-observation. - uint64_t keeper_wall = 500; - MountLeaseKeeper keeper(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(500), - [&] { return keeper_wall; }, [] { return uint64_t{0}; }, {}, - std::chrono::milliseconds(0), [&] { return keeper_boot; }); - keeper.start(); + uint64_t renewer_wall = 500; + MountLeaseRenewer renewer(ops.mount, ops.farewell, l, "r", UInt128(1), 7, std::chrono::milliseconds(500), + [&] { return renewer_wall; }, [] { return uint64_t{0}; }, {}, + std::chrono::milliseconds(0), [&] { return renewer_boot; }); + renewer.start(); const uint64_t threshold_ms = 500 + 500 / 20 + 50; /// = 575 uint64_t mono = 0; @@ -1862,7 +1857,7 @@ TEST(CASMountObservation, RenewalDuringObservationRestartsIt) if (!renewed && mono >= threshold_ms - 50) { renewed = true; - renewKeeperOrThrow(keeper); + renewOrThrow(renewer); } }, /*on_wait_start=*/[&](const MountLease &, uint64_t) { ++wait_starts; }); @@ -1893,7 +1888,7 @@ TEST(CASMountObservation, GcFencedIsReclaimedInstantlyWithPriorFenced) MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; fenced.seq += 1; - mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(fenced), got->incarnation, Retry::standard()), + mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(fenced), got->etag, Retry::standard()), "fence-out"); } @@ -1940,7 +1935,7 @@ TEST(CASFenceTerminal, GcFencedIsTerminal) ASSERT_TRUE(got.has_value()); MountLease fenced = decodeMountLease(got->bytes); fenced.gc_fenced = true; - mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(fenced), got->incarnation, Retry::standard()), + mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(fenced), got->etag, Retry::standard()), "fence-out"); EXPECT_TRUE(isCreatorFenceTerminal(ops.op, l, "r", 7)); @@ -1955,7 +1950,7 @@ TEST(CASFenceTerminal, CleanFarewellIsTerminal) ASSERT_TRUE(got.has_value()); MountLease retired = decodeMountLease(got->bytes); retired.min_active_build_sequence = std::numeric_limits::max(); - mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(retired), got->incarnation, Retry::standard()), + mustCommit(ops.op.replace(l.mountKey("r"), encodeMountLease(retired), got->etag, Retry::standard()), "farewell"); EXPECT_TRUE(isCreatorFenceTerminal(ops.op, l, "r", 7)); @@ -2030,7 +2025,7 @@ TEST(CASMountLease, ClaimAdoptIsTwoRequests) /// The absent-slot mint. backend->reads = backend->heads = backend->writes = 0; - MountLeaseKeeper minting(ops.mount, ops.farewell, l, "fresh", UInt128(1), 7, + MountLeaseRenewer minting(ops.mount, ops.farewell, l, "fresh", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }); minting.start(); EXPECT_EQ(backend->reads, 1u); @@ -2041,7 +2036,7 @@ TEST(CASMountLease, ClaimAdoptIsTwoRequests) ASSERT_EQ(claimMount(ops.op, l, "adopted", UInt128(1), /*epoch*/ 7, now, /*ttl*/ 100).kind, MountClaimResult::Claimed); backend->reads = backend->heads = backend->writes = 0; - MountLeaseKeeper adopting(ops.mount, ops.farewell, l, "adopted", UInt128(1), 7, + MountLeaseRenewer adopting(ops.mount, ops.farewell, l, "adopted", UInt128(1), 7, std::chrono::milliseconds(100), [&] { return now; }, [] { return uint64_t{0}; }); adopting.start(); EXPECT_EQ(backend->reads, 1u); @@ -2051,7 +2046,7 @@ TEST(CASMountLease, ClaimAdoptIsTwoRequests) /// A mount whose fence has dropped must still hand its slot back: the renewal is refused (it would be /// writing under authority this node no longer holds), while the farewell runs on the open plane and -/// lands. Deliberately two keepers: `release` is admitted only from `Active`, so a keeper whose +/// lands. Deliberately two renewers: `release` is admitted only from `Active`, so a renewer whose /// renewal already went terminal never reaches its own farewell -- the ordering the two halves below /// pin separately. TEST(CASMountLease, FarewellRunsOnAnOpenFenceAfterTheMountFenceIsLost) @@ -2080,10 +2075,10 @@ TEST(CASMountLease, FarewellRunsOnAnOpenFenceAfterTheMountFenceIsLost) ASSERT_EQ(claimMount(seed, l, "renewing", UInt128(1), 7, now, /*ttl*/ 1000).kind, MountClaimResult::Claimed); ASSERT_EQ(claimMount(seed, l, "departing", UInt128(1), 7, now, /*ttl*/ 1000).kind, MountClaimResult::Claimed); - MountLeaseKeeper renewing(mount_requests, open_requests, l, "renewing", UInt128(1), 7, + MountLeaseRenewer renewing(mount_requests, open_requests, l, "renewing", UInt128(1), 7, std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); - MountLeaseKeeper departing(mount_requests, open_requests, l, "departing", UInt128(1), 7, + MountLeaseRenewer departing(mount_requests, open_requests, l, "departing", UInt128(1), 7, std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); renewing.start(); @@ -2121,10 +2116,10 @@ TEST(CASMountLease, ClaimIsNotAdmittedUnderTheMountFence) open_requests.setNowFnForTest([&boot] { return boot; }); open_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); - MountLeaseKeeper keeper(mount_requests, open_requests, l, "r", UInt128(1), 7, + MountLeaseRenewer renewer(mount_requests, open_requests, l, "r", UInt128(1), 7, std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); - EXPECT_NO_THROW(keeper.start()); + EXPECT_NO_THROW(renewer.start()); CasOperation reader = open_requests.admit(); const MountLease claimed = decodeMountLease(reader.read(l.mountKey("r"), Retry::standard())->bytes); @@ -2181,12 +2176,12 @@ TEST(CASMountLease, RemountRenewalIsAdmittedOffTheMountFence) open_requests.setNowFnForTest([&boot] { return boot; }); open_requests.setSleepFnForTest([&boot](uint64_t ms) { boot += ms; }); - MountLeaseKeeper keeper(mount_requests, open_requests, l, "r", UInt128(1), 7, + MountLeaseRenewer renewer(mount_requests, open_requests, l, "r", UInt128(1), 7, std::chrono::milliseconds(1000), [&] { return now; }, [] { return uint64_t{0}; }, {}, std::chrono::milliseconds(0), [&] { return boot; }); - keeper.start(); + renewer.start(); - const MountRenewResult redo = keeper.renewForRemount(); + const MountRenewResult redo = renewer.renewForRemount(); EXPECT_EQ(redo.outcome, MountRenewOutcome::Committed); CasOperation reader = open_requests.admit(); diff --git a/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp b/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp index bb1b8bd3adc6..313b7f7e362d 100644 --- a/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp +++ b/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp @@ -14,16 +14,16 @@ using DB::Cas::tests::expectThrowsCodeWithMessage; namespace { -/// One keeper for the mount slot of server-root "r", under (uuid=1, epoch=7) unless overridden. Both +/// One renewer for the mount slot of server-root "r", under (uuid=1, epoch=7) unless overridden. Both /// of its planes are the same open-fence one: what these tests exercise is the mount protocol's own /// exclusivity, not a fence's, and no test here renews, which is the only caller of the mount plane. -MountLeaseKeeper makeKeeper( +MountLeaseRenewer makeRenewer( CasRequests & requests, uint64_t & now, DB::UInt128 uuid = DB::UInt128(1), uint64_t epoch = 7) { - return MountLeaseKeeper( + return MountLeaseRenewer( requests, requests, Layout("p"), @@ -43,7 +43,7 @@ void markMountGcFenced(CasOperation & op, const Layout & layout, const String & MountLease lease = decodeMountLease(got->bytes); lease.gc_fenced = true; ASSERT_TRUE(std::holds_alternative( - op.replace(key, encodeMountLease(lease), got->incarnation, Retry::standard()))); + op.replace(key, encodeMountLease(lease), got->etag, Retry::standard()))); } } @@ -61,11 +61,11 @@ TEST(CASMountClaimConflicts, SlotAppearedBetweenTheReadAndTheCreate) CasOperation racer = requests.admit(); claimMount(racer, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100); }; - auto keeper = makeKeeper(requests, now); + auto renewer = makeRenewer(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "appeared between the read and the create", - [&] { keeper.start(); }); + [&] { renewer.start(); }); } TEST(CASMountClaimConflicts, SlotHeldByForeignServer) @@ -78,11 +78,11 @@ TEST(CASMountClaimConflicts, SlotHeldByForeignServer) ASSERT_EQ( claimMount(op, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - auto keeper = makeKeeper(requests, now); + auto renewer = makeRenewer(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "held by a foreign server", - [&] { keeper.start(); }); + [&] { renewer.start(); }); } TEST(CASMountClaimConflicts, SlotHeldByDifferentWriterEpoch) @@ -95,11 +95,11 @@ TEST(CASMountClaimConflicts, SlotHeldByDifferentWriterEpoch) ASSERT_EQ( claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); - auto keeper = makeKeeper(requests, now, DB::UInt128(1), /*epoch=*/8); + auto renewer = makeRenewer(requests, now, DB::UInt128(1), /*epoch=*/8); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "held by a different writer_epoch", - [&] { keeper.start(); }); + [&] { renewer.start(); }); } TEST(CASMountClaimConflicts, SlotChangedInsideAdoptionWindow) @@ -118,11 +118,11 @@ TEST(CASMountClaimConflicts, SlotChangedInsideAdoptionWindow) CasOperation racer = requests.admit(); claimMount(racer, layout, "r", DB::UInt128(1), 7, now + 1, /*ttl_ms=*/100); }; - auto keeper = makeKeeper(requests, now); + auto renewer = makeRenewer(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "changed while adopting our own mount slot", - [&] { keeper.start(); }); + [&] { renewer.start(); }); } TEST(CASMountClaimConflicts, SlotVanishedInsideAdoptionWindow) @@ -140,11 +140,11 @@ TEST(CASMountClaimConflicts, SlotVanishedInsideAdoptionWindow) CasOperation racer = requests.admit(); ASSERT_EQ(racer.removeCurrent(layout.mountKey("r"), Retry::standard()), Removal::Removed); }; - auto keeper = makeKeeper(requests, now); + auto renewer = makeRenewer(requests, now); expectThrowsCodeWithMessage( DB::ErrorCodes::ABORTED, "vanished while adopting our own mount slot", - [&] { keeper.start(); }); + [&] { renewer.start(); }); } /// The two fenced branches keep their own type, and keep PRECEDENCE over the conflicts above: the @@ -161,8 +161,8 @@ TEST(CASMountClaimConflicts, FencedBeforeAdoptionRaisesMountFenced) claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, MountClaimResult::Claimed); markMountGcFenced(op, layout, "r"); - auto keeper = makeKeeper(requests, now); - EXPECT_THROW(keeper.start(), MountFencedException); + auto renewer = makeRenewer(requests, now); + EXPECT_THROW(renewer.start(), MountFencedException); } TEST(CASMountClaimConflicts, FencedInsideAdoptionWindowRaisesMountFencedNotAborted) @@ -182,6 +182,6 @@ TEST(CASMountClaimConflicts, FencedInsideAdoptionWindowRaisesMountFencedNotAbort CasOperation racer = requests.admit(); markMountGcFenced(racer, layout, "r"); }; - auto keeper = makeKeeper(requests, now); - EXPECT_THROW(keeper.start(), MountFencedException); + auto renewer = makeRenewer(requests, now); + EXPECT_THROW(renewer.start(), MountFencedException); } diff --git a/src/Disks/tests/gtest_cas_mount_runtime.cpp b/src/Disks/tests/gtest_cas_mount_runtime.cpp index 697d486c171a..25d7f5e3b25e 100644 --- a/src/Disks/tests/gtest_cas_mount_runtime.cpp +++ b/src/Disks/tests/gtest_cas_mount_runtime.cpp @@ -12,7 +12,7 @@ using namespace DB::Cas; namespace { -/// A `CasMountRuntime` with nothing running on it: no keeper, no workers, an injected boot clock and a +/// A `CasMountRuntime` with nothing running on it: no renewer, no workers, an injected boot clock and a /// fence the test arms by hand. Enough to exercise admission, which reads only the fence's own state. class RuntimeFixture { diff --git a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp index 45fca6d2f1b1..4f5e17c1931f 100644 --- a/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp +++ b/src/Disks/tests/gtest_cas_namespace_file_request_profile.cpp @@ -29,7 +29,7 @@ namespace DB::ContentAddressedSetting /// /// WHAT THIS FILE PINS: the last clause, per key. The counts below were READ OFF this tree before any /// key change and pasted as literals, which is the whole point of the file -- expectations re-derived -/// after a change measure the change against itself. Incarnation qualification changes the KEY a +/// after a change measure the change against itself. Etag qualification changes the KEY a /// namespace file is stored under, so the keys are derived from `Layout` rather than spelled out; what /// must not move is the count per key and the set of keys touched. /// @@ -519,7 +519,8 @@ TEST(CASNamespaceFileDiskProfile, RemovalOnANeverOpenedTableLeavesTheCatalogUnto /// A valid pool already owns its explicit empty mandatory catalog. Nothing has opened this table: /// no namespace file written, no part published, and no ref operation has changed that object. - const auto catalog_before = storage->store()->backend().get(layout.refCatalogKey()); + OperationForTest catalog_probe(storage->store()->poolBackendPtr()); + const auto catalog_before = (*catalog_probe).read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog_before); EXPECT_TRUE(decodeRefCatalog(catalog_before->bytes).entries.empty()); object_storage->resetRecords(); @@ -538,10 +539,10 @@ TEST(CASNamespaceFileDiskProfile, RemovalOnANeverOpenedTableLeavesTheCatalogUnto EXPECT_EQ(object_storage->writtenContaining(layout.refCatalogKey()), std::vector{}) << "a removal must not write the catalog: it must not birth the namespace it is removing from"; - const auto catalog_after_removal = storage->store()->backend().get(layout.refCatalogKey()); + const auto catalog_after_removal = (*catalog_probe).read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog_after_removal); EXPECT_EQ(catalog_after_removal->bytes, catalog_before->bytes); - EXPECT_EQ(catalog_after_removal->token, catalog_before->token) + EXPECT_EQ(catalog_after_removal->etag, catalog_before->etag) << "the mandatory catalog must remain byte-for-byte and token-for-token unchanged"; /// Not vacuous: the SAME operations on the same table after a write do reach the file, so the zeros @@ -553,8 +554,8 @@ TEST(CASNamespaceFileDiskProfile, RemovalOnANeverOpenedTableLeavesTheCatalogUnto EXPECT_FALSE(storage->existsFile(kTablePath + "/format_version.txt")); /// Positive control: the write really did birth the namespace and mutate the same catalog object /// whose stability the removal assertions pin above. - const auto catalog_after_birth = storage->store()->backend().get(layout.refCatalogKey()); + const auto catalog_after_birth = (*catalog_probe).read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog_after_birth); EXPECT_NE(catalog_after_birth->bytes, catalog_after_removal->bytes); - EXPECT_NE(catalog_after_birth->token, catalog_after_removal->token); + EXPECT_NE(catalog_after_birth->etag, catalog_after_removal->etag); } diff --git a/src/Disks/tests/gtest_cas_namespace_janitor.cpp b/src/Disks/tests/gtest_cas_namespace_janitor.cpp index 76d2f4aa5be4..0ada8882e7b0 100644 --- a/src/Disks/tests/gtest_cas_namespace_janitor.cpp +++ b/src/Disks/tests/gtest_cas_namespace_janitor.cpp @@ -86,9 +86,11 @@ class ReplaceBeforeJanitorDeleteBackend : public CountingBackend if (!replaced) { replaced = true; - const auto current = InMemoryBackend::get(key); + /// The qualified primitive, exactly as the sibling concurrent-actor doubles in this file: a + /// simulated concurrent write must not be counted as the janitor's own. + const auto current = InMemoryBackend::read(key, access); if (current) - (void)InMemoryBackend::casPut(key, "winner", current->token); + (void)InMemoryBackend::write(key, "winner", current->value, access); } return CountingBackend::remove(key, expected_value, access); } @@ -229,9 +231,23 @@ class ThrowingListAndAmbiguousWriteBackend : public CountingBackend uint64_t write_attempts = 0; }; -void seedCatalog(CountingBackend & backend, const Layout & layout, RefCatalog catalog = {}) +/// A one-shot `create`, asserting it committed (mirrors the retired `backend->putIfAbsent(key, bytes)`). +void createObj(Backend & backend, const String & key, const String & bytes) { - ASSERT_EQ(backend.putIfAbsent(layout.refCatalogKey(), encodeRefCatalog(catalog)).outcome, PutOutcome::Done); + OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// An exact read (mirrors the retired `backend->get(key)`). +std::optional readObj(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +void seedCatalog(Backend & backend, const Layout & layout, RefCatalog catalog = {}) +{ + createObj(backend, layout.refCatalogKey(), encodeRefCatalog(catalog)); } NamespaceLifeId life(const char * name, uint64_t id) @@ -251,8 +267,8 @@ TEST(CASNamespaceJanitor, DeletesDeadFilesAndCheckpointFromOnePostListCatalogCut const auto dead = life("dead", 41); const String file = layout.namespaceFilesPrefix(dead) + "part/data.bin"; const String ckpt = layout.refCkptKey(dead); - ASSERT_EQ(backend->putIfAbsent(file, "file-bytes").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(ckpt, "ckpt-bytes").outcome, PutOutcome::Done); + createObj(*backend, file, "file-bytes"); + createObj(*backend, ckpt, "ckpt-bytes"); backend->resetCounts(); NamespaceJanitor janitor(requests, layout, 100); @@ -261,8 +277,8 @@ TEST(CASNamespaceJanitor, DeletesDeadFilesAndCheckpointFromOnePostListCatalogCut EXPECT_EQ(result.pages, 1u); EXPECT_EQ(result.keys, 2u); EXPECT_EQ(result.deleted, 2u); - EXPECT_FALSE(backend->get(file)); - EXPECT_FALSE(backend->get(ckpt)); + EXPECT_FALSE(readObj(*backend, file).has_value()); + EXPECT_FALSE(readObj(*backend, ckpt).has_value()); EXPECT_EQ(backend->listCount(layout.namespaceRootPrefix()), 1u); EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); EXPECT_EQ(readState(requests, layout).state, GcMaintenanceState{}); @@ -282,8 +298,8 @@ TEST(CASNamespaceJanitor, RetainsEveryCurrentLifecycleAndSuppressesAmbiguousCut) catalog.entries = {creating, live, removing}; seedCatalog(*backend, layout, catalog); for (const auto & entry : catalog.entries) - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey( - NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation)), "keep").outcome, PutOutcome::Done); + createObj(*backend, layout.refCkptKey( + NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation)), "keep"); NamespaceJanitor janitor(requests, layout, 100); const auto result = janitor.runOnePage(false, [] { return true; }); @@ -308,8 +324,8 @@ TEST(CASNamespaceJanitor, CatalogFirstCreatingRetainsEveryObjectOfTheNewLife) = NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation); const String ckpt = layout.refCkptKey(creating_life); const String file = layout.namespaceFilesPrefix(creating_life) + "data"; - ASSERT_EQ(backend->putIfAbsent(ckpt, "checkpoint").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(file, "file").outcome, PutOutcome::Done); + createObj(*backend, ckpt, "checkpoint"); + createObj(*backend, file, "file"); backend->resetCounts(); const NamespaceJanitorResult result @@ -318,8 +334,8 @@ TEST(CASNamespaceJanitor, CatalogFirstCreatingRetainsEveryObjectOfTheNewLife) EXPECT_EQ(result.deleted, 0u); EXPECT_EQ(backend->deleteTotal(), 0u); EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); - EXPECT_TRUE(backend->get(ckpt)); - EXPECT_TRUE(backend->get(file)); + EXPECT_TRUE(readObj(*backend, ckpt).has_value()); + EXPECT_TRUE(readObj(*backend, file).has_value()); } TEST(CASNamespaceJanitor, CancelledCreatingCheckpointIsReclaimedThroughPublicLifecycle) @@ -335,7 +351,7 @@ TEST(CASNamespaceJanitor, CancelledCreatingCheckpointIsReclaimedThroughPublicLif seedCatalog(*backend, layout, RefCatalog{.entries = {creating}}); const String ckpt = layout.refCkptKey( NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation)); - ASSERT_EQ(backend->putIfAbsent(ckpt, "cancelled-checkpoint").outcome, PutOutcome::Done); + createObj(*backend, ckpt, "cancelled-checkpoint"); auto cancel_op = requests.admit(); ASSERT_EQ(CasRefCatalog::cancelStalledCreating( @@ -347,7 +363,7 @@ TEST(CASNamespaceJanitor, CancelledCreatingCheckpointIsReclaimedThroughPublicLif const NamespaceJanitorResult result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); - EXPECT_FALSE(backend->get(ckpt)); + EXPECT_FALSE(readObj(*backend, ckpt).has_value()); } TEST(CASNamespaceJanitor, SuppressionAndFenceLossDeleteNothing) @@ -358,8 +374,8 @@ TEST(CASNamespaceJanitor, SuppressionAndFenceLossDeleteNothing) seedCatalog(*backend, layout); const String first = layout.refCkptKey(life("dead-a", 61)); const String second = layout.refCkptKey(life("dead-b", 62)); - ASSERT_EQ(backend->putIfAbsent(first, "first").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(second, "second").outcome, PutOutcome::Done); + createObj(*backend, first, "first"); + createObj(*backend, second, "second"); /// The seeding above (the catalog + the two checkpoints) lands through the same write primitive /// CountingBackend counts, so reset before measuring what the suppressed page itself does. @@ -378,8 +394,8 @@ TEST(CASNamespaceJanitor, SuppressionAndFenceLossDeleteNothing) [&] { (void)janitor.runOnePage(false, [] { return false; }); }); EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "fence loss must not mint progress past a page whose deletion was not authorized"; - EXPECT_TRUE(backend->get(first)); - EXPECT_TRUE(backend->get(second)); + EXPECT_TRUE(readObj(*backend, first).has_value()); + EXPECT_TRUE(readObj(*backend, second).has_value()); EXPECT_EQ(backend->deleteTotal(), 0u); } @@ -395,8 +411,8 @@ TEST(CASNamespaceJanitor, FenceLossOnRetainedOnlyPageDoesNotAdvanceCursor) = NamespaceLifeId::fromCatalogEntry(current.ns, current.incarnation); const String ckpt = layout.refCkptKey(current_life); const String file = layout.namespaceFilesPrefix(current_life) + "data"; - ASSERT_EQ(backend->putIfAbsent(ckpt, "checkpoint").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(file, "file").outcome, PutOutcome::Done); + createObj(*backend, ckpt, "checkpoint"); + createObj(*backend, file, "file"); /// A liveness sample false from the start is refused at the maintenance read, before the page ever /// gets to examine an object -- retained-only or not; the page ends by exception. @@ -404,8 +420,8 @@ TEST(CASNamespaceJanitor, FenceLossOnRetainedOnlyPageDoesNotAdvanceCursor) [&] { (void)NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return false; }); }); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_TRUE(backend->get(ckpt)); - EXPECT_TRUE(backend->get(file)); + EXPECT_TRUE(readObj(*backend, ckpt).has_value()); + EXPECT_TRUE(readObj(*backend, file).has_value()); EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "a tenure that observes fence loss cannot publish progress even when every object was retained"; } @@ -417,13 +433,13 @@ TEST(CASNamespaceJanitor, FenceLossAfterLastDeleteRetainsCursorWithoutRollingBac const Layout layout("p"); seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead-after-delete", 64)); - ASSERT_EQ(backend->putIfAbsent(dead, "dead").outcome, PutOutcome::Done); + createObj(*backend, dead, "dead"); const NamespaceJanitorResult result = NamespaceJanitor(requests, layout, 1).runOnePage( false, [&] { return !backend->delete_done; }); EXPECT_EQ(result.deleted, 1u); - EXPECT_FALSE(backend->get(dead)) + EXPECT_FALSE(readObj(*backend, dead).has_value()) << "the exact delete completed under the fence and is never rolled back"; EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "losing the fence after the delete keeps this page selected for an idempotent retry"; @@ -436,8 +452,8 @@ TEST(CASNamespaceJanitor, CursorResumesThenResetsAtEnd) const Layout layout("p"); seedCatalog(*backend, layout); const auto dead = life("dead", 71); - ASSERT_EQ(backend->putIfAbsent(layout.namespaceFilesPrefix(dead) + "a", "a").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(layout.namespaceFilesPrefix(dead) + "b", "b").outcome, PutOutcome::Done); + createObj(*backend, layout.namespaceFilesPrefix(dead) + "a", "a"); + createObj(*backend, layout.namespaceFilesPrefix(dead) + "b", "b"); NamespaceJanitor first_process(requests, layout, 1); EXPECT_EQ(first_process.runOnePage(false, [] { return true; }).deleted, 1u); @@ -460,17 +476,17 @@ TEST(CASNamespaceJanitor, TakesOneCatalogCutAfterListingAndContinuesPastMalforme const String valid = layout.namespaceFilesPrefix(dead) + "data"; const String malformed = layout.namespaceStreamRootPrefix() + "not-a-life/_log/1-1.zst"; const String malformed_state = layout.namespaceStateRootPrefix() + "not-a-life/_ckpt"; - ASSERT_EQ(backend->putIfAbsent(valid, "v").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(malformed, "bad").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(malformed_state, "bad-state").outcome, PutOutcome::Done); + createObj(*backend, valid, "v"); + createObj(*backend, malformed, "bad"); + createObj(*backend, malformed_state, "bad-state"); backend->resetCounts(); backend->events.clear(); const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); EXPECT_FALSE(result.anomalies.empty()); - EXPECT_TRUE(backend->get(malformed)); - EXPECT_TRUE(backend->get(malformed_state)); + EXPECT_TRUE(readObj(*backend, malformed).has_value()); + EXPECT_TRUE(readObj(*backend, malformed_state).has_value()); ASSERT_EQ(backend->events.size(), 2u); EXPECT_EQ(backend->events[0], "list"); EXPECT_EQ(backend->events[1], "catalog"); @@ -485,16 +501,16 @@ TEST(CASNamespaceJanitor, MalformedKeyIsFinalAndAdvancesCursor) seedCatalog(*backend, layout); const String first = layout.namespaceStreamRootPrefix() + "bad-a/_log/1-1.zst"; const String second = layout.namespaceStreamRootPrefix() + "bad-b/_log/1-1.zst"; - ASSERT_EQ(backend->putIfAbsent(first, "first").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(second, "second").outcome, PutOutcome::Done); + createObj(*backend, first, "first"); + createObj(*backend, second, "second"); const NamespaceJanitorResult result = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); EXPECT_FALSE(result.anomalies.empty()); - EXPECT_TRUE(backend->get(first)); - EXPECT_TRUE(backend->get(second)); + EXPECT_TRUE(readObj(*backend, first).has_value()); + EXPECT_TRUE(readObj(*backend, second).has_value()); const GcMaintenanceReadResult progress = readState(requests, layout); ASSERT_EQ(progress.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(progress.state); @@ -514,13 +530,13 @@ TEST(CASNamespaceJanitor, DuplicateCurrentLifeSuppressesWholePage) seedCatalog(*backend, layout, catalog); const String dead_a = layout.refCkptKey(life("dead-a", 92)); const String dead_b = layout.refCkptKey(life("dead-b", 93)); - ASSERT_EQ(backend->putIfAbsent(dead_a, "a").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(dead_b, "b").outcome, PutOutcome::Done); + createObj(*backend, dead_a, "a"); + createObj(*backend, dead_b, "b"); const auto result = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_TRUE(backend->get(dead_a)); - EXPECT_TRUE(backend->get(dead_b)); + EXPECT_TRUE(readObj(*backend, dead_a).has_value()); + EXPECT_TRUE(readObj(*backend, dead_b).has_value()); EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Absent) << "an ambiguous catalog cut leaves the selected page undecided for an authoritative retry"; } @@ -532,15 +548,15 @@ TEST(CASNamespaceJanitor, CorruptProgressResetsWithoutDeletingAndFilesOnlyOmitte const Layout layout("p"); seedCatalog(*backend, layout); const String dead = layout.namespaceFilesPrefix(life("dead", 101)) + "only-residue"; - ASSERT_EQ(backend->putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(layout.gcMaintenanceStateKey(), "corrupt").outcome, PutOutcome::Done); + createObj(*backend, dead, "bytes"); + createObj(*backend, layout.gcMaintenanceStateKey(), "corrupt"); EXPECT_EQ(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }).deleted, 0u); - EXPECT_TRUE(backend->get(dead)); + EXPECT_TRUE(readObj(*backend, dead).has_value()); EXPECT_EQ(readState(requests, layout).status, GcMaintenanceReadStatus::Valid); EXPECT_EQ(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }).deleted, 0u); - EXPECT_TRUE(backend->get(dead)); + EXPECT_TRUE(readObj(*backend, dead).has_value()); EXPECT_EQ(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }).deleted, 1u); - EXPECT_FALSE(backend->get(dead)); + EXPECT_FALSE(readObj(*backend, dead).has_value()); } TEST(CASNamespaceJanitor, ExactTokenMismatchRetainsConcurrentReplacement) @@ -551,13 +567,13 @@ TEST(CASNamespaceJanitor, ExactTokenMismatchRetainsConcurrentReplacement) seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead-a", 111)); const String later = layout.refCkptKey(life("dead-b", 112)); - ASSERT_EQ(backend->putIfAbsent(dead, "old").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(later, "later").outcome, PutOutcome::Done); + createObj(*backend, dead, "old"); + createObj(*backend, later, "later"); const auto result = NamespaceJanitor(requests, layout, 1).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); - ASSERT_TRUE(backend->get(dead)); - EXPECT_EQ(backend->get(dead)->bytes, "winner"); - EXPECT_TRUE(backend->get(later)); + ASSERT_TRUE(readObj(*backend, dead).has_value()); + EXPECT_EQ(readObj(*backend, dead)->bytes, "winner"); + EXPECT_TRUE(readObj(*backend, later).has_value()); const GcMaintenanceReadResult progress = readState(requests, layout); ASSERT_EQ(progress.status, GcMaintenanceReadStatus::Valid); ASSERT_TRUE(progress.state); @@ -576,9 +592,9 @@ TEST(CASNamespaceJanitor, TokenlessListHeadsDeadKeysAndRetainsConcurrentReplacem const String live_key = layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(current.ns, current.incarnation)); const String dead_key = layout.refCkptKey(life("dead", 162)); const String raced_key = layout.namespaceFilesPrefix(life("raced", 163)) + "data"; - ASSERT_EQ(backend->putIfAbsent(live_key, "live").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(dead_key, "dead").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(raced_key, "old").outcome, PutOutcome::Done); + createObj(*backend, live_key, "live"); + createObj(*backend, dead_key, "dead"); + createObj(*backend, raced_key, "old"); backend->replace_on_head = raced_key; backend->resetCounts(); @@ -586,10 +602,10 @@ TEST(CASNamespaceJanitor, TokenlessListHeadsDeadKeysAndRetainsConcurrentReplacem EXPECT_EQ(result.deleted, 1u); EXPECT_TRUE(result.anomalies.empty()); - EXPECT_TRUE(backend->get(live_key)); - EXPECT_FALSE(backend->get(dead_key)); - ASSERT_TRUE(backend->get(raced_key)); - EXPECT_EQ(backend->get(raced_key)->bytes, "winner"); + EXPECT_TRUE(readObj(*backend, live_key).has_value()); + EXPECT_FALSE(readObj(*backend, dead_key).has_value()); + ASSERT_TRUE(readObj(*backend, raced_key).has_value()); + EXPECT_EQ(readObj(*backend, raced_key)->bytes, "winner"); EXPECT_EQ(backend->headCount(live_key), 0u); EXPECT_EQ(backend->headCount(dead_key), 1u); EXPECT_EQ(backend->headCount(raced_key), 1u); @@ -604,7 +620,7 @@ TEST(CASNamespaceJanitor, TokenlessListRechecksFenceAfterHeadBeforeDelete) const Layout layout("p"); seedCatalog(*backend, layout); const String dead_key = layout.refCkptKey(life("dead", 164)); - ASSERT_EQ(backend->putIfAbsent(dead_key, "dead").outcome, PutOutcome::Done); + createObj(*backend, dead_key, "dead"); backend->resetCounts(); const auto result = NamespaceJanitor(requests, layout, 100).runOnePage( @@ -613,7 +629,7 @@ TEST(CASNamespaceJanitor, TokenlessListRechecksFenceAfterHeadBeforeDelete) EXPECT_EQ(result.deleted, 0u); EXPECT_EQ(backend->headCount(dead_key), 1u); EXPECT_EQ(backend->deleteCount(dead_key), 0u); - EXPECT_TRUE(backend->get(dead_key)); + EXPECT_TRUE(readObj(*backend, dead_key).has_value()); } TEST(CASNamespaceJanitor, PostListCatalogCutProtectsConcurrentCreationWithOneGet) @@ -625,15 +641,15 @@ TEST(CASNamespaceJanitor, PostListCatalogCutProtectsConcurrentCreationWithOneGet seedCatalog(*backend, layout); const String first = layout.refCkptKey(created); const String second = layout.namespaceFilesPrefix(created) + "data"; - ASSERT_EQ(backend->putIfAbsent(first, "ckpt").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(second, "file").outcome, PutOutcome::Done); + createObj(*backend, first, "ckpt"); + createObj(*backend, second, "file"); backend->resetCounts(); const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 0u); EXPECT_EQ(backend->deleteTotal(), 0u); EXPECT_EQ(backend->getCount(layout.refCatalogKey()), 1u); - EXPECT_TRUE(backend->get(first)); - EXPECT_TRUE(backend->get(second)); + EXPECT_TRUE(readObj(*backend, first).has_value()); + EXPECT_TRUE(readObj(*backend, second).has_value()); } TEST(CASNamespaceJanitor, BackendRejectedCursorResetsExactlyAndDeletesNothing) @@ -643,12 +659,12 @@ TEST(CASNamespaceJanitor, BackendRejectedCursorResetsExactlyAndDeletesNothing) const Layout layout("p"); seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead", 131)); - ASSERT_EQ(backend->putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(layout.gcMaintenanceStateKey(), - encodeGcMaintenanceState({.janitor_cursor = "rejected"})).outcome, PutOutcome::Done); + createObj(*backend, dead, "bytes"); + createObj(*backend, layout.gcMaintenanceStateKey(), + encodeGcMaintenanceState({.janitor_cursor = "rejected"})); EXPECT_THROW(NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }), std::runtime_error); EXPECT_EQ(backend->deleteTotal(), 0u); - EXPECT_TRUE(backend->get(dead)); + EXPECT_TRUE(readObj(*backend, dead).has_value()); EXPECT_TRUE(readState(requests, layout).state->janitor_cursor.empty()); } @@ -659,12 +675,12 @@ TEST(CASNamespaceJanitor, CursorPublicationFailureIsLeakOnly) const Layout layout("p"); seedCatalog(*backend, layout); const String dead = layout.refCkptKey(life("dead", 141)); - ASSERT_EQ(backend->putIfAbsent(dead, "bytes").outcome, PutOutcome::Done); + createObj(*backend, dead, "bytes"); backend->fail_publication = true; const auto result = NamespaceJanitor(requests, layout, 100).runOnePage(false, [] { return true; }); EXPECT_EQ(result.deleted, 1u); EXPECT_FALSE(result.anomalies.empty()); - EXPECT_FALSE(backend->get(dead)); + EXPECT_FALSE(readObj(*backend, dead).has_value()); } /// The write inside `catch (...)` (the reset after a LIST failure) is admitted `once`: an unmodeled, @@ -695,12 +711,11 @@ TEST(CASNamespaceJanitorIntegration, RegularGcRoundDeletesDeadNamespaceBytes) const Layout & layout = store->layout(); const RootNamespace live_namespace{"00/live@cas@"}; fixture::admitLive(*backend, layout, live_namespace); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(fixture::fixtureLife(live_namespace)), + createObj(*backend, layout.refCkptKey(fixture::fixtureLife(live_namespace)), encodeRefCkpt(RefCkpt{.life_epoch = std::optional{1}, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})).outcome, - PutOutcome::Done); + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})); const String dead = layout.refCkptKey(life("dead", 151)); - ASSERT_EQ(backend->putIfAbsent(dead, "checkpoint").outcome, PutOutcome::Done); + createObj(*backend, dead, "checkpoint"); std::map namespace_cleanup; Gc gc(store, UInt128{152}); @@ -713,7 +728,7 @@ TEST(CASNamespaceJanitorIntegration, RegularGcRoundDeletesDeadNamespaceBytes) gc.setPhaseSink({}); ASSERT_TRUE(report.acquired_lease); - EXPECT_FALSE(backend->get(dead)); + EXPECT_FALSE(readObj(*backend, dead).has_value()); ASSERT_FALSE(namespace_cleanup.empty()); EXPECT_EQ(namespace_cleanup["janitor_pages"], 1u); EXPECT_GE(namespace_cleanup["janitor_keys"], 1u); diff --git a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp index f97590ab3e12..89d55e8dbb65 100644 --- a/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_incarnation.cpp @@ -74,12 +74,14 @@ TEST(CASNsFileIncarnation, ColdReaderUsesCatalogCutWhileOldFileSurvivesRemoval) const String old_key = layout.namespaceFileKey(*old_life, kFile); backend->hide(old_key); - ASSERT_TRUE(backend->head(old_key).exists) << "the lie must be in LIST only -- the object is durable"; + CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation catalog_op = catalog_requests.admit(); + + ASSERT_TRUE(catalog_op.head(old_key, Retry::standard()).has_value()) + << "the lie must be in LIST only -- the object is durable"; ASSERT_TRUE(store->listNamespaceFiles(*old_life).empty()) << "precondition: enumeration omits the file, so no cleanup pass can ever find it"; const size_t holes_before_gc = backend->holesServed(); - CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); - CasOperation catalog_op = catalog_requests.admit(); store->dropNamespace(ns); ASSERT_TRUE(CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns)); @@ -94,9 +96,9 @@ TEST(CASNsFileIncarnation, ColdReaderUsesCatalogCutWhileOldFileSurvivesRemoval) ASSERT_GT(backend->holesServed(), holes_before_gc) << "the GC janitor must observe the injected LIST hole after the explicit precondition LIST"; - const auto old_head = backend->head(old_key); - ASSERT_TRUE(old_head.exists) << "logical removal must not depend on physical empty"; - const auto old_object = backend->get(old_key); + const auto old_head = catalog_op.head(old_key, Retry::standard()); + ASSERT_TRUE(old_head.has_value()) << "logical removal must not depend on physical empty"; + const auto old_object = catalog_op.read(old_key, Retry::standard()); ASSERT_TRUE(old_object); EXPECT_EQ(old_object->bytes, old_bytes); @@ -211,20 +213,21 @@ TEST(CASNsFileIncarnation, RebirthDoesNotWaitForFilesToBeEmpty) .last_epoch_seal = std::nullopt, }); const String debris_key = layout.namespaceFileKey(life, kFile); - backend->putIfAbsent(debris_key, "1\n"); - backend->putIfAbsent(layout.namespaceFileKey(life, "deduplication_logs/deduplication_log_1.txt"), "records"); + catalog_op.create(debris_key, "1\n", Retry::once()); + catalog_op.create(layout.namespaceFileKey(life, "deduplication_logs/deduplication_log_1.txt"), "records", Retry::once()); Gc gc(store, kGcId); gc.runRegularRound(); /// Folding the terminal records positive evidence on the same life row even though files remain. - const GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState state = decodeGcState(catalog_op.read(layout.gcStateKey(), Retry::standard())->bytes); ASSERT_GT(state.snap_generation, 0u); const CasFoldSeal seal = decodeFoldSeal( - backend->get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + catalog_op.read(layout.foldSealKey(state.snap_generation, state.snap_attempt), Retry::standard())->bytes); const auto row_it = seal.ref_lives.find(life.incarnation); ASSERT_NE(row_it, seal.ref_lives.end()); ASSERT_TRUE(row_it->second.cleanup_evidence.has_value()); EXPECT_EQ(row_it->second.cleanup_evidence->remove_txn_id, (RefTxnId{1, 1})); - EXPECT_TRUE(backend->head(debris_key).exists) << "cleanup evidence does not gate on physical deletion"; + EXPECT_TRUE(catalog_op.head(debris_key, Retry::standard()).has_value()) + << "cleanup evidence does not gate on physical deletion"; } diff --git a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp index 5bd58de528d5..a843ba3fbce3 100644 --- a/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ns_file_read_contract.cpp @@ -81,9 +81,8 @@ void writeVerbatimThroughDisk( void deleteCatalogLife( DB::ContentAddressedMetadataStorage & storage, const NamespaceLifeId & life1) { - Backend & backend = storage.store()->backend(); const Layout & layout = storage.store()->layout(); - CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasRequests requests = DB::Cas::tests::openRequestsForTest(storage.store()->poolBackendPtr()); CasOperation op = requests.admit(); CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & current) { @@ -126,7 +125,7 @@ NamespaceLifeId admitReplacementLife( if (life1.incarnation == kLife2Id) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Fixture life ids unexpectedly collide"); const NamespaceLifeId life2 = NamespaceLifeId::fromCatalogEntry(life1.ns, kLife2Id); - CasRequests requests = DB::Cas::tests::openRequestsForTest(storage.store()->backend()); + CasRequests requests = DB::Cas::tests::openRequestsForTest(storage.store()->poolBackendPtr()); CasOperation op = requests.admit(); CasRefCatalog::casAdmitEntry( op, storage.store()->layout(), storage.store()->poolConfig().gc_shards, CatalogEntry{ @@ -186,14 +185,15 @@ TEST(CASNamespaceFileReadContract, DelayedInlineFinalizeCannotChangeSuccessorTok const NamespaceLifeId life2 = replaceCatalogLife(*fixture.storage, life1); fixture.storage->store()->putNamespaceFile(life2, kFile, "life-2-stable\n"); - Backend & backend = fixture.storage->store()->backend(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(fixture.storage->store()->poolBackendPtr()); + CasOperation op = requests.admit(); const Layout & layout = fixture.storage->store()->layout(); const String life1_key = layout.namespaceFileKey(life1, kFile); const String life2_key = layout.namespaceFileKey(life2, kFile); ASSERT_TRUE(std::filesystem::exists(nativeKeyUnder(fixture.object_storage, life2_key))); - const HeadResult life2_before = backend.head(life2_key); - ASSERT_TRUE(life2_before.exists); - const auto life2_body_before = backend.get(life2_key); + const auto life2_before = op.head(life2_key, Retry::standard()); + ASSERT_TRUE(life2_before.has_value()); + const auto life2_body_before = op.read(life2_key, Retry::standard()); ASSERT_TRUE(life2_body_before.has_value()); ASSERT_EQ(life2_body_before->bytes, "life-2-stable\n"); @@ -209,17 +209,17 @@ TEST(CASNamespaceFileReadContract, DelayedInlineFinalizeCannotChangeSuccessorTok EXPECT_NE(e.message().find("retrying later"), String::npos); } - const HeadResult life2_after = backend.head(life2_key); - ASSERT_TRUE(life2_after.exists); - EXPECT_EQ(life2_after.token, life2_before.token); - const auto life2_body_after = backend.get(life2_key); + const auto life2_after = op.head(life2_key, Retry::standard()); + ASSERT_TRUE(life2_after.has_value()); + EXPECT_EQ(life2_after->etag, life2_before->etag); + const auto life2_body_after = op.read(life2_key, Retry::standard()); ASSERT_TRUE(life2_body_after.has_value()); EXPECT_EQ(life2_body_after->bytes, "life-2-stable\n"); if (!stale_failure) { ASSERT_TRUE(std::filesystem::exists(nativeKeyUnder(fixture.object_storage, life1_key))); - const auto life1_body = backend.get(life1_key); + const auto life1_body = op.read(life1_key, Retry::standard()); ASSERT_TRUE(life1_body.has_value()); EXPECT_EQ(life1_body->bytes, "life-1-delayed\n"); } diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index 23ce4eba8341..605a78cd8dbe 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -75,15 +75,11 @@ class RenewalCounterBackend final : public InMemoryBackend } }; -CasRequestBudget renewalCounterBudget(uint32_t max_attempts = 2) +CasRequestBudget renewalCounterBudget(uint32_t /*max_attempts*/ = 2) { return CasRequestBudget{ .attempt_timeout_ms = 10, - .operation_deadline_ms = 500, - .max_attempts = max_attempts, .lease_safety_margin_ms = 20, - .retry_initial_backoff_ms = 0, - .retry_max_backoff_ms = 0, }; } @@ -306,10 +302,12 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) const RootNamespace ns{"test/tbl"}; const String P = "republish-payload-audit"; + DB::Cas::tests::OperationForTest head_op(*b); + /// 1. Publish ref r1 -> token A referenced; drop it; ONE GC round condemns A (retired, not deleted). publishOneBlobPart(s, ns, "r1", P); - const HeadResult hA = b->head(s->layout().blobKey(idOf(P))); - ASSERT_TRUE(hA.exists); + const auto hA = (*head_op).head(s->layout().blobKey(idOf(P)), Retry::standard()); + ASSERT_TRUE(hA.has_value()); s->dropRef(ns, "r1"); s->renewWatermarkOnce(); @@ -326,9 +324,10 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) /// 2. RESURRECT: r2 dedup-hits P while A is condemned -> mints a fresh incarnation B; drop it too. publishOneBlobPart(s, ns, "r2", P); - const HeadResult hB = b->head(s->layout().blobKey(idOf(P))); - ASSERT_TRUE(hB.exists); - ASSERT_NE(hB.token.value, hA.token.value) << "republication must mint a new incarnation token B"; + const auto hB = (*head_op).head(s->layout().blobKey(idOf(P)), Retry::standard()); + ASSERT_TRUE(hB.has_value()); + ASSERT_NE(PersistedEtag::capture(hB->etag).value, PersistedEtag::capture(hA->etag).value) + << "republication must mint a new incarnation token B"; s->dropRef(ns, "r2"); s->renewWatermarkOnce(); @@ -360,14 +359,13 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) std::copy_if(seen.begin(), seen.end(), std::back_inserter(replaced_events), [&](const CasEvent & e){ return is_this_blob(e) && e.type == CasEventType::BlobRetireReplaced; }); ASSERT_EQ(replaced_events.size(), 1u) << "exactly one blob_retire_replaced for the supersede"; - /// The event's token text is dialect-qualified ("emulated:", matching `Incarnation::render` - /// and `PersistedIncarnation`'s wire word) -- `Token::value` alone (from the legacy `head()` this - /// test reads hA/hB through) is only the bare value. - EXPECT_EQ(replaced_events[0].token, "emulated:" + hB.token.value) + /// The event's token text is dialect-qualified ("emulated:", matching `Etag::render` + /// and `PersistedEtag`'s wire word). + EXPECT_EQ(replaced_events[0].token, hB->etag.render()) << "the event's own token is the fresh CURRENT token B"; ASSERT_TRUE(replaced_events[0].detail.count("superseded_token")); EXPECT_FALSE(replaced_events[0].detail.at("superseded_token").empty()); - EXPECT_EQ(replaced_events[0].detail.at("superseded_token"), "emulated:" + hA.token.value) + EXPECT_EQ(replaced_events[0].detail.at("superseded_token"), hA->etag.render()) << "superseded_token must name the stale token (A) that republication replaced"; EXPECT_EQ(replaced_after - replaced_before, 1u) << "CASGCRetireReplaced increments exactly once"; @@ -385,7 +383,7 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) const auto it = std::find_if(retired.begin(), retired.end(), [&](const RetiredEntry & e){ return e.kind == ObjectKind::Blob && e.ref == DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of(P))}; }); ASSERT_NE(it, retired.end()) << "the superseded entry must be present in the current retired set"; - EXPECT_EQ(it->token.value, hB.token.value) << "the persisted entry names the fresh CURRENT token B"; + EXPECT_EQ(it->token.value, PersistedEtag::capture(hB->etag).value) << "the persisted entry names the fresh CURRENT token B"; EXPECT_EQ(it->size, P.size()) << "supersede must persist the LOGICAL size (payload length, header stripped), matching what " "a fresh condemn of the same blob would carry -- not the raw physical (header-included) size"; diff --git a/src/Disks/tests/gtest_cas_operation_gate.cpp b/src/Disks/tests/gtest_cas_operation_gate.cpp index 6d16c3946fae..2dbddf6c5f54 100644 --- a/src/Disks/tests/gtest_cas_operation_gate.cpp +++ b/src/Disks/tests/gtest_cas_operation_gate.cpp @@ -113,13 +113,14 @@ const std::string kSrid = "test"; /// gtest_cas_lifecycle_condition.cpp's helper. void fenceOutMount(DB::Cas::Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, DB::Cas::Retry::standard()); ASSERT_TRUE(got.has_value()); DB::Cas::MountLease m = DB::Cas::decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, DB::Cas::encodeMountLease(m), got->token).outcome, - DB::Cas::PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(mount_key, DB::Cas::encodeMountLease(m), got->etag, DB::Cas::Retry::standard()))); } } @@ -420,7 +421,7 @@ TEST(CASOperationGate, RemoveThrowsDuringTransientAndDrainsAfterRecovery) << "the gap message must name the transient (auto-recovering) condition: " << gap_msg; /// The lease is restored: the disk self-remounts a fresh incarnation and auto-recovers to Live. - fenceOutMount(pool->backend(), pool->layout().mountKey(kSrid)); + fenceOutMount(*pool->poolBackendPtr(), pool->layout().mountKey(kSrid)); ASSERT_TRUE(pool->tryRemountOnce()) << "the self-remount must reclaim a fresh incarnation"; ASSERT_EQ(pool->lifecycle(), PoolLifecycle::Live) << "the pool must auto-recover to Live"; diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index 8b2e8d4478ac..cc46229aafdd 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -28,6 +28,12 @@ ManifestRef ref(uint64_t seq, uint64_t inst) return ManifestRef{.writer_epoch = kWriterEpoch, .build_sequence = seq, .manifest_ordinal = static_cast(inst)}; } +bool headExists(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + /// The §6 deletion premise (`manifestDeletionPremise`) is a SECOND precondition on every deletion below, /// alongside the watermark eligibility these tests are about: a manifest of an epoch-`E` build is /// deletable only once the namespace's sealed fold cursor sits in an epoch strictly above `E`. Tests @@ -51,11 +57,11 @@ void seedEmptyRecoveryAuthority(InMemoryBackend & backend, const Layout & layout [&](const CatalogEntry & candidate) { return candidate.ns == ns; }); ASSERT_NE(entry, catalog.catalog.entries.end()); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation); - ASSERT_EQ(backend.putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = std::optional{kWriterEpoch}, .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), Retry::standard()))); } /// Replaces the catalog row immediately before its second read after arming. The legacy orphan path @@ -169,7 +175,7 @@ TEST(CASOrphanManifestSweep, EligibleAndUnownedIsDeleted) seedEmptyRecoveryAuthority(*backend, store->layout(), ns); sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}); - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } /// The orphan sweep must not turn a forged same-id snapshot at an OLDER `EpochSeal` into an empty owner @@ -205,11 +211,11 @@ TEST(CASOrphanManifestSweep, CheckpointSnapshotAtOlderEpochSealSkipsDeletion) applyRefLogTxn(through_seal, birth); applyRefLogTxn(through_seal, seal_txn); writeRefSnapshotRaw(*backend, layout, snapshotOf(through_seal, ns.string())); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{2, 1}, .checkpoint_snapshot_id = RefTxnId{1, 2}, - .last_epoch_seal = RefTxnId{2, 1}})).outcome, PutOutcome::Done); + .last_epoch_seal = RefTxnId{2, 1}}), Retry::standard()))); const ManifestRef candidate = ref(5, 0xAC); const String candidate_key = layout.manifestKey(ManifestId{ns, candidate}); @@ -219,7 +225,7 @@ TEST(CASOrphanManifestSweep, CheckpointSnapshotAtOlderEpochSealSkipsDeletion) std::vector warnings; EXPECT_EQ(sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}, &warnings), 0u); - EXPECT_TRUE(backend->head(candidate_key).exists); + EXPECT_TRUE(headExists(*backend, candidate_key)); ASSERT_FALSE(warnings.empty()); } @@ -235,7 +241,7 @@ TEST(CASOrphanManifestSweep, OwnedBodyIsSkipped) setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, 6); sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } /// GC-WEDGE regression (2026-07-10): a COMMITTED ref that has been DROPPED but whose removal `-1` is NOT @@ -265,7 +271,7 @@ TEST(CASOrphanManifestSweep, PendingCommittedRemovalBodyIsSkipped) setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, 6); // 6 > 5 => prefix eligible sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists) + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))) << "a dropped-but-unsealed committed manifest body must survive the sweep (delete-after-sealed-" "decrements) — else the removal-fold clamps forever on the missing body (GC-WEDGE-2026-07-10)"; } @@ -322,7 +328,7 @@ TEST(CASOrphanManifestSweep, NoWatermarkIsNotAuthority) writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); // No setWatermarkMinActive — no durable fact => not eligible. sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } TEST(CASOrphanManifestSweep, CursorPageDeletesEligibleUnownedBody) @@ -340,7 +346,7 @@ TEST(CASOrphanManifestSweep, CursorPageDeletesEligibleUnownedBody) const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget*/100, /*delete_budget*/10); EXPECT_GE(result.listed, 1u); EXPECT_EQ(result.deleted, 1u); - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } TEST(CASOrphanManifestSweep, CursorPageRespectsDeleteBudget) @@ -359,8 +365,8 @@ TEST(CASOrphanManifestSweep, CursorPageRespectsDeleteBudget) const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget*/100, /*delete_budget*/1); EXPECT_EQ(result.deleted, 1u); - const bool first_exists = backend->head(store->layout().manifestKey(ManifestId{ns, r1})).exists; - const bool second_exists = backend->head(store->layout().manifestKey(ManifestId{ns, r2})).exists; + const bool first_exists = headExists(*backend, store->layout().manifestKey(ManifestId{ns, r1})); + const bool second_exists = headExists(*backend, store->layout().manifestKey(ManifestId{ns, r2})); EXPECT_NE(first_exists, second_exists); } @@ -380,7 +386,7 @@ TEST(CASOrphanManifestSweep, CursorPageDeletesObservedBodyWhenCatalogOmitsNamesp const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); EXPECT_EQ(result.deleted, 1u); - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } /// The candidate body and token must be frozen before the later catalog cut. A concurrent same-key @@ -403,7 +409,7 @@ TEST(CASOrphanManifestSweep, CursorPageCannotDeleteManifestReplacedAfterObservat EXPECT_TRUE(backend->didReplace()); EXPECT_EQ(result.deleted, 0u); - EXPECT_TRUE(backend->head(key).exists); + EXPECT_TRUE(headExists(*backend, key)); } /// Any duplicate current life id makes the catalog-to-physical join ambiguous. The cursor page is @@ -421,7 +427,7 @@ TEST(CASOrphanManifestSweep, CursorPageRefusesAmbiguousCatalogLifeIndex) seedConsumedSealCursor(*backend, store->layout(), ns); seedEmptyRecoveryAuthority(*backend, store->layout(), ns); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CasRefCatalog::Snapshot before = CasRefCatalog::read(op, store->layout()); RefCatalog damaged = before.catalog; CatalogEntry duplicate = damaged.entries.front(); @@ -429,13 +435,13 @@ TEST(CASOrphanManifestSweep, CursorPageRefusesAmbiguousCatalogLifeIndex) damaged.entries.push_back(duplicate); std::sort(damaged.entries.begin(), damaged.entries.end(), [](const CatalogEntry & lhs, const CatalogEntry & rhs) { return lhs.ns.string() < rhs.ns.string(); }); - ASSERT_TRUE(before.incarnation.has_value()); + ASSERT_TRUE(before.etag.has_value()); ASSERT_TRUE(std::holds_alternative( - op.replace(store->layout().refCatalogKey(), encodeRefCatalog(damaged), *before.incarnation, Retry::standard()))); + op.replace(store->layout().refCatalogKey(), encodeRefCatalog(damaged), *before.etag, Retry::standard()))); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); }); - EXPECT_TRUE(backend->head(key).exists); + EXPECT_TRUE(headExists(*backend, key)); } TEST(CASOrphanManifestSweep, CursorPageSkipsOwnedBody) @@ -450,7 +456,7 @@ TEST(CASOrphanManifestSweep, CursorPageSkipsOwnedBody) const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget*/100, /*delete_budget*/10); EXPECT_EQ(result.deleted, 0u); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } /// A catalog-named life cannot be treated as an empty table merely because its mandatory recovery @@ -467,13 +473,13 @@ TEST(CASOrphanManifestSweep, MissingRequiredCheckpointSuppressesDestructiveDecis setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, 6); seedConsumedSealCursor(*backend, store->layout(), ns); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); ASSERT_FALSE(readCkpt(op, store->layout(), NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation))); sweepNamespace(*store, ns, BuildPrefix{.writer_epoch = kWriterEpoch, .build_sequence = 5}); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists) + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))) << "without the exact _ckpt required by a Live catalog row, the sweep must retain rather than " "derive an empty owner set"; } @@ -489,7 +495,7 @@ TEST(CASOrphanManifestSweep, EpochSealFoldCursorCrossesTailByExactDecodedSuccess auto store = openPoolForTest(backend); const RootNamespace ns{"00/seal-cursor-tail@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); @@ -505,27 +511,27 @@ TEST(CASOrphanManifestSweep, EpochSealFoldCursorCrossesTailByExactDecodedSuccess writeSealAt(*backend, store->layout(), ns, RefTxnId{5, 1}, RefTxnId{4, 1}); writeSealAt(*backend, store->layout(), ns, RefTxnId{6, 1}, RefTxnId{5, 1}); for (uint64_t epoch = 3; epoch <= 6; ++epoch) - ASSERT_TRUE(backend->head(store->layout().refLogKey(life, RefTxnId{epoch, 1})).exists) + ASSERT_TRUE(headExists(*backend, store->layout().refLogKey(life, RefTxnId{epoch, 1}))) << "fixture must deposit every intermediate exact successor in the catalog life"; writeTxnAt(*backend, store->layout(), ns, RefTxnId{7, 1}, {ownerTransitionOp(RefOwnerBinding{RefOwnerKind::Committed, "dropped", removed}, std::nullopt)}, RefTxnId{6, 1}); writeSealAt(*backend, store->layout(), ns, RefTxnId{7, 2}); writeManifestRaw(*backend, store->layout(), ns, unowned, {blobEntryFor("unowned", DB::UInt128(0xA2))}); - ASSERT_EQ(backend->putIfAbsent(store->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(store->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{7, 2}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = RefTxnId{7, 2}})).outcome, PutOutcome::Done); + .last_epoch_seal = RefTxnId{7, 2}}), Retry::standard()))); setWatermarkMinActive(*backend, store->layout(), kServerRoot, kWriterEpoch, 7); seedFoldCursorForTest(*backend, store->layout(), ns, RefTxnId{2, 2}); const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); EXPECT_EQ(result.deleted, 1u); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, removed})).exists) + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, removed}))) << "the exact successor of the folded epoch seal contains this body's unconsumed -1"; - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, unowned})).exists) + EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, unowned}))) << "an unrelated eligible body must still drain; retaining it would mask a geometry failure"; } @@ -538,16 +544,15 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip auto store = openPoolForTest(backend); const RootNamespace ns{"00/missing-next-epoch@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); const RefTxnId cursor{2, 2}; writeSealAt(*backend, store->layout(), ns, cursor); - const HeadResult cursor_head = backend->head(store->layout().refLogKey(life, cursor)); - ASSERT_TRUE(cursor_head.exists); - ASSERT_EQ(classifyDeleteOutcome( - backend->deleteExact(store->layout().refLogKey(life, cursor), cursor_head.token)), DeleteClass::Deleted); + const auto cursor_head = op.head(store->layout().refLogKey(life, cursor), Retry::standard()); + ASSERT_TRUE(cursor_head.has_value()); + ASSERT_EQ(op.remove(store->layout().refLogKey(life, cursor), cursor_head->etag, Retry::standard()), Removal::Removed); const ManifestRef phantom{.writer_epoch = 7, .build_sequence = 1, .manifest_ordinal = 1}; /// The codec refuses this skipped predecessor when a writer tries to create it. Inject the malformed @@ -563,19 +568,18 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip ASSERT_NE(predecessor_pos, String::npos); malformed_later_link.replace( predecessor_pos, encoded_predecessor.size(), R"("!prev_epoch":"2")"); - ASSERT_EQ(backend->putIfAbsent( - store->layout().refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, malformed_later_link)).outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(op.create( + store->layout().refLogKey(life, RefTxnId{7, 1}), sealObject(FormatId::RefLog, malformed_later_link), Retry::standard()))); writeRefSnapshotRaw(*backend, store->layout(), RefTableSnapshot{ .ns = ns.string(), .snapshot_id = RefTxnId{7, 1}, .committed = {committedRow("phantom", phantom)}, .precommits = {}}); - ASSERT_EQ(backend->putIfAbsent(store->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(store->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{7, 1}, .checkpoint_snapshot_id = RefTxnId{7, 1}, - .last_epoch_seal = RefTxnId{6, 1}})).outcome, PutOutcome::Done); + .last_epoch_seal = RefTxnId{6, 1}}), Retry::standard()))); const ManifestRef victim{.writer_epoch = 1, .build_sequence = 5, .manifest_ordinal = 1}; writeManifestRaw(*backend, store->layout(), ns, victim, {blobEntryFor("victim", DB::UInt128(0xC1))}); @@ -585,7 +589,7 @@ TEST(CASOrphanManifestSweep, MissingImmediateEpochAfterCleanedCursorCannotBeSkip const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); EXPECT_EQ(result.deleted, 0u); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, victim})).exists); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, victim}))); } /// Control for the cleaned-cursor path: the exact immediately-next epoch head exists and names the @@ -599,16 +603,15 @@ TEST(CASOrphanManifestSweep, CleanedCursorCrossesOnlyThroughExactImmediateEpochH auto store = openPoolForTest(backend); const RootNamespace ns{"00/exact-next-epoch@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CatalogEntry entry = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry.ns, entry.incarnation); const RefTxnId cursor{2, 2}; writeSealAt(*backend, store->layout(), ns, cursor); - const HeadResult cursor_head = backend->head(store->layout().refLogKey(life, cursor)); - ASSERT_TRUE(cursor_head.exists); - ASSERT_EQ(classifyDeleteOutcome( - backend->deleteExact(store->layout().refLogKey(life, cursor), cursor_head.token)), DeleteClass::Deleted); + const auto cursor_head = op.head(store->layout().refLogKey(life, cursor), Retry::standard()); + ASSERT_TRUE(cursor_head.has_value()); + ASSERT_EQ(op.remove(store->layout().refLogKey(life, cursor), cursor_head->etag, Retry::standard()), Removal::Removed); const ManifestRef removed{.writer_epoch = 1, .build_sequence = 5, .manifest_ordinal = 1}; writeTxnAt(*backend, store->layout(), ns, RefTxnId{3, 1}, @@ -618,11 +621,11 @@ TEST(CASOrphanManifestSweep, CleanedCursorCrossesOnlyThroughExactImmediateEpochH {ownerTransitionOp(RefOwnerBinding{RefOwnerKind::Committed, "absent-anchor", absent_anchor}, std::nullopt)}); writeRefSnapshotRaw(*backend, store->layout(), RefTableSnapshot{ .ns = ns.string(), .snapshot_id = RefTxnId{3, 2}, .committed = {}, .precommits = {}}); - ASSERT_EQ(backend->putIfAbsent(store->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(op.create(store->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{3, 2}, .checkpoint_snapshot_id = RefTxnId{3, 2}, - .last_epoch_seal = cursor})).outcome, PutOutcome::Done); + .last_epoch_seal = cursor}), Retry::standard()))); const ManifestRef unowned{.writer_epoch = 1, .build_sequence = 6, .manifest_ordinal = 1}; writeManifestRaw(*backend, store->layout(), ns, removed, {blobEntryFor("removed", DB::UInt128(0xC2))}); @@ -633,8 +636,8 @@ TEST(CASOrphanManifestSweep, CleanedCursorCrossesOnlyThroughExactImmediateEpochH const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); EXPECT_EQ(result.deleted, 1u); - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, removed})).exists); - EXPECT_FALSE(backend->head(store->layout().manifestKey(ManifestId{ns, unowned})).exists); + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, removed}))); + EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, unowned}))); } /// The catalog row used to obtain coverage and the life used to recover ownership must be ONE frozen @@ -646,7 +649,7 @@ TEST(CASOrphanManifestSweep, LaterCatalogCutCannotSpliceOwnershipAuthority) auto store = openPoolForTest(backend); const RootNamespace ns{"00/frozen-catalog-cut@cas@"}; fixture::admitLive(*backend, store->layout(), ns); - CasOperation op = store->gcRequests().admit(); + CasOperation op = store->openRequests().admit(); const CatalogEntry predecessor = CasRefCatalog::read(op, store->layout()).catalog.entries.front(); const ManifestRef r = ref(5, 0xB1); @@ -663,7 +666,7 @@ TEST(CASOrphanManifestSweep, LaterCatalogCutCannotSpliceOwnershipAuthority) EXPECT_FALSE(backend->didSwitch()) << "the sweep must not resolve a second catalog cut after it starts using the frozen entry"; - EXPECT_TRUE(backend->head(store->layout().manifestKey(ManifestId{ns, r})).exists) + EXPECT_TRUE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))) << "the committed predecessor manifest must remain protected by the same frozen authority cut"; } diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index 664b9763b650..5dda01e60328 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -29,7 +29,8 @@ ManifestRef candidateRef() bool manifestExists(Backend & backend, const Layout & layout, const ManifestId & id) { - return backend.head(layout.manifestKey(id)).exists; + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(layout.manifestKey(id), Retry::standard()).has_value(); } bool activeSourceExists(CasOperation & op, const Layout & layout, const UInt128 & source_id) @@ -177,18 +178,18 @@ ReadyFixture makeReadyFixture() /// Seed the exact S42 precondition: the candidate manifest's `+1` edges are already in the adopted /// run, yet the recovered owner view does not name the body. Four blobs also have another source. - const auto state_got = f.backend->get(f.store->layout().gcStateKey()); + CasOperation seed_op = f.store->openRequests().admit(); + const auto state_got = seed_op.read(f.store->layout().gcStateKey(), Retry::standard()); EXPECT_TRUE(state_got.has_value()); GcState state = decodeGcState(state_got->bytes); - const auto parent_got = f.backend->get( - f.store->layout().foldSealKey(state.snap_generation, state.snap_attempt)); + const auto parent_got = seed_op.read( + f.store->layout().foldSealKey(state.snap_generation, state.snap_attempt), Retry::standard()); EXPECT_TRUE(parent_got.has_value()); CasFoldSeal seal = decodeFoldSeal(parent_got->bytes); const uint64_t new_generation = state.snap_generation + 1; const uint64_t new_attempt = state.snap_attempt + 1000; std::vector runs; RetiredMergeResult retired; - CasOperation seed_op = f.store->gcRequests().admit(); foldDeltasIntoGeneration( seed_op, f.store->layout(), seal.blob_target_runs, new_generation, new_attempt, /*shard=*/0, std::move(seeded_edges), runs, @@ -202,7 +203,7 @@ ReadyFixture makeReadyFixture() seed_op, f.store->layout().foldSealKey(new_generation, new_attempt), encodeFoldSeal(seal)); state.snap_generation = new_generation; state.snap_attempt = new_attempt; - f.backend->putOverwrite(f.store->layout().gcStateKey(), encodeGcState(state), state_got->token); + seed_op.replace(f.store->layout().gcStateKey(), encodeGcState(state), state_got->etag, Retry::standard()); f.backend->watched_manifest_key = f.store->layout().manifestKey(f.candidate); f.backend->watched_source_id = sourceEdgeId(f.candidate, "blob-0"); @@ -229,7 +230,7 @@ TEST(CASOrphanNomination, RetiresExactManifestSourcesBeforeDelete) EXPECT_FALSE(manifestExists(*f.backend, f.store->layout(), f.candidate)); EXPECT_TRUE(f.backend->source_absent_when_delete_started) << "the adopted in-degree run must retire the manifest source before exact deletion begins"; - CasOperation op = f.store->gcRequests().admit(); + CasOperation op = f.store->openRequests().admit(); for (size_t i = 0; i < f.blobs.size(); ++i) { EXPECT_FALSE(activeSourceExists( @@ -252,9 +253,10 @@ TEST(CASOrphanNomination, RetiresExactManifestSourcesBeforeDelete) TEST(CASOrphanNomination, CorruptManifestIsRetainedAndSurfaced) { ReadyFixture f = makeReadyFixture(); - const auto got = f.backend->get(f.backend->watched_manifest_key); + DB::Cas::tests::OperationForTest op(f.backend); + const auto got = (*op).read(f.backend->watched_manifest_key, Retry::standard()); ASSERT_TRUE(got.has_value()); - f.backend->putOverwrite(f.backend->watched_manifest_key, "not a sealed manifest", got->token); + (*op).replace(f.backend->watched_manifest_key, "not a sealed manifest", got->etag, Retry::standard()); std::optional orphan_sweep; f.gc->setPhaseSink([&](const GcPhaseRecord & rec) { if (rec.phase == "orphan_sweep") orphan_sweep = rec; }); @@ -264,7 +266,10 @@ TEST(CASOrphanNomination, CorruptManifestIsRetainedAndSurfaced) EXPECT_TRUE(report.acquired_lease); ASSERT_TRUE(orphan_sweep.has_value()); EXPECT_EQ(orphan_sweep->metrics.at("undecodable"), 1u); - EXPECT_TRUE(f.backend->head(f.backend->watched_manifest_key).exists); + { + DB::Cas::tests::OperationForTest head_op(f.backend); + EXPECT_TRUE((*head_op).head(f.backend->watched_manifest_key, Retry::standard()).has_value()); + } } /// Manifest identities are immutable. A changed token at the same key is illegal ABA, not an ordinary @@ -275,7 +280,10 @@ TEST(CASOrphanNomination, TokenAbaIsRetainedAndSurfaced) f.backend->replace_manifest_before_delete = true; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { runRegularRoundReclaiming(*f.gc); }); - EXPECT_TRUE(f.backend->head(f.backend->watched_manifest_key).exists); + { + DB::Cas::tests::OperationForTest head_op(f.backend); + EXPECT_TRUE((*head_op).head(f.backend->watched_manifest_key, Retry::standard()).has_value()); + } } /// Nomination PLANNING itself is gated on `!suppress_destructive` diff --git a/src/Disks/tests/gtest_cas_part_folder_access.cpp b/src/Disks/tests/gtest_cas_part_folder_access.cpp index 03d7ef33186b..6462d0ce4878 100644 --- a/src/Disks/tests/gtest_cas_part_folder_access.cpp +++ b/src/Disks/tests/gtest_cas_part_folder_access.cpp @@ -108,9 +108,6 @@ class RollbackFaultBackend final : public Cas::InMemoryBackend class PromoteConflictOnceBackend final : public Cas::InMemoryBackend { public: - /// Unhide the legacy `putIfAbsent` overloads the primitive override below would otherwise hide. - using InMemoryBackend::putIfAbsent; - String fault_key_substr; int skip = 0; int fault_count = 0; @@ -118,8 +115,7 @@ class PromoteConflictOnceBackend final : public Cas::InMemoryBackend /// cleanup path ran its ref-log append at all, on a table where that append can no longer succeed. int matching_put_attempts = 0; - /// Sabotages the PRIMITIVE, which every legacy forwarder (including `putIfAbsent`) reaches too, so - /// the fault fires whichever surface issued the create. + /// Sabotages the sole write primitive, so the fault fires whichever verb (`create`/`replace`) issued it. std::expected write(const String & key, const String & bytes, const std::optional & expected_value, Cas::TransportAccess & access) override @@ -152,15 +148,12 @@ class PromoteConflictOnceBackend final : public Cas::InMemoryBackend class PromoteDefiniteFailureBackend final : public Cas::InMemoryBackend { public: - /// Unhide the legacy `putIfAbsent` overloads the primitive override below would otherwise hide. - using InMemoryBackend::putIfAbsent; - String fault_key_substr; int skip = 0; int fault_count = 0; int matching_put_attempts = 0; - /// Sabotages the PRIMITIVE, which every legacy forwarder (including `putIfAbsent`) reaches too. + /// Sabotages the sole write primitive, so the fault fires whichever verb (`create`/`replace`) issued it. std::expected write(const String & key, const String & bytes, const std::optional & expected_value, Cas::TransportAccess & access) override @@ -419,14 +412,15 @@ TEST(CASPartFolderAccess, PublishEntriesAbandonsBuildOnPromoteFailure) /// fresh id to get past the foreign object would sort above it. String greatest_key; size_t foreign_objects = 0; + DB::Cas::tests::OperationForTest scan(*backend); for (String cursor;;) { - const Cas::ListPage page = backend->list(backend->fault_key_substr, cursor, 1000); + const Cas::ListPage page = (*scan).list(backend->fault_key_substr, cursor, 1000, Cas::Retry::standard()); for (const auto & listed : page.keys) { if (listed.key > greatest_key) greatest_key = listed.key; - const auto body = backend->get(listed.key); + const auto body = (*scan).read(listed.key, Cas::Retry::standard()); if (body && body->bytes.find("_FOREIGN_DIFFERENT") != String::npos) ++foreign_objects; } @@ -436,7 +430,7 @@ TEST(CASPartFolderAccess, PublishEntriesAbandonsBuildOnPromoteFailure) } EXPECT_EQ(foreign_objects, 1u) << "the foreign object must still own the key it took"; ASSERT_FALSE(greatest_key.empty()); - const auto greatest_body = backend->get(greatest_key); + const auto greatest_body = (*scan).read(greatest_key, Cas::Retry::standard()); ASSERT_TRUE(greatest_body.has_value()); EXPECT_NE(greatest_body->bytes.find("_FOREIGN_DIFFERENT"), String::npos) << "the foreign occupant must still be the highest id in this table's stream: a log object above " @@ -553,7 +547,10 @@ TEST(CASPartFolderAccess, PrepareThenAbortAppendsThePrecommitRemoval) /// The precommit BODY survives (delete-after-sealed-decrements) -- the removal queues GC's `-1`, /// it does not writer-delete the manifest. Mirrors /// `CASPartWriteTxn.AbandonAppendsPrecommitRemovalAndKeepsLivePrecommitBody`. - EXPECT_TRUE(backend->head(store->layout().manifestKey(id)).exists); + { + DB::Cas::tests::OperationForTest op(*backend); + EXPECT_TRUE((*op).head(store->layout().manifestKey(id), Cas::Retry::standard()).has_value()); + } } /// A forgotten terminal must be impossible, not merely discouraged: `~PartWriteTxn` only retires the diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 570407814093..afec5b1bfbe1 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -182,20 +182,10 @@ ManifestId publishOneBlobPart( class HeadThenDeleteOnceBackend final : public DB::Cas::Backend { public: - HeadThenDeleteOnceBackend(BackendPtr inner_, String target_key_, DB::Cas::Token condemned_) - : inner(std::move(inner_)), target_key(std::move(target_key_)), condemned(condemned_) {} - - std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & meta) override { return inner->putIfAbsent(k, b, meta); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & meta) override { return inner->putOverwrite(k, b, e, meta); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & meta) override { return inner->casPut(k, b, e, meta); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } + HeadThenDeleteOnceBackend(BackendPtr inner_, String target_key_, Etag condemned_) + : inner(std::move(inner_)), target_key(std::move(target_key_)), + condemned_value(PersistedEtag::capture(condemned_).value) {} + bool supportsListTokens() const override { return inner->supportsListTokens(); } /// The fault sits on the HEAD primitive, which is the only path a writer's mandatory HEAD takes. @@ -207,7 +197,7 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend { fired = true; /// GC's single content-delete site, landing in the HEAD->GET window. - inner->remove(target_key, condemned.value, access); + inner->remove(target_key, condemned_value, access); } return observed; } @@ -227,7 +217,7 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend private: BackendPtr inner; String target_key; - DB::Cas::Token condemned; + String condemned_value; }; /// A delegating backend that counts head()/get() calls per key. Lets a test assert the promote gate @@ -241,22 +231,10 @@ class KeyCountingBackend final : public DB::Cas::Backend size_t headCountFor(const String & k) const { auto it = head_counts.find(k); return it == head_counts.end() ? 0 : it->second; } size_t getCountFor(const String & k) const { auto it = get_counts.find(k); return it == get_counts.end() ? 0 : it->second; } - DB::Cas::HeadResult head(const String & k) override { ++head_counts[k]; return inner->head(k); } - std::optional get(const String & k, DB::Cas::Range r) override { ++get_counts[k]; return inner->get(k, r); } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::ListPage list(const String & pfx, const String & c, size_t l) override { return inner->list(pfx, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & meta) override { return inner->putIfAbsent(k, b, meta); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & meta) override { return inner->putOverwrite(k, b, e, meta); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & meta) override { return inner->casPut(k, b, e, meta); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The primitives count too: `Backend::probeSentinelRaw` reaches the store through them, so a - /// per-key observation on the primitive path would otherwise go uncounted. + /// Counted on the primitives: `Backend::probeSentinelRaw` reaches the store through them, and + /// `CasOperation` is the only caller of `Backend` now, so this is the one place left to count. std::optional read(const String & key, TransportAccess & access) override { ++get_counts[key]; @@ -376,7 +354,8 @@ TEST(CASPartWrite, RacingWritersBothHeadMissAndPublishEquivalentBodies) << "both equivalent writers may publish after racing absent observations"; EXPECT_EQ(first->dependencyProof(ref), BlobDependencyProof::Materialized); EXPECT_EQ(second->dependencyProof(ref), BlobDependencyProof::Materialized); - const auto stored = backend->get(store->layout().blobKey(ref)); + OperationForTest op(*backend); + const auto stored = (*op).read(store->layout().blobKey(ref), Retry::once()); ASSERT_TRUE(stored.has_value()); EXPECT_EQ(stored->bytes.substr(store->poolMeta().blob_header_len), payload); } @@ -401,7 +380,8 @@ TEST(CASPartWrite, WrongSizeSourcePublishesNothing) { build->putBlob(ref, std::move(source)); }); - EXPECT_FALSE(backend->head(store->layout().blobKey(ref)).exists); + OperationForTest op(*backend); + EXPECT_FALSE((*op).head(store->layout().blobKey(ref), Retry::once()).has_value()); } TEST(CASPartWriteTxn, PutBlobWritesEnvelopeWithFixedHeader) @@ -413,7 +393,8 @@ TEST(CASPartWriteTxn, PutBlobWritesEnvelopeWithFixedHeader) auto ref = build->putBlob(idOf("hello world"), BlobSource::fromString("hello world")); EXPECT_EQ(ref.size, 11u); - auto raw = b->get(s->layout().blobKey(ref.ref)); + OperationForTest op(*b); + auto raw = (*op).read(s->layout().blobKey(ref.ref), Retry::once()); ASSERT_TRUE(raw.has_value()); auto h = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); EXPECT_EQ(h.header_len, s->poolMeta().blob_header_len); /// 256 @@ -467,7 +448,10 @@ TEST(CASPartWriteTxn, PutBlobDedupSecondWriterAdopts) /// First writer publishes under its durable precommit edge. auto build_a = precommittedBuildForPayload(s, RootNamespace{"srv/tbl-a"}, "ref_a", "dup"); auto ref_a = build_a->putBlob(idOf("dup"), BlobSource::fromString("dup")); - const Token token_a = b->head(s->layout().blobKey(ref_a.ref)).token; + OperationForTest op(*b); + const auto head_a = (*op).head(s->layout().blobKey(ref_a.ref), Retry::once()); + ASSERT_TRUE(head_a.has_value()); + const Etag token_a = head_a->etag; /// Second writer ADOPTS — the adopt must happen under a durable precommit edge (EDGE-BEFORE-OBSERVE: /// stageManifest -> precommitAdd -> putBlob), so give build_b the wiring order. @@ -479,7 +463,9 @@ TEST(CASPartWriteTxn, PutBlobDedupSecondWriterAdopts) EXPECT_EQ(ref_b.ref, ref_a.ref); /// A's incarnation survives — the second writer adopts, nothing was overwritten. - EXPECT_EQ(b->head(s->layout().blobKey(ref_a.ref)).token, token_a); + const auto head_a_after = (*op).head(s->layout().blobKey(ref_a.ref), Retry::once()); + ASSERT_TRUE(head_a_after.has_value()); + EXPECT_EQ(head_a_after->etag, token_a); } /// Task 3 (spec §meta-protocols v3): the writer's dedup gate no longer consults the RetireView for the @@ -584,7 +570,10 @@ TEST(CASPartWriteTxn, PutBlobAdoptsWhenMetaCleanNoRetireView) raw_body += payload; writeRawBlobBody(*b, s->layout(), hash, raw_body); writeMetaClean(*b, s->layout(), hash, payload.size()); - const Token t0 = b->head(blob_key).token; + OperationForTest op(*b); + const auto head0 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(head0.has_value()); + const Etag t0 = head0->etag; /// Adopt must happen under a durable precommit edge (EDGE-BEFORE-OBSERVE), mirroring /// PutBlobDedupSecondWriterAdopts above. @@ -596,7 +585,9 @@ TEST(CASPartWriteTxn, PutBlobAdoptsWhenMetaCleanNoRetireView) EXPECT_EQ(ref.ref, id); /// Adopted: the pre-seeded incarnation survives untouched — no putOverwrite/re-upload happened. - EXPECT_EQ(b->head(blob_key).token, t0); + const auto head1 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(head1.has_value()); + EXPECT_EQ(head1->etag, t0); const auto lm = loadMetaForTest(*b, s->layout(), hash); ASSERT_TRUE(lm.has_value()); @@ -655,7 +646,10 @@ TEST(CASPartWriteTxn, PutBlobRepublishesWhenMetaCondemned) writeRawBlobBody(*b, s->layout(), hash, raw_body); writeMetaClean(*b, s->layout(), hash, payload.size()); condemnMeta(*b, s->layout(), hash, /*condemn_round*/ 1); - const Token t0 = b->head(blob_key).token; + OperationForTest op(*b); + const auto head0 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(head0.has_value()); + const Etag t0 = head0->etag; /// No retire-view seeding: the replacement is decided from the metadata point-read. auto build = precommittedBuildForPayload( @@ -664,10 +658,10 @@ TEST(CASPartWriteTxn, PutBlobRepublishesWhenMetaCondemned) EXPECT_EQ(ref.ref, id); /// Resurrected: the condemned incarnation was displaced by a fresh one. - const HeadResult hr = b->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_NE(hr.token, t0) << "a condemned incarnation must be displaced by a fresh publication"; - EXPECT_EQ(b->deleteExact(blob_key, t0).kind, DeleteOutcome::Kind::TokenMismatch) + const auto hr = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(hr.has_value()); + EXPECT_NE(hr->etag, t0) << "a condemned incarnation must be displaced by a fresh publication"; + EXPECT_EQ((*op).remove(blob_key, t0, Retry::once()), Removal::Mismatch) << "the condemned token must never return (INV-NO-RETURN)"; const auto lm = loadMetaForTest(*b, s->layout(), hash); @@ -723,7 +717,8 @@ TEST(CASPartWriteTxn, PutBlobWrongSizeFailsClosed) build->putBlob(id, std::move(lying)); }); /// The cancelled stream created nothing. - EXPECT_FALSE(b->head(s->layout().blobKey(id)).exists); + OperationForTest op(*b); + EXPECT_FALSE((*op).head(s->layout().blobKey(id), Retry::once()).has_value()); } /// The happy-path upload STREAMS the source directly into the put sink — it does NOT pre-materialize the @@ -753,7 +748,8 @@ TEST(CASPartWriteTxn, PutBlobStreamsSourceOnceNoFullMaterialization) EXPECT_EQ(invocations, 1) << "happy-path upload must stream the source exactly once (no pre-materialization pass)"; /// And the object really landed with the streamed payload (at the fixed header offset). - auto raw = b->get(s->layout().blobKey(ref.ref)); + OperationForTest op(*b); + auto raw = (*op).read(s->layout().blobKey(ref.ref), Retry::once()); ASSERT_TRUE(raw.has_value()); auto h = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); EXPECT_EQ(raw->bytes.substr(h.header_len), payload); @@ -809,13 +805,16 @@ TEST(CASPartWriteTxn, PutBlobRepublishesVanishedBodyFromHeldSource) /// 1. Write payload-X via a throwaway build to create the blob; capture its token t0. BlobRef id; - Token t0; + std::optional t0; { auto s0 = openPool(b); auto build0 = precommittedBuildForPayload( s0, RootNamespace{"srv1/republish-vanished-seed"}, "part", "payload-X"); id = build0->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")).ref; - t0 = b->head(s0->layout().blobKey(id)).token; + OperationForTest op(*b); + const auto head = (*op).head(s0->layout().blobKey(id), Retry::once()); + ASSERT_TRUE(head.has_value()); + t0 = head->etag; build0->abandon(); } @@ -829,7 +828,7 @@ TEST(CASPartWriteTxn, PutBlobRepublishesVanishedBodyFromHeldSource) /// 3. Wrap the backend so the NEXT head(blob_key) returns the (present) result and THEN deletes that /// exact incarnation once — GC emptying the key underneath the writer's observation. Open a FRESH /// Pool over the hook so its retire view (refreshed at open) sees the condemnation. - auto hook = std::make_shared(b, blob_key, t0); + auto hook = std::make_shared(b, blob_key, *t0); auto s = Pool::open(hook, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); auto build = precommittedBuildForPayload( s, RootNamespace{"srv1/republish-vanished"}, "part", "payload-X"); @@ -847,17 +846,18 @@ TEST(CASPartWriteTxn, PutBlobRepublishesVanishedBodyFromHeldSource) /// 5. The blob is present again under a FRESH token, with the same payload; and the condemned token /// never returns (INV-NO-RETURN). - const HeadResult hr = b->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_NE(hr.token, t0); + OperationForTest op(*b); + const auto hr = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(hr.has_value()); + EXPECT_NE(hr->etag, *t0); - auto raw = b->get(blob_key); + auto raw = (*op).read(blob_key, Retry::once()); ASSERT_TRUE(raw.has_value()); auto h = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); EXPECT_EQ(h.header_len, s->poolMeta().blob_header_len); EXPECT_EQ(raw->bytes.substr(h.header_len), "payload-X"); - EXPECT_EQ(b->deleteExact(blob_key, t0).kind, DeleteOutcome::Kind::TokenMismatch); + EXPECT_EQ((*op).remove(blob_key, *t0, Retry::once()), Removal::Mismatch); /// The freshness meta must be reconciled to Clean too, not left stale at Condemned: the fresh /// re-upload's meta write (writeFreshMetaClean) must find and fix the pre-existing Condemned @@ -946,13 +946,14 @@ TEST(CASPartWriteTxn, PutBlobFreshMetaExhaustionThrowsRetryLater) /// The body itself landed (only .meta writes are faulted) -- confirming the failure is /// specifically the freshness marker, not the blob body. - const HeadResult hr = b->head(s->layout().blobKey(idOf(payload))); - EXPECT_TRUE(hr.exists) << "the body PUT is unaffected by the meta-only fault"; + OperationForTest op(*b); + const auto hr = (*op).head(s->layout().blobKey(idOf(payload)), Retry::once()); + EXPECT_TRUE(hr.has_value()) << "the body PUT is unaffected by the meta-only fault"; } /// INV-1 (revival-from-source): a condemned blob is NEVER read via GET to revive it. -/// putBlob on a condemned-dedup hit must re-upload from its OWN source bytes — never calling -/// backend().get(blob_key). This test counts backend GETs on the blob key and asserts zero. +/// putBlob on a condemned-dedup hit must re-upload from its OWN source bytes — never reading the +/// blob key's body. This test counts backend GETs on the blob key and asserts zero. TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) { /// A delegating backend that counts get() calls on a specific key to assert INV-1. @@ -962,23 +963,6 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) : inner(std::move(inner_)), watched_key(std::move(watched_key_)) {} size_t get_count = 0; - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - std::optional get(const String & k, DB::Cas::Range r) override - { - if (k == watched_key) - ++get_count; - return inner->get(k, r); - } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & bts, const DB::Cas::ObjectMeta & m) override { return inner->putIfAbsent(k, bts, m); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & bts, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & m) override { return inner->putOverwrite(k, bts, e, m); } - DB::Cas::CasResult casPut(const String & k, const String & bts, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, bts, e, m); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & tok) override { return inner->deleteExact(k, tok); } bool supportsListTokens() const override { return inner->supportsListTokens(); } /// `read` is a GET, so it counts on the watched key too -- otherwise the INV-1 fence below @@ -1010,7 +994,7 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) /// 1. Upload blob Y via a throwaway build; capture the incarnation t0 the way GC persists it, and /// let GC's exact-incarnation delete land before the writer's dedup hit. BlobRef id; - PersistedIncarnation t0; + PersistedEtag t0; { auto s0 = openPool(b); auto build0 = precommittedBuildForPayload( @@ -1020,8 +1004,8 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) const String seed_key = s0->layout().blobKey(id); const auto seeded = op0.head(seed_key, Retry::standard()); ASSERT_TRUE(seeded.has_value()); - t0 = PersistedIncarnation::capture(seeded->incarnation); - ASSERT_EQ(op0.remove(seed_key, seeded->incarnation, Retry::standard()), Removal::Removed); + t0 = PersistedEtag::capture(seeded->etag); + ASSERT_EQ(op0.remove(seed_key, seeded->etag, Retry::standard()), Removal::Removed); build0->abandon(); } @@ -1030,7 +1014,10 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) const String blob_key = layout.blobKey(id); injectRetire(*b, layout, /*round*/ 1, /*shard*/ 0, {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-Y"))}, .token = t0, .size = 9}}); - ASSERT_FALSE(b->head(blob_key).exists); + { + OperationForTest raw_op(*b); + ASSERT_FALSE((*raw_op).head(blob_key, Retry::once()).has_value()); + } /// 3. Open a fresh Pool over a GET-counting wrapper; the retire view sees the condemnation at open. auto counting = std::make_shared(b, blob_key); @@ -1051,8 +1038,8 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) CasOperation probe_op = probe.admit(); const auto after = probe_op.head(blob_key, Retry::standard()); ASSERT_TRUE(after.has_value()); - EXPECT_FALSE(t0.matches(after->incarnation)) << "a fresh publication must have a fresh incarnation"; - const auto raw = b->get(blob_key); + EXPECT_FALSE(t0.matches(after->etag)) << "a fresh publication must have a fresh incarnation"; + const auto raw = probe_op.read(blob_key, Retry::standard()); ASSERT_TRUE(raw.has_value()); const auto hdr = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); EXPECT_EQ(raw->bytes.substr(hdr.header_len), "payload-Y"); @@ -1068,23 +1055,6 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupPresentNeverGetsTheDyingObject) : inner(std::move(inner_)), watched_key(std::move(watched_key_)) {} size_t get_count = 0; - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - std::optional get(const String & k, DB::Cas::Range r) override - { - if (k == watched_key) - ++get_count; - return inner->get(k, r); - } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & bts, const DB::Cas::ObjectMeta & m) override { return inner->putIfAbsent(k, bts, m); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & bts, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & m) override { return inner->putOverwrite(k, bts, e, m); } - DB::Cas::CasResult casPut(const String & k, const String & bts, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, bts, e, m); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & tok) override { return inner->deleteExact(k, tok); } bool supportsListTokens() const override { return inner->supportsListTokens(); } /// `read` is a GET, so it counts on the watched key too -- otherwise the INV-1 fence below @@ -1115,13 +1085,16 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupPresentNeverGetsTheDyingObject) /// 1. Upload blob Z via a throwaway build; capture the token t0. BlobRef id; - Token t0; + std::optional t0; { auto s0 = openPool(b); auto build0 = precommittedBuildForPayload( s0, RootNamespace{"srv1/condemned-present-seed"}, "part", "payload-Z"); id = build0->putBlob(idOf("payload-Z"), BlobSource::fromString("payload-Z")).ref; - t0 = b->head(s0->layout().blobKey(id)).token; + OperationForTest seed_op(*b); + const auto head0 = (*seed_op).head(s0->layout().blobKey(id), Retry::once()); + ASSERT_TRUE(head0.has_value()); + t0 = head0->etag; build0->abandon(); } @@ -1130,7 +1103,10 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupPresentNeverGetsTheDyingObject) const String blob_key = layout.blobKey(id); /// v3: condemn via the per-hash meta (the writer's freshness point-read), object still PRESENT. condemnMeta(*b, layout, u128Of("payload-Z"), /*condemn_round*/ 1); - ASSERT_TRUE(b->head(blob_key).exists) << "blob must be PRESENT for the condemned-present path"; + { + OperationForTest raw_op(*b); + ASSERT_TRUE((*raw_op).head(blob_key, Retry::once()).has_value()) << "blob must be PRESENT for the condemned-present path"; + } /// 3. Open a fresh Pool over a GET-counting wrapper. auto counting = std::make_shared(b, blob_key); @@ -1143,10 +1119,11 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupPresentNeverGetsTheDyingObject) EXPECT_EQ(ref.ref, id); EXPECT_EQ(counting->get_count, 0u) << "INV-1: putBlob must not GET the condemned object"; - const HeadResult hr = b->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_NE(hr.token, t0) << "condemned incarnation must be displaced by a fresh token"; - const auto raw = b->get(blob_key); + OperationForTest raw_op(*b); + const auto hr = (*raw_op).head(blob_key, Retry::once()); + ASSERT_TRUE(hr.has_value()); + EXPECT_NE(hr->etag, *t0) << "condemned incarnation must be displaced by a fresh token"; + const auto raw = (*raw_op).read(blob_key, Retry::once()); ASSERT_TRUE(raw.has_value()); const auto hdr = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); EXPECT_EQ(raw->bytes.substr(hdr.header_len), "payload-Z"); @@ -1278,7 +1255,10 @@ TEST(CASPartWriteTxn, PromoteTrustsAdoptedLeafEvenIfBackendRaced) seed->promote(seed_ns, "part", seed->buildId(), seed_manifest); } const String blob_key = s->layout().blobKey(streamRefOf("payload-RACE")); - const Token t0 = b->head(blob_key).token; + OperationForTest op(*b); + const auto head0 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(head0.has_value()); + const Etag t0 = head0->etag; auto build = startBuildFor(s, ns, "part_1"); const ManifestEntry entry = blobManifestEntryStreaming("data.bin", "payload-RACE"); @@ -1286,8 +1266,8 @@ TEST(CASPartWriteTxn, PromoteTrustsAdoptedLeafEvenIfBackendRaced) const ManifestId id = build->stageManifest({entry}); build->precommitAdd(ns, "part_1", id); - ASSERT_EQ(b->deleteExact(blob_key, t0).kind, DeleteOutcome::Kind::Deleted); - ASSERT_FALSE(b->head(blob_key).exists); + ASSERT_EQ((*op).remove(blob_key, t0, Retry::once()), Removal::Removed); + ASSERT_FALSE((*op).head(blob_key, Retry::once()).has_value()); EXPECT_NO_THROW(build->promote(ns, "part_1", build->buildId(), id)); EXPECT_TRUE(s->resolveRef(ns, "part_1").has_value()); @@ -1351,7 +1331,10 @@ TEST(CASPartWriteTxn, MissingDependencyProofFailsClosed) seed->promote(seed_ns, "part", seed->buildId(), seed_manifest); } const String blob_key = s->layout().blobKey(streamRefOf("payload-NODEP")); - const Token t0 = b->head(blob_key).token; + OperationForTest op(*b); + const auto head0 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(head0.has_value()); + const Etag t0 = head0->etag; auto build = startBuildFor(s, ns, "part_1"); const ManifestEntry entry = blobManifestEntryStreaming("data.bin", "payload-NODEP"); @@ -1371,7 +1354,9 @@ TEST(CASPartWriteTxn, MissingDependencyProofFailsClosed) "no dependency proof"); EXPECT_FALSE(s->resolveRef(ns, "part_1").has_value()); /// The pool blob was never touched (no probe, no displacement). - EXPECT_EQ(b->head(blob_key).token, t0); + const auto head1 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(head1.has_value()); + EXPECT_EQ(head1->etag, t0); } TEST(CASPartWriteTxn, PromoteRevalidatesBlobPresenceFailClosed) @@ -1451,14 +1436,15 @@ TEST(CASPartWriteTxn, AbandonRemovesStagedDebrisAndDisables) const ManifestId mid = build->stageManifest({blobManifestEntry("f", "kept")}); /// The staged manifest body and the blob are present before abandon. - EXPECT_TRUE(b->head(s->layout().blobKey(blob_ref.ref)).exists); - EXPECT_TRUE(b->head(s->layout().manifestKey(mid)).exists); + OperationForTest op(*b); + EXPECT_TRUE((*op).head(s->layout().blobKey(blob_ref.ref), Retry::once()).has_value()); + EXPECT_TRUE((*op).head(s->layout().manifestKey(mid), Retry::once()).has_value()); build->abandon(); /// Blob stays (debris — full GC reclaims it). The staged manifest debris is best-effort cleaned now. - EXPECT_TRUE(b->head(s->layout().blobKey(blob_ref.ref)).exists); - EXPECT_FALSE(b->head(s->layout().manifestKey(mid)).exists) + EXPECT_TRUE((*op).head(s->layout().blobKey(blob_ref.ref), Retry::once()).has_value()); + EXPECT_FALSE((*op).head(s->layout().manifestKey(mid), Retry::once()).has_value()) << "abandon must best-effort delete this build's staged manifest debris"; /// Further operations throw via requireAlive. @@ -1507,7 +1493,8 @@ TEST(CASPartWriteTxn, PublishHappyPathRoundTrip) ASSERT_TRUE(entry != nullptr); const auto loc = s->locate(*entry); /// The located window of the blob object, sliced by the test: the seam reads whole objects. - auto got = b->get(loc.key); + OperationForTest op(*b); + auto got = (*op).read(loc.key, Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), "hello world"); } @@ -1566,7 +1553,10 @@ TEST(CASPartWriteTxn, PublishIntoSecondNamespaceSameBlob) build1->precommitAdd(ns1, "part_1", id1); auto blob = build1->putBlob(idOf("hello world"), BlobSource::fromString("hello world")); const String blob_key = s->layout().blobKey(blob.ref); - const Token blob_token = b->head(blob_key).token; + OperationForTest op(*b); + const auto blob_head0 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(blob_head0.has_value()); + const Etag blob_token = blob_head0->etag; build1->promote(ns1, "part_1", build1->buildId(), id1); /// Second build publishes part_1 in ns2 referencing the SAME blob: putBlob dedup-hits and ADOPTS the @@ -1586,7 +1576,9 @@ TEST(CASPartWriteTxn, PublishIntoSecondNamespaceSameBlob) EXPECT_EQ(r2->manifest_id, id2); /// The blob object was uploaded once: its token is unchanged after both publishes. - EXPECT_EQ(b->head(blob_key).token, blob_token); + const auto blob_head1 = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(blob_head1.has_value()); + EXPECT_EQ(blob_head1->etag, blob_token); } /// Task 10: refs are no longer sharded (one whole-table cache per namespace, spec §Table State), so @@ -1657,23 +1649,6 @@ TEST(CASPartWriteTxn, AdoptEvidenceRecordsTrustedManifestDependencyProofWithoutI size_t puts = 0; size_t gets = 0; - HeadResult head(const String & k) override { ++heads; return inner->head(k); } - void publishBlob(const BlobPublishRequest & request) override - { - ++puts; - inner->publishBlob(request); - } - std::optional get(const String & k, Range r) override { ++gets; return inner->get(k, r); } - std::optional getStream(const String & k, Range r) override { return inner->getStream(k, r); } - ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - PutResult putIfAbsent(const String & k, const String & bts, const ObjectMeta & m) override - { - ++puts; - return inner->putIfAbsent(k, bts, m); - } - PutResult putOverwrite(const String & k, const String & bts, const Token & e, const ObjectMeta & m) override { return inner->putOverwrite(k, bts, e, m); } - CasResult casPut(const String & k, const String & bts, const std::optional & e, const ObjectMeta & m) override { return inner->casPut(k, bts, e, m); } - DeleteOutcome deleteExact(const String & k, const Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } /// The primitives count on the same three counters, so "no backend op" stays a total claim @@ -1772,7 +1747,7 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) /// 1. PartWriteTxn A creates H ("shared-content"), publishes a part referencing it, then drops the ref. /// Capture H's first incarnation token so we can condemn exactly it. BlobRef h; - PersistedIncarnation h_token0; + PersistedEtag h_token0; { auto s0 = Pool::open(b, cfg); publishOneBlobPart(s0, ns, "part_1", "f", content); @@ -1780,7 +1755,7 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) CasOperation op0 = s0->mountRequests().admit(); const auto seeded = op0.head(s0->layout().blobKey(h), Retry::standard()); ASSERT_TRUE(seeded.has_value()); - h_token0 = PersistedIncarnation::capture(seeded->incarnation); + h_token0 = PersistedEtag::capture(seeded->etag); s0->dropRef(ns, "part_1"); } @@ -1815,7 +1790,7 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) CasOperation reupload_op = s->mountRequests().admit(); const auto after_reupload = reupload_op.head(blob_key, Retry::standard()); ASSERT_TRUE(after_reupload.has_value()); - EXPECT_FALSE(h_token0.matches(after_reupload->incarnation)); /// a genuinely fresh incarnation + EXPECT_FALSE(h_token0.matches(after_reupload->etag)); /// a genuinely fresh incarnation /// 4. THE ADVERSARIAL LOOP. A real, productive GC keeps trying to reclaim. It reclaims the now- /// unreferenced part_1 manifest (build A's, UNprotected) but H stays pinned by B's PRECOMMIT edge @@ -1832,10 +1807,11 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) /// (a frozen B would have its precommit reclaimed; an advancing seq keeps it). s->renewWatermarkOnce(); gc.runRegularRound(); - const HeadResult hr = b->head(blob_key); - ASSERT_TRUE(hr.exists) << "H was deleted by GC at round " << round_no + OperationForTest op(*b); + const auto hr = (*op).head(blob_key, Retry::once()); + ASSERT_TRUE(hr.has_value()) << "H was deleted by GC at round " << round_no << " despite being pinned by the live build B's precommit (B167 livelock would do this)"; - const auto raw = b->get(blob_key); + const auto raw = (*op).read(blob_key, Retry::once()); ASSERT_TRUE(raw.has_value()); const auto hdr = decodeEnvelopeHeader(raw->bytes, raw->bytes.size(), ObjectKind::Blob); EXPECT_EQ(raw->bytes.substr(hdr.header_len), content) @@ -1882,7 +1858,8 @@ TEST(CASPartWriteTxn, ConvergesUnderProductiveGc) const auto * entry = findEntry(manifest.entries, "f"); ASSERT_TRUE(entry != nullptr); const auto loc = s->locate(*entry); - const auto got = b->get(loc.key); + OperationForTest op(*b); + const auto got = (*op).read(loc.key, Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), content); } @@ -2024,12 +2001,13 @@ TEST(CASPartWriteTxn, AbandonAppendsPrecommitRemovalAndKeepsLivePrecommitBody) build->putBlob(idOf("kept"), BlobSource::fromString("kept")); /// The precommit manifest body is present before abandon. - ASSERT_TRUE(b->head(manifest_key).exists); + OperationForTest op(*b); + ASSERT_TRUE((*op).head(manifest_key, Retry::once()).has_value()); build->abandon(); /// (a) the LIVE precommit body must SURVIVE abandon (left for GC after the sealed decrement). - EXPECT_TRUE(b->head(manifest_key).exists) + EXPECT_TRUE((*op).head(manifest_key, Retry::once()).has_value()) << "abandon must NOT writer-delete a live precommit body (delete-after-sealed-decrements)"; /// (b) the exact precommit binding is gone (spec §Remove Precommit: an exact owner_transition @@ -2066,9 +2044,10 @@ TEST(CASPartWriteTxn, AbandonStillDeletesNeverPrecommittedStagedDebris) build->abandon(); /// The never-precommitted debris is best-effort deleted; the live precommit body survives. - EXPECT_FALSE(b->head(s->layout().manifestKey(debris)).exists) + OperationForTest op(*b); + EXPECT_FALSE((*op).head(s->layout().manifestKey(debris), Retry::once()).has_value()) << "never-precommitted staged debris must still be best-effort deleted by abandon"; - EXPECT_TRUE(b->head(s->layout().manifestKey(precommitted)).exists) + EXPECT_TRUE((*op).head(s->layout().manifestKey(precommitted), Retry::once()).has_value()) << "the live precommit body must be spared"; } @@ -2198,14 +2177,15 @@ TEST(CASPartWriteTxn, AbandonRetryableAfterAppendFailure) /// no longer take one. String greatest_key; size_t foreign_objects = 0; + OperationForTest op(*b); for (String cursor;;) { - const ListPage page = b->list(b->corrupt_key_substr, cursor, 1000); + const ListPage page = (*op).list(b->corrupt_key_substr, cursor, 1000, Retry::once()); for (const auto & listed : page.keys) { if (listed.key > greatest_key) greatest_key = listed.key; - const auto body = b->get(listed.key); + const auto body = (*op).read(listed.key, Retry::once()); if (body && body->bytes.find("_FOREIGN_DIFFERENT") != String::npos) ++foreign_objects; } @@ -2215,7 +2195,7 @@ TEST(CASPartWriteTxn, AbandonRetryableAfterAppendFailure) } EXPECT_EQ(foreign_objects, 1u) << "the foreign object must still own the key it took"; ASSERT_FALSE(greatest_key.empty()); - const auto greatest_body = b->get(greatest_key); + const auto greatest_body = (*op).read(greatest_key, Retry::once()); ASSERT_TRUE(greatest_body.has_value()); EXPECT_NE(greatest_body->bytes.find("_FOREIGN_DIFFERENT"), String::npos) << "the foreign occupant must still be the highest id in this table's stream: a log object above " @@ -2323,7 +2303,8 @@ TEST(CASPartWriteTxn, ManifestCapEncodedBytesJustUnderStagesSuccessfully) auto build = startBuildFor(s, ns, "wide_part"); const ManifestId id = build->stageManifest({wideBlobManifestEntry(path_len_under)}); EXPECT_EQ(id.root_namespace, ns); - EXPECT_TRUE(b->head(s->layout().manifestKey(id)).exists) + OperationForTest op(*b); + EXPECT_TRUE((*op).head(s->layout().manifestKey(id), Retry::once()).has_value()) << "a just-under-cap manifest must actually be written"; } @@ -2341,7 +2322,8 @@ TEST(CASPartWriteTxn, ManifestCapEncodedBytesOverThrowsBeforeBodyWrite) auto build = startBuildFor(s, ns, "wide_part"); - const size_t keys_before = b->list("", "", 100).keys.size(); + OperationForTest op(*b); + const size_t keys_before = (*op).list("", "", 100, Retry::once()).keys.size(); bool threw = false; try { @@ -2357,7 +2339,7 @@ TEST(CASPartWriteTxn, ManifestCapEncodedBytesOverThrowsBeforeBodyWrite) /// Fail-closed BEFORE the body write: the over-cap attempt must not have created ANY new object /// (no partial state, no orphaned blob/manifest debris for a manifest that was never accepted). - const size_t keys_after = b->list("", "", 100).keys.size(); + const size_t keys_after = (*op).list("", "", 100, Retry::once()).keys.size(); EXPECT_EQ(keys_before, keys_after) << "stageManifest must fail closed before writing the manifest body, leaving no new objects"; } @@ -2415,9 +2397,9 @@ TEST(CASPartWriteTxn, WDepSetCrossAlgoSatisfactionFailsClosed) } /// ===================================================================================== -/// Task B (chaos-tolerance-report §Task B): stageManifest's part-manifest conditional PUT rides the -/// shared CasRequestController — budgeted attempts + resolve-before-reissue — instead of the old -/// single bare attempt (which a 19s object-store pause killed while every read path survived). +/// Task B (chaos-tolerance-report §Task B): stageManifest's part-manifest conditional PUT rides +/// budgeted attempts with resolve-before-reissue, instead of the old single bare attempt (which a +/// 19s object-store pause killed while every read path survived). /// ===================================================================================== namespace @@ -2490,7 +2472,8 @@ TEST(CASPartWriteTxnStageManifestRetry, AmbiguousTimeoutsThenCommitSucceedsWithi const ManifestId id = build->stageManifest({blobManifestEntry("a.bin", "a")}); EXPECT_EQ(b->put_attempts, 3) << "two faulted attempts + the committing third"; - const auto got = b->get(s->layout().manifestKey(id)); + OperationForTest op(*b); + const auto got = (*op).read(s->layout().manifestKey(id), Retry::once()); ASSERT_TRUE(got.has_value()) << "the staged manifest body must be durable"; EXPECT_EQ(decodePartManifest(openObject(FormatId::PartManifest, got->bytes)).ref, id.ref); } @@ -2516,7 +2499,10 @@ TEST(CASPartWriteTxnStageManifestRetry, AmbiguousLandedWriteResolvesToCommittedW EXPECT_EQ(b->put_attempts, 1) << "a landed ambiguous attempt must be resolved, never reissued"; const String key = s->layout().manifestKey(id); - ASSERT_TRUE(b->get(key).has_value()); + { + OperationForTest op(*b); + ASSERT_TRUE((*op).read(key, Retry::once()).has_value()); + } const auto ev = std::find_if(events.begin(), events.end(), [](const CasEvent & e) { return e.type == CasEventType::ManifestPut; }); @@ -2525,7 +2511,7 @@ TEST(CASPartWriteTxnStageManifestRetry, AmbiguousLandedWriteResolvesToCommittedW CasOperation probe_op = probe.admit(); const auto landed = probe_op.head(key, Retry::standard()); ASSERT_TRUE(landed.has_value()); - EXPECT_EQ(ev->token, landed->incarnation.render()) + EXPECT_EQ(ev->token, landed->etag.render()) << "the audit token must be the landed incarnation, rendered"; } @@ -2578,7 +2564,8 @@ TEST(CASPartWriteTxnStageManifestRetry, BudgetExhaustionMapsToNetworkError) << "the reissues were paced on the injected clock, so this exhaustion cost no wall time"; /// Over the namespace's whole manifest prefix rather than one computed key: the ordinal the build /// would have used is arithmetic this assertion should not have to reproduce to stay true. - EXPECT_TRUE(b->list(s->layout().manifestNamespacePrefix(ns), "", 10).keys.empty()) + OperationForTest op(*b); + EXPECT_TRUE((*op).list(s->layout().manifestNamespacePrefix(ns), "", 10, Retry::once()).keys.empty()) << "an exhausted stage names nothing durable"; } @@ -2700,7 +2687,8 @@ TEST(CASPartWrite, AmbiguousTimeoutsThenCommitRestreamsFromSource) EXPECT_EQ(b->publish_stream_attempts, 3) << "two ambiguous publications + the committing third"; EXPECT_EQ(b->blob_head_attempts, 3) << "every outer retry restarts from a fresh blob HEAD"; EXPECT_EQ(payload_streams, 3) << "every reissue must RE-STREAM from the writer's own source (INV-1)"; - EXPECT_TRUE(b->head(s->layout().blobKey(idOf(payload))).exists) << "the blob body must be durable"; + OperationForTest op(*b); + EXPECT_TRUE((*op).head(s->layout().blobKey(idOf(payload)), Retry::once()).has_value()) << "the blob body must be durable"; } /// Ambiguous-but-landed: the FIRST attempt's response is lost AFTER the write actually landed @@ -2740,7 +2728,7 @@ TEST(CASPartWrite, AmbiguousLandedWriteAdoptsOccupantWithoutReupload) CasOperation probe_op = probe.admit(); const auto landed = probe_op.head(key, Retry::standard()); ASSERT_TRUE(landed.has_value()); - EXPECT_EQ(adopt->token, landed->incarnation.render()) + EXPECT_EQ(adopt->token, landed->etag.render()) << "the adopted token must be the landed incarnation, rendered"; EXPECT_EQ(std::count_if(events.begin(), events.end(), [](const CasEvent & e) { return e.type == CasEventType::BlobPut; }), 0) @@ -2815,9 +2803,6 @@ namespace class RearmAfterMetaReadBackend final : public InMemoryBackend { public: - /// Unhide the legacy overload the primitive override below would otherwise hide. - using InMemoryBackend::get; - String watched_key; std::function trigger; @@ -2919,7 +2904,10 @@ TEST(CASPartWrite, AmbiguousCopyLandedAdoptsDestinationWithoutRecopy) /// The staging object: [pool-fixed-length envelope header][payload], promoted VERBATIM by the copy. const String staging_key = "p/staging/test/blob-a"; const String staging_bytes = String(s->poolMeta().blob_header_len, 'h') + payload; - ASSERT_EQ(b->putIfAbsent(staging_key, staging_bytes).outcome, PutOutcome::Done); + { + OperationForTest seed_op(*b); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(staging_key, staging_bytes, Retry::once()))); + } s->setEventSink([&](const CasEvent & e) { events.push_back(e); }); @@ -2939,7 +2927,8 @@ TEST(CASPartWrite, AmbiguousCopyLandedAdoptsDestinationWithoutRecopy) EXPECT_EQ(b->publish_stream_attempts, 0); EXPECT_EQ(b->blob_head_attempts, 2); const String key = s->layout().blobKey(idOf(payload)); - const auto got = b->get(key); + OperationForTest op(*b); + const auto got = (*op).read(key, Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes, staging_bytes) << "the destination is the staging object's verbatim copy"; EXPECT_NE(std::find_if(events.begin(), events.end(), @@ -2958,7 +2947,10 @@ TEST(CASPartWrite, AmbiguousCopyAbsentReattemptsAndCommits) const String payload = "staged-payload-B"; const String staging_key = "p/staging/test/blob-b"; const String staging_bytes = String(s->poolMeta().blob_header_len, 'h') + payload; - ASSERT_EQ(b->putIfAbsent(staging_key, staging_bytes).outcome, PutOutcome::Done); + { + OperationForTest seed_op(*b); + ASSERT_TRUE(std::holds_alternative((*seed_op).create(staging_key, staging_bytes, Retry::once()))); + } auto build = startBuildFor(s, ns, "part_copy_retry"); const ManifestId id = build->stageManifest({blobManifestEntry("a.bin", payload)}); @@ -2979,7 +2971,8 @@ TEST(CASPartWrite, AmbiguousCopyAbsentReattemptsAndCommits) EXPECT_EQ(b->publish_stream_attempts, 1) << "the absent retry must retag and stream"; EXPECT_EQ(b->blob_head_attempts, 2); const String key = s->layout().blobKey(idOf(payload)); - const auto got = b->get(key); + OperationForTest op(*b); + const auto got = (*op).read(key, Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_NE(got->bytes, staging_bytes); EXPECT_EQ(got->bytes.substr(s->poolMeta().blob_header_len), payload); diff --git a/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp b/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp index 1f86c6fe7d47..23cce62494e2 100644 --- a/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp +++ b/src/Disks/tests/gtest_cas_part_write_root_dangle.cpp @@ -120,7 +120,8 @@ TEST(CASPartWriteTxnRootDangle, SharedBlobSurvivesSourceDropDuringBuild) << "B171: PartWriteTxn B's promote must succeed — the precommit should have kept P alive"; /// The blob B references must still be present (no dangle), and refB must resolve. - ASSERT_TRUE(backend->head(s->layout().blobKey(idOf(P))).exists) + DB::Cas::tests::OperationForTest dangle_op(*backend); + ASSERT_TRUE((*dangle_op).head(s->layout().blobKey(idOf(P)), Retry::once()).has_value()) << "B171-dangle: GC deleted the shared blob P that PartWriteTxn B adopted — its cas_owner was the " << "retired PartWriteTxn A and the stub precommit published no build-root edge, so inDeg(P) hit 0 " << "and the single content-delete site removed it. refB now dangles."; @@ -173,18 +174,19 @@ TEST(CASPartWriteTxnRootDangle, PrematureReclaimCommitFailsClosed) /// RAW removal append would collide with the writer's own `RefTxnId` sequence allocation on the next /// flush; the property under test is the COMMIT gate's fail-closed behavior against a missing /// dependency, not the reclaim mechanics -- so we go straight to the reclaimed state.) + DB::Cas::tests::OperationForTest reclaim_op(*backend); { const String pkey = s->layout().blobKey(idOf(P)); - const HeadResult h = backend->head(pkey); - ASSERT_TRUE(h.exists) << "P must be present before the simulated reclaim"; - ASSERT_EQ(backend->deleteExact(pkey, h.token).kind, DeleteOutcome::Kind::Deleted); + const auto h = (*reclaim_op).head(pkey, Retry::once()); + ASSERT_TRUE(h.has_value()) << "P must be present before the simulated reclaim"; + ASSERT_EQ((*reclaim_op).remove(pkey, h->etag, Retry::once()), Removal::Removed); } /// Drop the source ref too (the state a real premature reclaim leaves: P unprotected and gone). s->dropRef(ns, "refA"); s->renewWatermarkOnce(); /// The shared blob must be GONE (the premature reclaim collected it). - ASSERT_FALSE(backend->head(s->layout().blobKey(idOf(P))).exists) + ASSERT_FALSE((*reclaim_op).head(s->layout().blobKey(idOf(P)), Retry::once()).has_value()) << "premature-reclaim setup invalid: P should have been collected after losing its precommit"; /// §4 manifest-trust (test name is legacy — B171 INV-COMMIT-FAILCLOSED for an ADOPTED leaf now moves to @@ -199,7 +201,7 @@ TEST(CASPartWriteTxnRootDangle, PrematureReclaimCommitFailsClosed) << "§4: an adopted leaf is trusted at promote — a missing dependency is not re-observed here"; /// Trust never fabricates the missing blob (it never touches P); refB IS committed (naming absent P). - ASSERT_FALSE(backend->head(s->layout().blobKey(idOf(P))).exists) + ASSERT_FALSE((*reclaim_op).head(s->layout().blobKey(idOf(P)), Retry::once()).has_value()) << "trust never fabricates the missing blob — P stays absent"; ASSERT_TRUE(s->resolveRef(ns, "refB").has_value()) << "§4: refB commits under trust (the D4 trade-off); the dangle is caught by fsck, below"; @@ -247,7 +249,8 @@ TEST(CASPartWriteTxnRoot, LivePrecommitNotReclaimed) runGcToFixpoint(gc); /// Q must still be present (the live precommit's +1 edge pins it across GC). - ASSERT_TRUE(backend->head(s->layout().blobKey(idOf(Q))).exists) + DB::Cas::tests::OperationForTest live_op(*backend); + ASSERT_TRUE((*live_op).head(s->layout().blobKey(idOf(Q)), Retry::once()).has_value()) << "B8 conservatism: the live precommit must keep its blob alive across GC"; /// B can still commit (the precommit is intact). diff --git a/src/Disks/tests/gtest_cas_pluggable_hash.cpp b/src/Disks/tests/gtest_cas_pluggable_hash.cpp index 32dbe2a0ad94..724f5e6a905c 100644 --- a/src/Disks/tests/gtest_cas_pluggable_hash.cpp +++ b/src/Disks/tests/gtest_cas_pluggable_hash.cpp @@ -58,6 +58,29 @@ using namespace DB::Cas::tests; namespace { +/// ---- Small raw-fixture request-engine wrappers shared by the tests below ---- + +/// True iff `key` exists. +bool existsAt(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::once()).has_value(); +} + +/// The durable object at `key`, or `nullopt`. +std::optional readAt(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::once()); +} + +/// Unconditional create of a fresh key (the fixture's own setup, never a real conflict). +void createAt(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + EXPECT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + /// A deterministic, non-repeating-byte payload spanning several `DBMS_DEFAULT_HASHING_BLOCK_SIZE` /// (2048 B) blocks, so a chunked-vs-one-shot divergence (the CityHash128 pitfall documented on /// `poolContentHash`) would not accidentally go unnoticed. @@ -99,7 +122,10 @@ SeededBlob seedReferencedBlob(Pool & store, Backend & backend, const RootNamespa header.kind = ObjectKind::Blob; header.incarnation_tag = UInt128(0x1234); header.build_id = UInt128(0x5678); - backend.putIfAbsent(key, encodeEnvelopeHeader(header, static_cast(store.poolMeta().blob_header_len)) + payload); + { + OperationForTest op(backend); + (*op).create(key, encodeEnvelopeHeader(header, static_cast(store.poolMeta().blob_header_len)) + payload, Retry::once()); + } ManifestEntry entry; entry.path = "data_" + std::to_string(build_sequence) + ".bin"; @@ -328,7 +354,7 @@ TEST(CASPluggableHash, Xxh3BlobLandsUnderAlgoSegmentAndIsDiscoveredCleanByFsck) const String blob_key = store->layout().blobKey(id); EXPECT_NE(blob_key.find("/blobs/xxh3/"), String::npos) << blob_key; EXPECT_EQ(blob_key.find("/blobs/ch128/"), String::npos) << blob_key; - EXPECT_TRUE(backend->head(blob_key).exists); + EXPECT_TRUE(existsAt(*backend, blob_key)); build->promote(ns, "rb", build->buildId(), mid); store->renewWatermarkOnce(); @@ -395,7 +421,7 @@ TEST(CASPluggableHash, Sha256BlobSeenByCondemnSweepAndFsckNotSilentlySkipped) const BlobRef id = seeded.ref; const String blob_key = seeded.key; EXPECT_NE(blob_key.find("/blobs/sha256/"), String::npos) << blob_key; - ASSERT_TRUE(backend->head(blob_key).exists) << "the sha256 blob body must be present before the fold"; + ASSERT_TRUE(existsAt(*backend, blob_key)) << "the sha256 blob body must be present before the fold"; ASSERT_EQ(codecFor(store->writeAlgo()).fromHex(hex), digest) << "fixture sanity: the seeded digest is ours"; /// ---- Site 1: the fold's condemn path ---- @@ -403,11 +429,11 @@ TEST(CASPluggableHash, Sha256BlobSeenByCondemnSweepAndFsckNotSilentlySkipped) dropSeededRef(*store, *backend, ns, /*build_sequence=*/1, "tbl_sha"); runRegularRoundReclaiming(gc); /// folds the -1: transition to zero => condemned - const auto state_bytes = backend->get(store->layout().gcStateKey()); + const auto state_bytes = readAt(*backend, store->layout().gcStateKey()); ASSERT_TRUE(state_bytes.has_value()); const GcState state = decodeGcState(state_bytes->bytes); ASSERT_GT(state.snap_generation, 0u); - const auto seal_bytes = backend->get(store->layout().foldSealKey(state.snap_generation, state.snap_attempt)); + const auto seal_bytes = readAt(*backend, store->layout().foldSealKey(state.snap_generation, state.snap_attempt)); ASSERT_TRUE(seal_bytes.has_value()); const CasFoldSeal seal = decodeFoldSeal(seal_bytes->bytes); ASSERT_TRUE(seal.condemned_summary.contains(0)) << "the seal's condemned_summary must be total over gc_shards"; @@ -512,14 +538,14 @@ TEST(CASPluggableHash, Sha256BuildWritesFullWidthDigestAndInlineEqualsBlob) /// to a 32-hex (128-bit) key. const String blob_key = store->layout().blobKey(id); EXPECT_NE(blob_key.find("/blobs/sha256/"), String::npos) << blob_key; - ASSERT_TRUE(backend->head(blob_key).exists); + ASSERT_TRUE(existsAt(*backend, blob_key)); build->promote(ns, "part1", build->buildId(), mid); store->renewWatermarkOnce(); /// Read the committed manifest back -- the on-disk `blob_hash` must be the FULL 32-byte digest, not /// truncated by the manifest codec or by anything upstream of `stageManifest`. - const auto manifest_bytes = backend->get(store->layout().manifestKey(mid)); + const auto manifest_bytes = readAt(*backend, store->layout().manifestKey(mid)); ASSERT_TRUE(manifest_bytes.has_value()); const PartManifest read_back = decodePartManifest(openObject(FormatId::PartManifest, manifest_bytes->bytes)); ASSERT_EQ(read_back.entries.size(), 2u); @@ -637,7 +663,7 @@ TEST(CASPluggableHash, ForeignAlgoSegmentIsDebrisNotOurs) /// A FOREIGN object under an algo segment `blobHashAlgoName` never renders ("md5") -- not one of /// ours under any circumstance. const String foreign_key = store->layout().blobsPrefix() + "md5/aa/" + std::string(32, 'a'); - backend->putIfAbsent(foreign_key, std::string("not a real envelope")); + createAt(*backend, foreign_key, std::string("not a real envelope")); runRegularRoundReclaiming(gc); /// folds both +1s dropSeededRef(*store, *backend, ns, /*build_sequence=*/1, "tbl_ch"); @@ -655,7 +681,7 @@ TEST(CASPluggableHash, ForeignAlgoSegmentIsDebrisNotOurs) } EXPECT_TRUE(condemned_refs.count(ch_ref)); EXPECT_TRUE(condemned_refs.count(sh_ref)); - EXPECT_TRUE(backend->head(foreign_key).exists) << "the foreign object must never be touched by the fold"; + EXPECT_TRUE(existsAt(*backend, foreign_key)) << "the foreign object must never be touched by the fold"; const FsckReport frep = runFsck(*store, /*detail=*/true); /// The physical listing counts all THREE unreferenced objects (two ours + one foreign). @@ -695,7 +721,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); EXPECT_EQ(store->poolMeta().min_reader_generation, G_BUILD); - const auto meta_bytes = backend->get(store->layout().poolMetaKey()); + const auto meta_bytes = readAt(*backend, store->layout().poolMetaKey()); ASSERT_TRUE(meta_bytes.has_value()); EXPECT_EQ(decodePoolMeta(meta_bytes->bytes).min_reader_generation, G_BUILD); } @@ -708,7 +734,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) OperationForTest meta_op(*backend); PoolMeta pm = PoolMeta::createOrValidate(*meta_op, layout, /*blob_header_len*/ 256, BlobHashAlgo::CityHash128, /*allow_new*/ false, /*allow_mint*/ true); pm.min_reader_generation = G_BUILD + 1; - ASSERT_TRUE(backend->casPut(layout.poolMetaKey(), encodePoolMeta(pm), backend->get(layout.poolMetaKey())->token).outcome == CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative((*meta_op).replace(layout.poolMetaKey(), encodePoolMeta(pm), (*meta_op).head(layout.poolMetaKey(), Retry::once())->etag, Retry::once()))); expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); }); @@ -730,7 +756,7 @@ TEST(CASPluggableHash, ReaderGenerationIsRaisedToGBuild) ASSERT_NE(pos, String::npos); // sanity: a fresh pool stamps the header at the floor String downgraded = fresh_bytes; downgraded.replace(pos, from.size(), to); - ASSERT_TRUE(backend->casPut(layout.poolMetaKey(), downgraded, backend->get(layout.poolMetaKey())->token).outcome == CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative((*meta_op).replace(layout.poolMetaKey(), downgraded, (*meta_op).head(layout.poolMetaKey(), Retry::once())->etag, Retry::once()))); /// `decodePoolMeta`'s backward floor rejects the downgraded bytes directly... expectThrowsCode(DB::ErrorCodes::UNKNOWN_FORMAT_VERSION, [&] { decodePoolMeta(downgraded); }); @@ -781,8 +807,8 @@ TEST(CASPluggableHash, TwoAlgoBlobsBothFullyReclaimed) /*build_sequence=*/2, /*payload_size=*/5002, "tbl_sh"); const String ch_key = ch.key; const String sh_key = sh.key; - ASSERT_TRUE(backend->head(ch_key).exists); - ASSERT_TRUE(backend->head(sh_key).exists); + ASSERT_TRUE(existsAt(*backend, ch_key)); + ASSERT_TRUE(existsAt(*backend, sh_key)); runRegularRoundReclaiming(gc); /// folds both +1s dropSeededRef(*store, *backend, ns, /*build_sequence=*/1, "tbl_ch"); @@ -806,8 +832,8 @@ TEST(CASPluggableHash, TwoAlgoBlobsBothFullyReclaimed) { const RoundReport rep1 = runRegularRoundReclaiming(gc); EXPECT_EQ(rep1.graduated, 2u) << "both algos' blobs must graduate together in one round"; - EXPECT_TRUE(backend->head(ch_key).exists); // pending: still present this pass - EXPECT_TRUE(backend->head(sh_key).exists); + EXPECT_TRUE(existsAt(*backend, ch_key)); // pending: still present this pass + EXPECT_TRUE(existsAt(*backend, sh_key)); } { const RoundReport rep2 = runRegularRoundReclaiming(gc); @@ -815,8 +841,8 @@ TEST(CASPluggableHash, TwoAlgoBlobsBothFullyReclaimed) } /// THE CRUX: after graduation the backend holds ZERO blob bodies of EITHER algo. - EXPECT_FALSE(backend->head(ch_key).exists) << "the ch128 blob must be physically reclaimed"; - EXPECT_FALSE(backend->head(sh_key).exists) << "the sha256 blob must be physically reclaimed"; + EXPECT_FALSE(existsAt(*backend, ch_key)) << "the ch128 blob must be physically reclaimed"; + EXPECT_FALSE(existsAt(*backend, sh_key)) << "the sha256 blob must be physically reclaimed"; const FsckReport frep = runFsck(*store, /*detail=*/true); EXPECT_TRUE(frep.clean()); @@ -885,8 +911,8 @@ TEST(CASPluggableHash, SameDigestDifferentAlgoDistinctBodiesAndSettlement) const String key_ch = store->layout().blobKey(ref_ch); const String key_xx = store->layout().blobKey(ref_xx); EXPECT_NE(key_ch, key_xx); - const auto raw_ch = backend->get(key_ch); - const auto raw_xx = backend->get(key_xx); + const auto raw_ch = readAt(*backend, key_ch); + const auto raw_xx = readAt(*backend, key_xx); ASSERT_TRUE(raw_ch.has_value()); ASSERT_TRUE(raw_xx.has_value()); EXPECT_NE(raw_ch->bytes.find(body_ch), String::npos); @@ -898,17 +924,17 @@ TEST(CASPluggableHash, SameDigestDifferentAlgoDistinctBodiesAndSettlement) const String meta_ch = store->layout().blobMetaKey(ref_ch); const String meta_xx = store->layout().blobMetaKey(ref_xx); EXPECT_NE(meta_ch, meta_xx); - EXPECT_TRUE(backend->head(meta_ch).exists); - EXPECT_TRUE(backend->head(meta_xx).exists); + EXPECT_TRUE(existsAt(*backend, meta_ch)); + EXPECT_TRUE(existsAt(*backend, meta_xx)); /// Distinct settlement (in-degree per ref, keyed on the FULL `BlobRef` pair -- never the shared /// bare digest, which would alias the two rows into one). Gc gc(store, UInt128(1)); runRegularRoundReclaiming(gc); { - const GcState st = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + const GcState st = decodeGcState(readAt(*backend, store->layout().gcStateKey())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + readAt(*backend, store->layout().foldSealKey(st.snap_generation, st.snap_attempt))->bytes); EXPECT_EQ(inDegreeInRuns(*backend, seal.blob_target_runs, ref_ch), 1); EXPECT_EQ(inDegreeInRuns(*backend, seal.blob_target_runs, ref_xx), 1); } @@ -920,11 +946,11 @@ TEST(CASPluggableHash, SameDigestDifferentAlgoDistinctBodiesAndSettlement) runRegularRoundReclaiming(gc); // graduates ch128:X runRegularRoundReclaiming(gc); // executes the exact-token delete for ch128:X - EXPECT_FALSE(backend->head(key_ch).exists) << "ch128:X must be reclaimed once its ref is dropped"; - EXPECT_TRUE(backend->head(key_xx).exists) + EXPECT_FALSE(existsAt(*backend, key_ch)) << "ch128:X must be reclaimed once its ref is dropped"; + EXPECT_TRUE(existsAt(*backend, key_xx)) << "THE CRUX: xxh3:X (same digest value, different algo) must remain readable after ch128:X " "is reclaimed -- a digest-only settlement would have condemned/deleted both together"; - const auto still_readable = backend->get(key_xx); + const auto still_readable = readAt(*backend, key_xx); ASSERT_TRUE(still_readable.has_value()); EXPECT_NE(still_readable->bytes.find(body_xx), String::npos); diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 58c5636358a3..f524dc9b81bb 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -65,23 +65,10 @@ class WriteCountingBackend final : public DB::Cas::Backend explicit WriteCountingBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} size_t writes = 0; - std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & meta) override { ++writes; return inner->putIfAbsent(k, b, meta); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - ++writes; - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & meta) override { ++writes; return inner->putOverwrite(k, b, e, meta); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & meta) override { ++writes; return inner->casPut(k, b, e, meta); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { ++writes; return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The primitives count too: `Backend::probeSentinelRaw` reaches the store through them, so a - /// write that took the primitive path would otherwise go unseen by `writes`. + /// Every write reaches the store through these primitives, so `writes` sees it whichever verb + /// (`create`/`replace`/`remove`/`publish`) issued it. std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } @@ -107,6 +94,27 @@ class WriteCountingBackend final : public DB::Cas::Backend std::shared_ptr inner; }; +/// A one-shot `create`, asserting it committed (mirrors the retired `backend.putIfAbsent(key, bytes)`). +void createObj(Backend & backend, const String & key, const String & bytes) +{ + DB::Cas::tests::OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// An exact read (mirrors the retired `backend.get(key)`). +std::optional readObj(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +/// A HEAD (mirrors the retired `backend.head(key)`). +std::optional headObj(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, Retry::standard()); +} + /// Publish one part `ref` through the REAL PartWriteTxn write path: stage a manifest holding a single content /// blob whose payload is `payload`, precommit-add into the owning shard, then promote precommit -> /// committed. Returns the published ManifestId. This is the canonical write-side fixture for the @@ -167,7 +175,7 @@ ManifestId publishPartWithEntries( { /// Materialize the blob body so the promote-time HEAD revalidation succeeds, then record the /// tokenless W-EVIDENCE dep (the gate re-observes the current token at promote). - DB::Cas::tests::writeBlobBody(s->backend(), s->layout(), e.ref.digest.toU128()); + DB::Cas::tests::writeBlobBody(*s->poolBackendPtr(), s->layout(), e.ref.digest.toU128()); build->adoptEvidence(e); } const ManifestId id = build->stageManifest(std::move(entries)); @@ -207,23 +215,10 @@ class ProbeWatchingBackend final : public DB::Cas::Backend explicit ProbeWatchingBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} bool probe_touched = false; - std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & m) override { note(k); return inner->putIfAbsent(k, b, m); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - note(request.destination_key); - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & m) override { note(k); return inner->putOverwrite(k, b, e, m); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & m) override { note(k); return inner->casPut(k, b, e, m); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { note(k); return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The primitives note too: `Backend::probeSentinelRaw` reaches the store through them, so a - /// probe-key mutation on the primitive path would otherwise go unseen. + /// Every mutation reaches the store through these primitives, so a probe-key touch is noted + /// whichever verb (`create`/`replace`/`remove`/`publish`) issued it. std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } @@ -300,22 +295,9 @@ class ForwardingBackend : public DB::Cas::Backend public: explicit ForwardingBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} - std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & m) override { return inner->putIfAbsent(k, b, m); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & m) override { return inner->putOverwrite(k, b, e, m); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, b, e, m); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The transport primitives forward to `inner`; the legacy overrides above are what this - /// double injects through. Declared because `Backend` declares them pure. + /// The transport primitives forward to `inner`. Declared because `Backend` declares them pure. std::optional read(const String & key, TransportAccess & access) override { return inner->read(key, access); } std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } @@ -597,7 +579,7 @@ TEST(CASPoolMeta, FailClosed) /// (createOrValidate path). The future-version fail-closed (v > G_BUILD => UNKNOWN_FORMAT_VERSION) /// is exercised at the codec level by the battery's per-row v+1 gate. auto b2 = std::make_shared(); - b2->putIfAbsent(layout.poolMetaKey(), "garbage"); + createObj(*b2, layout.poolMetaKey(), "garbage"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b2), layout, 256); }); } @@ -642,7 +624,7 @@ TEST(CASPoolMeta, RejectsBadConstantsAtCreation) [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 17 * 1024); }); /// A creation that fails config validation must not have written anything. - EXPECT_FALSE(b->get(layout.poolMetaKey()).has_value()); + EXPECT_FALSE(readObj(*b, layout.poolMetaKey()).has_value()); } TEST(CASPoolMeta, RejectsBadConstantsOnDecode) @@ -654,7 +636,7 @@ TEST(CASPoolMeta, RejectsBadConstantsOnDecode) bad_pm.pool_id = hexToU128("00000000000000000000000000000001"); bad_pm.blob_header_len = 100; /// violates 8-alignment invariant bad_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(bad_pm)); + createObj(*b, layout.poolMetaKey(), encodePoolMeta(bad_pm)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, 256); }); } @@ -679,7 +661,7 @@ TEST(CASPoolMeta, ConcurrentCreateRace) foreign_pm.pool_id = foreign; foreign_pm.blob_header_len = 256; foreign_pm.algos_used = {static_cast(BlobHashAlgo::CityHash128)}; - b->putIfAbsent(layout.poolMetaKey(), encodePoolMeta(foreign_pm)); + createObj(*b, layout.poolMetaKey(), encodePoolMeta(foreign_pm)); PoolMeta result = PoolMeta::createOrValidate(*DB::Cas::tests::OperationForTest(b), layout, /*blob_header_len*/ 512); EXPECT_EQ(result.pool_id, foreign); @@ -689,12 +671,12 @@ TEST(CASPoolMeta, ConcurrentCreateRace) TEST(CASPoolMeta, CasConflictReReadsWinner) { /// The subtlest branch: the initial GET sees ABSENT, so createOrValidate proceeds to the - /// create-if-absent casPut — and loses, because a racing creator committed in between. The loser + /// create-if-absent write — and loses, because a racing creator committed in between. The loser /// must then re-read and return the WINNER's pool identity, not LOGICAL_ERROR. A single-threaded /// `refuseNextWrite` alone cannot exercise this: it returns Conflict without leaving the object /// readable, so the re-read would fire the LOGICAL_ERROR guard. We model the real interleaving - /// with a backend whose casPut commits the winner's object (via the public putIfAbsent) and THEN - /// reports Conflict — exactly what the loser observes. + /// with a backend whose write primitive commits the winner's object and THEN reports Conflict -- + /// exactly what the loser observes. class RacingBackend : public InMemoryBackend { public: @@ -832,7 +814,7 @@ TEST(CASPool, ResolveReturnsManifestId) EXPECT_EQ(loc.offset, s->poolMeta().blob_header_len); EXPECT_EQ(loc.length, payload.size()); - auto bytes = b->get(loc.key); + auto bytes = readObj(*b, loc.key); ASSERT_TRUE(bytes.has_value()); /// The located window holds exactly the payload: the envelope header is outside it. EXPECT_EQ(bytes->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), payload); @@ -866,7 +848,7 @@ TEST(CASPool, ReadManifestValidatesBodyAndFailsClosed) body.root_namespace_id = RootNamespace{"srv1/other"}; /// namespace does NOT body.entries = {blobEntryFor("f", u128Of("x"), 1)}; body.payload_digest = computePayloadDigest(body); - b->putIfAbsent(layout.manifestKey(addressed), encodePartManifest(body)); + createObj(*b, layout.manifestKey(addressed), encodePartManifest(body)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { s->readManifest(addressed); }); } @@ -882,7 +864,7 @@ TEST(CASPool, ReadManifestValidatesBodyAndFailsClosed) body.root_namespace_id = ns; /// namespace matches body.entries = {blobEntryFor("f", u128Of("y"), 1)}; body.payload_digest = computePayloadDigest(body); - b->putIfAbsent(layout.manifestKey(addressed), encodePartManifest(body)); + createObj(*b, layout.manifestKey(addressed), encodePartManifest(body)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { s->readManifest(addressed); }); } @@ -1252,7 +1234,7 @@ TEST(CASPool, ListRefsSkipsForeignKeys) /// A stray key directly under the namespace's ref-object prefix that is not `_log`/ /// `_snap` shaped (also covers the legacy shard-number layout GC/dropNamespace still write). - b->putIfAbsent(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "garbage", "not-a-ref-object"); + createObj(*b, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "garbage", "not-a-ref-object"); std::map refs; EXPECT_NO_THROW(refs = s->listRefs(ns)); @@ -1273,7 +1255,7 @@ TEST(CASPool, ReadManifestFailsClosed) { const ManifestRef ref = manifestRefFor("garbage-body"); const ManifestId id{.root_namespace = ns, .ref = ref}; - b->putIfAbsent(layout.manifestKey(id), "not a valid manifest body"); + createObj(*b, layout.manifestKey(id), "not a valid manifest body"); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { s->readManifest(id); }); } @@ -1413,8 +1395,7 @@ TEST(CASPool, ListNamespacesDoesNotMintLogicalNamesFromFileKeys) s->putNamespaceFile(DB::Cas::tests::fixture::fixtureLife(ns), "format_version.txt", "1\n"); /// A second life of the SAME name, written by exact key because no helper mints two lives yet. const NamespaceLifeId other = NamespaceLifeId::fromCatalogEntry(ns, DB::UInt128(0x5eed)); - ASSERT_EQ(b->putIfAbsent(s->layout().namespaceFileKey(other, "format_version.txt"), "1\n").outcome, - PutOutcome::Done); + createObj(*b, s->layout().namespaceFileKey(other, "format_version.txt"), "1\n"); const NamespaceListing listing = s->listNamespaces(""); EXPECT_TRUE(listing.skipped.empty()); @@ -1438,8 +1419,8 @@ TEST(CASPool, ListNamespacesDoesNotTreatPhysicalDebrisAsCatalogAuthority) const String lifeless_ref = s->layout().casRefsPrefix() + ns.string() + "/_log/" + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; const String lifeless_file = s->layout().rootsPrefix() + ns.string() + "/_files/format_version.txt"; - ASSERT_EQ(b->putIfAbsent(lifeless_ref, "garbage").outcome, PutOutcome::Done); - ASSERT_EQ(b->putIfAbsent(lifeless_file, "garbage").outcome, PutOutcome::Done); + createObj(*b, lifeless_ref, "garbage"); + createObj(*b, lifeless_file, "garbage"); NamespaceListing listing; ASSERT_NO_THROW(listing = s->listNamespaces("")) @@ -1451,8 +1432,8 @@ TEST(CASPool, ListNamespacesDoesNotTreatPhysicalDebrisAsCatalogAuthority) EXPECT_EQ(listing.namespaces[0], ns.string()); EXPECT_TRUE(listing.skipped.empty()); - EXPECT_TRUE(b->head(lifeless_ref).exists); - EXPECT_TRUE(b->head(lifeless_file).exists); + EXPECT_TRUE(headObj(*b, lifeless_ref).has_value()); + EXPECT_TRUE(headObj(*b, lifeless_file).has_value()); } TEST(CASPool, ListMirroredChildren) @@ -1475,7 +1456,7 @@ namespace /// Delegating backend that fences the mount slot IN PLACE the first time a `get` returns a present /// body for the armed key — reproducing the S13 window: the GC's token-guarded fence-out lands -/// between the keeper adopt's GET and its CAS. The caller's subsequent token-guarded `putOverwrite` +/// between the renewer adopt's GET and its CAS. The caller's subsequent token-guarded `putOverwrite` /// then fails `PreconditionFailed`, the adopt re-reads, sees `gc_fenced`, and throws /// `MountFencedException` — which `Pool::open`'s fence-recovery loop must turn into a fresh-epoch /// retry rather than a permanent wedge (P3.1 vector C). @@ -1485,21 +1466,9 @@ class FenceInAdoptWindowBackend final : public DB::Cas::Backend explicit FenceInAdoptWindowBackend(std::shared_ptr inner_) : inner(std::move(inner_)) {} String fence_key; /// empty = fault disarmed; set to the mount key to arm the one-shot fence - std::optional get(const String & k, DB::Cas::Range r) override { return inner->get(k, r); } - std::optional getStream(const String & k, DB::Cas::Range r) override { return inner->getStream(k, r); } - DB::Cas::HeadResult head(const String & k) override { return inner->head(k); } - DB::Cas::ListPage list(const String & p, const String & c, size_t l) override { return inner->list(p, c, l); } - DB::Cas::PutResult putIfAbsent(const String & k, const String & b, const DB::Cas::ObjectMeta & m) override { return inner->putIfAbsent(k, b, m); } - void publishBlob(const DB::Cas::BlobPublishRequest & request) override - { - inner->publishBlob(request); - } - DB::Cas::PutResult putOverwrite(const String & k, const String & b, const DB::Cas::Token & e, const DB::Cas::ObjectMeta & m) override { return inner->putOverwrite(k, b, e, m); } - DB::Cas::CasResult casPut(const String & k, const String & b, const std::optional & e, const DB::Cas::ObjectMeta & m) override { return inner->casPut(k, b, e, m); } - DB::Cas::DeleteOutcome deleteExact(const String & k, const DB::Cas::Token & t) override { return inner->deleteExact(k, t); } bool supportsListTokens() const override { return inner->supportsListTokens(); } - /// The fault sits on the READ PRIMITIVE: the keeper's adopt reads the mount slot through it. + /// The fault sits on the READ PRIMITIVE: the renewer's adopt reads the mount slot through it. std::optional read(const String & key, TransportAccess & access) override { auto got = inner->read(key, access); @@ -1539,7 +1508,7 @@ TEST(CASPoolMountFence, OpenRecoversFromFenceInAdoptWindowWithFreshEpoch) auto inner = std::make_shared(); auto fencing = std::make_shared(inner); /// Arm the one-shot fence on the mount slot. Pool::open first claims the mount (fresh mint), then - /// the keeper adopts it — the adopt's GET trips the fence, its CAS fails, and open must recover. + /// the renewer adopts it — the adopt's GET trips the fence, its CAS fails, and open must recover. const DB::Cas::Layout layout("p"); fencing->fence_key = layout.mountKey("test"); @@ -1559,7 +1528,7 @@ TEST(CASPoolMountFence, OpenRecoversFromFenceInAdoptWindowWithFreshEpoch) /// The final live lease is unfenced and at a HIGHER writer_epoch than the first attempt (a fence /// costs an epoch): the first claim took epoch 1, got fenced, the retry took epoch 2 and mounted. - const auto got = inner->get(layout.mountKey("test")); + const auto got = readObj(*inner, layout.mountKey("test")); ASSERT_TRUE(got.has_value()); const MountLease final_lease = decodeMountLease(got->bytes); EXPECT_FALSE(final_lease.gc_fenced); @@ -1605,13 +1574,14 @@ namespace /// GC's fence-out, applied directly: preserve the body, set gc_fenced, bump seq (token-guarded). void fenceOutMount(DB::Cas::Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, Retry::standard()); ASSERT_TRUE(got.has_value()); MountLease m = decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, encodeMountLease(m), got->token).outcome, - DB::Cas::PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(mount_key, encodeMountLease(m), got->etag, Retry::standard()))); } } @@ -1621,22 +1591,22 @@ TEST(CASPoolRemount, FenceOutThenSelfRemountRestoresWrites) auto backend = std::make_shared(); auto store = DB::Cas::tests::openPoolForTest(backend); const String mount_key = store->layout().mountKey("test"); - const uint64_t epoch_before = decodeMountLease(backend->get(mount_key)->bytes).writer_epoch; + const uint64_t epoch_before = decodeMountLease(readObj(*backend, mount_key)->bytes).writer_epoch; EXPECT_EQ(store->liveWriterEpoch(), epoch_before); fenceOutMount(*backend, mount_key); - /// The keeper's next renewal fails closed (foreign touch — never re-mint). + /// The renewer's next renewal fails closed (foreign touch — never re-mint). EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); /// Self-remount claims a FRESH incarnation: epoch bumped, gc_fenced cleared, writes restored. ASSERT_TRUE(store->tryRemountOnce()); - const MountLease after = decodeMountLease(backend->get(mount_key)->bytes); + const MountLease after = decodeMountLease(readObj(*backend, mount_key)->bytes); EXPECT_EQ(after.writer_epoch, epoch_before + 1); EXPECT_FALSE(after.gc_fenced); EXPECT_EQ(store->liveWriterEpoch(), epoch_before + 1); - /// The renewal path works again (the new keeper owns the slot). (The follow-on "...and so does a + /// The renewal path works again (the new renewer owns the slot). (The follow-on "...and so does a /// ref-shard mutation" check used `mutateShardForTest` -- the held Phase-E shard lane -- and moves /// to Phase E's own tests; the self-remount liveness assertion above is the point of this test.) EXPECT_NO_THROW(store->renewWatermarkOnce()); @@ -1673,19 +1643,20 @@ TEST(CASPoolRemount, ForeignOwnerIsNeverTakenOver) const String mount_key = store->layout().mountKey("test"); /// A genuinely foreign uuid holds the mount (live or not — foreign is terminal for the claim). - const auto got = backend->get(mount_key); + DB::Cas::tests::OperationForTest overwrite_op(*backend); + const auto got = (*overwrite_op).read(mount_key, Retry::standard()); MountLease foreign = decodeMountLease(got->bytes); foreign.server_uuid = foreign.server_uuid + DB::UInt128(1); foreign.seq += 1; - ASSERT_EQ(backend->putOverwrite(mount_key, encodeMountLease(foreign), got->token).outcome, - DB::Cas::PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*overwrite_op).replace(mount_key, encodeMountLease(foreign), got->etag, Retry::standard()))); EXPECT_FALSE(store->tryRemountOnce()); /// The foreign body is untouched (no takeover, ever). - EXPECT_EQ(decodeMountLease(backend->get(mount_key)->bytes).server_uuid, foreign.server_uuid); + EXPECT_EQ(decodeMountLease(readObj(*backend, mount_key)->bytes).server_uuid, foreign.server_uuid); /// Move the parent fixture to the production-recognized fenced terminal state before explicitly - /// destroying its superseded keeper. The unfenced foreign-release guard is covered separately below. + /// destroying its superseded renewer. The unfenced foreign-release guard is covered separately below. fenceOutMount(*backend, mount_key); store.reset(); @@ -1697,15 +1668,15 @@ TEST(CASPoolRemount, ForeignOwnerIsNeverTakenOver) auto foreign_backend = std::make_shared(); auto invalid_store = DB::Cas::tests::openPoolForTest(foreign_backend); const String foreign_mount_key = invalid_store->layout().mountKey("test"); - const auto foreign_got = foreign_backend->get(foreign_mount_key); + DB::Cas::tests::OperationForTest foreign_overwrite_op(*foreign_backend); + const auto foreign_got = (*foreign_overwrite_op).read(foreign_mount_key, Retry::standard()); ASSERT_TRUE(foreign_got.has_value()); MountLease foreign_lease = decodeMountLease(foreign_got->bytes); foreign_lease.server_uuid = foreign_lease.server_uuid + DB::UInt128(1); foreign_lease.seq += 1; - ASSERT_EQ( - foreign_backend->putOverwrite(foreign_mount_key, encodeMountLease(foreign_lease), foreign_got->token).outcome, - DB::Cas::PutOutcome::Done); - const auto occupant_before = foreign_backend->get(foreign_mount_key); + ASSERT_TRUE(std::holds_alternative((*foreign_overwrite_op).replace( + foreign_mount_key, encodeMountLease(foreign_lease), foreign_got->etag, Retry::standard()))); + const auto occupant_before = readObj(*foreign_backend, foreign_mount_key); ASSERT_TRUE(occupant_before.has_value()); EXPECT_FALSE(invalid_store->tryRemountOnce()) << "a foreign owner is never taken over at remount"; @@ -1717,7 +1688,7 @@ TEST(CASPoolRemount, ForeignOwnerIsNeverTakenOver) EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountExclusivityViolation].load(), violations_before + 1) << "the release must report the broken single-writer guarantee rather than dying on it"; - const auto occupant_after = foreign_backend->get(foreign_mount_key); + const auto occupant_after = readObj(*foreign_backend, foreign_mount_key); ASSERT_TRUE(occupant_after.has_value()) << "nor is it taken over at release"; EXPECT_EQ(occupant_after->bytes, occupant_before->bytes) << "the slot must be left byte-for-byte as the foreign owner wrote it"; @@ -1763,18 +1734,18 @@ struct SequencedBootClock } /// Phase B addendum 2 (task 5b review, reviewer's probe): the self-remount arm must anchor at the -/// claim attempt's pre-I/O instant (`remount_anchor_boot_ms`, captured right after `installKeeper` -/// and right before `keeperStart()` in `Pool::tryRemountOnce`), never at a later reading taken after -/// `keeperStart`/`quiesceRefTablesForRemount` have already run. +/// claim attempt's pre-I/O instant (`remount_anchor_boot_ms`, captured right after `installRenewer` +/// and right before `renewerStart()` in `Pool::tryRemountOnce`), never at a later reading taken after +/// `renewerStart`/`quiesceRefTablesForRemount` have already run. /// /// The two `bootMsNow()` calls of interest, in the ORDER each code version issues them: -/// - FIXED code: call #1 = the new anchor (`remount_anchor_boot_ms`, before `keeperStart`); -/// call #2 = `MountLeaseKeeper::prepareRenew`'s own internal boot read inside `keeperStart`'s -/// `doStart` (feeds only the keeper's OWN internal `confirmed_deadline_ms` -- unrelated to the +/// - FIXED code: call #1 = the new anchor (`remount_anchor_boot_ms`, before `renewerStart`); +/// call #2 = `MountLeaseRenewer::prepareRenew`'s own internal boot read inside `renewerStart`'s +/// `doStart` (feeds only the renewer's OWN internal `confirmed_deadline_ms` -- unrelated to the /// Pool-level arm -- so its value is irrelevant to the arm post-fix). /// - PRE-FIX code (no anchor line): call #1 = that SAME `prepareRenew` read (now the first boot -/// call of the attempt, since nothing reads the clock before `keeperStart`); call #2 = the -/// arm-site's own `mount_runtime.bootMsNow()`, read AFTER `keeperStart` returns -- the stale, +/// call of the attempt, since nothing reads the clock before `renewerStart`); call #2 = the +/// arm-site's own `mount_runtime.bootMsNow()`, read AFTER `renewerStart` returns -- the stale, /// response-time reading this whole fix exists to stop using. /// A sequenced clock returning 10000 then 11000 (a later response-time reading that remains inside /// the normal renewal window) therefore arms the FIXED code from 10000 and the PRE-FIX code from @@ -1812,7 +1783,7 @@ TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) clock.steady = 40000; EXPECT_FALSE(store->mayMutate()) << "the remount arm must anchor at the claim attempt's pre-I/O instant, not a later " - "response-time reading taken after keeperStart/quiesceRefTablesForRemount"; + "response-time reading taken after renewerStart/quiesceRefTablesForRemount"; } /// ==== rev.6 Task 5: clean-release drain gates the farewell marker ==== @@ -1900,7 +1871,7 @@ class RuntimeRenewBackend final : public DB::Cas::tests::CountingBackend } }; -CasRequestBudget runtimeRenewBudget(uint32_t max_attempts); +CasRequestBudget runtimeRenewBudget(); /// A directly-constructed `CasMountRuntime` plus the two request planes it needs. `Pool` builds those /// from its own members; a test has no `Pool`, so the mount plane's fence reaches the runtime through @@ -2003,21 +1974,23 @@ void verifyForeignConflictSinkIsNonInterfering(ForeignConflictSinkBehavior behav }, server_root_id, sink, - runtimeRenewBudget(1), + runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); - auto ours = backend->get(key); + DB::Cas::tests::OperationForTest successor_op(*backend); + auto ours = (*successor_op).read(key, Retry::standard()); ASSERT_TRUE(ours.has_value()); MountLease successor = decodeMountLease(ours->bytes); successor.server_uuid = UInt128{2}; successor.writer_epoch = 9; successor.seq += 1; - ASSERT_EQ(backend->putOverwrite(key, encodeMountLease(successor), ours->token).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*successor_op).replace(key, encodeMountLease(successor), ours->etag, Retry::standard()))); const uint64_t skipped_before = ProfileEvents::global_counters[ProfileEvents::CASMountReleaseSkippedForeignOccupant].load(); const uint64_t violations_before @@ -2063,7 +2036,7 @@ void verifyForeignConflictSinkIsNonInterfering(ForeignConflictSinkBehavior behav if (failed != events.end()) EXPECT_EQ(failed->detail.at("classification"), "conflict"); - const auto successor_before_teardown = backend->get(key); + const auto successor_before_teardown = readObj(*backend, key); ASSERT_TRUE(successor_before_teardown.has_value()); const uint64_t heads_before_teardown = backend->headCount(key); const uint64_t gets_before_teardown = backend->getCount(key); @@ -2077,7 +2050,7 @@ void verifyForeignConflictSinkIsNonInterfering(ForeignConflictSinkBehavior behav EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountReleaseSkippedForeignOccupant].load(), skipped_before_teardown); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountExclusivityViolation].load(), violations_before); - const auto successor_after_teardown = backend->get(key); + const auto successor_after_teardown = readObj(*backend, key); ASSERT_TRUE(successor_after_teardown.has_value()); EXPECT_EQ(successor_after_teardown->bytes, successor_before_teardown->bytes); } @@ -2151,7 +2124,7 @@ class ScopedParkedRenewalLogCapture { public: ScopedParkedRenewalLogCapture() - : logger(getLogger("CasMountLeaseKeeper")) + : logger(getLogger("CasMountLeaseRenewer")) , channel(new Poco::StreamChannel(stream)) , old_channel(logger->getChannel(), /*shared=*/true) , old_level(logger->getLevel()) @@ -2214,15 +2187,15 @@ class WorkerExitLatch uint64_t exits = 0; }; -CasRequestBudget runtimeRenewBudget(uint32_t max_attempts = 1) +/// The budget every runtime-renewal test uses. `attempt_timeout_ms`/`lease_safety_margin_ms` bound the +/// mount lease's own admission arithmetic; a renewal write's own attempt count and backoff are the +/// request engine's fence-derived `Retry::standard()` policy now, not a budget knob -- every caller of +/// this helper used to pass `max_attempts=1` and no other value, so that parameter carried nothing. +CasRequestBudget runtimeRenewBudget() { return CasRequestBudget{ .attempt_timeout_ms = 10, - .operation_deadline_ms = 500, - .max_attempts = max_attempts, .lease_safety_margin_ms = 20, - .retry_initial_backoff_ms = 0, - .retry_max_backoff_ms = 0, }; } } @@ -2236,7 +2209,7 @@ TEST(CASPoolShutdown, CleanStopDrainsAndWritesFarewell) const String mount_key = store->layout().mountKey("test"); store.reset(); /// drives ~Pool(): with no in-flight ref-log PUT, the drain must succeed. - const auto got = backend->get(mount_key); + const auto got = readObj(*backend, mount_key); ASSERT_TRUE(got.has_value()); const MountLease lease = decodeMountLease(got->bytes); EXPECT_EQ(lease.min_active_build_sequence, std::numeric_limits::max()) @@ -2246,9 +2219,7 @@ TEST(CASPoolShutdown, CleanStopDrainsAndWritesFarewell) TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) { CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); @@ -2280,7 +2251,7 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) const String mount_key = store->layout().mountKey("test"); store.reset(); /// drives ~Pool(): the still-wedged lane must skip the farewell marker. - const auto got = backend->get(mount_key); + const auto got = readObj(*backend, mount_key); ASSERT_TRUE(got.has_value()); const MountLease lease = decodeMountLease(got->bytes); EXPECT_NE(lease.min_active_build_sequence, std::numeric_limits::max()) @@ -2310,7 +2281,7 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) auto b = std::make_shared(); Layout l{"p"}; DB::Cas::tests::seedPoolMetaForRestart(*b); - /// Predecessor: claim epoch 7, no farewell (simulate crash: just drop the keeper) -- a bare + /// Predecessor: claim epoch 7, no farewell (simulate crash: just drop the renewer) -- a bare /// `claimMount` plants the lease directly, with no clean-farewell `min_active_build_sequence` marker and no /// `gc_fenced`, so the successor below has no certificate of death until it observes one itself. ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(b), l, "test", UInt128(1), /*epoch*/ 7, /*now_ms*/ 1000, /*ttl_ms*/ 500).kind, @@ -2318,13 +2289,13 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) /// A real predecessor at epoch 7 durably minted it first (`allocateWriterEpoch` always runs /// before the mount claim); seed that durable epoch object here too, or the successor's own /// `allocateWriterEpoch` trips the Phase C guard (epoch absent, mount present -> fail closed). - b->putIfAbsent(l.epochKey("test"), encodeServerEpoch(ServerEpoch{.next_writer_epoch = 8})); + createObj(*b, l.epochKey("test"), encodeServerEpoch(ServerEpoch{.next_writer_epoch = 8})); /// A 500ms lease TTL is far below the default `cas_request_budget` (RFC /// cas-s3-timeout-retry-control §required-timeout-model requires attempt_timeout + safety_margin < /// lease TTL), so scale the budget down to fit -- mirrors `CasMountStartup::StaleSelfMountReclaimedAfterWait`. const CasRequestBudget tiny_budget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = 500, .max_attempts = 1, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; uint64_t fake_boot = 0; std::vector waits; @@ -2386,14 +2357,14 @@ TEST(CASMountOpenWaits, FencedPriorReclaimsWithoutAnyWait) /// A real predecessor at epoch 7 durably minted it first (`allocateWriterEpoch` always runs /// before the mount claim); seed that durable epoch object here too, or the successor's own /// `allocateWriterEpoch` trips the Phase C guard (epoch absent, mount present -> fail closed). - b->putIfAbsent(l.epochKey("test"), encodeServerEpoch(ServerEpoch{.next_writer_epoch = 8})); + createObj(*b, l.epochKey("test"), encodeServerEpoch(ServerEpoch{.next_writer_epoch = 8})); /// Predecessor lease carries gc_fenced=true: fence it directly, exactly as `computeHeartbeatFloor`'s /// fence-out does (preserve the body, gc_fenced = true, seq + 1, token-guarded). fenceOutMount(*b, l.mountKey("test")); /// See UncleanOpenPaysOnlyTheObservationWindow above: a 500ms TTL needs a scaled-down budget too. const CasRequestBudget tiny_budget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = 500, .max_attempts = 1, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; std::vector waits; PoolPtr store; @@ -2418,7 +2389,7 @@ namespace /// Stalls the CLAIM ITSELF past the lease TTL, and counts what the open writes afterwards. /// /// The mount key is written twice before the write fence arms: once by `claimMount`'s reclaim, then -/// once by the keeper's adopt -- and the fence's anchor is taken BETWEEN them. So advancing the +/// once by the renewer's adopt -- and the fence's anchor is taken BETWEEN them. So advancing the /// injected boot clock on the SECOND write models exactly the thing the Phase B redo exists for: the /// claim's own I/O outliving the lease it is about to arm a fence under. (This used to be modelled by /// a materialization grace long enough to consume the TTL; that wait is retired, and the guard it @@ -2431,10 +2402,10 @@ class StalledMountClaimBackend final : public DB::Cas::InMemoryBackend std::atomic mount_writes{0}; std::atomic mount_writes_after_stall{0}; - /// The hook sits on the WRITE PRIMITIVE: both the reclaim and the keeper's adopt reach the mount + /// The hook sits on the WRITE PRIMITIVE: both the reclaim and the renewer's adopt reach the mount /// slot through it. It counts only CONDITIONAL overwrites, which is what every production mount-slot - /// write is -- the legacy `putIfAbsent` that seeds the predecessor lease forwards through this same - /// virtual, and counting it would shift the stall onto the reclaim instead of the adopt. + /// write is -- the unconditional create that seeds the predecessor lease reaches this same virtual + /// too, and counting it would shift the stall onto the reclaim instead of the adopt. std::expected write(const String & key, const String & bytes, const std::optional & expected_value, DB::Cas::TransportAccess & access) override { @@ -2478,12 +2449,12 @@ TEST(CASPool, StartupArmRedoesLeaseWriteWhenTheClaimConsumesTtl) prior.expires_at_ms = 1; /// long expired prior.gc_fenced = true; prior.write_attempt_id = DB::UInt128{7}; - backend->putIfAbsent(layout.mountKey(srid), DB::Cas::encodeMountLease(prior)); + createObj(*backend, layout.mountKey(srid), DB::Cas::encodeMountLease(prior)); } /// A real predecessor at epoch 7 durably minted it first (`allocateWriterEpoch` always runs /// before the mount claim); seed that durable epoch object here too, or `Pool::open`'s own /// `allocateWriterEpoch` trips the Phase C guard (epoch absent, mount present -> fail closed). - backend->putIfAbsent(layout.epochKey(srid), DB::Cas::encodeServerEpoch(DB::Cas::ServerEpoch{.next_writer_epoch = 8})); + createObj(*backend, layout.epochKey(srid), DB::Cas::encodeServerEpoch(DB::Cas::ServerEpoch{.next_writer_epoch = 8})); uint64_t fake_boot_ms = 10'000; DB::Cas::PoolConfig cfg; cfg.pool_prefix = "pool"; @@ -2492,7 +2463,7 @@ TEST(CASPool, StartupArmRedoesLeaseWriteWhenTheClaimConsumesTtl) cfg.background_watermark = true; cfg.mount_lease_ttl_ms = std::chrono::milliseconds(30'000); cfg.boot_ms_fn = [&] { return fake_boot_ms; }; - /// The keeper's adopt write stalls for 15 s of boot clock. That consumes the publication horizon + /// The renewer's adopt write stalls for 15 s of boot clock. That consumes the publication horizon /// (one 10 s cadence plus one 5 s attempt) while leaving one physical attempt admissible inside /// the old lease's safety window, so the synchronous redo can safely re-anchor. backend->on_second_mount_write = [&] { fake_boot_ms += 15'000; }; @@ -2501,7 +2472,7 @@ TEST(CASPool, StartupArmRedoesLeaseWriteWhenTheClaimConsumesTtl) ASSERT_NE(store, nullptr); ASSERT_EQ(backend->mount_writes.load(), 3) - << "the fixture assumes exactly two mount writes before the redo (the reclaim and the keeper's " + << "the fixture assumes exactly two mount writes before the redo (the reclaim and the renewer's " "adopt, with the fence anchor between them); a different sequence would make the stall land " "somewhere else and this test would stop testing the redo"; EXPECT_EQ(backend->mount_writes_after_stall.load(), 1) @@ -2549,9 +2520,7 @@ TEST(CASRemountWaits, DrainedRemountPaysNoWait) TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) { CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); @@ -2617,9 +2586,7 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPredecessorsDied) { CasRequestBudget budget; - budget.max_attempts = 1; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is a wall-clock race (validateCasRequestBudget) budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); @@ -2679,12 +2646,12 @@ TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPrede EXPECT_EQ(global_counters[ProfileEvents::CASRefRecoveryEpochSealed].load(), sealed_before + 2) << "both dead epochs must be closed -- the chain link is what a later reader needs to tell an " "EMPTY epoch from a LOST one, and that is independent of how each mount ended"; - EXPECT_TRUE(backend->get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns2), RefTxnId{1, 2})).has_value()) + EXPECT_TRUE(readObj(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns2), RefTxnId{1, 2})).has_value()) << "epoch 1 closes at the slot right after its last durable id, in-band"; - EXPECT_TRUE(backend->get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns2), RefTxnId{2, 1})).has_value()) + EXPECT_TRUE(readObj(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns2), RefTxnId{2, 1})).has_value()) << "empty epoch 2 closes at its own sequence 1, chained to the epoch-1 seal"; const RefTxnId retired_sentinel_id{2, std::numeric_limits::max()}; - EXPECT_FALSE(backend->get(layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns2), retired_sentinel_id)).has_value()) + EXPECT_FALSE(readObj(*backend, layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns2), retired_sentinel_id)).has_value()) << "and NO synthetic seal snapshot is written: that shape is retired"; } @@ -2777,14 +2744,16 @@ TEST(CASPool, StaleSnapshotServesCachedManifestAndBlobAbsenceSurfacesOnRead) /// What GC does after the owner is removed and the decrement is adopted: exact-token deletes of /// the body and of the now-unreferenced blob. { - const HeadResult h = b->head(manifest_key); - ASSERT_TRUE(h.exists); - b->deleteExact(manifest_key, h.token); + DB::Cas::tests::OperationForTest op(*b); + const auto h = (*op).head(manifest_key, Retry::standard()); + ASSERT_TRUE(h.has_value()); + (*op).remove(manifest_key, h->etag, Retry::once()); } { - const HeadResult h = b->head(blob_key); - ASSERT_TRUE(h.exists); - b->deleteExact(blob_key, h.token); + DB::Cas::tests::OperationForTest op(*b); + const auto h = (*op).head(blob_key, Retry::standard()); + ASSERT_TRUE(h.has_value()); + (*op).remove(blob_key, h->etag, Retry::once()); } b->resetCounts(); @@ -2797,7 +2766,7 @@ TEST(CASPool, StaleSnapshotServesCachedManifestAndBlobAbsenceSurfacesOnRead) const BlobLocation location = s->locate(m2->entries[0]); EXPECT_EQ(location.key, blob_key); EXPECT_EQ(b->getCount(blob_key), 0u); /// locate is pure: no I/O until the read - EXPECT_FALSE(b->get(location.key).has_value()); /// the read observes the absence + EXPECT_FALSE(readObj(*b, location.key).has_value()); /// the read observes the absence } /// The scoped contract for mutation evidence, executable. A carry-forward from a committed source @@ -2825,9 +2794,10 @@ TEST(CASPool, CachedSourceDecodeLetsAdoptionCommitAnAbsentBlobThatFsckReports) /// Out of band: both objects gone, the committed source ref untouched. for (const String & key : {src_manifest_key, blob_key}) { - const HeadResult h = b->head(key); - ASSERT_TRUE(h.exists); - b->deleteExact(key, h.token); + DB::Cas::tests::OperationForTest op(*b); + const auto h = (*op).head(key, Retry::standard()); + ASSERT_TRUE(h.has_value()); + (*op).remove(key, h->etag, Retry::once()); } b->resetCounts(); @@ -2872,7 +2842,7 @@ TEST(CASPool, CachedSourceDecodeLetsAdoptionCommitAnAbsentBlobThatFsckReports) #define EXPECT_RUNTIME_STATE_REJECTION(statement) EXPECT_THROW(statement, DB::Exception) #endif -TEST(CASPoolRemount, DirectRenewCannotRaceWorkerStartOrKeeperReplacement) +TEST(CASPoolRemount, DirectRenewCannotRaceWorkerStartOrRenewerReplacement) { auto backend = std::make_shared(); const Layout layout("runtime-direct"); @@ -2886,8 +2856,8 @@ TEST(CASPoolRemount, DirectRenewCannotRaceWorkerStartOrKeeperReplacement) .boot_ms_fn = [&] { return boot_ms; }}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); DB::Cas::tests::ManualBarrier barrier; @@ -2896,8 +2866,8 @@ TEST(CASPoolRemount, DirectRenewCannotRaceWorkerStartOrKeeperReplacement) auto direct = std::async(std::launch::async, [&] { runtime.renewWatermarkOnce(); }); barrier.waitUntilArrived(); EXPECT_RUNTIME_STATE_REJECTION(runtime.startBackgroundWorkers(std::chrono::milliseconds(10))); - EXPECT_RUNTIME_STATE_REJECTION(runtime.installKeeper(uuid, 2, [&] { return wall_ms; })); - EXPECT_RUNTIME_STATE_REJECTION(runtime.keeperReset()); + EXPECT_RUNTIME_STATE_REJECTION(runtime.installRenewer(uuid, 2, [&] { return wall_ms; })); + EXPECT_RUNTIME_STATE_REJECTION(runtime.renewerReset()); barrier.release(); EXPECT_NO_THROW(direct.get()); runtime.finishTeardown(true); @@ -2927,8 +2897,8 @@ TEST(CASPoolRemount, DueWorkerAdmissionIsReservedBeforeParkRequest) return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::milliseconds(0)); admitted.waitUntilArrived(); @@ -2962,8 +2932,8 @@ TEST(CASPoolRemount, DueWorkerAdmissionIsReservedBeforeStop) .renewal_admitted_hook_for_test = [&] { admitted.arriveAndWait(); }}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::milliseconds(0)); admitted.waitUntilArrived(); @@ -2990,8 +2960,8 @@ TEST(CASPoolRemount, DirectRenewIsRefusedForBackgroundConfiguredRuntimeAfterStop .boot_ms_fn = [&] { return boot_ms; }}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::hours(1)); runtime.stopBackgroundWorkers(); @@ -3024,8 +2994,8 @@ TEST(CASPoolRemount, RemountWaitsForRenewalParkedBeforeReplacement) return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); backend->barrier = &renewal_barrier; backend->fault = RuntimeRenewBackend::Fault::BlockThenDelegate; @@ -3067,14 +3037,14 @@ TEST(CASPoolRemount, TeardownJoinsBothWorkersBeforeRelease) .boot_ms_fn = [&] { return boot_ms; }, .worker_factory = factory}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::hours(1)); runtime.stopBackgroundWorkers(); EXPECT_EQ(worker_exits.load(), 2u); runtime.finishTeardown(true); - EXPECT_EQ(decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active_build_sequence, + EXPECT_EQ(decodeMountLease(readObj(*backend, layout.mountKey("test"))->bytes).min_active_build_sequence, std::numeric_limits::max()); } @@ -3120,8 +3090,8 @@ TEST(CASPoolRemount, NaturalTerminalTransitionMakesBothPersistentWorkersSelfExit }); CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::hours(1)); runtime.tripMountLost(); @@ -3229,8 +3199,8 @@ TEST(CASPoolRemount, ParkedRenewalCannotMissNaturalTerminalPublication) }); CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::hours(1)); renewal_before_driver_lock.wait(); @@ -3282,8 +3252,8 @@ TEST(CASPoolRemount, VanishedReasonPreparationFailureLeavesTerminalTransitionRet }}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::hours(1)); runtime.tripMountLost(); @@ -3334,8 +3304,8 @@ TEST(CASPoolRemount, WorkerConstructionRollbackFailsOpenClosed) .boot_ms_fn = [&] { return boot_ms; }, .worker_factory = factory}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); EXPECT_THROW(runtime.startBackgroundWorkers(std::chrono::milliseconds(10)), DB::Exception); EXPECT_FALSE(runtime.mayMutate()); @@ -3373,8 +3343,8 @@ TEST(CASPoolRemount, ExternalLossDuringRenewalUsesOneRecoveryGeneration) EXPECT_EQ(fresh.kind, MountClaimResult::Claimed); if (fresh.kind != MountClaimResult::Claimed) return false; - runtime_ptr->installKeeper(uuid, 2, [&] { return wall_ms; }); - const uint64_t fresh_anchor = runtime_ptr->startKeeper(); + runtime_ptr->installRenewer(uuid, 2, [&] { return wall_ms; }); + const uint64_t fresh_anchor = runtime_ptr->startRenewer(); runtime_ptr->setProcessEpoch(2, std::memory_order_release); runtime_ptr->setLiveWriterEpoch(2); runtime_ptr->armMountFence(uuid, 2, fresh_anchor + 1000); @@ -3384,8 +3354,8 @@ TEST(CASPoolRemount, ExternalLossDuringRenewalUsesOneRecoveryGeneration) }); CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); backend->barrier = &renewal_barrier; backend->fault = RuntimeRenewBackend::Fault::BlockThenDelegate; @@ -3405,7 +3375,7 @@ TEST(CASPoolRemount, ExternalLossDuringRenewalUsesOneRecoveryGeneration) runtime.finishTeardown(false); } -TEST(CASPoolRemount, TerminalDepositionDoesNotTouchKeeperAfterReplacement) +TEST(CASPoolRemount, TerminalDepositionDoesNotTouchRenewerAfterReplacement) { auto backend = std::make_shared(); const Layout layout("runtime-terminal-replacement"); @@ -3426,9 +3396,9 @@ TEST(CASPoolRemount, TerminalDepositionDoesNotTouchKeeperAfterReplacement) .boot_ms_fn = [&] { return boot_ms; }, .renewal_terminal_deposited_hook_for_test = [&] { - runtime_ptr->keeperReset(); - runtime_ptr->installKeeper(uuid, 2, [&] { return wall_ms; }); - runtime_ptr->keeperReset(); + runtime_ptr->renewerReset(); + runtime_ptr->installRenewer(uuid, 2, [&] { return wall_ms; }); + runtime_ptr->renewerReset(); replaced.store(true, std::memory_order_release); terminal_deposited.arriveAndWait(); }}, @@ -3439,8 +3409,8 @@ TEST(CASPoolRemount, TerminalDepositionDoesNotTouchKeeperAfterReplacement) }); CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; /// Expire the lease from inside the attempt. The fault alone no longer ends a renewal: the engine @@ -3481,8 +3451,8 @@ TEST(CASPoolRemount, ConcurrentRemountRequestIsProcessedAfterActiveGeneration) return true; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); runtime.startBackgroundWorkers(std::chrono::hours(1)); runtime.tripMountLost(); @@ -3527,8 +3497,8 @@ TEST(CASPoolRemount, ImmediatePostRemountRenewalFailureIsNotDropped) EXPECT_EQ(fresh.kind, MountClaimResult::Claimed); if (fresh.kind != MountClaimResult::Claimed) return false; - runtime_ptr->installKeeper(uuid, 2, [&] { return wall_ms; }); - const uint64_t fresh_anchor = runtime_ptr->startKeeper(); + runtime_ptr->installRenewer(uuid, 2, [&] { return wall_ms; }); + const uint64_t fresh_anchor = runtime_ptr->startRenewer(); runtime_ptr->armMountFence(uuid, 2, fresh_anchor + 10'000); runtime_ptr->noteRemounted(); boot_ms = 2'000; @@ -3545,8 +3515,8 @@ TEST(CASPoolRemount, ImmediatePostRemountRenewalFailureIsNotDropped) }); CasMountRuntime & runtime = *runtime_holder; runtime_ptr = &runtime; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 10'000); runtime.startBackgroundWorkers(std::chrono::milliseconds(1000)); runtime.tripMountLost(); @@ -3588,7 +3558,7 @@ TEST(CASPoolRemount, StaleRemountAnchorPerformsParkedRedo) ASSERT_TRUE(store->scheduleRemountForTest()); committed.waitUntilArrived(); EXPECT_GE(backend->putOverwriteCount(key), writes_before + 3) - << "claim, keeper start, and the stale-anchor parked redo must all write"; + << "claim, renewer start, and the stale-anchor parked redo must all write"; committed.release(); } @@ -3777,8 +3747,8 @@ TEST(CASPoolShutdown, PreSendCancellationAllowsFarewellButAmbiguityDoesNot) .boot_ms_fn = [&] { return boot_ms; }}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); if (ambiguous) { @@ -3796,7 +3766,7 @@ TEST(CASPoolShutdown, PreSendCancellationAllowsFarewellButAmbiguityDoesNot) runtime.stopBackgroundWorkers(); } runtime.finishTeardown(true); - return decodeMountLease(backend->get(layout.mountKey("test"))->bytes).min_active_build_sequence; + return decodeMountLease(readObj(*backend, layout.mountKey("test"))->bytes).min_active_build_sequence; }; EXPECT_EQ(run(false), std::numeric_limits::max()); @@ -3824,8 +3794,8 @@ TEST(CASPool, DirectAndStartupTerminalFailuresRethrowTypedExceptions) .renewal_live_for_test = [&] { return renewal_live.load(std::memory_order_acquire); }}, "test", sink, runtimeRenewBudget(), [] { return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); if (refusal == Refusal::PreAttemptDeadline) /// Past the point where the lease has more room left than the safety margin (deadline @@ -3836,7 +3806,7 @@ TEST(CASPool, DirectAndStartupTerminalFailuresRethrowTypedExceptions) try { if (startup) - (void)runtime.renewKeeperForStartupOnce(); + (void)runtime.renewRenewerForStartupOnce(); else runtime.renewWatermarkOnce(); ADD_FAILURE() << "terminal renewal did not propagate"; @@ -3931,8 +3901,8 @@ TEST(CASPool, DeterministicWorkerFailureFencesWithoutWaitingForCadence) return false; }); CasMountRuntime & runtime = *runtime_holder; - runtime.installKeeper(uuid, 1, [&] { return wall_ms; }); - const uint64_t anchor = runtime.startKeeper(); + runtime.installRenewer(uuid, 1, [&] { return wall_ms; }); + const uint64_t anchor = runtime.startRenewer(); runtime.armMountFence(uuid, 1, anchor + 1000); backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; /// Expire the lease from inside the attempt, so the ambiguity can be neither settled by a read nor @@ -4016,9 +3986,9 @@ TEST(CASPoolRemount, WholeChainResultsAreNumberedAndStepLabelled) EXPECT_EQ(countRemountFinalLogs(logs.captured()), 2u) << logs.captured(); } -/// The remount's `keeper_redo` step re-anchors the lease BEFORE `armMountFence`, so it runs with the +/// The remount's `renewer_redo` step re-anchors the lease BEFORE `armMountFence`, so it runs with the /// fence still latched lost. Admitted on the mount plane it could only ever give up, and every remount -/// that reached the step would fail -- so it renews on the keeper's open plane instead. +/// that reached the step would fail -- so it renews on the renewer's open plane instead. /// /// Driven the way production reaches the step, which is the only way it CAN be reached: the persistent /// renewal worker runs, `scheduleRemount` parks it, and the redo is the parked driver's one call. A @@ -4029,7 +3999,7 @@ TEST(CASPoolRemount, WholeChainResultsAreNumberedAndStepLabelled) /// renewal window fits and the step is skipped entirely. So the quiesce hook advances the injected boot /// clock to just inside the safety margin, and the paired run with no quiesce cost is the control that /// proves the step was reached rather than skipped. -TEST(CASPoolRemount, TheKeeperRedoRenewsOnTheOpenPlane) +TEST(CASPoolRemount, TheRenewerRedoRenewsOnTheOpenPlane) { /// One successful self-remount whose quiescence costs `quiesce_ms`; returns the conditional /// mount-slot writes it issued. Counted while the remount worker is still held inside the event @@ -4040,7 +4010,7 @@ TEST(CASPoolRemount, TheKeeperRedoRenewsOnTheOpenPlane) uint64_t fake_boot = 1'000'000; DB::Cas::tests::ManualBarrier committed; auto store = Pool::open(backend, PoolConfig{ - .pool_prefix = "remount-keeper-redo", + .pool_prefix = "remount-renewer-redo", .server_root_id = "test", .background_watermark = true, .event_sink = [&committed](const CasEvent & event) diff --git a/src/Disks/tests/gtest_cas_pool_meta.cpp b/src/Disks/tests/gtest_cas_pool_meta.cpp index fc2a668e1620..f82a5f8e6091 100644 --- a/src/Disks/tests/gtest_cas_pool_meta.cpp +++ b/src/Disks/tests/gtest_cas_pool_meta.cpp @@ -55,8 +55,10 @@ TEST(CASPoolMeta, AdmitOrValidateEndsAtTheDeadlineUnderPerpetualConflict) if (inside_hook) return; inside_hook = true; - if (auto cur = backend->get(key)) - (void)backend->putOverwrite(key, cur->bytes, cur->token); + auto hook_requests = DB::Cas::tests::openRequestsForTest(BackendPtr(backend)); + auto hook_op = hook_requests.admit(); + if (auto cur = hook_op.read(key, Retry::once())) + (void)hook_op.replace(key, cur->bytes, cur->etag, Retry::once()); inside_hook = false; }); diff --git a/src/Disks/tests/gtest_cas_probe.cpp b/src/Disks/tests/gtest_cas_probe.cpp index 4d87da4543f2..d7acb16a0ff3 100644 --- a/src/Disks/tests/gtest_cas_probe.cpp +++ b/src/Disks/tests/gtest_cas_probe.cpp @@ -30,7 +30,7 @@ TEST(CASProbe, PassesOnEnforcingBackend) auto requests = makeRequests(*b); auto op = requests.admit(); EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")); - EXPECT_TRUE(b->list("p/.cas_probe", "", 10).keys.empty()); // probe cleans up after itself + EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()); // probe cleans up after itself } TEST(CASProbe, FailsClosedOnNonEnforcingDelete) @@ -73,9 +73,12 @@ TEST(CASProbe, PassesOnEmulatedLocal) TEST(CASProbe, ConcurrentMountsDoNotCollide) { auto b = std::make_shared(); + auto probe_requests = makeRequests(*b); + auto probe_op = probe_requests.admit(); /// Simulate a concurrent mounter whose probe object under the legacy fixed key is still present. - ASSERT_EQ(b->putIfAbsent("p/_probe/token", "concurrent-mounter-in-flight").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + probe_op.create("p/_probe/token", "concurrent-mounter-in-flight", Retry::once()))); /// A real (second) mount over the same shared pool must still succeed — its probe runs under a /// fresh per-mount-unique prefix and never touches the seeded fixed key. @@ -85,7 +88,7 @@ TEST(CASProbe, ConcurrentMountsDoNotCollide) EXPECT_NO_THROW(Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"})); /// The seeded fixed-key artifact is untouched (the probe never collided with it). - EXPECT_TRUE(b->get("p/_probe/token").has_value()); + EXPECT_TRUE(probe_op.read("p/_probe/token", Retry::once()).has_value()); } /// RFC cas-s3-timeout-retry-control: a Native-mode mount over an object storage that does not support @@ -205,7 +208,7 @@ class DialectOverrideBackend : public InMemoryBackend } -/// The probe's reordered "wrong incarnation" steps always reuse a REAL, backend-minted `Incarnation` +/// The probe's reordered "wrong incarnation" steps always reuse a REAL, backend-minted `Etag` /// from the same key rather than a synthesized value — see CasProbe.cpp's step comments — and every /// such value is grammar-valid under every dialect by construction (`InMemoryBackend`'s minted values are /// a monotonically increasing decimal starting at "1": non-empty and comma/`*`-free for ETag, a canonical @@ -222,7 +225,7 @@ TEST(CASProbe, ReorderedProbePassesOnAllThreeDialects) auto requests = makeRequests(b); auto op = requests.admit(); EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")) << "dialect " << static_cast(dialect); - EXPECT_TRUE(b.list("p/.cas_probe", "", 10).keys.empty()) << "dialect " << static_cast(dialect); + EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()) << "dialect " << static_cast(dialect); EXPECT_EQ(b.write_reached, 4) << "dialect " << static_cast(dialect); EXPECT_EQ(b.remove_reached, 2) << "dialect " << static_cast(dialect); } diff --git a/src/Disks/tests/gtest_cas_protocol_scenarios.cpp b/src/Disks/tests/gtest_cas_protocol_scenarios.cpp index 422cc934c67d..f33018415c0a 100644 --- a/src/Disks/tests/gtest_cas_protocol_scenarios.cpp +++ b/src/Disks/tests/gtest_cas_protocol_scenarios.cpp @@ -61,18 +61,18 @@ PoolPtr openPool(const std::shared_ptr & b) /// The object's incarnation as the store reports it now -- what these scenarios compare when they /// assert an object was, or was not, displaced. -Incarnation currentIncarnation(Backend & b, const String & key) +Etag currentIncarnation(Backend & b, const String & key) { DB::Cas::tests::OperationForTest operation(b); const std::optional meta = (*operation).head(key, Retry::standard()); if (!meta) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "object {} is absent", key); - return meta->incarnation; + return meta->etag; } /// An exact-incarnation delete attempt, for the scenarios whose discriminator is that a displaced /// incarnation can never be current again. -Removal removeAtIncarnation(Backend & b, const String & key, const Incarnation & seen) +Removal removeAtIncarnation(Backend & b, const String & key, const Etag & seen) { DB::Cas::tests::OperationForTest operation(b); return (*operation).remove(key, seen, Retry::standard()); @@ -140,7 +140,8 @@ void assertPartReads( const auto * entry = findEntry(manifest.entries, path); ASSERT_TRUE(entry != nullptr); auto loc = s->locate(*entry); - auto got = b->get(loc.key); + DB::Cas::tests::OperationForTest op(*b); + auto got = (*op).read(loc.key, Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes.substr(static_cast(loc.offset), static_cast(loc.length)), payload); } @@ -165,11 +166,11 @@ TEST(CASProtocol, FenceConflictCondemnedTokenedBlobCommitsWithTokenUnchanged) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); /// GC condemns X at t0 in round 1 and fences the namespace to round 1. injectRetire(*b, s->layout(), /*round*/ 1, /*shard*/ 0, - {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedIncarnation::capture(t0), .size = 9}}); + {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedEtag::capture(t0), .size = 9}}); /// promote: mutateShard refreshes the view (fence_round 1 > view round 0), but the materialized leaf is /// edge-protected — skipped, not re-validated ⇒ commit, token unchanged. @@ -192,7 +193,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenKeepsWhenUnchanged) /// X pre-exists out-of-band; the build dedup-adopts it via putBlob (records the current token t0). writeBlobRaw(*b, s->layout(), "payload-X", s->poolMeta().blob_header_len, s->poolMeta().pool_id); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); /// Wiring order: stage + precommit (durable edge) BEFORE the adopting putBlob. auto build = startBuildFor(s, ns, "part_1"); @@ -224,7 +225,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenAdoptsWhenDisplaced) writeBlobRaw(*b, s->layout(), "payload-X", s->poolMeta().blob_header_len, s->poolMeta().pool_id); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); auto build = startBuildFor(s, ns, "part_1"); /// Wiring order (EDGE-BEFORE-OBSERVE): stageManifest -> precommitAdd -> putBlob. @@ -233,7 +234,7 @@ TEST(CASProtocol, RevalidateReObservesStaleTokenAdoptsWhenDisplaced) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); /// dedup → adopts t0 /// Another writer displaces X out-of-band ⇒ a new current token t1 (same payload, fresh tag). - const Incarnation t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); + const Etag t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); EXPECT_NE(t1, t0); /// GC advanced to round 1 with an EMPTY retired set; fence to 1. @@ -270,8 +271,8 @@ TEST(CASProtocol, RevalidateAdoptsLiveTokenWhenOnlyPhantomCondemnedAtDifferentTo writeBlobRaw(*b, s0->layout(), "payload-X", s0->poolMeta().blob_header_len, s0->poolMeta().pool_id); } const String blob_key = layout.blobKey(idOf("payload-X")); - const Incarnation t0 = currentIncarnation(*b, blob_key); - const PersistedIncarnation t_other{"emulated", "emulated-phantom"}; + const Etag t0 = currentIncarnation(*b, blob_key); + const PersistedEtag t_other{"emulated", "emulated-phantom"}; ASSERT_FALSE(t_other.matches(t0)) << "the phantom must name a DIFFERENT incarnation than the live one"; injectRetire(*b, layout, /*round*/ 1, /*shard*/ 0, @@ -318,7 +319,7 @@ TEST(CASProtocol, EvidenceHitCondemnedPresentBlobCopiesForwardInClosure) const BlobRef seeded_ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128(hex))}; seedBlobWithDurablePrecommit(s, seeded_ref, "payload-X"); const String blob_key = s->layout().blobKey(seeded_ref); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); auto build = startBuildFor(s, ns, "part_1"); ManifestEntry entry = blobEntry("data.bin", "payload-X"); @@ -363,11 +364,11 @@ TEST(CASProtocol, WedgedHeartbeatCondemnedTokenedBlobCommitsWithTokenUnchanged) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); /// Full GC condemned the build's OWN upload. injectRetire(*b, s->layout(), /*round*/ 1, /*shard*/ 0, - {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedIncarnation::capture(t0), .size = 9}}); + {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedEtag::capture(t0), .size = 9}}); /// promote: the materialized leaf is edge-protected — skipped, not revalidated ⇒ commit, token unchanged. build->promote(ns, "part_1", build->buildId(), id); @@ -392,8 +393,11 @@ TEST(CASProtocol, AbandonLeavesDebrisAndDisables) /// Both bodies remain as debris: once the manifest has named a durable precommit edge, its body /// must survive until GC folds the matching owner removal. - EXPECT_TRUE(b->head(s->layout().blobKey(blob.ref)).exists); - EXPECT_TRUE(b->head(s->layout().manifestKey(id)).exists); + { + DB::Cas::tests::OperationForTest op(*b); + EXPECT_TRUE((*op).head(s->layout().blobKey(blob.ref), Retry::once()).has_value()); + EXPECT_TRUE((*op).head(s->layout().manifestKey(id), Retry::once()).has_value()); + } EXPECT_TRUE(s->listRefs(ns).empty()); /// Further build ops ⇒ LOGICAL_ERROR (requireAlive). @@ -418,7 +422,7 @@ TEST(CASProtocol, DropReattachThroughDetachedNamespace) publishBlobPart(s, ns, "part_1", "data.bin", "payload-X"); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Incarnation blob_tok = currentIncarnation(*b, blob_key); + const Etag blob_tok = currentIncarnation(*b, blob_key); EXPECT_TRUE(s->listRefs(ns).contains("part_1")); EXPECT_TRUE(s->listRefs(detached).empty()); @@ -477,7 +481,7 @@ TEST(CASProtocol, DisplacedToLiveTokenCommitsAtCurrentIncarnation) writeBlobRaw(*b, s->layout(), "payload-X", s->poolMeta().blob_header_len, s->poolMeta().pool_id); const String blob_key = s->layout().blobKey(idOf("payload-X")); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); auto build = startBuildFor(s, ns, "part_1"); /// Wiring order (EDGE-BEFORE-OBSERVE): stageManifest -> precommitAdd -> putBlob. @@ -486,12 +490,12 @@ TEST(CASProtocol, DisplacedToLiveTokenCommitsAtCurrentIncarnation) build->putBlob(idOf("payload-X"), BlobSource::fromString("payload-X")); /// dedup → adopts t0 /// Another writer displaces X to t1 (uncondemned) before our gate runs. - const Incarnation t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); + const Etag t1 = displaceBlobToken(*b, s->layout(), idOf("payload-X")); ASSERT_NE(t1, t0); /// The view still condemns the OLD t0 at round 1, fenced. injectRetire(*b, s->layout(), /*round*/ 1, /*shard*/ 0, - {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedIncarnation::capture(t0), .size = 9}}); + {RetiredEntry{.kind = ObjectKind::Blob, .ref = DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of("payload-X"))}, .token = PersistedEtag::capture(t0), .size = 9}}); /// promote: revalidate X ⇒ HEAD current t1 (NOT condemned; only the defunct t0 is) ⇒ commit. build->promote(ns, "part_1", build->buildId(), id); @@ -538,15 +542,16 @@ TEST(CASProtocol, NewNamespacePublishGatedByShardFenceFloor) build_a.reset(); s->renewWatermarkOnce(); Gc gc(s, hexToU128("00000000000000000000000000000001")); + DB::Cas::tests::OperationForTest blob_op(*b); for (size_t r = 0; r < 16; ++r) { const RoundReport rep = DB::Cas::tests::runRegularRoundReclaiming(gc); s->renewWatermarkOnce(); - if (!b->head(blob_key).exists) + if (!(*blob_op).head(blob_key, Retry::once()).has_value()) break; } /// The blob (unreachable) was deleted at t0. - EXPECT_FALSE(b->head(blob_key).exists); + EXPECT_FALSE((*blob_op).head(blob_key, Retry::once()).has_value()); /// 4. build B publishes into a BRAND-NEW namespace. §4 manifest-trust: the adopted leaf is trusted at /// promote (no probe) ⇒ promote SUCCEEDS and commits a manifest naming the deleted blob (the dangle). @@ -582,7 +587,7 @@ TEST(CASProtocol, FreshEvidenceDepWithViewHitIsResolvedByGate) "payload-fresh-ev"); } const String blob_key = layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128(hex))}); - const Incarnation t0 = currentIncarnation(*b, blob_key); + const Etag t0 = currentIncarnation(*b, blob_key); condemnMeta(*b, layout, hexToU128(hex), /*condemn_round*/ 1); auto s = openPool(b); diff --git a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp index 5973f35d0c48..3fd231d9de78 100644 --- a/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp +++ b/src/Disks/tests/gtest_cas_rebuild_condemn_nothing.cpp @@ -92,9 +92,21 @@ BlobRef blobRefOf(const DB::UInt128 & hash) return BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)}; } +std::optional readOf(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +bool headExists(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + bool blobPresent(Backend & backend, const Layout & layout, const DB::UInt128 & hash) { - return backend.head(layout.blobKey(blobRefOf(hash))).exists; + return headExists(backend, layout.blobKey(blobRefOf(hash))); } /// Whether ANY run the newest fold seal references carries a `RunMarker::Condemned` row for `hash`. This is where @@ -102,12 +114,12 @@ bool blobPresent(Backend & backend, const Layout & layout, const DB::UInt128 & h /// than by watching for a deletion several rounds later. bool condemnedInSealedRuns(Backend & backend, const Layout & layout, const DB::UInt128 & hash) { - const GcState st = decodeGcState(backend.get(layout.gcStateKey())->bytes); - const auto sealed = backend.get(layout.foldSealKey(st.snap_generation, st.snap_attempt)); + DB::Cas::tests::OperationForTest operation(backend); + const GcState st = decodeGcState((*operation).read(layout.gcStateKey(), Retry::standard())->bytes); + const auto sealed = (*operation).read(layout.foldSealKey(st.snap_generation, st.snap_attempt), Retry::standard()); if (!sealed) return false; const CasFoldSeal seal = decodeFoldSeal(sealed->bytes); - DB::Cas::tests::OperationForTest operation(backend); for (const RunRef & r : seal.blob_target_runs) { auto reader = openSourceEdgeRun(*operation, r.key); @@ -156,7 +168,7 @@ RefTableState stateAfter(Backend & backend, const Layout & layout, const RootNam RefReplayBuilder builder(std::nullopt); for (const RefTxnId & id : ids) { - const auto got = backend.get(layout.refLogKey(fixture::fixtureLife(ns), id)); + const auto got = readOf(backend, layout.refLogKey(fixture::fixtureLife(ns), id)); if (!got) throw std::runtime_error("stateAfter: fixture log " + std::to_string(id.writer_epoch) + "-" + std::to_string(id.ref_sequence) + " is missing"); @@ -220,8 +232,8 @@ TEST(CASRebuildCondemnNothing, HiddenLiveManifestBlobIsNotCondemned) backend->hide(hidden_manifest); /// Precondition: both objects really are durable and really are hidden. - ASSERT_TRUE(backend->get(layout.refLogKey(fixture::fixtureLife(kNsA), RefTxnId{1, 2})).has_value()); - ASSERT_TRUE(backend->get(hidden_manifest).has_value()); + ASSERT_TRUE(readOf(*backend, layout.refLogKey(fixture::fixtureLife(kNsA), RefTxnId{1, 2})).has_value()); + ASSERT_TRUE(readOf(*backend, hidden_manifest).has_value()); Gc gc(store, kGc); const RebuildReport rep = gc.rebuildBaseline(/*force=*/true); @@ -267,8 +279,8 @@ TEST(CASRebuildCondemnNothing, OrphanBlobIsRetainedNotCondemned) ASSERT_TRUE(rep.performed) << rep.refusal; EXPECT_FALSE(condemnedInSealedRuns(*backend, layout, DB::UInt128(2))); - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); - const CasFoldSeal seal = decodeFoldSeal(backend->get(layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + const GcState st = decodeGcState(readOf(*backend, layout.gcStateKey())->bytes); + const CasFoldSeal seal = decodeFoldSeal(readOf(*backend, layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); ASSERT_TRUE(seal.condemned_summary.contains(0)) << "the summary stays TOTAL over gc_shards"; EXPECT_EQ(seal.condemned_summary.at(0).condemned_total, 0u) << "a rebuild condemns nothing"; @@ -299,7 +311,10 @@ TEST(CASRebuildCondemnNothing, NonCanonicalLifeKeyDoesNotAbortTheRebuild) /// Hand-built: no helper can mint this shape any more. const String noncanonical_life = layout.casRefsPrefix() + kNsA.string() + "/_log/" + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; - ASSERT_EQ(backend->putIfAbsent(noncanonical_life, "garbage").outcome, PutOutcome::Done); + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).create(noncanonical_life, "garbage", Retry::standard()))); + } Gc gc(store, kGc); RebuildReport rep; @@ -335,19 +350,23 @@ TEST(CASRebuildCondemnNothing, NestedLifelessKeyUnderTheLifePrefixDoesNotAbortTh Gc gc(store, kGc); ASSERT_TRUE(gc.runRegularRound().acquired_lease); { - const auto got = backend->get(layout.gcStateKey()); + const auto got = readOf(*backend, layout.gcStateKey()); ASSERT_TRUE(got.has_value()); GcState st = decodeGcState(got->bytes); st.snap_generation = 0; - ASSERT_EQ(backend->putOverwrite(layout.gcStateKey(), encodeGcState(st), got->token).outcome, - PutOutcome::Done); + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.gcStateKey(), encodeGcState(st), got->etag, Retry::standard()))); } /// Hand-built, and planted AFTER the round so the round itself is clean: one segment too deep under /// the life prefix, so the segment where the incarnation belongs holds `x`. No helper mints this. const String nested = layout.namespaceStreamPrefix(fixture::fixtureLife(kNsA)) + "x/_log/" + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; - ASSERT_EQ(backend->putIfAbsent(nested, "garbage").outcome, PutOutcome::Done); + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).create(nested, "garbage", Retry::standard()))); + } RebuildReport rep; ASSERT_NO_THROW(rep = gc.rebuildBaseline(/*force=*/false)) @@ -376,12 +395,13 @@ TEST(CASRebuildCondemnNothing, OneCatalogCutDrivesHealthCheckAndRebuild) Gc gc(store, kGc); ASSERT_TRUE(gc.runRegularRound().acquired_lease); { - const auto got = backend->get(layout.gcStateKey()); + const auto got = readOf(*backend, layout.gcStateKey()); ASSERT_TRUE(got); GcState state = decodeGcState(got->bytes); state.snap_generation = 0; - ASSERT_EQ(backend->putOverwrite( - layout.gcStateKey(), encodeGcState(state), got->token).outcome, PutOutcome::Done); + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(layout.gcStateKey(), encodeGcState(state), got->etag, Retry::standard()))); } backend->armCatalogMutation(layout.refCatalogKey()); @@ -416,7 +436,7 @@ TEST(CASRebuildCondemnNothing, CarriesHoldsVerbatimWhileCondemningNothing) const RefHold planted{.reason = HoldReason::GapBelowWitness, .offending_position = RefTxnId{4, 9}, .retry_count = 17, .next_retry_round = 23}; { - const GcState adopted = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState adopted = decodeGcState(readOf(*backend, layout.gcStateKey())->bytes); seedFoldCursorForTest(*backend, layout, kNsA, RefTxnId{1, 1}, planted, adopted.snap_generation, adopted.snap_attempt); } @@ -424,8 +444,8 @@ TEST(CASRebuildCondemnNothing, CarriesHoldsVerbatimWhileCondemningNothing) const RebuildReport rep = gc.rebuildBaseline(/*force=*/true); ASSERT_TRUE(rep.performed) << rep.refusal; - const GcState st = decodeGcState(backend->get(layout.gcStateKey())->bytes); - const CasFoldSeal seal = decodeFoldSeal(backend->get(layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); + const GcState st = decodeGcState(readOf(*backend, layout.gcStateKey())->bytes); + const CasFoldSeal seal = decodeFoldSeal(readOf(*backend, layout.foldSealKey(st.snap_generation, st.snap_attempt))->bytes); const auto it = seal.ref_lives.find(catalogLifeIdForTest(*backend, layout, kNsA)); ASSERT_NE(it, seal.ref_lives.end()); EXPECT_EQ(it->second.coverage.classification, CoverageClass::Clamped); @@ -480,9 +500,10 @@ TEST(CASRebuildCondemnNothingFsck, MidChainHoleBelowAWitnessIsChainBroken) /// Punch the hole: {1,2} is gone while {1,3} stays durable and listed. const String holed = layout.refLogKey(fixture::fixtureLife(kNsA), RefTxnId{1, 2}); - const HeadResult h = backend->head(holed); - ASSERT_TRUE(h.exists); - backend->deleteExact(holed, h.token); + OperationForTest hole_op(*backend); + const auto h = (*hole_op).head(holed, Retry::standard()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ((*hole_op).remove(holed, h->etag, Retry::standard()), Removal::Removed); FsckReport rep; ASSERT_NO_THROW(rep = runFsck(*store, /*detail=*/true)) @@ -616,9 +637,10 @@ TEST(CASRebuildCondemnNothingFsck, OneBadNamespaceDoesNotAbortTheAudit) RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 3}, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); const String holed = layout.refLogKey(fixture::fixtureLife(kNsA), RefTxnId{1, 2}); - const HeadResult h = backend->head(holed); - ASSERT_TRUE(h.exists); - backend->deleteExact(holed, h.token); + OperationForTest hole_op(*backend); + const auto h = (*hole_op).head(holed, Retry::standard()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ((*hole_op).remove(holed, h->etag, Retry::standard()), Removal::Removed); publishAt(*backend, layout, kNsB, RefTxnId{1, 1}, "ref_z", 1, DB::UInt128(9), /*birth=*/true); writeCkptRaw(*backend, layout, kNsB, diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index 25c4a36a465d..a5309f4d916a 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -37,7 +37,7 @@ SourceEdgeRecord zero(const BlobRef & ref) return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Zero}; } -SourceEdgeRecord condemned(const BlobRef & ref, const PersistedIncarnation & token, uint64_t size, uint64_t round, bool pend) +SourceEdgeRecord condemned(const BlobRef & ref, const PersistedEtag & token, uint64_t size, uint64_t round, bool pend) { return SourceEdgeRecord{.ref = ref, .source_id = UInt128(0), .marker = RunMarker::Condemned, .delete_pending = pend, .token = token, .size = size, .condemn_round = round}; @@ -105,7 +105,7 @@ TEST(CASRecordStream, EdgeZeroCondemnedRoundTrip) /// an edge; c has a zero marker. Blobs ascend a < b < c, so the sequence is already non-decreasing. std::vector recs = { edge(a, 10), - condemned(b, PersistedIncarnation{"etag", "e-1"}, 4242, 7, /*pend*/ true), + condemned(b, PersistedEtag{"etag", "e-1"}, 4242, 7, /*pend*/ true), zero(c), }; const String bytes = encodeRun(recs); @@ -146,7 +146,7 @@ TEST(CASRecordStream, ClosedSetPinsRunMarkerWords) /// which is a different retention decision than the writer recorded. TEST(CASRecordStream, CondemnedRowMissingOneOfItsSixFieldsFailsClosed) { - const String good = encodeRun({condemned(chRef(2), PersistedIncarnation{"etag", "e-1"}, 4242, 7, /*pend*/ true)}); + const String good = encodeRun({condemned(chRef(2), PersistedEtag{"etag", "e-1"}, 4242, 7, /*pend*/ true)}); for (const std::string_view field : {R"(,"pending":true)", R"(,"token_type":"etag")", R"(,"token":"e-1")", R"(,"size":4242)", R"(,"condemn_round":"7")", R"(,"confirmed":false)"}) { @@ -197,7 +197,7 @@ TEST(CASRecordStream, WriterIsByteDeterministic) std::vector recs = { edge(chRef(1), 5), edge(chRef(1), 9), - condemned(chRef(2), PersistedIncarnation{"etag", "t/with/slashes"}, 1, 2, false), + condemned(chRef(2), PersistedEtag{"etag", "t/with/slashes"}, 1, 2, false), }; EXPECT_EQ(encodeRun(recs), encodeRun(recs)); /// pure function of the sorted record set } diff --git a/src/Disks/tests/gtest_cas_recovery_grounding.cpp b/src/Disks/tests/gtest_cas_recovery_grounding.cpp index ddf267df7e22..3e31f4f5650c 100644 --- a/src/Disks/tests/gtest_cas_recovery_grounding.cpp +++ b/src/Disks/tests/gtest_cas_recovery_grounding.cpp @@ -375,8 +375,7 @@ TEST(CASRecoveryGrounding, CatalogLifecycleAndCheckpointAreMandatoryForReadOnlyR const CatalogEntry live{.ns = ns, .state = NsState::Live, .incarnation = 9}; CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, live); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(live.ns, live.incarnation); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), "not a sealed checkpoint").outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(catalog_op.create(layout.refCkptKey(life), "not a sealed checkpoint", Retry::once()))); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); } { @@ -386,18 +385,18 @@ TEST(CASRecoveryGrounding, CatalogLifecycleAndCheckpointAreMandatoryForReadOnlyR CatalogEntry creating{.ns = ns, .state = NsState::Creating, .incarnation = 7, .creator = CreatorFence{"srv1", 1, 1}}; CasRefCatalog::casAdmitEntry(catalog_op, layout, 1, creating); - backend->putIfAbsent(layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation)), + catalog_op.create(layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(creating.ns, creating.incarnation)), encodeRefCkpt(RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 1}, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})); + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}), Retry::once()); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::INVALID_STATE); } { auto backend = std::make_shared(ListingMode::Full); CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(backend); CasOperation catalog_op = catalog_requests.admit(); - backend->putIfAbsent(layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)), + catalog_op.create(layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)), encodeRefCkpt(RefCkpt{.life_epoch = 1, .committed_through = RefTxnId{1, 1}, - .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt})); + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}), Retry::once()); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::INVALID_STATE); } } @@ -523,11 +522,11 @@ TEST(CASRecoveryGrounding, SemanticallyMalformedCheckpointSnapshotIsCorruptionAf const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); const String snapshot_key = layout.refSnapshotKey(life, {1, 1}); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(catalog_op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = RefTxnId{1, 1}, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}), Retry::once()))); backend->resetCounts(); try @@ -572,7 +571,7 @@ TEST(CASRecoveryGrounding, CheckpointSnapshotEqualToLastEpochSealIsRejectedBefor .checkpoint_snapshot_id = RefTxnId{1, 2}, .last_epoch_seal = RefTxnId{1, 2}}; ASSERT_TRUE(std::holds_alternative(catalog_op.replace( - layout.refCkptKey(life), encodeRefCkpt(with_sealed_base), before.incarnation, Retry::standard()))); + layout.refCkptKey(life), encodeRefCkpt(with_sealed_base), before.etag, Retry::standard()))); backend->resetCounts(); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -608,7 +607,7 @@ TEST(CASRecoveryGrounding, SameEpochFrontierAfterDecodedEpochSealIsCorruption) const size_t frontier_sequence = malformed_ckpt.find(R"("committed_seq":"2")"); ASSERT_NE(frontier_sequence, String::npos); malformed_ckpt.replace(frontier_sequence, String{R"("committed_seq":"2")"}.size(), R"("committed_seq":"3")"); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(catalog_op.create(layout.refCkptKey(life), malformed_ckpt, Retry::once()))); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); } @@ -646,7 +645,7 @@ TEST(CASRecoveryGrounding, OlderCheckpointSnapshotAtSealIsCorruption) .checkpoint_snapshot_id = RefTxnId{1, 2}, .last_epoch_seal = RefTxnId{2, 2}}; ASSERT_TRUE(std::holds_alternative(catalog_op.replace( - layout.refCkptKey(life), encodeRefCkpt(with_old_sealed_base), before.incarnation, Retry::standard()))); + layout.refCkptKey(life), encodeRefCkpt(with_old_sealed_base), before.etag, Retry::standard()))); backend->resetCounts(); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); @@ -679,7 +678,7 @@ TEST(CASRecoveryGrounding, TerminalGapBelowFrontierIsCorruptionNotARebirth) const size_t frontier_epoch = malformed_ckpt.find(R"("committed_epoch":"1")"); ASSERT_NE(frontier_epoch, String::npos); malformed_ckpt.replace(frontier_epoch, String{R"("committed_epoch":"1")"}.size(), R"("committed_epoch":"2")"); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), malformed_ckpt).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(catalog_op.create(layout.refCkptKey(life), malformed_ckpt, Retry::once()))); expectCode([&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, DB::ErrorCodes::CORRUPTED_DATA); } @@ -703,11 +702,11 @@ TEST(CASRecoveryGrounding, LaterEpochCheckpointBaseRequiresItsContextualBacklink writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base_id)); const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(catalog_op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = base_id, .checkpoint_snapshot_id = base_id, - .last_epoch_seal = seal_id})).outcome, PutOutcome::Done); + .last_epoch_seal = seal_id}), Retry::once()))); expectCode( [&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, @@ -726,11 +725,11 @@ TEST(CASRecoveryGrounding, LaterEpochCheckpointBaseRequiresItsContextualBacklink writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), base_id)); const NamespaceLifeId life = *CasRefCatalog::lifeIfCataloged(catalog_op, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(std::holds_alternative(catalog_op.create(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = base_id, .checkpoint_snapshot_id = base_id, - .last_epoch_seal = seal_id})).outcome, PutOutcome::Done); + .last_epoch_seal = seal_id}), Retry::once()))); expectCode( [&] { (void)recoverFromCurrentCatalogCut(*backend, layout, ns); }, diff --git a/src/Disks/tests/gtest_cas_recovery_streaming.cpp b/src/Disks/tests/gtest_cas_recovery_streaming.cpp index 085baaf7e231..5b087b738532 100644 --- a/src/Disks/tests/gtest_cas_recovery_streaming.cpp +++ b/src/Disks/tests/gtest_cas_recovery_streaming.cpp @@ -310,9 +310,10 @@ TEST(CASRecoveryStreaming, MaterializingControlExceedsMemoryBound) /// tail has been applied -- exactly the memory profile streaming recovery replaced. std::vector resident_txns; int64_t held = 0; + OperationForTest tail_op(*backend); for (size_t t = 1; t <= kTxns; ++t) { - const auto got = backend->get(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, t})); + const auto got = (*tail_op).read(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, t}), Retry::once()); ASSERT_TRUE(got.has_value()); RefLogTxn txn = decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), RefTxnId{1, t}); const int64_t footprint = static_cast(decodedRefLogTxnFootprint(txn)); @@ -578,7 +579,8 @@ TEST(CASRecoveryStreaming, RecoveryResultInventoryComplete) base_txn.ops = publishCommittedOps("c_two", mref(12)); fixture::writeRefLogRaw(*backend, layout, base_txn); writeRefSnapshotRaw(*backend, layout, base); - const auto base_got = backend->get(layout.refSnapshotKey(fixture::fixtureLife(ns), base.snapshot_id)); + OperationForTest inv_op(*backend); + const auto base_got = (*inv_op).read(layout.refSnapshotKey(fixture::fixtureLife(ns), base.snapshot_id), Retry::once()); ASSERT_TRUE(base_got.has_value()); const uint64_t base_stored_bytes = base_got->bytes.size(); @@ -599,8 +601,8 @@ TEST(CASRecoveryStreaming, RecoveryResultInventoryComplete) .checkpoint_snapshot_id = RefTxnId{1, 5}, .last_epoch_seal = std::nullopt}); - const uint64_t tail6 = backend->get(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, 6}))->bytes.size(); - const uint64_t tail7 = backend->get(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, 7}))->bytes.size(); + const uint64_t tail6 = (*inv_op).read(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, 6}), Retry::once())->bytes.size(); + const uint64_t tail7 = (*inv_op).read(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, 7}), Retry::once())->bytes.size(); backend->resetCounts(); auto store = openPoolForTest(backend); diff --git a/src/Disks/tests/gtest_cas_ref_carve.cpp b/src/Disks/tests/gtest_cas_ref_carve.cpp index e12706d64f69..a06fe1335634 100644 --- a/src/Disks/tests/gtest_cas_ref_carve.cpp +++ b/src/Disks/tests/gtest_cas_ref_carve.cpp @@ -66,7 +66,7 @@ PoolPtr openPool(const BackendPtr & backend) /// injection/verification that separately computes a key via `DB::Cas::tests::fixture::fixtureLife(ns)`. void publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const String & ref) { - DB::Cas::tests::casAdmitRecoverableEntry(s->backend(), s->layout(), ns, s->liveWriterEpoch()); + DB::Cas::tests::casAdmitRecoverableEntry(*s->poolBackendPtr(), s->layout(), ns, s->liveWriterEpoch()); PartWriteInfo info; info.intended_namespace = ns; info.intended_ref = ns.string() + "/" + ref; @@ -96,11 +96,12 @@ struct CaseSync /// cache). Used to inspect exactly what a flush durably committed. std::optional newestLogTxn(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const RootNamespace & ns) { + DB::Cas::tests::OperationForTest operation(backend); std::optional newest; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*operation).list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -115,7 +116,7 @@ std::optional newestLogTxn(DB::Cas::Backend & backend, const DB::Cas: } if (!newest) return std::nullopt; - const auto got = backend.get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), *newest)); + const auto got = (*operation).read(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), *newest), Retry::standard()); if (!got) return std::nullopt; return decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), *newest); @@ -126,18 +127,19 @@ std::optional newestLogTxn(DB::Cas::Backend & backend, const DB::Cas: size_t committedRemovalCountForRef(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const RootNamespace & ns, const String & ref_name) { + DB::Cas::tests::OperationForTest operation(backend); size_t count = 0; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*operation).list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (!parsed || parsed->life_id != DB::Cas::tests::fixture::fixtureLife(ns).incarnation || parsed->kind != RefObjectKind::Log) continue; - const auto got = backend.get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), parsed->txn_id)); + const auto got = (*operation).read(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), parsed->txn_id), Retry::standard()); if (!got) continue; const RefLogTxn txn = decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), parsed->txn_id); diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index de93bd54c817..a7aaaa0fbbdb 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -42,7 +42,7 @@ class GcRoundPlanSignatureAccess public: using FoldSignature = decltype(&Gc::fold); using ExpectedFoldSignature = Gc::FoldResult (Gc::*)( - GcState &, std::optional &, RoundReport &, uint64_t, const RefPlan &, UniversePolicy, + GcState &, std::optional &, RoundReport &, uint64_t, const RefPlan &, UniversePolicy, GcRoundWorkBudget &); using BuilderSignature = decltype(&buildRefWalkPlan); using ExpectedBuilderSignature = RefPlan (*)(RoundInput &&); @@ -360,7 +360,7 @@ TEST(CASRefCatalogLifeIndex, AmbiguityStopsCatalogMutationButNotUnrelatedPointLo EXPECT_THROW(CasRefCatalog::casUpdate(op, layout, [](const RefCatalog & current) { return current; }), DB::Exception); const auto after = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(after); - EXPECT_EQ(after->incarnation, before->incarnation); + EXPECT_EQ(after->etag, before->etag); EXPECT_EQ(after->bytes, before->bytes); const auto unique = CasRefCatalog::lifeIfCataloged(op, layout, RootNamespace{"c"}); @@ -1020,13 +1020,13 @@ TEST(CASRefCatalog, CasUpdateThrowsOnVanishMidRetryInsteadOfReplacingTheCatalog) CasRefCatalog::initializeEmptyForNewPool(op, layout); CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); const CasRefCatalog::Snapshot seeded = CasRefCatalog::read(op, layout); - ASSERT_TRUE(seeded.incarnation.has_value()); + ASSERT_TRUE(seeded.etag.has_value()); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & cur) { - EXPECT_EQ(op.remove(layout.refCatalogKey(), *seeded.incarnation, Retry::standard()), Removal::Removed); + EXPECT_EQ(op.remove(layout.refCatalogKey(), *seeded.etag, Retry::standard()), Removal::Removed); RefCatalog next = cur; next.entries[0].state = NsState::Removing; next.entries[0].removal_started_round = 1; @@ -1050,13 +1050,13 @@ TEST(CASRefCatalogDeathTest, CasUpdateThrowsOnVanishMidRetryInsteadOfReplacingTh CasRefCatalog::initializeEmptyForNewPool(op, layout); CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("a", 1)); const CasRefCatalog::Snapshot seeded = CasRefCatalog::read(op, layout); - ASSERT_TRUE(seeded.incarnation.has_value()); + ASSERT_TRUE(seeded.etag.has_value()); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { CasRefCatalog::casUpdate(op, layout, [&](const RefCatalog & cur) { - EXPECT_EQ(op.remove(layout.refCatalogKey(), *seeded.incarnation, Retry::standard()), Removal::Removed); + EXPECT_EQ(op.remove(layout.refCatalogKey(), *seeded.etag, Retry::standard()), Removal::Removed); RefCatalog next = cur; next.entries[0].state = NsState::Removing; next.entries[0].removal_started_round = 1; @@ -1281,7 +1281,7 @@ TEST(CASRefCatalogRemoval, ExactDeletionRefusesChangedEntryAndAdmissionCannotCar seedObject(op, "unrelated", "sentinel"); ASSERT_TRUE(std::holds_alternative(op.replace( layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {current}}), - *CasRefCatalog::read(op, layout).incarnation, Retry::standard()))); + *CasRefCatalog::read(op, layout).etag, Retry::standard()))); CasFoldSeal ready_parent; ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ @@ -1709,7 +1709,7 @@ TEST(CASGCRefWalkPlan, CatalogIsSoleRowAdmissionAuthorityAcrossOrdinaryAndRebuil .removal_started_round = 8}, }; const CasRefCatalog::Snapshot cut{ - .catalog = catalog, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .etag = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary ordinary_scan; ordinary_scan.parent_ref_lives.emplace(UInt128{1}, RefLifeFoldState{ @@ -1878,7 +1878,7 @@ TEST(CASGCStuckRemoval, AdoptedRoundWarnsEveryRestartWithoutAppending) ASSERT_TRUE(catalog); ASSERT_TRUE(std::holds_alternative(op.replace( layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}}), - catalog->incarnation, Retry::standard()))); + catalog->etag, Retry::standard()))); CasFoldSeal seal; seal.generation = 1; @@ -1939,7 +1939,7 @@ TEST(CASGCRefWalkPlan, UnmatchedAdoptedParentLifeIsObservedWithoutEnteringThePla hexToU128("fedcba98765432100123456789abcdef"); RefCatalog catalog{.entries = {liveEntry("live", 2)}}; const CasRefCatalog::Snapshot cut{ - .catalog = catalog, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .etag = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary scan; scan.parent_ref_lives.emplace(current_life, RefLifeFoldState{ .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{2, 3}}}); @@ -1975,7 +1975,7 @@ TEST(CASGCRefPlan, RoundInputOwnsObservationsAndSuccessorStateCannotChangePlan) RefCatalog catalog; catalog.entries = {liveEntry("live", 2)}; CasRefCatalog::Snapshot cut{ - .catalog = catalog, .incarnation = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; + .catalog = catalog, .etag = std::nullopt, .life_index = CatalogLifeIndex(catalog)}; RefScanSummary observations; observations.max_log_by_life.emplace(UInt128{2}, RefTxnId{2, 7}); diff --git a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp index f25a303371d8..f2628ab62b7b 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp @@ -201,7 +201,7 @@ TEST(CASRefCatalogBirthWiring, CatalogLossAfterMountCannotRecreateAOneRowAuthori publishBirth(store, RootNamespace{"srv1/existing"}, "old"); const auto catalog = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog); - ASSERT_EQ(op.remove(layout.refCatalogKey(), catalog->incarnation, Retry::standard()), Removal::Removed); + ASSERT_EQ(op.remove(layout.refCatalogKey(), catalog->etag, Retry::standard()), Removal::Removed); backend->resetCounts(); backend->resetWriteCounts(); @@ -294,7 +294,7 @@ TEST(CASRefCatalogBirthWiring, ExistingPoolMetaWithMissingCatalogStillFailsClose PoolPtr first = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const auto catalog = op.read(first->layout().refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog); - ASSERT_EQ(op.remove(first->layout().refCatalogKey(), catalog->incarnation, Retry::standard()), Removal::Removed); + ASSERT_EQ(op.remove(first->layout().refCatalogKey(), catalog->etag, Retry::standard()), Removal::Removed); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); }); @@ -314,7 +314,7 @@ TEST(CASRefCatalogBirthWiring, RestartFixturePreservesItsExistingNonemptyCatalog .ns = RootNamespace{"test/preserved"}, .state = NsState::Live, .incarnation = UInt128{1}, .creator = std::nullopt}}}; const String bytes = encodeRefCatalog(nonempty); ASSERT_TRUE(std::holds_alternative( - op.replace(layout.refCatalogKey(), bytes, empty->incarnation, Retry::standard()))); + op.replace(layout.refCatalogKey(), bytes, empty->etag, Retry::standard()))); const auto before = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(before); @@ -322,7 +322,7 @@ TEST(CASRefCatalogBirthWiring, RestartFixturePreservesItsExistingNonemptyCatalog const auto after = op.read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(after); EXPECT_EQ(after->bytes, before->bytes); - EXPECT_EQ(after->incarnation, before->incarnation); + EXPECT_EQ(after->etag, before->etag); } /// A namespace whose catalog entry is ALREADY `Live` (e.g. admitted by an earlier mount that this diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index bcb6861cba31..7775169a36af 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -99,7 +99,7 @@ PoolPtr openPool(const BackendPtr & backend) /// production birth mints a random incarnation and those computed keys land nowhere real. void publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const String & ref) { - DB::Cas::tests::casAdmitRecoverableEntry(s->backend(), s->layout(), ns, s->liveWriterEpoch()); + DB::Cas::tests::casAdmitRecoverableEntry(*s->poolBackendPtr(), s->layout(), ns, s->liveWriterEpoch()); PartWriteInfo info; info.intended_namespace = ns; info.intended_ref = ns.string() + "/" + ref; @@ -534,11 +534,12 @@ std::vector addRemovePrecommitPairs(const String & ref, size_t num_pairs, /// breaks the inventory. Reads the backend directly (no Pool cache). std::vector listLogTxns(DB::Cas::Backend & backend, const DB::Cas::Layout & layout, const RootNamespace & ns) { + DB::Cas::tests::OperationForTest operation(backend); std::vector ids; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*operation).list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -554,7 +555,7 @@ std::vector listLogTxns(DB::Cas::Backend & backend, const DB::Cas::La std::vector txns; for (const RefTxnId & id : ids) { - const auto got = backend.get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id)); + const auto got = (*operation).read(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id), Retry::standard()); if (!got) continue; try @@ -820,7 +821,6 @@ ChunkFailureOutcome runChunkFailureCase(const String & ns_suffix, ChunkFaultBack /// stays armed for the whole call while the injected clock carries the call to its own deadline. CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; auto store = openPoolWith(backend, cfg); diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index 60a219eb5e00..f9c5e0e29bb1 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -686,7 +686,7 @@ TEST(CASRefCheckpoint, AnIdenticalMergedBodyIssuesNoWrite) EXPECT_EQ(backend->writes(key), writes_after_create) << "a skip must issue no write at all"; const auto meta_after_skips = op.head(key, Retry::standard()); ASSERT_TRUE(meta_after_skips.has_value()); - EXPECT_EQ(meta_after_skips->incarnation, meta_after_create->incarnation) + EXPECT_EQ(meta_after_skips->etag, meta_after_create->etag) << "and must not mint a new incarnation"; } @@ -723,7 +723,7 @@ TEST(CASRefCheckpoint, AnAdmissionLossBetweenTheReadAndTheWriteWritesNothing) EXPECT_EQ(backend->writes(key), writes_before) << "the check precedes the write, so nothing is sent"; const auto meta_after = reader.head(key, Retry::standard()); ASSERT_TRUE(meta_after.has_value()); - EXPECT_EQ(meta_after->incarnation, meta_before->incarnation); + EXPECT_EQ(meta_after->etag, meta_before->etag); EXPECT_EQ(readCkptOrFail(reader, layout, life), base); } @@ -1036,8 +1036,8 @@ TEST(CASRefCheckpoint, AMissingSampledBaseRestartsOnAnAdvancedIncarnationAndIsCo overwriteObject(op, key, "second"); const auto second = op.read(key, Retry::standard()); ASSERT_TRUE(second.has_value()); - const Incarnation sampled = first->incarnation; - const Incarnation advanced = second->incarnation; + const Etag sampled = first->etag; + const Etag advanced = second->etag; ASSERT_FALSE(sampled == advanced) << "the rewrite must mint a different incarnation"; EXPECT_EQ(classifyMissingSampledBase(sampled, advanced), MissingBaseVerdict::RestartRecovery) diff --git a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp index b9cc21481d3c..31648e992ceb 100644 --- a/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp +++ b/src/Disks/tests/gtest_cas_ref_contiguous_alloc.cpp @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include #include @@ -83,7 +83,6 @@ PoolPtr openPoolFenceControlled(const std::shared_ptr & backend cfg.mount_renew_period = std::chrono::milliseconds{3600000}; CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); @@ -117,9 +116,10 @@ size_t eraseKeysContaining(Backend & backend, const String & substr) size_t removed = 0; String cursor; std::vector keys; + OperationForTest op(backend); while (true) { - const ListPage page = backend.list("", cursor, 1000); + const ListPage page = (*op).list("", cursor, 1000, Retry::standard()); for (const ListedKey & listed : page.keys) if (substr.empty() || listed.key.find(substr) != String::npos) keys.push_back(listed.key); @@ -129,8 +129,8 @@ size_t eraseKeysContaining(Backend & backend, const String & substr) } for (const String & key : keys) { - const HeadResult h = backend.head(key); - if (h.exists && backend.deleteExact(key, h.token).kind == DeleteOutcome::Kind::Deleted) + const auto h = (*op).head(key, Retry::standard()); + if (h && (*op).remove(key, h->etag, Retry::standard()) == Removal::Removed) ++removed; } return removed; @@ -364,7 +364,7 @@ TEST(CASRefContiguousAlloc, NeedsRecoveryReplaysBeforeAllocatingTheNextId) CasOperation catalog_op = catalog_requests.admit(); const NamespaceLifeId life = CasRefCatalog::lifeIfCataloged(catalog_op, store->layout(), ns).value(); for (uint64_t seq = 1; seq <= 3; ++seq) - EXPECT_TRUE(backend->head(store->layout().refLogKey(life, RefTxnId{epoch, seq})).exists) + EXPECT_TRUE(catalog_op.head(store->layout().refLogKey(life, RefTxnId{epoch, seq}), Retry::once()).has_value()) << "log object " << epoch << "-" << seq << " must exist: the durable stream has no hole"; } @@ -448,7 +448,7 @@ TEST(CASRefContiguousAlloc, RecreationProceedsOnceTheHolderIsTerminal) { auto holder = openPool(backend); publishRef(holder, ns, "ref_1", 1); - } /// destroyed: the keeper stamps the farewell, making the slot terminal + } /// destroyed: the renewer stamps the farewell, making the slot terminal ASSERT_EQ(eraseKeysContaining(*backend, "_pool_meta"), 1u); /// The prefix still holds this pool's data, so the bootstrap still refuses -- but on the ORDINARY @@ -476,14 +476,14 @@ TEST(CASRefContiguousAlloc, RecreationProceedsOnceTheHolderIsTerminal) /// survivor's renewal conclusive. Clearing the prefix also resets the durable writer-epoch counter, so /// a recreation by the SAME server uuid can be handed the very same `(uuid, epoch)` the survivor still /// holds -- and the two are then indistinguishable to the lease protocol, which reads the survivor's -/// renewal as its own keeper adopting a refreshed body. That is precisely why the refusal above is the +/// renewal as its own renewer adopting a refreshed body. That is precisely why the refusal above is the /// primary defence and this fence is only the backstop: quiescing the holder BEFORE the prefix is /// cleared is what keeps the ambiguous case from arising at all. TEST(CASRefContiguousAlloc, SurvivingWriterIsFencedByTheRecreatedPoolsMount) { auto backend = std::make_shared(); /// The survivor uses the runtime-owned renewal worker, as a real mount does: the runtime terminal - /// consumer is what latches the write fence when a renewal fails, so a keeper-only call would + /// consumer is what latches the write fence when a renewal fails, so a renewer-only call would /// reproduce the failure but not the lifecycle effect it causes. PoolConfig survivor_cfg{.pool_prefix = "p", .server_root_id = "test"}; survivor_cfg.background_watermark = true; @@ -522,11 +522,12 @@ TEST(CASRefContiguousAlloc, SurvivingWriterIsFencedByTheRecreatedPoolsMount) EXPECT_EQ(publishRef(recreated, ns, "ref_1", 1), (RefTxnId{recreated->writerEpoch(), 1})); /// The survivor's TEARDOWN is the other half, and it is asserted here rather than left to the - /// destructor at scope exit. A terminal keeper must skip release without backend I/O: the renewal + /// destructor at scope exit. A terminal renewer must skip release without backend I/O: the renewal /// conflict already counted the conclusive foreign successor, and teardown must neither double-count /// it nor stamp a farewell over the successor's slot. const String survivor_mount_key = recreated->layout().mountKey("test"); - const auto successor_slot_before = backend->get(survivor_mount_key); + OperationForTest teardown_op(*backend); + const auto successor_slot_before = (*teardown_op).read(survivor_mount_key, Retry::once()); ASSERT_TRUE(successor_slot_before.has_value()); const uint64_t skipped_after_deposition = ProfileEvents::global_counters[ProfileEvents::CASMountReleaseSkippedForeignOccupant].load(); @@ -539,7 +540,7 @@ TEST(CASRefContiguousAlloc, SurvivingWriterIsFencedByTheRecreatedPoolsMount) EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASMountExclusivityViolation].load(), violations_before) << "and must NOT report an exclusivity violation: this is a failover, not a broken guarantee"; - const auto successor_slot_after = backend->get(survivor_mount_key); + const auto successor_slot_after = (*teardown_op).read(survivor_mount_key, Retry::once()); ASSERT_TRUE(successor_slot_after.has_value()); EXPECT_EQ(successor_slot_after->bytes, successor_slot_before->bytes) << "the deposed writer must not stamp its farewell over the successor's lease"; diff --git a/src/Disks/tests/gtest_cas_ref_gc.cpp b/src/Disks/tests/gtest_cas_ref_gc.cpp index 0d34129d2d81..5c20e6298403 100644 --- a/src/Disks/tests/gtest_cas_ref_gc.cpp +++ b/src/Disks/tests/gtest_cas_ref_gc.cpp @@ -75,7 +75,7 @@ size_t runToFixpoint(const PoolPtr & s, Gc & gc, size_t max_rounds = 64) s->renewWatermarkOnce(); const bool no_work = rep.candidates == 0 && rep.deleted == 0 && rep.absent == 0 && rep.replaced == 0 && rep.spared == 0; - if (no_work && !anyCondemnedInSeal(s->backend(), s->layout())) + if (no_work && !anyCondemnedInSeal(*s->poolBackendPtr(), s->layout())) break; } return rounds; @@ -83,7 +83,8 @@ size_t runToFixpoint(const PoolPtr & s, Gc & gc, size_t max_rounds = 64) bool blobPresent(Backend & b, const Layout & layout, const UInt128 & hash) { - return b.head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)})).exists; + OperationForTest op(b); + return (*op).head(layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hash)}), Retry::once()).has_value(); } /// Denies ONCE the single round-commit `gc/state` CAS that advances `snap_generation` (the losing @@ -100,7 +101,7 @@ class DeposeRoundCommitBackend : public InMemoryBackend { if (arm && key == "p/gc/state") { - const auto stored = get(key); + const auto stored = read(key, access); const uint64_t stored_gen = stored ? decodeGcState(stored->bytes).snap_generation : 0; if (decodeGcState(bytes).snap_generation > stored_gen) { @@ -152,7 +153,7 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend { auto result = CountingBackend::head(key, access); if (armed && timing == Timing::BeforeFirstDelete && key == first_cleanup_key) - moveAuthority(); + moveAuthority(access); return result; } @@ -161,16 +162,20 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend { auto result = CountingBackend::remove(key, expected_value, access); if (armed && timing == Timing::AfterFirstDelete && key == first_cleanup_key) - moveAuthority(); + moveAuthority(access); return result; } private: - void moveAuthority() + /// `access` is the token the caller's own primitive override already holds for its in-flight + /// request; reused here for this method's extra read+write rather than minting a new CasRequests, + /// exactly as `Backend::probeSentinelRaw`'s default implementation reuses one `access` across its + /// own head-then-more sequence. + void moveAuthority(TransportAccess & access) { armed = false; const String & key = authority == Authority::Catalog ? catalog_key : gc_state_key; - const auto got = CountingBackend::get(key); + const auto got = read(key, access); if (!got) throw std::runtime_error("test-injected cleanup authority object is absent"); @@ -181,7 +186,7 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend ++moved.lease.seq; bytes = encodeGcState(moved); } - if (CountingBackend::casPut(key, bytes, got->token).outcome != CasOutcome::Committed) + if (!write(key, bytes, got->value, access).has_value()) throw std::runtime_error("test-injected cleanup authority move lost its CAS"); } @@ -368,7 +373,8 @@ TEST(CASRefGc, LosingGenerationCommitAdoptsNothingDeletesNothing) Gc gc(store, kGc); gc.runRegularRound(); /// round 1: folds the +1 and adopts it cleanly store->renewWatermarkOnce(); - const auto adopted = decodeGcState(backend->get(layout.gcStateKey())->bytes); + OperationForTest raw_op(*backend); + const auto adopted = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); ASSERT_GT(adopted.snap_generation, 0u); /// Drop the ref, then run the round whose commit is DENIED (losing leader). @@ -378,7 +384,7 @@ TEST(CASRefGc, LosingGenerationCommitAdoptsNothingDeletesNothing) backend->arm = false; /// The deposed round adopted NOTHING: the durable pointers are unchanged... - const auto after = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const auto after = decodeGcState((*raw_op).read(layout.gcStateKey(), Retry::once())->bytes); EXPECT_EQ(after.snap_generation, adopted.snap_generation) << "a denied round-commit CAS must not advance the adopted generation"; EXPECT_EQ(after.snap_attempt, adopted.snap_attempt); @@ -424,22 +430,23 @@ TEST(CASRefGc, RefObjectCleanupRetainsCheckpointNamedTriple) const String log_v2_key = layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, v2}); const String old_snap_key = layout.refSnapshotKey(fixture::fixtureLife(ns), RefTxnId{1, v1}); const String new_snap_key = layout.refSnapshotKey(fixture::fixtureLife(ns), RefTxnId{1, v2}); - ASSERT_TRUE(backend->head(log_v1_key).exists); - ASSERT_TRUE(backend->head(log_v2_key).exists); - ASSERT_TRUE(backend->head(old_snap_key).exists); + OperationForTest raw_op(*backend); + ASSERT_TRUE((*raw_op).head(log_v1_key, Retry::once()).has_value()); + ASSERT_TRUE((*raw_op).head(log_v2_key, Retry::once()).has_value()); + ASSERT_TRUE((*raw_op).head(old_snap_key, Retry::once()).has_value()); Gc gc(store, kGc); runToFixpoint(store, gc); /// folds v1,v2 (cursor -> v2) then cleans covered ref objects post-CAS /// The old log lies below both the durable cursor and the validated checkpoint base => DELETED. - EXPECT_FALSE(backend->head(log_v1_key).exists) + EXPECT_FALSE((*raw_op).head(log_v1_key, Retry::once()).has_value()) << "a log below the checkpoint-named snapshot base and durable cursor must be deleted"; /// The same-id ordinary log is part of recovery's triple and must survive. - EXPECT_TRUE(backend->head(log_v2_key).exists) + EXPECT_TRUE((*raw_op).head(log_v2_key, Retry::once()).has_value()) << "the checkpoint-named non-seal log must survive with its snapshot"; /// The older snapshot is deleted; the checkpoint-named snapshot is retained. - EXPECT_FALSE(backend->head(old_snap_key).exists) << "an older snapshot must be deleted"; - EXPECT_TRUE(backend->head(new_snap_key).exists) << "the checkpoint-named snapshot must be retained"; + EXPECT_FALSE((*raw_op).head(old_snap_key, Retry::once()).has_value()) << "an older snapshot must be deleted"; + EXPECT_TRUE((*raw_op).head(new_snap_key, Retry::once()).has_value()) << "the checkpoint-named snapshot must be retained"; } /// `cleanupRefObjects`'s per-round cap. Five deletable logs share one @@ -487,11 +494,12 @@ TEST(CASRefGc, RefObjectCleanupRespectsRoundBudgetAndConvergesAcrossRounds) deletable_log_keys.push_back(layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, static_cast(i)})); Gc gc(store, kGc); + OperationForTest raw_op(*backend); auto countSurviving = [&] { size_t n = 0; for (const String & k : deletable_log_keys) - if (backend->head(k).exists) + if ((*raw_op).head(k, Retry::once()).has_value()) ++n; return n; }; @@ -559,10 +567,10 @@ TEST(CASRefGc, RefObjectCleanupRetainsCheckpointPredecessorSealProof) Gc gc(store, kGc); runToFixpoint(store, gc); - EXPECT_TRUE(backend->head(layout.refLogKey(life, seal_id)).exists) - << "cleanup must retain the predecessor seal that proves the checkpoint base's epoch transition"; CasRequests requests(backend, Fence::open()); CasOperation op = requests.admit(); + EXPECT_TRUE(op.head(layout.refLogKey(life, seal_id), Retry::once()).has_value()) + << "cleanup must retain the predecessor seal that proves the checkpoint base's epoch transition"; const CasRefCatalog::Snapshot cut = CasRefCatalog::read(op, layout); const auto entry = std::find_if(cut.catalog.entries.begin(), cut.catalog.entries.end(), [&](const CatalogEntry & candidate) { return candidate.ns == ns; }); @@ -585,8 +593,9 @@ TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBeforeFirstDeleteRefusesEveryRefO Gc gc(store, kGc); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - EXPECT_TRUE(backend->head(keys.first_log_key).exists); - EXPECT_TRUE(backend->head(keys.second_log_key).exists); + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(keys.first_log_key), 0u); EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } @@ -604,8 +613,9 @@ TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBetweenKeysAllowsFirstAndRefusesS Gc gc(store, kGc); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - EXPECT_FALSE(backend->head(keys.first_log_key).exists); - EXPECT_TRUE(backend->head(keys.second_log_key).exists); + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(keys.first_log_key), 1u); EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } @@ -623,8 +633,9 @@ TEST(CASRefGcCleanupAuthority, GcFenceMoveBeforeFirstDeleteRefusesEveryRefObject Gc gc(store, kGc); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - EXPECT_TRUE(backend->head(keys.first_log_key).exists); - EXPECT_TRUE(backend->head(keys.second_log_key).exists); + OperationForTest raw_op(*backend); + EXPECT_TRUE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(keys.first_log_key), 0u); EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } @@ -642,8 +653,9 @@ TEST(CASRefGcCleanupAuthority, GcFenceMoveBetweenKeysAllowsFirstAndRefusesSecond Gc gc(store, kGc); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); - EXPECT_FALSE(backend->head(keys.first_log_key).exists); - EXPECT_TRUE(backend->head(keys.second_log_key).exists); + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(keys.first_log_key), 1u); EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } @@ -736,16 +748,16 @@ TEST(CASRefGc, RefSnaplogLifecycleE2E) after_snapshot_publish.checkpoint_snapshot_id = RefTxnId{1, va2}; ASSERT_TRUE(std::holds_alternative(op.replace( layout.refCkptKey(life_a), encodeRefCkpt(after_snapshot_publish), - before_snapshot_publish.incarnation, Retry::standard()))); + before_snapshot_publish.etag, Retry::standard()))); Gc gc(store, kGc); runToFixpoint(store, gc); /// Snapshot lifecycle: the covering snapshot is retained; the covered logs (folded + snapshot-covered) /// are cleaned; the replaced manifest's blob is reclaimed while the live blobs survive. - EXPECT_TRUE(backend->head(layout.refSnapshotKey(fixture::fixtureLife(ns_a), RefTxnId{1, va2})).exists) + EXPECT_TRUE(op.head(layout.refSnapshotKey(fixture::fixtureLife(ns_a), RefTxnId{1, va2}), Retry::once()).has_value()) << "covering snapshot retained"; - EXPECT_FALSE(backend->head(layout.refLogKey(fixture::fixtureLife(ns_a), RefTxnId{1, va1})).exists) << "covered log cleaned"; + EXPECT_FALSE(op.head(layout.refLogKey(fixture::fixtureLife(ns_a), RefTxnId{1, va1}), Retry::once()).has_value()) << "covered log cleaned"; EXPECT_FALSE(blobPresent(*backend, layout, DB::UInt128(1))) << "replaced blob reclaimed"; EXPECT_TRUE(blobPresent(*backend, layout, DB::UInt128(2))) << "live blob survives"; EXPECT_TRUE(blobPresent(*backend, layout, DB::UInt128(3))) << "other table's blob survives"; @@ -774,7 +786,10 @@ TEST(CASRefGc, MalformedRefKeyAbortsRefFoldingNoPartialDelta) /// Plant a malformed ref key under the ref prefix (a `_log` with a non-canonical id render). const NamespaceLifeId life = store->namespaceLife(ns); - backend->putIfAbsent(layout.namespaceStreamPrefix(life) + "_log/not-a-valid-txn-id", "garbage"); + { + OperationForTest seed_op(*backend); + (*seed_op).create(layout.namespaceStreamPrefix(life) + "_log/not-a-valid-txn-id", "garbage", Retry::once()); + } Gc gc(store, kGc); /// The fold's `groupRefKeys` rejects the unrecognized key and ABORTS ref folding for the round (spec @@ -815,7 +830,8 @@ TEST(CASRefGc, NonCanonicalLifeKeyAbortsRefFoldingWithoutWedgingTheRound) /// opaque id. Only a foreign or corrupt writer can put this key here, and the pool must survive it. const String noncanonical_life = layout.casRefsPrefix() + ns.string() + "/_log/" + renderRefTxnId(RefTxnId{1, 1}) + ".zst"; - ASSERT_EQ(backend->putIfAbsent(noncanonical_life, "garbage").outcome, PutOutcome::Done); + OperationForTest raw_op(*backend); + ASSERT_TRUE(std::holds_alternative((*raw_op).create(noncanonical_life, "garbage", Retry::once()))); Gc gc(store, kGc); RoundReport rep; @@ -834,7 +850,7 @@ TEST(CASRefGc, NonCanonicalLifeKeyAbortsRefFoldingWithoutWedgingTheRound) /// The wedge is only visible over time: the key is still there (nothing deletes it), so a second /// round meets it again. It must survive that one too. - ASSERT_TRUE(backend->head(noncanonical_life).exists) << "precondition: nothing removed the key"; + ASSERT_TRUE((*raw_op).head(noncanonical_life, Retry::once()).has_value()) << "precondition: nothing removed the key"; ASSERT_NO_THROW(gc.runRegularRound()) << "a round that dies on this key would die on it forever"; } @@ -874,7 +890,10 @@ TEST(CASRefGc, InvalidRefLogBodyHoldsNamespaceNoPartialDelta) /// A canonical `_log` key (groupRefKeys accepts it) whose body cannot be decoded: the fold GETs it /// and `decodeRefLogTxn` throws. const String garbage_key = layout.refLogKey(fixture::fixtureLife(ns), RefTxnId{1, dropped + 1}); - backend->putIfAbsent(garbage_key, "garbage-not-a-valid-reflog-body"); + { + OperationForTest seed_op(*backend); + (*seed_op).create(garbage_key, "garbage-not-a-valid-reflog-body", Retry::once()); + } /// The corruption claims the next committed position. Advance only the durable frontier, not the /// log body, so recovery must exact-GET and hold this malformed object instead of ignoring F+1. advanceRecoverableCkptForRawFixture(*backend, layout, ns, RefTxnId{1, dropped + 1}); @@ -902,9 +921,10 @@ TEST(CASRefGc, InvalidRefLogBodyHoldsNamespaceNoPartialDelta) /// precisely what made the hold necessary; if an absent could clear it, the whole mechanism would /// be defeated by the corruption it exists to survive. (Before durable holds this delete DID /// release the namespace, which is the hole Task 8 closed.) - const HeadResult h = backend->head(garbage_key); - ASSERT_TRUE(h.exists); - ASSERT_EQ(backend->deleteExact(garbage_key, h.token).kind, DeleteOutcome::Kind::Deleted); + OperationForTest evidence_op(*backend); + const auto h = (*evidence_op).head(garbage_key, Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ((*evidence_op).remove(garbage_key, h->etag, Retry::once()), Removal::Removed); for (int i = 0; i < 4; ++i) { @@ -996,8 +1016,8 @@ TEST(CASRefGc, CatalogAdmittedFreshLifeWithoutParentSeedsSuccessorSeal) Gc gc(store, kGc); ASSERT_NO_THROW(gc.runRegularRound()); - const GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState state = decodeGcState(op.read(layout.gcStateKey(), Retry::once())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend->get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + op.read(layout.foldSealKey(state.snap_generation, state.snap_attempt), Retry::once())->bytes); EXPECT_TRUE(seal.ref_lives.contains(life_id)); } diff --git a/src/Disks/tests/gtest_cas_ref_install_safety.cpp b/src/Disks/tests/gtest_cas_ref_install_safety.cpp index fc215264f398..b6c747a8ee37 100644 --- a/src/Disks/tests/gtest_cas_ref_install_safety.cpp +++ b/src/Disks/tests/gtest_cas_ref_install_safety.cpp @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include @@ -59,6 +58,12 @@ using namespace DB::Cas; namespace { +bool manifestKeyExists(const BackendPtr & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + PoolPtr openPool(const BackendPtr & backend) { /// A fresh pool with no residue, mirroring `gtest_cas_ref_chunked_flush.cpp`'s `openPool`. @@ -79,7 +84,6 @@ PoolPtr openPoolWedgeBudget(const BackendPtr & backend) PoolConfig cfg{.pool_prefix = "p", .server_root_id = "test"}; CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; return Pool::open(backend, cfg); @@ -181,7 +185,6 @@ PoolPtr openPoolFenceControlled(const std::shared_ptr cfg.mount_renew_period = std::chrono::milliseconds{3600000}; CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget @@ -900,7 +903,10 @@ TEST(CASRefInstallSafety, WedgeResolutionProvenForeignFaultsTheLane) backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::None; const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(wedged_key.empty()); - ASSERT_EQ(backend->putIfAbsent(wedged_key, "a-different-object").outcome, PutOutcome::Done); + { + DB::Cas::tests::OperationForTest foreign_op(backend); + ASSERT_TRUE(std::holds_alternative((*foreign_op).create(wedged_key, "a-different-object", Retry::once()))); + } DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { store->dropRef(ns, "y"); }); @@ -1037,7 +1043,7 @@ TEST(CASRefInstallSafety, UncertainPrecommitKeepsItsCleanupOwnerAndItsBody) auto build = store->beginPartWrite(info); const ManifestId id = build->stageManifest({}); const String manifest_key = store->layout().manifestKey(id); - ASSERT_TRUE(backend->head(manifest_key).exists) << "the staged body must exist before the precommit"; + ASSERT_TRUE(manifestKeyExists(backend, manifest_key)) << "the staged body must exist before the precommit"; /// Scoped to THIS namespace's ref log so the manifest body's own PUT cannot consume the fault. The /// object LANDS and only its acknowledgement is lost, and no settling read can prove otherwise, so @@ -1054,7 +1060,7 @@ TEST(CASRefInstallSafety, UncertainPrecommitKeepsItsCleanupOwnerAndItsBody) /// precommit durable) and appends the exact removal in the same flush. build->abandon(); - EXPECT_TRUE(backend->head(manifest_key).exists) + EXPECT_TRUE(manifestKeyExists(backend, manifest_key)) << "abandon writer-deleted the body of a precommit that may be live -- GC's fold barrier would " "clamp on it forever"; EXPECT_TRUE(store->livePrecommitsForTest(ns).empty()) diff --git a/src/Disks/tests/gtest_cas_ref_read_contract.cpp b/src/Disks/tests/gtest_cas_ref_read_contract.cpp index 5fa0b77db52c..ba8456b0c2a8 100644 --- a/src/Disks/tests/gtest_cas_ref_read_contract.cpp +++ b/src/Disks/tests/gtest_cas_ref_read_contract.cpp @@ -238,20 +238,21 @@ TEST(CASRefReadContract, StaleLifeDropRefusesAfterRebirthAndNeverTouchesSuccesso publishCommittedTransition(*backend, layout, ns, ref_name, std::nullopt, life2_ref); const ManifestId life2_manifest{ns, life2_ref}; - const HeadResult catalog_head_before = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(catalog_head_before.exists); - const auto catalog_get_before = backend->get(layout.refCatalogKey()); + OperationForTest catalog_probe(*backend); + const auto catalog_head_before = (*catalog_probe).head(layout.refCatalogKey(), Retry::standard()); + ASSERT_TRUE(catalog_head_before.has_value()); + const auto catalog_get_before = (*catalog_probe).read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog_get_before.has_value()); /// The held life-1 handle names an incarnation the catalog no longer carries: refused, not /// resolved against the current (life-2) row. expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropNamespace(life1); }); - const HeadResult catalog_head_after = backend->head(layout.refCatalogKey()); - ASSERT_TRUE(catalog_head_after.exists); - EXPECT_EQ(catalog_head_after.token, catalog_head_before.token) + const auto catalog_head_after = (*catalog_probe).head(layout.refCatalogKey(), Retry::standard()); + ASSERT_TRUE(catalog_head_after.has_value()); + EXPECT_EQ(catalog_head_after->etag, catalog_head_before->etag) << "a refused stale-life drop must not touch the catalog object at all"; - const auto catalog_get_after = backend->get(layout.refCatalogKey()); + const auto catalog_get_after = (*catalog_probe).read(layout.refCatalogKey(), Retry::standard()); ASSERT_TRUE(catalog_get_after.has_value()); EXPECT_EQ(catalog_get_after->bytes, catalog_get_before->bytes); diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index d0c98a62bbaf..1bfdf7c5ac52 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -103,7 +103,7 @@ CasRefCatalog::Snapshot readCatalogForTest(const BackendPtr & backend, const Lay /// A fixture's own conditional replace, for the races these tests stage by hand. bool replaceForTest(const BackendPtr & backend, const String & key, const String & bytes, - const Incarnation & expected) + const Etag & expected) { CasRequests requests(backend, Fence::open()); CasOperation op = requests.admit(); @@ -114,13 +114,28 @@ bool replaceForTest(const BackendPtr & backend, const String & key, const String /// generation can drive the production remount boundary without paying a live-lease expiry wait. void fenceOutMountForRemount(Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + OperationForTest op(backend); + const auto got = (*op).read(mount_key, Retry::once()); ASSERT_TRUE(got.has_value()); MountLease mount = decodeMountLease(got->bytes); mount.gc_fenced = true; mount.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, encodeMountLease(mount), got->token).outcome, - PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(mount_key, encodeMountLease(mount), got->etag, Retry::once()))); +} + +/// The durable object at `key`, or `nullopt`. +std::optional readAt(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, Retry::once()); +} + +/// Unconditional create of a fresh key (the fixture's own setup, never a real conflict). +void createAt(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + EXPECT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); } /// A backend whose `LIST` can lie by omission. `hidden_keys` remain readable by exact key, so these @@ -338,7 +353,7 @@ constexpr int kFaultsBeyondTheRetryWindow = 100'000; CasRequestBudget tinyBudget() { return CasRequestBudget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = 500, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; } PoolConfig walkTestConfig() @@ -423,7 +438,7 @@ void seedTxn(Backend & backend, const Layout & layout, const RootNamespace & ns, /// never run a birth through the append lane. void seedCkpt(Backend & backend, const Layout & layout, const RootNamespace & ns, const RefCkpt & ckpt) { - backend.putIfAbsent(layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)), encodeRefCkpt(ckpt)); + createAt(backend, layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)), encodeRefCkpt(ckpt)); } RefCkpt lifeEpochCkpt(uint64_t life_epoch, std::optional committed_through = std::nullopt) @@ -438,7 +453,7 @@ RefCkpt lifeEpochCkpt(uint64_t life_epoch, std::optional committed_thr /// disengaged optional: an aborted binary would take every later suite's result with it. std::optional readLogTxn(Backend & backend, const Layout & layout, const RootNamespace & ns, RefTxnId id) { - const auto got = backend.get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id)); + const auto got = readAt(backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id)); if (!got) return std::nullopt; return decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), id); @@ -503,9 +518,9 @@ CatalogEntry replaceCatalogLifeForTest( { return entry.ns == predecessor.ns && entry.incarnation == predecessor.incarnation; }); - if (!before_delete.incarnation + if (!before_delete.etag || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(without_predecessor), - *before_delete.incarnation)) + *before_delete.etag)) throw std::runtime_error("test failed to retire exact predecessor catalog life"); CatalogEntry successor{ @@ -516,8 +531,8 @@ CatalogEntry replaceCatalogLifeForTest( const CasRefCatalog::Snapshot after_delete = readCatalogForTest(backend, layout); RefCatalog reborn = after_delete.catalog; reborn.entries.push_back(successor); - if (!after_delete.incarnation - || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.incarnation)) + if (!after_delete.etag + || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.etag)) throw std::runtime_error("test failed to publish successor catalog life"); return successor; } @@ -618,11 +633,11 @@ TEST(CASRefRecoveryCasWalk, MissingExactIdAtOrBelowCommittedFrontierIsCorruption DB::Cas::tests::fixture::admitLive(*backend, layout, ns); const NamespaceLifeId life = catalogLife(backend, layout, ns); seedTxn(*backend, layout, ns, RefTxnId{1, 1}, "a", /*birth=*/true); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + createAt(*backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = std::optional{1}, .committed_through = frontier, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt})); const auto ckpt_before = readCkptForTest(backend, layout, life); ASSERT_TRUE(ckpt_before); @@ -631,7 +646,7 @@ TEST(CASRefRecoveryCasWalk, MissingExactIdAtOrBelowCommittedFrontierIsCorruption const auto ckpt_after = readCkptForTest(backend, layout, life); ASSERT_TRUE(ckpt_after); - EXPECT_EQ(ckpt_after->incarnation, ckpt_before->incarnation) + EXPECT_EQ(ckpt_after->etag, ckpt_before->etag) << "an unchanged checkpoint makes the missing committed id corruption, not a shorter stream"; } @@ -649,11 +664,11 @@ TEST(CASRefRecoveryCasWalk, UncommittedSnapshotIsUnobservedWithoutStreamList) writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), uncommitted_snapshot_id, {committedRow("laundered", manifestRef(1, 2, 1))})); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + createAt(*backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = std::optional{1}, .committed_through = frontier, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt})); auto store = openWalkPool(backend); backend->resetCounts(); @@ -695,20 +710,24 @@ TEST(CASRefRecoveryCasWalk, ListingShapeDoesNotAffectCheckpointRecovery) String cursor; do { - const KeyPage page = seed_op.list("", cursor, 1000, Retry::standard()); - for (const KeyEntry & listed : page.keys) + const ListPage page = seed_op.list("", cursor, 1000, Retry::standard()); + for (const ListedKey & listed : page.keys) { - const auto object = seed->get(listed.key); + const auto object = seed_op.read(listed.key, Retry::standard()); if (!object) throw std::runtime_error("seed LIST returned a key that exact GET could not read"); - const auto existing = backend->get(listed.key); + const auto existing = readAt(*backend, listed.key); if (existing) { - if (existing->bytes != object->bytes || existing->attributes != object->attributes) + if (existing->bytes != object->bytes) throw std::runtime_error("clone backend constructor disagreed with seeded object"); } - else if (backend->putIfAbsent(listed.key, object->bytes, object->attributes).outcome != PutOutcome::Done) - throw std::runtime_error("clone backend failed to copy seeded object"); + else + { + OperationForTest clone_op(*backend); + if (!std::holds_alternative((*clone_op).create(listed.key, object->bytes, Retry::once()))) + throw std::runtime_error("clone backend failed to copy seeded object"); + } } cursor = page.next_cursor; } while (!cursor.empty()); @@ -807,8 +826,8 @@ TEST(CASRefRecoveryCasWalk, DuplicateCatalogLifeIsCorruptionBeforeColdRuntimeAdm .incarnation = life.incarnation}); std::sort(ambiguous.entries.begin(), ambiguous.entries.end(), [](const CatalogEntry & lhs, const CatalogEntry & rhs) { return lhs.ns.string() < rhs.ns.string(); }); - ASSERT_TRUE(sampled.incarnation); - ASSERT_TRUE(replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(ambiguous), *sampled.incarnation)); + ASSERT_TRUE(sampled.etag); + ASSERT_TRUE(replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(ambiguous), *sampled.etag)); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { @@ -836,7 +855,7 @@ TEST(CASRefRecoveryCasWalk, CheckpointAdvanceAfterLastLogProbeRestartsBeforeInst ASSERT_TRUE(sampled); RefCkpt advanced = sampled->ckpt; advanced.committed_through = concurrent_frontier; - ASSERT_TRUE(replaceForTest(backend, layout.refCkptKey(life), encodeRefCkpt(advanced), sampled->incarnation)); + ASSERT_TRUE(replaceForTest(backend, layout.refCkptKey(life), encodeRefCkpt(advanced), sampled->etag)); }; auto store = openWalkPool(backend); @@ -1028,8 +1047,8 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEveryOccupiedObjectBeforeAdvancingP backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->listRefs(ns); }); - EXPECT_TRUE(backend->get(layout.refLogKey(life, test_case.occupant))); - EXPECT_FALSE(backend->get(layout.refLogKey(life, test_case.forbidden_successor))) + EXPECT_TRUE(readAt(*backend, layout.refLogKey(life, test_case.occupant))); + EXPECT_FALSE(readAt(*backend, layout.refLogKey(life, test_case.forbidden_successor))) << "recovery advanced before exact _ckpt certified the occupied object"; EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, initial_frontier); EXPECT_FALSE(store->refTableRecoveredForTest(ns)); @@ -1105,9 +1124,9 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEachCreatedSealBeforeCreatingTheNex backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->listRefs(ns); }); - EXPECT_TRUE(backend->get(layout.refLogKey(life, first_seal))) + EXPECT_TRUE(readAt(*backend, layout.refLogKey(life, first_seal))) << "the first recovery seal became durable before its frontier attempt"; - EXPECT_FALSE(backend->get(layout.refLogKey(life, second_seal))) + EXPECT_FALSE(readAt(*backend, layout.refLogKey(life, second_seal))) << "recovery may not create a second object while the first is still above exact _ckpt"; EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, initial_frontier); EXPECT_FALSE(store->refTableRecoveredForTest(ns)); @@ -1119,8 +1138,8 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEachCreatedSealBeforeCreatingTheNex auto cold_store = openWalkPool(backend); ASSERT_EQ(cold_store->liveWriterEpoch(), 4u); ASSERT_EQ(cold_store->listRefs(ns).size(), 1u); - EXPECT_TRUE(backend->get(layout.refLogKey(life, second_seal))); - EXPECT_TRUE(backend->get(layout.refLogKey(life, cold_remount_frontier))); + EXPECT_TRUE(readAt(*backend, layout.refLogKey(life, second_seal))); + EXPECT_TRUE(readAt(*backend, layout.refLogKey(life, cold_remount_frontier))); EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, cold_remount_frontier); } @@ -1157,16 +1176,16 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesAnAdoptedStragglerBeforeCreatingIts backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->listRefs(ns); }); - EXPECT_TRUE(backend->get(layout.refLogKey(life, straggler))) + EXPECT_TRUE(readAt(*backend, layout.refLogKey(life, straggler))) << "the straggler occupied the recovery seal slot"; - EXPECT_FALSE(backend->get(layout.refLogKey(life, following_seal))) + EXPECT_FALSE(readAt(*backend, layout.refLogKey(life, following_seal))) << "recovery may not create a seal after an adopted straggler above exact _ckpt"; EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, initial_frontier); EXPECT_FALSE(store->refTableRecoveredForTest(ns)); backend->ambiguous_cas_count = 0; ASSERT_EQ(store->listRefs(ns).size(), 2u); - EXPECT_TRUE(backend->get(layout.refLogKey(life, following_seal))); + EXPECT_TRUE(readAt(*backend, layout.refLogKey(life, following_seal))); EXPECT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, following_seal); } @@ -1248,7 +1267,7 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo const CatalogEntry predecessor = readCatalogForTest(backend, layout).catalog.entries.front(); const NamespaceLifeId predecessor_life = NamespaceLifeId::fromCatalogEntry(predecessor.ns, predecessor.incarnation); - const auto predecessor_ckpt_before = backend->get(layout.refCkptKey(predecessor_life)); + const auto predecessor_ckpt_before = readAt(*backend, layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_before); auto store = openWalkPool(backend); @@ -1288,9 +1307,8 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo const CatalogEntry successor = replaceCatalogLifeForTest(backend, layout, predecessor, UInt128{0x5152}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(successor_life), encodeRefCkpt(lifeEpochCkpt(2))).outcome, - PutOutcome::Done); - const auto successor_ckpt_before = backend->get(layout.refCkptKey(successor_life)); + createAt(*backend, layout.refCkptKey(successor_life), encodeRefCkpt(lifeEpochCkpt(2))); + const auto successor_ckpt_before = readAt(*backend, layout.refCkptKey(successor_life)); ASSERT_TRUE(successor_ckpt_before); store->invalidateRemovedCatalogLife(predecessor_life); backend->resetCounts(); @@ -1307,9 +1325,9 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo << "no predecessor seal retry may be sent after exact retirement"; EXPECT_EQ(backend->writeCount(layout.refCkptKey(predecessor_life)), 0u) << "no predecessor checkpoint CAS may be sent after exact retirement"; - const auto predecessor_ckpt_after = backend->get(layout.refCkptKey(predecessor_life)); + const auto predecessor_ckpt_after = readAt(*backend, layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_after); - EXPECT_EQ(predecessor_ckpt_after->token, predecessor_ckpt_before->token); + EXPECT_EQ(predecessor_ckpt_after->etag, predecessor_ckpt_before->etag); EXPECT_FALSE(store->refTableRecoveredForTest(ns)) << "the detached predecessor result was installed"; EXPECT_EQ(store->recoveryInstallCountForTest(), recovery_installs_before) << "the detached predecessor reached the recovery publication point"; @@ -1317,9 +1335,9 @@ TEST(CASRefRecoveryCasWalk, RetiredLifePausedInRealRecoveryIoWritesAndInstallsNo for (const String & key : backend->touchedKeys()) EXPECT_EQ(key.find(successor_prefix), String::npos) << "predecessor recovery retargeted storage I/O into successor key " << key; - const auto successor_ckpt_after = backend->get(layout.refCkptKey(successor_life)); + const auto successor_ckpt_after = readAt(*backend, layout.refCkptKey(successor_life)); ASSERT_TRUE(successor_ckpt_after); - EXPECT_EQ(successor_ckpt_after->token, successor_ckpt_before->token); + EXPECT_EQ(successor_ckpt_after->etag, successor_ckpt_before->etag); EXPECT_EQ(successor_ckpt_after->bytes, successor_ckpt_before->bytes); } @@ -1352,7 +1370,7 @@ TEST(CASRefRecoveryCasWalk, FenceBumpedAfterSlotOccupyBeforeCkptCasAdvancesNoChe ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->ckpt.last_epoch_seal, std::nullopt) << "the seal is durable but the checkpoint must not record it under a generation that moved"; - EXPECT_EQ(ckpt_after->incarnation, ckpt_before->incarnation) << "no CAS was sent at all"; + EXPECT_EQ(ckpt_after->etag, ckpt_before->etag) << "no CAS was sent at all"; } /// Bump point 2: AFTER the `_ckpt` CAS, BEFORE the install. The checkpoint advance is harmless (the @@ -1610,7 +1628,7 @@ TEST(CASRefRecoveryCasWalk, NeedsRecoveryReplaysTheStrandedTxn) entry = &e; ASSERT_NE(entry, nullptr) << "the birth above must have minted a catalog entry for " << ns.string(); const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation); - ASSERT_TRUE(backend->get(layout.refLogKey(life, RefTxnId{1, 2})).has_value()) + ASSERT_TRUE(readAt(*backend, layout.refLogKey(life, RefTxnId{1, 2})).has_value()) << "the stranded transaction must be durable -- otherwise recovery is not owed"; } @@ -1656,7 +1674,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsOneExactUnfrontieredSuccessorAnd EXPECT_GT(clock->pauseCount(), 1u) << "the reissues must pace through the injected sleep, never a real one"; ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); - ASSERT_TRUE(backend->get(layout.refLogKey(life, RefTxnId{1, 2}))) + ASSERT_TRUE(readAt(*backend, layout.refLogKey(life, RefTxnId{1, 2}))) << "the sole deterministic successor must be durable before recovery"; ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, (RefTxnId{1, 1})); @@ -1710,8 +1728,7 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsFirstCommittedTxnAboveLifeEpochO /// readable checkpoint whose `committed_through` is absent. DB::Cas::tests::fixture::admitLive(*backend, layout, ns); const NamespaceLifeId life = catalogLife(backend, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(lifeEpochCkpt(1))).outcome, - PutOutcome::Done); + createAt(*backend, layout.refCkptKey(life), encodeRefCkpt(lifeEpochCkpt(1))); ASSERT_TRUE(readCkptForTest(backend, layout, life)->ckpt.life_epoch); ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, std::nullopt); @@ -1732,9 +1749,9 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryAdoptsFirstCommittedTxnAboveLifeEpochO }, RootMutationOrigin::Writer, RootMutationKind::Publish); }); ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); - ASSERT_TRUE(backend->get(layout.refLogKey(life, RefTxnId{1, 1}))); + ASSERT_TRUE(readAt(*backend, layout.refLogKey(life, RefTxnId{1, 1}))); ASSERT_EQ(readCkptForTest(backend, layout, life)->ckpt.committed_through, std::nullopt); - ASSERT_FALSE(backend->get(layout.refSnapshotKey(life, RefTxnId{1, 1}))) + ASSERT_FALSE(readAt(*backend, layout.refSnapshotKey(life, RefTxnId{1, 1}))) << "the grounding test must exercise the exact log successor, not a hinted snapshot"; backend->ambiguous_cas_count = 0; @@ -1766,21 +1783,22 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPriv return; injected = true; ASSERT_TRUE(expected); - ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(life, later.txn_id), - sealObject(FormatId::RefLog, encodeRefLogTxn(later))).outcome, - PutOutcome::Done); - const auto current = backend->get(key); + createAt(*backend, layout.refLogKey(life, later.txn_id), + sealObject(FormatId::RefLog, encodeRefLogTxn(later))); + OperationForTest op(*backend); + const auto current = (*op).read(key, Retry::once()); ASSERT_TRUE(current); - /// `expected` is the transport value the publisher is presenting; the legacy read hands back - /// the same observation wrapped in a `Token`, so its `value` is what compares. - ASSERT_EQ(current->token.value, *expected); + /// `expected` is the raw transport value the publisher is presenting; `PersistedEtag::capture` + /// re-derives the same raw value from the minted incarnation, so the two compare. + ASSERT_EQ(PersistedEtag::capture(current->etag).value, *expected); const RefCkpt advanced = mergeCkpt( decodeRefCkpt(current->bytes), RefCkpt{.life_epoch = std::nullopt, .committed_through = later.txn_id, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); - ASSERT_EQ(backend->putOverwrite(key, encodeRefCkpt(advanced), current->token).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative( + (*op).replace(key, encodeRefCkpt(advanced), current->etag, Retry::once()))); }; const auto refs = store->listRefs(ns); @@ -1826,9 +1844,8 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsTwoUnfrontieredSuccessorsAfterE ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); const RefLogTxn second_successor = makeOrdinaryTxn(ns, RefTxnId{1, 3}, "c", /*birth=*/false); - ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(life, second_successor.txn_id), - sealObject(FormatId::RefLog, encodeRefLogTxn(second_successor))).outcome, - PutOutcome::Done); + createAt(*backend, layout.refLogKey(life, second_successor.txn_id), + sealObject(FormatId::RefLog, encodeRefLogTxn(second_successor))); backend->ambiguous_cas_count = 0; expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)store->listRefs(ns); }); @@ -1847,13 +1864,15 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRejectsDifferentOrdinaryBytesAtTheReta ASSERT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); const String successor_key = layout.refLogKey(life, RefTxnId{1, 2}); - const auto original = backend->get(successor_key); + const auto original = readAt(*backend, successor_key); ASSERT_TRUE(original); const RefLogTxn different = makeOrdinaryTxn(ns, RefTxnId{1, 2}, "different", /*birth=*/false); - ASSERT_EQ(backend->putOverwrite(successor_key, - sealObject(FormatId::RefLog, encodeRefLogTxn(different)), - original->token).outcome, - PutOutcome::Done); + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).replace(successor_key, + sealObject(FormatId::RefLog, encodeRefLogTxn(different)), + original->etag, Retry::once()))); + } expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)store->listRefs(ns); }); EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery); @@ -1871,13 +1890,15 @@ TEST(CASRefRecoveryCasWalk, RetainedOldWriterAttemptLosesConclusiveToASuccessorS const RefTxnId successor_id{1, 2}; const String successor_key = layout.refLogKey(life, successor_id); - const auto original = backend->get(successor_key); + const auto original = readAt(*backend, successor_key); ASSERT_TRUE(original); const RefLogTxn successor_seal = makeSealTxn(ns, successor_id); - ASSERT_EQ(backend->putOverwrite(successor_key, - sealObject(FormatId::RefLog, encodeRefLogTxn(successor_seal)), - original->token).outcome, - PutOutcome::Done); + { + OperationForTest op(*backend); + ASSERT_TRUE(std::holds_alternative((*op).replace(successor_key, + sealObject(FormatId::RefLog, encodeRefLogTxn(successor_seal)), + original->etag, Retry::once()))); + } const auto refs = store->listRefs(ns); @@ -1953,8 +1974,10 @@ TEST(CASRefRecoveryCasWalk, ALatePredecessorPutAtTheSealedSlotIsRefusedByTheStor const RefTxnId ghost_id{1, 2}; const String ghost_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(makeOrdinaryTxn(ns, ghost_id, "ghost", /*birth=*/false))); - const PutResult put = backend->putIfAbsent(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), ghost_id), ghost_bytes); - EXPECT_EQ(put.outcome, PutOutcome::PreconditionFailed) + OperationForTest ghost_op(*backend); + const WriteResult put = (*ghost_op).create( + layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), ghost_id), ghost_bytes, Retry::once()); + EXPECT_TRUE(std::holds_alternative(put)) << "the seal occupies the ghost's own key, so the store itself is the fence"; /// And the object at that key is still the seal, byte for byte -- nothing adopted the ghost. @@ -2112,8 +2135,9 @@ TEST(CASRefRecoveryCasWalk, PutHookBackendComposesHidingListBackendCasPutFaultIn /// `HidingListBackend::write` only runs `before_cas_put` on the CONDITIONAL branch (an `expected` /// token present) -- a bare create-shaped `casPut(..., std::nullopt)` takes the other branch and /// can never reach it. Seed the key first so the probed call below is a genuine replace. - const auto seeded = backend->putIfAbsent("p/probe", "seed"); - ASSERT_EQ(seeded.outcome, PutOutcome::Done); + OperationForTest op(*backend); + const WriteResult seeded = (*op).create("p/probe", "seed", Retry::once()); + ASSERT_TRUE(std::holds_alternative(seeded)); bool before_cas_put_fired = false; backend->before_cas_put = [&](const String &, const String &, const std::optional &) @@ -2125,7 +2149,8 @@ TEST(CASRefRecoveryCasWalk, PutHookBackendComposesHidingListBackendCasPutFaultIn bool on_key_fired = false; backend->on_key = [&] { on_key_fired = true; }; - ASSERT_EQ(backend->casPut("p/probe", "x", seeded.token).outcome, CasOutcome::Committed); + ASSERT_TRUE(std::holds_alternative( + (*op).replace("p/probe", "x", std::get(seeded).etag, Retry::once()))); EXPECT_TRUE(before_cas_put_fired) << "HidingListBackend's before_cas_put hook must still fire for a PutHookBackend instance"; diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp index 0ac0b9e93644..1aa5ab55cb6f 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp @@ -363,7 +363,6 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) /// backoff decisions on is a DIFFERENT clock, and stays frozen between steps. CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; uint64_t fake_now = 1'000'000; @@ -489,7 +488,6 @@ TEST(CASRefSnapshotPublishOrdering, NotReadyRefusalBacksOffAndResetsAfterDurable /// so the budget only has to keep the mount lease admitting. CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; uint64_t fake_now = 2'000'000; diff --git a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp index 60df4db7809c..48645110e50a 100644 --- a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp +++ b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp @@ -3,7 +3,7 @@ #include "config.h" #include -#include +#include #include #include #include @@ -83,16 +83,30 @@ PoolPtr openPool(const BackendPtr & backend, CasRequestBudget budget = {}) return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); } +/// An exact read (mirrors the retired `backend.get(key)`). +std::optional readObj(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +/// A one-shot `create`, asserting it committed (mirrors the retired `backend.putIfAbsent(key, bytes)`). +void createObj(Backend & backend, const String & key, const String & bytes) +{ + DB::Cas::tests::OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + /// The budget every wedge test uses. It bounds the mount lease's own admission arithmetic /// (`attempt_timeout_ms` is what one attempt reserves, `lease_safety_margin_ms` the room kept past it) -/// and nothing else: a write's ATTEMPT COUNT is the `Retry` policy's, so no budget field can make an -/// injected fault conclusive. What makes a fault conclusive here is that it stays armed for the whole -/// call while `VirtualRetryClock` carries the call to its own deadline. +/// and nothing else: a write's ATTEMPT COUNT is the `Retry` policy's, and a call's own deadline is +/// fence-derived (`Retry::untilLeaseSafe`/`Retry.bind`), so no budget field can make an injected fault +/// conclusive. What makes a fault conclusive here is that it stays armed for the whole call while +/// `VirtualRetryClock` carries the call to its own deadline. CasRequestBudget wedgeTestBudget() { CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; /// strictly above attempt_timeout_ms: equality is refused by validateCasRequestBudget budget.lease_safety_margin_ms = 100; return budget; } @@ -119,8 +133,6 @@ void publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const String class WedgeTestBackend : public CountingBackend { public: - using CountingBackend::putIfAbsent; - /// One-shot ambiguity that writes NOTHING: the response is lost and the key stays absent, which is /// the input that makes a later bounded create commit. String ambiguous_substr; @@ -176,7 +188,7 @@ class WedgeTestBackend : public CountingBackend String refuse_precondition_substr; bool refuse_read_after_precondition = false; - /// Straight past every seam below, for the writes a test makes on its own behalf. `putIfAbsent` + /// Straight past every seam below, for the writes a test makes on its own behalf. `create` /// reaches the store through the VIRTUAL `write`, so a qualified call cannot bypass this override -- /// only this flag can. std::atomic bypass_seams{false}; @@ -252,13 +264,13 @@ class WedgeTestBackend : public CountingBackend /// Write straight through, bypassing every fault and block seam above -- how a test models what a /// SUCCESSOR (another process entirely) put at a key. Routing it through the seams instead would /// park the test's own write on the very gate it is trying to drive a scenario through, or spend - /// the fault meant for the lane's attempt. Qualifying the call does NOT achieve that: - /// `Backend::putIfAbsent` reaches the store through the VIRTUAL `write`, so the override above runs - /// either way -- `bypass_seams` is what it reads to step aside. - PutResult putAsSuccessor(const String & key, const String & bytes) + /// the fault meant for the lane's attempt. `create` reaches the store through the VIRTUAL `write`, + /// so the override above runs either way -- `bypass_seams` is what it reads to step aside. + WriteResult putAsSuccessor(const String & key, const String & bytes) { bypass_seams.store(true, std::memory_order_release); - const PutResult result = CountingBackend::putIfAbsent(key, bytes, ObjectMeta{}); + DB::Cas::tests::OperationForTest op(*this); + const WriteResult result = (*op).create(key, bytes, Retry::once()); bypass_seams.store(false, std::memory_order_release); return result; } @@ -379,9 +391,9 @@ CatalogEntry replaceCatalogLifeForWedgeRace( { return entry.ns == predecessor.ns && entry.incarnation == predecessor.incarnation; }); - if (!before_delete.incarnation + if (!before_delete.etag || !std::holds_alternative(op.replace( - layout.refCatalogKey(), encodeRefCatalog(without_predecessor), *before_delete.incarnation, + layout.refCatalogKey(), encodeRefCatalog(without_predecessor), *before_delete.etag, Retry::standard()))) throw std::runtime_error("test failed to retire exact predecessor catalog life"); @@ -393,9 +405,9 @@ CatalogEntry replaceCatalogLifeForWedgeRace( const CasRefCatalog::Snapshot after_delete = CasRefCatalog::read(op, layout); RefCatalog reborn = after_delete.catalog; reborn.entries.push_back(successor); - if (!after_delete.incarnation + if (!after_delete.etag || !std::holds_alternative(op.replace( - layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.incarnation, Retry::standard()))) + layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.etag, Retry::standard()))) throw std::runtime_error("test failed to publish successor catalog life"); return successor; } @@ -404,7 +416,8 @@ CatalogEntry replaceCatalogLifeForWedgeRace( /// hand-rolled parse), so an assertion about `prev_epoch_seal` is an assertion about the WIRE. RefLogTxn readRefLogTxn(Backend & backend, const Layout & layout, const RootNamespace & ns, const RefTxnId & id) { - const auto got = backend.get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id)); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id), Retry::standard()); if (!got) throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "no ref-log object at {}-{}", id.writer_epoch, id.ref_sequence); return decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), id); @@ -536,7 +549,7 @@ TEST(CASRefWedgeEveryAttempt, AmbiguousPutWedgesTheLaneAndTheNextFlushsCreateAdo wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->resolveRef(ns, "x").has_value()) << "a wedged transaction is not applied"; const String wedged_key = store->wedgedKeyForTest(ns); - ASSERT_FALSE(backend->get(wedged_key).has_value()) << "the ambiguous attempt wrote nothing"; + ASSERT_FALSE(readObj(*backend, wedged_key).has_value()) << "the ambiguous attempt wrote nothing"; const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); /// The next caller's flush resolves the wedge with ONE create, adopts it, and only then carves and @@ -546,7 +559,7 @@ TEST(CASRefWedgeEveryAttempt, AmbiguousPutWedgesTheLaneAndTheNextFlushsCreateAdo EXPECT_FALSE(store->refLaneWedgedForTest(ns)); EXPECT_FALSE(store->resolveRef(ns, "x").has_value()) << "the adopted wedge applied its drop"; EXPECT_FALSE(store->resolveRef(ns, "y").has_value()) << "the resolving flush committed its own drop"; - EXPECT_TRUE(backend->get(wedged_key).has_value()) << "the wedged transaction is durable at its own key"; + EXPECT_TRUE(readObj(*backend, wedged_key).has_value()) << "the wedged transaction is durable at its own key"; EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_before + 2) << "the adopted wedge and the ordinary commit must each join the tail exactly once"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Ready); @@ -564,12 +577,12 @@ TEST(CASRefWedgeEveryAttempt, DurableCreatedWedgeNeedsRecoveryWhenItsFrontierCan wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); - ASSERT_FALSE(backend->get(wedged_key)); + ASSERT_FALSE(readObj(*backend, wedged_key)); const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); const NamespaceLifeId life = *store->refTableLifeForTest(ns); const String ckpt_key = store->layout().refCkptKey(life); - const RefCkpt ckpt_before = decodeRefCkpt(backend->get(ckpt_key)->bytes); + const RefCkpt ckpt_before = decodeRefCkpt(readObj(*backend, ckpt_key)->bytes); /// Latched, not counted: the frontier publish reissues an ambiguous replace until ITS window /// closes, so a bounded fault would simply be outlived and the publication would succeed. backend->fail_cas_substr = ckpt_key; @@ -582,12 +595,12 @@ TEST(CASRefWedgeEveryAttempt, DurableCreatedWedgeNeedsRecoveryWhenItsFrontierCan EXPECT_GT(clock->pauseCount(), pauses_before + 1) << "the publication's reissues must pace through the injected sleep, never a real one"; - EXPECT_TRUE(backend->get(wedged_key)) << "the exact wedged log was proven durable"; + EXPECT_TRUE(readObj(*backend, wedged_key)) << "the exact wedged log was proven durable"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::NeedsRecovery) << "a durable log without a confirmed frontier must not return to Ready"; EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), tail_before) << "the unfrontiered wedge must not be installed into the resident table"; - EXPECT_EQ(decodeRefCkpt(backend->get(ckpt_key)->bytes), ckpt_before); + EXPECT_EQ(decodeRefCkpt(readObj(*backend, ckpt_key)->bytes), ckpt_before); } TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdoption) @@ -605,7 +618,7 @@ TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdop wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { store->dropRef(ns, "x"); }); const String wedged_key = store->wedgedKeyForTest(ns); - ASSERT_FALSE(backend->get(wedged_key)); + ASSERT_FALSE(readObj(*backend, wedged_key)); std::mutex mutex; std::condition_variable cv; @@ -640,10 +653,10 @@ TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdop = replaceCatalogLifeForWedgeRace(backend, store->layout(), predecessor, UInt128{0x71f2}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); - ASSERT_EQ(backend->putIfAbsent(store->layout().refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ + createObj(*backend, store->layout().refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ .life_epoch = store->liveWriterEpoch(), .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt})); store->invalidateRemovedCatalogLife(predecessor_life); backend->resetCounts(); @@ -658,7 +671,7 @@ TEST(CASRefWedgeEveryAttempt, RetiredLifeRefusesWedgeRetryBeforeAnyRequestOrAdop EXPECT_TRUE(retry_error); EXPECT_EQ(backend->putCount(wedged_key), 0u) << "retirement must refuse before the retry send"; EXPECT_EQ(backend->getCount(wedged_key), 0u) << "a refused retry needs no occupant resolution read"; - EXPECT_FALSE(backend->get(wedged_key)) << "the predecessor wedge was adopted or made durable"; + EXPECT_FALSE(readObj(*backend, wedged_key)) << "the predecessor wedge was adopted or made durable"; EXPECT_NO_THROW((void)store->listRefs(ns)); ASSERT_TRUE(store->refTableLifeForTest(ns)); EXPECT_EQ(*store->refTableLifeForTest(ns), successor_life); @@ -682,7 +695,7 @@ TEST(CASRefWedgeEveryAttempt, OwnLandedAttemptIsAdoptedFromOccupiedWithoutDouble wedgeLaneOverADurableObject(*clock, *backend, store, ns, "x"); const String wedged_key = store->wedgedKeyForTest(ns); - ASSERT_TRUE(backend->get(wedged_key).has_value()) << "this fault LANDS the write; only the ack was lost"; + ASSERT_TRUE(readObj(*backend, wedged_key).has_value()) << "this fault LANDS the write; only the ack was lost"; ASSERT_TRUE(store->resolveRef(ns, "x").has_value()) << "durable, but not applied while wedged"; const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); const uint64_t puts_before = backend->putCount(wedged_key); @@ -722,7 +735,7 @@ TEST(CASRefWedgeEveryAttempt, DefiniteRefusalOfARetryAttemptKeepsTheLaneWedged) EXPECT_TRUE(store->refLaneWedgedForTest(ns)) << "a definite refusal AFTER an ambiguous attempt must not unwedge"; EXPECT_EQ(store->wedgedKeyForTest(ns), wedged_key) << "the SAME wedge, not a fresh one"; EXPECT_TRUE(store->resolveRef(ns, "x").has_value()) << "nothing was adopted"; - EXPECT_FALSE(backend->get(wedged_key).has_value()) << "and nothing became durable"; + EXPECT_FALSE(readObj(*backend, wedged_key).has_value()) << "and nothing became durable"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged) << "a wedged lane's steady state is 'may be durable, not applied'"; @@ -838,7 +851,7 @@ TEST(CASRefWedgeEveryAttempt, ADefiniteRefusalAfterAnAmbiguousAttemptOfTheSameCa EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(), wedged_before + 1); EXPECT_TRUE(store->resolveRef(ns, "x").has_value()) << "nothing is applied while the lane is wedged"; const String wedged_key = store->wedgedKeyForTest(ns); - EXPECT_FALSE(backend->get(wedged_key).has_value()) << "and nothing became durable"; + EXPECT_FALSE(readObj(*backend, wedged_key).has_value()) << "and nothing became durable"; /// And it still recovers by the ordinary route: the next flush's bounded create lands the wedged /// transaction and adopts it, so wedging costs availability only until the next caller arrives. @@ -879,7 +892,7 @@ TEST(CASRefWedgeEveryAttempt, SuccessorSealAtTheWedgedKeyRejectsConclusivelyAndS const size_t tail_before = store->tailSinceSnapshotCountForTest(ns); /// A successor closes our epoch at exactly the slot our attempt was aiming at. - ASSERT_EQ(backend->putAsSuccessor(wedged_key, epochSealBytes(ns, seal_id)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(backend->putAsSuccessor(wedged_key, epochSealBytes(ns, seal_id)))); /// The next caller's resolution meets the seal. Its own items fail -- permanently, not "retry /// later": nothing about this lane's epoch will ever accept a write again. @@ -903,7 +916,7 @@ TEST(CASRefWedgeEveryAttempt, SuccessorSealAtTheWedgedKeyRejectsConclusivelyAndS /// wedge-resolve site does. const uint64_t remounts_before = store->scheduleRemountCallCountForTest(); expectThrowsCode(DB::ErrorCodes::INVALID_STATE, [&] { store->dropRef(ns, "y"); }); - EXPECT_EQ(backend->get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), RefTxnId{epoch, seal_id.ref_sequence + 1})), std::nullopt) + EXPECT_EQ(readObj(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), RefTxnId{epoch, seal_id.ref_sequence + 1})), std::nullopt) << "nothing of ours may exist above the seal in the closed epoch"; EXPECT_TRUE(store->mayMutate()) << "meeting a successor's seal is the protocol working, not an anomaly"; EXPECT_EQ(store->scheduleRemountCallCountForTest(), remounts_before) @@ -1001,7 +1014,7 @@ TEST(CASRefWedgeEveryAttempt, ForeignNonSealOccupantIsCorruptedDataAndSchedulesA const uint64_t remounts_before = store->scheduleRemountCallCountForTest(); /// Something that is neither our bytes nor a seal occupies the slot. - ASSERT_EQ(backend->putAsSuccessor(wedged_key, "not a ref-log object at all").outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(backend->putAsSuccessor(wedged_key, "not a ref-log object at all"))); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { store->dropRef(ns, "y"); }); @@ -1027,8 +1040,8 @@ TEST(CASRefWedgeEveryAttempt, AppendSiteProvenDifferentObjectAlsoSchedulesARemou /// Occupy the id the next append will derive with a foreign object, so its create conflicts and /// the controller's resolve-before-reissue proves the occupant is not ours. const RefTxnId next{store->liveWriterEpoch(), 3}; - ASSERT_EQ(backend->putIfAbsent(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), next), - "a different object entirely").outcome, PutOutcome::Done); + createObj(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), next), + "a different object entirely"); const uint64_t remounts_before = store->scheduleRemountCallCountForTest(); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { store->dropRef(ns, "x"); }); @@ -1073,7 +1086,7 @@ TEST(CASRefWedgeEveryAttempt, RetryUnderAnOlderAdmissionGenerationSendsNothing) EXPECT_EQ(backend->putCount(wedged_key), puts_before) << "the retry must be refused pre-attempt: nothing may reach the store under a foreign generation"; EXPECT_TRUE(store->refLaneWedgedForTest(ns)) << "and the wedge is untouched"; - EXPECT_FALSE(backend->get(wedged_key).has_value()); + EXPECT_FALSE(readObj(*backend, wedged_key).has_value()); } /// The post-I/O recheck, deterministically. The retry's create is parked mid-flight; while it is @@ -1110,7 +1123,7 @@ TEST(CASRefWedgeEveryAttempt, ResultReleasedAfterAFenceBumpAndSuccessorSealIsIne backend->awaitBlockEntered(); /// Everything that makes this runtime superseded happens INSIDE the I/O window. - ASSERT_EQ(backend->putAsSuccessor(wedged_key, epochSealBytes(ns, seal_id)).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative(backend->putAsSuccessor(wedged_key, epochSealBytes(ns, seal_id)))); bumpFenceGeneration(store, epoch + 1); backend->releaseBlock(); resolver.join(); @@ -1186,7 +1199,7 @@ TEST(CASRefWedgeEveryAttempt, KnownDurableInstallFailureMovesDirectlyToRecovery) wedgeLaneOverADurableObject(*clock, *backend, store, ns, "x"); const String wedged_key = store->wedgedKeyForTest(ns); - ASSERT_TRUE(backend->get(wedged_key).has_value()) << "the wedged transaction is durable"; + ASSERT_TRUE(readObj(*backend, wedged_key).has_value()) << "the wedged transaction is durable"; /// The adoption reaches its install region and the install throws. armOneShotInstallFailure(store); expectThrowsCode(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED, [&] { store->dropRef(ns, "y"); }); @@ -1275,7 +1288,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesAConclusiveFirstRefLogRejectio const RefTxnId genesis{store->liveWriterEpoch(), 1}; const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); - const auto ckpt_before = backend->get(ckpt_key); + const auto ckpt_before = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation checkpoint must exist before the first ref-log attempt"; /// A successor's epoch seal lands at exactly the id this first `NamespaceBirth` transaction derives. @@ -1285,7 +1298,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesAConclusiveFirstRefLogRejectio expectThrowsCode(DB::ErrorCodes::INVALID_STATE, [&] { publishEmptyPart(store, ns, "x"); }); - const auto ckpt_after = backend->get(ckpt_key); + const auto ckpt_after = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) << "the creation checkpoint must survive a conclusively rejected first ref-log PUT unchanged"; @@ -1315,7 +1328,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesALaterConclusiveRejection) publishEmptyPart(store, ns, "x"); const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); - const auto ckpt_before = backend->get(ckpt_key); + const auto ckpt_before = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation step must have published a real _ckpt"; const RefTxnId next{store->liveWriterEpoch(), 3}; @@ -1325,7 +1338,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesALaterConclusiveRejection) expectThrowsCode(DB::ErrorCodes::INVALID_STATE, [&] { store->dropRef(ns, "x"); }); - const auto ckpt_after = backend->get(ckpt_key); + const auto ckpt_after = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_after.has_value()) << "a Live namespace's _ckpt must never be deleted by this path"; EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) << "not merely present but UNCHANGED -- no code anywhere on this path deletes _ckpt any more " @@ -1349,13 +1362,13 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthIsAm const RootNamespace ns{"srv1/birth_ckpt_ambiguous"}; admitProperlyBornEntry(backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); - const auto ckpt_before = backend->get(ckpt_key); + const auto ckpt_before = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation checkpoint must exist before the first ref-log attempt"; /// The wedge is the assertion: an ambiguous outcome must not resolve into one of the conclusive /// branches, so `wedgeLaneOnUnresolvedAppend` insisting on it is what this row needs. wedgeLaneOnUnresolvedAppend(*clock, *backend, store, ns, [&] { publishEmptyPart(store, ns, "x"); }); - const auto ckpt_after = backend->get(ckpt_key); + const auto ckpt_after = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) << "the creation checkpoint must survive an ambiguous first ref-log outcome unchanged"; @@ -1377,7 +1390,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthName const RefTxnId genesis{store->liveWriterEpoch(), 1}; const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); - const auto ckpt_before = backend->get(ckpt_key); + const auto ckpt_before = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation checkpoint must exist before the first ref-log attempt"; /// The store refuses the birth create's precondition while the key is in fact ABSENT, so the @@ -1386,7 +1399,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesWhenTheFirstNamespaceBirthName expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { publishEmptyPart(store, ns, "x"); }); - const auto ckpt_after = backend->get(ckpt_key); + const auto ckpt_after = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) << "the creation checkpoint must survive an unnameable first ref-log occupant unchanged"; @@ -1404,7 +1417,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesFirstNamespaceBirthForeignInte const RefTxnId genesis{store->liveWriterEpoch(), 1}; const String ckpt_key = layout.refCkptKey(DB::Cas::tests::fixture::fixtureLife(ns)); - const auto ckpt_before = backend->get(ckpt_key); + const auto ckpt_before = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_before.has_value()) << "the fixture's creation checkpoint must exist before the first ref-log attempt"; /// A perfectly decodable transaction for this exact namespace and id -- just not an epoch seal, and @@ -1418,7 +1431,7 @@ TEST(CASRefWedgeEveryAttempt, CreationCkptSurvivesFirstNamespaceBirthForeignInte expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { publishEmptyPart(store, ns, "x"); }); - const auto ckpt_after = backend->get(ckpt_key); + const auto ckpt_after = readObj(*backend, ckpt_key); ASSERT_TRUE(ckpt_after.has_value()); EXPECT_EQ(ckpt_after->bytes, ckpt_before->bytes) << "the creation checkpoint must survive a foreign-interference first ref-log outcome unchanged"; @@ -1601,7 +1614,7 @@ TEST(CASRefWedgeEveryAttempt, ALiveEpochSealIsNeverStampedAsItsOwnPrevEpochSeal) EXPECT_NE(e.message().find("resumes only under a later epoch"), String::npos) << "the deposition must be surfaced, not just the failure: " << e.message(); } - EXPECT_FALSE(backend->get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), RefTxnId{epoch + 1, 1})).has_value()) + EXPECT_FALSE(readObj(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), RefTxnId{epoch + 1, 1})).has_value()) << "nothing may be written: the lane could not construct a legal transaction, so it sent none"; EXPECT_EQ(backend->putCount(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), RefTxnId{epoch + 1, 1})), puts_before) << "and no request was spent learning what the lane could already prove about itself"; @@ -1693,7 +1706,7 @@ TEST(CASRefLane, PostCommitFenceLossWedges) << "the object is durable, so the lane must not be returned to Ready"; EXPECT_EQ(store->laneStateForTest(ns), RefLaneState::Wedged); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefAppendWedged].load(), wedged_before + 1); - EXPECT_TRUE(backend->get(store->wedgedKeyForTest(ns)).has_value()) + EXPECT_TRUE(readObj(*backend, store->wedgedKeyForTest(ns)).has_value()) << "the write landed -- what was refused is the CLAIM, not the object"; } diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index d3fefe8109b4..f0fc0fafb2be 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -72,12 +72,6 @@ using DB::Cas::tests::writeSealAt; namespace { -/// The operation deadline every wedge fixture in this file uses, distinct from `attempt_timeout_ms` -/// because `validateCasRequestBudget` requires the two to differ strictly: `Pool::open` refuses a -/// budget where `attempt_timeout_ms >= operation_deadline_ms` with `BAD_ARGUMENTS`, so an equal or -/// smaller value would refuse to open the pool at all rather than exercise any fixture below. -constexpr uint64_t kSingleAttemptDeadlineMs = 5000; - /// The budget every wedge fixture here uses. It bounds the mount lease's own admission arithmetic and /// nothing else: a write's ATTEMPT COUNT is the `Retry` policy's, and the ref lane's is `standard`, so /// no budget field can make an injected fault conclusive. What does is `driveToTheWedge` below -- @@ -86,7 +80,6 @@ CasRequestBudget wedgeTestBudget() { CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = kSingleAttemptDeadlineMs; budget.lease_safety_margin_ms = 100; return budget; } @@ -198,7 +191,7 @@ PoolPtr openPoolWithConfig(const BackendPtr & backend, PoolConfig config) /// production birth mints a random incarnation and those computed keys land nowhere real. PartWriteTxnPtr startBuildFor(const PoolPtr & s, const RootNamespace & ns, const String & ref) { - DB::Cas::tests::casAdmitRecoverableEntry(s->backend(), s->layout(), ns, s->liveWriterEpoch()); + DB::Cas::tests::casAdmitRecoverableEntry(*s->poolBackendPtr(), s->layout(), ns, s->liveWriterEpoch()); PartWriteInfo info; info.intended_namespace = ns; info.intended_ref = ns.string() + "/" + ref; @@ -244,7 +237,7 @@ CasRefCatalog::Snapshot readCatalogForTest(const BackendPtr & backend, const Lay /// A fixture's own conditional replace, for the races these tests stage by hand. bool replaceForTest(const BackendPtr & backend, const String & key, const String & bytes, - const Incarnation & expected) + const Etag & expected) { CasRequests requests(backend, Fence::open()); CasOperation op = requests.admit(); @@ -277,13 +270,29 @@ uint64_t allocateWriterEpochForTest(const BackendPtr & backend, const Layout & l /// A fixture enumeration on an open fence: the primitive `list` override in this file's test backend /// hides the legacy name, and a test walking a prefix should ride the same engine production does. -KeyPage listForTest(const BackendPtr & backend, const String & prefix, const String & cursor, size_t limit) +ListPage listForTest(const BackendPtr & backend, const String & prefix, const String & cursor, size_t limit) { CasRequests requests(backend, Fence::open()); CasOperation op = requests.admit(); return op.list(prefix, cursor, limit, Retry::standard()); } +/// The same open-fence idiom as `listForTest` above, for the raw observations this file's fixtures make +/// directly against a `RefWriterTestBackend` outside any Pool operation. +std::optional readOf(const BackendPtr & backend, const String & key) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return op.read(key, Retry::standard()); +} + +bool createRaw(const BackendPtr & backend, const String & key, const String & bytes) +{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + return std::holds_alternative(op.create(key, bytes, Retry::standard())); +} + CatalogEntry catalogEntryOrThrow(const BackendPtr & backend, const Layout & layout, const RootNamespace & ns) { const RefCatalog catalog = readCatalogForTest(backend, layout).catalog; @@ -306,9 +315,9 @@ CatalogEntry replaceCatalogLifeForRuntimeRace( { return entry.ns == predecessor.ns && entry.incarnation == predecessor.incarnation; }); - if (!before_delete.incarnation + if (!before_delete.etag || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(without_predecessor), - *before_delete.incarnation)) + *before_delete.etag)) throw std::runtime_error("test failed to retire exact predecessor catalog life"); CatalogEntry successor{ @@ -319,8 +328,8 @@ CatalogEntry replaceCatalogLifeForRuntimeRace( const CasRefCatalog::Snapshot after_delete = readCatalogForTest(backend, layout); RefCatalog reborn = after_delete.catalog; reborn.entries.push_back(successor); - if (!after_delete.incarnation - || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.incarnation)) + if (!after_delete.etag + || !replaceForTest(backend, layout.refCatalogKey(), encodeRefCatalog(reborn), *after_delete.etag)) throw std::runtime_error("test failed to publish successor catalog life"); return successor; } @@ -331,11 +340,12 @@ std::optional listGreatestLogIdForTest( std::optional listGreatestLogIdForLifeForTest( Backend & backend, const Layout & layout, const NamespaceLifeId & life) { + DB::Cas::tests::OperationForTest op(backend); std::optional greatest; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(life), cursor, 1000); + const ListPage page = (*op).list(layout.namespaceStreamPrefix(life), cursor, 1000, Retry::standard()); for (const ListedKey & listed : page.keys) { const auto parsed = layout.parseRefObjectKey(listed.key); @@ -373,9 +383,10 @@ CompletedRemovingFixture prepareResidentRemovalForDrain( if (runRegularRoundReclaiming(gc).deferred) throw std::runtime_error("fixture terminal fold unexpectedly deferred"); - const GcState state = decodeGcState(backend->get(store->layout().gcStateKey())->bytes); + DB::Cas::tests::OperationForTest drain_op(backend); + const GcState state = decodeGcState((*drain_op).read(store->layout().gcStateKey(), Retry::standard())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend->get(store->layout().foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + (*drain_op).read(store->layout().foldSealKey(state.snap_generation, state.snap_attempt), Retry::standard())->bytes); const auto row = seal.ref_lives.find(predecessor.incarnation); if (row == seal.ref_lives.end() || !row->second.cleanup_evidence) throw std::runtime_error("fixture terminal fold produced no cleanup evidence"); @@ -395,11 +406,12 @@ ManifestRef manifestRef(uint64_t epoch, uint64_t seq, uint32_t ordinal) RefTableState independentFullReplayForTest(Backend & backend, const Layout & layout, const RootNamespace & ns, std::optional up_to = std::nullopt) { + DB::Cas::tests::OperationForTest op(backend); std::vector ids; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*op).list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -416,7 +428,7 @@ RefTableState independentFullReplayForTest(Backend & backend, const Layout & lay RefTableState state; for (const RefTxnId & id : ids) { - const auto got = backend.get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id)); + const auto got = (*op).read(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), id), Retry::standard()); applyRefLogTxn(state, decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), id)); } return state; @@ -426,11 +438,12 @@ RefTableState independentFullReplayForTest(Backend & backend, const Layout & lay /// of the Pool's own cached bookkeeping). std::optional listGreatestSnapshotIdForTest(Backend & backend, const Layout & layout, const RootNamespace & ns) { + DB::Cas::tests::OperationForTest op(backend); std::optional greatest; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*op).list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -448,7 +461,7 @@ std::optional listGreatestSnapshotIdForTest(Backend & backend, const L /// A backend that can (a) force one `get()` on a chosen exact key to return absent exactly once /// (simulating an object vanishing after recovery sampled its exact checkpoint, with an optional side effect /// fired at that exact moment -- e.g. publishing a covering newer snapshot, mirroring a concurrent GC -/// cleanup+republish race), and (b) force `putIfAbsent` on keys matching a chosen substring to throw an +/// cleanup+republish race), and (b) force a create-shaped write on keys matching a chosen substring to throw an /// ambiguous (Unresolved-classified) exception a bounded number of times, optionally still capturing /// the (key, bytes) so a test can later "deliver" it -- simulating a request whose RESPONSE was lost /// even though the write eventually landed server-side. @@ -460,8 +473,6 @@ class RefWriterTestBackend : public CountingBackend DB::Cas::tests::seedPoolMetaForRestart(*this); } - using CountingBackend::getStream; - void clearRequestJournal() { std::lock_guard lock(request_journal_mutex); @@ -516,7 +527,7 @@ class RefWriterTestBackend : public CountingBackend int fault_skip = 0; std::optional> pending_delayed_write; - /// (I1) On a matching `putIfAbsent`, a FOREIGN writer lands a DIFFERENT object at the exact key and + /// (I1) On a matching create-shaped write, a FOREIGN writer lands a DIFFERENT object at the exact key and /// then this attempt's response is lost -- so the controller's resolve-before-reissue GET observes /// different bytes and must raise CORRUPTED_DATA (a proven conflict, never a retry signal). /// By default the foreign object is the attempt's own bytes plus a trailing marker -- UNDECODABLE @@ -733,8 +744,8 @@ class RefWriterTestBackend : public CountingBackend block_cv.notify_all(); return r; } - /// See `putIfAbsent`'s `block_this` branch. Set before spawning any thread that could race - /// `putIfAbsent`, like `corrupt_key_substr`/`fault_key_substr` above -- not itself lock-protected. + /// See `createForTest`'s `block_this` branch. Set before spawning any thread that could race + /// a create-shaped write, like `corrupt_key_substr`/`fault_key_substr` above -- not itself lock-protected. bool block_throw_corrupted_on_release = false; /// "Deliver" the earlier ambiguous write: the request DID eventually land server-side, the caller @@ -743,12 +754,13 @@ class RefWriterTestBackend : public CountingBackend { if (pending_delayed_write) { - (void)putIfAbsent(pending_delayed_write->first, pending_delayed_write->second); + DB::Cas::tests::OperationForTest op(*this); + (void)(*op).create(pending_delayed_write->first, pending_delayed_write->second, DB::Cas::Retry::once()); pending_delayed_write.reset(); } } - /// Task 11: blocks EVERY `putIfAbsent()` whose key contains `armed_block_substr` until + /// Task 11: blocks EVERY create-shaped write whose key contains `armed_block_substr` until /// `releaseBlock()` is called, notifying `awaitBlockEntered()` the first time one is reached. Used /// to prove snapshot publication never holds up an unrelated concurrent append. void armPutBlock(const String & substr) @@ -762,7 +774,7 @@ class RefWriterTestBackend : public CountingBackend blocked_key.clear(); } - /// Task 11 (monotonic-adoption harness): block ONLY the FIRST `putIfAbsent` whose key contains + /// Task 11 (monotonic-adoption harness): block ONLY the FIRST create-shaped write whose key contains /// `substr`, capturing that exact key; every LATER put -- including a DIFFERENT `_snap/` key -- /// proceeds unblocked. Lets a test pin one in-flight publish's PUT mid-flight while a second, /// higher-id publish runs to completion, deterministically forcing the out-of-order overlap. @@ -789,8 +801,8 @@ class RefWriterTestBackend : public CountingBackend } block_cv.notify_all(); } - /// Blocks until the PREVIOUSLY-blocked `putIfAbsent` call has actually RETURNED (not merely been - /// unblocked) -- i.e. its underlying `CountingBackend::putIfAbsent` has completed. Deterministic, + /// Blocks until the PREVIOUSLY-blocked create-shaped write call has actually RETURNED (not merely been + /// unblocked) -- i.e. its underlying `CountingBackend::write` has completed. Deterministic, /// sleep-free way to observe a detached background caller's own work finishing when the TEST no /// longer holds anything (e.g. a Pool handle) that call would otherwise let it wait on. void awaitBlockedCallCompleted() @@ -799,7 +811,7 @@ class RefWriterTestBackend : public CountingBackend block_cv.wait(lk, [&] { return block_call_completed; }); } - /// (I1 regression harness) Arms independent per-key blocking for every `putIfAbsent` matching + /// (I1 regression harness) Arms independent per-key blocking for every create-shaped write matching /// `substr`: unlike `armPutBlock`/`armPutBlockFirstMatchOnly` (one shared release gate), each /// blocked key parks on ITS OWN release (`releaseKey`), so two distinct `_snap/` PUTs can be /// parked concurrently -- both past their capture point, neither yet adopted -- and released in a @@ -900,7 +912,7 @@ TEST(CASRefWriterNonMinting, ListRefsOnAbsentNamespaceDoesNotMutateCatalog) auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/list_absent_non_minting"}; - const auto catalog_before = backend->get(layout.refCatalogKey()); + const auto catalog_before = readOf(backend, layout.refCatalogKey()); ASSERT_TRUE(catalog_before); backend->resetCounts(); @@ -910,10 +922,10 @@ TEST(CASRefWriterNonMinting, ListRefsOnAbsentNamespaceDoesNotMutateCatalog) EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->writeCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->deleteCount(layout.refCatalogKey()), 0u); - const auto catalog_after = backend->get(layout.refCatalogKey()); + const auto catalog_after = readOf(backend, layout.refCatalogKey()); ASSERT_TRUE(catalog_after); EXPECT_EQ(catalog_after->bytes, catalog_before->bytes); - EXPECT_EQ(catalog_after->token, catalog_before->token); + EXPECT_EQ(catalog_after->etag, catalog_before->etag); } TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsCatalogLifeReplacedWithoutLocalInvalidation) @@ -959,10 +971,10 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsCatalogLifeReplacedWithoutLocal = replaceCatalogLifeForRuntimeRace(backend, layout, predecessor, UInt128{0xabc002}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ .life_epoch = store->liveWriterEpoch(), .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); { std::lock_guard lock(mutex); resume = true; @@ -997,12 +1009,13 @@ TEST(CASRefWriterRuntimeIdentity, ColdReadRejectsReplacementByExternalPoolActor) external_store->poolBackendPtr(), external_store->layout(), predecessor, UInt128{0xabc003}); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(successor.ns, successor.incarnation); - if (external_store->backend().putIfAbsent( + OperationForTest successor_op(*external_store->poolBackendPtr()); + if (!std::holds_alternative((*successor_op).create( external_store->layout().refCkptKey(successor_life), encodeRefCkpt(RefCkpt{ .life_epoch = external_store->liveWriterEpoch(), .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome != PutOutcome::Done) + .last_epoch_seal = std::nullopt}), Retry::standard()))) throw std::runtime_error("test failed to publish external successor checkpoint"); }); @@ -1098,7 +1111,7 @@ TEST(CASRefWriterNonMinting, ResolveRefOnAbsentNamespaceDoesNotMutateCatalog) auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/resolve_absent_non_minting"}; - const auto catalog_before = backend->get(layout.refCatalogKey()); + const auto catalog_before = readOf(backend, layout.refCatalogKey()); ASSERT_TRUE(catalog_before); backend->resetCounts(); @@ -1108,10 +1121,10 @@ TEST(CASRefWriterNonMinting, ResolveRefOnAbsentNamespaceDoesNotMutateCatalog) EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->writeCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->deleteCount(layout.refCatalogKey()), 0u); - const auto catalog_after = backend->get(layout.refCatalogKey()); + const auto catalog_after = readOf(backend, layout.refCatalogKey()); ASSERT_TRUE(catalog_after); EXPECT_EQ(catalog_after->bytes, catalog_before->bytes); - EXPECT_EQ(catalog_after->token, catalog_before->token); + EXPECT_EQ(catalog_after->etag, catalog_before->etag); } TEST(CASRefWriterNonMinting, DropNamespaceOnAbsentNamespaceDoesNotMutateCatalog) @@ -1120,7 +1133,7 @@ TEST(CASRefWriterNonMinting, DropNamespaceOnAbsentNamespaceDoesNotMutateCatalog) auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/drop_absent_non_minting"}; - const auto catalog_before = backend->get(layout.refCatalogKey()); + const auto catalog_before = readOf(backend, layout.refCatalogKey()); ASSERT_TRUE(catalog_before); backend->resetCounts(); @@ -1130,10 +1143,10 @@ TEST(CASRefWriterNonMinting, DropNamespaceOnAbsentNamespaceDoesNotMutateCatalog) EXPECT_EQ(backend->putOverwriteCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->writeCount(layout.refCatalogKey()), 0u); EXPECT_EQ(backend->deleteCount(layout.refCatalogKey()), 0u); - const auto catalog_after = backend->get(layout.refCatalogKey()); + const auto catalog_after = readOf(backend, layout.refCatalogKey()); ASSERT_TRUE(catalog_after); EXPECT_EQ(catalog_after->bytes, catalog_before->bytes); - EXPECT_EQ(catalog_after->token, catalog_before->token); + EXPECT_EQ(catalog_after->etag, catalog_before->etag); } /// A table born by a log tail alone (no snapshot yet): `namespace_birth` with nothing else is a legal @@ -1146,11 +1159,11 @@ TEST(CASRefWriterRecovery, BirthOnlyLogNoSnapshotRecoversToEmptyLiveTable) DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ns.string(), RefTxnId{1, 1}, {namespaceBirthOp()}, std::nullopt}); const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 1}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); auto store = openPool(backend); EXPECT_TRUE(store->listRefs(ns).empty()); @@ -1170,11 +1183,11 @@ TEST(CASRefWriterRecovery, BirthPlusPrecommitPromoteAcrossTwoLogsNoSnapshot) DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ns.string(), RefTxnId{1, 2}, {publishCommittedOps("part_1", m1)[1]}, std::nullopt}); const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 2}, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); auto store = openPool(backend); const auto resolved = store->resolveRef(ns, "part_1"); @@ -1252,11 +1265,11 @@ TEST(CASRefWriterRecovery, SnapshotPlusTailRecovery) tail_ops.push_back(publishCommittedOps("b", mb)[1]); DB::Cas::tests::fixture::writeRefLogRaw(*backend, layout, RefLogTxn{ns.string(), RefTxnId{1, 6}, tail_ops, std::nullopt}); const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, layout, ns); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 6}, .checkpoint_snapshot_id = RefTxnId{1, 5}, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); backend->resetCounts(); auto store = openPool(backend); @@ -1295,11 +1308,11 @@ TEST(CASRefWriterRecovery, RestartOnVanishConvergesOnNewerSnapshot) .ops = publishCommittedOps("a", ma), .prev_epoch_seal = std::nullopt}); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), snap_x, {committedRow("a", ma)})); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 10}, .checkpoint_snapshot_id = snap_x, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); backend->vanish_once_keys.insert(layout.refSnapshotKey(life, snap_x)); bool vanish_fired = false; backend->on_vanish_fire = [&] @@ -1312,13 +1325,15 @@ TEST(CASRefWriterRecovery, RestartOnVanishConvergesOnNewerSnapshot) .ops = publishCommittedOps("b", mb), .prev_epoch_seal = std::nullopt}); writeRefSnapshotRaw(*backend, layout, minimalLiveSnapshot(ns.string(), snap_y, {committedRow("b", mb)})); - const auto before = backend->get(layout.refCkptKey(life)); + const auto before = readOf(backend, layout.refCkptKey(life)); ASSERT_TRUE(before); - ASSERT_EQ(backend->casPut(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.replace(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = RefTxnId{1, 20}, .checkpoint_snapshot_id = snap_y, - .last_epoch_seal = std::nullopt}), before->token).outcome, CasOutcome::Committed); + .last_epoch_seal = std::nullopt}), before->etag, Retry::standard()))); }; backend->resetCounts(); @@ -1361,14 +1376,13 @@ TEST(CASRefWriterRecovery, DifferentBytesAtSelectedSnapshotIsCorruptionNotRestar foreign.ns = other_ns.string(); foreign.snapshot_id = snap_x; const String snapshot_key = layout.refSnapshotKey(life, snap_x); - ASSERT_EQ(backend->putIfAbsent(snapshot_key, - DB::Cas::sealObject(DB::Cas::FormatId::RefSnapshot, DB::Cas::encodeRefTableSnapshot(foreign))).outcome, - PutOutcome::Done); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, snapshot_key, + DB::Cas::sealObject(DB::Cas::FormatId::RefSnapshot, DB::Cas::encodeRefTableSnapshot(foreign)))); + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = 1, .committed_through = snap_x, .checkpoint_snapshot_id = snap_x, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); auto store = openPool(backend); backend->resetCounts(); @@ -1562,9 +1576,9 @@ TEST(CASRefWriterAppendLane, CheckpointConflictAfterLogCommitRequiresRecoveryWit const auto after = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(after); EXPECT_EQ(after->ckpt.committed_through, before->ckpt.committed_through); - EXPECT_TRUE(backend->get(store->layout().refLogKey(life, candidate))) + EXPECT_TRUE(readOf(backend, store->layout().refLogKey(life, candidate))) << "the log PUT committed before checkpoint publication failed"; - EXPECT_FALSE(backend->get(store->layout().refLogKey( + EXPECT_FALSE(readOf(backend, store->layout().refLogKey( life, RefTxnId{candidate.writer_epoch, candidate.ref_sequence + 1}))) << "no later id may be allocated above an unfrontiered durable transaction"; } @@ -1594,8 +1608,8 @@ TEST(CASRefWriterAppendLane, FenceMovementAtCheckpointPublicationRequiresRecover const auto after = readCkptForTest(backend, store->layout(), life); ASSERT_TRUE(after); EXPECT_EQ(after->ckpt.committed_through, before->ckpt.committed_through); - EXPECT_TRUE(backend->get(store->layout().refLogKey(life, candidate))); - EXPECT_FALSE(backend->get(store->layout().refLogKey( + EXPECT_TRUE(readOf(backend, store->layout().refLogKey(life, candidate))); + EXPECT_FALSE(readOf(backend, store->layout().refLogKey( life, RefTxnId{candidate.writer_epoch, candidate.ref_sequence + 1}))); } @@ -1906,13 +1920,16 @@ TEST(CASRefWriterAppendLane, I1AppendCorruptionSurfacesAndFencesTheMountForRemou /// substitute anymore: immutable runtimes retain the generation that admitted them and cannot be /// rebound to the new one. const String mount_key = layout.mountKey("test"); - const auto mount = backend->get(mount_key); + const auto mount = readOf(backend, mount_key); ASSERT_TRUE(mount); MountLease fenced_mount = decodeMountLease(mount->bytes); fenced_mount.gc_fenced = true; fenced_mount.seq += 1; - ASSERT_EQ(backend->putOverwrite(mount_key, encodeMountLease(fenced_mount), mount->token).outcome, - PutOutcome::Done); + { + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.replace(mount_key, encodeMountLease(fenced_mount), mount->etag, Retry::standard()))); + } ASSERT_TRUE(store->tryRemountOnce()); auto same = std::async(std::launch::async, [&] { store->dropRef(ns, "x"); }); @@ -1959,7 +1976,7 @@ TEST(CASRefWriterAppendLane, I1WedgeResolveCorruptionSurfacesAndFaultsLane) /// observes the mismatch and must raise `CORRUPTED_DATA` to that caller while faulting the lane. const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(wedged_key.empty()); - ASSERT_EQ(backend->putIfAbsent(wedged_key, "a-different-object").outcome, PutOutcome::Done); + ASSERT_TRUE(createRaw(backend, wedged_key, "a-different-object")); auto fut = std::async(std::launch::async, [&] { @@ -2025,7 +2042,7 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) /// Out-of-band, a foreign writer lands DIFFERENT bytes at the exact wedged key. const String wedged_key = store->wedgedKeyForTest(ns); ASSERT_FALSE(wedged_key.empty()); - ASSERT_EQ(backend->putIfAbsent(wedged_key, "a-different-object").outcome, PutOutcome::Done); + ASSERT_TRUE(createRaw(backend, wedged_key, "a-different-object")); /// The next append's wedge resolve observes the mismatch: CORRUPTED_DATA, the fence trips closed, /// and a ForeignInterference event is audited. @@ -2040,7 +2057,7 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) /// `scheduleRemount`'s own entry regardless of `background_watermark` -- see that accessor's /// comment for why this test deliberately does NOT enable `background_watermark` to observe a real /// automatic recovery: doing so was tried and makes the store's self-remount attempt race its own - /// still-live keeper for 30+ seconds per call (confirmed while building this test), which is not + /// still-live renewer for 30+ seconds per call (confirmed while building this test), which is not /// something a fast unit test should be driving. EXPECT_EQ(store->scheduleRemountCallCountForTest(), 1u) << "reportImpossibleInterference must have called scheduleRemount exactly once"; @@ -2078,8 +2095,8 @@ TEST(CASAnomalyPolicy, NonReadyAtNewIdAllocationFaultsAndFailsClosed) String cursor; for (;;) { - const KeyPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); - for (const KeyEntry & lk : page.keys) + const ListPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (parsed && parsed->life_id == DB::Cas::tests::fixture::fixtureLife(ns).incarnation && parsed->kind == RefObjectKind::Log) @@ -2108,7 +2125,7 @@ TEST(CASAnomalyPolicy, NonReadyAtNewIdAllocationFaultsAndFailsClosed) EXPECT_FALSE(store->mayMutate()) << "the local write fence must trip closed on the wedge-contract violation"; /// See the sibling test's comment on why this checks the call-count seam (never /// `background_watermark` plus automatic recovery -- that combination makes the store's self-remount - /// race its own still-live keeper). + /// race its own still-live renewer). EXPECT_EQ(store->scheduleRemountCallCountForTest(), 1u) << "reportImpossibleInterference must have called scheduleRemount exactly once"; @@ -2175,7 +2192,7 @@ TEST(CASRefWriteContract, CommittedSurfacesTheIncarnationFromTheWriteAndFromTheS CasOperation reader = requests.admit(); const auto observed = reader.head("k1", Retry::standard()); ASSERT_TRUE(observed.has_value()); - EXPECT_EQ(direct_committed->incarnation, observed->incarnation) + EXPECT_EQ(direct_committed->etag, observed->etag) << "the direct-commit incarnation is the write's own response"; /// The write of `k2` lands and its response is lost: the settling read proves the commit, and the @@ -2189,7 +2206,7 @@ TEST(CASRefWriteContract, CommittedSurfacesTheIncarnationFromTheWriteAndFromTheS CasOperation second_reader = requests.admit(); const auto second_observed = second_reader.head("k2", Retry::standard()); ASSERT_TRUE(second_observed.has_value()); - EXPECT_EQ(resolved_committed->incarnation, second_observed->incarnation) + EXPECT_EQ(resolved_committed->etag, second_observed->etag) << "the resolve-commit incarnation is the observed one"; } @@ -2299,7 +2316,7 @@ TEST(CASRefWriterSnapshotPublish, ThresholdTriggerPublishesCacheReplayEquivalent EXPECT_TRUE(store->newestPublishedSnapshotIdForTest(ns) == snap_id); EXPECT_EQ(store->tailSinceSnapshotCountForTest(ns), 0u) << "a snapshot covering everything prunes the whole tail"; - const auto got = backend->get(layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); + const auto got = readOf(backend, layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); ASSERT_TRUE(got.has_value()); /// The independent oracle: replay every `_log/` object directly, ignoring the snapshot entirely. @@ -2374,7 +2391,7 @@ TEST(CASRefWriterSnapshotPublish, CapturedPredecessorCannotPublishAfterSameNameR const uint64_t successor_runtime = store->refTableRuntimeIdentityForTest(ns); const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, predecessor.incarnation); - const auto predecessor_ckpt_before_resume = backend->get(layout.refCkptKey(predecessor_life)); + const auto predecessor_ckpt_before_resume = readOf(backend, layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_before_resume) << "the removal protocol leaves this checkpoint as janitor-owned predecessor debris"; @@ -2387,11 +2404,11 @@ TEST(CASRefWriterSnapshotPublish, CapturedPredecessorCannotPublishAfterSameNameR EXPECT_FALSE(publisher.get()); store->setSnapshotAfterCaptureHookForTest(nullptr); - EXPECT_FALSE(backend->get(layout.refSnapshotKey(predecessor_life, *predecessor_snapshot))) + EXPECT_FALSE(readOf(backend, layout.refSnapshotKey(predecessor_life, *predecessor_snapshot))) << "a stale publisher recreated the retired predecessor snapshot"; - const auto predecessor_ckpt_after_resume = backend->get(layout.refCkptKey(predecessor_life)); + const auto predecessor_ckpt_after_resume = readOf(backend, layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_after_resume); - EXPECT_EQ(predecessor_ckpt_after_resume->token, predecessor_ckpt_before_resume->token) + EXPECT_EQ(predecessor_ckpt_after_resume->etag, predecessor_ckpt_before_resume->etag) << "a stale publisher replaced the retired predecessor checkpoint"; EXPECT_EQ(predecessor_ckpt_after_resume->bytes, predecessor_ckpt_before_resume->bytes) << "a stale publisher changed the retired predecessor checkpoint"; @@ -2451,7 +2468,7 @@ TEST(CASRefWriterSnapshotPublish, RetiredPredecessorCannotAdvanceCkptAfterSnapsh std::unique_lock lock(mutex); ASSERT_TRUE(cv.wait_for(lock, std::chrono::seconds(10), [&] { return before_ckpt_cas; })); } - EXPECT_TRUE(backend->get(layout.refSnapshotKey(predecessor_life, *candidate_id))) + EXPECT_TRUE(readOf(backend, layout.refSnapshotKey(predecessor_life, *candidate_id))) << "the hook must run after the snapshot body PUT and immediately before `_ckpt` admission"; EXPECT_NO_THROW(store->dropNamespace(ns)); @@ -2465,7 +2482,7 @@ TEST(CASRefWriterSnapshotPublish, RetiredPredecessorCannotAdvanceCkptAfterSnapsh const uint64_t successor_runtime = store->refTableRuntimeIdentityForTest(ns); const CatalogEntry successor = catalogEntryOrThrow(backend, layout, ns); EXPECT_NE(successor.incarnation, predecessor.incarnation); - const auto predecessor_ckpt_before_resume = backend->get(layout.refCkptKey(predecessor_life)); + const auto predecessor_ckpt_before_resume = readOf(backend, layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_before_resume); { @@ -2477,9 +2494,9 @@ TEST(CASRefWriterSnapshotPublish, RetiredPredecessorCannotAdvanceCkptAfterSnapsh EXPECT_FALSE(publisher.get()); store->setSnapshotBeforeCkptCasHookForTest(nullptr); - const auto predecessor_ckpt_after_resume = backend->get(layout.refCkptKey(predecessor_life)); + const auto predecessor_ckpt_after_resume = readOf(backend, layout.refCkptKey(predecessor_life)); ASSERT_TRUE(predecessor_ckpt_after_resume); - EXPECT_EQ(predecessor_ckpt_after_resume->token, predecessor_ckpt_before_resume->token); + EXPECT_EQ(predecessor_ckpt_after_resume->etag, predecessor_ckpt_before_resume->etag); EXPECT_EQ(predecessor_ckpt_after_resume->bytes, predecessor_ckpt_before_resume->bytes); EXPECT_EQ(store->refTableRuntimeIdentityForTest(ns), successor_runtime); EXPECT_TRUE(store->resolveRef(ns, "successor")); @@ -2744,7 +2761,7 @@ TEST(CASRefWriterSnapshotPublish, MountTimeRecoveredLargeTailPublishesAfterOrdin << "the ordinary successor must make the inherited mount-time tail publishable"; EXPECT_EQ(successor->tailSinceSnapshotCountForTest(ns), 0u); - const auto got = backend->get(layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); + const auto got = readOf(backend, layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); ASSERT_TRUE(got.has_value()); const RefTableState oracle = independentFullReplayForTest(*backend, layout, ns, snap_id); EXPECT_EQ(openObject(FormatId::RefSnapshot, got->bytes), encodeRefTableSnapshot(snapshotOf(oracle, ns.string()))); @@ -2780,7 +2797,7 @@ TEST(CASRefWriterPublishFromLive, YoungTxnIsCoveredImmediately) << "publish-from-live: a just-committed txn is immediately coverable, with no grace window"; const auto snap_id = listGreatestSnapshotIdForTest(*backend, layout, ns); ASSERT_TRUE(snap_id.has_value()); - const auto got = backend->get(layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); + const auto got = readOf(backend, layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); ASSERT_TRUE(got.has_value()); const RefTableSnapshot snap = decodeRefTableSnapshot(openObject(FormatId::RefSnapshot, got->bytes), ns.string(), *snap_id); ASSERT_EQ(snap.committed.size(), 1u); @@ -2963,7 +2980,7 @@ TEST(CASRefWriterSnapshotPublish, ConcurrentOutOfOrderPublishDoesNotRegressBaseN ASSERT_TRUE(store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns)); const auto snap_id = listGreatestSnapshotIdForTest(*backend, layout, ns); ASSERT_TRUE(snap_id.has_value()); - const auto got = backend->get(layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); + const auto got = readOf(backend, layout.refSnapshotKey(DB::Cas::tests::fixture::fixtureLife(ns), *snap_id)); ASSERT_TRUE(got.has_value()); const RefTableState oracle = independentFullReplayForTest(*backend, layout, ns, snap_id); EXPECT_EQ(openObject(FormatId::RefSnapshot, got->bytes), encodeRefTableSnapshot(snapshotOf(oracle, ns.string()))) @@ -3110,13 +3127,13 @@ TEST(CASRefWriterSnapshotPublish, RecoveredSealAboveThresholdDoesNotRedispatchUn auto predecessor = openPoolWithConfig(backend, predecessor_config); DB::Cas::tests::fixture::admitLive(*backend, predecessor->layout(), ns); const NamespaceLifeId life = *lifeIfCatalogedForTest(backend, predecessor->layout(), ns); - ASSERT_EQ(backend->putIfAbsent(predecessor->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, predecessor->layout().refCkptKey(life), encodeRefCkpt(RefCkpt{ .life_epoch = predecessor->liveWriterEpoch(), .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); publishEmptyPart(predecessor, ns, "before_seal"); - const auto before = backend->get(predecessor->layout().refCkptKey(life)); + const auto before = readOf(backend, predecessor->layout().refCkptKey(life)); ASSERT_TRUE(before); const RefCkpt before_seal = decodeRefCkpt(before->bytes); ASSERT_TRUE(before_seal.committed_through); @@ -3127,8 +3144,10 @@ TEST(CASRefWriterSnapshotPublish, RecoveredSealAboveThresholdDoesNotRedispatchUn RefCkpt recovered_seal = before_seal; recovered_seal.committed_through = seal_id; recovered_seal.last_epoch_seal = seal_id; - ASSERT_EQ(backend->casPut(predecessor->layout().refCkptKey(life), encodeRefCkpt(recovered_seal), before->token).outcome, - CasOutcome::Committed); + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.replace( + predecessor->layout().refCkptKey(life), encodeRefCkpt(recovered_seal), before->etag, Retry::standard()))); } PoolConfig successor_config; @@ -3450,8 +3469,8 @@ TEST(CASRefWriterStalePrecommitSweep, BoundedBatchesAndInterruptionResumeAcrossM String cursor; for (;;) { - const KeyPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); - for (const KeyEntry & lk : page.keys) + const ListPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (parsed && parsed->life_id == DB::Cas::tests::fixture::fixtureLife(ns).incarnation @@ -3629,21 +3648,23 @@ namespace /// gtest_cas_pool.cpp's fenceOutMount, without its ASSERT_ macros so it can run outside a fixture). void fenceOutRefMount(Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, Retry::standard()); MountLease m = decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - backend.putOverwrite(mount_key, encodeMountLease(m), got->token); + (void)(*op).replace(mount_key, encodeMountLease(m), got->etag, Retry::standard()); } /// The greatest `_log/` transaction id currently present for `ns` (independent of any Pool cache). std::optional listGreatestLogIdForTest(Backend & backend, const Layout & layout, const RootNamespace & ns) { + DB::Cas::tests::OperationForTest op(backend); std::optional greatest; String cursor; for (;;) { - const ListPage page = backend.list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*op).list(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -3672,7 +3693,7 @@ uint64_t seedTwinDrop(const BackendPtr & backend, const Layout & layout, const R { uint64_t greatest_in_previous_epoch = 0; uint64_t previous_epoch = 0; - for (const KeyEntry & lk : listForTest( + for (const ListedKey & lk : listForTest( backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), "", 1000).keys) { const auto parsed = layout.parseRefObjectKey(lk.key); @@ -3721,13 +3742,16 @@ TEST(CASRefWriterRemount, FailedRemountPublishesNoRuntimeUnderFenceLossGeneratio ASSERT_EQ(predecessor_generation, store->fenceGeneration()); const String mount_key = layout.mountKey("test"); - const auto got = backend->get(mount_key); + const auto got = readOf(backend, mount_key); ASSERT_TRUE(got); MountLease foreign = decodeMountLease(got->bytes); foreign.server_uuid = foreign.server_uuid + UInt128{1}; foreign.seq += 1; - ASSERT_EQ(backend->putOverwrite(mount_key, encodeMountLease(foreign), got->token).outcome, - PutOutcome::Done); + { + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.replace(mount_key, encodeMountLease(foreign), got->etag, Retry::standard()))); + } store->tripMountLost(); const uint64_t rejected_generation = store->fenceGeneration(); @@ -4040,8 +4064,8 @@ TEST(CASRefWriterNamespaceRemoval, TxnNamesEveryOwnerThenRemoveNamespace) String cursor; for (;;) { - const KeyPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); - for (const KeyEntry & lk : page.keys) + const ListPage page = listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + for (const ListedKey & lk : page.keys) { const auto parsed = layout.parseRefObjectKey(lk.key); if (parsed && parsed->life_id == DB::Cas::tests::fixture::fixtureLife(ns).incarnation && parsed->kind == RefObjectKind::Log @@ -4054,7 +4078,7 @@ TEST(CASRefWriterNamespaceRemoval, TxnNamesEveryOwnerThenRemoveNamespace) } } ASSERT_TRUE(newest_log.has_value()); - const auto got = backend->get(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), *newest_log)); + const auto got = readOf(backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), *newest_log)); ASSERT_TRUE(got.has_value()); const RefLogTxn removal_txn = decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), *newest_log); @@ -4091,12 +4115,12 @@ TEST(CASRefWriterNamespaceRemoval, RemovalPublishesTerminalLogWithoutTerminalSna << "the terminal transaction remains ordinary immutable stream work until GC folds it"; size_t terminal_logs = 0; - for (const KeyEntry & listed : listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), "", 1000).keys) + for (const ListedKey & listed : listForTest(backend, layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), "", 1000).keys) { const auto parsed = layout.parseRefObjectKey(listed.key); if (!parsed || parsed->kind != RefObjectKind::Log) continue; - const auto got = backend->get(listed.key); + const auto got = readOf(backend, listed.key); ASSERT_TRUE(got.has_value()); const RefLogTxn txn = decodeRefLogTxn(openObject(FormatId::RefLog, got->bytes), ns.string(), parsed->txn_id); if (!txn.ops.empty() && txn.ops.back().kind == RefOpKind::RemoveNamespace) @@ -4218,7 +4242,7 @@ TEST(CASRefWriterNamespaceRemoval, GenericTerminalOnAbsentNamePerformsZeroDurabl EXPECT_EQ(backend->writeTotal(), 0u); EXPECT_EQ(backend->deleteTotal(), 0u); const CasRefCatalog::Snapshot catalog_after = readCatalogForTest(backend, store->layout()); - EXPECT_EQ(catalog_after.incarnation, catalog_before.incarnation); + EXPECT_EQ(catalog_after.etag, catalog_before.etag); EXPECT_EQ(catalog_after.catalog, catalog_before.catalog); EXPECT_FALSE(store->refTableLifeForTest(ns)); } @@ -4242,11 +4266,11 @@ TEST(CASRefWriterNamespaceRemoval, CatalogedNamespaceFilesOnlyLifeCompletesRemov EXPECT_NO_THROW(store->dropNamespace(ns)); ASSERT_EQ(catalogEntryOrThrow(backend, layout, ns).state, NsState::Removing); - const KeyPage terminal_page = listForTest(backend, layout.namespaceStreamPrefix(life), "", 100); + const ListPage terminal_page = listForTest(backend, layout.namespaceStreamPrefix(life), "", 100); ASSERT_EQ(terminal_page.keys.size(), 1u); const auto parsed = layout.parseRefObjectKey(terminal_page.keys.front().key); ASSERT_TRUE(parsed); - const auto terminal_body = backend->get(terminal_page.keys.front().key); + const auto terminal_body = readOf(backend, terminal_page.keys.front().key); ASSERT_TRUE(terminal_body); const RefLogTxn terminal = decodeRefLogTxn( openObject(FormatId::RefLog, terminal_body->bytes), ns.string(), parsed->txn_id); @@ -4421,7 +4445,7 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeCreatingIsPresentAndRemovalWaits dead.gc_fenced = true; dead.seq = 1; dead.write_attempt_id = UInt128{1}; - backend->putIfAbsent(layout.mountKey("srv1"), encodeMountLease(dead)); + createRaw(backend, layout.mountKey("srv1"), encodeMountLease(dead)); EXPECT_NO_THROW(store->dropNamespace(ns)); EXPECT_FALSE(store->namespaceStillLogicallyPresent(ns)); @@ -4578,12 +4602,16 @@ TEST(CASRefWriterNamespaceRemoval, PresenceProbeFenceLossPropagatesRatherThanAns publishEmptyPart(store, ns, "x"); const String mount_key = store->layout().mountKey("test"); - const auto got = backend->get(mount_key); + const auto got = readOf(backend, mount_key); ASSERT_TRUE(got); MountLease foreign = decodeMountLease(got->bytes); foreign.server_uuid = foreign.server_uuid + UInt128{1}; foreign.seq += 1; - ASSERT_EQ(backend->putOverwrite(mount_key, encodeMountLease(foreign), got->token).outcome, PutOutcome::Done); + { + CasRequests requests(backend, Fence::open()); + CasOperation op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.replace(mount_key, encodeMountLease(foreign), got->etag, Retry::standard()))); + } store->tripMountLost(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)store->namespaceStillLogicallyPresent(ns); }); @@ -4878,8 +4906,8 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi Gc gc(store, gc_id); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); - GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); - CasFoldSeal seal = decodeFoldSeal(backend->get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + GcState state = decodeGcState(readOf(backend, layout.gcStateKey())->bytes); + CasFoldSeal seal = decodeFoldSeal(readOf(backend, layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); const auto predecessor_row = seal.ref_lives.find(predecessor.incarnation); ASSERT_NE(predecessor_row, seal.ref_lives.end()); ASSERT_NE(predecessor_row->second.coverage.last_folded_ref_id, RefTxnId{}); @@ -4889,7 +4917,7 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi ASSERT_GT(backend->putTotal(), puts_before_drop) << "control: the real removal call returned after durably writing its terminal artifacts"; std::optional terminal_id; - for (const KeyEntry & listed : listForTest(backend, + for (const ListedKey & listed : listForTest(backend, layout.namespaceStreamPrefix(NamespaceLifeId::fromCatalogEntry(ns, predecessor.incarnation)), "", 1000).keys) { @@ -4899,7 +4927,7 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi terminal_id = parsed->txn_id; } ASSERT_TRUE(terminal_id.has_value()); - const auto terminal_body = backend->get(layout.refLogKey( + const auto terminal_body = readOf(backend, layout.refLogKey( NamespaceLifeId::fromCatalogEntry(ns, predecessor.incarnation), *terminal_id)); ASSERT_TRUE(terminal_body.has_value()); const RefLogTxn terminal = decodeRefLogTxn( @@ -4916,8 +4944,8 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi << "the removal path must invalidate the resident runtime's life, not pass through eviction"; ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred) << "the terminal delta must fold"; - state = decodeGcState(backend->get(layout.gcStateKey())->bytes); - seal = decodeFoldSeal(backend->get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + state = decodeGcState(readOf(backend, layout.gcStateKey())->bytes); + seal = decodeFoldSeal(readOf(backend, layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); ASSERT_TRUE(seal.ref_lives.at(predecessor.incarnation).cleanup_evidence.has_value()); const RoundReport drain = runRegularRoundReclaiming(gc); @@ -4936,7 +4964,7 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi ASSERT_EQ(store->refTableLifeForTest(ns)->incarnation, successor.incarnation); const NamespaceLifeId successor_life = NamespaceLifeId::fromCatalogEntry(ns, successor.incarnation); - const KeyPage successor_stream = listForTest(backend, layout.namespaceStreamPrefix(successor_life), "", 1000); + const ListPage successor_stream = listForTest(backend, layout.namespaceStreamPrefix(successor_life), "", 1000); ASSERT_FALSE(successor_stream.keys.empty()) << "the real successor writer produced foldable stream work"; std::vector successor_phases; gc.setPhaseSink([&](const GcPhaseRecord & phase) { successor_phases.push_back(phase); }); @@ -4950,8 +4978,8 @@ TEST(CASRefWriterNamespaceRemoval, SameNameSameWriterEpochRebirthInvalidatesResi << "successor stream keys=" << successor_stream.keys.size() << ", changed_shards=" << decision->metrics.at("changed_shards") << ", dead_life_debris=" << decision->metrics.at("dead_life_debris"); - state = decodeGcState(backend->get(layout.gcStateKey())->bytes); - seal = decodeFoldSeal(backend->get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + state = decodeGcState(readOf(backend, layout.gcStateKey())->bytes); + seal = decodeFoldSeal(readOf(backend, layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); EXPECT_FALSE(seal.ref_lives.contains(predecessor.incarnation)); const auto successor_row = seal.ref_lives.find(successor.incarnation); ASSERT_NE(successor_row, seal.ref_lives.end()); @@ -4995,9 +5023,9 @@ TEST(CASRefWriterNamespaceRemoval, CommitThenThrowEraseResolvesAndRebindsResiden EXPECT_NE(store->refTableRuntimeIdentityForTest(ns), ready.runtime_identity); ASSERT_FALSE(runRegularRoundReclaiming(gc).deferred); - const GcState state = decodeGcState(backend->get(layout.gcStateKey())->bytes); + const GcState state = decodeGcState(readOf(backend, layout.gcStateKey())->bytes); const CasFoldSeal seal = decodeFoldSeal( - backend->get(layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); + readOf(backend, layout.foldSealKey(state.snap_generation, state.snap_attempt))->bytes); EXPECT_FALSE(seal.ref_lives.contains(ready.predecessor.incarnation)); ASSERT_TRUE(seal.ref_lives.contains(successor.incarnation)); EXPECT_EQ(seal.ref_lives.at(successor.incarnation).coverage.last_folded_ref_id, @@ -5025,10 +5053,10 @@ TEST(CASRefWriterNamespaceRemoval, OtherWinnerReplacementInvalidatesExactPredece .creator = std::nullopt}; ASSERT_NE(replacement.incarnation, ready.predecessor.incarnation); const NamespaceLifeId replacement_life = NamespaceLifeId::fromCatalogEntry(ns, replacement.incarnation); - ASSERT_EQ(backend->putIfAbsent(layout.refCkptKey(replacement_life), encodeRefCkpt(RefCkpt{ + ASSERT_TRUE(createRaw(backend, layout.refCkptKey(replacement_life), encodeRefCkpt(RefCkpt{ .life_epoch = ready.writer_epoch, .checkpoint_snapshot_id = std::nullopt, - .last_epoch_seal = std::nullopt})).outcome, PutOutcome::Done); + .last_epoch_seal = std::nullopt}))); backend->catalog_fault_key = layout.refCatalogKey(); backend->catalog_replacement_bytes = encodeRefCatalog(RefCatalog{.entries = {replacement}}); backend->catalog_cas_fault = RefWriterTestBackend::CatalogCasFault::OtherWriterReplacement; @@ -5261,9 +5289,7 @@ void seedUncleanPredecessorMount(const BackendPtr & backend, const Layout & layo /// lease TTL) -- mirrors `CASMountOpenWaits.FencedPriorReclaimsWithoutAnyWait` exactly. CasRequestBudget sealTestTinyBudget() { - return CasRequestBudget{ - .attempt_timeout_ms = 50, .operation_deadline_ms = kSingleAttemptDeadlineMs, - .lease_safety_margin_ms = 50}; + return CasRequestBudget{.attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; } } diff --git a/src/Disks/tests/gtest_cas_request_control.cpp b/src/Disks/tests/gtest_cas_request_control.cpp deleted file mode 100644 index 69b6da4c2c96..000000000000 --- a/src/Disks/tests/gtest_cas_request_control.cpp +++ /dev/null @@ -1,1498 +0,0 @@ -#include - -#include "config.h" - -#include -#include - -#include - -#if USE_AWS_S3 -#include -#include -#include -#include -#include -#endif - -using namespace DB::Cas; - -namespace DB::ErrorCodes -{ - extern const int NETWORK_ERROR; - extern const int ABORTED; - extern const int CAS_WRITE_UNATTRIBUTED; -} - -namespace ProfileEvents -{ - extern const Event CASConditionalWriteAttempts; - extern const Event CASConditionalWriteCommitted; - extern const Event CASConditionalWriteDefiniteFailure; - extern const Event CASConditionalWriteUnresolved; -} - -#if USE_AWS_S3 -namespace DB::ErrorCodes -{ - extern const int CORRUPTED_DATA; - extern const int BAD_ARGUMENTS; - extern const int LOGICAL_ERROR; - extern const int UNKNOWN_EXCEPTION; -} -#endif - -/// The success path (buf.finalize() returned without throwing) is always Committed. No exception -/// object is needed — the caller distinguishes success from failure before calling either overload. -TEST(CASRequestControl, SuccessIsAlwaysCommitted) -{ - EXPECT_EQ(classifyConditionalWriteResult(), CasWriteOutcome::Committed); -} - -/// Fix #37 phase 2: the retry-later throw must be NETWORK_ERROR, never ABORTED -- ABORTED is silently -/// swallowed by ReplicatedMergeMutateTaskBase (no backoff, no last_exception), which is exactly the -/// defect this fix closes. -TEST(CASWriteRetryLater, ThrowsNetworkErrorNotAborted) -{ - bool threw = false; - try - { - throwCasWriteRetryLater("test cause"); - FAIL() << "throwCasWriteRetryLater must always throw"; - } - catch (const DB::Exception & e) - { - threw = true; - EXPECT_EQ(e.code(), DB::ErrorCodes::NETWORK_ERROR); - EXPECT_NE(e.code(), DB::ErrorCodes::ABORTED); - EXPECT_NE(e.message().find("test cause"), String::npos) << e.message(); - EXPECT_NE(e.message().find("retrying later"), String::npos) << e.message(); - } - EXPECT_TRUE(threw); -} - -/// The exception_ptr twin (for call sites that fail a pending future/promise rather than throw -/// directly, e.g. CasRefLedger's queued-append completion paths) must carry the SAME classification. -TEST(CASWriteRetryLater, ExceptionPtrVariantCarriesSameClassification) -{ - const std::exception_ptr eptr = makeCasWriteRetryLaterExceptionPtr("another cause"); - bool threw = false; - try - { - std::rethrow_exception(eptr); - FAIL() << "expected a thrown exception"; - } - catch (const DB::Exception & e) - { - threw = true; - EXPECT_EQ(e.code(), DB::ErrorCodes::NETWORK_ERROR); - EXPECT_NE(e.message().find("another cause"), String::npos) << e.message(); - } - EXPECT_TRUE(threw); -} - -#if USE_AWS_S3 - -/// One row per RFC cas-s3-timeout-retry-control §operation-classes classification. PreconditionFailed -/// is NEVER DefiniteFailure — it means the key exists, not that the request was rejected — and every -/// unrecognized/ambiguous error also falls to Unresolved, never to a false DefiniteFailure. -TEST(CASRequestControl, ClassifiesPreconditionFailedAsUnresolved) -{ - DB::S3Exception e("412 from backend", Aws::S3::S3Errors::UNKNOWN, "PreconditionFailed"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::Unresolved); -} - -TEST(CASRequestControl, ClassifiesTimeoutAsUnresolved) -{ - Poco::TimeoutException e("simulated client-side receive timeout"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::Unresolved); -} - -TEST(CASRequestControl, ClassifiesConnectionResetAsUnresolved) -{ - Poco::Net::ConnectionResetException e("simulated connection reset"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::Unresolved); -} - -TEST(CASRequestControl, Classifies5xxAsUnresolved) -{ - DB::S3Exception e("simulated internal error", Aws::S3::S3Errors::INTERNAL_FAILURE, "InternalError"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::Unresolved); - /// SlowDown / ServiceUnavailable are also 5xx-class and equally Unresolved. - DB::S3Exception slow_down("simulated throttle", Aws::S3::S3Errors::SLOW_DOWN, "SlowDown"); - EXPECT_EQ(classifyConditionalWriteResult(slow_down), CasWriteOutcome::Unresolved); -} - -TEST(CASRequestControl, ClassifiesMalformedRequestAsDefiniteFailure) -{ - DB::S3Exception e("bad xml", Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::DefiniteFailure); - /// The modeled-enum path (no canonical name attached) must classify identically. - DB::S3Exception by_code("bad argument", Aws::S3::S3Errors::INVALID_REQUEST); - EXPECT_EQ(classifyConditionalWriteResult(by_code), CasWriteOutcome::DefiniteFailure); -} - -TEST(CASRequestControl, ClassifiesEntityTooLargeAsDefiniteFailure) -{ - DB::S3Exception e("body exceeds the maximum object size", Aws::S3::S3Errors::UNKNOWN, "EntityTooLarge"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::DefiniteFailure); -} - -TEST(CASRequestControl, ClassifiesAccessDeniedAsDefiniteFailure) -{ - DB::S3Exception e("simulated 403", Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied"); - EXPECT_EQ(classifyConditionalWriteResult(e), CasWriteOutcome::DefiniteFailure); - /// The modeled-enum path (no canonical name attached) must classify identically. - DB::S3Exception by_code("simulated 403, no name", Aws::S3::S3Errors::ACCESS_DENIED); - EXPECT_EQ(classifyConditionalWriteResult(by_code), CasWriteOutcome::DefiniteFailure); -} - -/// Anything the classifier does not recognize (an unmodeled/unnamed S3 error, or an entirely -/// unrelated exception type) must fail toward Unresolved — never toward a false DefiniteFailure or a -/// false Committed (RFC §resolve-before-reissuing: ambiguity always resolves toward "resolve before -/// reissuing"). -TEST(CASRequestControl, UnrecognizedErrorsFailSafeToUnresolved) -{ - DB::S3Exception unknown_named("weird service error", Aws::S3::S3Errors::UNKNOWN, "SomeFutureErrorCode"); - EXPECT_EQ(classifyConditionalWriteResult(unknown_named), CasWriteOutcome::Unresolved); - - /// UNKNOWN_EXCEPTION (not LOGICAL_ERROR): any arbitrary non-S3 exception type works here -- the - /// point is that the classifier doesn't recognize it, not which specific code it carries. - /// LOGICAL_ERROR would abort the whole process under debug/sanitizer builds merely by being - /// constructed (Exception's constructor calls handle_error_code unconditionally). - DB::Exception unrelated(DB::ErrorCodes::UNKNOWN_EXCEPTION, "not an S3 error at all"); - EXPECT_EQ(classifyConditionalWriteResult(unrelated), CasWriteOutcome::Unresolved); -} - -/// recordConditionalWriteAttemptStarted / recordConditionalWriteOutcome bump the per-class counters -/// (RFC §observability): attempts, and exactly one of Committed/DefiniteFailure/Unresolved per call. -TEST(CASRequestControl, CountersHookupIncrementsPerClass) -{ - using ProfileEvents::global_counters; - const auto attempts_before = global_counters[ProfileEvents::CASConditionalWriteAttempts].load(); - const auto committed_before = global_counters[ProfileEvents::CASConditionalWriteCommitted].load(); - const auto definite_before = global_counters[ProfileEvents::CASConditionalWriteDefiniteFailure].load(); - const auto unresolved_before = global_counters[ProfileEvents::CASConditionalWriteUnresolved].load(); - - recordConditionalWriteAttemptStarted(); - recordConditionalWriteOutcome(CasWriteOutcome::Committed); - recordConditionalWriteAttemptStarted(); - recordConditionalWriteOutcome(CasWriteOutcome::DefiniteFailure); - recordConditionalWriteAttemptStarted(); - recordConditionalWriteOutcome(CasWriteOutcome::Unresolved); - -#if !WITH_COVERAGE - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteAttempts].load() - attempts_before, 3u); - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteCommitted].load() - committed_before, 1u); - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteDefiniteFailure].load() - definite_before, 1u); - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteUnresolved].load() - unresolved_before, 1u); -#else - (void)attempts_before; (void)committed_before; (void)definite_before; (void)unresolved_before; -#endif -} - -/// Wiring smoke test: a real conditional write through ObjectStorageBackend (Native mode) counts one -/// attempt and one Committed outcome via the SAME instrumented call site nativeConditionalPut uses — -/// see finalizeConditionalWriteInstrumented in CasObjectStorageBackend.cpp. The write itself lands -/// and is counted at that site; a local object storage then returns no incarnation for it, so the -/// call refuses to attribute the write rather than reading one back. The counters are what this pins. -TEST(CASRequestControl, NativeConditionalPutCountsOneAttemptAndCommitted) -{ - using ProfileEvents::global_counters; - const auto attempts_before = global_counters[ProfileEvents::CASConditionalWriteAttempts].load(); - const auto committed_before = global_counters[ProfileEvents::CASConditionalWriteCommitted].load(); - - auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); - auto b = std::make_shared(storage, ObjectStorageBackend::Mode::Native); - const String key = DB::Cas::tests::nativeKeyUnder(storage, "p/rc/one"); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CAS_WRITE_UNATTRIBUTED, [&] { b->putIfAbsent(key, "v1"); }); - -#if !WITH_COVERAGE - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteAttempts].load() - attempts_before, 1u); - EXPECT_EQ(global_counters[ProfileEvents::CASConditionalWriteCommitted].load() - committed_before, 1u); -#else - (void)attempts_before; (void)committed_before; -#endif -} - -/// Mechanism property (RFC §disable-transparent-conditional-write-retries), tested at the layer -/// actually reachable from a unit-test binary: NO live/fake S3 endpoint is available here (the Native -/// conditional-write path is exercised end-to-end only at M-W against RustFS — see the HONEST NOTE in -/// CasObjectStorageBackend.cpp), so driving a real socket-level retry against a real client is not -/// reachable from this binary. What IS reachable and asserted here: every Native conditional write -/// selects the SingleAttempt object-storage retry profile, and a non-S3 backend such as -/// LocalObjectStorage reports it as UNSUPPORTED via IObjectStorage::supportsRetryProfile — the property -/// checkConditionalWriteSingleAttemptSupport's fail-closed mount-time gate relies on. -TEST(CASRequestControl, SingleAttemptProfileRequestedAndLocalBackendRejected) -{ - auto b = std::make_shared( - DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - const auto ws = b->conditionalWriteSettingsForTest(); - EXPECT_EQ(ws.object_storage_retry_profile, DB::ObjectStorageRetryProfile::SingleAttempt); - /// LocalObjectStorage does not implement the profile — the capability check must say no. - EXPECT_FALSE(DB::Cas::tests::makeLocalObjectStorageForTest()->supportsRetryProfile(DB::ObjectStorageRetryProfile::SingleAttempt)); -} - -/// The SECOND retry-affecting layer above the S3 client (review finding): WriteBufferFromS3's OWN -/// makeSinglepartUpload/completeMultipartUpload retry loop reissues the identical conditional request -/// on a NO_SUCH_KEY response, driven by S3RequestSetting::max_unexpected_write_error_retries (default -/// 4) — a client-level override alone does not bound it (see WriteSettings:: -/// s3_max_unexpected_write_error_retries_override). Asserted at the reachable seam: no live/fake S3 -/// endpoint exists in this binary to drive the retry loop itself, so this proves the settings -/// plumbing conditionalWriteSettings() -> WriteSettings produces the override value that -/// S3ObjectStorage::writeObject then applies to request_settings — NOT a real single-attempt -/// assertion against a live wire attempt. -TEST(CASRequestControl, ConditionalWriteSettingsForceSingleUnexpectedWriteErrorRetry) -{ - auto b = std::make_shared( - DB::Cas::tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); - const auto ws = b->conditionalWriteSettingsForTest(); - EXPECT_EQ(ws.s3_max_unexpected_write_error_retries_override, 1u); -} - -/// ================================================================================================ -/// Task 5: CasRequestController — retry controller (deadlines, fence gating, exact-key resolution) -/// ================================================================================================ - -namespace -{ - -/// A per-call scripted Backend for CasRequestController tests: `putIfAbsent` optionally throws a -/// caller-supplied exception (models one classified HTTP-attempt outcome) or returns a forced -/// `PutOutcome` directly (models a `PreconditionFailed` observed WITHOUT an exception); with neither -/// set it delegates to the real in-memory conditional-write semantics. `get` optionally returns a -/// forced result, independent of what `putIfAbsent` actually did, so a test can drive exact-key -/// resolution (identical / different / absent) without the scripted put and the resolve GET needing to -/// agree on a shared, real backing store. -class ScriptedControllerBackend : public InMemoryBackend -{ -public: - std::function put_thrower; - std::optional put_forced_outcome; - std::atomic put_attempts{0}; - - std::function put_overwrite_thrower; - std::function put_overwrite_handler; - std::optional put_overwrite_forced_outcome; - std::atomic put_overwrite_attempts{0}; - - std::function(const String &, Range)> get_handler; - std::atomic get_attempts{0}; - bool get_overridden = false; - std::optional get_override_value; /// meaningful only when get_overridden - - void setGetOverride(std::optional value) - { - get_overridden = true; - get_override_value = std::move(value); - } - - PutResult putIfAbsent(const String & key, const String & bytes, const ObjectMeta & meta) override - { - ++put_attempts; - if (put_thrower) - put_thrower(); - if (put_forced_outcome) - return {*put_forced_outcome, {}}; - return InMemoryBackend::putIfAbsent(key, bytes, meta); - } - - PutResult putOverwrite(const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) override - { - ++put_overwrite_attempts; - if (put_overwrite_handler) - return put_overwrite_handler(key, bytes, expected, meta); - if (put_overwrite_thrower) - put_overwrite_thrower(); - if (put_overwrite_forced_outcome) - return {*put_overwrite_forced_outcome, {}}; - return InMemoryBackend::putOverwrite(key, bytes, expected, meta); - } - - std::optional get(const String & key, Range range) override - { - ++get_attempts; - if (get_handler) - return get_handler(key, range); - if (get_overridden) - return get_override_value; - return InMemoryBackend::get(key, range); - } -}; - -GetResult resultWithBytes(const String & bytes) -{ - return GetResult{.bytes = bytes, .token = Token{"t", TokenType::Emulated}, .attributes = {}}; -} - -CasOverwriteOperationContext overwriteContext( - uint64_t absolute_deadline_ms, - CasOverwriteDeadlineSource deadline_source = CasOverwriteDeadlineSource::RequestBudget, - std::function stop_cause = {}, - std::function wait_before_retry = {}, - std::function observe = {}) -{ - return CasOverwriteOperationContext{ - .absolute_deadline_ms = absolute_deadline_ms, - .deadline_source = deadline_source, - .stop_cause = stop_cause ? std::move(stop_cause) : [] { return CasOverwriteStopCause::Continue; }, - .wait_before_retry = wait_before_retry ? std::move(wait_before_retry) : [](uint64_t) { return true; }, - .observe = observe ? std::move(observe) : [](const CasOverwriteProgress &) {}, - }; -} - -void expectOverwriteDiagnostics( - const CasOverwriteResult & result, - uint32_t attempts_sent, - bool resolved_by_get, - CasUnresolvedReason unresolved_reason, - CasOverwriteDeadlineSource deadline_source, - CasOverwriteStopCause stop_cause) -{ - EXPECT_EQ(result.diagnostics.attempts_sent, attempts_sent); - EXPECT_EQ(result.diagnostics.resolved_by_get, resolved_by_get); - EXPECT_EQ(result.diagnostics.unresolved_reason, unresolved_reason); - EXPECT_EQ(result.diagnostics.deadline_source, deadline_source); - EXPECT_EQ(result.diagnostics.stop_cause, stop_cause); -} - -} - -TEST(CASRequestController, UncertainResolvesIdenticalAsCommitted) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(resultWithBytes("payload")); - - CasRequestController controller(backend, CasRequestBudget{}); - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return true; }); - EXPECT_EQ(outcome, CasWriteOutcome::Committed); - EXPECT_EQ(backend->put_attempts.load(), 1u); -} - -TEST(CASRequestController, UncertainResolvesDifferentThrowsCorruption) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(resultWithBytes("someone-elses-bytes")); - - CasRequestController controller(backend, CasRequestBudget{}); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] - { - controller.putIfAbsentControlled("k", "payload", [] { return true; }); - }); -} - -/// GET-absent NEVER yields DefiniteFailure (spec §writer-side-linearization): the SAME (key, bytes) is -/// retried up to `max_attempts`, and only THEN does the call give up with Unresolved. -TEST(CASRequestController, UncertainResolvesAbsentRetriesSameKeyWithinBudget) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(std::nullopt); /// absent on every resolve - - CasRequestBudget budget; - budget.max_attempts = 3; - budget.retry_initial_backoff_ms = 0; /// backoff behavior is pinned by its own tests below - CasRequestController controller(backend, budget); - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return true; }); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 3u); /// every attempt targeted the SAME key/bytes -} - -/// The operation deadline — not just the attempt-count budget — cuts a retry loop short: a fake clock -/// advances by a fixed step per now_ms() call (no sleeps), and max_attempts is generous enough that only -/// the deadline check can be what stops the loop. -TEST(CASRequestController, OperationDeadlineExhaustionReturnsUnresolvedBeforeMaxAttempts) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(std::nullopt); /// absent on every resolve - - uint64_t clock = 0; - auto now_ms = [&clock]() -> uint64_t { const uint64_t t = clock; clock += 200; return t; }; - - CasRequestBudget budget; - budget.max_attempts = 10; - budget.attempt_timeout_ms = 50; - budget.operation_deadline_ms = 450; - budget.retry_initial_backoff_ms = 0; /// isolate the deadline check from the backoff's own deadline guard - CasRequestController controller(backend, budget, now_ms); - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return true; }); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 2u); /// cut off well before the 10-attempt budget -} - -/// WHY `attempt_timeout_ms == operation_deadline_ms` IS REJECTED AT STARTUP, demonstrated on the -/// mechanism itself before the rejection is asserted below. -/// -/// The deadline is captured as `now + operation_deadline_ms` and the pre-send gate asks -/// `now + attempt_timeout_ms > deadline`. Equal values collapse that to `now_2 > now_1`: ONE elapsed -/// millisecond between the capture and the gate refuses the whole operation with NOTHING SENT. That is -/// not a bounded operation, it is a coin flip on the scheduler -- "mostly works, occasionally refuses -/// having sent nothing", which is the flakiness class validation exists to prevent. Single-attempt -/// semantics is what `max_attempts = 1` is for; the equality contributes only the race. -/// -/// The controller is constructed DIRECTLY here, bypassing `validateCasRequestBudget`, because the -/// point is to show the behaviour the validator now forbids. Three tests were flaky on exactly this -/// before it was forbidden: `8f9e63c7a19`'s sweep-interruption test, -/// `CASRefInstallSafety.UncertainPrecommitKeepsItsCleanupOwnerAndItsBody`, and -/// `CASRefWriterAppendLane.WedgedLaneBlocksSameTableWhileOtherTableProceeds`. -TEST(CASRequestController, EqualAttemptTimeoutAndDeadlineWouldRefuseAfterASingleTick) -{ - auto backend = std::make_shared(); - - /// The smallest possible passage of time: one millisecond per clock read. - uint64_t clock = 0; - auto now_ms = [&clock]() -> uint64_t { const uint64_t t = clock; clock += 1; return t; }; - - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 100; - CasRequestController razor(backend, budget, now_ms); - EXPECT_EQ(razor.putIfAbsentControlled("k", "payload", [] { return true; }), CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 0u) - << "the refusal came from the clock, not from the backend: nothing was sent at all"; - - /// STRICTLY LESS -- the shape the validator now requires -- sends the request over the SAME one-tick - /// clock. So what the inequality buys is the request actually happening, not merely a bigger number. - clock = 0; - budget.operation_deadline_ms = 5000; - CasRequestController wide(backend, budget, now_ms); - EXPECT_EQ(wide.putIfAbsentControlled("k", "payload", [] { return true; }), CasWriteOutcome::Committed); - EXPECT_EQ(backend->put_attempts.load(), 1u); -} - -/// And the same equality is refused at startup, so no budget can reach the controller in that shape. -/// The boundary is asserted from BOTH sides: equality throws, one millisecond more is accepted. -TEST(CASRequestController, ValidateBudgetRejectsAttemptTimeoutEqualToOperationDeadline) -{ - CasRequestBudget budget; - budget.attempt_timeout_ms = 5000; - budget.operation_deadline_ms = 5000; - budget.lease_safety_margin_ms = 1000; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { - validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); - }); - - budget.operation_deadline_ms = 5001; - EXPECT_NO_THROW(validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000)) - << "one millisecond of headroom is the whole requirement -- the rule is strictness, not size"; -} - -TEST(CASRequestController, OverwriteAmbiguousResolvesIntendedBytesAsCommitted) -{ - auto backend = std::make_shared(); - bool first_attempt = true; - backend->put_overwrite_thrower = [&first_attempt] - { - if (first_attempt) - { - first_attempt = false; - throw Poco::TimeoutException("scripted: ambiguous"); - } - }; - backend->setGetOverride(resultWithBytes("new-payload")); - - CasRequestController controller(backend, CasRequestBudget{}); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, [] { return true; }); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Committed); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(result.token, (Token{"t", TokenType::Emulated})); -} - -TEST(CASRequestController, OverwriteAmbiguousResolvesExpectedTokenAndRetriesWithinBudget) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - const Token expected{"old", TokenType::Emulated}; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - - CasRequestBudget budget; - budget.max_attempts = 3; - budget.retry_initial_backoff_ms = 0; - CasRequestController controller(backend, budget); - const auto result = controller.putOverwriteControlled("k", "new-payload", expected, [] { return true; }); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 3u); -} - -TEST(CASRequestController, OverwriteAmbiguousResolvesDifferentTokenAndBytesAsConflict) -{ - auto backend = std::make_shared(); - bool first_attempt = true; - backend->put_overwrite_thrower = [&first_attempt] - { - if (first_attempt) - { - first_attempt = false; - throw Poco::TimeoutException("scripted: ambiguous"); - } - }; - backend->setGetOverride(GetResult{ - .bytes = "someone-elses-payload", .token = Token{"other", TokenType::Emulated}, .attributes = {}}); - - CasRequestController controller(backend, CasRequestBudget{}); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, [] { return true; }); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Conflict); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); -} - -TEST(CASRequestController, OverwriteOperationDeadlineExhaustionReturnsUnresolvedBeforeMaxAttempts) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - const Token expected{"old", TokenType::Emulated}; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - - uint64_t clock = 0; - auto now_ms = [&clock]() -> uint64_t { const uint64_t t = clock; clock += 200; return t; }; - - CasRequestBudget budget; - budget.max_attempts = 10; - budget.attempt_timeout_ms = 50; - budget.operation_deadline_ms = 450; - budget.retry_initial_backoff_ms = 0; - CasRequestController controller(backend, budget, now_ms); - const auto result = controller.putOverwriteControlled("k", "new-payload", expected, [] { return true; }); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 2u); -} - -TEST(CASRequestController, AbsoluteDeadlineCannotBeReanchoredAfterPreemption) -{ - auto backend = std::make_shared(); - uint64_t clock = 60; - - CasRequestBudget budget; - budget.attempt_timeout_ms = 50; - budget.operation_deadline_ms = 10000; - CasRequestController controller(backend, budget, [&clock] { return clock; }); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 0u); - expectOverwriteDiagnostics( - result, - 0, - false, - CasUnresolvedReason::NoAttemptSent, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); -} - -TEST(CASRequestController, MaxAttemptsOneStillResolvesLostResponseByGet) -{ - auto backend = std::make_shared(); - const PutResult predecessor = backend->InMemoryBackend::putIfAbsent("k", "old-payload"); - ASSERT_EQ(predecessor.outcome, PutOutcome::Done); - - Token landed_token; - auto * backend_ptr = backend.get(); - backend->put_overwrite_handler = [backend_ptr, &landed_token]( - const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) -> PutResult - { - const PutResult landed = backend_ptr->InMemoryBackend::putOverwrite(key, bytes, expected, meta); - landed_token = landed.token; - throw Poco::TimeoutException("scripted: response lost after overwrite landed"); - }; - std::vector> progress; - - CasRequestBudget budget; - budget.max_attempts = 1; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 0; - CasRequestController controller(backend, budget, [] { return static_cast(0); }); - auto context = overwriteContext( - 1000, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - {}, - {}, - [&progress](const CasOverwriteProgress & event) { progress.emplace_back(event.kind, event.attempt_no); }); - const auto result = controller.putOverwriteControlled("k", "new-payload", predecessor.token, context); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Committed); - EXPECT_EQ(result.token, landed_token); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 1, - true, - CasUnresolvedReason::NotUnresolved, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); - const std::vector> expected_progress{ - {CasOverwriteProgressKind::PutStarted, 1}, - {CasOverwriteProgressKind::BecameAmbiguous, 1}, - {CasOverwriteProgressKind::ResolveStarted, 1}, - {CasOverwriteProgressKind::ResolvedByGet, 1}, - }; - EXPECT_EQ(progress, expected_progress); -} - -TEST(CASRequestController, StopBeforeFirstPutReportsExactCause) -{ - auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}, [] { return static_cast(0); }); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext( - 10000, - CasOverwriteDeadlineSource::RequestBudget, - [] { return CasOverwriteStopCause::Cancelled; })); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 0u); - EXPECT_EQ(backend->get_attempts.load(), 0u); - expectOverwriteDiagnostics( - result, - 0, - false, - CasUnresolvedReason::NoAttemptSent, - CasOverwriteDeadlineSource::RequestBudget, - CasOverwriteStopCause::Cancelled); -} - -TEST(CASRequestController, StopAfterPutSuppressesResolveAndReportsMidWay) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - uint32_t stop_samples = 0; - auto stop_cause = [&stop_samples] - { - ++stop_samples; - return stop_samples == 1 ? CasOverwriteStopCause::Continue : CasOverwriteStopCause::Cancelled; - }; - - CasRequestBudget budget; - budget.retry_initial_backoff_ms = 0; - CasRequestController controller(backend, budget, [] { return static_cast(0); }); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext(10000, CasOverwriteDeadlineSource::RequestBudget, stop_cause)); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 0u); - expectOverwriteDiagnostics( - result, - 1, - false, - CasUnresolvedReason::FenceLostMidWay, - CasOverwriteDeadlineSource::RequestBudget, - CasOverwriteStopCause::Cancelled); -} - -TEST(CASRequestController, StopAfterResolvedCommitReportsPostWrite) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(resultWithBytes("new-payload")); - uint32_t stop_samples = 0; - auto stop_cause = [&stop_samples] - { - ++stop_samples; - return stop_samples < 3 ? CasOverwriteStopCause::Continue : CasOverwriteStopCause::FenceOrLifecycleLost; - }; - - CasRequestController controller(backend, CasRequestBudget{}, [] { return static_cast(0); }); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext(10000, CasOverwriteDeadlineSource::RequestBudget, stop_cause)); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 1, - true, - CasUnresolvedReason::FenceLostPostWrite, - CasOverwriteDeadlineSource::RequestBudget, - CasOverwriteStopCause::FenceOrLifecycleLost); -} - -TEST(CASRequestController, FenceWinsCancellationAndExternalDeadlineWinsDeadlineTie) -{ - auto backend = std::make_shared(); - uint64_t clock = 100; - CasRequestBudget budget; - budget.attempt_timeout_ms = 10; - CasRequestController controller(backend, budget, [&clock] { return clock; }); - - bool cancelled = true; - bool fenced = true; - const auto simultaneous_stop = [&] - { - if (fenced) - return CasOverwriteStopCause::FenceOrLifecycleLost; - if (cancelled) - return CasOverwriteStopCause::Cancelled; - return CasOverwriteStopCause::Continue; - }; - const auto stopped = controller.putOverwriteControlled( - "stopped", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety, simultaneous_stop)); - EXPECT_EQ(stopped.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - stopped, - 0, - false, - CasUnresolvedReason::NoAttemptSent, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::FenceOrLifecycleLost); - - const auto deadline_tie = controller.putOverwriteControlled( - "deadline", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - EXPECT_EQ(deadline_tie.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - deadline_tie, - 0, - false, - CasUnresolvedReason::NoAttemptSent, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 0u); -} - -TEST(CASRequestController, InterruptedWaitResamplesStopCause) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - const Token expected{"old", TokenType::Emulated}; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - CasOverwriteStopCause stop = CasOverwriteStopCause::Continue; - uint32_t waits = 0; - - CasRequestBudget budget; - budget.max_attempts = 3; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 100; - budget.retry_max_backoff_ms = 100; - CasRequestController controller(backend, budget, [] { return static_cast(0); }); - auto context = overwriteContext( - 10000, - CasOverwriteDeadlineSource::RequestBudget, - [&stop] { return stop; }, - [&stop, &waits](uint64_t wait_ms) - { - EXPECT_EQ(wait_ms, 100u); - ++waits; - stop = CasOverwriteStopCause::Cancelled; - return false; - }); - const auto result = controller.putOverwriteControlled("k", "new-payload", expected, context); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(waits, 1u); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 1, - false, - CasUnresolvedReason::FenceLostMidWay, - CasOverwriteDeadlineSource::RequestBudget, - CasOverwriteStopCause::Cancelled); -} - -#ifndef DEBUG_OR_SANITIZER_BUILD -TEST(CASRequestController, InterruptedWaitWithoutPublishedStopIsAProgrammingException) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - const Token expected{"old", TokenType::Emulated}; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - uint32_t stop_samples = 0; - uint32_t waits = 0; - - CasRequestBudget budget; - budget.max_attempts = 3; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 100; - budget.retry_max_backoff_ms = 100; - CasRequestController controller(backend, budget, [] { return static_cast(0); }); - auto context = overwriteContext( - 10000, - CasOverwriteDeadlineSource::RequestBudget, - [&stop_samples] - { - ++stop_samples; - return CasOverwriteStopCause::Continue; - }, - [&waits](uint64_t wait_ms) - { - EXPECT_EQ(wait_ms, 100u); - ++waits; - return false; - }); - - bool threw = false; - try - { - (void)controller.putOverwriteControlled("k", "new-payload", expected, context); - FAIL() << "an interrupted wait must publish a non-Continue stop cause"; - } - catch (const DB::Exception & e) - { - threw = true; - EXPECT_EQ(e.code(), DB::ErrorCodes::LOGICAL_ERROR); - EXPECT_EQ( - e.message(), - "CasRequestController: wait_before_retry returned false while stop_cause remained Continue"); - } - - EXPECT_TRUE(threw); - EXPECT_EQ(stop_samples, 5u) << "the false wait was followed by an authoritative stop-cause resample"; - EXPECT_EQ(waits, 1u); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); -} -#endif - -#if defined(DEBUG_OR_SANITIZER_BUILD) -TEST(CASRequestControllerDeathTest, InterruptedWaitWithoutPublishedStopIsAProgrammingExceptionAborts) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - const Token expected{"old", TokenType::Emulated}; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - - CasRequestBudget budget; - budget.max_attempts = 3; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 100; - budget.retry_max_backoff_ms = 100; - CasRequestController controller(backend, budget, [] { return static_cast(0); }); - auto context = overwriteContext( - 10000, - CasOverwriteDeadlineSource::RequestBudget, - [] { return CasOverwriteStopCause::Continue; }, - [](uint64_t) { return false; }); - - EXPECT_DEATH( - { (void)controller.putOverwriteControlled("k", "new-payload", expected, context); }, - "wait_before_retry returned false while stop_cause remained Continue"); -} -#endif - -TEST(CASRequestController, CompletedWaitCrossingDeadlineSendsNoRetry) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - const Token expected{"old", TokenType::Emulated}; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - uint64_t clock = 0; - uint32_t waits = 0; - - CasRequestBudget budget; - budget.max_attempts = 3; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 50; - budget.retry_max_backoff_ms = 50; - CasRequestController controller(backend, budget, [&clock] { return clock; }); - auto context = overwriteContext( - 100, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - {}, - [&clock, &waits](uint64_t wait_ms) - { - EXPECT_EQ(wait_ms, 50u); - ++waits; - clock = 101; - return true; - }); - const auto result = controller.putOverwriteControlled("k", "new-payload", expected, context); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_TRUE(result.token.empty()); - EXPECT_EQ(waits, 1u); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 1, - false, - CasUnresolvedReason::DeadlineMidWay, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); -} - -TEST(CASRequestController, DirectPutCompletingAtDeadlineIsNotAccepted) -{ - auto backend = std::make_shared(); - const PutResult predecessor = backend->InMemoryBackend::putIfAbsent("k", "old-payload"); - ASSERT_EQ(predecessor.outcome, PutOutcome::Done); - uint64_t clock = 0; - Token landed_token; - auto * backend_ptr = backend.get(); - backend->put_overwrite_handler = [backend_ptr, &clock, &landed_token]( - const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) -> PutResult - { - const PutResult landed = backend_ptr->InMemoryBackend::putOverwrite(key, bytes, expected, meta); - landed_token = landed.token; - clock = 100; - return landed; - }; - - CasRequestBudget budget; - budget.attempt_timeout_ms = 10; - CasRequestController controller(backend, budget, [&clock] { return clock; }); - const auto result = controller.putOverwriteControlled( - "k", - "new-payload", - predecessor.token, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_TRUE(result.token.empty()); - ASSERT_FALSE(landed_token.empty()); - const auto durable = backend->InMemoryBackend::get("k", Range{}); - ASSERT_TRUE(durable.has_value()); - EXPECT_EQ(durable->bytes, "new-payload"); - EXPECT_EQ(durable->token, landed_token); - EXPECT_NE(durable->token, predecessor.token); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 0u); - expectOverwriteDiagnostics( - result, - 1, - false, - CasUnresolvedReason::DeadlineMidWay, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); -} - -TEST(CASRequestController, ReadProofCompletingAtDeadlineIsNotAccepted) -{ - auto backend = std::make_shared(); - const PutResult predecessor = backend->InMemoryBackend::putIfAbsent("k", "old-payload"); - ASSERT_EQ(predecessor.outcome, PutOutcome::Done); - uint64_t clock = 0; - Token landed_token; - auto * backend_ptr = backend.get(); - backend->put_overwrite_handler = [backend_ptr, &landed_token]( - const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) -> PutResult - { - const PutResult landed = backend_ptr->InMemoryBackend::putOverwrite(key, bytes, expected, meta); - landed_token = landed.token; - throw Poco::TimeoutException("scripted: response lost after overwrite landed"); - }; - backend->get_handler = [backend_ptr, &clock](const String & key, Range range) -> std::optional - { - const auto durable = backend_ptr->InMemoryBackend::get(key, range); - clock = 100; - return durable; - }; - - CasRequestBudget budget; - budget.attempt_timeout_ms = 10; - CasRequestController controller(backend, budget, [&clock] { return clock; }); - const auto result = controller.putOverwriteControlled( - "k", - "new-payload", - predecessor.token, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_TRUE(result.token.empty()); - ASSERT_FALSE(landed_token.empty()); - const auto durable = backend->InMemoryBackend::get("k", Range{}); - ASSERT_TRUE(durable.has_value()); - EXPECT_EQ(durable->bytes, "new-payload"); - EXPECT_EQ(durable->token, landed_token); - EXPECT_NE(durable->token, predecessor.token); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 1, - true, - CasUnresolvedReason::DeadlineMidWay, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); -} - -TEST(CASRequestController, ObserverFailureCannotChangeOutcome) -{ - auto backend = std::make_shared(); - const PutResult predecessor = backend->InMemoryBackend::putIfAbsent("k", "old-payload"); - ASSERT_EQ(predecessor.outcome, PutOutcome::Done); - auto * backend_ptr = backend.get(); - bool first_attempt = true; - backend->put_overwrite_handler = [backend_ptr, &first_attempt]( - const String & key, const String & bytes, const Token & expected, const ObjectMeta & meta) -> PutResult - { - if (first_attempt) - { - first_attempt = false; - throw Poco::TimeoutException("scripted: transient failure before landing"); - } - return backend_ptr->InMemoryBackend::putOverwrite(key, bytes, expected, meta); - }; - uint32_t observer_calls = 0; - - CasRequestBudget budget; - budget.max_attempts = 2; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 0; - CasRequestController controller(backend, budget, [] { return static_cast(0); }); - auto context = overwriteContext( - 10000, - CasOverwriteDeadlineSource::RequestBudget, - {}, - {}, - [&observer_calls](const CasOverwriteProgress &) - { - ++observer_calls; - throw DB::Exception(DB::ErrorCodes::UNKNOWN_EXCEPTION, "scripted observer failure"); - }); - - CasOverwriteResult result; - EXPECT_NO_THROW(result = controller.putOverwriteControlled("k", "new-payload", predecessor.token, context)); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Committed); - EXPECT_EQ(observer_calls, 5u); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 2u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 2, - false, - CasUnresolvedReason::NotUnresolved, - CasOverwriteDeadlineSource::RequestBudget, - CasOverwriteStopCause::Continue); -} - -TEST(CASRequestController, ResolveFailuresExhaustDeadlineWithoutSendingLatePut) -{ - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - uint64_t clock = 0; - backend->get_handler = [&clock](const String &, Range) -> std::optional - { - clock = 30; - throw Poco::TimeoutException("scripted: resolving GET failed at deadline"); - }; - - CasRequestBudget budget; - budget.max_attempts = 5; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 0; - CasRequestController controller(backend, budget, [&clock] { return clock; }); - const auto result = controller.putOverwriteControlled( - "k", "new-payload", Token{"old", TokenType::Emulated}, - overwriteContext(30, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 1u); - EXPECT_EQ(backend->get_attempts.load(), 1u); - expectOverwriteDiagnostics( - result, - 1, - false, - CasUnresolvedReason::DeadlineMidWay, - CasOverwriteDeadlineSource::ExternalLeaseSafety, - CasOverwriteStopCause::Continue); -} - -TEST(CASRequestController, EveryTerminalShapeReportsExactDiagnostics) -{ - const Token expected{"old", TokenType::Emulated}; - auto make_controller = [](const std::shared_ptr & backend, uint64_t & clock, uint32_t max_attempts = 2) - { - CasRequestBudget budget; - budget.max_attempts = max_attempts; - budget.attempt_timeout_ms = 10; - budget.retry_initial_backoff_ms = 0; - return CasRequestController(backend, budget, [&clock] { return clock; }); - }; - - for (const auto stop : {CasOverwriteStopCause::Cancelled, CasOverwriteStopCause::FenceOrLifecycleLost}) - { - auto backend = std::make_shared(); - uint64_t clock = 0; - auto controller = make_controller(backend, clock); - const auto result = controller.putOverwriteControlled( - "pre-stop", "new-payload", expected, - overwriteContext(100, CasOverwriteDeadlineSource::RequestBudget, [stop] { return stop; })); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 0, false, CasUnresolvedReason::NoAttemptSent, - CasOverwriteDeadlineSource::RequestBudget, stop); - } - - for (const auto source : {CasOverwriteDeadlineSource::RequestBudget, CasOverwriteDeadlineSource::ExternalLeaseSafety}) - { - auto backend = std::make_shared(); - uint64_t clock = 100; - auto controller = make_controller(backend, clock); - const auto result = controller.putOverwriteControlled( - "pre-deadline", "new-payload", expected, overwriteContext(100, source)); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 0, false, CasUnresolvedReason::NoAttemptSent, source, CasOverwriteStopCause::Continue); - } - - for (const auto stop : {CasOverwriteStopCause::Cancelled, CasOverwriteStopCause::FenceOrLifecycleLost}) - { - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - uint64_t clock = 0; - uint32_t stop_samples = 0; - auto controller = make_controller(backend, clock); - const auto result = controller.putOverwriteControlled( - "mid-stop", "new-payload", expected, - overwriteContext(100, CasOverwriteDeadlineSource::RequestBudget, [&stop_samples, stop] - { - ++stop_samples; - return stop_samples == 1 ? CasOverwriteStopCause::Continue : stop; - })); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 1, false, CasUnresolvedReason::FenceLostMidWay, - CasOverwriteDeadlineSource::RequestBudget, stop); - } - - for (const auto stop : {CasOverwriteStopCause::Cancelled, CasOverwriteStopCause::FenceOrLifecycleLost}) - { - auto backend = std::make_shared(); - backend->put_overwrite_forced_outcome = PutOutcome::Done; - uint64_t clock = 0; - uint32_t stop_samples = 0; - auto controller = make_controller(backend, clock); - const auto result = controller.putOverwriteControlled( - "post-stop", "new-payload", expected, - overwriteContext(100, CasOverwriteDeadlineSource::RequestBudget, [&stop_samples, stop] - { - ++stop_samples; - return stop_samples == 1 ? CasOverwriteStopCause::Continue : stop; - })); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 1, false, CasUnresolvedReason::FenceLostPostWrite, - CasOverwriteDeadlineSource::RequestBudget, stop); - } - - for (const auto source : {CasOverwriteDeadlineSource::RequestBudget, CasOverwriteDeadlineSource::ExternalLeaseSafety}) - { - auto backend = std::make_shared(); - uint64_t clock = 0; - backend->put_overwrite_handler = [&clock]( - const String &, const String &, const Token &, const ObjectMeta &) -> PutResult - { - clock = 100; - throw Poco::TimeoutException("scripted: ambiguous at deadline"); - }; - auto controller = make_controller(backend, clock); - const auto result = controller.putOverwriteControlled( - "mid-deadline", "new-payload", expected, overwriteContext(100, source)); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 1, false, CasUnresolvedReason::DeadlineMidWay, source, CasOverwriteStopCause::Continue); - } - - { - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(std::nullopt); - uint64_t clock = 0; - auto controller = make_controller(backend, clock, 1); - const auto result = controller.putOverwriteControlled( - "attempts", "new-payload", expected, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 1, false, CasUnresolvedReason::AttemptsExhausted, - CasOverwriteDeadlineSource::ExternalLeaseSafety, CasOverwriteStopCause::Continue); - } - - { - auto backend = std::make_shared(); - auto * backend_ptr = backend.get(); - backend->put_overwrite_handler = [backend_ptr]( - const String &, const String &, const Token &, const ObjectMeta &) -> PutResult - { - if (backend_ptr->put_overwrite_attempts.load() == 1) - throw Poco::TimeoutException("scripted: first attempt remains ambiguous"); - throw DB::S3Exception("scripted: later attempt definitely refused", Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); - }; - backend->setGetOverride(GetResult{.bytes = "old-payload", .token = expected, .attributes = {}}); - uint64_t clock = 0; - auto controller = make_controller(backend, clock); - const auto result = controller.putOverwriteControlled( - "definite-after-ambiguity", "new-payload", expected, - overwriteContext(100, CasOverwriteDeadlineSource::RequestBudget)); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - EXPECT_EQ(backend->put_overwrite_attempts.load(), 2u); - expectOverwriteDiagnostics( - result, 2, false, CasUnresolvedReason::DefiniteFailureAfterAmbiguity, - CasOverwriteDeadlineSource::RequestBudget, CasOverwriteStopCause::Continue); - } - - { - auto backend = std::make_shared(); - backend->put_overwrite_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - uint64_t clock = 0; - backend->get_handler = [&clock](const String &, Range) -> std::optional - { - clock = 95; - return std::nullopt; - }; - auto controller = make_controller(backend, clock, 1); - const auto result = controller.putOverwriteControlled( - "deadline-before-attempt-limit", "new-payload", expected, - overwriteContext(100, CasOverwriteDeadlineSource::ExternalLeaseSafety)); - EXPECT_EQ(result.outcome, CasOverwriteOutcome::Unresolved); - expectOverwriteDiagnostics( - result, 1, false, CasUnresolvedReason::DeadlineMidWay, - CasOverwriteDeadlineSource::ExternalLeaseSafety, CasOverwriteStopCause::Continue); - } -} - -TEST(CASRequestController, FenceLostBeforeAttemptSendsNoAttempt) -{ - auto backend = std::make_shared(); - CasRequestController controller(backend, CasRequestBudget{}); - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return false; }); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 0u); -} - -/// The write itself may have landed, but a fence lost between the write and this call's own final -/// check must never surface as Committed (RFC §ack-and-cache-rules: no ACK, no cache update on that -/// path) — the caller sees Unresolved and must not treat the operation as acknowledged. -TEST(CASRequestController, FenceLostAfterWriteNeverReturnsCommitted) -{ - auto backend = std::make_shared(); /// real in-memory commit path - int fence_calls = 0; - auto fence_ok = [&fence_calls] { return fence_calls++ == 0; }; /// true once, then false - - CasRequestController controller(backend, CasRequestBudget{}); - const auto outcome = controller.putIfAbsentControlled("k", "payload", fence_ok); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 1u); /// the write itself DID happen - EXPECT_TRUE(backend->head("k").exists); /// ...it is durable; never claimed as Committed here -} - -TEST(CASRequestController, DefiniteFailurePropagatesImmediatelyWithoutResolve) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw DB::S3Exception("scripted: malformed", Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); }; - - CasRequestController controller(backend, CasRequestBudget{}); - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return true; }); - EXPECT_EQ(outcome, CasWriteOutcome::DefiniteFailure); - EXPECT_EQ(backend->put_attempts.load(), 1u); /// no retry, no resolve GET issued -} - -/// ================================================================================================ -/// Inter-attempt backoff (chaos-tolerance-report §Task B follow-up / stagefix-review M3): the -/// controller paces reissues with a capped-exponential, fence-gated, deadline-aware sleep instead of -/// hammering a recovering store with immediate retries. -/// ================================================================================================ - -/// The full event-ordered schedule: fence checked before EVERY attempt AND before EVERY sleep, sleeps -/// strictly between attempts, capped exponential (initial 100ms, cap 200ms), no sleep after the final -/// attempt. The exact interleaving is the contract — a sleep served before its fence check would keep -/// a fenced writer dozing past its lease. -TEST(CASRequestControllerBackoff, CappedExponentialSleepsAreFenceCheckedAndOrdered) -{ - auto backend = std::make_shared(); - std::vector events; - backend->put_thrower = [&] { events.emplace_back("put"); throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(std::nullopt); /// absent on every resolve - - CasRequestBudget budget; - budget.max_attempts = 5; - budget.attempt_timeout_ms = 1; - budget.operation_deadline_ms = 1000000; /// never the binding constraint here - budget.retry_initial_backoff_ms = 100; - budget.retry_max_backoff_ms = 200; - CasRequestController controller( - backend, budget, - /*now_ms=*/[] { return static_cast(0); }, - /*sleep_ms=*/[&](uint64_t ms) { events.push_back("sleep:" + std::to_string(ms)); }); - - const auto fence_ok = [&] { events.emplace_back("fence"); return true; }; - const auto outcome = controller.putIfAbsentControlled("k", "payload", fence_ok); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 5u); - - const std::vector expected{ - "fence", "put", "fence", "sleep:100", - "fence", "put", "fence", "sleep:200", - "fence", "put", "fence", "sleep:200", - "fence", "put", "fence", "sleep:200", - "fence", "put"}; /// budget spent: no fence-for-sleep, no sleep after the last attempt - EXPECT_EQ(events, expected); -} - -/// A fence lost between an ambiguous attempt's resolve and its backoff sleep aborts INSTANTLY: no -/// sleep is served, no further attempt is sent, and the outcome is Unresolved (never a false -/// Committed, never a retry under a lost lease). -TEST(CASRequestControllerBackoff, FenceLostBeforeSleepAbortsWithoutSleeping) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(std::nullopt); - - CasRequestBudget budget; - budget.max_attempts = 5; - budget.retry_initial_backoff_ms = 100; - budget.retry_max_backoff_ms = 200; - uint64_t sleeps = 0; - int fence_calls = 0; - CasRequestController controller( - backend, budget, /*now_ms=*/[] { return static_cast(0); }, - /*sleep_ms=*/[&](uint64_t) { ++sleeps; }); - - /// True for the pre-attempt check (call 1), lost by the pre-sleep check (call 2). - const auto fence_ok = [&fence_calls] { return ++fence_calls <= 1; }; - const auto outcome = controller.putIfAbsentControlled("k", "payload", fence_ok); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 1u) << "no attempt may be sent after the fence is lost"; - EXPECT_EQ(sleeps, 0u) << "a fence lost mid-backoff must abort BEFORE the sleep, not after it"; - EXPECT_EQ(fence_calls, 2); -} - -/// A backoff sleep the operation deadline cannot afford is never served: when sleep + one more -/// attempt would cross the deadline, the loop gives up immediately (Unresolved) instead of sleeping -/// into a guaranteed exhaustion. -TEST(CASRequestControllerBackoff, SleepThatWouldCrossOperationDeadlineIsSkipped) -{ - auto backend = std::make_shared(); - backend->put_thrower = [] { throw Poco::TimeoutException("scripted: ambiguous"); }; - backend->setGetOverride(std::nullopt); - - uint64_t clock = 0; - CasRequestBudget budget; - budget.max_attempts = 10; - budget.attempt_timeout_ms = 10; - budget.operation_deadline_ms = 100; - budget.retry_initial_backoff_ms = 1000; /// any sleep would blow the 100ms deadline - budget.retry_max_backoff_ms = 1000; - uint64_t sleeps = 0; - CasRequestController controller( - backend, budget, /*now_ms=*/[&clock] { return clock; }, - /*sleep_ms=*/[&](uint64_t) { ++sleeps; }); - - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return true; }); - EXPECT_EQ(outcome, CasWriteOutcome::Unresolved); - EXPECT_EQ(backend->put_attempts.load(), 1u); - EXPECT_EQ(sleeps, 0u) << "the deadline guard must refuse the sleep, not serve it and then fail"; -} - -/// THE ENVELOPE CONTRACT (chaos-tolerance-report §Task B follow-up): the DEFAULT budget rides a -/// simulated 60-second S3 outage — every conditional-write attempt fails (≈3s adaptive first-attempt -/// timeout each, the observed incident shape) until the store recovers at t=60s, then the next -/// attempt commits, all inside the default 90s operation deadline and 16-attempt budget. The fake -/// clock advances 3s per failed attempt and by each backoff sleep, so this test pins the arithmetic -/// documented on CasRequestBudget without any wall-clock waiting. -TEST(CASRequestControllerBackoff, DefaultBudgetRidesSixtySecondOutage) -{ - auto backend = std::make_shared(); - uint64_t clock = 0; - backend->put_thrower = [&clock] - { - if (clock < 60000) - { - clock += 3000; /// the failed attempt's own ~3s adaptive receive timeout - throw Poco::TimeoutException("scripted: store paused"); - } - /// store recovered: fall through to the real in-memory conditional write (Done) - }; - backend->setGetOverride(std::nullopt); /// nothing ever landed while the store was paused - - CasRequestController controller( - backend, CasRequestBudget{}, /*now_ms=*/[&clock] { return clock; }, - /*sleep_ms=*/[&clock](uint64_t ms) { clock += ms; }); - - const auto outcome = controller.putIfAbsentControlled("k", "payload", [] { return true; }); - EXPECT_EQ(outcome, CasWriteOutcome::Committed) << "the default budget must absorb a 60s outage"; - /// Schedule: attempts fail at 3s each with sleeps 0.2,0.4,0.8,1.6,3.2 then 5s (cap); the first - /// attempt scheduled at clock >= 60000 (attempt 11, t=61.2s) commits — well inside 16 attempts - /// and the 90s deadline. - EXPECT_EQ(backend->put_attempts.load(), 11u); - EXPECT_LT(clock, CasRequestBudget{}.operation_deadline_ms); -} - -/// Startup validation (RFC §required-timeout-model): a consistent default budget is accepted silently; -/// either inequality violated on its own is rejected with BAD_ARGUMENTS. -TEST(CASRequestController, ValidateBudgetAcceptsConsistentDefaults) -{ - EXPECT_NO_THROW(validateCasRequestBudget(CasRequestBudget{}, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000)); -} - -TEST(CASRequestController, ValidateBudgetRejectsAttemptTimeoutPlusMarginAtOrAboveLeaseTtl) -{ - CasRequestBudget budget; - budget.attempt_timeout_ms = 25000; - budget.lease_safety_margin_ms = 5000; /// sums to EXACTLY the lease TTL below — not strictly less - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { - validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); - }); -} - -TEST(CASRequestController, ValidateBudgetRejectsAttemptTimeoutAboveOperationDeadline) -{ - CasRequestBudget budget; - budget.attempt_timeout_ms = 6000; - budget.operation_deadline_ms = 5000; - budget.lease_safety_margin_ms = 1000; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { - validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); - }); -} - -/// max_attempts == 0 would let putIfAbsentControlled return Unresolved without ever sending an -/// attempt — reject at startup rather than silently accepting a no-op budget. -TEST(CASRequestController, ValidateBudgetRejectsZeroMaxAttempts) -{ - CasRequestBudget budget; - budget.max_attempts = 0; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { - validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); - }); -} - -/// A capped-exponential backoff whose cap sits below its own starting value is inconsistent — reject -/// at startup (0/0 disables backoff and stays accepted, covered by the defaults test above since the -/// defaults are nonzero and consistent). -TEST(CASRequestController, ValidateBudgetRejectsInitialBackoffAboveMaxBackoff) -{ - CasRequestBudget budget; - budget.retry_initial_backoff_ms = 500; - budget.retry_max_backoff_ms = 100; - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { - validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); - }); -} - -/// attempt_timeout_ms + lease_safety_margin_ms must not be computed by a wrapping uint64 sum: absurd -/// config values near UINT64_MAX must fail validation (correctly, as inconsistent), never wrap around -/// to a spuriously small sum that would pass the "< lease TTL" check. -TEST(CASRequestController, ValidateBudgetRejectsOverflowingSumRatherThanWrapping) -{ - CasRequestBudget budget; - budget.attempt_timeout_ms = std::numeric_limits::max() - 10; - budget.lease_safety_margin_ms = 20; /// sum would wrap past UINT64_MAX to a tiny value - budget.operation_deadline_ms = std::numeric_limits::max(); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] - { - validateCasRequestBudget(budget, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); - }); -} - -#endif diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index 9f696c2d2397..6e689395a368 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -1,13 +1,12 @@ #include -#include +#include #include #include #include #include #include #include -#include #include #include "cas_test_helpers.h" @@ -55,9 +54,9 @@ CasRequests makeRequests(BackendPtr backend, FakeClock & clock, Fence fence = Fe } -static_assert(!std::is_default_constructible_v); -static_assert(!std::is_constructible_v); -static_assert(!std::is_constructible_v); +static_assert(!std::is_default_constructible_v); +static_assert(!std::is_constructible_v); +static_assert(!std::is_constructible_v); static_assert(!std::is_default_constructible_v); static_assert(!std::is_copy_constructible_v); @@ -128,7 +127,7 @@ TEST(CASWriteResult, OrThrowMapsEveryAlternative) auto op = requests.admit(); WriteResult committed = op.create("k", "v", Retry::standard()); ASSERT_TRUE(std::holds_alternative(committed)); - const Incarnation landed = std::get(committed).incarnation; + const Etag landed = std::get(committed).etag; const auto returned = orThrow(std::move(committed), "create"); ASSERT_TRUE(returned.has_value()); EXPECT_EQ(*returned, landed); @@ -170,20 +169,20 @@ TEST(CASBackendPrimitives, InMemoryWriteReadRemoveRoundTripThroughOneOperation) auto requests = makeRequests(b, clock); auto op = requests.admit(); - const std::optional w1 = orThrow(op.create("k", "v1", Retry::once()), "create"); + const std::optional w1 = orThrow(op.create("k", "v1", Retry::once()), "create"); ASSERT_TRUE(w1); const std::optional r = op.read("k", Retry::once()); ASSERT_TRUE(r); EXPECT_EQ(r->bytes, "v1"); - EXPECT_EQ(r->incarnation, *w1); + EXPECT_EQ(r->etag, *w1); const std::optional h = op.head("k", Retry::once()); ASSERT_TRUE(h); EXPECT_EQ(h->size, 2u); - EXPECT_EQ(h->incarnation, *w1); + EXPECT_EQ(h->etag, *w1); EXPECT_TRUE(std::holds_alternative(op.create("k", "v2", Retry::once()))); /// must be absent - const std::optional w3 = orThrow(op.replace("k", "v2", *w1, Retry::once()), "replace"); + const std::optional w3 = orThrow(op.replace("k", "v2", *w1, Retry::once()), "replace"); ASSERT_TRUE(w3); EXPECT_NE(*w3, *w1); /// incarnations never repeat @@ -200,23 +199,23 @@ TEST(CASBackendPrimitives, ListSurfacesTheIncarnationAndPaginates) auto requests = makeRequests(b, clock); auto op = requests.admit(); - const std::optional a = orThrow(op.create("p/a", "0123456789", Retry::once()), "create"); + const std::optional a = orThrow(op.create("p/a", "0123456789", Retry::once()), "create"); ASSERT_TRUE(a); orThrow(op.create("p/b", "xy", Retry::once()), "create"); orThrow(op.create("q/c", "z", Retry::once()), "create"); - const KeyPage page = op.list("p/", "", 10, Retry::once()); + const ListPage page = op.list("p/", "", 10, Retry::once()); ASSERT_EQ(page.keys.size(), 2u); /// sorted, prefix-scoped EXPECT_EQ(page.keys[0].key, "p/a"); EXPECT_EQ(page.keys[0].size, 10u); - ASSERT_TRUE(page.keys[0].incarnation.has_value()); - EXPECT_EQ(*page.keys[0].incarnation, *a); + ASSERT_TRUE(page.keys[0].etag.has_value()); + EXPECT_EQ(*page.keys[0].etag, *a); EXPECT_TRUE(page.next_cursor.empty()); - const KeyPage first = op.list("p/", "", 1, Retry::once()); + const ListPage first = op.list("p/", "", 1, Retry::once()); ASSERT_EQ(first.keys.size(), 1u); EXPECT_EQ(first.next_cursor, "p/a"); - const KeyPage second = op.list("p/", first.next_cursor, 1, Retry::once()); + const ListPage second = op.list("p/", first.next_cursor, 1, Retry::once()); ASSERT_EQ(second.keys.size(), 1u); EXPECT_EQ(second.keys[0].key, "p/b"); } @@ -230,57 +229,24 @@ TEST(CASBackendPrimitives, EveryBackendInstanceHasItsOwnId) EXPECT_EQ(a->dialect(), Dialect::Emulated); } -namespace -{ - -/// Counts every call that reaches the primitive `write`, whichever surface it entered through. -struct WriteCountingBackend : InMemoryBackend -{ - size_t writes = 0; - - std::expected write(const String & k, const String & v, const std::optional & e, - TransportAccess & a) override - { - ++writes; - return InMemoryBackend::write(k, v, e, a); - } -}; - -} +/// `EveryLegacyVerbReachesAnOverrideOfThePrimitiveItForwardsTo` pinned the legacy verbs +/// (`putIfAbsent`/`casPut`/`putOverwrite`) forwarding through the primitive `write`. Those verbs are +/// gone -- `CasOperation` is the only caller of `Backend` now -- so the property is a type-level +/// guarantee rather than a runtime check; every fault double in this file that overrides `write` (e.g. +/// `EachWriteKnobIsKeyedAndOneShotOnThePrimitiveWrite` below) is what proves a double sees every write. -TEST(CASBackendPrimitives, EveryLegacyVerbReachesAnOverrideOfThePrimitiveItForwardsTo) +TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotOnThePrimitiveWrite) { - /// The migration rule: the new methods are the primitives, and a fault injection written against a - /// NEW signature intercepts a legacy caller too, because every legacy verb forwards through the - /// virtual. No verb is exempt -- an exemption would leave a double blind to whichever surface its - /// subject happens to use. - auto b = std::make_shared(); - FakeClock clock; - auto requests = makeRequests(b, clock); - auto op = requests.admit(); - const std::optional first = orThrow(op.create("k", "v", Retry::once()), "create"); - ASSERT_TRUE(first); - b->writes = 0; - - EXPECT_EQ(b->putIfAbsent("k2", "v").outcome, PutOutcome::Done); - EXPECT_EQ(b->casPut("k3", "v", std::nullopt).outcome, CasOutcome::Committed); - EXPECT_EQ(b->putOverwrite("k", "w", b->head("k").token).outcome, PutOutcome::Done); - EXPECT_EQ(b->writes, 3u); -} - -TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotWhicheverSurfaceConsumesIt) -{ - /// A knob names a KEY, not a verb: the keyed `write` every surface reaches cannot see which verb - /// its caller used, so a knob scoped to one verb would fire or not fire on where the caller - /// happened to enter rather than on what it did. + /// A knob names a KEY, not a call site: the keyed `write` every write reaches, whichever + /// `CasOperation` verb (`create`/`replace`) issued it. auto b = std::make_shared(); FakeClock clock; auto requests = makeRequests(b, clock); auto op = requests.admit(); b->refuseNextWrite("k"); - EXPECT_EQ(b->putIfAbsent("k", "v").outcome, PutOutcome::PreconditionFailed); /// consumed here - EXPECT_EQ(b->putIfAbsent("k", "v").outcome, PutOutcome::Done); /// and only once + EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::once()))); /// consumed here + EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::once()))); /// and only once expectBytes(b, "k", "v"); b->refuseNextWrite("k2"); @@ -288,9 +254,9 @@ TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotWhicheverSurfaceConsume EXPECT_TRUE(std::holds_alternative(op.create("k2", "v", Retry::once()))); b->injectAmbiguousWrite("k3"); - EXPECT_THROW(b->casPut("k3", "v", std::nullopt), Poco::TimeoutException); - EXPECT_FALSE(b->get("k3").has_value()) << "an ambiguous write leaves the store untouched"; - EXPECT_EQ(b->casPut("k3", "v", std::nullopt).outcome, CasOutcome::Committed); + EXPECT_TRUE(std::holds_alternative(op.create("k3", "v", Retry::once()))); + EXPECT_FALSE(op.read("k3", Retry::once()).has_value()) << "an ambiguous write leaves the store untouched"; + EXPECT_TRUE(std::holds_alternative(op.create("k3", "v", Retry::once()))); /// Both knobs on one key, each consumed by the next write in turn. b->injectAmbiguousWrite("k4"); @@ -300,48 +266,26 @@ TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotWhicheverSurfaceConsume EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); } -TEST(CASBackendPrimitives, LegacyGetRefusesAValueThatIsNotAnIncarnation) +TEST(CASBackendPrimitives, ReadRefusesAValueThatIsNotAnIncarnation) { - /// `read` hands back whatever the store said, malformed included -- settling that is the caller's. - /// The legacy forwarder has no caller to settle it: it would hand the value on as a `Token` that - /// the next conditional operation refuses as a caller bug, one layer too late to name the key. + /// `read` hands back whatever the store said, malformed included -- `CasRequests::mint` is what + /// refuses it, naming the key, before any caller can see it as an `Etag`. struct EmptyValueBackend : InMemoryBackend { std::optional read(const String &, TransportAccess &) override { return Raw{"body", ""}; } }; auto b = std::make_shared(); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { b->get("k"); }); -} - -namespace -{ - -/// A double whose fault injection is written against the LEGACY surface, the way almost every one in -/// this suite is. A `Pool` hands its callers a decorator, so a legacy call reaches the decorator -/// first: if the decorator converted it to a primitive before forwarding, this override would never -/// run and the injection would be silently dead. -struct LegacyCasPutFlaggingBackend : DB::Cas::tests::CountingBackend -{ - bool legacy_cas_put_ran = false; - - CasResult casPut(const String & key, const String & bytes, const std::optional & expected, - const ObjectMeta & meta) override - { - legacy_cas_put_ran = true; - return CountingBackend::casPut(key, bytes, expected, meta); - } -}; - + FakeClock clock; + auto requests = makeRequests(b, clock); + auto op = requests.admit(); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { op.read("k", Retry::once()); }); } -TEST(CASBackendPrimitives, InstrumentedBackendPassesALegacyCallThroughAsLegacy) -{ - auto inner = std::make_shared(); - InstrumentedBackend instrumented(inner); - EXPECT_EQ(instrumented.casPut("k", "v", std::nullopt).outcome, CasOutcome::Committed); - EXPECT_TRUE(inner->legacy_cas_put_ran); - EXPECT_EQ(inner->writeCount("k"), 1u); -} +/// `InstrumentedBackendPassesALegacyCallThroughAsLegacy` pinned `InstrumentedBackend` delegating the +/// legacy `casPut` verb to its inner backend unconverted. `Backend` has no legacy verbs left -- +/// `InstrumentedBackend` is a pure primitive decorator now -- and its primitive delegation (`write` and +/// every other primitive, classified and counted) is what `CASInstrumentedBackend.ClassifierAndPerNamespaceOpEvents` +/// (gtest_cas_backend.cpp) pins. TEST(CASBackendPrimitives, RefreshCredentialsIsOffUntilAskedFor) { @@ -385,15 +329,10 @@ TEST(CASThrottlingBackend, RefusalsAreRetryableUnderBothStatuses) } } -TEST(CASThrottlingBackend, PassesALegacyCallThroughAsLegacy) -{ - auto inner = std::make_shared(); - /// A period no call here reaches, so nothing is refused: what this pins is the pass-through. - auto t = std::make_shared(inner, ThrottlingBackend::Mode::EveryNth, 1000, 503); - EXPECT_EQ(t->casPut("k", "v", std::nullopt).outcome, CasOutcome::Committed); - EXPECT_TRUE(inner->legacy_cas_put_ran); - EXPECT_EQ(inner->writeCount("k"), 1u); -} +/// `PassesALegacyCallThroughAsLegacy` pinned `ThrottlingBackend` delegating the legacy `casPut` verb +/// unconverted. `Backend` has no legacy verbs left; `ThrottlingBackend`'s primitive pass-through is +/// pinned by `FirstPerKeyRefusesOnceAndTheCallStillSucceeds` above and `EveryNthRefusesOnThePeriodAcrossKeys` +/// below, both of which drive it through `CasOperation`. TEST(CASThrottlingBackend, EveryNthRefusesOnThePeriodAcrossKeys) { @@ -516,20 +455,20 @@ TEST(CASIncarnation, RenderAndPersistedCompare) auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation first = *orThrow(op.create("k", "v", Retry::standard()), "create"); + const Etag first = *orThrow(op.create("k", "v", Retry::standard()), "create"); EXPECT_EQ(first.render(), "emulated:1"); EXPECT_EQ(first.key(), "k"); EXPECT_EQ(first.dialect(), Dialect::Emulated); - const PersistedIncarnation persisted = PersistedIncarnation::capture(first); + const PersistedEtag persisted = PersistedEtag::capture(first); EXPECT_EQ(persisted.dialect, "emulated"); EXPECT_EQ(persisted.value, "1"); EXPECT_TRUE(persisted.matches(first)); - const Incarnation second = *orThrow(op.replace("k", "w", first, Retry::standard()), "replace"); + const Etag second = *orThrow(op.replace("k", "w", first, Retry::standard()), "replace"); EXPECT_EQ(second.render(), "emulated:2"); EXPECT_FALSE(persisted.matches(second)); /// a captured record never re-matches a later incarnation - EXPECT_TRUE(PersistedIncarnation::capture(second).matches(second)); + EXPECT_TRUE(PersistedEtag::capture(second).matches(second)); } TEST(CASRetry, BindSaturatesAndLeavesAnEqualLeaseOffTheLeaseSource) @@ -559,13 +498,13 @@ TEST(CASRequests, CreateThenReplaceThenRemove) auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation first = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + const Etag first = *orThrow(op.create("k", "v1", Retry::standard()), "create"); const auto seen = op.read("k", Retry::standard()); ASSERT_TRUE(seen.has_value()); EXPECT_EQ(seen->bytes, "v1"); - EXPECT_EQ(seen->incarnation, first); + EXPECT_EQ(seen->etag, first); - const Incarnation second = *orThrow(op.replace("k", "v2", first, Retry::standard()), "replace"); + const Etag second = *orThrow(op.replace("k", "v2", first, Retry::standard()), "replace"); EXPECT_NE(second, first); EXPECT_EQ(op.remove("k", first, Retry::standard()), Removal::Mismatch); /// the incarnation is stale @@ -585,7 +524,7 @@ TEST(CASRequests, KeyBindingThrowsBeforeAnyRequest) auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation of_a = *orThrow(op.create("a", "v", Retry::standard()), "create"); + const Etag of_a = *orThrow(op.create("a", "v", Retry::standard()), "create"); backend->resetCounts(); expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] { (void)op.replace("b", "w", of_a, Retry::standard()); }); @@ -599,7 +538,7 @@ TEST(CASRequestsDeathTest, KeyBindingThrowsBeforeAnyRequest) auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation of_a = *orThrow(op.create("a", "v", Retry::standard()), "create"); + const Etag of_a = *orThrow(op.create("a", "v", Retry::standard()), "create"); EXPECT_DEATH({ (void)op.replace("b", "w", of_a, Retry::standard()); }, ""); } @@ -754,7 +693,7 @@ TEST(CASRequests, ForEachListedKeyStopsEarlyAndBudgetsPerPage) size_t seen = 0; size_t pages = 0; - op.forEachListedKey("p/", [&](const KeyEntry &) { return ++seen < 3; }, Retry::standard(), + op.forEachListedKey("p/", [&](const ListedKey &) { return ++seen < 3; }, Retry::standard(), /*page_limit=*/10, [&] { ++pages; }); EXPECT_EQ(seen, 3u); /// The walk stops where the caller stops it: the remaining two pages are never fetched. @@ -768,7 +707,7 @@ TEST(CASRequests, DeleteMarkerIsANamedException) auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation inc = *orThrow(op.create("k", "v", Retry::standard()), "create"); + const Etag inc = *orThrow(op.create("k", "v", Retry::standard()), "create"); backend->setSimulateDeleteMarkers(true); expectThrowsCode(DB::ErrorCodes::CAS_DELETE_MARKER, [&] { (void)op.remove("k", inc, Retry::standard()); }); @@ -925,7 +864,7 @@ TEST(CASRequests, AResolveReadRefusedForLeaseBudgetIsReportedAsTheLeaseDeadline) [](uint64_t) {}}; auto requests = makeRequests(backend, clock, fence); auto op = requests.admit(); - const Incarnation seen = *orThrow(op.create("k", "v", Retry::standard()), "create"); + const Etag seen = *orThrow(op.create("k", "v", Retry::standard()), "create"); /// The store refuses the precondition, and the lease budget is gone by the time the read that /// would say WHO holds the key is due. The call learned nothing about the key, so what it reports @@ -1112,7 +1051,7 @@ TEST(CASRequests, AmbiguousReplaceWhoseResolveShowsThePreconditionUnchangedIsRei auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation seen = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + const Etag seen = *orThrow(op.create("k", "v1", Retry::standard()), "create"); backend->resetCounts(); /// The attempt's fate is lost and the store is untouched. The incarnation it named is still the @@ -1139,7 +1078,7 @@ TEST(CASRequests, AmbiguousReplaceOfIdenticalBytesIsReissuedNotClaimedByByteEqua auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation seen = *orThrow(op.create("k", "B", Retry::standard()), "create"); + const Etag seen = *orThrow(op.create("k", "B", Retry::standard()), "create"); backend->resetCounts(); backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); @@ -1150,7 +1089,7 @@ TEST(CASRequests, AmbiguousReplaceOfIdenticalBytesIsReissuedNotClaimedByByteEqua /// never made; the reissue is what actually put these bytes there under a new incarnation. EXPECT_EQ(committed->attempts_sent, 2u); EXPECT_FALSE(committed->resolved_by_read); - EXPECT_NE(committed->incarnation, seen); + EXPECT_NE(committed->etag, seen); EXPECT_EQ(backend->writeTotal(), 2u); EXPECT_EQ(backend->getTotal(), 1u); EXPECT_EQ(clock.sleeps.size(), 1u); @@ -1160,7 +1099,7 @@ TEST(CASRequests, AmbiguousReplaceOfIdenticalBytesIsReissuedNotClaimedByByteEqua auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation seen = *orThrow(op.create("k", "B", Retry::standard()), "create"); + const Etag seen = *orThrow(op.create("k", "B", Retry::standard()), "create"); backend->resetCounts(); backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); @@ -1181,7 +1120,7 @@ TEST(CASRequests, AmbiguousReplaceWhoseResolveShowsAnotherIncarnationIsAConflict auto backend = std::make_shared(); auto requests = makeRequests(backend, clock); auto op = requests.admit(); - const Incarnation stale = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + const Etag stale = *orThrow(op.create("k", "v1", Retry::standard()), "create"); orThrow(op.replace("k", "theirs", stale, Retry::standard()), "the competitor's replace"); backend->resetCounts(); @@ -1194,7 +1133,7 @@ TEST(CASRequests, AmbiguousReplaceWhoseResolveShowsAnotherIncarnationIsAConflict const auto * occupant = std::get_if(&conflict->seen); ASSERT_NE(occupant, nullptr); EXPECT_EQ(occupant->bytes, "theirs"); - EXPECT_NE(occupant->incarnation, stale); + EXPECT_NE(occupant->etag, stale); EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getTotal(), 1u); /// The count the conflict reports is the count the transport saw, not a constant that happens to @@ -1229,7 +1168,7 @@ TEST(CASRequests, ReadModifyWriteDoesNotClaimACompetitorsIdenticalBytesAfterAnEa { /// The bytes we are about to send, under an incarnation that is not ours. if (const auto current = rival.read("k", Retry::once())) - (void)rival.replace("k", "B", current->incarnation, Retry::once()); + (void)rival.replace("k", "B", current->etag, Retry::once()); } ++staged; inside = false; @@ -1292,7 +1231,7 @@ TEST(CASRequests, ForEachListedKeyGivesEachPageItsOwnPolicyWindow) const uint64_t start = clock.now; /// A window that comfortably covers ONE page's refusal and its reissue, and could not have covered /// the walk: the policy governs each page, because a walk is an unbounded number of requests. - op.forEachListedKey("p/", [&](const KeyEntry &) { ++seen; return true; }, Retry::within(1'000), + op.forEachListedKey("p/", [&](const ListedKey &) { ++seen; return true; }, Retry::within(1'000), /*page_limit=*/10, [&] { ++pages; }); EXPECT_EQ(seen, 25u); EXPECT_EQ(pages, 3u); @@ -1310,7 +1249,7 @@ TEST(CASRequests, ForEachListedKeyThrowsRatherThanTruncateWhenAPageNeverArrives) for (int i = 0; i < 25; ++i) orThrow(op.create("p/" + std::to_string(i), "v", Retry::standard()), "create"); - const KeyPage first = op.list("p/", "", 10, Retry::within(1'000)); + const ListPage first = op.list("p/", "", 10, Retry::within(1'000)); ASSERT_FALSE(first.next_cursor.empty()); backend->always_refuse_cursor = first.next_cursor; /// the second page never arrives backend->refused_cursors.clear(); @@ -1321,7 +1260,7 @@ TEST(CASRequests, ForEachListedKeyThrowsRatherThanTruncateWhenAPageNeverArrives) /// reports the page it could not fetch instead of returning what it managed to read. expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { - op.forEachListedKey("p/", [&](const KeyEntry &) { ++seen; return true; }, Retry::within(1'000), + op.forEachListedKey("p/", [&](const ListedKey &) { ++seen; return true; }, Retry::within(1'000), /*page_limit=*/10, [&] { ++pages; }); }); EXPECT_EQ(pages, 1u); @@ -1395,7 +1334,7 @@ TEST(CASRequests, ReadModifyWriteLosesNoIncrementUnderContentionAndBoundsAHotKey return; inside_hook = true; if (const auto current = rival.read("ctr", Retry::once())) - (void)rival.replace("ctr", "999", current->incarnation, Retry::once()); + (void)rival.replace("ctr", "999", current->etag, Retry::once()); inside_hook = false; }); diff --git a/src/Disks/tests/gtest_cas_retirement_sweep.cpp b/src/Disks/tests/gtest_cas_retirement_sweep.cpp index 2c312c323772..36d9d6895bca 100644 --- a/src/Disks/tests/gtest_cas_retirement_sweep.cpp +++ b/src/Disks/tests/gtest_cas_retirement_sweep.cpp @@ -154,14 +154,21 @@ class UnresolvedPutBackend final : public InMemoryBackend /// GC's fence-out applied directly to the mount lease: preserve the body, set `gc_fenced`, bump `seq` /// (token-guarded). A subsequent `tryRemountOnce` then reclaims a fresh incarnation. +bool headExists(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + void fenceOutMount(Backend & backend, const String & mount_key) { - const auto got = backend.get(mount_key); + DB::Cas::tests::OperationForTest op(backend); + const auto got = (*op).read(mount_key, Retry::standard()); ASSERT_TRUE(got.has_value()); MountLease m = decodeMountLease(got->bytes); m.gc_fenced = true; m.seq += 1; - ASSERT_EQ(backend.putOverwrite(mount_key, encodeMountLease(m), got->token).outcome, PutOutcome::Done); + ASSERT_TRUE(std::holds_alternative((*op).replace(mount_key, encodeMountLease(m), got->etag, Retry::standard()))); } /// Publish one part `ref` with a single content blob whose payload is `payload`. @@ -188,11 +195,12 @@ ManifestId publishOneBlobPart(const PoolPtr & s, const RootNamespace & ns, const /// Every ref-log key of `ns` currently listed, in key order. std::set listRefLogKeys(Backend & b, const Layout & l, const RootNamespace & ns) { + DB::Cas::tests::OperationForTest op(b); std::set out; String cursor; while (true) { - const ListPage page = b.list(l.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000); + const ListPage page = (*op).list(l.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)), cursor, 1000, Retry::standard()); for (const ListedKey & k : page.keys) if (const auto parsed = l.parseRefObjectKey(k.key); parsed && parsed->kind == RefObjectKind::Log) out.insert(k.key); @@ -254,7 +262,7 @@ TEST(CASRetirementSweep, AHiddenRemovalStillReclaimsItsBlob) store->renewWatermarkOnce(); const String blob_key = layout.blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of(payload))}); - ASSERT_TRUE(backend->head(blob_key).exists); + ASSERT_TRUE(headExists(*backend, blob_key)); const std::set before_drop = listRefLogKeys(*backend, layout, ns); store->dropRef(ns, "part_a"); @@ -276,7 +284,7 @@ TEST(CASRetirementSweep, AHiddenRemovalStillReclaimsItsBlob) } ASSERT_TRUE(backend->holeServed()) << "the sabotage never fired"; - EXPECT_FALSE(backend->head(blob_key).exists) + EXPECT_FALSE(headExists(*backend, blob_key)) << "the removal was hidden from one enumeration and never folded -- the retention half of the " "skipped-transaction class, which arithmetic intake is supposed to close"; } @@ -332,7 +340,6 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS { CasRequestBudget budget; budget.attempt_timeout_ms = 100; - budget.operation_deadline_ms = 5000; budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); @@ -377,7 +384,7 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS const RefTxnId greatest = greatestLoggedId(*backend, layout, ns); ASSERT_EQ(greatest.writer_epoch, 1u); const RefTxnId straggler_slot{greatest.writer_epoch, greatest.ref_sequence + 1}; - ASSERT_FALSE(backend->head(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), straggler_slot)).exists) + ASSERT_FALSE(headExists(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), straggler_slot))) << "the slot must be empty before recovery -- otherwise this test proves nothing about who won"; /// Fence and remount. No wait: this is the case that used to cost 30 seconds. @@ -393,13 +400,15 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS /// landed, which is precisely the state that leaves a straggler outstanding. backend->fault_key_substr.clear(); EXPECT_EQ(store->listRefs(ns).size(), 1u); - ASSERT_TRUE(backend->head(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), straggler_slot)).exists) + ASSERT_TRUE(headExists(*backend, layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), straggler_slot))) << "recovery did not seal the dead epoch at the slot a straggler would take -- without that " "seal there is nothing for the straggler's create to lose to"; /// THE STRAGGLER ARRIVES. Its conditional create is refused, whenever it happens to land. - const PutResult put = backend->putIfAbsent(layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), straggler_slot), "ghost-body"); - EXPECT_EQ(put.outcome, PutOutcome::PreconditionFailed) + DB::Cas::tests::OperationForTest straggler_op(backend); + const WriteResult put = (*straggler_op).create( + layout.refLogKey(DB::Cas::tests::fixture::fixtureLife(ns), straggler_slot), "ghost-body", Retry::once()); + EXPECT_TRUE(std::holds_alternative(put)) << "the dying epoch's straggler overwrote (or joined) a slot the successor had already sealed"; } diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 6d0e9f2d8335..0370bea9afb2 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -49,6 +49,29 @@ namespace DB::ErrorCodes namespace { +/// ---- Small raw-fixture request-engine wrappers shared by the tests below ---- + +/// The durable object at `key`, or `nullopt`. +std::optional readAt(DB::Cas::Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).read(key, DB::Cas::Retry::once()); +} + +/// Unconditional create of a fresh key (the fixture's own setup, never a real conflict). +void createAt(DB::Cas::Backend & backend, const String & key, const String & bytes) +{ + DB::Cas::tests::OperationForTest op(backend); + EXPECT_TRUE(std::holds_alternative((*op).create(key, bytes, DB::Cas::Retry::once()))); +} + +/// The current metadata at `key`, or `nullopt`. +std::optional headAt(DB::Cas::Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, DB::Cas::Retry::once()); +} + /// Build a `Poco::Util::XMLConfiguration` with `inner_xml` nested under a `` element (mirrors /// the shape a real CAS disk config has under `storage_configuration.disks.`, so /// `config_prefix = "disk"` reads exactly like the disk factory's `config_prefix`). @@ -165,16 +188,15 @@ class RecordingStagingBackend : public DB::Cas::InMemoryBackend DB::Cas::InMemoryBackend::publish(request, access); } - /// Every key read as a stream, with a count. Republishing opens its source with `getStream`, so + /// Every key read as a stream, with a count. Republishing opens its source with `stream`, so /// this counts exactly those reads -- and deliberately not the materializing - /// `get`, which the assertions themselves use to inspect bodies. + /// `read`, which the assertions themselves use to inspect bodies. std::map reads_of; - using DB::Cas::InMemoryBackend::getStream; - std::optional getStream(const String & key, DB::Cas::Range range) override + std::unique_ptr stream(const String & key, DB::Cas::TransportAccess & access) override { ++reads_of[key]; - return DB::Cas::InMemoryBackend::getStream(key, range); + return DB::Cas::InMemoryBackend::stream(key, access); } @@ -202,13 +224,6 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend explicit EtagFaithfulPublicationBackend(FaultScript script_) : script(script_) {} - /// Unhide the legacy convenience overloads that the primitive overrides below would otherwise - /// hide: `head(key)` and `deleteExact(key, token)` are inherited UNCHANGED (Backend's own default - /// implementations), so a caller through either legacy name still reaches the ETag-faithful - /// primitives below by virtual dispatch -- which is what the production sites this double - /// instruments actually call now. - using DB::Cas::Backend::head; - std::optional head(const String & key, DB::Cas::TransportAccess & access) override { std::optional result = DB::Cas::InMemoryBackend::head(key, access); @@ -250,10 +265,20 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend { fault_fired = true; DB::Cas::InMemoryBackend::publish(request, access); - queued_delete_token = head(request.destination_key).token; + queued_delete_token = head(request.destination_key, access)->value; + /// The test wants to replay this exact captured value later as a delete precondition, to + /// prove a retag defeats it. `Etag` is never constructible from a raw string, so the only + /// way to hold a replayable one is to mint it -- through a nested admitted operation, over + /// this same backend instance -- at the exact moment the raw value above was observed. + { + DB::Cas::tests::OperationForTest mint_op(*this); + const auto meta = (*mint_op).head(request.destination_key, DB::Cas::Retry::once()); + if (meta) + queued_delete_incarnation = meta->etag; + } if (script != FaultScript::CopyLandsThenCondemned) - first_delete = deleteExact(request.destination_key, queued_delete_token); + first_delete = remove(request.destination_key, queued_delete_token, access); throw Poco::TimeoutException("ETag-faithful staged publication response lost"); } @@ -265,8 +290,9 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend bool fault_fired = false; size_t copy_publications = 0; size_t streaming_publications = 0; - DB::Cas::Token queued_delete_token; - DB::Cas::DeleteOutcome first_delete; + String queued_delete_token; + std::optional queued_delete_incarnation; + DB::Cas::Backend::RawRemoval first_delete{}; private: static bool isBlobBodyKey(const String & key) @@ -309,12 +335,13 @@ DB::Cas::BlobSource reReadableStagedSource( source.server_side_copy_from = staging_key; source.open = [backend, staging_key, header_len]() -> std::unique_ptr { - auto staged = backend->getStream(staging_key); + DB::Cas::tests::OperationForTest op(backend); + auto staged = (*op).stream(staging_key, DB::Cas::Retry::standard()); if (!staged) throw DB::Exception(DB::ErrorCodes::FILE_DOESNT_EXIST, "staging object {} is absent", staging_key); String encoded_header(header_len, '\0'); - staged->stream->readStrict(encoded_header.data(), encoded_header.size()); + staged->readStrict(encoded_header.data(), encoded_header.size()); const DB::Cas::EnvelopeHeader decoded = DB::Cas::decodeEnvelopeHeader(encoded_header, encoded_header.size(), DB::Cas::ObjectKind::Blob); if (decoded.header_len != header_len) @@ -324,7 +351,7 @@ DB::Cas::BlobSource reReadableStagedSource( staging_key, decoded.header_len, header_len); - return std::move(staged->stream); + return staged; }; return source; } @@ -349,7 +376,7 @@ TEST(CASS3Staging, StagedCopyCondemnedRetryRetagsBeforeQueuedDelete) const DB::Cas::BlobRef ref = DB::Cas::tests::idOf(payload); const String staging_key = "p/staging/mount1/etag-condemned.tmp"; const String staging_bytes = stagedBytes(store->poolMeta().blob_header_len, payload, DB::UInt128{101}); - backend->putIfAbsent(staging_key, staging_bytes); + createAt(*backend, staging_key, staging_bytes); DB::Cas::tests::writeMetaClean(*backend, store->layout(), DB::Cas::tests::u128Of(payload), payload.size()); DB::Cas::tests::condemnMeta(*backend, store->layout(), DB::Cas::tests::u128Of(payload), 31); auto build = precommittedBuildFor( @@ -362,10 +389,13 @@ TEST(CASS3Staging, StagedCopyCondemnedRetryRetagsBeforeQueuedDelete) EXPECT_EQ(backend->copy_publications, 1u); EXPECT_EQ(backend->streaming_publications, 1u); - EXPECT_EQ( - backend->deleteExact(store->layout().blobKey(ref), backend->queued_delete_token).kind, - DB::Cas::DeleteOutcome::Kind::TokenMismatch); - const auto current = backend->get(store->layout().blobKey(ref)); + ASSERT_TRUE(backend->queued_delete_incarnation.has_value()); + { + DB::Cas::tests::OperationForTest op(*backend); + EXPECT_EQ((*op).remove(store->layout().blobKey(ref), *backend->queued_delete_incarnation, DB::Cas::Retry::once()), + DB::Cas::Removal::Mismatch); + } + const auto current = readAt(*backend, store->layout().blobKey(ref)); ASSERT_TRUE(current.has_value()); EXPECT_NE(current->bytes, staging_bytes); EXPECT_EQ(current->bytes.substr(store->poolMeta().blob_header_len), payload); @@ -381,7 +411,7 @@ TEST(CASS3Staging, StagedCopyDeletedBeforeAbsentRetryRetagsBeforeQueuedDelete) const DB::Cas::BlobRef ref = DB::Cas::tests::idOf(payload); const String staging_key = "p/staging/mount1/etag-deleted.tmp"; const String staging_bytes = stagedBytes(store->poolMeta().blob_header_len, payload, DB::UInt128{202}); - backend->putIfAbsent(staging_key, staging_bytes); + createAt(*backend, staging_key, staging_bytes); auto build = precommittedBuildFor( store, DB::Cas::RootNamespace{"srv1/etag-deleted"}, "part", DB::Cas::tests::u128Of(payload), payload.size()); @@ -390,15 +420,18 @@ TEST(CASS3Staging, StagedCopyDeletedBeforeAbsentRetryRetagsBeforeQueuedDelete) ref, reReadableStagedSource(backend, staging_key, payload.size(), store->poolMeta().blob_header_len)); - EXPECT_EQ(backend->first_delete.kind, DB::Cas::DeleteOutcome::Kind::Deleted); + EXPECT_EQ(backend->first_delete, DB::Cas::Backend::RawRemoval::Removed); EXPECT_EQ(backend->copy_publications, 1u) << "the absent retry must not copy the original staged envelope again"; EXPECT_EQ(backend->streaming_publications, 1u); - EXPECT_EQ( - backend->deleteExact(store->layout().blobKey(ref), backend->queued_delete_token).kind, - DB::Cas::DeleteOutcome::Kind::TokenMismatch) - << "the second queued exact delete for the copied ETag must miss the retagged replacement"; - const auto current = backend->get(store->layout().blobKey(ref)); + ASSERT_TRUE(backend->queued_delete_incarnation.has_value()); + { + DB::Cas::tests::OperationForTest op(*backend); + EXPECT_EQ((*op).remove(store->layout().blobKey(ref), *backend->queued_delete_incarnation, DB::Cas::Retry::once()), + DB::Cas::Removal::Mismatch) + << "the second queued exact delete for the copied ETag must miss the retagged replacement"; + } + const auto current = readAt(*backend, store->layout().blobKey(ref)); ASSERT_TRUE(current.has_value()); EXPECT_NE(current->bytes, staging_bytes); EXPECT_EQ(current->bytes.substr(store->poolMeta().blob_header_len), payload); @@ -414,11 +447,17 @@ TEST(CASS3Staging, FirstCondemnedAttemptThenAbsentRetryNeverRecopies) const DB::Cas::BlobRef ref = DB::Cas::tests::idOf(payload); const String staging_key = "p/staging/mount1/etag-first-condemned.tmp"; const String staging_bytes = stagedBytes(store->poolMeta().blob_header_len, payload, DB::UInt128{303}); - backend->putIfAbsent(staging_key, staging_bytes); - backend->putIfAbsent(store->layout().blobKey(ref), staging_bytes); + createAt(*backend, staging_key, staging_bytes); + createAt(*backend, store->layout().blobKey(ref), staging_bytes); DB::Cas::tests::writeMetaClean(*backend, store->layout(), DB::Cas::tests::u128Of(payload), payload.size()); DB::Cas::tests::condemnMeta(*backend, store->layout(), DB::Cas::tests::u128Of(payload), 37); - const DB::Cas::Token original_staged_etag = backend->head(store->layout().blobKey(ref)).token; + /// Captured through a real admitted operation, so it is a genuinely replayable `Etag` -- never + /// constructible from a bare raw value -- for the later mismatch check below. + DB::Cas::Etag original_staged_etag = [&] + { + DB::Cas::tests::OperationForTest op(*backend); + return (*op).head(store->layout().blobKey(ref), DB::Cas::Retry::once())->etag; + }(); auto build = precommittedBuildFor( store, DB::Cas::RootNamespace{"srv1/etag-first-condemned"}, "part", DB::Cas::tests::u128Of(payload), payload.size()); @@ -427,14 +466,16 @@ TEST(CASS3Staging, FirstCondemnedAttemptThenAbsentRetryNeverRecopies) ref, reReadableStagedSource(backend, staging_key, payload.size(), store->poolMeta().blob_header_len)); - EXPECT_EQ(backend->first_delete.kind, DB::Cas::DeleteOutcome::Kind::Deleted); + EXPECT_EQ(backend->first_delete, DB::Cas::Backend::RawRemoval::Removed); EXPECT_EQ(backend->copy_publications, 0u) << "a first condemned publication and every later absent retry must stream, never copy"; EXPECT_EQ(backend->streaming_publications, 2u); - EXPECT_EQ( - backend->deleteExact(store->layout().blobKey(ref), original_staged_etag).kind, - DB::Cas::DeleteOutcome::Kind::TokenMismatch); - const auto current = backend->get(store->layout().blobKey(ref)); + { + DB::Cas::tests::OperationForTest op(*backend); + EXPECT_EQ((*op).remove(store->layout().blobKey(ref), original_staged_etag, DB::Cas::Retry::once()), + DB::Cas::Removal::Mismatch); + } + const auto current = readAt(*backend, store->layout().blobKey(ref)); ASSERT_TRUE(current.has_value()); EXPECT_NE(current->bytes, staging_bytes); EXPECT_EQ(current->bytes.substr(store->poolMeta().blob_header_len), payload); @@ -611,7 +652,7 @@ TEST(CASS3Staging, PromoteViaServerSideCopyCreatesFreshBlobMaterializedProof) const std::string staging_key = "p/staging/mount1/aaa.tmp"; const std::string staging_bytes = stagedBytes( store->poolMeta().blob_header_len, payload, DB::UInt128{0xA}); - backend->putIfAbsent(staging_key, staging_bytes); + createAt(*backend, staging_key, staging_bytes); auto build = precommittedBuildFor(store, ns, ref, hash, payload.size()); const DB::Cas::PutBlobResult bref = build->putBlob( @@ -627,13 +668,13 @@ TEST(CASS3Staging, PromoteViaServerSideCopyCreatesFreshBlobMaterializedProof) /// Successful publication records materialized evidence; the backend still owns the destination token. EXPECT_EQ(build->dependencyProof(blob_id), DB::Cas::BlobDependencyProof::Materialized); - const DB::Cas::HeadResult hr = backend->head(blob_key); - ASSERT_TRUE(hr.exists); - EXPECT_FALSE(hr.token.empty()); + const auto hr = headAt(*backend, blob_key); + ASSERT_TRUE(hr.has_value()); + EXPECT_FALSE(DB::Cas::PersistedEtag::capture(hr->etag).value.empty()); EXPECT_EQ(bref.size, payload.size()); /// The promoted blob body IS the staging bytes (server-side copy moved them verbatim). - const auto got = backend->get(blob_key); + const auto got = readAt(*backend, blob_key); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes, staging_bytes); } @@ -651,17 +692,19 @@ TEST(CASS3Staging, PromoteOverExistingCleanBlobAdoptsAndNeverOverwrites) const DB::Cas::BlobRef blob_id{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(hash)}; const std::string blob_key = store->layout().blobKey(blob_id); const std::string staging_key = "p/staging/mount1/bbb.tmp"; - backend->putIfAbsent( + createAt( + *backend, staging_key, stagedBytes(store->poolMeta().blob_header_len, payload, DB::UInt128{0xB})); /// A pre-existing, well-formed, CLEAN blob (envelope + payload) already at the content key. - backend->putIfAbsent( + createAt( + *backend, blob_key, stagedBytes(store->poolMeta().blob_header_len, payload, DB::UInt128{0xBB})); DB::Cas::tests::writeMetaClean(*backend, store->layout(), hash, payload.size()); - const DB::Cas::HeadResult before = backend->head(blob_key); - ASSERT_TRUE(before.exists); + const auto before = headAt(*backend, blob_key); + ASSERT_TRUE(before.has_value()); auto build = precommittedBuildFor(store, ns, ref, hash, payload.size()); build->putBlob( @@ -673,8 +716,9 @@ TEST(CASS3Staging, PromoteOverExistingCleanBlobAdoptsAndNeverOverwrites) EXPECT_EQ(backend->streamingPublicationCount(), 0u); /// The existing incarnation is untouched: same token, same bytes. - const DB::Cas::HeadResult after = backend->head(blob_key); - EXPECT_EQ(after.token, before.token); + const auto after = headAt(*backend, blob_key); + ASSERT_TRUE(after.has_value()); + EXPECT_EQ(after->etag, before->etag); /// Observing the existing incarnation records materialized evidence without retaining its token. EXPECT_EQ(build->dependencyProof(blob_id), DB::Cas::BlobDependencyProof::Materialized); @@ -707,16 +751,16 @@ TEST(CASS3Staging, PublishOverCondemnedBlobUsesFreshTagNotVerbatim) staging_h, static_cast(store->poolMeta().blob_header_len)); ASSERT_EQ(staging_header.size(), store->poolMeta().blob_header_len); const std::string staging_bytes = staging_header + payload; - backend->putIfAbsent(staging_key, staging_bytes); + createAt(*backend, staging_key, staging_bytes); /// Seed the condemned blob body = EXACTLY what a verbatim promote of this staging object would have /// produced (the writer's OWN create, later observed condemned). This is the adversarial shape: a /// verbatim republication WOULD reproduce these identical bytes ⇒ identical ETag ⇒ collision. - backend->putIfAbsent(blob_key, staging_bytes); + createAt(*backend, blob_key, staging_bytes); DB::Cas::tests::writeMetaClean(*backend, store->layout(), hash, /*size=*/payload.size()); DB::Cas::tests::condemnMeta(*backend, store->layout(), hash, /*condemn_round=*/5); - const DB::Cas::HeadResult before = backend->head(blob_key); - ASSERT_TRUE(before.exists); + const auto before = headAt(*backend, blob_key); + ASSERT_TRUE(before.has_value()); auto build = precommittedBuildFor(store, ns, ref, hash, payload.size()); build->putBlob( @@ -735,11 +779,11 @@ TEST(CASS3Staging, PublishOverCondemnedBlobUsesFreshTagNotVerbatim) EXPECT_EQ(backend->streamingPublicationCount(), 1u); /// The incarnation token is REFRESHED (a fresh incarnation displaced the condemned one). - const DB::Cas::HeadResult after = backend->head(blob_key); - EXPECT_NE(after.token, before.token); - ASSERT_TRUE(after.exists); + const auto after = headAt(*backend, blob_key); + ASSERT_TRUE(after.has_value()); + EXPECT_NE(after->etag, before->etag); - const auto got = backend->get(blob_key); + const auto got = readAt(*backend, blob_key); ASSERT_TRUE(got.has_value()); const uint64_t header_len = store->poolMeta().blob_header_len; @@ -941,7 +985,7 @@ namespace /// A `LocalObjectStorage` that reports the GCS generation dialect /// (`conditionalOpsUseGenerationTokens() == true`) and a non-`Local` `getType()`, so /// `ContentAddressedMetadataStorage::openPoolView` builds its backend in `Mode::Native` with -/// `native_token_type == TokenType::Generation`. The fake also advertises native copy so generation +/// `native_token_type == Dialect::Generation`. The fake also advertises native copy so generation /// token mode can exercise explicit S3 staging without endpoint/provider heuristics. /// /// Holds every object entirely in memory, keyed by the BARE CAS key exactly as `Backend` hands it to @@ -1141,7 +1185,7 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage /// Checks the write-once/exact-token precondition against the current generation and, on success, /// stores `bytes` and mints the next generation. Throws an `S3Exception` naming `PreconditionFailed` /// on a lost condition -- the one signal `finalizeConditionalWrite` classifies as - /// `PutOutcome::PreconditionFailed` rather than an ordinary failure. + /// `ConditionalWriteOutcome::PreconditionLost` rather than an ordinary failure. /// Returns the generation it minted, the way a real store returns it in the write response: the /// backend attributes the write to that generation and nothing reads it back. uint64_t commitConditionalWrite(const std::string & key, const std::string & bytes, diff --git a/src/Disks/tests/gtest_cas_sentinel_probe.cpp b/src/Disks/tests/gtest_cas_sentinel_probe.cpp index 380343aa195f..201087cdb1c1 100644 --- a/src/Disks/tests/gtest_cas_sentinel_probe.cpp +++ b/src/Disks/tests/gtest_cas_sentinel_probe.cpp @@ -58,10 +58,6 @@ CasRequests makeRequests(Backend & backend, DB::Cas::tests::FakeClock * clock = class TransportFaultBackend final : public InMemoryBackend { public: - using Backend::getStream; - using Backend::head; - using Backend::list; - std::optional head(const String & key, TransportAccess & access) override { if (fail.load()) @@ -92,10 +88,10 @@ class TransportFaultBackend final : public InMemoryBackend TEST(CASSentinelProbe, PresentKeyReturnsPresentWithBody) { InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent("k", "hello").outcome, PutOutcome::Done); - auto requests = makeRequests(backend); auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("k", "hello", Retry::once()))); + const auto result = probeSentinel(op, "k", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::Present); ASSERT_TRUE(result.body.has_value()); @@ -106,10 +102,10 @@ TEST(CASSentinelProbe, PresentKeyReturnsPresentWithBody) TEST(CASSentinelProbe, AbsentKeyWithContainerAliveReturnsKeyAbsent) { InMemoryBackend backend; - ASSERT_EQ(backend.putIfAbsent("other", "x").outcome, PutOutcome::Done); // proves the backend is alive - auto requests = makeRequests(backend); auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("other", "x", Retry::once()))); // proves the backend is alive + const auto result = probeSentinel(op, "missing", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::KeyAbsent); EXPECT_FALSE(result.body.has_value()); @@ -126,10 +122,9 @@ TEST(CASSentinelProbe, ContainerDirectoryRemovedReturnsContainerAbsent) auto storage = tests::makeLocalObjectStorageForTest(); ObjectStorageBackend backend(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); - ASSERT_EQ(backend.putIfAbsent("k", "hello").outcome, PutOutcome::Done); - auto requests = makeRequests(backend); auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("k", "hello", Retry::once()))); /// Sanity, container alive: Present vs. KeyAbsent are genuinely distinct before we remove anything. EXPECT_EQ(probeSentinel(op, "k", Retry::standard()).outcome, ProbeOutcome::Present); diff --git a/src/Disks/tests/gtest_cas_shutdown_context.cpp b/src/Disks/tests/gtest_cas_shutdown_context.cpp index 5884d8b47860..b1fd2ba48d6f 100644 --- a/src/Disks/tests/gtest_cas_shutdown_context.cpp +++ b/src/Disks/tests/gtest_cas_shutdown_context.cpp @@ -88,7 +88,8 @@ void emitTestEvent(DB::ContentAddressedMetadataStorage & storage) /// A failed ref-lane drain must not leave a clean-release marker behind. That marker lets a /// successor skip the observation window, so a phase-2 failure must leave it absent. - const auto mount = backend->get(Layout(config.pool_prefix).mountKey(config.server_root_id)); + DB::Cas::tests::OperationForTest op(backend); + const auto mount = (*op).read(Layout(config.pool_prefix).mountKey(config.server_root_id), Retry::standard()); const bool clean_release = mount && decodeMountLease(mount->bytes).min_active_build_sequence == std::numeric_limits::max(); const bool marker_must_be_absent = phase == 2; diff --git a/src/Disks/tests/gtest_cas_slot_occupy.cpp b/src/Disks/tests/gtest_cas_slot_occupy.cpp index 901b155df99c..1613d350520e 100644 --- a/src/Disks/tests/gtest_cas_slot_occupy.cpp +++ b/src/Disks/tests/gtest_cas_slot_occupy.cpp @@ -133,7 +133,7 @@ TEST(CASSlotOccupy, PreExistingKeyConflictsWithExactBytesAndIncarnationInTwoRequ const WriteResult seeded = seeder.create("k", "occupant-bytes", Retry::once()); const auto * seeded_committed = std::get_if(&seeded); ASSERT_TRUE(seeded_committed != nullptr); - const Incarnation seeded_incarnation = seeded_committed->incarnation; + const Etag seeded_incarnation = seeded_committed->etag; backend->resetCounts(); CasOperation op = requests.admit(); @@ -141,7 +141,7 @@ TEST(CASSlotOccupy, PreExistingKeyConflictsWithExactBytesAndIncarnationInTwoRequ const Object * occupant = conflictObject(result); ASSERT_TRUE(occupant != nullptr) << "the settling read must have named the occupant"; EXPECT_EQ(occupant->bytes, "occupant-bytes"); - EXPECT_EQ(occupant->incarnation, seeded_incarnation); + EXPECT_EQ(occupant->etag, seeded_incarnation); EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 1u); @@ -341,7 +341,7 @@ TEST(CASSlotOccupy, OwnLandedAmbiguousWriteIsObservedOnTheNextAttempt) const auto * committed = std::get_if(&first); ASSERT_TRUE(committed != nullptr) << "the write landed; the settling read proves it"; EXPECT_TRUE(committed->resolved_by_read); - const Incarnation landed_incarnation = committed->incarnation; + const Etag landed_incarnation = committed->etag; EXPECT_EQ(backend->writeTotal(), 1u); EXPECT_EQ(backend->getCount("k"), 1u); @@ -350,7 +350,7 @@ TEST(CASSlotOccupy, OwnLandedAmbiguousWriteIsObservedOnTheNextAttempt) const Object * occupant = conflictObject(second); ASSERT_TRUE(occupant != nullptr); EXPECT_EQ(occupant->bytes, "my-bytes"); - EXPECT_EQ(occupant->incarnation, landed_incarnation) << "both calls must observe the SAME landed incarnation"; + EXPECT_EQ(occupant->etag, landed_incarnation) << "both calls must observe the SAME landed incarnation"; EXPECT_EQ(backend->writeTotal(), 2u); EXPECT_EQ(backend->getCount("k"), 2u); } diff --git a/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp b/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp index 71f936c7a3c7..80b7e7eaddfa 100644 --- a/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp +++ b/src/Disks/tests/gtest_cas_sweep_deletion_premise.cpp @@ -65,7 +65,11 @@ struct OrphanFixture } String orphanKey() const { return store->layout().manifestKey(ManifestId{ns, orphan}); } - bool orphanExists() const { return backend->head(orphanKey()).exists; } + bool orphanExists() const + { + OperationForTest op(*backend); + return (*op).head(orphanKey(), Retry::once()).has_value(); + } }; /// The same admissible orphan shape as `OrphanFixture`, but without its legal manifest write: the @@ -85,7 +89,11 @@ struct UndecodableOrphanFixture } String orphanKey() const { return store->layout().manifestKey(ManifestId{ns, orphan}); } - bool orphanExists() const { return backend->head(orphanKey()).exists; } + bool orphanExists() const + { + OperationForTest op(*backend); + return (*op).head(orphanKey(), Retry::once()).has_value(); + } }; } @@ -239,10 +247,11 @@ TEST(CASSweepDeletionPremise, AnUndecodableManifestDoesNotWedgeTheCursorPage) const size_t at = bytes.find("==> \"a.txt\""); ASSERT_NE(at, String::npos) << "no banner line to corrupt -- the entry must be Inline, not Blob"; bytes[at + 5] = 'X'; /// Inside the quoted path, same length, so no other offset shifts. - const PutResult put = f.backend->putIfAbsent(f.orphanKey(), sealObject(FormatId::PartManifest, bytes)); - /// `putIfAbsent` over an existing key writes nothing and reports `PreconditionFailed`, so a - /// silently legal body would make every assertion below pass against the wrong object. - ASSERT_EQ(put.outcome, PutOutcome::Done) << "the poison body was not the one planted"; + OperationForTest poison_op(*f.backend); + const WriteResult put = (*poison_op).create(f.orphanKey(), sealObject(FormatId::PartManifest, bytes), Retry::once()); + /// `create` over an existing key writes nothing and reports a `Conflict`, so a silently legal body + /// would make every assertion below pass against the wrong object. + ASSERT_TRUE(std::holds_alternative(put)) << "the poison body was not the one planted"; const ManifestId legal = writeManifestRaw( *f.backend, f.store->layout(), f.ns, ref(5, 0xCD), {blobEntryFor("b", DB::UInt128(2))}); @@ -262,7 +271,7 @@ TEST(CASSweepDeletionPremise, AnUndecodableManifestDoesNotWedgeTheCursorPage) /// keys remain, so a moved-cursor assertion fails after a correct fix, not before it. EXPECT_TRUE(result.wrapped); /// And the strong form: the object beyond the poison key was still decided this page. - EXPECT_FALSE(f.backend->head(legal_key).exists) + EXPECT_FALSE((*poison_op).head(legal_key, Retry::once()).has_value()) << "the sweep stopped at the poison key instead of walking past it"; } @@ -377,8 +386,9 @@ TEST(CASSweepDeletionPremise, AnExhaustedDeleteBudgetRetainsAndDoesNotStepOverTh << "the cursor must not have stepped over the candidates the exhausted budget left undecided"; size_t surviving = 0; + OperationForTest survive_op(*f.backend); for (const ManifestRef & r : {f.orphan, second, third}) - if (f.backend->head(f.store->layout().manifestKey(ManifestId{f.ns, r})).exists) + if ((*survive_op).head(f.store->layout().manifestKey(ManifestId{f.ns, r}), Retry::once()).has_value()) ++surviving; EXPECT_EQ(surviving, 0u); } @@ -457,8 +467,9 @@ TEST(CASSweepDeletionPremise, RecoveryWorkBudgetRetainsAndConvergesWithoutWedgin EXPECT_EQ(total_skipped, static_cast(kCandidates)) << "every one of the six candidates was decided (skipped), none silently dropped from the page"; EXPECT_GE(total_retained_work_budget, 1u); + OperationForTest survive_op(*backend); for (int i = 1; i <= kCandidates; ++i) - EXPECT_TRUE(backend->head(layout.manifestKey(ManifestId{ns, ref(i, 1)})).exists) + EXPECT_TRUE((*survive_op).head(layout.manifestKey(ManifestId{ns, ref(i, 1)}), Retry::once()).has_value()) << "candidate " << i << " must survive: it was never proven safe to delete"; } @@ -497,8 +508,9 @@ TEST(CASSweepDeletionPremise, NamespaceWorkBudgetCapsDistinctViewsPerPage) EXPECT_EQ(budget.sweep_namespaces_used, 1u); size_t surviving = 0; + OperationForTest survive_op(*backend); for (const auto & p : std::vector>{{ns_a, ref_a}, {ns_b, ref_b}}) - if (backend->head(layout.manifestKey(ManifestId{p.first, p.second})).exists) + if ((*survive_op).head(layout.manifestKey(ManifestId{p.first, p.second}), Retry::once()).has_value()) ++surviving; EXPECT_EQ(surviving, 1u) << "exactly one candidate remains -- the one whose namespace had no budget left"; } diff --git a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp index fefed9949677..f155ca9a35ff 100644 --- a/src/Disks/tests/gtest_cas_truncate_reclaim.cpp +++ b/src/Disks/tests/gtest_cas_truncate_reclaim.cpp @@ -77,7 +77,7 @@ bool anyRetiredPending(const PoolPtr & s) { /// Condemned state rides the adopted fold seal's RunMarker::Condemned rows, not a /// separate retired list — reconstruct the in-flight set from the seal. - return DB::Cas::tests::anyCondemnedInSeal(s->backend(), s->layout()); + return DB::Cas::tests::anyCondemnedInSeal(*s->poolBackendPtr(), s->layout()); } /// Run regular GC rounds until a fixpoint over the ACK-FLOOR round. A condemned blob is deleted only a @@ -276,7 +276,7 @@ TEST(CASTruncateReclaim, DropNamespaceLeavesSharedBlobDebrisForPerpetualSweep) << "an emptied pool must drain instead of standing still; the sweep owned these blobs and " "reclaimed them within " << rounds << " GC rounds"; EXPECT_EQ(after.reachable, 0u); - DB::Cas::CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(s->backend()); + DB::Cas::CasRequests catalog_requests = DB::Cas::tests::openRequestsForTest(s->poolBackendPtr()); DB::Cas::CasOperation catalog_op = catalog_requests.admit(); EXPECT_FALSE(CasRefCatalog::lifeIfCataloged(catalog_op, s->layout(), ns)) << "physical debris must not keep the logical namespace life cataloged"; diff --git a/src/Disks/tests/gtest_cas_upload_detached.cpp b/src/Disks/tests/gtest_cas_upload_detached.cpp index 999bab3f400e..89ced775f08b 100644 --- a/src/Disks/tests/gtest_cas_upload_detached.cpp +++ b/src/Disks/tests/gtest_cas_upload_detached.cpp @@ -49,13 +49,14 @@ BlobSource reReadableStagedSource( source.server_side_copy_from = staging_key; source.open = [backend, staging_key, header_len, payload_size]() -> std::unique_ptr { - auto staged = backend->getStream(staging_key); + DB::Cas::tests::OperationForTest op(backend); + std::unique_ptr staged = (*op).stream(staging_key, Retry::once()); if (!staged) throw DB::Exception(DB::ErrorCodes::FILE_DOESNT_EXIST, "staging object {} is absent", staging_key); String encoded_header(header_len, '\0'); - staged->stream->readStrict(encoded_header.data(), encoded_header.size()); + staged->readStrict(encoded_header.data(), encoded_header.size()); (void)decodeEnvelopeHeader(encoded_header, header_len + payload_size, ObjectKind::Blob); - return std::move(staged->stream); + return staged; }; return source; } @@ -72,6 +73,27 @@ PartWriteTxnPtr precommitBuildFor( return build; } +/// A one-shot `create`, asserting it committed (mirrors the retired `backend.putIfAbsent(key, bytes)`). +void createObj(Backend & backend, const String & key, const String & bytes) +{ + DB::Cas::tests::OperationForTest op(backend); + ASSERT_TRUE(std::holds_alternative((*op).create(key, bytes, Retry::once()))); +} + +/// An exact read (mirrors the retired `backend.get(key)`). +std::optional readObj(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).read(key, Retry::standard()); +} + +/// Whether `key` has a value, through a HEAD (mirrors the retired `backend.head(key).exists`). +bool headPresent(Backend & backend, const String & key) +{ + DB::Cas::tests::OperationForTest op(backend); + return (*op).head(key, Retry::standard()).has_value(); +} + /// Seed a present, well-formed blob body whose LOGICAL bytes are exactly `payload` (a fixed envelope /// header followed by the payload), so a later HEAD returns a token and a logical size of `payload.size()`. void seedPresentBody( @@ -82,13 +104,13 @@ void seedPresentBody( h.incarnation_tag = DB::UInt128(0xABCD); h.build_id = DB::UInt128(0x1111); const String head = encodeEnvelopeHeader(h, static_cast(pm.blob_header_len)); - b.putIfAbsent(layout.blobKey(ref), head + payload); + createObj(b, layout.blobKey(ref), head + payload); } /// The logical payload stored at `key` (object body minus the fixed blob header), or empty when absent. String logicalPayloadAt(InMemoryBackend & b, const String & key, uint64_t header_len) { - const auto got = b.get(key); + const auto got = readObj(b, key); if (!got || got->bytes.size() < header_len) return {}; return got->bytes.substr(header_len); @@ -269,7 +291,8 @@ TEST(CASUploadDetached, PresentCondemnedPublishesFreshAndQueuedOldDeleteMisses) condemnMeta(*backend, store->layout(), u128Of(payload), 19); auto build = precommitBuildFor(store, RootNamespace{"srv1/protocol-condemned"}, "part", payload); const String blob_key = store->layout().blobKey(ref); - const Token condemned_token = backend->head(blob_key).token; + DB::Cas::tests::OperationForTest condemned_probe(*backend); + const Etag condemned_token = (*condemned_probe).head(blob_key, Retry::standard())->etag; backend->watch(blob_key, store->layout().blobMetaKey(ref)); const uint64_t avoided_before = ProfileEvents::global_counters[ProfileEvents::CASBlobBodyPutAvoided].load(); @@ -282,8 +305,11 @@ TEST(CASUploadDetached, PresentCondemnedPublishesFreshAndQueuedOldDeleteMisses) EXPECT_EQ(backend->blob_heads, 1u); EXPECT_EQ(backend->publish_calls, 1u); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASBlobBodyPutAvoided].load(), avoided_before); - EXPECT_EQ(backend->deleteExact(blob_key, condemned_token).kind, DeleteOutcome::Kind::TokenMismatch); - EXPECT_TRUE(backend->head(blob_key).exists); + { + DB::Cas::tests::OperationForTest op(*backend); + EXPECT_EQ((*op).remove(blob_key, condemned_token, Retry::once()), Removal::Mismatch); + EXPECT_TRUE((*op).head(blob_key, Retry::standard()).has_value()); + } } /// A present body with absent metadata is observed and backfilled `Clean` without publication. @@ -358,7 +384,7 @@ TEST(CASUploadDetached, FreshLocalStreaming) arrange(b1, s1, build1); const String key = s1->layout().blobKey(blob); - ASSERT_FALSE(b1->head(key).exists); /// precondition: absent + ASSERT_FALSE(headPresent(*b1, key)); /// precondition: absent EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); const BlobUploadResult r = build1->uploadBlobDetached( @@ -372,7 +398,7 @@ TEST(CASUploadDetached, FreshLocalStreaming) EXPECT_EQ(r.dep.size, payload.size()); EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); - EXPECT_TRUE(b1->head(key).exists); + EXPECT_TRUE(headPresent(*b1, key)); EXPECT_EQ(logicalPayloadAt(*b1, key, s1->poolMeta().blob_header_len), payload); EXPECT_EQ(metaStateAt(*b1, s1->layout(), payload), std::optional(MetaState::Clean)); @@ -408,7 +434,7 @@ TEST(CASUploadDetached, S3StagingPromotion) h.kind = ObjectKind::Blob; h.incarnation_tag = DB::UInt128(0xC0FFEE); staging_bytes = encodeEnvelopeHeader(h, static_cast(s->poolMeta().blob_header_len)) + payload; - b->putIfAbsent(staging_key, staging_bytes); + createObj(*b, staging_key, staging_bytes); build = precommitBuildFor(s, ns, ref_name, payload); }; @@ -419,7 +445,7 @@ TEST(CASUploadDetached, S3StagingPromotion) arrange(b1, s1, build1, staging_bytes1); const String key = s1->layout().blobKey(blob); - ASSERT_FALSE(b1->head(key).exists); + ASSERT_FALSE(headPresent(*b1, key)); EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); const BlobUploadResult r = build1->uploadBlobDetached( @@ -435,9 +461,9 @@ TEST(CASUploadDetached, S3StagingPromotion) EXPECT_EQ(r.dep.size, payload.size()); EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); - ASSERT_TRUE(b1->head(key).exists); + ASSERT_TRUE(headPresent(*b1, key)); /// The server-side copy moved the staging bytes verbatim to the blob key. - const auto got = b1->get(key); + const auto got = readObj(*b1, key); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes, staging_bytes1); @@ -451,7 +477,7 @@ TEST(CASUploadDetached, S3StagingPromotion) reReadableStagedSource(b2, staging_key, payload.size(), s2->poolMeta().blob_header_len)); EXPECT_EQ(build2->dependencyProof(blob), BlobDependencyProof::Materialized); - const auto got2 = b2->get(key); + const auto got2 = readObj(*b2, key); ASSERT_TRUE(got2.has_value()); EXPECT_EQ(got->bytes, got2->bytes); } @@ -481,7 +507,8 @@ TEST(CASUploadDetached, CondemnedLocalResurrection) PartWriteTxnPtr build1; arrange(b1, s1, build1); const String key = s1->layout().blobKey(blob); - const Token condemned_token = b1->head(key).token; + DB::Cas::tests::OperationForTest token_probe(*b1); + const Etag condemned_token = (*token_probe).head(key, Retry::standard())->etag; ASSERT_EQ(metaStateAt(*b1, s1->layout(), payload), std::optional(MetaState::Condemned)); EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); @@ -497,8 +524,8 @@ TEST(CASUploadDetached, CondemnedLocalResurrection) EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); /// The condemned incarnation was displaced by a fresh one (token changed) and the meta is Clean again. - const Token after_token = b1->head(key).token; - EXPECT_NE(after_token.value, condemned_token.value); + const Etag after_token = (*token_probe).head(key, Retry::standard())->etag; + EXPECT_NE(after_token, condemned_token); EXPECT_EQ(metaStateAt(*b1, s1->layout(), payload), std::optional(MetaState::Clean)); EXPECT_EQ(logicalPayloadAt(*b1, key, s1->poolMeta().blob_header_len), payload); @@ -533,9 +560,9 @@ TEST(CASUploadDetached, CondemnedS3Resurrection) h.kind = ObjectKind::Blob; h.incarnation_tag = DB::UInt128(0xC0FFEE); const String staging_bytes = encodeEnvelopeHeader(h, static_cast(s->poolMeta().blob_header_len)) + payload; - b->putIfAbsent(staging_key, staging_bytes); + createObj(*b, staging_key, staging_bytes); /// Seed the condemned blob body = exactly a verbatim promote of the staging object would produce. - b->putIfAbsent(s->layout().blobKey(blob), staging_bytes); + createObj(*b, s->layout().blobKey(blob), staging_bytes); writeMetaClean(*b, s->layout(), u128Of(payload), payload.size()); condemnMeta(*b, s->layout(), u128Of(payload), /*condemn_round=*/9); build = precommitBuildFor(s, ns, ref_name, payload); @@ -546,7 +573,8 @@ TEST(CASUploadDetached, CondemnedS3Resurrection) PartWriteTxnPtr build1; arrange(b1, s1, build1); const String key = s1->layout().blobKey(blob); - const Token condemned_token = b1->head(key).token; + DB::Cas::tests::OperationForTest token_probe(*b1); + const Etag condemned_token = (*token_probe).head(key, Retry::standard())->etag; ASSERT_EQ(metaStateAt(*b1, s1->layout(), payload), std::optional(MetaState::Condemned)); EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); @@ -565,8 +593,8 @@ TEST(CASUploadDetached, CondemnedS3Resurrection) EXPECT_EQ(build1->dependencyProof(blob), std::nullopt); /// A fresh incarnation displaced the condemned one (INV-NO-RETURN: fresh tag ⇒ different token). - const Token after_token = b1->head(key).token; - EXPECT_NE(after_token.value, condemned_token.value); + const Etag after_token = (*token_probe).head(key, Retry::standard())->etag; + EXPECT_NE(after_token, condemned_token); EXPECT_EQ(metaStateAt(*b1, s1->layout(), payload), std::optional(MetaState::Clean)); std::shared_ptr b2; diff --git a/src/Disks/tests/gtest_cas_upload_fanout.cpp b/src/Disks/tests/gtest_cas_upload_fanout.cpp index d83bec5bdf96..5989ea44ccd3 100644 --- a/src/Disks/tests/gtest_cas_upload_fanout.cpp +++ b/src/Disks/tests/gtest_cas_upload_fanout.cpp @@ -41,6 +41,7 @@ using DB::Cas::tests::expectThrowsCode; using DB::Cas::tests::runRoundsUntilAbsent; using DB::Cas::tests::blobAbsent; using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::OperationForTest; namespace DB::ErrorCodes { @@ -77,6 +78,24 @@ PoolPtr openPool(const std::shared_ptr & b) return Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); } +std::optional readOf(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).read(key, DB::Cas::Retry::standard()); +} + +bool headExists(Backend & backend, const String & key) +{ + OperationForTest op(backend); + return (*op).head(key, DB::Cas::Retry::standard()).has_value(); +} + +void createRaw(Backend & backend, const String & key, const String & bytes) +{ + OperationForTest op(backend); + (*op).create(key, bytes, DB::Cas::Retry::standard()); +} + /// Stage a one-blob seed manifest and precommit it, so every adopt branch of `uploadBlobDetached` /// passes its EDGE-BEFORE-OBSERVE fail-closed gate (which only checks the `precommitted` flag). One /// precommit covers an arbitrary number of subsequently-uploaded blobs, mirroring @@ -100,13 +119,13 @@ void seedPresentBody(InMemoryBackend & b, const Layout & layout, const PoolMeta h.incarnation_tag = DB::UInt128(0xABCD); h.build_id = DB::UInt128(0x1111); const String head = encodeEnvelopeHeader(h, static_cast(pm.blob_header_len)); - b.putIfAbsent(layout.blobKey(idOf(payload)), head + payload); + createRaw(b, layout.blobKey(idOf(payload)), head + payload); } /// The logical payload stored at a blob key (object body minus the fixed blob header), or empty when absent. String logicalPayloadAt(InMemoryBackend & b, const String & key, uint64_t header_len) { - const auto got = b.get(key); + const auto got = readOf(b, key); if (!got || got->bytes.size() < header_len) return {}; return got->bytes.substr(header_len); @@ -264,7 +283,7 @@ TEST(CASUploadFanout, CopiedAndMovedRequestsSharePublicationAttemptedState) header.incarnation_tag = DB::UInt128(0xC0FFEE); const String staging_bytes = encodeEnvelopeHeader(header, static_cast(store->poolMeta().blob_header_len)) + payload; - backend->putIfAbsent(staging_key, staging_bytes); + createRaw(*backend, staging_key, staging_bytes); BlobSource source; source.size = payload.size(); @@ -293,7 +312,7 @@ TEST(CASUploadFanout, CopiedAndMovedRequestsSharePublicationAttemptedState) EXPECT_EQ(backend->streaming_publications, 1u) << "the request copied and moved through fan-out must retain the consumed first-attempt state"; EXPECT_EQ(build->dependencyProof(ref), BlobDependencyProof::Materialized); - const auto stored = backend->get(store->layout().blobKey(ref)); + const auto stored = readOf(*backend, store->layout().blobKey(ref)); ASSERT_TRUE(stored.has_value()); EXPECT_EQ(stored->bytes.substr(store->poolMeta().blob_header_len), payload); } @@ -347,7 +366,7 @@ WorldA arrangeWorldA() h.kind = ObjectKind::Blob; h.incarnation_tag = DB::UInt128(0xC0FFEE); const String staging = encodeEnvelopeHeader(h, static_cast(w.s->poolMeta().blob_header_len)) + kStaging; - w.b->putIfAbsent("p/staging/mount1/A-staging.tmp", staging); + createRaw(*w.b, "p/staging/mount1/A-staging.tmp", staging); } /// condemned-local resurrection: present body + condemned meta, local source. @@ -362,8 +381,8 @@ WorldA arrangeWorldA() h.kind = ObjectKind::Blob; h.incarnation_tag = DB::UInt128(0xC0FFEE); const String staging = encodeEnvelopeHeader(h, static_cast(w.s->poolMeta().blob_header_len)) + kResS3; - w.b->putIfAbsent("p/staging/mount1/A-republish.tmp", staging); - w.b->putIfAbsent(w.s->layout().blobKey(idOf(kResS3)), staging); + createRaw(*w.b, "p/staging/mount1/A-republish.tmp", staging); + createRaw(*w.b, w.s->layout().blobKey(idOf(kResS3)), staging); writeMetaClean(*w.b, w.s->layout(), u128Of(kResS3), std::string(kResS3).size()); condemnMeta(*w.b, w.s->layout(), u128Of(kResS3), /*condemn_round=*/9); } @@ -436,8 +455,8 @@ TEST(CASUploadFanout, CondemnedBranchesNeverGet) h.kind = ObjectKind::Blob; h.incarnation_tag = DB::UInt128(0xC0FFEE); const String staging = encodeEnvelopeHeader(h, static_cast(s->poolMeta().blob_header_len)) + s3_payload; - counting->putIfAbsent(s3_staging, staging); - counting->putIfAbsent(s->layout().blobKey(idOf(s3_payload)), staging); + createRaw(*counting, s3_staging, staging); + createRaw(*counting, s->layout().blobKey(idOf(s3_payload)), staging); writeMetaClean(*counting, s->layout(), u128Of(s3_payload), s3_payload.size()); condemnMeta(*counting, s->layout(), u128Of(s3_payload), /*condemn_round=*/5); } @@ -598,22 +617,24 @@ TEST(CASUploadFanout, CondemnedLocalResurrectStreamsAndFlipsMetaClean) condemnMeta(*b, s->layout(), u128Of(payload), /*condemn_round=*/13); const String blob_key = s->layout().blobKey(idOf(payload)); - const Token condemned_token = b->head(blob_key).token; + OperationForTest op(*b); + const auto condemned_meta = (*op).head(blob_key, Retry::standard()); + ASSERT_TRUE(condemned_meta.has_value()); std::vector reqs{localRequest(payload)}; auto pool = makePool(2); fanOutBlobUploads(*build, reqs, *pool, nullptr); - /// A fresh incarnation displaced the condemned one; INV-NO-RETURN: the queued exact-token delete - /// of the condemned incarnation must miss the resurrection. - const HeadResult after = b->head(blob_key); - ASSERT_TRUE(after.exists); - EXPECT_NE(after.token, condemned_token); - EXPECT_EQ(b->deleteExact(blob_key, condemned_token).kind, DeleteOutcome::Kind::TokenMismatch); - EXPECT_TRUE(b->head(blob_key).exists); + /// A fresh incarnation displaced the condemned one; INV-NO-RETURN: the queued exact-incarnation + /// delete of the condemned incarnation must miss the resurrection. + const auto after = (*op).head(blob_key, Retry::standard()); + ASSERT_TRUE(after.has_value()); + EXPECT_NE(after->etag, condemned_meta->etag); + EXPECT_EQ((*op).remove(blob_key, condemned_meta->etag, Retry::standard()), Removal::Mismatch); + EXPECT_TRUE(headExists(*b, blob_key)); /// The payload survived verbatim under the fresh header. - const auto got = b->get(blob_key); + const auto got = readOf(*b, blob_key); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes.substr(s->poolMeta().blob_header_len), payload); @@ -670,12 +691,14 @@ TEST(CASUploadFanout, DuplicateCondemnedS3ResurrectsCorrectly) h.kind = ObjectKind::Blob; h.incarnation_tag = DB::UInt128(0xC0FFEE); const String staging_bytes = encodeEnvelopeHeader(h, static_cast(s->poolMeta().blob_header_len)) + payload; - b->putIfAbsent(staging_key, staging_bytes); - b->putIfAbsent(s->layout().blobKey(idOf(payload)), staging_bytes); + createRaw(*b, staging_key, staging_bytes); + createRaw(*b, s->layout().blobKey(idOf(payload)), staging_bytes); writeMetaClean(*b, s->layout(), u128Of(payload), payload.size()); condemnMeta(*b, s->layout(), u128Of(payload), /*condemn_round=*/11); - const Token condemned_token = b->head(s->layout().blobKey(idOf(payload))).token; + OperationForTest op(*b); + const auto condemned_meta = (*op).head(s->layout().blobKey(idOf(payload)), Retry::standard()); + ASSERT_TRUE(condemned_meta.has_value()); std::atomic dispatched{0}; BlobUploadFanoutHooksForTest hooks; @@ -687,8 +710,9 @@ TEST(CASUploadFanout, DuplicateCondemnedS3ResurrectsCorrectly) EXPECT_EQ(dispatched.load(), 1) << "duplicate condemned records collapse to one republication task"; EXPECT_EQ(build->dependencyProof(idOf(payload)), BlobDependencyProof::Materialized); - const Token after_token = b->head(s->layout().blobKey(idOf(payload))).token; - EXPECT_NE(after_token.value, condemned_token.value) << "a fresh incarnation displaced the condemned one"; + const auto after_meta = (*op).head(s->layout().blobKey(idOf(payload)), Retry::standard()); + ASSERT_TRUE(after_meta.has_value()); + EXPECT_NE(after_meta->etag, condemned_meta->etag) << "a fresh incarnation displaced the condemned one"; EXPECT_EQ(metaStateAt(*b, s->layout(), payload), std::optional(MetaState::Clean)); EXPECT_EQ(logicalPayloadAt(*b, s->layout().blobKey(idOf(payload)), s->poolMeta().blob_header_len), payload); } @@ -860,7 +884,7 @@ TEST(CASUploadFanout, DrainPrecedesUnwind) fanOutBlobUploads(*build, reqs, *pool, &hooks); }); - EXPECT_TRUE(b->head(s->layout().blobKey(idOf(slow))).exists) + EXPECT_TRUE(headExists(*b, s->layout().blobKey(idOf(slow)))) << "the sibling's upload was drained by the join before the failure surfaced"; EXPECT_EQ(build->dependencyProof(idOf(slow)), std::nullopt) << "merge-nothing: the drained sibling's dep is not merged"; @@ -920,7 +944,7 @@ TEST(CASUploadFanout, DispatchThrowStillDrains) EXPECT_EQ(dispatch_calls.load(), 2) << "the throw fired on the second dispatch"; /// The already-RUNNING first task was drained before the stack unwound, so its body is present /// although nothing was merged. - EXPECT_TRUE(b->head(s->layout().blobKey(idOf(enqueued))).exists) + EXPECT_TRUE(headExists(*b, s->layout().blobKey(idOf(enqueued)))) << "the already-dispatched task was drained before the stack unwound"; EXPECT_EQ(build->depsSnapshotForTest().size(), 0u) << "merge-nothing on a dispatch throw"; } @@ -978,7 +1002,7 @@ TEST(CASUploadFanout, TrackingSeamThrowStillDrains) /// The already-scheduled first task was drained before `results` was destroyed, so its body is /// present; nothing was merged (merge-nothing on any fan-out throw). - EXPECT_TRUE(b->head(s->layout().blobKey(idOf(smaller))).exists) + EXPECT_TRUE(headExists(*b, s->layout().blobKey(idOf(smaller)))) << "an already-scheduled task was not drained before the stack unwound"; EXPECT_EQ(build->depsSnapshotForTest().size(), 0u) << "merge-nothing on a tracking-seam throw"; } diff --git a/src/Disks/tests/gtest_cas_wire_vocab.cpp b/src/Disks/tests/gtest_cas_wire_vocab.cpp index a375389d3c7b..97bef46f5e73 100644 --- a/src/Disks/tests/gtest_cas_wire_vocab.cpp +++ b/src/Disks/tests/gtest_cas_wire_vocab.cpp @@ -38,16 +38,16 @@ void expectThrowsCode(int expected_code, F && fn) } } -static_assert(DB::Cas::casEnumTableCoversEnum()); +static_assert(DB::Cas::casEnumTableCoversEnum()); static_assert(DB::Cas::casEnumTableCoversEnum()); static_assert(DB::Cas::casEnumTableCoversEnum()); TEST(CASWireVocab, EnumTablesPinTheCurrentWords) { using namespace DB::Cas; - EXPECT_EQ(kTokenTypeWords.toWord(TokenType::ETag, "t"), "etag"); - EXPECT_EQ(kTokenTypeWords.toWord(TokenType::Generation, "t"), "generation"); - EXPECT_EQ(kTokenTypeWords.toWord(TokenType::Emulated, "t"), "emulated"); + EXPECT_EQ(kTokenTypeWords.toWord(Dialect::ETag, "t"), "etag"); + EXPECT_EQ(kTokenTypeWords.toWord(Dialect::Generation, "t"), "generation"); + EXPECT_EQ(kTokenTypeWords.toWord(Dialect::Emulated, "t"), "emulated"); EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::CityHash128, "t"), "ch128"); EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::XXH3_128, "t"), "xxh3"); EXPECT_EQ(kBlobHashAlgoWords.toWord(BlobHashAlgo::Sha256, "t"), "sha256"); @@ -61,7 +61,7 @@ TEST(CASWireVocab, EnumTablesPinTheCurrentWords) /// would otherwise round-trip silently through the untested value. TEST(CASWireVocab, ClosedSetsRoundTripEveryEnumeratorExhaustively) { - for (const auto t : magic_enum::enum_values()) + for (const auto t : magic_enum::enum_values()) EXPECT_EQ(kTokenTypeWords.fromWord(kTokenTypeWords.toWord(t, "t"), "t"), t); for (const auto k : magic_enum::enum_values()) EXPECT_EQ(objectKindFromWord(objectKindToWord(k), "k"), k); @@ -73,7 +73,7 @@ TEST(CASWireVocab, ClosedSetsRoundTripEveryEnumeratorExhaustively) TEST(CASWireVocab, EnumWordsRoundTrip) { - for (TokenType t : {TokenType::ETag, TokenType::Generation, TokenType::Emulated}) + for (Dialect t : {Dialect::ETag, Dialect::Generation, Dialect::Emulated}) EXPECT_EQ(kTokenTypeWords.fromWord(kTokenTypeWords.toWord(t, "t"), "t"), t); for (BlobHashAlgo a : {BlobHashAlgo::CityHash128, BlobHashAlgo::XXH3_128, BlobHashAlgo::Sha256}) EXPECT_EQ(blobHashAlgoFromWord(blobHashAlgoName(a), "a"), a); @@ -86,7 +86,7 @@ TEST(CASWireVocab, SiblingFieldsWriteAndReadBack) { CasJsonWriter out; bool first = true; - writeTokenFields(out, first, PersistedIncarnation{"etag", "etag-abc\"x"}); + writeTokenFields(out, first, PersistedEtag{"etag", "etag-abc\"x"}); const BlobRef ref{BlobHashAlgo::CityHash128, BlobDigest::fromU128(hexToU128("00112233445566778899aabbccddeeff"))}; writeBlobRefFields(out, first, ref); closeObject(out, first); @@ -229,7 +229,7 @@ TEST(CASWireVocab, TokenFieldsBuildsInAnyKeyOrderAndRequiresBothFields) continue; r.skipUnknown(key); } - const PersistedIncarnation built = fields.build("t"); + const PersistedEtag built = fields.build("t"); EXPECT_EQ(built.dialect, "etag"); EXPECT_EQ(built.value, "abc"); @@ -264,14 +264,14 @@ TEST(CASWireVocab, OldManifestEpochKeyDoesNotAliasTheSemanticKey) } } -/// A `PersistedIncarnation` survives every encoding a durable CAS record uses for one, and the type +/// A `PersistedEtag` survives every encoding a durable CAS record uses for one, and the type /// system refuses the reverse direction: a persisted value must never be trusted to mint a live -/// `Incarnation`, which only an admitted request may produce. -static_assert(!std::is_constructible_v); +/// `Etag`, which only an admitted request may produce. +static_assert(!std::is_constructible_v); -TEST(CASPersistedIncarnation, RoundTripsThroughEveryFormatAndNeverBecomesAnIncarnation) +TEST(CASPersistedEtag, RoundTripsThroughEveryFormatAndNeverBecomesAnIncarnation) { - const PersistedIncarnation recorded{"generation", R"(17"3)"}; /// a quote the JSON encodings must escape + const PersistedEtag recorded{"generation", R"(17"3)"}; /// a quote the JSON encodings must escape /// 1. The shared `token_type`/`token` JSON pair. { @@ -286,7 +286,7 @@ TEST(CASPersistedIncarnation, RoundTripsThroughEveryFormatAndNeverBecomesAnIncar String key; while (r.nextKey(key)) ASSERT_TRUE(matchTokenFields(key, r, fields)) << "unexpected key " << key; - const PersistedIncarnation back = fields.build("t"); + const PersistedEtag back = fields.build("t"); EXPECT_EQ(back.dialect, recorded.dialect); EXPECT_EQ(back.value, recorded.value); } @@ -332,7 +332,7 @@ TEST(CASPersistedIncarnation, RoundTripsThroughEveryFormatAndNeverBecomesAnIncar /// Both directions of the dialect vocabulary fail closed, so neither encoding can carry a value the /// other cannot name. -TEST(CASPersistedIncarnation, UnknownDialectWordAndByteAreBothRefused) +TEST(CASPersistedEtag, UnknownDialectWordAndByteAreBothRefused) { expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectWordFromString("etags", "t"); }); expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [] { dialectByteFromWord("etags", "t"); }); diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index fc3b22b96951..e248a55e3ca3 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -31,9 +31,7 @@ PoolConfig singleAttemptConfig() .server_root_id = "test", .background_watermark = false, }; - config.cas_request_budget.max_attempts = 1; config.cas_request_budget.attempt_timeout_ms = 100; - config.cas_request_budget.operation_deadline_ms = 5000; config.cas_request_budget.lease_safety_margin_ms = 100; return config; } @@ -182,8 +180,11 @@ TEST(CASWriterDuties, UncertainAdoptedGrantStaysActiveUntilTheNextMutationRemove EXPECT_EQ( store->livePrecommitsForTest(ns), (std::set>{{"successor", successor_id.ref}})); - EXPECT_TRUE(backend->head(abandoned_manifest_key).exists) - << "the removed precommit body remains GC-owned until its decrement is sealed"; + { + DB::Cas::tests::OperationForTest verify_op(*backend); + EXPECT_TRUE((*verify_op).head(abandoned_manifest_key, Retry::once()).has_value()) + << "the removed precommit body remains GC-owned until its decrement is sealed"; + } successor->abandon(); EXPECT_TRUE(store->livePrecommitsForTest(ns).empty()); @@ -374,8 +375,6 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem DB::Cas::tests::seedPoolMetaForRestart(*backend); const CasRequestBudget budget{ .attempt_timeout_ms = 50, - .operation_deadline_ms = 500, - .max_attempts = 1, .lease_safety_margin_ms = 50, }; const RootNamespace ns{"srv1/writer_duty_crash"}; @@ -402,7 +401,8 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem abandoned.reset(); predecessor.reset(); - const auto mount = backend->get(mount_key); + DB::Cas::tests::OperationForTest mount_op(*backend); + const auto mount = (*mount_op).read(mount_key, Retry::once()); ASSERT_TRUE(mount.has_value()); EXPECT_NE(decodeMountLease(mount->bytes).min_active_build_sequence, std::numeric_limits::max()) << "a live writer-cleanup duty forbids the clean-release certificate"; @@ -450,8 +450,6 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) DB::Cas::tests::seedPoolMetaForRestart(*backend); const CasRequestBudget budget{ .attempt_timeout_ms = 50, - .operation_deadline_ms = 500, - .max_attempts = 1, .lease_safety_margin_ms = 50, }; /// Rooted under the POOL's OWN `server_root_id` ("test", unlike this file's other fixtures, which @@ -484,8 +482,11 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) ManifestId rejected_id; auto rejected = stageEmptyManifest(predecessor, ns, "rejected", rejected_id); const String rejected_manifest_key = predecessor->layout().manifestKey(rejected_id); - ASSERT_TRUE(backend->head(rejected_manifest_key).exists) - << "stageManifest's body write is unconditional; only the owner grant is refused below"; + { + DB::Cas::tests::OperationForTest verify_op(*backend); + ASSERT_TRUE((*verify_op).head(rejected_manifest_key, Retry::once()).has_value()) + << "stageManifest's body write is unconditional; only the owner grant is refused below"; + } /// `Unresolved` lands nothing, so the wedge it leaves resolves as a conclusive REJECT once the /// successor's own recovery walks past it -- unlike the ADOPT-arm crash-remnant test, this @@ -537,10 +538,11 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) EXPECT_EQ(seal->writer_epoch, predecessor_epoch); Gc gc(successor_store, hexToU128("000000000000000000000000000000e1")); - for (int round = 0; round < 16 && backend->head(rejected_manifest_key).exists; ++round) + DB::Cas::tests::OperationForTest sweep_op(*backend); + for (int round = 0; round < 16 && (*sweep_op).head(rejected_manifest_key, Retry::once()).has_value(); ++round) DB::Cas::tests::runRegularRoundReclaiming(gc); - EXPECT_FALSE(backend->head(rejected_manifest_key).exists) + EXPECT_FALSE((*sweep_op).head(rejected_manifest_key, Retry::once()).has_value()) << "the rejected attempt's orphan manifest must eventually be nominated and swept once its " "build epoch is durably closed"; diff --git a/src/IO/S3/tests/gtest_gcs_conditional_dialect.cpp b/src/IO/S3/tests/gtest_gcs_conditional_dialect.cpp index 348ceaf005d6..f2dabef765d8 100644 --- a/src/IO/S3/tests/gtest_gcs_conditional_dialect.cpp +++ b/src/IO/S3/tests/gtest_gcs_conditional_dialect.cpp @@ -74,8 +74,8 @@ TEST(GCSConditionalDialect, IfMatchUnquotedDigitsAlsoAccepted) TEST(GCSConditionalDialect, NonNumericIfMatchThrows) { /// CORRUPTED_DATA, not a broken internal invariant: the value can come from a persisted manifest - /// token or from a storage HEAD whose response carried no generation, and `mintingTypeMatches` - /// upstream only compares the token KIND, never the shape of its value. + /// etag or from a storage HEAD whose response carried no generation, and nothing upstream + /// validates the shape of that value before it reaches this function. auto r = makeRequest(); r.SetHeaderValue("if-match", "\"6654c734ccab8f440ff0825eb443dc7f\""); EXPECT_THROW(applyGcsConditionalDialectToRequest(r), DB::Exception); diff --git a/src/Storages/System/StorageSystemContentAddressedMounts.cpp b/src/Storages/System/StorageSystemContentAddressedMounts.cpp index 3aaa5a10af22..ba8b8670cc10 100644 --- a/src/Storages/System/StorageSystemContentAddressedMounts.cpp +++ b/src/Storages/System/StorageSystemContentAddressedMounts.cpp @@ -163,7 +163,7 @@ Pipe StorageSystemContentAddressedMounts::read( /// Introspection reads on the open fence: a row describing this disk's mount slots must /// still be produced when the local mount fence has already run down -- that is exactly /// the state an operator opens this table to look at. - Cas::CasOperation op = store->gcRequests().admit(); + Cas::CasOperation op = store->openRequests().admit(); mounts = Cas::listMounts(op, store->layout(), now_ms, skew_margin_ms); } catch (...) From d528693ef2492fa92468fbe6ee653ff0c462a28d Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:05:13 +0200 Subject: [PATCH 17/81] cas: review follow-ups and test hardening after the request-engine migration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Small correctness and hardening fixes found while the engine migration was under review: a read now ends only on an authoritative absence, not on every unretryable code; a raced `claimMount` reports the occupant it actually observed instead of its own proposal; an absent-key read through the request engine is no longer logged as an error (it's an ordinary outcome the engine already models); a hand-written retry loop now freezes one deadline and shares it across every call it makes, instead of re-deriving it per call; and an unobserved conflict is named for what it actually is — a vanish or a competing leader, never assumed corruption. The bulk of this is test hardening that follows from the engine actually enforcing pacing and admission where the old ad-hoc calls didn't: transport-fault doubles now inject `Poco::TimeoutException` (what production code actually throws) instead of `std::runtime_error`; several tests that asserted a schedule the engine never promised, or counted requests instead of asserting an outcome, are corrected; retry/backoff-dependent tests get their own virtual clock so they assert the engine actually reissued, rather than timing a real sleep; and the throttling coverage gate gains both a unit and an integration leg. Two properties orphaned by the old controller's test-file deletion (previous commit) are restored under the new API. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../ContentAddressed/Backend/CasBackend.h | 20 ++- .../Backend/CasObjectStorageBackend.cpp | 7 + .../ContentAddressed/Backend/CasRequests.cpp | 31 +++- .../ContentAddressed/Backend/CasRequests.h | 8 + .../ContentAddressed/Backend/CasRetry.cpp | 12 +- .../ContentAddressed/Backend/CasRetry.h | 30 +++- .../Backend/CasSentinelProbe.h | 2 +- .../Backend/CasThrottlingBackend.h | 10 ++ .../ContentAddressedTransaction.cpp | 7 +- .../ContentAddressed/Gc/CasGc.cpp | 26 ++- .../ContentAddressed/Pool/CasBlobMeta.cpp | 9 +- .../ContentAddressed/Pool/CasBlobMeta.h | 8 +- .../ContentAddressed/Pool/CasMountRuntime.h | 4 +- .../ContentAddressed/Pool/CasPartWriteTxn.cpp | 33 ++-- .../ContentAddressed/Pool/CasPool.cpp | 20 ++- .../ContentAddressed/Pool/CasPool.h | 7 +- .../ContentAddressed/Pool/CasRefCatalog.cpp | 58 ++++--- .../ContentAddressed/Pool/CasRefCatalog.h | 44 +++-- .../ContentAddressed/Pool/CasRefCkpt.cpp | 4 +- .../ContentAddressed/Pool/CasRefCkpt.h | 2 +- .../ContentAddressed/Pool/CasRefLedger.cpp | 61 +++++-- .../ContentAddressed/Pool/CasRefLedger.h | 11 +- .../ContentAddressed/Pool/CasRefProtocol.cpp | 14 +- .../ContentAddressed/Pool/CasServerRoot.cpp | 81 +++++---- .../ContentAddressed/Pool/CasServerRoot.h | 15 +- .../ContentAddressed/README.md | 13 +- .../Tools/CasDecommission.cpp | 15 +- .../ContentAddressed/Tools/CasDecommission.h | 10 +- src/Disks/tests/cas_test_helpers.h | 41 ++++- src/Disks/tests/gtest_cas_backend.cpp | 55 ++++-- .../tests/gtest_cas_confirm_exact_ref.cpp | 16 +- src/Disks/tests/gtest_cas_decommission.cpp | 55 ++++-- src/Disks/tests/gtest_cas_detached_work.cpp | 124 +++++++++++++- src/Disks/tests/gtest_cas_event_log.cpp | 3 + src/Disks/tests/gtest_cas_forget.cpp | 6 +- src/Disks/tests/gtest_cas_gc_ack_floor.cpp | 81 +++++++++ src/Disks/tests/gtest_cas_gc_round.cpp | 70 ++++++++ .../tests/gtest_cas_lifecycle_condition.cpp | 46 +++-- src/Disks/tests/gtest_cas_mount.cpp | 42 ++++- .../tests/gtest_cas_mount_claim_conflicts.cpp | 100 +++++++++++ src/Disks/tests/gtest_cas_observability.cpp | 8 +- src/Disks/tests/gtest_cas_part_write.cpp | 30 ++-- src/Disks/tests/gtest_cas_pool.cpp | 17 +- src/Disks/tests/gtest_cas_probe.cpp | 7 +- .../tests/gtest_cas_record_stream_format.cpp | 33 +++- src/Disks/tests/gtest_cas_ref_catalog.cpp | 162 ++++++++++++++++-- .../gtest_cas_ref_catalog_birth_wiring.cpp | 122 +++++++++++-- .../tests/gtest_cas_ref_chunked_flush.cpp | 3 + src/Disks/tests/gtest_cas_ref_ckpt.cpp | 43 +++-- .../tests/gtest_cas_ref_install_safety.cpp | 51 +----- .../tests/gtest_cas_ref_recovery_cas_walk.cpp | 71 ++++---- ...test_cas_ref_snapshot_publish_ordering.cpp | 6 + .../gtest_cas_ref_wedge_every_attempt.cpp | 6 +- src/Disks/tests/gtest_cas_ref_writer.cpp | 32 +++- src/Disks/tests/gtest_cas_requests.cpp | 137 ++++++++++++++- .../tests/gtest_cas_retirement_sweep.cpp | 29 +++- src/Disks/tests/gtest_cas_s3_staging.cpp | 39 +++-- src/Disks/tests/gtest_cas_sentinel_probe.cpp | 6 + src/Disks/tests/gtest_cas_throttling_gate.cpp | 129 ++++++++++++++ src/Disks/tests/gtest_cas_writer_duties.cpp | 90 +++++----- src/Storages/MergeTree/DataPartsExchange.cpp | 9 +- .../test_cas_gcs/gcs_mocks/server.py | 135 ++++++++++----- tests/integration/test_cas_gcs/test.py | 54 ++++++ 63 files changed, 1901 insertions(+), 519 deletions(-) create mode 100644 src/Disks/tests/gtest_cas_throttling_gate.cpp diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index 9187c15aa430..56ca4dffa772 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -22,13 +22,14 @@ namespace DB::Cas { -/// Typed erasure evidence for one key or one prefix. `head`/ -/// `get` deliberately flatten every kind of miss (a clean absence, a missing bucket/container, a -/// permission failure, a transport fault) into one "not found" result, which is exactly right for -/// their callers (a plain read) but wrong for lifecycle recovery, which must never treat a -/// transport/permission failure as proof that data is gone. `ProbeOutcome` keeps the four cases -/// distinct: only a backend's OWN authoritative "not found" evidence earns `KeyAbsent` — a timeout, -/// a 5xx, or an unclassifiable error is ALWAYS `Indeterminate`, never promoted to absence. +/// Typed erasure evidence for one key or one prefix. Ordinary `head`/`read` answer only two things +/// for a plain caller: the object, or nullopt for the store's own authoritative "key not found" (see +/// `isObjectNotFound`) -- every other fault (a missing bucket/container, a permission failure, a +/// transport fault) propagates as an exception instead of being flattened into absence. That +/// present/absent/throw shape is exactly right for a plain read, but wrong for lifecycle recovery, +/// which must never treat a fault it cannot classify as proof that data is gone. `ProbeOutcome` keeps +/// the four cases distinct: only a backend's OWN authoritative "not found" evidence earns `KeyAbsent` +/// -- a timeout, a 5xx, or an unclassifiable error is ALWAYS `Indeterminate`, never promoted to absence. enum class ProbeOutcome : uint8_t { Present, /// the key (or, for a prefix probe, at least one object under it) exists @@ -263,8 +264,9 @@ class Backend /// Fail-closed precondition: a Native-mode backend MUST have a /// working single-attempt conditional-write path before it coordinates a WRITABLE pool — silently /// running CAS conditional writes under the disk's default (~500-attempt) transparent retry policy - /// is exactly the hazard this seam forbids. Checked by the capability probe alongside - /// checkPoolPreconditions. Default: nothing to check (EmulatedSingleProcess and non-S3 backends + /// is exactly the hazard this seam forbids. Asked by `Pool::open` through + /// `backendForCapabilityPredicates()`, alongside `checkPoolPreconditions`, before the probe + /// operation is admitted. Default: nothing to check (EmulatedSingleProcess and non-S3 backends /// are not gated here — see ObjectStorageBackend's override for the one backend that is). virtual void checkConditionalWriteSingleAttemptSupport() {} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index a1f08aa2ec17..6f9530a4dee7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -625,6 +626,10 @@ std::optional ObjectStorageBackend::readUnder( /// incarnation, and it is the one place that can decide what a malformed one means. try { + /// An absent key is an ordinary answer here, exactly as for the HEAD helpers this GET + /// replaced: without the scope the HTTP client logs the 404 at Error, and a stateless + /// test whose stderr is checked fails on the log line alone. + Expect404ResponseScope scope; auto got = object_storage->readSmallObjectAndGetObjectMetadata( StoredObject(key), readSettingsFor(profile, timeout_ms), casMaxStoredObjectBytes()); return Raw{std::move(got.data), normalizeTokenValue(got.metadata.etag)}; @@ -677,6 +682,8 @@ std::unique_ptr ObjectStorageBackend::stream(const String & key, Tra /// the request marked NativeConditional: a stream observes no incarnation to answer with. try { + /// Same reason as `readUnder`: a not-found is an ordinary answer, not an error to log. + Expect404ResponseScope scope; std::unique_ptr buf; if (mode == Mode::Native) buf = object_storage->readObject(StoredObject(key), getReadSettings(), /*read_hint=*/std::nullopt); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index aa13743e2f77..301960426318 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -148,12 +148,20 @@ bool isRefreshableCredentialError([[maybe_unused]] const std::exception & e) namespace { -/// An answer from the store rather than a fault in reaching it: reissuing replays it unchanged. -bool isDefiniteStoreRefusal([[maybe_unused]] const std::exception & e) +/// The store's authoritative answer that there is nothing at the key -- a fact about the OBJECT, which +/// reissuing only replays. Deliberately narrower than "not retryable": a missing bucket or an +/// unmodeled name is an answer about reaching the store, which an S3-compatible store gives +/// transiently when it misroutes a request, so it stays in the ambiguous class and is reissued until +/// the deadline. The credential codes never reach here -- `refreshAndClassifyReadFault` consumes them +/// first. +bool isAuthoritativeAbsence([[maybe_unused]] const std::exception & e) { #if USE_AWS_S3 if (const auto * s3 = dynamic_cast(&e)) - return !s3->isRetryableError(); + { + const Aws::S3::S3Errors code = s3->getS3ErrorCode(); + return code == Aws::S3::S3Errors::NO_SUCH_KEY || code == Aws::S3::S3Errors::NO_SUCH_UPLOAD; + } #endif return false; } @@ -297,6 +305,15 @@ uint64_t CasOperation::reservedFor(uint64_t sleep_ms, uint32_t envelopes) const return total; } +Retry CasOperation::freeze(const Retry & policy) const +{ + if (policy.policy_deadline_ms) + return policy; + Retry frozen = policy; + frozen.policy_deadline_ms = saturatingAdd(owner.now_ms(), policy.window_ms); + return frozen; +} + bool CasOperation::fits(uint64_t needed_ms, const Retry::Bound & bound) const { /// Strict at the boundary. A backend with no attempt timeout reserves 0 and full jitter can draw a @@ -328,10 +345,10 @@ bool CasOperation::refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted = true; return !owner.backend->refreshCredentials(); } - /// The store's own answer decides. A definite refusal replays identically whether the store proved - /// the request never applied or merely named an error it will keep naming; everything else -- a - /// throttle, a 5xx, an unmodeled name an S3-compatible store reports -- may still be transient. - return isDefinitelyRefusedWrite(e) || isDefiniteStoreRefusal(e); + /// The store's own answer decides. A refusal it proved never applied, and an authoritative absence, + /// both replay identically; everything else -- a throttle, a 5xx, a missing bucket, an unmodeled + /// name an S3-compatible store reports -- may still be transient. + return isDefinitelyRefusedWrite(e) || isAuthoritativeAbsence(e); } void CasOperation::giveUpReadFenceLost(std::string_view verb, const String & subject, std::string_view when) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index de61d8be400a..b10979739c99 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -219,6 +219,14 @@ class CasOperation /// than a request. bool admitted() const { return gate(0) == Gate::Ok; } + /// `policy` with its window turned into an absolute deadline on this operation's clock, taken NOW. + /// A hand-written loop freezes its policy once before it starts and passes the frozen value to + /// every call it makes, so the loop ends when the window it was given ends -- rather than granting + /// each verb of each iteration a fresh one, which is how a bounded document promise became hours + /// of paced retrying. A policy that already carries a deadline is returned unchanged, so freezing + /// twice cannot extend it. + Retry freeze(const Retry & policy) const; + std::optional read(const String & key, const Retry & policy); std::optional head(const String & key, const Retry & policy); ListPage list(const String & prefix, const String & cursor, size_t limit, const Retry & policy); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp index e138afb5f39e..c7b4c27215b5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.cpp @@ -19,12 +19,14 @@ uint64_t Retry::backoff(uint32_t attempt) Retry::Bound Retry::bind(uint64_t now_ms) const { - const uint64_t policy_deadline_ms = now_ms > std::numeric_limits::max() - window_ms - ? std::numeric_limits::max() - : now_ms + window_ms; - if (lease_deadline_ms && *lease_deadline_ms < policy_deadline_ms) + const uint64_t own_deadline_ms = policy_deadline_ms + ? *policy_deadline_ms + : (now_ms > std::numeric_limits::max() - window_ms + ? std::numeric_limits::max() + : now_ms + window_ms); + if (lease_deadline_ms && *lease_deadline_ms < own_deadline_ms) return {*lease_deadline_ms, true}; - return {policy_deadline_ms, false}; + return {own_deadline_ms, false}; } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h index 6e9b32d9c293..984892caac55 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h @@ -17,13 +17,19 @@ struct Retry uint64_t window_ms; std::optional lease_deadline_ms; bool single_attempt; + /// An ABSOLUTE bound on the caller's own clock, replacing `window_ms` in `bind`. A hand-written + /// loop freezes one before it starts (`CasOperation::freeze`) and shares it across every call it + /// makes, so the loop as a whole ends when that deadline does instead of granting each of its + /// iterations a fresh window. Empty for a single verb, which gets its window from where it is + /// called. + std::optional policy_deadline_ms = std::nullopt; /// Full jitter: uniform(0, min(5000, 200 << (attempt-1))) milliseconds. `attempt` is 1-based; /// `attempt == 0` returns 0. static uint64_t backoff(uint32_t attempt); /// A policy with `ms` milliseconds of its own budget and no lease bound. - static Retry within(uint64_t ms) { return {ms, std::nullopt, false}; } + static Retry within(uint64_t ms) { return {.window_ms = ms, .lease_deadline_ms = std::nullopt, .single_attempt = false}; } /// `within(90'000)` -- the default write policy. static Retry standard() { return within(90'000); } /// The standard policy, additionally bound by the mount lease: `lease_deadline_ms` minus @@ -31,10 +37,21 @@ struct Retry /// gone. static Retry untilLeaseSafe(uint64_t lease_deadline_ms, uint64_t margin) { - return {90'000, lease_deadline_ms > margin ? lease_deadline_ms - margin : 0, false}; + return {.window_ms = 90'000, + .lease_deadline_ms = lease_deadline_ms > margin ? lease_deadline_ms - margin : 0, + .single_attempt = false}; } /// The standard policy, but at most one attempt is ever sent. - static Retry once() { return {90'000, std::nullopt, true}; } + static Retry once() { return {.window_ms = 90'000, .lease_deadline_ms = std::nullopt, .single_attempt = true}; } + + /// This policy, made single-attempt. A frozen loop policy keeps its absolute deadline through it, + /// which is what lets a loop send an unrepeatable request under the same bound as the rest. + Retry asSingleAttempt() const + { + Retry copy = *this; + copy.single_attempt = true; + return copy; + } /// A policy bound to an absolute deadline on the caller's own clock, plus which bound produced /// it -- the caller's own budget, or the (smaller) lease bound. @@ -43,9 +60,10 @@ struct Retry uint64_t deadline_ms; bool lease_bound; }; - /// Bind this policy to `now_ms`: `deadline_ms = min(now_ms + window_ms, lease_deadline_ms)`, - /// `lease_bound` true exactly when the lease bound was the smaller of the two. Called once at - /// call entry with the owner's `now_ms()`. + /// Bind this policy to `now_ms`: `deadline_ms = min(policy_deadline_ms ?: now_ms + window_ms, + /// lease_deadline_ms)`, `lease_bound` true exactly when the lease bound was the smaller of the + /// two. Called once at call entry with the owner's `now_ms()`. A frozen policy therefore keeps + /// the lease bound honest: the smaller of the two still wins, and still says so. Bound bind(uint64_t now_ms) const; }; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h index c4e15ee014c4..39a3ca700552 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h @@ -9,7 +9,7 @@ namespace DB::Cas /// timeouts / 5xx / connection errors => Indeterminate; permission errors => AccessDenied; /// missing container/bucket/prefix-parent => ContainerAbsent; a clean authoritative miss => KeyAbsent. /// -/// Free-function entry point (spec §2) — a thin dispatch to `op`'s own typed-evidence classification +/// Free-function entry point — a thin dispatch to `op`'s own typed-evidence classification /// (`CasOperation::probeSentinel`, which in turn reaches `Backend::probeSentinelRaw`; see there for the /// per-backend semantics: the S3-native raw HEAD error, the Local container-directory stat, or the /// generic head/get-based default for a backend without sharper evidence). diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h index 4e00208ac57c..6ac7860c276b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h @@ -54,6 +54,16 @@ class ThrottlingBackend final : public Backend return it == refusal_counts.end() ? 0 : it->second; } + /// Every key (or list prefix / publish destination) this backend has ever decided a request for, in + /// `FirstPerKey` mode every one of them by construction (`refused_keys` records the key whether or + /// not this particular call refused it), sorted. Lets a coverage test enumerate what it must check + /// without carrying its own duplicate list of keys. + std::vector decidedKeys() const + { + std::lock_guard lock(mutex); + return std::vector(refused_keys.begin(), refused_keys.end()); + } + std::optional read(const String & key, TransportAccess & access) override { refuseOrPass(key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp index 95eea5cd319b..3a78e4a5eb3f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedTransaction.cpp @@ -908,9 +908,10 @@ std::unique_ptr ContentAddressedTransaction::writeFile( /// content key stays the pool's hash of `payload` and `blob_size` stays the payload size. std::string envelope_header = buildS3StagingBlobHeader(*r); /// The staged upload becomes durable in `sink->finalize()`, outside this request contract by - /// design, so its admission is re-checked there through the callback below. The generation - /// comes off the operation that admitted it, so the two can never name different - /// incarnations. + /// design, so its admission is re-checked there through the callback below, against the + /// generation captured HERE. `admit().generation()` reads a plain value off a temporary + /// `CasOperation` that does not outlive this statement -- it names the fence generation at + /// THIS instant, not a live operation the two calls share. const Cas::PoolPtr pool = metadata_storage.store(); const uint64_t admitted_generation = pool->mountRequests().admit().generation(); return std::make_unique( diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index f7501805a165..f13e1a7fe763 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -727,7 +727,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al ++report.redeleted; ProfileEvents::increment(ProfileEvents::CASGCRetiredRedeleted); /// Drop the per-hash meta only on a removal or a proven absence — a mismatch means a - /// writer already resurrected a fresh incarnation at this hash (INV-1), and that writer's + /// writer already resurrected a fresh incarnation at this hash, and that writer's /// own republication path already flipped the meta back to Clean; blindly deleting here /// would race that legitimate Clean write for no reason (the meta is advisory, but there is /// no reason to touch it on that path at all). @@ -844,11 +844,18 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al WriteResult written = op.create(key, body, Retry::standard()); if (const auto * conflict = std::get_if(&written)) { - /// The conflict's observation IS the read that used to follow the refused create. + /// The conflict's observation IS the read that used to follow the refused create. Only + /// something that read actually OBSERVED can support a verdict about the key; a resolve + /// read that settled nothing says nothing about whether the object is there. + if (std::holds_alternative(conflict->seen)) + throw Exception(ErrorCodes::ABORTED, + "CAS gc: the create of the outcome log at {} was refused and its resolve read " + "observed nothing, so what the key holds is unknown", key); const auto * existing = std::get_if(&conflict->seen); if (!existing) throw Exception(ErrorCodes::ABORTED, - "CAS gc: outcome log at {} vanished between the create and the read that settled it", key); + "CAS gc: outcome log at {} refused the create and its resolve read observed {}", + key, detail::renderObservation(conflict->seen)); if (existing->bytes != body) { try { log = decodeOutcomeLog(openObject(FormatId::GcOutcomes, existing->bytes)); } @@ -948,9 +955,18 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al round_commit_timer->metric("generations_referenced", referenced_generations.size()); WriteResult commit = op.replace(layout.gcStateKey(), encodeGcState(next), *state_etag, Retry::standard()); - if (std::holds_alternative(commit)) + if (const auto * conflict = std::get_if(&commit)) + { + /// A refused precondition whose resolve read settled nothing proves only that this commit did + /// not apply -- naming a competing leader would assert something nobody observed. + if (std::holds_alternative(conflict->seen)) + throw Exception(ErrorCodes::ABORTED, + "CAS gc round: the gc/state commit was refused and its resolve read observed nothing; " + "retry next round"); throw Exception(ErrorCodes::ABORTED, - "CAS gc round: gc/state moved during the round (another leader advanced it); retry next round"); + "CAS gc round: gc/state moved during the round (another leader advanced it, observed {}); " + "retry next round", detail::renderObservation(conflict->seen)); + } state_etag = orThrow(std::move(commit), "CAS gc round commit"); state = std::move(next); report.round = state.round; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp index ae3f1c69c317..a6ded95e363c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.cpp @@ -12,19 +12,20 @@ namespace ProfileEvents namespace DB::Cas { -std::optional loadMeta(CasOperation & op, const Layout & layout, const BlobRef & ref) +std::optional loadMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, + const Retry & policy) { - auto got = op.read(layout.blobMetaKey(ref), Retry::standard()); + auto got = op.read(layout.blobMetaKey(ref), policy); if (!got) return std::nullopt; return LoadedMeta{.meta = decodeBlobMeta(got->bytes), .etag = std::move(got->etag)}; } WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const BlobRef & ref, - const BlobMeta & meta) + const BlobMeta & meta, const Retry & policy) { ProfileEvents::increment(ProfileEvents::CASMetaPut); - return op.create(layout.blobMetaKey(ref), encodeBlobMeta(meta), Retry::standard()); + return op.create(layout.blobMetaKey(ref), encodeBlobMeta(meta), policy); } WriteResult casMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h index 401c115cccb7..df84aee4d9f2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasBlobMeta.h @@ -32,7 +32,11 @@ struct LoadedMeta /// /// Returns the current decoded marker and the incarnation to guard the next write with, or nullopt /// when the meta key is absent. Decoding errors propagate as exceptions. -std::optional loadMeta(CasOperation & op, const Layout & layout, const BlobRef & ref); +/// +/// `policy` lets a hand-written loop pass the bound it froze at entry, so this read ends with the rest +/// of the loop instead of starting a fresh window. Same for `putMetaIfAbsent` below. +std::optional loadMeta(CasOperation & op, const Layout & layout, const BlobRef & ref, + const Retry & policy = Retry::standard()); /// Creates the marker only when its key is absent, on the plane `op` belongs to -- like its siblings, /// so one caller's decision cannot end up split across two fences. Anything at the key that this call @@ -41,7 +45,7 @@ std::optional loadMeta(CasOperation & op, const Layout & layout, con /// was observed, never as a throw: this marker is mutable, so a pre-existing different value is an /// expected outcome rather than corruption. WriteResult putMetaIfAbsent(CasOperation & op, const Layout & layout, const BlobRef & ref, - const BlobMeta & meta); + const BlobMeta & meta, const Retry & policy = Retry::standard()); /// Replaces the marker only when its current incarnation is `expected`, on the plane `op` belongs to. /// A competing write is reported as `Conflict` carrying what the resolve read observed, never thrown, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h index 9264b33f0c5a..635b41d58029 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h @@ -304,10 +304,10 @@ class CasMountRuntime /// TRUE once the pool has reached — or is being driven toward — a state on which the self-remount /// worker must stop: a published terminal `Vanished` intent (`vanished_intent` — set early by /// FORGET, or by a natural `enterVanished`, and already subsuming every settled `Vanished*` state since - /// it is published before the state store) OR `IdentityLost` (rev.8: a fail-loud TERMINAL state — no + /// it is published before the state store) OR `IdentityLost` (a fail-loud TERMINAL state — no /// demoted observer; recovery is restart or FORGET). Consulted by `scheduleRemount` before arming and by /// the remount loop at every step boundary. (The GC scheduler applies the same three-way test through - /// `Pool`, spec §9 rev.8 item 8.) + /// `Pool`.) bool remountTerminal() const { return vanished_intent.load(std::memory_order_acquire) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp index ea0082d6c64f..c42684b72339 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp @@ -273,9 +273,16 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) /// a dependency under an incarnation the precommit never saw. The build's own facts -- cancellation /// and a superseded writer epoch -- stay in `requireAlive`, where each states which one refused. CasOperation op = store->mountRequests().resume(txn_generation); - /// One policy value, `Retry::standard()`, named here for the requests this function issues - /// directly; the shared helpers it calls construct the same value themselves. - const Retry policy = Retry::standard(); + /// ONE bound for the whole publication loop, frozen before it starts: every HEAD and marker write + /// below shares this deadline, so a body whose publication keeps coming back ambiguous is refused + /// as retry-later inside one standard window instead of spending a fresh window per verb across + /// eight iterations. The paced retry is a bare sleep that does not consult the deadline, so the + /// loop can sleep one backoff (at most 5 s) past it before the next verb refuses to start. The + /// attempt cap below is the secondary bound. + const Retry policy = op.freeze(Retry::standard()); + /// The unrepeatable publication, under the SAME bound: the engine may never reissue an envelope + /// (see the publication call below), but the loop's deadline still governs whether one may start. + const Retry publication_policy = policy.asSingleAttempt(); const BlobRef & ref = req.ref; const BlobSource & source = req.source; @@ -405,7 +412,7 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) logical_size, source.size); - const std::optional loaded = loadMeta(op, store->layout(), ref); + const std::optional loaded = loadMeta(op, store->layout(), ref, policy); if (loaded) validateMetaSize(loaded->meta); @@ -422,7 +429,8 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) op, store->layout(), ref, - BlobMeta{.state = MetaState::Clean, .condemn_round = 0, .size = logical_size}); + BlobMeta{.state = MetaState::Clean, .condemn_round = 0, .size = logical_size}, + policy); } requireAdmitted("before the body-put-avoided observation is recorded"); ProfileEvents::increment(ProfileEvents::CASBlobBodyPutAvoided); @@ -469,14 +477,13 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) try { - /// A single physical publication, deliberately not under the shared policy: the engine - /// would reissue this exact envelope, and a re-sent envelope re-publishes the same - /// `incarnation_tag`, so on a content-derived-ETag dialect the republished body carries the - /// incarnation GC condemned and the exact-incarnation delete would remove a live body. - /// Every physical publication therefore mints its own envelope -- each iteration of this - /// loop builds one -- and the engine may never reissue one. The verbatim staged copy is - /// additionally a once-only privilege `beginPublication` spends. - op.publish(BlobPublishRequest{key, std::move(publication)}, Retry::once()); + /// A single physical publication, which the engine may never reissue: a re-sent envelope + /// re-publishes the same `incarnation_tag`, so on a content-derived-ETag dialect the + /// republished body carries the incarnation GC condemned and the exact-incarnation delete + /// would remove a live body. Every physical publication therefore mints its own envelope -- + /// each iteration of this loop builds one. The verbatim staged copy is additionally a + /// once-only privilege `beginPublication` spends. + op.publish(BlobPublishRequest{key, std::move(publication)}, publication_policy); } catch (const std::exception & error) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index c7a61fbc76cc..40229890bdf0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -726,14 +726,22 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol if (claim.kind != MountClaimResult::Claimed) { if (policy == MountClaimPolicy::NoWait) + { /// No FORCE variant, no wait-and-observe: the decommission gate treats any live-looking - /// or foreign-owner lease as an immediate refusal. + /// or foreign-owner lease as an immediate refusal. A raced write that observed nobody + /// has no holder to describe, and the lease this server merely proposed is not one -- + /// naming it would send an operator after this very process. + const String holder = claim.body + ? fmt::format("mount lease held by uuid={} epoch={} pid={} hostname={} (expires_at_ms={})", + u128ToHex(claim.body->server_uuid), claim.body->writer_epoch, + claim.body->pid, claim.body->hostname, claim.body->expires_at_ms) + : String("the conditional write that lost the mount slot observed nothing at the key, " + "so the holder is unknown to this server"); throw Exception(ErrorCodes::ABORTED, - "CAS decommission '{}': pool member is alive or contended — mount lease held by " - "uuid={} epoch={} pid={} hostname={} (expires_at_ms={}). Refusing (no FORCE variant " - "exists; stop the server or wait for its lease to lapse).", - srid, u128ToHex(claim.body.server_uuid), claim.body.writer_epoch, claim.body.pid, - claim.body.hostname, claim.body.expires_at_ms); + "CAS decommission '{}': pool member is alive or contended — {}. Refusing (no FORCE " + "variant exists; stop the server or wait for its lease to lapse).", + srid, holder); + } /// LiveDoubleStart (waited out the bound → a live twin) or ForeignOwner → fail closed /// with the actionable, multi-line startup error. throw Exception(ErrorCodes::ABORTED, "{}", mountDoubleStartMessage(srid, claim.body)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 180b7ba525a9..17369359538a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -714,9 +714,10 @@ class Pool : public std::enable_shared_from_this /// operation admitted here is refused the moment the fence trips, is re-armed under a fresh lease /// incarnation, or runs out of room before the lease expires. CasRequests & mountRequests() { return mount_requests; } - /// The farewell plane, on an open fence. Releasing the lease is the last thing a departing mount - /// does, and refusing it because the mount fence has already run down would leave the slot looking - /// live until GC fences it out. + /// The farewell plane, on an open fence -- shared with the mount-lease renewer's own claim/adopt, + /// not only its release: a self-remount claims with the fence already latched lost, so gating the + /// claim on the fence could never reclaim, and refusing the farewell because the mount fence has + /// already run down would leave the slot looking live until GC fences it out. CasRequests & farewellRequests() { return farewell_requests; } /// The open-fence plane: GC, the offline tools, this pool's own reads, and the bootstrap-control /// claims. None of them hold a mount lease -- the claims are what ESTABLISHES one, so gating them diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp index 84653aabc3a6..074c210ad6b7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp @@ -43,9 +43,10 @@ namespace throw Exception(ErrorCodes::LOGICAL_ERROR, "{}: the write was declined, which this call cannot produce", what); } -CasRefCatalog::Snapshot readOptionalForBootstrap(CasOperation & op, const Layout & layout) +CasRefCatalog::Snapshot readOptionalForBootstrap(CasOperation & op, const Layout & layout, + const Retry & policy = Retry::standard()) { - const std::optional got = op.read(layout.refCatalogKey(), Retry::standard()); + const std::optional got = op.read(layout.refCatalogKey(), policy); if (!got) { RefCatalog empty; @@ -59,9 +60,9 @@ CasRefCatalog::Snapshot readOptionalForBootstrap(CasOperation & op, const Layout } -CasRefCatalog::Snapshot CasRefCatalog::read(CasOperation & op, const Layout & layout) +CasRefCatalog::Snapshot CasRefCatalog::read(CasOperation & op, const Layout & layout, const Retry & policy) { - Snapshot snapshot = readOptionalForBootstrap(op, layout); + Snapshot snapshot = readOptionalForBootstrap(op, layout, policy); if (!snapshot.etag) throwMandatoryCatalogAbsent(layout.refCatalogKey()); return snapshot; @@ -139,11 +140,9 @@ struct CatalogCreatorStillLiveMarker : std::exception {}; /// the catalog is a single object mutated by every lifecycle transition of every namespace in the /// pool, so persistent contention is a real, not theoretical, exit condition to plan for. /// -/// It bounds ATTEMPTS, not time. Each iteration binds its own window per verb and pauses between -/// iterations, so the aggregate wall clock of one call is this cap times a backoff plus two verb -/// windows -- hours in the worst case. That is accepted because the only caller is the GC pre-fold -/// drain: a background round that may take as long as it takes, and whose next round re-derives -/// everything anyway. A foreground path must not adopt this loop without a wall-clock bound. +/// It bounds ATTEMPTS, and is the SECONDARY bound: the loop freezes one `Retry` before it starts and +/// every verb of every iteration shares that absolute deadline, so wall-clock time is already bounded +/// by one standard window. This cap exists so a call that somehow converges on neither still ends. constexpr size_t kMaxCatalogCasAttempts = 100; /// Shared body of `casUpdate`/`casAdmitEntry`. `encode` turns a freshly `mutate`d candidate into the @@ -153,7 +152,8 @@ constexpr size_t kMaxCatalogCasAttempts = 100; RefCatalog casUpdateImpl( CasOperation & op, const Layout & layout, const std::function & mutate, - const std::function & encode) + const std::function & encode, + const Retry & policy = Retry::standard()) { const String key = layout.refCatalogKey(); /// The candidate the LAST `decide` produced, which is the one the engine wrote: every earlier one @@ -174,7 +174,7 @@ RefCatalog casUpdateImpl( return bytes; }; - WriteResult result = op.readModifyWrite(key, decide, Retry::standard()); + WriteResult result = op.readModifyWrite(key, decide, policy); /// The fence can be lost in two places and both mean the same to a lifecycle caller: inside /// `decide`, which throws the marker itself, and between two attempts, where the engine notices it /// first and no further `decide` runs. Normalising the second onto the first is what keeps "the @@ -261,7 +261,8 @@ std::function create_namespace_step1_pre_read_hook_for_test; /// canonical-order/no-duplicate grammar check abort the process with `LOGICAL_ERROR` for what is, at /// this call site only, an ordinary race outcome. RefCatalog createNamespaceStep1( - CasOperation & op, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry) + CasOperation & op, const Layout & layout, uint64_t gc_shards, const CatalogEntry & entry, + const Retry & policy) { /// Moved into a local before invoking, not called on the global directly: a hook that reassigns /// `create_namespace_step1_pre_read_hook_for_test` from inside its own body (a test driving a @@ -289,7 +290,8 @@ RefCatalog createNamespaceStep1( [&entry, gc_shards, &layout](const RefCatalog & c) { return checkCatalogAdmission(c, gc_shards, layout, entry.ns); - }); + }, + policy); } } @@ -447,9 +449,13 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov .catalog_snapshot = std::move(catalog_snapshot)}; }; - /// ONE policy value for every erase this loop sends. Each call binds its own window when it is - /// made; what is shared is the policy, not a deadline. - const Retry policy = Retry::standard(); + /// ONE bound for the whole loop, frozen before the first iteration: every erase and every + /// resolution read below shares this deadline, so a permanently contended catalog gives up + /// retry-later within one standard window rather than spending a fresh window per verb per + /// iteration. The paced retry at the end of the loop is a bare sleep that does not consult the + /// deadline, so the loop can sleep one backoff (at most 5 s) past it before the next erase refuses + /// to start. + const Retry policy = op.freeze(Retry::standard()); for (size_t attempt = 0; attempt < kMaxCatalogCasAttempts; ++attempt) { @@ -484,7 +490,7 @@ CasRefCatalog::CompletedRemovingDeleteResult CasRefCatalog::deleteCompletedRemov /// The response to a conditional erase is not authority for what became durable. Resolve every /// attempted erase through one complete catalog read. This snapshot is also the next /// retry/selection cut, so no second read separates them. - catalog_snapshot = read(op, layout); + catalog_snapshot = read(op, layout, policy); const auto current_it = findEntry(catalog_snapshot.catalog, observed.ns); const bool old_life_still_cataloged = current_it != catalog_snapshot.catalog.entries.end() @@ -557,7 +563,7 @@ CasRefCatalog::StalledCreatingCancelOutcome CasRefCatalog::cancelStalledCreating } CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( - CasOperation & op, const Layout & layout, const CatalogEntry & observed) + CasOperation & op, const Layout & layout, const CatalogEntry & observed, const Retry & policy) { if (observed.state != NsState::Creating || !observed.creator) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -573,7 +579,7 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( .committed_through = std::nullopt, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}; if (publishCkpt(op, layout, NamespaceLifeId::fromCatalogEntry(observed.ns, observed.incarnation), - contribution) == CkptPublishOutcome::FencedOut) + contribution, policy) == CkptPublishOutcome::FencedOut) return NamespaceCreationOutcome::FencedOut; /// Step 3. `mutate` is the fence re-check point `casUpdate`'s header doc names -- checked FIRST, @@ -600,7 +606,7 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( try { casUpdateImpl(op, layout, identityPreserving(mutate), - [](const RefCatalog & c) { return encodeRefCatalog(c); }); + [](const RefCatalog & c) { return encodeRefCatalog(c); }, policy); } catch (const CatalogFenceMovedMarker &) { return NamespaceCreationOutcome::FencedOut; } catch (const CatalogEntryMismatchMarker &) { return NamespaceCreationOutcome::Superseded; } @@ -609,7 +615,7 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::completeCreation( CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( CasOperation & op, const Layout & layout, uint64_t gc_shards, - const RootNamespace & ns, const CreatorFence & creator) + const RootNamespace & ns, const CreatorFence & creator, const Retry & policy) { /// Read-first, per the Task 2 review's own note on `casAdmitEntry`: a namespace that already /// carries an entry is THIS function's job to reject with a clear message, not `casAdmitEntry`'s @@ -617,7 +623,7 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( /// -- true, but useless to a caller trying to understand why its create failed). A concurrent /// insert of the SAME namespace between this read and step 1 is still caught -- `casAdmitEntry`'s /// own grammar check is the backstop, not the only check. - const Snapshot snap = read(op, layout); + const Snapshot snap = read(op, layout, policy); const auto existing = findEntry(snap.catalog, ns); if (existing != snap.catalog.entries.end()) { @@ -652,13 +658,13 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( /// above) and reports the race as `Superseded` instead. try { - createNamespaceStep1(op, layout, gc_shards, entry); /// step 1 + createNamespaceStep1(op, layout, gc_shards, entry, policy); /// step 1 } catch (const CatalogEntryAlreadyPresentMarker &) { return NamespaceCreationOutcome::Superseded; } - return completeCreation(op, layout, entry); + return completeCreation(op, layout, entry, policy); } void CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest(std::function hook) @@ -668,7 +674,7 @@ void CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest(std::function & is_creator_fence_terminal) + const std::function & is_creator_fence_terminal, const Retry & policy) { if (observed.state != NsState::Creating || !observed.creator) throw Exception(ErrorCodes::LOGICAL_ERROR, @@ -701,7 +707,7 @@ CasRefCatalog::ReconcileCreatorOutcome CasRefCatalog::reconcileStaleCreator( try { casUpdateImpl(op, layout, identityPreserving(mutate), - [](const RefCatalog & c) { return encodeRefCatalog(c); }); + [](const RefCatalog & c) { return encodeRefCatalog(c); }, policy); } catch (const CatalogFenceMovedMarker &) { return ReconcileCreatorOutcome::FencedOut; } catch (const CatalogEntryMismatchMarker &) { return ReconcileCreatorOutcome::EntryChanged; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h index f65b1184834a..2926a7f33d6c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h @@ -15,8 +15,8 @@ namespace DB::Cas /// The `cas/ref_catalog` object (spec INV-3) as seen from the pool side: reading the current /// catalog, and the generic conditional-update primitive every lifecycle transition rides. This class /// builds ONLY that primitive -- the actual lifecycle steps (the three-conditional-write creation -/// sequence, the removal terminal-record-then-entry-delete sequence) are later tasks' job, built ON -/// TOP of `casUpdate`/`casAdmitEntry`. +/// sequence, the removal terminal-record-then-entry-delete sequence) are built ON TOP of +/// `casUpdate`/`casAdmitEntry`, further down in this same class. class CasRefCatalog { public: @@ -32,7 +32,9 @@ class CasRefCatalog /// Reads and decodes the mandatory current catalog. Absence is corruption, never an empty /// authority set: without the catalog, opaque life keys cannot prove ownership. - static Snapshot read(CasOperation & op, const Layout & layout); + /// `policy` lets a hand-written loop pass the bound it froze at entry, so its resolution reads end + /// with the rest of the loop instead of each starting a fresh window. + static Snapshot read(CasOperation & op, const Layout & layout, const Retry & policy = Retry::standard()); /// Materializes the explicit empty catalog for a prefix already proven new by /// `probePoolBootstrapResidual`. This is the only absence-tolerant catalog operation: no @@ -97,7 +99,7 @@ class CasRefCatalog /// entry point that accepted a free-form candidate could be handed a REMOVAL by a future caller /// that reads as correct, silently reopening Constraint 13 (removal is never refused) behind a /// name that says "admitting". A namespace `entry.ns` already carries an entry is a bug in the - /// caller (Task 3's creation lifecycle owns checking that first) and surfaces as + /// caller (`createNamespace` below owns checking that first) and surfaces as /// `encodeRefCatalog`'s own canonical-order/no-duplicate grammar check, inside /// `checkCatalogAdmission`. static RefCatalog casAdmitEntry( @@ -184,7 +186,7 @@ class CasRefCatalog CasOperation & op, const Layout & layout, const CatalogEntry & observed, const std::function & is_creator_fence_terminal); - /// === Task 3: the §3 creation lifecycle, built on the two primitives above === + /// === The creation lifecycle, built on the two primitives above === /// Outcome of the two-step tail every creation attempt ends in (`_ckpt` publish + `Creating -> /// Live` CAS) -- shared by a fresh `createNamespace` and a reconciler that just adopted a stalled @@ -232,10 +234,10 @@ class CasRefCatalog /// nonzero incarnation (spec: "fresh_random_128"), runs step 1 (`casAdmitEntry` inserting `{ns, /// Creating, incarnation, creator}`), then steps 2+3 via `completeCreation` below. /// - /// Per the Task 2 review's own note on `casAdmitEntry` ("a namespace `entry.ns` already carries an - /// entry is a bug in the caller -- Task 3's creation lifecycle owns checking that first"): this - /// function reads the catalog FIRST rather than handing `casAdmitEntry` a doomed insert and letting - /// its own grammar check report a confusing duplicate-namespace message. A namespace already + /// This function reads the catalog FIRST rather than handing `casAdmitEntry` a doomed insert and + /// letting its own grammar check report a confusing duplicate-namespace message: a namespace + /// `entry.ns` already carries an entry is a bug in THIS caller, not in `casAdmitEntry`, which is + /// why it is checked here first. A namespace already /// `Creating` is not this function's problem to solve -- that is exactly what `reconcileStaleCreator` /// + `completeCreation` are for, so this reports `Superseded` (never `LOGICAL_ERROR`) and sends the /// caller back through its own resume loop: sibling openers of the same namespace that all observed @@ -244,7 +246,8 @@ class CasRefCatalog /// creation's) and still throws `LOGICAL_ERROR` naming the observed state. static NamespaceCreationOutcome createNamespace( CasOperation & op, const Layout & layout, uint64_t gc_shards, - const RootNamespace & ns, const CreatorFence & creator); + const RootNamespace & ns, const CreatorFence & creator, + const Retry & policy = Retry::standard()); /// Fires once, synchronously, right after `createNamespace`'s own pre-check read observed no /// entry and right before its step 1 performs its own (first) catalog read -- the exact window a @@ -278,11 +281,12 @@ class CasRefCatalog /// `publishCkpt`); a caller that manages to make BOTH stale sees `FencedOut`, not `Superseded` -- /// both are truthful refusals of a CAS that was never sent. static NamespaceCreationOutcome completeCreation( - CasOperation & op, const Layout & layout, const CatalogEntry & observed); + CasOperation & op, const Layout & layout, const CatalogEntry & observed, + const Retry & policy = Retry::standard()); /// Stale-`Creating` reconciliation (spec INV-3: "stalled creators occupy entries until - /// fence-terminal reconciliation"; TLA Task 3 obligation 1: "the call-site is where - /// token-exactness is enforced"). `observed` must be a `Creating` entry this caller read a moment + /// fence-terminal reconciliation"; token-exactness is enforced right here, at this call site). + /// `observed` must be a `Creating` entry this caller read a moment /// ago (`LOGICAL_ERROR` otherwise -- a caller mistake, not a race). Refuses, WITHOUT writing /// anything, unless BOTH hold against a FRESH catalog read: /// - `is_creator_fence_terminal(*observed.creator)` -- injected rather than reaching into @@ -293,8 +297,8 @@ class CasRefCatalog /// `CreatorFence`, so the mount layer stays independent of the ref-catalog format), built from /// `writer_epoch` plus the mount-terminality certificates /// `probeNonTerminalMountSlots`/`computeHeartbeatFloor` already use -- NEVER from - /// `CreatorFence::fence_generation`. That field IS persisted (Task 2 serializes it into the - /// catalog entry), so it reaches the object store fine; what it is NOT is comparable across + /// `CreatorFence::fence_generation`. That field IS persisted (the catalog entry's own encoding + /// carries it), so it reaches the object store fine; what it is NOT is comparable across /// actors: it mirrors `CasMountRuntime::fence_generation`, an in-process atomic that each mount /// bumps from its OWN zero on every open, so a different actor's counter (or the SAME actor's /// after a restart) starts over at the same values and answers a different question than "is @@ -304,8 +308,8 @@ class CasRefCatalog /// invalidates this immediately). /// On success, CASes `creator` to `new_creator` -- `state` and `incarnation` are UNCHANGED, so the /// caller resumes with `completeCreation(op, layout, {..., .creator = new_creator})` over the SAME - /// incarnation, never a fresh one (rebirth under a fresh incarnation is Task 5/removal's business, - /// not a live reconciliation's). + /// incarnation, never a fresh one (rebirth under a fresh incarnation is removal's business, not a + /// live reconciliation's). /// /// `op.admitted()` is consulted FIRST on every fresh read this retries, exactly like /// `completeCreation`'s own placement -- a caller whose OWN mount fence has already moved must not @@ -316,14 +320,16 @@ class CasRefCatalog static ReconcileCreatorOutcome reconcileStaleCreator( CasOperation & op, const Layout & layout, const CatalogEntry & observed, const CreatorFence & new_creator, - const std::function & is_creator_fence_terminal); + const std::function & is_creator_fence_terminal, + const Retry & policy = Retry::standard()); /// Spec §3: "`Creating` forbids publication -- no ref writes admitted while the entry is /// Creating." Throws `throwCasWriteRetryLater`'s class (transient: `Creating` resolves once the /// creator finishes or is reconciled away) if `catalog`'s entry for `ns` is `Creating`; a no-op for /// every other case -- no entry, `Live`, or `Removing` -- since this is ONLY the birth-lifecycle /// gate on the catalog's own `Creating` state, never a general existence/removal check (that role - /// moves onto the catalog in Task 4/Task 6). Takes an already-read `RefCatalog` rather than + /// belongs to the catalog-governed append path described just below). Takes an already-read + /// `RefCatalog` rather than /// `Backend`/`Layout`, so a caller that is about to append anyway (and so already holds a fresh /// read for its OWN purposes) pays no second GET here. /// diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp index cea8f53619cd..d7a30304eb68 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.cpp @@ -193,7 +193,7 @@ std::optional readCkpt(CasOperation & op, const Layout & layout, con } CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life, - const RefCkpt & contribution) + const RefCkpt & contribution, const Retry & policy) { const String key = layout.refCkptKey(life); @@ -256,7 +256,7 @@ CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const N return encodeRefCkpt(merged); }; - WriteResult result = op.readModifyWrite(key, decide, Retry::standard()); + WriteResult result = op.readModifyWrite(key, decide, policy); if (std::holds_alternative(result)) return CkptPublishOutcome::Published; if (std::holds_alternative(result)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h index 7defb5f975c4..99ff787ee81c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCkpt.h @@ -89,7 +89,7 @@ enum class CkptPublishOutcome : uint8_t /// - exhausting the policy under persistent conflict throws the retry-later class. No partial state /// exists to clean up: every attempt either committed the complete merged body or changed nothing. CkptPublishOutcome publishCkpt(CasOperation & op, const Layout & layout, const NamespaceLifeId & life, - const RefCkpt & contribution); + const RefCkpt & contribution, const Retry & policy = Retry::standard()); /// One observation of a namespace's `_ckpt`: the decoded body and the incarnation it was read at. The /// incarnation is what the missing-base revalidation adjudicates against, so a reader that keeps only diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp index 7e6042679d35..196984a9aa68 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.cpp @@ -1191,9 +1191,9 @@ std::optional CasRefLedger::runRecoveryWalkOnce( { /// A STRAGGLER: an ordinary transaction of the dead epoch landed at `T+1` between /// our read and our create. Adopt it, advance `T` by exactly ONE, and try the seal - /// again at the NEW `T+1`. Never mint `T+2` around it: ids are state-derived - /// (INV-1/INV-2), and writing past an occupied slot puts a hole in the durable - /// stream that no later reader can distinguish from a lost object. + /// again at the NEW `T+1`. Never mint `T+2` around it: ids are state-derived, and + /// writing past an occupied slot puts a hole in the durable stream that no later + /// reader can distinguish from a lost object. ProfileEvents::increment(ProfileEvents::CASRefRecoveryStragglerAdopted); ++sequence; } @@ -1298,18 +1298,29 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( const RootNamespace & ns, uint64_t admitted_generation, uint64_t live_epoch, bool * lifecycle_refusal) { - /// Bounded exactly like `CasRefCatalog::casUpdateImpl`'s own live-lock brake, but against THIS - /// loop's re-read cycle only -- every primitive called below already bounds its OWN retry against - /// the catalog's single contended object. A duel between two openers (one creating, one - /// reconciling a stale creator) converges in a handful of rounds; this guards only against a - /// pathologically un-converging sequence of them. + /// The SECONDARY bound. Wall-clock time is already bounded by the frozen policy below, which + /// every verb of every iteration shares; this cap exists so a sequence that somehow converges on + /// neither a resolution nor the deadline still ends. A duel between two openers (one creating, one + /// reconciling a stale creator) converges in a handful of rounds. static constexpr size_t kMaxResolveAttempts = 32; const CreatorFence our_fence{server_root_id, live_epoch, admitted_generation}; CasOperation op = mount_requests.resume(admitted_generation); + /// ONE bound for the whole loop, frozen before the first iteration: every read and every protocol + /// call below shares this deadline, so a namespace whose catalog entry keeps moving gives up + /// retry-later within one standard window rather than spending a fresh window per verb per + /// iteration. The paced re-read below is a bare sleep that does not consult the deadline, so the + /// loop can sleep one backoff (at most 5 s) past it before the next read refuses to start. + const Retry policy = op.freeze(Retry::standard()); + for (size_t attempt = 0; attempt < kMaxResolveAttempts; ++attempt) { - const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); + /// Paces ONLY the re-reads a competing actor forced. A `Live` outcome is this open's own + /// success and its single confirming re-read is not contention, so it is never delayed. The + /// argument is the number of collisions so far, so the first one waits the shortest draw. + const auto paceReRead = [&] { op.pause(Retry::backoff(static_cast(attempt) + 1)); }; + + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout, policy); const auto it = std::find_if(snap.catalog.entries.begin(), snap.catalog.entries.end(), [&](const CatalogEntry & e) { return e.ns.string() == ns.string(); }); @@ -1321,12 +1332,14 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// catalog on the next loop iteration to learn it -- one extra GET, paid once per birth, /// never per write. const auto outcome = CasRefCatalog::createNamespace( - op, layout, config.gc_shards, ns, our_fence); + op, layout, config.gc_shards, ns, our_fence, policy); if (outcome == CasRefCatalog::NamespaceCreationOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " "birthing its catalog entry; the last attempt's fate is unresolved and nothing is " "installed", ns.string())); + if (outcome == CasRefCatalog::NamespaceCreationOutcome::Superseded) + paceReRead(); continue; /// Live or Superseded: re-read (Superseded means a DIFFERENT actor won birth) } @@ -1351,13 +1364,15 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// not dead), so this case is checked FIRST and unconditionally, before any terminality probe. if (it->creator->server_root_id == server_root_id && it->creator->writer_epoch == live_epoch) { - const auto outcome = CasRefCatalog::completeCreation(op, layout, *it); + const auto outcome = CasRefCatalog::completeCreation(op, layout, *it, policy); if (outcome == CasRefCatalog::NamespaceCreationOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " "resuming its own stalled creation; the last attempt's fate is unresolved and nothing " "is installed", ns.string())); + if (outcome == CasRefCatalog::NamespaceCreationOutcome::Superseded) + paceReRead(); continue; /// Live or Superseded: re-read either way } @@ -1366,7 +1381,8 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( /// steals it onto our own fence and this open resumes `completeCreation` itself. const auto reconcile_outcome = CasRefCatalog::reconcileStaleCreator( op, layout, *it, our_fence, - [&](const CreatorFence & f) { return isCreatorFenceTerminal(op, layout, f.server_root_id, f.writer_epoch); }); + [&](const CreatorFence & f) { return isCreatorFenceTerminal(op, layout, f.server_root_id, f.writer_epoch, policy); }, + policy); switch (reconcile_outcome) { case CasRefCatalog::ReconcileCreatorOutcome::FencedOut: @@ -1381,13 +1397,15 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( { CatalogEntry resumed = *it; resumed.creator = our_fence; - const auto outcome = CasRefCatalog::completeCreation(op, layout, resumed); + const auto outcome = CasRefCatalog::completeCreation(op, layout, resumed, policy); if (outcome == CasRefCatalog::NamespaceCreationOutcome::FencedOut) throwCasWriteRetryLater(fmt::format( "CAS ref-table recovery for namespace '{}': the mount incarnation moved while " "completing a reconciled creation; the last attempt's fate is unresolved and " "nothing is installed", ns.string())); + if (outcome == CasRefCatalog::NamespaceCreationOutcome::Superseded) + paceReRead(); continue; /// Live or Superseded: re-read either way } case CasRefCatalog::ReconcileCreatorOutcome::CreatorFenceStillLive: @@ -1397,7 +1415,9 @@ NamespaceLifeId CasRefLedger::resolveNamespaceLife( "CAS ref-table recovery for namespace '{}': its catalog entry is still Creating " "under a creator fence that is not yet provably dead; retry later", ns.string())); case CasRefCatalog::ReconcileCreatorOutcome::EntryChanged: - continue; /// token-exactness failed: someone else already moved this entry; re-read + /// Token-exactness failed: someone else already moved this entry. Pace before re-reading. + paceReRead(); + continue; } } @@ -3834,7 +3854,7 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// derives the SAME id and hits the SAME conflict, loudly, until a remount-level recovery (a /// fresh writer epoch is a fresh key namespace) clears it. Advancing past the occupant, which is /// what the pool-wide allocator did, would have written this table's stream around a foreign - /// object and hidden the violation -- and produced the hole INV-1 exists to forbid. + /// object and hidden the violation -- and produced a hole in a stream that must stay dense. /// /// Route it through the anomaly policy, exactly as the wedge-resolution site does for the /// identical observation. Failing closed is right, but failing closed FOREVER is @@ -3923,7 +3943,7 @@ bool CasRefLedger::commitRefChunk(const RootNamespace & ns, const std::shared_pt /// happened". A separate event keeps both readings available: the wedge counter now means /// only genuinely ambiguous appends, and this one means availability preserved. ProfileEvents::increment(ProfileEvents::CASRefAppendPreAttemptRefused); - /// The id is not consumed (INV-1): it was derived from `greatest_applied`, which this + /// The id is not consumed: it was derived from `greatest_applied`, which this /// refusal leaves exactly as it was, so the next caller on this table derives the SAME id /// and the durable stream keeps no trace of the refusal. That is the free half of the /// every-attempt rule -- an attempt that provably sent nothing owes nothing. @@ -4190,6 +4210,15 @@ void CasRefLedger::dispatchSnapshotPublisher(const RootNamespace & ns, const std } catch (...) { + { + /// Pace the exception exactly like an ordinary non-Committed publish. Every ordinary + /// failure arm inside the attempt arms this backoff before returning; an exception + /// thrown before any of them reaches here with the deadline unarmed, and settlement's + /// ONLY pacing gate is that deadline -- so without this the publisher redispatches at + /// full speed for as long as the fault persists. + std::lock_guard lock(rt->state_mutex); + advancePublishBackoff(*rt); + } if (publish_error_hook) publish_error_hook(); try diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h index ae96ed9d1e3e..73ce041c73da 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefLedger.h @@ -114,8 +114,8 @@ class CasRefLedger /// wedge captures the generation at admission and presents it back -- through `mount_requests`, /// by resuming an operation under it -- on every later retry and before every install, so a /// result that returns after a fence loss or a re-arm is inert for the superseded runtime instead - /// of installing a stale view (spec §3, "the mount-fence generation is captured at admission and - /// required on every slot-occupy and install"). + /// of installing a stale view: the generation is captured at admission and required on every + /// slot-occupy and install. std::function fence_generation_fn_, std::function boot_ms_now_fn_, std::function may_mutate_, @@ -1081,9 +1081,10 @@ class CasRefLedger /// Every `createNamespace`/`completeCreation`/`reconcileStaleCreator` outcome that writes nothing /// (`FencedOut`, `Superseded`, a reconciled entry, `EntryChanged`) re-reads the catalog and loops; /// `CreatorFenceStillLive` throws the retry-later class, which this function's caller (the transient - /// retry loop) or a higher one re-drives. Bounded against a pathological duel between two openers; - /// each primitive this loop calls has its OWN bounded retry against the catalog's single object, so - /// this bound is only against THIS loop's re-read cycle. + /// retry loop) or a higher one re-drives. One `Retry` is frozen before the loop and shared by every + /// read and every protocol call it makes, so the whole resolution ends within one standard window; + /// a re-read forced by a competing actor is paced by a jittered sleep, and an iteration cap is the + /// secondary bound. NamespaceLifeId resolveNamespaceLife( const RootNamespace & ns, uint64_t admitted_generation, uint64_t live_epoch, bool * lifecycle_refusal = nullptr); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp index 59210a22092d..a0dd85e1e158 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp @@ -870,13 +870,13 @@ EpochCrossResult crossEpochFromSeal(CasOperation & op, const Layout & layout, co return result; } - /// `life`: REQUIRED, not resolved here (review NEW-3) -- an internal fallback resolve was tried - /// once already (review C3, `Gc::fold`) and once more here (fsck's own independent walk defaulted - /// to `nullopt` and re-resolved), and both times a caller that had already committed to one `life` - /// for the rest of its walk could silently diverge from this function's OWN resolution if the - /// namespace is dropped and recreated between the two reads. `CasFsck.cpp`'s stream walk resolves - /// `life` once, at the top of its own function, and must pass that SAME value here rather than let - /// this function re-derive it a second time. + /// `life`: REQUIRED, not resolved here -- an internal fallback resolve was tried once already + /// (in `Gc::fold`) and once more here (fsck's own independent walk defaulted to `nullopt` and + /// re-resolved), and both times a caller that had already committed to one `life` for the rest of + /// its walk could silently diverge from this function's OWN resolution if the namespace is dropped + /// and recreated between the two reads. `CasFsck.cpp`'s stream walk resolves `life` once, at the + /// top of its own function, and must pass that SAME value here rather than let this function + /// re-derive it a second time. uint64_t target_epoch = witness.writer_epoch; while (target_epoch > from_seal.writer_epoch) { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 9f9d5eb8aade..44b4b55f8e83 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -74,6 +74,21 @@ std::optional conflictOrThrow(WriteResult && result, const String & return std::nullopt; } +/// A raced claim, reported as the OCCUPANT the write's own resolve read observed rather than as the +/// lease this server proposed and failed to install -- that body is what the caller renders into the +/// fail-closed operator message, and naming ourselves there points an operator at the wrong process. +/// The observed incarnation names the body returned beside it, so the caller's observation loop +/// compares like with like. A conflict the resolve read could not settle to a body saw NOBODY, and +/// reports nobody: the caller's own re-read is what identifies the holder there. +MountClaimResult racedDoubleStart(const Observation & seen) +{ + if (const Object * occupant = std::get_if(&seen)) + return {.kind = MountClaimResult::LiveDoubleStart, + .body = decodeMountLease(occupant->bytes), + .etag = occupant->etag}; + return {.kind = MountClaimResult::LiveDoubleStart, .body = std::nullopt, .etag = std::nullopt}; +} + uint64_t defaultBootMs() { struct timespec ts{}; @@ -781,12 +796,12 @@ MountClaimResult claimMount( if (!got) { const MountLease body = makeMountBody(our_uuid, our_epoch, /*seq=*/ 1, now_ms, ttl_ms); - if (conflictOrThrow(op.create(key, encodeMountLease(body), Retry::standard()), - fmt::format("CAS mount slot claim of '{}'", key))) + if (const std::optional raced + = conflictOrThrow(op.create(key, encodeMountLease(body), Retry::standard()), + fmt::format("CAS mount slot claim of '{}'", key))) /// Raced with a concurrent writer between the read and the create. Treat as a live double - /// start — fail closed; never overwrite a slot that appeared under us. The occupant was - /// not decoded here, so no conflicting identity is known to attach to an event. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .etag = std::nullopt}; + /// start — fail closed; never overwrite a slot that appeared under us. + return racedDoubleStart(*raced); emitMountEvent(sink, CasEventType::MountClaim, srid, "mint", nullptr, "fresh mount slot minted"); return {.kind = MountClaimResult::Claimed, .body = body, .etag = std::nullopt}; } @@ -816,13 +831,13 @@ MountClaimResult claimMount( return {.kind = MountClaimResult::FencedSelf, .body = existing, .etag = std::nullopt}; } const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); - if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->etag, Retry::standard()), - fmt::format("CAS mount slot refresh of '{}'", key))) - /// The mount changed under us between the read and the write: `got->etag` is now - /// KNOWN STALE (that mismatch is exactly why the write was refused), not merely unknown -- - /// leaving `.etag` unset (rather than handing back one the caller would wrongly - /// treat as current) is deliberate, matching the identical race below. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .etag = std::nullopt}; + if (const std::optional raced + = conflictOrThrow(op.replace(key, encodeMountLease(body), got->etag, Retry::standard()), + fmt::format("CAS mount slot refresh of '{}'", key))) + /// The mount changed under us between the read and the write, so `got->etag` is KNOWN + /// STALE -- that mismatch is exactly why the write was refused. What the write's resolve + /// read observed is current, and it is that pair that is reported. + return racedDoubleStart(*raced); emitMountEvent(sink, CasEventType::MountClaim, srid, "refresh", &existing, "own claim replayed — refreshed seq + expiry"); return {.kind = MountClaimResult::Claimed, .body = body, .etag = std::nullopt}; @@ -849,12 +864,13 @@ MountClaimResult claimMount( if (existing.gc_fenced || clean_marker || proven_dead) { const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); - if (conflictOrThrow(op.replace(key, encodeMountLease(body), got->etag, Retry::standard()), - fmt::format("CAS mount slot reclaim of '{}'", key))) + if (const std::optional raced + = conflictOrThrow(op.replace(key, encodeMountLease(body), got->etag, Retry::standard()), + fmt::format("CAS mount slot reclaim of '{}'", key))) /// The mount changed under us between the read and the write — someone else is racing the - /// reclaim. Fail closed. `got->etag` is now KNOWN STALE (that mismatch is exactly why - /// the write was refused) -- leaving `.etag` unset is deliberate, not an oversight. - return {.kind = MountClaimResult::LiveDoubleStart, .body = body, .etag = std::nullopt}; + /// reclaim. Fail closed, and report what the write's resolve read observed rather than + /// `got->etag`, which that mismatch just proved stale. + return racedDoubleStart(*raced); const MountPriorState prior = existing.gc_fenced ? MountPriorState::Fenced : clean_marker ? MountPriorState::Clean : MountPriorState::UncleanObserved; @@ -875,11 +891,17 @@ MountClaimResult claimMount( return {.kind = MountClaimResult::LiveDoubleStart, .body = existing, .etag = got->etag}; } -String mountDoubleStartMessage(const String & srid, const MountLease & existing) +String mountDoubleStartMessage(const String & srid, const std::optional & existing) { + const String identity = existing + ? fmt::format("server_uuid={} hostname={} pid={} last_seq={} expires_at_ms={}", + u128ToHex(existing->server_uuid), existing->hostname, existing->pid, + existing->seq, existing->expires_at_ms) + : String("could not be observed -- the conditional write that lost this slot saw nothing at " + "the key, so the holder's identity is unknown to this server"); return fmt::format( "Content-addressed disk cannot start: server_root_id '{}' is actively mounted by another LIVE server.\n" - " Existing mount: server_uuid={} hostname={} pid={} last_seq={} expires_at_ms={}\n" + " Existing mount: {}\n" "This server already waited for the mount lease to lapse, but it kept being renewed — a second\n" "server is holding the same CAS namespace. This prevents two ClickHouse servers from writing it.\n" " - If the other server is running intentionally, configure a unique for this disk.\n" @@ -891,8 +913,7 @@ String mountDoubleStartMessage(const String & srid, const MountLease & existing) " owner object gc/server-roots/{}/owner only after verifying no server uses this root.\n" " - As a LAST RESORT, after verifying that NO server is writing this root, manually delete the mount\n" " object gc/server-roots/{}/mount and restart; this server will then re-claim it.", - srid, u128ToHex(existing.server_uuid), existing.hostname, existing.pid, - existing.seq, existing.expires_at_ms, srid, srid); + srid, identity, srid, srid); } namespace @@ -943,10 +964,11 @@ MountClaimResult claimMountAwaitingExpiry( if (r.kind != MountClaimResult::LiveDoubleStart) return r; - /// `claimMount` already read the current body. Reuse `r.etag` whenever `claimMount` - /// set it (the common case: no write was attempted, so what it read is still current) instead of - /// re-reading the SAME key here. The rare stale-race branches deliberately leave `.etag` - /// unset (see their own comments), so this still falls back to a fresh read exactly there. + /// `claimMount` already read the current body, and a raced write reports whatever its own + /// resolve read observed. Reuse `r.etag` whenever it is set instead of re-reading the SAME key + /// here; only a raced write whose conflict observed nothing at all leaves it unset, and that is + /// exactly where this reads -- for the body as well as the incarnation, since a result with no + /// observation has no holder to report either. std::optional current_etag = r.etag; if (!current_etag) { @@ -967,6 +989,7 @@ MountClaimResult claimMountAwaitingExpiry( continue; } current_etag = got->etag; + r.body = decodeMountLease(got->bytes); } if (!observed || *observed != *current_etag) @@ -977,8 +1000,8 @@ MountClaimResult claimMountAwaitingExpiry( return r; observed = *current_etag; observed_since = mono_ms_fn(); - if (on_wait_start) - on_wait_start(r.body, threshold_ms); + if (on_wait_start && r.body) + on_wait_start(*r.body, threshold_ms); LOG_INFO(getLogger("CasMountLease"), "Attempting to mount content-addressed server root {} after node change or hard " "restart; waiting ~{} ms (incarnation-stability observation) to confirm the previous " @@ -1220,9 +1243,9 @@ FenceCertificate classifyFenceCertificate(const MountLease & lease, uint64_t fen } bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const String & server_root_id, - uint64_t writer_epoch) + uint64_t writer_epoch, const Retry & policy) { - const auto got = op.read(layout.mountKey(server_root_id), Retry::standard()); + const auto got = op.read(layout.mountKey(server_root_id), policy); if (!got) return false; /// absence proves nothing about liveness -- see the header doc diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index 5ab74f9fc971..f8f6c788925f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -234,7 +234,13 @@ struct MountClaimResult FencedSelf, }; Kind kind = ForeignOwner; - MountLease body; + /// The lease this result is ABOUT. For `Claimed` it is the body this server installed; for + /// `ForeignOwner`, `FencedSelf` and `LiveDoubleStart` it is the body that was observed at the key, + /// which is what `mountDoubleStartMessage` renders as the existing mount. EMPTY exactly when a + /// raced write's conflict settled to no observation at all -- nobody was seen, so no operator + /// message may name a holder, and the lease this server merely PROPOSED is not one. An optional + /// rather than the proposal, because a caller cannot check a convention it cannot see. + std::optional body; /// Which certificate of death justified a same-uuid, different-epoch `Claimed` reclaim (`None` for /// every other `Kind`, and for the absent-slot / same-epoch-refresh `Claimed` cases). MountPriorState prior = MountPriorState::None; @@ -274,7 +280,10 @@ MountClaimResult claimMount( /// live second server (the same `server_root_id` is mounted twice). Produced only AFTER this server /// has already waited for the lease to lapse (see `claimMountAwaitingExpiry`) and it did not — so the /// remediation is about a live twin, not about waiting. -String mountDoubleStartMessage(const String & srid, const MountLease & existing); +/// `existing` empty renders the identity block as "could not be observed": a raced write whose +/// conflict saw nothing has no holder to name, and naming the proposer would point an operator at this +/// very process. +String mountDoubleStartMessage(const String & srid, const std::optional & existing); /// Observation-based mount claim for restart recovery. /// Wraps `claimMount` in a loop: @@ -476,7 +485,7 @@ std::vector listMounts(CasOperation & op, const Layout & layout, uint /// "unknown" (`false`, refuse) is the fail-closed choice on every path already listed above; there is /// no path where this function answers `true` on evidence weaker than one of the three certificates. bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const String & server_root_id, - uint64_t writer_epoch); + uint64_t writer_epoch, const Retry & policy = Retry::standard()); /// Synchronous owner of the durable mount lease and merged build-watermark body. The stable /// `CasMountRuntime` is the sole driver: this class never creates a thread, invokes a callback into the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md index 4f757f072e69..44430e021ddb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/README.md @@ -92,13 +92,14 @@ Primitives → Formats → Backend → Pool → Gc → Tools ≈ Parts → facad text/format files (`CasFormat`, `CasTextFormat`, `CasPartManifestFormat`, `CasRefLogFormat`, …) plus `CasLayout` (the object-key schema). See `Formats/README.md` for the format registry. -- **`Backend/`** — the token-aware storage seam: `CasBackend` (the contract: - `get`/`putIfAbsent`/`casPut`/`deleteExact` with CAS tokens, plus unconditional - transport-only `publishBlob`), +- **`Backend/`** — the request-contract seam: `CasBackend` (the transport + primitives, dealing only in the store's own raw incarnation strings: + `read`/`head`/`list`/`remove`/`write`/`stream`/`publish`), `CasObjectStorageBackend`, `CasInMemoryBackend`, `CasInstrumentedBackend`, - `CasRequestControl` (single-attempt conditional non-blob writes, including create-if-absent - artifacts and conditional replacements, with explicit - state-aware retries), `CasProbe` (mount-time capability probe). + `CasRequests` (the retrying, `Etag`-typed layer above it — `create`/ + `replace`/`readModifyWrite`/`read`/`remove`/`probeSentinel`/`stream`/ + `publish`, admitted and budgeted by `Retry` and `CasRequestBudget`), + `CasProbe` (mount-time capability probe). - **`Pool/`** — the pool engine: `CasPool` (composition root), `CasPartWriteTxn` (one-part write transaction), `CasRefLedger` + `CasRefProtocol` (ref-table log/snapshot/replay + intake), `CasServerRoot` (mount-claim protocol + diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index 253f002fe1e3..eaee5cf485f4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -119,7 +119,9 @@ bool deleteSlotObject(CasOperation & op, const String & key, const Etag & etag, DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const String & victim_srid, const CasEventSink & sink, - const std::function & request_gc_round) + const std::function & request_gc_round, + const std::function & drain_now_fn, + const std::function & drain_sleep_fn) { DecommissionReport report; report.srid = victim_srid; @@ -149,10 +151,21 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, config.event_sink = sink; PoolPtr admin = Pool::openForDecommission(std::move(backend), std::move(config), victim_srid); + if (drain_now_fn && drain_sleep_fn) + { + /// `sweepNamespace` below issues its deletes on `admin`'s own GC plane. + admin->setCasRequestNowFnForTest(drain_now_fn); + admin->setCasRetrySleepForTest(drain_sleep_fn); + } /// A second engine over the pool's own (now instrumented) backend: `CasRequests` keeps its own /// shared_ptr to it, so `op` stays usable after `admin.reset()` retires the `Pool` below. CasRequests requests(admin->poolBackendPtr(), Fence::open()); + if (drain_now_fn && drain_sleep_fn) + { + requests.setNowFnForTest(drain_now_fn); + requests.setSleepFnForTest(drain_sleep_fn); + } CasOperation op = requests.admit(); EventEmitter{*admin}.emit([&](CasEvent & e) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h index a86edf9286c4..69c6875d27c7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h @@ -48,8 +48,16 @@ struct DecommissionReport /// terminated slot as a resume anchor; other failures, including refusal to claim the member, propagate /// as exceptions. When set, `sink` receives `MemberDecommission` audit events for the run's begin, /// per-namespace, and end milestones. +/// +/// `drain_now_fn`/`drain_sleep_fn`, when both set, replace the clock the drain's own request engine +/// paces its retries on -- the engine this function opens is a standalone one over the instrumented +/// backend, not one of `Pool`'s planes, so `Pool::setCasRetrySleepForTest` cannot reach it. A test +/// driving a latched per-object fault to `Retry::standard()`'s own give-up needs this seam, or it pays +/// the real 90-second deadline. DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const String & victim_srid, const CasEventSink & sink = {}, - const std::function & request_gc_round = {}); + const std::function & request_gc_round = {}, + const std::function & drain_now_fn = {}, + const std::function & drain_sleep_fn = {}); } diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 27a232219509..310051d792ea 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -844,8 +844,14 @@ inline DB::Cas::Etag displaceBlobToken( /// what Phase-4 Lever A (spec 2026-07-06-cas-gc-round-skip-unchanged) is designed to skip -- passes 0 /// here to force fold-every-round (shouldDeferRound's liveness bound: rounds_since_last_fold(0) >= 0 /// is always true). -inline DB::Cas::PoolPtr openPoolForTest( - std::shared_ptr backend, uint64_t gc_fold_max_defer_rounds = 8) +/// Templated on the backend's own pointer type (an `InMemoryBackend`, one of its many test subclasses, +/// or a decorator that is not itself an `InMemoryBackend`, e.g. `ThrottlingBackend`) rather than fixed +/// to `InMemoryBackend`/`BackendPtr`: a fixed pair of non-template overloads is genuinely AMBIGUOUS for +/// a `shared_ptr` argument, since "derived-to-`InMemoryBackend`" and "derived-to-`Backend`" are +/// equally-ranked conversions with no tiebreaker; template argument deduction has none of that problem. +template +DB::Cas::PoolPtr openPoolForTest( + std::shared_ptr backend, uint64_t gc_fold_max_defer_rounds = 8) { return DB::Cas::Pool::open(std::move(backend), DB::Cas::PoolConfig{.pool_prefix = "p", .server_root_id = "test", @@ -1708,6 +1714,14 @@ class CountingBackend : public DB::Cas::InMemoryBackend return InMemoryBackend::remove(key, expected_value, access); } + /// A blob publication reaches the store through `publish`, not `write` -- a "zero backend requests" + /// assertion built only from the primitives above would miss one landing. + void publish(const DB::Cas::BlobPublishRequest & request, DB::Cas::TransportAccess & access) override + { + tick(publish_counts, publish_total, request.destination_key); + InMemoryBackend::publish(request, access); + } + std::unique_ptr stream(const String & key, DB::Cas::TransportAccess & access) override { tick(get_stream_counts, get_stream_total, key); @@ -1741,6 +1755,7 @@ class CountingBackend : public DB::Cas::InMemoryBackend uint64_t putOverwriteCount(const String & key) const { return lookup(put_overwrite_counts, key); } uint64_t deleteCount(const String & key) const { return lookup(delete_counts, key); } uint64_t getStreamCount(const String & key) const { return lookup(get_stream_counts, key); } + uint64_t publishCount(const String & key) const { return lookup(publish_counts, key); } uint64_t getTotal() const { std::lock_guard lock(count_mutex); return get_total; } uint64_t headTotal() const { std::lock_guard lock(count_mutex); return head_total; } @@ -1750,6 +1765,7 @@ class CountingBackend : public DB::Cas::InMemoryBackend uint64_t putOverwriteTotal() const { std::lock_guard lock(count_mutex); return put_overwrite_total; } uint64_t deleteTotal() const { std::lock_guard lock(count_mutex); return delete_total; } uint64_t getStreamTotal() const { std::lock_guard lock(count_mutex); return get_stream_total; } + uint64_t publishTotal() const { std::lock_guard lock(count_mutex); return publish_total; } /// Attempted deletes against any key whose path CONTAINS `substr` — the per-site assertion the /// destructive-gate tests make ("the generation prune deleted nothing", "the sweep deleted nothing"). @@ -1804,9 +1820,10 @@ class CountingBackend : public DB::Cas::InMemoryBackend put_overwrite_counts.clear(); delete_counts.clear(); get_stream_counts.clear(); + publish_counts.clear(); largest_stream_chunk.clear(); get_total = head_total = list_total = write_total = put_total = put_overwrite_total - = delete_total = get_stream_total = 0; + = delete_total = get_stream_total = publish_total = 0; } private: @@ -1857,6 +1874,7 @@ class CountingBackend : public DB::Cas::InMemoryBackend std::map put_overwrite_counts; std::map delete_counts; std::map get_stream_counts; + std::map publish_counts; uint64_t get_total = 0; uint64_t head_total = 0; uint64_t list_total = 0; @@ -1865,6 +1883,7 @@ class CountingBackend : public DB::Cas::InMemoryBackend uint64_t put_overwrite_total = 0; uint64_t delete_total = 0; uint64_t get_stream_total = 0; + uint64_t publish_total = 0; }; /// Records the ORDER of writes (so a test can compare indices) and lets a test refuse or fail chosen @@ -2204,6 +2223,10 @@ class ChunkFaultBackend : public CountingBackend /// One-shot: the next read of exactly this key throws, then it is cleared. Armed by /// `Mode::LandedThenLost` (see above); settable directly for a bare lost-read fault. String fail_read_once_key; + /// How many times a fault actually fired (a `--fault_count` write, not a skipped or non-matching + /// one), so a caller can prove the double was hit rather than infer it from an outcome that a + /// weaker policy could also produce. + int fault_hits = 0; std::optional read(const String & key, DB::Cas::TransportAccess & access) override { @@ -2228,6 +2251,7 @@ class ChunkFaultBackend : public CountingBackend else if (fault_count > 0) { --fault_count; + ++fault_hits; switch (mode) { case Mode::Unresolved: @@ -2338,6 +2362,17 @@ class LatchedChunkFaultBackend : public ChunkFaultBackend fault_count = 1; return ChunkFaultBackend::write(key, bytes, expected_value, access); } + + /// Disarms completely (not just unlatches): what a caller does right after driving a call to its + /// give-up is a further mutation that must reach the store normally. + void disarm() + { + latched = false; + mode = Mode::None; + fault_count = 0; + fault_skip = 0; + fail_read_once_key.clear(); + } }; /// Fault decorator for the condemn-marker gate tests: while armed, every write against a blob `.meta` diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index b40b24e56445..73da22b314be 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -334,10 +334,15 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB return std::make_unique(String("payload")); }}}; - auto publication = std::async(std::launch::async, [&] { op.publish(request, Retry::once()); }); + /// `CasOperation` carries mutable per-call state and is single-threaded by design: the publish and + /// the concurrent read below each admit their OWN operation from the shared `requests` rather than + /// racing on `op`. + CasOperation publish_op = requests.admit(); + auto publication = std::async(std::launch::async, [&] { publish_op.publish(request, Retry::once()); }); source_opened.get_future().wait(); - auto observation = std::async(std::launch::async, [&] { return op.read("blob", Retry::once()); }); + CasOperation read_op = requests.admit(); + auto observation = std::async(std::launch::async, [&] { return read_op.read("blob", Retry::once()); }); const auto observation_status = observation.wait_for(2s); EXPECT_EQ(observation_status, std::future_status::ready) << "publication must not hold the visibility lock while draining its source"; @@ -672,31 +677,35 @@ TEST(CASCountingBackendShape, OneRequestIsCountedOnceWhicheverSurfaceIssuedIt) DB::Cas::CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); DB::Cas::CasOperation op = requests.admit(); - EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::standard()))); - EXPECT_TRUE(std::holds_alternative(op.create("k2", "v", Retry::standard()))); + /// `Retry::once()` on every verb below, not `standard()`: this test pins ONE physical request per + /// call, and a healthy backend never distinguishes the two policies by outcome -- only `once()` + /// forbids a reissue by construction, so a regression that made the engine reissue speculatively + /// would still fail here instead of passing on a backend too healthy to ever need the second attempt. + EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::once()))); + EXPECT_TRUE(std::holds_alternative(op.create("k2", "v", Retry::once()))); EXPECT_EQ(backend->putCount("k"), 1u); EXPECT_EQ(backend->putCount("k2"), 1u); EXPECT_EQ(backend->writeTotal(), 2u); EXPECT_EQ(backend->putOverwriteTotal(), 0u) << "neither write carried a precondition"; - const std::optional k_meta = op.head("k", Retry::standard()); + const std::optional k_meta = op.head("k", Retry::once()); ASSERT_TRUE(k_meta); EXPECT_EQ(backend->headCount("k"), 1u); expectBytes(*backend, "k", "v"); - EXPECT_TRUE(op.read("k", Retry::standard())); + EXPECT_TRUE(op.read("k", Retry::once())); EXPECT_EQ(backend->getCount("k"), 2u); - EXPECT_TRUE(std::holds_alternative(op.replace("k", "w", k_meta->etag, Retry::standard()))); + EXPECT_TRUE(std::holds_alternative(op.replace("k", "w", k_meta->etag, Retry::once()))); EXPECT_EQ(backend->putOverwriteCount("k"), 1u) << "a write with a precondition is the replace shape"; EXPECT_EQ(backend->writeCount("k"), 2u); - const std::optional k2_meta = op.head("k2", Retry::standard()); + const std::optional k2_meta = op.head("k2", Retry::once()); ASSERT_TRUE(k2_meta); - EXPECT_EQ(op.remove("k2", k2_meta->etag, Retry::standard()), Removal::Removed); - const std::optional k_meta_after = op.head("k", Retry::standard()); + EXPECT_EQ(op.remove("k2", k2_meta->etag, Retry::once()), Removal::Removed); + const std::optional k_meta_after = op.head("k", Retry::once()); ASSERT_TRUE(k_meta_after); - EXPECT_EQ(op.remove("k", k_meta_after->etag, Retry::standard()), Removal::Removed); + EXPECT_EQ(op.remove("k", k_meta_after->etag, Retry::once()), Removal::Removed); EXPECT_EQ(backend->deleteCount("k"), 1u); EXPECT_EQ(backend->deleteCount("k2"), 1u); EXPECT_EQ(backend->deleteTotal(), 2u); @@ -1096,6 +1105,26 @@ TEST(CASObjectStorageBackend, PublishBlobRefusesVerbatimCopyWithoutNativeTranspo EXPECT_FALSE(storage->exists(DB::StoredObject(destination))); } +/// Every Native conditional write selects the SingleAttempt object-storage retry profile (RFC +/// cas-s3-timeout-retry-control §disable-transparent-conditional-write-retries): the CAS request +/// engine, not the object-storage client, owns retry/backoff for a conditional PUT, so a client-level +/// retry loop reissuing the identical request underneath it would double the reissue and could land a +/// write the engine itself had already given up on. Two seams prove the property without a live/fake +/// S3 endpoint: `IObjectStorage::supportsRetryProfile` is the fail-closed capability check +/// `checkConditionalWriteSingleAttemptSupport` relies on at mount time, and +/// `s3_max_unexpected_write_error_retries_override` is the SECOND retry-affecting layer above the S3 +/// client -- `WriteBufferFromS3`'s own makeSinglepartUpload/completeMultipartUpload loop, bounded +/// independently of the client-level override. +TEST(CASObjectStorageBackend, ConditionalWriteSelectsSingleAttemptAndLocalStorageDoesNotSupportIt) +{ + auto backend = std::make_shared( + tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::Native); + const auto ws = backend->conditionalWriteSettingsForTest(); + EXPECT_EQ(ws.object_storage_retry_profile, DB::ObjectStorageRetryProfile::SingleAttempt); + EXPECT_EQ(ws.s3_max_unexpected_write_error_retries_override, 1u); + EXPECT_FALSE(tests::makeLocalObjectStorageForTest()->supportsRetryProfile(DB::ObjectStorageRetryProfile::SingleAttempt)); +} + /// The Native conditional-PUT path discriminates a lost precondition by the canonical S3 error code /// string ("PreconditionFailed", "NoSuchKey", ...) that `S3Exception` carries from the response XML /// `` — a 412 is UNMODELED for the AWS SDK (the enum value is UNKNOWN), so the name is the only @@ -1637,7 +1666,9 @@ TEST(CASObjectStorageBackend, EmuTokenStateEventuallyPrunesDistinctShortLivedKey /// refuse to construct an `Etag` from a malformed value in the first place (`CORRUPTED_DATA`), so no /// caller reaching the primitives through `CasOperation` can ever hold one -- the grammar guard inside /// `ObjectStorageBackend::write`/`removeUnder` is unreachable from the public engine surface and stays -/// as defense in depth only. The grammar predicate itself remains directly pinned by +/// as defense in depth only. The empty/`*`/quoted-list token cases the deleted tests drove are covered +/// at the `Etag::mint`/`tryMint` boundary by `CASIncarnation.GrammarRefusesTheNineWays` +/// (`gtest_cas_requests.cpp`); the grammar predicate itself remains directly pinned by /// `CASBackendGrammar.GenerationDialectAcceptsOnlyCanonicalPositiveDecimal` above. #endif diff --git a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp index 46f970ec64ed..c2a40218c0a7 100644 --- a/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp +++ b/src/Disks/tests/gtest_cas_confirm_exact_ref.cpp @@ -319,14 +319,15 @@ ManifestId publishEmptyPart(const PoolPtr & s, const RootNamespace & ns, const S return id; } -/// The reads, heads, stream opens, writes and lists `CountingBackend` observes, summed. The zero-I/O -/// contract is asserted against this total, so a confirm that quietly grew a HEAD or a GET fails the -/// test rather than the review. `writeTotal` and not `putTotal`: a write that carried a precondition -/// is still a write, and counting only the create-shaped ones left the replace path unwatched. -/// Deletes are NOT in this sum. +/// Every request `CountingBackend` observes, summed: reads, heads, stream opens, writes, lists, +/// deletes and publications. The zero-I/O contract is asserted against this total, so a confirm that +/// quietly grew any one of them fails the test rather than the review. `writeTotal` and not `putTotal`: +/// a write that carried a precondition is still a write, and counting only the create-shaped ones left +/// the replace path unwatched. uint64_t backendRequests(const CountingBackend & b) { - return b.headTotal() + b.getTotal() + b.getStreamTotal() + b.writeTotal() + b.listTotal(); + return b.headTotal() + b.getTotal() + b.getStreamTotal() + b.writeTotal() + b.listTotal() + + b.deleteTotal() + b.publishTotal(); } /// One refusal counter's current value. `confirmExactRef` attributes every `Unknown` to exactly one of @@ -756,6 +757,9 @@ TEST(CASConfirmExactRef, WedgedTransactionRefusesEveryRef) budget.attempt_timeout_ms = 100; budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); auto store = openPoolWithConfig(backend, cfg); auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/confirm_real_wedge"}; diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 3a3033e2bae7..554af1650c0b 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -47,18 +47,30 @@ void drainCompletedNamespaceRemovals(const std::shared_ptr & ba /// failure on one listed object must record a warning and let the rest of the sweep proceed, never /// abort the whole phase. Injects on the `remove` PRIMITIVE, not the legacy `deleteExact`, so it /// intercepts a caller on either surface. +/// +/// The thrown fault is a `Poco::TimeoutException`, the class the request engine classifies as a +/// transport failure and reissues (`Retry::standard()`); a `std::runtime_error` propagates immediately +/// and never exercises the retry path this test's own name claims to drive. `latch` keeps it armed +/// across every reissue of the SAME logical call, so the engine reaches its own retry deadline and +/// gives up rather than recovering on a later attempt -- a one-shot throw would be outlived by the +/// reissue and the delete would simply succeed. class FailingDeleteBackend : public InMemoryBackend { public: void failWithThrow(const String & key) { throw_key = key; } void failWithTokenMismatch(const String & key) { mismatch_key = key; } + void latch() { latched = true; } /// Clears every injected failure -- the resume half of a fail-then-retry test (Task 4). - void disarm() { throw_key.clear(); mismatch_key.clear(); } + void disarm() { throw_key.clear(); mismatch_key.clear(); latched = false; } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { if (key == throw_key) - throw std::runtime_error("injected transient delete failure for " + key); + { + if (!latched) + throw_key.clear(); + throw Poco::TimeoutException("injected transient delete failure for " + key); + } if (key == mismatch_key) return RawRemoval::Mismatch; return InMemoryBackend::remove(key, expected_value, access); @@ -67,6 +79,7 @@ class FailingDeleteBackend : public InMemoryBackend private: String throw_key; String mismatch_key; + bool latched = false; }; /// Replaces the durable catalog immediately after returning the first armed catalog read. This @@ -907,10 +920,15 @@ TEST(CASDecommission, PerObjectFailureWarnsAndContinuesDrain) (*seed_op).create("p/roots/victim/clickhouse_access_check_abc", "x", Retry::once()); } backend->failWithThrow("p/staging/victim/upload_throws.tmp"); + backend->latch(); backend->failWithTokenMismatch("p/roots/victim/clickhouse_access_check_abc"); + /// The engine reissues an unresolved delete until its own retry window closes, measured on this + /// clock, so the latched fault reaches a genuine give-up with no real time passing. + DB::Cas::tests::FakeClock clock; const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", + /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); EXPECT_EQ(report.staging_objects_removed, 1u) << "the OTHER staging object must still be deleted despite the injected failure on its sibling"; @@ -966,13 +984,18 @@ TEST(CASDecommission, ManifestDebrisDeleteFailureWarnsAndContinues) debris_key = victim->layout().manifestKey(debris_id); } backend->failWithThrow(debris_key); + backend->latch(); { OperationForTest seed_op(*backend); (*seed_op).create("p/staging/victim/upload_ok.tmp", "x", Retry::once()); } + /// The engine reissues an unresolved delete until its own retry window closes, measured on this + /// clock, so the latched fault reaches a genuine give-up with no real time passing. + DB::Cas::tests::FakeClock clock; const auto report = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim"); + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", + /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); EXPECT_EQ(report.namespaces_removed, 1u) << "victim/db/t1's namespace erasure (Task 2) is untouched by either injected failure"; @@ -1246,11 +1269,11 @@ class FailDeletesUnderPrefixBackend : public Backend RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { if (armed && key.starts_with(fail_prefix)) - /// A caller-bug-shaped exception (never `Poco::Exception`), so the engine surfaces it on - /// the first attempt instead of reissuing it for the whole policy window -- this fixture - /// models a single per-object failure the drain must warn on and move past, not a - /// transport fault the retry loop should absorb. - throw std::runtime_error("injected transient delete failure for " + key); + /// `Poco::TimeoutException`, the class the request engine classifies as a transport fault + /// and reissues (`Retry::standard()`): this fixture models a real backend transiently + /// failing, and `armed` stays set across every reissue of the same call, so the engine + /// reaches its own retry deadline and gives up rather than recovering on a later attempt. + throw Poco::TimeoutException("injected transient delete failure for " + key); return inner->remove(key, expected_value, access); } std::expected write(const String & key, const String & bytes, @@ -1283,8 +1306,13 @@ TEST(CASDecommission, FailedDrainKeepsSlotThenResumes) (*raw_op).create("p/roots/victim/loose_file", "x", Retry::once()); auto failing = std::make_shared(inner, "p/roots/victim/"); + /// The engine reissues an unresolved delete until its own retry window closes, measured on this + /// clock, so the fault (armed across every reissue) reaches a genuine give-up with no real time + /// passing. + DB::Cas::tests::FakeClock clock; const auto first = decommissionPoolMember( - failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim"); + failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", + /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) @@ -1320,9 +1348,14 @@ TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) debris_key = victim->layout().manifestKey(debris_id); } backend->failWithThrow(debris_key); + backend->latch(); + /// The engine reissues an unresolved delete until its own retry window closes, measured on this + /// clock, so the latched fault reaches a genuine give-up with no real time passing. + DB::Cas::tests::FakeClock clock; const auto first = decommissionPoolMember( - backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim"); + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", + /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); EXPECT_EQ(first.manifest_debris_removed, 0u); diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index c74181bd04d2..e309aea683d1 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -183,6 +183,10 @@ class ManualDetachedLedger {}, [](const RootNamespace &) {}) { + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the + /// `oneAttemptBudget()` field alone; pair the two so the ledger's own admission arithmetic sees + /// what the budget claims. + backend->setAttemptTimeoutMs(oneAttemptBudget().attempt_timeout_ms); /// The engine's own retry pauses, on a clock the engine reads: one call reaches its retry /// deadline against a latched fault with no real time passing. `setCasRetrySleepForTest` /// installs the sleep on both `mount_requests` and the recovery retry loop. @@ -286,6 +290,9 @@ PoolPtr openPublishingPool(const std::shared_ptrsetAttemptTimeoutMs(config.cas_request_budget.attempt_timeout_ms); return Pool::open(backend, config); } @@ -597,24 +604,127 @@ TEST(CASDetachedWork, FailedPublisherDispatchKeepsMutationAndClearsReservation) /// Settlement must survive a throwing error handler. Today it is a bare call after the handler, so a /// handler that throws skips it and strands the reservation for the life of the process. /// -/// The publish is FAULTED deliberately: with a healthy backend it would succeed, the handler would -/// never run, and this test would pass while exercising nothing. +/// The dispatched attempt is made to throw deliberately, via `snapshot_after_capture_hook_for_test` +/// (called inline, unguarded by any inner `catch`, so its throw reaches `dispatchSnapshotPublisher`'s +/// outer `catch (...)` that invokes `publish_error_hook_for_test`): with a healthy backend and no +/// injected throw the publish would succeed, the handler would never run, and this test would pass +/// while exercising nothing. An ordinary FAULTED WRITE does not reach the handler at all -- +/// `tryPublishSnapshotAndAdvanceCheckpointOnceOnRuntimeImpl` treats every non-`Committed` `WriteResult` +/// (including a genuine retry give-up) as an ordinary backoff-and-retry-later outcome, a plain return, +/// never a throw -- so only an exception from OUTSIDE that write (this hook stands in for one) ever +/// reaches the handler. +/// +/// The hook fires (and throws) EXACTLY ONCE, then disarms itself before throwing. An exception escaping +/// before any of the write's own failure arms now reaches the detached task's `catch (...)`, which arms +/// `advancePublishBackoff` on that exit exactly as the ordinary failure arms do, so this throw paces the +/// redispatch instead of driving it at full speed. Disarming after one throw lets the second dispatch +/// take the healthy path and settle, keeping this test's claim narrow: settlement survives ONE throwing +/// handler call. TEST(CASDetachedWork, SettlementSurvivesAThrowingErrorHandler) { auto backend = std::make_shared(); + std::atomic handler_ran{false}; PoolConfig config; - config.publish_error_hook_for_test - = [] { throw std::runtime_error("injected: the error handler itself throws"); }; + /// No real backoff wait: the injected throw leaves the tail over-threshold, and a REAL backoff + /// sleep here would still be paid at teardown drain even though the test's own assertions never + /// wait on it directly. + config.snapshot_publish_backoff_initial_ms = 0; + config.snapshot_publish_backoff_max_ms = 0; + config.publish_error_hook_for_test = [&handler_ran] + { + handler_ran.store(true); + throw std::runtime_error("injected: the error handler itself throws"); + }; auto store = openPublishingPool(backend, config); const RootNamespace ns{"srv1/handler_throws"}; - /// Arm the fault so the publisher's own PUT fails and its `catch` is entered. Use the same arming - /// call the snapshot-ordering suite uses against this backend. - backend->armWriteFailure("_snap/", 1); + std::atomic capture_hook_ran{false}; + std::atomic capture_hook_armed{true}; + store->setSnapshotAfterCaptureHookForTest([&capture_hook_ran, &capture_hook_armed] + { + capture_hook_ran.store(true); + if (capture_hook_armed.exchange(false)) + throw std::runtime_error("injected: the dispatched attempt itself throws"); + }); ASSERT_NO_THROW(publishRef(store, ns, "ref_1", 1)); store->waitForSnapshotPublishSettleForTest(ns); + ASSERT_TRUE(capture_hook_ran.load()) << "the dispatched attempt never reached the injected throw"; + EXPECT_TRUE(handler_ran.load()) << "the injected throw must have reached the (throwing) error handler"; EXPECT_EQ(store->pendingSnapshotPublishesForTest(ns), 0); + + store->setSnapshotAfterCaptureHookForTest(nullptr); +} + +/// A publish attempt that throws BEFORE any of the ordinary write-failure arms must pace exactly like +/// an ordinary failure: settlement's only pacing gate is the publish backoff deadline, so an exception +/// that leaves it unarmed redispatches the publisher at full speed for as long as the fault persists. +/// The clock is the injected boot clock, so the schedule is virtual and the test spends no wall time +/// waiting one out; the one real-time wait is the bounded observation window, because an unpaced +/// redispatch runs on a background thread and has to be caught in the act rather than waited out. +TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) +{ + auto backend = std::make_shared(); + constexpr uint64_t step_ms = 100; + std::atomic fake_boot{1000}; + std::atomic error_hook_calls{0}; + PoolConfig config; + config.boot_ms_fn = [&fake_boot] { return fake_boot.load(); }; + /// Initial == max, so every step of the schedule is the same virtual `step_ms` and the test can + /// advance the clock by a constant. + config.snapshot_publish_backoff_initial_ms = step_ms; + config.snapshot_publish_backoff_max_ms = step_ms; + config.publish_error_hook_for_test = [&error_hook_calls] { error_hook_calls.fetch_add(1); }; + auto store = openPublishingPool(backend, config); + const RootNamespace ns{"srv1/throwing_publisher_pacing"}; + + std::atomic attempts{0}; + store->setSnapshotAfterCaptureHookForTest([&attempts] + { + attempts.fetch_add(1); + throw std::runtime_error("injected: every publish attempt throws before its write"); + }); + + ASSERT_NO_THROW(publishRef(store, ns, "ref_1", 1)); + + /// The virtual clock does not move here, so the armed deadline is still in the future for the whole + /// window and exactly ONE attempt may have run. + const auto observe_until = std::chrono::steady_clock::now() + std::chrono::milliseconds(500); + while (std::chrono::steady_clock::now() < observe_until && attempts.load() <= 1) + std::this_thread::yield(); + EXPECT_EQ(attempts.load(), 1u) << "the throwing attempt redispatched without arming the publish backoff"; + EXPECT_GE(error_hook_calls.load(), 1u) << "the injected throw never reached the error handler"; + + /// One step of the schedule per iteration: the tail is still over threshold, so each mutation + /// re-evaluates admission, and exactly one attempt may pass per elapsed backoff interval. + for (uint64_t step = 1; step <= 3; ++step) + { + fake_boot.fetch_add(step_ms); + ASSERT_NO_THROW(publishRef(store, ns, "ref_" + std::to_string(step + 1), step + 1)); + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (attempts.load() < 1 + step) + { + ASSERT_LT(std::chrono::steady_clock::now(), deadline) + << "the elapsed backoff never admitted the next publish attempt"; + std::this_thread::yield(); + } + /// Bounded poll rather than `waitForSnapshotPublishSettleForTest`: that call waits on a condvar + /// predicate with no deadline, and on an unpaced-redispatch regression the reservation count + /// never rests at zero long enough for the predicate to observe it, hanging the test instead of + /// failing it. + const auto settle_deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (store->pendingSnapshotPublishesForTest(ns) != 0) + { + ASSERT_LT(std::chrono::steady_clock::now(), settle_deadline) + << "the snapshot publish for '" << ns.string() << "' never settled: " + << "pending_snapshot_publishes stayed nonzero"; + std::this_thread::yield(); + } + EXPECT_EQ(attempts.load(), 1 + step) << "more than one publish attempt ran within one backoff step"; + } + + EXPECT_EQ(error_hook_calls.load(), attempts.load()); + store->setSnapshotAfterCaptureHookForTest(nullptr); } /// A publisher asleep in recovery backoff must be woken by the stop, not waited out. The injected diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index 6c2ecbfa00f6..7beee4dc1348 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -136,6 +136,9 @@ PoolPtr openRenewalEventPool( String prefix = "renewal-events", String server_root_id = "test") { + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the renewal-boundary math these tests drive matches what admits. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); return Pool::open(backend, PoolConfig{ .pool_prefix = std::move(prefix), .server_root_id = std::move(server_root_id), diff --git a/src/Disks/tests/gtest_cas_forget.cpp b/src/Disks/tests/gtest_cas_forget.cpp index 094f46660eed..d1b49b4c41a7 100644 --- a/src/Disks/tests/gtest_cas_forget.cpp +++ b/src/Disks/tests/gtest_cas_forget.cpp @@ -96,14 +96,14 @@ class ToggleableTransportFaultBackend final : public DB::Cas::InMemoryBackend std::optional head(const String & key, DB::Cas::TransportAccess & access) override { if (fail.load()) - throw std::runtime_error("injected fault: transport error"); + throw Poco::TimeoutException("injected fault: transport error"); return InMemoryBackend::head(key, access); } std::optional read(const String & key, DB::Cas::TransportAccess & access) override { if (fail.load()) - throw std::runtime_error("injected fault: transport error"); + throw Poco::TimeoutException("injected fault: transport error"); return InMemoryBackend::read(key, access); } @@ -111,7 +111,7 @@ class ToggleableTransportFaultBackend final : public DB::Cas::InMemoryBackend DB::Cas::TransportAccess & access) override { if (fail.load()) - throw std::runtime_error("injected fault: transport error"); + throw Poco::TimeoutException("injected fault: transport error"); return InMemoryBackend::list(prefix, cursor, limit, access); } diff --git a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp index 8a4c04f6beb2..b89fa32d5526 100644 --- a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp +++ b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp @@ -14,6 +14,12 @@ #include #include #include "cas_test_helpers.h" +#include "config.h" + +namespace DB::ErrorCodes +{ +extern const int ABORTED; +} namespace ProfileEvents { @@ -1385,3 +1391,78 @@ TEST(CASGCCondemnMarker, LoadMetaFallbackConfirmsGraduationAfterLeaderRestart) EXPECT_TRUE(e->marker_confirmed) << "a delete_pending row confirmed via loadMeta still carries the bit"; EXPECT_TRUE(blobExists(*backend, store->layout(), blob)); } + +#if USE_AWS_S3 +/// The outcomes-log `create` meets the same shape as the round commit: a refused precondition whose +/// resolve read was itself refused. Nothing observed the key, so the round may not report that the log +/// vanished -- an absent key and an unreadable one are different answers. +/// +/// The S3 gate is the fault's, not the site's: the definitive-refusal classification that makes a +/// resolve read settle nothing rather than be reissued exists only for S3 errors. +TEST(CASGCRetire, OutcomeLogUnobservedConflictDoesNotReportItVanished) +{ + class UnobservedOutcomesBackend : public InMemoryBackend + { + public: + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override + { + if (arm && !expected_value && key.find("/outcomes/") != String::npos) + { + arm = false; + refused_key = key; + return std::unexpected(RawConflict{}); + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + std::optional read(const String & key, TransportAccess & access) override + { + if (!refused_key.empty() && key == refused_key) + { + refused_key.clear(); + throw DB::S3Exception("UnobservedOutcomesBackend: the settling read is definitively refused", + Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); + } + return InMemoryBackend::read(key, access); + } + + bool arm = false; + String refused_key; + }; + + auto backend = std::make_shared(); + auto store = openPoolForTest(backend); + const RootNamespace ns{"00/aa@cas@"}; + const ManifestRef r = ref("srv-a:1", 1, 0xAA); + writeBlobBody(*backend, store->layout(), DB::UInt128(1)); + writeManifestRaw(*backend, store->layout(), ns, r, {blobEntryFor("a", DB::UInt128(1))}); + publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, r); + Gc gc(store, kGc); + gc.runRegularRound(); + dropRefTransition(*backend, store->layout(), ns, "tbl", r); + + /// The condemn -> graduate -> delete pipeline needs several rounds before any round has an outcome + /// to log; the arm fires on the first one that does. + backend->arm = true; + bool refusal_reached = false; + for (int i = 0; i < 8 && !refusal_reached; ++i) + { + try + { + runRegularRoundReclaiming(gc); + store->renewWatermarkOnce(); + } + catch (const DB::Exception & e) + { + refusal_reached = true; + EXPECT_EQ(e.code(), DB::ErrorCodes::ABORTED); + EXPECT_NE(e.message().find("resolve read observed nothing"), String::npos) << e.message(); + EXPECT_EQ(e.message().find("vanished"), String::npos) + << "nothing observed the key, so it may not be called vanished: " << e.message(); + } + } + EXPECT_TRUE(refusal_reached) << "no round ever wrote an outcome log, so the arm was never reached"; +} +#endif diff --git a/src/Disks/tests/gtest_cas_gc_round.cpp b/src/Disks/tests/gtest_cas_gc_round.cpp index 8139c3fa23ad..eeaad2c7b86f 100644 --- a/src/Disks/tests/gtest_cas_gc_round.cpp +++ b/src/Disks/tests/gtest_cas_gc_round.cpp @@ -9,6 +9,7 @@ #include #include #include "cas_test_helpers.h" +#include "config.h" namespace DB::ErrorCodes { @@ -2252,3 +2253,72 @@ TEST(CASGc, RoundCommitConflictDropsTheRound) EXPECT_GT(after.lease.seq, before.lease.seq) << "the lease renewal that opened the round is a separate, earlier write and stays committed"; } + +#if USE_AWS_S3 +/// A refused `gc/state` precondition whose resolve read was ITSELF refused: nothing observed the key, +/// so the round must not report a competing leader nobody saw. It still aborts, and the next round +/// re-reads -- the behaviour is unchanged, only the claim the message makes. +/// +/// The S3 gate is the fault's, not the site's: the definitive-refusal classification that makes a +/// resolve read settle nothing rather than be reissued exists only for S3 errors. +TEST(CASGc, RoundCommitUnobservedConflictNamesNoCompetingLeader) +{ + class UnobservedCommitBackend : public InMemoryBackend + { + public: + std::expected write( + const String & key, const String & bytes, const std::optional & expected_value, + TransportAccess & access) override + { + /// Only the ROUND COMMIT advances `round`; the lease renewal writes the same key and must + /// pass through untouched. + if (arm && expected_value && key == gc_state_key) + { + const auto stored = InMemoryBackend::read(key, access); + if (stored && decodeGcState(bytes).round > decodeGcState(stored->bytes).round) + { + arm = false; + refuse_read = true; + return std::unexpected(RawConflict{}); + } + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + + std::optional read(const String & key, TransportAccess & access) override + { + if (refuse_read && key == gc_state_key) + { + refuse_read = false; + throw DB::S3Exception("UnobservedCommitBackend: the settling read is definitively refused", + Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); + } + return InMemoryBackend::read(key, access); + } + + String gc_state_key; + bool arm = false; + bool refuse_read = false; + }; + + auto backend = std::make_shared(); + auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + backend->gc_state_key = store->layout().gcStateKey(); + + Gc gc(store, kGc); + ASSERT_TRUE(gc.runRegularRound().acquired_lease); + backend->arm = true; + try + { + gc.runRegularRound(); + FAIL() << "a refused round commit must end the round"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::ABORTED); + EXPECT_NE(e.message().find("resolve read observed nothing"), String::npos) << e.message(); + EXPECT_EQ(e.message().find("another leader advanced it"), String::npos) + << "nothing observed the key, so no competing leader may be named: " << e.message(); + } +} +#endif diff --git a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp index f4d4e65cf91d..8d6133b75592 100644 --- a/src/Disks/tests/gtest_cas_lifecycle_condition.cpp +++ b/src/Disks/tests/gtest_cas_lifecycle_condition.cpp @@ -60,9 +60,21 @@ void fenceOutMount(Backend & backend, const String & mount_key) ASSERT_TRUE(std::holds_alternative(put)); } -/// A Backend decorator whose reads, heads and lists throw an untyped transport error while `fail` is -/// armed. Starts DISARMED so `Pool::open` succeeds; a test arms it only to make the identity probe -/// inconclusive. Mirrors gtest_cas_sentinel_probe.cpp's `TransportFaultBackend`, but toggleable AFTER open. +/// A Backend decorator whose reads, heads and lists throw a transport-classified error while `fail` is +/// armed, counting every attempt so a test can prove the probe path was actually reached (and stopped +/// where it should) rather than some other short-circuit. Starts DISARMED so `Pool::open` succeeds; a +/// test arms it only to make the identity probe inconclusive. Mirrors +/// gtest_cas_sentinel_probe.cpp's `TransportFaultBackend`, but toggleable AFTER open. +/// +/// The fault is `Poco::TimeoutException`: `Backend::probeSentinelRaw`'s default implementation (the one +/// `InMemoryBackend` uses) calls `head`/`read` directly and folds ANY exception from either into +/// `Indeterminate` with its own `catch (...)` -- so the exception never reaches `CasOperation`'s +/// transport-vs-local classification at all here. A `Poco::TimeoutException` is still the right class to +/// inject: it is what a real backend's probe would actually throw, and the point of the counters below +/// is to prove `head` was reached and actually failed, not skipped by some other short-circuit. +/// `tryRemountOnce` retries its own whole chain internally (well past the single probe attempt), so +/// the exact count per call is not pinned here -- only that a call growing it proves the fault path +/// stayed live across it, rather than a stale verdict being served from a cache. class ToggleableTransportFaultBackend final : public InMemoryBackend { public: @@ -70,32 +82,34 @@ class ToggleableTransportFaultBackend final : public InMemoryBackend using Backend::head; using Backend::list; - /// The faults sit on the TRANSPORT PRIMITIVES, because that is where every caller reaches the store: - /// the lifecycle gate probes `_pool_meta` through `probeSentinelRaw`, which speaks only these. A - /// legacy caller still reaches the fault, through the forwarder, so arming it here covers both - /// surfaces rather than only one. std::optional head(const String & key, TransportAccess & access) override { + ++head_attempts; if (fail.load()) - throw std::runtime_error("injected fault: transport error"); + throw Poco::TimeoutException("injected fault: transport error"); return InMemoryBackend::head(key, access); } std::optional read(const String & key, TransportAccess & access) override { + ++read_attempts; if (fail.load()) - throw std::runtime_error("injected fault: transport error"); + throw Poco::TimeoutException("injected fault: transport error"); return InMemoryBackend::read(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { + ++list_attempts; if (fail.load()) - throw std::runtime_error("injected fault: transport error"); + throw Poco::TimeoutException("injected fault: transport error"); return InMemoryBackend::list(prefix, cursor, limit, access); } std::atomic fail{false}; + std::atomic head_attempts{0}; + std::atomic read_attempts{0}; + std::atomic list_attempts{0}; }; } @@ -255,10 +269,18 @@ TEST(CASLifecycleCondition, ProbeTransportErrorStaysTransientAndRetries) EXPECT_EQ(store->lifecycle(), PoolLifecycle::TransientNotLive); EXPECT_FALSE(store->isVanished()); EXPECT_NO_THROW(store->throwIfLifecycleTerminal()); - - /// A second attempt with the fault still armed remains transient (retries continue). + /// The `_pool_meta` probe was actually reached and actually failed at `head` -- proving the + /// TransientNotLive verdict above came from the probe's own `Indeterminate` classification, not + /// from some other short-circuit that never touched the fault at all. + const uint64_t first_head_attempts = backend->head_attempts.load(); + EXPECT_GT(first_head_attempts, 0u); + + /// A second attempt with the fault still armed remains transient (retries continue) and probes + /// again -- proving each `tryRemountOnce` re-probes rather than caching the first call's + /// inconclusive verdict. EXPECT_FALSE(store->tryRemountOnce()); EXPECT_EQ(store->lifecycle(), PoolLifecycle::TransientNotLive); + EXPECT_GT(backend->head_attempts.load(), first_head_attempts); /// Disarm before teardown so `~Pool()`'s clean-farewell write is not fighting the injected fault. backend->fail.store(false); diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 1989c8cd5937..8d43ca7b7c4c 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -296,10 +296,21 @@ CasRequestBudget renewalLogBudget() } +/// `CASMountAudit.PhysicalRetryCannotBeDelayedByDebugLogging` was retired when mount renewal moved onto +/// `CasRequests`/`CasOperation` (the old hand-written renewal controller had a per-attempt progress +/// callback the test used to interleave a blocking debug log with the retry loop's own pacing; nothing +/// still exposes such a callback). Verified still true against the current engine, not just the +/// migration's own commit message: `grep -n "LOG_\|getLogger" .../Backend/CasRequests.cpp` finds exactly +/// one log call in the whole write-retry engine, `logCasWriteRetryLater`, reached only from the +/// `[[noreturn]]` `throwCasWriteRetryLater` -- the terminal give-up, called once, never between +/// attempts. No replacement test is needed: there is no per-attempt log call left to race. TEST(CASMountAudit, RenewalDefaultLogsAreBounded) { const auto open_store = [](const std::shared_ptr & backend, uint64_t & boot_ms, const String & prefix) { + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the + /// budget field alone; pair the two so the fence math below matches what admits. + backend->setAttemptTimeoutMs(renewalLogBudget().attempt_timeout_ms); return Pool::open(backend, PoolConfig{ .pool_prefix = prefix, .server_root_id = "test", @@ -347,8 +358,8 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) auto store = open_store(backend, boot_ms, "renewal-log-fenced"); ScopedRenewalLogCapture capture("information"); /// The lease was claimed at boot 100 with the 1000 ms TTL above, so it expires at 1100. The - /// fence admits only while the remaining time is strictly above the safety margin, and this - /// backend declares no attempt timeout, so the engine reserves nothing on top of that margin. + /// fence admits only while the remaining time strictly clears the safety margin plus whatever + /// the attempt reserves, so exactly `margin` remaining (with the reservation on top) refuses. boot_ms = 1100 - renewalLogBudget().lease_safety_margin_ms; EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); const String output = capture.captured(); @@ -1248,10 +1259,29 @@ TEST(CASMountReadOnly, ForeignOwnedPoolOpensWithoutMutation) EXPECT_EQ(epoch_after->bytes, epoch_before->bytes); } -/// Pool::open must call validateCasRequestBudget itself (not just the free function in isolation — -/// see gtest_cas_request_control.cpp for that): an inconsistent cas_request_budget must refuse a -/// writable mount end-to-end (RFC cas-s3-timeout-retry-control §required-timeout-model), never mount -/// silently with a budget that could let a controlled attempt outlive the lease it is fenced under. +/// `validateCasRequestBudget` itself, isolated from `Pool::open`: a consistent default budget is +/// accepted silently, and the overflow-safe comparison (subtraction against the TTL rather than +/// computing `attempt_timeout_ms + lease_safety_margin_ms` directly) really does reject an absurd +/// near-`UINT64_MAX` config rather than letting the sum wrap to a spuriously small value that would +/// pass the inequality when it should fail closed. +TEST(CASRequestBudget, ValidateAcceptsDefaultsAndRejectsAnOverflowingSumWithoutWrapping) +{ + EXPECT_NO_THROW(validateCasRequestBudget( + CasRequestBudget{}, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000)); + + const CasRequestBudget overflowing{ + .attempt_timeout_ms = std::numeric_limits::max() - 100, + .lease_safety_margin_ms = std::numeric_limits::max() - 100}; + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] + { + validateCasRequestBudget(overflowing, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); + }); +} + +/// Pool::open must call validateCasRequestBudget itself (not just the free function in isolation, +/// pinned directly above): an inconsistent cas_request_budget must refuse a writable mount end-to-end +/// (RFC cas-s3-timeout-retry-control §required-timeout-model), never mount silently with a budget that +/// could let a controlled attempt outlive the lease it is fenced under. TEST(CASMountStartup, RefusesWritableOpenWithInconsistentCasRequestBudget) { auto b = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp b/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp index 313b7f7e362d..6eb8f9c9b7b8 100644 --- a/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp +++ b/src/Disks/tests/gtest_cas_mount_claim_conflicts.cpp @@ -10,6 +10,7 @@ extern const int ABORTED; using namespace DB::Cas; using DB::Cas::tests::MountSlotRaceBackend; using DB::Cas::tests::expectThrowsCodeWithMessage; +using DB::Cas::tests::OperationForTest; namespace { @@ -185,3 +186,102 @@ TEST(CASMountClaimConflicts, FencedInsideAdoptionWindowRaisesMountFencedNotAbort auto renewer = makeRenewer(requests, now); EXPECT_THROW(renewer.start(), MountFencedException); } + +/// A raced claim reports a body its caller renders into the fail-closed operator message. Reporting +/// the PROPOSER's own lease there names this very server as the existing mount, which sends an +/// operator hunting a second process that is not the one holding the slot. The write's own resolve +/// read already observed the occupant, so that is what the result must carry. +TEST(CASMountClaimConflicts, ALostCreateReportsTheOccupantNotTheProposer) +{ + auto backend = std::make_shared(); + Layout layout("p"); + uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + /// Absent at our read; a foreign server mints the slot before our create lands. + backend->before_put_if_absent = [&] + { + CasOperation racer = requests.admit(); + ASSERT_EQ(claimMount(racer, layout, "r", DB::UInt128(2), 1, now, /*ttl_ms=*/100).kind, + MountClaimResult::Claimed); + }; + CasOperation op = requests.admit(); + const MountClaimResult claim = claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100); + + EXPECT_EQ(claim.kind, MountClaimResult::LiveDoubleStart); + ASSERT_TRUE(claim.body.has_value()); + EXPECT_EQ(claim.body->server_uuid, DB::UInt128(2)) << "the result named this server's own proposal"; + EXPECT_EQ(claim.body->writer_epoch, 1u); + ASSERT_TRUE(claim.etag.has_value()) << "the observed occupant's incarnation is what was read"; + EXPECT_NE(mountDoubleStartMessage("r", claim.body).find(u128ToHex(DB::UInt128(2))), String::npos) + << "the operator message must name the foreign holder"; +} + +/// The same for the refresh branch: a body that changed under our own adoption is the one the message +/// must name. The reclaim branch reaches the identical helper, so it is not repeated here. +TEST(CASMountClaimConflicts, ALostRefreshReportsTheObservedBodyNotTheProposer) +{ + auto backend = std::make_shared(); + Layout layout("p"); + uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation seed = requests.admit(); + ASSERT_EQ(claimMount(seed, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + MountClaimResult::Claimed); + /// A distinguishable body lands under our own refresh, so a result carrying the proposal cannot + /// pass by accident: our proposal would carry `seq` 2 and this process's pid. + backend->before_put_overwrite = [&] + { + OperationForTest racer(*backend); + const String key = layout.mountKey("r"); + const auto got = (*racer).read(key, Retry::standard()); + ASSERT_TRUE(got.has_value()); + MountLease raced = decodeMountLease(got->bytes); + raced.pid = 4242; + raced.seq = 99; + ASSERT_TRUE(std::holds_alternative( + (*racer).replace(key, encodeMountLease(raced), got->etag, Retry::standard()))); + }; + CasOperation op = requests.admit(); + const MountClaimResult claim = claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100); + + EXPECT_EQ(claim.kind, MountClaimResult::LiveDoubleStart); + ASSERT_TRUE(claim.body.has_value()); + EXPECT_EQ(claim.body->pid, 4242); + EXPECT_EQ(claim.body->seq, 99u); + ASSERT_TRUE(claim.etag.has_value()); +} + +/// The residue of the same rule: a raced write whose conflict settles to no observation saw nobody, so +/// there is no holder to name. Reporting the lease this server merely PROPOSED would put this very +/// process in the "Existing mount" line of an operator message -- the same defect as naming it after a +/// conflict that did observe someone. `ProvenAbsent` is the reachable half; `NotObserved` (the resolve +/// read itself failed) leaves through the same branch. +TEST(CASMountClaimConflicts, ARacedRefreshThatObservedNothingNamesNoHolder) +{ + auto backend = std::make_shared(); + Layout layout("p"); + uint64_t now = 1000; + CasRequests requests = openRequestsForTest(backend); + CasOperation seed = requests.admit(); + ASSERT_EQ(claimMount(seed, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100).kind, + MountClaimResult::Claimed); + /// The slot is removed under our own refresh, so the refused precondition resolves to a proven + /// absence rather than to an occupant. + backend->before_put_overwrite = [&] + { + OperationForTest racer(*backend); + const String key = layout.mountKey("r"); + const auto got = (*racer).read(key, Retry::standard()); + ASSERT_TRUE(got.has_value()); + ASSERT_EQ((*racer).remove(key, got->etag, Retry::standard()), Removal::Removed); + }; + CasOperation op = requests.admit(); + const MountClaimResult claim = claimMount(op, layout, "r", DB::UInt128(1), 7, now, /*ttl_ms=*/100); + + EXPECT_EQ(claim.kind, MountClaimResult::LiveDoubleStart); + EXPECT_FALSE(claim.etag.has_value()); + EXPECT_FALSE(claim.body.has_value()) << "a result that observed nobody reported a lease anyway"; + const String message = mountDoubleStartMessage("r", claim.body); + EXPECT_NE(message.find("could not be observed"), String::npos) + << "the message named a holder nobody saw: " << message; +} diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index 605a78cd8dbe..bffce59e2013 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -148,6 +148,7 @@ TEST(CASObservability, RenewalCountersHaveExactPhysicalAndLogicalDeltas) const auto run = [](RenewalCounterBackend::Fault fault, uint64_t attempts, uint64_t retries, uint64_t resolved, uint64_t recovered) { auto backend = std::make_shared(); + backend->setAttemptTimeoutMs(renewalCounterBudget().attempt_timeout_ms); uint64_t boot_ms = 100; auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "renewal-counter-" + std::to_string(attempts) + "-" + std::to_string(resolved), @@ -171,6 +172,7 @@ TEST(CASObservability, RenewalCountersHaveExactPhysicalAndLogicalDeltas) TEST(CASObservability, ExternalLeaseDeadlineCountsOnceWithoutReconstructingAttempts) { auto backend = std::make_shared(); + backend->setAttemptTimeoutMs(renewalCounterBudget().attempt_timeout_ms); uint64_t boot_ms = 100; auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "renewal-deadline-counter", @@ -181,9 +183,9 @@ TEST(CASObservability, ExternalLeaseDeadlineCountsOnceWithoutReconstructingAttem }); /// The fence deadline is 1100 and the safety margin 20, so admission refuses once fewer than - /// twenty milliseconds of lease remain. At 1090 nothing can be started and the logical renewal ends - /// without reconstructing a sent attempt. (Not 1071: the engine reserves the backend's own attempt - /// timeout, which is zero for an in-memory backend, so 29 ms of remaining lease is still room.) + /// twenty milliseconds of lease remain. At 1090 only ten milliseconds remain, short of the margin + /// however much a single attempt reserves, so nothing can be started and the logical renewal ends + /// without reconstructing a sent attempt. boot_ms = 1090; const RenewalCounterSnapshot before = renewalCounters(); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index afec5b1bfbe1..878fd536e1e5 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -196,7 +196,7 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend if (key == target_key && !fired) { fired = true; - /// GC's single content-delete site, landing in the HEAD->GET window. + /// Fires here: after the inner HEAD observation, before it is returned to the writer. inner->remove(target_key, condemned_value, access); } return observed; @@ -2766,12 +2766,16 @@ TEST(CASPartWrite, EveryPhysicalPublicationMintsAFreshIncarnationTag) << "a repeated incarnation_tag is a repeated incarnation: GC's condemn would name the live body"; } -/// "One `Retry::standard()`" is one policy VALUE, not one shared deadline: each verb of the -/// publication loop binds its own window from it, so a loop that has already spent far more than one -/// window still publishes. Here each ambiguous publication burns forty seconds of the injected clock, -/// so three of them exceed the standard ninety-second window; a loop that carried a single bound -/// across its iterations would refuse the fourth iteration's HEAD instead of committing. -TEST(CASPartWrite, EnsureBlobPresentSharesOneRetryAcrossItsLoop) +/// The publication loop captures ONE bound before it starts, and every verb of every iteration shares +/// it -- so eight iterations cannot spend eight ninety-second windows. Here each ambiguous +/// publication burns forty seconds of the injected clock, so the third one carries the loop past the +/// window it captured and the next iteration's HEAD refuses to start. The insert is refused as +/// retry-later, which is what a caller can act on; the alternative is a single blob upload sitting on +/// the request for twenty-five minutes. The eight-attempt cap stays as the secondary bound. +/// +/// The throw and the publication count are what fail if the shared bound regresses: without it this +/// same fixture publishes a fourth time and the call SUCCEEDS. +TEST(CASPartWrite, EnsureBlobPresentIsBoundedByTheOneWindowItCaptured) { auto b = std::make_shared(); auto s = openBlobFaultPool(b); @@ -2788,11 +2792,15 @@ TEST(CASPartWrite, EnsureBlobPresentSharesOneRetryAcrossItsLoop) int payload_streams = 0; b->fault_count = 3; - const PutBlobResult res = build->putBlob(idOf(payload), countingSource(payload, payload_streams)); - EXPECT_EQ(res.size, payload.size()); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + { + (void)build->putBlob(idOf(payload), countingSource(payload, payload_streams)); + }); - EXPECT_EQ(b->publish_stream_attempts, 4) << "three ambiguous publications + the committing fourth"; - EXPECT_GT(now->load(), 90'000u) << "the loop outlived one standard window, which is the point"; + EXPECT_EQ(b->publish_stream_attempts, 3) << "the fourth iteration must not start a publication"; + /// The clock is past the captured deadline when the loop stops, which is what stopped it -- the + /// third publication ends at 120 s of a window that closed at 90 s. + EXPECT_GT(now->load(), 90'000u); } namespace diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index f524dc9b81bb..c490332f50eb 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -1879,8 +1879,8 @@ CasRequestBudget runtimeRenewBudget(); class RuntimeUnderTest { public: - template - RuntimeUnderTest(DB::Cas::BackendPtr backend, Args &&... args) + template + RuntimeUnderTest(const std::shared_ptr & backend, Args &&... args) : mount(backend, DB::Cas::Fence{ [this] { return runtime.fenceGeneration(); }, [this](uint64_t g, uint64_t needed) { return runtime.admit(g, needed); }, @@ -1888,6 +1888,10 @@ class RuntimeUnderTest , farewell(backend, DB::Cas::Fence::open()) , runtime(backend, mount, farewell, std::forward(args)...) { + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the + /// budget field alone; every construction of this holder pairs the two via `runtimeRenewBudget`, + /// the sole budget it is ever built with in this file. + backend->setAttemptTimeoutMs(runtimeRenewBudget().attempt_timeout_ms); /// The runtime arms its lease deadline on ITS boot clock, and the engine measures that deadline /// against the clock it reads. Production runs both on `CLOCK_BOOTTIME`, so they agree; a test /// that injects one MUST inject the other, or `Retry::untilLeaseSafe` compares a synthetic @@ -2223,6 +2227,9 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); uint64_t fake_boot = 1'000'000; auto store = DB::Cas::Pool::open(backend, DB::Cas::PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget, @@ -2524,6 +2531,9 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); uint64_t fake_boot = 1'000'000; std::vector waits; auto store = Pool::open(backend, PoolConfig{ @@ -2590,6 +2600,9 @@ TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPrede budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); uint64_t fake_boot = 1'000'000; auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", diff --git a/src/Disks/tests/gtest_cas_probe.cpp b/src/Disks/tests/gtest_cas_probe.cpp index d7acb16a0ff3..6b9da190dcad 100644 --- a/src/Disks/tests/gtest_cas_probe.cpp +++ b/src/Disks/tests/gtest_cas_probe.cpp @@ -60,11 +60,10 @@ TEST(CASProbe, PassesOnEmulatedLocal) EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")); } -/// B135: two servers mounting the SAME shared CA pool concurrently must not race on the probe keys. +/// Two servers mounting the SAME shared CA pool concurrently must not race on the probe keys. /// We simulate "a concurrent mounter's probe is in flight" by PRE-SEEDING the fixed-name probe key -/// `/_probe/token` over a shared backend, then opening the Pool. With the OLD fixed-key probe -/// the open's `putIfAbsent("/_probe/token", …)` returns PreconditionFailed and `Pool::open` -/// throws NOT_IMPLEMENTED ("putIfAbsent on a fresh key returned PreconditionFailed"). With the +/// `/_probe/token` over a shared backend, then opening the Pool. A fixed-key probe would meet +/// the seeded object as a refused precondition on its own `create` and fail the open; with the /// per-mount unique probe prefix `/_probe//token`, the seeded key does not collide and /// the open succeeds — exactly the concurrent-shared-pool-mount behaviour we need. /// diff --git a/src/Disks/tests/gtest_cas_record_stream_format.cpp b/src/Disks/tests/gtest_cas_record_stream_format.cpp index a5309f4d916a..e796b394ca8b 100644 --- a/src/Disks/tests/gtest_cas_record_stream_format.cpp +++ b/src/Disks/tests/gtest_cas_record_stream_format.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -192,14 +193,42 @@ TEST(CASRecordStream, ActiveRowCarryingACondemnedFieldFailsClosed) } } +/// `CasBlobInDegree.cpp`'s fold (`SourceEdgeRunWriter writer(out); // sorted NDJSON; byte-deterministic +/// for write-once adoption`) sorts by `(ref, source_id)` before writing, so the run this fixture emits +/// is the writer's own straight-line append, never a reorder -- the property the run must actually have +/// is that whichever order the SAME set of edges was DISCOVERED in, the canonical `(ref, source_id)` +/// sort before the writer sees them converges on identical bytes. A bare `f(x) == f(x)` on one fixed, +/// already-sorted vector cannot fail on anything but genuine cross-call nondeterminism (a clock, a +/// pointer, hash randomization) -- none of which this format has -- so it passed vacuously; two +/// differently-DISCOVERED inputs, sorted by the same comparator production uses, is the real claim. TEST(CASRecordStream, WriterIsByteDeterministic) { - std::vector recs = { + const auto byRefThenSource = [](const SourceEdgeRecord & a, const SourceEdgeRecord & b) + { + if (a.ref != b.ref) + return a.ref < b.ref; + return a.source_id < b.source_id; + }; + + std::vector discovered_ascending = { edge(chRef(1), 5), edge(chRef(1), 9), condemned(chRef(2), PersistedEtag{"etag", "t/with/slashes"}, 1, 2, false), }; - EXPECT_EQ(encodeRun(recs), encodeRun(recs)); /// pure function of the sorted record set + /// The SAME three edges, as if a different GC shard or a different LIST page order had surfaced + /// them: reverse discovery order, still every one of them present. + std::vector discovered_reverse = { + condemned(chRef(2), PersistedEtag{"etag", "t/with/slashes"}, 1, 2, false), + edge(chRef(1), 9), + edge(chRef(1), 5), + }; + ASSERT_NE(discovered_ascending.front().ref, discovered_reverse.front().ref) + << "the two discovery orders must actually differ"; + + std::sort(discovered_ascending.begin(), discovered_ascending.end(), byRefThenSource); + std::sort(discovered_reverse.begin(), discovered_reverse.end(), byRefThenSource); + EXPECT_EQ(encodeRun(discovered_ascending), encodeRun(discovered_reverse)) + << "the run must be a pure function of the edge SET, not of the order it was discovered in"; } /// The run `ref` carries the algorithm as a raw leading BYTE, a second representation of the same diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index a7aaaa0fbbdb..129504bf4257 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -118,6 +118,8 @@ CatalogEntry entryInState(const String & ns, NsState state, uint64_t inc) class WriteCountingBackend : public DB::Cas::tests::CountingBackend { public: + enum class Verb { Read, Write }; + uint64_t writes(const String & key) const { std::lock_guard lock(write_count_mutex); @@ -125,6 +127,28 @@ class WriteCountingBackend : public DB::Cas::tests::CountingBackend return it == write_counts.end() ? 0 : it->second; } + /// The ordered READ/WRITE sequence issued against `key` since this backend was created. A count + /// alone cannot tell a settled-then-reissued attempt from a blind reissue that happened to read + /// more times; the order is what a caller actually needs to pin. + std::vector journalFor(const String & key) const + { + std::lock_guard lock(write_count_mutex); + std::vector filtered; + for (const auto & [journaled_key, verb] : journal) + if (journaled_key == key) + filtered.push_back(verb); + return filtered; + } + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + { + std::lock_guard lock(write_count_mutex); + journal.emplace_back(key, Verb::Read); + } + return CountingBackend::read(key, access); + } + std::expected write(const String & key, const String & bytes, const std::optional & expected_value, DB::Cas::TransportAccess & access) override @@ -132,6 +156,7 @@ class WriteCountingBackend : public DB::Cas::tests::CountingBackend { std::lock_guard lock(write_count_mutex); ++write_counts[key]; + journal.emplace_back(key, Verb::Write); } return CountingBackend::write(key, bytes, expected_value, access); } @@ -139,8 +164,19 @@ class WriteCountingBackend : public DB::Cas::tests::CountingBackend private: mutable std::mutex write_count_mutex; std::map write_counts; + std::vector> journal; }; +/// A compact 'R'/'W' rendering of `WriteCountingBackend::journalFor`, so a mismatch prints as one +/// readable string rather than a wall of enum values. +String renderJournal(const std::vector & journal) +{ + String rendered; + for (const auto verb : journal) + rendered += verb == WriteCountingBackend::Verb::Read ? 'R' : 'W'; + return rendered; +} + /// Lands a competing catalog body under the erase's own attempt and withdraws this actor's admission /// with it -- the concurrent winner an erase has to be resolved against, driven deterministically and /// without a second thread. @@ -1386,17 +1422,30 @@ TEST(CASRefCatalogRemoval, ATransientEraseFailureIsResolvedByAReadAndReissued) .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); - const uint64_t reads_before = backend->getCount(layout.refCatalogKey()); + const size_t journal_before = backend->journalFor(layout.refCatalogKey()).size(); backend->failNextWriteWith(layout.refCatalogKey(), std::make_exception_ptr( Poco::TimeoutException("injected erase failure whose outcome never reached the caller"))); EXPECT_EQ(CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, noAuthorityRefresh), CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + + /// A blind reissue still performs a baseline read, a post-write read and a verification read, + /// satisfying a bare count. The ORDER pins what a count cannot: the lost attempt's outcome is + /// settled by a read (the engine's own ambiguity resolution) before it is ever reissued, and the + /// reissue that lands is settled by the loop's mandatory resolution read in turn. Captured before + /// the test's own verification read below, which is not part of the call under test. + const std::vector full_journal = backend->journalFor(layout.refCatalogKey()); + const std::vector journal_since( + full_journal.begin() + static_cast(journal_before), full_journal.end()); + using Verb = WriteCountingBackend::Verb; + EXPECT_EQ(journal_since, (std::vector{Verb::Read, Verb::Write, Verb::Read, Verb::Write, Verb::Read})) + << "got " << renderJournal(journal_since) << "; a baseline read, the lost attempt settled by the " + << "engine's own resolution read BEFORE it is reissued, then the reissue settled by the loop's " + << "mandatory resolution read -- not a blind reissue that happens to read more times"; + EXPECT_TRUE(CasRefCatalog::read(op, layout).catalog.entries.empty()); EXPECT_EQ(backend->writes(layout.refCatalogKey()), 3u) << "the seed, the attempt whose outcome was lost, and the reissue that landed"; - EXPECT_GT(backend->getCount(layout.refCatalogKey()), reads_before + 1) - << "the lost attempt was settled by an exact read before anything was concluded from it"; } /// A refused precondition is the only thing the erase loop retries, and it PACES that retry on the @@ -1484,6 +1533,86 @@ TEST(CASRefCatalogRemoval, TheEraseLoopRefreshesItsLivenessBeforeEveryAttempt) << "the row a deposed leader must not erase is still there"; } +/// The refresh hook can fail the same way the read it wraps can -- and a failure of it is not a +/// negative liveness answer to fold into `FencedOut`; it is a fact this call cannot evaluate, so it +/// must escape rather than be swallowed into any of the loop's own outcomes. This is the case where +/// the hook fails before the loop has sent anything at all: no erase may reach the store on an +/// authority this call could not even ask about. +TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesBeforeEraseCas) +{ + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + const Layout layout("erase-refresh-throws-before"); + const CatalogEntry removing{ + .ns = RootNamespace{"a"}, + .state = NsState::Removing, + .incarnation = UInt128{7}, + .removal_started_round = 13}; + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const uint64_t writes_after_seed = backend->writes(layout.refCatalogKey()); + + EXPECT_THROW( + CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, + [] { throw std::runtime_error("injected authority-refresh failure before the first attempt"); }), + std::runtime_error); + EXPECT_EQ(backend->writes(layout.refCatalogKey()), writes_after_seed) + << "an authority this call could not even ask about must not send an erase"; + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{removing}) + << "nothing changed under an authority failure that reached no attempt"; +} + +/// The counterpart to the case above: the hook fails AFTER the loop has already sent one erase and +/// resolved it (a refusal, so the row is provably unchanged), on the refresh that would gate the next +/// attempt. The failure must still escape rather than be read as the fenced-out liveness answer this +/// class's default `admitted()` would otherwise report, and -- exactly as when it fails up front -- no +/// further erase may reach the store on an authority this call could not evaluate. +TEST(CASRefCatalogRemoval, NonFenceAuthorityExceptionPropagatesAfterEraseResolution) +{ + auto backend = std::make_shared(); + DB::Cas::tests::FakeClock clock; + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + requests.setNowFnForTest(clock.nowFn()); + requests.setSleepFnForTest(clock.sleepFn()); + CasOperation op = requests.admit(); + const Layout layout("erase-refresh-throws-after"); + const CatalogEntry removing{ + .ns = RootNamespace{"a"}, + .state = NsState::Removing, + .incarnation = UInt128{7}, + .removal_started_round = 13}; + seedObject(op, layout.refCatalogKey(), encodeRefCatalog(RefCatalog{.entries = {removing}})); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + + backend->refuseNextWrite(layout.refCatalogKey()); + const uint64_t writes_after_seed = backend->writes(layout.refCatalogKey()); + + size_t refreshes = 0; + EXPECT_THROW( + CasRefCatalog::deleteCompletedRemoving(op, layout, removing, ready_parent, [&] + { + ++refreshes; + if (refreshes > 1) + throw std::runtime_error("injected authority-refresh failure after the erase resolved"); + }), + std::runtime_error); + EXPECT_EQ(refreshes, 2u) << "once before the refused attempt, once before the retry it never sent"; + EXPECT_EQ(backend->writes(layout.refCatalogKey()) - writes_after_seed, 1u) + << "the one refused erase; the retry the resolution read would have paced never reached the store"; + EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{removing}) + << "the row a failed authority refresh must not let a later attempt erase is still there"; +} + /// The loop iterates on ONE alternative and one only: a refused precondition. Every other non-committed /// answer is terminal for the call and leaves through the same throw, so a future retry added for any /// of them would be retrying a write whose fate the store already settled. `Refused` is the alternative @@ -1583,11 +1712,13 @@ TEST(CASRefCatalogRemoval, ACommitTheResolutionReadContradictsFailsRetryLaterIns << "the message must name the life still observed: " << message; } -/// After the migration this cap is the ONLY bound the hand-written loop has of its own, so it is worth -/// proving it ends the call rather than letting a permanently contended catalog spin. Every erase is -/// refused, the injected clock absorbs every paced retry, and the loop stops on its attempt count -- -/// which the message says, and which is what tells it apart from a deadline. -TEST(CASRefCatalogRemoval, PerpetualConflictEndsAtTheAttemptCapAndSaysSo) +/// The loop captures ONE bound before it starts and every call it makes shares it, so a permanently +/// contended catalog gives up "retry later" inside one standard window rather than spending a fresh +/// window per verb across a hundred paced iterations -- which is hours against a document that +/// promises ninety seconds. Every erase is refused and the injected clock absorbs every paced retry, +/// so what ends the call is visible in the virtual time it took. The attempt cap stays as the +/// secondary bound; it is not what ends this call. +TEST(CASRefCatalogRemoval, PerpetualConflictGivesUpWithinOneWindowNotAtTheAttemptCap) { class AlwaysRefusesCatalogWrites final : public WriteCountingBackend { @@ -1637,6 +1768,7 @@ TEST(CASRefCatalogRemoval, PerpetualConflictEndsAtTheAttemptCapAndSaysSo) .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); backend->refused_key = layout.refCatalogKey(); + const uint64_t start = clock.now; String message; try { @@ -1648,12 +1780,14 @@ TEST(CASRefCatalogRemoval, PerpetualConflictEndsAtTheAttemptCapAndSaysSo) EXPECT_EQ(e.code(), DB::ErrorCodes::NETWORK_ERROR); message = e.message(); } - EXPECT_NE(message.find("did not converge"), String::npos) - << "the cap, not a deadline, is what ended this call: " << message; - /// One erase per iteration and one pause after each, so these agree exactly -- and both being - /// greater than one is what proves the loop iterated rather than failing on its first attempt. - EXPECT_EQ(backend->refusedAttempts(), clock.sleeps.size()); - EXPECT_GT(clock.sleeps.size(), 1u); + EXPECT_NE(message.find("deadline"), String::npos) + << "the shared bound, not the attempt cap, is what ended this call: " << message; + /// Two windows, so the assertion survives the jitter of the paced retries while still failing a + /// loop that binds a fresh window per iteration -- that one spends minutes here. + EXPECT_LT(clock.now - start, 2 * 90'000u) << "the loop outlived the window it captured"; + EXPECT_LT(backend->refusedAttempts(), 100u) << "the attempt cap must not be what ends this call"; + /// Greater than one is what proves the loop iterated rather than failing on its first attempt. + EXPECT_GT(backend->refusedAttempts(), 1u); EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{removing}); } diff --git a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp index f2628ab62b7b..18f279620f62 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp @@ -18,22 +18,22 @@ extern const int CORRUPTED_DATA; extern const int NETWORK_ERROR; } -/// Stage B Task 4-C: production birth wiring. `CasRefLedger::resolveNamespaceLife`, called from +/// Stage B: production birth wiring. `CasRefLedger::resolveNamespaceLife`, called from /// `ensureRefTableRecovered`, resolves a namespace's real catalog life ONCE per table-open -- /// create-if-absent, adopt an existing `Live`/`Removing` entry, or reconcile a stale `Creating` one via /// `CasRefCatalog::reconcileStaleCreator` + `isCreatorFenceTerminal` -- so every ref-layer object a /// mounted writer produces is keyed at a real, catalog-proven incarnation (spec INV-3), never the /// Stage-A sentinel. /// -/// OBLIGATION 3 (carried from Task 3's review, closed here): Task 3 could only enforce "`Creating` -/// forbids publication" (`CasRefCatalog::checkPublicationAdmittedOrThrow`) AT THE CATALOG LEVEL, because -/// nothing on the production ref-write path consulted the catalog at all. The refusal this suite pins -/// below rests on CONSTRUCTION, not a check: there is no `if (state == Creating) throw` anywhere in -/// `appendRefOps`'s path. `ensureRefTableRecovered` simply cannot make a table's runtime usable -/// (`rt.recovered` never becomes `true`, `rt.life` never gets set) while the catalog entry is `Creating` -/// under a fence that is not provably dead -- so no append can reach `commitRefChunk` for such a -/// namespace, by construction, stronger than any per-write check could prove. Stated here so nobody -/// later greps for a check and concludes the gap Task 3's review flagged is still open. +/// OBLIGATION 3 (closed here): `CasRefCatalog::checkPublicationAdmittedOrThrow` can only enforce +/// "`Creating` forbids publication" AT THE CATALOG LEVEL, because nothing on the production ref-write +/// path consulted the catalog at all. The refusal this suite pins below rests on CONSTRUCTION, not a +/// check: there is no `if (state == Creating) throw` anywhere in `appendRefOps`'s path. +/// `ensureRefTableRecovered` simply cannot make a table's runtime usable (`rt.recovered` never becomes +/// `true`, `rt.life` never gets set) while the catalog entry is `Creating` under a fence that is not +/// provably dead -- so no append can reach `commitRefChunk` for such a namespace, by construction, +/// stronger than any per-write check could prove. Stated here so nobody later greps for a check and +/// concludes this gap is still open. /// /// The suite name is prefixed `Cas` so it is covered by the `Cas*` unit-test gate filter. @@ -87,10 +87,14 @@ class WriteCountingBackend : public CountingBackend /// still be allowed to prove a new pool, and the failed first attempt must not have published /// `_pool_meta` without the catalog it makes mandatory. /// -/// A plain `std::runtime_error`, not a `Poco::Exception`: the engine's write loop treats any -/// `Poco`/transport exception as an ambiguity it settles itself with one resolve read, and a one-shot -/// fault of that class is retried and silently succeeds within the SAME `Pool::open` call -- it never -/// reaches the caller at all. A non-`Poco` `std::exception` is the engine's own signal for "this could +/// A plain `std::runtime_error`, not a `Poco::Exception`, and deliberately so: a `Poco`/transport +/// exception here would exercise the write loop's OWN ambiguity resolution rather than this suite's +/// subject, which is what `FailedCatalogBootstrapDoesNotPublishPoolMetaAndRetryConverges` actually +/// needs -- a fault that propagates out of the FIRST `Pool::open` call so a SEPARATE retry can be the +/// one that converges. The engine's write loop treats any `Poco`/transport exception as an ambiguity it +/// settles itself with one resolve read, and a one-shot fault of that class is retried and silently +/// succeeds within the SAME `Pool::open` call -- it never reaches the caller at all. A non-`Poco` +/// `std::exception` is the engine's own signal for "this could /// not have landed" and propagates unresolved, which is what "before it reaches durable storage" means. class CatalogBootstrapWriteFailsOnceBackend final : public WriteCountingBackend { @@ -395,10 +399,10 @@ TEST(CASRefCatalogBirthWiring, ANamespaceStuckCreatingUnderALiveForeignFenceRefu EXPECT_EQ(backend->writeTotal(), 0u); } -/// The mirror image, and Task 3's own deferred obligation ("wire `reconcileStaleCreator` and pin it -/// with a test that drives reconciliation through the discovery path rather than by calling the -/// primitive directly"): a dead predecessor's `Creating` entry is reconciled onto THIS mount and -/// completed to `Live`, over the SAME incarnation -- resumption, not rebirth. +/// The mirror image, and the deferred obligation to wire `reconcileStaleCreator` and pin it with a +/// test that drives reconciliation through the discovery path rather than by calling the primitive +/// directly: a dead predecessor's `Creating` entry is reconciled onto THIS mount and completed to +/// `Live`, over the SAME incarnation -- resumption, not rebirth. TEST(CASRefCatalogBirthWiring, AStaleCreatingEntryFromATerminatedForeignFenceIsReconciledThroughTheProductionPath) { auto backend = std::make_shared(); @@ -588,3 +592,85 @@ TEST(CASRefCatalogBirthWiring, ExactOldLifeCannotCancelReplacementTerminalCreati EXPECT_EQ(backend->deleteTotal(), 0u); EXPECT_EQ(CasRefCatalog::read(op, layout).catalog.entries, std::vector{successor}); } + +namespace +{ + +/// Leaves the catalog holding a body no previously observed entry equals: every read first bumps each +/// entry's creator fence generation, so `reconcileStaleCreator`'s token-exactness check refuses on +/// every attempt and `resolveNamespaceLife`'s state machine can never converge. +class ChurningCatalogBackend final : public InMemoryBackend +{ +public: + explicit ChurningCatalogBackend(String catalog_key_) + : catalog_key(std::move(catalog_key_)) + { + } + + bool churning = false; + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + if (churning && key == catalog_key) + bumpEveryCreatorFence(access); + return InMemoryBackend::read(key, access); + } + +private: + /// Qualified calls, never the virtual ones: this must not re-enter its own churn. + void bumpEveryCreatorFence(DB::Cas::TransportAccess & access) + { + const std::optional got = InMemoryBackend::read(catalog_key, access); + if (!got) + return; + RefCatalog catalog = decodeRefCatalog(got->bytes); + for (CatalogEntry & entry : catalog.entries) + if (entry.creator) + ++entry.creator->fence_generation; + (void)InMemoryBackend::write(catalog_key, encodeRefCatalog(catalog), got->value, access); + } + + const String catalog_key; +}; + +} + +/// A catalog entry that moves under every read drives `resolveNamespaceLife`'s state machine for ever. +/// One `Retry` frozen before the loop bounds the WHOLE resolution to a single standard window, and +/// every re-read a competing actor forces is paced by a jittered sleep -- so a permanently churning +/// catalog costs one window, not one fresh window per verb per iteration hammered with no wait between +/// them. +/// +/// What the clock bound below does NOT check: it bounds this call's own wall time, not the number of +/// requests the loop sent, and it says nothing about the paths that converge. +TEST(CASRefCatalogBirthWiring, APerpetuallyChurningCatalogEntryIsPacedAndEndsWithinOneWindow) +{ + auto backend = std::make_shared(Layout{"p"}.refCatalogKey()); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation op = requests.admit(); + auto store = openPoolForBirthTest(backend); + const Layout & layout = store->layout(); + const RootNamespace ns{"churning_creating"}; + + /// A FOREIGN creator, so the loop takes the reconciliation branch on every iteration. + const CatalogEntry creating{ + .ns = ns, + .state = NsState::Creating, + .incarnation = UInt128{0xc001}, + .creator = CreatorFence{.server_root_id = "foreign-creator", .writer_epoch = 7, .fence_generation = 1}}; + CasRefCatalog::casAdmitEntry(op, layout, 1, creating); + + auto clock = DB::Cas::tests::VirtualRetryClock::installOn(store); + backend->churning = true; + const size_t pauses_before = clock->pauseCount(); + const uint64_t now_before = clock->nowMs(); + + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->namespaceLife(ns); }); + + /// Sixteen jittered draws cannot exhaust a 90 s window even at their ceiling, so an unpaced loop + /// reaches its iteration cap having slept nothing at all. + EXPECT_GE(clock->pauseCount() - pauses_before, 16u) << "each forced re-read is paced"; + /// The frozen window, plus at most one backoff draw the pace does not consult the deadline for, + /// plus the virtual clock's one extra millisecond per pause. + EXPECT_LE(clock->nowMs() - now_before, 95'100u) << "one standard window bounds the whole loop"; +} diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index 7775169a36af..cbfe52b9e268 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -823,6 +823,9 @@ ChunkFailureOutcome runChunkFailureCase(const String & ns_suffix, ChunkFaultBack budget.attempt_timeout_ms = 100; budget.lease_safety_margin_ms = 100; cfg.cas_request_budget = budget; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); auto store = openPoolWith(backend, cfg); auto clock = VirtualRetryClock::installOn(store); const DB::Cas::Layout & layout = store->layout(); diff --git a/src/Disks/tests/gtest_cas_ref_ckpt.cpp b/src/Disks/tests/gtest_cas_ref_ckpt.cpp index f9c5e0e29bb1..f3f7e08efbd7 100644 --- a/src/Disks/tests/gtest_cas_ref_ckpt.cpp +++ b/src/Disks/tests/gtest_cas_ref_ckpt.cpp @@ -18,6 +18,7 @@ #include +#include #include #include #include @@ -104,7 +105,7 @@ RefCkpt readCkptOrFail(CasOperation & op, const Layout & layout, const Namespace return sample->ckpt; } -/// Stage B (Task 4-C): the incarnation `store`'s production birth wiring minted for `ns`, learned back +/// Stage B: the incarnation `store`'s production birth wiring minted for `ns`, learned back /// from the catalog exactly as a real reader would (`NamespaceLifeId::fromCatalogEntry`) -- once a real /// `Pool`/`CasRefLedger` has opened the table, its ref-layer objects are no longer keyed at the /// Stage-A sentinel, so every test below that drives the REAL append lane must ask the catalog what @@ -189,6 +190,9 @@ class CkptProbeBackend : public WriteCountingBackend std::function after_write; std::function after_read; std::vector journal; + /// How many reads `fail_reads_after_the_first` actually made throw, so a test can assert the fault + /// really fired rather than infer it from the journal's shape alone. + size_t read_fault_hits = 0; /// A test that must watch a namespace's `_ckpt` key cannot compute it before the pool exists -- /// the real incarnation is minted only once the namespace's first open resolves it. So the watch @@ -198,6 +202,7 @@ class CkptProbeBackend : public WriteCountingBackend watched_key = std::move(key); watched_reads = 0; journal.clear(); + read_fault_hits = 0; } void arm(const String & key, Fault fault_) @@ -214,7 +219,10 @@ class CkptProbeBackend : public WriteCountingBackend journal.push_back("READ"); ++watched_reads; if (watched_reads >= 2 && fail_reads_after_the_first) + { + ++read_fault_hits; throw Poco::TimeoutException("CkptProbeBackend: read response lost"); + } auto result = WriteCountingBackend::read(key, access); if (after_read) after_read(); @@ -876,10 +884,23 @@ TEST(CASRefCheckpoint, AFailedResolveReadNeverReportsACommitAndNeverSkipsTheRead publishCkpt(op, layout, life, RefCkpt{.life_epoch = std::nullopt, .committed_through = ID_1_2, .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); }); - for (size_t i = 1; i < backend->journal.size(); ++i) - EXPECT_FALSE(backend->journal[i] == "WRITE" && backend->journal[i - 1] == "WRITE") - << "two attempts with no exact observation between them, at journal position " << i; - EXPECT_GE(std::count(backend->journal.begin(), backend->journal.end(), String{"WRITE"}), 1); + /// `{READ, WRITE}` alone would satisfy "no adjacent writes" and "at least one write" without the + /// resolving read ever having been attempted, let alone failed. Pin that at least one read follows + /// the write, that the fault double actually fired on every one of them (a resolving read is itself + /// retried against the policy deadline, so several follow, not just one), and that no reissue was + /// sent while every resolution read fails. + ASSERT_GE(backend->journal.size(), 3u); + EXPECT_EQ(backend->journal.front(), "READ") << "the baseline read of the current state"; + EXPECT_EQ(backend->journal[1], "WRITE") << "the attempt that never committed"; + const size_t resolve_reads = backend->journal.size() - 2; + EXPECT_TRUE(std::all_of(backend->journal.begin() + 2, backend->journal.end(), + [](const String & verb) { return verb == "READ"; })) + << "no reissue can be sent while every resolution read fails, so nothing after the write is a WRITE"; + EXPECT_EQ(backend->read_fault_hits, resolve_reads) + << "every read after the baseline failed -- the fault double actually fired on all of them, not just " + << "the first"; + EXPECT_EQ(std::count(backend->journal.begin(), backend->journal.end(), String{"WRITE"}), 1) + << "no reissue can be sent while every resolution read fails"; backend->watched_key.clear(); backend->fail_reads_after_the_first = false; EXPECT_EQ(readCkptOrFail(reader, layout, life), base); @@ -1075,7 +1096,7 @@ TEST(CASRefCheckpoint, NamespaceBirthCreatesTheCheckpointCarryingItsLifeEpoch) CasOperation op = requests.admit(); const RootNamespace ns{"srv1/ckpt_birth"}; - /// Stage B (Task 4-C): the catalog carries no entry for `ns` before its first open, and the + /// Stage B: the catalog carries no entry for `ns` before its first open, and the /// namespace's real incarnation does not exist to name a key with yet -- the pre-birth analog of /// "nothing exists" is "nothing is even NAMED", checked at the catalog rather than at a key this /// test cannot yet compute. @@ -1317,7 +1338,7 @@ TEST(CASRefCheckpoint, APublishFencedOutMidAttemptDoesNotAdvanceTheCheckpoint) } /// =================================================================================== -/// Equivalence fences for the `prepareRefChunk` extraction (Stage B `{#extract-prepare-ref-chunk}`) +/// Equivalence fences for the `prepareRefChunk` extraction /// =================================================================================== /// /// An extraction is only safe to review if something pins what crosses its boundary. These three @@ -1342,9 +1363,9 @@ TEST(CASRefCheckpoint, CommitRefChunkDurableBytesUnchangedByExtraction) ASSERT_EQ(id.writer_epoch, 1u); ASSERT_EQ(id.ref_sequence, 1u); - /// The KEY carries the namespace incarnation, so its life segment is rendered rather than pasted - /// (Task 1c re-keys it); every other segment is literal. Stage B (Task 4-C): the incarnation is now - /// a REAL, randomly minted catalog value rather than the Stage-A sentinel, so it is learned back + /// The KEY carries the namespace incarnation, so its life segment is rendered rather than pasted; + /// every other segment is literal. Stage B: the incarnation is now a REAL, randomly minted catalog + /// value rather than the Stage-A sentinel, so it is learned back /// from the catalog (`liveLifeOrFail`) rather than pasted as a literal -- the shape assertion below /// is unaffected, since it names every OTHER segment literally and renders this one dynamically. const NamespaceLifeId life = liveLifeOrFail(op, store->layout(), ns); @@ -1398,7 +1419,7 @@ TEST(CASRefCheckpoint, AppendRequestCountUnchangedByExtraction) const String ckpt_key = store->layout().refCkptKey(life); EXPECT_EQ(backend->writes(log_key), 1u) << "exactly one write-once request per committed chunk"; - /// ONE GET, not zero, since Stage B (Task 4-C): `resolveNamespaceLife`'s `completeCreation` call + /// ONE GET, not zero, since Stage B: `resolveNamespaceLife`'s `completeCreation` call /// publishes this life's `_ckpt.life_epoch` BEFORE the birth chunk is prepared, so this table's /// OWN recovery walk (also inside this `appendRefOps`, ahead of the commit) grounds itself at the /// genesis position `_ckpt` now names and confirms it absent by exact key -- which is `log_key` diff --git a/src/Disks/tests/gtest_cas_ref_install_safety.cpp b/src/Disks/tests/gtest_cas_ref_install_safety.cpp index b6c747a8ee37..d3da467cb415 100644 --- a/src/Disks/tests/gtest_cas_ref_install_safety.cpp +++ b/src/Disks/tests/gtest_cas_ref_install_safety.cpp @@ -89,57 +89,8 @@ PoolPtr openPoolWedgeBudget(const BackendPtr & backend) return Pool::open(backend, cfg); } -/// The engine reissues an unresolved write until its OWN retry window closes, and that window is -/// measured on a clock the engine reads. Both seams here share one counter -- the sleep the engine -/// performs is what advances the clock -- so a fault that stays armed ends the call at its deadline -/// with no real time passing. Installed on the whole pool, because the ref-lane write, its settling -/// read and the recovery retry loop all pace through the same seam. The pool owns the closures and the -/// closures own the clock, so it outlives everything that can still read it. -class VirtualRetryClock -{ -public: - static std::shared_ptr installOn(const PoolPtr & store) - { - auto clock = std::make_shared(); - store->setCasRequestNowFnForTest([clock] { return clock->nowMs(); }); - store->setCasRetrySleepForTest([clock](uint64_t ms) { clock->advance(ms); }); - return clock; - } - - uint64_t nowMs() const - { - std::lock_guard lock(mutex); - return now_ms; - } - size_t pauseCount() const - { - std::lock_guard lock(mutex); - return pauses; - } - uint64_t longestPause() const - { - std::lock_guard lock(mutex); - return longest_pause; - } - - void advance(uint64_t ms) - { - std::lock_guard lock(mutex); - /// Plus one millisecond, because full jitter can draw a ZERO pause: a clock that does not move - /// would leave the loop reissuing for ever against a fault that never clears. - now_ms += ms + 1; - ++pauses; - longest_pause = std::max(longest_pause, ms); - } - -private: - mutable std::mutex mutex; - uint64_t now_ms = 0; - size_t pauses = 0; - uint64_t longest_pause = 0; -}; - using DB::Cas::tests::LatchedChunkFaultBackend; +using DB::Cas::tests::VirtualRetryClock; /// The mount-fence deadlines the pre-attempt tests drive, in the FROZEN boot clock of /// `openPoolFenceControlled` (which is pinned at 0, so these are also the remaining lease budgets). diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index 1bfdf7c5ac52..50c608d66469 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -156,12 +156,21 @@ class HidingListBackend : public CountingBackend std::set hidden_keys; std::set phantom_list_keys; - /// Every CREATING write of a key containing this substring throws a PLAIN (non-`DB::Exception`) - /// error, which is ambiguous by construction -- never a proven refusal. Persistent rather than - /// one-shot on purpose: the subject is what recovery does when the store KEEPS refusing to say - /// whether the write landed. + /// Every CREATING write of a key containing this substring throws `Poco::TimeoutException`, the + /// class the request engine classifies as an unresolved transport fault and reissues under + /// `Retry::standard()` -- a plain `std::exception` is instead the engine's signal for "this could + /// not have landed" and propagates on the FIRST attempt (`CasRequests.cpp`'s + /// `!dynamic_cast(&e)` arms), which is a proven-not-landed verdict, not the + /// ambiguity this fixture means to model. Persistent rather than one-shot on purpose: the subject is + /// what recovery does when the store KEEPS refusing to say whether the write landed. String ambiguous_put_substr; + /// Every attempt the `ambiguous_put_substr` fault intercepted, counted here because the throw below + /// happens before delegating to `CountingBackend::write` -- its own per-key counters never see a + /// faulted attempt at all. A test proves the engine actually reissued (rather than giving up after + /// one attempt) by reading this after the call. + std::atomic ambiguous_put_attempts{0}; + /// Persistent thrown response for a matching CONDITIONAL replace of the mutable checkpoint. The /// ref-log create has already completed when tests arm this, producing the exact one-successor /// recovery window. @@ -204,7 +213,10 @@ class HidingListBackend : public CountingBackend if (!expected_value) { if (!ambiguous_put_substr.empty() && key.find(ambiguous_put_substr) != String::npos) - throw std::runtime_error("injected ambiguous create"); + { + ambiguous_put_attempts.fetch_add(1); + throw Poco::TimeoutException("injected ambiguous create"); + } return CountingBackend::write(key, bytes, expected_value, access); } if (before_cas_put) @@ -371,9 +383,13 @@ PoolConfig walkTestConfig() return config; } -PoolPtr openWalkPool(const BackendPtr & backend, PoolConfig config = walkTestConfig()) +template +PoolPtr openWalkPool(const std::shared_ptr & backend, PoolConfig config = walkTestConfig()) { DB::Cas::tests::seedPoolMetaForRestart(*backend, config.pool_prefix); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the fence math the recovery walk drives matches what admits. + backend->setAttemptTimeoutMs(config.cas_request_budget.attempt_timeout_ms); return Pool::open(backend, std::move(config)); } @@ -1940,9 +1956,15 @@ TEST(CASRefRecoveryCasWalk, UnresolvedSealSlotFailsClosedWithoutInstalling) store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); backend->ambiguous_put_substr = "/_log/"; + const uint64_t fake_now_before = fake_now; EXPECT_ANY_THROW(store->listRefs(ns)); EXPECT_FALSE(store->refTableRecoveredForTest(ns)) << "a table whose dead epoch may or may not be closed must never be exposed as recovered"; + /// The engine reissued -- more than one physical attempt -- and paid a real retry pause on the + /// injected clock before giving up; a fault settled by a single, unretried attempt would not + /// exercise the transient-retry path this test's own name and docstring claim to drive. + EXPECT_GT(backend->ambiguous_put_attempts.load(), 1u); + EXPECT_GT(fake_now, fake_now_before); } /// --------------------------------------------------------------------------------------------- @@ -2125,34 +2147,9 @@ TEST(CASRefRecoveryCasWalk, RecoveryStartsAtRecreatedLifeGenesisAndLeavesPredece EXPECT_EQ(seal2->prev_epoch_seal, std::nullopt) << "sequence 2 carries no chain link"; } -/// `PutHookBackend::casPut` must route through its immediate parent `HidingListBackend::casPut`, not -/// past it to `CountingBackend`, so that a test arming BOTH layers on one `PutHookBackend` instance -/// gets both behaviors composed rather than one silently disabled by the other. -TEST(CASRefRecoveryCasWalk, PutHookBackendComposesHidingListBackendCasPutFaultInjection) -{ - auto backend = std::make_shared(); - - /// `HidingListBackend::write` only runs `before_cas_put` on the CONDITIONAL branch (an `expected` - /// token present) -- a bare create-shaped `casPut(..., std::nullopt)` takes the other branch and - /// can never reach it. Seed the key first so the probed call below is a genuine replace. - OperationForTest op(*backend); - const WriteResult seeded = (*op).create("p/probe", "seed", Retry::once()); - ASSERT_TRUE(std::holds_alternative(seeded)); - - bool before_cas_put_fired = false; - backend->before_cas_put = [&](const String &, const String &, const std::optional &) - { - before_cas_put_fired = true; - }; - - backend->watched_substr = "probe"; - bool on_key_fired = false; - backend->on_key = [&] { on_key_fired = true; }; - - ASSERT_TRUE(std::holds_alternative( - (*op).replace("p/probe", "x", std::get(seeded).etag, Retry::once()))); - - EXPECT_TRUE(before_cas_put_fired) - << "HidingListBackend's before_cas_put hook must still fire for a PutHookBackend instance"; - EXPECT_TRUE(on_key_fired) << "PutHookBackend's own on_key hook must still fire on top of it"; -} +/// `PutHookBackendComposesHidingListBackendCasPutFaultInjection` was retired: it pinned that +/// `PutHookBackend::casPut` reaches its immediate parent `HidingListBackend::casPut` rather than +/// bypassing it to `CountingBackend` -- a fact about this file's own fixture class hierarchy (ordinary +/// C++ virtual dispatch), not a claim any production change could falsify. `PutHookBackend` and +/// `HidingListBackend` are still exercised together, on real recovery-walk scenarios, elsewhere in this +/// file (search for `PutHookBackend>`). diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp index 1aa5ab55cb6f..918fa7391780 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp @@ -374,6 +374,9 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); config.boot_ms_fn = [&fake_now] { return fake_now; }; config.cas_request_budget = budget; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); auto store = openPool(backend, config); auto clock = VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/order_backoff"}; @@ -499,6 +502,9 @@ TEST(CASRefSnapshotPublishOrdering, NotReadyRefusalBacksOffAndResetsAfterDurable config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); config.boot_ms_fn = [&fake_now] { return fake_now; }; config.cas_request_budget = budget; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); auto store = openPool(backend, config); const RootNamespace ns{"srv1/order_not_ready_backoff"}; diff --git a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp index 48645110e50a..9fcd010643a9 100644 --- a/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp +++ b/src/Disks/tests/gtest_cas_ref_wedge_every_attempt.cpp @@ -77,9 +77,13 @@ using DB::Cas::tests::expectThrowsCode; namespace { -PoolPtr openPool(const BackendPtr & backend, CasRequestBudget budget = {}) +template +PoolPtr openPool(const std::shared_ptr & backend, CasRequestBudget budget = {}) { DB::Cas::tests::seedPoolMetaForRestart(*backend); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); } diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index f0fc0fafb2be..f2a9137fb7e4 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -160,24 +160,32 @@ class SynchronizedEventLog std::vector events; }; -PoolPtr openPool(const BackendPtr & backend, CasRequestBudget budget = {}) +template +PoolPtr openPool(const std::shared_ptr & backend, CasRequestBudget budget = {}) { /// Recovery tests seed ref-log/snapshot residue before opening; a pool with such residue always has a /// `_pool_meta` in production, so establish it first (Task 7's zero-write bootstrap check refuses to /// mint a fresh identity over residual data — see `seedPoolMetaForRestart`). Idempotent, and a no-op /// for the fresh-open tests that seed nothing (the subsequent open validates the just-created meta). DB::Cas::tests::seedPoolMetaForRestart(*backend); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); return Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget}); } /// Task 11: like `openPool`, but the caller supplies (and owns) the rest of the config -- snapshot /// thresholds, grace age, a fake `boot_ms_fn`, etc. `pool_prefix`/`server_root_id` are pinned so every /// test in this file addresses the same pool shape. -PoolPtr openPoolWithConfig(const BackendPtr & backend, PoolConfig config) +template +PoolPtr openPoolWithConfig(const std::shared_ptr & backend, PoolConfig config) { config.pool_prefix = "p"; config.server_root_id = "test"; DB::Cas::tests::seedPoolMetaForRestart(*backend); /// see `openPool` above + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(config.cas_request_budget.attempt_timeout_ms); return Pool::open(backend, std::move(config)); } @@ -3089,8 +3097,9 @@ TEST(CASRefWriterSnapshotPublish, C4LatchBoundedUnderSustainedNonCommittedPublis auto store = openPoolWithConfig(backend, config); /// The boot clock above is frozen (it is what keeps the publish backoff armed), so the REQUEST /// engine needs its own advancing clock or a saturated publish never reaches its retry window and - /// reissues for ever. - VirtualRetryClock::installOn(store); + /// reissues for ever. Retained (not discarded) so the assertion below can tell that retry window + /// from a `once` policy that would give up on the very first attempt. + auto clock = VirtualRetryClock::installOn(store); /// Every `_snap` create is unresolved (backend saturated) and stays that way for the whole call, so /// the publish gives up at its own window -- which is one dispatch, which is what this test counts. @@ -3099,6 +3108,11 @@ TEST(CASRefWriterSnapshotPublish, C4LatchBoundedUnderSustainedNonCommittedPublis publishEmptyPart(store, ns, "a"); /// crosses the threshold -> one dispatch -> fails -> backoff armed store->waitForSnapshotPublishSettleForTest(ns); + EXPECT_GT(clock->pauseCount(), 1u) + << "the one dispatch above must itself have reissued more than once against the saturated " + "backend before giving up at its retry window -- a single attempt would not distinguish " + "this from a non-retrying policy"; + EXPECT_GT(clock->longestPause(), 0u) << "at least one of those reissues must have paced with a real backoff"; const auto dispatched_before = global_counters[ProfileEvents::CASRefSnapshotPublishDispatched].load(); for (int i = 0; i < 30; ++i) @@ -3222,8 +3236,10 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) config.boot_ms_fn = [&fake_now] { return fake_now; }; config.cas_request_budget = budget; auto store = openPoolWithConfig(backend, config); - /// As above: the frozen boot clock drives the backoff decisions, so the request engine gets its own. - VirtualRetryClock::installOn(store); + /// As above: the frozen boot clock drives the backoff decisions, so the request engine gets its + /// own. Retained (not discarded) so the assertion below can tell the retry window that arms the + /// backoff from a `once` policy that would give up on the very first attempt. + auto clock = VirtualRetryClock::installOn(store); /// Fail the FIRST dispatch's `_snap` create for the whole call, so it gives up at its own retry /// window and arms the backoff; the fault is cleared before the retry below. @@ -3233,6 +3249,10 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) publishEmptyPart(store, ns, "a"); /// dispatch -> publish fails -> backoff armed store->waitForSnapshotPublishSettleForTest(ns); EXPECT_FALSE(listGreatestSnapshotIdForTest(*backend, layout, ns).has_value()); + EXPECT_GT(clock->pauseCount(), 1u) + << "the failing dispatch above must itself have reissued more than once before giving up and " + "arming the backoff -- a single attempt would not distinguish this from a non-retrying policy"; + EXPECT_GT(clock->longestPause(), 0u) << "at least one of those reissues must have paced with a real backoff"; /// A read within the backoff window (frozen clock) must not re-dispatch. const auto d1 = global_counters[ProfileEvents::CASRefSnapshotPublishDispatched].load(); diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index 6e689395a368..90d794d95e17 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -117,6 +117,50 @@ TEST(CASRetry, PoliciesAreShapedAsSpecified) EXPECT_EQ(Retry::within(1'000).bind(now).deadline_ms, now + 1'000); } +/// A frozen policy is ONE absolute deadline: time passing does not buy a later one, freezing again +/// cannot extend it, and the lease bound still wins when it is the smaller of the two -- which is what +/// keeps `GaveUp::Source` able to say which bound refused. +TEST(CASRetry, AFrozenPolicyIsOneDeadlineAndTheLeaseStillWins) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + const uint64_t start = clock.now; + const Retry frozen = op.freeze(Retry::standard()); + ASSERT_TRUE(frozen.policy_deadline_ms.has_value()); + EXPECT_EQ(frozen.bind(start).deadline_ms, start + 90'000); + EXPECT_EQ(frozen.bind(start + 50'000).deadline_ms, start + 90'000); + EXPECT_FALSE(frozen.bind(start + 50'000).lease_bound); + /// The single-attempt view of a frozen policy keeps the deadline rather than starting a window. + EXPECT_EQ(frozen.asSingleAttempt().policy_deadline_ms, frozen.policy_deadline_ms); + EXPECT_TRUE(frozen.asSingleAttempt().single_attempt); + + clock.now += 50'000; + EXPECT_EQ(op.freeze(frozen).policy_deadline_ms, frozen.policy_deadline_ms); + + const Retry::Bound leashed = op.freeze(Retry::untilLeaseSafe(start + 10'000, 2'000)).bind(clock.now); + EXPECT_EQ(leashed.deadline_ms, start + 8'000); + EXPECT_TRUE(leashed.lease_bound); +} + +/// Freezing belongs to a loop. A single verb still gets a full window from where it is called, however +/// long its caller has already been running. +TEST(CASRequests, ALoneReadUnderTheStandardPolicyStillGetsItsFullWindow) +{ + FakeClock clock; + auto throttled = std::make_shared( + std::make_shared(), ThrottlingBackend::Mode::EveryNth, 1, 429); + auto requests = makeRequests(throttled, clock); + auto op = requests.admit(); + + clock.now += 10 * 90'000; + const uint64_t start = clock.now; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_GE(clock.now - start, 85'000u); +} + TEST(CASWriteResult, OrThrowMapsEveryAlternative) { /// The two that are not failures: a commit hands back its incarnation, a decline hands back @@ -229,11 +273,12 @@ TEST(CASBackendPrimitives, EveryBackendInstanceHasItsOwnId) EXPECT_EQ(a->dialect(), Dialect::Emulated); } -/// `EveryLegacyVerbReachesAnOverrideOfThePrimitiveItForwardsTo` pinned the legacy verbs -/// (`putIfAbsent`/`casPut`/`putOverwrite`) forwarding through the primitive `write`. Those verbs are -/// gone -- `CasOperation` is the only caller of `Backend` now -- so the property is a type-level -/// guarantee rather than a runtime check; every fault double in this file that overrides `write` (e.g. -/// `EachWriteKnobIsKeyedAndOneShotOnThePrimitiveWrite` below) is what proves a double sees every write. +/// The legacy verbs (`putIfAbsent`/`casPut`/`putOverwrite`) that used to forward through the primitive +/// `write` are gone -- `CasOperation` is the only caller of `Backend` now -- so that forwarding is a +/// type-level guarantee rather than a runtime check. What remains to prove is that every fault double +/// in this file that overrides `write` sees an ATTEMPT under either shape `CasOperation` can send: +/// unconditional (`create`) and Etag-conditioned (`replace`). +/// `EachWriteKnobIsKeyedAndOneShotOnThePrimitiveWrite` below covers both. TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotOnThePrimitiveWrite) { @@ -264,6 +309,19 @@ TEST(CASBackendPrimitives, EachWriteKnobIsKeyedAndOneShotOnThePrimitiveWrite) EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); EXPECT_TRUE(std::holds_alternative(op.create("k4", "v", Retry::once()))); + + /// The Etag-conditioned shape: every write above was unconditional (`create`), so none of them + /// could have caught a fault double that only intercepts `write` when it carries an + /// `expected_value` -- the shape `replace` alone sends. + const std::optional k5_first = orThrow(op.create("k5", "v", Retry::once()), "create"); + ASSERT_TRUE(k5_first); + b->refuseNextWrite("k5"); + EXPECT_TRUE(std::holds_alternative(op.replace("k5", "v2", *k5_first, Retry::once()))) + << "consumed here"; + const std::optional k5_second + = orThrow(op.replace("k5", "v2", *k5_first, Retry::once()), "replace"); /// and only once + ASSERT_TRUE(k5_second); + expectBytes(b, "k5", "v2"); } TEST(CASBackendPrimitives, ReadRefusesAValueThatIsNotAnIncarnation) @@ -1023,12 +1081,16 @@ TEST(CASRequests, OnPresenceReportsMetaEvenWhenItHadToFetchTheBody) /// A competitor takes the key while our own create is in flight, and that create's own fate is /// lost. The ambiguity is armed from inside the hook so the competitor's write cannot consume it. bool staged = false; + std::optional rival_etag; backend->onBeforeWrite("k", [&] { if (staged) return; staged = true; - (void)rival.create("k", "theirs", Retry::once()); + const WriteResult rival_result = rival.create("k", "theirs", Retry::once()); + const auto * rival_committed = std::get_if(&rival_result); + ASSERT_NE(rival_committed, nullptr); + rival_etag = rival_committed->etag; backend->injectAmbiguousWrite("k"); }); @@ -1039,9 +1101,15 @@ TEST(CASRequests, OnPresenceReportsMetaEvenWhenItHadToFetchTheBody) const auto * conflict = std::get_if(&result); ASSERT_NE(conflict, nullptr); /// The ambiguity forced a body read, and the body stops at this boundary: a caller of the - /// presence loop can never come to depend on bytes the loop does not promise. - EXPECT_TRUE(std::holds_alternative(conflict->seen)); - EXPECT_FALSE(std::holds_alternative(conflict->seen)); + /// presence loop can never come to depend on bytes the loop does not promise. `get_if` plus + /// its field checks, not a bare `holds_alternative`: a variant that already proved it holds `Meta` + /// cannot also hold `Object`, so the field checks are what a regression could actually fail -- + /// proving the observed Meta is the RIVAL's own committed incarnation, not some other object. + ASSERT_TRUE(rival_etag.has_value()); + const auto * meta_seen = std::get_if(&conflict->seen); + ASSERT_NE(meta_seen, nullptr); + EXPECT_EQ(meta_seen->etag, *rival_etag); + EXPECT_EQ(meta_seen->size, String("theirs").size()); EXPECT_EQ(backend->getTotal(), 1u); } @@ -1406,6 +1474,24 @@ std::exception_ptr s3Error(Aws::S3::S3Errors code, const String & name) return std::make_exception_ptr(DB::S3Exception("the store answered " + name, code, name)); } +/// Answers EVERY read with the same store error. A classification that terminates on an error is then +/// visible as a single attempt, and one that keeps the error ambiguous as a policy spent to its +/// deadline -- which a one-shot arming could never tell apart. +class AlwaysFailingReadBackend final : public CountingBackend +{ +public: + explicit AlwaysFailingReadBackend(std::exception_ptr error_) : error(std::move(error_)) {} + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + (void)CountingBackend::read(key, access); + std::rethrow_exception(error); + } + +private: + std::exception_ptr error; +}; + } TEST(CASRequests, DeadlineIsTheOnlyBoundUnderZeroLatencyThrottling) @@ -1650,6 +1736,39 @@ TEST(CASRequests, ReadModifyWriteWhoseResolveAndFreshObservationBothFailGivesUpU EXPECT_EQ(clock.sleeps.size(), 1u); } +/// A missing bucket is an ANSWER the store gave, but not an answer about the object: an S3-compatible +/// store that transiently misroutes a bucket says exactly this, and a read that ended on it would turn +/// an availability blip into a hard failure. It stays in the ambiguous class -- reissued until the +/// policy's deadline -- like a throttle or a 5xx. +TEST(CASRequests, AMissingBucketOnAReadIsReissuedToTheDeadline) +{ + FakeClock clock; + auto backend = std::make_shared( + s3Error(Aws::S3::S3Errors::NO_SUCH_BUCKET, "NoSuchBucket")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + const uint64_t start = clock.now; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_GT(backend->getTotal(), 1u) << "the read ended on its first attempt instead of reissuing"; + EXPECT_GE(clock.now - start, 85'000u) << "the policy's own deadline is what must end this read"; +} + +/// The kept half of the same classification: a key miss IS an answer about the object, so reissuing it +/// only replays the same authoritative absence until the deadline. One attempt, no pause. +TEST(CASRequests, AnAuthoritativeKeyMissOnAReadEndsTheCallAtOnce) +{ + FakeClock clock; + auto backend = std::make_shared( + s3Error(Aws::S3::S3Errors::NO_SUCH_KEY, "NoSuchKey")); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + + expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { (void)op.read("k", Retry::standard()); }); + EXPECT_EQ(backend->getTotal(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); +} + TEST(CASRequests, AnUnmodeledStoreErrorOnAReadIsReissuedNotSurfaced) { FakeClock clock; diff --git a/src/Disks/tests/gtest_cas_retirement_sweep.cpp b/src/Disks/tests/gtest_cas_retirement_sweep.cpp index 36d9d6895bca..0a52f0055750 100644 --- a/src/Disks/tests/gtest_cas_retirement_sweep.cpp +++ b/src/Disks/tests/gtest_cas_retirement_sweep.cpp @@ -141,13 +141,19 @@ class UnresolvedPutBackend final : public InMemoryBackend { public: String fault_key_substr; + /// How many matching writes actually hit the fault, so a caller can prove the engine reissued + /// rather than infer it from a give-up that a non-retrying policy would also reach. + int fault_hits = 0; std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { if (!fault_key_substr.empty() && key.find(fault_key_substr) != String::npos) + { + ++fault_hits; throw Poco::TimeoutException("UnresolvedPutBackend: simulated ambiguous result (response lost)"); + } return InMemoryBackend::write(key, bytes, expected_value, access); } }; @@ -343,6 +349,9 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS budget.lease_safety_margin_ms = 100; auto backend = std::make_shared(); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); uint64_t fake_boot = 1'000'000; std::vector waits; auto store = Pool::open(backend, PoolConfig{ @@ -371,13 +380,25 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS /// Drive the next ref-log append into the Unresolved/wedge outcome: every attempt it makes fails /// ambiguously, so this process can never learn whether its conditional PUT landed. That /// undecidability is the whole reason the resolution is a conditional CREATE and not a GET. The - /// give-up is the append's own retry window, and the engine's inter-attempt sleeps pay it on the - /// same injected boot clock the fence and the deadline are measured against, so it costs no real - /// time and no lease. - store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms + 1; }); + /// give-up is the append's own retry window -- paced on ITS OWN virtual clock, separate from + /// `fake_boot` (the mount fence's), so the standard policy's full window is available to reissue + /// against rather than being cut short by the 30s lease `fake_boot` also measures. + uint64_t fake_retry = 0; + std::vector retry_sleeps; + store->setCasRequestNowFnForTest([&fake_retry] { return fake_retry; }); + store->setCasRetrySleepForTest([&fake_retry, &retry_sleeps](uint64_t ms) + { + fake_retry += ms + 1; + retry_sleeps.push_back(ms); + }); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); + EXPECT_GT(backend->fault_hits, 1) + << "the append must have reissued more than once against the persistent fault before giving up " + "-- a single attempt would not distinguish this from a non-retrying policy"; + EXPECT_GT(retry_sleeps.size(), 1u) << "more than one paced retry must have occurred before the give-up"; + EXPECT_GT(fake_retry, 0u) << "the retry clock must have advanced past the policy's own deadline"; /// The id the straggler would occupy: one past the greatest record that is actually durable in the /// dying epoch. That is also, by construction, where the recovery seal goes. diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 0370bea9afb2..a588fda71b36 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -274,7 +274,7 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend DB::Cas::tests::OperationForTest mint_op(*this); const auto meta = (*mint_op).head(request.destination_key, DB::Cas::Retry::once()); if (meta) - queued_delete_incarnation = meta->etag; + queued_delete_etag = meta->etag; } if (script != FaultScript::CopyLandsThenCondemned) @@ -291,7 +291,7 @@ class EtagFaithfulPublicationBackend final : public DB::Cas::InMemoryBackend size_t copy_publications = 0; size_t streaming_publications = 0; String queued_delete_token; - std::optional queued_delete_incarnation; + std::optional queued_delete_etag; DB::Cas::Backend::RawRemoval first_delete{}; private: @@ -389,10 +389,10 @@ TEST(CASS3Staging, StagedCopyCondemnedRetryRetagsBeforeQueuedDelete) EXPECT_EQ(backend->copy_publications, 1u); EXPECT_EQ(backend->streaming_publications, 1u); - ASSERT_TRUE(backend->queued_delete_incarnation.has_value()); + ASSERT_TRUE(backend->queued_delete_etag.has_value()); { DB::Cas::tests::OperationForTest op(*backend); - EXPECT_EQ((*op).remove(store->layout().blobKey(ref), *backend->queued_delete_incarnation, DB::Cas::Retry::once()), + EXPECT_EQ((*op).remove(store->layout().blobKey(ref), *backend->queued_delete_etag, DB::Cas::Retry::once()), DB::Cas::Removal::Mismatch); } const auto current = readAt(*backend, store->layout().blobKey(ref)); @@ -424,10 +424,10 @@ TEST(CASS3Staging, StagedCopyDeletedBeforeAbsentRetryRetagsBeforeQueuedDelete) EXPECT_EQ(backend->copy_publications, 1u) << "the absent retry must not copy the original staged envelope again"; EXPECT_EQ(backend->streaming_publications, 1u); - ASSERT_TRUE(backend->queued_delete_incarnation.has_value()); + ASSERT_TRUE(backend->queued_delete_etag.has_value()); { DB::Cas::tests::OperationForTest op(*backend); - EXPECT_EQ((*op).remove(store->layout().blobKey(ref), *backend->queued_delete_incarnation, DB::Cas::Retry::once()), + EXPECT_EQ((*op).remove(store->layout().blobKey(ref), *backend->queued_delete_etag, DB::Cas::Retry::once()), DB::Cas::Removal::Mismatch) << "the second queued exact delete for the copied ETag must miss the retagged replacement"; } @@ -966,12 +966,31 @@ TEST(CASStagingSweeper, RemovesOnlyObjectsUnderGivenMountPrefix) /// nested `staging/` under `blobs/` (or vice versa) would violate. TEST(CASS3Staging, GcBlobDiscoveryPrefixExcludesStagingObjects) { - const DB::Cas::Layout layout("p"); - const std::string blobs_prefix = layout.blobsPrefix(); - const std::string staging_prefix = "p/staging/mountA/"; + /// The REAL staging prefix, from the accessor every writer actually mints staging keys through + /// (`ContentAddressedMetadataStorage::stagingKeyPrefix`) -- not a hand-copied literal that a + /// staging-side rename would leave silently stale. + auto object_storage = makeFakeNativeCopyStorage(/*native_only_copy_supported=*/true); + auto metadata_storage = makeS3StagingMetadataStorageForTest(object_storage, "mountA"); + metadata_storage->startup(); + const std::string physical_root = object_storage->getCommonKeyPrefix(); + const std::string full_staging_prefix = metadata_storage->stagingKeyPrefix(); + ASSERT_TRUE(full_staging_prefix.starts_with(physical_root)) + << full_staging_prefix << " vs root " << physical_root; + /// Strip the physical object-storage root (and the '/' `physicalKey` joins it to the pool key + /// with): `Layout` (below) is root-agnostic, and comparing a physically-rooted key against a bare + /// `Layout` key would pass for the wrong reason (both simply fail to share the unrelated root, not + /// because the pool-relative prefixes are disjoint). + std::string staging_prefix = full_staging_prefix.substr(physical_root.size()); + if (!staging_prefix.empty() && staging_prefix.front() == '/') + staging_prefix.erase(0, 1); + staging_prefix += "/"; const std::string staging_key = staging_prefix + "aaa.tmp"; - EXPECT_EQ(blobs_prefix, "p/blobs/"); + const DB::Cas::Layout layout(metadata_storage->poolForTest()->poolConfig().pool_prefix); + const std::string blobs_prefix = layout.blobsPrefix(); + + EXPECT_EQ(staging_prefix, "pool/staging/mountA/") << "sanity: the accessor's own shape"; + EXPECT_EQ(blobs_prefix, "pool/blobs/"); EXPECT_FALSE(staging_prefix.starts_with(blobs_prefix)); EXPECT_FALSE(blobs_prefix.starts_with(staging_prefix)); EXPECT_FALSE(staging_key.starts_with(blobs_prefix)); diff --git a/src/Disks/tests/gtest_cas_sentinel_probe.cpp b/src/Disks/tests/gtest_cas_sentinel_probe.cpp index 201087cdb1c1..962bd85a30bc 100644 --- a/src/Disks/tests/gtest_cas_sentinel_probe.cpp +++ b/src/Disks/tests/gtest_cas_sentinel_probe.cpp @@ -177,6 +177,9 @@ TEST(CASSentinelProbe, TransportErrorNeverClassifiesAsAbsent) const auto result = probeSentinel(op, "k", Retry::standard()); EXPECT_EQ(result.outcome, ProbeOutcome::Indeterminate); EXPECT_FALSE(result.body.has_value()); + EXPECT_GT(clock.sleeps.size(), 1u) + << "a single attempt would not distinguish this reissue loop from a non-retrying policy that " + "reaches the same Indeterminate give-up on its first try"; } #if USE_AWS_S3 @@ -299,6 +302,9 @@ TEST(CASSentinelProbe, NativeClassifiesUnmodeledErrorAsIndeterminate) auto requests = makeRequests(backend, &clock); auto op = requests.admit(); EXPECT_EQ(probeSentinel(op, nativeKeyUnder(storage, "some/key"), Retry::standard()).outcome, ProbeOutcome::Indeterminate); + EXPECT_GT(clock.sleeps.size(), 1u) + << "a single attempt would not distinguish this reissue loop from a non-retrying policy that " + "reaches the same Indeterminate give-up on its first try"; } /// Production wiring (`Pool::open`) ALWAYS wraps the real backend in `InstrumentedBackend` before diff --git a/src/Disks/tests/gtest_cas_throttling_gate.cpp b/src/Disks/tests/gtest_cas_throttling_gate.cpp new file mode 100644 index 000000000000..d208eede19f9 --- /dev/null +++ b/src/Disks/tests/gtest_cas_throttling_gate.cpp @@ -0,0 +1,129 @@ +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "config.h" + +#if USE_AWS_S3 + +namespace ProfileEvents +{ +extern const Event CASRequestResolveRead; +} + +using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; + +namespace +{ + +/// The PartWriteTxn fixture every pool test uses: stage an empty manifest, precommit it under `ref`, +/// promote it. Empty content is enough -- this gate exercises the request contract under throttling, +/// not the blob path. +void publishEmptyPart(const PoolPtr & store, const RootNamespace & ns, const String & ref) +{ + PartWriteInfo info; + info.intended_namespace = ns; + info.intended_ref = ns.string() + "/" + ref; + auto build = store->beginPartWrite(info); + const ManifestId id = build->stageManifest({}); + build->precommitAdd(ns, ref, id); + build->promote(ns, ref, build->buildId(), id); +} + +/// Every key the total per-key request count a `CountingBackend` observed, summed across every +/// primitive: whichever verb the throttled key was refused on, this is what "requested again later" +/// means. +uint64_t totalRequestsFor(const CountingBackend & inner, const String & key) +{ + return inner.getCount(key) + inner.headCount(key) + inner.listCount(key) + inner.writeCount(key) + + inner.deleteCount(key) + inner.publishCount(key); +} + +} + +/// The `ThrottlingBackend` gate: every user-visible statement -- table creation, an +/// insert, a rename, a drop, the writable mount `Pool::open` itself performs, and one GC round -- must +/// still SUCCEED when every key it touches is refused exactly once (`FirstPerKey`, HTTP 429) before it +/// is honored. `refusals(key) == 1` for every key the gate recorded, and every one of them was requested +/// at least twice: once refused, at least once more to actually land. +/// +/// EXCLUDED, by design, and never reached by these scenarios: the in-band recovery walk's epoch seal at +/// `{E, T+1}` -- nothing here trips a fence, forces a remount, or drives recovery. +TEST(CASThrottlingGate, EveryUserVisibleStatementSucceedsUnderFirstPerKeyThrottling) +{ + auto inner = std::make_shared(); + auto throttled = std::make_shared( + inner, ThrottlingBackend::Mode::FirstPerKey, /*n=*/0, /*status=*/429); + + /// The writable mount at open, probe included: `Pool::open` itself issues the identity probe, the + /// mount claim and the epoch allocation under this same throttled backend. + auto store = DB::Cas::tests::openPoolForTest(throttled); + + const auto resolve_reads_before = ProfileEvents::global_counters[ProfileEvents::CASRequestResolveRead].load(); + + const RootNamespace ns{"test/throttle_gate"}; + + /// CREATE-shaped: the namespace's first part write births it. + ASSERT_NO_THROW(publishEmptyPart(store, ns, "created")); + EXPECT_TRUE(store->resolveRef(ns, "created").has_value()); + + /// INSERT-shaped: a second part write into the now-live namespace. + ASSERT_NO_THROW(publishEmptyPart(store, ns, "inserted")); + EXPECT_TRUE(store->resolveRef(ns, "inserted").has_value()); + + /// RENAME-shaped: content addressing has no rename primitive (`PartFolderAccess::republishRef`'s own + /// comment) -- a rename publishes equivalent content at the destination ref and drops the source. + ASSERT_NO_THROW(publishEmptyPart(store, ns, "renamed")); + ASSERT_NO_THROW(store->dropRef(ns, "inserted")); + EXPECT_TRUE(store->resolveRef(ns, "renamed").has_value()); + EXPECT_FALSE(store->resolveRef(ns, "inserted").has_value()); + + /// One GC round, still under throttling. + Gc gc(store, UInt128{7101}); + ASSERT_NO_THROW(DB::Cas::tests::runRegularRoundReclaiming(gc)); + + /// DROP-shaped: the whole namespace goes last, so the statements above still have something to act on. + ASSERT_NO_THROW(store->dropNamespace(ns)); + + /// `refusals(key) == 1` for every key is a class invariant of `FirstPerKey` mode itself + /// (`refuseOrPass` refuses a key at most once, ever, by construction of `refused_keys.insert`), so + /// asserting it here would prove nothing about THIS run's engine behavior -- it cannot fail. What + /// can fail, and is the actual content of the gate: every key the gate decided was requested again + /// afterwards (an inner post-refusal count of zero means the caller never retried at all, which the + /// statements above already ruled out by succeeding), and at least one of them took more than one + /// post-refusal request. + /// + /// That count alone does NOT prove a resolve read happened: a key that is read and then written + /// reaches two requests through two different verbs. The counter delta below is what pins the + /// engine's own ambiguity resolution -- it is incremented at exactly one site, the write loop's + /// settle-by-reading step. It does not attribute the reads to any particular key, and it counts a + /// refused precondition the same as a throttled ambiguity. + size_t keys_needing_more_than_one_request = 0; + for (const String & key : throttled->decidedKeys()) + { + const uint64_t total = totalRequestsFor(*inner, key); + EXPECT_GE(total, 1u) + << "key: " << key << " -- one refusal plus at least one later success is 'requested at least twice'"; + if (total >= 2) + ++keys_needing_more_than_one_request; + } + EXPECT_FALSE(throttled->decidedKeys().empty()) << "the gate must have actually decided some keys"; + EXPECT_GT(keys_needing_more_than_one_request, 0u) + << "no throttled key needed more than one post-refusal request -- every refusal landed on a " + "trivially-retried read/list/head"; + /// Under coverage builds ProfileEvents propagate into a thread-local subtree that does not reach + /// `global_counters`; deltas read 0 there only (see gtest_unique_key_index_cache). +#if !WITH_COVERAGE + EXPECT_GT(ProfileEvents::global_counters[ProfileEvents::CASRequestResolveRead].load() - resolve_reads_before, 0u) + << "no throttled write was settled by a read -- the engine's ambiguity-resolution path never ran"; +#endif +} + +#endif diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index e248a55e3ca3..5e34f05ee0a7 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -36,18 +36,24 @@ PoolConfig singleAttemptConfig() return config; } -PoolPtr openSingleAttemptPool(const BackendPtr & backend) +template +PoolPtr openSingleAttemptPool(const std::shared_ptr & backend) { DB::Cas::tests::seedPoolMetaForRestart(*backend); + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the fence math these single-attempt fixtures drive matches what admits. + backend->setAttemptTimeoutMs(singleAttemptConfig().cas_request_budget.attempt_timeout_ms); return Pool::open(backend, singleAttemptConfig()); } -PoolPtr openFrozenSingleAttemptPool(const BackendPtr & backend) +template +PoolPtr openFrozenSingleAttemptPool(const std::shared_ptr & backend) { DB::Cas::tests::seedPoolMetaForRestart(*backend); PoolConfig config = singleAttemptConfig(); config.boot_ms_fn = [] { return uint64_t{0}; }; config.mount_renew_period = std::chrono::hours{1}; + backend->setAttemptTimeoutMs(config.cas_request_budget.attempt_timeout_ms); return Pool::open(backend, config); } @@ -88,53 +94,30 @@ uint64_t leaveRejectedCleanupDuty(const PoolPtr & store, const RootNamespace & n return rejected_seq; } -/// `ChunkFaultBackend` counts its faults, and a count can no longer wedge one logical write: the write -/// engine settles every ambiguity by an exact read and reissues, so a bounded fault is outlived and the -/// call commits on a later attempt instead of exhausting its own retry window. Latching keeps the fault -/// (and, for `LandedThenLost`, the paired lost resolve-read) armed on every reissue, so the duty tests -/// below can drive a call all the way to a genuine give-up. -class LatchedChunkFaultBackend : public DB::Cas::tests::ChunkFaultBackend -{ -public: - bool latched = false; - - std::optional read(const String & key, DB::Cas::TransportAccess & access) override - { - if (latched && !fail_read_once_key.empty() && key == fail_read_once_key) - throw Poco::TimeoutException("LatchedChunkFaultBackend: the lost read stays lost"); - return ChunkFaultBackend::read(key, access); - } - - std::expected write(const String & key, const String & bytes, - const std::optional & expected_value, - DB::Cas::TransportAccess & access) override - { - if (latched && mode != Mode::None && fault_skip == 0 && !expected_value && !fault_substr.empty() - && key.find(fault_substr) != String::npos) - fault_count = 1; - return ChunkFaultBackend::write(key, bytes, expected_value, access); - } - - /// Disarms completely (not just unlatches): what a caller does right after driving a call to its - /// give-up is a further mutation that must reach the store normally. - void disarm() - { - latched = false; - mode = Mode::None; - fault_count = 0; - fault_skip = 0; - fail_read_once_key.clear(); - } -}; +using DB::Cas::tests::LatchedChunkFaultBackend; /// Latches `backend` and drives `f` to a NETWORK_ERROR give-up, then disarms the fault completely so a /// caller's next mutation reaches the store normally. The caller must have installed a /// `VirtualRetryClock` on the store first, or the give-up paces through a real sleep instead of a /// virtual one. -void driveToNetworkErrorGiveUp(LatchedChunkFaultBackend & backend, const std::function & f) +/// +/// A give-up is not by itself proof that the engine actually retried: a `once` policy reaches the same +/// outcome by propagating its first failure. Asserting the fault double's own hit count and the +/// clock's pause count is what tells the two apart -- both fire more than once only when reissues +/// really happened, whether the fault re-arms on every write attempt (`Unresolved`) or the resolving +/// read keeps retrying against a persistently lost response (`LandedThenLost`). +void driveToNetworkErrorGiveUp(LatchedChunkFaultBackend & backend, DB::Cas::tests::VirtualRetryClock & clock, + const std::function & f) { backend.latched = true; + const int fault_hits_before = backend.fault_hits; + const size_t pauses_before = clock.pauseCount(); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, f); + EXPECT_GE(backend.fault_hits - fault_hits_before, 1) << "the fault double must actually have fired"; + EXPECT_GT(clock.pauseCount() - pauses_before, 1u) + << "a give-up after a single attempt cannot distinguish a retrying `standard` policy from one " + << "that never reissues at all"; + EXPECT_GT(clock.longestPause(), 0u) << "at least one of the retry's pauses must be a real, nonzero backoff"; backend.disarm(); } @@ -148,7 +131,7 @@ TEST(CASWriterDuties, UncertainAdoptedGrantStaysActiveUntilTheNextMutationRemove { auto backend = std::make_shared(); auto store = openSingleAttemptPool(backend); - DB::Cas::tests::VirtualRetryClock::installOn(store); + auto clock = DB::Cas::tests::VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/writer_duty_adopt"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); @@ -160,7 +143,7 @@ TEST(CASWriterDuties, UncertainAdoptedGrantStaysActiveUntilTheNextMutationRemove backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::LandedThenLost; backend->fault_count = 1; - driveToNetworkErrorGiveUp(*backend, [&] { abandoned->precommitAdd(ns, "abandoned", abandoned_id); }); + driveToNetworkErrorGiveUp(*backend, *clock, [&] { abandoned->precommitAdd(ns, "abandoned", abandoned_id); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(abandoned->precommitState(), PartWriteTxn::PrecommitState::Uncertain); @@ -200,6 +183,9 @@ TEST(CASWriterDuties, ProvenAbsentGrantDrainsAsNoOpBeforeTheNextMutation) PoolConfig config = singleAttemptConfig(); config.boot_ms_fn = [] { return uint64_t{0}; }; config.mount_renew_period = std::chrono::hours{1}; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the fence math this single-attempt fixture drives matches what admits. + backend->setAttemptTimeoutMs(config.cas_request_budget.attempt_timeout_ms); auto store = Pool::open(backend, config); const RootNamespace ns{"srv1/writer_duty_reject"}; @@ -244,7 +230,7 @@ TEST(CASWriterDuties, WedgeResolvedAsRejectDrainsTheDutyAsNoOp) { auto backend = std::make_shared(); auto store = openSingleAttemptPool(backend); - DB::Cas::tests::VirtualRetryClock::installOn(store); + auto clock = DB::Cas::tests::VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/writer_duty_wedge_reject"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); @@ -255,7 +241,7 @@ TEST(CASWriterDuties, WedgeResolvedAsRejectDrainsTheDutyAsNoOp) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - driveToNetworkErrorGiveUp(*backend, [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); + driveToNetworkErrorGiveUp(*backend, *clock, [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); ASSERT_TRUE(store->refLaneWedgedForTest(ns)); ASSERT_EQ(rejected->precommitState(), PartWriteTxn::PrecommitState::Uncertain); @@ -377,6 +363,9 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, }; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); const RootNamespace ns{"srv1/writer_duty_crash"}; auto predecessor = Pool::open(backend, PoolConfig{ @@ -452,6 +441,9 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, }; + /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget + /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. + backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); /// Rooted under the POOL's OWN `server_root_id` ("test", unlike this file's other fixtures, which /// stay under "srv1" precisely because they never drive the orphan sweep): `prefixEligible`'s /// watermark floor is looked up by walking the NAMESPACE's own prefix segments for a live mount @@ -471,7 +463,7 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = budget, }); - DB::Cas::tests::VirtualRetryClock::installOn(predecessor); + auto clock = DB::Cas::tests::VirtualRetryClock::installOn(predecessor); /// A real, fully-promoted ref through the ordinary production write path (no seeded catalog/ckpt) /// gives the namespace genuine epoch-1 content, so the successor's recovery below has something @@ -496,7 +488,7 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) backend->fault_substr = predecessor->layout().namespaceStreamPrefix(predecessor->namespaceLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - driveToNetworkErrorGiveUp(*backend, [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); + driveToNetworkErrorGiveUp(*backend, *clock, [&] { rejected->precommitAdd(ns, "rejected", rejected_id); }); ASSERT_TRUE(predecessor->refLaneWedgedForTest(ns)); ASSERT_EQ(rejected->precommitState(), PartWriteTxn::PrecommitState::Uncertain); const uint64_t predecessor_epoch = predecessor->writerEpoch(); @@ -559,7 +551,7 @@ TEST(CASWriterDuties, DutySurvivesSettlementFailureForRetry) { auto backend = std::make_shared(); auto store = openFrozenSingleAttemptPool(backend); - DB::Cas::tests::VirtualRetryClock::installOn(store); + auto clock = DB::Cas::tests::VirtualRetryClock::installOn(store); const RootNamespace ns{"srv1/writer_duty_settlement_retry"}; DB::Cas::tests::casAdmitRecoverableEntry(*backend, store->layout(), ns, store->liveWriterEpoch()); publishEmptyRef(store, ns, "target"); @@ -577,7 +569,7 @@ TEST(CASWriterDuties, DutySurvivesSettlementFailureForRetry) backend->fault_substr = store->layout().namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; backend->mode = DB::Cas::tests::ChunkFaultBackend::Mode::Unresolved; backend->fault_count = 1; - driveToNetworkErrorGiveUp(*backend, [&] { store->dropRef(ns, "target"); }); + driveToNetworkErrorGiveUp(*backend, *clock, [&] { store->dropRef(ns, "target"); }); EXPECT_TRUE(store->writerCleanupDutiesPendingForTest()) << "a settlement that throws must retain the duty for retry, never lose it"; diff --git a/src/Storages/MergeTree/DataPartsExchange.cpp b/src/Storages/MergeTree/DataPartsExchange.cpp index 7a8ae2e54c42..b2f5e1d12b3e 100644 --- a/src/Storages/MergeTree/DataPartsExchange.cpp +++ b/src/Storages/MergeTree/DataPartsExchange.cpp @@ -879,10 +879,11 @@ std::pair Fetcher::fetchSelected readBinary(projections, *in); /// CAS replication 2b — fetch-by-relink (spec §4; B7 part_manifest_v2, all-tree task 7). The sender - /// chose to relink: it sent only the part's encoded PartManifest body, no file bytes. Build the part - /// by staging this server's OWN local manifest over the blobs already in the shared pool (adopt-by-hash - /// -> revalidate -> promote inside adoptPartFromManifest) — self-contained since task 6 routed - /// uuid.txt/metadata_version.txt through the content path, so there is no separate mutable header to + /// chose to relink: it sent only the part's encoded PartManifest body, no file bytes, and the + /// reservation above already went to the offered pool's disk. Build the part by staging this + /// server's OWN local manifest over the blobs already in the shared pool (adopt-by-hash -> revalidate + /// -> promote inside adoptPartFromManifest) — self-contained because uuid.txt and + /// metadata_version.txt travel through the content path, so there is no separate mutable header to /// reconstruct. If the relink is not possible (blob missing/condemned — a transient or a /// genuinely-different pool the cheap pre-filter let through, or a mixed-build pair offering an /// unrecognized cookie value), fall back to a normal byte fetch by re-requesting WITHOUT relink. diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 349c4068e4e5..7dee09763c0f 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -38,6 +38,12 @@ - ``POST /_control/mode?if_match=reject|ignore&omit_generation=0|1`` — select the adversarial behaviours below. Global, not per bucket: the client reuses connections across buckets and a per-bucket switch would invite a test to believe it had isolated something it had not. + - ``POST /_control/first_per_key_throttle?enabled=0|1`` — while enabled, the FIRST request naming + any given ``(bucket, key)`` -- of any method, ``_control/*`` excluded -- answers ``429 SlowDown`` + and touches nothing; every later request to that same key is served normally. Models a real + store's transient per-object throttling: the caller must resolve the refusal by reissuing, never + by treating it as a definite failure. ``enabled=0`` clears the seen-key set along with the flag, so + a later ``enabled=1`` throttles every key again from scratch. Adversarial behaviours, each off by default: @@ -136,6 +142,11 @@ def __init__(self): # a modelled rate cap — see the module docstring's `/_control/delay` bullet. self.delay_substr = "" self.delay_ms = 0 + # `/_control/first_per_key_throttle`: while enabled, every key in `throttled_keys_seen` has + # already been refused once and is now served normally; a key not yet in the set gets added + # and refused with 429 instead of being dispatched. + self.first_per_key_throttle = False + self.throttled_keys_seen = set() self._next_generation = _GENERATION_SEED self._next_etag_ordinal = 1 self._next_upload_ordinal = 1 @@ -212,6 +223,14 @@ def _bad_request(message): ) +def _slow_down(key): + return Reply( + 429, + _error_xml("SlowDown", "throttled by /_control/first_per_key_throttle: " + key), + {"Content-Type": "application/xml"}, + ) + + def _precondition_failed(message): return Reply( 412, @@ -817,11 +836,21 @@ def handle_control(path, method, query): json.dumps({"substr": STORE.delay_substr, "ms": STORE.delay_ms}).encode(), {"Content-Type": "application/json"}, ) + if path == "/_control/first_per_key_throttle" and method == "POST": + STORE.first_per_key_throttle = query.get("enabled", ["0"])[0] == "1" + STORE.throttled_keys_seen = set() + return Reply( + 200, + json.dumps({"enabled": STORE.first_per_key_throttle}).encode(), + {"Content-Type": "application/json"}, + ) if path == "/_control/reset" and method == "POST": STORE.requests = [] STORE.counters = {} STORE.delay_substr = "" STORE.delay_ms = 0 + STORE.first_per_key_throttle = False + STORE.throttled_keys_seen = set() return Reply(200, b"OK") return Reply(404, _error_xml("NoSuchControl", "unknown control path " + path)) @@ -881,49 +910,71 @@ def _dispatch(self, method, want_body=True): # sleep above actually ran. if delayed: STORE.count("DelayedPut") - request_class = _request_class(bucket, key) - operation = _request_operation(bucket, request_class, method, query, headers) - if method == "PUT": - if "partNumber" in query: - STORE.count("UploadPart") - reply = handle_put(bucket, key, query, headers, body) - elif method == "DELETE": - reply = handle_delete(bucket, key, query, headers) - elif method == "POST": - if "delete" in query: - STORE.count("DeleteObjects") - reply = handle_batch_delete(bucket, body) + throttled = STORE.first_per_key_throttle and (bucket, key) not in STORE.throttled_keys_seen + if throttled: + STORE.throttled_keys_seen.add((bucket, key)) + STORE.count("FirstPerKeyThrottled") + reply = _slow_down(key) + STORE.requests.append( + { + "seq": len(STORE.requests), + "method": method, + "bucket": bucket, + "key": key, + "query": parsed.query, + "headers": headers, + "request_class": _request_class(bucket, key), + "operation": "first_per_key_throttled", + "request_body": "", + "status": reply.status, + "response_generation": None, + "response_etag": None, + } + ) + if not throttled: + request_class = _request_class(bucket, key) + operation = _request_operation(bucket, request_class, method, query, headers) + if method == "PUT": + if "partNumber" in query: + STORE.count("UploadPart") + reply = handle_put(bucket, key, query, headers, body) + elif method == "DELETE": + reply = handle_delete(bucket, key, query, headers) + elif method == "POST": + if "delete" in query: + STORE.count("DeleteObjects") + reply = handle_batch_delete(bucket, body) + else: + if "uploads" in query: + STORE.count("CreateMultipartUpload") + if "uploadId" in query: + STORE.count("CompleteMultipartUpload") + reply = handle_post(bucket, key, query, headers, body) + elif method in ("GET", "HEAD"): + reply = handle_get_or_head(bucket, key, query, headers) else: - if "uploads" in query: - STORE.count("CreateMultipartUpload") - if "uploadId" in query: - STORE.count("CompleteMultipartUpload") - reply = handle_post(bucket, key, query, headers, body) - elif method in ("GET", "HEAD"): - reply = handle_get_or_head(bucket, key, query, headers) - else: - reply = _unsupported(method) - - STORE.requests.append( - { - "seq": len(STORE.requests), - "method": method, - "bucket": bucket, - "key": key, - "query": parsed.query, - "headers": headers, - "request_class": request_class, - "operation": operation, - "request_body": ( - body.decode("utf-8", "replace") - if request_class in ("blob_meta", "cas_control") - else "" - ), - "status": reply.status, - "response_generation": reply.headers.get("x-goog-generation"), - "response_etag": reply.headers.get("ETag"), - } - ) + reply = _unsupported(method) + + STORE.requests.append( + { + "seq": len(STORE.requests), + "method": method, + "bucket": bucket, + "key": key, + "query": parsed.query, + "headers": headers, + "request_class": request_class, + "operation": operation, + "request_body": ( + body.decode("utf-8", "replace") + if request_class in ("blob_meta", "cas_control") + else "" + ), + "status": reply.status, + "response_generation": reply.headers.get("x-goog-generation"), + "response_etag": reply.headers.get("ETag"), + } + ) self._send(reply, want_body) diff --git a/tests/integration/test_cas_gcs/test.py b/tests/integration/test_cas_gcs/test.py index 7df52e4dbbd3..e7571c86afb0 100644 --- a/tests/integration/test_cas_gcs/test.py +++ b/tests/integration/test_cas_gcs/test.py @@ -1302,6 +1302,60 @@ def test_a_reload_that_would_flip_the_token_dialect_is_refused(): ) +@pytest.mark.parametrize("disk", sorted(CAS_DISKS)) +def test_first_per_key_throttling_is_transparently_absorbed(disk): + """Task 21's coverage gate, over the wire: `/_control/first_per_key_throttle` refuses the FIRST + request naming every key with `429 SlowDown`, modelling a real store's transient per-object + throttling. A CAS mount's own request engine must resolve every one of those refusals by + reissuing rather than surfacing them, so `CREATE TABLE` / `INSERT` / `SELECT` / `DROP TABLE` + against a CAS disk must all still succeed with the mode on. + + Scoped to its own table and reset in `finally`: the throttle is a GLOBAL fake-service switch, and + leaving it on would refuse the first touch of every key every later test in this module makes. + """ + node = cluster.instances["node"] + bucket = CAS_DISKS[disk] + table = "t_throttled_" + disk + try: + assert _control_post("/_control/first_per_key_throttle?enabled=1")["enabled"] is True + start_seq = _next_seq() + + node.query("DROP TABLE IF EXISTS {} SYNC".format(table)) + node.query( + """ + CREATE TABLE {} (id Int64, data String) + ENGINE = MergeTree() ORDER BY id + SETTINGS storage_policy = '{}' + """.format( + table, disk + ) + ) + node.query( + "INSERT INTO {} SELECT number, toString(number) FROM numbers({})".format( + table, NUM_ROWS + ) + ) + assert int(node.query("SELECT count() FROM {}".format(table))) == NUM_ROWS + node.query("DROP TABLE {} SYNC".format(table)) + + records = _captured_since(start_seq, bucket) + throttled = [r for r in records if r["operation"] == "first_per_key_throttled"] + assert throttled, "the throttle control never fired -- this run exercises nothing" + assert all(r["status"] == 429 for r in throttled), throttled + # Every throttled key was reached again afterwards: `FirstPerKeyThrottled`'s own contract is + # refuse-once-then-pass, so a key throttled here but never seen again would mean the mount gave + # up on the refusal instead of absorbing it -- which the successful statements above already + # rule out, but this ties the failure (if any) to the exact key. + seen_again = {r["key"] for r in records if r["operation"] != "first_per_key_throttled"} + for record in throttled: + assert record["key"] in seen_again, "key '{}' was throttled once and never retried".format( + record["key"] + ) + finally: + assert _control_post("/_control/first_per_key_throttle?enabled=0")["enabled"] is False + node.query("DROP TABLE IF EXISTS {} SYNC".format(table)) + + # MUST STAY LAST IN THIS FILE. The fake's capture log is global and cumulative and nothing in this # module resets it, so this assertion covers exactly the traffic that precedes it. # Add new tests ABOVE this line. From 8f6cd74a3f52aeb56acec1b96a6400ffe7665bb0 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:05:43 +0200 Subject: [PATCH 18/81] cas: overlap the GC fold's small-object reads on a bounded pool (cas_gc_read_concurrency) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fold's `fold_ref_intake` and `fold_reduce` phases issue their checkpoint, walk-position, manifest-edge and zero-in-degree-HEAD reads one at a time, in the round's own decision order — a live-GCS soak measured `fold_ref_intake` at 2303 s of a 4352 s phase wall (53%), and a separate finding recorded one fold round holding the GC lease for hours on a real bucket, still unfinished after 97 minutes. `GcReadAhead` sits in front of the fold's one admitted `CasOperation`: callers hint keys the sequential walk will need next, workers fetch them on a bounded pool under the same admitted generation, and the walk takes results at exactly the sites and in exactly the order it reads today — no decision, decode, counter or event moves off the round thread. A key nobody hinted is still read inline. Concurrency 1 issues no hints and is byte-for-byte today's behavior; the new `cas_gc_read_concurrency` setting is plumbed like `gc_meta_pool_size` and refused at 0 like `gc_shards`, with three `ProfileEvent`s for hits, misses and wasted results. Measured against a fixed per-request latency: `fold_ref_intake` 2.4x, `fold_reduce` 1.2x, the round overall 1.65x. Intake's speedup stops there because the round issues ref-log and manifest `GET`s one to one and a manifest key is only known once its log is decoded — that chain, and the graduation gate's inline meta re-check, are recorded as follow-up items. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../cas/architecture/garbage-collection.md | 1 + docs/en/antalya/cas/configuration.md | 1 + docs/en/operations/storing-data.md | 3 + src/Common/ProfileEvents.cpp | 3 + .../ContentAddressedMetadataStorage.cpp | 3 + .../ContentAddressedMetadataStorage.h | 2 + .../ContentAddressedSettings.cpp | 10 +- .../ContentAddressed/Gc/CasGc.cpp | 265 ++++++++- .../ContentAddressed/Gc/CasGc.h | 11 +- .../ContentAddressed/Gc/CasGcReadAhead.cpp | 116 ++++ .../ContentAddressed/Gc/CasGcReadAhead.h | 88 +++ .../ContentAddressed/Pool/CasPool.h | 5 + src/Disks/tests/gtest_cas_gc_read_ahead.cpp | 553 ++++++++++++++++++ src/Disks/tests/gtest_cas_settings.cpp | 11 +- 14 files changed, 1050 insertions(+), 22 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h create mode 100644 src/Disks/tests/gtest_cas_gc_read_ahead.cpp diff --git a/docs/en/antalya/cas/architecture/garbage-collection.md b/docs/en/antalya/cas/architecture/garbage-collection.md index c918a0a35f14..a604104c7796 100644 --- a/docs/en/antalya/cas/architecture/garbage-collection.md +++ b/docs/en/antalya/cas/architecture/garbage-collection.md @@ -226,6 +226,7 @@ the user-facing configuration surface. | Setting | Default | Bounds | |---|---|---| | `cas_gc_meta_pool_size` | 16 | bounded pool for condemn-marker writes | +| `cas_gc_read_concurrency` | 16 | bounded pool for the fold's read-ahead; `1` disables | ## Observability {#observability} diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 7939db3034bf..c997ecb5fa30 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -100,6 +100,7 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_part_folder_cache_max_entry_bytes` | 16 MiB | Oversized part-folder views bypass retention above this size | | `cas_manifest_decode_cache_bytes` | 128 MiB | Manifest decode cache byte budget (`0` disables) | | `cas_gc_meta_pool_size` | `16` | Bounded pool size for GC per-hash freshness-meta writes | +| `cas_gc_read_concurrency` | `16` | Bounded pool size for the GC fold's read-ahead of checkpoints, ref logs, manifests and zero-candidate HEADs; `1` disables | | `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove) | | `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: `cas_attempt_timeout_ms + cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, or the disk refuses to open writable | | `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount | diff --git a/docs/en/operations/storing-data.md b/docs/en/operations/storing-data.md index 95e1273f83ea..099a3d0daa10 100644 --- a/docs/en/operations/storing-data.md +++ b/docs/en/operations/storing-data.md @@ -546,6 +546,9 @@ disk-level and server-level settings surface. - `cas_gc_meta_pool_size` — `16` by default. Bounded thread-pool size for the GC's per-hash freshness-meta writes (condemn/spare/delete), so a mass `DROP` condemning millions of blobs does not run fully sequentially. +- `cas_gc_read_concurrency` — `16` by default. Bounded thread-pool size for the GC fold's read-ahead of + checkpoints, ref logs, manifest bodies and zero-candidate `HEAD`s. The fold's decisions stay on the + round thread in their original order; only the fetches overlap. `1` disables read-ahead. - `skip_access_check` — `false` by default. Skips the disk's `CAS` capability probe ("start now, fix later"). The server-level `skip_access_check` flag skips the generic disk access check; this disk key governs the `CAS` capability probe. diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index ad3d22f1925b..5d65defbaad2 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -874,6 +874,9 @@ The server successfully detected this situation and will download merged part fr M(CASGCGetStream, "Number of streaming CAS GC GET requests. Grows with large collection or recovery reads.", ValueType::Number) \ M(CASGCDelete, "Number of CAS GC DELETE requests. Grows with successful cleanup attempts.", ValueType::Number) \ M(CASGCList, "Number of CAS GC LIST requests. Growing values indicate more collection enumeration.", ValueType::Number) \ + M(CASGCReadAheadHit, "Number of CAS GC fold reads and HEADs answered by the fold's read-ahead. Growth means the round's small-object round trips overlapped instead of serializing.", ValueType::Number) \ + M(CASGCReadAheadMiss, "Number of CAS GC fold reads and HEADs performed inline because nothing was hinted for the key. A large value against hits means a hint set is narrower than the walk.", ValueType::Number) \ + M(CASGCReadAheadWasted, "Number of CAS GC read-ahead results fetched and never taken: a namespace held below its lookahead, or a HEAD candidate that kept an edge. Bounded by the read-ahead window per namespace.", ValueType::Number) \ M(CASServerPut, "Number of CAS server-object PUT requests. Grows with server metadata writes.", ValueType::Number) \ M(CASServerPutDeduplicated, "Number of deduplicating CAS server-object PUT requests. Growth indicates reused server objects.", ValueType::Number) \ M(CASServerOverwrite,"Number of CAS server-object overwrite requests. Growing values indicate repeated replacement writes.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index 7e2d88658fe0..f1227ccf952a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -81,6 +81,7 @@ namespace ContentAddressedSetting extern const ContentAddressedSettingsUInt64 part_folder_cache_max_entry_bytes; extern const ContentAddressedSettingsUInt64 manifest_decode_cache_bytes; extern const ContentAddressedSettingsUInt64 gc_meta_pool_size; + extern const ContentAddressedSettingsUInt64 gc_read_concurrency; extern const ContentAddressedSettingsUInt64 attempt_timeout_ms; extern const ContentAddressedSettingsUInt64 lease_safety_margin_ms; extern const ContentAddressedSettingsBool blob_hash_allow_new; @@ -300,6 +301,7 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage( , cas_part_folder_cache_max_entry_bytes(settings_[ContentAddressedSetting::part_folder_cache_max_entry_bytes].value) , manifest_decode_cache_bytes(settings_[ContentAddressedSetting::manifest_decode_cache_bytes].value) , gc_meta_pool_size(settings_[ContentAddressedSetting::gc_meta_pool_size].value) + , gc_read_concurrency(settings_[ContentAddressedSetting::gc_read_concurrency].value) , cas_attempt_timeout_ms(settings_[ContentAddressedSetting::attempt_timeout_ms].value) , cas_lease_safety_margin_ms(settings_[ContentAddressedSetting::lease_safety_margin_ms].value) , staging_backend(settings_.stagingBackend()) @@ -765,6 +767,7 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.gc_round_handoff_prefix_wholesale_budget = gc_round_handoff_prefix_wholesale_budget; pool_config.gc_round_outcome_entry_budget = gc_round_outcome_entry_budget; pool_config.gc_meta_pool_size = gc_meta_pool_size; + pool_config.gc_read_concurrency = gc_read_concurrency; pool_config.cas_request_budget.attempt_timeout_ms = cas_attempt_timeout_ms; pool_config.cas_request_budget.lease_safety_margin_ms = cas_lease_safety_margin_ms; pool_config.event_sink = makeCasEventSink(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 183b470367c9..c5c1d8c788b0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -610,6 +610,8 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC const uint64_t manifest_decode_cache_bytes; /// Bounded pool size for GC's per-hash freshness-metadata writes. const uint64_t gc_meta_pool_size; + /// Bounded pool size for the GC fold's read-ahead; 1 disables it. + const uint64_t gc_read_concurrency; /// The budget for one HTTP attempt of a writable Native mount's control-plane requests; feeds /// `Cas::PoolConfig::cas_request_budget.attempt_timeout_ms` and the backend's own /// `attemptTimeoutMs()`. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index d1fc07746645..76efbfe790f1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -76,6 +76,7 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, part_folder_cache_max_entry_bytes, 16ULL << 20, "Oversized part-folder views bypass retention above this size", 0) \ DECLARE(UInt64, manifest_decode_cache_bytes, 128ULL << 20, "Manifest DECODE cache byte budget (0 disables)", 0) \ DECLARE(UInt64, gc_meta_pool_size, 16, "Bounded pool size for GC per-hash freshness-meta writes", 0) \ + DECLARE(UInt64, gc_read_concurrency, 16, "Bounded pool size for the GC fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate HEADs; 1 disables read-ahead", 0) \ DECLARE(UInt64, attempt_timeout_ms, 5000, "Budget for one HTTP attempt of a writable Native mount's control-plane requests", 0) \ DECLARE(UInt64, lease_safety_margin_ms, 2000, "Startup-only margin validated against the mount lease TTL (attempt_timeout_ms + this must be strictly less than the lease TTL)", 0) \ DECLARE(String, staging_backend, "local", "Blob staging backend (local | s3); s3 is opt-in", 0) \ @@ -223,10 +224,13 @@ void ContentAddressedSettings::validate() { auto & settings = *this; - if (settings[ContentAddressedSetting::gc_interval_sec] == 0 || settings[ContentAddressedSetting::gc_shards] == 0) + if (settings[ContentAddressedSetting::gc_interval_sec] == 0 || settings[ContentAddressedSetting::gc_shards] == 0 + || settings[ContentAddressedSetting::gc_read_concurrency] == 0) throw Exception(ErrorCodes::BAD_ARGUMENTS, - "content_addressed disk: cas_gc_interval_sec and cas_gc_shards must be >= 1 (got {}, {})", - settings[ContentAddressedSetting::gc_interval_sec].value, settings[ContentAddressedSetting::gc_shards].value); + "content_addressed disk: cas_gc_interval_sec, cas_gc_shards and cas_gc_read_concurrency must be >= 1 " + "(got {}, {}, {})", + settings[ContentAddressedSetting::gc_interval_sec].value, settings[ContentAddressedSetting::gc_shards].value, + settings[ContentAddressedSetting::gc_read_concurrency].value); /// The layout subtree identity is explicit and REQUIRED — no default, so an ABSENT key throws a /// typed `NO_ELEMENTS_IN_CONFIG` (mirroring the `metadata_type` check in `MetadataStorageFactory`), diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index f13e1a7fe763..f364d8b9538b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -29,6 +29,13 @@ #include #include +namespace CurrentMetrics +{ + extern const Metric LocalThread; + extern const Metric LocalThreadActive; + extern const Metric LocalThreadScheduled; +} + namespace ProfileEvents { extern const Event CASGCClampSuppressedPasses; @@ -332,6 +339,14 @@ Gc::Gc(PoolPtr store_, UInt128 gc_id_, std::function now_ms_fn_, /// `store->poolConfig()` AFTER the null check above. meta_writer = std::make_unique( store, logger, static_cast(store->poolConfig().gc_meta_pool_size)); + /// The fold's read-ahead pool, built here for the same reason. The queue is UNBOUNDED because the + /// hinting sites throttle themselves against `GcReadAhead::window`; a bounded queue would only + /// move the throttle into `scheduleOrThrowOnError`, blocking the round thread instead of the + /// hint loop that already knows how much it wants in flight. + const size_t read_concurrency = std::max(1, store->poolConfig().gc_read_concurrency); + read_pool = std::make_unique( + CurrentMetrics::LocalThread, CurrentMetrics::LocalThreadActive, CurrentMetrics::LocalThreadScheduled, + /*max_threads*/ read_concurrency, /*max_free_threads*/ read_concurrency, /*queue_size*/ 0); } void Gc::runNamespaceJanitorPage( @@ -1196,7 +1211,7 @@ void Gc::reportStuckRemovals(const RefPlan & plan, uint64_t current_round) } } -bool Gc::foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, std::vector & deltas, +bool Gc::foldManifestEdges(GcReadAhead & reads, const ManifestId & id, int sign, std::vector & deltas, std::map & mf_cleanup, uint32_t txn_ordinal) { const Layout & layout = store->layout(); @@ -1208,7 +1223,11 @@ bool Gc::foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, s /// absent outcome the missing HEAD used to produce -- record-and-continue, and the caller decides /// what an absent body means for that edge (a missing-body precommit is a barrier; a committed one /// fails closed). Never a throw: a 404 during the fold is an observation, not an error. - const auto got = op.read(key, Retry::standard()); + /// + /// The bytes may already have been fetched when the log that named this edge was decoded (all of + /// that log's edges are hinted together, since one decode names them all). The absence signal, the + /// decode and every decision below still happen HERE, in edge order, exactly as they always did. + const auto got = reads.takeRead(key); if (!got) return false; /// absent body: caller decides (missing-body precommit OK; committed => fail closed) ProfileEvents::increment(ProfileEvents::CASRefManifestBodyFoldGets); /// one body GET per manifest fold @@ -1278,7 +1297,8 @@ bool Gc::foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, s return true; } -Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::map & ref_tables, +Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(GcReadAhead & reads, + const std::map & ref_tables, const CasRefCatalog::Snapshot & catalog_cut) { /// Read the checkpoint of every namespace in the round's catalog cut, every namespace `ref_tables` @@ -1290,7 +1310,12 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::mapopenRequests().admit(); + /// + /// READS UNDER THE CALLER'S ADMISSION, not one of its own. This function used to admit a fresh + /// operation, which meant that a fence moving mid-round would hand it a NEWER generation than the + /// round holds and let it read on regardless; taking the caller's read-ahead makes the same fence + /// movement fail this read the way it fails every other read of the round. Strictly the + /// fail-closed direction, and it is why there is no `admit` here any more. const Layout & layout = store->layout(); std::set witness_namespaces; @@ -1300,7 +1325,16 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::map witness_keys; + witness_keys.reserve(witness_namespaces.size()); for (const String & ns_str : witness_namespaces) { const RootNamespace ns{ns_str}; @@ -1313,13 +1347,30 @@ Gc::CheckpointWitnesses Gc::readCheckpointWitnesses(const std::mapns != ns || (entry_it->state != NsState::Live && entry_it->state != NsState::Removing)) continue; - const String ckpt_key = layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(entry_it->ns, entry_it->incarnation)); + witness_keys.push_back( + {ns_str, layout.refCkptKey(NamespaceLifeId::fromCatalogEntry(entry_it->ns, entry_it->incarnation))}); + } + + size_t next_hint = 0; + const auto topUpWitnessHints = [&] + { + while (next_hint < witness_keys.size() && reads.pending() < reads.window()) + reads.hintRead(witness_keys[next_hint++].ckpt_key); + }; + + CheckpointWitnesses out; + for (const WitnessKey & witness_key : witness_keys) + { + const String & ns_str = witness_key.ns; + const String & ckpt_key = witness_key.ckpt_key; + topUpWitnessHints(); /// THE GET AND THE DECODE ARE SPLIT HERE, rather than taken together through `readCkpt`, so the /// catch below can scope to the DECODE ALONE. Wrapping the read too would turn a transport /// failure -- which says nothing about this object and everything about the round's ability to /// read anything -- into a per-namespace hold, silently narrowing a pool-wide outage to one - /// namespace. A backend throw still propagates and fails the round, exactly as it always did. - const std::optional got = op.read(ckpt_key, Retry::standard()); + /// namespace. A backend throw still propagates and fails the round, exactly as it always did -- + /// a read-ahead worker's failure is rethrown by the take below, at this same site. + const std::optional got = reads.takeRead(ckpt_key); /// ABSENT IS NORMAL AND IS NOT A WITNESS: a namespace has no `_ckpt` until its first snapshot /// publication commits, and one that 404s mid-round is a namespace being reclaimed. Neither says /// anything about which ids exist, so neither may hold the walk -- and neither may throw @@ -1576,6 +1627,11 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, { CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); + /// The fold's read-ahead. It fetches through `op`'s own admitted generation and hands every result + /// back at the site that would otherwise have read inline, so the walk's order, its counters, its + /// holds and its events are what they were; only the moment of the fetch moves. At + /// `gc_read_concurrency` 1 it hints nothing and every take IS the original inline read. + GcReadAhead reads(op, store->openRequests(), *read_pool, store->poolConfig().gc_read_concurrency); FoldResult result; /// 1. Group the round's one enumeration of `cas/ns/stream/` (taken before the defer decision) into @@ -1703,9 +1759,33 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, const uint64_t condemn_round = state.round + 1; result.retired_merge.resize(state.gc_shards); + /// HEAD READ-AHEAD FOR THE REDUCE PHASE. `head_candidates[shard]` is filled in that phase with the + /// blobs the merge can bring to in-degree zero, in the merge's own ascending key order; `head_blob` + /// below tops the hints up a window deep before each take. It is EMPTY everywhere else, the whole of + /// intake included, so every take outside that phase is the plain inline HEAD. + /// + /// Hints are issued from INSIDE the lambda rather than in one burst at phase start, so the requests + /// this can ever add are bounded by one window past the last candidate the merge actually reaches. + /// A superset that overshoots badly therefore costs a window, not its own size -- which matters + /// because nothing bounds a round's condemnation count. + std::vector> head_candidates(state.gc_shards); + size_t head_hint_shard = 0; + size_t next_head_hint = 0; + const auto topUpHeadHints = [&] + { + const std::vector & shard_candidates = head_candidates[head_hint_shard]; + while (next_head_hint < shard_candidates.size() && reads.pending() < reads.window()) + reads.hintHead(layout.blobKey(shard_candidates[next_head_hint++])); + }; + /// Condemn-time observation: ONE HEAD per new zero-transition captures the exact incarnation token /// the eventual delete carries (absent => a prior landed delete => nothing to condemn). Emits the /// Candidate trail (IndegZero / GcRetireObserve / BlobRetire) exactly where the decision is made. + /// + /// THE READ-AHEAD NEVER RUNS THIS LAMBDA, only feeds it. Everything below the HEAD is + /// side-effecting -- the trail, the counters, the condemn-marker write -- and running it over a + /// SUPERSET would stamp `Condemned` on blobs this round never condemns, forcing a live writer to + /// republish each one. The prefetch is a bare HEAD; the decision stays here. const auto head_blob = [&](const BlobRef & ref) -> std::optional { EventEmitter{*store}.emit([&](CasEvent & e) @@ -1717,7 +1797,8 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, e.gen = state.snap_generation + 1; e.reason = "last folded owner edge dropped; in-degree reached 0"; }); - const std::optional observed = op.head(layout.blobKey(ref), Retry::standard()); + topUpHeadHints(); + const std::optional observed = reads.takeHead(layout.blobKey(ref)); EventEmitter{*store}.emit([&](CasEvent & e) { e.type = CasEventType::GcRetireObserve; @@ -1764,6 +1845,15 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// hook is `head_blob` above, reserved for a genuinely NEW zero-in-degree candidate. A supersede's /// own event is `blob_retire_replaced`, emitted once below from `merge.replaced`. Plain HEAD, no /// events, no counters. + /// + /// AND IT IS NOT READ AHEAD, deliberately, unlike `head_blob`'s. This HEAD is not data handed to a + /// decision made elsewhere -- it IS the decision: `hr && !stale.token.matches(hr->etag)` is the + /// supersede branch itself, so observing earlier narrows the window in which a republication can be + /// seen and would genuinely change which entries supersede. The consequence of a missed supersede is + /// benign (the stale entry graduates and its exact-token delete mismatches, so reclamation is + /// delayed, never wrong), but "only the moment of the fetch moves, never a decision" is the property + /// this whole read-ahead is worth trusting for, and it is not worth spending on the rare blob that + /// carries a condemned row AND is touched again in the same round. const auto peek_head = [&](const BlobRef & ref) -> std::optional { std::optional hr = op.head(layout.blobKey(ref), Retry::standard()); @@ -1908,6 +1998,15 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// It also carries probe B1's two numbers -- reported on EVERY healthy round, so /// "logs_accounted always equals logs_applied" becomes an observable property of the table rather than /// a claim in a comment. + /// + /// THE REQUESTS ARE THE SAME ONES; ONLY THEIR TIMING MOVED. Four sites below hand their keys to the + /// round's `GcReadAhead` before the walk reaches them -- the checkpoints, each namespace's first walk + /// position, this epoch's next positions, and a decoded log's manifest edges -- so the phase's round + /// trips overlap instead of running strictly one after another. Every take happens where the inline + /// read happened, in the same order, and increments the same counters, which is why this row's + /// semantic metrics are identical at any `gc_read_concurrency`. Its S3 VERB counts are not: a request + /// a worker performed lands on that worker's ProfileEvents, the same gap `meta_pool_wait` has always + /// had. Read `CASGCReadAheadHit`/`Miss`/`Wasted` on this row for the read-ahead's own behaviour. std::optional intake_timer; intake_timer.emplace(phase_sink, "fold_ref_intake"); uint64_t intake_tables_changed = 0; @@ -1926,7 +2025,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// The round's SECOND witness source, independent of the listing -- see `readCheckpointWitnesses` /// for what it decides and why a listing alone cannot decide it. Its `undecodable` half names the /// namespaces whose `_ckpt` is present and unreadable; each of those is HELD below, and only those. - const CheckpointWitnesses checkpoints = readCheckpointWitnesses(ref_tables, catalog_snapshot); + const CheckpointWitnesses checkpoints = readCheckpointWitnesses(reads, ref_tables, catalog_snapshot); const std::map & checkpoint_witness = checkpoints.witnesses; /// WHICH NAMESPACES THIS ROUND WALKS -- i.e. THE ROUND'S UNIVERSE, the set the destructive gate owes @@ -2047,12 +2146,75 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, ++intake_tails_below_cursor; } + /// READ-AHEAD OF EACH NAMESPACE'S FIRST WALK POSITION, in walk order, kept a window deep. On a wide + /// pool this is the phase's shape: many namespaces, one read each, previously taken strictly one + /// after another. + /// + /// The key is recomputed from exactly the inputs the walk below uses -- the sealed cursor, the + /// catalog entry and the checkpoint grounding -- and ONLY where the walk will actually read it. A + /// namespace whose first position sits above `committed_through` is refused by the ceiling test + /// before any read (this is the ordinary QUIET namespace: its frontier is proved by the ceiling, + /// and it costs no request at all today), so hinting it would ADD a request the round never makes. + /// A namespace whose checkpoint is unusable is held without reading, and is skipped here for the + /// same reason. + std::vector first_walk_keys; + first_walk_keys.reserve(walk_targets.size()); + for (const WalkTarget & target : walk_targets) + { + if (checkpoints.undecodable.contains(target.ns)) + continue; + const RootNamespace target_ns{target.ns}; + const auto target_entry_it = std::lower_bound( + catalog_snapshot.catalog.entries.begin(), catalog_snapshot.catalog.entries.end(), target_ns, + [](const CatalogEntry & entry, const RootNamespace & needle) { return entry.ns < needle; }); + if (target_entry_it == catalog_snapshot.catalog.entries.end() || target_entry_it->ns != target_ns) + continue; + + std::optional target_checkpoint; + if (const auto it = checkpoints.recovery_checkpoints.find(target.ns); + it != checkpoints.recovery_checkpoints.end()) + target_checkpoint = it->second; + + std::optional target_grounding; + try + { + target_grounding = chooseRecoveryGrounding(std::optional{*target_entry_it}, target_checkpoint); + } + catch (const Exception &) + { + continue; /// the walk below holds this namespace without reading; so does the hint pass + } + if (!target_grounding->committed_through) + continue; + + const auto target_cursor_it = parent_ref_lives.find(target.life_id); + const RefTxnId target_cursor = target_cursor_it != parent_ref_lives.end() + ? target_cursor_it->second.coverage.last_folded_ref_id : RefTxnId{}; + std::optional target_expected; + if (target_cursor != RefTxnId{}) + target_expected = RefTxnId{target_cursor.writer_epoch, target_cursor.ref_sequence + 1}; + else if (target_checkpoint && target_checkpoint->life_epoch) + target_expected = RefTxnId{*target_checkpoint->life_epoch, 1}; + if (!target_expected || *target_grounding->committed_through < *target_expected) + continue; + + first_walk_keys.push_back(layout.refLogKey( + NamespaceLifeId::fromCatalogEntry(target_ns, target.life_id), *target_expected)); + } + size_t next_first_walk_hint = 0; + const auto topUpFirstWalkHints = [&] + { + while (next_first_walk_hint < first_walk_keys.size() && reads.pending() < reads.window()) + reads.hintRead(first_walk_keys[next_first_walk_hint++]); + }; + for (const WalkTarget & target : walk_targets) { const String & ns_str = target.ns; const RefTableListing & listing = *target.listing; if (ref_folding_aborted) break; + topUpFirstWalkHints(); const RootNamespace ns{ns_str}; /// Every walk target came out of this round's own catalog read, so its incarnation is the REAL /// one and the life below is minted from a catalog entry rather than guessed from a key. @@ -2335,7 +2497,21 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// GET + decode the expected record. Absence is the decision point of the whole walk, and /// an invalid body is a per-namespace hold: the key belongs to exactly one namespace, so it /// can never be grounds for discarding another namespace's fold. - const auto got = op.read(layout.refLogKey(life, *expected), Retry::standard()); + /// + /// LOOKAHEAD over this epoch's next arithmetic positions, and never past the ceiling the + /// test above enforces: `committed_through` was snapshotted before the walk and bounds + /// what this round may read at all, so every position hinted here is one this walk goes on + /// to read unless something stops it first. A hold or an epoch crossing stops it, leaving + /// at most a window's worth of bodies fetched and untaken -- bounded, counted, and never a + /// read the sequential walk would not have made. + for (uint64_t ahead_k = 1; ahead_k <= reads.window(); ++ahead_k) + { + const RefTxnId ahead{expected->writer_epoch, expected->ref_sequence + ahead_k}; + if (*grounding->committed_through < ahead) + break; + reads.hintRead(layout.refLogKey(life, ahead)); + } + const auto got = reads.takeRead(layout.refLogKey(life, *expected)); if (!got) { ++intake_absent_probes; @@ -2426,12 +2602,21 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// re-fold would then clamp on that missing body forever. A missing manifest body is a per-table /// CLAMP (barrier), never a round abort: keep the cursor below THIS log and re-read it next /// round. A removed precommit whose body never existed emitted no edge -- skip, no clamp. + /// + /// The decode above named every manifest key this log folds, so they are fetched together + /// here and taken one at a time in edge order below. A log with a single edge gains + /// nothing; a merge or a mutation log with dozens turns dozens of serial round trips into + /// one. A clamp mid-log leaves the rest of this log's bodies fetched and untaken, which is + /// the bounded waste the read-ahead counts. + for (const RefManifestEdge & edge : edges) + reads.hintRead(layout.manifestKey(edge.manifest_id)); + std::vector log_deltas; std::map log_mf_cleanup; for (const RefManifestEdge & edge : edges) { ProfileEvents::increment(ProfileEvents::CASRefEmittedEdges); /// one manifest-edge event - if (foldManifestEdges(op, edge.manifest_id, edge.change, log_deltas, log_mf_cleanup, + if (foldManifestEdges(reads, edge.manifest_id, edge.change, log_deltas, log_mf_cleanup, txn_ordinal)) continue; @@ -3039,6 +3224,39 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, nomination.source_retirements.end()); } + /// WHICH BLOBS THE MERGE CAN CLOSE AT ZERO. `closeBlob` HEADs under `cur_edges == 0 && cur_touched`: + /// no key of the blob ended present, and at least one was touched. Every prior-run edge that survives + /// increments `cur_edges`, so reaching zero requires each of them to be killed at its own key by a + /// `-1` delta or a source retirement -- which puts a removal-last verdict in the map below -- while + /// any activation-last verdict leaves an edge standing and is the exclusion clause. Retirements are + /// folded in AFTER the deltas because the merge applies them that way, unconditionally: a key whose + /// deltas end in an activation but which a retirement then clears is a candidate too. + /// + /// IT IS A SUPERSET, AND NOTHING MAY COME TO DEPEND ON IT BEING EXACT. A blob named here that keeps + /// an untouched prior edge costs one HEAD the merge never takes; a candidate this set misses is + /// HEADed inline, which is simply the behaviour with no read-ahead at all. Both are counted, and + /// neither is asserted. + /// + /// PLACED HERE, not at the top of the phase: `orphan_source_retirements` is decided just above by + /// the sweep, and a retirement is a removal like any other. The round's cut is frozen well before + /// this point -- intake has finished and `deltas` has taken its final form -- so no HEAD is issued + /// before the round knows what it folded. + { + std::map, bool> last_verdict_is_remove; + for (const BlobDelta & delta : deltas) + last_verdict_is_remove[{delta.ref, delta.source_id}] = delta.remove; + for (const BlobSourceRetirement & retirement : orphan_source_retirements) + last_verdict_is_remove[{retirement.ref, retirement.source_id}] = true; + + std::set removed; + std::set surviving_add; + for (const auto & [edge, is_remove] : last_verdict_is_remove) + (is_remove ? removed : surviving_add).insert(edge.first); + for (const BlobRef & ref : removed) + if (!surviving_add.contains(ref)) + head_candidates[blobShard(ref, state.gc_shards)].push_back(ref); + } + if (state.gc_shards == 1) { /// SINGLE-SHARD PATH (gc_shards == 1). Every blob routes to shard 0, so the entire delta stream @@ -3054,6 +3272,8 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, { /// Either a real delta or a non-empty retired input: run the merge (empty deltas still settle /// the RunMarker::Condemned rows riding the parent run). The prior runs are the parent seal's shard-0 refs. + head_hint_shard = 0; + next_head_hint = 0; foldDeltasIntoGeneration(op, layout, priorRunsFor(0), new_generation, attempt, /*shard*/0, std::move(deltas), result.fold_seal.blob_target_runs, @@ -3096,6 +3316,8 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// A reducer owns exactly one disjoint shard. Two replicas may run reducers for DIFFERENT /// shards concurrently (CasGcScheduler ownership); their run-key namespaces never collide. std::vector shard_runs; + head_hint_shard = shard; + next_head_hint = 0; foldDeltasIntoGeneration( op, layout, priorRunsFor(shard), new_generation, attempt, shard, std::move(buckets[shard]), shard_runs, @@ -3763,6 +3985,17 @@ RebuildReport Gc::rebuildBaseline(bool force) RebuildReport rep; CasOperation op = store->openRequests().admit(); const Layout & layout = store->layout(); + /// The rebuild reads through the same seam the fold does, so the two share one implementation of + /// "read this key" rather than growing a second. + /// + /// WHAT THAT MEANS HERE, exactly: `readCheckpointWitnesses` hints its own keys and does not ask who + /// called it, so the rebuild's checkpoint reads ARE fetched ahead, on the same pool and by the same + /// rule as the fold's. That is a gain and not an accident -- a rebuild reads every namespace's + /// checkpoint too. Nothing on THIS path hints a manifest key, so `foldManifestEdges` below takes + /// them one at a time, exactly as it did before: a rebuild walks a plan it already holds rather + /// than discovering its next key from the body it just read, so a lookahead would have nothing to + /// hide behind. + GcReadAhead reads(op, store->openRequests(), *read_pool, store->poolConfig().gc_read_concurrency); /// Read bookkeeping health before the lease (the lease acquire on an absent state CREATES a /// bootstrap body, which must not make scenario (а) look healthy). A generation-0 ref-baseline @@ -3942,7 +4175,7 @@ RebuildReport Gc::rebuildBaseline(bool force) /// universe. `recoverRefTableDetailedFromAuthority` deliberately has no internal catalog or /// checkpoint read: a later cut could admit a different life or frontier than the one every other /// part of this rebuild is using. - const CheckpointWitnesses rebuild_checkpoints = readCheckpointWitnesses({}, rebuild_walk_plan.catalogCut()); + const CheckpointWitnesses rebuild_checkpoints = readCheckpointWitnesses(reads, {}, rebuild_walk_plan.catalogCut()); if (validate_generation_zero_ref_baseline) { @@ -4112,7 +4345,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { const ManifestId id{ns, row.manifest_ref}; owned_manifest_keys.insert(layout.manifestKey(id)); - if (!foldManifestEdges(op, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) + if (!foldManifestEdges(reads, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) { rep.refusal = "committed ref '" + ns.string() + "/" + ref_name + "' names a missing or invalid part manifest — that is DATA LOSS the rebuild " @@ -4128,7 +4361,7 @@ RebuildReport Gc::rebuildBaseline(bool force) { const ManifestId id{ns, manifest_ref}; owned_manifest_keys.insert(layout.manifestKey(id)); - if (foldManifestEdges(op, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) + if (foldManifestEdges(reads, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) ++rep.live_precommits; else { @@ -4200,7 +4433,7 @@ RebuildReport Gc::rebuildBaseline(bool force) if (prefixEligible(*store, ns, BuildPrefix{mref.writer_epoch, mref.build_sequence})) return true; /// provably dead — the orphan sweep's territory, never an edge const ManifestId id{ns, mref}; - if (foldManifestEdges(op, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) + if (foldManifestEdges(reads, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0)) { ++rep.unowned_alive_manifests; route_deltas(deltas); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index 4cbd9733c19f..4216912604e5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -745,7 +746,8 @@ class Gc /// (`HoldReason::CheckpointUndecodable`) and folds every other namespace normally. std::map undecodable; }; - CheckpointWitnesses readCheckpointWitnesses(const std::map & ref_tables, + CheckpointWitnesses readCheckpointWitnesses(GcReadAhead & reads, + const std::map & ref_tables, const CasRefCatalog::Snapshot & catalog_cut); /// What ONE generation's prefix says about itself: whether the generation exists at all, and the @@ -785,7 +787,7 @@ class Gc /// PRESENT but fails refMatchesBody / manifestNamespaceMatches throws CORRUPTED_DATA. /// `txn_ordinal` stamps every delta this call pushes with the round-local ordinal of the ref /// transaction that emitted it (probe B2 — see `TxnApplyLedger`). - bool foldManifestEdges(CasOperation & op, const ManifestId & id, int sign, std::vector & deltas, + bool foldManifestEdges(GcReadAhead & reads, const ManifestId & id, int sign, std::vector & deltas, std::map & mf_cleanup, uint32_t txn_ordinal); @@ -952,6 +954,11 @@ class Gc /// initialized before that check. std::unique_ptr meta_writer; + /// The fold's read-ahead pool, sized by `gc_read_concurrency`. A `unique_ptr` for the same reason + /// as `meta_writer`: the size comes from `store->poolConfig()`, which may only be read after the + /// constructor body has validated `store`. + std::unique_ptr read_pool; + /// Probe B1's two numbers for the round: the ref-log POSITIONS the sealed coverage declares covered /// (counted arithmetically over each namespace's cut -- not by listed ids, which under arithmetic /// intake say nothing about what was applied), and the ref logs that actually folded. They are EQUAL diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp new file mode 100644 index 000000000000..2c1d9cf313fb --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp @@ -0,0 +1,116 @@ +#include +#include + +#include + +namespace ProfileEvents +{ + extern const Event CASGCReadAheadHit; + extern const Event CASGCReadAheadMiss; + extern const Event CASGCReadAheadWasted; +} + +namespace DB::Cas +{ + +GcReadAhead::GcReadAhead(CasOperation & op_, CasRequests & requests_, ThreadPool & pool_, size_t concurrency_) + : op(op_), requests(requests_), pool(pool_), concurrency(concurrency_), generation(op_.generation()) +{ +} + +GcReadAhead::~GcReadAhead() +{ + /// A worker holds its own slot and touches `requests`, which outlives this object; waiting here is + /// what keeps every worker inside the round that issued it. `wait`, not `get`: an exception nobody + /// took is dropped with the result it belongs to, and a destructor may not throw. + size_t wasted = 0; + for (auto & [key, slot] : reads) + { + slot->future.wait(); + ++wasted; + } + for (auto & [key, slot] : heads) + { + slot->future.wait(); + ++wasted; + } + if (wasted != 0) + ProfileEvents::increment(ProfileEvents::CASGCReadAheadWasted, wasted); +} + +template +void GcReadAhead::hint(Slots & slots, const String & key, Request request) +{ + if (concurrency <= 1 || slots.contains(key)) + return; + + auto slot = std::make_shared>(); + slots.emplace(key, slot); + try + { + pool.scheduleOrThrowOnError([slot, key, requests_ptr = &requests, gen = generation, request] + { + try + { + CasOperation worker = requests_ptr->resume(gen); + slot->promise.set_value(request(worker, key)); + } + catch (...) + { + slot->promise.set_exception(std::current_exception()); + } + }); + } + catch (...) + { + /// Nothing will ever satisfy this slot's promise, so a later take would wait on it forever. + /// Drop it and let the take read inline; the scheduling failure itself propagates to the + /// hinting site, which is a round-thread site like any other. + slots.erase(key); + throw; + } +} + +template +std::optional GcReadAhead::take(Slots & slots, const String & key, Inline inline_request) +{ + const auto it = slots.find(key); + if (it == slots.end()) + { + ProfileEvents::increment(ProfileEvents::CASGCReadAheadMiss); + return inline_request(op, key); + } + + std::shared_ptr> slot = std::move(it->second); + slots.erase(it); + ProfileEvents::increment(ProfileEvents::CASGCReadAheadHit); + /// Rethrows the worker's exception at the site that would otherwise have read inline, so a + /// transport failure fails the round from the same place and with the same type it always did. + return slot->future.get(); +} + +void GcReadAhead::hintRead(const String & key) +{ + hint(reads, key, + [](CasOperation & worker, const String & k) { return worker.read(k, Retry::standard()); }); +} + +void GcReadAhead::hintHead(const String & key) +{ + hint(heads, key, + [](CasOperation & worker, const String & k) { return worker.head(k, Retry::standard()); }); +} + +std::optional GcReadAhead::takeRead(const String & key) +{ + return take(reads, key, + [](CasOperation & inline_op, const String & k) { return inline_op.read(k, Retry::standard()); }); +} + +std::optional GcReadAhead::takeHead(const String & key) +{ + return take(heads, key, + [](CasOperation & inline_op, const String & k) { return inline_op.head(k, Retry::standard()); }); +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h new file mode 100644 index 000000000000..691eaa2c746f --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h @@ -0,0 +1,88 @@ +#pragma once + +#include +#include +#include + +#include +#include +#include +#include + +namespace DB::Cas +{ + +/// Read-ahead in front of ONE admitted operation. A caller HINTS keys the sequential code will read +/// next; workers fetch them on `pool`, each through an operation resumed under the SAME admitted +/// generation as the caller's (no liveness -- exactly the fold's own admission); a TAKE returns the +/// fetched result, rethrows the worker's exception, or -- for a key nobody hinted -- performs the +/// request inline. This is a cache of RESULTS, never of decisions: every decode, counter and event +/// stays at the take site, so a round at concurrency 1 (nothing is ever hinted) and a round at 16 +/// read the same keys in the same order and decide the same way; only WHEN the bytes were fetched +/// moves. +/// +/// Why a result may be fetched early: the objects the fold reads are write-once (a present body is +/// the same body later), an absent position at or below a checkpoint's `committed_through` was +/// durable before the round began (it is a gap whenever it is read), and a manifest body still being +/// uploaded when the early read lands yields the same hold a slightly earlier sequential read yields +/// today. Nothing is hinted above `committed_through`, so no request is issued that the sequential +/// walk would not issue. +/// +/// Memory is the CALLER's to bound: `pending` counts hinted-but-untaken slots and `window` is how +/// many a hinting site keeps in flight. A key hinted twice is one request. Results never taken are +/// awaited by the destructor and counted as wasted. Only the owning thread touches the maps; a +/// worker touches only its own slot. +class GcReadAhead +{ +public: + GcReadAhead(CasOperation & op_, CasRequests & requests_, ThreadPool & pool_, size_t concurrency_); + ~GcReadAhead(); + + GcReadAhead(const GcReadAhead &) = delete; + GcReadAhead & operator=(const GcReadAhead &) = delete; + + void hintRead(const String & key); + void hintHead(const String & key); + + std::optional takeRead(const String & key); + std::optional takeHead(const String & key); + + /// Hinted but not yet taken, both verbs together: what a hinting site throttles itself against. + size_t pending() const { return reads.size() + heads.size(); } + + /// How many requests a hinting site should keep in flight. Zero at concurrency 1, which is what + /// makes every `while (pending() < window())` loop hint nothing at all on the sequential setting. + size_t window() const { return concurrency <= 1 ? 0 : 4 * concurrency; } + +private: + template + struct Slot + { + std::promise> promise; + std::future> future; + Slot() : future(promise.get_future()) {} + }; + + template + using Slots = std::unordered_map>>; + + template + void hint(Slots & slots, const String & key, Request request); + + template + std::optional take(Slots & slots, const String & key, Inline inline_request); + + CasOperation & op; + CasRequests & requests; + ThreadPool & pool; + const size_t concurrency; + /// The generation the caller's operation was admitted under. A worker resumes under exactly this + /// one, so a fence that moves under the round fails a worker's request the way it fails the + /// round's own -- never with a fresher admission the round itself would not have had. + const uint64_t generation; + + Slots reads; + Slots heads; +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 17369359538a..feee78c93229 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -168,6 +168,11 @@ struct PoolConfig /// feedback_ca_gc_never_throw_on_404) and `Gc::runRegularRound` waits for the round's whole batch /// before the round's single gc/state CAS, so the meta writes are durable before that CAS commits. uint64_t gc_meta_pool_size = 16; + /// Bounded pool size for the fold's read-ahead of checkpoints, ref logs, manifest bodies and + /// zero-candidate HEADs. Every decision stays on the round thread, in the order it always ran; + /// only the fetch overlaps. `1` issues no read-ahead at all and is the sequential round, request + /// for request. + uint64_t gc_read_concurrency = 16; /// Tests drive `renewWatermarkOnce` explicitly; gates both persistent runtime workers. bool background_watermark = false; /// Installed on the pool before a writable mount can start its runtime-owned workers. diff --git a/src/Disks/tests/gtest_cas_gc_read_ahead.cpp b/src/Disks/tests/gtest_cas_gc_read_ahead.cpp new file mode 100644 index 000000000000..cf8ad113a530 --- /dev/null +++ b/src/Disks/tests/gtest_cas_gc_read_ahead.cpp @@ -0,0 +1,553 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +namespace CurrentMetrics +{ + extern const Metric LocalThread; + extern const Metric LocalThreadActive; + extern const Metric LocalThreadScheduled; +} + +using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::idOf; +using DB::Cas::tests::openRequestsForTest; +using DB::Cas::tests::u128Of; + +namespace +{ + +/// ============================ the class, on its own ============================ + +struct ReadAheadRig +{ + std::shared_ptr backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ThreadPool pool{CurrentMetrics::LocalThread, CurrentMetrics::LocalThreadActive, + CurrentMetrics::LocalThreadScheduled, + /*max_threads*/ 4, /*max_free_threads*/ 4, /*queue_size*/ 0}; + + void put(const String & key, const String & bytes) + { + ASSERT_TRUE(std::holds_alternative(op.create(key, bytes, Retry::once()))) << key; + } +}; + +} + +TEST(CASGCReadAhead, HitReturnsTheHintedBytesWithOneRequest) +{ + ReadAheadRig rig; + rig.put("k1", "one"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + rig.backend->resetCounts(); + + reads.hintRead("k1"); + EXPECT_EQ(reads.pending(), 1u); + const auto got = reads.takeRead("k1"); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->bytes, "one"); + EXPECT_EQ(reads.pending(), 0u); + EXPECT_EQ(rig.backend->getCount("k1"), 1u); +} + +TEST(CASGCReadAhead, MissReadsInlineOnTheCallersOperation) +{ + ReadAheadRig rig; + rig.put("k2", "two"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + rig.backend->resetCounts(); + + const auto got = reads.takeRead("k2"); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->bytes, "two"); + EXPECT_EQ(rig.backend->getCount("k2"), 1u); +} + +TEST(CASGCReadAhead, AbsentKeyIsNulloptHintedOrNot) +{ + ReadAheadRig rig; + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + + reads.hintRead("absent-hinted"); + EXPECT_FALSE(reads.takeRead("absent-hinted").has_value()); + EXPECT_FALSE(reads.takeRead("absent-inline").has_value()); +} + +TEST(CASGCReadAhead, DuplicateHintIsOneRequest) +{ + ReadAheadRig rig; + rig.put("k1", "one"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + rig.backend->resetCounts(); + + reads.hintRead("k1"); + reads.hintRead("k1"); + EXPECT_EQ(reads.pending(), 1u); + ASSERT_TRUE(reads.takeRead("k1").has_value()); + EXPECT_EQ(rig.backend->getCount("k1"), 1u); +} + +TEST(CASGCReadAhead, WorkerExceptionRethrowsAtTheTakeSiteAndDoesNotPoisonTheKey) +{ + ReadAheadRig rig; + rig.put("k3", "three"); + /// A non-Poco exception is a deterministic local failure to the engine, so it is thrown on the + /// first attempt rather than reissued. + rig.backend->failNextReadWith("k3", std::make_exception_ptr(std::runtime_error("injected read fault"))); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + + reads.hintRead("k3"); + EXPECT_THROW(reads.takeRead("k3"), std::runtime_error); + EXPECT_EQ(reads.pending(), 0u); + + const auto again = reads.takeRead("k3"); /// the fault was consumed; an inline read now answers + ASSERT_TRUE(again.has_value()); + EXPECT_EQ(again->bytes, "three"); +} + +TEST(CASGCReadAhead, ConcurrencyOneNeverHints) +{ + ReadAheadRig rig; + rig.put("k1", "one"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 1); + rig.backend->resetCounts(); + + EXPECT_EQ(reads.window(), 0u); + reads.hintRead("k1"); + reads.hintHead("k1"); + EXPECT_EQ(reads.pending(), 0u); + ASSERT_TRUE(reads.takeRead("k1").has_value()); + ASSERT_TRUE(reads.takeHead("k1").has_value()); + EXPECT_EQ(rig.backend->getCount("k1"), 1u); + EXPECT_EQ(rig.backend->headCount("k1"), 1u); +} + +TEST(CASGCReadAhead, DestructorWaitsForOutstandingRequests) +{ + ReadAheadRig rig; + rig.put("k1", "one"); + rig.backend->resetCounts(); + { + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + reads.hintRead("k1"); + reads.hintHead("k1"); + } + EXPECT_EQ(rig.backend->getCount("k1"), 1u); + EXPECT_EQ(rig.backend->headCount("k1"), 1u); +} + +TEST(CASGCReadAhead, HeadHitCarriesSizeAndAbsentIsNullopt) +{ + ReadAheadRig rig; + rig.put("k1", "one"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + rig.backend->resetCounts(); + + reads.hintHead("k1"); + const auto meta = reads.takeHead("k1"); + ASSERT_TRUE(meta.has_value()); + EXPECT_EQ(meta->size, 3u); + EXPECT_FALSE(reads.takeHead("absent").has_value()); + EXPECT_EQ(rig.backend->headCount("k1"), 1u); +} + +TEST(CASGCReadAhead, WindowIsFourTimesConcurrency) +{ + ReadAheadRig rig; + GcReadAhead reads(rig.op, rig.requests, rig.pool, 8); + EXPECT_EQ(reads.window(), 32u); +} + +/// ============================ the fold, at 1 against 8 ============================ + +namespace +{ + +const UInt128 kGc = u128Of("gc-read-ahead"); + +ManifestId publishPart(const PoolPtr & s, const String & ns, const String & ref, const String & payload) +{ + const RootNamespace nsr{ns}; + PartWriteInfo info; + info.intended_ref = ns + "/" + ref; + auto build = s->beginPartWrite(info); + + ManifestEntry e; + e.path = "data.bin"; + e.placement = EntryPlacement::Blob; + e.ref = BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(u128Of(payload))}; + e.blob_size = payload.size(); + + const ManifestId id = build->stageManifest({e}); + build->precommitAdd(nsr, ref, id); + build->putBlob(idOf(payload), BlobSource::fromString(payload)); + build->promote(nsr, ref, build->buildId(), id); + return id; +} + +/// Three namespaces. `wide` carries a ref-log backlog longer than any window this file uses, so the +/// same-epoch lookahead is genuinely exercised; `quiet` publishes once and never drops, so its +/// frontier is proved by the checkpoint ceiling with no read at all; `gone` is emptied entirely, so +/// its blobs reach in-degree zero and the reduce phase HEADs them. Every blob is unique to its part. +void populate(const PoolPtr & store) +{ + for (int i = 0; i < 30; ++i) + publishPart(store, "srv1/wide", fmt::format("part_{}", i), fmt::format("wide-payload-{}", i)); + for (int i = 0; i < 15; ++i) + store->dropRef(RootNamespace{"srv1/wide"}, fmt::format("part_{}", i)); + + publishPart(store, "srv1/quiet", "only", "quiet-payload"); + + publishPart(store, "srv1/gone", "a", "gone-payload-a"); + publishPart(store, "srv1/gone", "b", "gone-payload-b"); + store->dropRef(RootNamespace{"srv1/gone"}, "a"); + store->dropRef(RootNamespace{"srv1/gone"}, "b"); + + store->renewWatermarkOnce(); +} + +/// TWO POOLS ARE NOT BYTE-COMPARABLE UNTIL THEIR IDENTITIES ARE MAPPED. A namespace's catalog +/// incarnation is minted from the process RNG at creation, and it appears BOTH inside every one of that +/// namespace's object keys and inside the fold seal's `life` rows -- so two independently created pools +/// running the identical workload produce identical decisions under different names, and the seal's +/// `ref_life` rows come out in a different order because they are keyed by that random id. +/// +/// Neither fact has anything to do with the read-ahead, and hiding them by weakening the comparison +/// would hide the read-ahead's own defects too. So the identities are MAPPED instead of dropped: each +/// run reports its own incarnation-hex -> namespace-name table, every 32-hex id in a key or a seal is +/// rewritten to the namespace it names, and the `ref_life` rows are sorted once their names are stable. +/// What survives the rewrite is everything the fold decided; what it removes is only the naming. +using IdNames = std::map; + +String normalizeIds(const String & text, const IdNames & names) +{ + String out = text; + for (const auto & [hex, name] : names) + { + size_t at = 0; + while ((at = out.find(hex, at)) != String::npos) + { + out.replace(at, hex.size(), name); + at += name.size(); + } + } + return out; +} + +/// The seal with its ids named and its `ref_life` rows sorted; every other row keeps its position. +String canonicalSeal(const String & seal, const IdNames & names) +{ + const String named = normalizeIds(seal, names); + std::vector out; + std::vector lives; + size_t pos = 0; + while (pos <= named.size()) + { + const size_t nl = named.find('\n', pos); + const String line = named.substr(pos, nl == String::npos ? String::npos : nl - pos); + if (line.find("\"kind\":\"ref_life\"") != String::npos) + { + lives.push_back(line); + } + else + { + if (!lives.empty()) + { + std::sort(lives.begin(), lives.end()); + out.insert(out.end(), lives.begin(), lives.end()); + lives.clear(); + } + out.push_back(line); + } + if (nl == String::npos) + break; + pos = nl + 1; + } + std::sort(lives.begin(), lives.end()); + out.insert(out.end(), lives.begin(), lives.end()); + + String joined; + for (const String & line : out) + { + joined += line; + joined += '\n'; + } + return joined; +} + +struct FoldRun +{ + std::vector seals; /// the fold seal's bytes after each round + std::vector> intake; /// `fold_ref_intake` metrics, per round + std::vector> reduce; /// `fold_reduce` metrics, per round + std::map gets; /// key -> GETs over the whole run + std::map heads; /// key -> HEADs over the whole run + std::vector condemned; + std::vector deleted; + IdNames id_names; /// incarnation hex -> namespace, for the comparison +}; + +void runFolds(uint64_t concurrency, size_t rounds, FoldRun & out) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, + PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_fold_max_defer_rounds = 0, .gc_read_concurrency = concurrency}); + populate(store); + + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const Layout & layout = store->layout(); + + /// Read the id table BEFORE the rounds: a namespace the fold reclaims loses its catalog row, and its + /// keys still have to be nameable when the two runs are compared. + for (const CatalogEntry & entry : CasRefCatalog::read(op, layout).catalog.entries) + out.id_names.emplace(u128ToHex(entry.incarnation), "<" + entry.ns.string() + ">"); + + Gc gc(store, kGc); + gc.setPhaseSink([&](const GcPhaseRecord & rec) + { + if (rec.phase == "fold_ref_intake") + out.intake.push_back(rec.metrics); + else if (rec.phase == "fold_reduce") + out.reduce.push_back(rec.metrics); + }); + + /// Only the rounds' own I/O is compared; the identical population above is not part of the claim. + backend->resetCounts(); + + for (size_t round = 0; round < rounds; ++round) + { + const RoundReport report = gc.runRegularRound(); + ASSERT_TRUE(report.acquired_lease) << "round " << round; + out.condemned.push_back(report.condemned); + out.deleted.push_back(report.deleted); + store->renewWatermarkOnce(); + + const GcState st = decodeGcState(op.read(layout.gcStateKey(), Retry::once())->bytes); + const auto seal = op.read(layout.foldSealKey(st.snap_generation, st.snap_attempt), Retry::once()); + out.seals.push_back(seal ? canonicalSeal(seal->bytes, out.id_names) : String{}); + } + + for (const String & key : backend->touchedKeys()) + { + const String named = normalizeIds(key, out.id_names); + if (const uint64_t n = backend->getCount(key); n != 0) + out.gets[named] += n; + if (const uint64_t n = backend->headCount(key); n != 0) + out.heads[named] += n; + } +} + +} + +TEST(CASGCReadAhead, FoldIsIdenticalAtConcurrencyOneAndEight) +{ + constexpr size_t kRounds = 6; + FoldRun one; + FoldRun eight; + ASSERT_NO_FATAL_FAILURE(runFolds(1, kRounds, one)); + ASSERT_NO_FATAL_FAILURE(runFolds(8, kRounds, eight)); + + ASSERT_EQ(one.seals.size(), kRounds); + ASSERT_EQ(eight.seals.size(), kRounds); + for (size_t i = 0; i < kRounds; ++i) + EXPECT_EQ(one.seals[i], eight.seals[i]) << "fold seal differs at round " << i; + + EXPECT_EQ(one.intake, eight.intake); + EXPECT_EQ(one.reduce, eight.reduce); + EXPECT_EQ(one.condemned, eight.condemned); + EXPECT_EQ(one.deleted, eight.deleted); + + /// THE REQUEST-SET CLAIM. Every namespace here is healthy, so nothing is hinted that the walk + /// does not go on to read: the hints stop at the checkpoint ceiling the walk stops at, a quiet + /// namespace is not hinted at all, and every decoded log's manifest edges are all folded. So the + /// read-ahead must issue the SAME GETs against the SAME keys, not merely produce the same answer. + EXPECT_EQ(one.gets, eight.gets); + + ASSERT_FALSE(one.intake.empty()); + EXPECT_GT(one.intake[0].at("logs_applied"), 32u) + << "the wide namespace must carry more logs than the window, or the lookahead is untested"; +} + +TEST(CASGCReadAhead, ReduceCondemnsTheSameBlobsWithTheSameHeadsAtConcurrencyOneAndEight) +{ + /// `populate` gives every part its own blob and drops whole parts, so a dropped blob loses its only + /// edge and no surviving blob has a removal: the hinted set equals the set `head_blob` takes, and the + /// per-key HEAD counts must match exactly rather than merely producing the same verdict. + constexpr size_t kRounds = 6; + FoldRun one; + FoldRun eight; + ASSERT_NO_FATAL_FAILURE(runFolds(1, kRounds, one)); + ASSERT_NO_FATAL_FAILURE(runFolds(8, kRounds, eight)); + + EXPECT_EQ(one.condemned, eight.condemned); + EXPECT_EQ(one.heads, eight.heads); + + uint64_t condemned_total = 0; + for (const size_t n : one.condemned) + condemned_total += n; + EXPECT_GT(condemned_total, 0u) << "the scenario must condemn, or the reduce read-ahead is untested"; +} + +namespace +{ + +/// Throws once on the first read issued from a thread other than the one that armed it: exactly a +/// read-ahead worker's request, never the round thread's own. +class WorkerReadFaultBackend : public CountingBackend +{ +public: + void armAgainstOtherThreads() + { + owner = std::this_thread::get_id(); + armed.store(true); + } + + bool fired() const { return !armed.load(); } + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + if (armed.load() && std::this_thread::get_id() != owner) + { + armed.store(false); + throw std::runtime_error("injected worker read fault"); + } + return CountingBackend::read(key, access); + } + +private: + std::atomic armed{false}; + std::thread::id owner; +}; + +} + +namespace +{ + +/// Proves OVERLAP, which no equality test can: it releases a read only once `k_overlap` reads are +/// inside the backend at the same time. If the fold's reads were still strictly one after another the +/// count could never reach two, so the round would block until the bounded wait expires and the flag +/// below would stay false. The wait is bounded and the last arrival wakes everyone, so nothing here can +/// hang the suite: a fold with no overlap finishes late, it does not finish never. +class OverlapWitnessBackend : public CountingBackend +{ +public: + explicit OverlapWitnessBackend(size_t k_overlap_) : k_overlap(k_overlap_) {} + + /// ARMED ONLY FOR THE ROUND. Holding reads is fatal to a WRITER: its checkpoint publication is a + /// CAS with a bounded retry budget, and a latch on every read exhausts it long before the round + /// under test ever starts. + void arm() { armed.store(true); } + + bool sawOverlap() const { return saw_overlap.load(); } + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + if (!armed.load()) + return CountingBackend::read(key, access); + { + std::unique_lock lock(mutex); + ++in_flight; + peak = std::max(peak, in_flight); + if (in_flight >= k_overlap) + { + saw_overlap.store(true); + gate.notify_all(); + } + else + { + gate.wait_for(lock, std::chrono::milliseconds(250), + [&] { return in_flight >= k_overlap || saw_overlap.load(); }); + } + } + /// The count stays raised ACROSS the read, so what it measures is requests genuinely in the + /// backend together. Releasing it before the read would leave a window of a few instructions + /// that two threads would have to hit simultaneously to be seen -- which is a measurement of + /// luck, not of overlap. + std::optional raw = CountingBackend::read(key, access); + { + std::lock_guard lock(mutex); + --in_flight; + } + return raw; + } + + size_t peakInFlight() const + { + std::lock_guard lock(mutex); + return peak; + } + +private: + const size_t k_overlap; + std::atomic armed{false}; + mutable std::mutex mutex; + std::condition_variable gate; + size_t in_flight = 0; + size_t peak = 0; + std::atomic saw_overlap{false}; +}; + +} + +TEST(CASGCReadAhead, TheFoldsReadsActuallyOverlap) +{ + auto backend = std::make_shared(/*k_overlap*/ 2); + auto store = Pool::open(backend, + PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_fold_max_defer_rounds = 0, .gc_read_concurrency = 8}); + populate(store); + + Gc gc(store, kGc); + backend->arm(); + ASSERT_TRUE(gc.runRegularRound().acquired_lease); + + EXPECT_TRUE(backend->sawOverlap()) + << "no two of the fold's reads were ever in the backend at the same time; peak in flight was " + << backend->peakInFlight(); + EXPECT_GT(backend->peakInFlight(), 1u); +} + +TEST(CASGCReadAhead, WorkerReadFaultFailsTheRoundAndTheNextRoundRecovers) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, + PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_fold_max_defer_rounds = 0, .gc_read_concurrency = 8}); + populate(store); + + Gc gc(store, kGc); + backend->armAgainstOtherThreads(); + EXPECT_ANY_THROW(gc.runRegularRound()); + EXPECT_TRUE(backend->fired()) << "no read-ahead worker ever issued a request"; + + store->renewWatermarkOnce(); + const RoundReport recovered = gc.runRegularRound(); + EXPECT_TRUE(recovered.acquired_lease); +} diff --git a/src/Disks/tests/gtest_cas_settings.cpp b/src/Disks/tests/gtest_cas_settings.cpp index 1e43c222d71a..faa2076038dd 100644 --- a/src/Disks/tests/gtest_cas_settings.cpp +++ b/src/Disks/tests/gtest_cas_settings.cpp @@ -174,7 +174,16 @@ TEST(CASContentAddressedSettings, InvalidBoundsDiagnosticNamesExternalConfigKeys expectLoadFailureWithExactMessage( "srv10", ErrorCodes::BAD_ARGUMENTS, - "content_addressed disk: cas_gc_interval_sec and cas_gc_shards must be >= 1 (got 60, 0)"); + "content_addressed disk: cas_gc_interval_sec, cas_gc_shards and cas_gc_read_concurrency must be >= 1 " + "(got 60, 0, 16)"); + /// The fold's read-ahead pool is refused at zero for the same reason the shard count is: a zero + /// would be a silently disabled subsystem rather than a configuration the pool can honour. One is + /// the sequential fold and is the way to turn the read-ahead off. + expectLoadFailureWithExactMessage( + "srv10", + ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: cas_gc_interval_sec, cas_gc_shards and cas_gc_read_concurrency must be >= 1 " + "(got 60, 1, 0)"); } TEST(CASContentAddressedSettings, InvalidEnumDiagnosticsNameExternalConfigKeys) From 01043cb1c6238132beeaa1449eefbabe328e798c Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:06:34 +0200 Subject: [PATCH 19/81] cas: a same-pool fetch always relinks onto the pool's disk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fetch-by-relink existed but was opportunistic: the receiver advertised ONE guessed pool identity before it knew where the sender actually kept the part (the caller's `dest_disk` if content-addressed, else the first content-addressed disk of the table's storage policy), then reserved the target disk the ordinary way — the TTL move rule's destination, `balancedReservation`, or the first volume with space — and accepted the relink offer only when the reservation happened to land on a disk of the advertised pool. Otherwise it re-requested the bytes. So a part already in the shared pool moved as bytes whenever the policy's placement disagreed with the guess: a tiered policy whose local volume comes first, a TTL rule naming the local tier for a fresh part, a policy holding two pools with the sender's in second place. The relink is the whole point of a shared pool — a fetch should move no bytes — and the storage policy could veto it by accident. The receiver now advertises every pool of its storage policy (any volume; a disk configured on the server but absent from the policy is not a candidate, since a part on it wouldn't load at startup), the sender names the one it matched, and the part lands on that pool's disk ahead of volume order, JBOD balancing and TTL move rules — the mover carries it to a TTL destination afterwards, the same way `perform_ttl_move_on_insert=0` already places first and moves later. A caller-supplied `dest_disk` (zero-copy `MOVE`) stays authoritative and untouched; a content-addressed disk never enters that path (`supportZeroCopyReplication()` is false for CAS). A read-only or broken disk on the right pool is not a candidate — nothing can publish a ref there. The offered pool must itself be an advertised pool (not matched by disk name), and the confirm's gate 0 compares mounts rather than disk names, closing a second-order gap the first pass left. Non-live pool disk = fail-close: a disk whose mount isn't live is left out of both the advertise and the placement, never guessed at. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../antalya/cas/architecture/replication.md | 40 ++- src/Common/FailPoint.cpp | 4 +- src/Storages/MergeTree/DataPartsExchange.cpp | 174 ++++++++--- src/Storages/MergeTree/DataPartsExchange.h | 4 + .../MergeTree/DataPartsExchangeCasRouting.cpp | 104 +++++++ .../MergeTree/DataPartsExchangeCasRouting.h | 65 ++++ .../tests/gtest_cas_relink_pool_routing.cpp | 158 ++++++++++ .../configs/storage_conf_tiered.xml | 32 ++ .../test_cas_replicated_relink/test.py | 284 ++++++++++++++++++ 9 files changed, 818 insertions(+), 47 deletions(-) create mode 100644 src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp create mode 100644 src/Storages/MergeTree/DataPartsExchangeCasRouting.h create mode 100644 src/Storages/tests/gtest_cas_relink_pool_routing.cpp create mode 100644 tests/integration/test_cas_replicated_relink/configs/storage_conf_tiered.xml diff --git a/docs/en/antalya/cas/architecture/replication.md b/docs/en/antalya/cas/architecture/replication.md index 8e8981c404ae..a9f3221badf7 100644 --- a/docs/en/antalya/cas/architecture/replication.md +++ b/docs/en/antalya/cas/architecture/replication.md @@ -31,11 +31,11 @@ sequenceDiagram participant Snd as Sender participant S3 as Shared pool - R->>Snd: GET part, cas_pool_uuid = R's pool uuid, client_protocol_version = 11 + R->>Snd: GET part, cas_pool_uuid = every pool of R's policy, client_protocol_version = 11 Note over R: advertising 11 is a promise to confirm before promoting Snd->>Snd: same disk pool uuid? identity, never endpoint plus prefix Snd->>S3: resolve the offer once -- manifest bytes and confirm token from the SAME view - Snd-->>R: cookie cas_relink = part_manifest_v2, cookie cas_source_token = ..., body = manifest bytes + Snd-->>R: cookies cas_relink = part_manifest_v2, cas_source_token = ..., cas_pool_uuid = the matched pool -- body = manifest bytes Note over Snd: sender is fire-and-forget -- it releases the part here rect rgba(120,160,255,0.12) @@ -63,7 +63,7 @@ sequenceDiagram | # | Gate | What it enforces | |---|---|---| -| 1 | Pool identity | The receiver advertises `cas_pool_uuid`; the sender offers relink only if its own disk's pool uuid is **equal**. Matching by endpoint and prefix was tried and rejected — a minted pool uuid is the identity | +| 1 | Pool identity | The receiver advertises `cas_pool_uuid` — the pool uuids of every content-addressed disk of its storage policy that is not read-only, as one list — and the sender offers relink only if its own disk's pool uuid is **in** it, naming that uuid in a `cas_pool_uuid` response cookie. Matching by endpoint and prefix was tried and rejected — a minted pool uuid is the identity | | 2 | Protocol version 11 | On the receiver side, advertising it is a promise to run the confirm round trip before promoting | | 3 | One resolution for two outputs | The manifest bytes and the confirm token come from the **same** view. Two separate calls would allow a repoint in between and hand the receiver a token naming a manifest whose entries it never adopted | | 4 | The receiver trusts nothing from the wire but the entry list | The sender's manifest id, namespace and payload digest are ignored; the target namespace and ref come from the receiver's own router, and manifest path hygiene is validated at decode | @@ -77,6 +77,40 @@ cannot be entered twice for one fetch. Byte-fetched files content-address and de anyway, so falling back never loses the dedup property, only the zero-byte-move property for that one fetch. +## Where a relinked part lands {#relink-placement} + +The offer decides the disk. Once the sender has named the pool, the receiver places the part on the +first disk of its storage policy that belongs to that pool, and reserves space there directly — ahead +of everything the policy would otherwise consult: volume order, JBOD balancing, +`max_data_part_size_bytes`, and `TTL ... TO DISK|VOLUME` move rules. A part that is already in the +pool never travels as bytes merely because the policy would have put it somewhere else. + +A TTL rule is not ignored, it is deferred: the background mover sees a part that is not in its TTL +destination and moves it there afterwards. The bytes then travel once, as a read from the pool on the +receiver, and the sender is never loaded. + +Two things do not bend to the offer. A disk the caller supplied (zero-copy `MOVE` re-fetching a shared +part onto the move's destination) is never overridden — a content-addressed disk cannot reach that path +at all, since it does not support zero-copy replication. And a read-only disk is never a candidate: its +pool is advertised only if some other disk of that pool in the policy is writable, and when none is, the +sender streams bytes and the ordinary placement applies. + +A pool disk that is not live — its mount lease lost, its identity lost, or the storage shut down — is +still the target. The relink's own write gate refuses it and the fetch fails; a replication-queue fetch +is retried by the queue, while a manual `FETCH PART` or `FETCH PARTITION` reports the error to the user. +The part is never quietly placed on another disk instead. This is the behaviour a single-disk +content-addressed policy always had, and a mixed policy now shares it. + +The byte-fetch fallback after a relink that failed for a mechanism reason (a corrupted manifest, a +body-absent precommit, a ref conflict) re-requests the bytes on the same pool disk, where they +content-address and deduplicate against the pool — the placement outlives the relink. A manifest of a +newer format generation is not degraded to bytes today (a tracked gap, `[relink-fallback-unknown-format-version]` +in the backlog). + +During a rolling upgrade a sender that predates the pool-set advertise compares the whole `cas_pool_uuid` +value with its own pool id, so a receiver whose policy holds several pools gets bytes from such a sender +until it is upgraded; a receiver with one pool is unaffected, its advertise is byte-for-byte the old one. + ## What actually seals "commit before release" {#relink-seal} The receiver's `+1` — its precommit binding — is durable **before** the sender is asked anything, diff --git a/src/Common/FailPoint.cpp b/src/Common/FailPoint.cpp index 12dbc03ee70a..6d42340eff42 100644 --- a/src/Common/FailPoint.cpp +++ b/src/Common/FailPoint.cpp @@ -233,7 +233,9 @@ static struct InitFiu REGULAR(tcp_handler_fail_connection_setup) \ REGULAR(distributed_plan_status_check_reenqueue_fault) \ REGULAR(cas_relink_receiver_force_mechanism_failure) \ - PAUSEABLE_ONCE(cas_relink_receiver_pause_before_confirm) + PAUSEABLE_ONCE(cas_relink_receiver_pause_before_confirm) \ + REGULAR(cas_relink_sender_omit_pool_cookie) \ + REGULAR(cas_relink_receiver_drop_forced_disk) namespace FailPoints { diff --git a/src/Storages/MergeTree/DataPartsExchange.cpp b/src/Storages/MergeTree/DataPartsExchange.cpp index b2f5e1d12b3e..487e3654216a 100644 --- a/src/Storages/MergeTree/DataPartsExchange.cpp +++ b/src/Storages/MergeTree/DataPartsExchange.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -54,6 +55,12 @@ namespace FailPoints /// neither is reachable from configuration, so an integration test cannot produce them any other way. extern const char cas_relink_receiver_force_mechanism_failure[]; extern const char cas_relink_receiver_pause_before_confirm[]; + /// Stands in for a sender that predates the `cas_pool_uuid` response cookie: the offer is made + /// without naming the pool, and the receiver has to fall back on "the single advertised pool". + extern const char cas_relink_sender_omit_pool_cookie[]; + /// Stands in for an offer this policy has no disk for: the receiver forgets the forced disk it + /// resolved and must take the ordinary placement and a byte fetch. + extern const char cas_relink_receiver_drop_forced_disk[]; } namespace MergeTreeSetting @@ -112,8 +119,10 @@ std::string getEndpointId(const std::string & node_id) return "DataPartsExchange:" + node_id; } -/// CAS replication 2b. The receiver advertises its target pool's identity under this request param so -/// the sender can decide whether a fetch-by-relink (same pool) is possible. +/// CAS replication 2b. The receiver advertises the pool ids of its candidate content-addressed disks +/// under this request param (one id, or several joined with ", ", see `encodeCasPoolAdvertise`) so the +/// sender can decide whether a fetch-by-relink (same pool) is possible. On the offer the same name is a +/// response cookie naming the pool the sender matched. constexpr auto CA_POOL_UUID_PARAM = "cas_pool_uuid"; /// Set on the response when the sender chose the relink path; the receiver then reads the relink payload /// (the opaque encoded PartManifest body — self-contained, see part_manifest_v2 below) instead of the @@ -227,21 +236,22 @@ CasConfirmAnswer Service::resolveContentAddressedConfirm( /// Routing. A pool UUID identifies the shared pool, not the mount: every server root writing into it /// reports the same one, so the namespace's owner decides which instance may answer. EXACTLY one /// match is required — zero means this table has no such disk, several mean the question is - /// ambiguous, and both are `Unknown` rather than a guess. - const IContentAddressedExchange * matched = nullptr; - DiskPtr matched_disk; + /// ambiguous, and both are `Unknown` rather than a guess. A cache disk over a content-addressed disk + /// shares the base disk's exchange object, so the two are one mount and count once. + std::vector routing; + Disks routing_disks; for (const auto & disk : data.getDisks()) { const auto * ca_meta = tryGetContentAddressedExchange(disk); - if (!ca_meta || ca_meta->getPoolUUID() != pool_uuid || !ca_meta->ownsNamespace(server_root_id, root_namespace)) + if (!ca_meta) continue; - if (matched) - return CasConfirmAnswer::Unknown; - matched = ca_meta; - matched_disk = disk; + routing.push_back({ca_meta, ca_meta->getPoolUUID(), ca_meta->ownsNamespace(server_root_id, root_namespace)}); + routing_disks.push_back(disk); } - if (!matched) + const auto routed = resolveConfirmRoutingCandidate(routing, pool_uuid); + if (!routed) return CasConfirmAnswer::Unknown; + const IContentAddressedExchange * matched = tryGetContentAddressedExchange(routing_disks[*routed]); /// Gate 0 — the part-anchored fast filter. It is an AVAILABILITY filter and never a proof (spec /// §confirm-primitive, demoted in rev.5): `rollbackDeletingParts` puts a part back to `Outdated` @@ -250,8 +260,8 @@ CasConfirmAnswer Service::resolveContentAddressedConfirm( /// a cheap `No` that costs no ledger work; every `Yes` is earned by gate 1 alone. /// /// `Deleting` is excluded by the state filter, an unknown name yields no part at all, and a part of - /// this name living on ANOTHER disk is rejected explicitly — `MOVE ... TO DISK` leaves a same-name - /// `Active` part behind on the destination disk, and only the instance the token routed to may be + /// this name living on ANOTHER mount is rejected explicitly — `MOVE ... TO DISK` leaves a same-name + /// `Active` part behind on the destination disk, and only the mount the token routed to may be /// the one the confirm is about. The parts set is read under its own lock, which /// `getPartIfExists` takes and releases, and the part reference is dropped before any ledger lock. { @@ -260,7 +270,14 @@ CasConfirmAnswer Service::resolveContentAddressedConfirm( return CasConfirmAnswer::Unknown; const auto part = data.getPartIfExists( *part_info, {MergeTreeDataPartState::Active, MergeTreeDataPartState::Outdated}); - if (!part || part->getDataPartStorage().getDiskName() != matched_disk->getName()) + if (!part) + return CasConfirmAnswer::No; + /// Compared by mount, not by disk name: a base disk and its cache wrapper are two names for one + /// exchange object, and a part living on either of them is a part of the mount the token routed + /// to. A different mount (a distinct exchange object) is the "another disk" this gate rejects. + const auto * part_exchange = tryGetContentAddressedExchange( + data.getStoragePolicy()->tryGetDiskByName(part->getDataPartStorage().getDiskName())); + if (part_exchange != matched) return CasConfirmAnswer::No; } @@ -389,8 +406,8 @@ void Service::processQuery(const HTMLForm & params, ReadBufferPtr body, WriteBuf } /// CAS replication 2b — fetch-by-relink (spec §4). If the part is on a content-addressed disk and - /// the receiver advertised a `cas_pool_uuid` equal to THIS server's own pool_uuid - /// (same shared pool), send only the part's content id + the mutable header — no file bytes — so + /// the pool of the disk this part sits on is among the pools the receiver advertised in + /// `cas_pool_uuid`, send only the part's content id + the mutable header — no file bytes — so /// the receiver can "fetch" by publishing its own ref to the blobs already in the shared pool. /// Strictly gated on a matching pool_uuid: a non-CA part, a CA part on a different pool, or a /// receiver without the capability all fall through to the unchanged byte path below. @@ -404,21 +421,34 @@ void Service::processQuery(const HTMLForm & params, ReadBufferPtr body, WriteBuf if (client_protocol_version >= REPLICATION_PROTOCOL_VERSION_WITH_CA_CONFIRM && part->getDataPartStorage().isContentAddressed()) { - const String receiver_pool_uuid = parse(params.get(CA_POOL_UUID_PARAM, "")); + /// The receiver advertises every pool its storage policy has a writable content-addressed + /// disk for, as one list; this server's decision stays local — is the pool of the disk THIS + /// part sits on among them. The matched pool goes back as a cookie so a receiver with several + /// pools can place the part on that pool's disk instead of guessing which disk the offer is for. + const Strings receiver_pools = decodeCasPoolAdvertise(parse(params.get(CA_POOL_UUID_PARAM, ""))); DiskPtr part_disk = data.getStoragePolicy()->tryGetDiskByName(part->getDataPartStorage().getDiskName()); auto * ca_meta = tryGetContentAddressedExchange(part_disk); - if (ca_meta && !receiver_pool_uuid.empty() && receiver_pool_uuid == ca_meta->getPoolUUID()) + const String matched_pool = ca_meta ? ca_meta->getPoolUUID() : String{}; + if (ca_meta && !matched_pool.empty() + && std::find(receiver_pools.begin(), receiver_pools.end(), matched_pool) != receiver_pools.end()) { auto offer = ca_meta->getRelinkOffer(part->getDataPartStorage().getRelativePath()); if (offer) { LOG_DEBUG(log, "Sending part {} by relink (content-addressed, shared pool {}), manifest payload {} bytes", - part_name, receiver_pool_uuid, offer->manifest_bytes.size()); + part_name, matched_pool, offer->manifest_bytes.size()); response.addCookie({CA_RELINK_COOKIE, CA_RELINK_COOKIE_VALUE}); /// The source token for the confirm request the receiver makes before it promotes /// (spec §wire-protocol). It always accompanies the offer, and its ABSENCE is what /// tells a confirm-capable receiver that this sender predates the handshake. response.addCookie({CA_CONFIRM_TOKEN_COOKIE, offer->confirm_token}); + /// Which of the advertised pools this offer is for. A receiver with one pool does not + /// need it (an offer can only be for that pool); the failpoint stands in for a sender + /// that predates the cookie. + bool omit_pool_cookie = false; + fiu_do_on(FailPoints::cas_relink_sender_omit_pool_cookie, { omit_pool_cookie = true; }); + if (!omit_pool_cookie) + response.addCookie({CA_POOL_UUID_PARAM, matched_pool}); /// The relink payload (B7 part_manifest_v2, all-tree task 7): the opaque encoded /// PartManifest body (the receiver decodes it, ignores the sender identity, and /// stages its OWN local manifest over the shared-pool blobs; the legacy part_id wire @@ -690,11 +720,17 @@ std::pair Fetcher::fetchSelected if (disk) LOG_TRACE(log, "Will fetch to disk {} with type {}", disk->getName(), disk->getDataSourceDescription().toString()); - /// CAS replication 2b — fetch-by-relink (spec §4). Advertise this replica's target content-addressed - /// pool identity so a same-pool sender can relink instead of streaming bytes. The target disk is the - /// provided one if it is CA, else the first CA disk among the table's disks. A non-CA fetch adds - /// nothing here and is byte-for-byte unchanged. - /// Gated on `allow_ca_relink` alone (B66b). That flag is the RECURSION BRAKE and nothing else: not + /// CAS fetch-by-relink: advertise the content-addressed pools this fetch may land in, so a sender + /// holding the part in one of them relinks instead of streaming bytes. With a caller-supplied disk + /// that is its pool alone (the disk is the caller's contract and is never overridden); otherwise it + /// is every content-addressed disk of the table's storage policy that is not read-only, in policy + /// order. The sender names the pool it matched in a response cookie, and the reservation below then + /// goes to THAT pool's disk — ahead of the policy's volume order and of any TTL move rule, because a + /// part that is already in the pool must never travel as bytes merely because the policy would have + /// put it elsewhere (the mover carries it to a TTL destination afterwards). A pool disk that is not + /// live is still the target: the relink's own write gate refuses it, the fetch fails and the queue + /// retries — never a quiet landing on another disk. A non-CA fetch adds nothing here. + /// Gated on `allow_ca_relink` alone. That flag is the RECURSION BRAKE and nothing else: not /// advertising is what makes the sender stream bytes, so every same-sender byte re-request below /// clears it, and a persistent relink-mechanism failure therefore costs exactly one relink attempt. /// The gate used to be `try_zero_copy && !to_detached`, and BOTH halves were accidents of that same @@ -702,26 +738,34 @@ std::pair Fetcher::fetchSelected /// because the relink path staged at the ACTIVE part path and ignored `to_detached`. `to_detached` /// is now a parameter of `relinkPartToDisk` (it stages under the `detached/` parent), and /// `try_zero_copy` goes back to meaning real zero-copy only. - String advertised_pool_uuid; + Strings advertised_pools; + std::vector ca_candidates; + Disks ca_candidate_disks; if (allow_ca_relink) { - if (auto * ca_meta = tryGetContentAddressedExchange(disk)) + if (disk) { - advertised_pool_uuid = ca_meta->getPoolUUID(); - uri.addQueryParameter(CA_POOL_UUID_PARAM, advertised_pool_uuid); + if (auto * ca_meta = tryGetContentAddressedExchange(disk)) + advertised_pools.push_back(ca_meta->getPoolUUID()); } - else if (!disk) + else { for (const auto & data_disk : data.getDisks()) { - if (auto * ca_disk_meta = tryGetContentAddressedExchange(data_disk)) - { - advertised_pool_uuid = ca_disk_meta->getPoolUUID(); - uri.addQueryParameter(CA_POOL_UUID_PARAM, advertised_pool_uuid); - break; - } + auto * ca_disk_meta = tryGetContentAddressedExchange(data_disk); + if (!ca_disk_meta) + continue; + ca_candidates.push_back({data_disk->getName(), ca_disk_meta->getPoolUUID(), data_disk->isReadOnly()}); + ca_candidate_disks.push_back(data_disk); + if (!data_disk->isReadOnly()) + advertised_pools.push_back(ca_disk_meta->getPoolUUID()); } } + const String advertise = encodeCasPoolAdvertise(advertised_pools); + if (!advertise.empty()) + uri.addQueryParameter(CA_POOL_UUID_PARAM, advertise); + /// The deduplicated form is what "the single advertised pool" is measured against below. + advertised_pools = decodeCasPoolAdvertise(advertise); } Strings capability; @@ -786,6 +830,36 @@ std::pair Fetcher::fetchSelected int server_protocol_version = parse(in->getResponseCookie("server_protocol_version", "0")); String remote_fs_metadata = parse(in->getResponseCookie("remote_fs_metadata", "")); + /// The relink offer, if any, is already visible: response cookies arrive with the headers, before any + /// body field is consumed. Resolve the forced disk NOW, so the reservation below goes to it and the + /// body reads keep their order. `offered_pool` is what the relink block later checks the chosen disk + /// against; with a caller-supplied disk there is nothing to force and that check is all there is. + const String ca_relink = parse(in->getResponseCookie(CA_RELINK_COOKIE, "")); + String offered_pool; + DiskPtr forced_ca_disk; + if (!ca_relink.empty()) + { + const String offered_pool_cookie = parse(in->getResponseCookie(CA_POOL_UUID_PARAM, "")); + offered_pool = resolveOfferedCasPool(advertised_pools, offered_pool_cookie); + if (!disk) + { + auto chosen = resolveForcedCaCandidate(ca_candidates, advertised_pools, offered_pool_cookie); + fiu_do_on(FailPoints::cas_relink_receiver_drop_forced_disk, + { + LOG_INFO(log, "Failpoint cas_relink_receiver_drop_forced_disk: forgetting the forced disk for part {}", part_name); + chosen.reset(); + }); + if (chosen) + { + forced_ca_disk = ca_candidate_disks[*chosen]; + LOG_DEBUG(log, "Part {} is offered by relink for content-addressed pool {}; placing it on disk {} " + "ahead of the storage policy's volume order and TTL rules", part_name, offered_pool, ca_candidates[*chosen].disk_name); + /// From here on the target is decided: every `!disk` reservation branch below is skipped. + disk = forced_ca_disk; + } + } + } + DiskPtr preffered_disk = disk; if (!preffered_disk) @@ -806,6 +880,13 @@ std::pair Fetcher::fetchSelected { readBinary(sum_files_size, *in); + if (forced_ca_disk) + { + /// An object-storage disk reports no capacity, so this cannot decline for space; if it ever + /// does, the loud NOT_ENOUGH_SPACE is the right outcome — the part is not re-placed elsewhere. + reservation = MergeTreeData::reserveSpace(sum_files_size, forced_ca_disk); + } + if (server_protocol_version >= REPLICATION_PROTOCOL_VERSION_WITH_PARTS_SIZE_AND_TTL_INFOS) { IMergeTreeDataPart::TTLInfos ttl_infos; @@ -887,7 +968,6 @@ std::pair Fetcher::fetchSelected /// reconstruct. If the relink is not possible (blob missing/condemned — a transient or a /// genuinely-different pool the cheap pre-filter let through, or a mixed-build pair offering an /// unrecognized cookie value), fall back to a normal byte fetch by re-requesting WITHOUT relink. - String ca_relink = parse(in->getResponseCookie(CA_RELINK_COOKIE, "")); if (!ca_relink.empty()) { /// Re-request without the relink capability: pass the SAME (CA) disk but disable zero-copy/relink @@ -902,9 +982,9 @@ std::pair Fetcher::fetchSelected /// and recurses without bound. The failures it actually bounds are the ones that leave the CA /// disk resolved and matching: a mixed build offering an unrecognized cookie value, a sender that /// predates the confirm handshake, an undecodable manifest, a local ref conflict. (The - /// reservation-outside-the-pool exit below is bounded twice over — it re-requests with the - /// non-CA disk it resolved, which cannot advertise anything either way — so do not read that one - /// as evidence that the brake is redundant.) + /// no-disk-takes-it exit below is bounded twice over — it re-requests with the disk the ordinary + /// reservation resolved, which is outside the pool and cannot advertise it — so do not read that + /// one as evidence that the brake is redundant.) auto fall_back_to_byte_fetch = [&] { temporary_directory_lock = {}; @@ -924,12 +1004,20 @@ std::pair Fetcher::fetchSelected return fall_back_to_byte_fetch(); } + /// The disk is the forced one, so this holds by construction; it stays a real exit rather than + /// an assertion because it is also how an offer for a pool this policy has no disk for (no forced + /// disk, ordinary reservation) and a caller-supplied disk outside the pool leave the relink path. auto * chosen_ca = tryGetContentAddressedExchange(disk); - if (!chosen_ca || chosen_ca->getPoolUUID() != advertised_pool_uuid) + if (!chosen_ca || offered_pool.empty() || chosen_ca->getPoolUUID() != offered_pool) { - LOG_INFO(log, "Part {} was offered by relink for content-addressed pool '{}', but reservation landed " - "outside the advertised pool on disk {} (chosen pool: '{}'); falling back to a byte fetch", - part_name, advertised_pool_uuid, disk->getName(), chosen_ca ? chosen_ca->getPoolUUID() : ""); + if (offered_pool.empty()) + LOG_INFO(log, "Part {} was offered by relink, but the offer does not name one of the {} advertised " + "content-addressed pool(s) (cookie '{}'); falling back to a byte fetch onto disk {}", + part_name, advertised_pools.size(), parse(in->getResponseCookie(CA_POOL_UUID_PARAM, "")), disk->getName()); + else + LOG_INFO(log, "Part {} was offered by relink for content-addressed pool '{}', but no disk of this table's " + "storage policy takes it (chosen disk {}, pool '{}'); falling back to a byte fetch", + part_name, offered_pool, disk->getName(), chosen_ca ? chosen_ca->getPoolUUID() : ""); return fall_back_to_byte_fetch(); } diff --git a/src/Storages/MergeTree/DataPartsExchange.h b/src/Storages/MergeTree/DataPartsExchange.h index 5bec506eac21..acae2e503ae0 100644 --- a/src/Storages/MergeTree/DataPartsExchange.h +++ b/src/Storages/MergeTree/DataPartsExchange.h @@ -113,6 +113,10 @@ class Fetcher final : private boost::noncopyable const String & tmp_prefix_ = "", std::optional * tagger_ptr = nullptr, bool try_zero_copy = true, + /// The target disk when the CALLER has already decided it (zero-copy `MOVE` re-fetching a shared + /// part onto the move's destination); never overridden. When absent, a content-addressed relink + /// offer decides the disk — the policy disk on the sender's pool — ahead of the storage policy's + /// own placement; otherwise the ordinary reservation does. DiskPtr dest_disk = nullptr, /// CAS fetch-by-relink (spec §B66b): may this request advertise its content-addressed pool /// identity, i.e. may the sender answer with a relink offer instead of the part's bytes? diff --git a/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp b/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp new file mode 100644 index 000000000000..a3227e9de6da --- /dev/null +++ b/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp @@ -0,0 +1,104 @@ +#include + +#include + +#include + +#include + +namespace DB::DataPartsExchange +{ + +namespace +{ +const String CAS_POOL_ADVERTISE_DELIMITER = ", "; +} + +String encodeCasPoolAdvertise(Strings pool_uuids) +{ + std::erase_if(pool_uuids, [](const String & id) { return id.empty(); }); + ::sort(pool_uuids.begin(), pool_uuids.end()); + pool_uuids.erase(std::unique(pool_uuids.begin(), pool_uuids.end()), pool_uuids.end()); + return boost::algorithm::join(pool_uuids, CAS_POOL_ADVERTISE_DELIMITER); +} + +Strings decodeCasPoolAdvertise(const String & text) +{ + Strings pools; + if (text.empty()) + return pools; + + size_t pos_start = 0; + while (true) + { + const size_t pos_end = text.find(CAS_POOL_ADVERTISE_DELIMITER, pos_start); + if (pos_end == String::npos) + { + pools.push_back(text.substr(pos_start)); + return pools; + } + pools.push_back(text.substr(pos_start, pos_end - pos_start)); + pos_start = pos_end + CAS_POOL_ADVERTISE_DELIMITER.size(); + } +} + +String resolveOfferedCasPool(const Strings & advertised_pools, const String & offered_pool_cookie) +{ + if (!offered_pool_cookie.empty()) + { + /// A cookie naming a pool this receiver did not advertise is not an answer to its question. In + /// particular the byte re-request after a failed relink advertises NOTHING, and a peer that + /// offers a relink anyway must not be able to re-enter the relink path through the cookie. + if (std::find(advertised_pools.begin(), advertised_pools.end(), offered_pool_cookie) != advertised_pools.end()) + return offered_pool_cookie; + return {}; + } + if (advertised_pools.size() == 1) + return advertised_pools.front(); + return {}; +} + +std::optional resolveForcedCaCandidate( + const std::vector & candidates, + const Strings & advertised_pools, + const String & offered_pool_cookie) +{ + const String offered_pool = resolveOfferedCasPool(advertised_pools, offered_pool_cookie); + if (offered_pool.empty()) + return std::nullopt; + + for (size_t i = 0; i < candidates.size(); ++i) + { + const auto & candidate = candidates[i]; + if (!candidate.read_only && !candidate.pool_uuid.empty() && candidate.pool_uuid == offered_pool) + return i; + } + return std::nullopt; +} + +std::optional resolveConfirmRoutingCandidate( + const std::vector & candidates, + const String & pool_uuid) +{ + if (pool_uuid.empty()) + return std::nullopt; + + std::optional matched; + for (size_t i = 0; i < candidates.size(); ++i) + { + const auto & candidate = candidates[i]; + if (candidate.pool_uuid != pool_uuid || !candidate.owns_namespace) + continue; + if (!matched) + { + matched = i; + continue; + } + /// A second DISTINCT mount owning the namespace: ambiguous. An alias of the first is not. + if (candidates[*matched].exchange_identity != candidate.exchange_identity) + return std::nullopt; + } + return matched; +} + +} diff --git a/src/Storages/MergeTree/DataPartsExchangeCasRouting.h b/src/Storages/MergeTree/DataPartsExchangeCasRouting.h new file mode 100644 index 000000000000..c5e640271f98 --- /dev/null +++ b/src/Storages/MergeTree/DataPartsExchangeCasRouting.h @@ -0,0 +1,65 @@ +#pragma once + +#include + +#include +#include + +namespace DB::DataPartsExchange +{ + +/// The receiver's content-addressed pool advertise as it goes on the wire (the `cas_pool_uuid` request +/// parameter): the pool ids of every disk of its storage policy that could take a relink — sorted, +/// deduplicated, joined with ", ". The list form and the ", " delimiter are the ones the zero-copy +/// `remote_fs_metadata` capability list already uses, so the exchange keeps one list convention (the +/// decoder differs in one respect: an empty string is no pool at all, never one empty id). A single id +/// is written verbatim: a receiver with one pool puts on the wire exactly the string that a sender +/// comparing the whole value with its own pool id matches. Empty ids are dropped (a storage that never +/// started has no pool id and nothing to advertise). +String encodeCasPoolAdvertise(Strings pool_uuids); +Strings decodeCasPoolAdvertise(const String & text); + +/// Which pool a relink offer is for. The sender names it in the `cas_pool_uuid` response cookie, and +/// the answer is that cookie ONLY if it is one of the pools this receiver advertised — the advertise is +/// the receiver's question, and a byte re-request after a failed relink advertises nothing, so a peer +/// offering regardless can never select a disk. A sender that predates the cookie can only have matched +/// a one-element advertise, so an absent cookie means that single pool. Several advertised pools and no +/// cookie is not a state an honest sender can produce, and the answer is "no pool" — the receiver never +/// guesses. +String resolveOfferedCasPool(const Strings & advertised_pools, const String & offered_pool_cookie); + +/// One content-addressed disk of the RECEIVING table's storage policy, in policy order. +struct CasRelinkCandidate +{ + String disk_name; + String pool_uuid; /// empty: the storage never started; never a candidate + bool read_only = false; /// a static property of the disk's configuration; the one exclusion +}; + +/// Which candidate receives the offered relink: the index of the first candidate on the offered pool +/// (`resolveOfferedCasPool`) that is not read-only. `nullopt` means no disk of this policy may take +/// the offer, which the caller turns into a byte fetch. Whether the pool is LIVE is deliberately not +/// part of this decision — a not-live pool disk is still the target, the relink's own write gate +/// refuses it, and the fetch fails and is retried rather than landing on another disk. +std::optional resolveForcedCaCandidate( + const std::vector & candidates, + const Strings & advertised_pools, + const String & offered_pool_cookie); + +/// One content-addressed disk of the SENDING table's storage policy, as the confirm routing sees it. +struct CasConfirmRoutingCandidate +{ + const void * exchange_identity = nullptr; /// the `IContentAddressedExchange` behind the disk + String pool_uuid; + bool owns_namespace = false; +}; + +/// Which candidate answers a relink confirm for `pool_uuid`: EXACTLY one distinct mount that owns the +/// namespace, else `nullopt` — zero owners, or two distinct mounts, are both ambiguous and `Unknown` +/// is the only honest answer. Disks that alias one mount (a base disk and its cache wrapper share the +/// exchange object) count once, as the first of them. +std::optional resolveConfirmRoutingCandidate( + const std::vector & candidates, + const String & pool_uuid); + +} diff --git a/src/Storages/tests/gtest_cas_relink_pool_routing.cpp b/src/Storages/tests/gtest_cas_relink_pool_routing.cpp new file mode 100644 index 000000000000..2abd09738466 --- /dev/null +++ b/src/Storages/tests/gtest_cas_relink_pool_routing.cpp @@ -0,0 +1,158 @@ +#include +#include + +using namespace DB::DataPartsExchange; +using DB::Strings; + +/// ---- the advertise: sort, unique, ", " ---- + +TEST(CASRelinkPoolAdvertise, EmptySetIsEmptyStringBothWays) +{ + EXPECT_EQ(encodeCasPoolAdvertise({}), ""); + EXPECT_TRUE(decodeCasPoolAdvertise("").empty()); +} + +TEST(CASRelinkPoolAdvertise, SingleIdIsVerbatim) +{ + /// The one-element wire form must be byte-for-byte the pre-set-advertise form: a sender that compares + /// the whole parameter with its own pool id must still match. + EXPECT_EQ(encodeCasPoolAdvertise({"0123abcd"}), "0123abcd"); + EXPECT_EQ(decodeCasPoolAdvertise("0123abcd"), Strings{"0123abcd"}); +} + +TEST(CASRelinkPoolAdvertise, SortsAndDeduplicates) +{ + EXPECT_EQ(encodeCasPoolAdvertise({"bb", "aa", "bb", "aa"}), "aa, bb"); + EXPECT_EQ(decodeCasPoolAdvertise("aa, bb"), (Strings{"aa", "bb"})); +} + +TEST(CASRelinkPoolAdvertise, DropsEmptyIds) +{ + EXPECT_EQ(encodeCasPoolAdvertise({"", "aa", ""}), "aa"); + EXPECT_EQ(encodeCasPoolAdvertise({""}), ""); +} + +TEST(CASRelinkPoolAdvertise, RoundTripsThreeIds) +{ + const Strings ids{"cc", "aa", "bb"}; + EXPECT_EQ(decodeCasPoolAdvertise(encodeCasPoolAdvertise(ids)), (Strings{"aa", "bb", "cc"})); +} + +/// ---- which pool the offer is for ---- + +TEST(CASRelinkPoolAdvertise, OfferedPoolIsTheCookieWhenItWasAdvertised) +{ + EXPECT_EQ(resolveOfferedCasPool({"aa", "bb"}, "bb"), "bb"); + EXPECT_EQ(resolveOfferedCasPool({"aa"}, "aa"), "aa"); +} + +TEST(CASRelinkPoolAdvertise, UnadvertisedCookieIsNoPool) +{ + /// The byte re-request after a failed relink advertises nothing; an offer that arrives anyway must + /// not re-enter the relink path through its cookie. + EXPECT_EQ(resolveOfferedCasPool({"aa"}, "zz"), ""); + EXPECT_EQ(resolveOfferedCasPool({}, "aa"), ""); +} + +TEST(CASRelinkPoolAdvertise, AbsentCookieMeansTheSingleAdvertisedPool) +{ + EXPECT_EQ(resolveOfferedCasPool({"aa"}, ""), "aa"); +} + +TEST(CASRelinkPoolAdvertise, AbsentCookieWithSeveralPoolsIsNoPool) +{ + EXPECT_EQ(resolveOfferedCasPool({"aa", "bb"}, ""), ""); + EXPECT_EQ(resolveOfferedCasPool({}, ""), ""); +} + +/// ---- the receiver's forced candidate ---- + +static std::vector twoPools() +{ + return { + {"disk_other", "other", false}, + {"disk_shared", "shared", false}, + }; +} + +TEST(CASRelinkPoolAdvertise, CookieSelectsTheCandidateOnThatPool) +{ + EXPECT_EQ(resolveForcedCaCandidate(twoPools(), {"other", "shared"}, "shared"), std::optional{1}); + EXPECT_EQ(resolveForcedCaCandidate(twoPools(), {"other", "shared"}, "other"), std::optional{0}); +} + +TEST(CASRelinkPoolAdvertise, AbsentCookieWithOneAdvertisedPoolSelectsIt) +{ + const std::vector one{{"disk_shared", "shared", false}}; + EXPECT_EQ(resolveForcedCaCandidate(one, {"shared"}, ""), std::optional{0}); +} + +TEST(CASRelinkPoolAdvertise, AbsentCookieWithTwoAdvertisedPoolsSelectsNothing) +{ + EXPECT_EQ(resolveForcedCaCandidate(twoPools(), {"other", "shared"}, ""), std::nullopt); +} + +TEST(CASRelinkPoolAdvertise, UnknownPoolSelectsNothing) +{ + EXPECT_EQ(resolveForcedCaCandidate(twoPools(), {"other", "shared"}, "zz"), std::nullopt); +} + +TEST(CASRelinkPoolAdvertise, ReadOnlyCandidateIsSkipped) +{ + const std::vector ro{{"disk_shared_ro", "shared", true}}; + EXPECT_EQ(resolveForcedCaCandidate(ro, {"shared"}, "shared"), std::nullopt); + + const std::vector ro_then_rw{{"disk_shared_ro", "shared", true}, {"disk_shared", "shared", false}}; + EXPECT_EQ(resolveForcedCaCandidate(ro_then_rw, {"shared"}, "shared"), std::optional{1}); +} + +TEST(CASRelinkPoolAdvertise, EmptyPoolIdNeverMatches) +{ + const std::vector not_started{{"disk_cold", "", false}}; + EXPECT_EQ(resolveForcedCaCandidate(not_started, {}, ""), std::nullopt); + EXPECT_EQ(resolveForcedCaCandidate(not_started, {""}, ""), std::nullopt); +} + +TEST(CASRelinkPoolAdvertise, TwoCandidatesOnOnePoolTakeTheFirst) +{ + const std::vector two{{"disk_a", "shared", false}, {"disk_b", "shared", false}}; + EXPECT_EQ(resolveForcedCaCandidate(two, {"shared"}, "shared"), std::optional{0}); +} + +/// ---- the sender's confirm routing ---- + +static const void * const MOUNT_A = reinterpret_cast(0x10); +static const void * const MOUNT_B = reinterpret_cast(0x20); + +TEST(CASRelinkConfirmRouting, OneOwnerAnswers) +{ + const std::vector c{{MOUNT_A, "shared", true}}; + EXPECT_EQ(resolveConfirmRoutingCandidate(c, "shared"), std::optional{0}); +} + +TEST(CASRelinkConfirmRouting, NoOwnerIsNoAnswer) +{ + const std::vector c{{MOUNT_A, "shared", false}, {MOUNT_B, "other", true}}; + EXPECT_EQ(resolveConfirmRoutingCandidate(c, "shared"), std::nullopt); + EXPECT_EQ(resolveConfirmRoutingCandidate(c, ""), std::nullopt); + EXPECT_EQ(resolveConfirmRoutingCandidate({}, "shared"), std::nullopt); +} + +TEST(CASRelinkConfirmRouting, TwoDistinctOwnersAreAmbiguous) +{ + const std::vector c{{MOUNT_A, "shared", true}, {MOUNT_B, "shared", true}}; + EXPECT_EQ(resolveConfirmRoutingCandidate(c, "shared"), std::nullopt); +} + +TEST(CASRelinkConfirmRouting, AliasesOfOneMountCountOnce) +{ + /// A base disk and its cache wrapper share one exchange object: one mount, two disk names. + const std::vector c{{MOUNT_A, "shared", true}, {MOUNT_A, "shared", true}}; + EXPECT_EQ(resolveConfirmRoutingCandidate(c, "shared"), std::optional{0}); +} + +TEST(CASRelinkConfirmRouting, NonOwnerOnThePoolIsIgnored) +{ + const std::vector c{{MOUNT_A, "shared", false}, {MOUNT_B, "shared", true}}; + EXPECT_EQ(resolveConfirmRoutingCandidate(c, "shared"), std::optional{1}); +} diff --git a/tests/integration/test_cas_replicated_relink/configs/storage_conf_tiered.xml b/tests/integration/test_cas_replicated_relink/configs/storage_conf_tiered.xml new file mode 100644 index 000000000000..4ae77ab99a03 --- /dev/null +++ b/tests/integration/test_cas_replicated_relink/configs/storage_conf_tiered.xml @@ -0,0 +1,32 @@ + + + + + + + + default + + + disk_cas_shared + + + + + + + disk_cas_other + + + disk_cas_shared + + + + + + diff --git a/tests/integration/test_cas_replicated_relink/test.py b/tests/integration/test_cas_replicated_relink/test.py index 234cbbfbe7a4..e0bb176ea5a2 100644 --- a/tests/integration/test_cas_replicated_relink/test.py +++ b/tests/integration/test_cas_replicated_relink/test.py @@ -22,6 +22,14 @@ OTHER_STORAGE_POLICY = "cas_other" OTHER_CA_DISK = "disk_cas_other" +# node2-only policies (configs/storage_conf_tiered.xml). `cas_tiered` = [local `default`] then +# [disk_cas_shared]: an ordinary reservation lands on `default`, so a relink onto the pool's disk is a +# forced placement. `cas_two_pools` = [disk_cas_other] then [disk_cas_shared]: the first content-addressed +# disk is the WRONG pool, so a relink onto disk_cas_shared proves the whole pool set was advertised. +TIERED_STORAGE_POLICY = "cas_tiered" +TWO_POOLS_STORAGE_POLICY = "cas_two_pools" +LOCAL_DISK = "default" + # The shared pool's blob prefix inside the `test` RustFS bucket. The relink proof is that the fetch does # NOT create new objects under here: relink publishes a ref (per-server, under store/), never a blob. BLOBS_PREFIX = "shared_pool/blobs/" @@ -66,6 +74,7 @@ def start_cluster(): "configs/storage_conf.xml", "configs/server_root_id_node2.xml", "configs/storage_conf_other_pool.xml", + "configs/storage_conf_tiered.xml", ], macros={"replica": "node2"}, with_rustfs=True, @@ -183,6 +192,21 @@ def active_part_names(node, table): ).split() +def part_disk(node, table, part): + """The disk the ACTIVE part of this name sits on, from `system.parts`.""" + return node.query( + "SELECT disk_name FROM system.parts WHERE database = 'default' AND table = '{}' " + "AND name = '{}' AND active".format(table, part) + ).strip() + + +def detached_part_disk(node, table, part): + return node.query( + "SELECT disk FROM system.detached_parts WHERE database = 'default' AND table = '{}' " + "AND name = '{}'".format(table, part) + ).strip() + + def any_state_part_count(node, table, part): return int( node.query( @@ -931,3 +955,263 @@ def test_stalled_publish_protects_source_blobs_and_commits_nothing(): ) drop_everywhere(table) + + +# ---------------------------------------------------------------------------------------------------- +# FORCED PLACEMENT: a relink lands on the pool's disk even when the storage policy would put the part +# elsewhere. Every test here asserts the relink line AND `system.parts.disk_name`, because the first +# alone would also hold for a relink that then got moved, and the second alone would hold for a byte +# fetch that happened to be reserved on the pool's disk. +# ---------------------------------------------------------------------------------------------------- + + +def _fetch_via_queue(node1, node2, table, node2_policy, create_sql=None): + """INSERT on node1 while node2's fetches are stopped, then let node2 fetch exactly that one part. + + Returns `(part, blobs_before)`: the part name and the pool's blob keys as they were AFTER the + insert on node1 and BEFORE node2 fetched — the only snapshot `assert_no_new_blobs` can be measured + against, since the insert itself writes the part's blobs. `create_sql` overrides the table DDL (it + must contain `{policy}` and `{zk}`). + """ + drop_everywhere(table) + if create_sql is None: + create_replicated(node1, table) + create_replicated(node2, table, policy=node2_policy) + else: + node1.query(create_sql.format(policy=STORAGE_POLICY, zk="/clickhouse/tables/" + table)) + node2.query(create_sql.format(policy=node2_policy, zk="/clickhouse/tables/" + table)) + node2.query("SYSTEM STOP FETCHES {}".format(table)) + insert_rows(node1, table, 0) + part = active_part_names(node1, table)[0] + blobs_before = blob_keys() + node2.query("SYSTEM START FETCHES {}".format(table)) + node2.query("SYSTEM SYNC REPLICA {}".format(table), timeout=90) + return part, blobs_before + + +def test_tiered_policy_relinks_onto_cas_over_volume_order(): + """`[default] then [disk_cas_shared]`: the policy's own placement is the local volume, and before the + forced placement the fetch reserved there, failed the pool post-check and downloaded bytes onto + `default`. Now the offer decides the disk.""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "tiered_order" + + part, blobs_before = _fetch_via_queue(node1, node2, table, TIERED_STORAGE_POLICY) + + assert_relinked(node2, table, part) + assert part_disk(node2, table, part) == CA_DISK + assert_no_new_blobs(blobs_before) + assert int(node2.query("SELECT count() FROM {}".format(table))) == NUM_ROWS + drop_everywhere(table) + + +def test_relink_carries_projection_under_tiered_policy(): + """A projection-bearing part relinks like any other (the projection is loaded from the published + manifest), and the forced placement does not disturb that.""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "tiered_projection" + create_sql = ( + "CREATE TABLE " + table + " (id Int64, v UInt64, s String, " + "PROJECTION p_by_s (SELECT s, sum(v) GROUP BY s)) " + "ENGINE = ReplicatedMergeTree('{zk}', '{{replica}}') ORDER BY id " + "SETTINGS storage_policy = '{policy}'" + ) + + part, _ = _fetch_via_queue(node1, node2, table, TIERED_STORAGE_POLICY, create_sql=create_sql) + + assert_relinked(node2, table, part) + assert part_disk(node2, table, part) == CA_DISK + assert int(node2.query( + "SELECT count() FROM system.projection_parts WHERE database = 'default' AND table = '{}' " + "AND parent_name = '{}' AND name = 'p_by_s' AND active".format(table, part) + )) == 1 + assert int(node2.query("SELECT sum(v) FROM {} WHERE s = '7'".format(table))) == 70 + drop_everywhere(table) + + +def test_two_pool_policy_relinks_into_second_pool(): + """`[disk_cas_other] then [disk_cas_shared]`: the first content-addressed disk is the WRONG pool. + A single-pool advertise names `other`, the sender declines, and the bytes land on `disk_cas_other`; + advertising the whole set lets the sender match `shared` and the receiver place it there.""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "two_pools" + + part, blobs_before = _fetch_via_queue(node1, node2, table, TWO_POOLS_STORAGE_POLICY) + + assert_relinked(node2, table, part) + assert part_disk(node2, table, part) == CA_DISK + assert_no_new_blobs(blobs_before) + assert not log_lines(node2, download_finished_pattern(table, part, disk=OTHER_CA_DISK)) + drop_everywhere(table) + + +def test_mechanism_failure_falls_back_to_bytes_on_forced_disk(): + """A relink that fails for a mechanism reason re-requests the bytes on the SAME forced disk — the + placement decision outlives the relink. (The one-offer recursion bound is proven by + `test_recursion_brake_bounds_relink_to_one_attempt`; this test proves only the destination.)""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "tiered_fallback" + drop_everywhere(table) + create_replicated(node1, table) + create_replicated(node2, table, policy=TIERED_STORAGE_POLICY) + node2.query("SYSTEM STOP FETCHES {}".format(table)) + insert_rows(node1, table, 0) + part = active_part_names(node1, table)[0] + + node2.query("SYSTEM ENABLE FAILPOINT cas_relink_receiver_force_mechanism_failure") + try: + node2.query("SYSTEM START FETCHES {}".format(table)) + node2.query("SYSTEM SYNC REPLICA {}".format(table), timeout=90) + assert_byte_downloaded(node2, table, part, disk=CA_DISK) + assert part_disk(node2, table, part) == CA_DISK + assert not log_lines(node2, download_finished_pattern(table, part, disk=LOCAL_DISK)) + finally: + node2.query("SYSTEM DISABLE FAILPOINT cas_relink_receiver_force_mechanism_failure") + drop_everywhere(table) + + +def test_detached_fetch_relinks_onto_cas_under_tiered_policy(): + """`ALTER TABLE ... FETCH PART` into `detached/` under the tiered policy: the forced placement + applies to detached fetches too, and the detached part is on the pool's disk.""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + src, dst = "tiered_det_src", "tiered_det_dst" + drop_everywhere(src) + drop_everywhere(dst) + create_replicated(node1, src) + create_replicated(node2, dst, policy=TIERED_STORAGE_POLICY) + insert_rows(node1, src, 0) + part = active_part_names(node1, src)[0] + + node2.query( + "ALTER TABLE {dst} FETCH PART '{part}' FROM '/clickhouse/tables/{src}'".format( + dst=dst, part=part, src=src + ) + ) + + assert_relinked(node2, dst, part) + assert detached_part_disk(node2, dst, part) == CA_DISK + node2.query("ALTER TABLE {} ATTACH PART '{}'".format(dst, part)) + assert part_disk(node2, dst, part) == CA_DISK + assert int(node2.query("SELECT count() FROM {}".format(dst))) == NUM_ROWS + drop_everywhere(src) + drop_everywhere(dst) + + +def test_offer_for_unavailable_pool_falls_back_to_ordinary_placement(): + """The receiver resolved a forced disk and then lost it (the failpoint stands in for an offer this + policy has no disk for): the ordinary reservation runs, the relink block sees a disk outside the + offered pool, and the bytes go where the policy says — `default` under the tiered policy. Exactly + one offer is made: the byte re-request carries no advertise.""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "tiered_dropped" + drop_everywhere(table) + create_replicated(node1, table) + create_replicated(node2, table, policy=TIERED_STORAGE_POLICY) + node2.query("SYSTEM STOP FETCHES {}".format(table)) + insert_rows(node1, table, 0) + part = active_part_names(node1, table)[0] + + node2.query("SYSTEM ENABLE FAILPOINT cas_relink_receiver_drop_forced_disk") + try: + node2.query("SYSTEM START FETCHES {}".format(table)) + node2.query("SYSTEM SYNC REPLICA {}".format(table), timeout=90) + assert log_lines( + node2, + r"Failpoint cas_relink_receiver_drop_forced_disk: forgetting the forced disk for part {}".format( + re.escape(part) + ), + ) + assert_byte_downloaded(node2, table, part, disk=LOCAL_DISK) + assert part_disk(node2, table, part) == LOCAL_DISK + assert len(log_lines(node1, relink_offer_pattern(table, part))) == 1 + finally: + node2.query("SYSTEM DISABLE FAILPOINT cas_relink_receiver_drop_forced_disk") + drop_everywhere(table) + + +def test_offer_without_pool_cookie_resolves_to_single_advertised_pool(): + """The old-sender shape: an offer with no `cas_pool_uuid` cookie. With ONE advertised pool that is + the pool, and the relink is forced as usual; with TWO advertised pools the receiver refuses to guess, + the ordinary reservation lands on the first volume (`disk_cas_other`), and the bytes go there.""" + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + node1.query("SYSTEM ENABLE FAILPOINT cas_relink_sender_omit_pool_cookie") + try: + one_pool = "omit_cookie_one" + part, _ = _fetch_via_queue(node1, node2, one_pool, TIERED_STORAGE_POLICY) + assert_relinked(node2, one_pool, part) + assert part_disk(node2, one_pool, part) == CA_DISK + drop_everywhere(one_pool) + + two_pools = "omit_cookie_two" + part, _ = _fetch_via_queue(node1, node2, two_pools, TWO_POOLS_STORAGE_POLICY) + assert_byte_downloaded(node2, two_pools, part, disk=OTHER_CA_DISK) + assert part_disk(node2, two_pools, part) == OTHER_CA_DISK + assert len(log_lines(node1, relink_offer_pattern(two_pools, part))) == 1 + drop_everywhere(two_pools) + finally: + node1.query("SYSTEM DISABLE FAILPOINT cas_relink_sender_omit_pool_cookie") + + +def test_relink_wins_over_ttl_then_mover_converges(): + """A `TTL ... TO DISK` rule that names the LOCAL disk for this (already expired) part does not stop the + relink: the part lands on the pool's disk at zero byte cost, and the background mover — which sees a + part that is not in its TTL destination — carries it to `default` afterwards. Moves are stopped on + node2 around the fetch so the intermediate placement is observable, exactly as `test_ttl_move` does. + + `IF EXISTS` precedes the disk name in the grammar. It is there for node1, whose policy has no + `default` disk: without it `CREATE TABLE` on node1 fails with `BAD_TTL_EXPRESSION`, because + `MergeTreeData::checkTTLExpressions` rejects a `TO DISK` destination absent from the policy at + create time. + """ + node1 = cluster.instances["node1"] + node2 = cluster.instances["node2"] + table = "tiered_ttl" + drop_everywhere(table) + create_sql = ( + "CREATE TABLE " + table + " (id Int64, v UInt64, s String, ts DateTime) " + "ENGINE = ReplicatedMergeTree('/clickhouse/tables/" + table + "', '{{replica}}') ORDER BY id " + "TTL ts TO DISK IF EXISTS 'default' " + "SETTINGS storage_policy = '{policy}'" + ) + node1.query(create_sql.format(policy=STORAGE_POLICY)) + node2.query(create_sql.format(policy=TIERED_STORAGE_POLICY)) + + node2.query("SYSTEM STOP MOVES {}".format(table)) + node2.query("SYSTEM STOP FETCHES {}".format(table)) + try: + node1.query( + "INSERT INTO {table} SELECT number, number * 10, toString(number), now() - INTERVAL 1 DAY " + "FROM numbers({rows})".format(table=table, rows=NUM_ROWS) + ) + part = active_part_names(node1, table)[0] + assert part_disk(node1, table, part) == CA_DISK # the sender holds it in the pool + + node2.query("SYSTEM START FETCHES {}".format(table)) + node2.query("SYSTEM SYNC REPLICA {}".format(table), timeout=90) + + # The TTL rule says `default`; the relink put it on the pool's disk anyway, and moves are stopped. + assert_relinked(node2, table, part) + assert part_disk(node2, table, part) == CA_DISK + rows_before = node2.query("SELECT count(), sum(v) FROM {}".format(table)) + + node2.query("SYSTEM START MOVES {}".format(table)) + wait_until( + lambda: part_disk(node2, table, part) == LOCAL_DISK, + timeout=120, + what="the background mover carrying {} to {}".format(part, LOCAL_DISK), + ) + assert node2.query("SELECT count(), sum(v) FROM {}".format(table)) == rows_before + assert node2.query("SELECT count(), sum(v) FROM {}".format(table)) == node1.query( + "SELECT count(), sum(v) FROM {}".format(table) + ) + finally: + node2.query("SYSTEM START FETCHES {}".format(table)) + node2.query("SYSTEM START MOVES {}".format(table)) + drop_everywhere(table) From a4b3cbc1b1ddb09e30a72a081e9af542ef310de2 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:07:11 +0200 Subject: [PATCH 20/81] cas: a disk's teardown no longer waits out a GC round MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ContentAddressedMetadataStorage::shutdown` contained two unbounded waits on the GC round: a `std::lock_guard` on the same mutex a synchronous round (`SYSTEM CAS GC`, `GC REBUILD`) holds for its whole duration, and `CasGcScheduler::stop`'s join, which the scheduler loop only even looks at at the top of its wait — a round already in flight never observes it, and a comment in the loop recorded an accepted extra full round if `stop` lands while the loop is blocked behind a manual round. A round has no wall-clock budget at all: `GcRoundWorkBudget` caps destructive work, not time, and against a slow bucket the wall clock is whatever the bucket makes it. Nothing in this wait protects durable state — the round is one-pass, committed by a single `gc/state` conditional write at the end, so an interrupted round is a crash the protocol already survives — the wait existed purely so no thread would touch a freed object. Shutdown and the storage destructor now arm the pool's teardown flag before the lock or join they would otherwise wait behind. The open request plane carries that flag as its fence, so a round in flight is refused at its next request, its next retry sleep, or its next streamed refill — the check lives at the request because a phase is long from making thousands of requests, not from making one long one, and `CasOperation` already re-checks admission before every attempt and sleep. Every join and every object's ownership stay unchanged: the join became short, not optional. A round cut this way is recorded `Stopped` rather than `Aborted`. Decommission is deliberately not armed: an already-latched self-remount completes one more step whose pool-identity probe runs on the open plane, and no arm point early enough to bound the GC join leaves that step intact. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../en/operations/system-tables/cas_gc_log.md | 6 +- .../ContentAddressed/Backend/CasRequests.cpp | 88 +++- .../ContentAddressedMetadataStorage.cpp | 30 +- .../ContentAddressedMetadataStorage.h | 16 +- .../ContentAddressed/Gc/CasGcScheduler.cpp | 31 +- .../ContentAddressed/Gc/CasGcScheduler.h | 10 +- .../ContentAddressed/Pool/CasDetachedWork.cpp | 3 +- .../ContentAddressed/Pool/CasDetachedWork.h | 7 +- .../ContentAddressed/Pool/CasPool.cpp | 47 +- .../ContentAddressed/Pool/CasPool.h | 20 + src/Disks/tests/cas_test_helpers.h | 8 +- src/Disks/tests/gtest_cas_gc_log.cpp | 85 ++++ .../tests/gtest_cas_gc_teardown_stop.cpp | 454 ++++++++++++++++++ src/Disks/tests/gtest_cas_requests.cpp | 132 +++++ .../ContentAddressedGarbageCollectionLog.cpp | 9 +- .../ContentAddressedGarbageCollectionLog.h | 6 +- 16 files changed, 914 insertions(+), 38 deletions(-) create mode 100644 src/Disks/tests/gtest_cas_gc_teardown_stop.cpp diff --git a/docs/en/operations/system-tables/cas_gc_log.md b/docs/en/operations/system-tables/cas_gc_log.md index eb422cddc4d1..ebdce2038cdb 100644 --- a/docs/en/operations/system-tables/cas_gc_log.md +++ b/docs/en/operations/system-tables/cas_gc_log.md @@ -38,7 +38,7 @@ specified (it is enabled by default in the shipped `config.xml`). - `gc_id` ([String](/sql-reference/data-types/string)) — The GC scheduler instance id (which mounter ran the round). - `trigger` ([Enum8](/sql-reference/data-types/enum)) — `Scheduled` (background tick) or `Manual` (`SYSTEM` command). - `round` ([UInt64](/sql-reference/data-types/int-uint)) — The GC round number (`0` on a `Start` row). -- `outcome` ([Enum8](/sql-reference/data-types/enum)) — `Unknown` (on a `Start` row), `Success` (led, folded, and completed), `NotALeader` (another replica holds the GC lease), `Deferred` (led but took the skip-unchanged fast path — no fold ran, because no changed shard reached the fold threshold and no graduation was due), `Aborted` (the round threw a transient error — backend unavailability, a lost lease, a concurrent leader; the next scheduled round retries it), or `Error` (the round threw a non-transient error — investigate). +- `outcome` ([Enum8](/sql-reference/data-types/enum)) — `Unknown` (on a `Start` row), `Success` (led, folded, and completed), `NotALeader` (another replica holds the GC lease), `Deferred` (led but took the skip-unchanged fast path — no fold ran, because no changed shard reached the fold threshold and no graduation was due), `Aborted` (the round threw a transient error — backend unavailability, a lost lease, a concurrent leader; the next scheduled round retries it), `Stopped` (a transient error observed after the disk began shutting down: the round was cut short so the shutdown need not wait for it, and the next start re-derives its work), or `Error` (the round threw a non-transient error — investigate). - `candidates_marked` ([UInt64](/sql-reference/data-types/int-uint)) — Objects retired (marked) this round. - `objects_deleted` ([UInt64](/sql-reference/data-types/int-uint)) — Objects physically deleted this round. - `objects_absent` ([UInt64](/sql-reference/data-types/int-uint)) — Retire candidates found already absent. @@ -51,8 +51,8 @@ specified (it is enabled by default in the shipped `config.xml`). - `fence_outs` ([UInt64](/sql-reference/data-types/int-uint)) — Expired mounts fenced out by this round's heartbeat floor. - `anomalies` ([UInt64](/sql-reference/data-types/int-uint)) — Fold clamps surfaced (and survived) this round. A steady non-zero value warrants a look at the round log details. - `duration_ms` ([UInt64](/sql-reference/data-types/int-uint)) — The round wall-clock duration (on a `Finish` row). -- `error` ([String](/sql-reference/data-types/string)) — The exception text when `outcome = 'Aborted'` or `'Error'`. -- `error_code` ([Int32](/sql-reference/data-types/int-uint)) — The exception code when `outcome = 'Aborted'` or `'Error'`; `0` otherwise. Key monitoring on this column rather than on the `error` text. On an `Aborted` or `Error` row the counters still report everything the round completed before it threw, and `round != 0` on such a row means the round's closing compare-and-swap committed and the failure hit only post-commit cleanup. +- `error` ([String](/sql-reference/data-types/string)) — The exception text when `outcome = 'Aborted'`, `'Stopped'` or `'Error'`. On a `Stopped` row it names the request the shutdown refused, not the shutdown itself. +- `error_code` ([Int32](/sql-reference/data-types/int-uint)) — The exception code when `outcome = 'Aborted'`, `'Stopped'` or `'Error'`; `0` otherwise. Key monitoring on this column rather than on the `error` text. On an `Aborted`, `Stopped` or `Error` row the counters still report everything the round completed before it threw, and `round != 0` on such a row means the round's closing compare-and-swap committed and the failure hit only post-commit cleanup. - `ProfileEvents` ([Map(LowCardinality(String), UInt64)](/sql-reference/data-types/map)) — On a `Start`/`Finish` row, the per-round `ProfileEvents` delta (the `CAS*` counters and S3/disk events for this round). On a `Phase` row, **that phase's** delta, so `GROUP BY phase` over `ProfileEvents['S3ListObjects']` attributes the round's `LIST` budget to the phase that spent it. - `round_id` ([String](/sql-reference/data-types/string)) — The correlator for every row of one round attempt: its `Start`, each of its `Phase` rows, and its `Finish`. Minted per attempt, so unlike `round` it exists even for a round that never committed and for a round that never led. Group by this column to reconstruct one round. - `phase` ([LowCardinality(String)](/sql-reference/data-types/lowcardinality)) — The GC phase this row describes; empty on `Start`/`Finish`. See [Per-phase rows](#per-phase-rows) for the phase list. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 301960426318..4398784bda11 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -1,6 +1,7 @@ #include #include +#include #include #include #include @@ -282,6 +283,81 @@ const String & CasRequests::valueFor(const String & key, const Etag & inc) const return inc.value(); } +namespace +{ + +/// The one mapping from a refused admission to the exception a read-class caller sees. The request +/// gate and the streamed body below both refuse through it, so a body refused mid-transfer reads +/// exactly like an open refused before it: `NoBudget` is the retry-later class, everything else the +/// tripped-fence class. Free, not a member: the body's copy must not reference an operation. +[[noreturn]] void throwReadRefused(Fence::Admit admit, std::string_view verb, const String & subject, + std::string_view when) +{ + if (admit == Fence::Admit::NoBudget) + throwCasWriteRetryLater(fmt::format("{} of '{}': no lease budget {}", verb, subject, when)); + throwCasTransientUnavailable(fmt::format("CAS {} of '{}'", verb, subject), + fmt::format("mount fence tripped {}", when)); +} + +/// The body of a streamed object, re-admitted at every refill. `Backend::stream` bounds only the +/// open; the SDK reads the body at the consumer's pace, under the storage's ordinary settings, long +/// after the attempt that opened it returned -- so a fold parked in a multi-gigabyte run body would +/// outlive the fence that refused every other request of its operation. The check is the operation's +/// own admission, asked once per SDK buffer, and its refusal is the exception a refused open produces. +/// +/// The predicate is held BY VALUE -- the fence's `admit` closure, the admitted generation, the +/// caller's liveness -- never through the operation: `CasOperation::stream` returns a buffer that can +/// outlive the operation object (S3 staging opens its stream under a local mount-plane operation and +/// hands the buffer to the backend). Modelled on `LimitReadBuffer`: no byte is copied, the SDK's +/// window is exposed as this buffer's own. +class AdmittedBodyReadBuffer : public ReadBuffer +{ +public: + AdmittedBodyReadBuffer(std::unique_ptr in_, String key_, + std::function admit_, + uint64_t admitted_generation_, Liveness liveness_) + /// The open already loaded a window (`Backend::stream` forces the first GET): adopt it, so + /// the first refill this buffer asks for is the SECOND window and the SDK buffer is never + /// asked to advance over pending data. + : ReadBuffer(in_->position(), in_->available(), 0) + , in(std::move(in_)) + , key(std::move(key_)) + , admit(std::move(admit_)) + , admitted_generation(admitted_generation_) + , liveness(std::move(liveness_)) + { + } + +private: + bool nextImpl() override + { + /// Let the SDK buffer account the bytes the consumer took from the shared window. + in->position() = position(); + + Fence::Admit verdict = admit(admitted_generation, 0); + if (verdict == Fence::Admit::Ok && liveness && !liveness()) + verdict = Fence::Admit::LostOrRearmed; + if (verdict != Fence::Admit::Ok) + throwReadRefused(verdict, "stream body", key, "mid-body"); + + if (!in->next()) + { + BufferBase::set(in->position(), 0, 0); + return false; + } + BufferBase::set(in->position(), in->available(), 0); + return true; + } + + std::unique_ptr in; + String key; + std::function admit; + uint64_t admitted_generation; + Liveness liveness; +}; + +} + CasOperation::Gate CasOperation::gate(uint64_t needed_ms) const { switch (owner.fence.admit(admitted_generation, needed_ms)) @@ -354,14 +430,13 @@ bool CasOperation::refreshAndClassifyReadFault(const std::exception & e, bool & void CasOperation::giveUpReadFenceLost(std::string_view verb, const String & subject, std::string_view when) { last_read_stop = ReadStop::FenceLost; - throwCasTransientUnavailable(fmt::format("CAS {} of '{}'", verb, subject), - fmt::format("mount fence tripped {}", when)); + throwReadRefused(Fence::Admit::LostOrRearmed, verb, subject, when); } void CasOperation::giveUpReadNoBudget(std::string_view verb, const String & subject, std::string_view what) { last_read_stop = ReadStop::NoBudgetLease; - throwCasWriteRetryLater(fmt::format("{} of '{}': no lease budget {}", verb, subject, what)); + throwReadRefused(Fence::Admit::NoBudget, verb, subject, what); } void CasOperation::giveUpReadDeadline(std::string_view verb, const String & subject, @@ -572,10 +647,15 @@ SentinelProbeResult CasOperation::probeSentinel(const String & key, const Retry std::unique_ptr CasOperation::stream(const String & key, const Retry & policy) { const Retry::Bound bound = policy.bind(owner.now_ms()); - return readLoop("stream", key, policy, bound, [&](auto & access) + std::unique_ptr body = readLoop("stream", key, policy, bound, [&](auto & access) { return owner.backend->stream(key, access); }); + if (!body) + return nullptr; /// absent: the open already answered + /// By value, deliberately: nothing the buffer holds may reference this operation or its owner. + return std::make_unique(std::move(body), key, owner.fence.admit, + admitted_generation, liveness); } void CasOperation::publish(const BlobPublishRequest & request, const Retry & policy) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index f1227ccf952a..5bff526246df 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -465,15 +465,19 @@ CasLifecycleSnapshot ContentAddressedMetadataStorage::lifecycleSnapshot() const Cas::GcRoundLogger ContentAddressedMetadataStorage::makeGcRoundLogger() const { - /// Unit tests pass a null context (no system logs); the scheduler then runs without a sink. + /// Unit tests pass a null context (no system logs); the scheduler then runs with the test hook + /// as its only sink, or without a sink. + const std::function hook = gc_round_row_hook_for_test; if (!context) - return {}; + return hook ? Cas::GcRoundLogger(hook) : Cas::GcRoundLogger{}; const ContextWeakPtr weak_context = *context; /// The configured disk name (threaded from the metadata-storage factory); falls back to /// storage_path_prefix for callers that don't supply one (e.g. unit tests). const String disk = disk_name; - return [weak_context, disk](const Cas::GcRoundLogRecord & r) + return [weak_context, disk, hook](const Cas::GcRoundLogRecord & r) { + if (hook) + hook(r); auto ctx = weak_context.lock(); if (!ctx) { @@ -525,6 +529,9 @@ Cas::GcRoundLogger ContentAddressedMetadataStorage::makeGcRoundLogger() const case Cas::GcRoundLogRecord::Outcome::Aborted: e.outcome = ContentAddressedGarbageCollectionLogElement::ABORTED; break; + case Cas::GcRoundLogRecord::Outcome::Stopped: + e.outcome = ContentAddressedGarbageCollectionLogElement::STOPPED; + break; } e.round = r.round; e.candidates_marked = r.candidates_marked; @@ -898,8 +905,17 @@ void ContentAddressedMetadataStorage::startup() void ContentAddressedMetadataStorage::shutdown() { - /// Wait for any in-flight synchronous round to finish cleanly first (gc_scheduler_mutex is held - /// for a round's whole duration) -- unchanged priority: clean GC completion over fast shutdown. + /// Arm the pool BEFORE waiting for `gc_scheduler_mutex`: a synchronous round holds that mutex + /// for its whole duration and releases it only once its next request is refused. The arm frees, + /// nulls and swaps nothing -- every pointer swap still happens below, under the same locks as + /// before -- so taking `pointer_mutex` alone here, before the outer lock, inverts no order. + Cas::PoolPtr pool; + { + std::lock_guard ptr_lock(pointer_mutex); + pool = cas_store; + } + if (pool) + pool->beginTeardown(); std::lock_guard round_lock(gc_scheduler_mutex); shutdown_called = true; stopAndDrainForTeardown(); @@ -947,6 +963,10 @@ void ContentAddressedMetadataStorage::stopAndDrainForTeardown() noexcept } }; + /// The destructor reaches here without `shutdown`'s arm: arm now, before the join, so a + /// background round is refused at its next request rather than joined at its end. + if (old_pool) + old_pool->beginTeardown(); guarded([&] { if (old_scheduler) old_scheduler->stop(); }, "CAS storage teardown: stopping GC"); guarded([&] { old_part_access.reset(); }, "CAS storage teardown: releasing part access"); guarded([&] diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index c5c1d8c788b0..8200bfd1c5d0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -549,6 +549,15 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// default; production installs none. void setGcVerbAdmitWindowHookForTest(std::function fn) { gc_verb_admit_window_hook_for_test = std::move(fn); } + /// Test-only: sees every round-log row the storage's own scheduler emits (Start, Phase, Finish), + /// before and independently of the system-log path -- which a unit-test storage (null `Context`) + /// does not have at all. Lets a test park a synchronous round on a phase row while it holds + /// `gc_scheduler_mutex`, and read the round's outcome afterwards. Set before the first round. + void setGcRoundRowHookForTest(std::function fn) + { + gc_round_row_hook_for_test = std::move(fn); + } + /// Test-only fault-injection/hook seam for `ContentAddressedTransaction::publishStaging`'s /// promote/repoint call, keyed by the full `(ns, ref)` routed identity via `PartRefKey::cacheKey()` /// (mirrors `CasRefLedger::setRefPreCarveHookForTest`'s no-op-in-production shape) -- a bare ref @@ -661,9 +670,9 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// `gcStop`/`gcStart`. Serializes them against each other. Lock order when nested locks are needed: /// `lifecycle_mutex` -> `gc_scheduler_mutex` -> `pointer_mutex`, never the reverse. mutable std::mutex lifecycle_mutex; - /// Serializes ONE synchronous GC round at a time and makes `shutdown` wait for an in-flight round - /// to finish cleanly (clean GC completion has priority over fast shutdown) -- held for the WHOLE - /// round. Deliberately NOT the same mutex as `pointer_mutex` below: this one can be held for a + /// Serializes ONE synchronous GC round at a time. `shutdown` still takes it, but arms the pool + /// first, so a round in flight is refused at its next request once the pool is armed and the wait + /// is one request long -- held for the WHOLE round. Deliberately NOT the same mutex as `pointer_mutex` below: this one can be held for a /// long time, so nothing that only needs a brief pointer snapshot may share it. mutable std::mutex gc_scheduler_mutex; bool shutdown_called TSA_GUARDED_BY(gc_scheduler_mutex) = false; @@ -791,6 +800,7 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// TOCTOU tests). Empty in production; a `const` GC verb reads it and calls the const-qualified /// `std::function::operator()`, so it needs no `mutable`. std::function gc_verb_admit_window_hook_for_test; + std::function gc_round_row_hook_for_test; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp index 0ae8351de5ba..610d13879779 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.cpp @@ -266,7 +266,11 @@ Cas::RoundReport CasGcScheduler::runRoundLogged(Cas::Gc & round_gc, GcRoundLogRe catch (...) { fin.error_code = getCurrentExceptionCode(); - fin.outcome = isTransientGcRoundError(fin.error_code) ? Rec::Outcome::Aborted : Rec::Outcome::Failed; + /// Non-transient first, so a bug that coincides with a restart is never masked; then the + /// teardown flag, the only witness of a refused teardown fence (see `Outcome::Stopped`). + fin.outcome = !isTransientGcRoundError(fin.error_code) ? Rec::Outcome::Failed + : store->teardownBegun() ? Rec::Outcome::Stopped + : Rec::Outcome::Aborted; fin.error = getCurrentExceptionMessage(false); fill_counters(rep); fin.duration_ms = std::chrono::duration_cast( @@ -334,6 +338,13 @@ void CasGcScheduler::loop() /// correctness issue. std::lock_guard round_lock(gc_round_mutex); + /// A round that starts after the pool's teardown began would emit a Start row and be + /// refused at its first lease request -- a row that says nothing. Checked here, under the + /// round mutex, so it also covers the tick queued behind a manual round; the extra round + /// on a plain `stop` (above) is a different race and stays as described. + if (store->teardownBegun()) + return; + /// runRoundLogged emits the Start + Finish table rows (incl. the per-round /// ProfileEvents delta) and rethrows on a round exception (after an Aborted Finish). /// on_lease_acquired (onLeaseAcquired, shared with runOneRoundNow) fires the instant the @@ -378,6 +389,13 @@ void CasGcScheduler::loop() } catch (...) { + if (store->teardownBegun() && isTransientGcRoundError(getCurrentExceptionCode())) + { + /// The disk is being torn down and the round was cut at its next request: expected, + /// recorded as `Stopped` by `runRoundLogged`, not an error to raise. + LOG_INFO(log, "CA GC round stopped by the disk's teardown: {}", getCurrentExceptionMessage(false)); + continue; + } /// Idempotent round - the next tick retries; failures must never kill the pacing thread. /// runRoundLogged already emitted the classified (Aborted/Failed) Finish row before rethrowing. /// @@ -434,7 +452,16 @@ void CasGcScheduler::heartbeatLoop() } catch (...) { - tryLogCurrentException(log, "CA GC heartbeat pulse failed (advisory; will retry)"); + /// A pulse refused by the open plane during teardown is the expected end of this loop and + /// not a failure to report; `stop` joins it moments later. Only a TRANSIENT failure is + /// silent, for the same fail-closed reason the round classifier refuses to relabel a + /// non-transient one: a corrupt heartbeat that happens to coincide with a restart is an + /// incident, and swallowing it would be the one place this teardown path hides a defect. + const bool tearing_down = store->teardownBegun(); + if (!tearing_down || !isTransientGcRoundError(getCurrentExceptionCode())) + tryLogCurrentException(log, "CA GC heartbeat pulse failed (advisory; will retry)"); + if (tearing_down) + return; } } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h index a01186507074..80b50f2fc448 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcScheduler.h @@ -29,9 +29,13 @@ struct GcRoundLogRecord /// that genuinely folded and found nothing apart from one that never folded at all. /// `Aborted`: the round threw an exception whose code names a transient condition (backend /// unavailability, a lost lease, a concurrent leader) -- the next scheduled round retries it and - /// nothing durable is wrong. `Failed` is reserved for everything else (a logic error, corrupted - /// data, an unclassified code): fail-closed, an unrecognised failure reads as real. - enum class Outcome { Unknown, Success, NotALeader, Failed, Deferred, Aborted }; + /// nothing durable is wrong. `Stopped`: the same transient class, observed after the pool's + /// teardown began -- a correlation, not a cause: the engine reports a refused teardown fence + /// exactly like a lost mount fence, so the flag is the only witness, and the row says so honestly + /// rather than reading a clean restart as a backend incident. `Failed` is reserved for everything + /// else (a logic error, corrupted data, an unclassified code): fail-closed, an unrecognised + /// failure reads as real -- during a teardown too. + enum class Outcome { Unknown, Success, NotALeader, Failed, Deferred, Aborted, Stopped }; enum class Trigger { Scheduled, Manual }; EventType event_type = EventType::Start; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.cpp index c836f9932d79..e61145e03ce8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.cpp @@ -6,8 +6,7 @@ namespace DB::Cas bool DetachedStopToken::stopping() const { - std::lock_guard lock(state->mutex); - return state->stopping; + return state->stopping.load(std::memory_order_acquire); } struct DetachedTaskLease::Completion diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.h index 5e317e429071..1190dc1df939 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasDetachedWork.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -19,7 +20,11 @@ struct DetachedRegistryState std::mutex mutex; std::condition_variable cv; uint64_t in_flight = 0; - bool stopping = false; + /// Written under `mutex`, so the dispatch-side check-and-count and the drain's `in_flight == 0` + /// wait stay serialized against the stop; read WITHOUT it by the open request plane's fence before + /// every attempt and every sleep of every request -- a pool-wide mutex on that path is not + /// acceptable, and the readers need only the flag's current truth. + std::atomic stopping{false}; }; /// Read-only view of the registry, handed to every task. The ONLY way a task asks whether teardown has diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 40229890bdf0..8e51b551e9d7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -196,7 +196,22 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) config.boot_ms_fn, mountPlaneSleepFn()) , farewell_requests(pool_backend, Fence::open(), config.boot_ms_fn) - , gc_requests(pool_backend, Fence::open(), config.boot_ms_fn) + /// The open plane's fence is the pool's teardown flag: generation 0 forever, exactly like + /// `Fence::open`, but `admit` refuses once `beginTeardown` ran. A GC round, an FSCK or a probe in + /// flight is then refused at its next request instead of running to completion under a disk that + /// is being torn down. The ref ledger and the farewell live on the other two planes, so + /// teardown's own I/O never meets this fence. A write already proven durable is admitted ONCE + /// MORE (`postCommit`), so an armed teardown can turn a landed `gc/state` into a give-up rather + /// than a commit. That is safe and not merely tolerable: the round is one-pass, so the next round + /// reads the state this one committed and re-derives the rest, exactly as after a crash at that + /// instant -- and every step of the tail the give-up skipped is admitted on THIS plane, so it + /// would have been refused anyway. What it costs is the round number on the round's own row. + , gc_requests(pool_backend, Fence{ + [] { return uint64_t{0}; }, + [this](uint64_t, uint64_t) { return teardownBegun() ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [](uint64_t) {}}, + config.boot_ms_fn, + openPlaneSleepFn()) /// Seed the monotone admitted-algo cache from the pool state `createOrValidate` already /// established (fresh create, steady-state member, or a just-completed admission union) -- /// register-before-first-write means this Pool's own `writeAlgo()` is ALWAYS a @@ -1043,14 +1058,23 @@ bool Pool::tryDispatchDetached(std::function task) return true; } -bool Pool::stopAndDrainDetachedWork(uint64_t deadline_ms) +void Pool::beginTeardown() noexcept { { std::lock_guard lock(detached_work->mutex); - detached_work->stopping = true; + detached_work->stopping.store(true, std::memory_order_release); } detached_work->cv.notify_all(); +} + +bool Pool::teardownBegun() const noexcept +{ + return detached_work->stopping.load(std::memory_order_acquire); +} +bool Pool::stopAndDrainDetachedWork(uint64_t deadline_ms) +{ + beginTeardown(); std::unique_lock lock(detached_work->mutex); return detached_work->cv.wait_for(lock, std::chrono::milliseconds(deadline_ms), [this] { return detached_work->in_flight == 0; }); @@ -1064,8 +1088,7 @@ uint64_t Pool::detachedWorkInFlight() const bool Pool::detachedWorkStoppingForTest() const { - std::lock_guard lock(detached_work->mutex); - return detached_work->stopping; + return teardownBegun(); } void Pool::setDetachedDrainDeadlineBudgetForTest(uint64_t attempt_timeout_ms, uint64_t lease_safety_margin_ms) @@ -1094,6 +1117,12 @@ void Pool::forgetDisk(const std::function & stop_and_join_gc, const Stri if (mount_runtime.isVanished()) return; + /// This protocol deliberately does NOT arm the open plane. The GC join at (3+4) would be bounded + /// by it, but an already-latched self-remount completes its current step before the loop bails at + /// (5a), and that step's pool-identity probe is admitted on the open plane -- an arm here refuses + /// it, so the reclaim `finishTeardown` is written to override could never happen. Server shutdown + /// arms instead: it joins the same scheduler with no remount step to preserve. + /// /// (1) Publish the terminal-intent latch FIRST (spec §5). The runtime stops latching remounts and /// the remount loop bails at its next step boundary, so every join below is bounded to one step + one /// backend timeout. @@ -1885,11 +1914,11 @@ void Pool::setCasRetrySleepForTest(std::function sleep_fn) /// All three planes, not just the ledger's: a test that replaces the retry sleep must not be left /// with a real one on the plane the site under test happens to use. farewell_requests.setSleepFnForTest(sleep_fn); - gc_requests.setSleepFnForTest(sleep_fn); ref_ledger.setCasRetrySleepForTest(sleep_fn); - /// The ledger reaches the mount plane too, and `CasRequests` falls back to the engine's plain - /// sleep for an empty argument -- which is not this plane's default. Re-install ours last, so - /// clearing the seam cannot leave a parked or stopping renewal held for a whole capped backoff. + /// `CasRequests` falls back to the engine's plain sleep for an empty argument -- which is neither + /// the mount plane's nor the open plane's default. Re-install both, so clearing the seam cannot + /// leave a parked renewal held for a whole capped backoff, or the open plane deaf to a teardown. + gc_requests.setSleepFnForTest(sleep_fn ? sleep_fn : openPlaneSleepFn()); mount_requests.setSleepFnForTest(sleep_fn ? std::move(sleep_fn) : mountPlaneSleepFn()); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index feee78c93229..bf5ee66fb6ec 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -410,6 +410,12 @@ class Pool : public std::enable_shared_from_this ~Pool(); bool tryDispatchDetached(std::function task); + /// Marks this pool as being torn down. The open request plane refuses every further admission and + /// wakes a retry sleep on it, and no new detached task is accepted. Idempotent, and it frees, + /// nulls and swaps nothing: it can be called before any lock a teardown takes, so a GC round + /// holding such a lock is refused at its next request instead of being waited out. + void beginTeardown() noexcept; + bool teardownBegun() const noexcept; bool stopAndDrainDetachedWork(uint64_t deadline_ms); uint64_t detachedWorkInFlight() const; uint64_t detachedWorkInFlightForTest() const { return detachedWorkInFlight(); } @@ -1140,6 +1146,20 @@ class Pool : public std::enable_shared_from_this return [this](uint64_t ms) { mount_runtime.sleepInterruptibly(ms); }; } + /// The open plane's inter-attempt sleep: woken by `beginTeardown`, so a retry backing off on the + /// GC plane cannot hold a teardown for a whole capped backoff. A predicate wait, so the detached + /// tasks' own completions -- which notify the same variable -- do not cut a sleep short. Named + /// for the same reason as `mountPlaneSleepFn`: the test seam has to be able to put it back. + std::function openPlaneSleepFn() + { + return [this](uint64_t ms) + { + std::unique_lock lock(detached_work->mutex); + detached_work->cv.wait_for(lock, std::chrono::milliseconds(ms), + [this] { return detached_work->stopping.load(std::memory_order_acquire); }); + }; + } + BackendPtr pool_backend; PoolConfig config; PoolMeta meta; diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 310051d792ea..676473afe71e 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -1729,7 +1729,13 @@ class CountingBackend : public DB::Cas::InMemoryBackend const size_t chunk = stream_chunk.load(); if (!opened || chunk == 0) return opened; - return std::make_unique(std::move(opened), chunk, largestChunkSlot(key)); + auto chunked = std::make_unique(std::move(opened), chunk, largestChunkSlot(key)); + /// Hand back a buffer whose FIRST window is already loaded, as a network-backed store does: + /// `ObjectStorageBackend::stream` forces that GET so the open's own attempt is what pays for + /// it. A fixture that returned an empty buffer would let a consumer which drops the preloaded + /// window -- and so silently loses the head of every streamed body -- pass its tests. + chunked->nextIfAtEnd(); + return chunked; } /// Serve every stream opened from now on in windows of at most `bytes`, as a network-backed store diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index 63c672377d90..0812137a87df 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -28,6 +29,7 @@ namespace DB::ErrorCodes { extern const int BAD_ARGUMENTS; + extern const int CORRUPTED_DATA; extern const int NETWORK_ERROR; } @@ -738,3 +740,86 @@ TEST(CASGCHealth, ReflectsLeadershipAndPendingReclaim) EXPECT_EQ(h1.wedged_namespace_count, 0u); EXPECT_LT(h1.last_success_age_seconds, 60u); } + +namespace +{ + +/// A backend that arms the pool's teardown the moment a chosen key has been read -- after the read +/// returned, before the round can act on it -- so the arm lands mid-round at a known point. +class ArmAfterReadBackend : public InMemoryBackend +{ +public: + using InMemoryBackend::read; + + std::optional read(const String & key, TransportAccess & access) override + { + auto result = InMemoryBackend::read(key, access); + if (key == arm_key && on_read) + on_read(); + return result; + } + + String arm_key; + std::function on_read; +}; + +} + +/// `Stopped` is a transient failure observed after the pool's teardown began -- a correlation the row +/// records honestly. The arm lands right after the lease read; the round's next request is refused by +/// the open plane's fence, which the engine reports like any lost fence (a transient code). +TEST(CASGCLog, TransientFailureAfterTeardownBeganIsStopped) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + store->setCasRetrySleepForTest([](uint64_t) {}); + std::vector rows; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) { rows.push_back(r); }); + + backend->arm_key = store->layout().gcStateKey(); + backend->on_read = [&store] { store->beginTeardown(); }; + EXPECT_THROW(sched.runOneRoundNow(Rec::Trigger::Manual), DB::Exception); + + const std::vector round_rows = roundRowsOnly(rows); + ASSERT_EQ(round_rows.size(), 2u); + EXPECT_EQ(round_rows[1].event_type, Rec::EventType::Finish); + EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Stopped) + << "a transient refusal after the arm is the teardown cutting the round short, not an incident"; + EXPECT_EQ(round_rows[1].error_code, DB::ErrorCodes::NETWORK_ERROR); + EXPECT_FALSE(round_rows[1].error.empty()); +} + +/// The rule is fail-closed: a non-transient failure that coincides with the arm stays `Failed`. An +/// undecodable `gc/state` throws `CORRUPTED_DATA` out of the lease phase after the very read that arms. +TEST(CASGCLog, NonTransientFailureCoincidingWithTeardownStaysFailed) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + store->setCasRetrySleepForTest([](uint64_t) {}); + std::vector rows; + DB::Cas::CasGcScheduler sched( + store, std::chrono::seconds(1), "test::gc", "ca", + [&](const Rec & r) { rows.push_back(r); }); + + { + /// `gc/state` does not exist until a round writes it, so the undecodable value is planted, + /// not substituted: the lease phase's own decode is what must fail. + DB::Cas::tests::OperationForTest raw_op(*backend); + const auto current = (*raw_op).read(store->layout().gcStateKey(), Retry::once()); + const WriteResult planted = current + ? (*raw_op).replace(store->layout().gcStateKey(), "not a gc state", current->etag, Retry::once()) + : (*raw_op).create(store->layout().gcStateKey(), "not a gc state", Retry::once()); + ASSERT_TRUE(std::holds_alternative(planted)); + } + backend->arm_key = store->layout().gcStateKey(); + backend->on_read = [&store] { store->beginTeardown(); }; + EXPECT_THROW(sched.runOneRoundNow(Rec::Trigger::Manual), DB::Exception); + + const std::vector round_rows = roundRowsOnly(rows); + ASSERT_EQ(round_rows.size(), 2u); + EXPECT_EQ(round_rows[1].outcome, Rec::Outcome::Failed) + << "a bug that coincides with a restart is not masked as Stopped"; + EXPECT_EQ(round_rows[1].error_code, DB::ErrorCodes::CORRUPTED_DATA); +} diff --git a/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp b/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp new file mode 100644 index 000000000000..b15931e3221d --- /dev/null +++ b/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp @@ -0,0 +1,454 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/// A disk's teardown must not wait out a GC round. The pool's teardown flag is the open request +/// plane's fence, so a round in flight is refused at its next request, its next retry sleep or its +/// next streamed refill; the joins stay and the round becomes short. These tests pin the arm, the +/// plane wiring, the sleep wiring, and the scheduler's behaviour around a round that was cut. + +namespace DB::ErrorCodes +{ +extern const int NETWORK_ERROR; +} + +namespace CurrentMetrics +{ +extern const Metric LocalThread; +extern const Metric LocalThreadActive; +extern const Metric LocalThreadScheduled; +} + +using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::expectThrowsCode; + +namespace +{ + +PoolPtr openPlainPool(const std::shared_ptr & backend, PoolConfig config = {}) +{ + config.pool_prefix = "p"; + config.server_root_id = "test"; + return Pool::open(backend, config); +} + +/// A gate a test opens explicitly, so a thread can be held in flight without a sleep. Bounded, and it +/// names what it waited on: an unbounded wait on a premise that stopped holding hangs the binary. +struct Gate +{ + void wait(std::string_view name) + { + std::unique_lock lock(m); + if (!cv.wait_for(lock, std::chrono::seconds(60), [this] { return open_; })) + ADD_FAILURE() << "timed out waiting for '" << name << "'"; + } + void open() + { + std::lock_guard lock(m); + open_ = true; + cv.notify_all(); + } + std::mutex m; + std::condition_variable cv; + bool open_ = false; +}; + +/// Opens its gate on every exit from the scope, so a failing assertion cannot strand the thread +/// parked behind it. +struct GateOpenedOnExit +{ + explicit GateOpenedOnExit(Gate & gate_) : gate(gate_) {} + GateOpenedOnExit(const GateOpenedOnExit &) = delete; + GateOpenedOnExit & operator=(const GateOpenedOnExit &) = delete; + ~GateOpenedOnExit() { gate.open(); } + Gate & gate; +}; + +/// A real storage over a fresh local object storage, `context == nullptr`: no system log, no +/// scheduler until the first GC entry point creates one. +std::shared_ptr openTestStorage() +{ + static std::atomic counter{0}; + const auto scratch = std::filesystem::temp_directory_path() + / ("cas_gc_teardown_stop_scratch_" + std::to_string(::getpid()) + "_" + std::to_string(counter.fetch_add(1))); + auto settings = DB::Cas::tests::makeSettingsForTest("test", scratch); + auto storage = std::make_shared( + DB::Cas::tests::makeLocalObjectStorageForTest(), "pool", "srv1", "", nullptr, settings); + storage->startup(); + return storage; +} + +/// Completes the first read of `key`, then withholds its return until released, so the round that +/// issued it is parked with that request already accounted at the backend. +class ParkFirstReadBackend : public CountingBackend +{ +public: + void armParkFirstRead(String key_, std::shared_ptr entered_, std::shared_ptr release_) + { + key = std::move(key_); + entered = std::move(entered_); + release = std::move(release_); + armed.store(true); + } + + std::optional read(const String & read_key, TransportAccess & access) override + { + auto result = CountingBackend::read(read_key, access); + if (read_key == key && armed.exchange(false)) + { + entered->open(); + release->wait("release"); + } + return result; + } + + uint64_t requestsTotal() const { return getTotal() + headTotal() + listTotal() + writeTotal(); } + +private: + String key; + std::shared_ptr entered; + std::shared_ptr release; + std::atomic armed{false}; +}; + +/// A thread-safe sink for the scheduler's rows, with a wait that never sleeps. +class RoundLogSink +{ +public: + GcRoundLogger logger() + { + return [this](const GcRoundLogRecord & r) + { + std::lock_guard lock(mutex); + records.push_back(r); + cv.notify_all(); + }; + } + + std::vector all() + { + std::lock_guard lock(mutex); + return records; + } + + /// The first Finish row at index >= `from`, waiting up to `timeout`; nullopt on timeout. + std::optional waitForFinish(size_t from, std::chrono::milliseconds timeout) + { + std::unique_lock lock(mutex); + const auto is_finish = [&] + { + for (size_t i = from; i < records.size(); ++i) + if (records[i].event_type == GcRoundLogRecord::EventType::Finish) + return true; + return false; + }; + if (!cv.wait_for(lock, timeout, is_finish)) + return std::nullopt; + for (size_t i = from; i < records.size(); ++i) + if (records[i].event_type == GcRoundLogRecord::EventType::Finish) + return records[i]; + return std::nullopt; + } + +private: + std::mutex mutex; + std::condition_variable cv; + std::vector records; +}; + +size_t countStarts(const std::vector & rows) +{ + size_t n = 0; + for (const auto & r : rows) + if (r.event_type == GcRoundLogRecord::EventType::Start) + ++n; + return n; +} + +} + +/// The arm is idempotent, observable, and closes the door to new detached work; a drain after an +/// early arm has nothing to wait for. +TEST(CASGCTeardownStop, BeginTeardownIsIdempotentAndRefusesNewDetachedWork) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + + EXPECT_FALSE(store->teardownBegun()); + store->beginTeardown(); + EXPECT_TRUE(store->teardownBegun()); + store->beginTeardown(); + EXPECT_TRUE(store->teardownBegun()) << "a second arm changes nothing"; + EXPECT_TRUE(store->detachedWorkStoppingForTest()); + + EXPECT_FALSE(store->tryDispatchDetached([](DetachedStopToken) {})) + << "no detached task is accepted once teardown began"; + EXPECT_TRUE(store->stopAndDrainDetachedWork(/*deadline_ms=*/1000)) + << "the drain after an early arm finds nothing in flight and returns at once"; +} + +/// The open plane -- GC, FSCK, the probe -- refuses after the arm, before anything reaches the +/// backend; the mount plane, which the ref-lane drain and the farewell need alive, does not. +TEST(CASGCTeardownStop, OpenPlaneRefusesAfterTeardownBeganAndTheMountPlaneDoesNot) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + { + CasOperation op = store->openRequests().admit(); + orThrow(op.create("p/probe", "v", Retry::once()), "create"); + } + backend->resetCounts(); + + store->beginTeardown(); + + CasOperation refused = store->openRequests().admit(); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)refused.read("p/probe", Retry::standard()); }); + CasOperation resumed = store->openRequests().resume(/*admitted_generation=*/0); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)resumed.read("p/probe", Retry::standard()); }); + EXPECT_EQ(backend->getTotal(), 0u) << "a refused admission never reaches the backend"; + + CasOperation mount = store->mountRequests().admit(); + ASSERT_TRUE(mount.read("p/probe", Retry::once()).has_value()) + << "the mount plane is not the open plane: teardown's own drain and farewell run on it"; + EXPECT_EQ(backend->getTotal(), 1u); +} + +/// The open plane's sleep is the interruptible one, in production wiring and after the test seam is +/// cleared. Arming FIRST makes this a wiring test: a predicate `wait_for` whose predicate already +/// holds returns without waiting, so a plane still wired to the plain sleep cannot pass. The deadline +/// is the assertion; no sleep orders any thread. +TEST(CASGCTeardownStop, OpenPlaneSleepReturnsAtOnceOnceTeardownBegan) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + store->beginTeardown(); + + auto paused = std::async(std::launch::async, [&store] { store->openRequests().pause(60'000); }); + EXPECT_EQ(paused.wait_for(std::chrono::seconds(10)), std::future_status::ready) + << "the open plane's sleep must observe the arm; a plain sleep holds for the full minute"; + + /// Clearing the seam must put the interruptible sleep back, not the engine's plain one. + store->setCasRetrySleepForTest([](uint64_t) {}); + store->setCasRetrySleepForTest({}); + auto paused_again = std::async(std::launch::async, [&store] { store->openRequests().pause(60'000); }); + EXPECT_EQ(paused_again.wait_for(std::chrono::seconds(10)), std::future_status::ready) + << "resetting the retry-sleep seam left the open plane on the plain sleep"; +} + +/// A read-ahead worker resumes under the plane's generation and is refused at its first gate; the +/// fold learns it at the take site. An unconsumed future has its exception dropped by the read-ahead's +/// destructor, so the test consumes it. +TEST(CASGCTeardownStop, ReadAheadWorkerIsRefusedAndTheTakeSiteSeesIt) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + { + CasOperation op = store->openRequests().admit(); + orThrow(op.create("p/k1", "one", Retry::once()), "create"); + } + ThreadPool pool{CurrentMetrics::LocalThread, CurrentMetrics::LocalThreadActive, + CurrentMetrics::LocalThreadScheduled, + /*max_threads*/ 2, /*max_free_threads*/ 2, /*queue_size*/ 0}; + CasOperation op = store->openRequests().admit(); + GcReadAhead reads(op, store->openRequests(), pool, /*concurrency=*/2); + + store->beginTeardown(); + backend->resetCounts(); + reads.hintRead("p/k1"); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)reads.takeRead("p/k1"); }); + EXPECT_EQ(backend->getTotal(), 0u) << "the worker was refused before it reached the backend"; +} + +/// The defect itself: `shutdown` waits behind `gc_scheduler_mutex`, which a synchronous round holds +/// for its whole duration. After the fix it arms the pool first, the parked round is refused at its +/// next request, and `shutdown` returns. On the old code the arm never lands while the round is +/// parked, which is the assertion that goes red. +TEST(CASGCTeardownStop, ShutdownReturnsWhileASynchronousRoundIsParked) +{ + auto storage = openTestStorage(); + auto pool = storage->poolForTest(); + ASSERT_TRUE(pool); + + auto parked = std::make_shared(); + auto release = std::make_shared(); + GateOpenedOnExit opener(*release); + std::mutex rows_mutex; + std::vector rows; + storage->setGcRoundRowHookForTest([&](const GcRoundLogRecord & r) + { + { + std::lock_guard lock(rows_mutex); + rows.push_back(r); + } + /// Park on the `lease` phase row: the round holds `gc_scheduler_mutex` and has more + /// requests ahead of it. + if (r.event_type == GcRoundLogRecord::EventType::Phase && r.phase == "lease") + { + parked->open(); + release->wait("release"); + } + }); + + auto round = std::async(std::launch::async, [&storage] { storage->runOneGcRoundForTest(); }); + parked->wait("parked"); + + auto done = std::async(std::launch::async, [&storage] { storage->shutdown(); }); + + /// The state handshake: the arm must land while the round is still parked. Bounded, and its + /// expiry is the failure on the old code. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!pool->teardownBegun() && std::chrono::steady_clock::now() < deadline) + std::this_thread::yield(); + EXPECT_TRUE(pool->teardownBegun()) << "shutdown waited for the round instead of arming the pool first"; + + release->open(); + EXPECT_THROW(round.get(), DB::Exception) << "the released round must be refused at its next request"; + EXPECT_EQ(done.wait_for(std::chrono::seconds(30)), std::future_status::ready); + + std::optional finish; + { + std::lock_guard lock(rows_mutex); + for (const auto & r : rows) + if (r.event_type == GcRoundLogRecord::EventType::Finish) + finish = r; + } + ASSERT_TRUE(finish.has_value()); + EXPECT_EQ(finish->outcome, GcRoundLogRecord::Outcome::Stopped); +} + +/// A background round parked inside a request is refused at its NEXT request: nothing new reaches +/// the backend after the arm, the Finish row is `Stopped`, and `stop` returns with nothing in flight. +TEST(CASGCTeardownStop, BackgroundRoundIsCutAtItsNextRequest) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + RoundLogSink sink; + CasGcScheduler sched(store, std::chrono::seconds(1), "test::gc", "ca", sink.logger()); + + auto entered = std::make_shared(); + auto release = std::make_shared(); + GateOpenedOnExit opener(*release); + backend->armParkFirstRead(store->layout().gcStateKey(), entered, release); + sched.start(); + entered->wait("entered"); + + store->beginTeardown(); + const uint64_t requests_at_arm = backend->requestsTotal(); + release->open(); + + const auto finish = sink.waitForFinish(/*from=*/0, std::chrono::seconds(30)); + ASSERT_TRUE(finish.has_value()) << "the parked round never finished"; + EXPECT_EQ(finish->outcome, GcRoundLogRecord::Outcome::Stopped); + sched.stop(); + EXPECT_TRUE(sched.isQuiescent()); + EXPECT_EQ(backend->requestsTotal(), requests_at_arm) + << "after the arm no request may reach the backend: the round unwinds at the next gate"; + EXPECT_EQ(countStarts(sink.all()), 1u) << "no further round started after the arm"; +} + +/// The tick queued behind a manual round must not mint a Start row after the arm: it checks the +/// flag once it holds the round mutex, before it logs anything. +TEST(CASGCTeardownStop, AQueuedScheduledTickEmitsNoStartAfterTeardownBegan) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + RoundLogSink sink; + /// An hour-long interval: the loop ticks only when asked. + CasGcScheduler sched(store, std::chrono::seconds(3600), "test::gc", "ca", sink.logger()); + sched.start(); + + auto entered = std::make_shared(); + auto release = std::make_shared(); + GateOpenedOnExit opener(*release); + backend->armParkFirstRead(store->layout().gcStateKey(), entered, release); + auto manual = std::async(std::launch::async, + [&sched] { return sched.runOneRoundNow(GcRoundLogRecord::Trigger::Manual); }); + entered->wait("entered"); + + /// The loop wakes and queues on the round mutex behind the parked manual round. + sched.requestRoundSoon(); + store->beginTeardown(); + release->open(); + + EXPECT_THROW((void)manual.get(), DB::Exception); + sched.stop(); + const auto rows = sink.all(); + EXPECT_EQ(countStarts(rows), 1u) << "the queued tick minted a Start row after teardown began"; +} + +namespace +{ + +/// Arms the pool's teardown the first time a chosen prefix is listed, so the stop lands inside the +/// namespace janitor's page rather than in the round's own request chain. +class ArmOnJanitorListBackend : public CountingBackend +{ +public: + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + auto page = CountingBackend::list(prefix, cursor, limit, access); + if (!arm_prefix.empty() && prefix == arm_prefix && on_list) + { + on_list(); + on_list = {}; + } + return page; + } + + String arm_prefix; + std::function on_list; +}; + +} + +/// A stop that lands inside advisory work the round swallows is not `Stopped`: the deferred path +/// runs the namespace janitor's page and returns normally, and the janitor turns a refused request +/// into an anomaly. The row is `Deferred`; the round did finish. Pinned so a later change to this +/// behaviour is made on purpose. +TEST(CASGCTeardownStop, AStopInsideTheJanitorPageIsSwallowedAsDeferred) +{ + auto backend = std::make_shared(); + auto store = DB::Cas::tests::openPoolForTest(backend); + const RootNamespace ns{"00/aa@cas@"}; + const ManifestRef r{.writer_epoch = 1, .build_sequence = 1, .manifest_ordinal = 0xAA}; + DB::Cas::tests::writeBlobBody(*backend, store->layout(), DB::UInt128(1)); + DB::Cas::tests::writeManifestRaw(*backend, store->layout(), ns, r, + {DB::Cas::tests::blobEntryFor("a", DB::UInt128(1))}); + DB::Cas::tests::publishCommittedTransition(*backend, store->layout(), ns, "tbl", std::nullopt, r); + + Gc gc(store, DB::UInt128(0xAB)); + const RoundReport fold_rep = gc.runRegularRound(); + ASSERT_FALSE(fold_rep.deferred) << "the first round folds"; + + backend->arm_prefix = store->layout().namespaceRootPrefix(); + backend->on_list = [&store] { store->beginTeardown(); }; + RoundReport rep; + EXPECT_NO_THROW(rep = gc.runRegularRound()) << "the janitor page swallows the refusal"; + EXPECT_TRUE(store->teardownBegun()) << "sanity: the arm landed inside the round"; + EXPECT_TRUE(rep.deferred) << "an idle second round defers; the stop inside its janitor page is advisory"; +} diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index 90d794d95e17..a14cf0debae8 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -10,11 +10,14 @@ #include #include "cas_test_helpers.h" +#include + #include "config.h" #include #include +#include #include #include #include @@ -1814,3 +1817,132 @@ TEST(CASRequests, AWriteReservesTwoEnvelopesSoOneOfSurplusStartsNothing) EXPECT_TRUE(clock.sleeps.empty()); } + +/// The body of a streamed object is read at the consumer's pace, long after the opening attempt +/// returned; the wrapper re-admits it at every refill. The window the open already loaded is served +/// first -- the SDK buffer arrives with pending data -- and the check first fires on advancing past it. +TEST(CASRequests, StreamBodyKeepsThePreloadedWindowAndRefusesOnTheNextRefill) +{ + FakeClock clock; + auto backend = std::make_shared(); + std::atomic torn_down{false}; + Fence fence{[] { return uint64_t{0}; }, + [&](uint64_t, uint64_t) { return torn_down.load() ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + auto op = requests.admit(); + orThrow(op.create("k", "0123456789", Retry::once()), "create"); + backend->setStreamChunkForTest(4); /// the body arrives as "0123", "4567", "89" + + auto body = op.stream("k", Retry::once()); + ASSERT_TRUE(body); + String first(4, '\0'); + body->readStrict(first.data(), 4); + EXPECT_EQ(first, "0123") << "the window the open already loaded is served, not skipped"; + + torn_down.store(true); + char c; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { body->readStrict(&c, 1); }); + EXPECT_TRUE(body->isCanceled()) << "a refused refill leaves the buffer the consumer holds unusable"; +} + +TEST(CASRequests, StreamBodyServesEveryWindowThenEof) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "0123456789", Retry::once()), "create"); + backend->setStreamChunkForTest(4); + + auto body = op.stream("k", Retry::once()); + ASSERT_TRUE(body); + String all; + DB::readStringUntilEOF(all, *body); + EXPECT_EQ(all, "0123456789"); + EXPECT_TRUE(body->eof()); + EXPECT_FALSE(op.stream("absent", Retry::once())) << "an absent object is still the open's answer"; +} + +/// The mount plane's fence can answer `NoBudget`; a body refused for that reason must read like a +/// refused open on the same plane -- the retry-later class, not a tripped fence. +TEST(CASRequests, StreamBodyRefusalKeepsTheNoBudgetMapping) +{ + FakeClock clock; + auto backend = std::make_shared(); + std::atomic out_of_budget{false}; + Fence fence{[] { return uint64_t{0}; }, + [&](uint64_t, uint64_t) { return out_of_budget.load() ? Fence::Admit::NoBudget : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + auto op = requests.admit(); + orThrow(op.create("k", "0123456789", Retry::once()), "create"); + backend->setStreamChunkForTest(4); + + auto body = op.stream("k", Retry::once()); + ASSERT_TRUE(body); + String first(4, '\0'); + body->readStrict(first.data(), 4); + out_of_budget.store(true); + char c; + try + { + body->readStrict(&c, 1); + FAIL() << "the refill must be refused"; + } + catch (const DB::Exception & e) + { + EXPECT_NE(e.message().find("no lease budget"), String::npos) << e.message(); + } +} + +/// The caller's liveness is the second half of admission for the body too, in the gate's order. +TEST(CASRequests, StreamBodyHonoursTheCallersLiveness) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + std::atomic alive{true}; + auto op = requests.admit([&] { return alive.load(); }); + orThrow(op.create("k", "0123456789", Retry::once()), "create"); + backend->setStreamChunkForTest(4); + + auto body = op.stream("k", Retry::once()); + ASSERT_TRUE(body); + String first(4, '\0'); + body->readStrict(first.data(), 4); + alive.store(false); + char c; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { body->readStrict(&c, 1); }); +} + +/// The window the open already loaded is served WITHOUT a further admission: `Backend::stream` forces +/// that first GET and accounts it to the open's own attempt, so re-checking it here would refuse +/// bytes the caller has already paid for. The refusal belongs to the SECOND window, the first one the +/// body actually asks the store for. Armed before the first read, so a wrapper that discarded the +/// adopted window would refuse immediately instead of serving it. +TEST(CASRequests, StreamBodyServesTheAdoptedWindowEvenWhenAdmissionIsAlreadyRefused) +{ + FakeClock clock; + auto backend = std::make_shared(); + std::atomic torn_down{false}; + Fence fence{[] { return uint64_t{0}; }, + [&](uint64_t, uint64_t) { return torn_down.load() ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + auto op = requests.admit(); + orThrow(op.create("k", "0123456789", Retry::once()), "create"); + backend->setStreamChunkForTest(4); + + auto body = op.stream("k", Retry::once()); + ASSERT_TRUE(body); + torn_down.store(true); /// refused BEFORE the consumer touches the body + + String first(4, '\0'); + body->readStrict(first.data(), 4); + EXPECT_EQ(first, "0123") << "the window the open already paid for must be served, not re-admitted"; + EXPECT_EQ(body->count(), 4u) << "the adopted window is counted once, by the wrapper"; + + char c; + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { body->readStrict(&c, 1); }); +} diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp index 2438cab2801f..f71d8495dd03 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp @@ -21,7 +21,8 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri auto outcome_enum = std::make_shared(DataTypeEnum8::Values{ {"Unknown", static_cast(UNKNOWN)}, {"Success", static_cast(SUCCESS)}, {"NotALeader", static_cast(NOT_A_LEADER)}, {"Error", static_cast(FAILED)}, - {"Deferred", static_cast(DEFERRED)}, {"Aborted", static_cast(ABORTED)}}); + {"Deferred", static_cast(DEFERRED)}, {"Aborted", static_cast(ABORTED)}, + {"Stopped", static_cast(STOPPED)}}); auto trigger_enum = std::make_shared(DataTypeEnum8::Values{ {"Scheduled", static_cast(SCHEDULED)}, {"Manual", static_cast(MANUAL)}}); auto lc_string = std::make_shared(std::make_shared()); @@ -38,7 +39,7 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"gc_id", std::make_shared(), "GC scheduler instance id (which mounter)."}, {"trigger", trigger_enum, "Scheduled (background tick) or Manual (SYSTEM command)."}, {"round", std::make_shared(), "GC round number (0 on Start)."}, - {"outcome", outcome_enum, "Unknown (Start) / Success (led, folded, and completed) / NotALeader (another replica holds the GC lease) / Deferred (led but took the skip-unchanged fast path -- no fold ran) / Aborted (the round threw a transient error -- backend unavailability, a lost lease, a concurrent leader -- and the next scheduled round retries) / Error (the round threw a non-transient error)."}, + {"outcome", outcome_enum, "Unknown (Start) / Success (led, folded, and completed) / NotALeader (another replica holds the GC lease) / Deferred (led but took the skip-unchanged fast path -- no fold ran) / Aborted (the round threw a transient error -- backend unavailability, a lost lease, a concurrent leader -- and the next scheduled round retries) / Stopped (a transient error observed after the disk\'s teardown began: the round was cut short by a server shutdown or the storage\'s destructor, so neither waited for it; a correlation, not a cause -- a transient incident that started before the teardown is recorded the same way, and decommission does not arm the flag at all) / Error (the round threw a non-transient error -- during a teardown too)."}, {"candidates_marked", std::make_shared(), "Objects retired (marked) this round."}, {"objects_deleted", std::make_shared(), "Objects physically deleted this round."}, {"objects_absent", std::make_shared(), "Retire candidates found already absent."}, @@ -51,8 +52,8 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"fence_outs", std::make_shared(), "Expired mounts fenced out by this round's heartbeat floor."}, {"anomalies", std::make_shared(), "Fold clamps surfaced (and survived) this round; steady >0 warrants a look at the round log details."}, {"duration_ms", std::make_shared(), "Round wall-clock duration (Finish)."}, - {"error", std::make_shared(), "Exception text when outcome = Aborted or Error."}, - {"error_code", std::make_shared(), "Exception code when outcome = Aborted or Error; 0 otherwise. The structured twin of `error`: key monitoring on this column, not on message text."}, + {"error", std::make_shared(), "Exception text when outcome = Aborted, Stopped or Error. On a Stopped row it names the engine\'s refusal, not the teardown."}, + {"error_code", std::make_shared(), "Exception code when outcome = Aborted, Stopped or Error; 0 otherwise. The structured twin of `error`: key monitoring on this column, not on message text."}, {"ProfileEvents", std::make_shared(lc_string, std::make_shared()), "On a Start/Finish row: the per-round ProfileEvents delta (the Cas* counters and S3 events for this round). On a Phase row: THAT PHASE's delta, so `GROUP BY phase` over `ProfileEvents['S3ListObjects']` attributes the round's LIST budget to the phase that spent it. Empty on the `meta_pool_wait` row by construction — that phase's work runs on other threads (read its `phase_metrics` instead)."}, {"round_id", std::make_shared(), diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.h b/src/Interpreters/ContentAddressedGarbageCollectionLog.h index b7ffdc734c75..65c231b76eac 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.h +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.h @@ -17,8 +17,12 @@ struct ContentAddressedGarbageCollectionLogElement /// tell a round that genuinely folded and found nothing apart from one that never folded at all. /// `ABORTED`: the round threw an exception whose code names a transient condition (backend /// unavailability, a lost lease, a concurrent leader); the next scheduled round retries it. + /// `STOPPED`: a transient failure observed after the disk's teardown began -- the round was cut + /// short by a server shutdown or the storage's destructor, so neither had to wait for it; + /// `error` carries the engine's refusal. Decommission does not arm that flag and cannot produce + /// this outcome. /// `FAILED` is everything else -- fail-closed, an unclassified error reads as real. - enum Outcome : int8_t { UNKNOWN = 1, SUCCESS = 2, NOT_A_LEADER = 3, FAILED = 4, DEFERRED = 5, ABORTED = 6 }; + enum Outcome : int8_t { UNKNOWN = 1, SUCCESS = 2, NOT_A_LEADER = 3, FAILED = 4, DEFERRED = 5, ABORTED = 6, STOPPED = 7 }; enum Trigger : int8_t { SCHEDULED = 1, MANUAL = 2 }; time_t event_time = 0; From 9912221450bd9b218748de57ad1601cc43228160 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:07:36 +0200 Subject: [PATCH 21/81] cas: GC round cost on write-once keys (mount-floor memo, late reads, bulk delete) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A 15-minute real-GCS soak measured a sweep round costing 300-617 s per phase, all from per-object request loops on keys that are write-once by construction — an object whose only writer mints it once and whose only other mutations are exact-token deletes, so nothing about it needs re-reading once known. Three loops dominated: `fold_reduce`'s `GET` volume (2870-3400 per round) turned out to be the sweep's mount-floor probing, not manifest bodies — `floorForNamespace` reads the mount key of every `/`-prefix of a namespace for every listed manifest, though the floor is one value per server root; `manifest_deletes` cost 617 s for 3250 sequential conditional deletes at ~190 ms each; and `ref_object_cleanup` cost 199-204 s for 512-516 keys at four requests each. The fix cuts each loop to what the write-once property actually allows: one mount-floor read per namespace per sweep page (memoized), manifest bodies read only for nominated orphans and through the existing read-ahead instead of on every listed key, and a new write-once bulk-delete verb (`removeManyWriteOnce`, backed by `DeleteObjects` where the store has it) replacing the sequential per-key deletes for owner-removed manifests and for ref-object cleanup, which now revalidates its cohorts before batching them. None of this changes what gets deleted or when a namespace or manifest is judged eligible — only how many requests that judgment costs. Measured on real GCS: `fold_reduce` 300-380 s -> 2-5 s; `manifest_deletes` 617 s -> 2 s on a 1506-key round; `ref_object_cleanup` 204 s -> under 1 s. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- .../en/operations/system-tables/cas_gc_log.md | 2 +- src/Common/ProfileEvents.cpp | 3 +- .../ContentAddressed/Backend/CasBackend.h | 8 + .../Backend/CasInMemoryBackend.cpp | 55 +++ .../Backend/CasInMemoryBackend.h | 16 + .../Backend/CasInstrumentedBackend.cpp | 10 + .../Backend/CasInstrumentedBackend.h | 6 + .../Backend/CasObjectStorageBackend.cpp | 32 +- .../Backend/CasObjectStorageBackend.h | 6 + .../ContentAddressed/Backend/CasRequests.cpp | 17 + .../ContentAddressed/Backend/CasRequests.h | 9 + .../Backend/CasThrottlingBackend.h | 7 + .../ContentAddressedMetadataStorage.cpp | 3 + .../ContentAddressedMetadataStorage.h | 2 + .../ContentAddressedSettings.cpp | 9 +- .../ContentAddressed/Formats/CasLayout.h | 20 + .../ContentAddressed/Gc/CasGc.cpp | 146 +++--- .../ContentAddressed/Gc/CasGcKeyReader.h | 25 ++ .../ContentAddressed/Gc/CasGcReadAhead.cpp | 23 + .../ContentAddressed/Gc/CasGcReadAhead.h | 21 +- .../Gc/CasOrphanManifestSweep.cpp | 193 ++++---- .../Gc/CasOrphanManifestSweep.h | 37 +- .../ContentAddressed/Pool/CasKeyReader.cpp | 32 ++ .../ContentAddressed/Pool/CasKeyReader.h | 55 +++ .../ContentAddressed/Pool/CasPool.h | 2 + .../ContentAddressed/Pool/CasRefProtocol.cpp | 13 +- .../ContentAddressed/Pool/CasRefProtocol.h | 6 +- .../Primitives/CasWriteOnceKey.h | 26 ++ .../ObjectStorages/IObjectStorage.cpp | 5 + .../ObjectStorages/IObjectStorage.h | 7 + .../ObjectStorages/S3/S3ObjectStorage.cpp | 74 ++++ .../ObjectStorages/S3/S3ObjectStorage.h | 6 + src/Disks/tests/cas_test_helpers.h | 10 + .../tests/gtest_cas_bulk_delete_backend.cpp | 196 +++++++++ .../tests/gtest_cas_bulk_delete_engine.cpp | 91 ++++ src/Disks/tests/gtest_cas_decommission.cpp | 1 + src/Disks/tests/gtest_cas_gc_key_reader.cpp | 134 ++++++ .../gtest_cas_gc_manifest_bulk_delete.cpp | 163 +++++++ src/Disks/tests/gtest_cas_mount.cpp | 1 + .../tests/gtest_cas_orphan_manifest_sweep.cpp | 12 +- .../tests/gtest_cas_orphan_nomination.cpp | 29 +- .../tests/gtest_cas_orphan_sweep_requests.cpp | 390 +++++++++++++++++ src/Disks/tests/gtest_cas_part_write.cpp | 5 + src/Disks/tests/gtest_cas_pool.cpp | 13 + src/Disks/tests/gtest_cas_ref_gc.cpp | 414 ++++++++++++++++-- src/Disks/tests/gtest_cas_settings.cpp | 22 + src/Disks/tests/gtest_cas_write_once_key.cpp | 32 ++ .../ContentAddressedGarbageCollectionLog.cpp | 2 +- .../test_cas_gc_bulk_delete/__init__.py | 0 .../configs/storage_conf.xml | 36 ++ .../test_cas_gc_bulk_delete/test.py | 130 ++++++ 51 files changed, 2363 insertions(+), 194 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcKeyReader.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.h create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasWriteOnceKey.h create mode 100644 src/Disks/tests/gtest_cas_bulk_delete_backend.cpp create mode 100644 src/Disks/tests/gtest_cas_bulk_delete_engine.cpp create mode 100644 src/Disks/tests/gtest_cas_gc_key_reader.cpp create mode 100644 src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp create mode 100644 src/Disks/tests/gtest_cas_orphan_sweep_requests.cpp create mode 100644 src/Disks/tests/gtest_cas_write_once_key.cpp create mode 100644 tests/integration/test_cas_gc_bulk_delete/__init__.py create mode 100644 tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml create mode 100644 tests/integration/test_cas_gc_bulk_delete/test.py diff --git a/docs/en/operations/system-tables/cas_gc_log.md b/docs/en/operations/system-tables/cas_gc_log.md index ebdce2038cdb..e92c2a6e3bd6 100644 --- a/docs/en/operations/system-tables/cas_gc_log.md +++ b/docs/en/operations/system-tables/cas_gc_log.md @@ -44,7 +44,7 @@ specified (it is enabled by default in the shipped `config.xml`). - `objects_absent` ([UInt64](/sql-reference/data-types/int-uint)) — Retire candidates found already absent. - `objects_replaced` ([UInt64](/sql-reference/data-types/int-uint)) — `412`-saves (a resurrection won the race against the delete). - `objects_spared` ([UInt64](/sql-reference/data-types/int-uint)) — Candidates spared because their in-degree was greater than zero at recheck. -- `manifests_deleted` ([UInt64](/sql-reference/data-types/int-uint)) — Owner-removed manifest bodies physically deleted this round, counted separately from blob deletes. +- `manifests_deleted` ([UInt64](/sql-reference/data-types/int-uint)) — Owner-removed manifest bodies deleted or found already absent this round (a batch delete of write-once keys cannot tell the two apart), counted separately from blob deletes. - `entries_condemned` ([UInt64](/sql-reference/data-types/int-uint)) — Retired entries newly condemned this round (retired-cursor pipeline stage 1). - `entries_graduated` ([UInt64](/sql-reference/data-types/int-uint)) — Retired entries newly floor-passed and republished `delete_pending` this round (pipeline stage 2; deleted the next round). - `entries_redeleted` ([UInt64](/sql-reference/data-types/int-uint)) — Pending exact-token blob deletes executed this round (pipeline stage 3). diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 5d65defbaad2..1d46e9ce0ef3 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -823,7 +823,7 @@ The server successfully detected this situation and will download merged part fr M(CASRefLogBodyGets, "Number of CAS ref-log bodies read and decoded during GC. Growth indicates more reference history to process.", ValueType::Number) \ M(CASRefManifestBodyFoldGets, "Number of manifest bodies read while GC follows reference edges. High values indicate cache misses or many referenced manifests.", ValueType::Number) \ M(CASRefEmittedEdges, "Number of reachability edges emitted while GC folds CAS reference history. Growth indicates more reference relationships to process.", ValueType::Number) \ - M(CASRefCleanupObjectsDeleted, "Number of old CAS ref logs and snapshots deleted after safe coverage was confirmed. Growth indicates cleanup progress.", ValueType::Number) \ + M(CASRefCleanupObjectsDeleted, "Number of old CAS ref logs and snapshots deleted after safe coverage was confirmed. Includes keys that were already absent, since a batch delete of write-once keys cannot tell the two apart. Growth indicates cleanup progress.", ValueType::Number) \ M(CASRefSnapshotPutBytes, "Total bytes written to CAS ref-table snapshots. A high value indicates frequent or large snapshot publication.", ValueType::Bytes) \ M(CASRefSnapshotTailLogs, "Number of CAS ref-log entries compacted into published snapshots. Growth indicates snapshot maintenance work.", ValueType::Number) \ M(CASRefSnapshotPublishDispatched, "Number of background CAS ref-table snapshot publications started. High values indicate frequent threshold or read-triggered publishing.", ValueType::Number) \ @@ -916,6 +916,7 @@ The server successfully detected this situation and will download merged part fr M(CASMetaResurrectClean, "Number of condemned-body replacement paths that entered Clean metadata reconciliation. Counts the reason entry, not a guaranteed metadata reset.", ValueType::Number) \ M(CASGCMetaOps, "Number of per-hash metadata operations executed by CAS GC. Growing values indicate more GC candidates or metadata work.", ValueType::Number) \ M(CASGCEnumerationPages, "Number of CAS GC LIST pages fetched while enumerating the object universe. Growing values indicate a larger universe or more frequent scans.", ValueType::Number) \ + M(CASBulkDeleteRequests, "Number of CAS batch delete requests: one DeleteObjects carrying up to 1000 write-once keys (manifest bodies, ref logs, ref snapshots). The per-key class counters (CASManifestDelete, CASRootDelete) say how many keys each request carried.", ValueType::Number) \ M(CASGCRefWalkPlansBuilt, "Number of complete catalog-authoritative CAS ref walk plans constructed by ordinary GC and rebuild. A regular or rebuilding invocation that reaches the post-LIST catalog cut increments this exactly once, including a round that later defers.", ValueType::Number) \ M(CASGCUnmatchedAdoptedParentLives, "Number of adopted-parent CAS ref-life rows dropped because the post-LIST catalog cut has no matching physical life. Each occurrence is inert for planning and suppression and is logged with its exact physical life id; a persistent nonzero rate indicates old generation state is outliving catalog removal.", ValueType::Number) \ M(CASGCStuckRemovals, "Number of adopted CAS GC rounds that observed a Removing namespace at or beyond the diagnostic age threshold without terminal cleanup evidence. Incremented and warned every such round; diagnostic only, with no effect on folding, suppression, appends, or deletion.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index 56ca4dffa772..cbfdb4595fb2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -193,6 +194,13 @@ class Backend /// visible. Transport only: it observes no destination state and produces no incarnation. virtual void publish(const BlobPublishRequest & request, TransportAccess &) = 0; + /// Removes up to 1000 WRITE-ONCE keys in one request with no per-key precondition; an absent key + /// is success. The caller proves the keys are write-once by minting them as `WriteOnceKey`, so a + /// backend never sees a mutable control key through this verb. Throws on any failure; the whole + /// chunk is reissued by the engine, which is sound because a key already deleted is absent, and + /// absence is success. + virtual void removeManyWriteOnce(const std::vector & keys, TransportAccess &) = 0; + /// Authoritative, cache-bypassing probe of one key -- see `ProbeOutcome`. DEFAULT (used by every /// backend without sharper raw-error evidence, e.g. `InMemoryBackend`): derived from `head`/`read` /// alone, so it can only distinguish `Present` from `KeyAbsent`, and ANY exception from either diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp index 6ebdae19d037..b6bf981a40f1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp @@ -289,6 +289,61 @@ Backend::RawRemoval InMemoryBackend::remove(const String & key, const String & e return applyDelete(key, expected_value); } +void InMemoryBackend::removeManyWriteOnce(const std::vector & keys, TransportAccess &) +{ + std::exception_ptr armed; + std::function hook; + { + std::lock_guard lock(mutex_); + ++bulk_remove_calls_; + if (!armed_bulk_remove_failures_.empty()) + { + armed = armed_bulk_remove_failures_.front(); + armed_bulk_remove_failures_.erase(armed_bulk_remove_failures_.begin()); + } + hook = before_bulk_remove_hook_; + } + if (armed) + std::rethrow_exception(armed); + if (hook) + hook(); + + std::lock_guard lock(mutex_); + for (const WriteOnceKey & key : keys) + { + auto it = store_.find(key.str()); + if (it == store_.end()) + continue; /// absent is success + if (hold_deletes_) + { + PendingDelete pd; + pd.key = key.str(); + pd.value = it->second.value; + pending_deletes_.push_back(std::move(pd)); + continue; + } + store_.erase(it); + } +} + +void InMemoryBackend::failNextBulkRemoveWith(std::exception_ptr error) +{ + std::lock_guard lock(mutex_); + armed_bulk_remove_failures_.push_back(std::move(error)); +} + +void InMemoryBackend::onBeforeBulkRemove(std::function hook) +{ + std::lock_guard lock(mutex_); + before_bulk_remove_hook_ = std::move(hook); +} + +size_t InMemoryBackend::bulkRemoveCalls() const +{ + std::lock_guard lock(mutex_); + return bulk_remove_calls_; +} + Backend::RawListPage InMemoryBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess &) { if (limit == 0) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h index a41d0dd4ab3d..2074750e3ef6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h @@ -46,6 +46,19 @@ class InMemoryBackend : public Backend /// but its expected value is rechecked when it is landed. RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override; + /// Deletes every present key with no precondition; an absent key is success. Honours + /// `hold_deletes_` exactly as `remove` does: a held delete is queued and lands only on + /// `landPendingDelete`. + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override; + /// The next `removeManyWriteOnce` throws `error` instead of deleting anything; one-shot, like + /// `failNextWriteWith`. + void failNextBulkRemoveWith(std::exception_ptr error); + /// Runs before a `removeManyWriteOnce` applies, with no backend lock held, on the attempt that + /// will delete (an armed failure fires first and skips the hook). + void onBeforeBulkRemove(std::function hook); + /// How many `removeManyWriteOnce` calls reached the store, armed failures included. + size_t bulkRemoveCalls() const; + /// Creates the key when `expected_value` is empty, or replaces the incarnation it names. A /// refused precondition leaves the store unchanged. Value enforcement can be disabled with /// `setEnforceTokens` to model a backend that incorrectly ignores the condition. @@ -207,6 +220,9 @@ class InMemoryBackend : public Backend ArmedFailures head_failures_; Hooks before_write_hooks_; Hooks write_committed_hooks_; + std::vector armed_bulk_remove_failures_; + std::function before_bulk_remove_hook_; + size_t bulk_remove_calls_ = 0; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp index 7f4f97cd1bb6..4782bd3f1011 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp @@ -15,6 +15,8 @@ extern const Event CASBlobGetStream; extern const Event CASBlobDelete; extern const Event CASBlobList; +extern const Event CASBulkDeleteRequests; + extern const Event CASManifestPut; extern const Event CASManifestPutDeduplicated; extern const Event CASManifestOverwrite; @@ -144,4 +146,12 @@ void InstrumentedBackend::publish(const BlobPublishRequest & request, TransportA incrementCasEvent(classifyCasNs(request.destination_key), CasOp::Put); } +void InstrumentedBackend::removeManyWriteOnce(const std::vector & keys, TransportAccess & access) +{ + inner->removeManyWriteOnce(keys, access); + ProfileEvents::increment(ProfileEvents::CASBulkDeleteRequests); + for (const WriteOnceKey & key : keys) + incrementCasEvent(classifyCasNs(key.str()), CasOp::Delete); +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h index c321bfa27fd5..d0c853d29d69 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h @@ -125,6 +125,12 @@ class InstrumentedBackend final : public Backend return outcome; } + /// Delegate the batch removal, count the request once, and count each key it named as a `Delete` + /// in its own namespace class -- the per-key counters say how many keys of each class one request + /// carried, the request counter says how many requests it took. Out of line, like `publish`, so + /// the header need not declare the `CASBulkDeleteRequests` extern. + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override; + /// Count a create and a replacement separately, and each of them separately from its refusal: /// they cost the same one request, but a pool whose creates are mostly refused and one whose /// replacements mostly conflict are different problems. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index 6f9530a4dee7..32d02506bfdf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -979,6 +979,12 @@ Backend::RawRemoval ObjectStorageBackend::removeUnder( return RawRemoval::Mismatch; object_storage->removeObjectIfExists(StoredObject(emuPath(key))); + emuForgetDeletedToken(key); + return RawRemoval::Removed; +} + +void ObjectStorageBackend::emuForgetDeletedToken(const String & key) +{ /// Keep the deleted incarnation's last-minted etag around ONLY while a same-mtime-quantum /// collision with an immediate recreate is still possible (emuMintToken) — once it is /// comfortably old, erase it so `emu_token_state` does not grow for the lifetime of the backend @@ -991,7 +997,31 @@ Backend::RawRemoval ObjectStorageBackend::removeUnder( else emu_token_expiry.push_back(EmuTokenExpiry{now_ns, key, it->second}); } - return RawRemoval::Removed; +} + +void ObjectStorageBackend::removeManyWriteOnce(const std::vector & keys, TransportAccess &) +{ + if (keys.empty()) + return; + if (mode == Mode::Native) + { + StoredObjects objects; + objects.reserve(keys.size()); + for (const WriteOnceKey & key : keys) + objects.emplace_back(key.str()); + /// `NOT_IMPLEMENTED` from a storage without a batch delete propagates -- fail-closed by construction. + object_storage->removeObjectsIfExistUnderProfile(objects, controlPlaneProfile(), attempt_timeout_ms); + return; + } + + std::lock_guard lock(emu_mutex); + for (const WriteOnceKey & key : keys) + { + if (!emuExists(key.str())) + continue; + object_storage->removeObjectIfExists(StoredObject(emuPath(key.str()))); + emuForgetDeletedToken(key.str()); + } } Backend::RawListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess &) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index e0a4d95f971f..5aad6b1940c4 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -94,6 +94,10 @@ class ObjectStorageBackend final : public Backend /// returning a write-response value. Streaming uses ordinary write settings; staged bytes require /// a native same-store copy. void publish(const BlobPublishRequest & request, TransportAccess & access) override; + /// Removes up to 1000 write-once keys in one request. Native mode issues one `DeleteObjects` + /// under the control-plane profile; `EmulatedSingleProcess` deletes each present key under the + /// emulation lock with the same token bookkeeping as the single-key delete. + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override; /// Native mints its store's own dialect (ETag or GCS generation); the emulated adapter mints its /// own values. Dialect dialect() const override { return mode == Mode::Native ? native_token_type : Dialect::Emulated; } @@ -264,6 +268,8 @@ class ObjectStorageBackend final : public Backend /// has been validated, and the rename keeps publication atomic. Takes `emu_mutex` itself (for the /// rename + token-state bump only); the caller must NOT hold it. void emuPublishBlobAtomically(const String & key, const String & envelope, ReadBuffer & payload, uint64_t payload_size); + /// Caller holds emu_mutex: the token bookkeeping after a delete at `key`. + void emuForgetDeletedToken(const String & key); /// Return the current emulated token for a key we just read/HEAD'd, reflecting its on-disk etag — /// does NOT advance the same-etag disambiguator (that only applies to a just-completed write). String emuObserveToken(const String & key); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 4398784bda11..5184659ccbe8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -543,6 +543,23 @@ void CasOperation::forEachListedKey(const String & prefix, const ListedKeyFn & f } } +void CasOperation::removeManyWriteOnce(const std::vector & keys, const Retry & policy) +{ + if (keys.size() > kBulkDeleteMaxKeys) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS removeManyWriteOnce: {} keys in one chunk, the limit is {}; the consumer chunks its input", + keys.size(), kBulkDeleteMaxKeys); + if (keys.empty()) + return; + const Retry::Bound bound = policy.bind(owner.now_ms()); + const String subject = fmt::format("{} (+{} keys)", keys.front().str(), keys.size() - 1); + readLoop("removeManyWriteOnce", subject, policy, bound, [&](auto & access) + { + owner.backend->removeManyWriteOnce(keys, access); + return true; + }); +} + Removal CasOperation::remove(const String & key, const Etag & seen, const Retry & policy) { return removeUnder(key, owner.valueFor(key, seen), policy, policy.bind(owner.now_ms())); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index b10979739c99..82b292e1bd16 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -196,6 +196,9 @@ class CasRequests uint64_t attempt_reservation_ms; }; +/// The cap on one `removeManyWriteOnce` chunk -- also the ceiling a batch-delete request can carry. +inline constexpr size_t kBulkDeleteMaxKeys = 1000; + /// One admitted operation: the unit a policy, a fence generation and a liveness predicate apply to. /// Move-only, and every request it makes re-checks its admission -- before each attempt, before each /// sleep, and once more after a proven commit, so a write whose fence was lost while it was in flight @@ -241,6 +244,12 @@ class CasOperation /// absent; never returns `Mismatch` -- under `once`, where there is no reissue to resolve one, a /// `Mismatch` is the retry-later throw the read verbs use when their policy is exhausted. Removal removeCurrent(const String & key, const Retry & policy); + /// Deletes ONE chunk of up to `kBulkDeleteMaxKeys` write-once keys as one request under the + /// policy: admission, fence, budget, deadline, backoff and reissue exactly as `remove`. A reissue + /// resends the whole chunk; a key the failed attempt already deleted is absent, and absence is + /// success. Throws when the policy is exhausted. More keys than the cap is a caller bug: the + /// consumer chunks, so that every chunk that succeeded is recorded before a later one can fail. + void removeManyWriteOnce(const std::vector & keys, const Retry & policy); /// The one primitive that reports failure as a value, so the policy reissues on the OUTCOME: /// `Indeterminate` is retried, the four authoritative outcomes return at once, and an /// `Indeterminate` that outlives the bound is returned rather than thrown. Admission refused before diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h index 6ac7860c276b..1153f0c78232 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h @@ -88,6 +88,13 @@ class ThrottlingBackend final : public Backend return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override + { + for (const WriteOnceKey & key : keys) + refuseOrPass(key.str()); + inner->removeManyWriteOnce(keys, access); + } + std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index 5bff526246df..06b7074cb89a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -82,6 +82,7 @@ namespace ContentAddressedSetting extern const ContentAddressedSettingsUInt64 manifest_decode_cache_bytes; extern const ContentAddressedSettingsUInt64 gc_meta_pool_size; extern const ContentAddressedSettingsUInt64 gc_read_concurrency; + extern const ContentAddressedSettingsUInt64 gc_bulk_delete_chunk_keys; extern const ContentAddressedSettingsUInt64 attempt_timeout_ms; extern const ContentAddressedSettingsUInt64 lease_safety_margin_ms; extern const ContentAddressedSettingsBool blob_hash_allow_new; @@ -302,6 +303,7 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage( , manifest_decode_cache_bytes(settings_[ContentAddressedSetting::manifest_decode_cache_bytes].value) , gc_meta_pool_size(settings_[ContentAddressedSetting::gc_meta_pool_size].value) , gc_read_concurrency(settings_[ContentAddressedSetting::gc_read_concurrency].value) + , gc_bulk_delete_chunk_keys(settings_[ContentAddressedSetting::gc_bulk_delete_chunk_keys].value) , cas_attempt_timeout_ms(settings_[ContentAddressedSetting::attempt_timeout_ms].value) , cas_lease_safety_margin_ms(settings_[ContentAddressedSetting::lease_safety_margin_ms].value) , staging_backend(settings_.stagingBackend()) @@ -775,6 +777,7 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.gc_round_outcome_entry_budget = gc_round_outcome_entry_budget; pool_config.gc_meta_pool_size = gc_meta_pool_size; pool_config.gc_read_concurrency = gc_read_concurrency; + pool_config.gc_bulk_delete_chunk_keys = gc_bulk_delete_chunk_keys; pool_config.cas_request_budget.attempt_timeout_ms = cas_attempt_timeout_ms; pool_config.cas_request_budget.lease_safety_margin_ms = cas_lease_safety_margin_ms; pool_config.event_sink = makeCasEventSink(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 8200bfd1c5d0..dc68eccd68a0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -621,6 +621,8 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC const uint64_t gc_meta_pool_size; /// Bounded pool size for the GC fold's read-ahead; 1 disables it. const uint64_t gc_read_concurrency; + /// Keys per batch delete request for the write-once families. + const uint64_t gc_bulk_delete_chunk_keys; /// The budget for one HTTP attempt of a writable Native mount's control-plane requests; feeds /// `Cas::PoolConfig::cas_request_budget.attempt_timeout_ms` and the backend's own /// `attemptTimeoutMs()`. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index 76efbfe790f1..e88cddaee7de 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -61,7 +61,7 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, gc_snapshot_generations_to_keep, 3, "GC snapshot generations retained", 0) \ DECLARE(UInt64, gc_shards, 1, "Blob-hash-prefix reducer shards (>= 1); creation-time only", 0) \ DECLARE(UInt64, manifest_sweep_list_budget_keys, 1000, "Orphan-manifest sweep LIST budget per round", 0) \ - DECLARE(UInt64, manifest_sweep_delete_budget_keys, 100, "Orphan-manifest sweep DELETE budget per round", 0) \ + DECLARE(UInt64, manifest_sweep_delete_budget_keys, 100, "Orphan-manifest sweep candidate budget per round: keys that pass every retain check, whose bodies are read and decided", 0) \ DECLARE(UInt64, gc_round_graduation_budget, 5000, "Blob graduation (condemned -> delete_pending) cohort cap per round (0 = unbounded)", 0) \ DECLARE(UInt64, gc_round_redelete_budget, 5000, "Blob redelete (exact-token delete of a prior delete_pending row) cohort cap per round (0 = unbounded)", 0) \ DECLARE(UInt64, gc_round_sweep_namespace_budget, 20, "Orphan-manifest sweep: distinct namespaces per page whose protection view may be built (0 = unbounded)", 0) \ @@ -77,6 +77,7 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, manifest_decode_cache_bytes, 128ULL << 20, "Manifest DECODE cache byte budget (0 disables)", 0) \ DECLARE(UInt64, gc_meta_pool_size, 16, "Bounded pool size for GC per-hash freshness-meta writes", 0) \ DECLARE(UInt64, gc_read_concurrency, 16, "Bounded pool size for the GC fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate HEADs; 1 disables read-ahead", 0) \ + DECLARE(UInt64, gc_bulk_delete_chunk_keys, 1000, "Keys per batch delete request in GC's write-once families (owner-removed manifest bodies, covered ref logs and snapshots); 1 to 1000", 0) \ DECLARE(UInt64, attempt_timeout_ms, 5000, "Budget for one HTTP attempt of a writable Native mount's control-plane requests", 0) \ DECLARE(UInt64, lease_safety_margin_ms, 2000, "Startup-only margin validated against the mount lease TTL (attempt_timeout_ms + this must be strictly less than the lease TTL)", 0) \ DECLARE(String, staging_backend, "local", "Blob staging backend (local | s3); s3 is opt-in", 0) \ @@ -232,6 +233,12 @@ void ContentAddressedSettings::validate() settings[ContentAddressedSetting::gc_interval_sec].value, settings[ContentAddressedSetting::gc_shards].value, settings[ContentAddressedSetting::gc_read_concurrency].value); + if (settings[ContentAddressedSetting::gc_bulk_delete_chunk_keys] == 0 + || settings[ContentAddressedSetting::gc_bulk_delete_chunk_keys] > 1000) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: gc_bulk_delete_chunk_keys must be between 1 and 1000 (got {})", + settings[ContentAddressedSetting::gc_bulk_delete_chunk_keys].value); + /// The layout subtree identity is explicit and REQUIRED — no default, so an ABSENT key throws a /// typed `NO_ELEMENTS_IN_CONFIG` (mirroring the `metadata_type` check in `MetadataStorageFactory`), /// distinct from a PRESENT-but-invalid value, which falls through to `validateServerRootId`'s diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h index b1c556c600a5..5ab57bb7854b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasLayout.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -168,6 +169,18 @@ class Layout return namespaceStreamPrefix(ns_id) + "_snap/" + renderRefTxnId(id) + String(storedSuffix(FormatId::RefSnapshot)); } + /// The write-once forms of `refLogKey` and `refSnapshotKey`: the same strings, typed as keys a + /// precondition-free delete may take. Both objects are published with a create-only write at a + /// life-qualified key. + WriteOnceKey writeOnceRefLogKey(const NamespaceLifeId & ns_id, const RefTxnId & id) const + { + return WriteOnceKey(refLogKey(ns_id, id)); + } + WriteOnceKey writeOnceRefSnapshotKey(const NamespaceLifeId & ns_id, const RefTxnId & id) const + { + return WriteOnceKey(refSnapshotKey(ns_id, id)); + } + /// The life's checkpoint object (spec INV-4) at `/cas/ns/state//_ckpt`. Unlike /// immutable stream objects it is mutable (token-CAS), carries no transaction id, and therefore lives /// in the point/path-addressed state tree rather than a `_log`/`_snap` directory -- @@ -273,6 +286,13 @@ class Layout + manifestOrdinalFileName(id.ref.manifest_ordinal); } + /// The write-once form of `manifestKey`: a manifest is published with a create-only write at a key + /// whose epoch, build sequence and ordinal never repeat under one server root. + WriteOnceKey writeOnceManifestKey(const ManifestId & id) const + { + return WriteOnceKey(manifestKey(id)); + } + /// Inverse of `manifestKey`: parses `/cas/manifests//-/.zst`. /// Strict: rejects the old decimal directory shape (it is not two fixed-width hex fields joined by /// '-'), a missing namespace/build/ordinal segment, trailing garbage, a file not ending in the diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index f364d8b9538b..5890d1d150ae 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -1079,6 +1079,7 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// never re-derived by this pipeline, converting a bounded burst into a permanent leak. It drains /// the whole of `folded.mf_cleanup` every round it runs; only a crash (or the destructive-suppression /// gate below) leaves an entry for the orphan-manifest sweep to reclaim later. + /// The bodies go in batch requests of write-once keys; see the block. /// /// PHASE 15/18 `manifest_deletes`. { @@ -1091,30 +1092,54 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al static const std::map kNoManifestCleanup; const std::map & mf_cleanup_now = suppress_destructive ? kNoManifestCleanup : folded.mf_cleanup; + + /// Chunks of write-once keys, one request each, with no per-key precondition: a manifest key is + /// never written twice, so the body at it is the one the fold observed or nothing. The engine + /// reissues a failed chunk whole; a chunk that exhausts its policy throws here, and the chunks + /// before it are already recorded below. The etag the fold observed rides the event as + /// information only. + const size_t chunk_keys = std::clamp(store->poolConfig().gc_bulk_delete_chunk_keys, 1, kBulkDeleteMaxKeys); uint64_t attempted = 0; - for (const auto & [id, incarnation] : mf_cleanup_now) + uint64_t requests = 0; + std::vector chunk; + std::vector *> chunk_entries; + const auto flush = [&] { - ++attempted; - /// Gone and Mismatch are both tolerated: the body is already reclaimed, or a live - /// incarnation replaced the one this round's fold observed. - const Removal mdel = op.remove(layout.manifestKey(id), incarnation, Retry::standard()); - if (mdel == Removal::Removed) - ++report.manifests_deleted; - EventEmitter{*store}.emit([&](CasEvent & e) + if (chunk.empty()) + return; + op.removeManyWriteOnce(chunk, Retry::standard()); + ++requests; + for (const auto * entry : chunk_entries) { - e.type = CasEventType::ManifestDelete; - e.namespace_ = id.root_namespace.string(); - e.object_kind = CasEventObjectKind::Manifest; - e.object_hash = manifestRefDebugString(id.ref); - e.token = incarnation.render(); - e.round = new_round; - e.gen = generation; - e.outcome = String{removalName(mdel)}; - e.reason = "owner-removed manifest body; exact-incarnation delete after decrements adopted"; - }); + ++report.manifests_deleted; + EventEmitter{*store}.emit([&](CasEvent & e) + { + e.type = CasEventType::ManifestDelete; + e.namespace_ = entry->first.root_namespace.string(); + e.object_kind = CasEventObjectKind::Manifest; + e.object_hash = manifestRefDebugString(entry->first.ref); + e.token = entry->second.render(); + e.round = new_round; + e.gen = generation; + e.outcome = "deleted_or_absent"; + e.reason = "owner-removed manifest body; batch delete of a write-once key after decrements adopted"; + }); + } + chunk.clear(); + chunk_entries.clear(); + }; + for (const auto & entry : mf_cleanup_now) + { + ++attempted; + chunk.push_back(layout.writeOnceManifestKey(entry.first)); + chunk_entries.push_back(&entry); + if (chunk.size() >= chunk_keys) + flush(); } + flush(); t.metric("attempted", attempted); - t.metric("deleted", report.manifests_deleted - manifests_deleted_before); + t.metric("accepted", report.manifests_deleted - manifests_deleted_before); + t.metric("requests", requests); t.metric("suppressed", suppress_destructive ? 1 : 0); } @@ -1180,6 +1205,8 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al t.metric("list_budget_keys", store->poolConfig().manifest_sweep_list_budget_keys); t.metric("suppressed", suppress_destructive ? 1 : 0); t.metric("listed", sweep.listed); + t.metric("floor_lookups", sweep.floor_lookups); + t.metric("floor_reads", sweep.floor_reads); t.metric("deleted", sweep.deleted); t.metric("skipped", sweep.skipped); t.metric("undecodable", sweep.undecodable); @@ -3216,7 +3243,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional & /*state_etag*/, /// cut and `_ckpt` frontier the round's own universe came from -- which is exactly what an /// authoritative universe means, and is why this is the gate's term and not a separate one. universe_authoritative, - &work_budget); + &work_budget, read_pool.get(), store->poolConfig().gc_read_concurrency); for (const ManifestSweepResult::Nomination & nomination : result.orphan_sweep.nominations) orphan_source_retirements.insert( orphan_source_retirements.end(), @@ -3524,18 +3551,16 @@ void Gc::cleanupRefObjects( const CatalogEntry & observed_entry = *entry_it; const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(entry_it->ns, entry_it->incarnation); - /// Current-life ref cleanup is not the dead-life janitor: every irreversible key delete must - /// still be licensed by the SAME complete catalog observation and GC lease that adopted the - /// fold. Re-read both after the target observation and immediately before the removal. A moved - /// incarnation, changed row/life, missing or unreadable authority object, or changed - /// owner/sequence stops the whole cleanup pass. Continuing with another row/key would turn a - /// refusal into a fallback. - const auto deleteRefObject = [&](const String & key) + /// Current-life ref cleanup is not the dead-life janitor: every irreversible delete is still + /// licensed by the SAME complete catalog observation and GC lease that adopted the fold, + /// re-read immediately before the deletes it licenses. The unit + /// of licence is one chunk of write-once keys: a `_log` or `_snap` key is published once at a + /// life-qualified key and never rewritten, so there is nothing to HEAD, and the plan below is + /// derived from durable state a successor leader derives too. A moved incarnation, changed + /// row/life, missing or unreadable authority object, or changed owner/sequence stops the whole + /// cleanup pass; continuing with another chunk would turn a refusal into a fallback. + const auto authorityHolds = [&](const String & first_key) -> bool { - const std::optional h = op.head(key, Retry::standard()); - if (!h) - return true; - try { const CasRefCatalog::Snapshot current_catalog = CasRefCatalog::read(op, layout); @@ -3551,8 +3576,8 @@ void Gc::cleanupRefObjects( || !current_life || *current_life != life) { LOG_DEBUG(logger, - "CAS GC ref cleanup stopped before deleting '{}': catalog observation/life moved", - key); + "CAS GC ref cleanup stopped before the chunk starting at '{}': catalog observation/life moved", + first_key); return false; } @@ -3560,8 +3585,8 @@ void Gc::cleanupRefObjects( if (!current_state_object) { LOG_WARNING(logger, - "CAS GC ref cleanup stopped before deleting '{}': mandatory gc/state is absent", - key); + "CAS GC ref cleanup stopped before the chunk starting at '{}': mandatory gc/state is absent", + first_key); return false; } const GcState current_state = decodeGcState(current_state_object->bytes); @@ -3569,21 +3594,17 @@ void Gc::cleanupRefObjects( || current_state.lease.seq != adopted_lease.seq) { LOG_DEBUG(logger, - "CAS GC ref cleanup stopped before deleting '{}': GC fence moved", - key); + "CAS GC ref cleanup stopped before the chunk starting at '{}': GC fence moved", first_key); return false; } } catch (const std::exception & e) { LOG_WARNING(logger, - "CAS GC ref cleanup stopped before deleting '{}': authority revalidation failed: {}", - key, e.what()); + "CAS GC ref cleanup stopped before the chunk starting at '{}': authority revalidation failed: {}", + first_key, e.what()); return false; } - - op.remove(key, h->etag, Retry::standard()); - ProfileEvents::increment(ProfileEvents::CASRefCleanupObjectsDeleted); /// cleanup object deletion return true; }; @@ -3618,30 +3639,39 @@ void Gc::cleanupRefObjects( const RefCleanupPlan plan = planRefCleanup( listing, durable_cursor, checkpoint_snapshot_id, retained_log_proof); + std::vector cohort; + cohort.reserve(plan.deletable_logs.size() + plan.deletable_snapshots.size()); for (const RefTxnId & log_id : plan.deletable_logs) - { - /// Cumulative per-round cap, never amortized against the per-key fail-close - /// validation `deleteRefObject` performs (HEAD + catalog re-read + gc/state re-read before - /// every exact delete stays exactly as expensive per key as before). Exhaustion simply stops - /// the round's cleanup pass here; `planRefCleanup` recomputes the SAME remaining candidates - /// from durable state next round, so nothing here needs its own cursor. - if (!work_budget.refCleanupAvailable()) - return; - if (!deleteRefObject(layout.refLogKey(life, log_id))) - return; - ++work_budget.ref_cleanup_objects_used; - } + cohort.push_back(layout.writeOnceRefLogKey(life, log_id)); for (const RefTxnId & snap_id : plan.deletable_snapshots) { - /// Task 5's rule, asserted where it is acted on rather than only where it is computed: the - /// snapshot the checkpoint names is the one a recovering reader will sample, so it must + /// The snapshot the checkpoint names is the one a recovering reader will sample, so it must /// survive every cleanup that the same checkpoint authorized. chassert(snap_id < checkpoint_snapshot_id); + cohort.push_back(layout.writeOnceRefSnapshotKey(life, snap_id)); + } + + const size_t chunk_keys = std::clamp(store->poolConfig().gc_bulk_delete_chunk_keys, 1, kBulkDeleteMaxKeys); + for (size_t begin = 0; begin < cohort.size(); ) + { + /// Cumulative per-round cap in KEYS, exactly as before; a chunk is cut to what remains. The + /// plan recomputes the same remaining candidates from durable state next round, so nothing + /// here needs its own cursor. if (!work_budget.refCleanupAvailable()) return; - if (!deleteRefObject(layout.refSnapshotKey(life, snap_id))) + size_t end = std::min(cohort.size(), begin + chunk_keys); + if (work_budget.max_ref_cleanup_objects != 0) + end = std::min(end, begin + (work_budget.max_ref_cleanup_objects - work_budget.ref_cleanup_objects_used)); + std::vector chunk(cohort.begin() + begin, cohort.begin() + end); + if (!authorityHolds(chunk.front().str())) return; - ++work_budget.ref_cleanup_objects_used; + op.removeManyWriteOnce(chunk, Retry::standard()); + work_budget.ref_cleanup_objects_used += chunk.size(); + ProfileEvents::increment(ProfileEvents::CASRefCleanupObjectsDeleted, chunk.size()); /// cleanup object deletion + /// Advance by what was actually sent, not the nominal chunk size: the budget cap above can + /// truncate a chunk short of `chunk_keys`, and advancing by the full stride would skip the + /// untried remainder instead of retrying it next iteration. + begin = end; } } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcKeyReader.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcKeyReader.h new file mode 100644 index 000000000000..3d4d6ca82ffb --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcKeyReader.h @@ -0,0 +1,25 @@ +#pragma once +#include +#include + +namespace DB::Cas +{ + +/// The reader over the GC's read-ahead: a hint is a worker request, a take is the worker's result +/// (or an inline read for a key nobody hinted), and a discard drops a hinted key and counts it as +/// wasted at once. +class ReadAheadKeyReader final : public KeyReader +{ +public: + explicit ReadAheadKeyReader(GcReadAhead & reads_) : reads(reads_) {} + void hint(const String & key) override { reads.hintRead(key); } + std::optional take(const String & key) override { return reads.takeRead(key); } + void discard(const String & key) override { reads.discardRead(key); } + size_t pending() const override { return reads.pending(); } + size_t window() const override { return reads.window(); } + +private: + GcReadAhead & reads; +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp index 2c1d9cf313fb..fe6eecedd72a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.cpp @@ -101,6 +101,29 @@ void GcReadAhead::hintHead(const String & key) [](CasOperation & worker, const String & k) { return worker.head(k, Retry::standard()); }); } +template +void GcReadAhead::discard(Slots & slots, const String & key) +{ + const auto it = slots.find(key); + if (it == slots.end()) + return; + std::shared_ptr> slot = std::move(it->second); + slots.erase(it); + /// `wait`, not `get`: the result and any exception belong to a request nobody wanted. + slot->future.wait(); + ProfileEvents::increment(ProfileEvents::CASGCReadAheadWasted); +} + +void GcReadAhead::discardRead(const String & key) +{ + discard(reads, key); +} + +void GcReadAhead::discardHead(const String & key) +{ + discard(heads, key); +} + std::optional GcReadAhead::takeRead(const String & key) { return take(reads, key, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h index 691eaa2c746f..19b9df5a24c8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGcReadAhead.h @@ -26,12 +26,16 @@ namespace DB::Cas /// durable before the round began (it is a gap whenever it is read), and a manifest body still being /// uploaded when the early read lands yields the same hold a slightly earlier sequential read yields /// today. Nothing is hinted above `committed_through`, so no request is issued that the sequential -/// walk would not issue. +/// walk would not issue -- with one bounded exception: a ref-log hinting site does not yet know where +/// an epoch's closing seal is (it learns that only by decoding the log at that position), so it may +/// hint ids past the seal, inside the SAME epoch, that turn out not to exist. Those are discarded +/// rather than taken, overshooting by at most one window per epoch crossing, and counted wasted. /// /// Memory is the CALLER's to bound: `pending` counts hinted-but-untaken slots and `window` is how /// many a hinting site keeps in flight. A key hinted twice is one request. Results never taken are -/// awaited by the destructor and counted as wasted. Only the owning thread touches the maps; a -/// worker touches only its own slot. +/// awaited by the destructor and counted as wasted, or discarded explicitly by `discardRead`/ +/// `discardHead`, which counts them at once. Only the owning thread touches the maps; a worker +/// touches only its own slot. class GcReadAhead { public: @@ -47,6 +51,14 @@ class GcReadAhead std::optional takeRead(const String & key); std::optional takeHead(const String & key); + /// Drops a hinted key the caller will never take. The request is already in flight or done; the + /// wait is bounded by that one request, its result and its exception are dropped, and the slot is + /// counted as wasted now rather than in the destructor. The exception is dropped because no + /// sequential walk would have issued this request, so none could have failed on it; the fence and + /// the budget are re-observed by the very next request anyway. A key nobody hinted is a no-op. + void discardRead(const String & key); + void discardHead(const String & key); + /// Hinted but not yet taken, both verbs together: what a hinting site throttles itself against. size_t pending() const { return reads.size() + heads.size(); } @@ -72,6 +84,9 @@ class GcReadAhead template std::optional take(Slots & slots, const String & key, Inline inline_request); + template + void discard(Slots & slots, const String & key); + CasOperation & op; CasRequests & requests; ThreadPool & pool; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp index 5c3a88422846..28a471b673ff 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp @@ -10,11 +10,14 @@ #include #include #include +#include #include #include +#include #include #include #include +#include #include #include @@ -42,7 +45,8 @@ void onGcEnumerationPage() /// mount there is no deletion authority, so the caller must leave the prefix untouched. The mount's /// `writer_epoch` and `min_active_build_sequence` are the single durable epoch/floor pair used for eligibility, including /// across process replacement and the retired sentinel. -std::optional floorForNamespace(CasOperation & op, const Layout & layout, const RootNamespace & ns) +std::optional floorForNamespace(CasOperation & op, const Layout & layout, const RootNamespace & ns, + uint64_t * reads = nullptr) { const String & value = ns.string(); size_t pos = value.size(); @@ -55,6 +59,8 @@ std::optional floorForNamespace(CasOperation & op, const Layout & la const String server_root_id = value.substr(0, pos); if (!server_root_id.empty()) { + if (reads) + ++*reads; if (const auto got = op.read(layout.mountKey(server_root_id), Retry::standard())) return decodeMountLease(got->bytes); } @@ -170,9 +176,13 @@ struct NamespaceProtection /// invalid transaction (via the authority-grounded recovery / `decodeRefLogTxn`); the caller SKIPS the /// namespace's deletions on such a throw rather than substituting an empty owner set. /// -/// `work_budget`, when set, bounds the committed-tail walk below: each ref-log GET the walk issues -/// consumes one unit of `GcRoundWorkBudget::sweep_recovery_op_budget`, shared with every other -/// namespace this round touches. Exhaustion sets `NamespaceProtection::recovery_incomplete` and stops +/// `work_budget`, when set, bounds the committed-tail walk below: each ref-log the walk TAKES (decodes +/// and applies) consumes one unit of `GcRoundWorkBudget::sweep_recovery_op_budget`, shared with every +/// other namespace this round touches. A hinting reader may prefetch up to one window of logs beyond +/// the charged position before the walk reaches them -- those prefetches are not charged, since the +/// budget bounds decoded work, not requests in flight -- and a hint issued past an epoch's seal is +/// discarded rather than taken, bounding the waste of one crossing to one window, counted in +/// `CASGCReadAheadWasted`. Exhaustion sets `NamespaceProtection::recovery_incomplete` and stops /// the walk — it is deliberately NOT plumbed into `recoverRefTableDetailedFromAuthority` itself: that /// function is a shared recovery primitive also used by `fsck` (which needs a COMPLETE table to audit) /// and the GC rebuild path (which needs a complete table to reconstruct in-degree from scratch), so @@ -180,8 +190,8 @@ struct NamespaceProtection /// Reaching the budget before even calling `recoverRefTableDetailedFromAuthority` (already spent by an /// earlier namespace) skips that call entirely and reports incomplete immediately. NamespaceProtection activeManifestKeys( - CasOperation & op, const Layout & layout, const CatalogEntry & catalog_entry, const RefCkpt & ckpt, - const std::optional & coverage, GcRoundWorkBudget * work_budget = nullptr) + CasOperation & op, KeyReader & reader, const Layout & layout, const CatalogEntry & catalog_entry, + const RefCkpt & ckpt, const std::optional & coverage, GcRoundWorkBudget * work_budget = nullptr) { NamespaceProtection protection; if (work_budget && !work_budget->sweepRecoveryOpAvailable()) @@ -197,7 +207,7 @@ NamespaceProtection activeManifestKeys( /// The exact row and `_ckpt` come from the caller's frozen catalog cut. Do not resolve `ns` here: /// a later catalog cut can name a reborn life and turn this old life into an apparent orphan. const RecoveredRefTable recovered = recoverRefTableDetailedFromAuthority( - op, layout, catalog_entry, ckpt); + op, layout, catalog_entry, ckpt, &reader); if (work_budget) ++work_budget->sweep_recovery_ops_used; /// one coarse unit for the snapshot+tail recovery itself const RefTableState & state = recovered.state; @@ -285,6 +295,37 @@ NamespaceProtection activeManifestKeys( } } + /// Guards the walk's most recently hinted-but-not-yet-taken range (from `arm`'s `from` up to one + /// window) so that an early exit from the loop below -- the work-budget `break`, the corrupted-tail + /// `throw`, or the missing-cursor epoch cross -- frees it instead of leaving it pinned against the + /// SAME reader for whatever this page reads next (later candidates, later namespaces). An ordinary + /// same-epoch advance re-arms it on the new position without discarding: those hints are still + /// wanted. A seal crossing discards explicitly, right where the crossing happens, and disarms so + /// this guard's own destructor does not repeat it. + struct OutstandingHintGuard + { + KeyReader & reader; + const Layout & layout; + const NamespaceLifeId & life; + RefTxnId from{}; + RefTxnId committed_through{}; + bool armed = false; + + void arm(const RefTxnId & from_, const RefTxnId & committed_through_) + { + from = from_; + committed_through = committed_through_; + armed = true; + } + void discardNow() + { + if (armed) + discardRefLogHintsOfEpoch(reader, layout, life, from, committed_through); + armed = false; + } + ~OutstandingHintGuard() { discardNow(); } + } outstanding_hints{reader, layout, life}; + while (id <= *ckpt.committed_through) { /// UNCERTAINTY, work-budget arm: the committed-tail walk is a finite but potentially huge range @@ -292,13 +333,19 @@ NamespaceProtection activeManifestKeys( /// namespace it touches. Stopping HERE — before the next GET — leaves `active`/ /// `tail_removal_targets` genuinely partial, so the caller must treat the whole namespace as /// undecided this page (fail-closed retain), never authorize a deletion from what was collected - /// so far. + /// so far. `outstanding_hints`'s destructor frees whatever the walk hinted ahead of this point. if (work_budget && !work_budget->sweepRecoveryOpAvailable()) { protection.recovery_incomplete = true; break; } - const auto got = op.read(layout.refLogKey(life, id), Retry::standard()); + if (id.ref_sequence < std::numeric_limits::max()) + { + const RefTxnId hint_from{id.writer_epoch, id.ref_sequence + 1}; + hintRefLogsWithinEpoch(reader, layout, life, hint_from, *ckpt.committed_through); + outstanding_hints.arm(hint_from, *ckpt.committed_through); + } + const auto got = reader.take(layout.refLogKey(life, id)); if (work_budget) ++work_budget->sweep_recovery_ops_used; if (!got) @@ -310,6 +357,9 @@ NamespaceProtection activeManifestKeys( throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS orphan sweep: committed tail log {} is absent under the supplied _ckpt frontier", renderRefTxnId(id)); + /// The old epoch's hints past this missing cursor do not exist either; discard them before + /// crossing, exactly as the seal path below does. + outstanding_hints.discardNow(); id = cross_from_missing_cursor(*prior); prior.reset(); prior_is_seal.reset(); @@ -330,6 +380,7 @@ NamespaceProtection activeManifestKeys( { if (is_seal) { + outstanding_hints.discardNow(); prior.reset(); prior_is_seal.reset(); } @@ -480,7 +531,13 @@ namespace /// debris drains after a process restart even when its build_sequence is above the current min_active_build_sequence. bool prefixEligibleOn(CasOperation & op, const Layout & layout, const RootNamespace & ns, const BuildPrefix & prefix) { - const auto floor = floorForNamespace(op, layout, ns); + return prefixEligibleUnder(floorForNamespace(op, layout, ns), prefix); +} + +} + +bool prefixEligibleUnder(const std::optional & floor, const BuildPrefix & prefix) +{ if (!floor) return false; @@ -494,8 +551,6 @@ bool prefixEligibleOn(CasOperation & op, const Layout & layout, const RootNamesp return w.min_active_build_sequence > prefix.build_sequence; } -} - bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix) { CasOperation op = store.openRequests().admit(); @@ -541,7 +596,8 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi warnings->push_back(warning); return 0; } - protection = activeManifestKeys(op, layout, *catalog_entry, ckpt->ckpt, view.coverage); + InlineKeyReader reader(op); + protection = activeManifestKeys(op, reader, layout, *catalog_entry, ckpt->ckpt, view.coverage); view.tail_removal_targets = protection.tail_removal_targets; } @@ -614,7 +670,9 @@ ManifestSweepResult planManifestCursorPage( uint64_t list_budget, uint64_t nomination_budget, bool catalog_recovery_authoritative, - GcRoundWorkBudget * work_budget) + GcRoundWorkBudget * work_budget, + ThreadPool * read_pool, + size_t read_concurrency) { ManifestSweepResult result; result.next_cursor = cursor; @@ -623,46 +681,33 @@ ManifestSweepResult planManifestCursorPage( const Layout & layout = store.layout(); CasOperation op = store.openRequests().admit(); + /// The page's reader. With a pool and concurrency above one the candidates' bodies and the two + /// ref-stream walks overlap their round trips; otherwise every read is inline and the page is the + /// sequential one, request for request. + std::optional read_ahead; + std::unique_ptr reader; + if (read_pool && read_concurrency > 1) + { + read_ahead.emplace(op, store.openRequests(), *read_pool, read_concurrency); + reader = std::make_unique(*read_ahead); + } + else + reader = std::make_unique(op); const ListPage page = op.list(layout.casManifestsPrefix(), cursor, list_budget, Retry::standard()); /// This pass fetches exactly one page per round (the cursor advances across rounds, not within this /// call), so the metric increments once per call, not once per listed key. ProfileEvents::increment(ProfileEvents::CASGCEnumerationPages); - /// Freeze every possible destructive candidate BEFORE the later catalog cut. A same-name rebirth can - /// replace this logical manifest key between the observations; classifying the old bytes against the - /// later lifecycle cut is safe only when deletion retains the old exact incarnation, so the - /// replacement fails the caller's re-observation. Do not take a fresh read after the catalog read: - /// that would splice new-life bytes into old candidate selection and authorize their deletion with - /// the new incarnation. - /// - /// Bounded to `nomination_budget` well-formed keys — never the whole `list_budget`-sized - /// page — since `nomination_budget` is the hard ceiling on how many of them this call can ever - /// nominate. A well-formed key beyond this cap has no frozen body; it is retained where its absence - /// is discovered below, in the exact same "budget exhausted, cursor does not step over it" shape the - /// nomination-count exhaustion already uses. - std::map> observed_candidates; - if (nomination_budget > 0) - { - uint64_t frozen = 0; - for (const ListedKey & listed : page.keys) - { - if (frozen >= nomination_budget) - break; - if (parseListedManifestObject(layout, listed.key)) - { - observed_candidates.emplace(listed.key, op.read(listed.key, Retry::standard())); - ++frozen; - } - } - } - /// One seal and one later catalog cut for the whole page; every namespace joins through those same /// immutable observations. const std::optional adopted_seal = readAdoptedFoldSeal(op, layout); const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(op, layout); catalog_cut.life_index.throwIfAmbiguous("CAS orphan manifest sweep"); - std::map eligible_by_prefix; + /// One floor observation per namespace for the whole page. Every build of a namespace is judged + /// against the same mount body; reading it per build prefix would cost three requests per listed + /// key on a pool where every INSERT is its own build. + std::map> floor_by_ns; std::map view_by_ns; std::map> active_by_ns; std::set errored_namespaces; /// protection view unavailable => skip, never delete @@ -673,6 +718,11 @@ ManifestSweepResult planManifestCursorPage( String decided_through; bool budget_exhausted = false; + /// Keys that passed every retain check. Their bodies are read AFTER the loop, and after the + /// catalog cut: a manifest key is write-once, so its bytes do not depend on when they are read, + /// and the only thing a later read can observe differently is absence, which retains. + std::vector candidates; + for (const ListedKey & listed : page.keys) { ++result.listed; @@ -685,7 +735,7 @@ ManifestSweepResult planManifestCursorPage( /// A budget of ZERO is not exhaustion but a list-only pass: nothing is ever deletable, so /// freezing the cursor on it would make the sweep spin on one page forever. That pass keeps /// its pre-existing behaviour and advances. - if (budget_exhausted || (nomination_budget > 0 && result.nominations.size() >= nomination_budget)) + if (budget_exhausted || (nomination_budget > 0 && candidates.size() >= nomination_budget)) { budget_exhausted = true; ++result.skipped; @@ -711,13 +761,13 @@ ManifestSweepResult planManifestCursorPage( const CatalogEntry * catalog_entry = catalogEntryOf(catalog_cut, parsed->ns); if (catalog_entry) { - const String eligibility_key = parsed->ns.string() + "\n" - + std::to_string(parsed->prefix.writer_epoch) + "\n" - + std::to_string(parsed->prefix.build_sequence); - auto [eligible_it, eligible_inserted] = eligible_by_prefix.emplace(eligibility_key, false); - if (eligible_inserted) - eligible_it->second = prefixEligible(store, parsed->ns, parsed->prefix); - if (!eligible_it->second) + auto [floor_it, floor_inserted] = floor_by_ns.emplace(parsed->ns.string(), std::nullopt); + if (floor_inserted) + { + ++result.floor_lookups; + floor_it->second = floorForNamespace(op, layout, parsed->ns, &result.floor_reads); + } + if (!prefixEligibleUnder(floor_it->second, parsed->prefix)) { ++result.skipped; decided_through = listed.key; @@ -789,7 +839,7 @@ ManifestSweepResult planManifestCursorPage( else { NamespaceProtection protection = activeManifestKeys( - op, layout, *catalog_entry, ckpt->ckpt, view_it->second.coverage, work_budget); + op, *reader, layout, *catalog_entry, ckpt->ckpt, view_it->second.coverage, work_budget); if (protection.recovery_incomplete) { /// The committed-tail walk stopped early: `active`/`tail_removal_targets` @@ -865,26 +915,21 @@ ManifestSweepResult planManifestCursorPage( } } - /// This exact incarnation and bytes were captured before the catalog cut (see above). A missing - /// body has no deletion authority; a later replacement no longer matches what is recorded here, - /// so the caller's re-observation refuses the delete. - /// - /// A well-formed key can legitimately be ABSENT here: the freeze loop above caps - /// fan-out at `nomination_budget` candidates, so a key beyond that cap was never frozen. Treat - /// it exactly like nomination-count exhaustion -- retain, and do NOT advance the cursor past - /// it, so the very next page/round examines it with a fresh budget instead of losing it. - const auto observed_it = observed_candidates.find(parsed->key); - if (observed_it == observed_candidates.end()) - { - budget_exhausted = true; - ++result.skipped; - continue; - } - const std::optional & got = observed_it->second; + candidates.push_back(*parsed); + decided_through = listed.key; + } + + size_t next_body_hint = 0; + for (const ListedManifestObject & candidate : candidates) + { + while (next_body_hint < candidates.size() && reader->pending() < reader->window()) + reader->hint(candidates[next_body_hint++].key); + const std::optional got = reader->take(candidate.key); if (!got) { + /// Gone since the LIST: a fresh writer never reuses the key, so there is nothing to + /// nominate and nothing to retain. The key was decided above; the cursor stands. ++result.skipped; - decided_through = listed.key; continue; } std::optional body; @@ -901,21 +946,20 @@ ManifestSweepResult planManifestCursorPage( /// reclamation for the whole pool rather than for this one key. LOG_ERROR(getLogger("CasOrphanManifestSweep"), "CAS orphan sweep: manifest at {} cannot be decoded and was retained; run cas-fsck to " - "enumerate such objects", parsed->key); + "enumerate such objects", candidate.key); ++result.undecodable; ++result.skipped; - decided_through = listed.key; continue; } - const ManifestId id{parsed->ns, parsed->ref}; + const ManifestId id{candidate.ns, candidate.ref}; if (!refMatchesBody(id.ref, *body) || !manifestNamespaceMatches(id.root_namespace, *body)) throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS orphan sweep: manifest identity mismatch at {} while deriving exact source edges", - parsed->key); + candidate.key); ManifestSweepResult::Nomination nomination{ .id = id, - .key = parsed->key, + .key = candidate.key, .token = PersistedEtag::capture(got->etag), .source_retirements = {}}; for (const ManifestEntry & entry : body->entries) @@ -924,7 +968,6 @@ ManifestSweepResult planManifestCursorPage( .ref = entry.ref, .source_id = sourceEdgeId(id, entry.path)}); result.nominations.push_back(std::move(nomination)); - decided_through = listed.key; } if (budget_exhausted) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h index 72f723431f8c..d4da7e38f5a5 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -131,6 +132,13 @@ struct ManifestSweepResult uint64_t retained_tail_removal = 0; uint64_t retained_work_budget = 0; + /// The floor a namespace's builds are judged against is one mount body per server root. A page + /// resolves it once per namespace (`floor_lookups`); each lookup reads the mount key of every + /// `/`-prefix of the namespace until one answers (`floor_reads`), so an absent mount costs the + /// whole chain once. + uint64_t floor_lookups = 0; + uint64_t floor_reads = 0; + /// Exact-GET/decode candidates. The reducer must adopt every `source_retirements` entry before the /// caller may delete `key`, and only after re-observing `token` at it: a key whose incarnation /// moved on belongs to a fresh owner and must be left alone. @@ -187,24 +195,41 @@ uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefi /// judged-dead heuristic. A missing lease provides no deletion authority, so the prefix is not eligible. bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix); -/// Plan one cursor page without deleting. Every candidate is exact-GET, decoded and identity-validated; -/// its exact manifest-source edges are returned for accounting-neutral retirement in the next fold. +/// The pure half of `prefixEligible`: whether `prefix` is retired under one observation of the mount +/// floor. `nullopt` (no mount body under any prefix of the namespace) admits nothing. Retirement is +/// permanent -- the epoch and the acknowledgement floor only grow and the farewell is terminal -- so +/// an admission derived from any observation stays true afterwards, which is what lets a page judge +/// every build of a namespace against one read. +bool prefixEligibleUnder(const std::optional & floor, const BuildPrefix & prefix); + +/// Plan one cursor page without deleting. A key is decided from key-derived facts first; only a +/// candidate's body is read, after the catalog cut, then decoded and identity-validated; its exact +/// manifest-source edges are returned for accounting-neutral retirement in the next fold. /// Catalog-named namespaces are retain-only unless the caller explicitly authorizes recovery from its /// frozen catalog cut and the exact `_ckpt` frontier of the life named there. /// -/// `work_budget`, when set, bounds the body-GET/retention fan-out to `nomination_budget` well-formed -/// candidates (never the whole `list_budget`-sized page), caps how many DISTINCT namespaces this page -/// may build a fresh protection view for, and caps the committed-tail recovery walk's ref-log GET +/// `nomination_budget` is a candidate budget: the page stops deciding once it has that many +/// candidates, and reads exactly that many bodies at most. `work_budget`, when set, additionally +/// caps how many DISTINCT namespaces this page may build a fresh protection view for, and caps the +/// committed-tail recovery walk's ref-log GET /// count cumulatively across the round (shared with every other destructive-work family via the same /// `GcRoundWorkBudget` instance). Exhausting either cap retains every remaining candidate belonging to /// the affected namespace on THIS page rather than deciding it without a complete protection view; /// `nullptr` (the default) reproduces the pre-budget unbounded behavior. +/// +/// `read_pool` and `read_concurrency` (default `nullptr`/1, i.e. every read inline on the caller's +/// operation) drive the page's read-ahead: with a pool and a concurrency above one, the candidates' +/// bodies and the catalog-recovery/committed-tail ref-log walks overlap their round trips on `read_pool` +/// instead of serializing request by request. Every decision this function makes is unchanged by that +/// choice; see `GcReadAhead` for what may be fetched early and why. ManifestSweepResult planManifestCursorPage( Pool & store, const String & cursor, uint64_t list_budget, uint64_t nomination_budget, bool catalog_recovery_authoritative, - GcRoundWorkBudget * work_budget = nullptr); + GcRoundWorkBudget * work_budget = nullptr, + ThreadPool * read_pool = nullptr, + size_t read_concurrency = 1); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.cpp new file mode 100644 index 000000000000..5844902dd239 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.cpp @@ -0,0 +1,32 @@ +#include + +#include + +namespace DB::Cas +{ + +void hintRefLogsWithinEpoch(KeyReader & reader, const Layout & layout, const NamespaceLifeId & life, + RefTxnId first, const RefTxnId & committed_through) +{ + while (reader.pending() < reader.window() && first <= committed_through) + { + reader.hint(layout.refLogKey(life, first)); + if (first.ref_sequence == std::numeric_limits::max()) + return; + ++first.ref_sequence; + } +} + +void discardRefLogHintsOfEpoch(KeyReader & reader, const Layout & layout, const NamespaceLifeId & life, + RefTxnId first, const RefTxnId & committed_through) +{ + for (size_t n = 0; n < reader.window() && first <= committed_through; ++n) + { + reader.discard(layout.refLogKey(life, first)); + if (first.ref_sequence == std::numeric_limits::max()) + return; + ++first.ref_sequence; + } +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.h new file mode 100644 index 000000000000..a4bb79c91f28 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasKeyReader.h @@ -0,0 +1,55 @@ +#pragma once +#include +#include +#include +#include + +namespace DB::Cas +{ + +/// What a sequential walk needs from whoever fetches its objects. `take` returns the object (or +/// nullopt when absent) and is the only call that decides anything; `hint` may start fetching a key +/// the walk will take later; `discard` drops a hint the walk will never take. A walk that hints +/// nothing and takes everything in order behaves exactly like one that reads inline: the reader is a +/// cache of results, never of decisions. +class KeyReader +{ +public: + virtual ~KeyReader() = default; + virtual void hint(const String & key) = 0; + virtual std::optional take(const String & key) = 0; + virtual void discard(const String & key) = 0; + /// Hinted and not yet taken. + virtual size_t pending() const = 0; + /// How many hints a walk keeps outstanding; 0 means "do not hint". + virtual size_t window() const = 0; +}; + +/// The sequential reader: every take is one inline read on the caller's operation. +class InlineKeyReader final : public KeyReader +{ +public: + explicit InlineKeyReader(CasOperation & op_) : op(op_) {} + void hint(const String &) override {} + std::optional take(const String & key) override { return op.read(key, Retry::standard()); } + void discard(const String &) override {} + size_t pending() const override { return 0; } + size_t window() const override { return 0; } + +private: + CasOperation & op; +}; + +/// Hints the ref-log ids of `first`'s epoch, from `first` upward, while the reader has window and the +/// id is within the committed frontier. Only this epoch: past its seal the ids do not exist, and a +/// walk learns where the seal is only by decoding it. +void hintRefLogsWithinEpoch(KeyReader & reader, const Layout & layout, const NamespaceLifeId & life, + RefTxnId first, const RefTxnId & committed_through); + +/// The other half of the rule above, called when a walk crosses an epoch: every hint of the old epoch +/// from `first` up to one window is dropped, so the window is free for the new epoch. Discarding an +/// unhinted key is a no-op, so over-asking by a window is harmless. +void discardRefLogHintsOfEpoch(KeyReader & reader, const Layout & layout, const NamespaceLifeId & life, + RefTxnId first, const RefTxnId & committed_through); + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index bf5ee66fb6ec..fc6829d2c53c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -88,6 +88,8 @@ struct PoolConfig /// per completed GC round; the delete budget separately bounds exact-token destructive work. uint64_t manifest_sweep_list_budget_keys = 1000; uint64_t manifest_sweep_delete_budget_keys = 100; + /// Keys per batch delete request for the write-once families; tests lower it to exercise chunk boundaries. + uint64_t gc_bulk_delete_chunk_keys = 1000; /// Per-round blob-deletion work envelope: caps how many entries the fold's graduation /// (condemned -> delete_pending) and redelete (exact-token delete of a prior delete_pending row) /// arms move out of the durable retired pipeline in one round. Excess entries are carried diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp index a0dd85e1e158..15ffd10e9666 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.cpp @@ -1041,7 +1041,7 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( RecoveredRefTable recoverRefTableDetailedFromAuthority( CasOperation & op, const Layout & layout, const std::optional & catalog_entry, - const std::optional & ckpt) + const std::optional & ckpt, KeyReader * reader) { /// The frozen catalog row and `_ckpt` supplied by the caller determine every recovery boundary; /// this function must not re-read either mutable object, or enumerate the stream, because that @@ -1070,7 +1070,11 @@ RecoveredRefTable recoverRefTableDetailedFromAuthority( RefTxnId id = *grounding.walk_from; while (id <= *grounding.committed_through) { - const auto got = op.read(layout.refLogKey(life, id), Retry::standard()); + const String key = layout.refLogKey(life, id); + if (reader && id.ref_sequence < std::numeric_limits::max()) + hintRefLogsWithinEpoch(*reader, layout, life, RefTxnId{id.writer_epoch, id.ref_sequence + 1}, + *grounding.committed_through); + const auto got = reader ? reader->take(key) : op.read(key, Retry::standard()); if (!got) { /// `NamespaceLifeId` is opaque and unique to one logical life. A later birth has a @@ -1092,7 +1096,12 @@ RecoveredRefTable recoverRefTableDetailedFromAuthority( if (const std::optional next = nextRefLogIdWithinCommittedFrontier( id, is_seal, *grounding.committed_through)) + { + if (is_seal && reader && id.ref_sequence < std::numeric_limits::max()) + discardRefLogHintsOfEpoch(*reader, layout, life, RefTxnId{id.writer_epoch, id.ref_sequence + 1}, + *grounding.committed_through); id = *next; + } else break; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h index 4ba61f17f127..2a819e27eff8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefProtocol.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -795,8 +796,11 @@ CheckpointSnapshotBase readCheckpointSnapshotBase( /// under this immutable authority. In particular, this read-only API never probes or adopts `F+1`. There /// is deliberately no self-resolving compatibility overload: every consumer must pass the row from its /// frozen catalog cut explicitly. +/// +/// `reader`, when set, fetches the logs; it may prefetch ids of the current epoch and drops them at a +/// seal. Every decision and every error path is the same with and without it. RecoveredRefTable recoverRefTableDetailedFromAuthority( CasOperation & op, const Layout & layout, const std::optional & catalog_entry, - const std::optional & ckpt); + const std::optional & ckpt, KeyReader * reader = nullptr); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasWriteOnceKey.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasWriteOnceKey.h new file mode 100644 index 000000000000..058977b939a7 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Primitives/CasWriteOnceKey.h @@ -0,0 +1,26 @@ +#pragma once +#include +#include + +namespace DB::Cas +{ + +class Layout; + +/// The key of an object that is written once and never rewritten: a part manifest, a ref log, a ref +/// snapshot. Only `Layout` mints one, from the typed identity of such an object, so a verb that +/// accepts this type can delete without a precondition: whatever body the key holds is the one body +/// it ever held. A mutable control object (a checkpoint, the catalog, `gc/state`, a mount lease) has +/// no path to this type. +class WriteOnceKey +{ +public: + const String & str() const { return key; } + +private: + friend class Layout; + explicit WriteOnceKey(String key_) : key(std::move(key_)) {} + String key; +}; + +} diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp index fe8401e2d3b2..580b77941a55 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp @@ -96,6 +96,11 @@ ConditionalRemoveResult IObjectStorage::removeObjectIfTokenMatches( return removeObjectIfTokenMatches(object, etag); } +void IObjectStorage::removeObjectsIfExistUnderProfile(const StoredObjects &, ObjectStorageRetryProfile, uint64_t) +{ + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support batch removal under a retry profile", getName()); +} + ThreadPool & IObjectStorage::getThreadPoolWriter() { auto context = Context::getGlobalContextInstance(); diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index f770ce25c0ac..c56ecac18575 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -372,6 +372,13 @@ class IObjectStorage virtual ConditionalRemoveResult removeObjectIfTokenMatches( const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms); + /// Removes every object in ONE request with no per-key precondition; an absent object is success. + /// Content-addressed callers use it for write-once keys only, at most 1000 per call. Throws on a + /// request-level failure and on any per-key error other than "not found", naming the failed keys. + /// Same profile note as `iterate`. Backends without a batch delete keep the default, which refuses. + virtual void removeObjectsIfExistUnderProfile( + const StoredObjects & objects, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms); + /// Copy object with different attributes if required virtual void copyObject( /// NOLINT const StoredObject & object_from, diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 392ede659844..9f069239fe60 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -623,6 +623,80 @@ ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatchesImpl( err.GetMessage(), static_cast(err.GetErrorType()), err.GetExceptionName(), object.remote_path); } +void S3ObjectStorage::removeObjectsIfExistUnderProfile( + const StoredObjects & objects, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) +{ + refreshAndRetryOnExpiredCredentials([&] + { + removeObjectsIfExistImpl(objects, clientForRetryProfile(profile, request_timeout_ms)); + return 0; + }); +} + +void S3ObjectStorage::removeObjectsIfExistImpl(const StoredObjects & objects, const std::shared_ptr & used_client) +{ + if (objects.empty()) + return; + + std::vector identifiers; // STYLE_CHECK_ALLOW_STD_CONTAINERS + identifiers.reserve(objects.size()); + for (const auto & object : objects) + { + Aws::S3::Model::ObjectIdentifier identifier; + identifier.SetKey(object.remote_path); + identifiers.push_back(std::move(identifier)); + } + Aws::S3::Model::Delete to_delete; + to_delete.SetObjects(std::move(identifiers)); + /// Quiet: only failed keys come back. A key that is gone or was never there is not a failure here + /// (`NoSuchKey` below), and the caller has no use for the per-key successes. + to_delete.SetQuiet(true); + + S3::DeleteObjectsRequest request; + request.SetBucket(uri.bucket); + request.SetDelete(std::move(to_delete)); + + ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); + auto outcome = used_client->DeleteObjects(request); + + /// Every key lands in system.blob_storage_log, as the single-key paths do; the batch's outcome is + /// stamped on each of them. + if (auto blob_storage_log = BlobStorageLogWriter::create(disk_name)) + { + for (const auto & object : objects) + blob_storage_log->addEvent(BlobStorageLogElement::EventType::Delete, + uri.bucket, object.remote_path, + object.local_path, object.bytes_size, + /* elapsed_microseconds */ 0, + outcome.IsSuccess() ? 0 : static_cast(outcome.GetError().GetErrorType()), + outcome.IsSuccess() ? "" : outcome.GetError().GetMessage()); + } + + if (!outcome.IsSuccess()) + { + const auto & err = outcome.GetError(); + throw S3Exception(err.GetErrorType(), "{} (Code: {}) while removing {} objects from S3 in one request", + err.GetMessage(), static_cast(err.GetErrorType()), objects.size()); + } + + String failed_keys; + std::optional first_error_type; + for (const auto & err : outcome.GetResult().GetErrors()) + { + const auto error_type = static_cast( + Aws::S3::S3ErrorMapper::GetErrorForName(err.GetCode().c_str()).GetErrorType()); + if (S3::isNotFoundError(error_type)) + continue; + if (!failed_keys.empty()) + failed_keys += ", "; + failed_keys += err.GetKey() + " (" + err.GetCode() + ": " + err.GetMessage() + ")"; + if (!first_error_type) + first_error_type = error_type; + } + if (first_error_type) + throw S3Exception(*first_error_type, "batch removal left objects behind: [{}]", failed_keys); +} + bool S3ObjectStorage::conditionalOpsUseGenerationTokens() const { return client->get()->supportsGcsNativeConditionalRequests(); diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 866d9ab9fdd3..0b46038e9aa8 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -126,6 +126,10 @@ class S3ObjectStorage : public IObjectStorage ConditionalRemoveResult removeObjectIfTokenMatches( const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) override; + /// One `DeleteObjects` for the given objects (the caller chunks to at most 1000); absence is success. + void removeObjectsIfExistUnderProfile( + const StoredObjects & objects, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) override; + void tagObjects(const StoredObjects & objects, const std::string & tag_key, const std::string & tag_value) override; ObjectMetadata getObjectMetadata(const std::string & path, bool with_tags) const override; @@ -213,6 +217,8 @@ class S3ObjectStorage : public IObjectStorage ConditionalRemoveResult removeObjectIfTokenMatchesImpl( const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client); + void removeObjectsIfExistImpl(const StoredObjects & objects, const std::shared_ptr & used_client); + std::shared_ptr clientForRetryProfile(ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const; /// Runs `fn` and, if it failed because the vended credentials expired, refreshes this disk's diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 676473afe71e..24cf357ed110 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -1714,6 +1714,16 @@ class CountingBackend : public DB::Cas::InMemoryBackend return InMemoryBackend::remove(key, expected_value, access); } + /// One `removeManyWriteOnce` call is one bulk-delete request that names every key it carried; the + /// per-key `delete_counts` grow by one for each key named, exactly as a single-key `remove` would, + /// so `deleteCount(key)` reads the same whichever verb deleted it. + void removeManyWriteOnce(const std::vector & keys, DB::Cas::TransportAccess & access) override + { + for (const DB::Cas::WriteOnceKey & key : keys) + tick(delete_counts, delete_total, key.str()); + InMemoryBackend::removeManyWriteOnce(keys, access); + } + /// A blob publication reaches the store through `publish`, not `write` -- a "zero backend requests" /// assertion built only from the primitives above would miss one landing. void publish(const DB::Cas::BlobPublishRequest & request, DB::Cas::TransportAccess & access) override diff --git a/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp b/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp new file mode 100644 index 000000000000..27b7a01399e3 --- /dev/null +++ b/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp @@ -0,0 +1,196 @@ +#include + +#include "config.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/// `removeManyWriteOnce` deletes up to 1000 write-once keys in one request with no precondition: +/// an absent key is success, a present one is gone afterwards, and every backend honours the same +/// fault knobs the single-key delete has. + +namespace ProfileEvents +{ + extern const Event CASBulkDeleteRequests; + extern const Event CASManifestDelete; + extern const Event CASRootDelete; +} + +namespace DB::ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +} + +using namespace DB::Cas; +using DB::Cas::tests::expectThrowsCode; +using DB::Cas::tests::openRequestsForTest; + +namespace +{ + +const Layout kLayout{"p"}; +const RootNamespace kNs{"test/aa@cas@"}; + +ManifestId manifest(uint32_t ordinal) +{ + return ManifestId{kNs, ManifestRef{.writer_epoch = 1, .build_sequence = 1, .manifest_ordinal = ordinal}}; +} + +/// Three manifest keys with bodies and one that was never written. +struct Keys +{ + std::vector present; + WriteOnceKey absent; +}; + +Keys seed(CasOperation & op) +{ + Keys keys{.present = {}, .absent = kLayout.writeOnceManifestKey(manifest(4))}; + for (uint32_t ordinal = 1; ordinal <= 3; ++ordinal) + { + const WriteOnceKey key = kLayout.writeOnceManifestKey(manifest(ordinal)); + EXPECT_TRUE(std::holds_alternative(op.create(key.str(), "body-" + std::to_string(ordinal), Retry::once()))); + keys.present.push_back(key); + } + return keys; +} + +} + +TEST(CASBulkDeleteBackend, InMemoryDeletesPresentKeysAndTreatsAbsentAsSuccess) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + Keys keys = seed(op); + std::vector batch = keys.present; + batch.push_back(keys.absent); + + op.removeManyWriteOnce(batch, Retry::once()); + + for (const WriteOnceKey & key : batch) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); + EXPECT_EQ(backend->bulkRemoveCalls(), 1u); +} + +TEST(CASBulkDeleteBackend, InMemoryHeldDeletesLandLater) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + Keys keys = seed(op); + + backend->setHoldDeletes(true); + op.removeManyWriteOnce(keys.present, Retry::once()); + for (const WriteOnceKey & key : keys.present) + EXPECT_TRUE(op.head(key.str(), Retry::once()).has_value()) << "held, not landed: " << key.str(); + while (backend->pendingDeletes() > 0) + backend->landPendingDelete(0); + for (const WriteOnceKey & key : keys.present) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); +} + +TEST(CASBulkDeleteBackend, InMemoryArmedFailureFiresOnceAndTheHookRunsBeforeTheDeletes) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + Keys keys = seed(op); + + size_t hook_runs = 0; + backend->onBeforeBulkRemove([&] { ++hook_runs; }); + backend->failNextBulkRemoveWith(std::make_exception_ptr(Poco::TimeoutException("injected"))); + + op.removeManyWriteOnce(keys.present, Retry::standard()); /// the engine reissues the chunk + EXPECT_EQ(backend->bulkRemoveCalls(), 2u); + EXPECT_EQ(hook_runs, 1u) << "the hook runs on the attempt that deletes, not on the refused one"; + for (const WriteOnceKey & key : keys.present) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); +} + +TEST(CASBulkDeleteBackend, InstrumentedCountsOneRequestAndOneDeletePerKeyClass) +{ + auto inner = std::make_shared(); + auto backend = std::make_shared(inner); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + Keys keys = seed(op); + const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(kNs, DB::UInt128(0x55)); + const WriteOnceKey log = kLayout.writeOnceRefLogKey(life, RefTxnId{1, 1}); + ASSERT_TRUE(std::holds_alternative(op.create(log.str(), "log", Retry::once()))); + + const auto requests_before = ProfileEvents::global_counters[ProfileEvents::CASBulkDeleteRequests].load(); + const auto manifest_before = ProfileEvents::global_counters[ProfileEvents::CASManifestDelete].load(); + const auto root_before = ProfileEvents::global_counters[ProfileEvents::CASRootDelete].load(); + + std::vector batch = keys.present; + batch.push_back(log); + op.removeManyWriteOnce(batch, Retry::once()); + + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASBulkDeleteRequests].load() - requests_before, 1u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASManifestDelete].load() - manifest_before, 3u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRootDelete].load() - root_before, 1u); +} + +#if USE_AWS_S3 +TEST(CASBulkDeleteBackend, ThrottlingRefusesTheChunkOnceAndTheEngineReissuesIt) +{ + auto inner = std::make_shared(); + auto backend = std::make_shared(inner, ThrottlingBackend::Mode::FirstPerKey, 1, 429); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + /// Seeded under `Retry::standard()`: the FirstPerKey refusal on each key's own create is an + /// AMBIGUOUS attempt the engine must reissue to land at all, which only a reissuable policy grants. + std::vector present; + for (uint32_t ordinal = 1; ordinal <= 3; ++ordinal) + { + const WriteOnceKey key = kLayout.writeOnceManifestKey(manifest(ordinal)); + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "body-" + std::to_string(ordinal), Retry::standard()))); + present.push_back(key); + } + /// the FirstPerKey refusal is spent on these keys' writes above; the bulk delete's own request + /// each key names is the SECOND request naming it and passes unrefused. + op.removeManyWriteOnce(present, Retry::standard()); + for (const WriteOnceKey & key : present) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); + EXPECT_GE(backend->refusals(present.front().str()), 1u); +} +#endif + +#if USE_AWS_S3 +TEST(CASBulkDeleteBackend, EmulatedModeDeletesUnderTheEmulationLockAndForgetsTheTokens) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::EmulatedSingleProcess); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + Keys keys = seed(op); + std::vector batch = keys.present; + batch.push_back(keys.absent); + + op.removeManyWriteOnce(batch, Retry::once()); + + for (const WriteOnceKey & key : batch) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); + /// A recreate at a deleted key must mint a fresh incarnation, which is what the token bookkeeping + /// after a delete exists for. + EXPECT_TRUE(std::holds_alternative(op.create(keys.present.front().str(), "again", Retry::once()))); +} + +TEST(CASBulkDeleteBackend, LocalObjectStorageRefusesTheProfileOverload) +{ + auto storage = DB::Cas::tests::makeLocalObjectStorageForTest(); + DB::StoredObjects objects{DB::StoredObject("p/anything")}; + expectThrowsCode(DB::ErrorCodes::NOT_IMPLEMENTED, [&] + { + storage->removeObjectsIfExistUnderProfile(objects, DB::ObjectStorageRetryProfile::SingleAttempt, 1000); + }); +} +#endif diff --git a/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp b/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp new file mode 100644 index 000000000000..fc84d67ebc61 --- /dev/null +++ b/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp @@ -0,0 +1,91 @@ +#include + +#include +#include +#include +#include +#include + +/// The engine sends one chunk of at most 1000 write-once keys as one request under the ordinary +/// attempt loop; a failed attempt reissues the whole chunk, which is sound because a key the failed +/// attempt already deleted is absent on the reissue, and absence is success. + +namespace ProfileEvents +{ + extern const Event CASRequestReissue; +} + +using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::openRequestsForTest; + +namespace +{ + +const Layout kLayout{"p"}; +const RootNamespace kNs{"test/aa@cas@"}; + +std::vector manifestKeys(uint32_t count) +{ + std::vector keys; + for (uint32_t ordinal = 1; ordinal <= count; ++ordinal) + keys.push_back(kLayout.writeOnceManifestKey( + ManifestId{kNs, ManifestRef{.writer_epoch = 1, .build_sequence = 1, .manifest_ordinal = ordinal}})); + return keys; +} + +} + +TEST(CASBulkDeleteEngine, AFailedAttemptReissuesTheWholeChunkAndAlreadyDeletedKeysAreSuccess) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const std::vector keys = manifestKeys(5); + for (const WriteOnceKey & key : keys) + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "b", Retry::once()))); + /// Half the chunk is gone before the failed attempt reports: the reissue must still succeed. + { + const auto h = op.head(keys[0].str(), Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ(op.remove(keys[0].str(), h->etag, Retry::once()), Removal::Removed); + } + backend->failNextBulkRemoveWith(std::make_exception_ptr(Poco::TimeoutException("injected"))); + const auto reissues_before = ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load(); + + op.removeManyWriteOnce(keys, Retry::standard()); + + EXPECT_EQ(backend->bulkRemoveCalls(), 2u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load() - reissues_before, 1u); + for (const WriteOnceKey & key : keys) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); +} + +TEST(CASBulkDeleteEngine, AnEmptyChunkIsNoRequest) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + op.removeManyWriteOnce({}, Retry::once()); + EXPECT_EQ(backend->bulkRemoveCalls(), 0u); +} + +TEST(CASBulkDeleteEngine, ExactlyOneThousandKeysIsOneRequest) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + op.removeManyWriteOnce(manifestKeys(1000), Retry::once()); /// all absent: success, one request + EXPECT_EQ(backend->bulkRemoveCalls(), 1u); +} + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASBulkDeleteEngineDeathTest, MoreThanOneThousandKeysIsACallerBug) +{ + auto backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + EXPECT_DEATH({ op.removeManyWriteOnce(manifestKeys(1001), Retry::once()); }, "removeManyWriteOnce"); + EXPECT_EQ(backend->bulkRemoveCalls(), 0u); +} +#endif diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 554af1650c0b..f81fc7b8e591 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -1276,6 +1276,7 @@ class FailDeletesUnderPrefixBackend : public Backend throw Poco::TimeoutException("injected transient delete failure for " + key); return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { diff --git a/src/Disks/tests/gtest_cas_gc_key_reader.cpp b/src/Disks/tests/gtest_cas_gc_key_reader.cpp new file mode 100644 index 000000000000..4f3938f01698 --- /dev/null +++ b/src/Disks/tests/gtest_cas_gc_key_reader.cpp @@ -0,0 +1,134 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +/// A reader hands a sequential walk its next object and lets the walk say which keys it will want +/// (hint) and which hinted keys it will never take (discard). The inline reader ignores hints; the +/// read-ahead reader turns them into worker requests and counts a discarded one as wasted at once. + +namespace CurrentMetrics +{ + extern const Metric LocalThread; + extern const Metric LocalThreadActive; + extern const Metric LocalThreadScheduled; +} + +namespace ProfileEvents +{ + extern const Event CASGCReadAheadWasted; + extern const Event CASGCReadAheadHit; + extern const Event CASGCReadAheadMiss; +} + +using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::openRequestsForTest; + +namespace +{ + +struct Rig +{ + std::shared_ptr backend = std::make_shared(); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + ThreadPool pool{CurrentMetrics::LocalThread, CurrentMetrics::LocalThreadActive, + CurrentMetrics::LocalThreadScheduled, /*max_threads*/ 4, /*max_free_threads*/ 4, /*queue_size*/ 0}; + + void put(const String & key, const String & bytes) + { + ASSERT_TRUE(std::holds_alternative(op.create(key, bytes, Retry::once()))) << key; + } +}; + +uint64_t wasted() +{ + return ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadWasted].load(); +} + +} + +TEST(CASGCKeyReader, DiscardCountsWastedAtOnceAndALaterTakeReadsInline) +{ + Rig rig; + rig.put("k1", "one"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + ReadAheadKeyReader reader(reads); + rig.backend->resetCounts(); + + const uint64_t wasted_before = wasted(); + reader.hint("k1"); + EXPECT_EQ(reader.pending(), 1u); + reader.discard("k1"); + EXPECT_EQ(reader.pending(), 0u); + EXPECT_EQ(wasted() - wasted_before, 1u); + + const auto got = reader.take("k1"); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->bytes, "one"); + EXPECT_EQ(rig.backend->getCount("k1"), 2u) << "the discarded request and the inline one"; +} + +TEST(CASGCKeyReader, DiscardOfAnUnhintedKeyIsANoOp) +{ + Rig rig; + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + ReadAheadKeyReader reader(reads); + const uint64_t wasted_before = wasted(); + reader.discard("never-hinted"); + EXPECT_EQ(wasted() - wasted_before, 0u); + EXPECT_EQ(reader.pending(), 0u); +} + +TEST(CASGCKeyReader, DiscardSwallowsAWorkerFailureThatATakeWouldRethrow) +{ + Rig rig; + rig.put("k1", "one"); + GcReadAhead reads(rig.op, rig.requests, rig.pool, 4); + ReadAheadKeyReader reader(reads); + + rig.backend->failNextReadWith("k1", std::make_exception_ptr(std::runtime_error("injected worker fault"))); + reader.hint("k1"); + EXPECT_NO_THROW(reader.discard("k1")); + + rig.backend->failNextReadWith("k1", std::make_exception_ptr(std::runtime_error("injected worker fault"))); + reader.hint("k1"); + EXPECT_THROW(static_cast(reader.take("k1")), std::runtime_error); +} + +TEST(CASGCKeyReader, InlineReaderHintsNothingAndReadsOnTake) +{ + Rig rig; + rig.put("k1", "one"); + InlineKeyReader reader(rig.op); + rig.backend->resetCounts(); + EXPECT_EQ(reader.window(), 0u); + reader.hint("k1"); + EXPECT_EQ(rig.backend->getCount("k1"), 0u); + EXPECT_EQ(reader.pending(), 0u); + const auto got = reader.take("k1"); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->bytes, "one"); + EXPECT_EQ(rig.backend->getCount("k1"), 1u); + reader.discard("k1"); +} + +TEST(CASGCKeyReader, ReadAheadReaderWindowAndPendingAreTheReadAheads) +{ + Rig rig; + GcReadAhead reads(rig.op, rig.requests, rig.pool, 8); + ReadAheadKeyReader reader(reads); + EXPECT_EQ(reader.window(), reads.window()); + EXPECT_EQ(reader.window(), 32u); + rig.put("a", "1"); + reader.hint("a"); + EXPECT_EQ(reader.pending(), reads.pending()); + static_cast(reader.take("a")); +} diff --git a/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp b/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp new file mode 100644 index 000000000000..d17cffd1fbde --- /dev/null +++ b/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp @@ -0,0 +1,163 @@ +#include + +#include +#include +#include +#include +#include + +/// The manifest_deletes phase sends owner-removed manifest bodies to the store in chunks of +/// write-once keys, one request per chunk, and records every chunk that succeeded before a later +/// one can fail. + +namespace ProfileEvents +{ + extern const Event CASBulkDeleteRequests; +} + +using namespace DB::Cas; +using namespace DB::Cas::tests; + +namespace +{ + +const UInt128 kGc = hexToU128("00000000000000000000000000000001"); +const RootNamespace kNs{"00/aa@cas@"}; + +ManifestRef ref(uint64_t seq) +{ + return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = 1}; +} + +/// `count` tables, each with one manifest, published committed and then dropped, so the fold sees +/// `count` owner removals and `mf_cleanup` carries `count` bodies. +std::vector seedDroppedManifests(Backend & backend, const Layout & layout, uint64_t count) +{ + std::vector ids; + for (uint64_t i = 1; i <= count; ++i) + { + const ManifestRef r = ref(i); + writeBlobBody(backend, layout, DB::UInt128(0x1000 + i)); + writeManifestRaw(backend, layout, kNs, r, {blobEntryFor("a", DB::UInt128(0x1000 + i))}); + const String table = "t" + std::to_string(i); + publishCommittedTransition(backend, layout, kNs, table, std::nullopt, r); + dropRefTransition(backend, layout, kNs, table, r); + ids.push_back(ManifestId{kNs, r}); + } + return ids; +} + +/// Runs rounds until every listed manifest is gone or `max_rounds` passed; returns the sum of +/// `manifests_deleted` over the rounds that led. +uint64_t reclaim(Gc & gc, PoolPtr store, Backend & backend, const std::vector & ids, size_t max_rounds) +{ + uint64_t total = 0; + for (size_t round = 0; round < max_rounds; ++round) + { + const RoundReport rep = runRegularRoundReclaiming(gc); + if (rep.acquired_lease) + total += rep.manifests_deleted; + store->renewWatermarkOnce(); + bool any_left = false; + OperationForTest op(backend); + for (const ManifestId & id : ids) + any_left |= (*op).head(store->layout().manifestKey(id), Retry::once()).has_value(); + if (!any_left) + break; + } + return total; +} + +} + +TEST(CASGCManifestBulkDelete, FiveBodiesInChunksOfTwoAreThreeRequests) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_bulk_delete_chunk_keys = 2, .gc_fold_max_defer_rounds = 0}); + const auto ids = seedDroppedManifests(*backend, store->layout(), 5); + const auto requests_before = ProfileEvents::global_counters[ProfileEvents::CASBulkDeleteRequests].load(); + + Gc gc(store, kGc); + const uint64_t deleted = reclaim(gc, store, *backend, ids, 16); + + EXPECT_EQ(deleted, 5u); + EXPECT_EQ(backend->bulkRemoveCalls(), 3u) << "2 + 2 + 1"; + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASBulkDeleteRequests].load() - requests_before, 3u); + OperationForTest op(*backend); + for (const ManifestId & id : ids) + EXPECT_FALSE((*op).head(store->layout().manifestKey(id), Retry::once()).has_value()); +} + +TEST(CASGCManifestBulkDelete, AThrowInTheSecondChunkKeepsTheFirstChunksAuditAndAbortsTheRound) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_bulk_delete_chunk_keys = 2, .gc_fold_max_defer_rounds = 0}); + const auto ids = seedDroppedManifests(*backend, store->layout(), 5); + + std::vector> manifest_phase_rows; + Gc gc(store, kGc); + gc.setPhaseSink([&](const GcPhaseRecord & rec) + { + if (rec.phase == "manifest_deletes") + manifest_phase_rows.push_back(rec.metrics); + }); + /// Rounds until the fold has adopted the removals; the first round whose manifest_deletes phase + /// has work is the one the fault is armed for. + size_t calls = 0; + /// The hook throws on EVERY attempt of the second chunk, so the engine's policy is exhausted and + /// the round aborts; the counter keeps climbing across the reissues, which is why the arm is + /// "second call and later" rather than "exactly the second call". + backend->onBeforeBulkRemove([&] + { + if (++calls >= 2) + throw Poco::TimeoutException("injected into the second chunk, every attempt"); + }); + + bool aborted = false; + for (size_t round = 0; round < 16 && !aborted; ++round) + { + try + { + static_cast(runRegularRoundReclaiming(gc)); + } + catch (const Poco::Exception &) + { + aborted = true; + } + /// A round that exhausted the full retry window (up to `Retry::standard()`'s 90s) may have + /// outlasted the mount lease itself, so the aborted round's own lease-renewal attempt can + /// throw too -- irrelevant to what this test asserts, so skip it once aborted. + if (!aborted) + store->renewWatermarkOnce(); + } + ASSERT_TRUE(aborted); + ASSERT_FALSE(manifest_phase_rows.empty()); + /// The aborted round's row was never emitted (the phase threw), so the last emitted row belongs + /// to an earlier, empty round; what proves the first chunk's audit survived is the store: exactly + /// the first chunk's two bodies are gone. + OperationForTest op(*backend); + size_t gone = 0; + for (const ManifestId & id : ids) + gone += !(*op).head(store->layout().manifestKey(id), Retry::once()).has_value(); + EXPECT_EQ(gone, 2u); +} + +TEST(CASGCManifestBulkDelete, ASuppressedRoundMakesNoRequest) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_fold_max_defer_rounds = 0}); + const auto ids = seedDroppedManifests(*backend, store->layout(), 3); + Gc gc(store, kGc); + for (size_t round = 0; round < 4; ++round) + { + static_cast(gc.runRegularRound({}, /*allow_steal*/ true, UniversePolicy::StageA_Suppressed)); + store->renewWatermarkOnce(); + } + EXPECT_EQ(backend->bulkRemoveCalls(), 0u); + OperationForTest op(*backend); + for (const ManifestId & id : ids) + EXPECT_TRUE((*op).head(store->layout().manifestKey(id), Retry::once()).has_value()); +} diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 8d43ca7b7c4c..7200757f11d1 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -999,6 +999,7 @@ class AlwaysVanishesBackend final : public DB::Cas::Backend std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { diff --git a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp index cc46229aafdd..2c68bf8492fd 100644 --- a/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp +++ b/src/Disks/tests/gtest_cas_orphan_manifest_sweep.cpp @@ -389,9 +389,11 @@ TEST(CASOrphanManifestSweep, CursorPageDeletesObservedBodyWhenCatalogOmitsNamesp EXPECT_FALSE(headExists(*backend, store->layout().manifestKey(ManifestId{ns, r}))); } -/// The candidate body and token must be frozen before the later catalog cut. A concurrent same-key -/// replacement after that observation is a new physical incarnation and must lose the old-token delete. -TEST(CASOrphanManifestSweep, CursorPageCannotDeleteManifestReplacedAfterObservation) +/// Manifest keys are write-once, so a same-key rewrite after the observation below never happens in +/// production; this backend forces one anyway to prove the page does not need the old freeze-before-cut +/// discipline to stay correct. The body is read only after the catalog cut, so it sees whatever +/// incarnation is actually there at that point and deletes it under its own current token. +TEST(CASOrphanManifestSweep, CursorPageDeletesTheIncarnationSeenAfterTheCatalogCut) { auto backend = std::make_shared(); auto store = openPoolForTest(backend); @@ -408,8 +410,8 @@ TEST(CASOrphanManifestSweep, CursorPageCannotDeleteManifestReplacedAfterObservat const ManifestSweepResult result = sweepManifestCursorPageForTest(*store, "", /*list_budget=*/100, /*delete_budget=*/10); EXPECT_TRUE(backend->didReplace()); - EXPECT_EQ(result.deleted, 0u); - EXPECT_TRUE(headExists(*backend, key)); + EXPECT_EQ(result.deleted, 1u); + EXPECT_FALSE(headExists(*backend, key)); } /// Any duplicate current life id makes the catalog-to-physical join ambiguous. The cursor page is diff --git a/src/Disks/tests/gtest_cas_orphan_nomination.cpp b/src/Disks/tests/gtest_cas_orphan_nomination.cpp index 5dda01e60328..fb4cd6bd7a81 100644 --- a/src/Disks/tests/gtest_cas_orphan_nomination.cpp +++ b/src/Disks/tests/gtest_cas_orphan_nomination.cpp @@ -81,7 +81,7 @@ size_t condemnedCount(CasOperation & op, const Layout & layout) return count; } -class NominationBackend : public InMemoryBackend +class NominationBackend : public CountingBackend { public: /// The sweep's exact-token delete reaches the store through the keyed removal, so the fault is @@ -101,7 +101,7 @@ class NominationBackend : public InMemoryBackend static_cast(InMemoryBackend::write(key, got->bytes, got->value, access)); } } - return InMemoryBackend::remove(key, expected_value, access); + return CountingBackend::remove(key, expected_value, access); } Layout layout{"p"}; @@ -248,6 +248,31 @@ TEST(CASOrphanNomination, RetiresExactManifestSourcesBeforeDelete) "committed+produced ref transaction unapplied"; } +/// A page reads the body of a candidate only. Five live manifests share the namespace with the one +/// orphan candidate; they are active under the floor, are retained from their keys alone, and cost no +/// GET. The candidate costs exactly one. +TEST(CASOrphanNomination, OnlyCandidatesCostABodyRead) +{ + ReadyFixture f = makeReadyFixture(); + const Layout & layout = f.store->layout(); + std::vector live_keys; + for (uint32_t ordinal = 1; ordinal <= 5; ++ordinal) + { + const ManifestRef live{.writer_epoch = kCandidateEpoch, .build_sequence = 7, .manifest_ordinal = ordinal}; + writeManifestRaw(*f.backend, layout, f.ns, live, {blobEntryFor("live", DB::UInt128(0xA000 + ordinal))}); + live_keys.push_back(layout.manifestKey(ManifestId{f.ns, live})); + } + f.backend->resetCounts(); + + const ManifestSweepResult result = planManifestCursorPage( + *f.store, "", /*list_budget=*/100, /*nomination_budget=*/100, /*catalog_recovery_authoritative=*/true, nullptr); + + ASSERT_EQ(result.nominations.size(), 1u); + EXPECT_EQ(f.backend->getCount(layout.manifestKey(f.candidate)), 1u); + for (const String & key : live_keys) + EXPECT_EQ(f.backend->getCount(key), 0u) << key; +} + /// A nomination must exact-GET and decode the manifest before it can derive any source-edge identity. /// An undecodable body is retained and surfaced without aborting the rest of the round. TEST(CASOrphanNomination, CorruptManifestIsRetainedAndSurfaced) diff --git a/src/Disks/tests/gtest_cas_orphan_sweep_requests.cpp b/src/Disks/tests/gtest_cas_orphan_sweep_requests.cpp new file mode 100644 index 000000000000..34e87af705ed --- /dev/null +++ b/src/Disks/tests/gtest_cas_orphan_sweep_requests.cpp @@ -0,0 +1,390 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace CurrentMetrics +{ + extern const Metric LocalThread; + extern const Metric LocalThreadActive; + extern const Metric LocalThreadScheduled; +} + +namespace ProfileEvents +{ + extern const Event CASGCReadAheadWasted; + extern const Event CASGCReadAheadHit; +} + +/// The orphan-manifest sweep's request shape per page. The floor a namespace's builds are judged +/// against is one mount body per server root, so a page reads it once per namespace, not once per +/// listed build; the tests below count the mount-key reads and pin the pure eligibility predicate. + +using namespace DB::Cas; +using namespace DB::Cas::tests; + +namespace +{ + +/// A three-segment namespace, the shape a real table gets (`/store/<3hex>/@cas@`), so the +/// floor lookup has three `/`-prefixes to try and two of them miss. +const RootNamespace kNs{"test/store/465/aa@cas@"}; + +ManifestRef build(uint64_t seq) +{ + return ManifestRef{.writer_epoch = 1, .build_sequence = seq, .manifest_ordinal = 1}; +} + +struct PageFixture +{ + std::shared_ptr backend = std::make_shared(); + PoolPtr store; + uint64_t manifests = 0; + + explicit PageFixture(uint64_t manifests_, uint64_t min_active_build_sequence) + : manifests(manifests_) + { + PoolConfig config; + config.pool_prefix = "p"; + config.server_root_id = "test"; + config.manifest_sweep_list_budget_keys = 1000; + config.manifest_sweep_delete_budget_keys = 100; + config.gc_fold_max_defer_rounds = 0; + store = Pool::open(backend, config); + const Layout & layout = store->layout(); + casAdmitEntry(*backend, layout, kNs); + /// One committed birth log at {1,1} and a checkpoint naming it, so the namespace has a protection + /// view and every eligible key reaches the premise (which retains it for lack of fold coverage). + /// The live ref's own manifest occupies build_sequence == manifests (it is itself one of the + /// `manifests` listed objects, always active); the loop below fills the debris below it. + publishAt(*backend, layout, kNs, RefTxnId{1, 1}, "live", /*build_sequence=*/manifests, + DB::UInt128(0x7001), /*birth=*/true); + writeRecoverableCkptForRawFixture(*backend, layout, kNs, RefCkpt{ + .life_epoch = 1, + .committed_through = RefTxnId{1, 1}, + .checkpoint_snapshot_id = std::nullopt, + .last_epoch_seal = std::nullopt, + }); + for (uint64_t seq = 1; seq < manifests; ++seq) + writeManifestRaw(*backend, layout, kNs, build(seq), {blobEntryFor("a", DB::UInt128(0x100 + seq))}); + setWatermarkMinActive(*backend, layout, "test", /*writer_epoch=*/1, min_active_build_sequence); + backend->resetCounts(); + } + + ManifestSweepResult page() + { + return planManifestCursorPage(*store, "", /*list_budget=*/1000, /*nomination_budget=*/100, + /*catalog_recovery_authoritative=*/true, nullptr); + } +}; + +} + +TEST(CASOrphanSweepRequests, FloorIsReadOncePerNamespacePerPage) +{ + PageFixture f(/*manifests=*/50, /*min_active=*/25); + const Layout & layout = f.store->layout(); + const ManifestSweepResult result = f.page(); + + EXPECT_EQ(result.listed, 50u); + EXPECT_EQ(result.floor_lookups, 1u); + EXPECT_EQ(result.floor_reads, 3u); + EXPECT_EQ(f.backend->getCount(layout.mountKey("test/store/465")), 1u); + EXPECT_EQ(f.backend->getCount(layout.mountKey("test/store")), 1u); + EXPECT_EQ(f.backend->getCount(layout.mountKey("test")), 1u); + /// Builds 1..24 are retired under the floor and reach the premise, which retains them for lack of + /// coverage; builds 25..50 are active and never get that far. + EXPECT_EQ(result.retained_no_coverage, 24u); + EXPECT_TRUE(result.nominations.empty()); +} + +TEST(CASOrphanSweepRequests, AbsentFloorRetainsEverythingWithOneLookup) +{ + PageFixture f(/*manifests=*/10, /*min_active=*/100); + const Layout & layout = f.store->layout(); + { + OperationForTest op(*f.backend); + const auto h = (*op).head(layout.mountKey("test"), Retry::once()); + ASSERT_TRUE(h.has_value()); + ASSERT_EQ((*op).remove(layout.mountKey("test"), h->etag, Retry::once()), Removal::Removed); + } + f.backend->resetCounts(); + const ManifestSweepResult result = f.page(); + + EXPECT_EQ(result.floor_lookups, 1u); + EXPECT_EQ(result.floor_reads, 3u); + EXPECT_EQ(result.listed, 10u); + EXPECT_EQ(result.skipped, 10u); + EXPECT_EQ(result.retained_no_coverage, 0u) << "an absent floor admits nothing, so no key reaches the premise"; +} + +TEST(CASOrphanSweepRequests, RetainedKeysCostNoBodyRead) +{ + PageFixture f(/*manifests=*/50, /*min_active=*/25); + const Layout & layout = f.store->layout(); + const ManifestSweepResult result = f.page(); + EXPECT_EQ(result.retained_no_coverage, 24u); + for (uint64_t seq = 1; seq <= 50; ++seq) + EXPECT_EQ(f.backend->getCount(layout.manifestKey(ManifestId{kNs, build(seq)})), 0u) << seq; +} + +TEST(CASOrphanSweepRequests, PrefixEligibleUnderIsTheFourComparisons) +{ + MountLease floor; + floor.writer_epoch = 3; + floor.min_active_build_sequence = 10; + EXPECT_TRUE(prefixEligibleUnder(floor, BuildPrefix{.writer_epoch = 2, .build_sequence = 999})); + EXPECT_FALSE(prefixEligibleUnder(floor, BuildPrefix{.writer_epoch = 4, .build_sequence = 1})); + EXPECT_TRUE(prefixEligibleUnder(floor, BuildPrefix{.writer_epoch = 3, .build_sequence = 9})); + EXPECT_FALSE(prefixEligibleUnder(floor, BuildPrefix{.writer_epoch = 3, .build_sequence = 10})); + floor.min_active_build_sequence = std::numeric_limits::max(); + EXPECT_TRUE(prefixEligibleUnder(floor, BuildPrefix{.writer_epoch = 3, .build_sequence = 10})); + EXPECT_FALSE(prefixEligibleUnder(std::nullopt, BuildPrefix{.writer_epoch = 1, .build_sequence = 1})); +} + +namespace +{ + +/// Deletes the mount key the first time it is read, so the page decides with a floor whose object is +/// gone by the time it decides. Retirement is permanent, so the decisions must be the ones the floor +/// admitted when read, and no active build may be nominated. +class MountVanishesBackend final : public CountingBackend +{ +public: + using CountingBackend::read; + std::optional read(const String & key, TransportAccess & access) override + { + auto got = CountingBackend::read(key, access); + if (got && key == mount_key && !fired) + { + fired = true; + static_cast(InMemoryBackend::remove(key, got->value, access)); + } + return got; + } + String mount_key; + bool fired = false; +}; + +} + +TEST(CASOrphanSweepRequests, MountVanishingMidPageKeepsTheDecisionsOfTheFloorAsRead) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .manifest_sweep_list_budget_keys = 1000, + .manifest_sweep_delete_budget_keys = 100, + .gc_fold_max_defer_rounds = 0}); + const Layout & layout = store->layout(); + casAdmitEntry(*backend, layout, kNs); + publishAt(*backend, layout, kNs, RefTxnId{1, 1}, "live", /*build_sequence=*/51, DB::UInt128(0x7001), /*birth=*/true); + writeRecoverableCkptForRawFixture(*backend, layout, kNs, RefCkpt{ + .life_epoch = 1, .committed_through = RefTxnId{1, 1}, + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); + for (uint64_t seq = 1; seq <= 50; ++seq) + writeManifestRaw(*backend, layout, kNs, build(seq), {blobEntryFor("a", DB::UInt128(0x100 + seq))}); + setWatermarkMinActive(*backend, layout, "test", 1, /*min_active=*/25); + backend->mount_key = layout.mountKey("test"); + + const ManifestSweepResult result = planManifestCursorPage(*store, "", 1000, 100, true, nullptr); + EXPECT_TRUE(backend->fired); + EXPECT_EQ(result.floor_lookups, 1u); + EXPECT_EQ(result.retained_no_coverage, 24u) << "the 24 retired builds were decided under the floor as read"; + EXPECT_TRUE(result.nominations.empty()); + OperationForTest op(*backend); + EXPECT_FALSE((*op).head(layout.mountKey("test"), Retry::once()).has_value()); +} + +namespace +{ + +ThreadPool makeReadPool(size_t threads) +{ + return ThreadPool{CurrentMetrics::LocalThread, CurrentMetrics::LocalThreadActive, + CurrentMetrics::LocalThreadScheduled, threads, threads, /*queue_size*/ 0}; +} + +/// Every GET the page issued, by key, in whichever thread it ran. +std::map getsOf(CountingBackend & backend) +{ + std::map gets; + for (const String & key : backend.touchedKeys()) + if (const uint64_t n = backend.getCount(key); n != 0) + gets[key] = n; + return gets; +} + +struct PageOutcome +{ + uint64_t listed, skipped, deleted, undecodable, retained_no_coverage, retained_hold, + retained_unconsumed_seal, retained_tail_removal, retained_work_budget, floor_lookups, floor_reads; + bool wrapped; + String next_cursor; + bool operator==(const PageOutcome &) const = default; +}; + +PageOutcome outcomeOf(const ManifestSweepResult & r) +{ + return {r.listed, r.skipped, r.deleted, r.undecodable, r.retained_no_coverage, r.retained_hold, + r.retained_unconsumed_seal, r.retained_tail_removal, r.retained_work_budget, + r.floor_lookups, r.floor_reads, r.wrapped, r.next_cursor}; +} + +bool sameNomination(const ManifestSweepResult::Nomination & a, const ManifestSweepResult::Nomination & b) +{ + if (!(a.id == b.id) || a.key != b.key || a.token.dialect != b.token.dialect || a.token.value != b.token.value) + return false; + if (a.source_retirements.size() != b.source_retirements.size()) + return false; + for (size_t i = 0; i < a.source_retirements.size(); ++i) + if (!(a.source_retirements[i].ref == b.source_retirements[i].ref) + || !(a.source_retirements[i].source_id == b.source_retirements[i].source_id)) + return false; + return true; +} + +/// Both runs decide candidates from the SAME listed order and append nominations in that same order, +/// whichever reader fetched their bytes, so an index-wise comparison is exact -- no sort needed. +bool sameNominations(const std::vector & a, + const std::vector & b) +{ + if (a.size() != b.size()) + return false; + for (size_t i = 0; i < a.size(); ++i) + if (!sameNomination(a[i], b[i])) + return false; + return true; +} + +/// A fixture whose sweep has REAL candidates and a committed-tail walk long enough to hint ahead, +/// entirely WITHOUT crossing an epoch: this life is born directly at epoch 2 (`{2,1}`, no +/// `prev_epoch_seal` needed -- genesis, not a chain link), and `kDebrisManifests` unowned raw +/// manifests sit at prefix epoch 1 -- a legacy build-prefix number, never part of this life's own ref +/// stream, but eligible under the floor (an old epoch is always eligible) and covered by the folded +/// cursor sitting in epoch 2 (rule 1) all the same, so they become genuine nominations. `kEpochTwoLogs` +/// ordinary committed grants after the birth give the committed-tail walk (and the recovery walk, which +/// starts at this same genesis) a range comfortably longer than one read-ahead window, all inside the +/// ONE epoch neither walk ever leaves -- unlike the epoch-crossing fixture below, whose hints legitimately +/// overshoot a seal and so cannot be expected to read the identical key set at every concurrency, this +/// fixture's GET set is invariant to concurrency, which is what the comparison after it needs. +constexpr uint64_t kDebrisManifests = 6; +constexpr uint64_t kEpochTwoLogs = 80; + +struct CandidateFixture +{ + std::shared_ptr backend = std::make_shared(); + PoolPtr store; + + CandidateFixture() + { + PoolConfig config; + config.pool_prefix = "p"; + config.server_root_id = "test"; + config.manifest_sweep_list_budget_keys = 1000; + config.manifest_sweep_delete_budget_keys = 100; + config.gc_fold_max_defer_rounds = 0; + store = Pool::open(backend, config); + const Layout & layout = store->layout(); + casAdmitEntry(*backend, layout, kNs); + + for (uint64_t seq = 1; seq <= kDebrisManifests; ++seq) + writeManifestRaw(*backend, layout, kNs, build(seq), {blobEntryFor("a", DB::UInt128(0x100 + seq))}); + publishAt(*backend, layout, kNs, RefTxnId{2, 1}, "live", /*build_sequence=*/2000, + DB::UInt128(0x7001), /*birth=*/true); + for (uint64_t seq = 2; seq <= kEpochTwoLogs; ++seq) + publishAt(*backend, layout, kNs, RefTxnId{2, seq}, "epoch2-" + std::to_string(seq), + 2000 + seq, DB::UInt128(0x9000 + seq), /*birth=*/false); + writeRecoverableCkptForRawFixture(*backend, layout, kNs, RefCkpt{ + .life_epoch = 2, .committed_through = RefTxnId{2, kEpochTwoLogs}, + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = std::nullopt}); + seedFoldCursorForTest(*backend, layout, kNs, RefTxnId{2, 1}); + setWatermarkMinActive(*backend, layout, "test", /*writer_epoch=*/2, /*min_active=*/1); + backend->resetCounts(); + } +}; + +} + +/// The read-ahead reader must issue the same GETs against the same keys as the inline reader, decide +/// the same way, and nominate the exact same candidates; only when the bytes arrive moves. The fixture +/// gives the page real candidates and a committed-tail walk spanning more than one window, so the +/// comparison actually exercises the read-ahead instead of vacuously agreeing over nothing. +TEST(CASOrphanSweepRequests, PageIsIdenticalInlineAndWithReadAhead) +{ + CandidateFixture inline_f; + const ManifestSweepResult inline_r = planManifestCursorPage(*inline_f.store, "", 1000, 100, true, nullptr); + const auto inline_gets = getsOf(*inline_f.backend); + ASSERT_EQ(inline_r.nominations.size(), kDebrisManifests) + << "the fixture must actually produce candidates, or this test proves nothing"; + + CandidateFixture ahead_f; + ThreadPool pool = makeReadPool(4); + const uint64_t hits_before = ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadHit].load(); + const ManifestSweepResult ahead_r = planManifestCursorPage( + *ahead_f.store, "", 1000, 100, true, nullptr, &pool, /*read_concurrency=*/16); + const uint64_t hits = ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadHit].load() - hits_before; + const auto ahead_gets = getsOf(*ahead_f.backend); + + EXPECT_GT(hits, 0u) << "the fixture's committed-tail walk and candidates must actually hit the " + "read-ahead, or this oracle could never catch a hinting regression"; + EXPECT_EQ(outcomeOf(inline_r), outcomeOf(ahead_r)); + EXPECT_TRUE(sameNominations(inline_r.nominations, ahead_r.nominations)); + EXPECT_EQ(inline_gets, ahead_gets); +} + +/// A committed tail that spans two epochs: the walk hints ids past the seal in the old epoch, which +/// do not exist, discards them at the crossing (at most one window), and hints the new epoch's ids. +TEST(CASOrphanSweepRequests, EpochCrossingDiscardsAtMostOneWindowAndTheNewEpochIsHinted) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .manifest_sweep_list_budget_keys = 1000, + .manifest_sweep_delete_budget_keys = 100, + .gc_fold_max_defer_rounds = 0}); + const Layout & layout = store->layout(); + casAdmitEntry(*backend, layout, kNs); + /// Epoch 1: birth at {1,1}, six ordinary logs {1,2..7}, seal at {1,8}. Epoch 2: {2,1..40}. + publishAt(*backend, layout, kNs, RefTxnId{1, 1}, "t1", /*build_sequence=*/100, DB::UInt128(0x7001), /*birth=*/true); + for (uint64_t seq = 2; seq <= 7; ++seq) + publishAt(*backend, layout, kNs, RefTxnId{1, seq}, "t" + std::to_string(seq), 100 + seq, DB::UInt128(0x7000 + seq), /*birth=*/false); + writeSealAt(*backend, layout, kNs, RefTxnId{1, 8}); + publishAt(*backend, layout, kNs, RefTxnId{2, 1}, "u1", 200, DB::UInt128(0x8001), /*birth=*/false, /*prev_epoch_seal=*/RefTxnId{1, 8}); + for (uint64_t seq = 2; seq <= 40; ++seq) + publishAt(*backend, layout, kNs, RefTxnId{2, seq}, "u" + std::to_string(seq), 200 + seq, DB::UInt128(0x8000 + seq), /*birth=*/false); + writeRecoverableCkptForRawFixture(*backend, layout, kNs, RefCkpt{ + .life_epoch = 1, .committed_through = RefTxnId{2, 40}, + .checkpoint_snapshot_id = std::nullopt, .last_epoch_seal = RefTxnId{1, 8}}); + writeManifestRaw(*backend, layout, kNs, build(1), {blobEntryFor("a", DB::UInt128(0x100))}); + setWatermarkMinActive(*backend, layout, "test", 2, /*min_active=*/1000); + backend->resetCounts(); + + const uint64_t wasted_before = ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadWasted].load(); + const uint64_t hits_before = ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadHit].load(); + ThreadPool pool = makeReadPool(4); + const ManifestSweepResult result = planManifestCursorPage(*store, "", 1000, 100, true, nullptr, &pool, 16); + const uint64_t wasted = ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadWasted].load() - wasted_before; + const uint64_t hits = ProfileEvents::global_counters[ProfileEvents::CASGCReadAheadHit].load() - hits_before; + + /// `publishAt` writes a manifest as a side effect of every call above, so the page's LIST sees + /// every one of those (47) plus the one raw manifest -- the fixture is about the hint/discard + /// mechanics of the ref-log walks a namespace's first candidate triggers, not about the listing. + EXPECT_EQ(result.listed, 48u); + /// The SAME reader crosses the epoch twice on this fixture: once inside the recovery walk + /// `activeManifestKeys` runs via `recoverRefTableDetailedFromAuthority`, and again in its own + /// committed-tail walk over the same {1,1}..{2,40} range (today's pre-existing double walk, not + /// something this change introduces) -- so at most two windows are discarded, not one. + EXPECT_LE(wasted, 128u) << "at most two windows at concurrency 16: one per walk crossing the epoch"; + EXPECT_GE(hits, 30u) << "the new epoch's logs were hinted and taken"; + /// Both walks read epoch 2's logs once each, so every key is read twice -- today's behaviour with + /// or without read-ahead, not something the hint/discard rule changes. + for (uint64_t seq = 1; seq <= 40; ++seq) + EXPECT_EQ(backend->getCount(layout.refLogKey(NamespaceLifeId::fromCatalogEntry(kNs, catalogLifeIdForTest(*backend, layout, kNs)), RefTxnId{2, seq})), 2u) << seq; +} diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 878fd536e1e5..68fb07080b04 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -203,6 +203,7 @@ class HeadThenDeleteOnceBackend final : public DB::Cas::Backend } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -247,6 +248,7 @@ class KeyCountingBackend final : public DB::Cas::Backend } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -976,6 +978,7 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupNeverGetsTheDyingObject) std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -1068,6 +1071,7 @@ TEST(CASPartWriteTxn, PutBlobCondemnedDedupPresentNeverGetsTheDyingObject) std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -1665,6 +1669,7 @@ TEST(CASPartWriteTxn, AdoptEvidenceRecordsTrustedManifestDependencyProofWithoutI } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index c490332f50eb..b5264ba1b151 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -77,6 +77,11 @@ class WriteCountingBackend final : public DB::Cas::Backend ++writes; return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override + { + ++writes; + inner->removeManyWriteOnce(keys, access); + } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -227,6 +232,12 @@ class ProbeWatchingBackend final : public DB::Cas::Backend note(key); return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override + { + for (const WriteOnceKey & key : keys) + note(key.str()); + inner->removeManyWriteOnce(keys, access); + } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -302,6 +313,7 @@ class ForwardingBackend : public DB::Cas::Backend std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -1488,6 +1500,7 @@ class FenceInAdoptWindowBackend final : public DB::Cas::Backend std::optional head(const String & key, TransportAccess & access) override { return inner->head(key, access); } RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override { return inner->list(prefix, cursor, limit, access); } RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override { return inner->remove(key, expected_value, access); } + void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override { inner->removeManyWriteOnce(keys, access); } std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { diff --git a/src/Disks/tests/gtest_cas_ref_gc.cpp b/src/Disks/tests/gtest_cas_ref_gc.cpp index 5c20e6298403..1e38e91ef7d1 100644 --- a/src/Disks/tests/gtest_cas_ref_gc.cpp +++ b/src/Disks/tests/gtest_cas_ref_gc.cpp @@ -11,6 +11,7 @@ #include +#include #include /// Task 12 required GC tests over the snapshot+log ref model (spec 2026-07-11-cas-ref-table-snapshot-log-design). @@ -124,49 +125,139 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend { Catalog, GcFence, + CatalogRebirth, }; enum class Timing : uint8_t { - BeforeFirstDelete, AfterFirstDelete, + DuringChunk, }; void arm( Authority authority_, Timing timing_, const Layout & layout, const String & first_cleanup_key_) { - authority = authority_; - timing = timing_; + arm(authority_, timing_, layout, first_cleanup_key_, std::nullopt); + } + + void arm( + Authority authority_, Timing timing_, const Layout & layout, + const String & first_cleanup_key_, const RootNamespace & reborn_ns_) + { + arm(authority_, timing_, layout, first_cleanup_key_, std::optional{reborn_ns_}); + } + + /// "Before the first chunk" has no backend-request seam of its own to hang off: `cleanupRefObjects` + /// issues no `HEAD`, and the chunk's own catalog/`gc/state` reads only happen ONCE, back to back, + /// immediately before the delete they license -- by the time either is observable from a backend + /// override, the chunk's catalog snapshot is already cached in `authorityHolds`'s local, and moving + /// the authority no longer changes what THIS chunk decides. The window that must be hit instead is + /// the round's own hot-scan catalog cut: `Gc::setPostHotScanCatalogReadHookForTest` (`CasGc.h`) + /// fires the instant that cut is taken, before the round -- and later `authorityHolds` -- does + /// anything else with it, so a move landed there is exactly "before the first chunk starts" and the + /// chunk's later fresh reads observe it. + /// + /// `Authority::Catalog` moves the catalog's token directly, right there in the hook: the round's + /// own `round_commit` CAS (phase 13) never touches the catalog, so nothing downstream collides. + /// `Authority::GcFence` cannot do the same for `gc/state`: bumping its lease THERE lands strictly + /// BEFORE `round_commit`'s own `gc/state` replace (which still holds the etag from lease adoption, + /// phase 1), so that replace loses its own CAS and the round throws before `cleanupRefObjects` + /// (phase 17) ever runs -- the "nothing deleted" assertions would pass vacuously, not because + /// cleanup refused. Instead, the hook only ARMS `armGcFenceMoveOnAuthorityHoldsRevalidationForTest`: the + /// actual lease bump is deferred to a LATER read of `gc/state`. Not the next one -- namespace + /// janitor / orphan-sweep bookkeeping between `round_commit` and `cleanupRefObjects` also touches + /// the catalog and `gc/state`, just never the two BACK TO BACK the way `authorityHolds` does + /// (catalog, then `gc/state`, nothing in between): that adjacency is the one place in a round only + /// `authorityHolds`'s own revalidation produces, so gating on it -- rather than on the catalog key + /// alone -- is what actually lands the move inside that SAME call, well after `round_commit`. + static void moveRefCleanupAuthorityBeforeFirstChunk(Authority authority_, CasOperation & op, const Layout & layout) + { + if (authority_ == Authority::GcFence) + throw std::logic_error( + "moveRefCleanupAuthorityBeforeFirstChunk is for Authority::Catalog/CatalogRebirth only -- " + "use armGcFenceMoveOnAuthorityHoldsRevalidationForTest for Authority::GcFence"); + /// Same-content rewrite: only the catalog's TOKEN moves (mints a fresh etag), never its + /// parsed content -- the pure "someone else touched this row" race `Authority::Catalog` + /// models, as opposed to `Authority::CatalogRebirth`'s actual incarnation bump. + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); + op.replace(layout.refCatalogKey(), encodeRefCatalog(snap.catalog), *snap.etag, Retry::standard()); + } + + /// Arms the seam `read` below fires on: see the doc comment above. Called from the test body + /// before the round (and so before any read-ahead worker exists), but still under the mutex, so + /// this method and `read`'s critical sections never race even under future reordering. + void armGcFenceMoveOnAuthorityHoldsRevalidationForTest(const Layout & layout) + { catalog_key = layout.refCatalogKey(); gc_state_key = layout.gcStateKey(); - first_cleanup_key = first_cleanup_key_; - armed = true; + std::lock_guard lock(seam_mutex); + catalog_seam_armed = true; } - /// The primitive `head` override below hides every base overload of that name. - using CountingBackend::head; - - /// Both seams hang off the transport primitives, so they fire whichever surface the cleanup pass - /// reaches the store through. - std::optional head(const String & key, DB::Cas::TransportAccess & access) override + /// `read` also runs on the GC read-ahead pool's threads (`CasGcReadAhead.cpp` schedules + /// `CasOperation::read` there; `gc_read_concurrency` defaults to 16), concurrently with the round + /// thread's own reads -- `catalog_seam_armed` and `last_control_key_read` below are shared mutable + /// state a pool thread's read can land between `authorityHolds`'s two reads, so both are read AND + /// written only under `seam_mutex`. `last_control_key_read` tracks only the catalog and `gc/state` + /// keys, never any other key a read-ahead worker fetches: those workers never touch either control + /// key (they fetch ref-log/manifest bodies), so an unrelated concurrent read can never perturb the + /// adjacency signal even though it runs lock-free between this method's two critical sections. + std::optional read(const String & key, DB::Cas::TransportAccess & access) override { - auto result = CountingBackend::head(key, access); - if (armed && timing == Timing::BeforeFirstDelete && key == first_cleanup_key) - moveAuthority(access); - return result; + bool fires_here = false; + { + std::lock_guard lock(seam_mutex); + fires_here = catalog_seam_armed && key == gc_state_key && last_control_key_read == catalog_key; + if (fires_here) + catalog_seam_armed = false; + if (key == catalog_key || key == gc_state_key) + last_control_key_read = key; + } + if (fires_here) + { + const auto current = CountingBackend::read(key, access); + if (!current) + throw std::runtime_error("test-injected cleanup authority object is absent"); + GcState moved = decodeGcState(current->bytes); + ++moved.lease.seq; + if (!write(key, encodeGcState(moved), current->value, access).has_value()) + throw std::runtime_error("test-injected cleanup authority move lost its CAS"); + return CountingBackend::read(key, access); /// the FRESH, post-move bytes, for THIS read + } + return CountingBackend::read(key, access); } - DB::Cas::Backend::RawRemoval remove(const String & key, const String & expected_value, - DB::Cas::TransportAccess & access) override + /// The `AfterFirstDelete` seam moves the authority once the first chunk's batch delete has + /// landed (so that chunk keeps whatever it already observed and only the NEXT chunk's + /// revalidation refuses); `DuringChunk` moves it after the chunk's revalidation but before its + /// batch delete lands, so the chunk in flight still completes under the authority it observed. + void removeManyWriteOnce(const std::vector & keys, DB::Cas::TransportAccess & access) override { - auto result = CountingBackend::remove(key, expected_value, access); - if (armed && timing == Timing::AfterFirstDelete && key == first_cleanup_key) + const bool names_first = std::any_of(keys.begin(), keys.end(), + [&](const DB::Cas::WriteOnceKey & key) { return key.str() == first_cleanup_key; }); + if (armed && timing == Timing::DuringChunk && names_first) + moveAuthority(access); /// after the revalidation, before the deletes land + CountingBackend::removeManyWriteOnce(keys, access); + if (armed && timing == Timing::AfterFirstDelete && names_first) moveAuthority(access); - return result; } private: + void arm( + Authority authority_, Timing timing_, const Layout & layout, + const String & first_cleanup_key_, std::optional reborn_ns_) + { + authority = authority_; + timing = timing_; + catalog_key = layout.refCatalogKey(); + gc_state_key = layout.gcStateKey(); + first_cleanup_key = first_cleanup_key_; + reborn_ns = std::move(reborn_ns_); + layout_for_rebirth_seed = &layout; + armed = true; + } + /// `access` is the token the caller's own primitive override already holds for its in-flight /// request; reused here for this method's extra read+write rather than minting a new CasRequests, /// exactly as `Backend::probeSentinelRaw`'s default implementation reuses one `access` across its @@ -174,7 +265,7 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend void moveAuthority(TransportAccess & access) { armed = false; - const String & key = authority == Authority::Catalog ? catalog_key : gc_state_key; + const String & key = authority == Authority::GcFence ? gc_state_key : catalog_key; const auto got = read(key, access); if (!got) throw std::runtime_error("test-injected cleanup authority object is absent"); @@ -186,16 +277,73 @@ class RefCleanupAuthorityRaceBackend : public CountingBackend ++moved.lease.seq; bytes = encodeGcState(moved); } + UInt128 reborn_incarnation = 0; + if (authority == Authority::CatalogRebirth) + { + RefCatalog catalog = decodeRefCatalog(bytes); + for (CatalogEntry & entry : catalog.entries) + if (reborn_ns && entry.ns == *reborn_ns) + { + entry.incarnation = entry.incarnation + 1; + reborn_incarnation = entry.incarnation; + } + bytes = encodeRefCatalog(catalog); + } if (!write(key, bytes, got->value, access).has_value()) throw std::runtime_error("test-injected cleanup authority move lost its CAS"); + + /// Give the reborn life SOMETHING of its own, landed right after its catalog row exists (any + /// earlier and an "unknown incarnation" sweep elsewhere in the SAME round can claim it, since + /// no catalog entry yet names that incarnation) -- so the "reborn life untouched" assertions + /// below test something real instead of an empty listing. + if (authority == Authority::CatalogRebirth && reborn_ns && reborn_incarnation != 0 && layout_for_rebirth_seed) + { + const Layout & layout = *layout_for_rebirth_seed; + const NamespaceLifeId reborn_life = NamespaceLifeId::fromCatalogEntry(*reborn_ns, reborn_incarnation); + const RefTxnId reborn_log_id{1, 1}; + const RefLogTxn reborn_birth{ + .ns = reborn_ns->string(), .txn_id = reborn_log_id, .ops = {namespaceBirthOp()}, + .prev_epoch_seal = std::nullopt}; + if (!write(layout.refLogKey(reborn_life, reborn_log_id), + sealObject(FormatId::RefLog, encodeRefLogTxn(reborn_birth)), std::nullopt, access).has_value()) + throw std::runtime_error("test-injected reborn-life log seed lost its CAS"); + const RefTableSnapshot reborn_snap = minimalLiveSnapshot(reborn_ns->string(), reborn_log_id); + if (!write(layout.refSnapshotKey(reborn_life, reborn_log_id), + sealObject(FormatId::RefSnapshot, encodeRefTableSnapshot(reborn_snap)), std::nullopt, access).has_value()) + throw std::runtime_error("test-injected reborn-life snapshot seed lost its CAS"); + /// A checkpoint too, naming the seeded log/snapshot: without one, the NEXT round's recovery + /// grounding for this namespace finds "no usable checkpoint", which SUPPRESSES that round's + /// destructive work ENTIRELY (every namespace, not just this one) -- a test relying on the + /// old cohort surviving round 2 would then be observing a no-op round, not the plan moving + /// to the reborn life. `writeRecoverableCkptForRawFixture` resolves its own fresh catalog + /// read, which already sees the incarnation bump the write just above landed. + writeRecoverableCkptForRawFixture(*this, layout, *reborn_ns, RefCkpt{ + .life_epoch = 1, + .committed_through = reborn_log_id, + .checkpoint_snapshot_id = reborn_log_id, + .last_epoch_seal = std::nullopt, + }); + } } Authority authority = Authority::Catalog; - Timing timing = Timing::BeforeFirstDelete; + Timing timing = Timing::AfterFirstDelete; String catalog_key; String gc_state_key; String first_cleanup_key; + std::optional reborn_ns; + const Layout * layout_for_rebirth_seed = nullptr; bool armed = false; + /// Guards both members below: `read` runs concurrently on the GC read-ahead pool's threads, see + /// the doc comment on `read` itself. + std::mutex seam_mutex; + /// Independent of `armed`/`timing`/`authority` above: `armGcFenceMoveOnAuthorityHoldsRevalidationForTest` + /// arms this, and the `read` override consumes it once. + bool catalog_seam_armed = false; + /// The most recent CONTROL key (catalog or `gc/state`) read -- every other key a read-ahead + /// worker reads is ignored, so `read` can recognize the catalog-then-`gc/state` ADJACENCY + /// `authorityHolds` alone produces -- see the doc comment above `moveRefCleanupAuthorityBeforeFirstChunk`. + String last_control_key_read; }; struct RefCleanupFixture @@ -580,30 +728,40 @@ TEST(CASRefGc, RefObjectCleanupRetainsCheckpointPredecessorSealProof) EXPECT_NO_THROW((void)recoverRefTableDetailedFromAuthority(op, layout, *entry, checkpoint->ckpt)); } -TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBeforeFirstDeleteRefusesEveryRefObjectDelete) +TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBeforeFirstChunkRefusesEveryRefObjectDelete) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); const Layout & layout = store->layout(); const RefCleanupFixture keys = seedTwoCoveredLogs(*backend, layout, RootNamespace{"00/aa@cas@"}); - backend->arm( - RefCleanupAuthorityRaceBackend::Authority::Catalog, - RefCleanupAuthorityRaceBackend::Timing::BeforeFirstDelete, layout, keys.first_log_key); Gc gc(store, kGc); + /// Lands the move in the exact window between the round's own hot-scan catalog cut (what + /// `authorityHolds` later compares `folded.catalog_cut` against) and everything after it -- "the + /// catalog moved before the first chunk starts", the case spec §D's test (1) names. + OperationForTest race_op(*backend); + bool hook_fired = false; + gc.setPostHotScanCatalogReadHookForTest([&] + { + hook_fired = true; + RefCleanupAuthorityRaceBackend::moveRefCleanupAuthorityBeforeFirstChunk( + RefCleanupAuthorityRaceBackend::Authority::Catalog, *race_op, layout); + }); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + ASSERT_TRUE(hook_fired) << "the race hook never fired -- this test proves nothing about the race"; - OperationForTest raw_op(*backend); - EXPECT_TRUE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); - EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()); + OperationForTest head_op(*backend); + EXPECT_TRUE((*head_op).head(keys.first_log_key, Retry::once()).has_value()); + EXPECT_TRUE((*head_op).head(keys.second_log_key, Retry::once()).has_value()); EXPECT_EQ(backend->deleteCount(keys.first_log_key), 0u); EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } -TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBetweenKeysAllowsFirstAndRefusesSecondDelete) +TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBetweenChunksAllowsFirstAndRefusesSecondDelete) { auto backend = std::make_shared(); - auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_bulk_delete_chunk_keys = 1, .gc_fold_max_defer_rounds = 0}); const Layout & layout = store->layout(); const RefCleanupFixture keys = seedTwoCoveredLogs(*backend, layout, RootNamespace{"00/aa@cas@"}); backend->arm( @@ -620,18 +778,29 @@ TEST(CASRefGcCleanupAuthority, CatalogTokenMoveBetweenKeysAllowsFirstAndRefusesS EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } -TEST(CASRefGcCleanupAuthority, GcFenceMoveBeforeFirstDeleteRefusesEveryRefObjectDelete) +TEST(CASRefGcCleanupAuthority, GcFenceMoveBeforeFirstChunkRefusesEveryRefObjectDelete) { auto backend = std::make_shared(); auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); const Layout & layout = store->layout(); const RefCleanupFixture keys = seedTwoCoveredLogs(*backend, layout, RootNamespace{"00/aa@cas@"}); - backend->arm( - RefCleanupAuthorityRaceBackend::Authority::GcFence, - RefCleanupAuthorityRaceBackend::Timing::BeforeFirstDelete, layout, keys.first_log_key); Gc gc(store, kGc); + /// Bumping `gc/state`'s lease directly from the hot-scan hook (as `Authority::Catalog` bumps the + /// catalog above) would land BEFORE the round's own `round_commit` CAS (phase 13), which still + /// holds the etag from lease adoption (phase 1) -- that CAS would then lose and the round would + /// throw before `cleanupRefObjects` (phase 17) ever runs, so "nothing deleted" would hold + /// vacuously. Instead, only ARM the seam here: the actual bump happens on `authorityHolds`'s own + /// `gc/state` read (phase 17, long after `round_commit` landed) -- see the class doc comment above + /// `moveRefCleanupAuthorityBeforeFirstChunk`. + bool hook_fired = false; + gc.setPostHotScanCatalogReadHookForTest([&] + { + hook_fired = true; + backend->armGcFenceMoveOnAuthorityHoldsRevalidationForTest(layout); + }); ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + ASSERT_TRUE(hook_fired) << "the race hook never fired -- this test proves nothing about the race"; OperationForTest raw_op(*backend); EXPECT_TRUE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); @@ -640,10 +809,11 @@ TEST(CASRefGcCleanupAuthority, GcFenceMoveBeforeFirstDeleteRefusesEveryRefObject EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } -TEST(CASRefGcCleanupAuthority, GcFenceMoveBetweenKeysAllowsFirstAndRefusesSecondDelete) +TEST(CASRefGcCleanupAuthority, GcFenceMoveBetweenChunksAllowsFirstAndRefusesSecondDelete) { auto backend = std::make_shared(); - auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_bulk_delete_chunk_keys = 1, .gc_fold_max_defer_rounds = 0}); const Layout & layout = store->layout(); const RefCleanupFixture keys = seedTwoCoveredLogs(*backend, layout, RootNamespace{"00/aa@cas@"}); backend->arm( @@ -660,6 +830,176 @@ TEST(CASRefGcCleanupAuthority, GcFenceMoveBetweenKeysAllowsFirstAndRefusesSecond EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); } +TEST(CASRefGcCleanupAuthority, LeaseMoveDuringAChunkLetsTheChunkCompleteAndNothingElse) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_bulk_delete_chunk_keys = 1, .gc_fold_max_defer_rounds = 0}); + const Layout & layout = store->layout(); + const RefCleanupFixture keys = seedTwoCoveredLogs(*backend, layout, RootNamespace{"00/aa@cas@"}); + backend->arm(RefCleanupAuthorityRaceBackend::Authority::GcFence, + RefCleanupAuthorityRaceBackend::Timing::DuringChunk, layout, keys.first_log_key); + + Gc gc(store, kGc); + ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()) << "the chunk in flight completes"; + EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()) << "the next chunk's revalidation refuses"; + EXPECT_EQ(backend->deleteCount(keys.first_log_key), 1u); + EXPECT_EQ(backend->deleteCount(keys.second_log_key), 0u); +} + +TEST(CASRefGcCleanupAuthority, RebirthDuringAChunkDeletesOnlyTheOldLifesKeys) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_bulk_delete_chunk_keys = 1, .gc_fold_max_defer_rounds = 0}); + const Layout & layout = store->layout(); + const RootNamespace ns{"00/aa@cas@"}; + const RefCleanupFixture keys = seedTwoCoveredLogs(*backend, layout, ns); + backend->arm(RefCleanupAuthorityRaceBackend::Authority::CatalogRebirth, + RefCleanupAuthorityRaceBackend::Timing::DuringChunk, layout, keys.first_log_key, ns); + + Gc gc(store, kGc); + ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head(keys.first_log_key, Retry::once()).has_value()); + EXPECT_TRUE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()); + /// Whatever the reborn life owns is untouched: its keys carry a different life id and were never + /// in the cohort. Every key under the new life's stream prefix is still present -- `moveAuthority`'s + /// `CatalogRebirth` branch seeds the reborn life's own `_log` and `_snap` right after the catalog + /// rewrite lands (see its doc comment: any earlier and an "unknown incarnation" sweep elsewhere in + /// the SAME round could claim them, since no catalog entry names that incarnation yet). + const CasRefCatalog::Snapshot cut = CasRefCatalog::read(*raw_op, layout); + const auto entry = std::find_if(cut.catalog.entries.begin(), cut.catalog.entries.end(), + [&](const CatalogEntry & e) { return e.ns == ns; }); + ASSERT_NE(entry, cut.catalog.entries.end()); + const NamespaceLifeId reborn = NamespaceLifeId::fromCatalogEntry(entry->ns, entry->incarnation); + ListPage page = (*raw_op).list(layout.namespaceStreamPrefix(reborn), "", 1000, Retry::once()); + ASSERT_FALSE(page.keys.empty()) << "the seeded new-life objects must be listed, or this test proves nothing"; + for (const ListedKey & listed : page.keys) + { + EXPECT_EQ(backend->deleteCount(listed.key), 0u) << listed.key; + EXPECT_TRUE((*raw_op).head(listed.key, Retry::once()).has_value()) << listed.key; + } + + /// The next round revalidates against the NEW catalog row: the old (dead) life is no longer named + /// by any entry `cleanupRefObjects` walks, so its plan/cohort revalidation -- built fresh from the + /// catalog entry each round -- has nothing of the old life's to touch. What actually happens to the old cohort's + /// second key, once the round runs unsuppressed, is that the NAMESPACE JANITOR + /// (`CasNamespaceJanitor.cpp:131`, `catalog_cut.life_index.resolve(*life_id)` failing for a + /// physical life the catalog no longer names) reclaims it as leaked dead-life debris -- exactly + /// spec §D's own words: "a moved catalog row means either a dropped life, whose keys the + /// namespace janitor deletes anyway, or a reborn one". So the key does NOT survive; it survives + /// past `cleanupRefObjects` specifically, then is reclaimed by a wholly separate, pre-existing + /// mechanism this task never touches. Attribute the delete precisely rather than asserting + /// "survives" and being right for an unrelated reason: capture the `ref_object_cleanup` phase's + /// `suppressed` metric (to confirm the round actually ran, not merely suppressed everything, which + /// an unusable checkpoint ANYWHERE would do -- see the checkpoint seeded above) and the + /// `namespace_cleanup` phase's `janitor_deleted` metric, and independently confirm `cleanupRefObjects` + /// itself deleted nothing this round via the GLOBAL `CASRefCleanupObjectsDeleted` counter (the + /// `GcPhaseRecord::profile_events` delta is unavailable here: it needs a `CurrentThread` with an + /// attached `ThreadStatus`, which a bare gtest thread does not have). + std::optional ref_cleanup_suppressed; + std::optional janitor_deleted; + gc.setPhaseSink([&](const GcPhaseRecord & rec) + { + if (rec.phase == "ref_object_cleanup") + if (const auto it = rec.metrics.find("suppressed"); it != rec.metrics.end()) + ref_cleanup_suppressed = it->second; + if (rec.phase == "namespace_cleanup") + if (const auto it = rec.metrics.find("janitor_deleted"); it != rec.metrics.end()) + janitor_deleted = it->second; + }); + using ProfileEvents::global_counters; + const auto ref_cleanup_deleted_before = global_counters[ProfileEvents::CASRefCleanupObjectsDeleted].load(); + ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + ASSERT_TRUE(ref_cleanup_suppressed.has_value()) << "the ref_object_cleanup phase row never fired"; + EXPECT_EQ(*ref_cleanup_suppressed, 0u) + << "round two must actually run destructive work, not merely leave everything alone because it was suppressed"; + EXPECT_EQ(global_counters[ProfileEvents::CASRefCleanupObjectsDeleted].load(), ref_cleanup_deleted_before) + << "cleanupRefObjects' own plan/cohort must delete NOTHING this round: the old life is not in it " + "(no catalog entry names it) and the reborn life's own checkpoint-named log is not yet deletable"; + ASSERT_TRUE(janitor_deleted.has_value()) << "the namespace_cleanup phase row never fired"; + EXPECT_GE(*janitor_deleted, 1u) + << "the old cohort's second key is expected to be reclaimed by the namespace janitor, not to survive"; + EXPECT_FALSE((*raw_op).head(keys.second_log_key, Retry::once()).has_value()) + << "the old cohort's second key is dead-life debris once its life no longer resolves in the " + "catalog -- the namespace janitor reclaims it, exactly as spec §D says it would"; + /// The reborn life's own objects are untouched: round two's plan, cleanup and cohort are about the + /// reborn life now, and none of what it seeded for itself is in that plan. The namespace janitor + /// leaves them alone too, since they resolve fine against the CURRENT catalog entry. + for (const ListedKey & listed : page.keys) + EXPECT_TRUE((*raw_op).head(listed.key, Retry::once()).has_value()) << listed.key; +} + +TEST(CASRefGc, RefObjectCleanupDeletesExactlyThePlannedSet) +{ + auto backend = std::make_shared(); + auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + const Layout & layout = store->layout(); + const RootNamespace ns{"00/aa@cas@"}; + fixture::admitLive(*backend, store->layout(), ns); /// Stage B (Task 4-C): pin to the sentinel before the first real touch + + /// Two committed publishes -> logs {1,1} and {1,2}. + const ManifestRef r1 = mref(1); + const ManifestRef r2 = mref(2); + writeManifestRaw(*backend, layout, ns, r1, {blobEntryFor("a", DB::UInt128(1))}); + writeManifestRaw(*backend, layout, ns, r2, {blobEntryFor("b", DB::UInt128(2))}); + const uint64_t v1 = publishCommittedTransition(*backend, layout, ns, "t1", std::nullopt, r1); + const uint64_t v2 = publishCommittedTransition(*backend, layout, ns, "t2", std::nullopt, r2); + + /// Two observed snapshots: an OLD one covering only v1, and the NEWEST covering v2. Both are real + /// wire-format snapshot objects (the recovery codec reads them). + RefTableSnapshot old_snap = minimalLiveSnapshot(ns.string(), RefTxnId{1, v1}, + {committedRow("t1", r1)}); + RefTableSnapshot new_snap = minimalLiveSnapshot(ns.string(), RefTxnId{1, v2}, + {committedRow("t1", r1), committedRow("t2", r2)}); + writeRefSnapshotRaw(*backend, layout, old_snap); + writeRefSnapshotRaw(*backend, layout, new_snap); + replaceRecoverableCkptForRawFixture(*backend, layout, ns, RefCkpt{ + .life_epoch = 1, + .committed_through = RefTxnId{1, v2}, + .checkpoint_snapshot_id = RefTxnId{1, v2}, + .last_epoch_seal = std::nullopt, + }); + + /// The plan the pass computes: `listing` names every log and snapshot this round's scan would + /// observe, `durable_cursor` is the fold cursor after folding both logs, `checkpoint_snapshot_id` + /// is the checkpoint-named recovery snapshot, and this fixture never crosses an epoch, so there + /// is no retained-seal proof. + const NamespaceLifeId life = fixture::fixtureLife(ns); + const RefTableListing listing{ + .logs = {RefTxnId{1, v1}, RefTxnId{1, v2}}, + .snapshots = {RefTxnId{1, v1}, RefTxnId{1, v2}}}; + const RefTxnId durable_cursor{1, v2}; + const RefTxnId checkpoint_snapshot_id{1, v2}; + const std::optional retained_log_proof = std::nullopt; + + OperationForTest op(*backend); + const RefCleanupPlan plan = planRefCleanup(listing, durable_cursor, checkpoint_snapshot_id, retained_log_proof); + Gc gc(store, kGc); + ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + for (const RefTxnId & id : plan.deletable_logs) + EXPECT_FALSE((*op).head(layout.refLogKey(life, id), Retry::once()).has_value()); + for (const RefTxnId & id : plan.deletable_snapshots) + EXPECT_FALSE((*op).head(layout.refSnapshotKey(life, id), Retry::once()).has_value()); + /// and every listed key NOT in the plan is present -- the chunked implementation deletes exactly + /// the set the per-key implementation would have deleted, nothing more. + const std::set deleted_logs(plan.deletable_logs.begin(), plan.deletable_logs.end()); + const std::set deleted_snapshots(plan.deletable_snapshots.begin(), plan.deletable_snapshots.end()); + for (const RefTxnId & id : listing.logs) + if (!deleted_logs.contains(id)) + EXPECT_TRUE((*op).head(layout.refLogKey(life, id), Retry::once()).has_value()) + << "log " << renderRefTxnId(id) << " not in the plan must survive"; + for (const RefTxnId & id : listing.snapshots) + if (!deleted_snapshots.contains(id)) + EXPECT_TRUE((*op).head(layout.refSnapshotKey(life, id), Retry::once()).has_value()) + << "snapshot " << renderRefTxnId(id) << " not in the plan must survive"; +} + /// Task 13 (spec §implementation-impact / §GC Budget): one fold+clean round increments every ref-intake /// observability counter -- global LIST pages (Q), log-body GETs (K), manifest-body fold GETs (H), emitted /// manifest edges, and cleaned old ref objects (D). Before/after deltas prove each site actually fires. diff --git a/src/Disks/tests/gtest_cas_settings.cpp b/src/Disks/tests/gtest_cas_settings.cpp index faa2076038dd..1fa1b49311ca 100644 --- a/src/Disks/tests/gtest_cas_settings.cpp +++ b/src/Disks/tests/gtest_cas_settings.cpp @@ -25,6 +25,7 @@ namespace DB::ContentAddressedSetting extern const ContentAddressedSettingsBool gc_enabled; extern const ContentAddressedSettingsUInt64 gc_shards; extern const ContentAddressedSettingsUInt64 gc_interval_sec; + extern const ContentAddressedSettingsUInt64 gc_bulk_delete_chunk_keys; extern const ContentAddressedSettingsString scratch_path; } @@ -186,6 +187,27 @@ TEST(CASContentAddressedSettings, InvalidBoundsDiagnosticNamesExternalConfigKeys "(got 60, 1, 0)"); } +TEST(CASSettings, BulkDeleteChunkKeysBoundsAreEnforced) +{ + expectLoadFailureWithExactMessage( + "srv1" + "1001", + ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: gc_bulk_delete_chunk_keys must be between 1 and 1000 (got 1001)"); + expectLoadFailureWithExactMessage( + "srv1" + "0", + ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: gc_bulk_delete_chunk_keys must be between 1 and 1000 (got 0)"); + + auto cfg = makeConfig( + "srv1" + "1"); + ContentAddressedSettings s; + EXPECT_NO_THROW(s.loadFromConfig(*cfg, "disk", "/scratch", "/scratch", identity_macros)); + EXPECT_EQ(s[ContentAddressedSetting::gc_bulk_delete_chunk_keys].value, 1u); +} + TEST(CASContentAddressedSettings, InvalidEnumDiagnosticsNameExternalConfigKeys) { expectLoadFailureWithExactMessage( diff --git a/src/Disks/tests/gtest_cas_write_once_key.cpp b/src/Disks/tests/gtest_cas_write_once_key.cpp new file mode 100644 index 000000000000..c05eb6b1e345 --- /dev/null +++ b/src/Disks/tests/gtest_cas_write_once_key.cpp @@ -0,0 +1,32 @@ +#include + +#include +#include +#include + +#include + +/// A `WriteOnceKey` names an object of one of the three families that are written once and never +/// rewritten: a part manifest, a ref log, a ref snapshot. Only `Layout` can mint one, from a typed +/// identity, so a verb that takes the type cannot be handed a mutable control key. + +using namespace DB::Cas; + +static_assert(!std::is_default_constructible_v); +static_assert(!std::is_constructible_v); +static_assert(!std::is_constructible_v); + +TEST(CASWriteOnceKey, FactoriesMintTheSameStringsAsThePlainKeyFunctions) +{ + const Layout layout{"p"}; + const RootNamespace ns{"test/aa@cas@"}; + const ManifestId manifest{ns, ManifestRef{.writer_epoch = 3, .build_sequence = 9, .manifest_ordinal = 2}}; + const NamespaceLifeId life = NamespaceLifeId::fromCatalogEntry(ns, DB::UInt128(0x1234)); + const RefTxnId id{5, 7}; + + EXPECT_EQ(layout.writeOnceManifestKey(manifest).str(), layout.manifestKey(manifest)); + EXPECT_EQ(layout.writeOnceRefLogKey(life, id).str(), layout.refLogKey(life, id)); + EXPECT_EQ(layout.writeOnceRefSnapshotKey(life, id).str(), layout.refSnapshotKey(life, id)); + EXPECT_TRUE(layout.parseManifestKey(layout.writeOnceManifestKey(manifest).str()).has_value()); + EXPECT_TRUE(layout.parseRefObjectKey(layout.writeOnceRefLogKey(life, id).str()).has_value()); +} diff --git a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp index f71d8495dd03..356a6eed523f 100644 --- a/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp +++ b/src/Interpreters/ContentAddressedGarbageCollectionLog.cpp @@ -45,7 +45,7 @@ ColumnsDescription ContentAddressedGarbageCollectionLogElement::getColumnsDescri {"objects_absent", std::make_shared(), "Retire candidates found already absent."}, {"objects_replaced", std::make_shared(), "412-saves (a resurrection won the race)."}, {"objects_spared", std::make_shared(), "Candidates spared (in-degree > 0 at recheck)."}, - {"manifests_deleted", std::make_shared(), "Owner-removed manifest bodies physically deleted this round (counted separately from blob deletes, B11)."}, + {"manifests_deleted", std::make_shared(), "Owner-removed manifest bodies deleted or found already absent this round (a batch delete of write-once keys cannot tell the two apart), counted separately from blob deletes."}, {"entries_condemned", std::make_shared(), "Retired entries newly condemned this round (retired-cursor pipeline stage 1)."}, {"entries_graduated", std::make_shared(), "Retired entries newly floor-passed and republished delete_pending this round (stage 2; deleted the NEXT round)."}, {"entries_redeleted", std::make_shared(), "Pending exact-token blob deletes executed this round (stage 3)."}, diff --git a/tests/integration/test_cas_gc_bulk_delete/__init__.py b/tests/integration/test_cas_gc_bulk_delete/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml b/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml new file mode 100644 index 000000000000..7324f93c273d --- /dev/null +++ b/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml @@ -0,0 +1,36 @@ + + + + + object_storage + s3 + cas + + itest-content-addressed-gc-s3 + + http://rustfs1:11121/test/cas_gc_bulk/ + clickhouse + clickhouse + + 1 + 1 + + 2 + + + + + +
+ disk_cas_gc_s3 +
+ + + + + diff --git a/tests/integration/test_cas_gc_bulk_delete/test.py b/tests/integration/test_cas_gc_bulk_delete/test.py new file mode 100644 index 000000000000..3c09d05cb6fc --- /dev/null +++ b/tests/integration/test_cas_gc_bulk_delete/test.py @@ -0,0 +1,130 @@ +import math +import time + +import pytest + +from helpers.cluster import ClickHouseCluster + +cluster = ClickHouseCluster(__file__) + +STORAGE_POLICY = "cas_gc_bulk" +MANIFESTS_PREFIX = "cas_gc_bulk/cas/manifests/" +RECLAIM_RETRIES = 90 +RECLAIM_SLEEP = 1.0 + +# Must match configs/storage_conf.xml's : kept small on purpose so +# the dropped table's owner-removed manifests (>= NUM_INSERTS of them) span more than one chunk, +# which is what this test is proving on the wire. +CHUNK_KEYS = 2 +NUM_INSERTS = 6 + + +@pytest.fixture(scope="module", autouse=True) +def start_cluster(): + cluster.add_instance( + "node", + main_configs=["configs/storage_conf.xml"], + with_rustfs=True, + stay_alive=True, + ) + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def list_manifests(): + return [ + o.object_name + for o in cluster.rustfs_client.list_objects( + cluster.rustfs_bucket, MANIFESTS_PREFIX, recursive=True + ) + ] + + +def gc_manifest_delete_totals(node): + """ + Sum every `manifest_deletes` phase row with work across however many rounds it took: the + owner-removed manifests from one DROP TABLE are not guaranteed to fold in a single round, so + reading only the first row would silently under-count on a slow run. + + Chunking happens per round, not over the grand total, so the expected request count is the + sum over rows of `ceil(row_attempted / CHUNK_KEYS)`, not `ceil(total_attempted / CHUNK_KEYS)`. + """ + node.query("SYSTEM FLUSH LOGS") + rows = ( + node.query( + "SELECT phase_metrics['attempted'], phase_metrics['accepted'], phase_metrics['requests'], " + "ProfileEvents['CASBulkDeleteRequests'], ProfileEvents['DiskS3DeleteObjects'] " + "FROM system.cas_gc_log WHERE event_type = 'Phase' AND phase = 'manifest_deletes' " + "AND phase_metrics['attempted'] > 0" + ) + .strip() + .splitlines() + ) + totals = [0, 0, 0, 0, 0] + expected_requests = 0 + for row in rows: + values = [int(x) for x in row.split("\t")] + for i, value in enumerate(values): + totals[i] += value + expected_requests += math.ceil(values[0] / CHUNK_KEYS) + # attempted, accepted, requests, bulk_requests, s3_deletes, expected_requests + return tuple(totals) + (expected_requests,) + + +def test_manifest_deletes_go_in_one_request_per_chunk(): + """ + A dropped table's owner-removed manifest bodies are deleted through `manifest_deletes` in + chunks of `cas_gc_bulk_delete_chunk_keys` write-once keys, one `DeleteObjects` request per + chunk -- never one request per key. With the chunk size forced down to 2 and >= NUM_INSERTS + manifests to delete, this asserts the request count matches the chunking math exactly, and + that the engine's own `CASBulkDeleteRequests` / `DiskS3DeleteObjects` counters agree with it. + """ + node = cluster.instances["node"] + node.query("DROP TABLE IF EXISTS t SYNC") + node.query( + f"CREATE TABLE t (k UInt64, v String) ENGINE = MergeTree ORDER BY k " + f"SETTINGS storage_policy = '{STORAGE_POLICY}'" + ) + for i in range(NUM_INSERTS): + node.query( + "INSERT INTO t SELECT number, toString(number) FROM numbers(1000) " + "SETTINGS max_insert_block_size = 1000" + ) + manifests_before = list_manifests() + assert len(manifests_before) >= NUM_INSERTS + + node.query("DROP TABLE t SYNC") + + attempted = accepted = requests = bulk_requests = s3_deletes = expected_requests = 0 + for _ in range(RECLAIM_RETRIES): + node.query("SYSTEM CAS GC RUN") + ( + attempted, + accepted, + requests, + bulk_requests, + s3_deletes, + expected_requests, + ) = gc_manifest_delete_totals(node) + if attempted >= NUM_INSERTS: + break + time.sleep(RECLAIM_SLEEP) + assert attempted >= NUM_INSERTS, "manifest_deletes never reported the dropped table's manifests" + + assert accepted == attempted, "every owner-removed manifest body is deleted or already absent" + assert requests == expected_requests, ( + f"chunking is per round, so {attempted} keys at {CHUNK_KEYS} per chunk across however " + f"many rounds it took should sum to {expected_requests} requests, got {requests}" + ) + assert bulk_requests == requests + assert s3_deletes == requests, "one DeleteObjects per chunk, no singular fallback" + + for _ in range(RECLAIM_RETRIES): + if not list_manifests(): + break + node.query("SYSTEM CAS GC RUN") + time.sleep(RECLAIM_SLEEP) + assert not list_manifests() From 4ec755474fb63c0520fa664c9f87df1dd6fbbf7e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:08:01 +0200 Subject: [PATCH 22/81] =?UTF-8?q?cas:=20hot-key=20write=20lane,=20phase=20?= =?UTF-8?q?A=20=E2=80=94=20serialize=20and=20combine=20a=20process's=20own?= =?UTF-8?q?=20writes=20to=20a=20hot=20control=20object?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every `CREATE TABLE`/`DROP TABLE` on a content-addressed disk mutates one pool-wide object, `cas/ref_catalog`, through a conditional write. Measured on ten parallel stateless jobs: `DROP TABLE` p50 2.4 s, p90 11.9 s, max 34.7 s; 113 `PreconditionFailed` in 80 s from 53 threads; the losing writer alone racking up 35 attempts with gaps growing to the 5 s cap; one `DROP` losing eight races in a row for 15.4 s. `CasRefCatalog::casUpdateImpl` starts every write with a `GET` and paces a lost race with `Retry::backoff`, a schedule shared with transport faults — a writer that has lost several races sleeps for seconds while a fresh one starts at zero, so the oldest loser is the least likely to win next. Worse, every writer in one process races every other writer in the *same* process: compare-and-swap is only needed against other servers, so every intra-process race is pure waste, each costing a `GET`, a refused `PUT`, a resolve `GET` and a sleep. `CasHotKeys` sits above the request engine as one FIFO ticket per pool and key: writers to the same hot object queue instead of racing, their conditional writes are combined into one physical attempt where safe (as-if-serial semantics, a `Conflict` cascade on a lost race so combined members see the answer a serial retry would have given them), and a last-known- object cache lets a lane holder skip the leading `GET` under one rule. Losing a race against *another server* still paces with a flat jitter, not the transport-fault backoff. The GC erase over `ref_catalog` (`deleteCompletedRemovingAtSnapshot`) becomes the lane's first caller, and the pool owns the lane. This is phase A only — combining, spacing, the clamp and moving the GC erase itself onto the lane in full are follow-on work; the design and its 34 review revisions are recorded separately. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- src/Common/CurrentMetrics.cpp | 2 + src/Common/ProfileEvents.cpp | 5 + .../ContentAddressed/Backend/CasHotKeys.cpp | 360 ++++++++ .../ContentAddressed/Backend/CasHotKeys.h | 124 +++ .../ContentAddressed/Backend/CasRequests.cpp | 44 +- .../ContentAddressed/Backend/CasRequests.h | 26 +- .../ContentAddressed/Backend/CasRetry.h | 7 + .../ContentAddressed/Backend/CasWriteResult.h | 8 +- .../ContentAddressed/Pool/CasMountRuntime.h | 6 + .../ContentAddressed/Pool/CasPool.cpp | 9 +- .../ContentAddressed/Pool/CasPool.h | 9 + .../ContentAddressed/Pool/CasRefCatalog.cpp | 66 +- src/Disks/tests/gtest_cas_hot_keys.cpp | 771 ++++++++++++++++++ .../tests/gtest_cas_ns_creation_lifecycle.cpp | 11 +- src/Disks/tests/gtest_cas_pool.cpp | 56 ++ src/Disks/tests/gtest_cas_ref_catalog.cpp | 652 +++++++++++++++ src/Disks/tests/gtest_cas_requests.cpp | 183 +++++ 17 files changed, 2307 insertions(+), 32 deletions(-) create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.cpp create mode 100644 src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.h create mode 100644 src/Disks/tests/gtest_cas_hot_keys.cpp diff --git a/src/Common/CurrentMetrics.cpp b/src/Common/CurrentMetrics.cpp index fe754c0334cb..a36be7839914 100644 --- a/src/Common/CurrentMetrics.cpp +++ b/src/Common/CurrentMetrics.cpp @@ -235,6 +235,8 @@ M(CASPartFolderCacheEntries, "Entries retained by the CA part-folder view cache") \ M(CASManifestDecodeCacheBytes, "Bytes retained by the CA manifest decode cache") \ M(CASManifestDecodeCacheEntries, "Entries retained by the CA manifest decode cache") \ + M(CASHotKeyCacheBytes, "Bytes retained by the CA hot-key lane's cache of last known objects") \ + M(CASHotKeyCacheEntries, "Entries retained by the CA hot-key lane's cache of last known objects") \ M(CASBlobUploadPoolThreads, "Number of threads in the CA blob upload thread pool.") \ M(CASBlobUploadPoolThreadsActive, "Number of threads in the CA blob upload thread pool running a task.") \ M(CASBlobUploadPoolThreadsScheduled, "Number of queued or active jobs in the CA blob upload thread pool.") \ diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 1d46e9ce0ef3..fb230d620df0 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -800,6 +800,10 @@ The server successfully detected this situation and will download merged part fr M(CASRefBatchedMutations, "Number of CAS ref mutations committed through the per-namespace batching queue. Growth indicates reference-write activity.", ValueType::Number) \ M(CASRefBatchScopeCuts, "Number of CAS ref batches cut short by scope limits. Growing values indicate smaller batches and more write overhead.", ValueType::Number) \ M(CASRefQueueWaitMicroseconds, "Total time CAS ref writers spent queued, in microseconds. A rising value indicates ref-write contention or backend latency.", ValueType::Microseconds) \ + M(CASHotKeyQueueWaitMicroseconds, "Total time CAS writers of a shared key spent queued in the hot-key lane before holding it or leaving, in microseconds. A rising value with a flat write rate means the holder is slow, not the store.", ValueType::Microseconds) \ + M(CASHotKeyCacheStarts, "Number of hot-key lane holds that started from the pool's last known object instead of a read.", ValueType::Number) \ + M(CASHotKeyReadStarts, "Number of hot-key lane holds that started from a read of the key.", ValueType::Number) \ + M(CASHotKeyCacheVerdictsReread, "Number of verdicts (a refusal or a decline) a hot-key lane decide rendered on a cached object and that were re-rendered on a fresh read instead of delivered.", ValueType::Number) \ M(CASRefRecoveryRestarts, "Number of CAS ref-table recovery retries after a snapshot or log vanished during reading. A non-zero value indicates concurrent cleanup or backend inconsistency.", ValueType::Number) \ M(CASRefRecoveryRetries, "Number of CAS ref-table recovery attempts retried after a transient object-store error before the table's load fails. A non-zero value indicates transient object-store disruption during table startup.", ValueType::Number) \ M(CASRefAppendWedged, "Number of CAS ref-log append lanes that exhausted retries after an uncertain PUT. A non-zero value indicates ref-log progress may be stalled.", ValueType::Number) \ @@ -938,6 +942,7 @@ The server successfully detected this situation and will download merged part fr M(CASConditionalWriteFenceLostPostWrite, "Number of CAS writes that succeeded but lost the final mount-fence check. A non-zero value indicates late responses after the mount lifecycle changed.", ValueType::Number) \ M(CASRequestAttempt, "Number of physical requests the CAS request contract started. Each one was admitted by the mount fence and reserved against the call's deadline before it was sent.", ValueType::Number) \ M(CASRequestReissue, "Number of CAS requests re-sent after a jittered backoff sleep. Growth means the object store is throttling, failing, or contended.", ValueType::Number) \ + M(CASRequestConflictPause, "Number of clean lost races the CAS request contract repaid after a flat jitter instead of a growing backoff: the resolve read had settled the conflict and no transport fault preceded it.", ValueType::Number) \ M(CASRequestResolveRead, "Number of requests the CAS request contract made to settle a refused precondition or an ambiguous write: a body read, or a HEAD where the caller needs only presence. Every conflict and every ambiguity costs one.", ValueType::Number) \ M(CASRequestGaveUp, "Number of CAS writes that ended without a proven outcome, at a deadline, on a lost mount fence, or unresolved. A non-zero value means callers are being asked to retry later.", ValueType::Number) \ M(CASRequestRefused, "Number of CAS writes the store itself refused, proving they never applied: a malformed request, an entity too large, or an access or credential denial that no credential refresh was performed for, either because the disk has no refresh mechanism or because this write had already spent its one refresh.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.cpp new file mode 100644 index 000000000000..0f8c9363d144 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.cpp @@ -0,0 +1,360 @@ +#include +#include + +#include +#include +#include +#include +#include + +#include +#include + +namespace ProfileEvents +{ + extern const Event CASHotKeyQueueWaitMicroseconds; + extern const Event CASHotKeyCacheStarts; + extern const Event CASHotKeyReadStarts; + extern const Event CASHotKeyCacheVerdictsReread; +} + +namespace CurrentMetrics +{ + extern const Metric CASHotKeyCacheBytes; + extern const Metric CASHotKeyCacheEntries; +} + +namespace DB::Cas +{ + +namespace +{ + +GaveUp::Source sourceFor(const Retry::Bound & bound) +{ + return bound.lease_bound ? GaveUp::Source::Lease : GaveUp::Source::Policy; +} + +/// The wait slice: how late, at most, a waiter notices its own fence or deadline. A handover wakes it +/// at once through the lane's condition variable. +constexpr auto kWaitSlice = std::chrono::milliseconds(200); + +} + +CasHotKeys::CasHotKeys(uint64_t cache_budget_bytes_) + : cache_budget_bytes(cache_budget_bytes_) + , cache(cache_budget_bytes_ == 0 + ? nullptr + /// No count cap: the weight above is never zero, so the byte budget alone already bounds + /// how many entries the cache can hold. `size_ratio` is unused by the LRU policy. + : std::make_unique("LRU", CurrentMetrics::CASHotKeyCacheBytes, CurrentMetrics::CASHotKeyCacheEntries, + cache_budget_bytes_, Cache::NO_MAX_COUNT, Cache::DEFAULT_SIZE_RATIO)) +{ +} + +CasHotKeys::~CasHotKeys() +{ + /// A lane surviving the pool that owns it means some caller's stack still points into it: a + /// lifetime error, not a state this destructor could ever see in a correct program. + chassert(lanes.empty()); +} + +WriteResult CasHotKeys::submit(const String & key, CasOperation & op, const Retry & policy, const Decide & decide) +{ + /// Frozen here so that the queue wait and the write share one deadline; a policy already frozen + /// by the caller's loop is returned unchanged. + const Retry frozen = op.freeze(policy); + const Retry::Bound bound = frozen.bind(op.owner.now_ms()); + const uint64_t entered_ms = op.owner.now_ms(); + + Item item{}; + { + std::lock_guard lock(mutex); + auto [it, inserted] = lanes.try_emplace(key); + try + { + if (enter_after_lane_hook_for_test) + enter_after_lane_hook_for_test(); + item.ticket = ++next_ticket; + it->second.queue.push_back(&item); + } + catch (...) + { + /// A lane that holds nothing is nobody's; erasing it puts the map back as it was. + if (inserted) + lanes.erase(it); + throw; + } + } + + /// From here the item is in the queue, and this guard is the only thing that removes it: on every + /// exit, normal or by unwinding, it runs the leave step, which allocates nothing and cannot throw. + bool entered_hold = false; + struct Leave + { + CasHotKeys & owner; + const String & key; + Item & item; + const bool & entered_hold; + uint64_t entered_ms; + CasOperation & op; + ~Leave() noexcept { owner.leave(key, item, entered_hold, entered_ms, op); } + } guard{*this, key, item, entered_hold, entered_ms, op}; + + for (;;) + { + /// Outside the mutex: the engine's own admission in the engine's order (the fence generation, + /// the lease budget, then the caller's liveness, which the engine reports as a lost fence), + /// then the caller's bound. A lost fence is therefore never reported as a policy deadline, and + /// an exhausted lease is reported as the lease. + std::optional leaving; + CasOperation::WriteState nothing_sent; + switch (op.gate(0)) + { + case CasOperation::Gate::FenceLost: + leaving = op.gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), nothing_sent); + break; + case CasOperation::Gate::NoBudget: + leaving = op.gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, nothing_sent); + break; + case CasOperation::Gate::Ok: + break; + } + if (!leaving && !op.fits(0, bound)) + leaving = op.gaveUp(GaveUp::Why::Deadline, sourceFor(bound), nothing_sent); + + /// Read outside the mutex, as `leave` already does: the mutex is a leaf that calls nothing, + /// and `now_ms` is a closure the pool injects, not a fixed clock. + const uint64_t now = op.owner.now_ms(); + std::unique_lock lock(mutex); + if (leaving) + return std::move(*leaving); /// the lock is released before the guard erases the item + Lane & lane = lanes.at(key); + if (lane.queue.front() == &item) + { + lane.holder_since_ms = now; + entered_hold = true; + break; + } + lane.cv.wait_for(lock, kWaitSlice); + } + ProfileEvents::increment(ProfileEvents::CASHotKeyQueueWaitMicroseconds, (op.owner.now_ms() - entered_ms) * 1000); + return hold(key, op, frozen, bound, decide); +} + +void CasHotKeys::leave(const String & key, Item & item, bool entered_hold, uint64_t entered_ms, CasOperation & op) noexcept +{ + /// The front item's ticket and how long it has held, when this caller left at its own bound behind + /// it: what makes a stuck holder visible while it is stuck. + std::optional> stuck_behind; + { + std::lock_guard lock(mutex); + auto it = lanes.find(key); + chassert(it != lanes.end()); + Lane & lane = it->second; + auto found = std::find(lane.queue.begin(), lane.queue.end(), &item); + chassert(found != lane.queue.end()); + lane.queue.erase(found); + if (entered_hold) + lane.holder_since_ms.reset(); + else if (!lane.queue.empty() && lane.holder_since_ms) + { + /// `now_ms` is the pool's injected clock and can throw; the fallback is no snapshot, never + /// a throw out of this locked section, whose sole duty -- removing the item -- must complete + /// regardless. + try + { + const uint64_t now = op.owner.now_ms(); + stuck_behind = std::pair{lane.queue.front()->ticket, now - *lane.holder_since_ms}; + } + catch (...) + { + } + } + if (lane.queue.empty()) + lanes.erase(it); + else + lane.cv.notify_all(); + } + try + { + if (!entered_hold) + { + const uint64_t now = op.owner.now_ms(); + ProfileEvents::increment(ProfileEvents::CASHotKeyQueueWaitMicroseconds, (now - entered_ms) * 1000); + } + if (stuck_behind) + LOG_WARNING(getLogger("CasHotKeys"), + "hot key '{}': a writer left at its own bound while ticket {} has held the key for {} ms", + key, stuck_behind->first, stuck_behind->second); + } + catch (...) + { + tryLogCurrentException("CasHotKeys"); + } +} + +WriteResult CasHotKeys::hold(const String & key, CasOperation & op, const Retry & policy, const Retry::Bound & bound, + const Decide & decide) +{ + CasOperation::WriteState state; + std::optional base; + bool from_cache = false; + /// A single-attempt submission never starts from a hint: its one attempt is on fresh state, as + /// the engine's own verb reads before it. + if (!policy.single_attempt) + { + if (auto remembered = cached(key)) + { + base = std::move(remembered); + from_cache = true; + ProfileEvents::increment(ProfileEvents::CASHotKeyCacheStarts); + } + } + const auto read_base = [&]() -> std::optional + { + ProfileEvents::increment(ProfileEvents::CASHotKeyReadStarts); + CasOperation::Resolved resolved = op.observe(key, policy, bound); + state.last_seen = resolved.seen; + base.reset(); + if (const auto * object = std::get_if(&state.last_seen)) + base = *object; + else if (!std::holds_alternative(state.last_seen)) + return op.gaveUpAfterFailedObservation(resolved.stop, state, bound); + return std::nullopt; + }; + if (!from_cache) + if (auto refused = read_base()) + return *refused; + + std::optional candidate; + bool verdict_on_hint = false; + try + { + candidate = decide(base); + verdict_on_hint = from_cache && !candidate; + } + catch (...) + { + if (!from_cache) + throw; + verdict_on_hint = true; + } + if (verdict_on_hint) + { + /// A verdict rendered on a hint proves only that a hint is not a proof: the entry is dropped, + /// the key is read, and what the `decide` does on the read is the result. A read the caller's + /// own gate refuses is that give-up, as for any caller that cannot read. + forget(key); + ProfileEvents::increment(ProfileEvents::CASHotKeyCacheVerdictsReread); + if (auto refused = read_base()) + return *refused; + from_cache = false; + candidate = decide(base); + } + if (!candidate) + { + if (base) + remember(key, *base); + return Declined{state.last_seen}; + } + + WriteResult result = [&] + { + try + { + return base ? op.replace(key, *candidate, base->etag, policy) : op.create(key, *candidate, policy); + } + catch (...) + { + forget(key); + throw; + } + }(); + std::visit(detail::Overload{ + [&](const Committed & committed) { remember(key, Object{*candidate, committed.etag}); }, + [&](const Conflict & conflict) + { + if (const auto * object = std::get_if(&conflict.seen)) + remember(key, *object); + else + forget(key); + }, + [&](const Refused &) { forget(key); }, + [&](const GaveUp & gave_up) { if (gave_up.sent_any) forget(key); }, + [&](const Declined &) {}}, result); + return result; +} + +std::optional CasHotKeys::cached(const String & key) const +{ + if (!cache) + return std::nullopt; + if (auto hit = cache->get(key)) + return hit->object; + return std::nullopt; +} + +void CasHotKeys::remember(const String & key, Object object) noexcept +{ + if (!cache) + return; + try + { + if (cache_fill_hook_for_test) + cache_fill_hook_for_test(); + /// Computed here, where a throw (allocation, `Etag::render`) is still just a failed fill: never + /// zero, since the key, the incarnation and the containers weigh something even when the object + /// is empty, so the byte budget bounds the entry count as well as the bytes. + const size_t weight = key.size() + object.bytes.size() + object.etag.render().size() + 64; + /// An object above the budget is not a hint worth evicting everything else for. + if (weight > cache_budget_bytes) + { + forget(key); + return; + } + auto remembered = std::make_shared(Remembered{std::move(object), weight}); + cache->set(key, remembered); + } + catch (...) + { + /// A hint that could not be stored is no hint: the next hold reads. The result that led here + /// stands; a failed fill must never replace it. + tryLogCurrentException("CasHotKeys"); + forget(key); + } +} + +void CasHotKeys::forget(const String & key) noexcept +{ + if (!cache) + return; + try + { + cache->remove(key); + } + catch (...) + { + tryLogCurrentException("CasHotKeys"); + } +} + +size_t CasHotKeys::queueDepthForTest(const String & key) const +{ + std::lock_guard lock(mutex); + const auto it = lanes.find(key); + return it == lanes.end() ? 0 : it->second.queue.size(); +} + +size_t CasHotKeys::laneCountForTest() const +{ + std::lock_guard lock(mutex); + return lanes.size(); +} + +size_t CasHotKeys::cacheEntriesForTest() const +{ + return cache ? cache->count() : 0; +} + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.h new file mode 100644 index 000000000000..d41ec95e1652 --- /dev/null +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasHotKeys.h @@ -0,0 +1,124 @@ +#pragma once +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB::Cas +{ + +class CasOperation; + +/// One conditional write in flight per key from the operations that write through here, in arrival +/// order, plus the last object this pool knows per key so the next write needs no read. +/// +/// Compare-and-swap is needed only against other servers; inside one server every writer of a shared +/// key used to race every other, and each lost race cost a read, a refused write, a resolve read and a +/// growing sleep. `submit` takes a FIFO ticket for the key, waits its turn re-checking the caller's +/// own fence, lease and deadline, obtains a base (the cache's object, else the engine's own read), +/// runs the caller's `decide` on it, lands the candidate through the engine's `replace` or `create` +/// on the caller's own operation and thread, and returns the engine's result unchanged. The caller +/// keeps the retry loop: a `Conflict` is a lost race against another server, and the caller submits +/// again after `Retry::conflictBackoff`. +/// +/// The cache is a hint and never a source of truth: every write against it is conditional on its +/// etag, so a stale entry costs one 412 and one resolve read; and a verdict a `decide` renders on it +/// (a refusal by exception, or "nothing to write") is never delivered, because a refusal without a +/// write is the one thing a 412 cannot correct -- the entry is dropped, the key is read, and the +/// `decide` runs again on the read. The store's answer to a write decided on a hint is delivered +/// whatever it is: it is a fact about the store, and the caller's ordinary retry learns the rest. +/// +/// One instance per pool, shared by its three request planes; a `CasRequests` built without one owns +/// a private instance with no cache. Every callback (`decide`, a `Liveness` closure, a backend hook) +/// runs with `mutex` released: the mutex is a leaf that calls nothing. +class CasHotKeys +{ +public: + /// `cache_budget_bytes` bounds the remembered objects; 0 disables the cache, and every hold reads. + explicit CasHotKeys(uint64_t cache_budget_bytes); + ~CasHotKeys(); + CasHotKeys(const CasHotKeys &) = delete; + CasHotKeys & operator=(const CasHotKeys &) = delete; + + /// The caller's mutation of `key`, the engine's own decide shape: the candidate bytes to write over + /// `base`, nothing (`Declined`), or a refusal by exception. `base` is absent when the key does not + /// exist; a caller that refuses to bootstrap throws there. A `decide` may be run twice, and a + /// later run on a fresh read is the decision that counts; it issues no write through this lane; + /// its reads go through `op`; it reads `base->bytes` and never `base->etag`. + using Decide = std::function(const std::optional &)>; + + /// One hold on `key`: wait for the turn, obtain a base, run `decide`, one engine write, remember. + /// `policy` is frozen at entry, so time in the queue spends the caller's window. Returns the + /// engine's own result for that write, in class and content, and propagates a `decide`'s + /// exception as `readModifyWrite` does, never from a cached base. + WriteResult submit(const String & key, CasOperation & op, const Retry & policy, const Decide & decide); + + /// Items in the key's queue, the holder included; 0 for a key with no lane. + size_t queueDepthForTest(const String & key) const; + /// Lanes in existence: a lane lives while its queue holds an item. + size_t laneCountForTest() const; + /// Entries the cache holds. + size_t cacheEntriesForTest() const; + + /// TEST SEAM: runs under `mutex` after the key's lane was found or created and before the item + /// is queued; a throw here is the enqueue's allocation failing. + std::function enter_after_lane_hook_for_test; + /// TEST SEAM: runs inside `remember` before the cache is filled; a throw here is the fill failing. + std::function cache_fill_hook_for_test; + +private: + /// One queued submission, on its caller's stack: the caller's own guard is the only thing that + /// removes it, so nothing here outlives the stack it lives on. + struct Item + { + uint64_t ticket; + }; + /// One key. Created on first use, erased when its queue empties; referenced only by threads whose + /// item is in its queue. + struct Lane + { + std::deque queue; /// guarded by `mutex`; the front is the holder + std::optional holder_since_ms; /// guarded by `mutex`; for the log line + std::condition_variable cv; + }; + /// A remembered object together with its precomputed weight: `Etag::render` allocates and can + /// throw, and the cache's own accounting calls the weight after it has already subtracted the + /// entry being replaced, so the weight itself must never throw. + struct Remembered + { + Object object; + size_t weight; + }; + struct RememberedWeight + { + size_t operator()(const Remembered & remembered) const noexcept { return remembered.weight; } + }; + using Cache = CacheBase, RememberedWeight>; + + WriteResult hold(const String & key, CasOperation & op, const Retry & policy, const Retry::Bound & bound, + const Decide & decide); + void leave(const String & key, Item & item, bool entered_hold, uint64_t entered_ms, CasOperation & op) noexcept; + + std::optional cached(const String & key) const; + void remember(const String & key, Object object) noexcept; + void forget(const String & key) noexcept; + + mutable std::mutex mutex; /// `mutable` for the const test seams + uint64_t next_ticket = 0; /// guarded by `mutex`; the holder's identity in the log line + std::unordered_map lanes; /// guarded by `mutex` + const uint64_t cache_budget_bytes; + /// The last known object per key. Its synchronization is its own; it is read and written with + /// `mutex` released. Null when the budget is 0. + std::unique_ptr cache; +}; + +} diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 5184659ccbe8..1ccba8e23d15 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -26,6 +26,7 @@ namespace ProfileEvents { extern const Event CASRequestAttempt; extern const Event CASRequestReissue; + extern const Event CASRequestConflictPause; extern const Event CASRequestResolveRead; extern const Event CASRequestGaveUp; extern const Event CASRequestRefused; @@ -58,6 +59,11 @@ void recordReissue() ProfileEvents::increment(ProfileEvents::CASRequestReissue); } +void recordConflictPause() +{ + ProfileEvents::increment(ProfileEvents::CASRequestConflictPause); +} + } namespace @@ -230,12 +236,15 @@ bool isDefinitelyRefusedWrite([[maybe_unused]] const std::exception & e) } CasRequests::CasRequests(BackendPtr backend_, Fence fence_, - std::function now_ms_, std::function sleep_ms_) + std::function now_ms_, std::function sleep_ms_, + CasHotKeys * hot_keys_) : backend(std::move(backend_)) , fence(std::move(fence_)) , now_ms(now_ms_ ? std::move(now_ms_) : std::function(bootClockMs)) , sleep_ms(sleep_ms_ ? std::move(sleep_ms_) : std::function(sleepForMilliseconds)) , attempt_reservation_ms(backend->attemptTimeoutMs()) + , own_hot_keys(hot_keys_ ? nullptr : std::make_unique(0)) + , hot_keys(hot_keys_ ? hot_keys_ : own_hot_keys.get()) { } @@ -803,6 +812,23 @@ std::optional CasOperation::pauseAndReissue(WriteState & state, con return std::nullopt; } +std::optional CasOperation::pauseForConflict(WriteState & state, const Retry::Bound & bound) +{ + const uint64_t pause_ms = Retry::conflictBackoff(); + const uint64_t needed = reservedFor(pause_ms, 2); + switch (gate(needed)) + { + case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); + case Gate::NoBudget: return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); + case Gate::Ok: break; + } + if (!fits(needed, bound)) + return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); + detail::recordConflictPause(); + owner.sleep_ms(pause_ms); + return std::nullopt; +} + WriteResult CasOperation::writeLoop(const String & key, const String & bytes, const std::optional & expected, const Retry & policy, const Retry::Bound & bound, WriteState & state, ResolveWith resolve_refusal_with) @@ -923,7 +949,7 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// Identical bytes here are somebody else's object, and the caller that owns the key's meaning /// decides what that means. if (!state.any_ambiguous) - return Conflict{state.last_seen, state.attempts_sent}; + return Conflict{state.last_seen, state.attempts_sent, state.any_ambiguous}; if (!preconditionStillSatisfiable(state.last_seen, expected)) { @@ -934,7 +960,7 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co if (const auto * obj = std::get_if(&state.last_seen); obj && obj->bytes == bytes) return postCommit(obj->etag, /*resolved_by_read=*/true, state, bound); /// Nothing of this inner write's is at the key, and a reissue would be refused too. - return Conflict{state.last_seen, state.attempts_sent}; + return Conflict{state.last_seen, state.attempts_sent, state.any_ambiguous}; } /// Unresolved but repeatable: the precondition would still be met -- or nothing was observed at @@ -996,7 +1022,10 @@ WriteResult CasOperation::readModifyWrite(const String & key, const DecideOnObje if (policy.single_attempt) return result; - if (auto given_up = pauseAndReissue(state, bound)) + /// A clean lost race is settled: the resolve read holds the fresh object and the next + /// iteration decides on it. Only a conflict that settled a transport fault is paced by the + /// growing schedule. + if (auto given_up = state.any_ambiguous ? pauseAndReissue(state, bound) : pauseForConflict(state, bound)) return *given_up; /// Only when the resolve settled nothing is a fresh read owed; otherwise `current` already is @@ -1049,8 +1078,11 @@ WriteResult CasOperation::readModifyWriteOnPresence(const String & key, const De current.reset(); if (policy.single_attempt) - return Conflict{state.last_seen, state.attempts_sent}; - if (auto given_up = pauseAndReissue(state, bound)) + return Conflict{state.last_seen, state.attempts_sent, state.any_ambiguous}; + /// A clean lost race is settled: the resolve read holds the fresh object and the next + /// iteration decides on it. Only a conflict that settled a transport fault is paced by the + /// growing schedule. + if (auto given_up = state.any_ambiguous ? pauseAndReissue(state, bound) : pauseForConflict(state, bound)) return *given_up; if (std::holds_alternative(state.last_seen)) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index 82b292e1bd16..16ed2daf6ef1 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -122,6 +123,7 @@ namespace detail /// The engine's per-attempt counters, behind functions so the header need not declare the events. void recordAttempt(); void recordReissue(); +void recordConflictPause(); } class CasOperation; @@ -140,9 +142,12 @@ class CasRequests /// `now_ms` defaults to `CLOCK_BOOTTIME` milliseconds -- the same clock a mount lease deadline is /// expressed on, so `Retry::untilLeaseSafe` and this engine compare like with like. `sleep_ms` /// defaults to a real sleep. `attempt_reservation_ms` is taken from the backend's own attempt - /// timeout: it is what the engine reserves before it starts anything. + /// timeout: it is what the engine reserves before it starts anything. `hot_keys` is the pool's + /// write lane, shared by its planes; without one this object owns a private lane with no cache, + /// so a write through it costs today's read and write. CasRequests(BackendPtr backend_, Fence fence_, - std::function now_ms_ = {}, std::function sleep_ms_ = {}); + std::function now_ms_ = {}, std::function sleep_ms_ = {}, + CasHotKeys * hot_keys_ = nullptr); /// Both may be called concurrently on one `CasRequests`: neither writes a member, and the only /// state either reads is the backend and the fence -- whose closures must therefore be thread-safe @@ -168,6 +173,10 @@ class CasRequests private: friend class CasOperation; + /// `CasOperation::owner` is a `CasRequests &`: `CasOperation`'s own friendship with `CasHotKeys` + /// does not extend to what that reference points at, so the lane needs its own grant to reach the + /// clock and sleep it reads and paces through `op.owner`. + friend class CasHotKeys; /// The one place a transport key is created. Every verb reaches the store through this, so no /// engine code -- and nothing outside it -- can name the key's type, let alone construct one. @@ -194,6 +203,9 @@ class CasRequests std::function now_ms; std::function sleep_ms; uint64_t attempt_reservation_ms; + /// The private lane of a `CasRequests` built without a pool; null when `hot_keys` is the pool's. + std::unique_ptr own_hot_keys; + CasHotKeys * hot_keys; }; /// The cap on one `removeManyWriteOnce` chunk -- also the ceiling a batch-delete request can carry. @@ -222,6 +234,9 @@ class CasOperation /// than a request. bool admitted() const { return gate(0) == Gate::Ok; } + /// The write lane for keys several writers of this pool share. + CasHotKeys & hotKeys() const { return *owner.hot_keys; } + /// `policy` with its window turned into an absolute deadline on this operation's clock, taken NOW. /// A hand-written loop freezes its policy once before it starts and passes the frozen value to /// every call it makes, so the loop ends when the window it was given ends -- rather than granting @@ -272,6 +287,9 @@ class CasOperation private: friend class CasRequests; + /// The lane waits on this operation's own admission and reads and writes through its verbs; it + /// needs the gate, the reservation, the resolve read and the give-up helpers, never the transport. + friend class CasHotKeys; CasOperation(CasRequests & owner_, uint64_t admitted_generation_, Liveness liveness_) : owner(owner_), admitted_generation(admitted_generation_), liveness(std::move(liveness_)) @@ -357,6 +375,10 @@ class CasOperation /// Admission, then the jittered sleep. A value means the call ended during it; nullopt means the /// caller may send another attempt. std::optional pauseAndReissue(WriteState & state, const Retry::Bound & bound); + /// The sibling for a clean lost race: the same admission and the same reservation, a flat + /// `Retry::conflictBackoff` sleep, and `state.reissues` untouched, so a transport fault that follows + /// starts its own schedule at the beginning. + std::optional pauseForConflict(WriteState & state, const Retry::Bound & bound); /// `sleep_ms` plus `envelopes` attempt reservations, saturating. uint64_t reservedFor(uint64_t sleep_ms, uint32_t envelopes) const; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h index 984892caac55..6275eb7e41ca 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h @@ -28,6 +28,13 @@ struct Retry /// `attempt == 0` returns 0. static uint64_t backoff(uint32_t attempt); + /// The flat pause after a clean lost race: `backoff(1)`, uniform over [0, 200] ms. It does not grow + /// with the writer's loss count, because a settled conflict has nothing to wait for but + /// desynchronisation from its competitors, and growing it with the loss count made the oldest + /// loser the slowest and the likeliest to lose again. A conflict that settled a transport fault + /// keeps `backoff(attempt)`: the fault is what must pace the loop. + static uint64_t conflictBackoff() { return backoff(1); } + /// A policy with `ms` milliseconds of its own budget and no lease bound. static Retry within(uint64_t ms) { return {.window_ms = ms, .lease_deadline_ms = std::nullopt, .single_attempt = false}; } /// `within(90'000)` -- the default write policy. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h index 4312c9d9da0f..72679c54b64a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasWriteResult.h @@ -46,8 +46,12 @@ struct Declined { Observation seen; }; /// A competing write won: the key's current state does not match what this call expected. /// `attempts_sent` counts the HTTP attempts this call made, the same count `Committed` and `GaveUp` /// carry: an operator's attempt counters sum over ALL the endings of a write, and losing the key is -/// one an operator wants counted rather than dropped. -struct Conflict { Observation seen; uint32_t attempts_sent = 0; }; +/// one an operator wants counted rather than dropped. `any_ambiguous` is true when an attempt of the +/// inner write ended without proof of whether it applied before the resolve read settled the race: a +/// caller outside the engine paces such a conflict on the growing schedule, as the engine's own +/// loops do, and a clean lost race on the flat one. `attempts_sent` cannot stand in for it: a +/// throttled first attempt whose resolve read finds the key moved is one attempt, ambiguous. +struct Conflict { Observation seen; uint32_t attempts_sent = 0; bool any_ambiguous = false; }; /// The store itself refused the request (not a lost precondition) -- `store_error` is a ClickHouse /// error code and `message` explains it. `attempts_sent` is the same count `Conflict` carries, for /// the same reason. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h index 635b41d58029..bc278279aefe 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h @@ -76,6 +76,8 @@ struct MountConfig RuntimeWorkerFactory worker_factory = {}; /// Deterministic test interposition after the remount worker has confirmed renewal is parked, /// immediately before it releases `driver_mutex` and begins the real remount callback. + /// Runs with `driver_mutex` held: it must issue no backend request and never wait on the pool's + /// hot-key lane, whose holders sleep under that mutex, or the test deadlocks itself. std::function remount_parked_hook_for_test = {}; /// Deterministic test interposition at the top of the renewal loop, before it acquires /// `driver_mutex` to inspect cadence or parking state. @@ -88,6 +90,8 @@ struct MountConfig std::function renewal_terminal_deposited_hook_for_test = {}; /// Deterministic test interposition after the parked renewal predicate has sampled terminal false, /// immediately before the condition-variable wait atomically releases `driver_mutex`. + /// Runs with `driver_mutex` held: it must issue no backend request and never wait on the pool's + /// hot-key lane, whose holders sleep under that mutex, or the test deadlocks itself. std::function renewal_parked_predicate_false_hook_for_test = {}; /// Deterministic test interposition immediately before a terminal publisher attempts to acquire /// `driver_mutex`. @@ -96,6 +100,8 @@ struct MountConfig /// contention, but before it blocks acquiring the mutex. std::function terminal_publication_driver_lock_contended_hook_for_test = {}; /// Deterministic test interposition immediately after a terminal publisher acquires `driver_mutex`. + /// Runs with `driver_mutex` held: it must issue no backend request and never wait on the pool's + /// hot-key lane, whose holders sleep under that mutex, or the test deadlocks itself. std::function terminal_publication_driver_lock_acquired_hook_for_test = {}; /// Deterministic failure injection at the vanished-reason preparation boundary. std::function vanished_reason_prepare_hook_for_test = {}; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 8e51b551e9d7..5e637b72694b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -185,6 +185,7 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) : pool_backend(std::move(backend_)) , config(std::move(config_)) , meta(std::move(meta_)) + , hot_keys(config.hot_key_cache_bytes) /// The mount plane's fence reaches `mount_runtime`, declared far below: the closures capture /// `this` and run only after construction, exactly like `ref_ledger`'s callbacks. All three planes /// take the fence's own clock, so a policy bound to a mount-lease deadline and the fence that @@ -194,8 +195,9 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) [this](uint64_t g, uint64_t needed) { return mount_runtime.admit(g, needed); }, [this](uint64_t g) { mount_runtime.checkFenceOrThrow(g); }}, config.boot_ms_fn, - mountPlaneSleepFn()) - , farewell_requests(pool_backend, Fence::open(), config.boot_ms_fn) + mountPlaneSleepFn(), + &hot_keys) + , farewell_requests(pool_backend, Fence::open(), config.boot_ms_fn, {}, &hot_keys) /// The open plane's fence is the pool's teardown flag: generation 0 forever, exactly like /// `Fence::open`, but `admit` refuses once `beginTeardown` ran. A GC round, an FSCK or a probe in /// flight is then refused at its next request instead of running to completion under a disk that @@ -211,7 +213,8 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) [this](uint64_t, uint64_t) { return teardownBegun() ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, [](uint64_t) {}}, config.boot_ms_fn, - openPlaneSleepFn()) + openPlaneSleepFn(), + &hot_keys) /// Seed the monotone admitted-algo cache from the pool state `createOrValidate` already /// established (fresh create, steady-state member, or a just-completed admission union) -- /// register-before-first-write means this Pool's own `writeAlgo()` is ALWAYS a diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index fc6829d2c53c..7354fd4d08ea 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -75,6 +75,10 @@ struct PoolConfig /// was count-bounded only (16384 entries) — decoded manifests carry inline bytes, so the worst /// case was multi-GB. 0 disables decode caching (every read decodes fresh — diagnostic mode). uint64_t manifest_decode_cache_bytes = 128ULL << 20; + /// Byte bound for the hot-key lane's cache of last known objects (the catalog today). 0 disables + /// the cache and every catalog write reads first. 16 MiB: the catalog is under 1 MiB, and the + /// bound exists so a later opt-in of per-namespace keys has one. + uint64_t hot_key_cache_bytes = 16ULL << 20; /// How many superseded snapshot generations to retain. After committing /// generation G, generations <= G - this are pruned (bounded per round). 0 = keep ALL /// (debug/forensics — replay GC's in-degree view as-of a past round). Default 3 = the safety @@ -1166,6 +1170,11 @@ class Pool : public std::enable_shared_from_this PoolConfig config; PoolMeta meta; + /// The pool's write lane for keys several of its writers share, declared before the three planes + /// that carry a pointer to it, so it outlives every operation they admit. `mutable` for the same + /// reason the planes are. + mutable CasHotKeys hot_keys; + /// The three planes' engines, declared before every component that is handed one and after the /// config they take their clock from. `mutable` because issuing a request is not a change to the /// pool: a `const` observer still has to read the store. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp index 074c210ad6b7..db3c3c61966b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -148,7 +149,8 @@ constexpr size_t kMaxCatalogCasAttempts = 100; /// Shared body of `casUpdate`/`casAdmitEntry`. `encode` turns a freshly `mutate`d candidate into the /// bytes to write: the plain path just grammar-checks (`encodeRefCatalog`), the admitting path also /// runs both admission predicates (`checkCatalogAdmission`) first. A refused precondition re-runs -/// `mutate` against the FRESH body -- never re-encoding the stale candidate. +/// `mutate` against the resolve read's fresh body -- the one the pool's hot-key lane remembers and +/// the next hold starts from -- never re-encoding the stale candidate. RefCatalog casUpdateImpl( CasOperation & op, const Layout & layout, const std::function & mutate, @@ -174,16 +176,31 @@ RefCatalog casUpdateImpl( return bytes; }; - WriteResult result = op.readModifyWrite(key, decide, policy); - /// The fence can be lost in two places and both mean the same to a lifecycle caller: inside - /// `decide`, which throws the marker itself, and between two attempts, where the engine notices it - /// first and no further `decide` runs. Normalising the second onto the first is what keeps "the - /// fence moved" a returned outcome rather than an exception. - if (const auto * gave_up = std::get_if(&result); gave_up && gave_up->why == GaveUp::Why::FenceLost) - throw CatalogFenceMovedMarker{}; - if (!std::holds_alternative(result)) - throwCatalogWriteFailure(std::move(result), fmt::format("CAS ref catalog '{}' update", key)); - return std::move(*written); + /// One hold at a time per pool on this key, from the pool's last known catalog when the lane holds + /// one. The loop is this function's: a `Conflict` is a lost race against another server (the lane + /// never conflicts with itself), repaid after the flat jitter, or after the growing schedule when + /// the conflict settled a transport fault. Under a single-attempt policy the first `Conflict` is + /// the answer, as the engine's own verb answers it. + const Retry frozen = op.freeze(policy); + uint32_t settled_faults = 0; + for (;;) + { + WriteResult result = op.hotKeys().submit(key, op, frozen, decide); + if (const auto * conflict = std::get_if(&result); conflict && !frozen.single_attempt) + { + op.pause(conflict->any_ambiguous ? Retry::backoff(++settled_faults) : Retry::conflictBackoff()); + continue; + } + /// The fence can be lost in two places and both mean the same to a lifecycle caller: inside + /// `decide`, which throws the marker itself, and between two attempts, where the engine + /// notices it first and no further `decide` runs. Normalising the second onto the first is + /// what keeps "the fence moved" a returned outcome rather than an exception. + if (const auto * gave_up = std::get_if(&result); gave_up && gave_up->why == GaveUp::Why::FenceLost) + throw CatalogFenceMovedMarker{}; + if (!std::holds_alternative(result)) + throwCatalogWriteFailure(std::move(result), fmt::format("CAS ref catalog '{}' update", key)); + return std::move(*written); + } } /// `casUpdate`'s own guard, shared with the two lifecycle callers that need to catch the fence marker @@ -337,11 +354,22 @@ RefCatalog CasRefCatalog::casAdmitEntry( next.entries.insert(it, entry); return next; }; - return casUpdateImpl(op, layout, mutate, - [&entry, gc_shards, &layout](const RefCatalog & c) - { - return checkCatalogAdmission(c, gc_shards, layout, entry.ns); - }); + try + { + return casUpdateImpl(op, layout, mutate, + [&entry, gc_shards, &layout](const RefCatalog & c) + { + return checkCatalogAdmission(c, gc_shards, layout, entry.ns); + }); + } + catch (const CatalogFenceMovedMarker &) + { + /// The marker is this file's private signal; a caller outside it gets the exception class every + /// other admission refusal raises. + throwCasTransientUnavailable( + fmt::format("CAS ref catalog '{}' update", layout.refCatalogKey()), + "mount fence tripped: the update was admitted under an incarnation this node no longer holds"); + } } CasRefCatalog::BeginRemovingOutcome CasRefCatalog::beginRemoving( @@ -664,6 +692,12 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( { return NamespaceCreationOutcome::Superseded; } + catch (const CatalogFenceMovedMarker &) + { + /// The creator's own admission moved while step 1 waited its turn or wrote: an answer, not a + /// failure, and the same one the two later steps already give. + return NamespaceCreationOutcome::FencedOut; + } return completeCreation(op, layout, entry, policy); } diff --git a/src/Disks/tests/gtest_cas_hot_keys.cpp b/src/Disks/tests/gtest_cas_hot_keys.cpp new file mode 100644 index 000000000000..6dc4d5426383 --- /dev/null +++ b/src/Disks/tests/gtest_cas_hot_keys.cpp @@ -0,0 +1,771 @@ +#include + +#include +#include +#include +#include +#include +#include +#include "cas_test_helpers.h" + +#include +#include + +#include "config.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ProfileEvents +{ + extern const Event CASHotKeyQueueWaitMicroseconds; + extern const Event CASHotKeyCacheStarts; + extern const Event CASHotKeyReadStarts; + extern const Event CASHotKeyCacheVerdictsReread; + extern const Event CASRequestGaveUp; +} + +using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; + +namespace +{ + +/// The harness's `FakeClock` is single-threaded. The lane is not: its holders sleep on the engine's +/// clock from their own threads while the test thread advances it, so every access goes through one +/// mutex. A sleep still advances the clock by what it slept, so a holder's transport backoff is real +/// time to every waiter's deadline. +struct SyncClock +{ + std::mutex mutex; + uint64_t now = 1'000'000; + std::vector sleeps; + + std::function nowFn() + { + return [this] { std::lock_guard lock(mutex); return now; }; + } + std::function sleepFn() + { + return [this](uint64_t ms) { std::lock_guard lock(mutex); sleeps.push_back(ms); now += ms; }; + } + void advance(uint64_t ms) { std::lock_guard lock(mutex); now += ms; } + size_t sleepCount() { std::lock_guard lock(mutex); return sleeps.size(); } +}; + +/// The object under test lists the tickets that wrote it, comma-separated, so order is visible. +CasHotKeys::Decide appendTicket(int ticket) +{ + return [ticket](const std::optional & current) -> std::optional + { + if (!current) + return std::to_string(ticket); + return current->bytes + "," + std::to_string(ticket); + }; +} + +uint64_t counter(ProfileEvents::Event event) +{ + return ProfileEvents::global_counters[event].load(); +} + +/// A one-shot gate a write hook parks on: the first write of the key waits here until the test +/// releases it; every later write passes. +struct ParkFirstWrite +{ + std::latch parked{1}; + std::latch release{1}; + std::atomic seen{0}; + + void install(CountingBackend & backend, const String & key) + { + backend.onBeforeWrite(key, [this] + { + if (seen.fetch_add(1) != 0) + return; + parked.count_down(); + release.wait(); + }); + } +}; + +#if USE_AWS_S3 +std::exception_ptr s3Error(Aws::S3::S3Errors code, const String & name) +{ + return std::make_exception_ptr(DB::S3Exception("the store answered " + name, code, name)); +} +#endif + +} + +TEST(CASHotKeys, SubmissionsOfOneKeyAreSerializedInArrivalOrder) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(0); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + constexpr int N = 4; + ParkFirstWrite park; + park.install(*backend, "k"); + + std::vector threads; + std::vector> results(N); + std::deque go; /// a deque: `std::latch` is neither copyable nor movable + for (int i = 0; i < N; ++i) + go.emplace_back(1); + for (int i = 0; i < N; ++i) + { + threads.emplace_back([&, i] + { + go[i].wait(); + auto op = requests.admit(); + results[i] = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(i + 1)); + }); + } + /// The first holder is released into its write and parked there; every later thread is released + /// only after its item is seen queued, so arrival order is the release order. + go[0].count_down(); + park.parked.wait(); + for (int i = 1; i < N; ++i) + { + go[i].count_down(); + while (hot_keys.queueDepthForTest("k") < static_cast(i + 1)) + std::this_thread::yield(); + } + park.release.count_down(); + for (auto & t : threads) + t.join(); + + EXPECT_EQ(backend->writeCount("k"), static_cast(N)); + EXPECT_EQ(backend->getCount("k"), static_cast(N)); /// no cache in this task: a read per hold + std::vector etags; + for (const auto & result : results) + { + ASSERT_TRUE(result.has_value()); + const auto * committed = std::get_if(&*result); + ASSERT_NE(committed, nullptr); + etags.push_back(committed->etag); + } + for (size_t i = 1; i < etags.size(); ++i) + EXPECT_FALSE(etags[i] == etags[i - 1]); + DB::Cas::tests::expectBytes(*backend, "k", "1,2,3,4"); + auto reader = requests.admit(); + EXPECT_EQ(reader.read("k", Retry::standard())->etag, etags.back()); + EXPECT_EQ(hot_keys.laneCountForTest(), 0u); + EXPECT_EQ(hot_keys.queueDepthForTest("k"), 0u); +} + +TEST(CASHotKeys, ADecideRunsWithTheLaneMutexReleased) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(0); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + /// `queueDepthForTest` takes the lane's mutex; a `decide` run under it would deadlock this test, + /// which hangs the whole `CAS*` gate rather than being reported by a per-test timeout. The call is + /// the assertion. + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::standard()), + [&](const std::optional &) -> std::optional + { + EXPECT_EQ(hot_keys.queueDepthForTest("k"), 1u); + return String("1"); + }); + EXPECT_TRUE(std::holds_alternative(result)); +} + +TEST(CASHotKeys, AFailedEnqueueLeavesNoEmptyLaneBehind) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(0); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + hot_keys.enter_after_lane_hook_for_test = [] { throw std::bad_alloc(); }; + EXPECT_THROW(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)), std::bad_alloc); + EXPECT_EQ(hot_keys.laneCountForTest(), 0u); + EXPECT_EQ(backend->writeCount("k"), 0u); + hot_keys.enter_after_lane_hook_for_test = {}; + EXPECT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); +} + +TEST(CASHotKeys, ResultsAreTheEnginesOwn) +{ + SyncClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(false); + CasHotKeys hot_keys(0); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); + +#if USE_AWS_S3 + /// The store refuses the bytes: the caller gets that `Refused`, at once. + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + { + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)); + ASSERT_TRUE(std::holds_alternative(result)); + } +#endif + /// A clean refused precondition with the store unchanged: `Conflict` carrying the occupant. + backend->refuseNextWrite("k"); + { + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(3)); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + EXPECT_TRUE(std::holds_alternative(conflict->seen)); + EXPECT_FALSE(conflict->any_ambiguous); + } + /// The resolve read fails at the transport under `once`: nothing observed. The failure is armed + /// from a one-shot write hook, not up front, so it lands on the write's own resolve read rather + /// than on the hold's base read, which must succeed for this sub-case to reach the write at all. + backend->refuseNextWrite("k"); + backend->onBeforeWrite("k", [&] { backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("resolve"))); }); + { + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::once()), appendTicket(4)); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + EXPECT_TRUE(std::holds_alternative(conflict->seen)); + } + backend->onBeforeWrite("k", [] {}); + /// An ambiguous attempt whose resolve read fails at the transport under `once`: unresolved. Same + /// one-shot arming as above, for the same reason. + backend->injectAmbiguousWrite("k"); + backend->onBeforeWrite("k", [&] { backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("resolve"))); }); + { + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::once()), appendTicket(5)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any); + } + backend->onBeforeWrite("k", [] {}); + /// The fence trips inside the hold, before the write: nothing sent. + bool alive = true; + auto fenced = requests.admit([&] { return alive; }); + { + WriteResult result = hot_keys.submit("k", fenced, fenced.freeze(Retry::standard()), + [&](const std::optional & current) -> std::optional + { + alive = false; + return current->bytes + ",6"; + }); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any); + } + /// The fence trips after the landed write: the object carries the ticket, the caller is told so. + alive = true; + backend->onWriteCommitted("k", [&] { alive = false; }); + { + WriteResult result = hot_keys.submit("k", fenced, fenced.freeze(Retry::standard()), appendTicket(7)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(gave_up->sent_any); + DB::Cas::tests::expectBytes(*backend, "k", "1,7"); + } + backend->onWriteCommitted("k", [] {}); + /// The engine call throws a local fault: it reaches the caller and the key is handed over. + backend->failNextWriteWith("k", std::make_exception_ptr(std::logic_error("local"))); + EXPECT_THROW(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(8)), std::logic_error); + EXPECT_EQ(hot_keys.laneCountForTest(), 0u); + EXPECT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(9)))); +} + +TEST(CASHotKeys, ABaseReadThatFailsGivesUpAsReadModifyWriteDoes) +{ + SyncClock clock; + auto backend = std::make_shared(); + backend->setAttemptTimeoutMs(1000); + CasHotKeys hot_keys(0); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + (void)orThrow(op.create("k", "1", Retry::standard()), "seed"); + + /// Enough armed failures to outlast a standard window: the read loop gives up at its deadline. + for (int i = 0; i < 64; ++i) + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("read"))); + WriteResult lane = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)); + for (int i = 0; i < 64; ++i) + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("read"))); + WriteResult verb = op.readModifyWrite("k", appendTicket(2), Retry::standard()); + + const auto * a = std::get_if(&lane); + const auto * b = std::get_if(&verb); + ASSERT_NE(a, nullptr); + ASSERT_NE(b, nullptr); + EXPECT_EQ(a->why, b->why); + EXPECT_EQ(a->deadline_source, b->deadline_source); + EXPECT_EQ(a->sent_any, b->sent_any); + EXPECT_FALSE(a->sent_any); + EXPECT_EQ(a->last_seen.index(), b->last_seen.index()); + + /// A fence that refuses the read's own reservation, and nothing smaller: the wait step passes + /// (it asks for zero), the base read is refused before its first attempt. + bool refuse_reservations = false; + Fence fence{[] { return uint64_t{0}; }, + [&](uint64_t, uint64_t needed) { return refuse_reservations && needed > 0 ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [](uint64_t) {}}; + CasRequests fenced_requests(backend, fence, clock.nowFn(), clock.sleepFn(), &hot_keys); + auto fenced = fenced_requests.admit(); + refuse_reservations = true; + WriteResult result = hot_keys.submit("k", fenced, fenced.freeze(Retry::standard()), appendTicket(3)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any); +} + +TEST(CASHotKeys, WaitersLeaveOnTheirOwnFenceLeaseAndDeadline) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(0); + bool lease_spent = false; + Fence fence{[] { return uint64_t{0}; }, + [&](uint64_t, uint64_t) { return lease_spent ? Fence::Admit::NoBudget : Fence::Admit::Ok; }, + [](uint64_t) {}}; + CasRequests requests(backend, fence, clock.nowFn(), clock.sleepFn(), &hot_keys); + ParkFirstWrite park; + park.install(*backend, "k"); + + std::thread holder([&] + { + auto op = requests.admit(); + (void)hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)); + }); + park.parked.wait(); + + const auto gave_up_before = counter(ProfileEvents::CASRequestGaveUp); + /// A waiter whose own window ends while the holder is parked. + std::optional by_deadline; + std::thread deadline_waiter([&] + { + auto op = requests.admit(); + by_deadline = hot_keys.submit("k", op, op.freeze(Retry::within(500)), appendTicket(2)); + }); + /// A waiter whose task stops. + std::atomic alive{true}; + std::optional by_liveness; + std::thread liveness_waiter([&] + { + auto op = requests.admit([&] { return alive.load(); }); + by_liveness = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(3)); + }); + while (hot_keys.queueDepthForTest("k") < 3) + std::this_thread::yield(); + + clock.advance(600); + deadline_waiter.join(); + alive = false; + liveness_waiter.join(); + { + const auto * gave_up = std::get_if(&*by_deadline); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Policy); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(gave_up->attempts_sent, 0u); + EXPECT_TRUE(std::holds_alternative(gave_up->last_seen)); + } + { + const auto * gave_up = std::get_if(&*by_liveness); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + EXPECT_FALSE(gave_up->sent_any); + } + /// A waiter whose lease budget is gone, and whose task has stopped at the same slice: the lease + /// speaks first, as the engine's own gate orders it. Both refusals are already in place before + /// this thread is even spawned, so its first admission check sees both at once. + lease_spent = true; + std::optional by_lease; + std::thread lease_waiter([&] + { + auto op = requests.admit([&] { return alive.load(); }); + by_lease = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(4)); + }); + lease_waiter.join(); + { + const auto * gave_up = std::get_if(&*by_lease); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_EQ(gave_up->deadline_source, GaveUp::Source::Lease); + } + EXPECT_EQ(counter(ProfileEvents::CASRequestGaveUp) - gave_up_before, 3u); + EXPECT_EQ(hot_keys.queueDepthForTest("k"), 1u) << "only the parked holder remains"; + EXPECT_EQ(backend->writeCount("k"), 1u) << "no second write started"; + + lease_spent = false; + park.release.count_down(); + holder.join(); + DB::Cas::tests::expectBytes(*backend, "k", "1"); + EXPECT_EQ(hot_keys.laneCountForTest(), 0u); +} + +TEST(CASHotKeys, AThrottledHolderKeepsTheWaitersQueuedThroughItsBackoff) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(0); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + + /// The holder's own `PUT` parks here on its first attempt; once the two waiters are proven + /// queued behind it, the release makes that attempt ambiguous instead of letting it through, so + /// its resolve read (the key still absent) drives one reissue on the growing schedule while the + /// waiters sit queued through it. + std::latch parked{1}; + std::latch release{1}; + std::atomic seen{0}; + backend->onBeforeWrite("k", [&] + { + if (seen.fetch_add(1) != 0) + return; + parked.count_down(); + release.wait(); + EXPECT_EQ(hot_keys.queueDepthForTest("k"), 3u); + backend->injectAmbiguousWrite("k"); + }); + + std::vector threads; + std::vector> results(3); + threads.emplace_back([&] + { + auto op = requests.admit(); + results[0] = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)); + }); + parked.wait(); + /// Each waiter is spawned only once the previous one is seen queued, so arrival order -- and so + /// the order they write in once the holder releases -- is the spawn order. + for (int i = 1; i < 3; ++i) + { + threads.emplace_back([&, i] + { + auto op = requests.admit(); + results[i] = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(i + 1)); + }); + while (hot_keys.queueDepthForTest("k") < static_cast(i + 1)) + std::this_thread::yield(); + } + release.count_down(); + for (auto & t : threads) + t.join(); + + for (const auto & result : results) + EXPECT_TRUE(std::holds_alternative(*result)); + ASSERT_EQ(clock.sleepCount(), 1u) << "the one reissue pause, taken while the two waiters were queued"; + EXPECT_LE(clock.sleeps[0], 200u); + EXPECT_EQ(backend->writeCount("k"), 4u) << "the ambiguous attempt counts, then three landed"; + EXPECT_EQ(hot_keys.laneCountForTest(), 0u); + DB::Cas::tests::expectBytes(*backend, "k", "1,2,3"); +} + +TEST(CASHotKeys, TheNextHoldStartsFromTheLandedObjectWithoutARead) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + const auto cache_starts_before = counter(ProfileEvents::CASHotKeyCacheStarts); + + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); + EXPECT_EQ(backend->getCount("k"), 1u); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)))); + EXPECT_EQ(backend->getCount("k"), 1u) << "the second hold started from the cache"; + EXPECT_EQ(counter(ProfileEvents::CASHotKeyCacheStarts) - cache_starts_before, 1u); + + /// Under `once` the one attempt is on fresh state: a read, no cached start. + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::once()), appendTicket(3)))); + EXPECT_EQ(backend->getCount("k"), 2u); + /// `expectBytes` issues its own read, so it comes after every count assertion, not between them. + DB::Cas::tests::expectBytes(*backend, "k", "1,2,3"); +} + +TEST(CASHotKeys, AnExternalWriterCostsOneResolveReadAndOneRetry) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + CasRequests external_requests = DB::Cas::tests::openRequestsForTest(backend); + auto external = external_requests.admit(); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); + + const auto current = external.read("k", Retry::standard()); + (void)orThrow(external.replace("k", "E", current->etag, Retry::standard()), "external"); + const uint64_t gets_before = backend->getCount("k"); + const uint64_t writes_before = backend->writeCount("k"); + + /// The caller's loop: submit, and on a conflict submit again after the flat pause. + std::optional result; + for (int i = 0; i < 3 && !result; ++i) + { + WriteResult attempt = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)); + if (std::holds_alternative(attempt)) + op.pause(Retry::conflictBackoff()); + else + result = std::move(attempt); + } + ASSERT_TRUE(result && std::holds_alternative(*result)); + EXPECT_EQ(backend->getCount("k") - gets_before, 1u) << "one resolve read"; + EXPECT_EQ(backend->writeCount("k") - writes_before, 2u) << "one refused write, one that landed"; + /// The submission after that starts from the cache. + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(3)))); + EXPECT_EQ(backend->getCount("k") - gets_before, 1u); + /// `expectBytes` issues its own read, so it comes after every count assertion, not between them. + DB::Cas::tests::expectBytes(*backend, "k", "E,2,3"); +} + +TEST(CASHotKeys, MalformedBytesRepairedExternallyRaiseNoCorruptionVerdict) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + CasRequests external_requests = DB::Cas::tests::openRequestsForTest(backend); + auto external = external_requests.admit(); + /// A decide that refuses bytes it cannot decode, as the catalog's does. + const CasHotKeys::Decide strict = [](const std::optional & current) -> std::optional + { + if (current && current->bytes.find("garbage") != String::npos) + throw DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "not a ticket list"); + return current ? current->bytes + ",9" : String("9"); + }; + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), strict))); + + auto current = external.read("k", Retry::standard()); + const Etag garbage = *orThrow(external.replace("k", "garbage", current->etag, Retry::standard()), "break"); + /// The lane's next hold starts from its cache, loses to the garbage, and remembers the garbage + /// its resolve read saw; the caller pauses and submits again. + WriteResult first = hot_keys.submit("k", op, op.freeze(Retry::standard()), strict); + ASSERT_TRUE(std::holds_alternative(first)); + (void)orThrow(external.replace("k", "9", garbage, Retry::standard()), "repair"); + const auto reread_before = counter(ProfileEvents::CASHotKeyCacheVerdictsReread); + /// The verdict on the cached garbage is not delivered: one read, and the decide lands on the repair. + WriteResult second = hot_keys.submit("k", op, op.freeze(Retry::standard()), strict); + ASSERT_TRUE(std::holds_alternative(second)); + EXPECT_EQ(counter(ProfileEvents::CASHotKeyCacheVerdictsReread) - reread_before, 1u); + DB::Cas::tests::expectBytes(*backend, "k", "9,9"); + /// And when the read is garbage too, that is the real corruption. + current = external.read("k", Retry::standard()); + (void)orThrow(external.replace("k", "garbage", current->etag, Retry::standard()), "break again"); + (void)hot_keys.submit("k", op, op.freeze(Retry::standard()), strict); /// conflict: the cache now holds garbage + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, + [&] { (void)hot_keys.submit("k", op, op.freeze(Retry::standard()), strict); }); +} + +TEST(CASHotKeys, ADeclineOnAHintIsRerenderedOnARead) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + CasRequests external_requests = DB::Cas::tests::openRequestsForTest(backend); + auto external = external_requests.admit(); + /// Writes "1" once and declines while the object already says "1". + const CasHotKeys::Decide idempotent = [](const std::optional & current) -> std::optional + { + if (current && current->bytes == "1") + return std::nullopt; + return String("1"); + }; + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), idempotent))); + /// On a fresh read the decline is the caller's answer. + { + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::once()), idempotent); + const auto * declined = std::get_if(&result); + ASSERT_NE(declined, nullptr); + EXPECT_TRUE(std::holds_alternative(declined->seen)); + } + /// An external writer replaces the object; the cached hint still says "1", so the decide would + /// decline on it. The decline is not delivered: the lane reads and the decide writes. + const auto current = external.read("k", Retry::standard()); + (void)orThrow(external.replace("k", "0", current->etag, Retry::standard()), "external"); + const uint64_t gets_before = backend->getCount("k"); + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::standard()), idempotent); + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->getCount("k") - gets_before, 1u); + DB::Cas::tests::expectBytes(*backend, "k", "1"); + /// On an absent key the decline names absence. + WriteResult absent = hot_keys.submit("missing", op, op.freeze(Retry::standard()), + [](const std::optional &) -> std::optional { return std::nullopt; }); + const auto * declined = std::get_if(&absent); + ASSERT_NE(declined, nullptr); + EXPECT_TRUE(std::holds_alternative(declined->seen)); +} + +TEST(CASHotKeys, TheCacheForgetsWhatItCannotVouchFor) +{ + SyncClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(false); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); + const auto reads = [&] { return backend->getCount("k"); }; + + uint64_t before = reads(); +#if USE_AWS_S3 + /// Refused: dropped, the next hold reads. + backend->failNextWriteWith("k", s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)))); + EXPECT_EQ(reads(), before); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)))); + EXPECT_EQ(reads(), before + 1); +#endif + + /// Ticket 2 only lands when the S3-only Refused sub-case above runs it; the terminal bytes below + /// follow the same guard. +#if USE_AWS_S3 + constexpr auto kFinalBytes = "1,2,3,4,5,6,7"; +#else + constexpr auto kFinalBytes = "1,3,4,5,6,7"; +#endif + + /// Unresolved after a send: dropped. A single-attempt submission never starts from the cache, so + /// arming the ambiguity and the resolve-read failure up front would let the hold's own base read + /// consume them; a one-shot write hook lands both on the write's own resolve read instead. + backend->onBeforeWrite("k", [&] + { + backend->injectAmbiguousWrite("k"); + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("resolve"))); + }); + before = reads(); + { + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::once()), appendTicket(3)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_TRUE(gave_up->sent_any); + } + backend->onBeforeWrite("k", [] {}); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(3)))); + EXPECT_EQ(reads(), before + 3); /// the base read, the failed resolve read, the next hold's read after the entry was dropped + + /// An exception out of the write: dropped. + backend->failNextWriteWith("k", std::make_exception_ptr(std::logic_error("local"))); + before = reads(); + EXPECT_THROW(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(4)), std::logic_error); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(4)))); + EXPECT_EQ(reads(), before + 1); + + /// A give-up that sent nothing leaves the entry as it was: the next hold starts from it. + bool alive = true; + auto fenced = requests.admit([&] { return alive; }); + before = reads(); + WriteResult nothing_sent = hot_keys.submit("k", fenced, fenced.freeze(Retry::standard()), + [&](const std::optional & current) -> std::optional { alive = false; return current->bytes + ",5"; }); + ASSERT_TRUE(std::holds_alternative(nothing_sent)); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(5)))); + EXPECT_EQ(reads(), before); + + /// A fill that throws after a landed write: the result stands, the next hold reads. + hot_keys.cache_fill_hook_for_test = [] { throw std::bad_alloc(); }; + before = reads(); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(6)))); + hot_keys.cache_fill_hook_for_test = {}; + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(7)))); + EXPECT_EQ(reads(), before + 1); + DB::Cas::tests::expectBytes(*backend, "k", kFinalBytes); +} + +TEST(CASHotKeys, TheBudgetBoundsBytesAndEntries) +{ + SyncClock clock; + auto backend = std::make_shared(); + /// Two entries of one-byte objects weigh 2 x (1 + 1 + etag + 64); a budget of one entry and a half + /// holds one at a time. + auto probe_op_requests = DB::Cas::tests::openRequestsForTest(backend); + auto probe = probe_op_requests.admit(); + const size_t etag_bytes = orThrow(probe.create("probe", "x", Retry::standard()), "probe")->render().size(); + const uint64_t one_entry = 1 + 1 + etag_bytes + 64; + CasHotKeys hot_keys(one_entry + one_entry / 2); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + const CasHotKeys::Decide one_byte = [](const std::optional &) -> std::optional { return String("x"); }; + + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("a", op, op.freeze(Retry::standard()), one_byte))); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("b", op, op.freeze(Retry::standard()), one_byte))); + EXPECT_EQ(hot_keys.cacheEntriesForTest(), 1u) << "the older entry was evicted"; + const uint64_t gets_a = backend->getCount("a"); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("a", op, op.freeze(Retry::standard()), one_byte))); + EXPECT_EQ(backend->getCount("a"), gets_a + 1) << "the evicted key reads"; + + /// An object above the budget is not stored. + const CasHotKeys::Decide big = [&](const std::optional &) -> std::optional { return String(one_entry * 2, 'y'); }; + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("c", op, op.freeze(Retry::standard()), big))); + const uint64_t gets_c = backend->getCount("c"); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("c", op, op.freeze(Retry::standard()), big))); + EXPECT_EQ(backend->getCount("c"), gets_c + 1); + + /// Empty objects weigh their key and their allowance: N of them stay bounded by the budget. + CasHotKeys small(4 * one_entry); + CasRequests small_requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &small); + auto small_op = small_requests.admit(); + const CasHotKeys::Decide empty = [](const std::optional &) -> std::optional { return String(); }; + for (int i = 0; i < 40; ++i) + ASSERT_TRUE(std::holds_alternative(small.submit("e" + std::to_string(i), small_op, small_op.freeze(Retry::standard()), empty))); + EXPECT_LE(small.cacheEntriesForTest(), 4u); +} + +TEST(CASHotKeys, ACachedStartPastTheDeadlineSendsNothing) +{ + SyncClock clock; + auto backend = std::make_shared(); + backend->setAttemptTimeoutMs(1000); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); + + /// The window fits the wait's zero reservation but not the write's two attempt envelopes. + int decided = 0; + const uint64_t writes_before = backend->writeCount("k"); + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::within(500)), + [&](const std::optional & current) -> std::optional { ++decided; return current->bytes + ",2"; }); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_FALSE(gave_up->sent_any); + EXPECT_EQ(decided, 1); + EXPECT_EQ(backend->writeCount("k"), writes_before); +} + +TEST(CASHotKeys, AnIdenticalCandidateLandedByAnotherServerIsTheEnginesCommit) +{ + SyncClock clock; + auto backend = std::make_shared(); + CasHotKeys hot_keys(16ULL << 20); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys); + auto op = requests.admit(); + CasRequests external_requests = DB::Cas::tests::openRequestsForTest(backend); + auto external = external_requests.admit(); + ASSERT_TRUE(std::holds_alternative(hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(1)))); + + /// Another server lands exactly the candidate this hold will compute from its stale hint, and this + /// hold's own refused write loses its answer. The resolve read finds the candidate's bytes under + /// the moved incarnation: the engine's own rule calls that landed. + const auto current = external.read("k", Retry::standard()); + const Etag theirs = *orThrow(external.replace("k", "1,2", current->etag, Retry::standard()), "identical"); + backend->injectAmbiguousWrite("k"); + WriteResult result = hot_keys.submit("k", op, op.freeze(Retry::standard()), appendTicket(2)); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_TRUE(committed->resolved_by_read); + EXPECT_TRUE(committed->etag == theirs); + DB::Cas::tests::expectBytes(*backend, "k", "1,2"); +} diff --git a/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp b/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp index 7fee0407e45b..e86d55074bd8 100644 --- a/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp +++ b/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp @@ -361,8 +361,10 @@ TEST(CASNsCreationLifecycle, EntryStolenByAConcurrentReconcilerRefusesGoLiveAndL /// `completeCreation` performs, since step 2 touches only the `_ckpt`. `decide` therefore receives /// the POST-steal body and refuses it without sending anything, which is what the entry check is /// for. The steal itself must succeed (asserted), so the mismatch is the entry ACTUALLY changing, - /// not a contrived stub. It runs on its own operation, because it is a different actor. - CasOperation thief_op = requests.admit(); + /// not a contrived stub. It runs on its own operation and its own `CasRequests` (a rival is + /// another server; one server's writers never race each other on the pool's hot-key lane). + CasRequests rival_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation thief_op = rival_requests.admit(); backend->hook_before_read_of = layout.refCatalogKey(); backend->on_read = [&] { @@ -414,7 +416,10 @@ TEST(CASNsCreationLifecycle, BothFenceAndEntryStaleRefusesGoLiveViaTheFenceCheck /// check from that `mutate` and the entry check answers `Superseded` instead, because `decide` /// refuses the stale entry before any write is sent and the engine's own gate never speaks. The /// assertions below confirm the entry really did change too. - CasOperation thief_op = requests.admit(); + /// The rival gets its own `CasRequests`, on its own hot-key lane (a rival is another server; one + /// server's writers never race each other on the pool's lane). + CasRequests rival_requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation thief_op = rival_requests.admit(); CasOperation creator_op = requests.admit([&backend] { return backend->admitted; }); backend->hook_before_read_of = layout.refCatalogKey(); backend->withdraw_on_read = true; diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index b5264ba1b151..33f92e656d68 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -4122,3 +4122,59 @@ TEST(CASPool, MountpointObjectRoundTrip) EXPECT_FALSE(store->getMountpointObject(key).has_value()); EXPECT_FALSE(store->mountpointObjectExists(key)); } + +namespace ProfileEvents +{ + extern const Event CASHotKeyReadStarts; + extern const Event CASRequestResolveRead; + extern const Event CASRequestConflictPause; +} + +TEST(CASPool, ConcurrentNamespaceCreationsNeverRaceEachOtherOnTheCatalog) +{ + auto backend = std::make_shared(); + auto pool = DB::Cas::tests::openPoolForTest(backend); + const DB::Cas::Layout layout("p"); + const String key = layout.refCatalogKey(); + constexpr int N = 6; + + /// Drain whatever the pool's own bootstrap touched on the catalog key before measuring. + (void)pool->namespaceLife(DB::Cas::RootNamespace{"warmup"}); + + const uint64_t writes_before = backend->writeCount(key); + const auto reads_before = ProfileEvents::global_counters[ProfileEvents::CASHotKeyReadStarts].load(); + const auto resolves_before = ProfileEvents::global_counters[ProfileEvents::CASRequestResolveRead].load(); + std::vector threads; + for (int i = 0; i < N; ++i) + threads.emplace_back([&, i] { (void)pool->namespaceLife(DB::Cas::RootNamespace{"ns" + std::to_string(i)}); }); + for (auto & t : threads) + t.join(); + + EXPECT_EQ(backend->writeCount(key) - writes_before, 2u * N) << "two catalog steps per creation, each one write"; + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestResolveRead].load() - resolves_before, 0u) + << "no refused precondition, so no resolve read"; + EXPECT_LE(ProfileEvents::global_counters[ProfileEvents::CASHotKeyReadStarts].load() - reads_before, 1u) + << "at most one lane read; every later hold started from the cache"; + + /// Another server writes the catalog between two of this pool's mutations: one extra read and one + /// retry write, then the cache is current again. Raw `getCount` cannot isolate that cost: a + /// `namespaceLife` call on a fresh namespace also issues the ledger's own snapshot reads + /// (`CasRefCatalog::read`), which are outside the lane by design and fire the same number of times + /// whether or not an external write happened. The lane's own signals are what the external write + /// actually moves. + { + auto external_requests = DB::Cas::tests::openRequestsForTest(backend); + auto external = external_requests.admit(); + DB::Cas::CasRefCatalog::casAdmitEntry(external, layout, 1, + DB::Cas::CatalogEntry{.ns = DB::Cas::RootNamespace{"zz"}, .state = DB::Cas::NsState::Live, .incarnation = UInt128{99}}); + } + const uint64_t writes_mid = backend->writeCount(key); + const auto resolves_mid = ProfileEvents::global_counters[ProfileEvents::CASRequestResolveRead].load(); + const auto lane_reads_mid = ProfileEvents::global_counters[ProfileEvents::CASHotKeyReadStarts].load(); + (void)pool->namespaceLife(DB::Cas::RootNamespace{"after"}); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestResolveRead].load() - resolves_mid, 1u) + << "one resolve read for the external write"; + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASHotKeyReadStarts].load() - lane_reads_mid, 0u) + << "the next hold starts from what the resolve read saw"; + EXPECT_EQ(backend->writeCount(key) - writes_mid, 3u) << "one refused, two landed"; +} diff --git a/src/Disks/tests/gtest_cas_ref_catalog.cpp b/src/Disks/tests/gtest_cas_ref_catalog.cpp index 129504bf4257..8d4c364cb68f 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog.cpp @@ -1,5 +1,6 @@ #include "cas_format_test_battery.h" #include "cas_test_helpers.h" +#include #include #include #include @@ -12,12 +13,14 @@ #include #include #include +#include #include #include #include #include #include #include +#include #include #include #include @@ -25,6 +28,8 @@ #include using namespace DB::Cas; +using DB::Cas::tests::CountingBackend; +using DB::Cas::tests::FakeClock; namespace ProfileEvents { @@ -2136,3 +2141,650 @@ TEST(CASGCRefPlan, RoundInputOwnsObservationsAndSuccessorStateCannotChangePlan) EXPECT_EQ(plan.row(UInt128{2}).fold_state.coverage.last_folded_ref_id, (RefTxnId{2, 3})); EXPECT_FALSE(plan.contains(UInt128{9})); } + +/// ---------- Pool/CasRefCatalog: mutations through the pool's hot-key lane ---------- + +namespace ProfileEvents +{ + extern const Event CASHotKeyReadStarts; + extern const Event CASHotKeyCacheVerdictsReread; + extern const Event CASRequestConflictPause; + extern const Event CASRequestReissue; +} + +namespace +{ + +#if USE_AWS_S3 +std::exception_ptr s3Error(Aws::S3::S3Errors code, const String & name) +{ + return std::make_exception_ptr(DB::S3Exception("the store answered " + name, code, name)); +} +#endif + +/// A pool's own engine: a `CasRequests` over the pool's shared lane with a cache, on a fake clock, +/// beside a plain `CasRequests` that stands for another server. +struct PoolAndExternal +{ + explicit PoolAndExternal(std::shared_ptr backend_) + : backend(std::move(backend_)) + , hot_keys(16ULL << 20) + , pool(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), &hot_keys) + , external(DB::Cas::tests::openRequestsForTest(backend)) + { + } + std::shared_ptr backend; + FakeClock clock; + CasHotKeys hot_keys; + CasRequests pool; + CasRequests external; +}; + +uint64_t eventCount(ProfileEvents::Event event) +{ + return ProfileEvents::global_counters[event].load(); +} + +} + +TEST(CASRefCatalog, AStaleHintNeverProducesAFalseEntryChanged) +{ + /// The pool's last catalog write left "a" Creating; another server completed it to Live. The + /// caller reads fresh, sees Live, and calls beginRemoving: the cached Creating row differs from + /// what it observed and would say EntryChanged. That verdict is not delivered. + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry creating = entryInState("a", NsState::Creating, 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, creating); /// through the lane: the cache holds Creating + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::completeCreation(other, layout, creating), CasRefCatalog::NamespaceCreationOutcome::Live); + + const CatalogEntry observed = CasRefCatalog::read(other, layout).catalog.entries.at(0); + ASSERT_EQ(observed.state, NsState::Live); + const uint64_t reads_before = f.backend->getCount(layout.refCatalogKey()); + const uint64_t rereads_before = eventCount(ProfileEvents::CASHotKeyCacheVerdictsReread); + EXPECT_EQ(CasRefCatalog::beginRemoving(pool_op, layout, observed, 7), CasRefCatalog::BeginRemovingOutcome::Transitioned); + EXPECT_EQ(f.backend->getCount(layout.refCatalogKey()) - reads_before, 1u) << "one lane read"; + EXPECT_EQ(eventCount(ProfileEvents::CASHotKeyCacheVerdictsReread) - rereads_before, 1u); + EXPECT_EQ(CasRefCatalog::read(other, layout).catalog.entries.at(0).state, NsState::Removing); +} + +TEST(CASRefCatalog, ATrueRefusalOnTheReadIsTheSameAsWithoutACache) +{ + /// The row really changed: the re-rendered verdict is the one a cache-less call renders. + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry creating = entryInState("a", NsState::Creating, 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, creating); + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::completeCreation(other, layout, creating), CasRefCatalog::NamespaceCreationOutcome::Live); + const uint64_t writes_before = f.backend->writeCount(layout.refCatalogKey()); + const uint64_t reads_before = f.backend->getCount(layout.refCatalogKey()); + /// The caller believes it observed a Creating row under another incarnation. The pool's hint + /// (Creating, incarnation 1) differs from it, so the verdict is rendered on the hint and not + /// delivered; the read (Live, incarnation 1) differs from it too, and that verdict is the answer, + /// the one a cache-less call renders: one read, no write. (Had the hint matched `observed`, the + /// write path would run instead: a 412 and a resolve read, which the lane tests cover.) + const CatalogEntry observed_elsewhere = entryInState("a", NsState::Creating, 2); + EXPECT_EQ(CasRefCatalog::cancelStalledCreating(pool_op, layout, observed_elsewhere, [](const CreatorFence &) { return true; }), + CasRefCatalog::StalledCreatingCancelOutcome::EntryChanged); + EXPECT_EQ(f.backend->writeCount(layout.refCatalogKey()), writes_before) << "no write"; + EXPECT_EQ(f.backend->getCount(layout.refCatalogKey()) - reads_before, 1u) << "one lane read"; +} + +/// createNamespaceStep1 via createNamespace: the cache holds row "a" (Live), the external erased it +/// via a completed removal. +TEST(CASRefCatalog, AStaleHintCreateNamespaceStep1AdmitsANewIncarnation) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + const CreatorFence creator{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}; + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + ASSERT_EQ(CasRefCatalog::createNamespace(pool_op, layout, 1, RootNamespace{"a"}, creator), + CasRefCatalog::NamespaceCreationOutcome::Live); /// the cache now holds "a" Live + + CasOperation other = f.external.admit(); + const CatalogEntry observed = CasRefCatalog::read(other, layout).catalog.entries.at(0); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, observed, 9), CasRefCatalog::BeginRemovingOutcome::Transitioned); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(observed.incarnation, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const CatalogEntry removing{.ns = RootNamespace{"a"}, .state = NsState::Removing, + .incarnation = observed.incarnation, .removal_started_round = 9}; + ASSERT_EQ(CasRefCatalog::deleteCompletedRemoving(other, layout, removing, ready_parent, noAuthorityRefresh).outcome, + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + + const uint64_t reads_before = f.backend->getCount(layout.refCatalogKey()); + EXPECT_EQ(CasRefCatalog::createNamespace(pool_op, layout, 1, RootNamespace{"a"}, creator), + CasRefCatalog::NamespaceCreationOutcome::Live); + /// Two reads, not one: `createNamespace`'s own pre-check (`read`) never goes through the lane, so + /// it always issues its own request regardless of the cache; the lane's ONE read is the one + /// `createNamespaceStep1` pays for finding its cached hint stale. + EXPECT_EQ(f.backend->getCount(layout.refCatalogKey()) - reads_before, 2u) << "the pre-check read plus one lane read"; +} + +/// createNamespaceStep1: the store holds "a" Creating under another creator -- Superseded on the read. +TEST(CASRefCatalog, ATrueRefusalCreateNamespaceStep1SeesASuperseder) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + const CreatorFence creator{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}; + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + ASSERT_EQ(CasRefCatalog::createNamespace(pool_op, layout, 1, RootNamespace{"a"}, creator), + CasRefCatalog::NamespaceCreationOutcome::Live); + + CasOperation other = f.external.admit(); + const CatalogEntry observed = CasRefCatalog::read(other, layout).catalog.entries.at(0); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, observed, 9), CasRefCatalog::BeginRemovingOutcome::Transitioned); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(observed.incarnation, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const CatalogEntry removing{.ns = RootNamespace{"a"}, .state = NsState::Removing, + .incarnation = observed.incarnation, .removal_started_round = 9}; + ASSERT_EQ(CasRefCatalog::deleteCompletedRemoving(other, layout, removing, ready_parent, noAuthorityRefresh).outcome, + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + /// The store now carries no row for "a"; the pool's cache still holds its own stale "a" Live from + /// the first `createNamespace` above. `createNamespace`'s own pre-check read is unconditional and + /// always fresh, so if the rival lands its row BEFORE this call, the pre-check alone would already + /// return `Superseded` and step 1 -- the lane -- would never run. The rival instead lands INSIDE + /// the window `setCreateNamespaceStep1PreReadHookForTest` names: after the pre-check already saw + /// nothing, but before step 1's own (lane) read. `createNamespaceStep1`'s decide then runs first + /// against the CACHED hint (still "a" Live) -- which already has a row for "a", so it throws + /// immediately without ever reaching the store -- and that verdict-on-a-hint is dropped: the lane + /// re-reads for real and decides again, this time seeing the rival's fresh row, and throws again. + /// That second throw is the one that reaches `createNamespace`. + const CreatorFence other_creator{.server_root_id = "other", .writer_epoch = 1, .fence_generation = 1}; + CatalogEntry other_creating = entryInState("a", NsState::Creating, 5); + other_creating.creator = other_creator; + CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest([&] + { + CasRefCatalog::casAdmitEntry(other, layout, 1, other_creating); + }); + + /// The hook's own admission is a real write on the SAME shared backend the pool uses, so a raw + /// backend read/write count taken across this call would count the rival's own IO alongside the + /// lane's -- `CASHotKeyCacheVerdictsReread` does not: it increments only when a CACHED hint's + /// verdict is dropped and re-decided, which the rival's cache-less admission (its own private, + /// budget-0 `CasHotKeys`) never does, so it isolates the lane's own contribution cleanly. + const uint64_t rereads_before = eventCount(ProfileEvents::CASHotKeyCacheVerdictsReread); + EXPECT_EQ(CasRefCatalog::createNamespace(pool_op, layout, 1, RootNamespace{"a"}, creator), + CasRefCatalog::NamespaceCreationOutcome::Superseded); + EXPECT_EQ(eventCount(ProfileEvents::CASHotKeyCacheVerdictsReread) - rereads_before, 1u) + << "one lane read: the cached hint's verdict was dropped and redecided on a fresh read"; + + /// No write of ours landed: the row for "a" is exactly what the rival left it as. + const CasRefCatalog::Snapshot snap_after = CasRefCatalog::read(other, layout); + const CatalogEntry * after = &snap_after.catalog.entries.at(0); + EXPECT_EQ(after->state, NsState::Creating); + ASSERT_TRUE(after->creator.has_value()); + EXPECT_EQ(*after->creator, other_creator); + EXPECT_EQ(after->incarnation, other_creating.incarnation); +} + +/// completeCreation: the cache holds "a" Creating (incarnation 1) matching the store's row for "a", +/// but the store also gained row "b" -- the cache is stale only elsewhere, so the write goes through. +TEST(CASRefCatalog, AStaleHintCompleteCreationLandsWhenItsOwnRowIsUnchanged) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + const CreatorFence creator{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}; + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a_creating = entryInState("a", NsState::Creating, 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a_creating); /// cache: "a" Creating + + CasOperation other = f.external.admit(); + const CreatorFence other_creator{.server_root_id = "other", .writer_epoch = 1, .fence_generation = 1}; + ASSERT_EQ(CasRefCatalog::createNamespace(other, layout, 1, RootNamespace{"b"}, other_creator), + CasRefCatalog::NamespaceCreationOutcome::Live); + + EXPECT_EQ(CasRefCatalog::completeCreation(pool_op, layout, a_creating), CasRefCatalog::NamespaceCreationOutcome::Live); + EXPECT_EQ(CasRefCatalog::read(other, layout).catalog.entries.size(), 2u); +} + +/// completeCreation: "a" was completed to Live by the external -- Superseded on the read. +TEST(CASRefCatalog, ATrueRefusalCompleteCreationSeesItsOwnRowCompletedElsewhere) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a_creating = entryInState("a", NsState::Creating, 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a_creating); + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::completeCreation(other, layout, a_creating), CasRefCatalog::NamespaceCreationOutcome::Live); + + EXPECT_EQ(CasRefCatalog::completeCreation(pool_op, layout, a_creating), CasRefCatalog::NamespaceCreationOutcome::Superseded); +} + +/// reconcileStaleCreator: the cache holds "a" Creating under creator X; the store's row for "a" is +/// unchanged but the catalog gained "b" -- Reconciled. +TEST(CASRefCatalog, AStaleHintReconcileStaleCreatorLandsWhenItsOwnRowIsUnchanged) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a_creating = entryInState("a", NsState::Creating, 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a_creating); /// cache: "a" Creating under creator X + + CasOperation other = f.external.admit(); + const CreatorFence other_creator{.server_root_id = "other", .writer_epoch = 1, .fence_generation = 1}; + ASSERT_EQ(CasRefCatalog::createNamespace(other, layout, 1, RootNamespace{"b"}, other_creator), + CasRefCatalog::NamespaceCreationOutcome::Live); + + const CreatorFence new_creator{.server_root_id = "srv2", .writer_epoch = 1, .fence_generation = 1}; + EXPECT_EQ(CasRefCatalog::reconcileStaleCreator(pool_op, layout, a_creating, new_creator, + [](const CreatorFence &) { return true; }), + CasRefCatalog::ReconcileCreatorOutcome::Reconciled); + EXPECT_EQ(CasRefCatalog::read(other, layout).catalog.entries.size(), 2u); +} + +/// reconcileStaleCreator: "a" was completed to Live by the external -- EntryChanged on the read. +TEST(CASRefCatalog, ATrueRefusalReconcileStaleCreatorSeesItsOwnRowCompletedElsewhere) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a_creating = entryInState("a", NsState::Creating, 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a_creating); + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::completeCreation(other, layout, a_creating), CasRefCatalog::NamespaceCreationOutcome::Live); + + const CreatorFence new_creator{.server_root_id = "srv2", .writer_epoch = 1, .fence_generation = 1}; + EXPECT_EQ(CasRefCatalog::reconcileStaleCreator(pool_op, layout, a_creating, new_creator, + [](const CreatorFence &) { return true; }), + CasRefCatalog::ReconcileCreatorOutcome::EntryChanged); +} + +/// Admission refusal (LIMIT_EXCEEDED): the pool's cache holds a catalog at the namespace limit, but +/// the external erased one row, so the fresh catalog admits. +TEST(CASRefCatalog, AStaleHintAdmissionRefusalAdmitsWhenTheFreshCatalogHasRoom) +{ + const Layout layout("p"); + constexpr uint64_t gc_shards = 1; + const uint64_t cap = foldSealCaps().object_cap; + const uint64_t fixed = foldSealFixedBytes(); + const uint64_t reservation = worstCaseEntryFoldReservationBytes(); + const uint64_t nonentry = widestBlobTargetRunReservationBytes(layout, gc_shards) + + widestCondemnedSummaryReservationBytes(gc_shards); + const uint64_t max_entries = (cap - fixed - nonentry) / reservation; + + PoolAndExternal f(std::make_shared()); + CasOperation pool_op = f.pool.admit(); + RefCatalog full; + full.entries.reserve(max_entries); + for (uint64_t i = 0; i < max_entries; ++i) + full.entries.push_back(liveEntry(fmt::format("ns{:012}", i), i + 1)); + seedObject(pool_op, layout.refCatalogKey(), encodeRefCatalog(full)); + /// Prime the cache with the full catalog: any further admission through the pool refuses on it. + CasRefCatalog::casUpdate(pool_op, layout, [](const RefCatalog & cur) { return cur; }); + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, full.entries[0], 5), CasRefCatalog::BeginRemovingOutcome::Transitioned); + const CasRefCatalog::Snapshot after_remove = CasRefCatalog::read(other, layout); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(full.entries[0].incarnation, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const CatalogEntry removing{.ns = full.entries[0].ns, .state = NsState::Removing, + .incarnation = full.entries[0].incarnation, .removal_started_round = 5}; + ASSERT_EQ(CasRefCatalog::deleteCompletedRemovingAtSnapshot(other, layout, after_remove, removing, ready_parent, noAuthorityRefresh).outcome, + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + + const CreatorFence creator{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}; + EXPECT_EQ(CasRefCatalog::createNamespace(pool_op, layout, gc_shards, RootNamespace{"fresh"}, creator), + CasRefCatalog::NamespaceCreationOutcome::Live); +} + +/// Admission refusal (LIMIT_EXCEEDED): the fresh catalog is also full -- thrown, no write. +TEST(CASRefCatalog, ATrueRefusalAdmissionRefusalThrowsWhenTheFreshCatalogIsAlsoFull) +{ + const Layout layout("p"); + constexpr uint64_t gc_shards = 1; + const uint64_t cap = foldSealCaps().object_cap; + const uint64_t fixed = foldSealFixedBytes(); + const uint64_t reservation = worstCaseEntryFoldReservationBytes(); + const uint64_t nonentry = widestBlobTargetRunReservationBytes(layout, gc_shards) + + widestCondemnedSummaryReservationBytes(gc_shards); + const uint64_t max_entries = (cap - fixed - nonentry) / reservation; + + PoolAndExternal f(std::make_shared()); + CasOperation pool_op = f.pool.admit(); + RefCatalog full; + full.entries.reserve(max_entries); + for (uint64_t i = 0; i < max_entries; ++i) + full.entries.push_back(liveEntry(fmt::format("ns{:012}", i), i + 1)); + seedObject(pool_op, layout.refCatalogKey(), encodeRefCatalog(full)); + CasRefCatalog::casUpdate(pool_op, layout, [](const RefCatalog & cur) { return cur; }); /// prime the cache + + const uint64_t writes_before = f.backend->writeCount(layout.refCatalogKey()); + const CreatorFence creator{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}; + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LIMIT_EXCEEDED, [&] + { + (void)CasRefCatalog::createNamespace(pool_op, layout, gc_shards, RootNamespace{"overflow"}, creator); + }); + EXPECT_EQ(f.backend->writeCount(layout.refCatalogKey()), writes_before) << "no write"; +} + +/// casAdmitEntry: the cache holds "a"; the external erased it -- admits it again on the fresh read. +/// The stale hint's own decide sees "a" still Live and duplicates the namespace before the lane can +/// reread, so `encodeRefCatalog` throws `LOGICAL_ERROR` on the hint attempt -- split like the blocks +/// above, since constructing that exception aborts under debug/sanitizer builds before this test's own +/// rescue (the reread on a fresh, "a"-absent state) ever runs. +#ifndef DEBUG_OR_SANITIZER_BUILD +TEST(CASRefCatalog, AStaleHintCasAdmitEntryAdmitsWhenTheStoreHasRoom) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a = liveEntry("a", 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a); /// cache: "a" + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, a, 5), CasRefCatalog::BeginRemovingOutcome::Transitioned); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(a.incarnation, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const CatalogEntry removing{.ns = a.ns, .state = NsState::Removing, + .incarnation = a.incarnation, .removal_started_round = 5}; + ASSERT_EQ(CasRefCatalog::deleteCompletedRemoving(other, layout, removing, ready_parent, noAuthorityRefresh).outcome, + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + + const uint64_t reads_before = f.backend->getCount(layout.refCatalogKey()); + EXPECT_NO_THROW(CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("a", 2))); + EXPECT_EQ(f.backend->getCount(layout.refCatalogKey()) - reads_before, 1u) << "one lane read"; +} +#endif + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASRefCatalogDeathTest, AStaleHintCasAdmitEntryAdmitsWhenTheStoreHasRoomAborts) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a = liveEntry("a", 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a); /// cache: "a" + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, a, 5), CasRefCatalog::BeginRemovingOutcome::Transitioned); + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(a.incarnation, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + const CatalogEntry removing{.ns = a.ns, .state = NsState::Removing, + .incarnation = a.incarnation, .removal_started_round = 5}; + ASSERT_EQ(CasRefCatalog::deleteCompletedRemoving(other, layout, removing, ready_parent, noAuthorityRefresh).outcome, + CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + + /// The lane's re-render on a fresh read cannot rescue this, because constructing the + /// `LOGICAL_ERROR` for the hint's own duplicate aborts before the reread ever happens. + EXPECT_DEATH({ CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("a", 2)); }, "not canonically ordered"); +} +#endif + +/// casAdmitEntry: "a" is still present in the fresh read (Removing, not the Live hint the cache +/// holds) -- re-admitting it duplicates the namespace, and `encodeRefCatalog`'s own canonical-order +/// check aborts with `LOGICAL_ERROR` under debug/sanitizer builds -- split like the blocks above. +#ifndef DEBUG_OR_SANITIZER_BUILD +TEST(CASRefCatalog, ATrueRefusalCasAdmitEntrySeesItsRowStillPresent) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a = liveEntry("a", 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a); + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, a, 5), CasRefCatalog::BeginRemovingOutcome::Transitioned); + + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, + [&] { CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a); }); +} +#endif + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASRefCatalogDeathTest, ATrueRefusalCasAdmitEntrySeesItsRowStillPresentAborts) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry a = liveEntry("a", 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a); + + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, a, 5), CasRefCatalog::BeginRemovingOutcome::Transitioned); + + EXPECT_DEATH({ CasRefCatalog::casAdmitEntry(pool_op, layout, 1, a); }, "not canonically ordered"); +} +#endif + +TEST(CASRefCatalog, AFenceLostDuringStepOneIsFencedOutNotABareMarker) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation seed = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(seed, layout); + const CreatorFence creator{.server_root_id = "srv", .writer_epoch = 1, .fence_generation = 1}; + + /// Tripped before its turn: the lane's wait refuses it, nothing is written. + { + bool alive = true; + CasOperation op = f.pool.admit([&] { return alive; }); + f.hot_keys.enter_after_lane_hook_for_test = [&] { alive = false; }; + const uint64_t writes_before = f.backend->writeCount(layout.refCatalogKey()); + EXPECT_EQ(CasRefCatalog::createNamespace(op, layout, 1, RootNamespace{"a"}, creator), + CasRefCatalog::NamespaceCreationOutcome::FencedOut); + f.hot_keys.enter_after_lane_hook_for_test = {}; + EXPECT_EQ(f.backend->writeCount(layout.refCatalogKey()), writes_before); + } + /// Tripped between the landed step-1 write and the post-commit check: FencedOut, and the Creating + /// row is durable for a later reconciler. + { + bool alive = true; + CasOperation op = f.pool.admit([&] { return alive; }); + f.backend->onWriteCommitted(layout.refCatalogKey(), [&] { alive = false; }); + EXPECT_EQ(CasRefCatalog::createNamespace(op, layout, 1, RootNamespace{"b"}, creator), + CasRefCatalog::NamespaceCreationOutcome::FencedOut); + f.backend->onWriteCommitted(layout.refCatalogKey(), {}); + const auto snap = CasRefCatalog::read(seed, layout); + ASSERT_EQ(snap.catalog.entries.size(), 1u); + EXPECT_EQ(snap.catalog.entries[0].state, NsState::Creating); + } + /// casAdmitEntry under a tripped fence throws the transient-unavailable class, never a bare marker. + { + CasOperation op = f.pool.admit([] { return false; }); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, + [&] { CasRefCatalog::casAdmitEntry(op, layout, 1, liveEntry("c", 3)); }); + } +} + +#if USE_AWS_S3 +TEST(CASRefCatalog, TheStoresAnswerToAHintsWriteIsDeliveredAndTheRetryLearnsTheRest) +{ + PoolAndExternal f(std::make_shared()); + f.backend->setRefreshCredentialsResult(false); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry observed = liveEntry("a", 1); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, observed); /// the cache holds "a" Live + + /// Another server moved the row on; the hint still matches what this caller observed, so a write + /// goes out, and the store refuses it definitively. + CasOperation other = f.external.admit(); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, observed, 5), CasRefCatalog::BeginRemovingOutcome::Transitioned); + f.backend->failNextWriteWith(layout.refCatalogKey(), s3Error(Aws::S3::S3Errors::ACCESS_DENIED, "AccessDenied")); + const uint64_t writes_before = f.backend->writeCount(layout.refCatalogKey()); + const uint64_t reads_before = f.backend->getCount(layout.refCatalogKey()); + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::S3_ERROR, + [&] { (void)CasRefCatalog::beginRemoving(pool_op, layout, observed, 5); }); + EXPECT_EQ(f.backend->writeCount(layout.refCatalogKey()) - writes_before, 1u); + EXPECT_EQ(f.backend->getCount(layout.refCatalogKey()) - reads_before, 0u); + /// The caller's ordinary retry after a fresh observation learns the row moved. + EXPECT_EQ(CasRefCatalog::beginRemoving(pool_op, layout, observed, 5), CasRefCatalog::BeginRemovingOutcome::AlreadyRemoving); + + /// The same stale hint, and the fence trips between the refused write and its resolve read. + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("b", 2)); + const CatalogEntry b = liveEntry("b", 2); + ASSERT_EQ(CasRefCatalog::beginRemoving(other, layout, b, 6), CasRefCatalog::BeginRemovingOutcome::Transitioned); + bool alive = true; + CasOperation fenced = f.pool.admit([&] { return alive; }); + f.backend->onBeforeWrite(layout.refCatalogKey(), [&] { alive = false; }); + EXPECT_EQ(CasRefCatalog::beginRemoving(fenced, layout, b, 6), CasRefCatalog::BeginRemovingOutcome::FencedOut); + f.backend->onBeforeWrite(layout.refCatalogKey(), {}); + EXPECT_EQ(CasRefCatalog::read(other, layout).catalog.entries.at(1).state, NsState::Removing) << "nothing of ours landed"; + CasOperation readmitted = f.pool.admit(); + EXPECT_EQ(CasRefCatalog::beginRemoving(readmitted, layout, b, 6), CasRefCatalog::BeginRemovingOutcome::AlreadyRemoving); +} +#endif + +TEST(CASRefCatalog, TheCatalogLoopPacesAConflictByWhetherItSettledAFault) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("a", 1)); + CasOperation other = f.external.admit(); + const String key = layout.refCatalogKey(); + + /// K clean races: the external moves the catalog before each of the pool's first K writes. The + /// two scenarios below admit under DIFFERENT namespace prefixes -- the first scenario's "x" rows + /// are still on the catalog when the second starts, and re-admitting the same namespace would + /// duplicate it rather than race it. + constexpr int K = 3; + /// The settled-fault scenario draws 8, not 3: an upper-bound-only check on `backoff`'s draws + /// cannot tell the growing schedule from the flat one (both stay under 200 ms plenty often at + /// small K), so that scenario also needs a floor. With `backoff(attempt) = uniform(0, min(5000, + /// 200 << (attempt-1)))`, the flat schedule can NEVER draw over 200 ms, so a single draw over + /// 200 ms is proof by itself that the schedule grew; drawing 8 times keeps the growing schedule's + /// own chance of missing that floor purely by bad luck around 2^-28. + constexpr int K2 = 8; + int moved = 0; + int limit = K; + bool inside = false; + bool ambiguous = false; + String prefix = "x"; + uint64_t incarnation_base = 10; + f.backend->onBeforeWrite(key, [&] + { + if (inside || moved >= limit) + return; + inside = true; + CasRefCatalog::casAdmitEntry(other, layout, 1, liveEntry(prefix + std::to_string(moved), incarnation_base + moved)); + if (ambiguous) + f.backend->injectAmbiguousWrite(key); + ++moved; + inside = false; + }); + const auto pauses_before = eventCount(ProfileEvents::CASRequestConflictPause); + CasRefCatalog::casUpdate(pool_op, layout, [](const RefCatalog & cur) + { + RefCatalog next = cur; + for (auto & e : next.entries) + if (e.ns.string() == "a") { e.state = NsState::Removing; e.removal_started_round = 1; } + return next; + }); + ASSERT_EQ(f.clock.sleeps.size(), static_cast(K)); + for (uint64_t s : f.clock.sleeps) + EXPECT_LE(s, 200u); + EXPECT_EQ(eventCount(ProfileEvents::CASRequestConflictPause) - pauses_before, 0u) << "the lane's caller pauses itself; the engine's counter is for its own loops"; + + /// K2 conflicts that each settled a fault: the growing schedule, on the loop's own count. An + /// upper-bound check alone is satisfied by the flat schedule too (its draws are a subset of the + /// growing schedule's early-attempt range), so this also asserts a floor: at least one draw over + /// 200 ms, which the flat schedule can never produce. + f.clock.sleeps.clear(); + moved = 0; + limit = K2; + ambiguous = true; + prefix = "y"; + incarnation_base = 20; + CasRefCatalog::casUpdate(pool_op, layout, [](const RefCatalog & cur) { return cur; }); + ASSERT_EQ(f.clock.sleeps.size(), static_cast(K2)); + for (size_t i = 0; i < f.clock.sleeps.size(); ++i) + EXPECT_LE(f.clock.sleeps[i], std::min(5000, 200ull << i)); + EXPECT_TRUE(std::any_of(f.clock.sleeps.begin(), f.clock.sleeps.end(), [](uint64_t s) { return s > 200u; })) + << "the flat schedule can never draw over 200 ms; a growing schedule almost certainly does " + "somewhere in 8 draws, so this is what tells the two schedules apart"; +} + +TEST(CASRefCatalog, TheGCEraseRacesTheLaneAsItRacesEverything) +{ + PoolAndExternal f(std::make_shared()); + const Layout layout("p"); + CasOperation pool_op = f.pool.admit(); + CasRefCatalog::initializeEmptyForNewPool(pool_op, layout); + const CatalogEntry removing{.ns = RootNamespace{"a"}, .state = NsState::Removing, + .incarnation = UInt128{7}, .removal_started_round = 13}; + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("keep", 1)); + /// Seed the Removing row through the external, as raw bytes: `casUpdate` refuses to add rows, and + /// the point is that the pool's cache is stale about this one. + CasOperation other = f.external.admit(); + { + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(other, layout); + RefCatalog with_removing = snap.catalog; + with_removing.entries.insert(with_removing.entries.begin(), removing); /// "a" sorts before "keep" + (void)orThrow(other.replace(layout.refCatalogKey(), encodeRefCatalog(with_removing), *snap.etag, Retry::standard()), "seed"); + } + CasFoldSeal ready_parent; + ready_parent.ref_lives.emplace(UInt128{7}, RefLifeFoldState{ + .coverage = RefCoverage{.classification = CoverageClass::Folded, .last_folded_ref_id = RefTxnId{1, 2}}, + .cleanup_evidence = RefCleanupEvidence{.remove_txn_id = RefTxnId{1, 2}}}); + + /// The erase runs on an open plane of the same pool while a lane holder is parked in its write: + /// it does not wait. + CasRequests open_plane(f.backend, Fence::open(), f.clock.nowFn(), f.clock.sleepFn(), &f.hot_keys); + CasOperation erase_op = open_plane.admit(); + std::latch parked(1); + std::latch release(1); + int seen = 0; + f.backend->onBeforeWrite(layout.refCatalogKey(), [&] + { + if (seen++ != 0) + return; + parked.count_down(); + release.wait(); + }); + std::thread holder([&] + { + CasRefCatalog::casUpdate(pool_op, layout, [](const RefCatalog & cur) { return cur; }); + }); + parked.wait(); + const auto result = CasRefCatalog::deleteCompletedRemovingAtSnapshot( + erase_op, layout, CasRefCatalog::read(other, layout), removing, ready_parent, noAuthorityRefresh); + EXPECT_EQ(result.outcome, CasRefCatalog::CompletedRemovingDeleteOutcome::Deleted); + EXPECT_EQ(f.hot_keys.queueDepthForTest(layout.refCatalogKey()), 1u) << "the holder is still parked"; + release.count_down(); + holder.join(); + f.backend->onBeforeWrite(layout.refCatalogKey(), {}); + + /// The pool's next submission pays exactly one resolve read and one retry write for the erase. + const uint64_t reads_before = f.backend->getCount(layout.refCatalogKey()); + const uint64_t writes_before = f.backend->writeCount(layout.refCatalogKey()); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("c", 3)); + EXPECT_LE(f.backend->getCount(layout.refCatalogKey()) - reads_before, 1u); + EXPECT_LE(f.backend->writeCount(layout.refCatalogKey()) - writes_before, 2u); + const uint64_t reads_after = f.backend->getCount(layout.refCatalogKey()); + CasRefCatalog::casAdmitEntry(pool_op, layout, 1, liveEntry("d", 4)); + EXPECT_EQ(f.backend->getCount(layout.refCatalogKey()), reads_after) << "the one after starts from the cache"; +} diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index a14cf0debae8..6abb79e3def6 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -9,6 +9,7 @@ #include #include #include "cas_test_helpers.h" +#include #include @@ -38,6 +39,12 @@ extern const int S3_ERROR; extern const int NETWORK_ERROR; } +namespace ProfileEvents +{ + extern const Event CASRequestReissue; + extern const Event CASRequestConflictPause; +} + using namespace DB::Cas; using DB::Cas::tests::CountingBackend; @@ -1418,6 +1425,182 @@ TEST(CASRequests, ReadModifyWriteLosesNoIncrementUnderContentionAndBoundsAHotKey EXPECT_FALSE(clock.sleeps.empty()); /// it paced its retries rather than spinning } +namespace +{ + +/// Moves `key` under the caller before each of its first `moves` write attempts, and optionally makes +/// each of those attempts ambiguous (the store never answers it) so the resolve read is what settles +/// the race. +struct RaceMaker +{ + RaceMaker(std::shared_ptr backend_, FakeClock & clock, String key_, int moves_, bool ambiguous_) + : backend(std::move(backend_)), key(std::move(key_)), moves(moves_), ambiguous(ambiguous_) + , rival_requests(makeRequests(backend, clock)), rival(rival_requests.admit()) + { + backend->onBeforeWrite(key, [this] + { + if (inside || made >= moves) + return; + inside = true; + if (const auto current = rival.read(key, Retry::once())) + (void)rival.replace(key, current->bytes + "r", current->etag, Retry::once()); + else + (void)rival.create(key, "r", Retry::once()); + if (ambiguous) + backend->injectAmbiguousWrite(key); + ++made; + inside = false; + }); + } + + std::shared_ptr backend; + String key; + int moves; + bool ambiguous; + int made = 0; + bool inside = false; + CasRequests rival_requests; + CasOperation rival; +}; + +DecideOnObject appendX() +{ + return [](const std::optional & current) -> std::optional + { + return current ? current->bytes + "x" : String("x"); + }; +} + +} + +TEST(CASRequests, CleanConflictsArePacedFlatAndDoNotAdvanceTheReissueCounter) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + (void)orThrow(op.create("k", "v", Retry::standard()), "seed"); + constexpr int K = 4; + RaceMaker races(backend, clock, "k", K, /*ambiguous=*/false); + const auto pauses_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConflictPause].load(); + const auto reissues_before = ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load(); + + WriteResult result = op.readModifyWrite("k", appendX(), Retry::standard()); + + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConflictPause].load() - pauses_before, K); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load() - reissues_before, 0u); + ASSERT_EQ(clock.sleeps.size(), static_cast(K)); + for (uint64_t s : clock.sleeps) + EXPECT_LE(s, 200u); /// flat: every pause is one `backoff(1)` draw, whatever the loss count + EXPECT_EQ(backend->writeCount("k"), 1u + K + 1u + K); /// seed, K refused, K rival moves, the one that landed +} + +TEST(CASRequests, AConflictThatSettledAFaultKeepsTheGrowingSchedule) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + (void)orThrow(op.create("k", "v", Retry::standard()), "seed"); + constexpr int K = 3; + RaceMaker races(backend, clock, "k", K, /*ambiguous=*/true); + const auto pauses_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConflictPause].load(); + const auto reissues_before = ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load(); + + WriteResult result = op.readModifyWrite("k", appendX(), Retry::standard()); + + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConflictPause].load() - pauses_before, 0u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load() - reissues_before, K); + ASSERT_EQ(clock.sleeps.size(), static_cast(K)); + for (size_t i = 0; i < clock.sleeps.size(); ++i) + EXPECT_LE(clock.sleeps[i], std::min(5000, 200ull << i)) << "reissue " << i; +} + +TEST(CASRequests, ReplaceReportsWhetherAConflictSettledAFault) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Etag seed = *orThrow(op.create("k", "v", Retry::standard()), "seed"); + { + RaceMaker clean(backend, clock, "k", 1, /*ambiguous=*/false); + WriteResult result = op.replace("k", "w", seed, Retry::standard()); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + EXPECT_FALSE(conflict->any_ambiguous); + EXPECT_EQ(conflict->attempts_sent, 1u); + } + { + RaceMaker faulty(backend, clock, "k", 1, /*ambiguous=*/true); + WriteResult result = op.replace("k", "w", seed, Retry::standard()); + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + EXPECT_TRUE(conflict->any_ambiguous); + EXPECT_EQ(conflict->attempts_sent, 1u); /// one attempt, lost, settled as moved: `attempts_sent` cannot tell + } +} + +TEST(CASRequests, OnPresenceUnderOnceKeepsTheFaultFlagOnTheRebuiltConflict) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + (void)orThrow(op.create("k", "v", Retry::standard()), "seed"); + RaceMaker faulty(backend, clock, "k", 1, /*ambiguous=*/true); + + WriteResult result = op.readModifyWriteOnPresence("k", + [](const std::optional &) -> std::optional { return String("w"); }, Retry::once()); + + const auto * conflict = std::get_if(&result); + ASSERT_NE(conflict, nullptr); + EXPECT_TRUE(conflict->any_ambiguous); + EXPECT_TRUE(std::holds_alternative(conflict->seen)); /// presence-only, as before +} + +TEST(CASRequests, CleanConflictsBeforeAFaultDoNotInflateTheFaultsFirstBackoff) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + (void)orThrow(op.create("k", "v", Retry::standard()), "seed"); + constexpr int K = 3; + RaceMaker races(backend, clock, "k", K, /*ambiguous=*/false); + /// After the K clean races the next attempt is ambiguous with the precondition unchanged, so the + /// engine reissues it; that reissue's pause must be the schedule's first, not its (K+1)-th. + bool armed = false; + backend->onBeforeWrite("k", [&] + { + /// The RaceMaker's hook is replaced by this one; it moves the key itself for the first K writes. + if (races.inside) + return; + if (races.made < K) + { + races.inside = true; + if (const auto current = races.rival.read("k", Retry::once())) + (void)races.rival.replace("k", current->bytes + "r", current->etag, Retry::once()); + ++races.made; + races.inside = false; + return; + } + if (!armed) + { + armed = true; + backend->injectAmbiguousWrite("k"); + } + }); + + WriteResult result = op.readModifyWrite("k", appendX(), Retry::standard()); + + ASSERT_TRUE(std::holds_alternative(result)); + ASSERT_EQ(clock.sleeps.size(), static_cast(K + 1)); + EXPECT_LE(clock.sleeps[K], 200u) << "the first transport reissue sleeps within backoff(1)"; +} + TEST(CASRequests, ADeterministicLocalFailureSurfacesUnchangedWithoutAReissue) { FakeClock clock; From 38661e72127977a67bd68780415ad8e052265f0f Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:08:22 +0200 Subject: [PATCH 23/81] cas: fix ASan use-after-scope in test hooks that outlive their captured locals MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Several gtest fixtures declared a test-only backend hook or fault clock as a local, captured other locals in it by reference, and then let the store (declared before the hook) call the hook again during its own teardown — after the captured locals had already gone out of scope. Under ASan this is a use-after-scope: the store's destructor writes a "farewell" record that can invoke a still-armed hook whose captured references are already dead. Fixed by declaring the test clock/hook before the store that keeps calling it (two transient-round tests, the straggler-epoch test), and by clearing the checkpoint-advance recovery test's backend hook before the locals it captures die. A separate scripted S3 client fix allocates its response body with `Aws::New`, matching how the SDK actually frees it, instead of a mismatched allocator. Also corrects two suites that had started asserting a schedule the engine never promised. Also: the stateless CAS lanes now run the GC scheduler every 20 s instead of every 5 s, matching the interval those tests actually need. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_detached_work.cpp | 32 +++++++++++++------ src/Disks/tests/gtest_cas_gc_log.cpp | 9 +++--- .../tests/gtest_cas_ref_recovery_cas_walk.cpp | 5 +++ .../tests/gtest_cas_retirement_sweep.cpp | 6 ++-- src/Disks/tests/gtest_cas_upstream_slice.cpp | 5 ++- src/Disks/tests/gtest_cas_writer_duties.cpp | 16 +++++++++- ...orage_policy_for_merge_tree_by_default.xml | 2 +- ...orage_policy_for_merge_tree_by_default.xml | 2 +- 8 files changed, 58 insertions(+), 19 deletions(-) diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index e309aea683d1..056cfde61669 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -696,18 +696,16 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) EXPECT_GE(error_hook_calls.load(), 1u) << "the injected throw never reached the error handler"; /// One step of the schedule per iteration: the tail is still over threshold, so each mutation - /// re-evaluates admission, and exactly one attempt may pass per elapsed backoff interval. + /// re-evaluates admission, and AT MOST one attempt may pass per elapsed backoff interval. At most, + /// not exactly: a publisher dispatched by a mutation whose append has not yet returned the lane to + /// `Ready` is refused at that gate, and the refusal arms the same backoff without the attempt ever + /// reaching the hook below -- so a step can legitimately elapse with no attempt of its own. The + /// regression this test exists for is the opposite, an unpaced redispatch storm, and the ceiling + /// is what catches it; progress is asserted once after the loop. for (uint64_t step = 1; step <= 3; ++step) { fake_boot.fetch_add(step_ms); ASSERT_NO_THROW(publishRef(store, ns, "ref_" + std::to_string(step + 1), step + 1)); - const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10); - while (attempts.load() < 1 + step) - { - ASSERT_LT(std::chrono::steady_clock::now(), deadline) - << "the elapsed backoff never admitted the next publish attempt"; - std::this_thread::yield(); - } /// Bounded poll rather than `waitForSnapshotPublishSettleForTest`: that call waits on a condvar /// predicate with no deadline, and on an unpaced-redispatch regression the reservation count /// never rests at zero long enough for the predicate to observe it, hanging the test instead of @@ -720,8 +718,24 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) << "pending_snapshot_publishes stayed nonzero"; std::this_thread::yield(); } - EXPECT_EQ(attempts.load(), 1 + step) << "more than one publish attempt ran within one backoff step"; + EXPECT_LE(attempts.load(), 1 + step) << "more than one publish attempt ran within one backoff step"; + } + + /// Progress, on the injected clock so it is deterministic rather than a race with a worker: an + /// elapsed backoff must eventually admit a further attempt, or the pacing gate would be a wedge. + for (uint64_t extra = 0; attempts.load() < 2 && extra < 20; ++extra) + { + fake_boot.fetch_add(step_ms); + ASSERT_NO_THROW(publishRef(store, ns, "ref_progress_" + std::to_string(extra), 100 + extra)); + const auto settle = std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (store->pendingSnapshotPublishesForTest(ns) != 0) + { + ASSERT_LT(std::chrono::steady_clock::now(), settle) << "a publish never settled"; + std::this_thread::yield(); + } } + EXPECT_GE(attempts.load(), 2u) + << "no elapsed backoff ever admitted a further publish attempt: the gate is a wedge, not a pace"; EXPECT_EQ(error_hook_calls.load(), attempts.load()); store->setSnapshotAfterCaptureHookForTest(nullptr); diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index 0812137a87df..b0fd72cb5869 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -333,10 +333,11 @@ class NetworkThrowingBackend : public InMemoryBackend TEST(CASGCLog, TransientThrowIsClassifiedAborted) { auto backend = std::make_shared(); - auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); /// A PERSISTENT transient fault is reissued for the whole retry window, so the window has to run - /// on a clock this test advances -- otherwise one read spends ninety real seconds. + /// on a clock this test advances -- otherwise one read spends ninety real seconds. Declared BEFORE + /// the store: the store's teardown still calls the now-function, so the clock must outlive it. std::atomic engine_now_ms{0}; + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); store->setCasRequestNowFnForTest([&] { return engine_now_ms.fetch_add(10'000) + 10'000; }); store->setCasRetrySleepForTest([](uint64_t) {}); @@ -475,10 +476,10 @@ class ModalThrowingBackend : public InMemoryBackend TEST(CASGCScheduler, TransientRoundFailureKeepsLeadershipAndHeartbeat) { auto backend = std::make_shared(); - auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); /// See `TransientThrowIsClassifiedAborted`: the transient mode is persistent while it is armed, so - /// the retry window runs on a clock this test advances. + /// the retry window runs on a clock this test advances, declared before the store it outlives. std::atomic engine_now_ms{0}; + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); store->setCasRequestNowFnForTest([&] { return engine_now_ms.fetch_add(10'000) + 10'000; }); store->setCasRetrySleepForTest([](uint64_t) {}); diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index 50c608d66469..c26e96735be6 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -1,5 +1,7 @@ #include +#include + #include "config.h" #include @@ -1793,6 +1795,9 @@ TEST(CASRefRecoveryCasWalk, WriterRecoveryRestartsWhenCheckpointAdvancesPastPriv /// `{1,3}` and publish its frontier before recovery's own checkpoint CAS. The stale private /// candidate contains only `b`; it must restart and replay `c`, not accept an `IdenticalSkip` and /// install below the exact checkpoint it just observed. + /// The hook captures locals declared after the store, and the store's teardown still performs + /// checkpoint writes, so the hook is cleared before those locals die. + SCOPE_EXIT({ backend->before_cas_put = {}; }); backend->before_cas_put = [&](const String & key, const String &, const std::optional & expected) { if (injected || key != ckpt_key) diff --git a/src/Disks/tests/gtest_cas_retirement_sweep.cpp b/src/Disks/tests/gtest_cas_retirement_sweep.cpp index 0a52f0055750..57ce4d5c4d76 100644 --- a/src/Disks/tests/gtest_cas_retirement_sweep.cpp +++ b/src/Disks/tests/gtest_cas_retirement_sweep.cpp @@ -354,6 +354,10 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); uint64_t fake_boot = 1'000'000; std::vector waits; + /// The append's own retry clock and its sleep log, installed further down; declared here, before + /// the store, because the store's teardown still calls the now-function they back. + uint64_t fake_retry = 0; + std::vector retry_sleeps; auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30000), @@ -383,8 +387,6 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS /// give-up is the append's own retry window -- paced on ITS OWN virtual clock, separate from /// `fake_boot` (the mount fence's), so the standard policy's full window is available to reissue /// against rather than being cut short by the 30s lease `fake_boot` also measures. - uint64_t fake_retry = 0; - std::vector retry_sleeps; store->setCasRequestNowFnForTest([&fake_retry] { return fake_retry; }); store->setCasRetrySleepForTest([&fake_retry, &retry_sleeps](uint64_t ms) { diff --git a/src/Disks/tests/gtest_cas_upstream_slice.cpp b/src/Disks/tests/gtest_cas_upstream_slice.cpp index 73f894b28b9e..c35bead7baa6 100644 --- a/src/Disks/tests/gtest_cas_upstream_slice.cpp +++ b/src/Disks/tests/gtest_cas_upstream_slice.cpp @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -363,7 +364,9 @@ class ScriptedGetObjectClient : public DB::S3::Client Aws::S3::Model::GetObjectResult result; result.SetETag(step.etag); result.SetContentLength(static_cast(step.body.size())); - result.ReplaceBody(new ScriptedBodyStream(step.body, step.fail_mid_body)); + /// The SDK releases the body through `Aws::Delete`, which frees with the SDK's allocator, so + /// the stream must come from `Aws::New`; a plain `new` here is an alloc-dealloc mismatch. + result.ReplaceBody(Aws::New("ScriptedBodyStream", step.body, step.fail_mid_body)); return Aws::S3::Model::GetObjectOutcome(std::move(result)); } diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index 5e34f05ee0a7..d763737a4172 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -114,7 +114,13 @@ void driveToNetworkErrorGiveUp(LatchedChunkFaultBackend & backend, DB::Cas::test const size_t pauses_before = clock.pauseCount(); DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, f); EXPECT_GE(backend.fault_hits - fault_hits_before, 1) << "the fault double must actually have fired"; - EXPECT_GT(clock.pauseCount() - pauses_before, 1u) + /// ONE pause is the whole discriminator: a reissue is paced by a backoff the engine sleeps + /// through, and a `once` policy -- which reaches this same give-up by propagating its first + /// failure -- never sleeps at all. How many MORE pauses follow is deliberately not asserted: the + /// backoff is full jitter, so the reissues that fit before the call gives up are a random small + /// number, and demanding two of them failed about one run in eight against a schedule that was + /// behaving exactly as designed. + EXPECT_GE(clock.pauseCount() - pauses_before, 1u) << "a give-up after a single attempt cannot distinguish a retrying `standard` policy from one " << "that never reissues at all"; EXPECT_GT(clock.longestPause(), 0u) << "at least one of the retry's pauses must be a real, nonzero backoff"; @@ -451,6 +457,13 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) /// regardless of epoch/coverage. const RootNamespace ns{"test/writer_duty_rejected_sweep"}; + /// The mount fence's own budget is read off the BOOT clock, while the retry window below is read + /// off the virtual one `VirtualRetryClock` installs -- so with a real boot clock the wall time + /// this test spends publishing the anchor and staging the manifest is subtracted from a 500 ms + /// lease, and on a loaded machine the give-up below stops being a retry give-up and becomes a + /// no-budget refusal before the first attempt. Freeze the boot clock, exactly as the successor + /// pool further down already does, so the only bound on that give-up is the one it asserts. + uint64_t predecessor_boot = 0; auto predecessor = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), @@ -462,6 +475,7 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = budget, + .boot_ms_fn = [&] { return predecessor_boot; }, }); auto clock = DB::Cas::tests::VirtualRetryClock::installOn(predecessor); diff --git a/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml b/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml index d74dfe173095..204fc4e4aed3 100644 --- a/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml +++ b/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml @@ -22,7 +22,7 @@ replaces the old grace window (M-W D-W5); a short interval so reclamation happens during the run. --> 1 - 5 + 20 1 - 5 + 20 From 438e063e7d79002997d600847b0a94c37e13fc9c Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:08:39 +0200 Subject: [PATCH 24/81] cas: log the single-attempt S3 client's failed attempt below Error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `S3ObjectStorage::getSingleAttemptClient` (`SingleAttemptRetryStrategy`, `max_retries=0`) and the write path carrying `WriteSettings::object_storage_retry_profile == SingleAttempt` belong to the CAS control plane's conditional writes: a failed attempt there is not the final answer — `CasOperation::writeLoop` resolves the outcome by a read and reissues — yet two upstream sites logged it at Error as if it were terminal: `Client`'s network-error handler and the non-412 `S3Exception` site in `WriteBufferFromS3`. Both now log at Debug when the client carries `SingleAttemptRetryStrategy` or the write carries the `SingleAttempt` profile; an ordinary client configured with zero retries by a user setting (no outer loop resolving it) keeps logging at Error, since for that caller the failure really is final. The neighbouring 412 (`isPreconditionFailedError`) branch drops from Info to Debug for the same reason: a conditional write losing its precondition is the caller's expected answer, not an operator-facing event. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- src/IO/S3/Client.cpp | 12 +- src/IO/S3/Client.h | 4 + src/IO/S3/tests/gtest_aws_s3_client.cpp | 179 ++++++++++++++++++++++++ src/IO/WriteBufferFromS3.cpp | 10 +- src/IO/tests/gtest_writebuffer_s3.cpp | 115 +++++++++++++++ 5 files changed, 317 insertions(+), 3 deletions(-) diff --git a/src/IO/S3/Client.cpp b/src/IO/S3/Client.cpp index 5a2ed079a8d3..c5280421de19 100644 --- a/src/IO/S3/Client.cpp +++ b/src/IO/S3/Client.cpp @@ -199,6 +199,11 @@ bool SingleAttemptRetryStrategy::ShouldRetry(const Aws::Client::AWSError(client_configuration.retryStrategy.get()) != nullptr; +} + namespace { @@ -814,7 +819,12 @@ Client::doRequestWithRetryNetworkErrors(RequestType & request, RequestFn request if (isClientForDisk()) incrementProfileEvents(ProfileEvents::DiskS3ReadRequestsErrors, ProfileEvents::DiskS3WriteRequestsErrors); - tryLogCurrentException(log, fmt::format("Network error on S3 request, attempt {} of {}", attempt_no, max_attempts)); + /// A client with the single-attempt strategy is owned by an outer retry loop that resolves + /// the outcome and reissues; its one failed attempt is not terminal, so it is not an error. + if (usesSingleAttemptRetryStrategy()) + LOG_DEBUG(log, "Network error on S3 request, attempt {} of {}: {}", attempt_no, max_attempts, getCurrentExceptionMessage(/*with_stacktrace=*/false)); + else + tryLogCurrentException(log, fmt::format("Network error on S3 request, attempt {} of {}", attempt_no, max_attempts)); outcome = Aws::Client::AWSError( Aws::Client::CoreErrors::NETWORK_CONNECTION, diff --git a/src/IO/S3/Client.h b/src/IO/S3/Client.h index a3d43c80a2b6..a6cb4972fa97 100644 --- a/src/IO/S3/Client.h +++ b/src/IO/S3/Client.h @@ -249,6 +249,10 @@ class Client : private Aws::S3::S3Client return client_configuration.for_disk_s3; } + /// True when this client's one and only attempt is not the final answer: it belongs to an + /// outer retry loop (e.g. a conditional write) that resolves the outcome and reissues. + bool usesSingleAttemptRetryStrategy() const; + ProviderType getProviderType() const { return provider_type; } std::string getGCSOAuthToken() const; diff --git a/src/IO/S3/tests/gtest_aws_s3_client.cpp b/src/IO/S3/tests/gtest_aws_s3_client.cpp index e9b6b901f37f..09ed0bd6692f 100644 --- a/src/IO/S3/tests/gtest_aws_s3_client.cpp +++ b/src/IO/S3/tests/gtest_aws_s3_client.cpp @@ -17,7 +17,10 @@ #include +#include +#include #include +#include #include #include @@ -29,6 +32,7 @@ #include #include +#include #include #include #include @@ -55,6 +59,7 @@ namespace DB::S3RequestSetting namespace ProfileEvents { extern const Event S3SingleAttemptRetryConsultations; + extern const Event S3WriteRequestsErrors; } /* @@ -258,6 +263,180 @@ TEST(IOTestAwsS3Client, SingleAttemptRetryStrategyRefusesAndCounts) EXPECT_EQ(global_counters[ProfileEvents::S3SingleAttemptRetryConsultations].load() - before, 2u); } +namespace +{ + +/// Captures what the `S3Client` logger (`Client::log`) writes at ERROR and above. A message logged +/// below Error (e.g. Debug) never reaches the channel at this threshold, so an empty capture proves +/// the site logged below Error rather than merely that this particular text was absent. +class ScopedS3ClientErrorLogCapture +{ +public: + ScopedS3ClientErrorLogCapture() + : logger(getLogger("S3Client")) + , channel(new Poco::StreamChannel(stream)) + , old_channel(logger->getChannel(), /*shared=*/true) + , old_level(logger->getLevel()) + { + logger->setChannel(channel.get()); + logger->setLevel("error"); + } + + ~ScopedS3ClientErrorLogCapture() + { + logger->setChannel(old_channel); + logger->setLevel(old_level); + } + + std::string captured() const { return stream.str(); } + +private: + LoggerPtr logger; + std::ostringstream stream; + Poco::AutoPtr channel; + /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. + Poco::AutoPtr old_channel; + int old_level; +}; + +/// A `Client` whose `PutObject` always fails as though the connection dropped while the response body +/// was being read -- the scenario `Client::doRequestWithRetryNetworkErrors`'s `net_exception_handler` +/// exists for (the comment on that function: "network error happens when XML document is being read +/// from the response body"). Throwing here, through the same virtual `Aws::S3::S3Client::PutObject` +/// slot `Client::PutObject`'s retry loop calls, reaches `net_exception_handler` exactly as a genuine +/// mid-body network failure would, without adding a test seam to production code -- the protected +/// `Client` constructor is already exposed "for testing" (see `RecordingClient` above). +class NetworkFailingClient : public DB::S3::Client +{ +public: + NetworkFailingClient( + size_t max_redirects_, + DB::S3::ServerSideEncryptionKMSConfig sse_kms_config_, + const std::shared_ptr & credentials_provider_, + const DB::S3::PocoHTTPClientConfiguration & client_configuration_, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy sign_payloads_, + const DB::S3::ClientSettings & client_settings_) + : DB::S3::Client(max_redirects_, std::move(sse_kms_config_), credentials_provider_, client_configuration_, sign_payloads_, client_settings_) + { + } + + Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest &) const override + { + ++attempts; + throw Poco::TimeoutException("mock timeout reading the response body"); + } + + mutable size_t attempts = 0; +}; + +std::shared_ptr makeNetworkFailingClient(std::shared_ptr retry_strategy) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration client_configuration = DB::S3::ClientFactory::instance().createClientConfiguration( + /*force_region=*/"us-east-1", + remote_host_filter, + /*s3_max_redirects=*/100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /*s3_slow_all_threads_after_network_error=*/false, + /*s3_slow_all_threads_after_retryable_error=*/false, + /*enable_s3_requests_logging=*/false, + /*for_disk_s3=*/false, + /*opt_disk_name=*/{}, + /*request_throttler=*/{}); + /// `PutObject` never reaches the wire (it is overridden below), so the endpoint is irrelevant -- + /// only the installed retry strategy, which is what `usesSingleAttemptRetryStrategy` inspects. + client_configuration.retryStrategy = std::move(retry_strategy); + + DB::S3::ClientSettings client_settings{ + .use_virtual_addressing = true, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }; + + Aws::Auth::AWSCredentials credentials("ACCESS_KEY_ID", "SECRET_ACCESS_KEY"); + auto credentials_provider = DB::S3::getCredentialsProvider( + client_configuration, + credentials, + DB::S3::CredentialsConfiguration{.use_environment_credentials = false, .use_insecure_imds_request = false}); + + return std::make_shared( + /*max_redirects_=*/100, + DB::S3::ServerSideEncryptionKMSConfig{}, + credentials_provider, + client_configuration, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + client_settings); +} + +} + +/// The client's own retry strategy still says "do not retry" (`SingleAttemptRetryStrategy::ShouldRetry` +/// always false, see `SingleAttemptRetryStrategyRefusesAndCounts` above); `usesSingleAttemptRetryStrategy` +/// is a separate, purely descriptive check of which strategy is installed, tested directly here. +TEST(IOTestAwsS3Client, UsesSingleAttemptRetryStrategyIdentifiesTheInstalledStrategy) +{ + auto single_attempt_client = makeNetworkFailingClient(std::make_shared()); + EXPECT_TRUE(single_attempt_client->usesSingleAttemptRetryStrategy()); + + DB::S3::PocoHTTPClientConfiguration::RetryStrategy zero_retries{.max_retries = 0}; + auto ordinary_client = makeNetworkFailingClient(std::make_shared(zero_retries)); + EXPECT_FALSE(ordinary_client->usesSingleAttemptRetryStrategy()); +} + +/// A client carrying the `SingleAttemptRetryStrategy` (the CAS conditional-write client, see +/// `S3ObjectStorage::getSingleAttemptClient`) is owned by an outer retry loop that resolves the outcome +/// and reissues; its one failed attempt is not terminal, so the network-error log site must not reach +/// Error. +TEST(IOTestAwsS3Client, NetworkErrorLogsDebugForSingleAttemptStrategy) +{ + using ProfileEvents::global_counters; + const auto errors_before = global_counters[ProfileEvents::S3WriteRequestsErrors].load(); + + auto client = makeNetworkFailingClient(std::make_shared()); + DB::S3::PutObjectRequest request; + + /// Call through the `DB::S3::Client&` interface, exactly as production code (which only ever + /// holds a `Client`, never `NetworkFailingClient`) does: `NetworkFailingClient::PutObject` hides + /// `Client::PutObject(PutObjectRequest&)` -- the retry-loop wrapper under test -- from lookup on + /// the derived type, so calling through the base is what makes this test exercise that wrapper + /// rather than the override directly. + const DB::S3::Client & base_client = *client; + ScopedS3ClientErrorLogCapture log_capture; + const auto outcome = base_client.PutObject(request); + + EXPECT_FALSE(outcome.IsSuccess()); + EXPECT_EQ(outcome.GetError().GetErrorType(), Aws::S3::S3Errors::NETWORK_CONNECTION); + EXPECT_EQ(client->attempts, 1u); + EXPECT_EQ(global_counters[ProfileEvents::S3WriteRequestsErrors].load() - errors_before, 1u); + EXPECT_TRUE(log_capture.captured().empty()); +} + +/// `max_retries = 0` on the ORDINARY strategy is a supported user configuration (`s3_retry_attempts`) +/// with no outer retry loop: its one failed attempt IS the final answer, so it must keep logging at +/// Error -- this is exactly the case a signal keyed on `max_retries == 0` alone would misclassify. +TEST(IOTestAwsS3Client, NetworkErrorLogsErrorForOrdinaryZeroRetryStrategy) +{ + using ProfileEvents::global_counters; + const auto errors_before = global_counters[ProfileEvents::S3WriteRequestsErrors].load(); + + DB::S3::PocoHTTPClientConfiguration::RetryStrategy zero_retries{.max_retries = 0}; + auto client = makeNetworkFailingClient(std::make_shared(zero_retries)); + DB::S3::PutObjectRequest request; + + /// See the comment in `NetworkErrorLogsDebugForSingleAttemptStrategy`: calling through the base + /// is what reaches `Client::PutObject`'s retry-loop wrapper rather than the override directly. + const DB::S3::Client & base_client = *client; + ScopedS3ClientErrorLogCapture log_capture; + const auto outcome = base_client.PutObject(request); + + EXPECT_FALSE(outcome.IsSuccess()); + EXPECT_EQ(outcome.GetError().GetErrorType(), Aws::S3::S3Errors::NETWORK_CONNECTION); + EXPECT_EQ(client->attempts, 1u); + EXPECT_EQ(global_counters[ProfileEvents::S3WriteRequestsErrors].load() - errors_before, 1u); + EXPECT_NE(log_capture.captured().find("Network error on S3 request, attempt 1 of 1"), std::string::npos); +} + struct ConditionalPutWireObservation { bool negotiated_expect_continue = false; diff --git a/src/IO/WriteBufferFromS3.cpp b/src/IO/WriteBufferFromS3.cpp index d05c225238d1..662204851290 100644 --- a/src/IO/WriteBufferFromS3.cpp +++ b/src/IO/WriteBufferFromS3.cpp @@ -810,9 +810,15 @@ void WriteBufferFromS3::makeSinglepartUpload(WriteBufferFromS3::PartData && data else { /// PreconditionFailed is an expected response for conditional writes (e.g. If-None-Match: *), - /// not a genuine error — the caller handles it (see `S3::isPreconditionFailedError`). + /// not a genuine error — the caller handles it (see `S3::isPreconditionFailedError`), and it + /// says nothing to the operator. if (S3::isPreconditionFailedError(outcome.GetError())) - LOG_INFO(log, "S3Exception name {}, Message: {}, bucket {}, key {}, object size {}", + LOG_DEBUG(log, "S3Exception name {}, Message: {}, bucket {}, key {}, object size {}", + outcome.GetError().GetExceptionName(), outcome.GetError().GetMessage(), bucket, key, content_length); + /// A SingleAttempt write is owned by an outer retry loop that resolves the outcome and + /// reissues; its one failed attempt is not terminal, so it is not an error. + else if (write_settings.object_storage_retry_profile == ObjectStorageRetryProfile::SingleAttempt) + LOG_DEBUG(log, "S3Exception name {}, Message: {}, bucket {}, key {}, object size {}", outcome.GetError().GetExceptionName(), outcome.GetError().GetMessage(), bucket, key, content_length); else LOG_ERROR(log, "S3Exception name {}, Message: {}, bucket {}, key {}, object size {}", diff --git a/src/IO/tests/gtest_writebuffer_s3.cpp b/src/IO/tests/gtest_writebuffer_s3.cpp index 138f96ed7285..ac3453a16c32 100644 --- a/src/IO/tests/gtest_writebuffer_s3.cpp +++ b/src/IO/tests/gtest_writebuffer_s3.cpp @@ -41,9 +41,12 @@ #include #include +#include #include #include +#include +#include #include @@ -552,6 +555,15 @@ struct PutObjectFailIngection: InjectionModel } }; +/// A conditional-write 412, matched by `S3::isPreconditionFailedError` on the canonical `` name. +struct PutObjectPreconditionFailedIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::PutObjectRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::UNKNOWN, "PreconditionFailed", "precondition failed", false); + } +}; + struct HeadObjectFailIngection: InjectionModel { std::optional call(const Aws::S3::Model::HeadObjectRequest & /*request*/) override @@ -676,6 +688,44 @@ struct SimpleAsyncTasks : BaseSyncPolicy using namespace DB; +namespace +{ + +/// Captures what `WriteBufferFromS3` logs at `threshold` and above (default: Error). A message +/// logged below the threshold never reaches the channel, so an empty capture proves the site logged +/// below it rather than merely that this particular text was absent. +class ScopedWriteBufferS3ErrorLogCapture +{ +public: + explicit ScopedWriteBufferS3ErrorLogCapture(const std::string & threshold = "error") + : logger(getLogger("WriteBufferFromS3")) + , channel(new Poco::StreamChannel(stream)) + , old_channel(logger->getChannel(), /*shared=*/true) + , old_level(logger->getLevel()) + { + logger->setChannel(channel.get()); + logger->setLevel(threshold); + } + + ~ScopedWriteBufferS3ErrorLogCapture() + { + logger->setChannel(old_channel); + logger->setLevel(old_level); + } + + std::string captured() const { return stream.str(); } + +private: + LoggerPtr logger; + std::ostringstream stream; + Poco::AutoPtr channel; + /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. + Poco::AutoPtr old_channel; + int old_level; +}; + +} + static void writeAsOneBlock(WriteBuffer& buf, size_t size) { std::vector data(size, 'a'); @@ -906,6 +956,71 @@ TEST_P(SyncAsync, ExceptionOnPut) { } +/// A non-412 `PutObject` failure on the ordinary (Default) retry profile is a genuine error: the +/// client's one attempt IS the final answer, so the site logs it at Error. +TEST_P(SyncAsync, PutObjectErrorLogsErrorForDefaultProfile) +{ + setInjectionModel(std::make_shared()); + + ScopedWriteBufferS3ErrorLogCapture log_capture; + EXPECT_THROW({ + auto buffer = getWriteBuffer("put_object_error_default_profile"); + buffer->write('A'); + buffer->next(); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + }, DB::S3Exception); + + EXPECT_THAT(log_capture.captured(), testing::HasSubstr("S3Exception name FailInjection")); + EXPECT_THAT(log_capture.captured(), testing::HasSubstr("PutObjectFailIngection")); +} + +/// The same failure on the SingleAttempt profile (the CAS conditional-write client) is owned by an +/// outer retry loop that resolves the outcome and reissues; the one failed attempt is not terminal, +/// so nothing here reaches Error. +TEST_P(SyncAsync, PutObjectErrorLogsDebugForSingleAttemptProfile) +{ + setInjectionModel(std::make_shared()); + + WriteSettings write_settings; + write_settings.object_storage_retry_profile = ObjectStorageRetryProfile::SingleAttempt; + + ScopedWriteBufferS3ErrorLogCapture log_capture; + EXPECT_THROW({ + auto buffer = getWriteBuffer("put_object_error_single_attempt_profile", write_settings); + buffer->write('A'); + buffer->next(); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + }, DB::S3Exception); + + EXPECT_TRUE(log_capture.captured().empty()); +} + +/// A conditional write losing its precondition (412) is the caller's expected answer, handled one +/// frame up -- it says nothing to the operator, so it must stay below Information, independent of the +/// retry profile. The capture threshold is Information so that an Info-level line from the site would +/// be caught; the cancel path logs its own Info lines, so the assertion is on the site's text, not on +/// an empty capture. +TEST_P(SyncAsync, PreconditionFailedNeverLogsAtError) +{ + setInjectionModel(std::make_shared()); + + ScopedWriteBufferS3ErrorLogCapture log_capture("information"); + EXPECT_THROW({ + auto buffer = getWriteBuffer("put_object_precondition_failed"); + buffer->write('A'); + buffer->next(); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + }, DB::S3Exception); + + EXPECT_THAT(log_capture.captured(), testing::Not(testing::HasSubstr("S3Exception name"))); +} + TEST_P(SyncAsync, ExceptionOnCreateMPU) { setInjectionModel(std::make_shared()); From 67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 01:51:53 +0200 Subject: [PATCH 25/81] cas: fix a genuine race and a fatal-abort hazard in the chunked-flush gtests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `SnapshotPublisherLatchedAcrossChunks` had two independent, reachable races. With `snapshot_log_count_threshold` at 0 (only reachable in this test), `precommitAdd`'s post-commit trigger dispatches a background publisher whose capture can still be in flight when `promote`, moments later, becomes lane leader and moves the lane to `Writing` — a lost race then backs the publisher off, and this pool's frozen `boot_ms_fn` never advances past that backoff deadline, poisoning every later dispatch on the namespace for the rest of the test. Fixed by driving `precommitAdd`/`promote` directly instead of through the shared `publishEmptyPart` helper, draining and explicitly publishing between the two commits. Separately, the carve hook gated the leader on the publisher reaching its blocked `PUT`, but under contention the dispatch's own scheduling delay could outlast that wait's bound, letting a leader released by timeout (not by the capture it meant to prove) start chunk 2 before the publisher captured — fixed by gating on the publisher's capture instead, which is causally prior to the `PUT`. Verified with 20 isolated `gtest_repeat` iterations and two full `CAS*` gates (2437/2437 each), reproduced only under CPU contention after isolated repeats alone did not reproduce it. Separately: a fatal `ASSERT_*` between launching an `AppendCaller`/`Caller` thread and its explicit `join()` left `TestBody` with the thread still joinable, and `std::thread::~thread()` on a joinable thread calls `std::terminate`, aborting the whole `unit_tests_dbms` binary and discarding every test scheduled after it. Both structs now join in their destructor if still joinable, so a failed assertion costs one test instead of the whole gate. Also: `CASDetachedWork` now stops and drains its detached publisher before the locals its hooks read go out of scope (ASan stack-use-after-return on `fake_boot` via `boot_ms_fn`), the same class of bug as the earlier test-hook lifetime fixes. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_detached_work.cpp | 7 ++ .../tests/gtest_cas_ref_chunked_flush.cpp | 74 +++++++++++++++++-- 2 files changed, 75 insertions(+), 6 deletions(-) diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index 056cfde61669..a8cfb5e1fb7e 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -653,6 +653,9 @@ TEST(CASDetachedWork, SettlementSurvivesAThrowingErrorHandler) EXPECT_TRUE(handler_ran.load()) << "the injected throw must have reached the (throwing) error handler"; EXPECT_EQ(store->pendingSnapshotPublishesForTest(ns), 0); + /// Same lifetime rule as below: the detached publisher reads the hooks' captured locals, so it is + /// stopped and drained before they go out of scope. + ASSERT_TRUE(store->stopAndDrainDetachedWork(/*deadline_ms=*/10000)); store->setSnapshotAfterCaptureHookForTest(nullptr); } @@ -738,6 +741,10 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) << "no elapsed backoff ever admitted a further publish attempt: the gate is a wedge, not a pace"; EXPECT_EQ(error_hook_calls.load(), attempts.load()); + /// The publisher is detached work: a redispatch admitted by the last elapsed backoff can still be + /// running when this body returns, and it reads `fake_boot` through `boot_ms_fn`. Stop and drain it + /// while the locals it reads are alive. + ASSERT_TRUE(store->stopAndDrainDetachedWork(/*deadline_ms=*/10000)); store->setSnapshotAfterCaptureHookForTest(nullptr); } diff --git a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp index cbfe52b9e268..a79f58fdf424 100644 --- a/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp +++ b/src/Disks/tests/gtest_cas_ref_chunked_flush.cpp @@ -115,6 +115,12 @@ struct Caller { std::thread t; std::future fut; + + /// A fatal `ASSERT_*` between launch and the explicit `t.join()` below returns from `TestBody` with + /// `t` still joinable; `std::thread::~thread` on a joinable thread calls `std::terminate`, aborting + /// the whole binary and discarding every later test. This destructor is the backstop: on every + /// success path the explicit join already ran and left nothing for it to do. + ~Caller() { if (t.joinable()) t.join(); } }; Caller launchAppend(const PoolPtr & store, const RootNamespace & ns, MutationScope scope, @@ -583,6 +589,10 @@ struct AppendCaller { std::thread t; std::future fut; + + /// Same hazard as `Caller` above (see its destructor comment): a fatal `ASSERT_*` before the + /// explicit join leaves `t` joinable, and a joinable thread's destructor calls `std::terminate`. + ~AppendCaller() { if (t.joinable()) t.join(); } }; AppendCaller launchAppendOps(const PoolPtr & store, const RootNamespace & ns, MutationScope scope, @@ -1022,18 +1032,63 @@ TEST(CASRefWriterChunkedFlush, SnapshotPublisherLatchedAcrossChunks) auto store = openPoolWith(backend, cfg); const DB::Cas::Layout & layout = store->layout(); const RootNamespace ns{"srv1/chunk_snapshot_coalesce"}; - publishEmptyPart(store, ns, "seed"); + /// NOT `publishEmptyPart`: that helper makes `precommitAdd` (which folds an implicit + /// `namespace_birth` and the add into ONE transaction, since the namespace is not yet `Live`) and + /// `promote` two separate, back-to-back `appendRefOps` calls. `precommitAdd`'s own post-commit + /// trigger (`maybeScheduleSnapshotPublish` at the end of `commitRefChunk`) dispatches a background + /// publisher for what it just committed; with `snapshot_log_count_threshold` at 0 (every single + /// commit is eligible -- never reachable through a normal, 256-count threshold) that publisher can + /// still be in flight when `promote`, moments later, becomes lane leader for its OWN write and + /// moves the lane to `Writing` -- a real, reachable race between that capture and this transition. + /// A lost race backs the publisher off, and this pool's frozen `boot_ms_fn` (see `openPool`'s + /// comment) never advances past that deadline, so the backoff never clears on its own, poisoning + /// every later dispatch on `ns` for the rest of the test, including chunk 1's. Draining + /// (`waitForSnapshotPublishSettleForTest`) between the two commits removes the in-flight publisher + /// `promote` would otherwise race, and driving one explicitly + /// (`tryPublishSnapshotAndAdvanceCheckpointOnce`, the direct synchronous seam built for exactly this + /// -- "public so tests can drive one attempt deterministically without depending on the background + /// dispatch's timing") covers anything a dispatch was never even admitted for. + DB::Cas::tests::casAdmitRecoverableEntry(*store->poolBackendPtr(), store->layout(), ns, store->liveWriterEpoch()); + PartWriteInfo seed_info; + seed_info.intended_namespace = ns; + seed_info.intended_ref = ns.string() + "/seed"; + auto seed_build = store->beginPartWrite(seed_info); + const ManifestId seed_manifest_id = seed_build->stageManifest({}); + seed_build->precommitAdd(ns, "seed", seed_manifest_id); + store->waitForSnapshotPublishSettleForTest(ns); + store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns); + seed_build->promote(ns, "seed", seed_build->buildId(), seed_manifest_id); store->waitForSnapshotPublishSettleForTest(ns); /// drain the seed's publish chain -> tail == 0 + store->tryPublishSnapshotAndAdvanceCheckpointOnce(ns); /// Latch the FIRST `_snap/` PUT (chunk 1's publisher) at its conditional PUT -- i.e. AFTER it has /// captured chunk 1's prefix under state_mutex. backend->armBlock(layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_snap/"); - /// Gate the leader at the chunk boundary until that publisher has parked on its PUT, so its captured - /// candidate is EXACTLY chunk 1's prefix (not chunk 1 + chunk 2). - store->setCarveHookForTest([backend](CasRefLedger::CarvePhaseForTest ph) + /// Gate the leader at the chunk boundary on the publisher's CAPTURE (`snapshot_after_capture_hook_for_test`, + /// fired the moment the publish attempt has read `rt->state` under `state_mutex` and passed the + /// `lane_state == Ready` admission check), not on the publisher's later blocked PUT. Capture is + /// causally prior to the PUT the block intercepts -- it is the OTHER side of the very check + /// (`CasRefLedger.cpp`'s `tryPublishSnapshotAndAdvanceCheckpointOnceOnRuntimeImpl`) whose failure logs + /// "refusing snapshot publication while the append lane is not Ready" -- so waiting for it is the + /// exact fact the boundary needs, whereas waiting for the PUT also waits on however long it takes the + /// dispatched publish to be scheduled onto a worker thread at all. Under contention that scheduling + /// delay can outlast a bounded wait, and a leader released by the wait's own timeout (rather than by + /// the capture it was meant to prove) can start chunk 2 -- moving the lane to `Writing` -- before the + /// not-yet-scheduled publisher ever captures, so it captures Writing and is refused. + auto captured = std::make_shared(); + store->setSnapshotAfterCaptureHookForTest([captured] { - if (ph == CasRefLedger::CarvePhaseForTest::ChunkReseed) - backend->awaitBlockEntered(); + std::lock_guard lk(captured->m); + captured->entered = true; + captured->cv.notify_all(); + }); + store->setCarveHookForTest([captured](CasRefLedger::CarvePhaseForTest ph) + { + if (ph != CasRefLedger::CarvePhaseForTest::ChunkReseed) + return; + std::unique_lock lk(captured->m); + captured->cv.wait_for(lk, std::chrono::seconds(10), [&] { return captured->entered; }); + ASSERT_TRUE(captured->entered) << "chunk 1's snapshot publisher never captured its candidate within 10s"; }); auto sync = std::make_shared(); @@ -1057,11 +1112,18 @@ TEST(CASRefWriterChunkedFlush, SnapshotPublisherLatchedAcrossChunks) ++chunk2_id.ref_sequence; EXPECT_EQ(rb.id, chunk2_id); + /// Independently confirm the latched publisher actually reached its blocked PUT, on the main test + /// thread rather than as the leader's own gate: without this, a wiring regression that never parks + /// the publisher would let the settlement assertion below pass VACUOUSLY (a direct, non-coalesced + /// dispatch can still cover chunk 2). + backend->awaitBlockEntered(); + /// Release the latched chunk-1 publisher. Its settlement must re-fire the chunk-2 trigger the /// single-flight gate dropped -> a follow-up publication covers chunk 2. backend->releaseBlock(); store->waitForSnapshotPublishSettleForTest(ns); store->setCarveHookForTest(nullptr); + store->setSnapshotAfterCaptureHookForTest(nullptr); store->setRefPreCarveHookForTest(nullptr); const std::optional newest = store->newestPublishedSnapshotIdForTest(ns); From 249c994a6466b8db661dbb29edeb2dc721f57474 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 05:32:10 +0200 Subject: [PATCH 26/81] cas: guard the fault state of a gtest backend that GC reads from a pool thread `CASGCRetire.OutcomeLogUnobservedConflictDoesNotReportItVanished` let its `UnobservedOutcomesBackend` flip `arm` and `refused_key` from `write` on the test thread while `read` consulted them from a `GcMetaWriter` pool thread (`scheduleConfirmedMetaDelete` -> `loadMeta`). TSan reported the race in `Unit tests (tsan)`; the fault state now lives under its own mutex. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 1b2fd76ad9b98dae195ee589b67d528ee5db05df) Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_gc_ack_floor.cpp | 39 ++++++++++++++++------ 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp index b89fa32d5526..84dfe33f1400 100644 --- a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp +++ b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp @@ -1,6 +1,7 @@ #include #include +#include #include #include #include @@ -1401,6 +1402,8 @@ TEST(CASGCCondemnMarker, LoadMetaFallbackConfirmsGraduationAfterLeaderRestart) /// resolve read settle nothing rather than be reissued exists only for S3 errors. TEST(CASGCRetire, OutcomeLogUnobservedConflictDoesNotReportItVanished) { + /// The fault's state is shared between the round (writes, on the test thread) and the meta + /// writer's pool (reads), so it lives under its own mutex. class UnobservedOutcomesBackend : public InMemoryBackend { public: @@ -1408,28 +1411,42 @@ TEST(CASGCRetire, OutcomeLogUnobservedConflictDoesNotReportItVanished) const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { - if (arm && !expected_value && key.find("/outcomes/") != String::npos) { - arm = false; - refused_key = key; - return std::unexpected(RawConflict{}); + std::lock_guard lock(fault_mutex); + if (arm && !expected_value && key.find("/outcomes/") != String::npos) + { + arm = false; + refused_key = key; + return std::unexpected(RawConflict{}); + } } return InMemoryBackend::write(key, bytes, expected_value, access); } std::optional read(const String & key, TransportAccess & access) override { - if (!refused_key.empty() && key == refused_key) { - refused_key.clear(); - throw DB::S3Exception("UnobservedOutcomesBackend: the settling read is definitively refused", - Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); + std::lock_guard lock(fault_mutex); + if (!refused_key.empty() && key == refused_key) + { + refused_key.clear(); + throw DB::S3Exception("UnobservedOutcomesBackend: the settling read is definitively refused", + Aws::S3::S3Errors::UNKNOWN, "MalformedXML"); + } } return InMemoryBackend::read(key, access); } - bool arm = false; - String refused_key; + void armOnce() + { + std::lock_guard lock(fault_mutex); + arm = true; + } + + private: + std::mutex fault_mutex; + bool arm TSA_GUARDED_BY(fault_mutex) = false; + String refused_key TSA_GUARDED_BY(fault_mutex); }; auto backend = std::make_shared(); @@ -1445,7 +1462,7 @@ TEST(CASGCRetire, OutcomeLogUnobservedConflictDoesNotReportItVanished) /// The condemn -> graduate -> delete pipeline needs several rounds before any round has an outcome /// to log; the arm fires on the first one that does. - backend->arm = true; + backend->armOnce(); bool refusal_reached = false; for (int i = 0; i < 8 && !refusal_reached; ++i) { From 45325aabd2da1e05a3fc41141c7ff7fad56fd078 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 05:32:10 +0200 Subject: [PATCH 27/81] cas: a resurrect compare-swaps the marker it already read instead of reading it again Since the request-engine migration `reconcileMetaClean` routed the `Condemned` case through `readModifyWrite`, which read the `.meta` marker a second time although `loadMeta` a few lines earlier had already returned its bytes and its `Etag`. That cost one extra GET per resurrect and broke the request budget `test_blob_publication_request_budget_and_default_mode` guards. The lambda now takes the loaded marker and issues `replace` under the incarnation that read observed, falling through to the read-decide-write loop only when that precondition is refused -- the same shape the `Absent` case's `create` already has. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 9287f20668cc62875824f2bf4ce47ce17dca61cb) Signed-off-by: Mikhail Filimonov --- .../ContentAddressed/Pool/CasPartWriteTxn.cpp | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp index c42684b72339..516debe79ef7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPartWriteTxn.cpp @@ -330,10 +330,12 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) /// Bring the freshness marker to `Clean`. A publication that followed an ABSENT observation has /// nothing at the marker key to decide from, so its create IS the whole reconciliation; routing it /// through a read-decide-write would spend a GET on every insert to learn what the create settles - /// for itself. Only a create that loses -- a racing writer's marker, or the stale `Condemned` one a - /// resurrect always finds -- needs the read, and there the engine's own loop is what bounds the + /// for itself. A resurrect already READ the stale `Condemned` marker before it published, and the + /// incarnation that read observed is the precondition its compare-swap needs -- so it spends no + /// second GET either. Only a write that loses -- a racing writer's marker, a marker that moved + /// under the resurrect -- needs the read, and there the engine's own loop is what bounds the /// retries at the policy's deadline instead of a fixed count of unpaced attempts. - auto reconcileMetaClean = [&](BlobPublicationReason reason) + auto reconcileMetaClean = [&](const std::optional & loaded, BlobPublicationReason reason) { if (reason == BlobPublicationReason::Absent) ProfileEvents::increment(ProfileEvents::CASMetaCreateClean); @@ -345,17 +347,23 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) "PartWriteTxn::ensureBlobPresent: reconciling the freshness metadata of '{}' to `Clean` " "after blob publication", key); + std::optional first; if (reason == BlobPublicationReason::Absent) { ProfileEvents::increment(ProfileEvents::CASMetaPut); - WriteResult created = op.create(meta_key, encodeBlobMeta(clean), policy); - /// Anything but a lost race is this call's answer, and `orThrow` maps it exactly as it maps - /// the read-decide-write's own result. - if (!std::holds_alternative(created)) - { - orThrow(std::move(created), what); - return; - } + first = op.create(meta_key, encodeBlobMeta(clean), policy); + } + else if (loaded) + { + ProfileEvents::increment(ProfileEvents::CASMetaCompareSwap); + first = op.replace(meta_key, encodeBlobMeta(clean), loaded->etag, policy); + } + /// Anything but a lost race is this call's answer, and `orThrow` maps it exactly as it maps + /// the read-decide-write's own result. + if (first && !std::holds_alternative(*first)) + { + orThrow(std::move(*first), what); + return; } orThrow( @@ -393,6 +401,7 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) op.pause(Retry::backoff(attempt)); const std::optional present = op.head(key, policy); BlobPublicationReason reason = BlobPublicationReason::Absent; + std::optional loaded; if (present) { @@ -412,7 +421,7 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) logical_size, source.size); - const std::optional loaded = loadMeta(op, store->layout(), ref, policy); + loaded = loadMeta(op, store->layout(), ref, policy); if (loaded) validateMetaSize(loaded->meta); @@ -511,7 +520,7 @@ BlobUploadResult PartWriteTxn::ensureBlobPresent(const BlobUploadRequest & req) continue; } - reconcileMetaClean(reason); + reconcileMetaClean(loaded, reason); /// A publication may land just as this mount loses its fence. The bytes are harmless debris, /// but they cannot become dependency proof for the fenced transaction. requireAdmitted("before the publication is recorded"); From 9abc940fa88955da7258dedbefea7e85a9819599 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 05:32:10 +0200 Subject: [PATCH 28/81] cas: bring two test_cas_gcs tests onto the request engine's contract `test_a_write_whose_response_carries_no_generation_is_refused` expected the INSERT to fail with "carried no valid generation". That refusal went away with the request-engine migration: `CasOperation::writeLoop` treats a 2xx whose value no grammar accepts as an ambiguous attempt and settles it with one exact body read, adopting the observed incarnation only when the bytes are its own. The test now asserts that contract: the INSERT succeeds, the `.meta` creates of the blobs it published are answered without a generation, each is followed by a GET (never a HEAD) of its own key, and `CASRequestResolveRead` grows accordingly. GC and the table's merges are stopped for the window so nothing else publishes markers into the captured slice. `test_marked_and_default_heads_coexist_on_one_oauth_client` relied on the sentinel probe issuing an ordinary HEAD; the probe now reads with a marked GET, so no default HEAD exists on a CAS bucket any more. It becomes a partition by GET on one disk: CAS control and marker reads are marked, blob-body reads are not, and every HEAD is marked. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 26a2f52b763808f8d12c8041b318c3484ae2bf02) Signed-off-by: Mikhail Filimonov --- .../test_cas_gcs/gcs_mocks/server.py | 5 +- tests/integration/test_cas_gcs/test.py | 237 +++++++++++++----- 2 files changed, 179 insertions(+), 63 deletions(-) diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 7dee09763c0f..457b9f03653c 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -53,8 +53,9 @@ if it did, and therefore how much of the fixture's safety depends on the service being strict. - ``omit_generation=1`` — a successful object-write ``PUT`` answers without ``x-goog-generation``. A real GCS always sends one; this models the response a proxy or a future API version might - return, which is the only input that can reach the "write succeeded but carried no valid - generation" refusal in the CAS write path. + return, which is the only input that leaves a successful conditional write UNATTRIBUTED to an + incarnation. The CAS request engine settles such a write with an exact GET of the key (see + ``CasOperation::writeLoop``) rather than refusing it or adopting whatever a HEAD reports. Usage: ``python3 server.py ``. Started by ``helpers.mock_servers.start_mock_servers``, which probes ``GET /`` and expects the body ``OK``. diff --git a/tests/integration/test_cas_gcs/test.py b/tests/integration/test_cas_gcs/test.py index e7571c86afb0..a3bcb62a5dfe 100644 --- a/tests/integration/test_cas_gcs/test.py +++ b/tests/integration/test_cas_gcs/test.py @@ -31,6 +31,7 @@ import os import re import runpy +import time import urllib.parse import pytest @@ -258,6 +259,35 @@ def _minted(): return _control("/_control/minted") +def _quiesce_merges(node, table): + """Stop background merges of `table` and wait for the running ones to finish. + + A merge publishes blobs and `.meta` markers of its own; a capture that must attribute every + marker create to one query has to keep merges out of the window. + """ + node.query("SYSTEM STOP MERGES {}".format(table)) + for _ in range(300): + running = int( + node.query( + "SELECT count() FROM system.merges WHERE table = '{}'".format(table) + ).strip() + ) + if running == 0: + return + time.sleep(0.1) + raise AssertionError("merges of {} did not drain".format(table)) + + +def _resolve_reads(node): + """The engine's own count of reads it issued to settle a refused or ambiguous write.""" + return int( + node.query( + "SELECT value FROM system.events WHERE event = 'CASRequestResolveRead'" + ).strip() + or 0 + ) + + def _next_seq(): """The `seq` the fake's next captured request will carry, so a later slice can start here.""" return len(_control("/_control/requests")) @@ -833,37 +863,75 @@ def test_ordinary_goog4_traffic_keeps_upstream_semantics(): assert all(not _is_translated(r) for r in multipart), multipart -def test_marked_and_default_heads_coexist_on_one_oauth_client(): - """Per-request marking, observed on ONE client, for ONE method, in ONE bucket. +def test_marked_and_default_gets_partition_one_oauth_disk(): + """Per-request marking, observed on ONE disk, for ONE method, in ONE bucket. - The CAS `gcp_oauth` disk owns a single S3 client and issues HEADs of both kinds: CAS metadata - reads are marked, while `probeSentinelRaw` deliberately goes through the ordinary throwing - `getObjectMetadata` because it must tell no-such-key from no-such-bucket from a transient failure, - and it discards the metadata anyway. Marking deletes `x-amz-api-version`, so the two kinds are - distinguishable on the wire even though they are the same verb on the same key space. + The CAS `gcp_oauth` disk issues GETs of both kinds: CAS reads of control objects and blob metadata + are marked, so GCS answers them with a generation, while a query reading a part's data goes + through the disk's ordinary read path and fetches the blob body unmarked. Marking deletes + `x-amz-api-version`, so the two kinds are distinguishable on the wire even though they are the + same verb on the same key space. (The two kinds need not share one `S3::Client`: a writable CAS + mount routes its control-plane reads through the single-attempt client, and the body read uses + the disk's default one. What is asserted is the partition by request kind, not by client.) - This is the assertion I earlier reported the fixture could not make. I was wrong for a specific - reason worth keeping: marking adds no header, which is true, but it REMOVES one, and an absence is - just as observable as a presence. + Marking adds no header, which is true, but it REMOVES one, and an absence is just as observable + as a presence. Every HEAD this disk issues is a CAS one (the mandatory HEAD before a blob PUT; the + sentinel probe reads with a GET so that a 404 carries a parseable body), so HEADs cannot carry the + partition and are asserted to be marked without exception. - Would fail if: every request were marked (the `Default` HEAD would lose the header) or none were - (all the marked HEADs would keep it). Both directions fire, which is what makes it a partition - rather than a one-sided check. + Would fail if: every request were marked (the body read would lose the header), none were (the + CAS reads would keep it), a body read were marked, a CAS read were not, or a HEAD went out + unmarked. Both directions fire, which is what makes it a partition rather than a one-sided check. Only the OAuth disk can support this. On `gcs_hmac`, `prepareGcsRequestForGoog4Authentication` runs for every request the client sends, so the header is absent regardless of mode and carries no information. """ - heads = [r for r in _captured(CAS_DISKS["cas_gcs_oauth"]) if r["method"] == "HEAD"] - assert heads, "no HEAD reached the fake, so this test would be vacuous" + node = cluster.instances["node"] + table = "t_cas_gcs_oauth" + bucket = CAS_DISKS["cas_gcs_oauth"] + + # GC streams its own run artifacts with ordinary (unmarked) reads, so a round that overlaps this + # slice would put unmarked GETs of control keys into it. The disk's GC is stopped for the window. + node.query("SYSTEM CAS GC STOP 'cas_gcs_oauth'") + try: + seq = _next_seq() + node.query( + "INSERT INTO {} SELECT number, toString(number) FROM numbers(40000, 50)".format(table) + ) + # The disk has no cache in front of it, so the part's column data is fetched from the store. + assert int(node.query("SELECT sum(length(data)) FROM {} WHERE id >= 40000".format(table))) > 0 + records = _captured_since(seq, bucket) + finally: + node.query("SYSTEM CAS GC START 'cas_gcs_oauth'") - default_heads = [r for r in heads if _looks_default_on_oauth(r)] - marked_heads = [r for r in heads if not _looks_default_on_oauth(r)] + gets = [r for r in records if r["method"] == "GET" and r["key"]] + assert gets, "no object GET reached the fake, so this test would be vacuous" + marked_gets = [r for r in gets if not _looks_default_on_oauth(r)] + default_gets = [r for r in gets if _looks_default_on_oauth(r)] + + assert marked_gets, "no GET was marked — CAS reads lost their request mode" + assert default_gets, ( + "every GET was marked — the ordinary body read was marked too, which is the whole-disk " + "marking regression this test guards against" + ) + # The partition is exact: a CAS read is marked and a body read is not, with no exceptions in + # either direction. + assert all(r["request_class"] in ("cas_control", "blob_meta") for r in marked_gets), ( + "a marked GET reached something other than a CAS control object or a blob marker: {}".format( + [r["key"] for r in marked_gets if r["request_class"] not in ("cas_control", "blob_meta")] + ) + ) + assert all(r["request_class"] == "blob_body" for r in default_gets), ( + "an unmarked GET reached something other than a blob body: {}".format( + [r["key"] for r in default_gets if r["request_class"] != "blob_body"] + ) + ) - assert marked_heads, "no HEAD was marked — CAS metadata reads lost their request mode" - assert default_heads, ( - "every HEAD was marked — the sentinel probe's ordinary metadata read was marked too, " - "which is the whole-client marking regression this plan removes" + heads = [r for r in records if r["method"] == "HEAD"] + assert heads, "the insert published no blob, so no HEAD preceded a PUT" + assert all(not _looks_default_on_oauth(r) for r in heads), ( + "a HEAD went out unmarked: {}".format([r["key"] for r in heads if _looks_default_on_oauth(r)]) ) @@ -1068,7 +1136,7 @@ def test_interleaved_ordinary_and_cas_operations_do_not_leak_mode_or_build_a_cli it) and the CAS disk's traffic must still be marked, in a slice of the log where the two disks' traffic is interleaved rather than separated by phase. The contribution here is the INTERLEAVING; that a single OAuth client carries both marked and unmarked requests is established separately by - `test_marked_and_default_heads_coexist_on_one_oauth_client`, which asserts both halves non-empty in + `test_marked_and_default_gets_partition_one_oauth_disk`, which asserts both halves non-empty in one bucket. This test asserts only the marked half, deliberately -- duplicating the partition would add a second place to keep in step and no new fencing power. Would fail if: the mode became a property of the client rather than of the request. @@ -1151,65 +1219,112 @@ def test_interleaved_ordinary_and_cas_operations_do_not_leak_mode_or_build_a_cli ) -def test_a_write_whose_response_carries_no_generation_is_refused(): - """The one input that can reach the "no valid generation" refusal. +def test_a_write_whose_response_carries_no_generation_is_settled_by_an_exact_read(): + """The one input that can reach the "unattributed write" path. A real GCS always answers a successful object write with `x-goog-generation`, and the response adapter turns that into the SDK's `ETag`. When it is absent the SDK sees the store's real ETag - instead, which is not a generation, so `tokenFromWriteResult` must refuse to attribute the write to - an incarnation rather than patching the missing token over with a fresh HEAD — a HEAD returns - whatever incarnation happens to be current, which on a lost race is somebody else's. - - The error text is the whole discriminator, and it is tight: had the code HEADed and adopted the - current incarnation instead of refusing, the INSERT would have SUCCEEDED. It failed, naming the - missing generation. So a regression that replaced the strict branch with a HEAD-and-adopt fallback - turns the error assertion red on its own. - - Do NOT add an assertion here about which requests follow that write. The remaining conditional - metadata/control lane may classify an unattributed attempt as unresolved and call - `resolveByExactGet`, while the globally enabled injection can be consumed by more than one object - kind. Blob-body publication is no longer part of this test: it is unconditional, consumes no - response generation, and therefore cannot be the source of this refusal. - - What fences the behaviour is the error text above, and nothing else here needs to. - - Would fail if: the strict Generation branch in `tokenFromWriteResult` were replaced by, or fell - back to, the ETag dialect's HEAD path. - - The mode is global while it is on, so a background CAS operation on the other disk can fail during - the window too. That is logged, not fatal, and the restored-mode INSERT at the end is what says - the disk is healthy again. + instead, which is not a generation, so the request engine cannot attribute the write to an + incarnation. The engine treats that as an AMBIGUOUS attempt: the write may well have landed, so it + settles the attempt with one exact read of the key and adopts the incarnation it observes only when + the bytes there are its own. It must not patch the missing token over with a HEAD — a HEAD returns + whatever incarnation happens to be current, which on a lost race is somebody else's — and it must + not refuse the write outright either: the object may be there. + + The discriminator is the request that follows the ungenerated PUT: a body `GET` of the same key, + never a `HEAD`. Had the engine adopted the current incarnation via HEAD, the read would be a HEAD; + had it refused, the INSERT would have FAILED. It succeeded, and every selected `.meta` create was + followed by a GET of its own key. + + The per-key assertion is made only on the `.meta` creates of blobs whose body this INSERT itself + published in the captured slice: those keys are touched by this query alone, and their settle read + has completed by the time the query returns. Control-plane keys (`_ckpt`, `_log`, manifests, the + mount lease) are shared with the renewer, and GC writes `.meta` markers of its own, so a request + on any other key cannot be causally tied to the PUT before it. GC is stopped on the disk and the + table's merges are stopped and drained for the window on top of that, so no other publisher of + blobs or markers interleaves in the bucket. The mode is global while it is on, so a CAS operation + of the other disk can be settled during the window too; the `CASRequestResolveRead` bound is `>=` + for that reason. The restored-mode INSERT at the end is what says the disk is healthy again. + + Would fail if: the engine adopted the current incarnation from a HEAD instead of proving its own + bytes with a GET, or if it refused an unattributed write instead of settling it. """ node = cluster.instances["node"] table = "t_cas_gcs_oauth" cas_bucket = CAS_DISKS["cas_gcs_oauth"] - first_new_seq = _next_seq() + node.query("SYSTEM CAS GC STOP 'cas_gcs_oauth'") + _quiesce_merges(node, table) try: - assert _set_omit_generation(True)["omit_generation"] is True - error = node.query_and_get_error( - "INSERT INTO {} SELECT number, toString(number) FROM numbers(20000, 50)".format(table) - ) + first_new_seq = _next_seq() + resolve_reads_before = _resolve_reads(node) + try: + assert _set_omit_generation(True)["omit_generation"] is True + node.query( + "INSERT INTO {} SELECT number, toString(number) FROM numbers(20000, 50)".format(table) + ) + finally: + assert _set_omit_generation(False)["omit_generation"] is False + captured = _captured_since(first_new_seq, cas_bucket) finally: - assert _set_omit_generation(False)["omit_generation"] is False + node.query("SYSTEM START MERGES {}".format(table)) + node.query("SYSTEM CAS GC START 'cas_gcs_oauth'") - assert "carried no valid generation" in error, error + assert int(node.query("SELECT count() FROM {} WHERE id >= 20000 AND id < 20050".format(table))) == 50 # Positive proof that the fake actually produced the condition under test: a successful object - # write really did answer without a generation. Without this the error assertion above could be - # satisfied by an INSERT that failed for some entirely unrelated reason, and a mode switch that - # silently stopped working would look like a pass. + # write really did answer without a generation. Without this the assertions below could be + # satisfied by an INSERT that never met the condition, and a mode switch that silently stopped + # working would look like a pass. ungenerated = [ r - for r in _captured_since(first_new_seq, cas_bucket) + for r in captured if r["method"] == "PUT" and r["status"] == 200 and r["response_generation"] is None ] assert ungenerated, "the mode was on but no successful PUT answered without a generation" - # Restoring the mode must restore the disk, or the failure above was something other than the - # missing generation. + # Only the conditional lane attributes a write to an incarnation, so only its PUTs have anything + # to settle. Blob-body publication is unconditional and consumes no response generation; an + # ungenerated answer to it is nothing the engine has to resolve. Of the conditional PUTs, only + # the `.meta` creates of blobs this INSERT published itself are query-owned, and only a + # query-owned key can be tied to its settle read by capture order alone. + published_bodies = {r["key"] for r in _blob_publications(captured)} + unattributed = [ + r + for r in ungenerated + if r["operation"] == "conditional_put" + and r["request_class"] == "blob_meta" + and r["headers"].get("x-goog-if-generation-match") == "0" + and r["key"][: -len(".meta")] in published_bodies + ] + assert unattributed, ( + "no `.meta` create of a freshly published blob answered without a generation, so nothing " + "query-owned was unattributed" + ) + + # Every unattributed write is settled by a body read of ITS OWN key -- a GET, not a HEAD. + for put in unattributed: + observations = [ + r + for r in captured + if r["seq"] > put["seq"] and r["key"] == put["key"] and r["method"] in ("GET", "HEAD") + ] + assert observations, "the unattributed PUT of {} was never read back".format(put["key"]) + settling = observations[0] + assert settling["method"] == "GET", ( + "the unattributed PUT of {} was settled by a {}, not by an exact GET".format( + put["key"], settling["method"] + ) + ) + + assert _resolve_reads(node) - resolve_reads_before >= len(unattributed), ( + "the engine did not account a resolve read for every unattributed write" + ) + + # Restoring the mode must restore the disk, or the success above was something other than the + # settled write. node.query("INSERT INTO {} SELECT number, toString(number) FROM numbers(30000, 50)".format(table)) - assert int(node.query("SELECT count() FROM {} WHERE id >= 30000".format(table))) == 50 + assert int(node.query("SELECT count() FROM {} WHERE id >= 30000 AND id < 30050".format(table))) == 50 # --------------------------------------------------------------------------------------------------- From 7c7c6bda734d7b9d99c3e15948a2b4324a29a29e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 06:59:08 +0200 Subject: [PATCH 29/81] cas: prove an existing pool by reading `_pool_meta`, not by enumerating its prefix The startup residual probe proved that a pool exists through a LIST walk of the pool prefix (page 1000). Since the request-engine migration that walk runs under the control-plane single-attempt budget, and a 1000-key page over a large prefix on a slow store exceeds it reliably: the `cas_alter_attach_2` regression suite restarted a server whose every LIST attempt timed out at 5 s, 30 attempts gave up at the deadline, the probe answered `Indeterminate` and startup refused a pool the server could read perfectly well. `_pool_meta` present is decisive by the probe's own contract, and one exact read of that key answers it without enumerating anything. The probe now reads it first and runs the residual LIST only when the key is absent or the read could not settle it -- which is exactly the case that needs the absence-of-residue proof. `PoolMeta::createOrValidate` still re-reads and validates the object on the `PoolMetaPresent` path. `CASListLiarEndToEnd.RecoveryUnderTheSameLieReconstructsExactlyTheTruth` relied on that bootstrap LIST to prove its lying store was consulted at all; recovery itself reads by exact key. The test now proves the omission with an explicit LIST of the stream before comparing recovery against the oracle. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit aee2a4b6880f8f87d1961db151ff77cdffc8425d) Signed-off-by: Mikhail Filimonov --- .../Backend/CasSentinelProbe.cpp | 25 +++++++++- .../Backend/CasSentinelProbe.h | 6 ++- .../tests/gtest_cas_bootstrap_ordering.cpp | 47 ++++++++++++++++++- .../tests/gtest_cas_list_liar_end_to_end.cpp | 14 ++++++ 4 files changed, 87 insertions(+), 5 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp index 229847ff1d76..df47c0ddf638 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp @@ -36,10 +36,31 @@ BootstrapResidual probePoolBootstrapResidual(CasOperation & op, const Layout & l const String prefix = layout.poolPrefix() + "/"; const String probe_root = layout.poolPrefix() + "/_probe/"; + /// `_pool_meta` present is decisive on its own, and one exact read of that key answers it without + /// enumerating anything: a LIST page over a large pool is the most expensive request a store + /// answers (it must enumerate and sort the prefix), and a store that is merely slow to LIST would + /// otherwise refuse to reopen a pool it can read perfectly well. So the existing-pool case is + /// settled by the read, and the LIST below runs only when the key is absent -- which is exactly + /// the case that needs the absence-of-residue proof. A read that could not settle presence (a + /// refusal, a deadline) is not a verdict; it falls through to the LIST, which may still see the + /// key. + try + { + if (op.read(pool_meta_key, Retry::standard())) + return BootstrapResidual::PoolMetaPresent; + } + catch (...) + { + LOG_WARNING(getLogger("CasBootstrap"), + "Pool prefix '{}': the exact read of '{}' could not settle whether the pool exists; " + "falling back to the residual LIST: {}", + prefix, pool_meta_key, getCurrentExceptionMessage(/*with_stacktrace=*/false)); + } + /// Classification is order-independent for correctness: every listed key is examined, and finding /// `_pool_meta` anywhere is decisive. It relies on lexicographic LIST order only for COST — `_pool_meta` - /// sorts first under `/`, so a healthy pool short-circuits on the first page rather than - /// enumerating its whole content on every open. + /// sorts first under `/`, so a pool whose exact read above was refused still short-circuits + /// on the first page rather than enumerating its whole content. bool has_pool_meta = false; bool has_residual = false; bool has_catalog = false; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h index 39a3ca700552..54862280726f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.h @@ -48,8 +48,10 @@ enum class BootstrapResidual : uint8_t Indeterminate, }; -/// Zero-write authoritative classification of a pool prefix for the startup bootstrap decision. A single -/// paginated LIST of `layout.poolPrefix()`; each listed object is classified as the `_pool_meta` +/// Zero-write authoritative classification of a pool prefix for the startup bootstrap decision. One exact +/// read of `_pool_meta` first: present is decisive, and it costs no enumeration, so an existing pool +/// reopens without a LIST at all. Only when the key is absent (or the read could not settle it) does a +/// single paginated LIST of `layout.poolPrefix()` run; each listed object is classified as the `_pool_meta` /// sentinel, capability-battery debris under the reserved `/_probe/` subtree (a crash-mid-battery /// leftover OR a concurrent fresh opener's in-flight battery — [D2]), or genuine residual CAS state. It /// NEVER writes, and it IGNORES probe debris exactly so a normal restart after a crash-mid-battery still diff --git a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp index 680217bcf3e1..650796a3a9dd 100644 --- a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp +++ b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp @@ -7,6 +7,7 @@ #include "cas_test_helpers.h" #include +#include #include #include #include @@ -24,6 +25,7 @@ namespace DB::ErrorCodes { extern const int INVALID_STATE; +extern const int NOT_IMPLEMENTED; } using namespace DB::Cas; @@ -44,7 +46,7 @@ const String kProbeUid2 = "fedcba9876543210fedcba9876543210"; /// /// The `write` primitive covers create, replace and conditional-put alike, so the log distinguishes /// only writes from removals -- which is all the ordering assertions ask. -class RecordingBackend final : public InMemoryBackend +class RecordingBackend : public InMemoryBackend { public: /// Unhide the legacy `list` overloads the primitive override below would otherwise hide: the tests @@ -355,6 +357,49 @@ TEST(CASBootstrapOrdering, HealthyPoolReopenPreservesIdentity) << "a healthy reopen must NOT re-mint _pool_meta — the pool identity must be preserved"; } +/// (d') An existing pool whose prefix the store cannot LIST at the moment (a large prefix on a store +/// that enumerates slowly, a LIST budget that expires) still reopens: `_pool_meta` present is proven by +/// ONE exact read, and the residual LIST is only the absent-key path. Before this, a pool that could be +/// read perfectly well refused to start because the enumeration that would have found the same key +/// did not return in time. +TEST(CASBootstrapOrdering, HealthyPoolReopensWhenThePrefixCannotBeListed) +{ + /// Refuses every LIST of the pool root once armed; everything else is the ordinary store. + class UnlistableRootBackend final : public RecordingBackend + { + public: + using RecordingBackend::list; + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + if (refuse_root_list && prefix == kPrefix + "/") + throw DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, + "UnlistableRootBackend: the pool root cannot be enumerated right now"); + return RecordingBackend::list(prefix, cursor, limit, access); + } + std::atomic refuse_root_list{false}; + }; + + auto backend = std::make_shared(); + + UInt128 pool_id_first; + { + PoolPtr store = Pool::open(backend, makeConfig()); + pool_id_first = store->poolMeta().pool_id; + } /// clean teardown: drained farewell, so the reopen reclaims immediately + + backend->refuse_root_list = true; + backend->clearLog(); + PoolPtr store2; + ASSERT_NO_THROW(store2 = Pool::open(backend, makeConfig())) + << "an existing pool must reopen on the exact read of _pool_meta alone"; + EXPECT_EQ(store2->lifecycle(), PoolLifecycle::Live); + EXPECT_EQ(store2->poolMeta().pool_id, pool_id_first); + const auto log = backend->snapshot(); + EXPECT_FALSE(firstIndex(log, [](const RecordingBackend::Entry & e) + { return e.op == RecordingBackend::Op::List && e.key == kPrefix + "/"; }).has_value()) + << "a pool whose _pool_meta was read must not be enumerated to prove it exists"; +} + /// (e) [D2] concurrent-opener case: debris from a SECOND concurrent fresh opener's in-flight battery (a /// distinct probe uid) is skipped by the SAME structural rule as (c). Two openers racing over one shared /// pool prefix must not make each other's zero-write residual check fail. diff --git a/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp b/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp index d40d30fd334d..368430755275 100644 --- a/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp +++ b/src/Disks/tests/gtest_cas_list_liar_end_to_end.cpp @@ -11,6 +11,7 @@ #include "cas_test_helpers.h" #include +#include #include #include @@ -260,6 +261,19 @@ TEST(CASListLiarEndToEnd, RecoveryUnderTheSameLieReconstructsExactlyTheTruth) auto lying = openRecoveryPool(lying_backend); const std::map recovered = refsOf(lying, ns); + /// Recovery reads every record by exact key and asks no listing what to read next, and an existing + /// pool reopens on the exact read of `_pool_meta` alone -- so nothing above ever LISTed the stream. + /// That is the point, but it also means the lie has to be PROVEN in effect here, or the comparison + /// below would pass against a store that hid nothing. + { + DB::Cas::tests::OperationForTest op(*lying_backend); + std::set listed; + (*op).forEachListedKey(layout.namespaceStreamPrefix(fixture::fixtureLife(ns)), + [&](const ListedKey & key) { listed.insert(key.key); return true; }, + Retry::standard()); + for (const String & hidden : hiddenMiddleOf(layout, ns)) + ASSERT_FALSE(listed.contains(hidden)) << "the store was told to hide " << hidden << " and did not"; + } ASSERT_GT(lying_backend->holesServed(), 0u) << "the omission was never actually served -- the test would pass vacuously"; EXPECT_EQ(truth.size(), 5u) << "the oracle itself must see all five published refs"; From f7259840cb22e55c93ea4079033559296dedb272 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 09:07:31 +0200 Subject: [PATCH 30/81] cas: the startup residual check reads one small page and stops at the first residual key The residual LIST at startup exists to tell whether the pool prefix holds anything besides battery debris. It walked the whole prefix on the residual verdict and asked the store for its default page (a thousand keys) to do so -- the one request a slow store cannot answer within an attempt, spent on keys the check never reads. The walk now stops at the first residual key, and the first page is asked for at most 32 keys. `ObjectStorageBackend::listUnder` passes the caller's page size to the store for the first page only, as `limit + 1`: not every storage pages, and the fallback iterator lists `max_keys` keys once and ends, so a page that ended exactly at the limit would be indistinguishable from the end of the prefix; the extra key is what proves there is more. A resumed page keeps the storage's own page size, because a storage that ignores `start_after` would answer a bounded resumed page with the first keys of the prefix again. Tests: forty residue keys cost exactly one LIST and a typed refusal; a page-sized list over a non-paging store still reports the keys past the page. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 23b67e74a851abd5bd8ced9950df7f5355d43176) Signed-off-by: Mikhail Filimonov --- .../Backend/CasObjectStorageBackend.cpp | 14 +++++++++++- .../Backend/CasSentinelProbe.cpp | 21 ++++++++++++------ .../tests/gtest_cas_bootstrap_ordering.cpp | 22 +++++++++++++++++++ src/Disks/tests/gtest_cas_s3_staging.cpp | 21 ++++++++++++++++++ 4 files changed, 70 insertions(+), 8 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index 32d02506bfdf..2c9440a8f4b7 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -1086,8 +1086,20 @@ Backend::RawListPage ObjectStorageBackend::listUnder( ? std::nullopt : std::optional(cursor); + /// The FIRST page the caller asked for is the page the store is asked for: a request that + /// enumerates the storage's default page (a thousand keys) to answer a caller that wanted a + /// handful spends the caller's whole attempt on keys it will never read -- and on a large prefix + /// that enumeration is exactly what a slow store cannot deliver within an attempt. One key MORE + /// than the page, because not every storage pages: the fallback iterator lists `max_keys` keys + /// once and ends, and a page that ends exactly at the limit would then be indistinguishable from + /// the end of the prefix -- the extra key is what proves there is more, whichever iterator + /// answers. A RESUMED page is not bounded: a storage that ignores `start_after` would answer it + /// with the first keys of the prefix again, and a bound there would drop every key past it. The + /// first page is the one the emptiness probe needs; a walk keeps the storage's own page size. + static constexpr size_t max_store_page = 1'000'000; + const size_t store_page = cursor.empty() ? std::min(limit, max_store_page) + 1 : 0; RawListPage page; - auto it = object_storage->iterate(physical_prefix, /*max_keys=*/0, /*with_tags=*/false, start_after, profile, timeout_ms); + auto it = object_storage->iterate(physical_prefix, /*max_keys=*/store_page, /*with_tags=*/false, start_after, profile, timeout_ms); for (; it->isValid(); it->next()) { const auto child = it->current(); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp index df47c0ddf638..d53812acac21 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasSentinelProbe.cpp @@ -57,10 +57,17 @@ BootstrapResidual probePoolBootstrapResidual(CasOperation & op, const Layout & l prefix, pool_meta_key, getCurrentExceptionMessage(/*with_stacktrace=*/false)); } - /// Classification is order-independent for correctness: every listed key is examined, and finding - /// `_pool_meta` anywhere is decisive. It relies on lexicographic LIST order only for COST — `_pool_meta` - /// sorts first under `/`, so a pool whose exact read above was refused still short-circuits - /// on the first page rather than enumerating its whole content. + /// The LIST answers one question: is there anything under the prefix besides the ignorable + /// battery debris (and, as the one retryable exception, the canonical empty catalog)? The first + /// residual key settles it, so the walk stops there. A probe-only or catalog-only prefix is walked + /// to completion -- debris is normally a few keys but every interrupted open leaves one more, so + /// it may span pages -- and the page is kept small because the enumeration cost of a large prefix + /// is what a slow store cannot deliver within an attempt. `_pool_meta` is still recognised if the + /// walk meets it (the exact read above may have been refused): it sorts before every key family + /// CAS itself writes under `/`, so on a lexicographic listing it is met before anything + /// that could have stopped the walk. Foreign residue that sorts earlier, or a listing that is not + /// lexicographic, can only make this refuse an existing pool, never bootstrap over one. + constexpr size_t page_limit = 32; bool has_pool_meta = false; bool has_residual = false; bool has_catalog = false; @@ -80,9 +87,9 @@ BootstrapResidual probePoolBootstrapResidual(CasOperation & op, const Layout & l has_catalog = true; return true; } - has_residual = true; /// a non-`_probe` object, and no `_pool_meta` seen (so far) - return true; - }, Retry::standard()); + has_residual = true; /// a non-`_probe` object, and no `_pool_meta` seen — decisive too + return false; + }, Retry::standard(), page_limit); } catch (...) { diff --git a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp index 650796a3a9dd..2afec62dbd07 100644 --- a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp +++ b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp @@ -265,6 +265,28 @@ TEST(CASBootstrapOrdering, ResidualWithoutMetaFailsTypedWithZeroWrites) EXPECT_FALSE(readPresent(*backend, kPoolMetaKey)) << "a fresh _pool_meta must NOT have been minted"; } +/// (b') The residual verdict is decided by the first residual key, not by an enumeration of the whole +/// prefix: forty residue keys and a 32-key page must cost exactly ONE list request. Enumerating a +/// large prefix is the one request a slow store cannot answer within an attempt, and a refusal +/// needs none of it. +TEST(CASBootstrapOrdering, ResidualWithoutMetaIsDecidedByTheFirstPage) +{ + auto backend = std::make_shared(); + for (uint64_t i = 1; i <= 40; ++i) + seedObject(*backend, Layout{"p"}.refLogKey(DB::Cas::tests::fixture::fixtureLife(RootNamespace{"test%2Fabcd"}), RefTxnId{1, i}), "x"); + backend->clearLog(); + + expectThrowsCodeContaining(DB::ErrorCodes::INVALID_STATE, "refusing to bootstrap over residual data", + [&] { Pool::open(backend, makeConfig()); }); + + size_t root_lists = 0; + for (const auto & e : backend->snapshot()) + if (e.op == RecordingBackend::Op::List && e.key == kPrefix + "/") + ++root_lists; + EXPECT_EQ(root_lists, 1u) << "the first residual key settles the verdict; nothing past it may be enumerated"; + EXPECT_EQ(backend->writeCount(), 0u); +} + /// (c) A prefix containing ONLY stale, structurally-valid `_probe//…` debris (a crash-mid-battery /// leftover) → treated as empty → open succeeds and bootstraps a fresh pool. The debris-skip is what makes /// a normal restart-after-crash recover instead of wedging. diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index a588fda71b36..3c4a9805e9c8 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -1296,6 +1296,27 @@ std::shared_ptr makeFakeGenerationObjectStorageForT } +/// A store whose iterator does not page (the fallback `IObjectStorage::iterate` lists `max_keys` keys +/// once and ends) must still let a page-sized list report that more keys follow. A page that ended +/// exactly at the limit with an empty cursor would read as the end of the prefix, and the startup +/// residual check would then take a prefix of debris plus residue for an empty one. +TEST(CASS3Staging, ListPageOverANonPagingStoreStillReportsMoreKeys) +{ + auto object_storage = makeFakeGenerationObjectStorageForTest(); + auto backend = std::make_shared(object_storage, DB::Cas::ObjectStorageBackend::Mode::Native); + for (int i = 0; i < 40; ++i) + createAt(*backend, fmt::format("p/list/{:03}", i), "x"); + + DB::Cas::tests::OperationForTest op(*backend); + const DB::Cas::ListPage first = (*op).list("p/list/", "", 32, DB::Cas::Retry::once()); + EXPECT_EQ(first.keys.size(), 32u); + ASSERT_FALSE(first.next_cursor.empty()) << "a full page over a non-paging store must still say there is more"; + + const DB::Cas::ListPage rest = (*op).list("p/list/", first.next_cursor, 32, DB::Cas::Retry::once()); + EXPECT_EQ(rest.keys.size(), 8u); + EXPECT_TRUE(rest.next_cursor.empty()); +} + TEST(CASS3Staging, GenerationBackendMayUseNativeOnlyCopy) { auto object_storage = makeFakeGenerationObjectStorageForTest(); From b39b1b514c6d10fe63462451742e7d9d86606b97 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Sat, 5 Sep 2026 09:18:08 +0200 Subject: [PATCH 31/81] cas: test_cas_mount_renewal_retry asserts the renewal's terminal event, not a per-attempt one Since the request-engine migration the reissues of a mount renewal are paced inside the engine's write loop and the mount logs only the renewal's outcome: the per-attempt `retrying` event is gone, and the read-settled classification is named `committed_by_read`. The test still waited for a `retrying` row next to `recovered` and expected `committed_by_get`, so both scenarios timed out on `system.cas_log` although the renewals had recovered. The attempt count on the terminal row is what proves a retry happened. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit 5adccd5009b929988ff0d1849162fea656a04cba) Signed-off-by: Mikhail Filimonov --- .../integration/test_cas_mount_renewal_retry/test.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/integration/test_cas_mount_renewal_retry/test.py b/tests/integration/test_cas_mount_renewal_retry/test.py index 31fa1529c0c3..8f9fecbc5856 100644 --- a/tests/integration/test_cas_mount_renewal_retry/test.py +++ b/tests/integration/test_cas_mount_renewal_retry/test.py @@ -205,10 +205,12 @@ def recovered_snapshot(): mount_body = _decode_mount(body_after) delta = _event_delta(counters_before, counters_after) sequence = mount_after["sequence"] + # The engine paces the reissues inside one renewal; what the log records is the renewal's + # outcome, and the attempt count on that row is what says a retry happened. rows = _wait_until( lambda: ( found - if {row[0] for row in found} >= {"retrying", "recovered"} + if any(row[0] == "recovered" for row in found) else None ) if (found := _renewal_log_rows(node, since, sequence)) @@ -232,10 +234,8 @@ def recovered_snapshot(): assert stats["by_mode"].get("503") == 1, stats print("targeted request count (transient renewal): {}".format(stats["faults"]), flush=True) - retrying = next(row for row in rows if row[0] == "retrying") recovered = next(row for row in rows if row[0] == "recovered") - assert retrying[1] == recovered[1] == str(sequence), rows - assert retrying[2] == recovered[2], rows + assert recovered[1] == str(sequence), rows assert int(recovered[3]) > 1, rows assert recovered[4] == "committed_after_retry", rows @@ -291,7 +291,7 @@ def resolved_snapshot(): rows = _wait_until( lambda: ( found - if any(row[0] == "recovered" and row[4] == "committed_by_get" for row in found) + if any(row[0] == "recovered" and row[4] == "committed_by_read" for row in found) else None ) if (found := _renewal_log_rows(node, since, sequence)) @@ -329,5 +329,5 @@ def resolved_snapshot(): assert recovered[1] == str(sequence), rows assert recovered[2] and mount_body["write_attempt_id"].startswith(recovered[2]), rows assert recovered[3] == "1", rows - assert recovered[4] == "committed_by_get", rows + assert recovered[4] == "committed_by_read", rows print("targeted request count (landed response lost): {}".format(stats["faults"]), flush=True) From 96e94850508278daee35bd0d151e9b11ce5c2949 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 7 Sep 2026 07:24:23 +0200 Subject: [PATCH 32/81] cas: reissue a control-plane request at once when the failure says the connection never formed Problem (CI, `cas_selects` regression suite on PR #2300, first of four fixes): under ephemeral-port exhaustion (`EADDRNOTAVAIL`) a mount-lease renewal's first attempt failed before anything was sent, and the request engine treated that like a store fault: it paid the settle read and the paced backoff before trying again, so a renewal that should have taken milliseconds took 11 s, the lease lapsed and the mount was fenced. A transport error whose text names a failed CONNECTION (`isConnectFailureHint`: connection refused / reset / unreachable, address not available, resolver failures) never reached the store, so there is nothing to settle. The engine (`CasOperation::writeLoop` and `readLoop`) now reissues such an attempt after one flat pause through `reissueAtOnce`, skipping the settle read; the reissue stays under the fence and deadline gates and does not advance the ordinary backoff. The hint is a reissue hint, not a not-sent verdict: a refusal-class answer is never a hint (an earlier ambiguity of the same write still gets its exact read), and `Retry::once` forbids the reissue but still records the observation. Every hinted attempt is counted at classification in `CASRequestConnectFailureHint`. Tests: `CASRequestsConnectHint.*` in `gtest_cas_requests.cpp` (reissue without a settle read, the false-hint case that is settled by a read, refusal after an earlier ambiguity, the deadline edge), the renewal-level twin in `gtest_cas_heartbeat.cpp`, and `gtest_writebuffer_s3.cpp` pins that `WriteBufferFromS3` surfaces the connect-failure text unchanged. The request contract and the metrics tables in `docs/en/antalya/cas` describe the hint. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../cas/architecture/mounts-and-leases.md | 4 +- src/Common/ProfileEvents.cpp | 5 +- .../ContentAddressed/Backend/CasRequests.cpp | 73 +++- .../ContentAddressed/Backend/CasRequests.h | 15 +- src/Disks/tests/gtest_cas_heartbeat.cpp | 48 +++ src/Disks/tests/gtest_cas_requests.cpp | 355 ++++++++++++++++++ src/IO/tests/gtest_writebuffer_s3.cpp | 41 ++ 7 files changed, 530 insertions(+), 11 deletions(-) diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index db6f984b2d0b..9b3df46f04df 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -75,7 +75,9 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve write_attempt_id)` tuple before I/O. Every physical retry repeats it byte-for-byte; a later GC fence preserves the observed ID, while reclaim and successor bodies mint new IDs. - **Resolve before retry.** A transient or ambiguous conditional `PUT` is followed by one exact - `GET`. The renewer adopts the result only when the complete body, including `write_attempt_id`, + `GET`, except that an attempt whose transport error names a failed connection is reissued first + after a flat pause and settled by the reissue's own answer (a 2xx) or by the exact `GET` that + follows its `412`. The renewer adopts the result only when the complete body, including `write_attempt_id`, equals its immutable request. If the predecessor token is still current, another identical `PUT` may follow bounded backoff. A same-pair twin, GC-fenced body, successor, foreign holder, or absent body is never treated as this renewal. diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 18d35d9d8b44..fdbe44da666d 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -941,12 +941,13 @@ The server successfully detected this situation and will download merged part fr M(CASConditionalWriteUnresolved, "Number of CAS conditional writes with an unknown outcome after conflict, timeout, connection loss, or server error. A non-zero value indicates backend instability or state requiring resolution.", ValueType::Number) \ M(CASConditionalWriteFenceLostPostWrite, "Number of CAS writes that succeeded but lost the final mount-fence check. A non-zero value indicates late responses after the mount lifecycle changed.", ValueType::Number) \ M(CASRequestAttempt, "Number of physical requests the CAS request contract started. Each one was admitted by the mount fence and reserved against the call's deadline before it was sent.", ValueType::Number) \ - M(CASRequestReissue, "Number of CAS requests re-sent after a jittered backoff sleep. Growth means the object store is throttling, failing, or contended.", ValueType::Number) \ + M(CASRequestReissue, "Number of CAS requests re-sent: after a jittered backoff for an ordinary failure, after a flat pause for a connect-failure hint, or at once with no pause at all for a first-attempt fuse. Growth means the object store is throttling, failing, or contended.", ValueType::Number) \ M(CASRequestConflictPause, "Number of clean lost races the CAS request contract repaid after a flat jitter instead of a growing backoff: the resolve read had settled the conflict and no transport fault preceded it.", ValueType::Number) \ - M(CASRequestResolveRead, "Number of requests the CAS request contract made to settle a refused precondition or an ambiguous write: a body read, or a HEAD where the caller needs only presence. Every conflict and every ambiguity costs one.", ValueType::Number) \ + M(CASRequestResolveRead, "Number of requests the CAS request contract made to settle a refused precondition or an ambiguous write: a body read, or a HEAD where the caller needs only presence. A connect-hinted attempt reissues without one.", ValueType::Number) \ M(CASRequestGaveUp, "Number of CAS writes that ended without a proven outcome, at a deadline, on a lost mount fence, or unresolved. A non-zero value means callers are being asked to retry later.", ValueType::Number) \ M(CASRequestRefused, "Number of CAS writes the store itself refused, proving they never applied: a malformed request, an entity too large, or an access or credential denial that no credential refresh was performed for, either because the disk has no refresh mechanism or because this write had already spent its one refresh.", ValueType::Number) \ M(CASRequestFenceLostPostWrite, "Number of CAS writes that were proven durable but lost the mount fence before the call could claim them. A non-zero value indicates late responses after the mount lifecycle changed.", ValueType::Number) \ + M(CASRequestConnectFailureHint, "Number of CAS write attempts whose transport error named a failed connection (no free local port, refused or unreachable peer, connect timeout). Under a reissuing policy the engine reissues them after a flat pause without a settle read, when the deadline and the fence admit it. Growth means the server cannot open connections to the object store.", ValueType::Number) \ M(CASMountRenewalAttempts, "Number of physical conditional renewal PUTs sent for CAS mount leases. This counts transport attempts, not logical renewals.", ValueType::Number) \ M(CASMountRenewalRetries, "Number of physical conditional renewal PUTs sent after the first attempt of one logical CAS mount-lease renewal.", ValueType::Number) \ M(CASMountRenewalResolved, "Number of CAS mount-lease renewals whose committed outcome was proved by an exact resolving GET.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 1ccba8e23d15..12b213eda4ae 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -18,6 +18,7 @@ #include +#include #include #include #include @@ -31,6 +32,7 @@ namespace ProfileEvents extern const Event CASRequestGaveUp; extern const Event CASRequestRefused; extern const Event CASRequestFenceLostPostWrite; + extern const Event CASRequestConnectFailureHint; } namespace DB::ErrorCodes @@ -235,6 +237,24 @@ bool isDefinitelyRefusedWrite([[maybe_unused]] const std::exception & e) return false; } +bool isConnectFailureHint([[maybe_unused]] const std::exception & e) +{ +#if USE_AWS_S3 + const auto * s3 = dynamic_cast(&e); + if (!s3 || s3->getS3ErrorCode() != Aws::S3::S3Errors::NETWORK_CONNECTION) + return false; + /// This repository's Poco (`SocketImpl::error`, `SocketImpl::connect`) is the source of every text. + static constexpr std::array texts{ + "Cannot assign requested address", "Connection refused", "No route to host", + "Network is unreachable", "connect timed out"}; + const std::string_view message = s3->message(); + for (std::string_view text : texts) + if (message.find(text) != std::string_view::npos) + return true; +#endif + return false; +} + CasRequests::CasRequests(BackendPtr backend_, Fence fence_, std::function now_ms_, std::function sleep_ms_, CasHotKeys * hot_keys_) @@ -829,6 +849,25 @@ std::optional CasOperation::pauseForConflict(WriteState & state, co return std::nullopt; } +/// A flat pause before reissuing an attempt whose failure text named a failed connection. +static constexpr uint64_t kConnectHintPauseMs = 50; + +std::optional CasOperation::pauseFlat(WriteState & state, const Retry::Bound & bound) +{ + const uint64_t needed = reservedFor(kConnectHintPauseMs, 2); + switch (gate(needed)) + { + case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); + case Gate::NoBudget: return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); + case Gate::Ok: break; + } + if (!fits(needed, bound)) + return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); + detail::recordReissue(); + owner.sleep_ms(kConnectHintPauseMs); + return std::nullopt; +} + WriteResult CasOperation::writeLoop(const String & key, const String & bytes, const std::optional & expected, const Retry & policy, const Retry::Bound & bound, WriteState & state, ResolveWith resolve_refusal_with) @@ -868,6 +907,7 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// instead of resolved by a read and reissued to the deadline. bool credential_answer = false; bool refreshed = false; + bool connect_hint = false; try { outcome = owner.withTransportAccess([&](auto & access) @@ -893,11 +933,18 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co } /// A refusal that FOLLOWS an ambiguous attempt of this inner write proves nothing about that /// attempt, so it is settled by the read below instead of ending the call here. - if (!refreshed && isDefinitelyRefusedWrite(e) && !state.any_ambiguous) + const bool definitely_refused = !refreshed && isDefinitelyRefusedWrite(e); + if (definitely_refused && !state.any_ambiguous) { ProfileEvents::increment(ProfileEvents::CASRequestRefused); return Refused{e.code(), e.message(), state.attempts_sent}; } + /// A refusal-class exception is never a hint, even when an earlier ambiguity of this inner + /// write kept it from ending the call above: that earlier attempt's fate is what the read + /// below must settle, and a hint reissue would skip it. + connect_hint = !definitely_refused && isConnectFailureHint(e); + if (connect_hint) + ProfileEvents::increment(ProfileEvents::CASRequestConnectFailureHint); } catch (const std::exception & e) { @@ -930,11 +977,25 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co continue; } - /// Every refused precondition and every ambiguous attempt is settled by ONE exact read, under - /// every policy: a refused precondition does not say WHO holds the key, and a 404 and a 412 - /// reach here as the same answer. A refused precondition needs only to know WHAT is there, so a - /// presence-only caller settles it with a HEAD; proving an ambiguous attempt landed needs the - /// bytes, and there the body read is unavoidable. + /// The failure text named a failed CONNECTION -- counted above, at classification, whether or + /// not this policy or the deadline/fence gates below let the reissue actually happen. A read now + /// would meet the same broken condition, so the reissue itself is the cheaper probe: the attempt + /// stays ambiguous (`any_ambiguous` is set above), and if the reissue meets a refused precondition + /// the read below settles it. + if (connect_hint && !policy.single_attempt) + { + if (auto given_up = pauseFlat(state, bound)) + return *given_up; + continue; + } + + /// Every refused precondition, and every ambiguous attempt that was not reissued on a + /// connect-failure hint, is settled by ONE exact read, under every policy: a refused + /// precondition does not say WHO holds the key, and a 404 and a 412 reach here as the same + /// answer. A hinted attempt reaches this read only when its reissue meets a refused + /// precondition. A refused precondition needs only to know WHAT is there, so a presence-only + /// caller settles it with a HEAD; proving an ambiguous attempt landed needs the bytes, and there + /// the body read is unavoidable. ProfileEvents::increment(ProfileEvents::CASRequestResolveRead); const Resolved resolved = resolve_refusal_with == ResolveWith::Presence && !state.any_ambiguous ? observePresence(key, policy, bound) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index 16ed2daf6ef1..233b8404083a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -30,6 +30,12 @@ namespace DB::Cas /// refusal: an unmodeled error may have landed. bool isDefinitelyRefusedWrite(const std::exception & e); +/// TRUE when a transport failure's text says the CONNECTION itself failed: no free local port, a +/// refused or unreachable peer, or the connect poll's own timeout. A hint, not a verdict: the same +/// errno can be reported after `send` or `recv`, so a hinted attempt keeps every property of an +/// ambiguous one; what the hint changes is only that the engine reissues before spending a read. +bool isConnectFailureHint(const std::exception & e); + /// Deterministic caller/local bugs, surfaced unchanged by every loop here: reissuing only replays the /// same failure and buries the root cause behind a retryable exception. The set is `LOGICAL_ERROR`, /// `NOT_IMPLEMENTED`, `BAD_ARGUMENTS` and `CORRUPTED_DATA`. @@ -353,8 +359,10 @@ class CasOperation /// the attempt landed. enum class ResolveWith : uint8_t { Body, Presence }; - /// The write engine: one call, any policy. Settles every refused precondition and every ambiguity - /// by an exact read before it reports anything. + /// The write engine: one call, any policy. `Committed` and `Conflict` are proven by an exact read + /// or by the reissue's own 2xx before they are reported; an attempt whose transport error named a + /// failed connection is reissued before its read. `Refused`, `Declined` and `GaveUp` report what + /// the store or the bounds said. WriteResult writeLoop(const String & key, const String & bytes, const std::optional & expected, const Retry & policy, const Retry::Bound & bound, WriteState & state, ResolveWith resolve_refusal_with); @@ -379,6 +387,9 @@ class CasOperation /// `Retry::conflictBackoff` sleep, and `state.reissues` untouched, so a transport fault that follows /// starts its own schedule at the beginning. std::optional pauseForConflict(WriteState & state, const Retry::Bound & bound); + /// The sibling for a failure text that named a failed connection. The same admission and the same + /// reservation, a flat `kConnectHintPauseMs` sleep, and `state.reissues` untouched. + std::optional pauseFlat(WriteState & state, const Retry::Bound & bound); /// `sleep_ms` plus `envelopes` attempt reservations, saturating. uint64_t reservedFor(uint64_t sleep_ms, uint32_t envelopes) const; diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index 0810370cf343..c63f686299a3 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -7,6 +7,9 @@ #include #include +#include "config.h" +#include + #include #include #include @@ -89,6 +92,7 @@ class RenewalScriptBackend final : public InMemoryBackend LandThenThrow, ReturnThenCancel, ThrowBeforeThenLandAfterResolve, + ThrowConnectHint, }; struct Attempt @@ -117,6 +121,16 @@ class RenewalScriptBackend final : public InMemoryBackend if (!actions.empty()) actions.pop_front(); + if (action == Action::ThrowConnectHint) + { +#if USE_AWS_S3 + throw DB::S3Exception("Poco::Exception. Code: 1000, e.code() = 99, Cannot assign requested address: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION); +#else + throw Poco::TimeoutException("connect timed out"); +#endif + } + if (action == Action::ThrowBefore || action == Action::ThrowBeforeThenLandAfterResolve) { if (action == Action::ThrowBeforeThenLandAfterResolve) @@ -736,6 +750,40 @@ TEST(CASHeartbeat, RenewalRetriesOneImmutableBodyAndAdoptsLostResponse) decodeMountLease(backend->attempts.front().bytes).write_attempt_id); } +#if USE_AWS_S3 +TEST(CASHeartbeat, RenewalOverConnectFailuresRecoversWithoutASettleRead) +{ + auto backend = std::make_shared(); + Layout layout("pool"); + const String srid = "test"; + const UInt128 uuid{0x1234}; + uint64_t wall_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, 9, wall_ms, 30000); + MountLeaseRenewer renewer( + ops.mount, ops.farewell, layout, srid, uuid, 9, std::chrono::milliseconds(30000), + [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(2000), + [&] { return boot_ms; }); + renewer.start(); + + backend->attempts.clear(); + backend->read_calls = 0; + /// Three seconds of "no free port" at 50 ms per hint, then the store answers. + for (int i = 0; i < 60; ++i) + backend->actions.push_back(RenewalScriptBackend::Action::ThrowConnectHint); + backend->actions.push_back(RenewalScriptBackend::Action::Delegate); + const MountRenewResult renewed = renewer.renew(renewalEnvironment(boot_ms)); + ASSERT_EQ(renewed.outcome, MountRenewOutcome::Committed); + EXPECT_GT(renewed.attempts_sent, 1u); + EXPECT_FALSE(renewed.resolved_by_read); /// classification `committed_after_retry` + EXPECT_EQ(backend->read_calls, 0u); + EXPECT_EQ(backend->attempts.size(), 61u); + for (const auto & attempt : backend->attempts) + EXPECT_EQ(attempt.bytes, backend->attempts.front().bytes); +} +#endif + TEST(CASHeartbeat, DeadlineBeforeSendTerminalizesWithTypedFailure) { auto backend = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index 6abb79e3def6..cf4c3c504b41 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -16,9 +16,16 @@ #include "config.h" #include +#include +#include #include +#include + +#include + #include +#include #include #include #include @@ -43,6 +50,7 @@ namespace ProfileEvents { extern const Event CASRequestReissue; extern const Event CASRequestConflictPause; + extern const Event CASRequestConnectFailureHint; } using namespace DB::Cas; @@ -2129,3 +2137,350 @@ TEST(CASRequests, StreamBodyServesTheAdoptedWindowEvenWhenAdmissionIsAlreadyRefu char c; expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { body->readStrict(&c, 1); }); } + +/// The hint is a text match on this repository's Poco. These pins fail the build's own tests the day +/// `SocketImpl::error` changes a word, which is the only way a text match stays honest. +TEST(CASRequestsConnectHint, PocoTextsArePinned) +{ + const auto text_of = [](int err) + { + try + { + Poco::Net::SocketImpl::error(err); + } + catch (const Poco::Exception & e) + { + return e.displayText(); + } + return std::string("did not throw"); + }; + EXPECT_THAT(text_of(EADDRNOTAVAIL), testing::HasSubstr("Cannot assign requested address")); + EXPECT_THAT(text_of(ECONNREFUSED), testing::HasSubstr("Connection refused")); + EXPECT_THAT(text_of(EHOSTUNREACH), testing::HasSubstr("No route to host")); + EXPECT_THAT(text_of(ENETUNREACH), testing::HasSubstr("Network is unreachable")); + /// The fifth text is the connect poll's own: `SocketImpl::connect` throws + /// `Poco::TimeoutException("connect timed out", ...)` (SocketImpl.cpp ~138). + EXPECT_THAT(Poco::TimeoutException("connect timed out", "10.255.255.1:9").displayText(), + testing::HasSubstr("connect timed out")); +} + +#if USE_AWS_S3 +TEST(CASRequestsConnectHint, ClassifierGuards) +{ + using Aws::S3::S3Errors; + for (const char * text : {"Cannot assign requested address", "Connection refused", "No route to host", + "Network is unreachable", "connect timed out"}) + { + const DB::S3Exception hinted(fmt::format("Poco::Exception. Code: 1000, e.code() = 99, {}: 10.0.0.1:9000", text), + S3Errors::NETWORK_CONNECTION); + EXPECT_TRUE(isConnectFailureHint(hinted)) << text; + /// The same text under another S3 error is not a transport verdict. + const DB::S3Exception other(String(text), S3Errors::INTERNAL_FAILURE); + EXPECT_FALSE(isConnectFailureHint(other)) << text; + } + EXPECT_FALSE(isConnectFailureHint(DB::S3Exception("Timeout", S3Errors::NETWORK_CONNECTION))); + EXPECT_FALSE(isConnectFailureHint(DB::S3Exception("Connection reset by peer", S3Errors::NETWORK_CONNECTION))); + EXPECT_FALSE(isConnectFailureHint(Poco::TimeoutException("connect timed out"))); + EXPECT_FALSE(isConnectFailureHint(std::runtime_error("Connection refused"))); +} + +namespace +{ +std::exception_ptr connectHint() +{ + return std::make_exception_ptr(DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 99, Cannot assign requested address: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION)); +} +} + +TEST(CASRequestsConnectHint, HintedFailuresReissueWithoutARead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", connectHint()); + backend->failNextWriteWith("k", connectHint()); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto hints_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load(); + const auto reissues_before = ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 3u); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_EQ(backend->writeTotal(), 3u); + EXPECT_EQ(backend->getTotal(), 0u); /// no settle read before the commit + ASSERT_EQ(clock.sleeps.size(), 2u); + EXPECT_EQ(clock.sleeps[0], 50u); /// the flat pause, twice + EXPECT_EQ(clock.sleeps[1], 50u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load() - hints_before, 2u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestReissue].load() - reissues_before, 2u); +} + +TEST(CASRequestsConnectHint, ReissueMeetsPreconditionAndAdoptsOwnBytes) +{ + /// The hint was false: the write landed, its response was lost as a connect-failure text. The + /// reissue meets 412 (the store now holds our OWN new incarnation, minted by the attempt whose + /// response never arrived), one read follows and proves the bytes are ours. + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Etag seen = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + backend->resetCounts(); + bool thrown = false; + /// Runs after the write lands and before the caller ever sees a value, with no lock held -- + /// `InMemoryBackend::applyWrite` (CasInMemoryBackend.cpp) calls it right there. + backend->onWriteCommitted("k", [&] + { + if (!thrown) + { + thrown = true; + throw DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 99, Cannot assign requested address: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION); + } + }); + WriteResult result = op.replace("k", "v2", seen, Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_TRUE(committed->resolved_by_read); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_EQ(backend->writeTotal(), 2u); + EXPECT_EQ(backend->getTotal(), 1u); + ASSERT_EQ(clock.sleeps.size(), 1u); + EXPECT_EQ(clock.sleeps[0], 50u); + } + /// Different ETag, other bytes: a conflict, as today. + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Etag seen = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + backend->failNextWriteWith("k", connectHint()); + /// A competitor lands during the flat pause: the engine's own sleep is the seam. + bool competitor_landed = false; + requests.setSleepFnForTest([&](uint64_t ms) + { + clock.sleepFn()(ms); + if (!competitor_landed) + { + competitor_landed = true; + auto other = requests.admit(); + orThrow(other.replace("k", "theirs", seen, Retry::standard()), "competitor"); + } + }); + WriteResult result = op.replace("k", "v2", seen, Retry::standard()); + EXPECT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(backend->getTotal(), 1u); + } + /// The ORIGINAL ETag is still current after the hinted attempt: the reissue meets a CLEAN 412 (the + /// store untouched, unlike sub-block 1's landed write), the settle read observes the original bytes + /// still there under the original ETag, and a further reissue is what actually commits. + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const Etag seen = *orThrow(op.create("k", "v1", Retry::standard()), "create"); + backend->resetCounts(); + backend->failNextWriteWith("k", connectHint()); /// attempt 1: hint, flat pause, no read + backend->refuseNextWrite("k"); /// attempt 2: clean 412, store unchanged + WriteResult result = op.replace("k", "v2", seen, Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_EQ(committed->attempts_sent, 3u); + EXPECT_EQ(backend->writeTotal(), 3u); + /// The read after attempt 2's 412 saw the precondition still satisfiable (the original ETag, + /// untouched), so the loop reissued instead of adopting -- exactly one read, not zero. + EXPECT_EQ(backend->getTotal(), 1u); + ASSERT_EQ(clock.sleeps.size(), 2u); + EXPECT_EQ(clock.sleeps[0], 50u); /// the flat pause after attempt 1's hint + EXPECT_LE(clock.sleeps[1], 200u); /// the backoff after attempt 2's settle read + } +} + +TEST(CASRequestsConnectHint, OnceKeepsOneWriteAndOneRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", connectHint()); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto hints_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load(); + WriteResult result = op.create("k", "v", Retry::once()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Unresolved); + EXPECT_EQ(backend->writeTotal(), 1u); + EXPECT_EQ(backend->getTotal(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); + /// The counter is "hint seen", recorded at classification: `Retry::once` never acts on it, but the + /// attempt's transport error still named a failed connection. + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load() - hints_before, 1u); +} + +TEST(CASRequestsConnectHint, EarlierAmbiguityStillSettlesByRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->injectAmbiguousWrite("k"); /// attempt 1: ordinary ambiguity -> read, backoff + /// Attempt 2's hint has to come from the hook, not a second `failNextWriteWith`: the armed-failure + /// queue is checked BEFORE the ambiguous-key injection on every call, so a queued failure would win + /// attempt 1 regardless of install order. `writeTotal()` ticks before the request is served, so it + /// reads 2 while attempt 2 is in flight. + bool hint_fired_on_second_attempt = false; + backend->onBeforeWrite("k", [&] + { + if (backend->writeTotal() == 2) + { + /// The ordering claim in full: attempt 1's ambiguity must already have been settled by its + /// read before attempt 2 -- the one place `sleeps[1] == 50u` alone could be fooled by a + /// same-range jittered draw (`backoff(1)` is `uniform(0, 200)`, so a reversed order would + /// false-green about once in 200 runs). + EXPECT_EQ(backend->getTotal(), 1u) << "attempt 1's ambiguity read must already have run"; + hint_fired_on_second_attempt = true; + throw DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 99, Cannot assign requested address: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION); + } + }); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 3u); + EXPECT_EQ(backend->getTotal(), 1u); + ASSERT_EQ(clock.sleeps.size(), 2u); + EXPECT_LE(clock.sleeps[0], 200u); /// the backoff after attempt 1's ambiguity read + EXPECT_EQ(clock.sleeps[1], 50u); /// the flat pause after attempt 2's hint + EXPECT_TRUE(hint_fired_on_second_attempt); +} + +/// A single exception can be BOTH refusal-class (`isDefinitelyRefusedWrite` matches on the exception +/// NAME, independent of the S3 error code) and hint-text (`isConnectFailureHint` matches on the code +/// and the message): the classifier order, not the exception's shape, must decide which wins. An +/// earlier ambiguity of this inner write keeps the refusal from ending the call, but that must never +/// let the hint skip the read the earlier attempt still needs. +TEST(CASRequestsConnectHint, RefusalAfterAnEarlierAmbiguitySettlesByRead) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->injectAmbiguousWrite("k"); /// attempt 1: ordinary ambiguity -> read, backoff + /// Attempt 2's refusal-and-hint exception has to come from the hook, not a second + /// `failNextWriteWith`: the armed-failure queue is checked BEFORE the ambiguous-key injection on + /// every call, so a queued failure would win attempt 1 regardless of install order (see the sibling + /// `EarlierAmbiguityStillSettlesByRead` above). `writeTotal()` ticks before the request is served, + /// so it reads 2 while attempt 2 is in flight. + bool refusal_fired_on_second_attempt = false; + backend->onBeforeWrite("k", [&] + { + if (backend->writeTotal() == 2) + { + EXPECT_EQ(backend->getTotal(), 1u) << "attempt 1's ambiguity read must already have run"; + refusal_fired_on_second_attempt = true; + throw DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 99, Cannot assign requested address: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION, "MalformedXML"); /// refusal AND hint text + } + }); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto hints_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load(); + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 3u); + /// Two reads, one per settled attempt: a hint reissue for attempt 2 would have skipped its own + /// read and left this at 1. + EXPECT_EQ(backend->getTotal(), 2u); + ASSERT_EQ(clock.sleeps.size(), 2u); + EXPECT_LE(clock.sleeps[0], 200u); /// backoff(1) after attempt 1's read + EXPECT_LE(clock.sleeps[1], 400u); /// backoff(2) after attempt 2's read -- today's verdict, + /// never the flat 50 ms hint pause + EXPECT_TRUE(refusal_fired_on_second_attempt); + /// The refusal classification wins outright: a definite refusal is never a hint, so the counter + /// must not move even though the exception's code and text also match `isConnectFailureHint`. + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load() - hints_before, 0u); +} + +TEST(CASRequestsConnectHint, GatesRefuseTheReissue) +{ + /// Deadline: hints until the window closes. + { + FakeClock clock; + auto backend = std::make_shared(); + for (int i = 0; i < 100; ++i) + backend->failNextWriteWith("k", connectHint()); + auto requests = makeRequests(backend, clock); + requests.setAttemptReservationForTest(1'000); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::within(3'000)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_EQ(backend->getTotal(), 0u); + } + /// Fence: the fence trips during the pause. + { + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", connectHint()); + bool lost = false; + Fence fence{ + [] { return uint64_t{1}; }, + [&](uint64_t, uint64_t) { return lost ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto requests = makeRequests(backend, clock, fence); + requests.setSleepFnForTest([&](uint64_t ms) { clock.sleepFn()(ms); lost = true; }); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::FenceLost); + } + /// Two envelopes exactly for the attempt; today's ambiguous path would still have its read + /// envelope (2000 >= 1000), the hint path gives up instead -- the documented deadline-edge + /// difference. + { + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", connectHint()); + auto requests = makeRequests(backend, clock); + requests.setAttemptReservationForTest(1'000); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::within(2'000)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_TRUE(gave_up->sent_any); + EXPECT_EQ(backend->getTotal(), 0u); + } +} + +TEST(CASRequestsConnectHint, AmbiguityAfterHintsStartsAtFirstBackoff) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", connectHint()); + backend->failNextWriteWith("k", connectHint()); + backend->failNextWriteWith("k", std::make_exception_ptr(Poco::TimeoutException("the write timed out"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::standard()); + ASSERT_TRUE(std::holds_alternative(result)); + ASSERT_EQ(clock.sleeps.size(), 3u); + EXPECT_EQ(clock.sleeps[0], 50u); + EXPECT_EQ(clock.sleeps[1], 50u); + /// `backoff(1)` is full jitter over [0, 200] ms (`CasRetry.h`): the hints did not advance the index. + EXPECT_LE(clock.sleeps[2], 200u); +} + +#endif diff --git a/src/IO/tests/gtest_writebuffer_s3.cpp b/src/IO/tests/gtest_writebuffer_s3.cpp index ac3453a16c32..ca52dd8b2742 100644 --- a/src/IO/tests/gtest_writebuffer_s3.cpp +++ b/src/IO/tests/gtest_writebuffer_s3.cpp @@ -564,6 +564,18 @@ struct PutObjectPreconditionFailedIngection: InjectionModel } }; +/// A transport failure shaped as `PocoHTTPClient` shapes one: the S3 error is `NETWORK_CONNECTION` +/// and the message is the Poco text, exception name empty. +struct PutObjectNetworkTextIngection: InjectionModel +{ + explicit PutObjectNetworkTextIngection(std::string text_) : text(std::move(text_)) {} + std::optional call(const Aws::S3::Model::PutObjectRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::NETWORK_CONNECTION, "", text, false); + } + std::string text; +}; + struct HeadObjectFailIngection: InjectionModel { std::optional call(const Aws::S3::Model::HeadObjectRequest & /*request*/) override @@ -1021,6 +1033,35 @@ TEST_P(SyncAsync, PreconditionFailedNeverLogsAtError) EXPECT_THAT(log_capture.captured(), testing::Not(testing::HasSubstr("S3Exception name"))); } +/// The classifier a later change adds to the CAS request engine (`isConnectFailureHint`) reads the +/// Poco text a connection failure carries. This pins that the fake S3 client -- and, through it, the +/// same `WriteBufferFromS3` rethrow every real disk uses -- hands the caller that text unchanged, +/// under `NETWORK_CONNECTION`. +TEST_F(WBS3Test, NetworkConnectionTextSurvives) +{ + for (const char * text : {"Cannot assign requested address", "Connection refused", "No route to host", + "Network is unreachable", "connect timed out"}) + { + setInjectionModel(std::make_shared(text)); + WriteSettings write_settings; + write_settings.object_storage_retry_profile = ObjectStorageRetryProfile::SingleAttempt; + write_settings.s3_max_unexpected_write_error_retries_override = 1; + try + { + auto buffer = getWriteBuffer("network_text", write_settings); + buffer->write('A'); + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + FAIL() << "the injected failure must surface"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::NETWORK_CONNECTION) << text; + EXPECT_THAT(e.message(), testing::HasSubstr(text)); + } + } +} + TEST_P(SyncAsync, ExceptionOnCreateMPU) { setInjectionModel(std::make_shared()); From 9a6bcb68aca1c57be449b8e9135aade51eddbc41 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 7 Sep 2026 07:24:26 +0200 Subject: [PATCH 33/81] cas: budget an attempt by its whole envelope and fuse a first attempt that hangs in connect Problem (CI, `cas_selects` regression on PR #2300, second fix): a first attempt that hung in connect burned the whole `cas_attempt_timeout_ms`, the budget validation ignored the connect phase entirely, and the SDK could retry behind the engine's back, so the engine's arithmetic about how long a renewal can take was wrong on both ends and the lease-safety margin was not what the operator configured. The attempt now carries its number and timeouts to the transport: the control-plane requests pass an `ObjectStorageControlRequest` (profile, attempt timeout, connect cap, attempt number) through `ReadSettings` / `WriteSettings`, `ReadBufferFromS3::sendRequest`, `WriteBufferFromS3::getPutRequest`, the `attempt_seed` through `getObjectInfo`, and the LIST iterator's pages; a writable Native mount uses a single-attempt S3 client clone (`getSingleAttemptClient`) so the SDK never retries. The connect phase is capped by `connect_timeout_cap_ms`, frozen at open from the disk's `connect_timeout_ms` (zero meaning the attempt timeout) and reaching every verb through the client; a first attempt that times out in that window is a fuse (`isFirstAttemptFuseTimeout`, `CASRequestFirstAttemptFuse`) and is reissued at once after the settle read -- counted under `Retry::once` too, and on the read side. The budget is one envelope everywhere: `CasRequestBudget::attemptEnvelopeMs` = attempt + 2 x cap, validated as `attempt >= 1`, `envelope + margin < TTL` and `period + 2 x envelope + margin < TTL` (`BAD_ARGUMENTS`), forwarded by every backend decorator. The mount path derives its fixed windows from the same numbers: the farewell window is `max(kFarewellBudgetMs, 2 x reservation + slack)` bounded by `Retry::untilLeaseSafe` (the old fixed 10 s was smaller than the write's reservation, so every graceful restart failed its farewell and the successor paid the full token-stability observation), and the three gated pauses collapse into one helper. The hint and fuse counters are not inflated by a credential reissue that happens to match their text: they are suppressed only when the credential path owns the reissue (`refresh_owns_reissue`), so an earlier ambiguity's hint is still counted. Tests: `CASRequestBudget.*`, `CASRequestsFuse.*`, `CASEnvelopeWiring.*` (a genuine `S3ObjectStorage` against a delaying local HTTP server: for every verb the production dispatch picks the single-attempt clone with the caller's timeout, one request seen, while the default profile succeeds), `S3SingleAttemptClient.*`, the attempt-seed carry tests in `gtest_aws_s3_client.cpp` / `gtest_writebuffer_s3.cpp`, the farewell-window pins in `gtest_cas_heartbeat.cpp` / `gtest_cas_mount.cpp`, the detached-work budget hook now sets the whole budget, and `test_cas_gcs` proves a fuse timeout reissues as attempt 2 in PUT(1) -> GET -> PUT(2) order. `docs/en/antalya/cas/configuration.md` carries the envelope arithmetic with the default figures (envelope 7000 ms; a disk `connect_timeout_ms` of 2000 ms or more refuses a writable mount with the defaults) and `mounts-and-leases.md` the derived windows. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../cas/architecture/mounts-and-leases.md | 6 +- docs/en/antalya/cas/configuration.md | 15 +- src/Common/ProfileEvents.cpp | 1 + .../ContentAddressed/Backend/CasBackend.h | 6 + .../Backend/CasInstrumentedBackend.h | 1 + .../Backend/CasObjectStorageBackend.cpp | 92 ++- .../Backend/CasObjectStorageBackend.h | 71 +- .../Backend/CasRequestBudget.cpp | 62 +- .../Backend/CasRequestBudget.h | 36 +- .../ContentAddressed/Backend/CasRequests.cpp | 135 +++- .../ContentAddressed/Backend/CasRequests.h | 86 +- .../ContentAddressed/Backend/CasRetry.h | 13 +- .../Backend/CasThrottlingBackend.h | 1 + .../Backend/CasTransportAccess.h | 12 +- .../ContentAddressedMetadataStorage.cpp | 32 +- .../ContentAddressedMetadataStorage.h | 11 + .../ContentAddressedSettings.cpp | 9 +- .../ContentAddressed/Pool/CasMountRuntime.cpp | 10 +- .../ContentAddressed/Pool/CasPool.cpp | 55 +- .../ContentAddressed/Pool/CasPool.h | 9 +- .../ContentAddressed/Pool/CasServerRoot.cpp | 45 +- .../ObjectStorages/IObjectStorage.cpp | 15 +- .../ObjectStorages/IObjectStorage.h | 23 +- .../ObjectStorages/S3/S3ObjectStorage.cpp | 80 +- .../ObjectStorages/S3/S3ObjectStorage.h | 35 +- src/Disks/tests/gtest_cas_backend.cpp | 34 + .../tests/gtest_cas_bootstrap_ordering.cpp | 35 + .../tests/gtest_cas_bulk_delete_backend.cpp | 3 +- src/Disks/tests/gtest_cas_detached_work.cpp | 13 +- src/Disks/tests/gtest_cas_event_log.cpp | 1 + src/Disks/tests/gtest_cas_heartbeat.cpp | 260 +++++- src/Disks/tests/gtest_cas_mount.cpp | 7 +- src/Disks/tests/gtest_cas_mount_runtime.cpp | 34 +- src/Disks/tests/gtest_cas_observability.cpp | 1 + src/Disks/tests/gtest_cas_pool.cpp | 157 +++- .../tests/gtest_cas_ref_recovery_cas_walk.cpp | 2 +- src/Disks/tests/gtest_cas_ref_writer.cpp | 2 +- src/Disks/tests/gtest_cas_requests.cpp | 547 +++++++++++++ .../gtest_cas_s3_single_attempt_client.cpp | 757 ++++++++++++++++++ src/Disks/tests/gtest_cas_s3_staging.cpp | 6 +- src/Disks/tests/gtest_cas_sentinel_probe.cpp | 24 + src/Disks/tests/gtest_cas_upstream_slice.cpp | 37 +- src/Disks/tests/gtest_cas_writer_duties.cpp | 2 + src/IO/ReadBufferFromS3.cpp | 2 +- src/IO/ReadSettings.h | 8 + src/IO/S3/Requests.h | 7 + src/IO/S3/getObjectInfo.cpp | 16 +- src/IO/S3/getObjectInfo.h | 5 +- src/IO/S3/tests/gtest_aws_s3_client.cpp | 208 +++++ src/IO/WriteBufferFromS3.cpp | 3 + src/IO/WriteSettings.h | 19 + src/IO/tests/gtest_writebuffer_s3.cpp | 255 +++++- .../test_cas_gcs/gcs_mocks/server.py | 96 ++- tests/integration/test_cas_gcs/test.py | 126 +++ .../configs/request_budget_disks.xml | 51 ++ .../test_cas_mount_renewal_retry/test.py | 111 ++- 56 files changed, 3342 insertions(+), 348 deletions(-) create mode 100644 src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp create mode 100644 tests/integration/test_cas_mount_renewal_retry/configs/request_budget_disks.xml diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index 9b3df46f04df..1681c85c96c4 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -84,7 +84,7 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve - **Absolute deadline.** Renewal uses `CLOCK_BOOTTIME`, not `CLOCK_MONOTONIC`, so a VM resumed from suspend correctly observes itself expired. Its absolute deadline is the minimum of the existing request-operation budget and the last confirmed lease deadline minus the safety margin. The - controller checks that one configured attempt still fits before each backend `PUT` or resolving + controller checks that one attempt envelope still fits before each backend `PUT` or resolving `GET`, after each interruptible backoff, and before accepting success. A retry, `GET`, response timestamp, or wall-clock step never extends authority. - **Cadence.** The runtime normally starts a logical renewal every `cas_mount_renew_period_ms` (default @@ -96,8 +96,8 @@ watermark — there is no separate watermark object. `MountLease` fields: `serve and rechecks it immediately before the object-store call and on every conditional retry. Reads are not gated. - **Request-budget admission.** `refAppendFenceOk` refuses to *start* a ref-log attempt unless - `attempt_timeout + safety_margin` fits inside the remaining lease, rejecting with - `BAD_ARGUMENTS` at request-admission time rather than mid-flight. + `2 × envelope + safety_margin` fits inside the remaining lease (a write and its settlement read), + rejecting with `BAD_ARGUMENTS` at request-admission time rather than mid-flight. **Losing the lease is neither read-only mode nor a process abort.** `MountLeaseRenewer` is a synchronous durable-slot state machine. A committed result advances its token, sequence, confirmed diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 79e77b55f4cc..3b3ed87b6998 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -93,7 +93,7 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_blob_hash_allow_new` | `false` | Explicit opt-in to admit a new hash algorithm into an existing pool. One-way: once admitted, the pool carries both algorithms permanently | | `skip_access_check` | `false` | Skip the boot-time capability probe (start now, fix later). Only the preflight probe is skipped — the conditional-write correctness check still runs on every writable mount. **Not available on a writable generation-token (GCS) disk**, which refuses to mount with it: there, the probe battery is the only proof that a token-exact delete carries its generation precondition. Mount such a disk read-only if you need to defer the check | | `cas_mount_lease_ttl_ms` | `30000` | Milliseconds for which a mount lease remains valid after a successful claim or renewal (≥ 1). Lower values shorten stale-mount recovery but reduce tolerance for object-storage and scheduling delays | -| `cas_mount_renew_period_ms` | `10000` | Milliseconds between background mount-lease renewals (≥ 1). It must leave enough time for one request attempt and the lease safety margin before the TTL expires | +| `cas_mount_renew_period_ms` | `10000` | Milliseconds between background mount-lease renewals (≥ 1). It must leave enough time for two attempt envelopes (a renewal write and its settlement read) and the lease safety margin before the TTL expires: `period + 2 × envelope + margin < TTL` | | `cas_gc_snapshot_generations_to_keep` | `3` | GC snapshot generations retained | | `cas_gc_shards` | `1` | Blob-hash-prefix reducer shards (≥ 1). Recorded in the pool at creation; a mismatching config is refused at mount | | `gcs_max_conditional_put_bytes` | 1 GiB | Largest conditional non-blob `PUT` on a generation-token store, including create-if-absent metadata/control artifacts and conditional replacements. Blob publication is unconditional, uses ordinary multipart, and is not subject to this cap | @@ -103,10 +103,19 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_manifest_decode_cache_bytes` | 128 MiB | Manifest decode cache byte budget (`0` disables) | | `cas_gc_meta_pool_size` | `16` | Bounded pool size for GC per-hash freshness-meta writes | | `cas_gc_read_concurrency` | `16` | Bounded pool size for the GC fold's read-ahead of checkpoints, ref logs, manifests and zero-candidate HEADs; `1` disables | -| `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove) | -| `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: `cas_attempt_timeout_ms + cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, or the disk refuses to open writable | +| `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove, conditional write), at least 1. Together with the connect cap it forms the attempt envelope (`cas_attempt_timeout_ms + 2 × cap`; the cap is `cas_attempt_timeout_ms` itself when the disk's `connect_timeout_ms` is `0`, else `min(connect_timeout_ms, cas_attempt_timeout_ms)`) that the lease arithmetic reserves: one TCP connect and one TLS handshake under the cap each, send/receive bounded per socket operation by `cas_attempt_timeout_ms`. With background renewal the cadence check requires `cas_mount_renew_period_ms + 2 × envelope + cas_lease_safety_margin_ms < cas_mount_lease_ttl_ms`, which puts an effective ceiling on the frozen connect cap: under the defaults (TTL 30000, period 10000, margin 2000) the envelope must stay under 9000, so a disk `connect_timeout_ms` of 2000 ms or more refuses to open writable — lower the connect timeout or raise the TTL if you hit this | +| `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: the attempt envelope + `cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, and `cas_mount_renew_period_ms` + 2 × envelope + `cas_lease_safety_margin_ms` too, or the disk refuses to open writable | | `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount | +A shorter TTL reduces the tolerance for object-storage delays; a shorter renewal period increases it +(renewal starts earlier) at the cost of more background traffic. With the defaults, +`cas_mount_lease_ttl_ms − cas_lease_safety_margin_ms − cas_mount_renew_period_ms − 2 × envelope = +4000` ms is the scheduling-lateness budget before the first renewal attempt of a period can begin, +where `envelope = cas_attempt_timeout_ms + 2 × cap` (7000 ms with defaults) and `cap` is +`cas_attempt_timeout_ms` when the disk's `connect_timeout_ms` is `0`, else +`min(connect_timeout_ms, cas_attempt_timeout_ms)` (1000 ms with defaults); the renewal then keeps +retrying until `confirmed deadline − cas_lease_safety_margin_ms`. + ## Advanced GC pacing settings {#advanced-gc-pacing-settings} These settings bound individual phases of a `GC` round. The first two accept any `UInt64` value; diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index fdbe44da666d..1ce7bdb60a2b 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -948,6 +948,7 @@ The server successfully detected this situation and will download merged part fr M(CASRequestRefused, "Number of CAS writes the store itself refused, proving they never applied: a malformed request, an entity too large, or an access or credential denial that no credential refresh was performed for, either because the disk has no refresh mechanism or because this write had already spent its one refresh.", ValueType::Number) \ M(CASRequestFenceLostPostWrite, "Number of CAS writes that were proven durable but lost the mount fence before the call could claim them. A non-zero value indicates late responses after the mount lifecycle changed.", ValueType::Number) \ M(CASRequestConnectFailureHint, "Number of CAS write attempts whose transport error named a failed connection (no free local port, refused or unreachable peer, connect timeout). Under a reissuing policy the engine reissues them after a flat pause without a settle read, when the deadline and the fence admit it. Growth means the server cannot open connections to the object store.", ValueType::Number) \ + M(CASRequestFirstAttemptFuse, "Number of CAS control requests whose first HTTP attempt matched the adaptive first-attempt timeout; the engine reissues them at once as attempt 2 when the policy and the gates permit. Growth means the object store does not answer a fresh connection within the first-attempt timeout.", ValueType::Number) \ M(CASMountRenewalAttempts, "Number of physical conditional renewal PUTs sent for CAS mount leases. This counts transport attempts, not logical renewals.", ValueType::Number) \ M(CASMountRenewalRetries, "Number of physical conditional renewal PUTs sent after the first attempt of one logical CAS mount-lease renewal.", ValueType::Number) \ M(CASMountRenewalResolved, "Number of CAS mount-lease renewals whose committed outcome was proved by an exact resolving GET.", ValueType::Number) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index cbfdb4595fb2..96911defc15f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -237,6 +237,12 @@ class Backend /// request contract reserves it before every attempt it starts. virtual uint64_t attemptTimeoutMs() const { return 0; } + /// What one attempt may cost end to end, connect included; the contract reserves THIS. A backend + /// with no connect notion answers its attempt timeout. A decorator that forwards `attemptTimeoutMs` + /// to an inner backend must forward THIS too -- the default falls back to `attemptTimeoutMs`, which + /// would silently drop the inner backend's connect contribution. + virtual uint64_t attemptEnvelopeMs() const { return attemptTimeoutMs(); } + /// Asks the storage to re-acquire credentials. TRUE when fresh ones were installed, so the /// caller's reissue can sign with them; FALSE when this backend has no refresh mechanism, which /// makes an expired-credential failure terminal for the caller's policy rather than retryable. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h index d0c853d29d69..ed0c25dab861 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h @@ -160,6 +160,7 @@ class InstrumentedBackend final : public Backend Dialect dialect() const override { return inner->dialect(); } bool supportsListTokens() const override { return inner->supportsListTokens(); } uint64_t attemptTimeoutMs() const override { return inner->attemptTimeoutMs(); } + uint64_t attemptEnvelopeMs() const override { return inner->attemptEnvelopeMs(); } bool refreshCredentials() override { return inner->refreshCredentials(); } private: diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index 2c9440a8f4b7..0a302071c9fc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -125,11 +125,13 @@ void recordConditionalWriteOutcome(CasWriteOutcome outcome) } ObjectStorageBackend::ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_, - bool single_attempt_control_plane_, uint64_t attempt_timeout_ms_) + bool single_attempt_control_plane_, uint64_t attempt_timeout_ms_, + uint64_t connect_timeout_cap_ms_) : object_storage(std::move(object_storage_)) , mode(mode_) , single_attempt_control_plane(single_attempt_control_plane_) , attempt_timeout_ms(attempt_timeout_ms_) + , connect_timeout_cap_ms(connect_timeout_cap_ms_) , emu_root(object_storage->getCommonKeyPrefix()) { if (mode == Mode::Native && object_storage->conditionalOpsUseGenerationTokens()) @@ -221,10 +223,9 @@ bool ObjectStorageBackend::isValidTokenValue(Dialect type, const String & value) return isIncarnationValue(type, value); } -std::optional ObjectStorageBackend::nativeHead( - const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) +std::optional ObjectStorageBackend::nativeHead(const String & key, const ObjectStorageControlRequest & request) { - auto metadata = object_storage->tryGetObjectMetadataWithNativeToken(key, /*with_tags=*/false, profile, timeout_ms); + auto metadata = object_storage->tryGetObjectMetadataWithNativeToken(key, /*with_tags=*/false, request); if (!metadata) return std::nullopt; @@ -599,24 +600,25 @@ String ObjectStorageBackend::emuMintToken(const String & key, const String & eta /// Backend interface /// ========================================================================================= -ReadSettings ObjectStorageBackend::readSettingsFor(ObjectStorageRetryProfile profile, uint64_t timeout_ms) const +ReadSettings ObjectStorageBackend::readSettingsFor(const ObjectStorageControlRequest & request) const { ReadSettings rs = getReadSettings(); /// Mark the request for the store's native conditional dialect, so a GCS read is answered with a /// generation rather than an MD5-shaped ETag. rs.object_storage_request_mode = ObjectStorageRequestMode::NativeConditional; - rs.object_storage_retry_profile = profile; - rs.object_storage_attempt_timeout_ms = timeout_ms; + rs.object_storage_retry_profile = request.profile; + rs.object_storage_attempt_timeout_ms = request.attempt_timeout_ms; + rs.object_storage_connect_timeout_cap_ms = request.connect_timeout_cap_ms; + rs.object_storage_attempt_number = request.attempt_number; return rs; } -std::optional ObjectStorageBackend::read(const String & key, TransportAccess &) +std::optional ObjectStorageBackend::read(const String & key, TransportAccess & access) { - return readUnder(key, controlPlaneProfile(), attempt_timeout_ms); + return readUnder(key, controlRequest(access.attemptNo())); } -std::optional ObjectStorageBackend::readUnder( - const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) +std::optional ObjectStorageBackend::readUnder(const String & key, const ObjectStorageControlRequest & request) { if (mode == Mode::Native) { @@ -631,7 +633,7 @@ std::optional ObjectStorageBackend::readUnder( /// test whose stderr is checked fails on the log line alone. Expect404ResponseScope scope; auto got = object_storage->readSmallObjectAndGetObjectMetadata( - StoredObject(key), readSettingsFor(profile, timeout_ms), casMaxStoredObjectBytes()); + StoredObject(key), readSettingsFor(request), casMaxStoredObjectBytes()); return Raw{std::move(got.data), normalizeTokenValue(got.metadata.etag)}; } catch (const std::exception & e) @@ -705,16 +707,15 @@ std::unique_ptr ObjectStorageBackend::stream(const String & key, Tra } } -std::optional ObjectStorageBackend::head(const String & key, TransportAccess &) +std::optional ObjectStorageBackend::head(const String & key, TransportAccess & access) { - return headUnder(key, controlPlaneProfile(), attempt_timeout_ms); + return headUnder(key, controlRequest(access.attemptNo())); } -std::optional ObjectStorageBackend::headUnder( - const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) +std::optional ObjectStorageBackend::headUnder(const String & key, const ObjectStorageControlRequest & request) { if (mode == Mode::Native) - return nativeHead(key, profile, timeout_ms); + return nativeHead(key, request); std::lock_guard lock(emu_mutex); if (!emuExists(key)) @@ -731,13 +732,12 @@ std::optional ObjectStorageBackend::headUnder( } /// See Backend::probeSentinelRaw / CasBackend.h's ProbeOutcome for the semantics this classifies. -SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key, TransportAccess &) +SentinelProbeResult ObjectStorageBackend::probeSentinelRaw(const String & key, TransportAccess & access) { - return probeSentinelUnder(key, controlPlaneProfile(), attempt_timeout_ms); + return probeSentinelUnder(key, controlRequest(access.attemptNo())); } -SentinelProbeResult ObjectStorageBackend::probeSentinelUnder( - const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms) +SentinelProbeResult ObjectStorageBackend::probeSentinelUnder(const String & key, const ObjectStorageControlRequest & request) { if (mode == Mode::Native) { @@ -746,7 +746,7 @@ SentinelProbeResult ObjectStorageBackend::probeSentinelUnder( /// One `read`: unlike a bodyless HEAD 404, a GET 404 carries a response body, so the /// SDK can parse its `` and a missing key and a missing bucket arrive as different /// errors -- which is the whole distinction this probe exists to make. - auto raw = readUnder(key, profile, timeout_ms); + auto raw = readUnder(key, request); if (!raw) return {ProbeOutcome::KeyAbsent, std::nullopt}; return {ProbeOutcome::Present, std::move(raw->bytes)}; @@ -785,7 +785,7 @@ SentinelProbeResult ObjectStorageBackend::probeSentinelUnder( if (!object_storage->existsOrHasAnyChild(emu_root)) return {ProbeOutcome::ContainerAbsent, std::nullopt}; - auto raw = readUnder(key, profile, timeout_ms); + auto raw = readUnder(key, request); if (!raw) return {ProbeOutcome::KeyAbsent, std::nullopt}; return {ProbeOutcome::Present, std::move(raw->bytes)}; @@ -802,7 +802,7 @@ SentinelProbeResult ObjectStorageBackend::probeSentinelUnder( /// unconditional multipart-capable write. CAS-mutable keys (shard manifests, gc/state, the registry) /// also skip the racy post-upload existence/size check; a publish's manifest CAS was observed racing /// the GC fence there. -WriteSettings ObjectStorageBackend::conditionalWriteSettings() const +WriteSettings ObjectStorageBackend::conditionalWriteSettings(size_t attempt_no) const { WriteSettings ws; ws.object_storage_request_mode = ObjectStorageRequestMode::NativeConditional; @@ -821,11 +821,15 @@ WriteSettings ObjectStorageBackend::conditionalWriteSettings() const /// And that attempt is bounded by the same budget the caller reserved for it. Without this the /// write would run under the storage's own timeout while its caller waited on a shorter one. ws.object_storage_attempt_timeout_ms = attempt_timeout_ms; + /// And that attempt's connect is bounded by the same frozen cap every other control request carries. + ws.object_storage_connect_timeout_cap_ms = connect_timeout_cap_ms; + /// The caller's own physical-attempt count, so the HTTP client sees a reissue as attempt >= 2. + ws.object_storage_attempt_number = attempt_no; return ws; } std::expected ObjectStorageBackend::write( - const String & key, const String & bytes, const std::optional & expected_value, TransportAccess &) + const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) { /// An empty, wildcard or list value would turn the precondition into an unconditional write -- /// refuse it as a caller bug before anything else runs. @@ -837,7 +841,7 @@ std::expected ObjectStorageBackend::write( if (mode == Mode::Native) { - WriteSettings ws = conditionalWriteSettings(); + WriteSettings ws = conditionalWriteSettings(access.attemptNo()); if (expected_value) ws.object_storage_write_if_match = *expected_value; else @@ -937,13 +941,13 @@ void ObjectStorageBackend::publish(const BlobPublishRequest & request, Transport write_settings); } -Backend::RawRemoval ObjectStorageBackend::remove(const String & key, const String & expected_value, TransportAccess &) +Backend::RawRemoval ObjectStorageBackend::remove(const String & key, const String & expected_value, TransportAccess & access) { - return removeUnder(key, expected_value, controlPlaneProfile(), attempt_timeout_ms); + return removeUnder(key, expected_value, controlRequest(access.attemptNo())); } Backend::RawRemoval ObjectStorageBackend::removeUnder( - const String & key, const String & expected_value, ObjectStorageRetryProfile profile, uint64_t timeout_ms) + const String & key, const String & expected_value, const ObjectStorageControlRequest & request) { /// Same grammar guard as `write`, and for the same reason: an empty, wildcard or list value would /// turn the condition into an unconditional delete. @@ -957,7 +961,7 @@ Backend::RawRemoval ObjectStorageBackend::removeUnder( { /// `NOT_IMPLEMENTED` from a storage that does not enforce conditional removal propagates — /// fail-closed by construction. - auto result = object_storage->removeObjectIfTokenMatches(StoredObject(key), expected_value, profile, timeout_ms); + auto result = object_storage->removeObjectIfTokenMatches(StoredObject(key), expected_value, request); switch (result.outcome) { case ConditionalRemoveOutcome::Removed: @@ -999,7 +1003,7 @@ void ObjectStorageBackend::emuForgetDeletedToken(const String & key) } } -void ObjectStorageBackend::removeManyWriteOnce(const std::vector & keys, TransportAccess &) +void ObjectStorageBackend::removeManyWriteOnce(const std::vector & keys, TransportAccess & access) { if (keys.empty()) return; @@ -1010,7 +1014,7 @@ void ObjectStorageBackend::removeManyWriteOnce(const std::vector & for (const WriteOnceKey & key : keys) objects.emplace_back(key.str()); /// `NOT_IMPLEMENTED` from a storage without a batch delete propagates -- fail-closed by construction. - object_storage->removeObjectsIfExistUnderProfile(objects, controlPlaneProfile(), attempt_timeout_ms); + object_storage->removeObjectsIfExistUnderProfile(objects, controlRequest(access.attemptNo())); return; } @@ -1024,13 +1028,13 @@ void ObjectStorageBackend::removeManyWriteOnce(const std::vector & } } -Backend::RawListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess &) +Backend::RawListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) { - return listUnder(prefix, cursor, limit, controlPlaneProfile(), attempt_timeout_ms); + return listUnder(prefix, cursor, limit, controlRequest(access.attemptNo())); } Backend::RawListPage ObjectStorageBackend::listUnder( - const String & prefix, const String & cursor, size_t limit, ObjectStorageRetryProfile profile, uint64_t timeout_ms) + const String & prefix, const String & cursor, size_t limit, const ObjectStorageControlRequest & request) { /// Use the lazy object-storage iterator instead of `listObjects(..., max_keys=0)`: the latter /// materialized the whole prefix, then sliced client-side, so a paginated walk re-fetched the full @@ -1099,7 +1103,7 @@ Backend::RawListPage ObjectStorageBackend::listUnder( static constexpr size_t max_store_page = 1'000'000; const size_t store_page = cursor.empty() ? std::min(limit, max_store_page) + 1 : 0; RawListPage page; - auto it = object_storage->iterate(physical_prefix, /*max_keys=*/store_page, /*with_tags=*/false, start_after, profile, timeout_ms); + auto it = object_storage->iterate(physical_prefix, /*max_keys=*/store_page, /*with_tags=*/false, start_after, request); for (; it->isValid(); it->next()) { const auto child = it->current(); @@ -1131,5 +1135,21 @@ Backend::RawListPage ObjectStorageBackend::listUnder( return page; } +void ensureBackendMatchesBudget(const ObjectStorageBackend & backend, const CasRequestBudget & budget) +{ + const uint64_t budget_connect_cap = budget.connect_timeout_cap_ms.value_or(0); + if (backend.connectTimeoutCapMs() != budget_connect_cap) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS: backend connect timeout cap ({} ms) does not match the pool's request budget ({} ms); " + "the mount's lease arithmetic was validated against the budget alone, so a backend built with " + "another cap could silently outlive it", + backend.connectTimeoutCapMs(), budget_connect_cap); + if (backend.attemptTimeoutMs() != budget.attempt_timeout_ms) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS: backend attempt timeout ({} ms) does not match the pool's request budget ({} ms); " + "the mount's lease arithmetic was validated against the budget alone, so a backend built with " + "another timeout could silently outlive it", + backend.attemptTimeoutMs(), budget.attempt_timeout_ms); +} } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index 5aad6b1940c4..f9f51396efbb 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -1,5 +1,6 @@ #pragma once #include +#include #include #include #include @@ -65,10 +66,12 @@ class ObjectStorageBackend final : public Backend /// requests below: a writable Native mount owns its own retry policy and a transparently retried /// request would outlive the caller's deadline, while a read-only mount has no such deadline and /// keeps the storage's default. `attempt_timeout_ms` bounds ONE attempt of those requests; 0 - /// leaves the storage's own timeout in place. Both are supplied by the mount that opens the pool; - /// the defaults are what a narrow unit test constructing a bare backend gets. + /// leaves the storage's own timeout in place. `connect_timeout_cap_ms` caps the connect portion of + /// that same attempt (0 = no cap), frozen by the mount at open. All three are supplied by the mount + /// that opens the pool; the defaults are what a narrow unit test constructing a bare backend gets. ObjectStorageBackend(ObjectStoragePtr object_storage_, Mode mode_, - bool single_attempt_control_plane_ = false, uint64_t attempt_timeout_ms_ = 0); + bool single_attempt_control_plane_ = false, uint64_t attempt_timeout_ms_ = 0, + uint64_t connect_timeout_cap_ms_ = 0); /// Read the whole object, or return `nullopt` if it is absent. Native mode reads the incarnation /// value out of the GET response itself, so no HEAD precedes it; a not-found race is reported as @@ -103,6 +106,16 @@ class ObjectStorageBackend final : public Backend Dialect dialect() const override { return mode == Mode::Native ? native_token_type : Dialect::Emulated; } /// The budget for one attempt of a read-class request, as configured by the mount. uint64_t attemptTimeoutMs() const override { return attempt_timeout_ms; } + /// What one attempt may cost end to end, connect included: the attempt timeout plus two connect + /// caps (TCP, then TLS), saturating. Delegates to `CasRequestBudget::attemptEnvelopeMs` (the single + /// definition of this formula) rather than re-deriving it here. + uint64_t attemptEnvelopeMs() const override + { + return CasRequestBudget{.attempt_timeout_ms = attempt_timeout_ms, .connect_timeout_cap_ms = connect_timeout_cap_ms} + .attemptEnvelopeMs(); + } + /// The frozen connect cap this backend was constructed with; see the constructor. + uint64_t connectTimeoutCapMs() const { return connect_timeout_cap_ms; } /// Ask the storage to re-acquire credentials through its refresh callback. bool refreshCredentials() override { return object_storage->tryRefreshCredentialsViaCallback(); } @@ -184,9 +197,11 @@ class ObjectStorageBackend final : public Backend /// Settings for a Native COMPARE/CREATE write (create-if-absent, compare-and-set): mark the request /// conditional, make exactly one attempt at every retry layer, skip the racy post-upload /// existence/size check, and force a single PUT on generation stores because GCS does not - /// enforce the condition on multipart completion. - WriteSettings conditionalWriteSettings() const; - WriteSettings conditionalWriteSettingsForTest() const { return conditionalWriteSettings(); } + /// enforce the condition on multipart completion. `attempt_no` is the engine's own 1-based + /// physical-attempt count (see `TransportAccess::attemptNo`), carried into + /// `object_storage_attempt_number` so the HTTP client sees a reissue as attempt >= 2. + WriteSettings conditionalWriteSettings(size_t attempt_no) const; + WriteSettings conditionalWriteSettingsForTest() const { return conditionalWriteSettings(/*attempt_no=*/1); } /// Override the emulated backend's wall clock for deterministic expiry tests. void setEmuNowNsForTest(uint64_t now_ns); /// Return the guarded per-key token-state size for expiry tests. @@ -199,24 +214,34 @@ class ObjectStorageBackend final : public Backend /// See the constructor: what the READ-class requests (read, head, list, remove) carry. const bool single_attempt_control_plane; const uint64_t attempt_timeout_ms; + const uint64_t connect_timeout_cap_ms; ObjectStorageRetryProfile controlPlaneProfile() const { return single_attempt_control_plane ? ObjectStorageRetryProfile::SingleAttempt : ObjectStorageRetryProfile::Default; } - /// The read settings a request carries: the native conditional dialect, plus the retry profile and - /// per-attempt bound its caller is entitled to. - ReadSettings readSettingsFor(ObjectStorageRetryProfile profile, uint64_t timeout_ms) const; + /// The control-request context every read-class primitive builds from `access.attemptNo()`: this + /// backend's own retry profile, attempt timeout and frozen connect cap, plus the caller's attempt + /// number -- see `ObjectStorageControlRequest`. + ObjectStorageControlRequest controlRequest(size_t attempt_no) const + { + return ObjectStorageControlRequest{ + .profile = controlPlaneProfile(), + .attempt_timeout_ms = attempt_timeout_ms, + .connect_timeout_cap_ms = connect_timeout_cap_ms, + .attempt_number = attempt_no}; + } + /// The read settings a request carries: the native conditional dialect, plus the control-request + /// context (retry profile, per-attempt bound and connect cap, attempt number) its caller built. + ReadSettings readSettingsFor(const ObjectStorageControlRequest & request) const; - /// The bodies a keyed primitive and its legacy override share. They differ in one thing: the - /// keyed call passes `controlPlaneProfile(), attempt_timeout_ms`, the legacy one the storage's - /// defaults. The legacy arguments disappear with the legacy methods. - std::optional readUnder(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); - std::optional headUnder(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); + /// The keyed primitive's body, taking the control-request context its caller built + /// (`controlRequest(access.attemptNo())`) rather than deriving it again here. + std::optional readUnder(const String & key, const ObjectStorageControlRequest & request); + std::optional headUnder(const String & key, const ObjectStorageControlRequest & request); RawListPage listUnder(const String & prefix, const String & cursor, size_t limit, - ObjectStorageRetryProfile profile, uint64_t timeout_ms); - RawRemoval removeUnder(const String & key, const String & expected_value, - ObjectStorageRetryProfile profile, uint64_t timeout_ms); - SentinelProbeResult probeSentinelUnder(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); + const ObjectStorageControlRequest & request); + RawRemoval removeUnder(const String & key, const String & expected_value, const ObjectStorageControlRequest & request); + SentinelProbeResult probeSentinelUnder(const String & key, const ObjectStorageControlRequest & request); /// EmulatedSingleProcess state: per-key {etag, disambiguator} — see emuMintToken. A successfully /// deleted entry is retained only while its etag is recent enough that an immediate recreate could /// land in the same mtime quantum. `deleteExact` erases already-old entries immediately and queues @@ -237,7 +262,7 @@ class ObjectStorageBackend final : public Backend /// Look up Native metadata and normalize the storage ETag or generation into an incarnation /// value. The value is returned as the store gave it: whether it IS an incarnation is judged by /// whoever can act on the answer, never here. - std::optional nativeHead(const String & key, ObjectStorageRetryProfile profile, uint64_t timeout_ms); + std::optional nativeHead(const String & key, const ObjectStorageControlRequest & request); /// Write a body with the condition already encoded in `ws`, finalize it, map a lost precondition /// onto `RawConflict`, and return the write response's own value on success -- normalized, and @@ -285,4 +310,12 @@ class ObjectStorageBackend final : public Backend String emuMintToken(const String & key, const String & etag, bool just_wrote); }; +/// Fail-closed programmer-error guard: a mount opens exactly one backend and one `CasRequestBudget` +/// together (`ContentAddressedMetadataStorage::openPoolView`), and the pool's lease arithmetic +/// (`validateCasRequestBudget`, `CasMountRuntime::admit`) is validated against the budget alone -- +/// never against the backend it hands to the request layer. If the two ever disagree, the backend +/// would silently outlive (or underlive) the envelope the lease math was checked against. Throws +/// `LOGICAL_ERROR` naming both values; called once at open, before `Pool::open`. +void ensureBackendMatchesBudget(const ObjectStorageBackend & backend, const CasRequestBudget & budget); + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp index 4f22c85364f5..ca00928ef9e6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.cpp @@ -1,5 +1,6 @@ #include +#include #include #include @@ -14,24 +15,57 @@ namespace ErrorCodes namespace DB::Cas { -void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms) +uint64_t CasRequestBudget::attemptEnvelopeMs() const { - /// Overflow-safe: `attempt_timeout_ms + lease_safety_margin_ms` could wrap uint64 for absurd config - /// values, which would make the sum spuriously small and the inequality below pass when it should - /// fail closed. Compare via subtraction against the (unsigned, so already non-negative) TTL instead - /// of computing the sum directly. - if (!(budget.attempt_timeout_ms < mount_lease_ttl_ms - && budget.lease_safety_margin_ms < mount_lease_ttl_ms - budget.attempt_timeout_ms)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "CAS request budget rejected: attempt_timeout_ms ({}) + lease_safety_margin_ms ({}) must be " - "strictly less than the mount lease TTL ({} ms). A writable mount refuses to open with " - "this budget.", - budget.attempt_timeout_ms, budget.lease_safety_margin_ms, mount_lease_ttl_ms); + /// The TCP connect and the TLS handshake each get one connect interval from Poco, so an HTTPS + /// attempt may spend two caps before any request I/O. Scheme-agnostic on purpose: conservative for + /// plain HTTP, exact for HTTPS. + const uint64_t cap = connect_timeout_cap_ms.value_or(0); + const uint64_t connects = cap > std::numeric_limits::max() / 2 ? std::numeric_limits::max() : 2 * cap; + return attempt_timeout_ms > std::numeric_limits::max() - connects + ? std::numeric_limits::max() + : attempt_timeout_ms + connects; +} +void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, + uint64_t mount_renew_period_ms, bool background_renewal) +{ + if (budget.attempt_timeout_ms == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS request budget rejected: attempt_timeout_ms must be at least 1; a zero would reserve nothing " + "while the request keeps the storage's own timeout"); + const uint64_t envelope = budget.attemptEnvelopeMs(); + /// Subtractions against the unsigned TTL: the sums could wrap for absurd values and read as small. + const bool one_envelope_fits = envelope < mount_lease_ttl_ms + && budget.lease_safety_margin_ms < mount_lease_ttl_ms - envelope; + if (!one_envelope_fits) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS request budget rejected: the attempt envelope ({} ms = attempt_timeout_ms {} + connect cap {}) " + "plus lease_safety_margin_ms ({}) must be strictly less than the mount lease TTL ({} ms). " + "A writable mount refuses to open with this budget.", + envelope, budget.attempt_timeout_ms, budget.connect_timeout_cap_ms.value_or(0), budget.lease_safety_margin_ms, mount_lease_ttl_ms); + if (background_renewal) + { + /// A renewal is a write: two envelopes (the attempt and its settlement read) after one period. + /// Saturating doubling first (matching the production horizon checks' own arithmetic), then + /// subtraction-based comparisons against the TTL -- no truncating division, so this enforces + /// exactly the inequality the exception message states, not an off-by-one-tighter one. + const uint64_t two_envelope = envelope > std::numeric_limits::max() / 2 + ? std::numeric_limits::max() : 2 * envelope; + const bool cadence_fits = mount_renew_period_ms < mount_lease_ttl_ms + && two_envelope < mount_lease_ttl_ms - mount_renew_period_ms + && budget.lease_safety_margin_ms < mount_lease_ttl_ms - mount_renew_period_ms - two_envelope; + if (!cadence_fits) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "CAS mount renewal cadence rejected: mount_renew_period_ms ({}) + 2 × attempt envelope ({} ms) + " + "lease_safety_margin_ms ({}) must be strictly less than the mount lease TTL ({} ms)", + mount_renew_period_ms, envelope, budget.lease_safety_margin_ms, mount_lease_ttl_ms); + } LOG_INFO(getLogger("CasRequestBudget"), - "CAS request budget in effect: attempt_timeout_ms={} lease_safety_margin_ms={} " + "CAS request budget in effect: attempt_timeout_ms={} connect_timeout_cap_ms={} envelope_ms={} lease_safety_margin_ms={} " "(mount_lease_ttl_ms={} mount_renew_period_ms={})", - budget.attempt_timeout_ms, budget.lease_safety_margin_ms, mount_lease_ttl_ms, mount_renew_period_ms); + budget.attempt_timeout_ms, budget.connect_timeout_cap_ms.value_or(0), envelope, budget.lease_safety_margin_ms, + mount_lease_ttl_ms, mount_renew_period_ms); } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h index ab6f6300666f..1d96e41b199d 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequestBudget.h @@ -1,13 +1,16 @@ #pragma once #include +#include namespace DB::Cas { -/// The limits a writable mount is configured with. `attempt_timeout_ms` and `lease_safety_margin_ms` -/// are what `CasMountRuntime::admit` measures a request against, and the three `recovery_retry_*` fields -/// bound a whole ref-table recovery; see `validateCasRequestBudget` for the relationship a writable -/// mount enforces at startup. +/// The limits a writable mount is configured with. `CasMountRuntime::admit` measures a request against +/// `lease_safety_margin_ms` plus a caller-supplied need expressed in attempt envelopes (one for an +/// ordinary attempt, TWO for a ref-log append's write-plus-settlement-read -- see +/// `CasMountRuntime::refAppendFenceOk`), and the three `recovery_retry_*` fields bound a whole +/// ref-table recovery; see `validateCasRequestBudget` for the relationship a writable mount enforces +/// at startup. struct CasRequestBudget { /// Maximum client wait budgeted for one HTTP attempt. The request contract reserves this before @@ -19,6 +22,17 @@ struct CasRequestBudget /// write fence's own deadline) is what actually gates lease-relative timing per attempt. uint64_t lease_safety_margin_ms = 2000; + /// The cap the single-attempt client puts on one TCP connect and again on one TLS handshake, + /// frozen when the pool opens as `min(disk connect_timeout_ms, attempt_timeout_ms)` (a configured + /// zero means unbounded and is normalized to the attempt timeout) so a later client reload cannot + /// widen the envelope. Empty when the storage has no S3 client: the envelope is the attempt alone. + std::optional connect_timeout_cap_ms = 1000; + + /// What ONE physical attempt is allowed to cost end to end: two connect caps (TCP, then TLS) plus + /// the attempt timeout, saturating. The request contract reserves this, not the bare attempt timeout, before + /// every attempt it starts. + uint64_t attemptEnvelopeMs() const; + /// Recovery-level retry (`CasRefLedger::ensureRefTableRecovered`): a whole ref-table recovery /// attempt (LIST + snapshot/log GETs + seal PUT) that fails with a transient NETWORK_ERROR is /// retried, with capped-exponential backoff, until this total wall-clock budget is spent — then the @@ -35,18 +49,18 @@ struct CasRequestBudget /// Startup validation: a writable mount refuses to open with an inconsistent budget rather than /// silently falling back to an unbounded or unsafe retry policy. Throws /// `BAD_ARGUMENTS` unless: -/// attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms -/// -/// `mount_renew_period_ms` takes no part in the inequality (the renewer keeps the fence deadline -/// refreshed well ahead of the TTL by construction) — it is accepted only so the effective-values log -/// line records the full picture in one place. +/// attemptEnvelopeMs() + lease_safety_margin_ms < mount_lease_ttl_ms +/// and, when `background_renewal` is true (the mount runs a background renewer): +/// mount_renew_period_ms + 2 × attemptEnvelopeMs() + lease_safety_margin_ms < mount_lease_ttl_ms +/// (a renewal is a write: two envelopes for the attempt and its settlement read). /// /// A successor mounting over an unclean predecessor waits at least one lease TTL, plus its /// materialization grace period, before trusting recovery listings. This is long enough for any /// conditional PUT still in flight at the predecessor to either land or be abandoned by its own /// exhausted retry budget. The predecessor's budget is constrained by -/// `attempt_timeout_ms + lease_safety_margin_ms < mount_lease_ttl_ms`, so no additional handover +/// `attemptEnvelopeMs() + lease_safety_margin_ms < mount_lease_ttl_ms`, so no additional handover /// check is needed here. -void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms); +void validateCasRequestBudget(const CasRequestBudget & budget, uint64_t mount_lease_ttl_ms, uint64_t mount_renew_period_ms, + bool background_renewal); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index 12b213eda4ae..898c0c8c1504 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -33,6 +33,7 @@ namespace ProfileEvents extern const Event CASRequestRefused; extern const Event CASRequestFenceLostPostWrite; extern const Event CASRequestConnectFailureHint; + extern const Event CASRequestFirstAttemptFuse; } namespace DB::ErrorCodes @@ -66,6 +67,11 @@ void recordConflictPause() ProfileEvents::increment(ProfileEvents::CASRequestConflictPause); } +void recordFirstAttemptFuse() +{ + ProfileEvents::increment(ProfileEvents::CASRequestFirstAttemptFuse); +} + } namespace @@ -255,6 +261,23 @@ bool isConnectFailureHint([[maybe_unused]] const std::exception & e) return false; } +bool isFirstAttemptFuseTimeout([[maybe_unused]] const std::exception & e, [[maybe_unused]] size_t attempt_no) +{ +#if USE_AWS_S3 + /// The connect-failure hint is checked FIRST: `connect timed out` belongs to that classifier, and + /// an attempt whose text matches both stays a hint, reissued without a preceding settle read. + if (attempt_no != 1 || isConnectFailureHint(e)) + return false; + const auto * s3 = dynamic_cast(&e); + if (!s3 || s3->getS3ErrorCode() != Aws::S3::S3Errors::NETWORK_CONNECTION) + return false; + const std::string_view message = s3->message(); + return message.find("Timeout") != std::string_view::npos; +#else + return false; +#endif +} + CasRequests::CasRequests(BackendPtr backend_, Fence fence_, std::function now_ms_, std::function sleep_ms_, CasHotKeys * hot_keys_) @@ -262,7 +285,7 @@ CasRequests::CasRequests(BackendPtr backend_, Fence fence_, , fence(std::move(fence_)) , now_ms(now_ms_ ? std::move(now_ms_) : std::function(bootClockMs)) , sleep_ms(sleep_ms_ ? std::move(sleep_ms_) : std::function(sleepForMilliseconds)) - , attempt_reservation_ms(backend->attemptTimeoutMs()) + , attempt_reservation_ms(backend->attemptEnvelopeMs()) , own_hot_keys(hot_keys_ ? nullptr : std::make_unique(0)) , hot_keys(hot_keys_ ? hot_keys_ : own_hot_keys.get()) { @@ -430,7 +453,7 @@ bool CasOperation::fits(uint64_t needed_ms, const Retry::Bound & bound) const return needed_ms <= bound.deadline_ms - now; } -bool CasOperation::refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted) +bool CasOperation::refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted, bool & refreshed) { if (const auto * db_e = dynamic_cast(&e); db_e && isDeterministicLocalFailure(db_e->code())) return true; @@ -448,7 +471,8 @@ bool CasOperation::refreshAndClassifyReadFault(const std::exception & e, bool & if (refresh_attempted) return true; refresh_attempted = true; - return !owner.backend->refreshCredentials(); + refreshed = owner.backend->refreshCredentials(); + return !refreshed; } /// The store's own answer decides. A refusal it proved never applied, and an authoritative absence, /// both replay identically; everything else -- a throttle, a 5xx, a missing bucket, an unmodeled @@ -652,14 +676,17 @@ SentinelProbeResult CasOperation::probeSentinel(const String & key, const Retry SentinelProbeResult result{ProbeOutcome::Indeterminate, std::nullopt}; try { - result = owner.withTransportAccess([&](auto & access) + result = owner.withTransportAccess(attempt, [&](auto & access) { return owner.backend->probeSentinelRaw(key, access); }); } catch (const std::exception & e) { - if (refreshAndClassifyReadFault(e, refresh_attempted)) + /// The probe never counts a fuse, so whether this fault was a credential reissue is not + /// interesting here -- only the write and read loops guard a counter with it. + bool refreshed = false; + if (refreshAndClassifyReadFault(e, refresh_attempted, refreshed)) throw; /// The probe reports every transport failure as `Indeterminate` rather than by throwing, so /// a decorator that does throw is folded onto the same inconclusive outcome. @@ -815,10 +842,10 @@ WriteResult CasOperation::postCommit(Etag inc, bool resolved_by_read, WriteState return Committed{std::move(inc), state.attempts_sent, resolved_by_read}; } -std::optional CasOperation::pauseAndReissue(WriteState & state, const Retry::Bound & bound) +std::optional CasOperation::gatedPause(uint64_t pause_ms, uint32_t envelopes, WriteState & state, + const Retry::Bound & bound, void (*record)(), bool should_sleep) { - const uint64_t pause_ms = Retry::backoff(++state.reissues); - const uint64_t needed = reservedFor(pause_ms, 2); + const uint64_t needed = reservedFor(pause_ms, envelopes); switch (gate(needed)) { case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); @@ -827,26 +854,25 @@ std::optional CasOperation::pauseAndReissue(WriteState & state, con } if (!fits(needed, bound)) return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); - detail::recordReissue(); - owner.sleep_ms(pause_ms); + record(); + /// `should_sleep` is false ONLY for the fuse's zero-pause reissue: that pause is not "zero + /// milliseconds", it is NO PAUSE AT ALL, so it must not call `sleep_ms` even with a zero argument. + /// The three ordinary callers always sleep, even when a drawn jitter happens to be exactly zero -- + /// `Retry::backoff`'s full jitter includes zero -- so their call to `sleep_ms(0)` is kept + /// unconditional here to leave their observable behaviour exactly as it was before this collapse. + if (should_sleep) + owner.sleep_ms(pause_ms); return std::nullopt; } +std::optional CasOperation::pauseAndReissue(WriteState & state, const Retry::Bound & bound) +{ + return gatedPause(Retry::backoff(++state.reissues), 2, state, bound, detail::recordReissue, /*should_sleep=*/true); +} + std::optional CasOperation::pauseForConflict(WriteState & state, const Retry::Bound & bound) { - const uint64_t pause_ms = Retry::conflictBackoff(); - const uint64_t needed = reservedFor(pause_ms, 2); - switch (gate(needed)) - { - case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); - case Gate::NoBudget: return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); - case Gate::Ok: break; - } - if (!fits(needed, bound)) - return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); - detail::recordConflictPause(); - owner.sleep_ms(pause_ms); - return std::nullopt; + return gatedPause(Retry::conflictBackoff(), 2, state, bound, detail::recordConflictPause, /*should_sleep=*/true); } /// A flat pause before reissuing an attempt whose failure text named a failed connection. @@ -854,18 +880,12 @@ static constexpr uint64_t kConnectHintPauseMs = 50; std::optional CasOperation::pauseFlat(WriteState & state, const Retry::Bound & bound) { - const uint64_t needed = reservedFor(kConnectHintPauseMs, 2); - switch (gate(needed)) - { - case Gate::FenceLost: return gaveUp(GaveUp::Why::FenceLost, sourceFor(bound), state); - case Gate::NoBudget: return gaveUp(GaveUp::Why::Deadline, GaveUp::Source::Lease, state); - case Gate::Ok: break; - } - if (!fits(needed, bound)) - return gaveUp(GaveUp::Why::Deadline, sourceFor(bound), state); - detail::recordReissue(); - owner.sleep_ms(kConnectHintPauseMs); - return std::nullopt; + return gatedPause(kConnectHintPauseMs, 2, state, bound, detail::recordReissue, /*should_sleep=*/true); +} + +std::optional CasOperation::reissueAtOnce(WriteState & state, const Retry::Bound & bound) +{ + return gatedPause(0, 2, state, bound, detail::recordReissue, /*should_sleep=*/false); } WriteResult CasOperation::writeLoop(const String & key, const String & bytes, const std::optional & expected, @@ -907,10 +927,17 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// instead of resolved by a read and reissued to the deadline. bool credential_answer = false; bool refreshed = false; + /// Set once `refreshed` is known: true only when the credential-owned reissue below (the one + /// guarded by this exact expression) is what will actually resend this attempt. An earlier + /// ambiguity of this inner write routes the reissue through the ordinary hint/fuse/backoff + /// mechanisms instead -- the credential answer never gets to skip their read or their pacing -- + /// so a hint or fuse counter must still count in that case even though credentials were refreshed. + bool refresh_owns_reissue = false; bool connect_hint = false; + bool fuse = false; try { - outcome = owner.withTransportAccess([&](auto & access) + outcome = owner.withTransportAccess(state.attempts_sent, [&](auto & access) { return owner.backend->write(key, bytes, expected_value, access); }); @@ -931,6 +958,14 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co state.refresh_attempted = true; refreshed = owner.backend->refreshCredentials(); } + /// Mirrors the condition guarding the credential-owned reissue below exactly, so the two + /// can never drift: `state.any_ambiguous` here is still this attempt's INCOMING value, + /// because the update below only fires when `!credential_answer`, which `refreshed` implies + /// false for. `!policy.single_attempt` is redundant today -- `refreshed` can only be set + /// above under `!policy.single_attempt` already -- but it is kept so this stays an exact + /// copy of the reissue branch's condition rather than a hand-simplified one that could + /// silently stop matching it. + refresh_owns_reissue = refreshed && !policy.single_attempt && !state.any_ambiguous; /// A refusal that FOLLOWS an ambiguous attempt of this inner write proves nothing about that /// attempt, so it is settled by the read below instead of ending the call here. const bool definitely_refused = !refreshed && isDefinitelyRefusedWrite(e); @@ -941,10 +976,22 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co } /// A refusal-class exception is never a hint, even when an earlier ambiguity of this inner /// write kept it from ending the call above: that earlier attempt's fate is what the read - /// below must settle, and a hint reissue would skip it. + /// below must settle, and a hint reissue would skip it. Both counters below are recorded + /// here, at classification, regardless of `policy.single_attempt` (`Retry::once` never acts + /// on either, but the attempt's transport error still named what it named) -- except when + /// the credential answer OWNS the reissue: a credential answer whose text happens to also + /// match a hint or fuse text is a credential reissue, not a hint or fuse one, and must not + /// inflate these counts. When an earlier ambiguity of this inner write keeps the credential + /// answer from owning the reissue, the hint/fuse mechanism reissues it instead, exactly as + /// if credentials had never been refreshed, so the counter must still count it. connect_hint = !definitely_refused && isConnectFailureHint(e); - if (connect_hint) + if (connect_hint && !refresh_owns_reissue) ProfileEvents::increment(ProfileEvents::CASRequestConnectFailureHint); + /// Checked AFTER the hint, so a hinted attempt stays hinted (reissued before its read); a + /// fuse timeout is reissued after the settle read runs below. + fuse = isFirstAttemptFuseTimeout(e, state.attempts_sent); + if (fuse && !refresh_owns_reissue) + ProfileEvents::increment(ProfileEvents::CASRequestFirstAttemptFuse); } catch (const std::exception & e) { @@ -970,7 +1017,7 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// Nothing for a read to settle: this attempt did not apply, and no EARLIER attempt of this /// inner write is unresolved either. Re-send it under the credentials the refresh installed. - if (refreshed && !policy.single_attempt && !state.any_ambiguous) + if (refresh_owns_reissue) { if (auto given_up = pauseAndReissue(state, bound)) return *given_up; @@ -1030,6 +1077,16 @@ WriteResult CasOperation::writeLoop(const String & key, const String & bytes, co /// the store's own answer. A policy with no reissue has to say it settled nothing. if (policy.single_attempt) return gaveUp(GaveUp::Why::Unresolved, sourceFor(bound), state); + /// The first attempt met the adaptive first-attempt timeout: a connection-quality answer, not a + /// store fault. The read above settled nothing new about it, so re-send at once as attempt 2 + /// under the full attempt budget; the backoff index is untouched because no store fault was + /// seen yet. + if (fuse) + { + if (auto given_up = reissueAtOnce(state, bound)) + return *given_up; + continue; + } if (auto given_up = pauseAndReissue(state, bound)) return *given_up; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index 233b8404083a..dd79f1a9db4b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -36,6 +36,14 @@ bool isDefinitelyRefusedWrite(const std::exception & e); /// ambiguous one; what the hint changes is only that the engine reissues before spending a read. bool isConnectFailureHint(const std::exception & e); +/// TRUE when a FIRST physical attempt (`attempt_no == 1`) failed with the adaptive first-attempt +/// timeout: an `S3Exception` naming `NETWORK_CONNECTION` whose text is the generic transport-timeout +/// one, not a connect-failure hint (checked first, so a hinted attempt stays hinted -- the failed +/// connection it names is a different condition from the fuse, and must not be claimed by it). A +/// connection-quality answer about a fresh connection, not a store fault -- attempt 2 runs under the +/// full attempt budget, so the right response is to re-send at once rather than pace it like a fault. +bool isFirstAttemptFuseTimeout(const std::exception & e, size_t attempt_no); + /// Deterministic caller/local bugs, surfaced unchanged by every loop here: reissuing only replays the /// same failure and buries the root cause behind a retryable exception. The set is `LOGICAL_ERROR`, /// `NOT_IMPLEMENTED`, `BAD_ARGUMENTS` and `CORRUPTED_DATA`. @@ -130,6 +138,7 @@ namespace detail void recordAttempt(); void recordReissue(); void recordConflictPause(); +void recordFirstAttemptFuse(); } class CasOperation; @@ -148,7 +157,8 @@ class CasRequests /// `now_ms` defaults to `CLOCK_BOOTTIME` milliseconds -- the same clock a mount lease deadline is /// expressed on, so `Retry::untilLeaseSafe` and this engine compare like with like. `sleep_ms` /// defaults to a real sleep. `attempt_reservation_ms` is taken from the backend's own attempt - /// timeout: it is what the engine reserves before it starts anything. `hot_keys` is the pool's + /// envelope (attempt timeout plus its connect caps): it is what the engine reserves before it + /// starts anything. `hot_keys` is the pool's /// write lane, shared by its planes; without one this object owns a private lane with no cache, /// so a write through it costs today's read and write. CasRequests(BackendPtr backend_, Fence fence_, @@ -177,6 +187,11 @@ class CasRequests void setSleepFnForTest(std::function sleep_ms_); void setAttemptReservationForTest(uint64_t ms) { attempt_reservation_ms = ms; } + /// What every write on this plane reserves before it starts an attempt (the backend's own attempt + /// envelope). A caller that derives its OWN policy window from a write's cost -- rather than from a + /// constant that predates this reservation -- reads it here instead of duplicating the backend call. + uint64_t attemptReservationMs() const { return attempt_reservation_ms; } + private: friend class CasOperation; /// `CasOperation::owner` is a `CasRequests &`: `CasOperation`'s own friendship with `CasHotKeys` @@ -186,10 +201,12 @@ class CasRequests /// The one place a transport key is created. Every verb reaches the store through this, so no /// engine code -- and nothing outside it -- can name the key's type, let alone construct one. + /// `attempt_no` is the caller's own 1-based physical-attempt count, threaded to the transport + /// through `TransportAccess::attemptNo()` so a reissue is seen as attempt >= 2. template - auto withTransportAccess(Fn && fn) + auto withTransportAccess(size_t attempt_no, Fn && fn) { - TransportAccess access; + TransportAccess access(attempt_no); return std::forward(fn)(access); } @@ -380,6 +397,13 @@ class CasOperation /// state the read never saw, which is how a lease refusal used to be reported as a policy deadline. WriteResult gaveUpAfterFailedObservation(std::optional stop, WriteState & state, const Retry::Bound & bound) const; + /// The shared shape behind every gated pause below: admission for `envelopes` attempt reservations + /// plus `pause_ms`, the deadline check, the counter this pause records itself under, then the sleep + /// -- called even with a zero `pause_ms` UNLESS `should_sleep` is false, which is reserved for the + /// fuse's zero-pause reissue (a pace of "no pause", not "a zero-length one"). A value means the call + /// ended during it; nullopt means the caller may send another attempt. + std::optional gatedPause(uint64_t pause_ms, uint32_t envelopes, WriteState & state, + const Retry::Bound & bound, void (*record)(), bool should_sleep); /// Admission, then the jittered sleep. A value means the call ended during it; nullopt means the /// caller may send another attempt. std::optional pauseAndReissue(WriteState & state, const Retry::Bound & bound); @@ -390,6 +414,10 @@ class CasOperation /// The sibling for a failure text that named a failed connection. The same admission and the same /// reservation, a flat `kConnectHintPauseMs` sleep, and `state.reissues` untouched. std::optional pauseFlat(WriteState & state, const Retry::Bound & bound); + /// The sibling for a first-attempt fuse timeout: the same admission and the same reservation, NO + /// sleep at all, and `state.reissues` untouched -- the fuse is a connection-quality answer about a + /// fresh connection, not a store fault, so nothing here is paced against it. + std::optional reissueAtOnce(WriteState & state, const Retry::Bound & bound); /// `sleep_ms` plus `envelopes` attempt reservations, saturating. uint64_t reservedFor(uint64_t sleep_ms, uint32_t envelopes) const; @@ -400,8 +428,10 @@ class CasOperation /// One failed read-class attempt, classified. A credential failure is refreshed HERE so the reissue /// signs with the new client, at most once per call -- `refresh_attempted` is the caller's, and a /// second credential failure under the same call is classified as if no refresh were available. - /// TRUE means the failure must surface unchanged. - bool refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted); + /// `refreshed` is set true only when THIS call installed new credentials, mirroring the write + /// loop's own local of the same name, so the caller can keep a credential reissue from also + /// inflating a counter whose text it happens to match. TRUE means the failure must surface unchanged. + bool refreshAndClassifyReadFault(const std::exception & e, bool & refresh_attempted, bool & refreshed); /// Each records its cause in `last_read_stop` before throwing, so the resolve read can report it. [[noreturn]] void giveUpReadFenceLost(std::string_view verb, const String & subject, std::string_view when); @@ -422,7 +452,12 @@ auto CasOperation::readLoop(std::string_view verb, const String & subject, const const Retry::Bound & bound, Fn && once) { bool refresh_attempted = false; - for (uint32_t attempt = 1;; ++attempt) + /// Two counters, deliberately kept separate: `attempt_no` is the PHYSICAL attempt count handed to + /// the transport (so a reissue is seen as attempt >= 2); `ordinary_reissues` is the + /// exponential-backoff index. They advance together on an ordinary failure, but the first-attempt + /// fuse below advances `attempt_no` alone (via `continue`, skipping the backoff pause) so a + /// following ordinary failure's backoff still starts from `backoff(1)`, undisturbed. + for (uint32_t attempt_no = 1, ordinary_reissues = 0;; ++attempt_no) { const uint64_t reservation = reservedFor(0, 1); switch (gate(reservation)) @@ -432,20 +467,49 @@ auto CasOperation::readLoop(std::string_view verb, const String & subject, const case Gate::Ok: break; } if (!fits(reservation, bound)) - giveUpReadDeadline(verb, subject, bound, attempt - 1); + giveUpReadDeadline(verb, subject, bound, attempt_no - 1); detail::recordAttempt(); try { - return owner.withTransportAccess([&](auto & access) { return once(access); }); + return owner.withTransportAccess(attempt_no, [&](auto & access) { return once(access); }); } catch (const std::exception & e) { - if (refreshAndClassifyReadFault(e, refresh_attempted) || policy.single_attempt) + bool refreshed = false; + if (refreshAndClassifyReadFault(e, refresh_attempted, refreshed)) + throw; + /// Classified -- and counted -- before the single-attempt check below: `Retry::once` forbids + /// the REISSUE, not the observation that this attempt hit the fuse, and the write path + /// already counts at classification the same way -- except when `refreshed` is also true: a + /// credential answer whose text happens to also match the fuse text is a credential reissue, + /// not a fuse one, and must not inflate this count. + const bool fuse = isFirstAttemptFuseTimeout(e, attempt_no); + if (fuse && !refreshed) + detail::recordFirstAttemptFuse(); + if (policy.single_attempt) throw; + /// A first-attempt fuse is a connection-quality answer, not a store fault: re-check + /// admission for the reissue alone and send it at once. `ordinary_reissues` stays put, so a + /// following ordinary failure's backoff starts at `backoff(1)`, exactly as if this attempt + /// had never happened. + if (fuse) + { + const uint64_t needed = reservedFor(0, 1); + switch (gate(needed)) + { + case Gate::FenceLost: giveUpReadFenceLost(verb, subject, "before the reissue"); + case Gate::NoBudget: giveUpReadNoBudget(verb, subject, "for the reissue"); + case Gate::Ok: break; + } + if (!fits(needed, bound)) + giveUpReadDeadline(verb, subject, bound, attempt_no); + detail::recordReissue(); + continue; + } } - const uint64_t pause_ms = Retry::backoff(attempt); + const uint64_t pause_ms = Retry::backoff(++ordinary_reissues); const uint64_t needed = reservedFor(pause_ms, 1); switch (gate(needed)) { @@ -454,7 +518,7 @@ auto CasOperation::readLoop(std::string_view verb, const String & subject, const case Gate::Ok: break; } if (!fits(needed, bound)) - giveUpReadDeadline(verb, subject, bound, attempt); + giveUpReadDeadline(verb, subject, bound, attempt_no); detail::recordReissue(); owner.sleep_ms(pause_ms); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h index 6275eb7e41ca..de235e8fd3fa 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRetry.h @@ -39,12 +39,15 @@ struct Retry static Retry within(uint64_t ms) { return {.window_ms = ms, .lease_deadline_ms = std::nullopt, .single_attempt = false}; } /// `within(90'000)` -- the default write policy. static Retry standard() { return within(90'000); } - /// The standard policy, additionally bound by the mount lease: `lease_deadline_ms` minus - /// `margin`, clamped at 0 -- never risk a write landing after this node's fence may already be - /// gone. - static Retry untilLeaseSafe(uint64_t lease_deadline_ms, uint64_t margin) + /// A policy bound by the mount lease: `lease_deadline_ms` minus `margin`, clamped at 0 -- never + /// risk a write landing after this node's fence may already be gone. `window_ms` defaults to the + /// standard 90 s write budget; a caller whose own budget is deliberately much smaller (the + /// graceful-shutdown farewell, whose window is derived from what ONE write costs, not from the + /// standard policy) passes its own window explicitly, and `bind` still takes whichever of the two + /// bounds is smaller. + static Retry untilLeaseSafe(uint64_t lease_deadline_ms, uint64_t margin, uint64_t window_ms = 90'000) { - return {.window_ms = 90'000, + return {.window_ms = window_ms, .lease_deadline_ms = lease_deadline_ms > margin ? lease_deadline_ms - margin : 0, .single_attempt = false}; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h index 1153f0c78232..e620195a31c3 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h @@ -124,6 +124,7 @@ class ThrottlingBackend final : public Backend Dialect dialect() const override { return inner->dialect(); } bool supportsListTokens() const override { return inner->supportsListTokens(); } uint64_t attemptTimeoutMs() const override { return inner->attemptTimeoutMs(); } + uint64_t attemptEnvelopeMs() const override { return inner->attemptEnvelopeMs(); } bool refreshCredentials() override { return inner->refreshCredentials(); } void checkPoolPreconditions() override { inner->checkPoolPreconditions(); } void checkSkipAccessCheckSupport() override { inner->checkSkipAccessCheckSupport(); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h index 02ecdbc873cb..49a3bf419874 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasTransportAccess.h @@ -1,19 +1,25 @@ #pragma once +#include namespace DB::Cas { /// A capability token: holding one proves the holder is `CasRequests`. Not copyable, not -/// constructible outside that one friend, and carries no data -- its only job is to gate access at -/// compile time to the backend entry points that must not be called except through the contract. +/// constructible outside that one friend, and carries no data besides the engine's own attempt +/// count -- its main job is to gate access at compile time to the backend entry points that must +/// not be called except through the contract. class TransportAccess { friend class CasRequests; - TransportAccess() = default; + explicit TransportAccess(size_t attempt_no_) : attempt_no(attempt_no_) {} + size_t attempt_no; public: TransportAccess(const TransportAccess &) = delete; TransportAccess & operator=(const TransportAccess &) = delete; + /// The engine's 1-based count of physical attempts of the logical call this request belongs to, + /// for the transport to number its request with. Only "1 versus more than 1" is relied upon. + size_t attemptNo() const { return attempt_no; } }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index 0e53687f8924..fab17ff8eef0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -1,4 +1,5 @@ #include +#include "config.h" #include #include #include @@ -693,6 +694,23 @@ Cas::RebuildReport ContentAddressedMetadataStorage::runGcRebuildNow(bool force) return gc.rebuildBaseline(force); } +std::optional ContentAddressedMetadataStorage::freezeConnectTimeoutCapMs( + [[maybe_unused]] const ObjectStoragePtr & object_storage, [[maybe_unused]] uint64_t cas_attempt_timeout_ms) +{ +#if USE_AWS_S3 + const auto s3_client = object_storage->tryGetS3StorageClient(); + if (!s3_client) + return std::nullopt; + /// A configured zero is "unbounded" to Poco: the cap is then the attempt timeout itself. + const auto configured = static_cast(std::max(0, s3_client->getClientConfiguration().connectTimeoutMs)); + return configured == 0 ? cas_attempt_timeout_ms : std::min(configured, cas_attempt_timeout_ms); +#else + /// Without the AWS S3 client compiled in there is no S3 storage to read a connect timeout from, + /// so nothing is frozen: the same answer an S3-less object storage gets above. + return std::nullopt; +#endif +} + ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openPoolView(bool context_available) const { /// Native mode rides real conditional ops (probed fail-closed by Pool::open); Local object @@ -784,6 +802,9 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.gc_bulk_delete_chunk_keys = gc_bulk_delete_chunk_keys; pool_config.cas_request_budget.attempt_timeout_ms = cas_attempt_timeout_ms; pool_config.cas_request_budget.lease_safety_margin_ms = cas_lease_safety_margin_ms; + /// Frozen here, once: see `freezeConnectTimeoutCapMs`. A later reload that widens the disk's + /// connect timeout cannot widen the envelope the lease arithmetic was validated against. + pool_config.cas_request_budget.connect_timeout_cap_ms = freezeConnectTimeoutCapMs(object_storage, cas_attempt_timeout_ms); pool_config.mount_lease_ttl_ms = mount_lease_ttl; pool_config.mount_renew_period = mount_renew_period; pool_config.event_sink = makeCasEventSink(); @@ -795,7 +816,14 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP auto backend = std::make_shared( object_storage, mode, /*single_attempt_control_plane_=*/!read_only && mode == Cas::ObjectStorageBackend::Mode::Native, - pool_config.cas_request_budget.attempt_timeout_ms); + pool_config.cas_request_budget.attempt_timeout_ms, + pool_config.cas_request_budget.connect_timeout_cap_ms.value_or(0)); + + /// A programmer error, not an input one: the handoff above is code, not configuration. The pool's + /// lease arithmetic was validated against `pool_config.cas_request_budget` alone (never against the + /// backend), so a backend built with a different cap or timeout would silently outlive it. Fail + /// closed before `Pool::open` rather than trust the two stayed in sync. + Cas::ensureBackendMatchesBudget(*backend, pool_config.cas_request_budget); PoolView view; view.physical_key_prefix = physical_key_prefix_local; @@ -983,7 +1011,7 @@ void ContentAddressedMetadataStorage::stopAndDrainForTeardown() noexcept if (!old_pool) return; const auto & budget = old_pool->poolConfig().cas_request_budget; - const uint64_t deadline_ms = budget.attempt_timeout_ms + budget.lease_safety_margin_ms; + const uint64_t deadline_ms = budget.attemptEnvelopeMs() + budget.lease_safety_margin_ms; if (old_pool->stopAndDrainDetachedWork(deadline_ms)) return; ProfileEvents::increment(ProfileEvents::CASDetachedWorkDrainTimeouts); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index f4de160fea2f..65b6349e7367 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -183,6 +183,17 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC /// observe the detached-work stop latch without going through the lifecycle gate. Cas::PoolPtr poolForTest() const; + /// What `openPoolView` freezes into `pool_config.cas_request_budget.connect_timeout_cap_ms`: the + /// connect cap every control request and every single-attempt clone will carry for the life of + /// this mount. `nullopt` when `object_storage` has no S3 client (the envelope is then the attempt + /// alone); otherwise `min(disk connect_timeout_ms, cas_attempt_timeout_ms)`, with a configured zero + /// (Poco's "unbounded") normalized to `cas_attempt_timeout_ms` rather than treated as no limit. + /// Exposed here (not test-only) because it is a pure read of already-public state -- freezing it in + /// one place, unit-testable without opening a pool, is what keeps a later client reload from + /// widening the envelope the lease arithmetic was validated against. + static std::optional freezeConnectTimeoutCapMs( + const ObjectStoragePtr & object_storage, uint64_t cas_attempt_timeout_ms); + /// Runs one synchronous GC round on the caller's thread and emits Start and Finish rows to /// `system.cas_gc_log`. Throws `BAD_ARGUMENTS` when GC is disabled /// by read-only mode or configuration. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index 9ccbabc9206c..29b9d12e37f6 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -80,8 +80,8 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, gc_meta_pool_size, 16, "Bounded pool size for GC per-hash freshness-meta writes", 0) \ DECLARE(UInt64, gc_read_concurrency, 16, "Bounded pool size for the GC fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate HEADs; 1 disables read-ahead", 0) \ DECLARE(UInt64, gc_bulk_delete_chunk_keys, 1000, "Keys per batch delete request in GC's write-once families (owner-removed manifest bodies, covered ref logs and snapshots); 1 to 1000", 0) \ - DECLARE(UInt64, attempt_timeout_ms, 5000, "Budget for one HTTP attempt of a writable Native mount's control-plane requests", 0) \ - DECLARE(UInt64, lease_safety_margin_ms, 2000, "Startup-only margin validated against the mount lease TTL (attempt_timeout_ms + this must be strictly less than the lease TTL)", 0) \ + DECLARE(UInt64, attempt_timeout_ms, 5000, "Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove, conditional write), at least 1. With the connect cap it forms the attempt envelope the lease arithmetic reserves", 0) \ + DECLARE(UInt64, lease_safety_margin_ms, 2000, "Startup-only margin validated against the mount lease TTL: attempt envelope + this must be strictly less than the TTL, and renew period + 2 × envelope + this too", 0) \ DECLARE(String, staging_backend, "local", "Blob staging backend (local | s3); s3 is opt-in", 0) \ DECLARE_SETTINGS_TRAITS(ContentAddressedSettingsTraits, LIST_OF_CONTENT_ADDRESSED_SETTINGS, CONTENT_ADDRESSED_SETTINGS_SUPPORTED_TYPES) @@ -241,6 +241,11 @@ void ContentAddressedSettings::validate() "content_addressed disk: gc_bulk_delete_chunk_keys must be between 1 and 1000 (got {})", settings[ContentAddressedSetting::gc_bulk_delete_chunk_keys].value); + if (settings[ContentAddressedSetting::attempt_timeout_ms] == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "content_addressed disk: cas_attempt_timeout_ms must be >= 1 (got {})", + settings[ContentAddressedSetting::attempt_timeout_ms].value); + if (settings[ContentAddressedSetting::mount_lease_ttl_ms] == 0) throw Exception(ErrorCodes::BAD_ARGUMENTS, "content_addressed disk: cas_mount_lease_ttl_ms must be >= 1 (got {})", diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp index 700fed37dafa..e3f52fd7957a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp @@ -153,9 +153,13 @@ Fence::Admit CasMountRuntime::admit(uint64_t admitted_generation, uint64_t neede bool CasMountRuntime::refAppendFenceOk() const { - /// One attempt's worth of room under the live generation: a ref-log attempt is not started when it - /// cannot plausibly finish, safety margin included, before the lease expires. - return admit(fenceGeneration(), cas_request_budget.attempt_timeout_ms) == Fence::Admit::Ok; + /// Two envelopes' worth of room under the live generation -- a write and its settlement read, which + /// is what `writeLoop` reserves -- so a ref-log attempt is not started when it cannot plausibly + /// finish, safety margin included, before the lease expires. + const uint64_t envelope_ms = cas_request_budget.attemptEnvelopeMs(); + const uint64_t needed_ms = envelope_ms > std::numeric_limits::max() / 2 + ? std::numeric_limits::max() : 2 * envelope_ms; + return admit(fenceGeneration(), needed_ms) == Fence::Admit::Ok; } void CasMountRuntime::setMountDeadline(uint64_t deadline_boot_ms) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 5e637b72694b..4c3d6ad9cf22 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -88,21 +88,7 @@ void validateWritableMountTiming(const PoolConfig & config) "CAS mount timing rejected: lease TTL must be positive and renewal period non-negative"); const uint64_t ttl_ms = static_cast(config.mount_lease_ttl_ms.count()); const uint64_t period_ms = static_cast(config.mount_renew_period.count()); - validateCasRequestBudget(config.cas_request_budget, ttl_ms, period_ms); - if (!config.background_watermark) - return; - - const uint64_t safety_ms = config.cas_request_budget.lease_safety_margin_ms; - const uint64_t attempt_ms = config.cas_request_budget.attempt_timeout_ms; - const bool cadence_fits = safety_ms <= ttl_ms - && period_ms <= ttl_ms - safety_ms - && attempt_ms <= ttl_ms - safety_ms - period_ms; - if (!cadence_fits) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "CAS mount renewal cadence rejected: period ({} ms) + attempt timeout ({} ms) must be " - "at most TTL ({} ms) - safety margin ({} ms) when background renewal is enabled", - period_ms, attempt_ms, ttl_ms, safety_ms); + validateCasRequestBudget(config.cas_request_budget, ttl_ms, period_ms, config.background_watermark); } /// The verdict of the pool-lifecycle identity gate (step 0 of `tryRemountOnce`, spec §2). Exactly one @@ -838,14 +824,20 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol : claim_anchor_boot_ms + ttl_ms_u - safety_ms; const uint64_t now_boot_ms = store->bootMsNow(); const uint64_t period_ms = static_cast(store->config.mount_renew_period.count()); + const uint64_t envelope_ms = store->config.cas_request_budget.attemptEnvelopeMs(); + const uint64_t two_envelopes_ms = envelope_ms > std::numeric_limits::max() / 2 + ? std::numeric_limits::max() : 2 * envelope_ms; const uint64_t renewal_window_ms = store->config.background_watermark - ? period_ms + store->config.cas_request_budget.attempt_timeout_ms - : store->config.cas_request_budget.attempt_timeout_ms; - /// Preserve one ordinary cadence followed by one physical renewal attempt inside the safe lease - /// window. If that publication horizon was consumed, re-anchor synchronously before opening the - /// fence; the renewer independently retains its per-request deadline checks. + ? (two_envelopes_ms > std::numeric_limits::max() - period_ms + ? std::numeric_limits::max() : period_ms + two_envelopes_ms) + : two_envelopes_ms; + /// Preserve one ordinary cadence followed by one physical renewal attempt (a write and its + /// settlement read: two envelopes) inside the safe lease window. If that publication horizon was + /// consumed, re-anchor synchronously before opening the fence; the renewer independently retains + /// its per-request deadline checks. STRICT, like `CasMountRuntime::admit`: a horizon that fits + /// exactly still starts a renewal the fence would then refuse. const bool renewal_window_fits = now_boot_ms <= safe_deadline - && renewal_window_ms <= safe_deadline - now_boot_ms; + && renewal_window_ms < safe_deadline - now_boot_ms; if (!renewal_window_fits) { /// The claim path outlived the lease TTL: its anchor can no longer authorize an armed fence (a @@ -1001,7 +993,7 @@ Pool::~Pool() if (config.teardown_phase2_throw_for_test) config.teardown_phase2_throw_for_test(); const bool ref_lanes_drained = ref_ledger.drainRefLanesForShutdown( - config.cas_request_budget.attempt_timeout_ms + config.cas_request_budget.lease_safety_margin_ms); + config.cas_request_budget.attemptEnvelopeMs() + config.cas_request_budget.lease_safety_margin_ms); drained = ref_lanes_drained && !writerCleanupDutiesPending(); }, "CAS pool teardown: draining ref lanes"); @@ -1094,10 +1086,9 @@ bool Pool::detachedWorkStoppingForTest() const return teardownBegun(); } -void Pool::setDetachedDrainDeadlineBudgetForTest(uint64_t attempt_timeout_ms, uint64_t lease_safety_margin_ms) +void Pool::setDetachedDrainDeadlineBudgetForTest(const CasRequestBudget & budget) { - config.cas_request_budget.attempt_timeout_ms = attempt_timeout_ms; - config.cas_request_budget.lease_safety_margin_ms = lease_safety_margin_ms; + config.cas_request_budget = budget; } void Pool::forgetDisk(const std::function & stop_and_join_gc, const String & reason) @@ -1157,7 +1148,7 @@ void Pool::forgetDisk(const std::function & stop_and_join_gc, const Stri /// (5b) Drain the ref lanes (bounded by one attempt's budget + safety margin) to learn whether a clean /// farewell is EARNED — exactly the `~Pool` rule. const bool ref_lanes_drained = ref_ledger.drainRefLanesForShutdown( - config.cas_request_budget.attempt_timeout_ms + config.cas_request_budget.lease_safety_margin_ms); + config.cas_request_budget.attemptEnvelopeMs() + config.cas_request_budget.lease_safety_margin_ms); const bool drained = ref_lanes_drained && !writerCleanupDutiesPending(); /// (3+5c) Retire the merged heartbeat: a clean-release farewell ONLY if the lanes provably drained, @@ -1503,11 +1494,17 @@ bool Pool::tryRemountOnce() : remount_anchor_boot_ms + ttl_ms - safety_ms; const uint64_t now_boot_ms = mount_runtime.bootMsNow(); const uint64_t period_ms = static_cast(config.mount_renew_period.count()); + const uint64_t envelope_ms = config.cas_request_budget.attemptEnvelopeMs(); + const uint64_t two_envelopes_ms = envelope_ms > std::numeric_limits::max() / 2 + ? std::numeric_limits::max() : 2 * envelope_ms; const uint64_t renewal_window_ms = config.background_watermark - ? period_ms + config.cas_request_budget.attempt_timeout_ms - : config.cas_request_budget.attempt_timeout_ms; + ? (two_envelopes_ms > std::numeric_limits::max() - period_ms + ? std::numeric_limits::max() : period_ms + two_envelopes_ms) + : two_envelopes_ms; + /// STRICT, like the open path and `CasMountRuntime::admit`: a horizon that fits exactly still + /// starts a renewal the fence would then refuse. const bool renewal_window_fits = now_boot_ms <= safe_deadline - && renewal_window_ms <= safe_deadline - now_boot_ms; + && renewal_window_ms < safe_deadline - now_boot_ms; if (!renewal_window_fits) { step = "renewer_redo"; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 7354fd4d08ea..5e2b74e7ac91 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -428,7 +428,10 @@ class Pool : public std::enable_shared_from_this bool detachedWorkStoppingForTest() const; /// Test-only: shortens the metadata-storage teardown deadline without changing any request path. /// Call before dispatching detached work; production configuration remains immutable after open. - void setDetachedDrainDeadlineBudgetForTest(uint64_t attempt_timeout_ms, uint64_t lease_safety_margin_ms); + /// Replaces the whole budget (not just `attempt_timeout_ms`) because the drain deadline is read + /// from `attemptEnvelopeMs()`, which also folds in `connect_timeout_cap_ms` -- a caller that only + /// overrode the attempt timeout would silently keep whatever connect cap the pool froze at open. + void setDetachedDrainDeadlineBudgetForTest(const CasRequestBudget & budget); /// ---- per-server watermark surface ---- /// process_epoch: random nonzero per Pool (process). GC checks epoch EQUALITY, never ordering. @@ -892,8 +895,8 @@ class Pool : public std::enable_shared_from_this void cancelInflightBuildsForNamespace(const RootNamespace & ns); /// Delegate to `mount_runtime`: the write fence moved there. Extends `mayMutate` with the REMAINING - /// budget check -- work is not started unless there is enough of the mount lease left for one more - /// attempt timeout plus the safety margin. + /// budget check -- work is not started unless there is enough of the mount lease left for TWO full + /// attempt envelopes (a write and its settlement read) plus the safety margin. bool refAppendFenceOk() const; /// incidental-detection reaction for a foreign-interference diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 44b4b55f8e83..50e525b7d31c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -1283,10 +1283,23 @@ bool isCreatorFenceTerminal(CasOperation & op, const Layout & layout, const Stri return terminal; } -/// The farewell's whole budget. It is deliberately short: a departing mount is holding shutdown open, -/// and a slot it fails to hand back is fenced out by the next GC round anyway. +/// The farewell's FLOOR, not its whole budget: a departing mount is holding shutdown open, so the +/// window still wants to be short, but it can never be shorter than what the farewell's own write +/// needs to send even one attempt. `terminate` below takes the larger of this and that requirement. +/// A window below the requirement is strictly worse than a slightly longer shutdown: the write is +/// refused before it tries the wire, the slot is left holding the departing incarnation, and the next +/// start pays a full incarnation-stability observation (up to the mount lease TTL) instead of +/// reclaiming instantly off a clean farewell. constexpr uint64_t kFarewellBudgetMs = 10'000; +/// Slack added on top of the write's bare two-envelope reservation (see `terminate`). `fits` admits a +/// write whose reservation exactly equals the remaining window, but only at the instant it is checked; +/// with zero slack the farewell would be admitted only to immediately re-fail its own deadline check +/// once the clock advances by even one millisecond. This mirrors `lease_safety_margin_ms`'s default +/// (`CasRequestBudget.h`) -- the same order of magnitude already trusted elsewhere on this path for +/// "admission-time arithmetic needs room to actually run, not just to pass at t=0". +constexpr uint64_t kFarewellSlackMs = 2'000; + MountLeaseRenewer::MountLeaseRenewer( CasRequests & mount_requests_, CasRequests & open_requests_, const Layout & layout_, const String & srid_, UInt128 server_uuid_, @@ -1712,7 +1725,33 @@ void MountLeaseRenewer::terminate(CasOperation & op) .min_active_build_sequence = std::numeric_limits::max(), .write_attempt_id = newMountWriteAttemptId(), }); - WriteResult written = op.replace(key, body, precondition(), Retry::within(kFarewellBudgetMs)); + /// The farewell is admitted on `open_requests` (see `release`, which calls this via `open_requests.admit()`), + /// so its own reservation -- attempt plus the read that settles it, `reservedFor(0, 2)` in + /// `CasOperation::writeLoop` -- is exactly `2 * open_requests.attemptReservationMs()`. A window + /// below that value refuses the write before its first attempt, deterministically, on every call: + /// `kFarewellBudgetMs` alone predates the attempt-envelope reservation and can no longer be trusted + /// to admit it. Saturating, like every other deadline computation on this path (see the + /// `expires_at_ms`/`confirmed_deadline_boot_ms` arithmetic above): an operator-configured envelope + /// is not bounds-checked against this doubling, and wrapping past `UINT64_MAX` would turn a too-long + /// window into a too-SHORT one -- the exact failure mode this fix exists to remove. + const uint64_t reservation_ms = open_requests.attemptReservationMs(); + const uint64_t doubled_reservation_ms = reservation_ms > std::numeric_limits::max() / 2 + ? std::numeric_limits::max() + : reservation_ms * 2; + const uint64_t two_envelope_reservation_plus_slack_ms = doubled_reservation_ms > std::numeric_limits::max() - kFarewellSlackMs + ? std::numeric_limits::max() + : doubled_reservation_ms + kFarewellSlackMs; + const uint64_t farewell_window_ms = std::max(kFarewellBudgetMs, two_envelope_reservation_plus_slack_ms); + /// The derived window alone is not enough: mount-control activity must also never run past the + /// point this node's own fence may already be gone (the same rule `renew` enforces via + /// `Retry::untilLeaseSafe` above). The precondition on this write already stops it from clobbering + /// a successor if it DOES land late, but a shutdown holding the process open to retry a write past + /// its own lease-safe deadline serves no one -- the successor's own reclaim does not wait for it. + /// `confirmed_deadline_boot_ms` is set at `start()` and kept current by every successful `renew`, + /// so it is valid here whenever `terminate` runs (only reachable from `release`, which requires + /// `Active`, which `start` alone establishes). + WriteResult written = op.replace(key, body, precondition(), + Retry::untilLeaseSafe(confirmed_deadline_boot_ms, static_cast(lease_safety_margin.count()), farewell_window_ms)); if (Committed * committed = std::get_if(&written)) { diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp index 580b77941a55..a64f9a886ee9 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp @@ -72,31 +72,30 @@ ObjectStorageIteratorPtr IObjectStorage::iterate( size_t max_keys, bool with_tags, const std::optional & start_after, - ObjectStorageRetryProfile profile, - uint64_t /*request_timeout_ms*/) const + const ObjectStorageControlRequest & request) const { - if (profile == ObjectStorageRetryProfile::SingleAttempt) + if (request.profile == ObjectStorageRetryProfile::SingleAttempt) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support single-attempt listing requests", getName()); return iterate(path_prefix, max_keys, with_tags, start_after); } std::optional IObjectStorage::tryGetObjectMetadataWithNativeToken( - const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t /*request_timeout_ms*/) const + const std::string & path, bool with_tags, const ObjectStorageControlRequest & request) const { - if (profile == ObjectStorageRetryProfile::SingleAttempt) + if (request.profile == ObjectStorageRetryProfile::SingleAttempt) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support single-attempt metadata requests", getName()); return tryGetObjectMetadataWithNativeToken(path, with_tags); } ConditionalRemoveResult IObjectStorage::removeObjectIfTokenMatches( - const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t /*request_timeout_ms*/) + const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request) { - if (profile == ObjectStorageRetryProfile::SingleAttempt) + if (request.profile == ObjectStorageRetryProfile::SingleAttempt) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support single-attempt removal requests", getName()); return removeObjectIfTokenMatches(object, etag); } -void IObjectStorage::removeObjectsIfExistUnderProfile(const StoredObjects &, ObjectStorageRetryProfile, uint64_t) +void IObjectStorage::removeObjectsIfExistUnderProfile(const StoredObjects &, const ObjectStorageControlRequest &) { throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support batch removal under a retry profile", getName()); } diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index c56ecac18575..eec112d27059 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -272,17 +272,16 @@ class IObjectStorage bool with_tags, const std::optional & start_after) const; - /// Same, under a chosen retry profile, with `request_timeout_ms` bounding one attempt of it - /// (0 = the storage's own timeout). A storage that cannot execute the profile must refuse: a - /// caller that asked for one attempt has its own deadline, and a transparently retried request - /// would outlive it. + /// Same, under a chosen control-request context (retry profile, one attempt's timeout and connect + /// cap, and the caller's own attempt number -- see `ObjectStorageControlRequest`). A storage that + /// cannot execute the profile must refuse: a caller that asked for one attempt has its own + /// deadline, and a transparently retried request would outlive it. virtual ObjectStorageIteratorPtr iterate( const std::string & path_prefix, size_t max_keys, bool with_tags, const std::optional & start_after, - ObjectStorageRetryProfile profile, - uint64_t request_timeout_ms) const; + const ObjectStorageControlRequest & request) const; /// Get object metadata if supported. It should be possible to receive at least size of object virtual ObjectMetadata getObjectMetadata(const std::string & path, bool with_tags) const = 0; @@ -298,9 +297,9 @@ class IObjectStorage return tryGetObjectMetadata(path, with_tags); } - /// Same, under a chosen retry profile; see the note on `iterate`. + /// Same, under a chosen control-request context; see the note on `iterate`. virtual std::optional tryGetObjectMetadataWithNativeToken( - const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const; + const std::string & path, bool with_tags, const ObjectStorageControlRequest & request) const; /// Read single object virtual std::unique_ptr readObject( /// NOLINT @@ -368,16 +367,16 @@ class IObjectStorage "Conditional (token-exact) object removal is not implemented for {} object storage", getName()); } - /// Same, under a chosen retry profile; see the note on `iterate`. + /// Same, under a chosen control-request context; see the note on `iterate`. virtual ConditionalRemoveResult removeObjectIfTokenMatches( - const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms); + const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request); /// Removes every object in ONE request with no per-key precondition; an absent object is success. /// Content-addressed callers use it for write-once keys only, at most 1000 per call. Throws on a /// request-level failure and on any per-key error other than "not found", naming the failed keys. - /// Same profile note as `iterate`. Backends without a batch delete keep the default, which refuses. + /// Same context note as `iterate`. Backends without a batch delete keep the default, which refuses. virtual void removeObjectsIfExistUnderProfile( - const StoredObjects & objects, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms); + const StoredObjects & objects, const ObjectStorageControlRequest & request); /// Copy object with different attributes if required virtual void copyObject( /// NOLINT diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 9f069239fe60..3ef301e5f8d8 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -158,7 +158,8 @@ class S3IteratorAsync final : public IObjectStorageIteratorAsync std::shared_ptr client_, size_t max_list_size, bool with_tags_, - const std::optional & start_after_) + const std::optional & start_after_, + size_t attempt_seed_ = 0) : IObjectStorageIteratorAsync( CurrentMetrics::ObjectStorageS3Threads, CurrentMetrics::ObjectStorageS3ThreadsActive, @@ -168,12 +169,15 @@ class S3IteratorAsync final : public IObjectStorageIteratorAsync , request(std::make_unique()) , with_tags(with_tags_) , start_after_set(start_after_.has_value() && !start_after_->empty()) + , attempt_seed(attempt_seed_) { request->SetBucket(bucket_); request->SetPrefix(path_prefix); request->SetMaxKeys(static_cast(max_list_size)); if (start_after_set) request->SetStartAfter(*start_after_); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(*request, attempt_seed); } ~S3IteratorAsync() override @@ -210,6 +214,8 @@ class S3IteratorAsync final : public IObjectStorageIteratorAsync paginated_request->SetPrefix(request->GetPrefix()); paginated_request->SetMaxKeys(request->GetMaxKeys()); paginated_request->SetContinuationToken(next_continuation_token); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(*paginated_request, attempt_seed); request = std::move(paginated_request); start_after_set = false; } @@ -247,6 +253,7 @@ class S3IteratorAsync final : public IObjectStorageIteratorAsync std::unique_ptr request; const bool with_tags; bool start_after_set; + const size_t attempt_seed; }; } @@ -328,7 +335,10 @@ std::unique_ptr S3ObjectStorage::readObject( /// NOLINT } return std::make_unique( - clientForRetryProfile(read_settings.object_storage_retry_profile, read_settings.object_storage_attempt_timeout_ms), + clientForRetryProfile(ObjectStorageControlRequest{ + .profile = read_settings.object_storage_retry_profile, + .attempt_timeout_ms = read_settings.object_storage_attempt_timeout_ms, + .connect_timeout_cap_ms = read_settings.object_storage_connect_timeout_cap_ms}), uri.bucket, object.remote_path, uri.version_id, @@ -423,8 +433,10 @@ std::unique_ptr S3ObjectStorage::writeObject( /// NOLIN /// The SingleAttempt profile (e.g. CAS conditional writes, RFC cas-s3-timeout-retry-control) rides /// on WriteSettings instead of changing this disk's shared client — every other write keeps using /// client->get() and its normal retry policy unchanged. - auto used_client = clientForRetryProfile( - write_settings.object_storage_retry_profile, write_settings.object_storage_attempt_timeout_ms); + auto used_client = clientForRetryProfile(ObjectStorageControlRequest{ + .profile = write_settings.object_storage_retry_profile, + .attempt_timeout_ms = write_settings.object_storage_attempt_timeout_ms, + .connect_timeout_cap_ms = write_settings.object_storage_connect_timeout_cap_ms}); return std::make_unique( used_client, @@ -445,7 +457,7 @@ ObjectStorageIteratorPtr S3ObjectStorage::iterate( bool with_tags, const std::optional & start_after) const { - return iterate(path_prefix, max_keys, with_tags, start_after, ObjectStorageRetryProfile::Default, /*request_timeout_ms=*/0); + return iterate(path_prefix, max_keys, with_tags, start_after, ObjectStorageControlRequest{}); } ObjectStorageIteratorPtr S3ObjectStorage::iterate( @@ -453,14 +465,13 @@ ObjectStorageIteratorPtr S3ObjectStorage::iterate( size_t max_keys, bool with_tags, const std::optional & start_after, - ObjectStorageRetryProfile profile, - uint64_t request_timeout_ms) const + const ObjectStorageControlRequest & request) const { auto settings_ptr = s3_settings.get(); if (!max_keys) max_keys = settings_ptr->request_settings[S3RequestSetting::list_object_keys_size]; return std::make_shared( - uri.bucket, path_prefix, clientForRetryProfile(profile, request_timeout_ms), max_keys, with_tags, start_after); + uri.bucket, path_prefix, clientForRetryProfile(request), max_keys, with_tags, start_after, request.attempt_number); } void S3ObjectStorage::listObjects(const std::string & path, RelativePathsWithMetadata & children, size_t max_keys) const @@ -567,18 +578,20 @@ void S3ObjectStorage::removeObjectsIfExist(const StoredObjects & objects) ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches(const StoredObject & object, const std::string & etag) { - return removeObjectIfTokenMatchesImpl(object, etag, client->get()); + return removeObjectIfTokenMatchesImpl(object, etag, client->get(), /*attempt_seed=*/0); } ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches( - const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) + const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request) { return refreshAndRetryOnExpiredCredentials( - [&] { return removeObjectIfTokenMatchesImpl(object, etag, clientForRetryProfile(profile, request_timeout_ms)); }); + [&] { return removeObjectIfTokenMatchesImpl( + object, etag, clientForRetryProfile(request), request.attempt_number); }); } ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatchesImpl( - const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client) + const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client, + size_t attempt_seed) { S3::DeleteObjectRequest request; request.SetBucket(uri.bucket); @@ -587,6 +600,8 @@ ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatchesImpl( /// This is a content-addressed exact-token DELETE: mark it eligible for the typed NativeConditional /// mode, so a GCS-native client can send the generation token this etag actually encodes. request.setNativeConditional(); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(request, attempt_seed); ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); @@ -623,17 +638,17 @@ ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatchesImpl( err.GetMessage(), static_cast(err.GetErrorType()), err.GetExceptionName(), object.remote_path); } -void S3ObjectStorage::removeObjectsIfExistUnderProfile( - const StoredObjects & objects, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) +void S3ObjectStorage::removeObjectsIfExistUnderProfile(const StoredObjects & objects, const ObjectStorageControlRequest & request) { refreshAndRetryOnExpiredCredentials([&] { - removeObjectsIfExistImpl(objects, clientForRetryProfile(profile, request_timeout_ms)); + removeObjectsIfExistImpl(objects, clientForRetryProfile(request), request.attempt_number); return 0; }); } -void S3ObjectStorage::removeObjectsIfExistImpl(const StoredObjects & objects, const std::shared_ptr & used_client) +void S3ObjectStorage::removeObjectsIfExistImpl( + const StoredObjects & objects, const std::shared_ptr & used_client, size_t attempt_seed) { if (objects.empty()) return; @@ -655,6 +670,8 @@ void S3ObjectStorage::removeObjectsIfExistImpl(const StoredObjects & objects, co S3::DeleteObjectsRequest request; request.SetBucket(uri.bucket); request.SetDelete(std::move(to_delete)); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(request, attempt_seed); ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); auto outcome = used_client->DeleteObjects(request); @@ -813,7 +830,7 @@ std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeTok } std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeToken( - const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const + const std::string & path, bool with_tags, const ObjectStorageControlRequest & request) const { return refreshAndRetryOnExpiredCredentials( [&] @@ -822,7 +839,8 @@ std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeTok path, with_tags, ObjectStorageRequestMode::NativeConditional, - clientForRetryProfile(profile, request_timeout_ms)); + clientForRetryProfile(request), + request.attempt_number); }); } @@ -830,11 +848,12 @@ std::optional S3ObjectStorage::tryGetObjectMetadataImpl( const std::string & path, bool with_tags, ObjectStorageRequestMode request_mode, - const std::shared_ptr & used_client) const + const std::shared_ptr & used_client, + size_t attempt_seed) const { auto settings_ptr = s3_settings.get(); auto object_info = S3::getObjectInfoIfExists( - *used_client, uri.bucket, path, {}, /* with_metadata= */ true, with_tags, request_mode); + *used_client, uri.bucket, path, {}, /* with_metadata= */ true, with_tags, request_mode, attempt_seed); if (object_info.size == 0 && object_info.last_modification_time == 0 && object_info.metadata.empty()) return {}; @@ -1114,7 +1133,7 @@ std::shared_ptr S3ObjectStorage::tryGetS3StorageClient() return client->get(); } -std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64_t request_timeout_ms) const +std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64_t request_timeout_ms, uint64_t connect_timeout_cap_ms) const { auto base = client->get(); std::lock_guard lock(single_attempt_client_mutex); @@ -1124,7 +1143,8 @@ std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64 single_attempt_client_base = base; } - if (auto it = single_attempt_clients.find(request_timeout_ms); it != single_attempt_clients.end()) + const auto cache_key = std::make_pair(request_timeout_ms, connect_timeout_cap_ms); + if (auto it = single_attempt_clients.find(cache_key); it != single_attempt_clients.end()) return it->second; auto cfg = base->getClientConfiguration(); @@ -1141,16 +1161,22 @@ std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64 if (request_timeout_ms != 0) cfg.requestTimeoutMs = static_cast(request_timeout_ms); - return single_attempt_clients.emplace(request_timeout_ms, base->cloneWithConfigurationOverride(cfg)).first->second; + /// One TCP/TLS connect may not cost more than the cap the mount froze at open: the engine reserves + /// attempt + 2 × cap per envelope, and a reloaded base client with a wider connect timeout must not + /// widen what a reissue can spend. + if (connect_timeout_cap_ms != 0) + cfg.connectTimeoutMs = cfg.connectTimeoutMs <= 0 ? static_cast(connect_timeout_cap_ms) + : std::min(cfg.connectTimeoutMs, static_cast(connect_timeout_cap_ms)); + + return single_attempt_clients.emplace(cache_key, base->cloneWithConfigurationOverride(cfg)).first->second; } -std::shared_ptr S3ObjectStorage::clientForRetryProfile( - ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const +std::shared_ptr S3ObjectStorage::clientForRetryProfile(const ObjectStorageControlRequest & request) const { /// getSingleAttemptClient is only invoked when actually selected, so an ordinary request never /// pays for building or locking the clone. - if (profile == ObjectStorageRetryProfile::SingleAttempt) - return getSingleAttemptClient(request_timeout_ms); + if (request.profile == ObjectStorageRetryProfile::SingleAttempt) + return getSingleAttemptClient(request.attempt_timeout_ms, request.connect_timeout_cap_ms); return client->get(); } diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 0b46038e9aa8..a7a70e9616f6 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -24,7 +25,6 @@ namespace S3RequestSetting extern const S3RequestSettingsBool read_only; } - class S3ObjectStorage : public IObjectStorage { public: @@ -110,8 +110,7 @@ class S3ObjectStorage : public IObjectStorage size_t max_keys, bool with_tags, const std::optional & start_after, - ObjectStorageRetryProfile profile, - uint64_t request_timeout_ms) const override; + const ObjectStorageControlRequest & request) const override; /// Uses `DeleteObjectRequest`. void removeObjectIfExists(const StoredObject & object) override; @@ -124,11 +123,11 @@ class S3ObjectStorage : public IObjectStorage ConditionalRemoveResult removeObjectIfTokenMatches(const StoredObject & object, const std::string & etag) override; ConditionalRemoveResult removeObjectIfTokenMatches( - const StoredObject & object, const std::string & etag, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) override; + const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request) override; /// One `DeleteObjects` for the given objects (the caller chunks to at most 1000); absence is success. void removeObjectsIfExistUnderProfile( - const StoredObjects & objects, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) override; + const StoredObjects & objects, const ObjectStorageControlRequest & request) override; void tagObjects(const StoredObjects & objects, const std::string & tag_key, const std::string & tag_value) override; @@ -141,7 +140,7 @@ class S3ObjectStorage : public IObjectStorage std::optional tryGetObjectMetadataWithNativeToken(const std::string & path, bool with_tags) const override; std::optional tryGetObjectMetadataWithNativeToken( - const std::string & path, bool with_tags, ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const override; + const std::string & path, bool with_tags, const ObjectStorageControlRequest & request) const override; void copyObject( /// NOLINT const StoredObject & object_from, @@ -201,7 +200,10 @@ class S3ObjectStorage : public IObjectStorage /// disk client rotates (applyNewSettings/credentials refresh) — the cached clone is keyed by the /// base client's identity, so a stale clone can never outlive a rotation. /// `request_timeout_ms` overrides the clone's send/receive inactivity bound; 0 keeps the disk's. - std::shared_ptr getSingleAttemptClient(uint64_t request_timeout_ms) const; + /// `connect_timeout_cap_ms` caps the clone's connect timeout (0 = no cap); the cache key is the + /// pair, so two callers asking for the same request timeout but different caps get distinct clones. + std::shared_ptr getSingleAttemptClient(uint64_t request_timeout_ms, uint64_t connect_timeout_cap_ms = 0) const; + private: void removeObjectImpl(const StoredObject & object, bool if_exists); void removeObjectsImpl(const StoredObjects & objects, bool if_exists); @@ -212,14 +214,17 @@ class S3ObjectStorage : public IObjectStorage const std::string & path, bool with_tags, ObjectStorageRequestMode request_mode, - const std::shared_ptr & used_client) const; + const std::shared_ptr & used_client, + size_t attempt_seed = 0) const; ConditionalRemoveResult removeObjectIfTokenMatchesImpl( - const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client); + const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client, + size_t attempt_seed); - void removeObjectsIfExistImpl(const StoredObjects & objects, const std::shared_ptr & used_client); + void removeObjectsIfExistImpl( + const StoredObjects & objects, const std::shared_ptr & used_client, size_t attempt_seed); - std::shared_ptr clientForRetryProfile(ObjectStorageRetryProfile profile, uint64_t request_timeout_ms) const; + std::shared_ptr clientForRetryProfile(const ObjectStorageControlRequest & request) const; /// Runs `fn` and, if it failed because the vended credentials expired, refreshes this disk's /// client and runs it once more. `fn` must re-read the client itself, so the second run signs @@ -252,10 +257,10 @@ class S3ObjectStorage : public IObjectStorage std::atomic pinned_generation_dialect{-1}; /// -1 unpinned, 0 pinned ETag, 1 pinned generation mutable std::mutex single_attempt_client_mutex; - /// One clone per requested timeout: the verbs of one operation ask for different bounds, and a - /// single slot would rebuild a whole S3 client (and lose its connection pool) on every - /// alternation between them. - mutable std::map> single_attempt_clients; + /// One clone per (requested timeout, connect cap) pair: the verbs of one operation ask for + /// different bounds, and a single slot would rebuild a whole S3 client (and lose its connection + /// pool) on every alternation between them. + mutable std::map, std::shared_ptr> single_attempt_clients; /// The base client the cached clones above were built from. Deliberately held as a shared_ptr (not /// a raw pointer): a raw pointer would be compared for identity AFTER the object it once pointed /// to could have been freed and a new client reallocated at the same address by an unrelated diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index 73da22b314be..4c37bcd13a12 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -1648,6 +1648,40 @@ TEST(CASObjectStorageBackend, EmuTokenStateEventuallyPrunesDistinctShortLivedKey << "token state should track only the bounded recent-key window, not all " << key_count << " deleted keys"; } +TEST(CASObjectStorageBackend, EnsureBackendMatchesBudgetAcceptsAMatchingHandoff) +{ + CasRequestBudget budget; + budget.attempt_timeout_ms = 4000; + budget.connect_timeout_cap_ms = 900; + auto backend = std::make_shared( + tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess, + /*single_attempt_control_plane_=*/false, budget.attempt_timeout_ms, *budget.connect_timeout_cap_ms); + EXPECT_NO_THROW(ensureBackendMatchesBudget(*backend, budget)); +} + +TEST(CASObjectStorageBackend, EnsureBackendMatchesBudgetRejectsAMismatchedConnectCap) +{ + CasRequestBudget budget; + budget.attempt_timeout_ms = 4000; + budget.connect_timeout_cap_ms = 900; + /// The backend is built with a DIFFERENT connect cap than the budget it will be paired with -- + /// exactly the handoff mistake the production check at `openPoolView` guards against. + auto backend = std::make_shared( + tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess, + /*single_attempt_control_plane_=*/false, budget.attempt_timeout_ms, /*connect_timeout_cap_ms_=*/1500); + EXPECT_THROW(ensureBackendMatchesBudget(*backend, budget), DB::Exception); +} + +TEST(CASObjectStorageBackend, EnsureBackendMatchesBudgetRejectsAMismatchedAttemptTimeout) +{ + CasRequestBudget budget; + budget.attempt_timeout_ms = 4000; + budget.connect_timeout_cap_ms = 900; + auto backend = std::make_shared( + tests::makeLocalObjectStorageForTest(), ObjectStorageBackend::Mode::EmulatedSingleProcess, + /*single_attempt_control_plane_=*/false, /*attempt_timeout_ms_=*/6000, *budget.connect_timeout_cap_ms); + EXPECT_THROW(ensureBackendMatchesBudget(*backend, budget), DB::Exception); +} /// `NativeRejectsWrongDialectTokenBeforeTouchingTheWire` is deleted here: it built a `Token{value, /// Dialect::Emulated}` holding a NATIVE backend's live wire value under the WRONG dialect tag, to prove diff --git a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp index 2afec62dbd07..07b71ee96cff 100644 --- a/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp +++ b/src/Disks/tests/gtest_cas_bootstrap_ordering.cpp @@ -6,6 +6,7 @@ #include #include "cas_test_helpers.h" #include +#include #include #include @@ -265,6 +266,40 @@ TEST(CASBootstrapOrdering, ResidualWithoutMetaFailsTypedWithZeroWrites) EXPECT_FALSE(readPresent(*backend, kPoolMetaKey)) << "a fresh _pool_meta must NOT have been minted"; } +/// The engine's attempt number reaches the transport even through the bootstrap's own residual LIST. +/// A backend that fails only the FIRST attempt of every LIST +/// (as the adaptive-timeout fuse would) must still let the residual check succeed on attempt 2 -- if +/// propagation were broken every attempt would look like attempt 1 and the LIST would never succeed, +/// which the bootstrap reports as `BootstrapResidual::Indeterminate` ("could not authoritatively list"), +/// a DIFFERENT message from the one asserted below. Reuses `ResidualWithoutMetaFailsTypedWithZeroWrites`'s +/// exact seeding helper and expected error code so the assertion distinguishes "refused because listed" +/// from "refused because the LIST failed". +TEST(CASBootstrapOrdering, ResidualListSucceedsOnTheSecondAttempt) +{ + /// Every LIST whose attempt number is 1 fails as the first-attempt fuse would; attempt 2 answers. + struct FuseOnFirstList : RecordingBackend + { + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + if (access.attemptNo() == 1) + throw Poco::TimeoutException("Timeout"); + return RecordingBackend::list(prefix, cursor, limit, access); + } + }; + auto backend = std::make_shared(); + /// A healthy pool without `_pool_meta` is the shape that needs the LIST: seed one residual key. + seedObject(*backend, residualRefLogKey(), "x"); + backend->clearLog(); + + expectThrowsCodeContaining(DB::ErrorCodes::INVALID_STATE, "refusing to bootstrap over residual data", + [&] { Pool::open(backend, makeConfig()); }); + + bool listed_on_second = false; + for (const auto & e : backend->snapshot()) + listed_on_second |= (e.op == RecordingBackend::Op::List); + EXPECT_TRUE(listed_on_second) << "the residual LIST must have been answered (on attempt 2), not merely failed forever"; +} + /// (b') The residual verdict is decided by the first residual key, not by an enumeration of the whole /// prefix: forty residue keys and a 32-key page must cost exactly ONE list request. Enumerating a /// large prefix is the one request a slow store cannot answer within an attempt, and a refusal diff --git a/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp b/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp index 27b7a01399e3..68aeabeb9523 100644 --- a/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp +++ b/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp @@ -190,7 +190,8 @@ TEST(CASBulkDeleteBackend, LocalObjectStorageRefusesTheProfileOverload) DB::StoredObjects objects{DB::StoredObject("p/anything")}; expectThrowsCode(DB::ErrorCodes::NOT_IMPLEMENTED, [&] { - storage->removeObjectsIfExistUnderProfile(objects, DB::ObjectStorageRetryProfile::SingleAttempt, 1000); + storage->removeObjectsIfExistUnderProfile(objects, DB::ObjectStorageControlRequest{ + .profile = DB::ObjectStorageRetryProfile::SingleAttempt, .attempt_timeout_ms = 1000}); }); } #endif diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index a8cfb5e1fb7e..e08a51ea1c00 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -272,8 +272,14 @@ std::shared_ptr openTestStorage(bool tiny_b storage->startup(); if (tiny_budget) { - storage->poolForTest()->setDetachedDrainDeadlineBudgetForTest( - /*attempt_timeout_ms=*/10, /*lease_safety_margin_ms=*/10); + /// `connect_timeout_cap_ms` is set explicitly (not left at whatever the pool froze at open) + /// so the drain deadline -- `attemptEnvelopeMs() + lease_safety_margin_ms` -- is really the + /// tiny 20 ms this test wants, not 10 ms of attempt timeout plus a hidden connect-cap tax. + CasRequestBudget budget; + budget.attempt_timeout_ms = 10; + budget.lease_safety_margin_ms = 10; + budget.connect_timeout_cap_ms = 0; + storage->poolForTest()->setDetachedDrainDeadlineBudgetForTest(budget); } return storage; } @@ -290,6 +296,9 @@ PoolPtr openPublishingPool(const std::shared_ptrsetAttemptTimeoutMs(config.cas_request_budget.attempt_timeout_ms); diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index 7beee4dc1348..fea227a73bc8 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -126,6 +126,7 @@ CasRequestBudget renewalEventBudget() return CasRequestBudget{ .attempt_timeout_ms = 10, .lease_safety_margin_ms = 20, + .connect_timeout_cap_ms = std::nullopt, }; } diff --git a/src/Disks/tests/gtest_cas_heartbeat.cpp b/src/Disks/tests/gtest_cas_heartbeat.cpp index c63f686299a3..c39935d7ca7f 100644 --- a/src/Disks/tests/gtest_cas_heartbeat.cpp +++ b/src/Disks/tests/gtest_cas_heartbeat.cpp @@ -82,7 +82,10 @@ void seedOwnClaim(CasOperation & op, const Layout & l, const String & srid, UInt ASSERT_EQ(claimMount(op, l, srid, uuid, epoch, now_ms, ttl_ms).kind, MountClaimResult::Claimed); } -class RenewalScriptBackend final : public InMemoryBackend +/// Not `final`: `EnvelopeEatingBackend` (the envelope-cutoff test below) derives from it to +/// reuse its `Attempt`/`attempts` bookkeeping while overriding `write`/`read` with its own always-fail +/// behavior instead of the scripted-action queue. +class RenewalScriptBackend : public InMemoryBackend { public: enum class Action : uint8_t @@ -292,6 +295,201 @@ TEST(CASHeartbeat, StopStampsExpiredAndFarewellSentinel) EXPECT_EQ(m.min_active_build_sequence, std::numeric_limits::max()); } +namespace +{ +/// Reports the SHIPPED PRODUCTION defaults (`attempt_timeout_ms=5000`, two `connect_timeout_cap_ms=1000` +/// caps -> `attemptEnvelopeMs()=7000`, `CasRequestBudget.cpp`'s own defaults) while landing every attempt +/// immediately: the write's own success is not what is under test here, only whether the farewell's +/// policy window is wide enough to admit one attempt in the first place. +struct DefaultEnvelopeBackend : InMemoryBackend +{ + uint64_t attemptTimeoutMs() const override { return 5000; } + uint64_t attemptEnvelopeMs() const override { return 7000; } +}; + +/// A DIFFERENT envelope from `DefaultEnvelopeBackend`'s, for +/// `FarewellIsAdmittedUnderADifferentEnvelope` below: that test exists to pin the window's +/// ARITHMETIC, not just that some window admits the write, so it needs a reservation the +/// shipped-default window (16000 ms) could not have admitted by coincidence. +struct WiderEnvelopeBackend : InMemoryBackend +{ + uint64_t attemptTimeoutMs() const override { return 5000; } + uint64_t attemptEnvelopeMs() const override { return 9000; } +}; +} + +/// A write reserves two attempt envelopes before it starts (`CasOperation::writeLoop`'s +/// `reservedFor(0, 2)`), so at the shipped defaults the farewell needs a policy window that admits +/// 2 * 7000 = 14000 ms. A fixed window that predates that reservation (`kFarewellBudgetMs` alone is +/// 10000 ms) refuses the write before its first attempt on every graceful shutdown: no farewell is +/// published, and the next start pays a full incarnation-stability observation instead of reclaiming +/// the slot instantly. +TEST(CASHeartbeat, FarewellIsAdmittedUnderTheDefaultBudget) +{ + auto backend = std::make_shared(); + Layout layout("pool"); + const String srid = "test"; + const UInt128 uuid(0x1234); + uint64_t now_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/30000); + + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(30000), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(2000), + [&] { return boot_ms; }); + renewer.start(); + + now_ms = 2000; + EXPECT_NO_THROW(renewer.release()) + << "the farewell's policy window must admit the write's own two-envelope reservation " + "(2 * 7000 ms with the shipped defaults) -- otherwise a clean shutdown never hands the " + "mount slot back and every restart pays a full incarnation-stability observation"; + + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); + EXPECT_LE(m.expires_at_ms, now_ms); + EXPECT_EQ(m.min_active_build_sequence, std::numeric_limits::max()); +} + +/// Pins the window's ARITHMETIC, not just that some fixed window happens to be wide enough: a +/// regression that hardcoded the shipped-default window (16000 ms) instead of deriving it from +/// `attemptReservationMs()` would still pass `FarewellIsAdmittedUnderTheDefaultBudget` above (16000 +/// happens to equal what a 7000 ms envelope needs) but would refuse THIS write, whose reservation is +/// 2 * 9000 = 18000 ms -- strictly more than the shipped-default window. +TEST(CASHeartbeat, FarewellIsAdmittedUnderADifferentEnvelope) +{ + auto backend = std::make_shared(); + Layout layout("pool"); + const String srid = "test"; + const UInt128 uuid(0x1234); + uint64_t now_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/40000); + + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(40000), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(2000), + [&] { return boot_ms; }); + renewer.start(); + + now_ms = 2000; + EXPECT_NO_THROW(renewer.release()) + << "the farewell's policy window must be DERIVED from this backend's own envelope " + "(2 * 9000 ms), not hardcoded to the shipped-default window -- a window fixed at " + "16000 ms would refuse this write's 18000 ms reservation"; + + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); + EXPECT_LE(m.expires_at_ms, now_ms); + EXPECT_EQ(m.min_active_build_sequence, std::numeric_limits::max()); +} + +/// The derived window alone is not the whole story: mount-control activity must also never run past +/// the point this node's own fence may already be gone. A 5000 ms TTL with a 2000 ms safety margin +/// leaves only 3000 ms of lease-safe remaining time at release -- far short of the 7000 ms envelope's +/// own 16000 ms derived window (2 * 7000 + 2000 slack) -- so the LEASE bound, not the derived window, +/// must be what refuses this write, and it must refuse it before any physical attempt: a write that +/// cannot land inside the lease-safe remainder gains nothing by being sent anyway. +TEST(CASHeartbeat, FarewellIsRefusedWhenTheLeaseExpiresBeforeItsDerivedWindow) +{ + auto backend = std::make_shared(); + Layout layout("pool"); + const String srid = "test"; + const UInt128 uuid(0x1234); + uint64_t now_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/5000); + + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(5000), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(2000), + [&] { return boot_ms; }); + renewer.start(); + + now_ms = 2000; + String message; + int code = 0; + bool threw = false; + try + { + renewer.release(); + } + catch (const DB::Exception & e) + { + threw = true; + message = e.message(); + code = e.code(); + } + EXPECT_TRUE(threw) << "a farewell whose reservation cannot fit inside the lease-safe remaining " + "time must be refused, not admitted past the point this node's fence may " + "already be gone"; + EXPECT_EQ(code, DB::ErrorCodes::NETWORK_ERROR) << message; + EXPECT_NE(message.find("gave up at the lease deadline after zero attempt(s)"), String::npos) << message; + + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); + EXPECT_NE(m.min_active_build_sequence, std::numeric_limits::max()) + << "the refused write must not have landed"; +} + +/// The lease bound added above must not change what an ordinary Conflict outcome does: a successor +/// that took the slot (a different, unfenced incarnation) before this node's own shutdown could +/// publish its farewell must be left untouched, and the release must report the conflict rather than +/// silently succeeding or overwriting the successor's incarnation. +TEST(CASHeartbeat, ForeignIncarnationDuringFarewellLeavesTheSuccessorUntouchedAndReportsTheConflict) +{ + auto backend = std::make_shared(); + Layout layout("pool"); + const String srid = "test"; + const UInt128 uuid(0x1234); + uint64_t now_ms = 1000; + uint64_t boot_ms = 100; + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, srid, uuid, /*epoch=*/9, now_ms, /*ttl_ms=*/100); + + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, srid, uuid, /*writer_epoch=*/9, + std::chrono::milliseconds(100), [&] { return now_ms; }, + [] { return uint64_t{5}; }, {}, std::chrono::milliseconds(0), + [&] { return boot_ms; }); + renewer.start(); + + /// A successor (a different uuid/epoch, NOT gc_fenced) took the slot before this node's own + /// clean shutdown could publish its farewell -- the exact shape a live double-start reclaim + /// leaves behind. + const auto observed = ops.op.read(layout.mountKey(srid), Retry::standard()); + ASSERT_TRUE(observed.has_value()); + MountLease successor; + successor.server_uuid = UInt128(0x9999); + successor.writer_epoch = 1; + successor.seq = 1; + successor.write_attempt_id = UInt128{1}; + mustCommit(ops.op.replace(layout.mountKey(srid), encodeMountLease(successor), observed->etag, + Retry::standard()), "successor slot"); + + now_ms = 2000; + String message; + int code = 0; + try + { + renewer.release(); + FAIL() << "a farewell that finds a foreign, unfenced incarnation must report the conflict, " + "not silently succeed or clobber the successor"; + } + catch (const DB::Exception & e) + { + message = e.message(); + code = e.code(); + } + EXPECT_EQ(code, DB::ErrorCodes::ABORTED) << message; + EXPECT_NE(message.find("found a foreign incarnation"), String::npos) << message; + + auto m = decodeMountLease(ops.op.read(layout.mountKey(srid), Retry::standard())->bytes); + EXPECT_EQ(m.server_uuid, successor.server_uuid) + << "the successor's own incarnation must be untouched by the refused farewell"; + EXPECT_EQ(m.writer_epoch, successor.writer_epoch); +} + /// Phase A (spec rev.4 2026-07-24): a confirmed renewal mismatch whose re-read shows OUR OWN /// (uuid, epoch), unfenced, is state UNCERTAINTY (an ambiguous landed renewal of ours, or a /// same-pair twin after epoch-state loss) — fail closed via fence + self-remount, never an @@ -1087,3 +1285,63 @@ TEST(CASHeartbeat, WallClockStepsAndBootSuspendCannotExtendAuthority) EXPECT_EQ(failure.code(), DB::ErrorCodes::NETWORK_ERROR); EXPECT_TRUE(backend->attempts.empty()) << "suspend-sized BOOTTIME overshoot must close admission"; } + +/// Every attempt costs the whole envelope (attempt 100 + 2 * cap 50 = 200 ms) and fails ambiguously. +/// Under a 1000 ms lease with a 100 ms margin the renewal must stop issuing before the cutoff rather +/// than start an attempt that cannot finish inside it. +namespace +{ +/// Bypasses `RenewalScriptBackend`'s scripted-action queue for a guarded mount write and instead +/// always fails it (and every read) once armed, each failure costing the whole envelope on the +/// injected boot clock. Left unarmed during `seedOwnClaim` (an unconditional read then an unguarded +/// create -- neither is a guarded mount write, but the read would still hit the always-throwing +/// override below) and during `renewer.start()`'s adopt read, so the fixture itself can land. +struct EnvelopeEatingBackend : RenewalScriptBackend +{ + uint64_t * boot_ms = nullptr; + bool armed = false; + uint64_t attemptTimeoutMs() const override { return 100; } + uint64_t attemptEnvelopeMs() const override { return 200; } + std::expected write(const String & key, const String & bytes, + const std::optional & expected_value, TransportAccess & access) override + { + if (armed && expected_value && key.ends_with("/mount")) + { + attempts.push_back({key, bytes, expected_value}); + *boot_ms += 200; + throw Poco::TimeoutException("the whole envelope, gone"); + } + return InMemoryBackend::write(key, bytes, expected_value, access); + } + std::optional read(const String & key, TransportAccess & access) override + { + if (armed) + { + *boot_ms += 200; + throw Poco::TimeoutException("the read too"); + } + return InMemoryBackend::read(key, access); + } +}; +} + +TEST(CASHeartbeat, RenewalStopsBeforeTheCutoffWhenEveryAttemptConsumesTheEnvelope) +{ + auto backend = std::make_shared(); + uint64_t wall_ms = 1000; + uint64_t boot_ms = 100; + backend->boot_ms = &boot_ms; + Layout layout("pool"); + Ops ops(backend, &boot_ms); + seedOwnClaim(ops.op, layout, "test", UInt128{0x1234}, 9, wall_ms, 1000); + MountLeaseRenewer renewer(ops.mount, ops.farewell, layout, "test", UInt128{0x1234}, 9, std::chrono::milliseconds(1000), + [&] { return wall_ms; }, [] { return uint64_t{7}; }, {}, std::chrono::milliseconds(100), + [&] { return boot_ms; }); + renewer.start(); + const uint64_t cutoff = renewer.lastCommittedAttemptStartBootMs() + 1000 - 100; + backend->attempts.clear(); + backend->armed = true; + const MountRenewResult result = renewer.renew(renewalEnvironment(boot_ms)); + EXPECT_EQ(result.outcome, MountRenewOutcome::Terminal); + EXPECT_LE(boot_ms, cutoff) << "the last attempt started inside the cutoff and the engine did not start one that could not finish"; +} diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 7200757f11d1..4f008b6724ea 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -291,6 +291,7 @@ CasRequestBudget renewalLogBudget() return CasRequestBudget{ .attempt_timeout_ms = 10, .lease_safety_margin_ms = 20, + .connect_timeout_cap_ms = std::nullopt, }; } @@ -1268,14 +1269,14 @@ TEST(CASMountReadOnly, ForeignOwnedPoolOpensWithoutMutation) TEST(CASRequestBudget, ValidateAcceptsDefaultsAndRejectsAnOverflowingSumWithoutWrapping) { EXPECT_NO_THROW(validateCasRequestBudget( - CasRequestBudget{}, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000)); + CasRequestBudget{}, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000, /*background_renewal=*/false)); const CasRequestBudget overflowing{ .attempt_timeout_ms = std::numeric_limits::max() - 100, .lease_safety_margin_ms = std::numeric_limits::max() - 100}; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] { - validateCasRequestBudget(overflowing, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000); + validateCasRequestBudget(overflowing, /*mount_lease_ttl_ms=*/30000, /*mount_renew_period_ms=*/10000, /*background_renewal=*/false); }); } @@ -1312,7 +1313,7 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) /// lease TTL), so it also scales down cas_request_budget to fit — the budget itself is not /// exercised here, only Pool::open's validateCasRequestBudget startup gate. const CasRequestBudget tiny_budget{ - .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; auto a = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r", .mount_lease_ttl_ms = std::chrono::milliseconds(300), diff --git a/src/Disks/tests/gtest_cas_mount_runtime.cpp b/src/Disks/tests/gtest_cas_mount_runtime.cpp index 25d7f5e3b25e..0294fa178a35 100644 --- a/src/Disks/tests/gtest_cas_mount_runtime.cpp +++ b/src/Disks/tests/gtest_cas_mount_runtime.cpp @@ -6,6 +6,7 @@ #include #include +#include using namespace DB::Cas; @@ -17,7 +18,8 @@ namespace class RuntimeFixture { public: - explicit RuntimeFixture(uint64_t lease_safety_margin_ms, uint64_t attempt_timeout_ms = 10) + explicit RuntimeFixture(uint64_t lease_safety_margin_ms, uint64_t attempt_timeout_ms = 10, + std::optional connect_timeout_cap_ms = std::nullopt) : backend(std::make_shared()) , mount(backend, Fence{ [this] { return runtime.fenceGeneration(); }, @@ -29,7 +31,8 @@ class RuntimeFixture MountConfig{.boot_ms_fn = [this] { return boot_ms; }}, "test", sink, CasRequestBudget{.attempt_timeout_ms = attempt_timeout_ms, - .lease_safety_margin_ms = lease_safety_margin_ms}, + .lease_safety_margin_ms = lease_safety_margin_ms, + .connect_timeout_cap_ms = connect_timeout_cap_ms}, [] { return false; }) { } @@ -142,15 +145,32 @@ TEST(CASMountRuntime, AdmitAllowsAnUnarmedFence) } /// `refAppendFenceOk` is `admit` at one attempt's worth of budget under the live generation. -TEST(CASMountRuntime, RefAppendFenceOkIsAdmitAtTheAttemptTimeout) +TEST(CASMountRuntime, RefAppendFenceOkIsAdmitAtTwoEnvelopes) { + /// connect_timeout_cap_ms is nullopt (see RuntimeFixture), so the envelope equals the bare attempt + /// timeout (10 ms); refAppendFenceOk asks for TWO of them (a write and its settlement read). RuntimeFixture f(/*lease_safety_margin_ms=*/20, /*attempt_timeout_ms=*/10); f.boot_ms = 1'000; - f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'031); /// 31 ms left: one more than 10 + 20 + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'041); /// 41 ms left: one more than 2*10 + 20 EXPECT_TRUE(f->refAppendFenceOk()); - EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 10)), "Ok"); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 20)), "Ok"); - f->setMountDeadline(1'030); /// exactly 10 + 20 left + f->setMountDeadline(1'040); /// exactly 2*10 + 20 left EXPECT_FALSE(f->refAppendFenceOk()); - EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 10)), "NoBudget"); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 20)), "NoBudget"); +} + +/// Same boundary, with a nonzero connect cap so the envelope's connect contribution (not just the +/// doubling) is pinned: attempt 100, cap 50 -> envelope 200, refAppendFenceOk asks for 2*200 = 400. +TEST(CASMountRuntime, RefAppendFenceOkIsAdmitAtTwoEnvelopesWithANonzeroCap) +{ + RuntimeFixture f(/*lease_safety_margin_ms=*/20, /*attempt_timeout_ms=*/100, /*connect_timeout_cap_ms=*/50); + f.boot_ms = 1'000; + f->armMountFence(kUuid, 1, /*deadline_boot_ms=*/1'421); /// 421 ms left: one more than 2*200 + 20 + EXPECT_TRUE(f->refAppendFenceOk()); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 400)), "Ok"); + + f->setMountDeadline(1'420); /// exactly 2*200 + 20 left + EXPECT_FALSE(f->refAppendFenceOk()); + EXPECT_STREQ(admitName(f->admit(f->fenceGeneration(), 400)), "NoBudget"); } diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index bffce59e2013..1a114b2144c0 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -80,6 +80,7 @@ CasRequestBudget renewalCounterBudget(uint32_t /*max_attempts*/ = 2) return CasRequestBudget{ .attempt_timeout_ms = 10, .lease_safety_margin_ms = 20, + .connect_timeout_cap_ms = std::nullopt, }; } diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 33f92e656d68..320778e2dcc2 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -2213,6 +2213,9 @@ CasRequestBudget runtimeRenewBudget() return CasRequestBudget{ .attempt_timeout_ms = 10, .lease_safety_margin_ms = 20, + /// The default cap (1000 ms) would make the envelope (10 + 2*1000 = 2010) blow every tiny TTL + /// this budget is used against; no connect notion is exercised by these tests. + .connect_timeout_cap_ms = std::nullopt, }; } } @@ -2315,7 +2318,7 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) /// cas-s3-timeout-retry-control §required-timeout-model requires attempt_timeout + safety_margin < /// lease TTL), so scale the budget down to fit -- mirrors `CasMountStartup::StaleSelfMountReclaimedAfterWait`. const CasRequestBudget tiny_budget{ - .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; uint64_t fake_boot = 0; std::vector waits; @@ -2367,6 +2370,53 @@ TEST(CASMountOpenWaits, CleanOpenSkipsAllWaits) << "a clean farewell (Task 5) needs no observation window"; } +namespace +{ +/// Reports the SHIPPED PRODUCTION default envelope (`CasRequestBudget{}`'s own defaults -- +/// `attempt_timeout_ms=5000`, `connect_timeout_cap_ms=1000` -> `attemptEnvelopeMs()=7000`), so the +/// teardown below pays the SAME two-envelope reservation (14000 ms) production pays, not the +/// near-zero envelope a bare `InMemoryBackend` reports by default. +struct DefaultBudgetEnvelopeBackend : InMemoryBackend +{ + uint64_t attemptTimeoutMs() const override { return 5000; } + uint64_t attemptEnvelopeMs() const override { return 7000; } +}; +} + +/// `CleanOpenSkipsAllWaits` above proves a clean farewell skips the observation window, but its bare +/// `InMemoryBackend` reports a zero attempt envelope, so its teardown never exercises the farewell's +/// own policy window against a write's real cost. Pin the shipped default budget specifically: a +/// window that cannot admit the write's `2 * attemptEnvelopeMs()` reservation refuses the farewell +/// before its first attempt, and the successor below then pays a full incarnation-stability +/// observation instead of reclaiming instantly. +TEST(CASMountOpenWaits, CleanTeardownUnderDefaultBudgetLeavesAFarewell) +{ + auto b = std::make_shared(); + auto predecessor = Pool::open(b, PoolConfig{ + .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test"}); + predecessor.reset(); /// drives ~Pool(): with nothing in flight, this is the graceful-shutdown farewell. + + const Layout layout{"p"}; + const auto got = readObj(*b, layout.mountKey("test")); + ASSERT_TRUE(got.has_value()); + const MountLease lease = decodeMountLease(got->bytes); + EXPECT_EQ(lease.min_active_build_sequence, std::numeric_limits::max()) + << "the farewell's policy window must admit the write's own two-envelope reservation at the " + "shipped default budget (2 * 7000 ms) -- otherwise a clean teardown never hands the mount " + "slot back"; + + std::vector waits; + PoolPtr successor; + ASSERT_NO_THROW( + successor = Pool::open(b, PoolConfig{ + .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", + .wait_sleep_fn = [&](uint64_t ms) { waits.push_back(ms); }, + })); + ASSERT_TRUE(successor); + EXPECT_TRUE(waits.empty()) + << "a clean farewell needs no observation window on reopen, even at the shipped default budget"; +} + TEST(CASMountOpenWaits, FencedPriorReclaimsWithoutAnyWait) { auto b = std::make_shared(); @@ -2384,7 +2434,7 @@ TEST(CASMountOpenWaits, FencedPriorReclaimsWithoutAnyWait) /// See UncleanOpenPaysOnlyTheObservationWindow above: a 500ms TTL needs a scaled-down budget too. const CasRequestBudget tiny_budget{ - .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; std::vector waits; PoolPtr store; @@ -2404,6 +2454,109 @@ TEST(CASMountOpenWaits, FencedPriorReclaimsWithoutAnyWait) << "a certified-dead predecessor needs neither the observation window nor any grace period"; } +/// The open-time publication horizon must reserve TWO attempt envelopes (connect cap included), not +/// two bare attempt timeouts -- a slow connect could otherwise overrun the reservation the horizon +/// check was guarding. `background_watermark` defaults false (not set below), so `CasPool.cpp`'s +/// `renewal_window_ms` ternary takes its no-period branch: `2 * attemptEnvelopeMs()`. The check is also +/// STRICT (refuses equality), matching `CasMountRuntime::admit`. +TEST(CASMountOpenWaits, PublicationHorizonUsesTheEnvelope) +{ + /// Opens with a boot clock costing `per_call_ms` per read (models a faster or slower claim) and + /// returns how many times the mount key was written. attempt 100, cap 100: envelope = + /// 100 + 2*100 = 300, so 2*envelope = 600; the old code reserved 2*attempt = 200. Empirically the + /// claim path's own anchor read and the horizon check's own `now_boot_ms` read are five reads apart, + /// so `remaining = safe_deadline(TTL 1000 - margin 50 = 950) - now = 950 - 5 * per_call_ms`. + const auto mountWriteCount = [](uint64_t per_call_ms) -> uint64_t + { + auto b = std::make_shared(); + Layout l{"p"}; + DB::Cas::tests::seedPoolMetaForRestart(*b); + uint64_t fake_boot = 0; + PoolPtr store; + store = Pool::open(b, PoolConfig{ + .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", + .mount_lease_ttl_ms = std::chrono::milliseconds(1000), + .cas_request_budget = CasRequestBudget{.attempt_timeout_ms = 100, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = 100}, + .boot_ms_fn = [&] { const uint64_t now = fake_boot; fake_boot += per_call_ms; return now; }, + .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; }, + }); + if (!store) + return 0; + return b->putOverwriteCount(l.mountKey("test")) + b->putCount(l.mountKey("test")); + }; + + /// remaining = 945 (per_call_ms=1): both 2*attempt(200) and 2*envelope(600) fit -- two writes (the + /// claim's own reclaim, then the renewer's adopt) and no re-anchor. + EXPECT_EQ(mountWriteCount(1), 2u) << "a horizon that fits both windows must not re-anchor"; + /// remaining = 450 (per_call_ms=100): 2*attempt(200) fits, 2*envelope(600) does not -- the + /// re-anchor costs one extra write. This is the discriminator: reverting the reservation to + /// 2*attempt would make this case behave like the one above (two writes). + EXPECT_EQ(mountWriteCount(100), 3u) << "the old 2*attempt window fit here; only the envelope window must redo"; + /// remaining = 600 (per_call_ms=70) exactly equals 2*envelope: STRICT ("<", not "<=") refuses + /// equality too, so this must also redo -- reverting the strict comparison to "<=" would make this + /// case behave like the fits-both case (two writes). + EXPECT_EQ(mountWriteCount(70), 3u) << "an exact boundary (renewal_window_ms == remaining) must be refused, not accepted"; +} + +/// Same reservation change as `PublicationHorizonUsesTheEnvelope` above, exercised through the remount +/// path's own `renewer_redo` step (`CasPool.cpp` ~1503). Modelled directly on +/// `CASPoolRemount.TheRenewerRedoRenewsOnTheOpenPlane` above: the step's admission refuses a driver that +/// was never parked by a persistent renewal worker, so `background_watermark` must be true and the +/// remount must be driven through `scheduleRemountForTest` (which parks the worker before running it), +/// never through a bare `tryRemountOnce` with no workers -- the direct-driven attempt deadlocks in +/// exactly the way that test's own comment describes. +TEST(CASPoolRemount, RemountRenewerRedoUsesTheEnvelope) +{ + /// One successful self-remount whose quiescence costs `quiesce_ms`; returns the conditional + /// mount-slot writes it issued, counted while the remount worker is still held inside the event + /// sink that reported the result (so the renewal worker it un-parks cannot add one). + const auto remountConditionalMountWrites = [](uint64_t quiesce_ms) -> uint64_t + { + auto backend = std::make_shared(); + uint64_t fake_boot = 1'000'000; + DB::Cas::tests::ManualBarrier committed; + auto store = Pool::open(backend, PoolConfig{ + .pool_prefix = "remount-renewer-redo-envelope", + .server_root_id = "test", + .background_watermark = true, + .event_sink = [&committed](const CasEvent & event) + { + if (event.type == CasEventType::MountRemount && event.outcome == "ok") + committed.arriveAndWait(); + }, + .mount_lease_ttl_ms = std::chrono::milliseconds(1000), + .mount_renew_period = std::chrono::milliseconds(100), + .cas_request_budget = CasRequestBudget{.attempt_timeout_ms = 100, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = 100}, + .boot_ms_fn = [&fake_boot] { return fake_boot; }, + .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }, + .remount_quiesce_hook_for_test = [&fake_boot, quiesce_ms] { fake_boot += quiesce_ms; }, + }); + const String mount_key = store->layout().mountKey("test"); + + fenceOutMount(*backend, mount_key); + const uint64_t before = backend->putOverwriteCount(mount_key); + EXPECT_TRUE(store->scheduleRemountForTest()) + << "the remount must be latched with quiesce_ms=" << quiesce_ms; + committed.waitUntilArrived(); + const uint64_t writes = backend->putOverwriteCount(mount_key) - before; + committed.release(); + return writes; + }; + + /// attempt 100, cap 100: envelope = 100 + 2*100 = 300, so period(100) + 2*envelope(600) = 700. A + /// 450 ms quiescence leaves remaining = TTL(1000) - margin(50) - 450 = 500: the old + /// period + 2*attempt (300) window fit that, but the new period + 2*envelope (700) window does not + /// -- so only the envelope-based check must redo (validation: 100 + 600 + 50 = 750 < 1000). + const uint64_t control = remountConditionalMountWrites(0); + EXPECT_GT(remountConditionalMountWrites(450), control) + << "a quiescence that fits the old attempt-only window but not the envelope window must still cost the redo"; + /// A 250 ms quiescence leaves remaining = 950 - 250 = 700, exactly equal to + /// period + 2*envelope (700): STRICT ("<", not "<=") refuses equality too, so this must also + /// redo -- reverting the strict comparison to "<=" would make this case behave like the control. + EXPECT_GT(remountConditionalMountWrites(250), control) + << "an exact boundary (renewal_window_ms == remaining) must be refused, not accepted"; +} + namespace { /// Stalls the CLAIM ITSELF past the lease TTL, and counts what the open writes afterwards. diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index c26e96735be6..c3488fb44902 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -367,7 +367,7 @@ constexpr int kFaultsBeyondTheRetryWindow = 100'000; CasRequestBudget tinyBudget() { return CasRequestBudget{ - .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; } PoolConfig walkTestConfig() diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index f2a9137fb7e4..2f3d22ca1a37 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -5309,7 +5309,7 @@ void seedUncleanPredecessorMount(const BackendPtr & backend, const Layout & layo /// lease TTL) -- mirrors `CASMountOpenWaits.FencedPriorReclaimsWithoutAnyWait` exactly. CasRequestBudget sealTestTinyBudget() { - return CasRequestBudget{.attempt_timeout_ms = 50, .lease_safety_margin_ms = 50}; + return CasRequestBudget{.attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; } } diff --git a/src/Disks/tests/gtest_cas_requests.cpp b/src/Disks/tests/gtest_cas_requests.cpp index cf4c3c504b41..81086f27f433 100644 --- a/src/Disks/tests/gtest_cas_requests.cpp +++ b/src/Disks/tests/gtest_cas_requests.cpp @@ -2,6 +2,8 @@ #include #include +#include +#include #include #include #include @@ -10,8 +12,11 @@ #include #include "cas_test_helpers.h" #include +#include #include +#include +#include #include "config.h" @@ -20,6 +25,11 @@ #include #include +#include +#include +#include +#include + #include #include @@ -39,6 +49,7 @@ namespace DB::ErrorCodes { extern const int ABORTED; +extern const int BAD_ARGUMENTS; extern const int CAS_DELETE_MARKER; extern const int CORRUPTED_DATA; extern const int LOGICAL_ERROR; @@ -51,6 +62,7 @@ namespace ProfileEvents extern const Event CASRequestReissue; extern const Event CASRequestConflictPause; extern const Event CASRequestConnectFailureHint; + extern const Event CASRequestFirstAttemptFuse; } using namespace DB::Cas; @@ -680,6 +692,46 @@ TEST(CASRequests, AmbiguousCreateThatNeverLandedIsReissued) EXPECT_EQ(clock.sleeps.size(), 1u); } +/// The engine's own attempt number reaches the transport through `TransportAccess::attemptNo()`, for +/// every primitive -- write, read (the resolve read is its own call, with its own attempt count) and +/// list. +TEST(CASRequests, TheTransportSeesTheEngineAttemptNumber) +{ + struct AttemptRecordingBackend : CountingBackend + { + std::vector write_attempts, read_attempts, list_attempts; + std::expected write(const String & key, const String & bytes, + const std::optional & expected, TransportAccess & access) override + { + write_attempts.push_back(access.attemptNo()); + return CountingBackend::write(key, bytes, expected, access); + } + std::optional read(const String & key, TransportAccess & access) override + { + read_attempts.push_back(access.attemptNo()); + return CountingBackend::read(key, access); + } + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + list_attempts.push_back(access.attemptNo()); + return CountingBackend::list(prefix, cursor, limit, access); + } + }; + FakeClock clock; + auto backend = std::make_shared(); + backend->injectAmbiguousWrite("k"); + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("read timed out"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + ASSERT_TRUE(std::holds_alternative(op.create("k", "v", Retry::standard()))); + /// Attempt 1 ambiguous, attempt 2 commits. The settle read is its OWN read call: attempt 1 failed, 2 answered. + EXPECT_EQ(backend->write_attempts, (std::vector{1, 2})); + EXPECT_EQ(backend->read_attempts, (std::vector{1, 2})); + backend->list_attempts.clear(); + (void)op.list("p/", "", 10, Retry::standard()); + EXPECT_EQ(backend->list_attempts, (std::vector{1})); +} + TEST(CASRequests, OnceSendsOneWriteAndAtMostOneResolveRead) { FakeClock clock; @@ -2184,6 +2236,87 @@ TEST(CASRequestsConnectHint, ClassifierGuards) EXPECT_FALSE(isConnectFailureHint(std::runtime_error("Connection refused"))); } +namespace +{ + +DB::S3::PocoHTTPClientConfiguration networkFailureClientConfiguration() +{ + DB::RemoteHostFilter remote_host_filter; + return DB::S3::ClientFactory::instance().createClientConfiguration( + "some-region", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /* s3_slow_all_threads_after_network_error = */ true, + /* s3_slow_all_threads_after_retryable_error = */ true, + /* enable_s3_requests_logging = */ false, + /* for_disk_s3 = */ false, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); +} + +/// A client whose `PutObject` always fails with a `NETWORK_CONNECTION` `AWSError` carrying `text` +/// verbatim -- shaped exactly as `PocoHTTPClient` shapes a real connection failure (empty exception +/// name, the Poco text as the message) -- so a test built on it proves `WriteBufferFromS3`'s rethrow, +/// not a hand-built exception, is what `isConnectFailureHint` above actually has to classify. +struct NetworkFailurePutClient : DB::S3::Client +{ + explicit NetworkFailurePutClient(std::string text_) + : DB::S3::Client( + /*max_retries=*/100, + DB::S3::ServerSideEncryptionKMSConfig(), + std::make_shared("", ""), + networkFailureClientConfiguration(), + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + DB::S3::ClientSettings{ + .use_virtual_addressing = true, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }) + , text(std::move(text_)) + { + } + + Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest &) const override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::NETWORK_CONNECTION, "", text, /*retryable=*/false); + } + + std::string text; +}; + +} + +/// The classifier above reads the Poco text a connection failure carries off an `S3Exception`; this +/// pins that the REAL `WriteBufferFromS3` rethrow every CAS conditional write goes through -- not a +/// hand-built exception -- hands the caller that text unchanged, under `NETWORK_CONNECTION`. +TEST(CASRequestsConnectHint, WriteBufferFromS3SurfacesTheConnectFailureTextUnchanged) +{ + for (const char * text : {"Cannot assign requested address", "Connection refused", "No route to host", + "Network is unreachable", "connect timed out"}) + { + auto client = std::make_shared(text); + DB::WriteSettings write_settings; + write_settings.object_storage_retry_profile = DB::ObjectStorageRetryProfile::SingleAttempt; + DB::S3::S3RequestSettings request_settings; + DB::WriteBufferFromS3 buffer( + client, "bucket", "network_text", DB::DBMS_DEFAULT_BUFFER_SIZE, request_settings, + /*blob_log_=*/nullptr, /*object_metadata_=*/std::nullopt, /*schedule_=*/{}, write_settings); + buffer.write('A'); + try + { + buffer.finalize(); + FAIL() << "the injected failure must surface"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::NETWORK_CONNECTION) << text; + EXPECT_THAT(e.message(), testing::HasSubstr(text)); + } + } +} + namespace { std::exception_ptr connectHint() @@ -2410,6 +2543,78 @@ TEST(CASRequestsConnectHint, RefusalAfterAnEarlierAmbiguitySettlesByRead) EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load() - hints_before, 0u); } +/// A single exception can ALSO be both refreshable-credential-class (`isRefreshableCredentialError` +/// matches on the exception NAME, independent of the S3 error code) and hint-text +/// (`isConnectFailureHint` matches on the code and the message). The credential refresh drives the +/// reissue here, not the hint, so the hint counter must stay put. +TEST(CASRequestsConnectHint, RefreshedCredentialTextDoesNotDoubleCountTheHint) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + backend->failNextWriteWith("k", std::make_exception_ptr(DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 99, Connection refused: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION, "ExpiredToken"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto hints_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_EQ(backend->getTotal(), 0u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + /// The refresh -- not the hint's flat pause -- drove the reissue, so the hint counter must not move + /// even though the exception's code and text also match `isConnectFailureHint`. + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load() - hints_before, 0u); +} + +/// The counter's ambiguity-precedence twin: the credential-owned reissue above requires +/// `!state.any_ambiguous`, so an earlier ambiguity of this inner write keeps it from applying even +/// though attempt 2's exception matches the refreshable-credential class. Attempt 2 is then reissued +/// by the ordinary hint mechanism instead -- flat-paused, and after the resolve read attempt 1 still +/// owes -- so the hint counter must count it. +TEST(CASRequestsConnectHint, CredentialRefreshAfterAnEarlierAmbiguityStillCountsTheHint) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + backend->injectAmbiguousWrite("k"); /// attempt 1: ordinary ambiguity -> read, backoff + bool hint_fired_on_second_attempt = false; + backend->onBeforeWrite("k", [&] + { + if (backend->writeTotal() == 2) + { + EXPECT_EQ(backend->getTotal(), 1u) << "attempt 1's ambiguity read must already have run"; + hint_fired_on_second_attempt = true; + throw DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 99, Cannot assign requested address: 10.0.0.1:9000", + Aws::S3::S3Errors::NETWORK_CONNECTION, "ExpiredToken"); /// hint AND credential-refreshable + } + }); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto hints_before = ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 3u); + /// One read (attempt 1's) settles the earlier ambiguity; attempt 2's hint reissue skips its own + /// read, exactly as `EarlierAmbiguityStillSettlesByRead` pins for a non-credential hint. + EXPECT_EQ(backend->getTotal(), 1u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + ASSERT_EQ(clock.sleeps.size(), 2u); + EXPECT_LE(clock.sleeps[0], 200u); /// the backoff after attempt 1's ambiguity read + EXPECT_EQ(clock.sleeps[1], 50u); /// the flat pause after attempt 2's hint, not a credential backoff + EXPECT_TRUE(hint_fired_on_second_attempt); + /// The hint mechanism, not a credential-owned reissue, actually resent this attempt, so the counter + /// counts it even though the exception's name also matches the refreshable-credential class. + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestConnectFailureHint].load() - hints_before, 1u); +} + TEST(CASRequestsConnectHint, GatesRefuseTheReissue) { /// Deadline: hints until the window closes. @@ -2483,4 +2688,346 @@ TEST(CASRequestsConnectHint, AmbiguityAfterHintsStartsAtFirstBackoff) EXPECT_LE(clock.sleeps[2], 200u); } +TEST(CASRequestsFuse, MatcherPrecedence) +{ + using Aws::S3::S3Errors; + /// The generic transport-timeout text is Poco's exception name, pinned here. + EXPECT_THAT(Poco::TimeoutException("the socket").displayText(), testing::StartsWith("Timeout")); + const DB::S3Exception fuse("Poco::Exception. Code: 1000, e.code() = 0, Timeout: the socket", S3Errors::NETWORK_CONNECTION); + EXPECT_TRUE(isFirstAttemptFuseTimeout(fuse, 1)); + EXPECT_FALSE(isFirstAttemptFuseTimeout(fuse, 2)); + const DB::S3Exception hint("Poco::Exception. Code: 1000, e.code() = 0, Timeout: connect timed out: 10.0.0.1:9", S3Errors::NETWORK_CONNECTION); + EXPECT_FALSE(isFirstAttemptFuseTimeout(hint, 1)); /// the connect-failure hint owns it + EXPECT_TRUE(isConnectFailureHint(hint)); + EXPECT_FALSE(isFirstAttemptFuseTimeout(DB::S3Exception("Connection reset by peer", S3Errors::NETWORK_CONNECTION), 1)); + EXPECT_FALSE(isFirstAttemptFuseTimeout(DB::S3Exception("Timeout", S3Errors::INTERNAL_FAILURE), 1)); +} + +namespace +{ +std::exception_ptr fuseTimeout() +{ + return std::make_exception_ptr(DB::S3Exception("Poco::Exception. Code: 1000, e.code() = 0, Timeout: the socket", + Aws::S3::S3Errors::NETWORK_CONNECTION)); +} +} + +TEST(CASRequestsFuse, FirstAttemptTimeoutReissuesWithoutSleep) +{ + /// Write: the settle read still runs (the request may have been sent), then a no-sleep reissue. + { + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", fuseTimeout()); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_EQ(backend->getTotal(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); + } + /// Read: no settle read, no sleep. + { + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "seed"); + backend->resetCounts(); + backend->failNextReadWith("k", fuseTimeout()); + EXPECT_TRUE(op.read("k", Retry::standard()).has_value()); + EXPECT_EQ(backend->getTotal(), 2u); + EXPECT_TRUE(clock.sleeps.empty()); + } + /// LIST: no sleep either. LIST has no `failNextWith`-style armed queue (only write/read/head do), + /// so a small backend that throws the fuse on its first LIST and records the physical attempt + /// number stands in. + { + struct ListFuseOnceBackend : CountingBackend + { + bool armed = true; + std::vector list_attempts; + RawListPage list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) override + { + list_attempts.push_back(access.attemptNo()); + if (armed) + { + armed = false; + std::rethrow_exception(fuseTimeout()); + } + return CountingBackend::list(prefix, cursor, limit, access); + } + }; + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + (void)op.list("p/", "", 10, Retry::standard()); + EXPECT_EQ(backend->list_attempts, (std::vector{1, 2})); + EXPECT_TRUE(clock.sleeps.empty()); + } + /// Attempts 1 and 2 failing: attempt 2 is not a first attempt, so exactly one sleep, after it. + { + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", fuseTimeout()); + backend->failNextWriteWith("k", fuseTimeout()); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::standard()); + ASSERT_TRUE(std::holds_alternative(result)); + EXPECT_EQ(std::get(result).attempts_sent, 3u); + EXPECT_EQ(clock.sleeps.size(), 1u); + } +} + +TEST(CASRequestsFuse, GatesRefuseTheZeroPauseReissue) +{ + /// `setAttemptReservationForTest(1'000)`: the write's own admission reserves two envelopes + /// (`reservedFor(0, 2) == 2000`), which matches a 2000 ms window exactly -- `fits` is `needed <= + /// remaining`, so the boundary admits. The settle read that follows the fuse reserves only one + /// envelope (`reservedFor(0, 1) == 1000`), which still fits even after the clock below has moved. + /// What must NOT fit is the zero-pause reissue's own `reservedFor(0, 2) == 2000`. `FakeClock` never + /// moves on its own -- only a sleep advances it, and this path sleeps none -- so a naive `now()` + /// would see the SAME instant at every one of the four calls this operation makes (the initial + /// `bind`, the write's own admission, the settle read's admission, the reissue's admission) and + /// wrongly admit the reissue too. A real failing attempt spends wall time even though it never + /// lands, so this fixture's clock counts its own calls and adds 1 ms starting from the THIRD one + /// (the settle read's admission) onward: late enough that the write's own admission still sees the + /// pristine window, early enough that the reissue's admission sees one fewer millisecond than it + /// needs. + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextWriteWith("k", fuseTimeout()); + int now_calls = 0; + auto requests = makeRequests(backend, clock); + requests.setAttemptReservationForTest(1'000); + requests.setNowFnForTest([&clock, &now_calls]() -> uint64_t + { + ++now_calls; + return clock.now + (now_calls <= 2 ? 0 : 1); + }); + auto op = requests.admit(); + WriteResult result = op.create("k", "v", Retry::within(2'000)); + const auto * gave_up = std::get_if(&result); + ASSERT_NE(gave_up, nullptr); + EXPECT_EQ(gave_up->why, GaveUp::Why::Deadline); + EXPECT_TRUE(clock.sleeps.empty()) << "the zero-pause reissue never sleeps, even when refused"; + /// The fence, not the deadline, refuses the zero-pause reissue: three `Fence::admit` calls happen + /// in this scenario -- the write's own admission, the settle read's admission, and the reissue's + /// admission -- in that order, so tripping the fence on the THIRD call refuses only the reissue, + /// after the write attempt and its settle read both already went through. + { + FakeClock fence_clock; + auto fence_backend = std::make_shared(); + fence_backend->failNextWriteWith("k", fuseTimeout()); + int admit_calls = 0; + Fence fence{ + [] { return uint64_t{1}; }, + [&](uint64_t, uint64_t) { return ++admit_calls >= 3 ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, + [](uint64_t) {}}; + auto fence_requests = makeRequests(fence_backend, fence_clock, fence); + auto fence_op = fence_requests.admit(); + WriteResult fence_result = fence_op.create("k", "v", Retry::standard()); + const auto * fence_gave_up = std::get_if(&fence_result); + ASSERT_NE(fence_gave_up, nullptr); + EXPECT_EQ(fence_gave_up->why, GaveUp::Why::FenceLost); + EXPECT_TRUE(fence_gave_up->sent_any); + EXPECT_EQ(fence_backend->writeTotal(), 1u) << "the fence refuses before a second write is ever sent"; + } + /// `Retry::once()` never performs a second attempt. + auto once_backend = std::make_shared(); + once_backend->failNextWriteWith("k", fuseTimeout()); + auto once_requests = makeRequests(once_backend, clock); + auto once_op = once_requests.admit(); + (void)once_op.create("k", "v", Retry::once()); + EXPECT_EQ(once_backend->writeTotal(), 1u); +} + +TEST(CASRequestsFuse, ReadLoopZeroPauseKeepsTheBackoffIndex) +{ + struct ReadAttemptRecordingBackend : CountingBackend + { + std::vector read_attempts; + std::optional read(const String & key, TransportAccess & access) override + { + read_attempts.push_back(access.attemptNo()); + return CountingBackend::read(key, access); + } + }; + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "seed"); + backend->read_attempts.clear(); + backend->failNextReadWith("k", fuseTimeout()); + backend->failNextReadWith("k", std::make_exception_ptr(Poco::TimeoutException("attempt 2: an ordinary fault"))); + EXPECT_TRUE(op.read("k", Retry::standard()).has_value()); + ASSERT_EQ(clock.sleeps.size(), 1u); + /// The one sleep is `backoff(1)`: the zero-pause reissue did not advance the index. + EXPECT_LE(clock.sleeps[0], 200u); /// `backoff(1)` is full jitter over [0, 200] ms + /// The transport still sees every physical attempt: the zero-pause reissue (attempt 2) advances + /// `attempt_no` alone, so attempt 3 -- reached only after the one ordinary backoff -- follows it, + /// not a second attempt 1. + EXPECT_EQ(backend->read_attempts, (std::vector{1, 2, 3})); +} + +/// `Retry::once` forbids the REISSUE, not the observation: a fuse a single-attempt read hits still +/// counts (the write path already counts at classification, before its own single-attempt check), it +/// just throws unchanged instead of re-sending. +TEST(CASRequestsFuse, ReadUnderOnceCountsTheFuseWithoutReissuing) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->failNextReadWith("k", fuseTimeout()); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto fuses_before = ProfileEvents::global_counters[ProfileEvents::CASRequestFirstAttemptFuse].load(); + expectThrowsCode(DB::ErrorCodes::S3_ERROR, [&] { (void)op.read("k", Retry::once()); }); + EXPECT_EQ(backend->getTotal(), 1u) << "Retry::once performs no second attempt"; + EXPECT_TRUE(clock.sleeps.empty()); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestFirstAttemptFuse].load() - fuses_before, 1u); +} + +/// The fuse counter's credential-refresh twin of `CASRequestsConnectHint.RefreshedCredentialTextDoesNotDoubleCountTheHint`: +/// a first attempt whose exception is both fuse-text and refreshable-credential-name must be counted +/// as the credential reissue it actually is, not also as a fuse. +TEST(CASRequestsFuse, RefreshedCredentialTextDoesNotDoubleCountTheFuse) +{ + FakeClock clock; + auto backend = std::make_shared(); + backend->setRefreshCredentialsResult(true); + backend->failNextWriteWith("k", std::make_exception_ptr(DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 0, Timeout: the socket", + Aws::S3::S3Errors::NETWORK_CONNECTION, "ExpiredToken"))); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + const auto fuses_before = ProfileEvents::global_counters[ProfileEvents::CASRequestFirstAttemptFuse].load(); + + WriteResult result = op.create("k", "v", Retry::standard()); + const auto * committed = std::get_if(&result); + ASSERT_NE(committed, nullptr); + EXPECT_EQ(committed->attempts_sent, 2u); + EXPECT_FALSE(committed->resolved_by_read); + EXPECT_EQ(backend->getTotal(), 0u); + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + /// The refresh -- not the fuse's immediate reissue -- drove the resend, so the fuse counter must not + /// move even though the exception's code and text also match `isFirstAttemptFuseTimeout`. + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestFirstAttemptFuse].load() - fuses_before, 0u); +} + +/// The read loop's own twin of `RefreshedCredentialTextDoesNotDoubleCountTheFuse`: a first read attempt +/// whose exception is both fuse-text and refreshable-credential-name is a credential reissue, not a +/// fuse, so the counter must not move even though the reissue itself is immediate, exactly like a fuse. +TEST(CASRequestsFuse, ReadRefreshedCredentialTextDoesNotDoubleCountTheFuse) +{ + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + orThrow(op.create("k", "v", Retry::standard()), "seed"); + backend->resetCounts(); + backend->setRefreshCredentialsResult(true); + backend->failNextReadWith("k", std::make_exception_ptr(DB::S3Exception( + "Poco::Exception. Code: 1000, e.code() = 0, Timeout: the socket", + Aws::S3::S3Errors::NETWORK_CONNECTION, "ExpiredToken"))); + const auto fuses_before = ProfileEvents::global_counters[ProfileEvents::CASRequestFirstAttemptFuse].load(); + + const auto seen = op.read("k", Retry::standard()); + ASSERT_TRUE(seen.has_value()); + EXPECT_EQ(seen->bytes, "v"); + EXPECT_EQ(backend->getTotal(), 2u) << "the failed attempt and its immediate reissue both reached the store"; + EXPECT_EQ(backend->refreshCredentialsCalls(), 1u); + EXPECT_TRUE(clock.sleeps.empty()); + /// The refresh -- not the fuse's immediate reissue -- drove the resend, so the fuse counter must not + /// move even though the exception's code and text also match `isFirstAttemptFuseTimeout`. + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRequestFirstAttemptFuse].load() - fuses_before, 0u); +} + #endif + +TEST(CASRequestBudget, EnvelopeIsValidatedNotTheBareAttempt) +{ + CasRequestBudget budget{.attempt_timeout_ms = 5000, .lease_safety_margin_ms = 2000, .connect_timeout_cap_ms = 1000}; + EXPECT_EQ(budget.attemptEnvelopeMs(), 7000u); + EXPECT_EQ((CasRequestBudget{.attempt_timeout_ms = 5000, .lease_safety_margin_ms = 2000, .connect_timeout_cap_ms = std::nullopt}.attemptEnvelopeMs()), 5000u); + /// Defaults with the default TTL / period are accepted. + EXPECT_NO_THROW(validateCasRequestBudget(budget, 30000, 10000, /*background_renewal=*/true)); + /// A zero attempt timeout would reserve nothing while the request keeps the disk's own timeout. + expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] + { + validateCasRequestBudget(CasRequestBudget{.attempt_timeout_ms = 0, .lease_safety_margin_ms = 2000, + .connect_timeout_cap_ms = std::nullopt}, 30000, 10000, true); + }); + /// The old inequality (attempt <= TTL - margin - period: 5000 <= 13000) accepted this; two envelopes + /// of 15 s do not fit a 25 s lease behind a 10 s period and a 2 s margin. + const CasRequestBudget wide{.attempt_timeout_ms = 5000, .lease_safety_margin_ms = 2000, .connect_timeout_cap_ms = 5000}; + try + { + validateCasRequestBudget(wide, 25000, 10000, true); + FAIL() << "must refuse"; + } + catch (const DB::Exception & e) + { + EXPECT_THAT(e.message(), testing::HasSubstr("envelope")); + EXPECT_THAT(e.message(), testing::HasSubstr("15000")); + } + /// Without background renewal only `envelope + margin < TTL` applies (15000 + 2000 < 25000). + EXPECT_NO_THROW(validateCasRequestBudget(wide, 25000, 10000, /*background_renewal=*/false)); + /// Saturation: absurd values fail closed rather than wrap. + expectThrowsCode(DB::ErrorCodes::BAD_ARGUMENTS, [&] + { + validateCasRequestBudget(CasRequestBudget{.attempt_timeout_ms = std::numeric_limits::max(), + .lease_safety_margin_ms = 1, .connect_timeout_cap_ms = 1}, + 30000, 10000, true); + }); +} + +TEST(CASRequests, ReservationIsTheEnvelope) +{ + struct EnvelopeBackend : InMemoryBackend + { + uint64_t attemptTimeoutMs() const override { return 5000; } + uint64_t attemptEnvelopeMs() const override { return 7000; } + }; + FakeClock clock; + auto backend = std::make_shared(); + auto requests = makeRequests(backend, clock); + auto op = requests.admit(); + /// A write reserves two envelopes: 14 s fits a 14 s window, 13.999 s does not. + EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::within(14'000)))); + const WriteResult refused = op.create("k2", "v", Retry::within(13'999)); + const auto * gave_up = std::get_if(&refused); + ASSERT_NE(gave_up, nullptr); + EXPECT_FALSE(gave_up->sent_any); +} + +/// Every `Backend` decorator that forwards `attemptTimeoutMs` to an inner backend must forward +/// `attemptEnvelopeMs` too, or the default (`attemptEnvelopeMs() { return attemptTimeoutMs(); }`) +/// silently drops the inner backend's connect contribution -- exactly the gap `Pool::open`'s +/// `InstrumentedBackend` wrapper had. Pin the forwarding through the same engine construction +/// production uses. +TEST(CASRequests, ReservationIsTheEnvelopeThroughInstrumentedBackend) +{ + struct EnvelopeBackend : InMemoryBackend + { + uint64_t attemptTimeoutMs() const override { return 5000; } + uint64_t attemptEnvelopeMs() const override { return 7000; } + }; + FakeClock clock; + auto inner = std::make_shared(); + auto wrapped = std::make_shared(inner); + ASSERT_EQ(wrapped->attemptTimeoutMs(), 5000u); + ASSERT_EQ(wrapped->attemptEnvelopeMs(), 7000u) << "InstrumentedBackend must forward the envelope, not fall back to the bare attempt timeout"; + auto requests = makeRequests(wrapped, clock); + auto op = requests.admit(); + /// Same boundary as ReservationIsTheEnvelope, now through the wrapper `Pool::open` actually uses. + EXPECT_TRUE(std::holds_alternative(op.create("k", "v", Retry::within(14'000)))); + const WriteResult refused = op.create("k2", "v", Retry::within(13'999)); + const auto * gave_up = std::get_if(&refused); + ASSERT_NE(gave_up, nullptr); + EXPECT_FALSE(gave_up->sent_any); +} diff --git a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp new file mode 100644 index 000000000000..1e5f70782735 --- /dev/null +++ b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp @@ -0,0 +1,757 @@ +#include + +#include "config.h" + +#if USE_AWS_S3 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +/// The single-attempt client clone must cap its connect timeout at the value the mount froze at open, +/// never at the disk's (possibly wider, possibly reloaded, possibly unbounded) own connect timeout. + +namespace +{ + +/// A `PocoHTTPClientConfiguration` that never resolves a real socket: `endpointOverride` points at a +/// port nothing listens on, so a test that never issues a request (every assertion here reads +/// `getClientConfiguration()`, which needs no network) never blocks or flakes on connection refusal. +DB::S3::PocoHTTPClientConfiguration clientConfigurationForTest(long connect_ms) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /* s3_slow_all_threads_after_network_error = */ true, + /* s3_slow_all_threads_after_retryable_error = */ true, + /* enable_s3_requests_logging = */ false, + /* for_disk_s3 = */ true, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); + cfg.endpointOverride = "http://127.0.0.1:1"; + cfg.connectTimeoutMs = connect_ms; + cfg.requestTimeoutMs = 30000; + return cfg; +} + +DB::S3::ClientSettings clientSettingsForTest() +{ + return DB::S3::ClientSettings{ + .use_virtual_addressing = false, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }; +} + +/// A `` config section carrying `connect_timeout_ms`, for driving a reload through the real +/// `applyNewSettings` path (as a live disk's config reload would) rather than swapping the client +/// directly. Explicit static credentials keep the reload from falling through to the EC2 instance +/// metadata credentials provider (no access/secret key configured means "try every other provider"), +/// which would otherwise probe an unreachable metadata endpoint on every reload. +Poco::AutoPtr configWithConnectTimeout(long connect_timeout_ms) +{ + std::istringstream xml_stream( // STYLE_CHECK_ALLOW_STD_STRING_STREAM + "" + "" + std::to_string(connect_timeout_ms) + "" + "ACCESS_KEY_ID" + "SECRET_ACCESS_KEY" + ""); + return new Poco::Util::XMLConfiguration(xml_stream); +} + +std::shared_ptr makeStorageForTest(long connect_ms) +{ + auto client = DB::S3::ClientFactory::instance().create( + clientConfigurationForTest(connect_ms), clientSettingsForTest(), + "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); + return std::make_shared( + std::move(client), std::make_unique(), + DB::S3::URI("http://127.0.0.1:1/bucket/"), DB::S3Capabilities{}, + DB::ObjectStorageKeyGeneratorPtr{}, "disk"); +} + +/// A handler that always answers with a canned, verb-appropriate response after sleeping `delay` -- +/// simulating a slow-but-eventually-answering S3 endpoint. The sleep is deliberate test scaffolding for +/// a real elapsed-time discriminator, not a workaround for a race condition. Every request increments +/// `requests_seen`, the only way a caller can prove a client's retry strategy never reissued. +class DelayedResponseRequestHandler : public Poco::Net::HTTPRequestHandler +{ + std::atomic & requests_seen; + std::chrono::milliseconds delay; + std::function respond; + +public: + DelayedResponseRequestHandler( + std::atomic & requests_seen_, + std::chrono::milliseconds delay_, + std::function respond_) + : requests_seen(requests_seen_), delay(delay_), respond(std::move(respond_)) + { + } + + void handleRequest(Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) override + { + ++requests_seen; + std::this_thread::sleep_for(delay); + respond(response); + } +}; + +class DelayedResponseRequestHandlerFactory : public Poco::Net::HTTPRequestHandlerFactory +{ + std::atomic & requests_seen; + std::chrono::milliseconds delay; + std::function respond; + + Poco::Net::HTTPRequestHandler * createRequestHandler(const Poco::Net::HTTPServerRequest &) override + { + return new DelayedResponseRequestHandler(requests_seen, delay, respond); + } + +public: + DelayedResponseRequestHandlerFactory( + std::atomic & requests_seen_, + std::chrono::milliseconds delay_, + std::function respond_) + : requests_seen(requests_seen_), delay(delay_), respond(std::move(respond_)) + { + } + + ~DelayedResponseRequestHandlerFactory() override = default; +}; + +/// A real local HTTP server standing in for S3, one verb at a time: every request gets the same canned +/// response after `delay`. Pointing a genuine `S3ObjectStorage` at it and comparing a `Default` call +/// (succeeds -- the delay is well under the base client's request timeout) against a `SingleAttempt` +/// call with a short caller timeout (times out, and the server counts exactly one request) is what +/// actually discriminates PRODUCTION client selection: no subclass stands between the test and +/// `S3ObjectStorage`'s own verb implementations. +class DelayedResponseServer +{ + std::unique_ptr server_socket; + Poco::SharedPtr handler_factory; + Poco::AutoPtr server_params; + std::unique_ptr server; + std::atomic requests_seen{0}; + +public: + DelayedResponseServer(std::chrono::milliseconds delay, std::function respond) + : server_socket(std::make_unique(0)) + , handler_factory(new DelayedResponseRequestHandlerFactory(requests_seen, delay, std::move(respond))) + , server_params(new Poco::Net::HTTPServerParams()) + , server(std::make_unique(handler_factory, *server_socket, server_params)) + { + server->start(); + } + + std::string getUrl() const { return "http://" + server_socket->address().toString(); } + size_t requestsSeen() const { return requests_seen.load(); } + void resetRequestsSeen() { requests_seen = 0; } +}; + +/// A TCP listener whose accept queue is permanently full of connections it never accept()s: `backlog = +/// 1` requests a one-entry queue, but Linux's actual capacity for a given backlog is not exactly that +/// number (historically `backlog + 1`, and kernel-version-dependent besides), so a single pre-filled +/// connection is not reliably enough to make the very next connect attempt stall. Instead, this keeps +/// connecting -- each attempt bounded by a short timeout -- until an attempt itself times out: that IS +/// the proof the queue is now genuinely full, whatever slack the nominal backlog actually bought, and +/// every established connection up to that point is kept (never accepted) to hold the queue full for +/// the rest of this object's life. Once full, every FURTHER inbound SYN finds no room: Linux's default +/// `tcp_abort_on_overflow = 0` then just drops that SYN instead of answering it (no RST, no SYN-ACK), +/// so a connecting client's kernel silently retransmits in the background while the client's OWN +/// socket-connect timeout -- not the kernel's SYN retry timer -- is what actually bounds how long a +/// caller waits. Nothing here ever completes a handshake with a real peer, so a call against this +/// listener can only ever fail on CONNECT, never on request/response -- unlike `DelayedResponseServer` +/// above, which answers every request and so can only discriminate the request/response phase. +class ConnectStallServer +{ + Poco::Net::ServerSocket listener; + std::vector prefill_connections; + +public: + ConnectStallServer() : listener(Poco::Net::SocketAddress("127.0.0.1", 0), /*backlog=*/1) + { + /// The loop bound is only a safety net (the queue always fills well before it on Linux): without + /// one, an environment where the queue somehow never fills would hang the constructor forever. + for (size_t i = 0; i < 64; ++i) + { + Poco::Net::StreamSocket prefill; + try + { + prefill.connect(listener.address(), Poco::Timespan(200 * 1000)); + } + catch (const Poco::TimeoutException &) + { + return; + } + prefill_connections.push_back(prefill); + } + throw Poco::RuntimeException("ConnectStallServer: accept queue never filled"); + } + + std::string getUrl() const { return "http://" + listener.address().toString(); } +}; + +/// `ConnectStallServer` relies on Linux dropping the overflow SYN silently, which only happens while +/// `net.ipv4.tcp_abort_on_overflow` stays at its default 0; a host with it set to 1 resets the +/// connection instead, so the queue-full state this fixture depends on never actually stalls a connect. +/// An unreadable file is treated the same as "1": this is a fixture precondition, not the behaviour +/// under test, so silently assuming the default would risk fencing that at the fixture layer. +bool tcpAbortOnOverflowPreventsStallServer() +{ + std::ifstream sysctl_file("/proc/sys/net/ipv4/tcp_abort_on_overflow"); + char value = '\0'; + if (!(sysctl_file >> value)) + return true; + return value != '0'; +} + +/// A genuine `S3ObjectStorage` for the CONNECT-phase discriminator: `connect_timeout_ms` governs only +/// the base client's TCP connect deadline, while `requestTimeoutMs` is set far wider so a call against +/// `ConnectStallServer` can only ever fail on connect, never on request/response (the peer there never +/// completes a handshake at all, so no request is ever sent). Adaptive timeouts are disabled for the +/// same reason `makeDispatchStorageForTest` disables them: the adaptive strategy would shrink the first +/// attempt's own connect deadline below whatever this function configures. +std::shared_ptr makeConnectStallStorageForTest(const std::string & endpoint, long connect_timeout_ms) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /* s3_slow_all_threads_after_network_error = */ false, + /* s3_slow_all_threads_after_retryable_error = */ false, + /* enable_s3_requests_logging = */ false, + /* for_disk_s3 = */ true, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); + cfg.endpointOverride = endpoint; + cfg.connectTimeoutMs = connect_timeout_ms; + cfg.requestTimeoutMs = 30000; + cfg.s3_use_adaptive_timeouts = false; + auto client = DB::S3::ClientFactory::instance().create( + cfg, clientSettingsForTest(), "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); + return std::make_shared( + std::move(client), std::make_unique(), + DB::S3::URI(endpoint + "/test-bucket/"), DB::S3Capabilities{}, + DB::ObjectStorageKeyGeneratorPtr{}, "disk"); +} + +/// Runs `attempt`, expecting a connection-class failure -- a stalled connect is classified by +/// `PocoHTTPClient` as `Aws::Client::CoreErrors::NETWORK_CONNECTION` from the `Poco::TimeoutException` +/// its connect poll raises, never as a request/response error -- and returns how long it took. +template +std::chrono::milliseconds expectConnectFailureAndMeasure(F && attempt) +{ + const auto start = std::chrono::steady_clock::now(); + try + { + attempt(); + ADD_FAILURE() << "expected a connection failure, the call unexpectedly succeeded"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::NETWORK_CONNECTION); + } + return std::chrono::duration_cast(std::chrono::steady_clock::now() - start); +} + +/// A genuine `S3ObjectStorage` pointed at `endpoint`. `base_request_timeout_ms` is the base client's +/// request AND connect timeout -- comfortably above the server's simulated delay, so a `Default` call +/// succeeds. No SDK-level retry (`RetryStrategy{.max_retries = 0}`, +/// `s3_slow_all_threads_after_retryable_error = false`): a retry would blur "the single-attempt clone +/// made exactly one request" into "the SDK also tried again". +std::shared_ptr makeDispatchStorageForTest(const std::string & endpoint, long base_request_timeout_ms) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /* s3_slow_all_threads_after_network_error = */ false, + /* s3_slow_all_threads_after_retryable_error = */ false, + /* enable_s3_requests_logging = */ false, + /* for_disk_s3 = */ true, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); + cfg.endpointOverride = endpoint; + cfg.connectTimeoutMs = base_request_timeout_ms; + cfg.requestTimeoutMs = base_request_timeout_ms; + /// The adaptive-timeout strategy gives the FIRST attempt a much shorter deadline than + /// `requestTimeoutMs` and only widens it on a later retry -- with SDK retries disabled above, that + /// first (short) deadline is the only one this client ever gets, which would time out well under + /// `server_delay` regardless of `requestTimeoutMs`. Off, so `requestTimeoutMs` governs uniformly. + cfg.s3_use_adaptive_timeouts = false; + auto client = DB::S3::ClientFactory::instance().create( + cfg, clientSettingsForTest(), "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); + return std::make_shared( + std::move(client), std::make_unique(), + DB::S3::URI(endpoint + "/test-bucket/"), DB::S3Capabilities{}, + DB::ObjectStorageKeyGeneratorPtr{}, "disk"); +} + +DB::ContextPtr contextForTest() +{ + return getContext().context; +} + +} + +/// Test 6c of the spec: the clone's connect cap is the MIN of the base client's own connect timeout +/// and the requested cap, a configured-zero base is treated as unbounded (never "no limit"), the cache +/// key is the (request timeout, cap) pair, and a reloaded base client cannot widen a clone rebuilt for +/// the same cap. +TEST(S3SingleAttemptClient, ConnectTimeoutIsCappedAndFrozen) +{ + auto storage = makeStorageForTest(20000); + auto clone = storage->getSingleAttemptClient(/*request_timeout_ms=*/5000, /*connect_timeout_cap_ms=*/5000); + EXPECT_EQ(clone->getClientConfiguration().connectTimeoutMs, 5000); + EXPECT_EQ(clone->getClientConfiguration().requestTimeoutMs, 5000); + + auto narrow = makeStorageForTest(1000); + EXPECT_EQ(narrow->getSingleAttemptClient(5000, 5000)->getClientConfiguration().connectTimeoutMs, 1000); + /// A base of 0 means unbounded to Poco: it resolves to the cap, never to "no limit". + EXPECT_EQ(makeStorageForTest(0)->getSingleAttemptClient(5000, 1000)->getClientConfiguration().connectTimeoutMs, 1000); + /// Two caps under one request timeout are two clones: the cache key is the pair. + EXPECT_NE(narrow->getSingleAttemptClient(5000, 1000).get(), narrow->getSingleAttemptClient(5000, 500).get()); + + /// The reload path replaces the base client with a wider connect timeout, through the real + /// `applyNewSettings` config-reload path (as `SYSTEM RELOAD CONFIG` would drive it); a clone + /// rebuilt for the frozen cap 1000 stays at 1000. + auto reloaded = makeStorageForTest(1000); + (void)reloaded->getSingleAttemptClient(5000, 1000); + reloaded->applyNewSettings(*configWithConnectTimeout(5000), "disk", contextForTest(), + DB::IObjectStorage::ApplyNewSettingsOptions{.allow_client_change = true}); + EXPECT_EQ(reloaded->getSingleAttemptClient(5000, 1000)->getClientConfiguration().connectTimeoutMs, 1000); +} + +/// The freeze computation `openPoolView` uses to build `pool_config.cas_request_budget.connect_timeout_cap_ms`, +/// isolated from any particular verb: the cap is the MIN of the base client's own connect timeout and +/// the attempt timeout, a configured-zero base normalizes to the attempt timeout itself (never "no +/// limit"), and the resulting envelope arithmetic matches `CasRequestBudget::attemptEnvelopeMs`. +TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapSnapshot) +{ + /// A base client with connectTimeoutMs = 1000 and cas_attempt_timeout_ms = 5000: the narrower of + /// the two wins, and the envelope is attempt + 2 * cap = 7000. + auto storage = makeStorageForTest(1000); + const auto cap = DB::ContentAddressedMetadataStorage::freezeConnectTimeoutCapMs(storage, /*cas_attempt_timeout_ms=*/5000); + ASSERT_TRUE(cap.has_value()); + EXPECT_EQ(*cap, 1000u); + DB::Cas::CasRequestBudget budget{.attempt_timeout_ms = 5000, .connect_timeout_cap_ms = cap}; + EXPECT_EQ(budget.attemptEnvelopeMs(), 7000u); + + /// A base client with connectTimeoutMs = 0 (Poco "unbounded") and a TTL wide enough for the + /// resulting envelope (60000, per the spec's test 6e): the cap normalizes to the attempt timeout + /// itself, never to "no limit" -- a snapshot computing `min(0, attempt)` would report 0 here. + auto unbounded_storage = makeStorageForTest(0); + const auto wide_cap = DB::ContentAddressedMetadataStorage::freezeConnectTimeoutCapMs(unbounded_storage, /*cas_attempt_timeout_ms=*/5000); + ASSERT_TRUE(wide_cap.has_value()); + EXPECT_EQ(*wide_cap, 5000u); + DB::Cas::CasRequestBudget wide_budget{.attempt_timeout_ms = 5000, .connect_timeout_cap_ms = wide_cap}; + EXPECT_EQ(wide_budget.attemptEnvelopeMs(), 15000u); + EXPECT_NO_THROW(DB::Cas::validateCasRequestBudget(wide_budget, /*mount_lease_ttl_ms=*/60000, /*mount_renew_period_ms=*/10000, + /*background_renewal=*/false)); + + /// A storage with no S3 client (not exercised here -- every storage above is S3) freezes `nullopt`; + /// covered directly by `S3ObjectStorage::tryGetS3StorageClient` returning null for a non-S3 storage + /// and `freezeConnectTimeoutCapMs` short-circuiting on it. +} + +/// Every public verb whose retry profile is selectable is proven here to reach the client +/// `S3ObjectStorage::clientForRetryProfile` (private) actually picks for it -- through the storage's OWN +/// verb implementations, never a subclass override standing in for them. Per verb: a `Default` call +/// against a server that answers after `server_delay` succeeds (its client keeps the wide base timeout); +/// the SAME call under `SingleAttempt` with a caller timeout well under `server_delay` times out, and +/// the server counts exactly one request -- proving both that the short-timeout single-attempt clone +/// was selected (not the base client) and that its `SingleAttemptRetryStrategy` performs no +/// SDK-transparent retry. +TEST(CASEnvelopeWiring, ProductionDispatchSelectsTheFrozenSingleAttemptClientPerVerb) +{ + (void)contextForTest(); // getThreadPoolWriter/BlobStorageLogWriter::create fall back to the global context + + constexpr auto server_delay = std::chrono::milliseconds(1000); + constexpr long base_request_timeout_ms = 10000; + constexpr uint64_t single_attempt_timeout_ms = 100; + + auto singleAttemptRequest = [] + { + return DB::ObjectStorageControlRequest{ + .profile = DB::ObjectStorageRetryProfile::SingleAttempt, + .attempt_timeout_ms = single_attempt_timeout_ms, + .connect_timeout_cap_ms = single_attempt_timeout_ms}; + }; + + /// PUT: writeObject; the profile rides on WriteSettings, not an ObjectStorageControlRequest. + { + DelayedResponseServer server(server_delay, [](Poco::Net::HTTPServerResponse & response) + { + response.set("ETag", "\"put-etag\""); + response.setContentLength(0); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.send(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_request_timeout_ms); + + auto put = [&](DB::ObjectStorageRetryProfile profile, uint64_t timeout_ms) + { + DB::WriteSettings write_settings; + write_settings.object_storage_retry_profile = profile; + write_settings.object_storage_attempt_timeout_ms = timeout_ms; + write_settings.object_storage_connect_timeout_cap_ms = timeout_ms; + auto buffer = storage->writeObject( + DB::StoredObject("put-key"), DB::WriteMode::Rewrite, {}, DB::DBMS_DEFAULT_BUFFER_SIZE, write_settings); + buffer->write('A'); + buffer->finalize(); + }; + + EXPECT_NO_THROW(put(DB::ObjectStorageRetryProfile::Default, 0)); + EXPECT_EQ(server.requestsSeen(), 1u); + + server.resetRequestsSeen(); + EXPECT_THROW(put(DB::ObjectStorageRetryProfile::SingleAttempt, single_attempt_timeout_ms), DB::Exception); + EXPECT_EQ(server.requestsSeen(), 1u); + } + + /// HEAD: tryGetObjectMetadataWithNativeToken's ObjectStorageControlRequest-taking overload. + { + DelayedResponseServer server(server_delay, [](Poco::Net::HTTPServerResponse & response) + { + response.set("ETag", "\"head-etag\""); + response.setContentLength(5); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.send(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_request_timeout_ms); + + EXPECT_TRUE(storage->tryGetObjectMetadataWithNativeToken( + "head-key", /*with_tags=*/false, DB::ObjectStorageControlRequest{}).has_value()); + EXPECT_EQ(server.requestsSeen(), 1u); + + server.resetRequestsSeen(); + EXPECT_THROW( + storage->tryGetObjectMetadataWithNativeToken("head-key", /*with_tags=*/false, singleAttemptRequest()), + DB::Exception); + EXPECT_EQ(server.requestsSeen(), 1u); + } + + /// Conditional DELETE: removeObjectIfTokenMatches's ObjectStorageControlRequest-taking overload. + { + DelayedResponseServer server(server_delay, [](Poco::Net::HTTPServerResponse & response) + { + response.setStatus(Poco::Net::HTTPResponse::HTTP_NO_CONTENT); + response.setContentLength(0); + response.send(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_request_timeout_ms); + + auto result = storage->removeObjectIfTokenMatches( + DB::StoredObject("delete-key"), "\"etag\"", DB::ObjectStorageControlRequest{}); + EXPECT_EQ(result.outcome, DB::ConditionalRemoveOutcome::Removed); + EXPECT_EQ(server.requestsSeen(), 1u); + + server.resetRequestsSeen(); + EXPECT_THROW( + storage->removeObjectIfTokenMatches(DB::StoredObject("delete-key"), "\"etag\"", singleAttemptRequest()), + DB::Exception); + EXPECT_EQ(server.requestsSeen(), 1u); + } + + /// Bulk DELETE: removeObjectsIfExistUnderProfile (one DeleteObjects request for the whole batch). + { + DelayedResponseServer server(server_delay, [](Poco::Net::HTTPServerResponse & response) + { + static const std::string body = + "" + ""; + response.setContentType("application/xml"); + response.setContentLength(body.size()); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + auto & out = response.send(); + out << body; + out.flush(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_request_timeout_ms); + + EXPECT_NO_THROW(storage->removeObjectsIfExistUnderProfile( + {DB::StoredObject("bulk-delete-key")}, DB::ObjectStorageControlRequest{})); + EXPECT_EQ(server.requestsSeen(), 1u); + + server.resetRequestsSeen(); + EXPECT_THROW( + storage->removeObjectsIfExistUnderProfile({DB::StoredObject("bulk-delete-key")}, singleAttemptRequest()), + DB::Exception); + EXPECT_EQ(server.requestsSeen(), 1u); + } + + /// LIST: iterate's ObjectStorageControlRequest-taking overload. The ListObjectsV2 call happens + /// lazily, on the async iterator's first `isValid()`. + { + DelayedResponseServer server(server_delay, [](Poco::Net::HTTPServerResponse & response) + { + static const std::string body = + "" + "" + "test-bucket01000" + "false"; + response.setContentType("application/xml"); + response.setContentLength(body.size()); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + auto & out = response.send(); + out << body; + out.flush(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_request_timeout_ms); + + auto default_iterator = storage->iterate("p/", /*max_keys=*/10, /*with_tags=*/false, {}, DB::ObjectStorageControlRequest{}); + EXPECT_NO_THROW(default_iterator->isValid()); + EXPECT_EQ(server.requestsSeen(), 1u); + + server.resetRequestsSeen(); + auto single_attempt_iterator = storage->iterate("p/", /*max_keys=*/10, /*with_tags=*/false, {}, singleAttemptRequest()); + EXPECT_THROW(single_attempt_iterator->isValid(), DB::Exception); + EXPECT_EQ(server.requestsSeen(), 1u); + } + + /// GET: readObject; the profile rides on ReadSettings. The request happens lazily, on the buffer's + /// first read. + { + DelayedResponseServer server(server_delay, [](Poco::Net::HTTPServerResponse & response) + { + static const std::string body = "hello"; + response.set("ETag", "\"get-etag\""); + response.setContentType("binary/octet-stream"); + response.setContentLength(body.size()); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + auto & out = response.send(); + out << body; + out.flush(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_request_timeout_ms); + + auto get = [&](DB::ObjectStorageRetryProfile profile, uint64_t timeout_ms) + { + DB::ReadSettings read_settings; + read_settings.object_storage_retry_profile = profile; + read_settings.object_storage_attempt_timeout_ms = timeout_ms; + read_settings.object_storage_connect_timeout_cap_ms = timeout_ms; + auto buffer = storage->readObject(DB::StoredObject("get-key"), read_settings); + std::string content; + DB::readStringUntilEOF(content, *buffer); + return content; + }; + + EXPECT_EQ(get(DB::ObjectStorageRetryProfile::Default, 0), "hello"); + EXPECT_EQ(server.requestsSeen(), 1u); + + server.resetRequestsSeen(); + EXPECT_THROW(get(DB::ObjectStorageRetryProfile::SingleAttempt, single_attempt_timeout_ms), DB::Exception); + EXPECT_EQ(server.requestsSeen(), 1u); + } +} + +/// The test above proves production dispatch selects a short-REQUEST-timeout clone, but every server +/// there answers every request -- it can never tell whether the frozen `connect_timeout_cap_ms` reaches +/// the CONNECTION phase at all, only whether SOME clone with a short deadline was picked. This test +/// closes that gap with `ConnectStallServer`, which never completes a handshake with anyone: a call +/// against it can only fail on connect. Under `Default`, the base client's own 2000 ms connect timeout +/// governs; under `SingleAttempt` with a 100 ms `connect_timeout_cap_ms` and a much wider 5000 ms +/// `attempt_timeout_ms` (so the request/response budget, which this discriminator never reaches, is not +/// what is being measured), a dropped or ignored cap would fall back to the base client's 2000 ms +/// connect timeout -- making the SingleAttempt call take just as long as Default. The discrimination is +/// therefore specifically on the CAP, not merely on whether a SingleAttempt clone was selected at all. +TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTime) +{ + if (tcpAbortOnOverflowPreventsStallServer()) + GTEST_SKIP() << "net.ipv4.tcp_abort_on_overflow is not 0 (or unreadable): ConnectStallServer " + "cannot reliably stall a connect on this host"; + + (void)contextForTest(); // getThreadPoolWriter/BlobStorageLogWriter::create fall back to the global context + + constexpr long base_connect_timeout_ms = 2000; + constexpr uint64_t single_attempt_timeout_ms = 5000; + constexpr uint64_t single_attempt_connect_cap_ms = 100; + + /// PUT: writeObject; the profile and cap ride on WriteSettings, not an ObjectStorageControlRequest. + { + ConnectStallServer server; + auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); + + auto put = [&](DB::ObjectStorageRetryProfile profile, uint64_t attempt_timeout_ms, uint64_t connect_cap_ms) + { + DB::WriteSettings write_settings; + write_settings.object_storage_retry_profile = profile; + write_settings.object_storage_attempt_timeout_ms = attempt_timeout_ms; + write_settings.object_storage_connect_timeout_cap_ms = connect_cap_ms; + auto buffer = storage->writeObject( + DB::StoredObject("put-key"), DB::WriteMode::Rewrite, {}, DB::DBMS_DEFAULT_BUFFER_SIZE, write_settings); + buffer->write('A'); + buffer->finalize(); + }; + + const auto default_elapsed = expectConnectFailureAndMeasure( + [&] { put(DB::ObjectStorageRetryProfile::Default, 0, 0); }); + EXPECT_GE(default_elapsed.count(), 1500); + + const auto capped_elapsed = expectConnectFailureAndMeasure([&] + { + put(DB::ObjectStorageRetryProfile::SingleAttempt, single_attempt_timeout_ms, single_attempt_connect_cap_ms); + }); + EXPECT_LT(capped_elapsed.count(), 1000); + } + + /// HEAD: tryGetObjectMetadataWithNativeToken's ObjectStorageControlRequest-taking overload. + { + ConnectStallServer server; + auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); + + const auto default_elapsed = expectConnectFailureAndMeasure([&] + { + storage->tryGetObjectMetadataWithNativeToken("head-key", /*with_tags=*/false, DB::ObjectStorageControlRequest{}); + }); + EXPECT_GE(default_elapsed.count(), 1500); + + const auto capped_elapsed = expectConnectFailureAndMeasure([&] + { + storage->tryGetObjectMetadataWithNativeToken( + "head-key", /*with_tags=*/false, + DB::ObjectStorageControlRequest{ + .profile = DB::ObjectStorageRetryProfile::SingleAttempt, + .attempt_timeout_ms = single_attempt_timeout_ms, + .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); + }); + EXPECT_LT(capped_elapsed.count(), 1000); + } + + /// Conditional DELETE: removeObjectIfTokenMatches's ObjectStorageControlRequest-taking overload. + { + ConnectStallServer server; + auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); + + const auto default_elapsed = expectConnectFailureAndMeasure([&] + { + storage->removeObjectIfTokenMatches(DB::StoredObject("delete-key"), "\"etag\"", DB::ObjectStorageControlRequest{}); + }); + EXPECT_GE(default_elapsed.count(), 1500); + + const auto capped_elapsed = expectConnectFailureAndMeasure([&] + { + storage->removeObjectIfTokenMatches( + DB::StoredObject("delete-key"), "\"etag\"", + DB::ObjectStorageControlRequest{ + .profile = DB::ObjectStorageRetryProfile::SingleAttempt, + .attempt_timeout_ms = single_attempt_timeout_ms, + .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); + }); + EXPECT_LT(capped_elapsed.count(), 1000); + } +} + +/// `FreezeConnectTimeoutCapSnapshot` above pins the ARITHMETIC of `freezeConnectTimeoutCapMs` in +/// isolation; `ProductionDispatchAppliesTheFrozenConnectCapAtConnectTime` pins that a cap handed +/// DIRECTLY to `S3ObjectStorage` reaches the connect phase. Neither proves the composition +/// `ContentAddressedMetadataStorage::openPoolView` actually performs: freezing the cap from a real S3 +/// client (`ContentAddressedMetadataStorage.cpp` ~802) and handing it into `Cas::ObjectStorageBackend`'s +/// constructor (the backend handoff at ~812-822) exactly as a writable Native mount does. This test +/// drives that whole chain end to end -- real client -> freezeConnectTimeoutCapMs -> ObjectStorageBackend +/// -> CasRequests/CasOperation -> the SAME production S3ObjectStorage dispatch the tests above cover -- +/// with no recording subclass anywhere in it. `Pool::open` itself is not driven here: it needs a live +/// store (PoolMeta creation/validation) that a stalled-connect endpoint cannot provide, so the backend +/// composition above is the reachable end of the chain from a unit test. +/// +/// A read-only backend (`single_attempt_control_plane_ = false`, matching `openPoolView`'s own choice +/// for a read-only mount) keeps the storage's DEFAULT client for its read-class requests -- the base +/// 2000 ms connect timeout -- as the uncapped control. The SAME derived cap and attempt timeout, handed +/// to a WRITABLE Native backend exactly as `openPoolView` constructs one, must then fail an order of +/// magnitude faster: a dropped or corrupted handoff anywhere in the chain would silently fall back to +/// the uncapped control's timing instead. +TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapReachesTheBackendOverProductionDispatch) +{ + if (tcpAbortOnOverflowPreventsStallServer()) + GTEST_SKIP() << "net.ipv4.tcp_abort_on_overflow is not 0 (or unreadable): ConnectStallServer " + "cannot reliably stall a connect on this host"; + + (void)contextForTest(); + + constexpr long base_connect_timeout_ms = 2000; + constexpr uint64_t cas_attempt_timeout_ms = 100; + + ConnectStallServer server; + auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); + + /// The exact derivation `ContentAddressedMetadataStorage::openPoolView` uses: min(base connect + /// timeout, attempt timeout) = 100 here, never the wide 2000 ms base timeout. + const auto cap = DB::ContentAddressedMetadataStorage::freezeConnectTimeoutCapMs(storage, cas_attempt_timeout_ms); + ASSERT_TRUE(cap.has_value()); + EXPECT_EQ(*cap, cas_attempt_timeout_ms); + + auto uncapped_backend = std::make_shared( + storage, DB::Cas::ObjectStorageBackend::Mode::Native, + /*single_attempt_control_plane_=*/false, /*attempt_timeout_ms_=*/0, /*connect_timeout_cap_ms_=*/0); + { + DB::Cas::CasRequests requests(DB::Cas::BackendPtr(uncapped_backend), DB::Cas::Fence::open()); + auto op = requests.admit(); + const auto elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); + EXPECT_GE(elapsed.count(), 1500); + } + + /// The derived cap, handed to the backend exactly as `openPoolView` constructs it (:812-822) for a + /// WRITABLE Native mount. + auto capped_backend = std::make_shared( + storage, DB::Cas::ObjectStorageBackend::Mode::Native, + /*single_attempt_control_plane_=*/true, cas_attempt_timeout_ms, *cap); + { + DB::Cas::CasRequests requests(DB::Cas::BackendPtr(capped_backend), DB::Cas::Fence::open()); + auto op = requests.admit(); + const auto elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); + EXPECT_LT(elapsed.count(), 1000); + } +} + +#endif diff --git a/src/Disks/tests/gtest_cas_s3_staging.cpp b/src/Disks/tests/gtest_cas_s3_staging.cpp index 3c4a9805e9c8..74a82df51569 100644 --- a/src/Disks/tests/gtest_cas_s3_staging.cpp +++ b/src/Disks/tests/gtest_cas_s3_staging.cpp @@ -1110,20 +1110,20 @@ class FakeGenerationObjectStorage final : public DB::LocalObjectStorage /// vary, so the profile-aware overloads simply forward. A storage that claimed the capability /// without implementing them would refuse every control-plane request of a writable mount. std::optional tryGetObjectMetadataWithNativeToken( - const std::string & path, bool with_tags, DB::ObjectStorageRetryProfile, uint64_t) const override + const std::string & path, bool with_tags, const DB::ObjectStorageControlRequest &) const override { return tryGetObjectMetadata(path, with_tags); } DB::ObjectStorageIteratorPtr iterate( const std::string & path_prefix, size_t max_keys, bool with_tags, const std::optional & start_after, - DB::ObjectStorageRetryProfile, uint64_t) const override + const DB::ObjectStorageControlRequest &) const override { return DB::LocalObjectStorage::iterate(path_prefix, max_keys, with_tags, start_after); } DB::ConditionalRemoveResult removeObjectIfTokenMatches( - const DB::StoredObject & object, const std::string & etag, DB::ObjectStorageRetryProfile, uint64_t) override + const DB::StoredObject & object, const std::string & etag, const DB::ObjectStorageControlRequest &) override { return removeObjectIfTokenMatches(object, etag); } diff --git a/src/Disks/tests/gtest_cas_sentinel_probe.cpp b/src/Disks/tests/gtest_cas_sentinel_probe.cpp index 962bd85a30bc..cb52bed51399 100644 --- a/src/Disks/tests/gtest_cas_sentinel_probe.cpp +++ b/src/Disks/tests/gtest_cas_sentinel_probe.cpp @@ -84,6 +84,30 @@ class TransportFaultBackend final : public InMemoryBackend } +/// The probe loop's own attempt counter reaches the transport too -- propagation only, the probe +/// keeps its ordinary backoff. +TEST(CASSentinelProbe, AttemptNumberPropagates) +{ + struct ProbeRecording : InMemoryBackend + { + std::vector attempts; + SentinelProbeResult probeSentinelRaw(const String & key, TransportAccess & access) override + { + attempts.push_back(access.attemptNo()); + if (attempts.size() == 1) + return {ProbeOutcome::Indeterminate, std::nullopt}; + return InMemoryBackend::probeSentinelRaw(key, access); + } + }; + DB::Cas::tests::FakeClock clock; + auto backend = std::make_shared(); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn()); + auto op = requests.admit(); + (void)op.probeSentinel("probe", Retry::standard()); + EXPECT_EQ(backend->attempts, (std::vector{1, 2})); + EXPECT_EQ(clock.sleeps.size(), 1u); /// propagation only: the probe keeps its ordinary backoff +} + /// (a) A present key probes Present and carries the materialized body. TEST(CASSentinelProbe, PresentKeyReturnsPresentWithBody) { diff --git a/src/Disks/tests/gtest_cas_upstream_slice.cpp b/src/Disks/tests/gtest_cas_upstream_slice.cpp index c35bead7baa6..99d8ac0b1a4e 100644 --- a/src/Disks/tests/gtest_cas_upstream_slice.cpp +++ b/src/Disks/tests/gtest_cas_upstream_slice.cpp @@ -89,23 +89,26 @@ TEST(CASUpstreamSlice, HeadListRemoveOverloadsRefuseSingleAttemptOnTheBaseStorag { auto local = makeLocalObjectStorageForRetryProfileTest(); + const DB::ObjectStorageControlRequest single_attempt{.profile = DB::ObjectStorageRetryProfile::SingleAttempt}; + const DB::ObjectStorageControlRequest default_profile{.profile = DB::ObjectStorageRetryProfile::Default}; + expectThrowsNotImplementedSaying( "single-attempt metadata requests", - [&] { local->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::SingleAttempt, 0); }); + [&] { local->tryGetObjectMetadataWithNativeToken("k", false, single_attempt); }); expectThrowsNotImplementedSaying( "single-attempt listing requests", - [&] { local->iterate("", 1, false, {}, DB::ObjectStorageRetryProfile::SingleAttempt, 0); }); + [&] { local->iterate("", 1, false, {}, single_attempt); }); expectThrowsNotImplementedSaying( "single-attempt removal requests", - [&] { local->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::SingleAttempt, 0); }); + [&] { local->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", single_attempt); }); /// `Default` must keep reaching the ordinary implementation. For removal that is still a refusal, /// but the pre-existing one — matching its wording proves the profile overload forwarded. - EXPECT_NO_THROW(local->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::Default, 0)); - EXPECT_NO_THROW(local->iterate("", 1, false, {}, DB::ObjectStorageRetryProfile::Default, 0)); + EXPECT_NO_THROW(local->tryGetObjectMetadataWithNativeToken("k", false, default_profile)); + EXPECT_NO_THROW(local->iterate("", 1, false, {}, default_profile)); expectThrowsNotImplementedSaying( "Conditional (token-exact) object removal", - [&] { local->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::Default, 0); }); + [&] { local->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", default_profile); }); } #if USE_AWS_S3 @@ -640,7 +643,7 @@ TEST(CASUpstreamSlice, NativeTokenHeadRecoversFromAnExpiredTokenAndInstallsTheRe expired->scriptHead({controlExpiredToken()}); const auto metadata = storage->tryGetObjectMetadataWithNativeToken( - "k", /*with_tags=*/false, DB::ObjectStorageRetryProfile::Default, /*request_timeout_ms=*/0); + "k", /*with_tags=*/false, DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::Default}); ASSERT_TRUE(metadata.has_value()); ASSERT_NE(refreshed, nullptr); @@ -664,7 +667,7 @@ TEST(CASUpstreamSlice, ConditionalRemoveRecoversFromAnExpiredTokenAndInstallsThe expired->scriptDelete({controlExpiredToken()}); const auto result = storage->removeObjectIfTokenMatches( - DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::Default, /*request_timeout_ms=*/0); + DB::StoredObject("k"), "e", DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::Default}); EXPECT_EQ(result.outcome, DB::ConditionalRemoveOutcome::Removed); ASSERT_NE(refreshed, nullptr); @@ -689,7 +692,7 @@ TEST(CASUpstreamSlice, SingleAttemptConditionalRemoveIssuesExactlyOneRequestOnTh retrying->scriptDelete({controlThrottle()}); EXPECT_ANY_THROW(retrying_storage->removeObjectIfTokenMatches( - DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::Default, 0)); + DB::StoredObject("k"), "e", DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::Default})); EXPECT_EQ(retrying->deleteRequestTimeouts().size(), 3u); /// max_retries = 2, so three attempts ScriptedGetObjectClient * client = nullptr; @@ -697,7 +700,7 @@ TEST(CASUpstreamSlice, SingleAttemptConditionalRemoveIssuesExactlyOneRequestOnTh client->scriptDelete({controlThrottle()}); EXPECT_ANY_THROW(storage->removeObjectIfTokenMatches( - DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::SingleAttempt, 0)); + DB::StoredObject("k"), "e", DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::SingleAttempt})); EXPECT_EQ(client->deleteRequestTimeouts().size(), 1u); } @@ -711,18 +714,21 @@ TEST(CASUpstreamSlice, HeadAndRemoveUnderSingleAttemptRideTheClientBoundToTheReq ScriptedGetObjectClient * client = nullptr; auto storage = makeScriptedS3ObjectStorage(client); - storage->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::SingleAttempt, 4321); + storage->tryGetObjectMetadataWithNativeToken( + "k", false, DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::SingleAttempt, .attempt_timeout_ms = 4321}); ASSERT_EQ(client->headRequestTimeouts().size(), 1u); EXPECT_EQ(client->headRequestTimeouts().at(0), 4321); - storage->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", DB::ObjectStorageRetryProfile::SingleAttempt, 8765); + storage->removeObjectIfTokenMatches(DB::StoredObject("k"), "e", + DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::SingleAttempt, .attempt_timeout_ms = 8765}); ASSERT_EQ(client->deleteRequestTimeouts().size(), 1u); EXPECT_EQ(client->deleteRequestTimeouts().at(0), 8765); EXPECT_EQ(client->cloneRequestTimeouts(), (std::vector{4321, 8765})); /// Asking again for a bound already built must reuse that clone rather than evict the other one. - storage->tryGetObjectMetadataWithNativeToken("k", false, DB::ObjectStorageRetryProfile::SingleAttempt, 4321); + storage->tryGetObjectMetadataWithNativeToken( + "k", false, DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::SingleAttempt, .attempt_timeout_ms = 4321}); EXPECT_EQ(client->cloneRequestTimeouts(), (std::vector{4321, 8765})); EXPECT_EQ(client->headRequestTimeouts().at(1), 4321); } @@ -736,10 +742,11 @@ TEST(CASUpstreamSlice, IterateUnderSingleAttemptSelectsTheClientBoundToTheReques ScriptedGetObjectClient * client = nullptr; auto storage = makeScriptedS3ObjectStorage(client); - (void)storage->iterate("p", 1, false, {}, DB::ObjectStorageRetryProfile::Default, 0); + (void)storage->iterate("p", 1, false, {}, DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::Default}); EXPECT_TRUE(client->cloneRequestTimeouts().empty()); - (void)storage->iterate("p", 1, false, {}, DB::ObjectStorageRetryProfile::SingleAttempt, 4321); + (void)storage->iterate("p", 1, false, {}, + DB::ObjectStorageControlRequest{.profile = DB::ObjectStorageRetryProfile::SingleAttempt, .attempt_timeout_ms = 4321}); EXPECT_EQ(client->cloneRequestTimeouts(), (std::vector{4321})); } diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index d763737a4172..576a8b2728dc 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -368,6 +368,7 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem const CasRequestBudget budget{ .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, + .connect_timeout_cap_ms = std::nullopt, }; /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. @@ -446,6 +447,7 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) const CasRequestBudget budget{ .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, + .connect_timeout_cap_ms = std::nullopt, }; /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. diff --git a/src/IO/ReadBufferFromS3.cpp b/src/IO/ReadBufferFromS3.cpp index cf92f9c6f2a2..fc1d5fe38d3f 100644 --- a/src/IO/ReadBufferFromS3.cpp +++ b/src/IO/ReadBufferFromS3.cpp @@ -565,7 +565,7 @@ Aws::S3::Model::GetObjectResult ReadBufferFromS3::sendRequest(size_t attempt, si if (!version_id.empty()) req.SetVersionId(version_id); - S3::setClickhouseAttemptNumber(req, attempt); + S3::setClickhouseAttemptNumber(req, S3::seededAttemptNumber(read_settings.object_storage_attempt_number, attempt)); if (read_settings.object_storage_request_mode == ObjectStorageRequestMode::NativeConditional) req.setNativeConditional(); diff --git a/src/IO/ReadSettings.h b/src/IO/ReadSettings.h index 00f3b649a95d..981d5584e6d6 100644 --- a/src/IO/ReadSettings.h +++ b/src/IO/ReadSettings.h @@ -169,6 +169,14 @@ struct ReadSettings ObjectStorageRetryProfile object_storage_retry_profile = ObjectStorageRetryProfile::Default; uint64_t object_storage_attempt_timeout_ms = 0; + /// The cap the single-attempt client's clone puts on one TCP connect and again on one TLS + /// handshake, frozen by the mount at open; see `CasRequestBudget::attemptEnvelopeMs`. 0 = no cap. + uint64_t object_storage_connect_timeout_cap_ms = 0; + + /// The caller's own attempt number for the request built from these settings, 1-based; 0 leaves the + /// buffer's own numbering. A CAS reissue passes its count so the HTTP client sees attempt ≥ 2. + size_t object_storage_attempt_number = 0; + ReadSettings adjustBufferSize(size_t file_size) const; /// Verification/metadata-read mode: disable every read-side cache (and the diff --git a/src/IO/S3/Requests.h b/src/IO/S3/Requests.h index 53460a8fa099..7526f0049e3c 100644 --- a/src/IO/S3/Requests.h +++ b/src/IO/S3/Requests.h @@ -265,6 +265,13 @@ size_t getClickhouseAttemptNumber(const Aws::AmazonWebServiceRequest & request); size_t getClickhouseAttemptNumber(const Aws::Http::HttpRequest & request); void setClickhouseAttemptNumber(Aws::AmazonWebServiceRequest & request, size_t attempt); +/// The attempt number a request carries when its caller seeded one: the caller's attempt for the first +/// local try, then the local counter's increments. Seed 0 is "unseeded" and yields `local`. +inline size_t seededAttemptNumber(size_t seed, size_t local) +{ + return (seed == 0 ? 1 : seed) + local - 1; +} + } #endif diff --git a/src/IO/S3/getObjectInfo.cpp b/src/IO/S3/getObjectInfo.cpp index 63aeb736960c..5bd1b4cf52e1 100644 --- a/src/IO/S3/getObjectInfo.cpp +++ b/src/IO/S3/getObjectInfo.cpp @@ -25,7 +25,8 @@ namespace const String & bucket, const String & key, const String & version_id, - ObjectStorageRequestMode request_mode = ObjectStorageRequestMode::Default) + ObjectStorageRequestMode request_mode = ObjectStorageRequestMode::Default, + size_t attempt_seed = 0) { ProfileEvents::increment(ProfileEvents::S3HeadObject); if (client.isClientForDisk()) @@ -41,6 +42,9 @@ namespace req.setNativeConditional(request_mode == ObjectStorageRequestMode::NativeConditional); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(req, attempt_seed); + return client.HeadObject(req); } @@ -71,9 +75,10 @@ namespace const String & version_id, bool with_metadata, bool with_tags, - ObjectStorageRequestMode request_mode = ObjectStorageRequestMode::Default) + ObjectStorageRequestMode request_mode = ObjectStorageRequestMode::Default, + size_t attempt_seed = 0) { - auto outcome = headObject(client, bucket, key, version_id, request_mode); + auto outcome = headObject(client, bucket, key, version_id, request_mode, attempt_seed); if (!outcome.IsSuccess()) return {std::nullopt, outcome.GetError()}; @@ -146,11 +151,12 @@ ObjectInfo getObjectInfoIfExists( const String & version_id, bool with_metadata, bool with_tags, - ObjectStorageRequestMode request_mode) + ObjectStorageRequestMode request_mode, + size_t attempt_seed) { Expect404ResponseScope scope; // 404 is not an error - auto [object_info, error] = tryGetObjectInfo(client, bucket, key, version_id, with_metadata, with_tags, request_mode); + auto [object_info, error] = tryGetObjectInfo(client, bucket, key, version_id, with_metadata, with_tags, request_mode, attempt_seed); if (object_info) return *object_info; diff --git a/src/IO/S3/getObjectInfo.h b/src/IO/S3/getObjectInfo.h index 60683cbf3abf..d3a31260f473 100644 --- a/src/IO/S3/getObjectInfo.h +++ b/src/IO/S3/getObjectInfo.h @@ -25,6 +25,8 @@ struct ObjectInfo /// Ignore if object does not exist /// `request_mode` marks the HEAD wrapper as eligible for the typed NativeConditional request mode /// (see ObjectStorageRequestMode); the client's HTTP layer decides whether it actually takes effect. +/// `attempt_seed`, when nonzero, is set as the HEAD's `clickhouse-request` attempt number (see +/// `S3::seededAttemptNumber`); 0 leaves the request unseeded. ObjectInfo getObjectInfoIfExists( const S3::Client & client, const String & bucket, @@ -32,7 +34,8 @@ ObjectInfo getObjectInfoIfExists( const String & version_id = {}, bool with_metadata = false, bool with_tags = false, - ObjectStorageRequestMode request_mode = ObjectStorageRequestMode::Default); + ObjectStorageRequestMode request_mode = ObjectStorageRequestMode::Default, + size_t attempt_seed = 0); ObjectInfo getObjectInfo( const S3::Client & client, diff --git a/src/IO/S3/tests/gtest_aws_s3_client.cpp b/src/IO/S3/tests/gtest_aws_s3_client.cpp index 09ed0bd6692f..2b477dae1c91 100644 --- a/src/IO/S3/tests/gtest_aws_s3_client.cpp +++ b/src/IO/S3/tests/gtest_aws_s3_client.cpp @@ -148,6 +148,214 @@ static void doWriteRequest(std::shared_ptr client, const D using RequestFn = std::function, const DB::S3::URI &)>; +/// Parses the `attempt=N` value `S3::setClickhouseAttemptNumber` writes into the `clickhouse-request` +/// header, straight off the wire header a real HTTP server received -- mirrors +/// `S3::getAttemptFromInfo`/`getOrEmpty` (both `static` in `Requests.cpp`, not exported), 1 when the +/// header is missing. +static size_t attemptFromHeader(const Poco::Net::MessageHeader & header) +{ + const std::string & value = header.get("clickhouse-request", ""); + static const std::string key = "attempt="; + auto pos = value.find(key); + if (pos == std::string::npos) + return 1; + try + { + return static_cast(std::stol(value.substr(pos + key.size()))); + } + catch (const std::exception &) + { + return 1; + } +} + +static std::shared_ptr makeTestClient(const DB::S3::URI & uri) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration client_configuration = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", + remote_host_filter, + /*s3_max_redirects=*/100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /*s3_slow_all_threads_after_network_error=*/false, + /*s3_slow_all_threads_after_retryable_error=*/false, + /*enable_s3_requests_logging=*/false, + /*for_disk_s3=*/false, + /*opt_disk_name=*/{}, + /*request_throttler=*/{}, + uri.uri.getScheme()); + client_configuration.endpointOverride = uri.endpoint; + /// `ClientFactory::create` installs the SDK's actual retry strategy itself from + /// `client_configuration.retry_strategy`/`s3_slow_all_threads_after_retryable_error` (any + /// `retryStrategy` set here is overwritten) -- with `s3_slow_all_threads_after_retryable_error` + /// true it forces `max_retries = 1` regardless of the `RetryStrategy{.max_retries = 0}` passed + /// above, so the SDK itself retries a retryable error once before `ReadBufferFromS3`'s own + /// local-retry loop ever sees a failure, and both physical requests carry the same seeded header. + /// `false` here keeps the SDK to exactly one physical attempt, matching the CAS single-attempt + /// client's own setup. + + DB::S3::ClientSettings client_settings{ + .use_virtual_addressing = uri.is_virtual_hosted_style, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }; + + return DB::S3::ClientFactory::instance().create( + client_configuration, + client_settings, + "ACCESS_KEY_ID", + "SECRET_ACCESS_KEY", + /*server_side_encryption_customer_key_base64=*/"", + DB::S3::ServerSideEncryptionKMSConfig(), + DB::HTTPHeaderEntries(), + DB::S3::CredentialsConfiguration{ + .use_environment_credentials = false, + .use_insecure_imds_request = false, + }); +} + +/// Fails the first `fail_first_n` requests with `fail_status` (empty body), then serves `body` with a +/// 200 to every request after. Records every request's header (not just the last) so a caller can +/// check the sequence a local retry produced. +class SequenceRecordingRequestHandler : public Poco::Net::HTTPRequestHandler +{ + std::vector & all_request_headers; + size_t & requests_seen; + size_t fail_first_n; + Poco::Net::HTTPResponse::HTTPStatus fail_status; + std::string body; + +public: + SequenceRecordingRequestHandler( + std::vector & all_request_headers_, + size_t & requests_seen_, + size_t fail_first_n_, + Poco::Net::HTTPResponse::HTTPStatus fail_status_, + std::string body_) + : all_request_headers(all_request_headers_) + , requests_seen(requests_seen_) + , fail_first_n(fail_first_n_) + , fail_status(fail_status_) + , body(std::move(body_)) + { + } + + void handleRequest(Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) override + { + all_request_headers.push_back(request); + ++requests_seen; + + if (requests_seen <= fail_first_n) + { + response.setStatus(fail_status); + response.send(); + return; + } + + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.setContentLength(static_cast(body.size())); + auto & out = response.send(); + out << body; + out.flush(); + } +}; + +class SequenceRecordingRequestHandlerFactory : public Poco::Net::HTTPRequestHandlerFactory +{ + std::vector & all_request_headers; + size_t & requests_seen; + size_t fail_first_n; + Poco::Net::HTTPResponse::HTTPStatus fail_status; + std::string body; + + Poco::Net::HTTPRequestHandler * createRequestHandler(const Poco::Net::HTTPServerRequest &) override + { + return new SequenceRecordingRequestHandler(all_request_headers, requests_seen, fail_first_n, fail_status, body); + } + +public: + SequenceRecordingRequestHandlerFactory( + std::vector & all_request_headers_, + size_t & requests_seen_, + size_t fail_first_n_, + Poco::Net::HTTPResponse::HTTPStatus fail_status_, + std::string body_) + : all_request_headers(all_request_headers_) + , requests_seen(requests_seen_) + , fail_first_n(fail_first_n_) + , fail_status(fail_status_) + , body(std::move(body_)) + { + } + + ~SequenceRecordingRequestHandlerFactory() override = default; +}; + +/// Like `TestPocoHTTPServer`, but for driving a real local retry: the first `fail_first_n` requests +/// get `fail_status`, every one after gets `body` with a 200, and every request's header is kept (not +/// just the last). Its only user is the seed test right below -- localized here rather than in the +/// shared `TestPocoHTTPServer.h` header. +class TestPocoHTTPSequenceServer +{ + std::unique_ptr server_socket; + Poco::SharedPtr handler_factory; + Poco::AutoPtr server_params; + std::unique_ptr server; + std::vector all_request_headers; + size_t requests_seen = 0; + +public: + TestPocoHTTPSequenceServer(size_t fail_first_n, Poco::Net::HTTPResponse::HTTPStatus fail_status, std::string body = {}): + server_socket(std::make_unique(0)), + handler_factory(new SequenceRecordingRequestHandlerFactory(all_request_headers, requests_seen, fail_first_n, fail_status, std::move(body))), + server_params(new Poco::Net::HTTPServerParams()), + server(std::make_unique(handler_factory, *server_socket, server_params)) + { + server->start(); + } + + std::string getUrl() + { + return "http://" + server_socket->address().toString(); + } + + const std::vector & getAllRequestHeaders() const + { + return all_request_headers; + } +}; + +/// An unset seed sends `[1, 2]` across a local retry, a seed of 2 sends `[2, 3]` -- a real HTTP round +/// trip through `TestPocoHTTPSequenceServer` is the only way to drive the retry through +/// `ReadBufferFromS3`'s actual success path (the SDK's response stream wraps a real +/// `Poco::Net::HTTPBasicStreamBuf`, which `ReadBufferFromIStream` requires). +TEST(IOTestAwsS3Client, ReadBufferFromS3AttemptSeedCarriesAcrossLocalRetry) +{ + for (const auto [seed, first, second] : {std::tuple{0, 1, 2}, {2, 2, 3}}) + { + TestPocoHTTPSequenceServer http(/*fail_first_n=*/1, Poco::Net::HTTPResponse::HTTP_INTERNAL_SERVER_ERROR, "seeded-body"); + DB::S3::URI uri(http.getUrl() + "/seeded-bucket/seeded-key"); + auto client = makeTestClient(uri); + ASSERT_TRUE(client); + + DB::ReadSettings read_settings; + read_settings.object_storage_attempt_number = seed; + DB::S3::S3RequestSettings request_settings; + request_settings[DB::S3RequestSetting::max_single_read_retries] = 2; + DB::ReadBufferFromS3 read_buffer(client, uri.bucket, uri.key, /*version_id=*/{}, request_settings, read_settings); + + String content; + DB::readStringUntilEOF(content, read_buffer); + EXPECT_EQ(content, "seeded-body"); + + const auto & headers = http.getAllRequestHeaders(); + ASSERT_EQ(headers.size(), 2u); + EXPECT_EQ(attemptFromHeader(headers[0]), first); + EXPECT_EQ(attemptFromHeader(headers[1]), second); + } +} + static void testServerSideEncryption( RequestFn do_request, bool disable_checksum, diff --git a/src/IO/WriteBufferFromS3.cpp b/src/IO/WriteBufferFromS3.cpp index 662204851290..4be4973b0cb9 100644 --- a/src/IO/WriteBufferFromS3.cpp +++ b/src/IO/WriteBufferFromS3.cpp @@ -749,6 +749,9 @@ S3::PutObjectRequest WriteBufferFromS3::getPutRequest(PartData & data) /// If we don't do it, AWS SDK can mistakenly set it to application/xml, see https://github.com/aws/aws-sdk-cpp/issues/1840 req.SetContentType("binary/octet-stream"); + if (write_settings.object_storage_attempt_number != 0) + S3::setClickhouseAttemptNumber(req, write_settings.object_storage_attempt_number); + client_ptr->setKMSHeaders(req); /// The actual PUT that produces a CAS incarnation token: eligible for the typed NativeConditional diff --git a/src/IO/WriteSettings.h b/src/IO/WriteSettings.h index b68da174ab44..0cb5c0b31321 100644 --- a/src/IO/WriteSettings.h +++ b/src/IO/WriteSettings.h @@ -21,6 +21,17 @@ enum class ObjectStorageRetryProfile : uint8_t SingleAttempt, }; +/// What a CAS control request carries into the object storage: the retry profile, the per-attempt +/// budget and connect cap the storage's single-attempt client must honour, and the caller's own +/// attempt number (0 = unset) so the HTTP client sees a reissue as attempt ≥ 2. +struct ObjectStorageControlRequest +{ + ObjectStorageRetryProfile profile = ObjectStorageRetryProfile::Default; + uint64_t attempt_timeout_ms = 0; + uint64_t connect_timeout_cap_ms = 0; + size_t attempt_number = 0; +}; + /// Per-copy transport requirement, resolved by the object storage that executes the copy. /// `NativeOnly` requires a provider-native same-store copy and forbids a client-side fallback. enum class ObjectStorageCopyMode : uint8_t @@ -85,6 +96,14 @@ struct WriteSettings /// `object_storage_retry_profile == SingleAttempt`. 0 = the storage's configured timeout. uint64_t object_storage_attempt_timeout_ms = 0; + /// The cap the single-attempt client's clone puts on one TCP connect and again on one TLS + /// handshake, frozen by the mount at open; see `CasRequestBudget::attemptEnvelopeMs`. 0 = no cap. + uint64_t object_storage_connect_timeout_cap_ms = 0; + + /// The caller's own attempt number for the request built from these settings, 1-based; 0 leaves the + /// buffer's own numbering. A CAS reissue passes its count so the HTTP client sees attempt ≥ 2. + size_t object_storage_attempt_number = 0; + /// Selects the transport requirement for an object storage copy; see `ObjectStorageCopyMode`. ObjectStorageCopyMode object_storage_copy_mode = ObjectStorageCopyMode::Default; diff --git a/src/IO/tests/gtest_writebuffer_s3.cpp b/src/IO/tests/gtest_writebuffer_s3.cpp index ca52dd8b2742..ce3fc108b840 100644 --- a/src/IO/tests/gtest_writebuffer_s3.cpp +++ b/src/IO/tests/gtest_writebuffer_s3.cpp @@ -20,6 +20,8 @@ #include #include #include +#include +#include #include #include #include @@ -31,6 +33,8 @@ #include #include #include +#include +#include #include #include @@ -38,6 +42,7 @@ #include #include #include +#include #include #include @@ -45,6 +50,8 @@ #include #include +#include + #include #include #include @@ -236,6 +243,34 @@ struct InjectionModel #undef DeclareInjectCall }; +/// `DB::S3::getClickhouseAttemptNumber(const Aws::AmazonWebServiceRequest &)` reads `GetHeaders()`, +/// which for a plain S3 request never includes `SetAdditionalCustomHeaderValue`'s custom headers -- +/// only `AWSClient::BuildHttpRequest` merges those into the wire-level `Aws::Http::HttpRequest` that +/// `PocoHTTPClient` actually inspects (the overload production code reads). This mock overrides the +/// `S3Client` virtuals directly, below that merge, so it reads the custom header collection itself. +/// `nullopt` means the `clickhouse-request` header is absent -- distinct from an explicit `attempt=1`, +/// since a seed of 0 leaves every verb but the read path unseeded (no header at all; see +/// `S3::seededAttemptNumber`'s callers). +static std::optional attemptNumberFromCustomHeaders(const Aws::AmazonWebServiceRequest & request) +{ + const auto & headers = request.GetAdditionalCustomHeaders(); + auto it = headers.find("clickhouse-request"); + if (it == headers.end()) + return std::nullopt; + static const std::string key = "attempt="; + auto pos = it->second.find(key); + if (pos == std::string::npos) + return std::nullopt; + try + { + return static_cast(std::stol(it->second.substr(pos + key.size()))); + } + catch (const std::exception &) + { + return std::nullopt; + } +} + struct Client : DB::S3::Client { explicit Client(std::shared_ptr mock_s3_store) @@ -282,8 +317,54 @@ struct Client : DB::S3::Client injections = injections_; } + /// `clickhouse-request` attempt of every verb, in order -- test-only recorder for the attempt-seed tests. + mutable std::vector> attempts_seen; + + Aws::S3::Model::ListObjectsV2Outcome ListObjectsV2(const Aws::S3::Model::ListObjectsV2Request & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + auto & bStore = store->GetBucketStore(request.GetBucket()); + Aws::S3::Model::ListObjectsV2Result result; + result.SetPrefix(request.GetPrefix()); + int emitted = 0; + std::string last; + const std::string after = request.ContinuationTokenHasBeenSet() ? request.GetContinuationToken() + : request.StartAfterHasBeenSet() ? request.GetStartAfter() : ""; + for (const auto & [key, data] : bStore.objects) + { + if (!key.starts_with(request.GetPrefix()) || key <= after) + continue; + if (emitted == request.GetMaxKeys()) + { + result.SetIsTruncated(true); + result.SetNextContinuationToken(last); + break; + } + Aws::S3::Model::Object object; + object.SetKey(key); + object.SetSize(static_cast(data.size())); + result.AddContents(std::move(object)); + last = key; + ++emitted; + } + return Aws::S3::Model::ListObjectsV2Outcome(std::move(result)); + } + + Aws::S3::Model::DeleteObjectsOutcome DeleteObjects(const Aws::S3::Model::DeleteObjectsRequest & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + + auto & bStore = store->GetBucketStore(request.GetBucket()); + for (const auto & identifier : request.GetDelete().GetObjects()) + bStore.objects.erase(identifier.GetKey()); + + Aws::S3::Model::DeleteObjectsResult result; + return Aws::S3::Model::DeleteObjectsOutcome(std::move(result)); + } + Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest & request) const override { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); ++counters.putObject; if (const auto * wrapper = dynamic_cast(&request)) @@ -338,6 +419,7 @@ struct Client : DB::S3::Client Aws::S3::Model::HeadObjectOutcome HeadObject(const Aws::S3::Model::HeadObjectRequest & request) const override { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); ++counters.headObject; /// The request's DYNAMIC type is still the production `DB::S3::HeadObjectRequest` wrapper -- @@ -499,6 +581,7 @@ struct Client : DB::S3::Client Aws::S3::Model::DeleteObjectOutcome DeleteObject(const Aws::S3::Model::DeleteObjectRequest & request) const override { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); ++counters.deleteObject; if (const auto * wrapper = dynamic_cast(&request)) @@ -564,18 +647,6 @@ struct PutObjectPreconditionFailedIngection: InjectionModel } }; -/// A transport failure shaped as `PocoHTTPClient` shapes one: the S3 error is `NETWORK_CONNECTION` -/// and the message is the Poco text, exception name empty. -struct PutObjectNetworkTextIngection: InjectionModel -{ - explicit PutObjectNetworkTextIngection(std::string text_) : text(std::move(text_)) {} - std::optional call(const Aws::S3::Model::PutObjectRequest & /*request*/) override - { - return Aws::Client::AWSError(Aws::Client::CoreErrors::NETWORK_CONNECTION, "", text, false); - } - std::string text; -}; - struct HeadObjectFailIngection: InjectionModel { std::optional call(const Aws::S3::Model::HeadObjectRequest & /*request*/) override @@ -1033,35 +1104,6 @@ TEST_P(SyncAsync, PreconditionFailedNeverLogsAtError) EXPECT_THAT(log_capture.captured(), testing::Not(testing::HasSubstr("S3Exception name"))); } -/// The classifier a later change adds to the CAS request engine (`isConnectFailureHint`) reads the -/// Poco text a connection failure carries. This pins that the fake S3 client -- and, through it, the -/// same `WriteBufferFromS3` rethrow every real disk uses -- hands the caller that text unchanged, -/// under `NETWORK_CONNECTION`. -TEST_F(WBS3Test, NetworkConnectionTextSurvives) -{ - for (const char * text : {"Cannot assign requested address", "Connection refused", "No route to host", - "Network is unreachable", "connect timed out"}) - { - setInjectionModel(std::make_shared(text)); - WriteSettings write_settings; - write_settings.object_storage_retry_profile = ObjectStorageRetryProfile::SingleAttempt; - write_settings.s3_max_unexpected_write_error_retries_override = 1; - try - { - auto buffer = getWriteBuffer("network_text", write_settings); - buffer->write('A'); - getAsyncPolicy().setAutoExecute(true); - buffer->finalize(); - FAIL() << "the injected failure must surface"; - } - catch (const DB::S3Exception & e) - { - EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::NETWORK_CONNECTION) << text; - EXPECT_THAT(e.message(), testing::HasSubstr(text)); - } - } -} - TEST_P(SyncAsync, ExceptionOnCreateMPU) { setInjectionModel(std::make_shared()); @@ -1291,6 +1333,137 @@ TEST_F(WBS3Test, ResultObjectETagIsCaptured) { } } +TEST_F(WBS3Test, S3RequestAttemptSeedPutHeadDeleteCarryTheSeed) +{ + WriteSettings write_settings; + write_settings.object_storage_attempt_number = 3; + client->attempts_seen.clear(); + { + auto buffer = getWriteBuffer("seeded_put", write_settings); + buffer->write('A'); + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + } + ASSERT_FALSE(client->attempts_seen.empty()); + EXPECT_EQ(client->attempts_seen.front(), 3u); + /// Seed 0 adds no header at all (the spec's rule for every verb but the read path). + client->attempts_seen.clear(); + { + auto buffer = getWriteBuffer("unseeded_put"); + buffer->write('A'); + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + } + ASSERT_EQ(client->attempts_seen.size(), 1u); + EXPECT_FALSE(client->attempts_seen.front().has_value()); + + /// The native HEAD's seed: `S3ObjectStorage::tryGetObjectMetadataWithNativeToken`'s profile-aware + /// overload now forwards `request.attempt_number`, like every other verb here; this exercises the + /// seed-carrying layer directly -- `S3::getObjectInfoIfExists`, the same call + /// `tryGetObjectMetadataImpl` makes. + client->attempts_seen.clear(); + S3::getObjectInfoIfExists(*client, bucket, "seeded_head", /*version_id=*/{}, /*with_metadata=*/false, + /*with_tags=*/false, ObjectStorageRequestMode::Default, /*attempt_seed=*/4); + ASSERT_EQ(client->attempts_seen.size(), 1u); + EXPECT_EQ(client->attempts_seen.front(), 4u); + client->attempts_seen.clear(); + S3::getObjectInfoIfExists(*client, bucket, "unseeded_head"); + ASSERT_EQ(client->attempts_seen.size(), 1u); + EXPECT_FALSE(client->attempts_seen.front().has_value()); + + /// Conditional (single) and bulk DELETE: reachable now through `S3ObjectStorage`'s + /// `ObjectStorageControlRequest`-carrying overloads, which is what actually drives + /// `removeObjectIfTokenMatchesImpl`/`removeObjectsIfExistImpl` with a real nonzero seed, through the + /// object storage's own API rather than a lower-level free function. + (void)getContext(); // BlobStorageLogWriter::create falls back to the global context + auto delete_store = std::make_shared(); + delete_store->CreateBucket(bucket); + auto owned_delete_client = std::make_unique(delete_store); + MockS3::Client * delete_client = owned_delete_client.get(); + S3::URI delete_uri; + delete_uri.bucket = bucket; + auto delete_object_storage = std::make_shared( + std::move(owned_delete_client), + std::make_unique(), + delete_uri, + S3Capabilities{}, + ObjectStorageKeyGeneratorPtr{}, + "seed-delete-disk"); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectIfTokenMatches(StoredObject("unseeded-delete-key"), "etag-1"); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_FALSE(delete_client->attempts_seen.front().has_value()); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectIfTokenMatches( + StoredObject("seeded-delete-key"), "etag-1", ObjectStorageControlRequest{.attempt_number = 3}); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_EQ(delete_client->attempts_seen.front(), 3u); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectsIfExistUnderProfile({StoredObject("unseeded-bulk-key")}, ObjectStorageControlRequest{}); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_FALSE(delete_client->attempts_seen.front().has_value()); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectsIfExistUnderProfile( + {StoredObject("seeded-bulk-key")}, ObjectStorageControlRequest{.attempt_number = 3}); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_EQ(delete_client->attempts_seen.front(), 3u); +} + +TEST_F(WBS3Test, S3RequestAttemptSeedListPagesCarryTheSeed) +{ + /// Drives the seed through the public `iterate` overload a real caller (the CAS backend's LIST + /// primitive) uses, rather than the anonymous-namespace `S3IteratorAsync` directly -- that class is + /// an implementation detail of `S3ObjectStorage.cpp` and not reachable from a test in this file. + auto list_store = std::make_shared(); + list_store->CreateBucket(bucket); + auto owned_list_client = std::make_unique(list_store); + MockS3::Client * list_client = owned_list_client.get(); + S3::URI list_uri; + list_uri.bucket = bucket; + auto list_object_storage = std::make_shared( + std::move(owned_list_client), + std::make_unique(), + list_uri, + S3Capabilities{}, + ObjectStorageKeyGeneratorPtr{}, + "seed-list-disk"); + + auto & bucket_store = list_store->GetBucketStore(bucket); + for (int i = 0; i < 5; ++i) + bucket_store.PutObject(fmt::format("p/{}", i), "x"); + + /// Profile is left at Default (not SingleAttempt): that would route through + /// `clientForRetryProfile`'s single-attempt clone, whose `cloneWithConfigurationOverride` the mock + /// client does not override, and the test would stop exercising the mock entirely. + list_client->attempts_seen.clear(); + auto iterator = list_object_storage->iterate( + "p/", /*max_keys=*/2, /*with_tags=*/false, std::optional("p/0"), + ObjectStorageControlRequest{.attempt_number = 2}); + size_t seen = 0; + for (; iterator->isValid(); iterator->next()) + ++seen; + EXPECT_EQ(seen, 4u); + ASSERT_EQ(list_client->attempts_seen.size(), 2u); /// the initial page and one rebuilt page + EXPECT_EQ(list_client->attempts_seen[0], 2u); + EXPECT_EQ(list_client->attempts_seen[1], 2u); + + /// Seed 0 adds no header on either page. + list_client->attempts_seen.clear(); + auto unseeded_iterator = list_object_storage->iterate( + "p/", /*max_keys=*/2, /*with_tags=*/false, std::optional("p/0"), ObjectStorageControlRequest{}); + seen = 0; + for (; unseeded_iterator->isValid(); unseeded_iterator->next()) + ++seen; + EXPECT_EQ(seen, 4u); + ASSERT_EQ(list_client->attempts_seen.size(), 2u); + EXPECT_FALSE(list_client->attempts_seen[0].has_value()); + EXPECT_FALSE(list_client->attempts_seen[1].has_value()); +} + TEST_P(SyncAsync, EmptyFile) { getSettings()[Setting::s3_check_objects_after_upload] = true; diff --git a/tests/integration/test_cas_gcs/gcs_mocks/server.py b/tests/integration/test_cas_gcs/gcs_mocks/server.py index 457b9f03653c..2945578bf9b9 100644 --- a/tests/integration/test_cas_gcs/gcs_mocks/server.py +++ b/tests/integration/test_cas_gcs/gcs_mocks/server.py @@ -29,12 +29,19 @@ state seam for the writer retry test, not a model of the GC request sequence; - ``POST /_control/reset`` — drop the capture log and the counters (objects are kept), and clear the delay knob below; - - ``POST /_control/delay?substr=S&ms=N`` — every PUT whose key contains ``S`` sleeps ``N`` - milliseconds before it is served, outside the store lock so other requests keep flowing. A fixed + - ``POST /_control/delay?substr=S&ms=N&method=PUT|GET|LIST&once=0|1`` — every request matching + ``method`` (default ``PUT``) whose key contains ``S`` sleeps ``N`` milliseconds before it is + served, outside the store lock so other requests keep flowing. ``LIST`` matches a GET with an + empty key and a ``prefix`` query, against the prefix value rather than the key. A fixed per-request delay, not a modelled per-object rate cap: it charges an isolated write the same as a - burst. Each delayed PUT also increments the ``DelayedPut`` counter (visible at - ``/_control/counters``), so a caller can prove the knob fired rather than infer it from timing. - ``substr=&ms=0`` clears it; + burst. ``once=1`` clears the whole knob the instant it matches, so only the very first matching + request is ever delayed -- needed to fire a fault exactly once and let the retry through clean. + Each delayed request increments the ``DelayedRequest`` counter, and a delayed PUT also + increments ``DelayedPut`` (both visible at ``/_control/counters``), so a caller can prove the + knob fired rather than infer it from timing. Every capture record also carries ``arrival_seq``, + assigned when the request arrives, before any delay is applied -- unlike ``seq``, which is + assigned when the handler finishes and so can order a delayed request's own faster reissue + ahead of it. ``substr=&ms=0`` clears it; - ``POST /_control/mode?if_match=reject|ignore&omit_generation=0|1`` — select the adversarial behaviours below. Global, not per bucket: the client reuses connections across buckets and a per-bucket switch would invite a test to believe it had isolated something it had not. @@ -138,11 +145,17 @@ def __init__(self): # Adversarial behaviour, off by default — see the module docstring. self.if_match_mode = "reject" self.omit_generation = False - # `/_control/delay`: every PUT whose key contains `delay_substr` sleeps `delay_ms` before it is - # served, outside the store lock so other requests keep flowing. A fixed per-request delay, not - # a modelled rate cap — see the module docstring's `/_control/delay` bullet. + # `/_control/delay`: a request matching `delay_method`/`delay_substr` sleeps `delay_ms` before + # it is served, outside the store lock so other requests keep flowing. A fixed per-request + # delay, not a modelled rate cap — see the module docstring's `/_control/delay` bullet. + # `delay_method` is one of PUT (match the key, default), GET (match the key) or LIST (a GET + # with an empty key and a `prefix` query, matched against the prefix value). `delay_once` + # clears the whole knob the instant it matches, so only the first matching request is ever + # delayed — needed to fire a fault exactly once and let the retry through clean. self.delay_substr = "" self.delay_ms = 0 + self.delay_method = "PUT" + self.delay_once = False # `/_control/first_per_key_throttle`: while enabled, every key in `throttled_keys_seen` has # already been refused once and is now served normally; a key not yet in the set gets added # and refused with 429 instead of being dispatched. @@ -151,10 +164,20 @@ def __init__(self): self._next_generation = _GENERATION_SEED self._next_etag_ordinal = 1 self._next_upload_ordinal = 1 + self._next_arrival_seq = 0 def count(self, name): self.counters[name] = self.counters.get(name, 0) + 1 + def next_arrival_seq(self): + """A strictly increasing id assigned when a request ARRIVES (before any delay), unlike + ``seq`` on the capture record, which reflects when its handler FINISHES. A delayed + request's handler can finish after its own faster reissue, so ``seq`` alone cannot order + them; ``arrival_seq`` can. Must be called with ``_LOCK`` held.""" + value = self._next_arrival_seq + self._next_arrival_seq += 1 + return value + def mint_generation(self): value = str(self._next_generation) self._next_generation += _GENERATION_STRIDE @@ -172,6 +195,25 @@ def mint_etag(self): STORE = Store() +def _delay_matches(method, key, query): + """Whether this request is the one the `/_control/delay` knob targets. + + Must be called with `_LOCK` held: it reads `STORE.delay_*` and the caller pairs it with clearing + a `once` knob atomically. `query` is the parsed query dict (values are lists), as everywhere else + in this module. + """ + if not STORE.delay_ms or not STORE.delay_substr: + return False + if STORE.delay_method == "LIST": + # A LIST is a GET with an empty key and a `prefix` query; match the prefix value, not the key. + if method != "GET" or key or "prefix" not in query: + return False + return any(STORE.delay_substr in value for value in query["prefix"]) + if STORE.delay_method not in ("PUT", "GET"): + return False + return method == STORE.delay_method and bool(key) and STORE.delay_substr in key + + def _xml_escape(text): return ( str(text) @@ -830,11 +872,23 @@ def handle_control(path, method, query): {"Content-Type": "application/json"}, ) if path == "/_control/delay" and method == "POST": + delay_method = query.get("method", ["PUT"])[0] + if delay_method not in ("PUT", "GET", "LIST"): + return _bad_request("unknown delay method " + delay_method) STORE.delay_substr = query.get("substr", [""])[0] STORE.delay_ms = int(query.get("ms", ["0"])[0]) + STORE.delay_method = delay_method + STORE.delay_once = query.get("once", ["0"])[0] == "1" return Reply( 200, - json.dumps({"substr": STORE.delay_substr, "ms": STORE.delay_ms}).encode(), + json.dumps( + { + "substr": STORE.delay_substr, + "ms": STORE.delay_ms, + "method": STORE.delay_method, + "once": STORE.delay_once, + } + ).encode(), {"Content-Type": "application/json"}, ) if path == "/_control/first_per_key_throttle" and method == "POST": @@ -850,6 +904,8 @@ def handle_control(path, method, query): STORE.counters = {} STORE.delay_substr = "" STORE.delay_ms = 0 + STORE.delay_method = "PUT" + STORE.delay_once = False STORE.first_per_key_throttle = False STORE.throttled_keys_seen = set() return Reply(200, b"OK") @@ -899,9 +955,21 @@ def _dispatch(self, method, want_body=True): stripped = path.lstrip("/") bucket, _, key = stripped.partition("/") - delayed = method == "PUT" and STORE.delay_ms and STORE.delay_substr and STORE.delay_substr in key + # Whether this request matches the `/_control/delay` knob, and how long to sleep for it, is + # decided under the lock so a `once` knob is consumed by exactly one request even when + # several requests race here; the sleep itself still happens outside the lock so other + # requests keep flowing while this one is delayed. + with _LOCK: + arrival_seq = STORE.next_arrival_seq() + delayed = _delay_matches(method, key, query) + delay_ms = STORE.delay_ms if delayed else 0 + if delayed and STORE.delay_once: + STORE.delay_substr = "" + STORE.delay_ms = 0 + STORE.delay_method = "PUT" + STORE.delay_once = False if delayed: - time.sleep(STORE.delay_ms / 1000.0) + time.sleep(delay_ms / 1000.0) with _LOCK: STORE.count("method_" + method) @@ -910,7 +978,9 @@ def _dispatch(self, method, want_body=True): # substring that no longer matches the key) — this counter is the caller's proof the # sleep above actually ran. if delayed: - STORE.count("DelayedPut") + STORE.count("DelayedRequest") + if method == "PUT": + STORE.count("DelayedPut") throttled = STORE.first_per_key_throttle and (bucket, key) not in STORE.throttled_keys_seen if throttled: STORE.throttled_keys_seen.add((bucket, key)) @@ -919,6 +989,7 @@ def _dispatch(self, method, want_body=True): STORE.requests.append( { "seq": len(STORE.requests), + "arrival_seq": arrival_seq, "method": method, "bucket": bucket, "key": key, @@ -959,6 +1030,7 @@ def _dispatch(self, method, want_body=True): STORE.requests.append( { "seq": len(STORE.requests), + "arrival_seq": arrival_seq, "method": method, "bucket": bucket, "key": key, diff --git a/tests/integration/test_cas_gcs/test.py b/tests/integration/test_cas_gcs/test.py index a3bcb62a5dfe..0441f07a11e1 100644 --- a/tests/integration/test_cas_gcs/test.py +++ b/tests/integration/test_cas_gcs/test.py @@ -52,6 +52,14 @@ PLAIN_HMAC_DISK = "plain_gcs_hmac" PLAIN_HMAC_BUCKET = "plainhmacbucket" +# A CAS disk dedicated to the first-attempt-fuse proof: same fake bucket as `cas_gcs_hmac` but its +# own physical prefix (so it shares no keys with it) and a tightened `attempt_timeout_ms`, so a +# deliberate fake-server delay is a genuine transport timeout instead of sailing through the 5000 ms +# default. Deliberately excluded from `CAS_DISKS`, so no parametrized test or run-wide assertion +# picks it up. +FUSE_DISK = "cas_gcs_hmac_fuse" +FUSE_BUCKET = "hmacbucket" + NUM_ROWS = 200 # Where the fixture installs the disk configuration, so a test can rewrite it and reload. @@ -134,6 +142,26 @@ def start_cluster(): "
plain_gcs_hmac
" "", ) + # `FUSE_DISK`: same bucket as `cas_gcs_hmac`, its own physical prefix and `cas_server_root_id` + # (so it owns a disjoint key space), with `attempt_timeout_ms` tightened to 200 -- see + # `test_a_first_attempt_timeout_is_reissued_as_attempt_two` for why this needs its own disk + # rather than a reload of `cas_gcs_hmac`'s setting. + node.replace_in_config( + CONFIG_IN_CONTAINER, + "", + "object_storages3" + "casitest-cas-gcs-hmac-fuse" + "http://fakegcs:8080/hmacbucket/cas-fuse/" + "gcs_hmacGOOG1EFAKEACCESSKEYID" + "fake-goog4-hmac-secret" + "200", + ) + node.replace_in_config( + CONFIG_IN_CONTAINER, + "", + "
cas_gcs_hmac_fuse
" + "", + ) node.replace_in_config( CONFIG_IN_CONTAINER, "", @@ -1471,6 +1499,104 @@ def test_first_per_key_throttling_is_transparently_absorbed(disk): node.query("DROP TABLE IF EXISTS {} SYNC".format(table)) +def test_a_first_attempt_timeout_is_reissued_as_attempt_two(): + """A first-attempt fuse timeout is reissued at once, on the wire, as attempt 2. + + The fake's `/_control/delay` knob delays the first matching request past the first-attempt + fuse; the engine's zero-pause reissue must show up as a second request carrying + `clickhouse-request: ...attempt=2` — for a LIST issued directly by a GC round, and for a + conditional PUT of an INSERT's `.meta` marker, which settles with a `GET` before its own + attempt 2 and counts as a `CASRequestResolveRead`. + + The fuse only trips on a real transport timeout, and `content_addressed`'s + `attempt_timeout_ms` defaults to 5000 -- a 300 ms fake delay would sail through that budget on + the ordinary `cas_gcs_hmac` disk and never throw at all. `FUSE_DISK` is a second CAS disk + the fixture mounts alongside it (own `cas_server_root_id`, own physical prefix under the same + bucket, so it shares no keys with `cas_gcs_hmac`'s traffic) with `attempt_timeout_ms` tightened + to 200 -- the only way to make a 300 ms delay a genuine transport timeout without touching the + engine's C++ default. `attempt_timeout_ms` is frozen at pool-open time (mount-time), not + reloadable, hence a dedicated disk rather than a temporary `SYSTEM RELOAD CONFIG` on the + existing one. + """ + node = cluster.instances["node"] + disk = FUSE_DISK + # No `/_control/reset` here: it would wipe the fake's cumulative capture log, which the + # run-wide fence at the end of this file depends on seeing from the very start of the run. + # No earlier test in this file arms the delay knob, so there is nothing stale to clear. + node.query("DROP TABLE IF EXISTS fuse_probe SYNC") + node.query( + "CREATE TABLE fuse_probe (id UInt64) ENGINE = MergeTree ORDER BY id " + "SETTINGS storage_policy = '{}'".format(disk) + ) + node.query("INSERT INTO fuse_probe VALUES (1)") + _quiesce_merges(node, "fuse_probe") + node.query("SYSTEM CAS GC STOP '{}'".format(disk)) # only the explicit round below may LIST + try: + # LIST: a GC round's first LIST lists the `gc/server-roots/` family; the first matching + # LIST is delayed past the fuse and must be reissued at once as attempt 2. + seq = _next_seq() + delayed_before = _counters().get("DelayedRequest", 0) + assert _control_post("/_control/delay?substr=gc&ms=300&method=LIST&once=1")["method"] == "LIST" + node.query("SYSTEM CAS GC RUN '{}'".format(disk)) + assert _counters().get("DelayedRequest", 0) - delayed_before == 1, "the LIST delay must fire exactly once" + lists = [ + r + for r in _captured_since(seq, FUSE_BUCKET) + if r["method"] == "GET" and not r["key"] and "prefix=" in r["query"] + ] + # The fake appends a request's capture record's `seq` when ITS OWN handler finishes, not + # when the client issued it: the delayed LIST's handler is still sleeping out its 300 ms + # when the reissue (a fresh connection, unaffected by the knob once `once=1` cleared it) + # completes and gets a lower `seq`. So the pair is identified by sharing one `query` (the + # same prefix, reissued), and ordered by `arrival_seq` -- assigned when a request arrives, + # before any delay is applied, so it reflects issue order rather than completion order. + by_query = {} + for r in lists: + by_query.setdefault(r["query"], []).append(r) + retried = [group for group in by_query.values() if len(group) >= 2] + assert len(retried) == 1, (lists, by_query) + pair = retried[0] + assert len(pair) == 2, pair + list2 = [r for r in pair if r["headers"].get("clickhouse-request", "").endswith("attempt=2")] + list1 = [r for r in pair if r not in list2] + assert len(list1) == 1 and len(list2) == 1, pair + list1, list2 = list1[0], list2[0] + assert list1["headers"].get("clickhouse-request", "").endswith("attempt=1") or not list1["headers"].get( + "clickhouse-request", "" + ), pair + assert list1["arrival_seq"] < list2["arrival_seq"], (list1, list2) + + # Conditional PUT: delay the first `.meta` PUT of the next insert; expect a settlement GET + # arriving strictly between PUT(1) and its reissue PUT(2), and one settlement read counted. + seq = _next_seq() + resolve_reads_before = _resolve_reads(node) + delayed_before = _counters().get("DelayedRequest", 0) + assert _control_post("/_control/delay?substr=.meta&ms=300&method=PUT&once=1")["method"] == "PUT" + node.query("INSERT INTO fuse_probe VALUES (2)") + assert _counters().get("DelayedRequest", 0) - delayed_before == 1, "the PUT delay must fire exactly once" + rows = [r for r in _captured_since(seq, FUSE_BUCKET) if r["key"].endswith(".meta")] + assert rows, "no `.meta` PUT reached the fake, so this test would be vacuous" + meta_key = rows[0]["key"] + same_key = [r for r in rows if r["key"] == meta_key] + puts = [r for r in same_key if r["method"] == "PUT"] + gets = [r for r in same_key if r["method"] == "GET"] + assert len(puts) >= 2, same_key + assert gets, "no settlement GET reached the fake between PUT(1) and its reissue" + put2 = [r for r in puts if r["headers"].get("clickhouse-request", "").endswith("attempt=2")] + put1 = [r for r in puts if r not in put2] + assert len(put1) == 1 and len(put2) == 1, puts + put1, put2 = put1[0], put2[0] + assert put1["headers"].get("clickhouse-request", "").endswith("attempt=1") or not put1["headers"].get( + "clickhouse-request", "" + ), puts + settle_get = min(gets, key=lambda r: r["arrival_seq"]) + assert put1["arrival_seq"] < settle_get["arrival_seq"] < put2["arrival_seq"], (put1, settle_get, put2) + assert _resolve_reads(node) - resolve_reads_before >= 1 + finally: + node.query("SYSTEM CAS GC START '{}'".format(disk)) + node.query("DROP TABLE IF EXISTS fuse_probe SYNC") + + # MUST STAY LAST IN THIS FILE. The fake's capture log is global and cumulative and nothing in this # module resets it, so this assertion covers exactly the traffic that precedes it. # Add new tests ABOVE this line. diff --git a/tests/integration/test_cas_mount_renewal_retry/configs/request_budget_disks.xml b/tests/integration/test_cas_mount_renewal_retry/configs/request_budget_disks.xml new file mode 100644 index 000000000000..8030a2f5da0d --- /dev/null +++ b/tests/integration/test_cas_mount_renewal_retry/configs/request_budget_disks.xml @@ -0,0 +1,51 @@ + + + + + + object_storage + s3 + cas + 30 + 10000 + itest-cas-budget-capped + http://rustfs1:11121/test/cas_request_budget_capped/ + clickhouse + clickhouse + false + + 1000 + 5000 + + + object_storage + s3 + cas + 30 + 10000 + itest-cas-budget-unbounded + http://rustfs1:11121/test/cas_request_budget_unbounded/ + clickhouse + clickhouse + false + + 0 + 5000 + 50000 + + + + diff --git a/tests/integration/test_cas_mount_renewal_retry/test.py b/tests/integration/test_cas_mount_renewal_retry/test.py index 8f9fecbc5856..73b0f9008436 100644 --- a/tests/integration/test_cas_mount_renewal_retry/test.py +++ b/tests/integration/test_cas_mount_renewal_retry/test.py @@ -108,7 +108,26 @@ def _decode_mount(body): return json.loads(lines[1]) -def _renewal_log_rows(node, since, sequence): +def _log_count_since_last_restart(node, pattern): + # The shortened renewal period makes this server log heavily enough to rotate + # clickhouse-server.log mid-test, so a plain grep on the live file alone can miss matches that + # already rotated out to clickhouse-server.log.N.gz. Concatenate the rotated files (oldest + # first, by the numeric suffix) followed by the live file, then count matches only after the + # LAST "Starting ClickHouse" line -- i.e. since the current server incarnation's own start -- + # so the count is a clean per-restart delta regardless of how many times rotation happened. + script = ( + "combined=$(mktemp); " + "for f in $(ls /var/log/clickhouse-server/clickhouse-server.log.[0-9]*.gz 2>/dev/null " + "| sort -t. -k3,3rn); do zcat \"$f\" >> \"$combined\"; done; " + "cat /var/log/clickhouse-server/clickhouse-server.log >> \"$combined\"; " + "start_line=$(grep -n 'Starting ClickHouse' \"$combined\" | tail -1 | cut -d: -f1); " + "tail -n +\"${start_line:-1}\" \"$combined\" | grep -c -- '%s' || true; " + "rm -f \"$combined\"" + ) % pattern + return int(node.exec_in_container(["bash", "-c", script]).strip()) + + +def _renewal_log_rows(node, since): node.query("SYSTEM FLUSH LOGS") rows = node.query( "SELECT outcome, detail['seq'], detail['write_attempt_id'], " @@ -117,9 +136,8 @@ def _renewal_log_rows(node, since, sequence): "WHERE event_type = 'watermark_renew' AND disk_name = '{}' " "AND detail['server_root_id'] = '{}' " "AND event_time_microseconds >= toDateTime64('{}', 6) " - "AND detail['seq'] = '{}' " "ORDER BY event_time_microseconds FORMAT TSV".format( - DISK, SERVER_ROOT_ID, since, sequence + DISK, SERVER_ROOT_ID, since ) ) return [tuple(row.split("\t")) for row in rows.splitlines() if row] @@ -133,6 +151,15 @@ def start_cluster(): with_rustfs=True, stay_alive=True, ) + # A separate instance for the two `disk_cas_budget_*` probe disks (see + # configs/request_budget_disks.xml): the hard-restart tests below count EXACT occurrences of a + # production log line across every writable CAS disk on `node`, so sharing that node with more + # writable CAS disks would inflate their counts. + cluster.add_instance( + "budget_probe", + main_configs=["configs/request_budget_disks.xml"], + with_rustfs=True, + ) cluster.base_cmd.extend( ["--file", os.path.join(os.path.dirname(__file__), "docker_compose_proxy.yml")] ) @@ -155,7 +182,11 @@ def start_cluster(): ) ) node.query("INSERT INTO renewal_probe VALUES (0, 'before')") - yield {"node": node, "control_url": control_url} + yield { + "node": node, + "budget_probe": cluster.instances["budget_probe"], + "control_url": control_url, + } finally: if control_url is not None: try: @@ -165,6 +196,25 @@ def start_cluster(): cluster.shutdown() +def test_openpoolview_handoff_freezes_the_connect_cap_from_the_real_disk(start_cluster): + # `ContentAddressedMetadataStorage::openPoolView` derives `connect_timeout_cap_ms` from the + # disk's OWN S3 client (`freezeConnectTimeoutCapMs`) and hands it into the pool's request budget; + # `Pool::open` logs that budget once, at startup, through logger `CasRequestBudget`. This proves + # the handoff end to end through the two disks' real startup, not through a test-constructed + # backend: `disk_cas_budget_capped` (connect_timeout_ms=1000) must show the derived cap 1000 and + # envelope 7000; `disk_cas_budget_unbounded` (connect_timeout_ms=0, Poco's "unbounded") must show + # the cap falling back to the attempt timeout itself, 5000, and envelope 15000. Either disk + # opening with the base client's own timeout instead (1000/1000/... or 2000/...) would mean the + # freeze was skipped or the frozen value never reached the backend. + node = start_cluster["budget_probe"] + assert _log_count_since_last_restart( + node, "CAS request budget in effect: attempt_timeout_ms=5000 connect_timeout_cap_ms=1000 envelope_ms=7000" + ) == 1 + assert _log_count_since_last_restart( + node, "CAS request budget in effect: attempt_timeout_ms=5000 connect_timeout_cap_ms=5000 envelope_ms=15000" + ) == 1 + + def test_transient_mount_renewal_retries_without_remount(start_cluster): node = start_cluster["node"] control_url = start_cluster["control_url"] @@ -331,3 +381,56 @@ def resolved_snapshot(): assert recovered[3] == "1", rows assert recovered[4] == "committed_by_read", rows print("targeted request count (landed response lost): {}".format(stats["faults"]), flush=True) + + +def test_hard_restart_observes_then_the_unsafe_knob_skips_the_observation(start_cluster): + node = start_cluster["node"] + + def log_count_since_last_restart(pattern): + return _log_count_since_last_restart(node, pattern) + + # 1150 = mountObservationThresholdMs(ttl_ms=1000, poll=max(1, period/2)=100): ttl + ttl/20 + poll. + observation = "waiting ~1150 ms (token-stability observation)" + epoch_before = int( + node.query( + "SELECT writer_epoch FROM system.cas_mounts WHERE disk = '{}' LIMIT 1".format(DISK) + ).strip() + ) + + # A hard kill leaves the previous incarnation's mount slot claimed; the restart must pay the + # token-stability observation wait once before it can safely reclaim it. + node.stop_clickhouse(kill=True) + node.start_clickhouse() + assert log_count_since_last_restart(observation) == 1 + assert _mount_snapshot(node)["state"] == "live" + # This restart already reclaims the slot and advances the epoch on its own (via the observation + # wait, not the knob), so the knob-restart's own advance must be measured from THIS value, not + # from epoch_before -- otherwise a knob-restart that wrongly reused this same epoch would still + # pass an `epoch_after > epoch_before` check. + epoch_after_safe_restart = int( + node.query( + "SELECT writer_epoch FROM system.cas_mounts WHERE disk = '{}' LIMIT 1".format(DISK) + ).strip() + ) + assert epoch_after_safe_restart > epoch_before + + # Enable the unsafe knob while the server is stopped (a test-stand-only config.d overlay), then + # hard-kill again: this server's own uuid already holds the slot, so the knob may reclaim it at + # once and skip the observation wait entirely. + node.stop_clickhouse(kill=True) + node.copy_file_to_container( + os.path.join(os.path.dirname(__file__), "configs/unsafe_remount.xml"), + "/etc/clickhouse-server/config.d/unsafe_remount.xml", + ) + try: + node.start_clickhouse() + assert log_count_since_last_restart(observation) == 0 + assert _mount_snapshot(node)["state"] == "live" + epoch_after_knob_restart = int( + node.query( + "SELECT writer_epoch FROM system.cas_mounts WHERE disk = '{}' LIMIT 1".format(DISK) + ).strip() + ) + assert epoch_after_knob_restart > epoch_after_safe_restart + finally: + node.exec_in_container(["rm", "-f", "/etc/clickhouse-server/config.d/unsafe_remount.xml"]) From d651b2362344ceb448ee73090af517cfca9d5f70 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 7 Sep 2026 07:24:27 +0200 Subject: [PATCH 34/81] cas: an explicit unsafe no-delay remount after a lapsed lease Problem (CI, `cas_selects` regression on PR #2300, third fix): once the lease had lapsed and the mount was fenced, the only way back was a restart that paid the full token-stability observation (TTL + TTL/20 + a poll, 36-41 s with the defaults) although the operator knew the previous holder was this very process. There was no supported way to say so. `cas_unsafe_remount_no_delay` (disk setting, `PoolConfig::unsafe_remount_no_delay`) lets the startup claim take an explicit unsafe reclaim authorization: `claimMount` accepts it, records the result as `MountPriorState::UncleanUnsafe`, and the mount-conflict log lines, the operator text and `system.cas_mounts` name it. The knob is consulted at exactly one site, only at startup; `tryRemountOnce` ignores it and never reclaims a live successor sharing its uuid; GC's fence-out threshold is unaffected; the reclaim reasons use the token-stability observation wording consistently. The setting's description and `docs/en/antalya/cas` state the exposure in full: the risk is availability (a concurrent holder can be fenced), never data. Tests: `CASMount*` in `gtest_cas_mount.cpp` (authorization path, the exact one-millisecond-short fence-out discriminator, the live-successor case, GC threshold pin), `gtest_cas_pool.cpp`, and `test_cas_mount_renewal_retry` gains a hard-restart scenario: with the knob the server remounts at once, without it the observation runs, and the epoch assertion isolates the knob restart. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../cas/architecture/mounts-and-leases.md | 36 +++- docs/en/antalya/cas/configuration.md | 18 ++ .../ContentAddressedMetadataStorage.cpp | 3 + .../ContentAddressedMetadataStorage.h | 3 + .../ContentAddressedSettings.cpp | 1 + .../ContentAddressed/Gc/CasGc.cpp | 5 +- .../ContentAddressed/Pool/CasMountRuntime.h | 17 +- .../ContentAddressed/Pool/CasPool.cpp | 50 ++++- .../ContentAddressed/Pool/CasPool.h | 10 + .../ContentAddressed/Pool/CasServerRoot.cpp | 63 +++--- .../ContentAddressed/Pool/CasServerRoot.h | 37 +++- src/Disks/tests/gtest_cas_gc_ack_floor.cpp | 53 ++++- src/Disks/tests/gtest_cas_mount.cpp | 50 ++++- src/Disks/tests/gtest_cas_pool.cpp | 183 +++++++++++++++++- src/Disks/tests/gtest_cas_settings.cpp | 17 ++ .../configs/storage_conf.xml | 9 + .../configs/unsafe_remount.xml | 12 ++ .../test_cas_mount_renewal_retry/test.py | 77 +++++--- 18 files changed, 547 insertions(+), 97 deletions(-) create mode 100644 tests/integration/test_cas_mount_renewal_retry/configs/unsafe_remount.xml diff --git a/docs/en/antalya/cas/architecture/mounts-and-leases.md b/docs/en/antalya/cas/architecture/mounts-and-leases.md index 1681c85c96c4..eb852b070da0 100644 --- a/docs/en/antalya/cas/architecture/mounts-and-leases.md +++ b/docs/en/antalya/cas/architecture/mounts-and-leases.md @@ -61,7 +61,9 @@ Two failure modes this closes: over, regardless of lease expiry. - A **same-uuid live twin** (two processes sharing one uuid file and `server_root_id`) is caught separately, by the mount claim's token-stability observation, and aborts with an operator-facing message rather - than corrupting the pool. + than corrupting the pool — this is the default behavior, with `cas_unsafe_remount_no_delay` off. + With it on, a same-uuid claim over such a slot reclaims at once instead of observing (see + `cas_unsafe_remount_no_delay` in the configuration reference). ## The mount lease {#mount-lease} @@ -117,9 +119,25 @@ inside authority already proved by the last confirmed lease. GC's own view of a dead server is symmetric and clock-skew-immune: a slot becomes fence-eligible only after the leader observes the *same* renewal token hold stable, on its own monotonic clock, -for `TTL + TTL/20 + cadence` — the identical formula a re-mounting server uses to wait out a -predecessor. The stamped `expires_at_ms` never participates in that decision; wall-clock `now` is -audit-only. +for `TTL + floor(TTL/20) + period` — close to, but not identical to, the threshold a re-mounting +server uses to wait out a predecessor, which observes `TTL + floor(TTL/20) + max(1, +floor(period/2))`. Both thresholds are evaluated purely on the observer's own clock and its own +configured `TTL`/`period`; nothing about the writer's timing travels on the wire. The stamped +`expires_at_ms` never participates in either decision — it is a writer-stamped diagnostic used by +`system.cas_mounts` and by the non-authoritative decommission epoch-recovery precheck, never an +authorization; local fencing is derived instead from the confirmed request's pre-I/O `BOOTTIME` +anchor plus the TTL, and wall-clock `now` stays audit-only. + +Every server sharing a pool must therefore run the identical `cas_mount_lease_ttl_ms` and +`cas_mount_renew_period_ms`: a member or GC leader configured with a shorter threshold than its +peers can fence out a healthy peer whose token-update gap merely exceeds that shorter threshold — +a peer renewing frequently stays live, one that missed a renewal does not. Change these values only +with every member of the pool stopped; a graceful restart removes only that member's own startup +observation and does not make mixed thresholds safe. With the defaults (TTL 30 s, period 10 s, +margin 2 s), `TTL − margin − period − 2 × envelope = 4 s` is the scheduling-lateness budget before +the first renewal attempt of a period can begin, where `envelope = attempt_timeout + 2 × cap` and +`cap` is `attempt_timeout` when the disk's `connect_timeout_ms` is `0`, else +`min(connect_timeout_ms, attempt_timeout)` (7 s with defaults). ## The two monotone counters {#counters} @@ -158,6 +176,7 @@ a `MountClaimResult::Kind` together with a `MountPriorState` describing which ce | `Clean` | the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) | | `Fenced` | GC's own threshold-gated fence-out (`gc_fenced`) | | `UncleanObserved` | this claimant's own token-stability observation held for the full `TTL + drift` window | +| `UncleanUnsafe` | the operator's explicit `cas_unsafe_remount_no_delay` authorization carried the slot's exact token — not a certificate of death | ## Behavioral mount-slot model {#mount-state-machines} @@ -174,6 +193,7 @@ stateDiagram-v2 Fenced --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim Terminated --> Live: same-uuid claim with a fresh writer_epoch, instant reclaim Live --> Live: same-uuid claim, proven-dead token via UncleanObserved + Live --> Live: same-uuid claim under cas_unsafe_remount_no_delay, no observation Fenced --> Fenced: same uuid and epoch claim, FencedSelf, no write Live --> Absent: decommission tail, mount then epoch then owner tombstone Terminated --> [*] @@ -204,10 +224,10 @@ under a live mount is an operator-level event. **Writable open** runs in a strict order: bootstrap-residual proof, capability probe under a random per-mount prefix, pool-meta create-or-validate, `validateServerRootId`, owner claim, -`allocateWriterEpoch`, mount claim and synchronous renewer start, materialization grace if the -predecessor was unclean (default 30 s), arm the fence, then create and release the runtime-owned -renewal and remount workers before the writable pool becomes externally visible. If the grace period -consumed the TTL, one fresh synchronous renewal re-anchors the deadline before the fence is armed. +`allocateWriterEpoch`, mount claim and synchronous renewer start, arm the fence, then create and +release the runtime-owned renewal and remount workers before the writable pool becomes externally +visible. If the claim consumed the TTL, one fresh synchronous renewal re-anchors the deadline +before the fence is armed. Failure to construct either worker joins the partial pair, closes the fence, and fails the writable open. No incident path constructs a thread. diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 3b3ed87b6998..436cf143b267 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -105,8 +105,21 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_gc_read_concurrency` | `16` | Bounded pool size for the GC fold's read-ahead of checkpoints, ref logs, manifests and zero-candidate HEADs; `1` disables | | `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove, conditional write), at least 1. Together with the connect cap it forms the attempt envelope (`cas_attempt_timeout_ms + 2 × cap`; the cap is `cas_attempt_timeout_ms` itself when the disk's `connect_timeout_ms` is `0`, else `min(connect_timeout_ms, cas_attempt_timeout_ms)`) that the lease arithmetic reserves: one TCP connect and one TLS handshake under the cap each, send/receive bounded per socket operation by `cas_attempt_timeout_ms`. With background renewal the cadence check requires `cas_mount_renew_period_ms + 2 × envelope + cas_lease_safety_margin_ms < cas_mount_lease_ttl_ms`, which puts an effective ceiling on the frozen connect cap: under the defaults (TTL 30000, period 10000, margin 2000) the envelope must stay under 9000, so a disk `connect_timeout_ms` of 2000 ms or more refuses to open writable — lower the connect timeout or raise the TTL if you hit this | | `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: the attempt envelope + `cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, and `cas_mount_renew_period_ms` + 2 × envelope + `cas_lease_safety_margin_ms` too, or the disk refuses to open writable | +| `cas_unsafe_remount_no_delay` | `0` | Reclaim a mount slot that carries this server's own uuid at once after a hard restart, without observing the slot's token for the lease TTL. Unsafe whenever two processes can hold the same `server_uuid` (a copied uuid file, a stalled predecessor). After such a reclaim the predecessor can still start conditional writes until its own cutoff (`confirmed deadline − cas_lease_safety_margin_ms − 2 × envelope`) or until its next renewal meets the token guard, and a request it already sent may still materialize later. That is not a data hazard: ref-log keys carry `(writer_epoch, sequence)` and creates are conditional, so two writers can never commit different bodies to one key, and recovery's epoch seal settles any straggler (recovery fails closed after 64 successive seal-create attempts displaced by newly materializing old-epoch transactions). The exposure is availability, not data. Intended for test stands and deployments that guarantee one process per uuid | | `cas_staging_backend` | `local` | Blob staging backend (`local` \| `s3`); `s3` is opt-in and requires native same-store copy on writable mount | +All servers sharing a pool must run the same `cas_mount_lease_ttl_ms` and `cas_mount_renew_period_ms`. +Startup reclaim and GC's fence-out both judge liveness by the mount slot's write token holding stable +on the observer's own `CLOCK_BOOTTIME`, using the observer's own threshold — nothing about a writer's +timing travels on the wire. Startup observes `cas_mount_lease_ttl_ms + floor(cas_mount_lease_ttl_ms / +20) + max(1, floor(cas_mount_renew_period_ms / 2))`; GC observes `cas_mount_lease_ttl_ms + +floor(cas_mount_lease_ttl_ms / 20) + cas_mount_renew_period_ms`. A pool member or GC leader +configured with a shorter threshold than its peers can therefore fence out a healthy peer whose +token-update gap exceeds that shorter threshold — a peer renewing frequently stays live, one that +missed a renewal does not. Change these values only with every member of the pool stopped: a +graceful restart removes only that member's own startup observation and does not make mixed +thresholds safe. + A shorter TTL reduces the tolerance for object-storage delays; a shorter renewal period increases it (renewal starts earlier) at the cost of more background traffic. With the defaults, `cas_mount_lease_ttl_ms − cas_lease_safety_margin_ms − cas_mount_renew_period_ms − 2 × envelope = @@ -116,6 +129,11 @@ where `envelope = cas_attempt_timeout_ms + 2 × cap` (7000 ms with defaults) and `min(connect_timeout_ms, cas_attempt_timeout_ms)` (1000 ms with defaults); the renewal then keeps retrying until `confirmed deadline − cas_lease_safety_margin_ms`. +The `expires_at_ms` stamped into the mount object is a writer-stamped diagnostic used by +`system.cas_mounts` and by the non-authoritative decommission epoch-recovery precheck; it never +authorizes a reclaim or a GC fence-out. Local fencing is derived instead from the confirmed +request's pre-I/O `CLOCK_BOOTTIME` anchor plus the TTL. + ## Advanced GC pacing settings {#advanced-gc-pacing-settings} These settings bound individual phases of a `GC` round. The first two accept any `UInt64` value; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp index fab17ff8eef0..78666e0ad297 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp @@ -79,6 +79,7 @@ namespace ContentAddressedSetting extern const ContentAddressedSettingsUInt64 gc_round_outcome_entry_budget; extern const ContentAddressedSettingsUInt64 mount_lease_ttl_ms; extern const ContentAddressedSettingsUInt64 mount_renew_period_ms; + extern const ContentAddressedSettingsBool unsafe_remount_no_delay; extern const ContentAddressedSettingsUInt64 part_folder_cache_bytes; extern const ContentAddressedSettingsUInt64 part_folder_cache_max_entries; extern const ContentAddressedSettingsUInt64 part_folder_cache_max_entry_bytes; @@ -302,6 +303,7 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage( , gc_round_outcome_entry_budget(settings_[ContentAddressedSetting::gc_round_outcome_entry_budget].value) , mount_lease_ttl(std::chrono::milliseconds(settings_[ContentAddressedSetting::mount_lease_ttl_ms].value)) , mount_renew_period(std::chrono::milliseconds(settings_[ContentAddressedSetting::mount_renew_period_ms].value)) + , cas_unsafe_remount_no_delay(settings_[ContentAddressedSetting::unsafe_remount_no_delay].value) , cas_part_folder_cache_bytes(settings_[ContentAddressedSetting::part_folder_cache_bytes].value) , cas_part_folder_cache_max_entries(settings_[ContentAddressedSetting::part_folder_cache_max_entries].value) , cas_part_folder_cache_max_entry_bytes(settings_[ContentAddressedSetting::part_folder_cache_max_entry_bytes].value) @@ -807,6 +809,7 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP pool_config.cas_request_budget.connect_timeout_cap_ms = freezeConnectTimeoutCapMs(object_storage, cas_attempt_timeout_ms); pool_config.mount_lease_ttl_ms = mount_lease_ttl; pool_config.mount_renew_period = mount_renew_period; + pool_config.unsafe_remount_no_delay = cas_unsafe_remount_no_delay; pool_config.event_sink = makeCasEventSink(); /// Built here rather than above so it carries the budget the pool was configured with. Only a diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h index 65b6349e7367..bffb608688cc 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h @@ -624,6 +624,9 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC const uint64_t gc_round_outcome_entry_budget; const std::chrono::milliseconds mount_lease_ttl; const std::chrono::milliseconds mount_renew_period; + /// See `PoolConfig::unsafe_remount_no_delay` -- the operator's explicit acceptance of an + /// unobserved same-uuid reclaim. + const bool cas_unsafe_remount_no_delay; /// Part-folder view cache settings. `cas_part_folder_cache_bytes == 0` disables retention. const uint64_t cas_part_folder_cache_bytes; const uint64_t cas_part_folder_cache_max_entries; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp index 29b9d12e37f6..b2a3651b2472 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp @@ -72,6 +72,7 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_"; DECLARE(UInt64, gc_round_outcome_entry_budget, 5000, "GcOutcomes per-round entry cap across the redelete/spared audit log (0 = unbounded)", 0) \ DECLARE(UInt64, mount_lease_ttl_ms, 30000, "Mount lease validity after a successful claim or renewal, in milliseconds", 0) \ DECLARE(UInt64, mount_renew_period_ms, 10000, "Interval between background mount lease renewals, in milliseconds", 0) \ + DECLARE(Bool, unsafe_remount_no_delay, false, "Reclaim a mount slot that carries this server's own uuid at once after a hard restart, without observing the slot's token for the lease TTL. Unsafe whenever two processes can hold the same server_uuid (a copied uuid file, a stalled predecessor): after such a reclaim the predecessor can still start conditional writes until its own cutoff (confirmed deadline − margin − 2 × envelope) or until its next renewal meets the token guard, and a request it already sent may materialize later. Ref-log keys carry (writer_epoch, sequence) and creates are conditional, so two writers can never commit different bodies to one key, and recovery's epoch seal settles stragglers -- the exposure is availability, not data: recovery fails closed after 64 successive seal-create attempts displaced by newly materializing old-epoch transactions. Intended for test stands and deployments that guarantee one process per uuid", 0) \ DECLARE(String, server_root_id, "", "REQUIRED explicit layout subtree identity; macros expand as in the s3 endpoint", 0) \ DECLARE(UInt64, part_folder_cache_bytes, 64ULL << 20, "Part-folder view cache byte budget (0 disables retention)", 0) \ DECLARE(UInt64, part_folder_cache_max_entries, 10000, "Part-folder view cache entry cap", 0) \ diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 5890d1d150ae..d4241c0d50ce 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -465,8 +465,9 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// rounds via `new_round`, not on heartbeat acks). Fencing no longer trusts a predecessor's stamped /// `expires_at_ms` against our wall clock — it /// fences ONLY once `mount_obs` has watched the mount's write-token hold unchanged for the full - /// threshold on THIS leader's own monotonic clock (mirrors `claimMountAwaitingExpiry`'s identical - /// `TTL + Drift` threshold for a mount's own reopen). + /// threshold on THIS leader's own monotonic clock (shares `claimMountAwaitingExpiry`'s + /// `TTL + Drift` formula for a mount's own reopen wait, but with the full renewal period as the + /// cadence term below instead of half of it, so the two thresholds are close, not identical). const uint64_t ttl_ms = static_cast(store->poolConfig().mount_lease_ttl_ms.count()); /// The formula is shared with `claimMountAwaitingExpiry` via /// `mountObservationThresholdMs` -- see its doc comment (CasServerRoot.h). diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h index bc278279aefe..7f3ff1d76f36 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.h @@ -111,10 +111,12 @@ struct MountConfig }; /// Local, in-memory write fence. It is deliberately not checked by reading the object store for every -/// write: the `MountLeaseRenewer` is the sole lease reader/renewer. A successful renewal translates the -/// durable `expires_at_ms` into `deadline_boot_ms`; a foreign owner, newer `writer_epoch`, or failed -/// renewal latches `lost`. Mutable operations are allowed only while the latch is clear and the local -/// deadline has not passed. The `writer_epoch` is the durable fencing token. +/// write: the `MountLeaseRenewer` is the sole lease reader/renewer. A successful renewal computes +/// `deadline_boot_ms` from its own confirmed request's pre-I/O `CLOCK_BOOTTIME` anchor plus the lease +/// TTL, never from the durable `expires_at_ms` stamp, which is a writer-stamped diagnostic only; a +/// foreign owner, newer `writer_epoch`, or failed renewal latches `lost`. Mutable operations are +/// allowed only while the latch is clear and the local deadline has not passed. The `writer_epoch` is +/// the durable fencing token. /// /// The fence uses `CLOCK_BOOTTIME`, not `CLOCK_MONOTONIC`: monotonic time does not advance while a VM is /// suspended, so a resumed sleeper would compute the same "not yet expired" verdict it had before the nap @@ -389,6 +391,13 @@ class CasMountRuntime /// Sleep through the injected test hook when present; otherwise use the production thread sleep. /// `Pool` claim observation and materialization grace waits share this seam so tests control both. void waitSleep(uint64_t ms) const; + /// Swap the wait hook after construction -- a test that must change what a wait DOES partway + /// through a scenario (e.g. driving a second incarnation's renewal from inside the observed + /// incarnation's own poll) cannot express that through `PoolConfig::wait_sleep_fn` alone, since + /// that value is fixed at open time. Unsynchronized against `waitSleep`'s `const` read of the same + /// field: safe only called from the test's own thread before any worker is running (no persistent + /// renewal/remount worker reads `config.wait_sleep_fn` concurrently with this write). + void setWaitSleepForTest(std::function fn) { config.wait_sleep_fn = std::move(fn); } /// Forward renewer events to the injected sink. The sink is held by reference so it observes the /// owning pool's current event routing for the runtime's entire lifetime. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 4c3d6ad9cf22..041e8d1d9086 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -699,10 +699,32 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol if (policy == MountClaimPolicy::WaitForExpiry) { CasOperation claim_op = store->gc_requests.admit(); - claim = claimMountAwaitingExpiry( - claim_op, store->pool_layout, srid, our_uuid, writer_epoch, - [&now_ms]() { return now_ms(); }, [raw] { return raw->bootMsNow(); }, - ttl_ms, poll_interval_ms, sleep_ms, on_wait_start, emit_mount_event); + const bool unsafe = store->config.unsafe_remount_no_delay; + if (unsafe) + { + /// One bare attempt first: a slot held by OUR uuid under another epoch is reclaimed at + /// once under the operator's authorization, carrying the exact token this read saw so a + /// slot that moves in between is refused. Every outcome but a claim (an absent slot + /// freshly minted, or a same-epoch refresh) falls through to the ordinary observed path + /// below. + claim = claimMount(claim_op, store->pool_layout, srid, our_uuid, writer_epoch, now_ms(), ttl_ms, + /*proven_dead_incarnation=*/{}, emit_mount_event); + if (claim.kind == MountClaimResult::LiveDoubleStart && claim.etag + && claim.body && claim.body->server_uuid == our_uuid) + claim = claimMount(claim_op, store->pool_layout, srid, our_uuid, writer_epoch, now_ms(), ttl_ms, + {}, emit_mount_event, /*unsafe_reclaim_authorization=*/claim.etag); + } + /// A `FencedSelf`, a foreign-uuid `LiveDoubleStart`/`ForeignOwner`, or a raced + /// `LiveDoubleStart` the authorization above did not cover falls through here and re-runs + /// the bare `claimMount` a second time inside `claimMountAwaitingExpiry`'s own loop; under + /// the knob that means the same conflict is recorded twice in the mount audit stream for + /// one open. The outcome this open ends in is unaffected -- only the audit stream gains a + /// duplicate row, and only when the knob is set. + if (!unsafe || claim.kind != MountClaimResult::Claimed) + claim = claimMountAwaitingExpiry( + claim_op, store->pool_layout, srid, our_uuid, writer_epoch, + [&now_ms]() { return now_ms(); }, [raw] { return raw->bootMsNow(); }, + ttl_ms, poll_interval_ms, sleep_ms, on_wait_start, emit_mount_event); } else { @@ -778,9 +800,11 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol } /// A reclaim over a predecessor whose death was NOT proven clean may still have a conditional PUT - /// from that predecessor in flight -- `Fenced` and `UncleanObserved` are exactly the two - /// `MountPriorState`s with no such proof (`Clean`, drained farewell, and `None`, a fresh mount / - /// same-epoch refresh with nothing to hand over, are the proven ones). + /// from that predecessor in flight -- `Fenced`, `UncleanObserved`, and `UncleanUnsafe` are exactly + /// the three `MountPriorState`s with no such proof (`UncleanUnsafe` has no proof at all, not merely + /// no proof of a CLEAN death: it is the operator's explicit `cas_unsafe_remount_no_delay` + /// acceptance of that risk, with no observation behind it). `Clean`, drained farewell, and `None`, a + /// fresh mount / same-epoch refresh with nothing to hand over, are the proven ones. /// An EXHAUSTIVE switch, not a positive allowlist -- a future `MountPriorState` /// enumerator with no proof of clean death must fail the BUILD (a missing `-Wswitch` case), never /// silently fall through to "clean". @@ -802,6 +826,7 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol break; case MountPriorState::Fenced: case MountPriorState::UncleanObserved: + case MountPriorState::UncleanUnsafe: unclean_reclaim = true; break; } @@ -810,7 +835,10 @@ void Pool::mountWritable(PoolPtr & store, UInt128 our_uuid, MountClaimPolicy pol LOG_INFO(getLogger("CasPool"), "Content-addressed mount {} follows a predecessor whose death was not proven clean " "(writer_epoch {}). Opening without a grace period: a still-in-flight conditional PUT from " - "that predecessor is fenced by the recovery seal, whenever it arrives.", srid, writer_epoch); + "that predecessor is fenced by the recovery seal, whenever it arrives.{}", srid, writer_epoch, + claimed_prior == MountPriorState::UncleanUnsafe + ? " (reclaimed without observation under cas_unsafe_remount_no_delay)" + : ""); } /// Arm the local write fence: cache (uuid, epoch) and set the boottime deadline at the claim @@ -1413,7 +1441,11 @@ bool Pool::tryRemountOnce() /// unlike the initial `open`, every event fired below reaches the real sink immediately. const auto emit_mount_event = [this](CasEvent e) { emitEvent(std::move(e)); }; - const auto sleep_ms = [](uint64_t ms) { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); }; + /// Routes through `mount_runtime.waitSleep` (which itself routes through `config.wait_sleep_fn` + /// when a test injected one) rather than a bare `sleep_for` directly, so a test intercepting + /// `wait_sleep_fn` observes every wait a self-remount can block on, exactly like `Pool::open`'s + /// own observation poll above. + const auto sleep_ms = [this](uint64_t ms) { mount_runtime.waitSleep(ms); }; step = "mount_claim"; CasOperation claim_op = gc_requests.admit(); const MountClaimResult claim = claimMountAwaitingExpiry( diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 5e2b74e7ac91..71a10fc803a2 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -216,6 +216,11 @@ struct PoolConfig /// `mount_renew_period` (default ttl/3) so a healthy mount renews well before expiry. std::chrono::milliseconds mount_lease_ttl_ms{30000}; std::chrono::milliseconds mount_renew_period{10000}; /// = ttl/3 by default + + /// `cas_unsafe_remount_no_delay`: reclaim a same-uuid, different-epoch, uncertified mount slot at + /// once, with no token-stability observation at all. Unsafe whenever two processes can hold the + /// same server_uuid; see the setting's own description for the exact risk. + bool unsafe_remount_no_delay = false; bool read_only = false; /// observe-only open: skip the mutating capability probe; reads only /// Boot-time "start now, fix later": skip the access-check-class part of the capability probe @@ -469,6 +474,11 @@ class Pool : public std::enable_shared_from_this { mount_runtime.setArmMountFenceInterpositionHookForTest(std::move(hook)); } + /// Swap the observation-wait hook after open -- see `CasMountRuntime::setWaitSleepForTest`. + void setWaitSleepForTest(std::function fn) + { + mount_runtime.setWaitSleepForTest(std::move(fn)); + } /// The fence clock: CLOCK_BOOTTIME in milliseconds (includes VM-suspend time, unlike /// CLOCK_MONOTONIC — see `MountFence`). Consults the injected `config.boot_ms_fn` if set (tests), /// otherwise `bootMs`. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp index 50e525b7d31c..10de8e758b3c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp @@ -787,7 +787,7 @@ void emitMountEvent(const CasEventSink & sink, CasEventType type, const String & MountClaimResult claimMount( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation, - const CasEventSink & sink) + const CasEventSink & sink, const std::optional & unsafe_reclaim_authorization) { const String key = l.mountKey(srid); const auto got = op.read(key, Retry::standard()); @@ -856,12 +856,17 @@ MountClaimResult claimMount( /// observation threshold on its own clock; re-deriving that here from a bare wall-clock /// comparison is exactly the cross-node trust that makes a clock-skewed or delayed observer /// unsafe. + /// - `unsafe_reclaim_authorization` matches the one we just read → the operator's + /// `cas_unsafe_remount_no_delay` setting explicitly authorized this reclaim with NO + /// observation at all; the caller read this exact token and accepted the availability risk. /// Anything else → `LiveDoubleStart` (do NOT write): a same-uuid, different-epoch, not fenced, not - /// clean-marked, not (yet) proven-dead lease may simply be a live twin, and `expires_at_ms` alone - /// can never distinguish that from a dead predecessor across two different clocks. + /// clean-marked, not (yet) proven-dead, not unsafe-authorized lease may simply be a live twin, and + /// `expires_at_ms` alone can never distinguish that from a dead predecessor across two different + /// clocks. const bool clean_marker = existing.min_active_build_sequence == std::numeric_limits::max(); const bool proven_dead = proven_dead_incarnation && *proven_dead_incarnation == got->etag; - if (existing.gc_fenced || clean_marker || proven_dead) + const bool unsafe_authorized = unsafe_reclaim_authorization && *unsafe_reclaim_authorization == got->etag; + if (existing.gc_fenced || clean_marker || proven_dead || unsafe_authorized) { const MountLease body = makeMountBody(our_uuid, our_epoch, existing.seq + 1, now_ms, ttl_ms); if (const std::optional raced @@ -873,18 +878,22 @@ MountClaimResult claimMount( return racedDoubleStart(*raced); const MountPriorState prior = existing.gc_fenced ? MountPriorState::Fenced : clean_marker ? MountPriorState::Clean - : MountPriorState::UncleanObserved; + : proven_dead ? MountPriorState::UncleanObserved + : MountPriorState::UncleanUnsafe; emitMountEvent(sink, CasEventType::MountClaim, srid, "reclaim", &existing, existing.gc_fenced ? "same server_uuid, different writer_epoch, GC-fenced — reclaimed" : clean_marker ? "same server_uuid, different writer_epoch, clean farewell — reclaimed" - : "same server_uuid, different writer_epoch, observed dead by " - "incarnation stability — reclaimed"); + : proven_dead ? "same server_uuid, different writer_epoch, observed dead by " + "token-stability observation — reclaimed" + : "same server_uuid, different writer_epoch, reclaimed at once under " + "cas_unsafe_remount_no_delay — the operator accepted that a live " + "predecessor with this uuid may still be writing"); return {.kind = MountClaimResult::Claimed, .body = body, .prior = prior, .etag = std::nullopt}; } emitMountEvent(sink, CasEventType::MountConflict, srid, "live_double_start", &existing, "same server_uuid, different writer_epoch, not fenced/clean/proven-dead — no wall-clock trust; " - "the caller must run the incarnation-stability observation wait before reclaiming"); + "the caller must run the token-stability observation wait before reclaiming"); /// No write was attempted on this path -- `got->etag` is exactly the CURRENT body's /// etag (what we just read is what's still there), so it is safe to hand back for the /// caller's observation loop to compare across polls without a redundant re-read. @@ -906,28 +915,23 @@ String mountDoubleStartMessage(const String & srid, const std::optional for this disk.\n" " - If the other server is a stale/zombie process, stop it; this server will then reclaim the mount on restart.\n" - " - CLOCK SKEW CAVEAT: liveness is judged by comparing the lease's wall-clock expires_at_ms against\n" - " THIS server's clock, so a large clock skew between the two servers can misjudge it (a healthy holder\n" - " may look mounted here, or a dead one may look live). Verify both servers' clocks are in sync (NTP).\n" + " - LIVENESS: this wait judges the holder alive by its write token holding stable on THIS server's\n" + " own clock for the full observation threshold; the stamped expires_at_ms above never enters that\n" + " judgment on its own -- it is a writer-stamped diagnostic (also shown in system.cas_mounts), not\n" + " an authorization. Every server sharing this pool must run the SAME cas_mount_lease_ttl_ms and\n" + " cas_mount_renew_period_ms: a server configured with a shorter threshold than its peers can fence\n" + " out a healthy one.\n" " - If the local ClickHouse uuid file was regenerated, restore the old uuid file, or remove the stale\n" " owner object gc/server-roots/{}/owner only after verifying no server uses this root.\n" " - As a LAST RESORT, after verifying that NO server is writing this root, manually delete the mount\n" - " object gc/server-roots/{}/mount and restart; this server will then re-claim it.", + " object gc/server-roots/{}/mount and restart; this server will then re-claim it.\n" + " - For a test stand or a deployment that guarantees one process per server_uuid,\n" + " cas_unsafe_remount_no_delay reclaims a slot carrying this server's own uuid at once instead of\n" + " waiting -- but a live predecessor sharing this uuid may still be writing, so enable it only\n" + " under that guarantee.", srid, identity, srid, srid); } -namespace -{ -/// Bounded number of observation restarts before giving up on a same-uuid slot whose write-token keeps -/// changing: each restart means the token changed DURING our observation window — i.e. something is -/// actively renewing it. A genuinely dead predecessor's token never changes again after its last -/// renewal, so it is observed stable well within one window; only a truly LIVE writer (a real second -/// incarnation, or the predecessor's own background renewer racing our first few polls) keeps resetting -/// the clock. Bounding this converts "wait forever for a live twin" into the same bounded-then-report -/// shape the old wall-clock wait had, without ever trusting a wall-clock deadline to get there. -constexpr size_t kMaxObservationRestarts = 3; -} - uint64_t mountObservationThresholdMs(uint64_t ttl_ms, uint64_t cadence_ms) { return ttl_ms + ttl_ms / 20 + cadence_ms; @@ -948,8 +952,9 @@ MountClaimResult claimMountAwaitingExpiry( /// Rate-bound observation threshold: the full lease TTL, plus a 5% allowance for clock-rate /// mismatch between the holder's and our own local clock, plus one poll interval for observation /// discreteness. It is measured only with OUR OWN clock (`mono_ms_fn`); no cross-node wall-clock - /// comparison participates in this loop. The shared helper keeps the startup and GC thresholds - /// identical. + /// comparison participates in this loop. `poll` here is half the renewal period (the caller's own + /// poll cadence), so this threshold is close to, but not identical to, GC's heartbeat fence-out + /// threshold, which passes the full renewal period into the same shared helper. const uint64_t threshold_ms = mountObservationThresholdMs(ttl_ms, poll); std::optional observed; @@ -960,7 +965,7 @@ MountClaimResult claimMountAwaitingExpiry( { const bool threshold_met = observed && mono_ms_fn() - observed_since >= threshold_ms; MountClaimResult r = claimMount(op, l, srid, our_uuid, our_epoch, now_ms_fn(), ttl_ms, - threshold_met ? observed : std::nullopt, sink); + threshold_met ? observed : std::nullopt, sink, /*unsafe_reclaim_authorization=*/{}); if (r.kind != MountClaimResult::LiveDoubleStart) return r; @@ -1004,7 +1009,7 @@ MountClaimResult claimMountAwaitingExpiry( on_wait_start(*r.body, threshold_ms); LOG_INFO(getLogger("CasMountLease"), "Attempting to mount content-addressed server root {} after node change or hard " - "restart; waiting ~{} ms (incarnation-stability observation) to confirm the previous " + "restart; waiting ~{} ms (token-stability observation) to confirm the previous " "incarnation's operations are all finalized", srid, threshold_ms); } @@ -1105,7 +1110,7 @@ HeartbeatFloor computeHeartbeatFloor(CasOperation & op, const Layout & l, uint64 LOG_INFO(getLogger("CasHeartbeatFloor"), "CAS GC fenced out mount lease for content-addressed server root {} at " "wall-clock ms {}: its write incarnation held unchanged for >= {} ms on the GC " - "leader's own monotonic clock (incarnation-stability observation)", + "leader's own monotonic clock (token-stability observation)", srid, now_ms, stable_threshold_ms); return true; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h index f8f6c788925f..1c1fd8928f97 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h @@ -184,7 +184,8 @@ uint64_t allocateWriterEpoch(CasOperation & op, const Layout & l, const String & EpochMintPolicy policy, uint64_t now_ms, const ObserveRefCatalog & observe_catalog); -/// Which certificate of death justified a same-uuid, different-epoch mount reclaim. `None` when no +/// Which certificate of death, or the operator's explicit unsafe authorization, justified a +/// same-uuid, different-epoch mount reclaim. `None` when no /// reclaim of that kind happened (a fresh claim, a /// same-epoch refresh, `LiveDoubleStart`, `ForeignOwner`, `FencedSelf`). enum class MountPriorState @@ -193,6 +194,7 @@ enum class MountPriorState Clean, /// the predecessor's own graceful farewell (`min_active_build_sequence == UINT64_MAX`) Fenced, /// the GC leader's own (already threshold-gated) fence-out (`gc_fenced`) UncleanObserved, /// OUR observation watched the incarnation hold stable for the full threshold + UncleanUnsafe, /// the operator's explicit `cas_unsafe_remount_no_delay` authorization carried the slot's exact token }; /// Startup decision for the mount lease (`gc/server-roots//mount`), run AFTER the owner gate @@ -205,7 +207,8 @@ enum class MountPriorState /// - `gc_fenced` → terminal for THIS (uuid, epoch) — a fence costs an epoch, so refreshing it /// in place would reactivate a fenced incarnation → `FencedSelf` (no write); /// - otherwise → refresh (`replace` to bump seq + fresh `expires_at_ms`) → `Claimed`; -/// - same `server_uuid`, DIFFERENT `writer_epoch` → reclaimed ONLY on a certificate of death that +/// - same `server_uuid`, DIFFERENT `writer_epoch` → reclaimed ONLY on a certificate of death, or the +/// operator's explicit unsafe authorization, that /// needs no fresh wall-clock trust (see /// `claimMountAwaitingExpiry` below for how a plain "looks expired" reading is turned into one): /// - `gc_fenced` (the GC leader already, itself, threshold-gated this incarnation dead; a fence @@ -215,6 +218,9 @@ enum class MountPriorState /// - `proven_dead_incarnation` matches the CURRENTLY OBSERVED incarnation (the caller itself /// watched that exact incarnation hold stable for the full observation threshold) → reclaim, `prior = /// UncleanObserved`; +/// - `unsafe_reclaim_authorization` matches the CURRENTLY OBSERVED incarnation (the operator's +/// explicit `cas_unsafe_remount_no_delay` authorization, carrying the exact token read, with NO +/// observation at all) → reclaim, `prior = UncleanUnsafe`; /// - none of the above → `LiveDoubleStart` (do NOT write). In particular `expires_at_ms <= /// now_ms` ALONE is never sufficient — comparing a predecessor's stamp against OUR wall clock /// is unsafe because a clock-skewed or merely late-observing @@ -241,7 +247,8 @@ struct MountClaimResult /// message may name a holder, and the lease this server merely PROPOSED is not one. An optional /// rather than the proposal, because a caller cannot check a convention it cannot see. std::optional body; - /// Which certificate of death justified a same-uuid, different-epoch `Claimed` reclaim (`None` for + /// Which certificate of death, or the operator's explicit unsafe authorization, justified a + /// same-uuid, different-epoch `Claimed` reclaim (`None` for /// every other `Kind`, and for the absent-slot / same-epoch-refresh `Claimed` cases). MountPriorState prior = MountPriorState::None; /// The incarnation of the body this result observed, for @@ -271,10 +278,14 @@ class MountFencedException : public DB::Exception /// CURRENTLY observed incarnation is the ONLY way (besides `gc_fenced` / the clean marker) a /// same-uuid different-epoch lease is ever reclaimed. Absent (`{}`, the default) for a bare claim /// attempt with no such proof. +/// `unsafe_reclaim_authorization`: the exact token the operator's `cas_unsafe_remount_no_delay` read +/// off a same-uuid, different-epoch slot before authorizing this reclaim, with no observation at all. +/// Never reused from `proven_dead_incarnation`: that one says the token was OBSERVED dead, this one +/// says the operator accepted the risk. Absent (`{}`, the default) when the knob is off. MountClaimResult claimMount( CasOperation & op, const Layout & l, const String & srid, UInt128 our_uuid, uint64_t our_epoch, uint64_t now_ms, uint64_t ttl_ms, const std::optional & proven_dead_incarnation = {}, - const CasEventSink & sink = {}); + const CasEventSink & sink = {}, const std::optional & unsafe_reclaim_authorization = {}); /// Format the operator-actionable startup error shown when the mount lease is held by a genuinely /// live second server (the same `server_root_id` is mounted twice). Produced only AFTER this server @@ -308,12 +319,22 @@ String mountDoubleStartMessage(const String & srid, const std::optional & now_ms_fn, diff --git a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp index 84dfe33f1400..10b8ae92b3f1 100644 --- a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp +++ b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp @@ -891,6 +891,17 @@ TEST(CASGCAckFloor, PublishBeforeGraduationSpares) EXPECT_TRUE(blobExists(*backend, store->layout(), blob)); } +namespace +{ + +/// Shared body for the fence-out timing invariant: opens a pool from `config`, then drives the exact +/// two-round scenario described below. Parameterized only by `config` so the same scenario can be run +/// against the default `PoolConfig` (`ExpiredMountFencedOutAndExcluded`) and against +/// `unsafe_remount_no_delay = true` (`CASGcFenceOut.ThresholdUnchangedByUnsafeKnob`), proving the knob +/// changes nothing about the fence-out threshold or its round count. `backend` and `events` are declared +/// BEFORE the Pool inside this same function so they outlive the background syncer's emits (ASan +/// 2026-07-09) -- the Pool must never outlive the function that opened it. +/// /// A dead mount is fenced out by the round's heartbeat step: gc_fenced is set on its body (a /// token-guarded rewrite that bumps seq). The fence is pure liveness (re-arms the write fence so a /// resumed sleeper can never mutate again); reclaim itself no longer depends on any mount's heartbeat — @@ -901,14 +912,15 @@ TEST(CASGCAckFloor, PublishBeforeGraduationSpares) /// (`expires_at_ms`) against the GC's own clock — it fences ONLY once GC has watched a mount's write /// token hold unchanged for the full threshold on its OWN monotonic clock. That takes (at least) two /// `computeHeartbeatFloor` calls spanning the threshold, so this test drives the GC leader's own -/// (persistent) `mono_ms_fn` across two rounds: round 1 seeds the observation for both mounts; the -/// STORE's own mount is then renewed (as a live leader would) before round 2 crosses the threshold — -/// srid2, never renewed again after its one-shot claim, is the one that gets fenced. -TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) +/// (persistent) `mono_ms_fn` across three rounds: round 1 seeds the observation for both mounts; the +/// STORE's own mount is then renewed (as a live leader would); round 2, one millisecond short of the +/// threshold, discriminates the threshold's exact value (must NOT fence yet); round 3, exactly at the +/// threshold, is where srid2 — never renewed again after its one-shot claim — gets fenced. +void runExpiredMountFenceOutScenario(const PoolConfig & config) { auto backend = std::make_shared(); std::vector events; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) - auto store = openPoolForTest(backend); + auto store = Pool::open(backend, config); const Layout & layout = store->layout(); // srid2's renewer claims ONE lease via `start` and is never renewed again — tests never enable @@ -950,9 +962,19 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) // The store's OWN mount renews between rounds (as a live leader would); srid2 never does. store->renewWatermarkOnce(); + gc_mono = threshold_ms - 1; + + // Round 2 (mono == threshold - 1): a discriminator for the threshold's EXACT value, not just its + // existence — one millisecond short of the full threshold, srid2's original token must NOT be fenced + // yet. Without this round, any knob-shortened positive threshold would also satisfy the fence-out + // assertion taken only at the full threshold below. + const RoundReport rep_before_threshold = gc.runRegularRound(); + EXPECT_EQ(rep_before_threshold.fence_outs, 0u); + EXPECT_FALSE(decodeMountLease(readObj(*backend, layout.mountKey(srid2))->bytes).gc_fenced); + gc_mono = threshold_ms; - // Round 2 (mono == threshold): srid2's original token has held stable for the full threshold — + // Round 3 (mono == threshold): srid2's original token has held stable for the full threshold — // fenced. The store's own (just-renewed) mount restarts its observation and stays live. const RoundReport rep = gc.runRegularRound(); @@ -988,6 +1010,25 @@ TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) EXPECT_TRUE(runRoundsUntilAbsent(store, gc, *backend, layout, blob)); } +} + +TEST(CASGCAckFloor, ExpiredMountFencedOutAndExcluded) +{ + runExpiredMountFenceOutScenario(PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); +} + +/// GC's fence-out threshold (`ttl + 5% drift allowance + one round's worth of renewal slack`, computed in +/// `Gc::runRegularRound`) never reads `PoolConfig::unsafe_remount_no_delay` -- that knob is consulted only +/// by `Pool::mountWritable`'s own reclaim decision, never by GC's heartbeat-floor observation. Runs the +/// IDENTICAL three-round scenario as `ExpiredMountFencedOutAndExcluded` with the knob turned on, and +/// asserts the SAME round-by-round fence-out counts -- including the one-millisecond-short discriminator +/// round -- proving the threshold's exact value and its timing are unaffected. +TEST(CASGcFenceOut, ThresholdUnchangedByUnsafeKnob) +{ + runExpiredMountFenceOutScenario( + PoolConfig{.pool_prefix = "p", .server_root_id = "test", .unsafe_remount_no_delay = true}); +} + /// fix-round F6 (author-review: `Gc`'s own `mono_ms_fn` used to default to the RAW static `Pool:: /// bootMs()`, bypassing the Pool's own injectable `config.boot_ms_fn` -- a time-controlled test can /// desync the mount side's fake clock from the GC side's real one). This mirrors diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 4f008b6724ea..31865606e788 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -1,4 +1,5 @@ #include +#include #include "cas_test_helpers.h" #include #include @@ -14,6 +15,7 @@ #include #include #include +#include namespace DB::ErrorCodes { @@ -890,11 +892,13 @@ TEST(CASMountMessage, DoubleStartTextHasIdentityAndRemediation) EXPECT_NE(msg.find("unique"), std::string::npos); EXPECT_NE(msg.find("reclaim the mount on restart"), std::string::npos); EXPECT_NE(msg.find("uuid file"), std::string::npos); - /// Clock-skew caveat + manual mount-object delete escape hatch. - EXPECT_NE(msg.find("CLOCK SKEW"), std::string::npos); - EXPECT_NE(msg.find("NTP"), std::string::npos); + /// Token-stability liveness statement (replaces the old wall-clock CLOCK SKEW caveat) + manual + /// mount-object delete escape hatch + the unsafe-knob escape hatch. + EXPECT_NE(msg.find("own clock"), std::string::npos); + EXPECT_NE(msg.find("diagnostic"), std::string::npos); EXPECT_NE(msg.find("manually delete the mount"), std::string::npos); EXPECT_NE(msg.find("gc/server-roots/replica-a/mount"), std::string::npos); + EXPECT_NE(msg.find("cas_unsafe_remount_no_delay"), std::string::npos); } /// rev.6: a stamped `expires_at_ms` that already looks past-due on our wall clock must NOT shortcut @@ -1094,6 +1098,46 @@ TEST(CASMountAwaitExpiry, SkewedFarFutureExpiryHasNoEffectOnObservationThreshold EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 8u); // reclaimed } +TEST(CASMountClaim, UnsafeAuthorizationIsTokenExact) +{ + auto b = std::make_shared(); + Layout l("p"); + Ops ops(b); + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 30000).kind, MountClaimResult::Claimed); + const Etag stale = ops.op.read(l.mountKey("r"), Retry::standard())->etag; + + /// `Etag` equality compares (key, value), and it is minted only through the request planes -- there + /// is no cross-key comparison to exercise here. Build the stale token by refreshing the SAME slot a + /// second time (same uuid, same epoch): `stale`, read before this refresh, is then a genuinely stale + /// token for the slot's CURRENT value, without touching an unrelated key. + ASSERT_EQ(claimMount(ops.op, l, "r", UInt128(1), 7, /*now*/ 1000, /*ttl*/ 30000).kind, MountClaimResult::Claimed); + const Etag current = ops.op.read(l.mountKey("r"), Retry::standard())->etag; + + /// A stale token is refused: nothing authorizes a reclaim over a slot that moved. + const MountClaimResult refused = claimMount(ops.op, l, "r", UInt128(1), 8, 1000, 30000, /*proven_dead=*/{}, + /*sink=*/{}, /*unsafe_reclaim_authorization=*/stale); + EXPECT_EQ(refused.kind, MountClaimResult::LiveDoubleStart); + + /// A foreign uuid is refused before the authorization is consulted. + const MountClaimResult foreign = claimMount(ops.op, l, "r", UInt128(2), 8, 1000, 30000, {}, {}, current); + EXPECT_EQ(foreign.kind, MountClaimResult::ForeignOwner); + + /// The exact token reclaims, with the prior state and the audit reason naming the setting. + std::vector events; + const MountClaimResult reclaimed = claimMount(ops.op, l, "r", UInt128(1), 8, 1000, 30000, {}, + [&](CasEvent e) { events.push_back(std::move(e)); }, current); + ASSERT_EQ(reclaimed.kind, MountClaimResult::Claimed); + EXPECT_EQ(reclaimed.prior, MountPriorState::UncleanUnsafe); + ASSERT_FALSE(events.empty()); + EXPECT_THAT(events.back().reason, testing::HasSubstr("cas_unsafe_remount_no_delay")); + /// Pin the fields `system.cas_log` consumers actually key on, not just the free-form reason: the + /// unsafe reclaim shares the same event type and outcome as every other reclaim (`emitMountEvent`'s + /// "reclaim" branch argument), so nothing about this path is a separate, unaudited channel. + EXPECT_EQ(events.back().type, CasEventType::MountClaim); + EXPECT_EQ(events.back().outcome, "reclaim"); + EXPECT_EQ(decodeMountLease(ops.op.read(l.mountKey("r"), Retry::standard())->bytes).writer_epoch, 8u); +} + TEST(CASMountLease, RenewerStartAdoptsOurOwnClaimNotDoubleStart) { auto b = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 320778e2dcc2..115912061d70 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -1799,6 +1799,127 @@ TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) "response-time reading taken after renewerStart/quiesceRefTablesForRemount"; } +/// ==== self-remount vs. a live successor carrying the same uuid under the unsafe-reclaim knob ==== +/// +/// `cas_unsafe_remount_no_delay` is consulted at exactly one site: the writable `Pool::open` claim. +/// `Pool::tryRemountOnce` (self-remount after a fence loss) does NOT consult it -- an incarnation +/// superseded by a duplicate-uuid process must still OBSERVE the slot's write-token before it may +/// reclaim, or two processes sharing a uuid (a copied uuid file, a stalled predecessor restarted under +/// the knob) would alternate authority indefinitely. + +TEST(CASMountRemount, SupersededIncarnationDoesNotReclaimALiveSuccessor) +{ + auto backend = std::make_shared(); + uint64_t boot_a = 0; + uint64_t boot_b = 0; + /// Mirrors `UncleanOpenPaysOnlyTheObservationWindow`'s tiny budget: the 1s lease TTL below is far + /// under the default `cas_request_budget`, so it must be scaled down to fit the required-timeout + /// inequality (attempt_timeout + safety_margin < lease TTL). + const CasRequestBudget tiny_budget{ + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; + auto config_for = [&](uint64_t * boot, bool unsafe) + { + return PoolConfig{ + .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", + .mount_lease_ttl_ms = std::chrono::milliseconds(1000), + .mount_renew_period = std::chrono::milliseconds(200), + .unsafe_remount_no_delay = unsafe, + .cas_request_budget = tiny_budget, + .boot_ms_fn = [boot] { return *boot; }, + .wait_sleep_fn = [boot](uint64_t ms) { *boot += ms; }, + }; + }; + + PoolPtr pool_a = Pool::open(backend, config_for(&boot_a, /*unsafe=*/false)); + ASSERT_TRUE(pool_a); + /// B carries the SAME (server_root_id, server_id) as A -- a copied uuid file -- and opens over A's + /// still-live slot under the operator's unsafe knob, reclaiming it at once (no observation). + PoolPtr pool_b = Pool::open(backend, config_for(&boot_b, /*unsafe=*/true)); + ASSERT_TRUE(pool_b); + EXPECT_NE(pool_a->liveWriterEpoch(), pool_b->liveWriterEpoch()) + << "the unsafe reclaim must have minted B a fresh epoch over A's slot"; + + /// A's next renewal meets the token guard: same uuid, a newer epoch now sits on the slot. Pin the + /// terminal classification directly (the "superseded" branch of `throwRenewConflict`, the one + /// that maps to `MountRenewOutcome::Terminal`) rather than accepting any exception -- no accessor + /// exposes the renewer's outcome/state today, so the error code and the classification's own + /// wording are what distinguish this from every other terminal reason (foreign owner, GC fence, + /// vanished slot, an unresolved write). + try + { + pool_a->renewWatermarkOnce(); + FAIL() << "A's renewal must be refused once B's reclaim superseded its epoch"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::ABORTED); + EXPECT_NE(e.message().find("superseded by a newer incarnation"), std::string::npos) + << "actual message: " << e.message(); + } + EXPECT_FALSE(pool_a->mayMutate()) << "the superseded classification must trip A's local write fence closed"; + + /// A's self-remount now observes the slot's write-token. Drive B's renewal from INSIDE every one + /// of A's observation polls, so the token never stabilizes across the whole bounded observation -- + /// the knob is not consulted by `tryRemountOnce` (only by `Pool::open`), so nothing else could let + /// A reclaim a slot a live successor keeps renewing. This cannot deadlock: A and B are distinct + /// `Pool` objects, so B's `renewWatermarkOnce` takes none of A's locks (each `Pool` owns its own + /// `remount_mutex`), and the wait fires between `claimMountAwaitingExpiry`'s polls -- with no + /// backend request of A's own in flight -- so B's call is the only one touching the shared + /// in-memory backend at that instant. + size_t polls = 0; + pool_a->setWaitSleepForTest([&](uint64_t ms) + { + boot_a += ms; + ++polls; + boot_b += ms; + EXPECT_NO_THROW(pool_b->renewWatermarkOnce()); + }); + EXPECT_FALSE(pool_a->tryRemountOnce()) + << "a superseded incarnation must never reclaim a live successor's slot"; + /// Bounded, not merely nonzero: B renews on every poll, so the observed token changes every + /// iteration and the FIRST (non-restart) observation start plus `kMaxObservationRestarts` further + /// restarts is exactly the number of polls before `claimMountAwaitingExpiry` gives up -- one + /// `sleep_ms_fn` call per iteration that does not itself exceed the bound, and none on the + /// terminal iteration that does. A widened or removed restart bound would make this hang instead + /// of failing, so pin the exact count rather than only asserting it ran. + EXPECT_EQ(polls, DB::Cas::kMaxObservationRestarts + 1) + << "the observation must give up after exactly kMaxObservationRestarts restarts, not wait " + "indefinitely for a live twin to go quiet"; + + const MountLease final_lease = decodeMountLease(readObj(*backend, pool_a->layout().mountKey("test"))->bytes); + EXPECT_EQ(final_lease.writer_epoch, pool_b->liveWriterEpoch()) + << "the mount slot must still belong to B's incarnation -- A never reclaimed it"; +} + +/// Cutoff-only fencing: with no renewals and no competing incarnation at all, crossing the armed +/// deadline on the local BOOTTIME clock alone must fence a mount closed -- the mechanism +/// `SupersededIncarnationDoesNotReclaimALiveSuccessor` above relies on is not special-cased to a +/// renewal conflict; the plain boot-clock cutoff fences unconditionally. +TEST(CASMountRemount, CutoffFencesWithoutRenewals) +{ + auto backend = std::make_shared(); + uint64_t boot = 0; + const CasRequestBudget tiny_budget{ + .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; + PoolPtr store = Pool::open(backend, PoolConfig{ + .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", + .mount_lease_ttl_ms = std::chrono::milliseconds(1000), + .mount_renew_period = std::chrono::milliseconds(200), + .cas_request_budget = tiny_budget, + .boot_ms_fn = [&] { return boot; }, + .wait_sleep_fn = [&](uint64_t ms) { boot += ms; }, + }); + ASSERT_TRUE(store); + EXPECT_TRUE(store->mayMutate()) << "freshly armed at open, well within the ttl"; + + /// No renewals at all -- advance the boot clock past the armed deadline (open's claim anchor plus + /// the lease ttl) on this incarnation's own clock alone. + boot += 1001; + EXPECT_FALSE(store->mayMutate()) + << "crossing the armed deadline must fence closed on the boot clock alone, with no renewal " + "conflict needed to trip it"; +} + /// ==== rev.6 Task 5: clean-release drain gates the farewell marker ==== namespace @@ -2334,20 +2455,70 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) })); ASSERT_TRUE(store); - /// The token-stability observation window (>= the 500ms ttl) is paid, because this predecessor's - /// death was never certified -- only observed. + /// The token-stability observation window is paid in full, pinned to the exact configured + /// formula (`mountObservationThresholdMs`): threshold_ms = ttl_ms + ttl_ms/20 + poll_interval_ms + /// = 500 + 25 + 50 = 575 ms, where poll_interval_ms = max(1, mount_renew_period/2) = 50 ms. The + /// loop only re-checks the threshold between polls, so the observed wait rounds UP to the next + /// whole poll: ceil(575 / 50) * 50 = 600 ms, i.e. exactly 12 polls of 50 ms each -- because this + /// predecessor's death was never certified, only observed. uint64_t total = 0; for (uint64_t w : waits) total += w; - EXPECT_GE(total, 500u) << "the observation window must have been paid"; - /// And NOTHING is paid on top of it. Every recorded wait is a poll of that window, bounded by the - /// lease TTL; a wait longer than the whole window can only be a reintroduced grace period. + EXPECT_EQ(total, 600u) << "the observation window must be paid in full, poll-rounded to the " + "configured threshold -- neither less (a shortened wait) nor more " + "(a reintroduced grace period)"; + /// And every one of those polls is exactly one poll interval -- no wait beyond the observation + /// poll (the straggler it used to wait out is fenced by the recovery seal instead). for (uint64_t w : waits) - EXPECT_LE(w, 500u) + EXPECT_EQ(w, 50u) << "an unclean reclaim must not block on any wait beyond the observation poll -- the " "straggler it used to wait out is fenced by the recovery seal instead"; } +TEST(CASMountOpenWaits, UnsafeNoDelayOpensWithoutTheObservationWindow) +{ + auto b = std::make_shared(); + Layout l{"p"}; + DB::Cas::tests::seedPoolMetaForRestart(*b); + /// Same predecessor shape as UncleanOpenPaysOnlyTheObservationWindow above: a bare `claimMount` + /// plants the lease directly, with no clean-farewell marker and no `gc_fenced`, so this slot has no + /// certificate of death -- only `cas_unsafe_remount_no_delay` below will let the successor skip + /// observing it. + ASSERT_EQ(claimMount(*DB::Cas::tests::OperationForTest(b), l, "test", UInt128(1), 7, 1000, 500).kind, MountClaimResult::Claimed); + /// A real predecessor at epoch 7 durably minted this first; seed it here too, or the successor's + /// own `allocateWriterEpoch` trips the Phase C guard (epoch absent, mount present -> fail closed). + createObj(*b, l.epochKey("test"), encodeServerEpoch(ServerEpoch{.next_writer_epoch = 8})); + std::vector events; + uint64_t fake_boot = 0; + std::vector waits; + PoolPtr store; + /// Same server_id (uuid) as the seeded predecessor and a different epoch -- exactly the shape + /// `unsafe_remount_no_delay` is for. Unlike the neighbour test, no wait is expected: the bare + /// `claimMount` reclaims at once under the operator's authorization. + ASSERT_NO_THROW(store = Pool::open(b, PoolConfig{ + .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", + .event_sink = [&](CasEvent e) { events.push_back(std::move(e)); }, + .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), + .unsafe_remount_no_delay = true, + .cas_request_budget = CasRequestBudget{.attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}, + .boot_ms_fn = [&] { return fake_boot; }, + .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + })); + ASSERT_TRUE(store); + EXPECT_TRUE(waits.empty()) << "no observation window under the unsafe setting"; + /// `Pool` has no test accessor for the adopted `MountPriorState`, so the `UncleanUnsafe` + /// classification is asserted through the mount audit event instead: `claimMount`'s unsafe-reclaim + /// branch (`CasServerRoot.cpp`) emits exactly one `MountClaim`/"reclaim" event whose reason names + /// the setting, and `CASMountClaim.UnsafeAuthorizationIsTokenExact` already pins the classification + /// itself at the `claimMount` level. + const auto reclaim_event = std::ranges::find_if(events, + [](const CasEvent & e) { return e.reason.find("cas_unsafe_remount_no_delay") != String::npos; }); + ASSERT_NE(reclaim_event, events.end()); + EXPECT_EQ(reclaim_event->type, CasEventType::MountClaim); + EXPECT_EQ(reclaim_event->outcome, "reclaim"); + EXPECT_EQ(decodeMountLease((*DB::Cas::tests::OperationForTest(b)).read(l.mountKey("test"), Retry::standard())->bytes).writer_epoch, 8u); +} + TEST(CASMountOpenWaits, CleanOpenSkipsAllWaits) { auto b = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_settings.cpp b/src/Disks/tests/gtest_cas_settings.cpp index 1fa1b49311ca..53b3b50b4fbb 100644 --- a/src/Disks/tests/gtest_cas_settings.cpp +++ b/src/Disks/tests/gtest_cas_settings.cpp @@ -27,6 +27,7 @@ namespace DB::ContentAddressedSetting extern const ContentAddressedSettingsUInt64 gc_interval_sec; extern const ContentAddressedSettingsUInt64 gc_bulk_delete_chunk_keys; extern const ContentAddressedSettingsString scratch_path; + extern const ContentAddressedSettingsBool unsafe_remount_no_delay; } namespace @@ -577,3 +578,19 @@ TEST(CASContentAddressedSettings, AbsentScratchPathUsesDefaultVerbatim) s.loadFromConfig(*cfg, "disk", "/data", "/data/disks/x/cas_scratch", identity_macros); EXPECT_EQ(s[ContentAddressedSetting::scratch_path].value, "/data/disks/x/cas_scratch"); } + +TEST(CASContentAddressedSettings, UnsafeRemountNoDelayIsOffByDefault) +{ + { + auto cfg = makeConfig("srv1"); + ContentAddressedSettings s; + s.loadFromConfig(*cfg, "disk", "/data", "/data/scratch", identity_macros); + EXPECT_FALSE(s[ContentAddressedSetting::unsafe_remount_no_delay].value); + } + { + auto cfg = makeConfig("srv11"); + ContentAddressedSettings s; + s.loadFromConfig(*cfg, "disk", "/data", "/data/scratch", identity_macros); + EXPECT_TRUE(s[ContentAddressedSetting::unsafe_remount_no_delay].value); + } +} diff --git a/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml b/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml index 997e9a217631..9aa72e53ce36 100644 --- a/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml +++ b/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml @@ -19,6 +19,15 @@ clickhouse clickhouse false + + 1000 + 200 + 50 + 50 diff --git a/tests/integration/test_cas_mount_renewal_retry/configs/unsafe_remount.xml b/tests/integration/test_cas_mount_renewal_retry/configs/unsafe_remount.xml new file mode 100644 index 000000000000..9d2fa4c0f306 --- /dev/null +++ b/tests/integration/test_cas_mount_renewal_retry/configs/unsafe_remount.xml @@ -0,0 +1,12 @@ + + + + + + 1 + + + + diff --git a/tests/integration/test_cas_mount_renewal_retry/test.py b/tests/integration/test_cas_mount_renewal_retry/test.py index 73b0f9008436..114908ae8ae3 100644 --- a/tests/integration/test_cas_mount_renewal_retry/test.py +++ b/tests/integration/test_cas_mount_renewal_retry/test.py @@ -238,8 +238,13 @@ def test_transient_mount_renewal_retries_without_remount(start_cluster): ) def recovered_snapshot(): - mount = _mount_snapshot(node) + # Read the counter before the mount row: `CASMountRenewalRecovered` is incremented as soon as + # the renewal decides its outcome, strictly before the mount row's `renewal_sequence` (and the + # matching cas_log row) is updated to the new sequence. With the shortened renewal period a + # background (fault-free) renewal can land between the two reads; reading counters first makes + # the subsequent mount read very unlikely to still observe the pre-recovery sequence. counters = _profile_events(node) + mount = _mount_snapshot(node) if ( mount["sequence"] > mount_before["sequence"] and counters["CASMountRenewalRecovered"] @@ -254,19 +259,23 @@ def recovered_snapshot(): body_after, token_after = _read_mount_object() mount_body = _decode_mount(body_after) delta = _event_delta(counters_before, counters_after) - sequence = mount_after["sequence"] # The engine paces the reissues inside one renewal; what the log records is the renewal's - # outcome, and the attempt count on that row is what says a retry happened. + # outcome, and the attempt count on that row is what says a retry happened. Look the row up by + # outcome rather than by a snapshot-derived sequence (see _renewal_log_rows): with the shortened + # renewal period, background renewals can advance `system.cas_mounts` past the exact sequence this + # recovery landed on before either of these two reads gets to it. rows = _wait_until( lambda: ( found if any(row[0] == "recovered" for row in found) else None ) - if (found := _renewal_log_rows(node, since, sequence)) + if (found := _renewal_log_rows(node, since)) else None, timeout=20, ) + recovered = next(row for row in rows if row[0] == "recovered") + sequence = int(recovered[1]) assert delta["CASMountRenewalAttempts"] > 1, delta assert delta["CASMountRenewalRetries"] > 0, delta @@ -278,14 +287,14 @@ def recovered_snapshot(): assert mount_after["state"] == "live", mount_after assert mount_after["lifecycle"] == "live", mount_after assert mount_after["gc_fenced"] == 0, mount_after - assert int(mount_body["seq"]) == sequence + # >= rather than == : body_after may reflect a later, unrelated background renewal that landed + # after the one this test is verifying. + assert int(mount_body["seq"]) >= sequence assert token_after != token_before assert stats["faults"] == 1, stats assert stats["by_mode"].get("503") == 1, stats print("targeted request count (transient renewal): {}".format(stats["faults"]), flush=True) - recovered = next(row for row in rows if row[0] == "recovered") - assert recovered[1] == str(sequence), rows assert int(recovered[3]) > 1, rows assert recovered[4] == "committed_after_retry", rows @@ -318,9 +327,32 @@ def test_landed_response_lost_adopts_exact_mount_write(start_cluster): }, ) + # The proxy records the dropped-after-forward request (and the upstream_etag the real PUT landed + # under) as soon as it happens -- the physical write itself already reached the object store; only + # the response back to ClickHouse was dropped. That is well before ClickHouse's own request notices + # the lost response and resolves it by re-reading. With the renewal period this short, a plain + # "read the object once resolution is confirmed" can just as easily observe a LATER, unrelated + # background renewal that started immediately after this one resolved (see the mount_before/after + # sequence race this replaced). Wait for the proxy's own record first, then poll the object for + # its exact upstream_etag, so body_after is unambiguously the write this test is about. + def dropped_record(): + found_stats = _control(control_url, "/stats") + found_records = found_stats["drop_after_forward"] + return (found_stats, found_records[0]) if len(found_records) == 1 else None + + stats, record = _wait_until(dropped_record) + target_etag = record["upstream_etag"].strip('"') + + def matching_object(): + body, token = _read_mount_object() + return (body, token) if token == target_etag else None + + body_after, token_after = _wait_until(matching_object) + mount_body = _decode_mount(body_after) + def resolved_snapshot(): - mount = _mount_snapshot(node) counters = _profile_events(node) + mount = _mount_snapshot(node) if ( mount["sequence"] > mount_before["sequence"] and counters["CASMountRenewalResolved"] @@ -333,37 +365,40 @@ def resolved_snapshot(): mount_after, counters_after = _wait_until(resolved_snapshot) _control(control_url, "/config", {"rate": 0.0}) - stats = _control(control_url, "/stats") - body_after, token_after = _read_mount_object() - mount_body = _decode_mount(body_after) delta = _event_delta(counters_before, counters_after) - sequence = mount_after["sequence"] + # Look the recovered row up by outcome/classification rather than by a snapshot-derived sequence + # (see _renewal_log_rows): background renewals can advance `system.cas_mounts` past the exact + # sequence this recovery landed on before either of these reads gets to it. rows = _wait_until( lambda: ( found if any(row[0] == "recovered" and row[4] == "committed_by_read" for row in found) else None ) - if (found := _renewal_log_rows(node, since, sequence)) + if (found := _renewal_log_rows(node, since)) else None, timeout=20, ) + recovered = next(row for row in rows if row[0] == "recovered" and row[4] == "committed_by_read") + sequence = int(recovered[1]) - records = stats["drop_after_forward"] assert stats["faults"] == 1, stats assert stats["by_mode"].get("drop_after_forward") == 1, stats - assert len(records) == 1, records - record = records[0] assert record["method"] == "PUT", record assert record["path"].split("?", 1)[0] == MOUNT_REQUEST_PATH, record assert 200 <= record["upstream_status"] < 300, record assert record["request_body_sha256"] == hashlib.sha256(body_after).hexdigest(), record - assert record["upstream_etag"].strip('"') == token_after, record assert body_after != body_before assert token_after != token_before - assert int(mount_body["seq"]) == sequence - - assert delta["CASMountRenewalAttempts"] == 1, delta + # >= rather than == : a later, unrelated background renewal may have advanced the mount object + # again between the capture above and this read of the confirmed sequence from the log. + assert int(mount_body["seq"]) >= sequence + + # >= rather than == : with the shortened renewal period, an unrelated fault-free background + # renewal can complete (and count its own single attempt) right before or after this one, in the + # gap between configuring the fault and observing this specific renewal's resolution. Resolved and + # Recovered stay exact -- only a lost-response renewal like this one increments them. + assert delta["CASMountRenewalAttempts"] >= 1, delta assert delta["CASMountRenewalRetries"] == 0, delta assert delta["CASMountRenewalResolved"] == 1, delta assert delta["CASMountRenewalRecovered"] == 1, delta @@ -375,8 +410,6 @@ def resolved_snapshot(): assert mount_after["lifecycle"] == "live", mount_after assert mount_after["gc_fenced"] == 0, mount_after - recovered = next(row for row in rows if row[0] == "recovered") - assert recovered[1] == str(sequence), rows assert recovered[2] and mount_body["write_attempt_id"].startswith(recovered[2]), rows assert recovered[3] == "1", rows assert recovered[4] == "committed_by_read", rows From 5882d6212a81cc515d2e8aa070ab5d07e1b9948d Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 7 Sep 2026 07:26:41 +0200 Subject: [PATCH 35/81] cas: recommend a longer S3 keep-alive on every CAS disk, in docs and test configs Problem (CI, `cas_selects` regression on PR #2300, root cause and fourth fix): the data-plane read load created a new connection to the store every hundred requests -- the generic default `http_keep_alive_max_requests = 100` is the connection's whole lifetime under CAS's control-plane request rate, not a headroom margin -- so `TIME_WAIT` sockets exhausted the ephemeral port range (`EADDRNOTAVAIL`) and the mount lease's control-plane requests could not connect at all. The two S3 disk settings that fix it already exist, so this is configuration and documentation, not code: every S3-backed CAS disk should carry `http_keep_alive_timeout = 30` and `http_keep_alive_max_requests = 10000`. `docs/en/antalya/cas/configuration.md` gets a recommendation with an example disk snippet and the reason, and every in-repo S3-backed CAS disk configuration (the stateless CAS storage policy, the CAS integration test modules) sets both values, so the CI lanes run with them. On a three-node stand the two values cut connection creation and `TIME_WAIT` by two orders of magnitude at the same request rate; the churn was attributed with temporary connection-pool reason counters during that spike, which are not part of this change. The `cas_selects` regression stand lives in the clickhouse-regression repository and needs the same two lines in its disk configuration. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2300&sha=67d337fdf1c2c145a8ee0b1a85d7df2b59f8440e&name_0=PR PR: https://github.com/Altinity/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- docs/en/antalya/cas/configuration.md | 38 ++++++++++++++++++- docs/en/antalya/cas/operations/migration.md | 6 ++- docs/en/antalya/cas/quick-start.md | 6 ++- docs/en/operations/storing-data.md | 6 ++- ...orage_policy_for_merge_tree_by_default.xml | 6 +++ .../configs/storage_conf.xml | 4 ++ .../configs/storage_conf.xml | 2 + .../configs/storage_conf.xml | 2 + .../test_cas_gc_s3/configs/storage_conf.xml | 2 + .../configs/storage_conf.xml | 2 + .../test_cas_gcs/configs/config.xml | 4 ++ tests/integration/test_cas_gcs/test.py | 5 ++- .../configs/storage_conf.xml | 2 + .../configs/storage_conf.xml | 2 + .../configs/storage_conf.xml | 2 + .../configs/storage_conf.xml | 2 + .../configs/unsafe_remount.xml | 2 + .../configs/storage_conf.xml | 4 ++ .../configs/storage_conf.xml | 2 + .../configs/storage_conf_other_pool.xml | 2 + .../test_cas_s3/configs/storage_conf.xml | 4 +- .../configs/storage_conf.xml | 2 + tests/integration/test_gcs_live/test.py | 2 + 23 files changed, 102 insertions(+), 7 deletions(-) diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 436cf143b267..40e248146684 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -15,7 +15,9 @@ A `CAS` disk is an `object_storage` disk with `metadata_type` set to `cas` and a `cas_server_root_id`. The recommended shape layers a `type=cache` disk in front of it — the local filesystem cache absorbs repeated reads of the same blob, while the `CAS` disk underneath stays the single source of truth the pool's other members and GC also read from. The storage policy references -the **cached** disk, not the raw `CAS` disk directly: +the **cached** disk, not the raw `CAS` disk directly. `http_keep_alive_timeout` and +`http_keep_alive_max_requests` are set here for the reason explained under +[recommended keep-alive settings](#recommended-keep-alive-settings): ```xml @@ -29,6 +31,8 @@ the **cached** disk, not the raw `CAS` disk directly: https://bucket.s3.amazonaws.com/cas/ ... ... + 30 + 10000 cache @@ -134,6 +138,38 @@ The `expires_at_ms` stamped into the mount object is a writer-stamped diagnostic authorizes a reclaim or a GC fence-out. Local fencing is derived instead from the confirmed request's pre-I/O `CLOCK_BOOTTIME` anchor plus the TTL. +## Recommended keep-alive settings {#recommended-keep-alive-settings} + +On a `CAS` disk, set `http_keep_alive_timeout` to `30` and `http_keep_alive_max_requests` to `10000`, +alongside the disk's other settings: + +```xml + + + + + object_storage + s3 + cas + {replica} + https://example-bucket.s3.amazonaws.com/cas/ + ... + ... + 30 + 10000 + + + + +``` + +The generic S3 default, `http_keep_alive_max_requests = 100`, is the whole lifetime of a +connection under `CAS`'s control-plane request rate rather than a headroom margin: every ~100 +requests, a connection is torn down and recreated, and its local port then cycles through +`TIME_WAIT`. Under sustained load this churn exhausts the ephemeral port range +(`EADDRNOTAVAIL`) and starves the mount-lease renewal request. Raising the two settings above +removes that churn, with no measured cost. + ## Advanced GC pacing settings {#advanced-gc-pacing-settings} These settings bound individual phases of a `GC` round. The first two accept any `UInt64` value; diff --git a/docs/en/antalya/cas/operations/migration.md b/docs/en/antalya/cas/operations/migration.md index df51e79144a3..e8258b96d61d 100644 --- a/docs/en/antalya/cas/operations/migration.md +++ b/docs/en/antalya/cas/operations/migration.md @@ -19,7 +19,9 @@ disk and its data are untouched until a partition is explicitly moved. A storage policy can carry both an ordinary disk and a `CAS` disk as separate volumes. `ALTER TABLE ... MOVE PARTITION ... TO DISK` then moves data between them without an `INSERT`/`DROP` cycle. As on the [configuration](/antalya/cas/configuration#disk-config) page, the recommended shape layers a -`type=cache` disk over the `CAS` disk, and the policy's volume references the **cached** disk name: +`type=cache` disk over the `CAS` disk, and the policy's volume references the **cached** disk name. +`http_keep_alive_timeout` and `http_keep_alive_max_requests` are set here for the reason explained +under [recommended keep-alive settings](/antalya/cas/configuration#recommended-keep-alive-settings): ```xml @@ -37,6 +39,8 @@ the [configuration](/antalya/cas/configuration#disk-config) page, the recommende https://bucket.s3.amazonaws.com/cas/ ... ... + 30 + 10000 cache diff --git a/docs/en/antalya/cas/quick-start.md b/docs/en/antalya/cas/quick-start.md index a54d9345fbb0..0491f6972e8b 100644 --- a/docs/en/antalya/cas/quick-start.md +++ b/docs/en/antalya/cas/quick-start.md @@ -56,7 +56,9 @@ literal string, as above, is enough; on a replicated cluster where every replica disk's `endpoint` already uses, giving each replica a distinct subtree from one template. **S3 endpoint variant.** Swap `object_storage_type` to `s3` and add the usual object-storage -connection keys; nothing else in this config changes: +connection keys, plus `http_keep_alive_timeout` and `http_keep_alive_max_requests` — see +[recommended keep-alive settings](/antalya/cas/configuration#recommended-keep-alive-settings) for +why; nothing else in this config changes: ```xml @@ -67,6 +69,8 @@ connection keys; nothing else in this config changes: https://bucket.s3.amazonaws.com/cas/ ... ... + 30 + 10000 ``` diff --git a/docs/en/operations/storing-data.md b/docs/en/operations/storing-data.md index 099a3d0daa10..a3503bc1eb2c 100644 --- a/docs/en/operations/storing-data.md +++ b/docs/en/operations/storing-data.md @@ -467,7 +467,9 @@ and the [`system.cas_gc_log`](/operations/system-tables/cas_gc_log), [content-addressed storage documentation](/antalya/cas) for the architecture, operations runbooks, and a live-validated quick start. -Configuration: +Configuration: `http_keep_alive_timeout` and `http_keep_alive_max_requests` are set here for the +reason explained under +[recommended keep-alive settings](/antalya/cas/configuration#recommended-keep-alive-settings). ```xml @@ -476,6 +478,8 @@ Configuration: cas https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/ 1 + 30 + 10000 server-{replica} disks/s3_cas/cas_scratch/ diff --git a/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml b/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml index 204fc4e4aed3..e8ac8c5dbe90 100644 --- a/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml +++ b/tests/config/config.d/cas_s3_storage_policy_for_merge_tree_by_default.xml @@ -5,6 +5,12 @@ object_storage s3 cas + + 30 + 10000 stateless-ca-s3 diff --git a/tests/integration/test_cas_drop_pool_member/configs/storage_conf.xml b/tests/integration/test_cas_drop_pool_member/configs/storage_conf.xml index ca865ce6e8d2..7cde804e2696 100644 --- a/tests/integration/test_cas_drop_pool_member/configs/storage_conf.xml +++ b/tests/integration/test_cas_drop_pool_member/configs/storage_conf.xml @@ -9,6 +9,8 @@ object_storage s3 cas + 30 + 10000 http://rustfs1:11121/test/cas_dpm_data/ clickhouse clickhouse @@ -26,6 +28,8 @@ object_storage s3 cas + 30 + 10000 http://rustfs1:11121/test/cas_dpm_data/ clickhouse clickhouse diff --git a/tests/integration/test_cas_file_cache/configs/storage_conf.xml b/tests/integration/test_cas_file_cache/configs/storage_conf.xml index b75531fc8b9b..ca4197e28e5f 100644 --- a/tests/integration/test_cas_file_cache/configs/storage_conf.xml +++ b/tests/integration/test_cas_file_cache/configs/storage_conf.xml @@ -7,6 +7,8 @@ object_storage s3 cas + 30 + 10000 itest-cas-file-cache http://rustfs1:11121/test/cas_cache_data/ clickhouse diff --git a/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml b/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml index 7324f93c273d..92a9d0ed3e64 100644 --- a/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml +++ b/tests/integration/test_cas_gc_bulk_delete/configs/storage_conf.xml @@ -5,6 +5,8 @@ object_storage s3 cas + 30 + 10000 itest-content-addressed-gc-s3 itest-content-addressed-gc-s3 __SERVER_ROOT_ID__ diff --git a/tests/integration/test_cas_insert_fault_recovery/configs/storage_conf.xml b/tests/integration/test_cas_insert_fault_recovery/configs/storage_conf.xml index 0149d398aa1c..4959b16fadff 100644 --- a/tests/integration/test_cas_insert_fault_recovery/configs/storage_conf.xml +++ b/tests/integration/test_cas_insert_fault_recovery/configs/storage_conf.xml @@ -5,6 +5,8 @@ object_storage s3 cas + 30 + 10000 diff --git a/tests/integration/test_cas_lazy_load_recovery/configs/storage_conf.xml b/tests/integration/test_cas_lazy_load_recovery/configs/storage_conf.xml index 0149d398aa1c..4959b16fadff 100644 --- a/tests/integration/test_cas_lazy_load_recovery/configs/storage_conf.xml +++ b/tests/integration/test_cas_lazy_load_recovery/configs/storage_conf.xml @@ -5,6 +5,8 @@ object_storage s3 cas + 30 + 10000 diff --git a/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml b/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml index 9aa72e53ce36..11ab2a4c08b6 100644 --- a/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml +++ b/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml @@ -11,6 +11,8 @@ object_storage s3 cas + 30 + 10000 itest-cas-renewal 1 + 30 + 10000 diff --git a/tests/integration/test_cas_ref_snaplog/configs/storage_conf.xml b/tests/integration/test_cas_ref_snaplog/configs/storage_conf.xml index 74a0ba8de29c..4ccd3a04688d 100644 --- a/tests/integration/test_cas_ref_snaplog/configs/storage_conf.xml +++ b/tests/integration/test_cas_ref_snaplog/configs/storage_conf.xml @@ -8,6 +8,8 @@ object_storage s3 cas + 30 + 10000 itest-ref-snaplog http://rustfs1:11121/test/cas_snaplog_data/ clickhouse @@ -22,6 +24,8 @@ object_storage s3 cas + 30 + 10000 itest-ref-snaplog http://rustfs1:11121/test/cas_snaplog_data/ clickhouse diff --git a/tests/integration/test_cas_replicated_relink/configs/storage_conf.xml b/tests/integration/test_cas_replicated_relink/configs/storage_conf.xml index 4513d345a2fb..67c3d444d9be 100644 --- a/tests/integration/test_cas_replicated_relink/configs/storage_conf.xml +++ b/tests/integration/test_cas_replicated_relink/configs/storage_conf.xml @@ -5,6 +5,8 @@ object_storage s3 cas + 30 + 10000 - 60 - 100 + 30 + 10000 5000 7 3 diff --git a/tests/integration/test_cas_shared_pool/configs/storage_conf.xml b/tests/integration/test_cas_shared_pool/configs/storage_conf.xml index a29a03741ef9..8638d5530ef5 100644 --- a/tests/integration/test_cas_shared_pool/configs/storage_conf.xml +++ b/tests/integration/test_cas_shared_pool/configs/storage_conf.xml @@ -5,6 +5,8 @@ object_storage s3 cas + 30 + 10000 30 10000 - 5000 + 1000 7 3
X-Cas-Test: 1
From 61c6caacf5b582a5c3ea36a74bf7f84a18c49c24 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Mon, 7 Sep 2026 17:49:53 +0200 Subject: [PATCH 43/81] cas: a sibling that already won namespace birth is Superseded, not a logical error CI evidence (PR #2300 run 3, https://github.com/Altinity/ClickHouse/pull/2300, regression suite tiered_storage_cas, server log 11:51:36): seven MergeTreeBackgroundExecutor movers made the first writes of a fresh table onto a CAS disk concurrently for part all_1_1_0. Every one ran CasRefLedger::resolveNamespaceLife, whose loop read "no entry" and called CasRefCatalog::createNamespace. The winner's three steps completed within ~120 ms, so a loser's own pre-check read inside createNamespace found the entry already Live and hit the LOGICAL_ERROR "already carries a catalog entry (state 'live')", aborting the server under DEBUG_OR_SANITIZER_BUILD. createNamespace's pre-check read is a snapshot taken AFTER the caller's own "no entry" read, so a sibling opener of the SAME namespace (concurrent threads of one server: parallel background movers, inserts, per-part FREEZE) can land anywhere in its own three-step sequence in the gap between those two reads -- Creating, Live and Removing are all reachable outcomes of that race, never a caller bug. This is the third catch-point of the same sibling-opener race already documented in this file (still-Creating in the pre-check, and a sibling's step 1 landing between the pre-check and createNamespaceStep1's own read); the pre-check's Live/Removing branch was the one spot still treating the race as a bug. Fix: report NamespaceCreationOutcome::Superseded for every state the pre-check observes, not only Creating, and drop the LOGICAL_ERROR throw. resolveNamespaceLife's loop already knows what to do with Superseded: it re-reads and dispatches from the fresh entry (Live is adopted directly, Removing is refused by the loop's own branch, a still-Creating entry resumes through reconcileStaleCreator + completeCreation). Tests (src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp, gtest_cas_ref_catalog_birth_wiring.cpp): converted the direct Live pre-check test from an expected LOGICAL_ERROR/death pair to an expected Superseded outcome, added the equivalent Removing case, and added a new setCreateNamespacePreCheckHookForTest (modelled on the existing step1 hook) to reproduce the exact CI shape through the production namespaceLife/resolveNamespaceLife path: a sibling's entire createNamespace call completes to Live inside the window right before the loser's own pre-check read, and the loser's namespaceLife call still returns the sibling's incarnation without throwing. Verification (failing-first): with the old throwing pre-check restored temporarily and only the new hook/test infrastructure added, all three new tests failed with the exact CI text ("... already carries a catalog entry (state 'live'/'removing')"). After the fix: release unit_tests_dbms --gtest_filter='CAS*' 2490 tests, 0 FAIL; ASan build_asan/src/unit_tests_dbms --gtest_filter='CAS*' 2494 tests, 0 FAIL. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit f9b7c0a7dedaca2d03ee914eb89fb6fdc9ce7185) --- .../ContentAddressed/Pool/CasRefCatalog.cpp | 56 ++++++++++++------- .../ContentAddressed/Pool/CasRefCatalog.h | 24 +++++--- .../tests/gtest_cas_ns_creation_lifecycle.cpp | 43 ++++++++------ .../gtest_cas_ref_catalog_birth_wiring.cpp | 44 +++++++++++++++ 4 files changed, 120 insertions(+), 47 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp index db3c3c61966b..d4e12fc65c97 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.cpp @@ -270,6 +270,13 @@ struct CatalogEntryAlreadyPresentMarker : std::exception {}; /// Empty (no-op) in production, mirroring every other `*_hook_for_test` in this tree. std::function create_namespace_step1_pre_read_hook_for_test; +/// Fires once, synchronously, right before `createNamespace`'s own pre-check read -- the window in +/// which a sibling opener of the SAME namespace can complete an entire birth (or begin a removal) that +/// this call's pre-check then observes. Lets a test land that interleaving deterministically instead of +/// relying on real thread scheduling. Empty (no-op) in production, mirroring every other +/// `*_hook_for_test` in this tree. +std::function create_namespace_pre_check_hook_for_test; + /// Step 1 of `createNamespace`, split out so it can recheck presence on EVERY catalog read this loop /// performs (the first one, and any `Conflict` retry's re-read), not only the snapshot-in-time read /// `createNamespace` itself already did before calling in. That single upfront read cannot see a @@ -646,33 +653,35 @@ CasRefCatalog::NamespaceCreationOutcome CasRefCatalog::createNamespace( const RootNamespace & ns, const CreatorFence & creator, const Retry & policy) { /// Read-first, per the Task 2 review's own note on `casAdmitEntry`: a namespace that already - /// carries an entry is THIS function's job to reject with a clear message, not `casAdmitEntry`'s - /// duplicate-namespace grammar refusal (which would report a `LOGICAL_ERROR` about canonical order - /// -- true, but useless to a caller trying to understand why its create failed). A concurrent + /// carries an entry is THIS function's job to notice and report `Superseded` for, not + /// `casAdmitEntry`'s duplicate-namespace grammar refusal (which would report a `LOGICAL_ERROR` about + /// canonical order -- true, but useless to a caller whose create merely lost a race). A concurrent /// insert of the SAME namespace between this read and step 1 is still caught -- `casAdmitEntry`'s /// own grammar check is the backstop, not the only check. + if (create_namespace_pre_check_hook_for_test) + { + std::function hook_to_run; + std::swap(hook_to_run, create_namespace_pre_check_hook_for_test); + hook_to_run(); + } const Snapshot snap = read(op, layout, policy); const auto existing = findEntry(snap.catalog, ns); if (existing != snap.catalog.entries.end()) { - /// `Creating` is not this function's problem to solve (the class-level doc above says so) -- - /// it is exactly the race `resolveNamespaceLife`'s own loop is built to absorb: sibling openers - /// of the SAME namespace (e.g. concurrent per-part freeze threads of one query, which share one - /// mount's fence) can all observe "no entry" before any of them lands step 1, then race into - /// this call. Reporting `Superseded` sends the loser back through the loop, where it re-reads - /// and takes the documented resume path (its own fence: `completeCreation`; a foreign one: - /// `reconcileStaleCreator`) instead of aborting the server for an outcome the design already - /// names and handles. `Live`/`Removing` stay a `LOGICAL_ERROR`: `namespaceLife`'s caller filters - /// `Live` before ever reaching here and refuses `Removing` outright, so seeing either here means - /// a caller bypassed that dispatch, not a race. - if (existing->state == NsState::Creating) - return NamespaceCreationOutcome::Superseded; - throw Exception(ErrorCodes::LOGICAL_ERROR, - "CasRefCatalog::createNamespace: namespace '{}' already carries a catalog entry (state " - "'{}') -- a stalled Creating entry is resumed through reconcileStaleCreator + " - "completeCreation, never a fresh createNamespace call; an existing Live or Removing " - "namespace must complete its current lifecycle before a fresh creation can be admitted", - ns.string(), nsStateToWord(existing->state)); + /// This read is a snapshot taken AFTER the caller's own "no entry" read (`resolveNamespaceLife`'s + /// loop, or any other dispatcher that only reaches `createNamespace` once it has seen nothing to + /// adopt). A sibling opener of the SAME namespace -- concurrent threads of one server: parallel + /// background movers, inserts, or per-part `FREEZE` -- can land anywhere in its own three-step + /// sequence in the gap between those two reads, so EVERY state observed here is a race outcome, + /// never a caller bug: `Creating` (a sibling landed step 1 only), `Live` (a sibling completed all + /// three steps and already won birth), and `Removing` (a concurrent drop) are all reported + /// `Superseded`, sending the loser back through its own resume loop rather than aborting the + /// server for an outcome the design already names and handles. There the loop's fresh re-read + /// tells the loser what actually happened: `Live` is adopted directly, `Removing` is refused by + /// the loop's own `Removing` branch, and a still-`Creating` entry resumes through + /// `reconcileStaleCreator` + `completeCreation` (or, if it is this caller's own fence, straight + /// through `completeCreation`). + return NamespaceCreationOutcome::Superseded; } const CatalogEntry entry{.ns = ns, .state = NsState::Creating, @@ -706,6 +715,11 @@ void CasRefCatalog::setCreateNamespaceStep1PreReadHookForTest(std::function hook) +{ + create_namespace_pre_check_hook_for_test = std::move(hook); +} + CasRefCatalog::ReconcileCreatorOutcome CasRefCatalog::reconcileStaleCreator( CasOperation & op, const Layout & layout, const CatalogEntry & observed, const CreatorFence & new_creator, const std::function & is_creator_fence_terminal, const Retry & policy) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h index 2926a7f33d6c..b1ed539def66 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasRefCatalog.h @@ -235,15 +235,14 @@ class CasRefCatalog /// Creating, incarnation, creator}`), then steps 2+3 via `completeCreation` below. /// /// This function reads the catalog FIRST rather than handing `casAdmitEntry` a doomed insert and - /// letting its own grammar check report a confusing duplicate-namespace message: a namespace - /// `entry.ns` already carries an entry is a bug in THIS caller, not in `casAdmitEntry`, which is - /// why it is checked here first. A namespace already - /// `Creating` is not this function's problem to solve -- that is exactly what `reconcileStaleCreator` - /// + `completeCreation` are for, so this reports `Superseded` (never `LOGICAL_ERROR`) and sends the - /// caller back through its own resume loop: sibling openers of the same namespace that all observed - /// "no entry" before any of them landed step 1 race in here exactly this way. A namespace already - /// `Live`/`Removing` IS a caller bug (recreating an existing name is removal's business, not - /// creation's) and still throws `LOGICAL_ERROR` naming the observed state. + /// letting its own grammar check report a confusing duplicate-namespace message. That read is a + /// snapshot taken AFTER the caller's own "no entry" read, so a sibling opener of the SAME namespace + /// (concurrent threads of one server: parallel background movers, inserts, `FREEZE`) can have landed + /// anywhere in its own three-step sequence in between -- ANY state observed here (`Creating`, `Live`, + /// `Removing`) is that race, never a caller bug, and is reported `Superseded` uniformly, sending the + /// caller back through its own resume loop (`resolveNamespaceLife`'s loop re-reads and dispatches: + /// `Live` is adopted, `Removing` is refused there, a still-`Creating` entry resumes through + /// `reconcileStaleCreator` + `completeCreation`). static NamespaceCreationOutcome createNamespace( CasOperation & op, const Layout & layout, uint64_t gc_shards, const RootNamespace & ns, const CreatorFence & creator, @@ -257,6 +256,13 @@ class CasRefCatalog /// `CasRefCatalog` itself carries no state. static void setCreateNamespaceStep1PreReadHookForTest(std::function hook); + /// Fires once, synchronously, right before `createNamespace`'s own pre-check read -- the exact + /// window a sibling opener of the same namespace can complete an entire birth (or begin a removal) + /// in, for a test to drive that interleaving deterministically instead of relying on real thread + /// scheduling. Empty (no-op) hook in production; a stateless class-scope hook (rather than an + /// instance member) because `CasRefCatalog` itself carries no state. + static void setCreateNamespacePreCheckHookForTest(std::function hook); + /// Steps 2 (`_ckpt` publish) + 3 (`Creating -> Live` CAS) alone, given an entry the caller already /// owns as `observed` -- either the entry `createNamespace`'s own step 1 just inserted, or one a /// caller just reconciled onto itself via `reconcileStaleCreator`. Exposed separately (rather than diff --git a/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp b/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp index e86d55074bd8..26ea174533f1 100644 --- a/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp +++ b/src/Disks/tests/gtest_cas_ns_creation_lifecycle.cpp @@ -146,12 +146,13 @@ TEST(CASNsCreationLifecycle, HappyPathReachesLiveWithADurableCkptAndAStableIncar } /// --------------------------------------------------------------------------------------------- -/// `createNamespace` refuses a namespace that already has an entry (Task 2 review's own note: this -/// is Task 3's job, not `casAdmitEntry`'s duplicate-namespace grammar refusal). +/// `createNamespace`'s pre-check reports `Superseded` for a namespace that already has an entry, in +/// EVERY state -- not a caller bug, but a sibling opener of the same namespace (CI PR#2300 run 3, +/// `tiered_storage_cas`: seven concurrent `MergeTreeBackgroundExecutor` movers) that landed somewhere in +/// its own three-step sequence between the caller's "no entry" read and this pre-check's read. /// --------------------------------------------------------------------------------------------- -#ifndef DEBUG_OR_SANITIZER_BUILD -TEST(CASNsCreationLifecycle, CreateNamespaceRejectsAnAlreadyExistingEntry) +TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsLiveEntryReportsSupersededNotAbort) { auto backend = initializedCatalogBackend(); CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); @@ -162,15 +163,18 @@ TEST(CASNsCreationLifecycle, CreateNamespaceRejectsAnAlreadyExistingEntry) ASSERT_EQ(CasRefCatalog::createNamespace(op, layout, 1, ns, creator), CasRefCatalog::NamespaceCreationOutcome::Live); - DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::LOGICAL_ERROR, [&] - { - CasRefCatalog::createNamespace(op, layout, 1, ns, creatorFence("srv2", 2)); - }); + EXPECT_EQ(CasRefCatalog::createNamespace(op, layout, 1, ns, creatorFence("srv2", 2)), + CasRefCatalog::NamespaceCreationOutcome::Superseded); + + /// The refused call left the winner's `Live` entry exactly as it was. + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); + const CatalogEntry * entry = findEntryForTest(snap.catalog, ns); + ASSERT_NE(entry, nullptr); + EXPECT_EQ(entry->state, NsState::Live); + EXPECT_EQ(entry->creator, std::nullopt); } -#endif -#if defined(DEBUG_OR_SANITIZER_BUILD) -TEST(CASNsCreationLifecycleDeathTest, CreateNamespaceRejectsAnAlreadyExistingEntryAborts) +TEST(CASNsCreationLifecycle, CreateNamespaceRacingASiblingsRemovingEntryReportsSupersededNotAbort) { auto backend = initializedCatalogBackend(); CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); @@ -180,14 +184,19 @@ TEST(CASNsCreationLifecycleDeathTest, CreateNamespaceRejectsAnAlreadyExistingEnt const CreatorFence creator = creatorFence("srv1", 1); ASSERT_EQ(CasRefCatalog::createNamespace(op, layout, 1, ns, creator), CasRefCatalog::NamespaceCreationOutcome::Live); + ASSERT_EQ(CasRefCatalog::beginRemoving(op, layout, *findEntryForTest(CasRefCatalog::read(op, layout).catalog, ns), + /*removal_started_round=*/1), + CasRefCatalog::BeginRemovingOutcome::Transitioned); - EXPECT_DEATH( - { - CasRefCatalog::createNamespace(op, layout, 1, ns, creatorFence("srv2", 2)); - }, - "already carries a catalog entry"); + EXPECT_EQ(CasRefCatalog::createNamespace(op, layout, 1, ns, creatorFence("srv2", 2)), + CasRefCatalog::NamespaceCreationOutcome::Superseded); + + /// The refused call left the concurrent drop's `Removing` entry exactly as it was. + const CasRefCatalog::Snapshot snap = CasRefCatalog::read(op, layout); + const CatalogEntry * entry = findEntryForTest(snap.catalog, ns); + ASSERT_NE(entry, nullptr); + EXPECT_EQ(entry->state, NsState::Removing); } -#endif /// --------------------------------------------------------------------------------------------- /// `Creating` forbids publication diff --git a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp index 18f279620f62..2c094fa122af 100644 --- a/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp +++ b/src/Disks/tests/gtest_cas_ref_catalog_birth_wiring.cpp @@ -366,6 +366,50 @@ TEST(CASRefCatalogBirthWiring, AnExistingLiveEntryIsAdoptedRatherThanReminted) EXPECT_TRUE(op.head(layout.refLogKey(life, id), Retry::standard()).has_value()); } +/// Regression (CI PR#2073, `tiered_storage_cas`, part `all_1_1_0` of a fresh table): seven concurrent +/// `MergeTreeBackgroundExecutor` movers all reached `resolveNamespaceLife`'s "no entry" read for the +/// SAME namespace before any of them landed a row. The winner's `createNamespace` ran its full three +/// steps to `Live` inside the WINDOW between the loop's own "no entry" read and the loser's own +/// `createNamespace` pre-check read -- so the loser's pre-check itself observed `Live`, not "no entry". +/// That must be adopted through the loop's normal re-read, never abort the server. +TEST(CASRefCatalogBirthWiring, ASiblingsFullCreateInsideCreateNamespacesOwnPreCheckWindowIsAdoptedNotAbort) +{ + auto backend = std::make_shared(); + CasRequests requests = DB::Cas::tests::openRequestsForTest(backend); + CasOperation sibling_op = requests.admit(); + auto store = openPoolForBirthTest(backend, "loser-server"); + const Layout & layout = store->layout(); + const RootNamespace ns{"srv1/precheck_race"}; + const CreatorFence sibling_fence{.server_root_id = "sibling-server", .writer_epoch = 1, .fence_generation = 1}; + + /// Fires once, inside the LOSER's own `store->namespaceLife` -> `resolveNamespaceLife` -> + /// `createNamespace` call, right before that call's pre-check read -- i.e. AFTER + /// `resolveNamespaceLife`'s own loop already observed no entry. Runs a sibling's entire + /// `createNamespace` to completion in that window, so the loser's own pre-check read is the one + /// that observes the sibling's `Live` row. + CasRefCatalog::setCreateNamespacePreCheckHookForTest([&] + { + const auto sibling_outcome = CasRefCatalog::createNamespace(sibling_op, layout, 1, ns, sibling_fence); + ASSERT_EQ(sibling_outcome, CasRefCatalog::NamespaceCreationOutcome::Live); + }); + + std::optional life; + EXPECT_NO_THROW(life = store->namespaceLife(ns)); + + /// The read result must outlive the returned pointer -- findEntry points into its entries. + const auto snap = CasRefCatalog::read(sibling_op, layout); + size_t rows_for_ns = 0; + for (const CatalogEntry & e : snap.catalog.entries) + if (e.ns.string() == ns.string()) + ++rows_for_ns; + EXPECT_EQ(rows_for_ns, 1u) << "the loser's refused pre-check left no trace of its own"; + const CatalogEntry * entry = findEntry(snap.catalog, ns); + ASSERT_NE(entry, nullptr); + EXPECT_EQ(entry->state, NsState::Live); + ASSERT_TRUE(life.has_value()); + EXPECT_EQ(*life, NamespaceLifeId::fromCatalogEntry(ns, entry->incarnation)) << "the sibling's incarnation, adopted"; +} + /// OBLIGATION 3, pinned through the PRODUCTION path: a `Creating` entry left by a DIFFERENT, still-live /// (or at least not provably dead) actor refuses every append -- no test-only seam, no direct call to /// `resolveNamespaceLife`/`reconcileStaleCreator`, just an ordinary `appendRefOps`. From 8be816a130dcb0c845db7996ff6fabdbebe90e90 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 03:03:08 +0200 Subject: [PATCH 44/81] s3: the read identity check fires only when bytes of two incarnations were delivered `ReadBufferFromS3::initialize` set `response_identity_changed` whenever a reissued GET answered with a different ETag than the first response, even when the first attempt failed before delivering a single byte and the reissue read the new incarnation whole. `S3ObjectStorage::readSmallObjectAndGetObjectMetadata` then threw CANNOT_READ_ALL_DATA; the Iceberg `version-hint.text` read (rewritten on every commit, no try/catch, fails the INSERT) and Paimon are callers outside CAS, so the false positive was a regression there. The flag now means what its consumers need: bytes from two different object incarnations reached the consumer. The comparison happens at delivery time in `nextImpl`: the baseline is the ETag of the last response that actually delivered bytes, it is compared only when a response delivers its first byte, and any number of empty failed attempts in between (with any ETags) are transparent. The delivery-time bookkeeping is allocation-free and `next_result` is set only after it, so a throwing operation there can no longer leave the retry loop with a null `impl`. Explicit repositioning (`seek` that reissues, `setReadUntilPosition`, `setReadUntilEnd`) starts a fresh identity; an in-buffer seek keeps it. Tests (failing-first against the old code): a failed attempt that delivered no bytes followed by a different ETag is accepted; bytes delivered then a different ETag is flagged; the three-response sequences A-bytes/A-empty/B-bytes (flagged) and A-bytes/B-empty/A-bytes (accepted); each reposition path; the external-buffer (prefetch) path; a partial fill that exposed no bytes. The CAS upstream-slice test that asserted the old semantics on a failure that delivered nothing now delivers two bytes first, and its sibling pins the accepted case. (squashed from ba274b2dd5b, 533f9f86edb, a2d672edf42, 67d4becabee) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_upstream_slice.cpp | 28 +- src/IO/ReadBufferFromS3.cpp | 48 ++- src/IO/ReadBufferFromS3.h | 27 +- src/IO/tests/gtest_readbuffer_s3.cpp | 412 +++++++++++++++++++ 4 files changed, 502 insertions(+), 13 deletions(-) diff --git a/src/Disks/tests/gtest_cas_upstream_slice.cpp b/src/Disks/tests/gtest_cas_upstream_slice.cpp index 99d8ac0b1a4e..367f3b2a915a 100644 --- a/src/Disks/tests/gtest_cas_upstream_slice.cpp +++ b/src/Disks/tests/gtest_cas_upstream_slice.cpp @@ -524,8 +524,13 @@ TEST(CASUpstreamSlice, NativeConditionalReadSettingMarksTheGetRequest) EXPECT_FALSE(plain_client->nativeConditionalMarks().at(0)); } -/// The buffer's own retry loop can straddle a replacement of the object: the first response is the old -/// incarnation, the reissue the new one. The bytes handed back are then from neither one alone. +/// The buffer's own retry loop can straddle a replacement of the object: the first response delivers +/// some of the old incarnation's bytes to the consumer before its stream breaks mid-body, and the +/// reissue answers with a different ETag. The bytes handed back are then from neither incarnation +/// alone. `buffer_size` is pinned to 2 so the first fill (of "AAAA"'s 4 bytes) completes and is +/// exposed to the consumer before the second fill hits the scripted mid-body failure - with the +/// default (much larger) buffer, that failure happens inside the very first fill, before any byte of +/// "e1" ever reaches the consumer, which is the "nothing to mix with" case covered below instead. TEST(CASUpstreamSlice, ReadSmallObjectThrowsWhenAReissueAnswersWithADifferentETag) { (void)getContext(); @@ -536,6 +541,7 @@ TEST(CASUpstreamSlice, ReadSmallObjectThrowsWhenAReissueAnswersWithADifferentETa DB::ReadSettings read_settings; read_settings.object_storage_request_mode = DB::ObjectStorageRequestMode::NativeConditional; + read_settings.remote_fs_settings.buffer_size = 2; /// Default profile here: the buffer's own multi-attempt loop is what straddles the replacement. expectThrowsCodeSaying( DB::ErrorCodes::CANNOT_READ_ALL_DATA, @@ -545,6 +551,24 @@ TEST(CASUpstreamSlice, ReadSmallObjectThrowsWhenAReissueAnswersWithADifferentETa EXPECT_EQ(client->getObjectCalls(), 2u); } +/// Scoped to an identity change that actually mixed bytes: with the default (large) buffer, "e1"'s +/// mid-body failure happens inside its very first fill attempt, before any byte crosses into the +/// consumer's buffer - so the reissue under a different ETag is an ordinary retry of a request that +/// never delivered anything, not a coherence problem, even though the ETag changed. +TEST(CASUpstreamSlice, ReadSmallObjectAcceptsAReissueThatDeliveredNoBytesEvenWithADifferentETag) +{ + (void)getContext(); + + ScriptedGetObjectClient * client = nullptr; + auto storage = makeScriptedS3ObjectStorage(client); + client->script({okStep("\"e1\"", "AAAA", /*fail_mid_body=*/true), okStep("\"e2\"", "BBBB")}); + + const auto result = storage->readSmallObjectAndGetObjectMetadata(DB::StoredObject("k"), DB::ReadSettings{}, 1 << 20); + EXPECT_EQ(result.data, "BBBB"); + EXPECT_EQ(result.metadata.etag, "\"e2\""); + EXPECT_EQ(client->getObjectCalls(), 2u); +} + /// Scoped to an identity CHANGE: a reissue is ordinary, and refusing every retried read would turn a /// dropped connection into a hard error. TEST(CASUpstreamSlice, ReadSmallObjectAcceptsAReissueThatAnswersWithTheSameETag) diff --git a/src/IO/ReadBufferFromS3.cpp b/src/IO/ReadBufferFromS3.cpp index fc1d5fe38d3f..e1f4bd15837f 100644 --- a/src/IO/ReadBufferFromS3.cpp +++ b/src/IO/ReadBufferFromS3.cpp @@ -209,7 +209,29 @@ bool ReadBufferFromS3::nextImpl() } /// Try to read a next portion of data. - next_result = impl->next(); + const bool delivered_more_data = impl->next(); + if (delivered_more_data && !pending_response_bytes_delivered) + { + /// This response just delivered its first byte: check it against whichever response + /// last delivered bytes, then it becomes the new baseline. A response that never + /// reaches this point (fails before delivering anything) never touches the baseline, + /// so any number of empty failed attempts in between are transparent to the check. + /// + /// Nothing here may throw: `last_delivering_response_etag = pending_response_etag` + /// used to be a copy, which can allocate and throw for a non-SSO ETag; if that throw + /// happened after `next_result` was already set to true, the catch block below resets + /// `impl` (since `processException` retries), but the loop's `!next_result` condition + /// is already false, so it exits with `impl` null while the code past the loop still + /// dereferences it. A `std::string` move is noexcept, so this block cannot throw; as a + /// second line of defense, `next_result` itself is set only once this block is done, so + /// even a future throwing addition here would leave the loop's retry invariant intact + /// instead of exiting with a dangling `impl`. + if (last_delivering_response_etag && *last_delivering_response_etag != pending_response_etag) + response_identity_changed = true; + last_delivering_response_etag = std::move(pending_response_etag); + pending_response_bytes_delivered = true; + } + next_result = delivered_more_data; break; } catch (...) @@ -448,6 +470,7 @@ off_t ReadBufferFromS3::seek(off_t offset_, int whence) if (!atEndOfRequestedRangeGuess()) ProfileEvents::increment(ProfileEvents::ReadBufferSeekCancelConnection); impl.reset(); + forgetResponseIdentityBaseline(); } } @@ -493,6 +516,7 @@ void ReadBufferFromS3::setReadUntilPosition(size_t position) offset = getPosition(); resetWorkingBuffer(); impl.reset(); + forgetResponseIdentityBaseline(); } read_until_position = position; } @@ -512,6 +536,7 @@ void ReadBufferFromS3::setReadUntilEnd() offset = getPosition(); resetWorkingBuffer(); impl.reset(); + forgetResponseIdentityBaseline(); } } } @@ -527,6 +552,12 @@ bool ReadBufferFromS3::atEndOfRequestedRangeGuess() return false; } +void ReadBufferFromS3::forgetResponseIdentityBaseline() +{ + last_delivering_response_etag.reset(); + pending_response_bytes_delivered = false; +} + std::unique_ptr ReadBufferFromS3::initialize(size_t attempt) { stop_reason = ""; @@ -545,13 +576,14 @@ std::unique_ptr ReadBufferFromS3::initialize( Stopwatch watch{CLOCK_MONOTONIC}; auto read_result = sendRequest(attempt, offset, right_offset); - /// Compared per reissue rather than only against the first response, so an A -> B -> A' sequence - /// is caught at B: a later response equal to the first is not evidence that nothing changed. - const String etag = read_result.GetETag(); - if (!first_response_etag) - first_response_etag = etag; - else if (*first_response_etag != etag) - response_identity_changed = true; + /// Record the new response's identity; the coherence check itself happens in nextImpl(), at the + /// moment this response actually delivers its first byte. Comparing here instead (against + /// whatever the previous attempt's ETag was) would flag a mismatch as soon as a differently-ETagged + /// response is merely attempted, before it is known whether that attempt will ever deliver + /// anything - and would just as easily lose track of an earlier delivering response across an + /// intervening empty failed attempt with yet another ETag. + pending_response_etag = read_result.GetETag(); + pending_response_bytes_delivered = false; size_t buffer_size = use_external_buffer ? 0 : read_settings.remote_fs_settings.buffer_size; return std::make_unique(std::move(read_result), buffer_size, std::move(watch)); diff --git a/src/IO/ReadBufferFromS3.h b/src/IO/ReadBufferFromS3.h index bc37ce7e3606..fdbf581f75aa 100644 --- a/src/IO/ReadBufferFromS3.h +++ b/src/IO/ReadBufferFromS3.h @@ -91,8 +91,11 @@ class ReadBufferFromS3 : public ReadBufferFromFileBase /// This method returns metadata from the last request. If there were no requests, it will throw exception. ObjectMetadata getObjectMetadataFromTheLastRequest() const; - /// True when a reissued GET answered with a different ETag than the first one did, i.e. the bytes - /// this buffer produced may come from more than one incarnation of the object. + /// True when bytes already delivered to the consumer came from a response whose ETag turned out to + /// differ from a later, reissued response's ETag, i.e. the bytes this buffer produced may come from + /// more than one incarnation of the object. A response that never delivered a byte (e.g. the GET + /// succeeded but the body read failed before any data arrived) does not count: reissuing it and + /// getting a different ETag is an ordinary retry, not a coherence problem. bool responseIdentityChanged() const { return response_identity_changed; } size_t getReadUntilPosition() const { return read_until_position; } @@ -115,7 +118,25 @@ class ReadBufferFromS3 : public ReadBufferFromFileBase Aws::S3::Model::GetObjectResult sendRequest(size_t attempt, size_t range_begin, std::optional range_end_incl) const; - std::optional first_response_etag; + /// Drops the identity baseline. Called when the next request is a reissue for a range the caller + /// explicitly repositioned to (seek, or a change of the read-until bound), as opposed to a retry of + /// the same range after a failure: the bytes already delivered before the reposition reached the + /// consumer as their own self-consistent range, so the next response is not compared against them. + void forgetResponseIdentityBaseline(); + + /// ETag of the last response that has delivered at least one byte to the consumer: the baseline a + /// newly-delivering response is checked against. A response that never delivers a byte (e.g. it + /// fails before the body starts) leaves this untouched, however many such empty attempts happen in + /// a row, so the baseline always reflects the last response that actually contributed bytes. + std::optional last_delivering_response_etag; + + /// ETag of the response `impl` currently represents, and whether that response has delivered a byte + /// yet. Both are set together in initialize(); nextImpl() flips `pending_response_bytes_delivered` + /// to true (and advances last_delivering_response_etag) the moment this response's first byte + /// reaches the consumer. + String pending_response_etag; + bool pending_response_bytes_delivered = false; + bool response_identity_changed = false; ReadSettings read_settings; diff --git a/src/IO/tests/gtest_readbuffer_s3.cpp b/src/IO/tests/gtest_readbuffer_s3.cpp index 4bf502a366bf..d2d2947e55bc 100644 --- a/src/IO/tests/gtest_readbuffer_s3.cpp +++ b/src/IO/tests/gtest_readbuffer_s3.cpp @@ -21,9 +21,15 @@ #include #include #include +#include #include #include +namespace DB::ErrorCodes +{ + extern const int S3_ERROR; +} + static constexpr auto TEST_LOG_LEVEL = "debug"; static fs::path caches_dir = fs::current_path() / "readbuffer_s3"; static std::string cache_base_path = caches_dir / "cache1" / ""; @@ -111,6 +117,57 @@ class StringHTTPBasicStreamBuf : public Poco::Net::HTTPBasicStreamBuf } }; +/// A response body stream that throws once `bytes_before_failure` bytes have been handed out (0 means +/// the very first read fails), simulating a GET whose headers arrived successfully but whose body read +/// broke before delivering that many bytes to the consumer. +class BreakingHTTPBasicStreamBuf : public Poco::Net::HTTPBasicStreamBuf +{ +public: + BreakingHTTPBasicStreamBuf(std::string body, size_t bytes_before_failure_) + : BasicBufferedStreamBuf(body.size(), IOS::in), bodyStream(std::stringstream(std::move(body))), bytes_before_failure(bytes_before_failure_) + { + } + +private: + std::stringstream bodyStream; + size_t bytes_before_failure; + + int readFromDevice(char_type * buf, std::streamsize n) override + { + if (bytes_before_failure == 0) + throw DB::Exception(DB::ErrorCodes::S3_ERROR, "Simulated S3 body read failure"); + + bodyStream.read(buf, std::min(n, static_cast(bytes_before_failure))); + const auto got = bodyStream.gcount(); + bytes_before_failure -= static_cast(got); + return static_cast(got); + } +}; + +/// The byte offset the request's Range header asks for, or 0 when no Range was set. sendRequest() +/// always emits "bytes=-" or "bytes=-", so parsing out lets a mock GetObject +/// serve the bytes a reissued request should actually receive. +static size_t rangeStart(const Aws::S3::Model::GetObjectRequest & request) +{ + if (!request.RangeHasBeenSet()) + return 0; + const std::string & range = request.GetRange(); + const size_t begin_pos = range.find('=') + 1; + const size_t dash_pos = range.find('-', begin_pos); + return std::stoull(range.substr(begin_pos, dash_pos - begin_pos)); +} + +static Aws::S3::Model::GetObjectOutcome makeGetObjectOutcome(std::streambuf * sb, const std::string & etag) +{ + Aws::Http::HeaderValueCollection headers; + headers["etag"] = etag; + auto response_stream = Aws::Utils::Stream::ResponseStream( + Aws::New>("test response stream", std::make_shared(), sb)); + Aws::AmazonWebServiceResult aws_result(std::move(response_stream), std::move(headers)); + DB::S3::Model::GetObjectResult result(std::move(aws_result)); + return Aws::S3::Model::GetObjectOutcome(std::move(result)); +} + using GetObjectFn = std::function; struct ClientFake : DB::S3::Client @@ -255,6 +312,361 @@ TEST_F(ReadBufferFromS3Test, ReleaseSessionWhenReadUntilPosition) ASSERT_FALSE(subject.nextImpl()); } +TEST_F(ReadBufferFromS3Test, IdentityNotFlaggedWhenFailedAttemptDeliveredNoBytes) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 20; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto failing_buf = std::make_shared(body, /* bytes_before_failure */ 0); + auto full_buf = std::make_shared(body); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + EXPECT_EQ(rangeStart(request), 0); + if (call == 1) + return makeGetObjectOutcome(failing_buf.get(), "A"); + return makeGetObjectOutcome(full_buf.get(), "B"); + }; + + /// First attempt's headers carried ETag "A", but its body read fails before any byte reaches the + /// consumer; the reissue delivers the whole object under ETag "B". No bytes of "A" were ever + /// consumed, so this must not be flagged as a coherence problem. + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, IdentityFlaggedWhenBytesDeliveredBeforeFailure) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + /// The first response (ETag "A") delivers 3 bytes before its stream breaks; the reissue, resuming + /// from offset 3, answers with ETag "B". Bytes from two different incarnations reached the + /// consumer, so this must be flagged. + readAndAssert(subject, body.c_str()); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, IdentityNotFlaggedWhenReissuedEtagMatches) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "A"); + }; + + /// Same as above, but the reissue answers with the same ETag "A": both attempts belong to the same + /// incarnation, so this must not be flagged. + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, ThreeResponsesABytesThenAEmptyFailThenBBytesIsFlagged) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto delivers_then_fails = std::make_shared(body, /* bytes_before_failure */ 3); + auto fails_empty = std::make_shared(body, /* bytes_before_failure */ 0); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(delivers_then_fails.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + if (call == 2) + return makeGetObjectOutcome(fails_empty.get(), "A"); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + /// A delivers 3 bytes, then breaks. The reissue (same ETag "A") fails before delivering anything. + /// The next reissue answers with ETag "B" and delivers the rest: A-bytes and B-bytes were mixed, so + /// this must be flagged, even though an empty failed attempt for "A" sat in between. + readAndAssert(subject, body.c_str()); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, ThreeResponsesABytesThenBEmptyFailThenABytesIsNotFlagged) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto delivers_then_fails = std::make_shared(body, /* bytes_before_failure */ 3); + auto fails_empty = std::make_shared(body, /* bytes_before_failure */ 0); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(delivers_then_fails.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + if (call == 2) + return makeGetObjectOutcome(fails_empty.get(), "B"); + return makeGetObjectOutcome(rest_buf.get(), "A"); + }; + + /// A delivers 3 bytes, then breaks. The reissue under ETag "B" fails before delivering anything, so + /// it never contributes to the read. The next reissue answers with ETag "A" (matching the only + /// response that ever delivered bytes) and delivers the rest: the read is coherent and must not be + /// flagged, even though a differently-ETagged empty failed attempt sat in between. + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); + ASSERT_EQ(subject.getObjectMetadataFromTheLastRequest().etag, "A"); +} + +TEST_F(ReadBufferFromS3Test, SeekReissueAcceptsNewEtagWithoutFlag) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + read_settings.remote_fs_settings.min_bytes_for_seek = 0; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto first_buf = std::make_shared(body); + auto after_seek_buf = std::make_shared(body.substr(8)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(first_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 8); + return makeGetObjectOutcome(after_seek_buf.get(), "B"); + }; + + readAndAssert(subject, "123"); + /// A seek far enough ahead to force a reissue (not an in-buffer rewind, not a small forward skip): + /// the caller explicitly repositioned to a different range, so the new response's ETag "B" must not + /// be compared against "A". + subject.seek(8, SEEK_SET); + readAndAssert(subject, "9"); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, SetReadUntilPositionReissueAcceptsNewEtagWithoutFlag) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 2; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456"; + auto first_buf = std::make_shared(body); + auto after_reposition_buf = std::make_shared(body.substr(2, 3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(first_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 2); + return makeGetObjectOutcome(after_reposition_buf.get(), "B"); + }; + + readAndAssert(subject, "12"); + /// impl is still open (no read-until-position was set yet, so nothing released it). Narrowing the + /// read-until bound now tears impl down to reissue for the new bound: an explicit reposition, so + /// the new response's ETag "B" must not be compared against "A". + subject.setReadUntilPosition(5); + readAndAssert(subject, "345"); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, SetReadUntilEndReissueAcceptsNewEtagWithoutFlag) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + subject.setReadUntilPosition(3); + + const std::string body = "123456789"; + auto first_buf = std::make_shared(body.substr(0, 3)); + auto after_reposition_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(first_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(after_reposition_buf.get(), "B"); + }; + + readAndAssert(subject, "123"); + /// Reading exactly up to the bound releases the result (does not reset impl). Removing the bound + /// now tears impl down to reissue for the rest of the object: an explicit reposition, so the new + /// response's ETag "B" must not be compared against "A". + subject.setReadUntilEnd(); + readAndAssert(subject, "456789"); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, InBufferSeekPreservesBaselineAndLaterMixedRetryIsFlagged) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + readAndAssert(subject, "123"); + /// Rewind within the bytes already buffered: this hits the in-buffer fast path in seek(), which + /// never touches impl, so it must not forget the identity baseline. + subject.seek(1, SEEK_SET); + readAndAssert(subject, "23"); + /// Reading past the buffer now reissues on the SAME impl (a retry after a stream break, not an + /// explicit reposition); the baseline from "A" must have survived the harmless seek above, so the + /// mismatched ETag "B" here must still be flagged. + readAndAssert(subject, "456789"); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, ExternalBufferFlagsMixedIncarnations) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + auto subject = DB::ReadBufferFromS3( + client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings, /* use_external_buffer */ true); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + std::vector external_memory(3); + + /// Drive the external-buffer path the way a prefetching/threadpool reader does: supply the memory + /// with set() and pull one chunk with next(), rather than relying on the buffer's own allocation. + subject.set(external_memory.data(), external_memory.size()); + ASSERT_TRUE(subject.next()); + ASSERT_EQ(std::string(subject.buffer().begin(), subject.buffer().end()), "123"); + ASSERT_FALSE(subject.responseIdentityChanged()); + + /// This next() call breaks the "A" stream and reissues; the reissue answers with ETag "B" and + /// delivers bytes via the external buffer. Bytes were consumed on this path too, so it must flag. + subject.set(external_memory.data(), external_memory.size()); + ASSERT_TRUE(subject.next()); + ASSERT_EQ(std::string(subject.buffer().begin(), subject.buffer().end()), "456"); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(ReadBufferFromS3Test, PartialInternalFillNeverExposedDoesNotCountAsDelivery) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 5; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + /// internal_buffer is 5 bytes but only 2 bytes are ever produced before the stream throws, so + /// ReadBufferFromIStream's fill loop calls readFromDevice a second time (asking for more) and gets + /// the exception before it ever assigns `working_buffer` - those 2 bytes are read off the wire but + /// never exposed to the consumer. + auto partial_then_fails = std::make_shared(body, /* bytes_before_failure */ 2); + auto full_buf = std::make_shared(body); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + EXPECT_EQ(rangeStart(request), 0); + if (call == 1) + return makeGetObjectOutcome(partial_then_fails.get(), "A"); + return makeGetObjectOutcome(full_buf.get(), "B"); + }; + + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + TEST_F(ReadBufferFromS3Test, IterateUsesStartAfter) { std::unique_ptr client = std::make_unique(); From 140ad05fff0cc3975564bc3ccfcf9c91223a6997 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 03:03:08 +0200 Subject: [PATCH 45/81] s3: the CAS batch delete honours support_batch_delete; the GC admits each fallback delete `S3ObjectStorage::removeObjectsIfExistImpl` (the CAS GC bulk-delete verb) sent `DeleteObjects` unconditionally, ignoring `S3Capabilities::isBatchDeleteSupported` that the generic `deleteFilesFromS3` consults, including an explicit `false`. Now: exactly one object is always a plain `DeleteObject`; for more than one, a capability known false, or learned false from a "not supported" reply (the same error classes `deleteFilesFromS3` uses, recorded through the shared `S3Capabilities`), throws NOT_IMPLEMENTED without sending anything else. The S3 layer never substitutes a per-key loop of its own, because such a loop would run up to a thousand destructive requests inside ONE admitted CAS request; the caller is the one that can admit each physical delete. `Cas::Gc` gained `removeChunkWriteOnceOrOneByOne`: it tries the chunk as one request and on NOT_IMPLEMENTED only issues one admitted request per key; it is used at both bulk-delete sites (the manifest_deletes flush and the ref-log/ref-snapshot cleanup, whose budget bookkeeping counts objects and is unchanged). Other error classes still fail closed. Also fixed on the way: the per-key errors of a `DeleteObjects` reply were classified with `S3ErrorMapper` alone, which knows only S3-specific names, so `AccessDenied` came back UNKNOWN; the lookup now falls back to `CoreErrorsMapper`, the order `S3ErrorMarshaller::Marshall` uses. Tests: per-key errors (NoSuchKey ignored, ACCESS_DENIED asserted by type with the key), configured-false / learned-false / second-call-after-learning all assert NOT_IMPLEMENTED, other request failures throw with no fallback, the size-one shortcut in both capability states; the CAS helper against the fake backend (bulk path, fallback path with N admitted requests, a teardown begun mid-fallback stopping the remainder at admission, a real error on a key stopping the remaining keys and propagating); a GC round through each call site. (squashed from a11f7f3aada, 78bef1d5c24, 4f8f1b3e3f6, 13bf6a92df0, 76a37852b65 and the batch-delete part of f8278ca6db5) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../Backend/CasObjectStorageBackend.cpp | 4 +- .../ContentAddressed/Gc/CasGc.cpp | 30 +- .../ContentAddressed/Gc/CasGc.h | 22 + .../ObjectStorages/S3/S3ObjectStorage.cpp | 85 +++- .../ObjectStorages/S3/S3ObjectStorage.h | 6 + .../gtest_cas_gc_bulk_delete_fallback.cpp | 184 ++++++++ .../gtest_cas_gc_manifest_bulk_delete.cpp | 33 ++ src/Disks/tests/gtest_cas_ref_gc.cpp | 64 +++ .../gtest_cas_s3_bulk_delete_fallback.cpp | 397 ++++++++++++++++++ 9 files changed, 818 insertions(+), 7 deletions(-) create mode 100644 src/Disks/tests/gtest_cas_gc_bulk_delete_fallback.cpp create mode 100644 src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index 0a302071c9fc..0589929d2912 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -1013,7 +1013,9 @@ void ObjectStorageBackend::removeManyWriteOnce(const std::vector & objects.reserve(keys.size()); for (const WriteOnceKey & key : keys) objects.emplace_back(key.str()); - /// `NOT_IMPLEMENTED` from a storage without a batch delete propagates -- fail-closed by construction. + /// `NOT_IMPLEMENTED` from a storage without a batch delete propagates -- this layer never + /// substitutes a per-key loop of its own (that would run under a single admission for up to + /// 1000 keys). The caller in CasGc.cpp catches it and retries one key per admitted request. object_storage->removeObjectsIfExistUnderProfile(objects, controlRequest(access.attemptNo())); return; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index d4241c0d50ce..5083c822a10e 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -71,6 +71,7 @@ namespace ErrorCodes extern const int BAD_ARGUMENTS; extern const int CORRUPTED_DATA; extern const int LOGICAL_ERROR; + extern const int NOT_IMPLEMENTED; } } @@ -379,6 +380,24 @@ void Gc::runNamespaceJanitorPage( t.metric("leaked", janitor_result.leaked); } +uint64_t removeChunkWriteOnceOrOneByOne(CasOperation & op, const std::vector & chunk, const Retry & policy) +{ + try + { + op.removeManyWriteOnce(chunk, policy); + return 1; + } + catch (const Exception & e) + { + if (e.code() != ErrorCodes::NOT_IMPLEMENTED) + throw; + for (const WriteOnceKey & key : chunk) + op.removeManyWriteOnce({key}, policy); + /// +1: the failed bulk attempt above is itself a call this helper made. + return 1 + chunk.size(); + } +} + RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool allow_steal, UniversePolicy policy, RoundReport * progress) { @@ -1108,8 +1127,10 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al { if (chunk.empty()) return; - op.removeManyWriteOnce(chunk, Retry::standard()); - ++requests; + /// A backend without `DeleteObjects` (GCS) falls back to one admitted delete per key here; + /// `chunk_entries`' per-key bookkeeping below is unaffected either way -- it counts objects + /// that are gone after this call returns, not how many requests it took to get them there. + requests += removeChunkWriteOnceOrOneByOne(op, chunk, Retry::standard()); for (const auto * entry : chunk_entries) { ++report.manifests_deleted; @@ -3666,7 +3687,10 @@ void Gc::cleanupRefObjects( std::vector chunk(cohort.begin() + begin, cohort.begin() + end); if (!authorityHolds(chunk.front().str())) return; - op.removeManyWriteOnce(chunk, Retry::standard()); + /// A backend without `DeleteObjects` (GCS) falls back to one admitted delete per key here. + /// The budget and the profile event below count OBJECTS in `chunk`, which is the same + /// `chunk.size()` whichever way `removeChunkWriteOnceOrOneByOne` actually sent them. + removeChunkWriteOnceOrOneByOne(op, chunk, Retry::standard()); work_budget.ref_cleanup_objects_used += chunk.size(); ProfileEvents::increment(ProfileEvents::CASRefCleanupObjectsDeleted, chunk.size()); /// cleanup object deletion /// Advance by what was actually sent, not the nominal chunk size: the budget cap above can diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index 4216912604e5..bd0d65e2e573 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -71,6 +71,28 @@ enum class UniversePolicy : uint8_t /// decision ever reads them. uint64_t retiredLogicalSize(ObjectKind kind, uint64_t object_size, uint64_t blob_header_len); +/// Deletes `chunk` as one bulk `removeManyWriteOnce` request, falling back to one admitted request per +/// key when the object storage answers with `NOT_IMPLEMENTED` -- the signal +/// `S3ObjectStorage::removeObjectsIfExistImpl` gives (without sending anything else itself) once +/// `DeleteObjects` is known unsupported (a configured GCS backend, or one that just failed a batch +/// attempt this same call). The fallback is not merely "the same deletes issued more slowly": each +/// `op.removeManyWriteOnce({key}, policy)` is its OWN admission (fence, budget, deadline checked +/// afresh), which one bulk call covering up to `kBulkDeleteMaxKeys` physical deletes under a SINGLE +/// admission cannot be -- exactly the gap a storage-side per-key loop would have left open. Every other +/// failure propagates unchanged: retry/reissue for it is the engine's own policy, applied to each +/// admitted attempt -- bulk or single -- the same way it always was. +/// +/// Returns the number of `op.removeManyWriteOnce` calls THIS HELPER issued: 1 for the bulk path, or +/// 1 + `chunk.size()` for the fallback -- the failed bulk attempt counted alongside the one call per key +/// that followed it, since that attempt is a call this helper made whether or not it reached the network +/// (there is no signal available here to tell "sent and rejected" apart from "refused locally, unsent"; +/// `S3ObjectStorage::removeObjectsIfExistImpl` reports both as the same NOT_IMPLEMENTED). This is call +/// COUNT, not a distinct network-request count -- the same granularity `CountingBackend::bulkRemoveCalls` +/// and the `CASBulkDeleteRequests` profile event already use elsewhere for "request". +/// Declared here (not file-local) so a unit test can drive it directly against a scripted backend, +/// rather than only through a full GC round. +uint64_t removeChunkWriteOnceOrOneByOne(CasOperation & op, const std::vector & chunk, const Retry & policy); + /// Pure skip-unchanged decision. Returns true iff the current round may be /// DEFERRED (re-adopt the sealed generation, no fold/delete). A round MUST fold when: enough shards /// changed (>= fold_threshold), OR a destructive decision is due (graduation_due), OR the defer bound diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 3ef301e5f8d8..eb430667fa45 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -40,6 +40,7 @@ #include #include +#include #include #include @@ -144,6 +145,23 @@ void logIfError(const Aws::Utils::Outcome & response, std::functi } } +/// Classifies a per-key error `Code` string from a `DeleteObjects` response body (data the SDK never +/// builds an `Aws::S3::S3Error` for, since the response as a whole was a success) the same way the SDK's +/// own `S3ErrorMarshaller::Marshall` classifies a whole-response error: `Aws::S3::S3ErrorMapper` first +/// (the S3-specific extension names -- `NoSuchKey`, `NoSuchBucket`, ...), falling back to +/// `Aws::Client::CoreErrorsMapper` for a name shared across every AWS service (`AccessDenied`, +/// `InternalError`, ...), which `S3ErrorMapper` alone does not recognize and would otherwise leave +/// classified as `UNKNOWN`. The two mappers' shared codes carry identical numeric values by construction +/// (see the "// From Core//" section of `Aws::S3::S3Errors`), so reinterpreting a `CoreErrors` result as +/// `S3Errors` is exactly what the SDK's own marshaller does. +Aws::S3::S3Errors classifyDeleteObjectsErrorCode(const String & code) +{ + if (const auto s3_specific = Aws::S3::S3ErrorMapper::GetErrorForName(code.c_str()).GetErrorType(); + s3_specific != Aws::Client::CoreErrors::UNKNOWN) + return static_cast(s3_specific); + return static_cast(Aws::Client::CoreErrorsMapper::GetErrorForName(code.c_str()).GetErrorType()); +} + } namespace @@ -653,6 +671,56 @@ void S3ObjectStorage::removeObjectsIfExistImpl( if (objects.empty()) return; + /// A batch of exactly one object is always a plain `DeleteObject`, never `DeleteObjects` -- mirroring + /// `deleteFilesFromS3`'s own `keys.size() == 1` rule (IO/S3/deleteFileFromS3.cpp), which skips the + /// batch request for the same single key for the same reason: there is no need for it. This is what + /// makes the CAS-side per-key fallback actually delete anything on a backend with no `DeleteObjects` + /// at all (GCS): that backend rejects the VERB outright, not by key count, so a "batch" of one + /// object sent as `DeleteObjects` would fail there identically to a bigger one. A single physical + /// request is never a loop, so this does not reintroduce what the capability check below exists to + /// rule out. + if (objects.size() == 1) + { + const StoredObject & object = objects.front(); + S3::DeleteObjectRequest request; + request.SetBucket(uri.bucket); + request.SetKey(object.remote_path); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(request, attempt_seed); + + ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); + Stopwatch watch; + auto outcome = used_client->DeleteObject(request); + auto elapsed = watch.elapsedMicroseconds(); + + if (auto blob_storage_log = BlobStorageLogWriter::create(disk_name)) + blob_storage_log->addEvent(BlobStorageLogElement::EventType::Delete, + uri.bucket, object.remote_path, + object.local_path, object.bytes_size, elapsed, + outcome.IsSuccess() ? 0 : static_cast(outcome.GetError().GetErrorType()), + outcome.IsSuccess() ? "" : outcome.GetError().GetMessage()); + + if (outcome.IsSuccess()) + return; + + const auto & err = outcome.GetError(); + if (S3::isNotFoundError(err.GetErrorType())) + return; + + throw S3Exception(err.GetErrorType(), "{} (Code: {}) while removing object with path {} from S3", + err.GetMessage(), static_cast(err.GetErrorType()), object.remote_path); + } + + /// GCS has no `DeleteObjects`: a capability the config declared false, or that an earlier batch + /// attempt on this same storage already learned false, must not be retried here. This storage never + /// loops over `objects` itself to work around it -- a CAS caller admits one request per physical + /// delete (see `ObjectStorageBackend::removeManyWriteOnce` and its own caller in CasGc.cpp), which an + /// internal loop over more than one object, running under a SINGLE admission, cannot be. Report the + /// absence of the capability instead, and let that caller decide how to retry. + if (auto support_batch_delete = s3_capabilities.isBatchDeleteSupported(); + support_batch_delete.has_value() && !support_batch_delete.value()) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support DeleteObjects", getName()); + std::vector identifiers; // STYLE_CHECK_ALLOW_STD_CONTAINERS identifiers.reserve(objects.size()); for (const auto & object : objects) @@ -677,7 +745,8 @@ void S3ObjectStorage::removeObjectsIfExistImpl( auto outcome = used_client->DeleteObjects(request); /// Every key lands in system.blob_storage_log, as the single-key paths do; the batch's outcome is - /// stamped on each of them. + /// stamped on each of them. This still happens when the batch turns out to be unsupported and the + /// call falls back below: the failed batch attempt is itself an event, same as in `deleteFilesFromS3`. if (auto blob_storage_log = BlobStorageLogWriter::create(disk_name)) { for (const auto & object : objects) @@ -692,6 +761,17 @@ void S3ObjectStorage::removeObjectsIfExistImpl( if (!outcome.IsSuccess()) { const auto & err = outcome.GetError(); + /// Same classification `deleteFilesFromS3` uses to detect a backend that rejects `DeleteObjects` + /// itself (as opposed to a request that reached S3 and failed for an ordinary reason). + if ((err.GetExceptionName() == "InvalidRequest") || (err.GetExceptionName() == "InvalidArgument") + || (err.GetExceptionName() == "NotImplemented")) + { + LOG_TRACE(log, "DeleteObjects is not supported: {} (Code: {}). The caller must delete one object at a time.", + err.GetMessage(), static_cast(err.GetErrorType())); + s3_capabilities.setIsBatchDeleteSupported(false); + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support DeleteObjects", getName()); + } + throw S3Exception(err.GetErrorType(), "{} (Code: {}) while removing {} objects from S3 in one request", err.GetMessage(), static_cast(err.GetErrorType()), objects.size()); } @@ -700,8 +780,7 @@ void S3ObjectStorage::removeObjectsIfExistImpl( std::optional first_error_type; for (const auto & err : outcome.GetResult().GetErrors()) { - const auto error_type = static_cast( - Aws::S3::S3ErrorMapper::GetErrorForName(err.GetCode().c_str()).GetErrorType()); + const auto error_type = classifyDeleteObjectsErrorCode(err.GetCode()); if (S3::isNotFoundError(error_type)) continue; if (!failed_keys.empty()) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index a7a70e9616f6..fa823b2f7296 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -126,6 +126,12 @@ class S3ObjectStorage : public IObjectStorage const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request) override; /// One `DeleteObjects` for the given objects (the caller chunks to at most 1000); absence is success. + /// Exactly one object is always a plain `DeleteObject` instead (never gated on `s3_capabilities`: a + /// single physical request per call, so there is nothing here for that capability to say no to). + /// For more than one object, throws `NOT_IMPLEMENTED` without sending anything once `DeleteObjects` + /// is known unsupported (a configured or a just-learned `S3Capabilities::isBatchDeleteSupported() == + /// false`) -- this storage never substitutes a per-key loop of its own, since the caller is the one + /// that can admit each physical delete as its own request. void removeObjectsIfExistUnderProfile( const StoredObjects & objects, const ObjectStorageControlRequest & request) override; diff --git a/src/Disks/tests/gtest_cas_gc_bulk_delete_fallback.cpp b/src/Disks/tests/gtest_cas_gc_bulk_delete_fallback.cpp new file mode 100644 index 000000000000..c490ada2c3ae --- /dev/null +++ b/src/Disks/tests/gtest_cas_gc_bulk_delete_fallback.cpp @@ -0,0 +1,184 @@ +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +/// `removeChunkWriteOnceOrOneByOne` (CasGc.h) is what both GC bulk-delete call sites (manifest_deletes' +/// flush() and cleanupRefObjects' chunk loop) use to survive a backend without `DeleteObjects`. Tested +/// here in isolation, directly against the engine, rather than only through the much larger machinery of +/// a full GC round. + +namespace DB::ErrorCodes +{ +extern const int CORRUPTED_DATA; +extern const int NETWORK_ERROR; +extern const int NOT_IMPLEMENTED; +} + +using namespace DB::Cas; +using DB::Cas::tests::expectThrowsCode; + +namespace +{ + +const Layout kLayout{"p"}; +const RootNamespace kNs{"test/aa@cas@"}; + +WriteOnceKey manifestKey(uint32_t ordinal) +{ + return kLayout.writeOnceManifestKey( + ManifestId{kNs, ManifestRef{.writer_epoch = 1, .build_sequence = 1, .manifest_ordinal = ordinal}}); +} + +std::vector manifestKeys(uint32_t count) +{ + std::vector keys; + for (uint32_t ordinal = 1; ordinal <= count; ++ordinal) + keys.push_back(manifestKey(ordinal)); + return keys; +} + +PoolPtr openPlainPool(const std::shared_ptr & backend) +{ + PoolConfig config; + config.pool_prefix = "p"; + config.server_root_id = "test"; + return Pool::open(backend, config); +} + +} + +TEST(CASGCBulkDeleteFallback, HappyPathIsOneRequest) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + CasOperation op = store->openRequests().admit(); + const std::vector keys = manifestKeys(3); + for (const WriteOnceKey & key : keys) + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "b", Retry::once()))); + + const uint64_t requests_issued = removeChunkWriteOnceOrOneByOne(op, keys, Retry::once()); + + EXPECT_EQ(requests_issued, 1u); + EXPECT_EQ(backend->bulkRemoveCalls(), 1u); + for (const WriteOnceKey & key : keys) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); +} + +TEST(CASGCBulkDeleteFallback, NotImplementedFallsBackToOneRequestPerKeyEachDeleted) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + CasOperation op = store->openRequests().admit(); + const std::vector keys = manifestKeys(3); + for (const WriteOnceKey & key : keys) + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "b", Retry::once()))); + + backend->failNextBulkRemoveWith(std::make_exception_ptr(DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "no batch delete"))); + + const uint64_t requests_issued = removeChunkWriteOnceOrOneByOne(op, keys, Retry::once()); + + EXPECT_EQ(requests_issued, 4u) << "the failed bulk attempt is itself a call, counted alongside the 3 that followed it"; + EXPECT_EQ(backend->bulkRemoveCalls(), 4u) << "1 failed bulk attempt + 3 single-key fallback requests"; + for (const WriteOnceKey & key : keys) + EXPECT_FALSE(op.head(key.str(), Retry::once()).has_value()) << key.str(); +} + +/// A teardown begun WHILE the fallback is mid-loop stops the remainder at admission, exactly as any +/// other CAS request would be: `removeChunkWriteOnceOrOneByOne`'s per-key loop is not a special path +/// around the engine's own fence, it is ordinary calls through it. +TEST(CASGCBulkDeleteFallback, TeardownBegunBetweenTwoFallbackKeysStopsTheRemainderAtAdmission) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + CasOperation op = store->openRequests().admit(); + const std::vector keys = manifestKeys(4); + for (const WriteOnceKey & key : keys) + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "b", Retry::once()))); + + backend->failNextBulkRemoveWith(std::make_exception_ptr(DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "no batch delete"))); + + /// The hook does not run on the armed (failing) bulk attempt (it is rethrown before the hook would + /// fire), so this counts only the fallback's own per-key calls that actually reached the backend. + /// Teardown is armed once the SECOND such call has been served, so it is the THIRD key's own + /// admission -- checked at the start of its own `removeManyWriteOnce`, before this hook could run + /// again -- that is refused; the fourth key is never attempted at all. + size_t backend_calls_served = 0; + backend->onBeforeBulkRemove([&] + { + if (++backend_calls_served == 2) + store->beginTeardown(); + }); + + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { (void)removeChunkWriteOnceOrOneByOne(op, keys, Retry::standard()); }); + + /// `store->beginTeardown()` is irreversible here (this test never re-opens the pool), so `op` itself + /// -- the open plane -- refuses every further request, verification reads included. Read through the + /// mount plane instead: a different fence over the SAME backend, unaffected by open-plane teardown + /// (see `CASGCTeardownStop.OpenPlaneRefusesAfterTeardownBeganAndTheMountPlaneDoesNot`). + CasOperation verify = store->mountRequests().admit(); + EXPECT_FALSE(verify.head(keys[0].str(), Retry::once()).has_value()) << "deleted before teardown began"; + EXPECT_FALSE(verify.head(keys[1].str(), Retry::once()).has_value()) << "deleted before teardown began"; + EXPECT_TRUE(verify.head(keys[2].str(), Retry::once()).has_value()) << "refused at admission, never reached the backend"; + EXPECT_TRUE(verify.head(keys[3].str(), Retry::once()).has_value()) << "never attempted"; + EXPECT_EQ(backend->bulkRemoveCalls(), 3u) << "1 failed bulk attempt + 2 single-key fallback requests that landed"; +} + +/// A REAL error on one of the fallback's per-key deletes (not "batch not supported", so not caught and +/// retried again) stops the loop exactly where it happened: the keys before it are deleted, the ones +/// from it on are never attempted, and the error itself propagates out of the helper. +TEST(CASGCBulkDeleteFallback, ARealErrorOnAFallbackKeyStopsTheRemainderAndPropagates) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + CasOperation op = store->openRequests().admit(); + const std::vector keys = manifestKeys(4); + for (const WriteOnceKey & key : keys) + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "b", Retry::once()))); + + backend->failNextBulkRemoveWith(std::make_exception_ptr(DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "no batch delete"))); + + /// The hook does not run on an armed (failing) call, so this fires only on the fallback's own + /// per-key calls that actually reached the backend -- the FIRST of which (key[0]'s own delete) arms + /// a real, non-capability failure for the call right after it, i.e. key[1]'s. + backend->onBeforeBulkRemove([&] + { + backend->failNextBulkRemoveWith(std::make_exception_ptr(DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "not a capability problem"))); + }); + + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)removeChunkWriteOnceOrOneByOne(op, keys, Retry::once()); }); + + EXPECT_FALSE(op.head(keys[0].str(), Retry::once()).has_value()) << "deleted before the real error"; + EXPECT_TRUE(op.head(keys[1].str(), Retry::once()).has_value()) << "this delete is the one that failed"; + EXPECT_TRUE(op.head(keys[2].str(), Retry::once()).has_value()) << "never attempted"; + EXPECT_TRUE(op.head(keys[3].str(), Retry::once()).has_value()) << "never attempted"; + EXPECT_EQ(backend->bulkRemoveCalls(), 3u) << "1 failed bulk attempt + key[0]'s delete + key[1]'s failed attempt"; +} + +/// A failure outside the "batch delete not supported" class must propagate as-is, with no fallback: +/// the helper does not treat every `removeManyWriteOnce` failure as "try one key at a time". +TEST(CASGCBulkDeleteFallback, OtherFailureClassPropagatesWithNoFallback) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + CasOperation op = store->openRequests().admit(); + const std::vector keys = manifestKeys(3); + for (const WriteOnceKey & key : keys) + ASSERT_TRUE(std::holds_alternative(op.create(key.str(), "b", Retry::once()))); + + backend->failNextBulkRemoveWith(std::make_exception_ptr(DB::Exception(DB::ErrorCodes::CORRUPTED_DATA, "not a capability problem"))); + + expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { (void)removeChunkWriteOnceOrOneByOne(op, keys, Retry::once()); }); + + EXPECT_EQ(backend->bulkRemoveCalls(), 1u) << "no per-key fallback for a non-capability failure"; + for (const WriteOnceKey & key : keys) + EXPECT_TRUE(op.head(key.str(), Retry::once()).has_value()) << "nothing was deleted"; +} diff --git a/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp b/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp index d17cffd1fbde..3a7c2ca3ba7f 100644 --- a/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp +++ b/src/Disks/tests/gtest_cas_gc_manifest_bulk_delete.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include /// The manifest_deletes phase sends owner-removed manifest bodies to the store in chunks of @@ -15,6 +16,11 @@ namespace ProfileEvents extern const Event CASBulkDeleteRequests; } +namespace DB::ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +} + using namespace DB::Cas; using namespace DB::Cas::tests; @@ -89,6 +95,33 @@ TEST(CASGCManifestBulkDelete, FiveBodiesInChunksOfTwoAreThreeRequests) EXPECT_FALSE((*op).head(store->layout().manifestKey(id), Retry::once()).has_value()); } +/// The object storage rejects the chunk's one bulk `removeManyWriteOnce` as NOT_IMPLEMENTED (a +/// GCS-backed pool): the phase's `flush()` falls back to one admitted request per key +/// (`removeChunkWriteOnceOrOneByOne`, CasGc.h), and every manifest in the chunk is still recorded +/// deleted -- the per-key event emission this phase does is unaffected by how the deletes were sent. +TEST(CASGCManifestBulkDelete, NotImplementedFallsBackToOneRequestPerKeyAndStillRecordsAllOfThem) +{ + auto backend = std::make_shared(); + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", + .gc_fold_max_defer_rounds = 0}); + const auto ids = seedDroppedManifests(*backend, store->layout(), 5); + + /// One armed failure: the chunk's own bulk attempt (all 5 land in one chunk under the default + /// chunk size) fails as "batch delete not supported"; the 5 single-key fallback calls that follow + /// are not armed and succeed. + backend->failNextBulkRemoveWith(std::make_exception_ptr( + DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "no batch delete"))); + + Gc gc(store, kGc); + const uint64_t deleted = reclaim(gc, store, *backend, ids, 16); + + EXPECT_EQ(deleted, 5u) << "the per-key fallback must still record every manifest as deleted"; + EXPECT_EQ(backend->bulkRemoveCalls(), 6u) << "1 failed bulk attempt + 5 single-key fallback requests"; + OperationForTest op(*backend); + for (const ManifestId & id : ids) + EXPECT_FALSE((*op).head(store->layout().manifestKey(id), Retry::once()).has_value()); +} + TEST(CASGCManifestBulkDelete, AThrowInTheSecondChunkKeepsTheFirstChunksAuditAndAbortsTheRound) { auto backend = std::make_shared(); diff --git a/src/Disks/tests/gtest_cas_ref_gc.cpp b/src/Disks/tests/gtest_cas_ref_gc.cpp index 1e38e91ef7d1..c76f68d0489b 100644 --- a/src/Disks/tests/gtest_cas_ref_gc.cpp +++ b/src/Disks/tests/gtest_cas_ref_gc.cpp @@ -25,6 +25,7 @@ using namespace DB::Cas::tests; namespace DB::ErrorCodes { extern const int CORRUPTED_DATA; +extern const int NOT_IMPLEMENTED; } namespace ProfileEvents @@ -1000,6 +1001,69 @@ TEST(CASRefGc, RefObjectCleanupDeletesExactlyThePlannedSet) << "snapshot " << renderRefTxnId(id) << " not in the plan must survive"; } +/// The same planned set as above, but the object storage rejects the cohort's one bulk +/// `removeManyWriteOnce` as NOT_IMPLEMENTED (a GCS-backed pool): `cleanupRefObjects`' call site falls +/// back to one admitted request per key (`removeChunkWriteOnceOrOneByOne`, CasGc.h), and the outcome -- +/// which keys are gone, and the budget/profile-event accounting -- must be identical to the plain +/// bulk-request path above. +TEST(CASRefGc, RefObjectCleanupFallsBackToOnePerKeyWhenBatchDeleteIsUnsupported) +{ + auto backend = std::make_shared(); + auto store = openPoolForTest(backend, /*gc_fold_max_defer_rounds*/ 0); + const Layout & layout = store->layout(); + const RootNamespace ns{"00/aa@cas@"}; + fixture::admitLive(*backend, store->layout(), ns); + + const ManifestRef r1 = mref(1); + const ManifestRef r2 = mref(2); + writeManifestRaw(*backend, layout, ns, r1, {blobEntryFor("a", DB::UInt128(1))}); + writeManifestRaw(*backend, layout, ns, r2, {blobEntryFor("b", DB::UInt128(2))}); + const uint64_t v1 = publishCommittedTransition(*backend, layout, ns, "t1", std::nullopt, r1); + const uint64_t v2 = publishCommittedTransition(*backend, layout, ns, "t2", std::nullopt, r2); + + RefTableSnapshot old_snap = minimalLiveSnapshot(ns.string(), RefTxnId{1, v1}, + {committedRow("t1", r1)}); + RefTableSnapshot new_snap = minimalLiveSnapshot(ns.string(), RefTxnId{1, v2}, + {committedRow("t1", r1), committedRow("t2", r2)}); + writeRefSnapshotRaw(*backend, layout, old_snap); + writeRefSnapshotRaw(*backend, layout, new_snap); + replaceRecoverableCkptForRawFixture(*backend, layout, ns, RefCkpt{ + .life_epoch = 1, + .committed_through = RefTxnId{1, v2}, + .checkpoint_snapshot_id = RefTxnId{1, v2}, + .last_epoch_seal = std::nullopt, + }); + + const NamespaceLifeId life = fixture::fixtureLife(ns); + const RefTableListing listing{ + .logs = {RefTxnId{1, v1}, RefTxnId{1, v2}}, + .snapshots = {RefTxnId{1, v1}, RefTxnId{1, v2}}}; + const RefTxnId durable_cursor{1, v2}; + const RefTxnId checkpoint_snapshot_id{1, v2}; + const RefCleanupPlan plan = planRefCleanup(listing, durable_cursor, checkpoint_snapshot_id, std::nullopt); + const uint64_t cohort_size = plan.deletable_logs.size() + plan.deletable_snapshots.size(); + ASSERT_GT(cohort_size, 0u) << "the fixture must actually have something to delete for this test to prove anything"; + + /// One armed failure: the cohort's own bulk `removeManyWriteOnce` call fails as "batch delete not + /// supported"; the fallback's per-key calls that follow are not armed and succeed. + backend->failNextBulkRemoveWith(std::make_exception_ptr( + DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "no batch delete"))); + const auto cleaned_before = ProfileEvents::global_counters[ProfileEvents::CASRefCleanupObjectsDeleted].load(); + + OperationForTest op(*backend); + Gc gc(store, kGc); + ASSERT_TRUE(runRegularRoundReclaiming(gc).acquired_lease); + + for (const RefTxnId & id : plan.deletable_logs) + EXPECT_FALSE((*op).head(layout.refLogKey(life, id), Retry::once()).has_value()); + for (const RefTxnId & id : plan.deletable_snapshots) + EXPECT_FALSE((*op).head(layout.refSnapshotKey(life, id), Retry::once()).has_value()); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefCleanupObjectsDeleted].load() - cleaned_before, cohort_size) + << "the budget/profile-event accounting counts objects, unaffected by the fallback"; + /// 1 failed bulk attempt + one request per key in the cohort. + EXPECT_EQ(backend->bulkRemoveCalls(), 1 + cohort_size); +} + /// Task 13 (spec §implementation-impact / §GC Budget): one fold+clean round increments every ref-intake /// observability counter -- global LIST pages (Q), log-body GETs (K), manifest-body fold GETs (H), emitted /// manifest edges, and cleaned old ref objects (D). Before/after deltas prove each site actually fires. diff --git a/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp b/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp new file mode 100644 index 000000000000..e3c7590fa136 --- /dev/null +++ b/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp @@ -0,0 +1,397 @@ +#include + +#include "config.h" + +#if USE_AWS_S3 + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB::ErrorCodes +{ +extern const int NOT_IMPLEMENTED; +} + +/// `S3ObjectStorage::removeObjectsIfExistImpl` (the CAS bulk-delete path, reached through +/// `removeObjectsIfExistUnderProfile`) must honour `S3Capabilities::isBatchDeleteSupported()` the same +/// way the generic `deleteFilesFromS3` does, but WITHOUT looping over the objects itself: once the +/// capability is known false (a configured `false`, or one just learned from a `DeleteObjects` reply in +/// the "batch delete not implemented" error class), it throws `NOT_IMPLEMENTED` without sending anything +/// else, and leaves per-key retry to the caller (the CAS engine admits each such retry as its own +/// request -- see CasGc.cpp's `removeChunkWriteOnceOrOneByOne`). A request failure of any other class +/// must keep today's fail-close behaviour. The one exception to all of this is a batch of exactly one +/// object, which is always a plain `DeleteObject` -- never gated on the capability at all, since a +/// single physical request is never something the capability check exists to rule out. + +namespace +{ + +/// A real local HTTP server standing in for S3. `DeleteObjects` arrives as a POST to the bucket root; +/// a per-key `DeleteObject` arrives as a plain HTTP DELETE to the key's path -- the two are +/// distinguished by HTTP method alone, with no need to parse the request body or query string. +class ScriptedS3Server +{ +public: + using Responder = std::function; + +private: + class Handler : public Poco::Net::HTTPRequestHandler + { + ScriptedS3Server & owner; + + public: + explicit Handler(ScriptedS3Server & owner_) : owner(owner_) { } + + void handleRequest(Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) override + { + { + std::lock_guard lock(owner.mutex); + owner.methods_seen.push_back(request.getMethod()); + } + /// `DeleteObjects` carries a request body (the XML `` payload); leaving it unread on a + /// keep-alive connection makes Poco parse those leftover bytes as the start of the NEXT + /// request once this handler returns, corrupting the very next `DeleteObject` this test expects. + request.stream().ignore(std::numeric_limits::max()); + owner.responder(request, response); + } + }; + + class Factory : public Poco::Net::HTTPRequestHandlerFactory + { + ScriptedS3Server & owner; + + Poco::Net::HTTPRequestHandler * createRequestHandler(const Poco::Net::HTTPServerRequest &) override + { + return new Handler(owner); + } + + public: + explicit Factory(ScriptedS3Server & owner_) : owner(owner_) { } + }; + + std::unique_ptr server_socket; + Poco::SharedPtr handler_factory; + Poco::AutoPtr server_params; + std::unique_ptr server; + Responder responder; + mutable std::mutex mutex; + std::vector methods_seen; + +public: + explicit ScriptedS3Server(Responder responder_) + : server_socket(std::make_unique(0)) + , handler_factory(new Factory(*this)) + , server_params(new Poco::Net::HTTPServerParams()) + , server(std::make_unique(handler_factory, *server_socket, server_params)) + , responder(std::move(responder_)) + { + server->start(); + } + + std::string getUrl() const { return "http://" + server_socket->address().toString(); } + + size_t countMethod(const std::string & method) const + { + std::lock_guard lock(mutex); + return static_cast(std::count(methods_seen.begin(), methods_seen.end(), method)); + } +}; + +void sendXml(Poco::Net::HTTPServerResponse & response, Poco::Net::HTTPResponse::HTTPStatus status, const std::string & body) +{ + response.setContentType("application/xml"); + response.setContentLength(body.size()); + response.setStatus(status); + auto & out = response.send(); + out << body; + out.flush(); +} + +/// A quiet-mode `DeleteObjects` success (HTTP 200) whose body lists only the failed keys, exactly as a +/// real S3 backend would report a mixed outcome. +void sendBatchSuccessWithErrors(Poco::Net::HTTPServerResponse & response, const std::string & not_found_key, const std::string & denied_key) +{ + const std::string body = + "" + "" + "" + not_found_key + "NoSuchKeyThe specified key does not exist." + "" + denied_key + "AccessDeniedAccess Denied" + ""; + sendXml(response, Poco::Net::HTTPResponse::HTTP_OK, body); +} + +/// A request-level `DeleteObjects` failure in the "batch delete is not implemented" class that +/// `deleteFileFromS3.cpp`'s `deleteFilesFromS3` also treats as "fall back to plain `DeleteObject`". +void sendBatchNotImplemented(Poco::Net::HTTPServerResponse & response) +{ + const std::string body = + "" + "NotImplementedA header you provided implies functionality that is not implemented"; + sendXml(response, Poco::Net::HTTPResponse::HTTP_BAD_REQUEST, body); +} + +/// A request-level `DeleteObjects` failure in an ordinary (not "unsupported") class: this must keep +/// today's fail-close behaviour and never fall back to per-key deletes. +void sendBatchInternalError(Poco::Net::HTTPServerResponse & response) +{ + const std::string body = + "" + "InternalErrorWe encountered an internal error, please try again."; + sendXml(response, Poco::Net::HTTPResponse::HTTP_INTERNAL_SERVER_ERROR, body); +} + +void sendDeleteObjectSuccess(Poco::Net::HTTPServerResponse & response) +{ + response.setContentLength(0); + response.setStatus(Poco::Net::HTTPResponse::HTTP_NO_CONTENT); + response.send(); +} + +/// A single-key `DeleteObject` failure -- used to script the size-one path's own error handling, as +/// distinct from the batch response's per-key `` elements covered by the test above. +void sendSingleDeleteError(Poco::Net::HTTPServerResponse & response, Poco::Net::HTTPResponse::HTTPStatus status, const std::string & code, const std::string & message) +{ + const std::string body = + "" + "" + code + "" + message + ""; + sendXml(response, status, body); +} + +std::shared_ptr makeStorageForTest(const std::string & endpoint, const DB::S3Capabilities & capabilities) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /* s3_slow_all_threads_after_network_error = */ false, + /* s3_slow_all_threads_after_retryable_error = */ false, + /* enable_s3_requests_logging = */ false, + /* for_disk_s3 = */ true, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); + cfg.endpointOverride = endpoint; + cfg.connectTimeoutMs = 10000; + cfg.requestTimeoutMs = 10000; + cfg.s3_use_adaptive_timeouts = false; + auto client = DB::S3::ClientFactory::instance().create( + cfg, + DB::S3::ClientSettings{ + .use_virtual_addressing = false, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }, + "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); + return std::make_shared( + std::move(client), std::make_unique(), + DB::S3::URI(endpoint + "/test-bucket/"), capabilities, + DB::ObjectStorageKeyGeneratorPtr{}, "disk"); +} + +DB::ContextPtr contextForTest() +{ + return getContext().context; +} + +/// The CAS-side fallback (CasGc.cpp's `removeChunkWriteOnceOrOneByOne`) keys specifically on +/// `NOT_IMPLEMENTED`; a capability-rejection test that only checks "threw a DB::Exception" would still +/// pass if this storage started throwing, say, BAD_ARGUMENTS instead -- which would silently break that +/// fallback while every assertion here kept passing. +void expectNotImplemented(const std::function & fn) +{ + try + { + fn(); + FAIL() << "expected a NOT_IMPLEMENTED exception"; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::NOT_IMPLEMENTED) << e.message(); + } +} + +} + +TEST(S3BulkDeleteFallback, PerKeyErrorsWithinASuccessfulBatchAreUnchanged) +{ + (void)contextForTest(); + + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendBatchSuccessWithErrors(response, "notfound-key", "denied-key"); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + try + { + storage->removeObjectsIfExistUnderProfile( + {DB::StoredObject("present-key"), DB::StoredObject("notfound-key"), DB::StoredObject("denied-key")}, + DB::ObjectStorageControlRequest{}); + FAIL() << "expected removeObjectsIfExistUnderProfile to throw on the AccessDenied key"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::ACCESS_DENIED); + EXPECT_NE(e.message().find("denied-key"), std::string::npos) << e.message(); + } + + /// Exactly one DeleteObjects request; NoSuchKey and AccessDenied are both surfaced by the same + /// batch response, no fallback is expected here. + EXPECT_EQ(server.countMethod("POST"), 1u); + EXPECT_EQ(server.countMethod("DELETE"), 0u); +} + +TEST(S3BulkDeleteFallback, UnsupportedBatchReplyRecordsCapabilityFalseAndThrowsNotImplemented) +{ + (void)contextForTest(); + + std::atomic batch_attempts{0}; + ScriptedS3Server server([&](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + ASSERT_EQ(request.getMethod(), "POST") << "capability false must never send anything, batch or per-key"; + ++batch_attempts; + sendBatchNotImplemented(response); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + DB::StoredObjects objects{DB::StoredObject("key-a"), DB::StoredObject("key-b")}; + + expectNotImplemented([&] { storage->removeObjectsIfExistUnderProfile(objects, DB::ObjectStorageControlRequest{}); }); + EXPECT_EQ(batch_attempts.load(), 1u); + EXPECT_EQ(server.countMethod("DELETE"), 0u) << "this storage never loops over objects itself"; + + /// The capability is now known false on this storage: a second call must throw at once, without + /// even a `DeleteObjects` probe. + expectNotImplemented([&] { storage->removeObjectsIfExistUnderProfile(objects, DB::ObjectStorageControlRequest{}); }); + EXPECT_EQ(batch_attempts.load(), 1u) << "a second DeleteObjects attempt means the learned capability was not honoured"; + EXPECT_EQ(server.countMethod("DELETE"), 0u); +} + +/// A batch of exactly one object is always a plain `DeleteObject`: never sent as `DeleteObjects`, and +/// never gated on `s3_capabilities` at all -- proven here with the capability both explicitly false AND +/// left unknown (the default), since a single physical request is never something that check exists to +/// refuse. This is what makes the CAS engine's per-key fallback (CasGc.cpp) actually delete anything on +/// a backend that rejects `DeleteObjects` outright (GCS): a "batch" of one sent as `DeleteObjects` would +/// fail there identically to a bigger one. +TEST(S3BulkDeleteFallback, ExactlyOneObjectIsAlwaysAPlainDeleteObjectRegardlessOfCapability) +{ + (void)contextForTest(); + + for (const bool explicit_false : {false, true}) + { + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + ASSERT_EQ(request.getMethod(), "DELETE"); + sendDeleteObjectSuccess(response); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{explicit_false ? std::optional{false} : std::nullopt}); + + EXPECT_NO_THROW(storage->removeObjectsIfExistUnderProfile({DB::StoredObject("solo-key")}, DB::ObjectStorageControlRequest{})); + + EXPECT_EQ(server.countMethod("POST"), 0u); + EXPECT_EQ(server.countMethod("DELETE"), 1u); + } +} + +/// The size-one path's own error handling, exactly as thorough as the batch path's: an absence is +/// ignored, and a real error is reported with the object's path. +TEST(S3BulkDeleteFallback, ExactlyOneObjectIgnoresAbsenceAndThrowsOnARealError) +{ + (void)contextForTest(); + + { + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendSingleDeleteError(response, Poco::Net::HTTPResponse::HTTP_NOT_FOUND, "NoSuchKey", "The specified key does not exist."); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + EXPECT_NO_THROW(storage->removeObjectsIfExistUnderProfile({DB::StoredObject("absent-key")}, DB::ObjectStorageControlRequest{})); + } + { + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendSingleDeleteError(response, Poco::Net::HTTPResponse::HTTP_FORBIDDEN, "AccessDenied", "Access Denied"); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + try + { + storage->removeObjectsIfExistUnderProfile({DB::StoredObject("denied-key")}, DB::ObjectStorageControlRequest{}); + FAIL() << "expected removeObjectsIfExistUnderProfile to throw on the AccessDenied key"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::ACCESS_DENIED); + EXPECT_NE(e.message().find("denied-key"), std::string::npos) << e.message(); + } + } +} + +TEST(S3BulkDeleteFallback, OtherFailureClassesKeepFailingClosedWithNoFallback) +{ + (void)contextForTest(); + + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendBatchInternalError(response); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + EXPECT_THROW( + storage->removeObjectsIfExistUnderProfile( + {DB::StoredObject("key-a"), DB::StoredObject("key-b")}, DB::ObjectStorageControlRequest{}), + DB::Exception); + + EXPECT_EQ(server.countMethod("POST"), 1u); + EXPECT_EQ(server.countMethod("DELETE"), 0u) << "an ordinary batch failure must not fall back to per-key deletes"; +} + +TEST(S3BulkDeleteFallback, ExplicitlyDisabledCapabilityThrowsNotImplementedWithoutSendingAnything) +{ + (void)contextForTest(); + + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse &) + { + FAIL() << "an explicit false capability must never send anything, batch or per-key"; + }); + /// `false` in a disk's config resolves to this. + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{/*support_batch_delete_=*/false}); + + expectNotImplemented([&] + { + storage->removeObjectsIfExistUnderProfile( + {DB::StoredObject("key-a"), DB::StoredObject("key-b")}, DB::ObjectStorageControlRequest{}); + }); + + EXPECT_EQ(server.countMethod("POST"), 0u); + EXPECT_EQ(server.countMethod("DELETE"), 0u); +} + +#endif From bbdc54a8599e3336632a89733643f97f4c4d52e4 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 03:03:08 +0200 Subject: [PATCH 46/81] s3: shutdown and startup toggle request processing on the single-attempt client clones too `S3ObjectStorage::shutdown` called `DisableRequestProcessing` only on the main client; the single-attempt clones cached by `getSingleAttemptClient`, which every CAS control-plane verb can run on, were left as they were. Now shutdown and startup apply the toggle to every cached clone under the same lock, and a clone built while shutdown is in effect comes into being already disabled. This is parity with the main client, not a mechanism that bounds a request: the AWS SDK checks the flag only after an attempt has failed, right before deciding whether to retry, and the clones run with zero retries, so the flag cannot prevent a dispatch or interrupt an attempt in flight. What keeps a NEW request of the CAS engine's open plane from reaching a clone after shutdown is admission: `DiskObjectStorage::shutdown` arms `Pool::beginTeardown` through the metadata storage before this object storage is shut down, and `CasOperation` checks that fence before every attempt. The comments say exactly this. Tests: the flag on an existing clone, on a clone created after shutdown, and after startup; a `removeManyWriteOnce` issued after teardown began is refused before it reaches the backend. (squashed from the shutdown part of f8278ca6db5, fbfad2361bd, 0195225f0f4) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ObjectStorages/S3/S3ObjectStorage.cpp | 39 ++++++++++++++++++- .../ObjectStorages/S3/S3ObjectStorage.h | 11 ++++++ .../tests/gtest_cas_gc_teardown_stop.cpp | 30 ++++++++++++++ .../gtest_cas_s3_single_attempt_client.cpp | 33 ++++++++++++++++ src/IO/S3/Client.h | 4 ++ 5 files changed, 116 insertions(+), 1 deletion(-) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index eb430667fa45..388444afec30 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -1105,12 +1105,39 @@ void S3ObjectStorage::shutdown() /// If S3 is healthy nothing wrong will be happened and S3 requests will be processed in a regular way without errors. /// This should significantly speed up shutdown process if S3 is unhealthy. const_cast(*client->get()).DisableRequestProcessing(); + + /// Parity with the main client above, not a stronger guarantee. `DisableRequestProcessing` cannot + /// prevent a request's INITIAL dispatch, and cannot interrupt an attempt already in flight: contrib/aws's + /// AWSClient checks it only after an attempt has already failed and returned, right before deciding + /// whether to retry (AWSClient.cpp, between `ShouldRetry` and the backoff sleep). Every cached clone + /// here runs `SingleAttemptRetryStrategy` (max_retries=0), whose `ShouldRetry` already always says no + /// -- so the flag is still consulted on a clone's failed attempt, it just changes nothing observable + /// there, since no retry was ever going to happen regardless of the flag's value. + /// + /// What actually stops a NEW request on the OPEN plane -- GC, FSCK, the probe; the plane the write-once + /// bulk-delete verb this storage serves runs on -- from being dispatched at all is admission, refused + /// earlier and on a different plane: `DiskObjectStorage::shutdown()` calls `metadata_storage->shutdown()` + /// (which arms `Pool::beginTeardown()`, tripping the open-plane fence `CasPool.cpp` wires to + /// `teardownBegun()`) BEFORE it calls this object storage's `shutdown()`, and `CasOperation::readLoop` + /// (CasRequests.h) checks that fence before every attempt, including the first -- so an open-plane + /// request issued after the disk's shutdown began throws at admission and never reaches a clone at + /// all. The mount and farewell planes are NOT covered by this: they intentionally stay admitting + /// through this same window, since teardown's own drain and farewell I/O run on them. + std::lock_guard lock(single_attempt_client_mutex); + single_attempt_clients_disabled = true; + for (const auto & [_, clone] : single_attempt_clients) + const_cast(*clone).DisableRequestProcessing(); } void S3ObjectStorage::startup() { /// Need to be enabled if it was disabled during shutdown() call. const_cast(*client->get()).EnableRequestProcessing(); + + std::lock_guard lock(single_attempt_client_mutex); + single_attempt_clients_disabled = false; + for (const auto & [_, clone] : single_attempt_clients) + const_cast(*clone).EnableRequestProcessing(); } void S3ObjectStorage::applyNewSettings( @@ -1247,7 +1274,17 @@ std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64 cfg.connectTimeoutMs = cfg.connectTimeoutMs <= 0 ? static_cast(connect_timeout_cap_ms) : std::min(cfg.connectTimeoutMs, static_cast(connect_timeout_cap_ms)); - return single_attempt_clients.emplace(cache_key, base->cloneWithConfigurationOverride(cfg)).first->second; + const auto & clone = single_attempt_clients.emplace(cache_key, base->cloneWithConfigurationOverride(cfg)).first->second; + + /// A fresh clone's own `Aws::Http::HttpClient` starts with request processing enabled regardless of + /// the main client's state; kept in parity with `shutdown()` for the same reason that flag is set + /// there in the first place (see the comment on `shutdown()`) -- this does not, by itself, stop a + /// request already dispatched on this clone, which a single-attempt clone never reaches anyway once + /// admission is refused (see `shutdown()`). + if (single_attempt_clients_disabled) + const_cast(*clone).DisableRequestProcessing(); + + return clone; } std::shared_ptr S3ObjectStorage::clientForRetryProfile(const ObjectStorageControlRequest & request) const diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index fa823b2f7296..1da1a6b38d99 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -275,6 +275,17 @@ class S3ObjectStorage : public IObjectStorage /// released as soon as the next rotation is observed and the clones are dropped — which is what /// makes the identity comparison in getSingleAttemptClient sound. mutable std::shared_ptr single_attempt_client_base; + /// Set for the duration of a `shutdown()` (cleared by the matching `startup()`), kept in parity with + /// the main client's `DisableRequestProcessing`/`EnableRequestProcessing` toggle. Every clone already + /// cached at the moment `shutdown()` runs is disabled there and then, under the same lock; this flag + /// is what makes a clone built afterwards by `getSingleAttemptClient` come into being already + /// disabled too. NOTE: this flag cannot prevent a request's initial dispatch or interrupt one already + /// in flight -- the AWS SDK checks it only after an attempt has failed, right before deciding whether + /// to retry, and every clone here runs `SingleAttemptRetryStrategy` (max_retries=0), whose own answer + /// to that question is already always no. What actually prevents a NEW request on the CAS engine's + /// open plane (GC, FSCK, the probe) from reaching a clone after shutdown is admission, refused + /// earlier at that engine's own fence (see the comment on `S3ObjectStorage::shutdown()`). + mutable bool single_attempt_clients_disabled = false; }; } diff --git a/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp b/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp index b15931e3221d..8b5e31dafff8 100644 --- a/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp +++ b/src/Disks/tests/gtest_cas_gc_teardown_stop.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -238,6 +239,35 @@ TEST(CASGCTeardownStop, OpenPlaneRefusesAfterTeardownBeganAndTheMountPlaneDoesNo EXPECT_EQ(backend->getTotal(), 1u); } +/// `removeManyWriteOnce` -- the verb the CAS GC bulk-delete phases call, and the one an +/// `S3ObjectStorage`-backed pool ultimately dispatches to `removeObjectsIfExistUnderProfile` -- runs on +/// the same open plane as `read` above, so a control-plane bulk delete issued after the disk's shutdown +/// (which arms teardown on this plane before the object storage's own `shutdown()` even runs, see +/// `DiskObjectStorage::shutdown()`) is refused at admission and never reaches the backend at all. +TEST(CASGCTeardownStop, RemoveManyWriteOnceIsRefusedAfterTeardownBeganAndNeverReachesTheBackend) +{ + auto backend = std::make_shared(); + auto store = openPlainPool(backend); + { + CasOperation op = store->openRequests().admit(); + orThrow(op.create("p/probe", "v", Retry::once()), "create"); + } + backend->resetCounts(); + + store->beginTeardown(); + + const Layout layout{"p"}; + const ManifestId manifest_id{RootNamespace{"probe/ns@cas@"}, + ManifestRef{.writer_epoch = 1, .build_sequence = 1, .manifest_ordinal = 1}}; + + CasOperation refused = store->openRequests().admit(); + expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] + { + refused.removeManyWriteOnce({layout.writeOnceManifestKey(manifest_id)}, Retry::standard()); + }); + EXPECT_EQ(backend->deleteTotal(), 0u) << "a refused admission never reaches the backend, not even for one key"; +} + /// The open plane's sleep is the interruptible one, in production wiring and after the test seam is /// cleared. Arming FIRST makes this a wiring test: a predicate `wait_for` whose predicate already /// holds returns without waiting, so a plane still wired to the plain sleep cannot pass. The deadline diff --git a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp index 1e5f70782735..94d2ca0288a9 100644 --- a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp +++ b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp @@ -372,6 +372,39 @@ TEST(S3SingleAttemptClient, ConnectTimeoutIsCappedAndFrozen) EXPECT_EQ(reloaded->getSingleAttemptClient(5000, 1000)->getClientConfiguration().connectTimeoutMs, 1000); } +/// `shutdown()` used to call `DisableRequestProcessing` only on the main client, leaving every cached +/// single-attempt clone (`getSingleAttemptClient`) at its default enabled state. This test verifies only +/// that the flag now propagates to every clone and is restored by `startup()` -- it does NOT prove a +/// disabled clone rejects or interrupts a request: `DisableRequestProcessing` cannot prevent a request's +/// initial dispatch or interrupt one in flight, and a clone that is not currently retrying (every clone +/// here runs `SingleAttemptRetryStrategy`, which never retries) never has an occasion to consult it at +/// all. See the comment on `S3ObjectStorage::shutdown()` for what actually stops a new request after +/// shutdown (CAS engine admission, on a different plane). +TEST(S3SingleAttemptClient, ShutdownDisablesRequestProcessingOnCachedAndFutureClones) +{ + auto storage = makeStorageForTest(20000); + auto clone = storage->getSingleAttemptClient(/*request_timeout_ms=*/5000, /*connect_timeout_cap_ms=*/5000); + ASSERT_TRUE(clone->GetHttpClient()->IsRequestProcessingEnabled()); + + storage->shutdown(); + EXPECT_FALSE(clone->GetHttpClient()->IsRequestProcessingEnabled()) + << "a clone cached before shutdown() ran must have the flag propagated to it"; + + /// A clone for a (timeout, cap) pair never requested before, built WHILE shutdown is in effect, must + /// come into being with the flag already set -- not just the ones that existed when shutdown() ran. + auto clone_after_shutdown = storage->getSingleAttemptClient(/*request_timeout_ms=*/6000, /*connect_timeout_cap_ms=*/6000); + EXPECT_FALSE(clone_after_shutdown->GetHttpClient()->IsRequestProcessingEnabled()) + << "a clone built after shutdown() started must come into being with the flag already set too"; + + storage->startup(); + EXPECT_TRUE(clone->GetHttpClient()->IsRequestProcessingEnabled()); + EXPECT_TRUE(clone_after_shutdown->GetHttpClient()->IsRequestProcessingEnabled()); + + /// The ordinary case: a clone built with no shutdown in effect is enabled from the start. + auto clone_after_startup = storage->getSingleAttemptClient(/*request_timeout_ms=*/7000, /*connect_timeout_cap_ms=*/7000); + EXPECT_TRUE(clone_after_startup->GetHttpClient()->IsRequestProcessingEnabled()); +} + /// The freeze computation `openPoolView` uses to build `pool_config.cas_request_budget.connect_timeout_cap_ms`, /// isolated from any particular verb: the cap is the MIN of the base client's own connect timeout and /// the attempt timeout, a configured-zero base normalizes to the attempt timeout itself (never "no diff --git a/src/IO/S3/Client.h b/src/IO/S3/Client.h index a6cb4972fa97..bf74689138fd 100644 --- a/src/IO/S3/Client.h +++ b/src/IO/S3/Client.h @@ -236,6 +236,10 @@ class Client : private Aws::S3::S3Client using Aws::S3::S3Client::EnableRequestProcessing; using Aws::S3::S3Client::DisableRequestProcessing; + /// Lets a caller (a shutdown-state test, in particular) observe whether Enable/DisableRequestProcessing + /// last took effect on this client's own `Aws::Http::HttpClient`, without exposing the rest of the + /// privately-inherited `Aws::S3::S3Client` surface. + using Aws::S3::S3Client::GetHttpClient; void BuildHttpRequest(const Aws::AmazonWebServiceRequest& request, const std::shared_ptr& httpRequest) const override; From 6dd36566cc829064bd376da8b343ee1ccc0b7917 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 00:30:24 +0200 Subject: [PATCH 47/81] cas: move the CAS error codes out of the range upstream uses `CAS_WRITE_UNATTRIBUTED` and `CAS_DELETE_MARKER` were assigned 1012 and 1013, which upstream ClickHouse already uses for `HANDLER_DOESNT_EXIST` and `AMBIGUOUS_HANDLER`. Move them to 1037/1038, comfortably past upstream's current maximum (1017), so a future merge from upstream does not silently swap the meaning of these codes. Verified no test, doc, or source file references the old numeric literals for these codes; all call sites use the symbolic `ErrorCodes::` names. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit a74d5ae2f9964fb87f1e26f41ce6dcf1641765ac) --- src/Common/ErrorCodes.cpp | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index 299f3bf4d391..b6fe2074ebce 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -679,8 +679,11 @@ M(1009, PENDING_MUTATIONS_NOT_ALLOWED) \ M(1010, EXPORT_PARTITION_ALREADY_EXPORTED) \ M(1011, PARTITION_EXPORT_FAILED) \ - M(1012, CAS_WRITE_UNATTRIBUTED) \ - M(1013, CAS_DELETE_MARKER) \ + /* 1012 and 1013 are intentionally skipped: they collide with upstream ClickHouse's \ + * HANDLER_DOESNT_EXIST and AMBIGUOUS_HANDLER. CAS codes resume at 1037, comfortably \ + * past upstream's current maximum, to leave headroom for future upstream additions. */ \ + M(1037, CAS_WRITE_UNATTRIBUTED) \ + M(1038, CAS_DELETE_MARKER) \ /* See END */ #ifdef APPLY_FOR_EXTERNAL_ERROR_CODES @@ -697,7 +700,7 @@ namespace ErrorCodes APPLY_FOR_ERROR_CODES(M) #undef M - constexpr ErrorCode END = 1013; + constexpr ErrorCode END = 1038; ErrorPairHolder values[END + 1]{}; struct ErrorCodesNames From 90ac6958ad74c1ad8821749c3d98a40a06b35cf7 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 00:31:05 +0200 Subject: [PATCH 48/81] io: ObjectStorageRetryProfile and ObjectStorageControlRequest live in their own header `ReadSettings.h` included `WriteSettings.h` purely to reach these two types, an odd read-depends-on-write coupling. Move them into a new `ObjectStorageRequestProfile.h` that both settings headers include directly, so neither depends on the other for these shared types. `gtest_cas_backend.cpp` picked up `WriteSettings.h` transitively through `ReadSettings.h`; add the direct include now that the chain is gone. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit fb845d11cc9214c82c9713d6c3923ab383a57cdb) --- src/Disks/tests/gtest_cas_backend.cpp | 1 + src/IO/ObjectStorageRequestProfile.h | 31 +++++++++++++++++++++++++++ src/IO/ReadSettings.h | 2 +- src/IO/WriteSettings.h | 23 +------------------- 4 files changed, 34 insertions(+), 23 deletions(-) create mode 100644 src/IO/ObjectStorageRequestProfile.h diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index 68e6f9afa658..5d7604ff7e02 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include diff --git a/src/IO/ObjectStorageRequestProfile.h b/src/IO/ObjectStorageRequestProfile.h new file mode 100644 index 000000000000..185c5c625b3a --- /dev/null +++ b/src/IO/ObjectStorageRequestProfile.h @@ -0,0 +1,31 @@ +#pragma once + +#include +#include + +namespace DB +{ + +/// Per-write retry-behavior selector, resolved by the object storage that executes the write. +/// SingleAttempt: exactly one HTTP attempt, no SDK-transparent retries — for conditional writes +/// whose retry loop lives above the storage client (it must resolve an uncertain PUT before +/// reissuing). Backends without a SingleAttempt implementation report it via +/// IObjectStorage::supportsRetryProfile; writers must fail closed rather than fall through. +enum class ObjectStorageRetryProfile : uint8_t +{ + Default, + SingleAttempt, +}; + +/// What a CAS control request carries into the object storage: the retry profile, the per-attempt +/// budget and connect cap the storage's single-attempt client must honour, and the caller's own +/// attempt number (0 = unset) so the HTTP client sees a reissue as attempt ≥ 2. +struct ObjectStorageControlRequest +{ + ObjectStorageRetryProfile profile = ObjectStorageRetryProfile::Default; + uint64_t attempt_timeout_ms = 0; + uint64_t connect_timeout_cap_ms = 0; + size_t attempt_number = 0; +}; + +} diff --git a/src/IO/ReadSettings.h b/src/IO/ReadSettings.h index 981d5584e6d6..c810c7311402 100644 --- a/src/IO/ReadSettings.h +++ b/src/IO/ReadSettings.h @@ -4,8 +4,8 @@ #include #include #include +#include #include -#include #include #include #include diff --git a/src/IO/WriteSettings.h b/src/IO/WriteSettings.h index 0cb5c0b31321..2e1a8b5c8daf 100644 --- a/src/IO/WriteSettings.h +++ b/src/IO/WriteSettings.h @@ -4,34 +4,13 @@ #include #include #include +#include #include namespace DB { -/// Per-write retry-behavior selector, resolved by the object storage that executes the write. -/// SingleAttempt: exactly one HTTP attempt, no SDK-transparent retries — for conditional writes -/// whose retry loop lives above the storage client (it must resolve an uncertain PUT before -/// reissuing). Backends without a SingleAttempt implementation report it via -/// IObjectStorage::supportsRetryProfile; writers must fail closed rather than fall through. -enum class ObjectStorageRetryProfile : uint8_t -{ - Default, - SingleAttempt, -}; - -/// What a CAS control request carries into the object storage: the retry profile, the per-attempt -/// budget and connect cap the storage's single-attempt client must honour, and the caller's own -/// attempt number (0 = unset) so the HTTP client sees a reissue as attempt ≥ 2. -struct ObjectStorageControlRequest -{ - ObjectStorageRetryProfile profile = ObjectStorageRetryProfile::Default; - uint64_t attempt_timeout_ms = 0; - uint64_t connect_timeout_cap_ms = 0; - size_t attempt_number = 0; -}; - /// Per-copy transport requirement, resolved by the object storage that executes the copy. /// `NativeOnly` requires a provider-native same-store copy and forbids a client-side fallback. enum class ObjectStorageCopyMode : uint8_t From 81ab92c31d9c9bb1c17a7d9ed409fd15c989b5ea Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 00:32:37 +0200 Subject: [PATCH 49/81] object-storage: the new iterate overload stays visible in AzureObjectStorage `AzureObjectStorage` overrides only the 4-argument `iterate`, which hides the 5-argument `IObjectStorage::iterate` (the control-request overload CAS calls) from this class's scope by C++ name-hiding rules. Bring it back with a `using` declaration. Checked `CachedObjectStorage` and `LocalObjectStorage`: neither overrides `iterate`, `tryGetObjectMetadataWithNativeToken`, `removeObjectIfTokenMatches`, or `removeObjectsIfExistUnderProfile`, so they are not affected by this hiding problem. `S3ObjectStorage` overrides both overloads of each name explicitly, so it needs no `using` declaration either. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit c0c8f2d536f28d613b80d158b6a5fd92f4b23002) --- .../ObjectStorages/AzureBlobStorage/AzureObjectStorage.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h index c9abf7339e2d..9c829b5a78fa 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/AzureBlobStorage/AzureObjectStorage.h @@ -49,6 +49,8 @@ class AzureObjectStorage : public IObjectStorage size_t max_keys, bool with_tags, const std::optional & start_after) const override; + /// Overriding one `iterate` overload hides the other from this class's scope; bring both back. + using IObjectStorage::iterate; std::string getName() const override { return "Azure"; } From 93089cdb99d5fc99b019e374cad904e97f663d9f Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 00:32:59 +0200 Subject: [PATCH 50/81] cas: inspect holds the pool for the whole operation `clickhouse-disks cas-inspect` called `ca->store()` twice, once to open the request and once to read the layout. `store()` hands back a snapshot of the pool pointer under a mutex, so two separate calls can observe different pools across a concurrent remount, and the `Layout` reference returned by the second call would then belong to a different `Pool` than the one the read went through. Take one snapshot and reuse it for both. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit 66eaba7afa59245198d43b1a5af1394e0fd6b6cc) --- programs/disks/CommandCaInspect.cpp | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/programs/disks/CommandCaInspect.cpp b/programs/disks/CommandCaInspect.cpp index 80f80324f62e..32df388f2cdb 100644 --- a/programs/disks/CommandCaInspect.cpp +++ b/programs/disks/CommandCaInspect.cpp @@ -48,12 +48,17 @@ class CommandCaInspect final : public ICommand if (!ca->isReadOnly()) throw Exception(ErrorCodes::BAD_ARGUMENTS, "cas-inspect: open the CA disk read-only"); - Cas::CasOperation op = ca->store()->openRequests().admit(); + /// `store()` hands back a snapshot of the pool pointer; `openRequests()`/`layout()` return + /// references into that Pool object. Keeping the shared_ptr alive for the whole operation, + /// rather than letting each `store()` call's temporary expire, is what keeps those references + /// valid and pins both calls to the SAME pool if a concurrent remount swaps it out from under `ca`. + const Cas::PoolPtr pool = ca->store(); + Cas::CasOperation op = pool->openRequests().admit(); const auto got = op.read(key, Cas::Retry::standard()); if (!got) throw Exception(ErrorCodes::BAD_ARGUMENTS, "cas-inspect: key '{}' does not exist", key); - const Cas::Layout & layout = ca->store()->layout(); + const Cas::Layout & layout = pool->layout(); std::optional resolved_life; std::optional life_id; if (const auto parsed = layout.parseRefObjectKey(key)) From 6f16a82b6e02d6f661d5067c4fb51069c5762831 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 00:34:37 +0200 Subject: [PATCH 51/81] docs(cas): document cas_gc_bulk_delete_chunk_keys and the Stopped outcome Add the `cas_gc_bulk_delete_chunk_keys` row to the settings table (default and range match `gc_bulk_delete_chunk_keys` in ContentAddressedSettings.cpp), and note in the monitoring guide that an alert filtering on `outcome = 'Error'` alone misses `Aborted` and `Stopped` rows, both of which are real outcomes worth watching. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit 09f0c2321bbc38b8bba232003103a41a31f6b60c) --- docs/en/antalya/cas/configuration.md | 1 + docs/en/antalya/cas/operations/monitoring.md | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 40e248146684..bf0fe17f7dd3 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -179,6 +179,7 @@ for the remaining caps, `0` means unbounded. |---|---|---|---| | `cas_manifest_sweep_list_budget_keys` | `1000` | `UInt64` | Orphan-manifest sweep `LIST` budget per round | | `cas_manifest_sweep_delete_budget_keys` | `100` | `UInt64` | Orphan-manifest sweep `DELETE` budget per round | +| `cas_gc_bulk_delete_chunk_keys` | `1000` | `1`–`1000` | Keys per batch delete request in GC's write-once families (owner-removed manifest bodies, covered ref logs and snapshots) | | `cas_gc_round_graduation_budget` | `5000` | `0` = unbounded | Blob-graduation (`condemned` → `delete_pending`) cohort cap per round | | `cas_gc_round_redelete_budget` | `5000` | `0` = unbounded | Exact-token re-delete cohort cap for prior `delete_pending` rows per round | | `cas_gc_round_sweep_namespace_budget` | `20` | `0` = unbounded | Distinct namespaces per orphan-manifest sweep page whose protection view may be built | diff --git a/docs/en/antalya/cas/operations/monitoring.md b/docs/en/antalya/cas/operations/monitoring.md index 3261d91891bf..8bde03328575 100644 --- a/docs/en/antalya/cas/operations/monitoring.md +++ b/docs/en/antalya/cas/operations/monitoring.md @@ -124,6 +124,12 @@ changed shard needing a fold and no graduation was due — a cheap round, not a `Finish` row is worth a steady watch: it is fold clamps surfaced and survived, so a non-zero value that persists across rounds is more interesting than an isolated one. +A dashboard alert that filters on `outcome = 'Error'` alone misses `Aborted` and `Stopped` rows too +— see the [`outcome` column](/operations/system-tables/cas_gc_log#columns) for what each one means. +A round that is recurring `Aborted` rather than `Error` still deserves attention: it keeps retrying, +but the underlying transient condition (backend unavailability, a lost lease, a competing leader) +has not gone away. + Which phase dominates round duration or the `LIST` budget — reproduced from the [per-phase rows](/operations/system-tables/cas_gc_log#per-phase-rows) reference: From 15f4fd89f60db62abc380996f51b9db5520aeb6e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 00:46:13 +0200 Subject: [PATCH 52/81] cas: say what happens to the keys of a bulk-delete chunk that exhausts its retries The comment covered the chunks before and after a failing chunk but not the failing chunk's own keys: deletion and recording are all-or-nothing per request, so a key deleted by one of that chunk's attempts is not recorded in this round and shows up as already gone in the next fold. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit dbd558fad504c0bb848d8ceb1591596dac21ba41) --- .../MetadataStorages/ContentAddressed/Gc/CasGc.cpp | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index 5083c822a10e..336ab21fbefa 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -1116,8 +1116,10 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al /// Chunks of write-once keys, one request each, with no per-key precondition: a manifest key is /// never written twice, so the body at it is the one the fold observed or nothing. The engine /// reissues a failed chunk whole; a chunk that exhausts its policy throws here, and the chunks - /// before it are already recorded below. The etag the fold observed rides the event as - /// information only. + /// before it are already recorded below. A key that one of the exhausted chunk's own attempts + /// did delete is not recorded either: deletion and recording are all-or-nothing per request, + /// never per key, so the next round's fold sees that key as already gone. The etag the fold + /// observed rides the event as information only. const size_t chunk_keys = std::clamp(store->poolConfig().gc_bulk_delete_chunk_keys, 1, kBulkDeleteMaxKeys); uint64_t attempted = 0; uint64_t requests = 0; From e5605ee72153ff0fd30759b74cddec0783ff6f79 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 01:56:54 +0200 Subject: [PATCH 53/81] s3: the readObject refresh-callback comment names what the copied callback may capture "Never `this`" was true only of this storage's own pointer: the callback copied into the buffer can capture a shorter-lived object (the StorageS3Configuration refresher does), so the lifetime obligation moves to the caller and the comment says so. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit 6e213305169ddf0057f8f532d0589efb892dbdd6) --- .../ObjectStorages/S3/S3ObjectStorage.cpp | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 388444afec30..d6f8df65b1a0 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -336,9 +336,12 @@ std::unique_ptr S3ObjectStorage::readObject( /// NOLINT request_settings[S3RequestSetting::max_single_read_retries] = 1; if (credentials_refresh_callback) { - /// Captures the client SLOT and a copy of the callback, never `this`: the buffer this - /// returns can outlive the storage, and a credential expiry firing afterwards would - /// otherwise install a fresh client into a destroyed object. + /// Captures the client SLOT and a copy of the caller's refresh callback, never this + /// storage's `this`: the buffer this returns can outlive the storage, and a credential + /// expiry firing afterwards would otherwise install a fresh client into a destroyed + /// object. The copied callback can itself capture a shorter-lived object -- + /// `StorageS3Configuration::createObjectStorage`'s refresher captures the configuration + /// it was built from -- so the caller must keep that object alive as long as the buffer. refresh_callback = [client_slot = client, refresh = credentials_refresh_callback]() -> std::unique_ptr { From 9271b16847d554629b9ec2004b06593ee39da3e8 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 03:08:48 +0200 Subject: [PATCH 54/81] cas: gtest pool hooks own their clock and sleep state instead of referencing test-frame locals A Pool can outlive the test frame that configured it (a background publish holds `shared_from_this()`), and its deferred teardown on another thread then calls the `boot_ms_fn` / `wait_sleep_fn` / retry-sleep hooks; ASan reported a stack-use-after-return in the next test when such a hook dereferenced a local of the previous one (gtest_cas_ref_writer.cpp, the frozen-clock publish test). Every hook in this file now owns its state: captured by value where the value never changes after the hook is installed, otherwise a shared atomic created before the pool and captured by value, with the test body reading and writing through it. Assertions are unchanged. ASan CASRefWriter* five consecutive runs 109/109 with no reports; ASan gate CAS*/S3/ObjectStorage/Teardown 2700/2701 (one pre-existing skip); release CAS* 2499/2499. The same capture shape remains in ten other CAS gtests (census in the backlog). (squashed from a726e933419 and the retry-sleep follow-up) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_ref_writer.cpp | 109 ++++++++++++++--------- 1 file changed, 69 insertions(+), 40 deletions(-) diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index 2f3d22ca1a37..5835f1867010 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -2825,7 +2825,10 @@ TEST(CASRefWriterSnapshotPublish, TriggerFiresOnCountAboveThresholdWithoutAging) PoolConfig config; config.snapshot_log_count_threshold = 3; config.snapshot_log_bytes_threshold = 1ULL << 40; - config.boot_ms_fn = [&fake_now] { return fake_now; }; + /// Captured by value: the value never changes, and the Pool can outlive this stack frame (a + /// background publish holds `shared_from_this()`), so a by-reference capture of `fake_now` would + /// dangle once the frame returns. + config.boot_ms_fn = [fake_now] { return fake_now; }; config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); auto store = openPoolWithConfig(backend, config); @@ -3092,7 +3095,9 @@ TEST(CASRefWriterSnapshotPublish, C4LatchBoundedUnderSustainedNonCommittedPublis config.snapshot_log_bytes_threshold = 1ULL << 40; config.snapshot_publish_backoff_initial_ms = 5000; /// the frozen clock keeps the backoff armed config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + /// Captured by value: `fake_now` never changes and the Pool can outlive this stack frame (a + /// background publish holds `shared_from_this()`), so a by-reference capture would dangle. + config.boot_ms_fn = [fake_now] { return fake_now; }; config.cas_request_budget = budget; auto store = openPoolWithConfig(backend, config); /// The boot clock above is frozen (it is what keeps the publish backoff armed), so the REQUEST @@ -3227,13 +3232,16 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) const CasRequestBudget budget = wedgeTestBudget(); - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: this test mutates the clock after the Pool exists + /// (below), and the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config; config.snapshot_log_count_threshold = 1; config.snapshot_log_bytes_threshold = 1ULL << 40; config.snapshot_publish_backoff_initial_ms = 1000; config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] { return fake_now->load(); }; config.cas_request_budget = budget; auto store = openPoolWithConfig(backend, config); /// As above: the frozen boot clock drives the backoff decisions, so the request engine gets its @@ -3264,7 +3272,7 @@ TEST(CASRefWriterSnapshotPublish, C4BackoffDefersThenRetriesAndPublishes) /// Advance past the backoff, with the fault cleared: exactly one retry is dispatched and it publishes. backend->disarmFaults(); - fake_now += 2000; + *fake_now += 2000; store->resolveRef(ns, "a"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(global_counters[ProfileEvents::CASRefSnapshotPublishDispatched].load(), d1 + 1) @@ -3467,10 +3475,12 @@ TEST(CASRefWriterStalePrecommitSweep, BoundedBatchesAndInterruptionResumeAcrossM /// pays a real ~36.5s token-stability observation wait here. Inject a fake `boot_ms_fn` + /// `wait_sleep_fn` (mirroring `CASMountOpenWaits.UncleanOpenPaysOnlyTheObservationWindow`) so it /// resolves instantly. - uint64_t resumer_fake_boot = 0; + /// Held in a shared atomic, not a plain local: the Pool can outlive this stack frame (a background + /// publish holds `shared_from_this()`), so a by-reference capture of a local would dangle. + auto resumer_fake_boot = std::make_shared>(0); PoolConfig resumer_config; - resumer_config.boot_ms_fn = [&resumer_fake_boot] { return resumer_fake_boot; }; - resumer_config.wait_sleep_fn = [&resumer_fake_boot](uint64_t ms) { resumer_fake_boot += ms; }; + resumer_config.boot_ms_fn = [resumer_fake_boot] { return resumer_fake_boot->load(); }; + resumer_config.wait_sleep_fn = [resumer_fake_boot](uint64_t ms) { *resumer_fake_boot += ms; }; auto resumer = openPoolWithConfig(backend, resumer_config); EXPECT_NO_THROW(resumer->listRefs(ns)); @@ -3519,11 +3529,14 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) const Layout layout("p"); const RootNamespace ns{"srv1/precommit_sweep_retry"}; - /// One shared injected clock for both incarnations. The successor's wait hook below advances this - /// same clock, so both mount observation and the later sweep-backoff deadline are deterministic. - uint64_t fake_now = 1'000'000; - size_t mount_wait_calls = 0; - const auto fake_clock = [&fake_now] { return fake_now; }; + /// One shared injected clock for both incarnations, held in a shared atomic rather than a plain + /// local: the successor Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. The successor's wait + /// hook below advances this same clock, so both mount observation and the later sweep-backoff + /// deadline are deterministic. + auto fake_now = std::make_shared>(1'000'000); + auto mount_wait_calls = std::make_shared>(0); + const auto fake_clock = [fake_now] { return fake_now->load(); }; { /// A predecessor writer leaves THREE precommits dangling (a crash before promote). @@ -3549,14 +3562,14 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) config.cas_request_budget = budget; config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); config.boot_ms_fn = fake_clock; - config.wait_sleep_fn = [&fake_now, &mount_wait_calls](uint64_t ms) + config.wait_sleep_fn = [fake_now, mount_wait_calls](uint64_t ms) { - ++mount_wait_calls; - fake_now += ms; + ++(*mount_wait_calls); + *fake_now += ms; }; SynchronizedEventLog seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) auto successor = openPoolWithConfig(backend, config); - EXPECT_GT(mount_wait_calls, 0u) + EXPECT_GT(mount_wait_calls->load(), 0u) << "the unclean predecessor must exercise the injected mount-observation wait"; successor->setEventSink([&](const CasEvent & e) { seen.add(e); }); @@ -3600,7 +3613,7 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) /// mutation this time) retries: the lane resolves its wedge (the first chunk's removals become /// durable and applied), the re-pass verifies clean, and the flag clears permanently. backend->materializePendingDelayedWrite(); - fake_now += 60'000; /// beyond any armed backoff (initial 200 ms, max 30 s) + *fake_now += 60'000; /// beyond any armed backoff (initial 200 ms, max 30 s) EXPECT_NO_THROW(publishEmptyPart(successor, ns, "fresh")); EXPECT_FALSE(successor->refLaneWedgedForTest(ns)); EXPECT_FALSE(successor->needsStalePrecommitSweepForTest(ns)) @@ -3869,12 +3882,14 @@ TEST(CASRefWriterRemount, DiscardsWedgeAndLaneRemainsUsable) auto backend = std::make_shared(); /// The self-remount below blocks on nothing (see /// `CASRemountWaits.UnresolvedWedgeRemountPaysNoWaitEither`, `gtest_cas_pool.cpp`); the injected - /// `boot_ms_fn`/`wait_sleep_fn` keep this test off the real clock anyway. - uint64_t fake_boot = 0; + /// `boot_ms_fn`/`wait_sleep_fn` keep this test off the real clock anyway. Held in a shared atomic, + /// not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(0); PoolConfig config; config.cas_request_budget = budget; - config.boot_ms_fn = [&fake_boot] { return fake_boot; }; - config.wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }; + config.boot_ms_fn = [fake_boot] { return fake_boot->load(); }; + config.wait_sleep_fn = [fake_boot](uint64_t ms) { *fake_boot += ms; }; auto store = openPoolWithConfig(backend, config); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/remount_wedge"}; @@ -3916,11 +3931,13 @@ TEST(CASRefWriterRemount, SupersededLeaderMidFlushFailsClosedCreatesNoObject) CasRequestBudget budget; budget.attempt_timeout_ms = 100; budget.lease_safety_margin_ms = 100; - uint64_t fake_boot = 0; + /// Held in a shared atomic, not a plain local: the Pool can outlive this stack frame (a background + /// publish holds `shared_from_this()`), so a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(0); PoolConfig config; config.cas_request_budget = budget; - config.boot_ms_fn = [&fake_boot] { return fake_boot; }; - config.wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }; + config.boot_ms_fn = [fake_boot] { return fake_boot->load(); }; + config.wait_sleep_fn = [fake_boot](uint64_t ms) { *fake_boot += ms; }; auto store = openPoolWithConfig(backend, config); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/remount_midflush"}; @@ -5359,7 +5376,10 @@ TEST(CASRefWriterRecoveryRetry, TransientSealFailureIsRetriedThenSucceeds) config.cas_request_budget.recovery_retry_budget_ms = 120000; config.cas_request_budget.recovery_retry_initial_backoff_ms = 1000; config.cas_request_budget.recovery_retry_max_backoff_ms = 30000; - config.boot_ms_fn = [&fake_now] { return fake_now; }; + /// Captured by value: `fake_now` stays frozen for the whole test (see below), and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture would dangle. + config.boot_ms_fn = [fake_now] { return fake_now; }; config.wait_sleep_fn = [](uint64_t) {}; auto store = openPoolWithConfig(backend, config); ASSERT_TRUE(store); @@ -5412,7 +5432,10 @@ TEST(CASRefWriterRecoveryRetry, RecoveryDoesNotEnumerateItsStream) config.mount_lease_ttl_ms = std::chrono::milliseconds(500); config.cas_request_budget = sealTestTinyBudget(); config.cas_request_budget.recovery_retry_budget_ms = 120000; - config.boot_ms_fn = [&fake_now] { return fake_now; }; + /// Captured by value: `fake_now` is never mutated in this test, and the Pool can outlive this + /// stack frame (a background publish holds `shared_from_this()`), so a by-reference capture would + /// dangle. + config.boot_ms_fn = [fake_now] { return fake_now; }; config.wait_sleep_fn = [](uint64_t) {}; auto store = openPoolWithConfig(backend, config); ASSERT_TRUE(store); @@ -5445,7 +5468,10 @@ TEST(CASRefWriterRecoveryRetry, TransientFailureLongerThanBudgetPropagates) seedSealFixtureDeadEpochs(backend, layout, ns); seedUncleanPredecessorMount(backend, layout, /*epoch=*/2); - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: this test mutates the clock via the retry-sleep + /// hook below, and the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config; config.server_id = UInt128(1); @@ -5456,11 +5482,11 @@ TEST(CASRefWriterRecoveryRetry, TransientFailureLongerThanBudgetPropagates) config.cas_request_budget.recovery_retry_budget_ms = 5000; /// small, deterministic config.cas_request_budget.recovery_retry_initial_backoff_ms = 1000; config.cas_request_budget.recovery_retry_max_backoff_ms = 30000; - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] { return fake_now->load(); }; config.wait_sleep_fn = [](uint64_t) {}; auto store = openPoolWithConfig(backend, config); ASSERT_TRUE(store); - store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); + store->setCasRetrySleepForTest([fake_now](uint64_t ms) { *fake_now += ms; }); /// The seal is an in-band LOG transaction at the slot after the dead epoch's last durable id, not a /// snapshot at a synthetic id: epoch 1 closes at `{1,2}`, which is the FIRST write the walk attempts. @@ -5488,8 +5514,9 @@ TEST(CASRefWriterRecoveryRetry, NonNetworkErrorIsNotRetried) auto store = openPoolWithConfig(backend, config); ASSERT_TRUE(store); - size_t sleep_calls = 0; - store->setCasRetrySleepForTest([&sleep_calls](uint64_t) { ++sleep_calls; }); + /// Owned by the closure: the pool may outlive this frame and retry a farewell request. + auto sleep_calls = std::make_shared>(0); + store->setCasRetrySleepForTest([sleep_calls](uint64_t) { ++*sleep_calls; }); /// A foreign writer lands DIFFERENT valid bytes at the seal key; resolve-before-reissue then throws /// CORRUPTED_DATA (a real cross-process seal conflict), which must NOT be retried. @@ -5500,7 +5527,7 @@ TEST(CASRefWriterRecoveryRetry, NonNetworkErrorIsNotRetried) expectThrowsCode(DB::ErrorCodes::CORRUPTED_DATA, [&] { store->listRefs(ns); }); EXPECT_EQ(backend->corrupt_count, 0) << "the test must reach the injected foreign seal conflict, not fail on fixture validation"; - EXPECT_EQ(sleep_calls, 0u) << "a non-transient error must fail fast with zero backoff sleeps"; + EXPECT_EQ(sleep_calls->load(), 0u) << "a non-transient error must fail fast with zero backoff sleeps"; } TEST(CASRefWriterRecoveryRetry, VanishBrakeStaysTerminalNotRetried) @@ -5529,8 +5556,9 @@ TEST(CASRefWriterRecoveryRetry, VanishBrakeStaysTerminalNotRetried) auto store = openPool(backend); - size_t sleep_calls = 0; - store->setCasRetrySleepForTest([&sleep_calls](uint64_t) { ++sleep_calls; }); + /// Owned by the closure: the pool may outlive this frame and retry a farewell request. + auto sleep_calls = std::make_shared>(0); + store->setCasRetrySleepForTest([sleep_calls](uint64_t) { ++*sleep_calls; }); /// A checkpoint-named snapshot belongs to the caller's immutable authority cut. If that exact /// object is absent, recovery must report corruption immediately; it must neither reinterpret a @@ -5547,7 +5575,7 @@ TEST(CASRefWriterRecoveryRetry, VanishBrakeStaysTerminalNotRetried) << "the test must reach the checkpoint-named snapshot GET, not fail on earlier fixture validation"; EXPECT_EQ(global_counters[ProfileEvents::CASRefRecoveryRetries].load(), retries_before) << "missing immutable checkpoint authority is terminal; the outer transient-retry loop must NOT re-drive it"; - EXPECT_EQ(sleep_calls, 0u) << "no backoff sleep for missing immutable checkpoint authority"; + EXPECT_EQ(sleep_calls->load(), 0u) << "no backoff sleep for missing immutable checkpoint authority"; } TEST(CASRefWriterRecoveryRetry, ThrowingBackoffSleepDoesNotWedgeRecovery) @@ -5575,10 +5603,11 @@ TEST(CASRefWriterRecoveryRetry, ThrowingBackoffSleepDoesNotWedgeRecovery) /// First touch: the seal create fails transiently and the next thing either loop does is sleep on /// this one seam -- the write engine's reissue pause is simply the first to reach it -- so the throw /// lands while `recovery_in_progress` is set, which is the state this test is about. - bool sleep_should_throw = true; - store->setCasRetrySleepForTest([&sleep_should_throw](uint64_t) + /// Owned by the closure: the pool may outlive this frame and retry a farewell request. + auto sleep_should_throw = std::make_shared>(true); + store->setCasRetrySleepForTest([sleep_should_throw](uint64_t) { - if (sleep_should_throw) + if (sleep_should_throw->load()) throw std::runtime_error("injected backoff-sleep failure"); }); const RefTxnId seal_id{1, 2}; @@ -5590,7 +5619,7 @@ TEST(CASRefWriterRecoveryRetry, ThrowingBackoffSleepDoesNotWedgeRecovery) /// The lane must NOT be wedged: with the fault now spent and the sleep no longer throwing, a second /// touch recovers cleanly. If recovery_in_progress had leaked (SCOPE_EXIT run unlocked / not run), a /// concurrent-safe second recovery would deadlock or mis-behave. - sleep_should_throw = false; + sleep_should_throw->store(false); EXPECT_EQ(store->listRefs(ns).size(), 2u) << "a second touch must recover; the retry lane is not wedged"; } From 112a2018a4c347bf49eae272c573e65f254c67a4 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 07:52:13 +0200 Subject: [PATCH 55/81] ci: move the local-run tooling fixes off this branch ci/jobs/functional_tests.py, ci/jobs/scripts/clickhouse_proc.py and utils/c++expr are local-run/tooling fixes unrelated to content-addressed storage; restoring them to altinity/antalya-26.6's version here so this branch only carries CAS changes. They now live on ci/local-run-fixes (commit 165f1bc9110), branched off altinity/antalya-26.6. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- ci/jobs/functional_tests.py | 8 -------- ci/jobs/scripts/clickhouse_proc.py | 9 +-------- utils/c++expr | 4 +--- 3 files changed, 2 insertions(+), 19 deletions(-) diff --git a/ci/jobs/functional_tests.py b/ci/jobs/functional_tests.py index 227bc50bfc17..b73f6f308211 100644 --- a/ci/jobs/functional_tests.py +++ b/ci/jobs/functional_tests.py @@ -506,14 +506,6 @@ def main(): # for local run check if stateful tests are present to skip prepare_stateful_data and start faster if not has_stateful_tests = True - if info.is_local_run and not tests: - # A local run of the WHOLE suite cannot prepare the stateful datasets: `create.sql` attaches - # them from a web disk on `dockerhub-proxy.dockerhub-proxy-zone`, which resolves only inside - # CI, so the step dies with DNS_ERROR before a single test runs. Skipping it lets the - # stateless suite run locally; the tests that genuinely need `test.hits`/`test.visits` fail - # and are triaged as environment rather than taking the whole job with them. - print("Local full-suite run: skipping stateful data preparation (datasets are CI-hosted)") - has_stateful_tests = False if tests and info.is_local_run: from glob import glob diff --git a/ci/jobs/scripts/clickhouse_proc.py b/ci/jobs/scripts/clickhouse_proc.py index cbcdb2e0a3e0..d4ce6313bf24 100644 --- a/ci/jobs/scripts/clickhouse_proc.py +++ b/ci/jobs/scripts/clickhouse_proc.py @@ -533,11 +533,6 @@ def start(self, replica_num=0): os.environ["LLVM_PROFILE_FILE"] = "ft-server-%m.profraw" env = os.environ.copy() - # Leave ASAN_OPTIONS at the sanitizer defaults here. Under ASan, a stateless run's resident set is - # proportional to the number of live server threads (each gets a fake stack via - # `detect_stack_use_after_return`), so when an ASan lane hits the memory ceiling, check the thread - # count first (`system.metrics` GlobalThread / LocalThread, `system.stack_trace` grouped by - # thread_name) rather than tuning ASan away from its defaults. env["TSAN_OPTIONS"] = " ".join( filter( lambda x: x is not None, @@ -844,9 +839,7 @@ def prepare_stateful_data(self, with_s3_storage, is_db_replicated): command = bootstrap_vars + command if with_s3_storage: command = "USE_S3_STORAGE_FOR_MERGE_TREE=1\n" + command - # verbose: this step loads the stateful datasets and it is the only place in the job - # that can fail without printing anything at all, which is exactly what happened. - return Shell.check(command, verbose=True) + return Shell.check(command) def insert_system_zookeeper_config(self): for _ in range(10): diff --git a/utils/c++expr b/utils/c++expr index 54c44c8ecdbd..08ddff2e1363 100755 --- a/utils/c++expr +++ b/utils/c++expr @@ -235,9 +235,7 @@ size_t max_tests = $BENCHMARK_TESTS; size_t max_steps = $BENCHMARK_STEPS; $GLOBAL -/// Internal linkage: the ClickHouse build enables -Werror,-Wmissing-prototypes, which rejects a -/// definition of an external function that has no preceding declaration. -static int work(int thread_id = 0) { +int work(int thread_id = 0) { (void)thread_id; try { EOF From 1ca3d621b2f531c258448b90e880ac3bb87ba168 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 08:04:59 +0200 Subject: [PATCH 56/81] fetch: CAS routing comments keep the reason and drop the provenance Rewrite the DataPartsExchange comments that cited a spec section, an all-tree/B66b/B7 task id, a review revision, a BACKLOG anchor, a GitHub issue number, or an internal codename (codex-6, M-W design section 4, CaRelinkConfirmCore.tla). Every one of them already stated the reason the code exists; only the citation of where that reason was written down is removed. No code changes. git diff --stat altinity/antalya-26.6...HEAD -- src/Storages/MergeTree/DataPartsExchange.cpp src/Storages/MergeTree/DataPartsExchange.h (before this commit): src/Storages/MergeTree/DataPartsExchange.cpp | 208 +++++++++++++++------ src/Storages/MergeTree/DataPartsExchange.h | 4 + (after this commit, same two files): src/Storages/MergeTree/DataPartsExchange.cpp | 174 ++++++++++++------ src/Storages/MergeTree/DataPartsExchange.h | 4 + Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Storages/MergeTree/DataPartsExchange.cpp | 68 ++++++++++---------- src/Storages/MergeTree/DataPartsExchange.h | 8 +-- 2 files changed, 38 insertions(+), 38 deletions(-) diff --git a/src/Storages/MergeTree/DataPartsExchange.cpp b/src/Storages/MergeTree/DataPartsExchange.cpp index 487e3654216a..12052535735b 100644 --- a/src/Storages/MergeTree/DataPartsExchange.cpp +++ b/src/Storages/MergeTree/DataPartsExchange.cpp @@ -108,7 +108,7 @@ constexpr auto REPLICATION_PROTOCOL_VERSION_WITH_COLUMNS_SUBSTREAMS = 9; /// 10 is a version peers still advertise, and deleting the record of what it meant would leave the next /// reader unable to tell what an incoming 10 promises (a relink it will NOT confirm). [[maybe_unused]] constexpr auto REPLICATION_PROTOCOL_VERSION_WITH_CA_RELINK = 10; -/// CAS replication, publish-then-confirm (spec §wire-protocol). A relink offer is now accompanied by a +/// CAS replication, publish-then-confirm. A relink offer is now accompanied by a /// source token, and the endpoint answers a second, part-less request that asks whether that token is /// still exactly what the sender's ref names. A server advertising this version serves the confirm /// action; a receiver advertising it must confirm before it promotes. @@ -128,15 +128,15 @@ constexpr auto CA_POOL_UUID_PARAM = "cas_pool_uuid"; /// (the opaque encoded PartManifest body — self-contained, see part_manifest_v2 below) instead of the /// byte stream. constexpr auto CA_RELINK_COOKIE = "cas_relink"; -/// All-tree task 7: the manifest is now self-contained (uuid.txt/metadata_version.txt are ordinary -/// manifest entries, task 6), so the wire payload dropped its trailing metadata_version field (the +/// The manifest is now self-contained (uuid.txt/metadata_version.txt are ordinary +/// manifest entries), so the wire payload dropped its trailing metadata_version field (the /// manifest bytes are now the ONLY field). Bumped from `part_manifest_v1` so a mixed-build pair (old /// sender, new receiver) does not try to parse the old two-field payload under the new one-field shape /// — the receiver rejects a cookie value it does not recognize and falls back to a byte fetch instead /// of desyncing on the wire format. constexpr auto CA_RELINK_COOKIE_VALUE = "part_manifest_v2"; -/// CAS fetch-by-relink, publish-then-confirm (spec §wire-protocol). Three names make up the second +/// CAS fetch-by-relink, publish-then-confirm. Three names make up the second /// request of the handshake. /// /// The request parameter both selects the confirm action and carries its only argument: the opaque @@ -163,7 +163,7 @@ constexpr auto CA_CONFIRM_ANSWER_UNPROVEN = "unproven"; /// Resolve a disk to the content-addressed exchange facade, or nullptr if the disk is not CA. The /// cast targets the purpose-built INTERFACE (IContentAddressedExchange), never the concrete -/// metadata-storage class (M-W design section 4). Used by both the relink sender (the part's +/// metadata-storage class. Used by both the relink sender (the part's /// disk) and the relink receiver (the target disk). IContentAddressedExchange * tryGetContentAddressedExchange(const DiskPtr & disk) { @@ -226,7 +226,7 @@ CasConfirmAnswer Service::resolveContentAddressedConfirm( const String & part_name, const String & manifest_ref_text) const { - /// CAS fetch-by-relink, publish-then-confirm (spec §confirm-primitive). The receiver's own `+1` is + /// CAS fetch-by-relink, publish-then-confirm. The receiver's own `+1` is /// already durable when this runs; a `Yes` is what authorizes it to promote a part whose blobs are /// protected only by THIS server's committed binding of that exact manifest. Every field below comes /// from a remote peer, so nothing here is trusted beyond being used as a lookup key. @@ -253,8 +253,8 @@ CasConfirmAnswer Service::resolveContentAddressedConfirm( return CasConfirmAnswer::Unknown; const IContentAddressedExchange * matched = tryGetContentAddressedExchange(routing_disks[*routed]); - /// Gate 0 — the part-anchored fast filter. It is an AVAILABILITY filter and never a proof (spec - /// §confirm-primitive, demoted in rev.5): `rollbackDeletingParts` puts a part back to `Outdated` + /// Gate 0 — the part-anchored fast filter. It is an AVAILABILITY filter and never a proof: + /// `rollbackDeletingParts` puts a part back to `Outdated` /// after a failed filesystem removal, and the in-memory part path is deliberately not updated by a /// `delete_tmp_*` rename, so an `Active`/`Outdated` part object authorizes nothing. What it buys is /// a cheap `No` that costs no ledger work; every `Yes` is earned by gate 1 alone. @@ -318,7 +318,7 @@ void Service::answerContentAddressedConfirm(const String & token_text, HTTPServe void Service::processQuery(const HTMLForm & params, ReadBufferPtr body, WriteBuffer & out, HTTPServerResponse & response) { - /// CAS fetch-by-relink, publish-then-confirm (spec §wire-protocol): the second request of the + /// CAS fetch-by-relink, publish-then-confirm: the second request of the /// handshake, dispatched before `part` is required because a confirm carries none — the part name /// is inside the token. Authentication parity with the fetch is inherent: the shared handler /// authenticates before it dispatches to any endpoint. @@ -405,7 +405,7 @@ void Service::processQuery(const HTMLForm & params, ReadBufferPtr body, WriteBuf writeBinary(projections.size(), out); } - /// CAS replication 2b — fetch-by-relink (spec §4). If the part is on a content-addressed disk and + /// CAS replication — fetch-by-relink. If the part is on a content-addressed disk and /// the pool of the disk this part sits on is among the pools the receiver advertised in /// `cas_pool_uuid`, send only the part's content id + the mutable header — no file bytes — so /// the receiver can "fetch" by publishing its own ref to the blobs already in the shared pool. @@ -438,8 +438,8 @@ void Service::processQuery(const HTMLForm & params, ReadBufferPtr body, WriteBuf LOG_DEBUG(log, "Sending part {} by relink (content-addressed, shared pool {}), manifest payload {} bytes", part_name, matched_pool, offer->manifest_bytes.size()); response.addCookie({CA_RELINK_COOKIE, CA_RELINK_COOKIE_VALUE}); - /// The source token for the confirm request the receiver makes before it promotes - /// (spec §wire-protocol). It always accompanies the offer, and its ABSENCE is what + /// The source token for the confirm request the receiver makes before it promotes. + /// It always accompanies the offer, and its ABSENCE is what /// tells a confirm-capable receiver that this sender predates the handshake. response.addCookie({CA_CONFIRM_TOKEN_COOKIE, offer->confirm_token}); /// Which of the advertised pools this offer is for. A receiver with one pool does not @@ -449,11 +449,11 @@ void Service::processQuery(const HTMLForm & params, ReadBufferPtr body, WriteBuf fiu_do_on(FailPoints::cas_relink_sender_omit_pool_cookie, { omit_pool_cookie = true; }); if (!omit_pool_cookie) response.addCookie({CA_POOL_UUID_PARAM, matched_pool}); - /// The relink payload (B7 part_manifest_v2, all-tree task 7): the opaque encoded + /// The relink payload (`part_manifest_v2`): the opaque encoded /// PartManifest body (the receiver decodes it, ignores the sender identity, and /// stages its OWN local manifest over the shared-pool blobs; the legacy part_id wire /// field carries it). Self-contained: uuid.txt/metadata_version.txt are ordinary - /// manifest entries now (task 6), so no separate mutable-header field is sent. + /// manifest entries now, so no separate mutable-header field is sent. writeStringBinary(offer->manifest_bytes, out); data.addLastSentPart(part->info); return; @@ -959,7 +959,7 @@ std::pair Fetcher::fetchSelected if (server_protocol_version >= REPLICATION_PROTOCOL_VERSION_WITH_PARTS_PROJECTION) readBinary(projections, *in); - /// CAS replication 2b — fetch-by-relink (spec §4; B7 part_manifest_v2, all-tree task 7). The sender + /// CAS replication — fetch-by-relink (`part_manifest_v2`). The sender /// chose to relink: it sent only the part's encoded PartManifest body, no file bytes, and the /// reservation above already went to the offered pool's disk. Build the part by staging this /// server's OWN local manifest over the blobs already in the shared pool (adopt-by-hash -> revalidate @@ -973,7 +973,7 @@ std::pair Fetcher::fetchSelected /// Re-request without the relink capability: pass the SAME (CA) disk but disable zero-copy/relink /// so the sender streams bytes; on CA the downloaded files content-address and dedup. /// - /// THE RECURSION BRAKE (B66b). `allow_ca_relink=false` is what bounds this: the re-request does + /// THE RECURSION BRAKE. `allow_ca_relink=false` is what bounds this: the re-request does /// not advertise the pool identity, so the sender cannot offer relink again, so this lambda /// cannot be reached a second time for the same fetch. Before relink had its own capability the /// brake was implicit in `try_zero_copy=false`; with the two decoupled it has to be spelled out, @@ -1025,7 +1025,7 @@ std::pair Fetcher::fetchSelected readStringBinary(sender_manifest_bytes, *in); assertEOF(*in); - /// Publish-then-confirm (spec §core-idea) happens inside `relinkPartToDisk`, including the second + /// Publish-then-confirm happens inside `relinkPartToDisk`, including the second /// interserver request; the token cookie is the sender's offer identity and is opaque here. A /// `nullptr` means the mechanism cannot work but the sender still has the part, so the byte /// re-request below is sound; a THROW means the source did not prove the binding, and the whole @@ -1407,8 +1407,8 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// re-request goes back to the very source whose state is in doubt. That code, deliberately: /// both queue executors (`processQueueEntry`, `ReplicatedMergeMutateTaskBase::executeStep`) /// demote it to INFO with no stack trace -- a refusal is the designed outcome of racing a source -/// whose ref moved on, not a network fault (issue #2219 records a multi-hour false triage chasing -/// that label) -- yet, unlike `ABORTED`, it still records the exception on the queue entry, so a +/// whose ref moved on, not a network fault, and logging it as an error invites exactly that +/// misdiagnosis -- yet, unlike `ABORTED`, it still records the exception on the queue entry, so a /// refusal storm stays visible in `system.replication_queue`. It is also the one fetch-transient /// code the stateless corpus already tolerates in `part_log` checks (e.g. `02265_column_ttl`). /// Lose a part? No -- the queue stores the exception, backs off, and re-executes the entry, which @@ -1453,7 +1453,7 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// the source itself. `adoptPartFromManifest` used to collapse the two by catching every `Exception` /// and returning `false`. /// -/// B66b — WHAT CHANGES WHEN THE TARGET IS `detached/`. Every row above still holds, and the two columns +/// WHAT CHANGES WHEN THE TARGET IS `detached/`. Every row above still holds, and the two columns /// that matter are unchanged in every one of them, but two rows hold for a DIFFERENT reason and that /// difference is worth stating rather than rediscovering: /// @@ -1476,11 +1476,11 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk( /// keeps a failed detached relink from ever being visible as a live part: the abandoned precommit and /// the abandoned staging directory both live in the detached ref space. /// -/// What a `yes` does NOT prove: `CaRelinkConfirmCore.tla` config `_sab_holeylist` shows that with every -/// confirm rule intact and one incomplete listing page permitted, `ConfirmedRelinkNeverDangles` still -/// breaks (BACKLOG `{#list-as-journal-dataloss-2026-07-25}`). A confirmed relink is therefore NOT proven -/// dangle-free; a `yes` means only "the source still holds exactly this manifest right now", which is -/// what closes the codex-6 handoff window and nothing more. +/// What a `yes` does NOT prove: formal modelling of this protocol found that even with every confirm +/// rule intact, one incomplete storage LIST page during a GC fold is enough to let a confirmed relink's +/// blobs be reclaimed anyway. A confirmed relink is therefore NOT proven dangle-free; a `yes` means only +/// "the source still holds exactly this manifest right now", which closes the window between this +/// receiver's publish and the source's answer, and nothing more. MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( const String & part_name, const String & tmp_prefix, @@ -1531,7 +1531,7 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( }); /// Stage under the tmp-fetch dir OF THE TARGET PARENT — the table dir, or `TABLE/detached` when - /// the caller asked for a detached fetch (B66b). The parent is composed exactly as + /// the caller asked for a detached fetch. The parent is composed exactly as /// `downloadPartToDisk` composes it, so the two fetch paths put a part in the same place and the /// caller's finalization is unchanged: `renameTempPartAndReplace`'s moveDirectory(tmp-fetch_ /// -> ) for the active path, `renameTo(detached/)` for the detached one. Both are ref @@ -1550,16 +1550,16 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( LOG_DEBUG(log, "Relinking part {} (staged as {}) onto content-addressed disk {} from a {}-byte transferred manifest.", part_name, part_path, disk->getName(), sender_manifest_bytes.size()); - /// T1 — PUBLISH. Adopt-from-manifest and precommit, stopping short of the promote (B7 - /// part_manifest_v2, all-tree task 7): the receiver decodes the transferred body and stages its OWN + /// T1 — PUBLISH. Adopt-from-manifest and precommit, stopping short of the promote (`part_manifest_v2`): + /// the receiver decodes the transferred body and stages its OWN /// local manifest over the shared-pool blobs (adopt-by-hash). Self-contained: - /// uuid.txt/metadata_version.txt are ordinary entries in the transferred manifest (task 6), so there + /// uuid.txt/metadata_version.txt are ordinary entries in the transferred manifest, so there /// is no sidecar to reconstruct. Trust boundary is the interserver channel, as for a normal part /// fetch — see `prepareAdoptFromManifest`. /// /// The order is the whole protocol. This `+1` must be DURABLE before the source is asked anything, /// because the question "do you still hold it?" only excludes a later removal if the receiver's own - /// reference is already in the ref log when that removal is appended (spec §correctness). Asking + /// reference is already in the ref log when that removal is appended. Asking /// first and publishing after would prove nothing about the interval in between. What it does NOT /// establish is that every subsequent GC fold OBSERVES that reference -- see "What a `yes` does NOT /// prove" above; ordering is necessary here, not sufficient. @@ -1581,10 +1581,10 @@ MergeTreeData::MutableDataPartPtr Fetcher::relinkPartToDisk( /// Test-only, and this is the ONE seam worth injecting on the whole path: it opens the window the /// protocol exists to make safe. The receiver's `+1` is durable and its release is armed, and the - /// source has not been asked anything yet, so a test that holds the fetch here can do to the source - /// exactly what codex-6 described — merge the part away, run GC to fixpoint — and then observe both - /// halves of the contract: the source's blobs survive the round (this receiver's binding protects - /// them) and the confirm that follows refuses to authorize a promote (the binding it named is gone). + /// source has not been asked anything yet, so a test that holds the fetch here can merge the part + /// away on the source and run GC to fixpoint, then observe both halves of the contract: the source's + /// blobs survive the round (this receiver's binding protects them) and the confirm that follows + /// refuses to authorize a promote (the binding it named is gone). FailPointInjection::pauseFailPoint(FailPoints::cas_relink_receiver_pause_before_confirm); /// T2 — CONFIRM. One read-only interserver question, aimed at the endpoint copied out of the fetch diff --git a/src/Storages/MergeTree/DataPartsExchange.h b/src/Storages/MergeTree/DataPartsExchange.h index acae2e503ae0..3258833a91a0 100644 --- a/src/Storages/MergeTree/DataPartsExchange.h +++ b/src/Storages/MergeTree/DataPartsExchange.h @@ -118,7 +118,7 @@ class Fetcher final : private boost::noncopyable /// offer decides the disk — the policy disk on the sender's pool — ahead of the storage policy's /// own placement; otherwise the ordinary reservation does. DiskPtr dest_disk = nullptr, - /// CAS fetch-by-relink (spec §B66b): may this request advertise its content-addressed pool + /// CAS fetch-by-relink: may this request advertise its content-addressed pool /// identity, i.e. may the sender answer with a relink offer instead of the part's bytes? /// /// It is a capability of its own rather than a rider on `try_zero_copy`, and it carries the @@ -160,14 +160,14 @@ class Fetcher final : private boost::noncopyable ThrottlerPtr throttler, bool sync); - /// CAS replication 2b — fetch-by-relink (spec §4), publish-then-confirm (spec §core-idea). Build a + /// CAS replication — fetch-by-relink, publish-then-confirm. Build a /// part WITHOUT downloading any bytes by publishing this server's own ref to the blobs already in the /// shared content-addressed pool. Stages the ref under the tmp-fetch dir of the target parent — the - /// table dir, or `detached/` when `to_detached` (B66b) — so the caller's finalization re-keys it to + /// table dir, or `detached/` when `to_detached` — so the caller's finalization re-keys it to /// the final part name, exactly as for a byte-fetched part: `renameTempPartAndReplace` for the /// active path, `renameTo(detached/)` for the detached one. Then it ASKS THE SOURCE whether it /// still holds exactly the manifest it offered, and only then promotes and loads the part. - /// Self-contained (all-tree task 7): the transferred manifest alone is enough to rebuild the part — + /// Self-contained: the transferred manifest alone is enough to rebuild the part — /// no separate uuid/metadata_version wire fields to reconstruct as a sidecar. /// /// The whole failure taxonomy lives at the definition; the two outcomes a CALLER must distinguish: From 850acfa3e7e91776d745279150cba333ea315db0 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:31:37 +0200 Subject: [PATCH 57/81] fetch: the forced content-addressed disk choice lives next to the other routing helpers Move the ~24-line forced-disk block out of the generic Fetcher::fetchSelectedPart and into chooseForcedCaDisk in DataPartsExchangeCasRouting.{cpp,h}, alongside resolveForcedCaCandidate and resolveOfferedCasPool that it already called. The generic function now makes one call and one conditional assignment (the resolved pool string is moved, not copied, matching the pre-extraction assignment sequence); behaviour, log text, and the cas_relink_receiver_drop_forced_disk failpoint (which moves with the code that uses it) are unchanged. Including Common/FailPoint.h as DataPartsExchangeCasRouting.cpp's first standard-library-touching header hit a known libfiu footgun: its include lands inside libfiu's own `extern "C"` block unless a real C++ was already pulled in first, so Common/logger_useful.h (which the file needs anyway) is included before it. DataPartsExchange.cpp's own diff against upstream shrinks by 15 lines and can no longer conflict with an upstream change to the relink block's control flow. The two LOG_DEBUG call sites now carry the helper's own source location in system.text_log metadata; message text is unchanged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Storages/MergeTree/DataPartsExchange.cpp | 27 +++--------- .../MergeTree/DataPartsExchangeCasRouting.cpp | 43 +++++++++++++++++++ .../MergeTree/DataPartsExchangeCasRouting.h | 38 ++++++++++++++++ 3 files changed, 87 insertions(+), 21 deletions(-) diff --git a/src/Storages/MergeTree/DataPartsExchange.cpp b/src/Storages/MergeTree/DataPartsExchange.cpp index 12052535735b..5d7e3ba6ac93 100644 --- a/src/Storages/MergeTree/DataPartsExchange.cpp +++ b/src/Storages/MergeTree/DataPartsExchange.cpp @@ -58,9 +58,6 @@ namespace FailPoints /// Stands in for a sender that predates the `cas_pool_uuid` response cookie: the offer is made /// without naming the pool, and the receiver has to fall back on "the single advertised pool". extern const char cas_relink_sender_omit_pool_cookie[]; - /// Stands in for an offer this policy has no disk for: the receiver forgets the forced disk it - /// resolved and must take the ordinary placement and a byte fetch. - extern const char cas_relink_receiver_drop_forced_disk[]; } namespace MergeTreeSetting @@ -840,24 +837,12 @@ std::pair Fetcher::fetchSelected if (!ca_relink.empty()) { const String offered_pool_cookie = parse(in->getResponseCookie(CA_POOL_UUID_PARAM, "")); - offered_pool = resolveOfferedCasPool(advertised_pools, offered_pool_cookie); - if (!disk) - { - auto chosen = resolveForcedCaCandidate(ca_candidates, advertised_pools, offered_pool_cookie); - fiu_do_on(FailPoints::cas_relink_receiver_drop_forced_disk, - { - LOG_INFO(log, "Failpoint cas_relink_receiver_drop_forced_disk: forgetting the forced disk for part {}", part_name); - chosen.reset(); - }); - if (chosen) - { - forced_ca_disk = ca_candidate_disks[*chosen]; - LOG_DEBUG(log, "Part {} is offered by relink for content-addressed pool {}; placing it on disk {} " - "ahead of the storage policy's volume order and TTL rules", part_name, offered_pool, ca_candidates[*chosen].disk_name); - /// From here on the target is decided: every `!disk` reservation branch below is skipped. - disk = forced_ca_disk; - } - } + auto choice = chooseForcedCaDisk( + static_cast(disk), ca_candidates, ca_candidate_disks, advertised_pools, offered_pool_cookie, part_name, log); + offered_pool = std::move(choice.offered_pool); + /// From here on the target is decided: every `!disk` reservation branch below is skipped. + if (choice.disk) + disk = forced_ca_disk = choice.disk; } DiskPtr preffered_disk = disk; diff --git a/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp b/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp index a3227e9de6da..e5071e8cc4f8 100644 --- a/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp +++ b/src/Storages/MergeTree/DataPartsExchangeCasRouting.cpp @@ -1,11 +1,25 @@ #include +/// libfiu's header wraps a C11 include in `extern "C"`; pulling in the real C++ +/// first (transitively, via logger_useful.h) keeps that redefinition from landing inside the extern "C" +/// block, which is what FailPoint.h being the first standard-library-touching include here would do. +#include +#include + #include #include #include +namespace DB +{ +namespace FailPoints +{ + extern const char cas_relink_receiver_drop_forced_disk[]; +} +} + namespace DB::DataPartsExchange { @@ -76,6 +90,35 @@ std::optional resolveForcedCaCandidate( return std::nullopt; } +ForcedCaDiskChoice chooseForcedCaDisk( + bool caller_supplied_disk, + const std::vector & candidates, + const Disks & candidate_disks, + const Strings & advertised_pools, + const String & offered_pool_cookie, + const String & part_name, + LoggerPtr log) +{ + ForcedCaDiskChoice result; + result.offered_pool = resolveOfferedCasPool(advertised_pools, offered_pool_cookie); + if (caller_supplied_disk) + return result; + + auto chosen = resolveForcedCaCandidate(candidates, advertised_pools, offered_pool_cookie); + fiu_do_on(FailPoints::cas_relink_receiver_drop_forced_disk, + { + LOG_INFO(log, "Failpoint cas_relink_receiver_drop_forced_disk: forgetting the forced disk for part {}", part_name); + chosen.reset(); + }); + if (chosen) + { + result.disk = candidate_disks[*chosen]; + LOG_DEBUG(log, "Part {} is offered by relink for content-addressed pool {}; placing it on disk {} " + "ahead of the storage policy's volume order and TTL rules", part_name, result.offered_pool, candidates[*chosen].disk_name); + } + return result; +} + std::optional resolveConfirmRoutingCandidate( const std::vector & candidates, const String & pool_uuid) diff --git a/src/Storages/MergeTree/DataPartsExchangeCasRouting.h b/src/Storages/MergeTree/DataPartsExchangeCasRouting.h index c5e640271f98..f1d58fb261f7 100644 --- a/src/Storages/MergeTree/DataPartsExchangeCasRouting.h +++ b/src/Storages/MergeTree/DataPartsExchangeCasRouting.h @@ -2,9 +2,24 @@ #include +#include #include #include +namespace DB +{ +class IDisk; +using DiskPtr = std::shared_ptr; +using Disks = std::vector; +} + +namespace Poco +{ +class Logger; +using LoggerPtr = std::shared_ptr; +} +using LoggerPtr = Poco::LoggerPtr; + namespace DB::DataPartsExchange { @@ -46,6 +61,29 @@ std::optional resolveForcedCaCandidate( const Strings & advertised_pools, const String & offered_pool_cookie); +/// The outcome of resolving a relink offer against this receiver's candidates: the pool the offer is +/// for (needed even when no disk is forced, to check a caller-supplied disk against it later), and the +/// disk to force the fetch onto — null when the caller already supplied a disk, or no live-policy +/// candidate matches the offered pool. +struct ForcedCaDiskChoice +{ + String offered_pool; + DiskPtr disk; +}; + +/// Resolve a relink offer's pool, and — only when the caller left disk selection to the fetch itself — +/// pick the forced candidate (`resolveForcedCaCandidate`) to place it on, ahead of the storage policy's +/// own placement. `part_name` and `log` are for the log lines only; the `cas_relink_receiver_drop_forced_disk` +/// failpoint (test-only) lives here so it can stand in for an offer this policy has no disk for. +ForcedCaDiskChoice chooseForcedCaDisk( + bool caller_supplied_disk, + const std::vector & candidates, + const Disks & candidate_disks, + const Strings & advertised_pools, + const String & offered_pool_cookie, + const String & part_name, + LoggerPtr log); + /// One content-addressed disk of the SENDING table's storage policy, as the confirm routing sees it. struct CasConfirmRoutingCandidate { From 24a0afee11203f302659cc1dc356b8ca5b109c30 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:31:37 +0200 Subject: [PATCH 58/81] object-storage: the default removeObjectsIfExistUnderProfile forwards a Default-profile request to removeObjectsIfExist Its three siblings (`iterate`, `tryGetObjectMetadataWithNativeToken`, `removeObjectIfTokenMatches`) already forward a Default-profile request to the plain, no-profile method and refuse only `SingleAttempt`; the batch remove's default threw `NOT_IMPLEMENTED` unconditionally, which was the odd one out. The only content-addressed caller of `removeObjectsIfExistUnderProfile` is `Cas::ObjectStorageBackend::removeManyWriteOnce`, and it only reaches `Default` on a read-only Native-mode mount; GC -- the sole caller of `removeManyWriteOnce` -- never starts on a read-only mount, so that branch is unreachable for CAS today. Every reachable CAS caller still gets `SingleAttempt` and is refused exactly as before, so this is a no-op for CAS and only brings the default in line with its siblings for other callers. Adds a minimal `IObjectStorage` stub gtest pinning both branches: `Default` forwards once with the same objects, `SingleAttempt` still throws `NOT_IMPLEMENTED` without calling `removeObjectsIfExist`. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ObjectStorages/IObjectStorage.cpp | 6 +- .../ObjectStorages/IObjectStorage.h | 4 +- .../gtest_cas_iobjectstorage_defaults.cpp | 161 ++++++++++++++++++ 3 files changed, 168 insertions(+), 3 deletions(-) create mode 100644 src/Disks/tests/gtest_cas_iobjectstorage_defaults.cpp diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp index a64f9a886ee9..fba16861e8e2 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp @@ -95,9 +95,11 @@ ConditionalRemoveResult IObjectStorage::removeObjectIfTokenMatches( return removeObjectIfTokenMatches(object, etag); } -void IObjectStorage::removeObjectsIfExistUnderProfile(const StoredObjects &, const ObjectStorageControlRequest &) +void IObjectStorage::removeObjectsIfExistUnderProfile(const StoredObjects & objects, const ObjectStorageControlRequest & request) { - throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support batch removal under a retry profile", getName()); + if (request.profile == ObjectStorageRetryProfile::SingleAttempt) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support batch removal under a retry profile", getName()); + removeObjectsIfExist(objects); } ThreadPool & IObjectStorage::getThreadPoolWriter() diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index eec112d27059..436f8c05bd51 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -374,7 +374,9 @@ class IObjectStorage /// Removes every object in ONE request with no per-key precondition; an absent object is success. /// Content-addressed callers use it for write-once keys only, at most 1000 per call. Throws on a /// request-level failure and on any per-key error other than "not found", naming the failed keys. - /// Same context note as `iterate`. Backends without a batch delete keep the default, which refuses. + /// Same context note as `iterate`: the default forwards a Default-profile request to + /// `removeObjectsIfExist` and refuses a SingleAttempt one, so a backend without a real batch + /// delete still needs to override this for SingleAttempt to behave correctly under that profile. virtual void removeObjectsIfExistUnderProfile( const StoredObjects & objects, const ObjectStorageControlRequest & request); diff --git a/src/Disks/tests/gtest_cas_iobjectstorage_defaults.cpp b/src/Disks/tests/gtest_cas_iobjectstorage_defaults.cpp new file mode 100644 index 000000000000..e89b1a28466a --- /dev/null +++ b/src/Disks/tests/gtest_cas_iobjectstorage_defaults.cpp @@ -0,0 +1,161 @@ +#include + +#include + +#include + +/// `IObjectStorage::removeObjectsIfExistUnderProfile` has three siblings (`iterate`, +/// `tryGetObjectMetadataWithNativeToken`, `removeObjectIfTokenMatches`) whose defaults all forward a +/// Default-profile request to the plain, no-profile method and refuse only SingleAttempt. This file +/// pins that `removeObjectsIfExistUnderProfile` follows the same rule, using a minimal stub storage +/// that implements nothing beyond what `IObjectStorage` requires. + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +} + +namespace +{ + +/// Implements only what `IObjectStorage` declares pure; every method a case below does not exercise +/// throws if called, so a test that reaches one it did not expect fails loudly instead of silently +/// doing the wrong thing. +class MinimalObjectStorage : public IObjectStorage +{ +public: + std::string getName() const override + { + return "MinimalObjectStorage"; + } + + ObjectStorageType getType() const override + { + return ObjectStorageType::None; + } + + std::string getCommonKeyPrefix() const override + { + return ""; + } + + std::string getDescription() const override + { + return "MinimalObjectStorage (test stub)"; + } + + bool exists(const StoredObject &) const override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + ObjectMetadata getObjectMetadata(const std::string &, bool) const override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + std::optional tryGetObjectMetadata(const std::string &, bool) const override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + std::unique_ptr readObject( + const StoredObject &, const ReadSettings &, std::optional, bool, bool) const override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + std::unique_ptr writeObject( + const StoredObject &, WriteMode, std::optional, size_t, const WriteSettings &) override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + bool isRemote() const override + { + return true; + } + + void removeObjectIfExists(const StoredObject &) override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + /// The method under test: `removeObjectsIfExistUnderProfile`'s Default-profile default forwards here. + void removeObjectsIfExist(const StoredObjects & objects) override + { + ++remove_objects_if_exist_calls; + last_removed_objects = objects; + } + + void copyObject( + const StoredObject &, const StoredObject &, const ReadSettings &, const WriteSettings &, std::optional) override + { + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "not used by this test"); + } + + void shutdown() override + { + } + + void startup() override + { + } + + String getObjectsNamespace() const override + { + return ""; + } + + ObjectStorageKeyGeneratorPtr createKeyGenerator() const override + { + return nullptr; + } + + size_t remove_objects_if_exist_calls = 0; + StoredObjects last_removed_objects; +}; + +} + +TEST(CASIObjectStorageDefaults, RemoveObjectsIfExistUnderProfileDefaultForwards) +{ + MinimalObjectStorage storage; + const StoredObjects objects{StoredObject("a"), StoredObject("b")}; + + ObjectStorageControlRequest request; + request.profile = ObjectStorageRetryProfile::Default; + + storage.removeObjectsIfExistUnderProfile(objects, request); + + EXPECT_EQ(storage.remove_objects_if_exist_calls, 1u); + ASSERT_EQ(storage.last_removed_objects.size(), 2u); + EXPECT_EQ(storage.last_removed_objects[0].remote_path, "a"); + EXPECT_EQ(storage.last_removed_objects[1].remote_path, "b"); +} + +TEST(CASIObjectStorageDefaults, RemoveObjectsIfExistUnderProfileSingleAttemptThrows) +{ + MinimalObjectStorage storage; + const StoredObjects objects{StoredObject("a")}; + + ObjectStorageControlRequest request; + request.profile = ObjectStorageRetryProfile::SingleAttempt; + + try + { + storage.removeObjectsIfExistUnderProfile(objects, request); + FAIL() << "expected a SingleAttempt batch-remove request to be refused"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::NOT_IMPLEMENTED); + } + + EXPECT_EQ(storage.remove_objects_if_exist_calls, 0u); +} + +} From 0c8da46759c55f77f74ca9da026aa6228778c853 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 08:07:48 +0200 Subject: [PATCH 59/81] errors: the fork's error-code range is written down Names the policy behind the existing "CAS codes resume at 1037" comment: fork-specific error codes live in 1030-1099, chosen to sit well above upstream ClickHouse's maximum error code (1017, per github.com/ClickHouse/ClickHouse's src/Common/ErrorCodes.cpp at the time this range was reserved) so upstream can keep adding codes below it without a future collision, and says a new fork code goes in that range. Also notes that 1010 (EXPORT_PARTITION_ALREADY_EXPORTED) and 1011 (PARTITION_EXPORT_FAILED) predate this policy and are NOT renumbered to fit it: upstream has since taken 1010 and 1011 for its own UNIQUE_KEY_DENSE_INDEX_UNREADABLE and HANDLER_ALREADY_EXISTS, so the fork's two codes currently collide with upstream's. Renumbering an already-shipped fork error code is a compatibility break of its own, so this only records the collision rather than fixing it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Common/ErrorCodes.cpp | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index b6fe2074ebce..cfdc94b31795 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -677,11 +677,16 @@ M(1007, ILLEGAL_STREAM) \ M(1008, TEMPORARY_DATA_NOT_IN_CACHE) \ M(1009, PENDING_MUTATIONS_NOT_ALLOWED) \ + /* 1010 and 1011 predate the fork's error-code range policy stated below, and are kept as-is \ + * rather than renumbered: they currently collide with upstream ClickHouse's own 1010 \ + * (UNIQUE_KEY_DENSE_INDEX_UNREADABLE) and 1011 (HANDLER_ALREADY_EXISTS). */ \ M(1010, EXPORT_PARTITION_ALREADY_EXPORTED) \ M(1011, PARTITION_EXPORT_FAILED) \ /* 1012 and 1013 are intentionally skipped: they collide with upstream ClickHouse's \ - * HANDLER_DOESNT_EXIST and AMBIGUOUS_HANDLER. CAS codes resume at 1037, comfortably \ - * past upstream's current maximum, to leave headroom for future upstream additions. */ \ + * HANDLER_DOESNT_EXIST and AMBIGUOUS_HANDLER. Fork-specific error codes live in the 1030-1099 \ + * range, chosen to sit well above upstream's maximum error code (1017 at the time this range \ + * was reserved) so upstream can keep adding codes below it without colliding with the fork's. \ + * A new fork error code goes in this range, not below 1030. CAS codes occupy 1037-1038. */ \ M(1037, CAS_WRITE_UNATTRIBUTED) \ M(1038, CAS_DELETE_MARKER) \ /* See END */ From cacaff1ee30607357b7f67da737a7bd69c04aac4 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:31:37 +0200 Subject: [PATCH 60/81] s3: the object storage keeps its MultiVersion client member; the refresh path captures the slot S3ObjectStorage::client used to be a plain MultiVersion value member, read via client.get() and written via client.set() from every call site. To let the SingleAttempt read path's credential-refresh lambda outlive this storage safely, the member had become a shared_ptr>, which turned all 27 of those call sites into client->get()/client->set(). Restore client as a reference bound to a new private client_slot (the shared_ptr the refresh lambda still captures), so every original client.get()/client.set() call site is textually unchanged. Only the one lambda that must own the slot independently of this object's lifetime captures client_slot directly. The class has no copy/move operations and is never copied (the only constructor call site is the delegating constructor), so a reference member is safe here; MultiVersion's own get()/set() split is preserved by the reference the same way the prior shared_ptr did. Also restores the double blank line before the class declaration that a driveby whitespace edit had removed, so that hunk disappears from the diff against upstream. diff --stat vs altinity/antalya-26.6 for the two touched files: S3ObjectStorage.cpp: before 376+/44-, after 355+/23- S3ObjectStorage.h: before 74+/8-, after 73+/7- Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ObjectStorages/S3/S3ObjectStorage.cpp | 58 +++++++++---------- .../ObjectStorages/S3/S3ObjectStorage.h | 11 +++- 2 files changed, 38 insertions(+), 31 deletions(-) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index d6f8df65b1a0..97ba900a0966 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -290,7 +290,7 @@ auto S3ObjectStorage::refreshAndRetryOnExpiredCredentials(Fn && fn) const auto new_client = credentials_refresh_callback(); if (!new_client) throw; - client->set(std::move(new_client)); + client.set(std::move(new_client)); return fn(); } } @@ -298,7 +298,7 @@ auto S3ObjectStorage::refreshAndRetryOnExpiredCredentials(Fn && fn) const bool S3ObjectStorage::exists(const StoredObject & object) const { auto settings_ptr = s3_settings.get(); - const bool e = S3::objectExists(*client->get(), uri.bucket, object.remote_path, {}); + const bool e = S3::objectExists(*client.get(), uri.bucket, object.remote_path, {}); return e; } @@ -342,12 +342,12 @@ std::unique_ptr S3ObjectStorage::readObject( /// NOLINT /// object. The copied callback can itself capture a shorter-lived object -- /// `StorageS3Configuration::createObjectStorage`'s refresher captures the configuration /// it was built from -- so the caller must keep that object alive as long as the buffer. - refresh_callback = [client_slot = client, refresh = credentials_refresh_callback]() + refresh_callback = [slot = client_slot, refresh = credentials_refresh_callback]() -> std::unique_ptr { auto new_client = refresh(); if (new_client) - client_slot->set(std::move(new_client)); + slot->set(std::move(new_client)); /// The buffer will not reissue this read, so it has no use for a client; refreshing /// the disk's is what lets the caller's next request sign with the new credentials. return nullptr; @@ -453,7 +453,7 @@ std::unique_ptr S3ObjectStorage::writeObject( /// NOLIN /// The SingleAttempt profile (e.g. CAS conditional writes, RFC cas-s3-timeout-retry-control) rides /// on WriteSettings instead of changing this disk's shared client — every other write keeps using - /// client->get() and its normal retry policy unchanged. + /// client.get() and its normal retry policy unchanged. auto used_client = clientForRetryProfile(ObjectStorageControlRequest{ .profile = write_settings.object_storage_retry_profile, .attempt_timeout_ms = write_settings.object_storage_attempt_timeout_ms, @@ -515,7 +515,7 @@ void S3ObjectStorage::listObjects(const std::string & path, RelativePathsWithMet { ProfileEventTimeIncrement watch(ProfileEvents::S3ListObjectsMicroseconds); - outcome = client->get()->ListObjectsV2(request); + outcome = client.get()->ListObjectsV2(request); } throwIfError(outcome, "while listing objects in bucket '{}' with prefix '{}' on disk '{}'", uri.bucket, path, disk_name); @@ -553,7 +553,7 @@ void S3ObjectStorage::removeObjectImpl(const StoredObject & object, bool if_exis { auto blob_storage_log = BlobStorageLogWriter::create(disk_name); - deleteFileFromS3(client->get(), uri.bucket, object.remote_path, if_exists, + deleteFileFromS3(client.get(), uri.bucket, object.remote_path, if_exists, blob_storage_log, object.local_path, object.bytes_size, ProfileEvents::DiskS3DeleteObjects); } @@ -581,7 +581,7 @@ void S3ObjectStorage::removeObjectsImpl(const StoredObjects & objects, bool if_e auto settings_ptr = s3_settings.get(); - deleteFilesFromS3(client->get(), uri.bucket, keys, if_exists, + deleteFilesFromS3(client.get(), uri.bucket, keys, if_exists, s3_capabilities, settings_ptr->request_settings[S3RequestSetting::objects_chunk_size_to_delete], blob_storage_log, local_paths_for_blob_storage_log, file_sizes_for_blob_storage_log, ProfileEvents::DiskS3DeleteObjects); @@ -599,7 +599,7 @@ void S3ObjectStorage::removeObjectsIfExist(const StoredObjects & objects) ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches(const StoredObject & object, const std::string & etag) { - return removeObjectIfTokenMatchesImpl(object, etag, client->get(), /*attempt_seed=*/0); + return removeObjectIfTokenMatchesImpl(object, etag, client.get(), /*attempt_seed=*/0); } ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches( @@ -798,7 +798,7 @@ void S3ObjectStorage::removeObjectsIfExistImpl( bool S3ObjectStorage::conditionalOpsUseGenerationTokens() const { - return client->get()->supportsGcsNativeConditionalRequests(); + return client.get()->supportsGcsNativeConditionalRequests(); } bool S3ObjectStorage::supportsCopyMode(ObjectStorageCopyMode mode) const @@ -818,7 +818,7 @@ std::optional S3ObjectStorage::isBucketVersioningEnabled() const S3::GetBucketVersioningRequest request; request.SetBucket(uri.bucket); - auto outcome = client->get()->GetBucketVersioning(request); + auto outcome = client.get()->GetBucketVersioning(request); if (!outcome.IsSuccess()) { /// The caller only learns "unknown"; the reason is what the operator needs to act on. @@ -898,17 +898,17 @@ static void putObjectsTagOnS3( void S3ObjectStorage::tagObjects(const StoredObjects & objects, const std::string & tag_key, const std::string & tag_value) { Strings keys = collectRemotePaths(objects); - putObjectsTagOnS3(client->get(), uri.bucket, keys, tag_key, tag_value); + putObjectsTagOnS3(client.get(), uri.bucket, keys, tag_key, tag_value); } std::optional S3ObjectStorage::tryGetObjectMetadata(const std::string & path, bool with_tags) const { - return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::Default, client->get()); + return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::Default, client.get()); } std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeToken(const std::string & path, bool with_tags) const { - return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::NativeConditional, client->get()); + return tryGetObjectMetadataImpl(path, with_tags, ObjectStorageRequestMode::NativeConditional, client.get()); } std::optional S3ObjectStorage::tryGetObjectMetadataWithNativeToken( @@ -957,7 +957,7 @@ ObjectMetadata S3ObjectStorage::getObjectMetadata(const std::string & path, bool S3::ObjectInfo object_info; try { - object_info = S3::getObjectInfo(*client->get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); + object_info = S3::getObjectInfo(*client.get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); } catch (DB::Exception & e) { @@ -967,8 +967,8 @@ ObjectMetadata S3ObjectStorage::getObjectMetadata(const std::string & path, bool auto new_client = credentials_refresh_callback(); if (new_client) { - client->set(std::move(new_client)); - object_info = S3::getObjectInfo(*client->get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); + client.set(std::move(new_client)); + object_info = S3::getObjectInfo(*client.get(), uri.bucket, path, /*version_id=*/ {}, /*with_metadata=*/ true, /*with_tags=*/ with_tags); updated = true; } } @@ -1001,9 +1001,9 @@ void S3ObjectStorage::copyObjectToAnotherObjectStorage( // NOLINT /// Shortcut for S3 if (auto * dest_s3 = dynamic_cast(&object_storage_to); dest_s3 != nullptr) { - auto current_client = dest_s3->client->get(); + auto current_client = dest_s3->client.get(); auto settings_ptr = s3_settings.get(); - auto size = S3::getObjectSize(*client->get(), uri.bucket, object_from.remote_path, {}); + auto size = S3::getObjectSize(*client.get(), uri.bucket, object_from.remote_path, {}); auto scheduler = threadPoolCallbackRunnerUnsafe(getThreadPoolWriter(), ThreadName::S3_COPY_POOL); const auto read_settings_to_use = patchSettings(read_settings); @@ -1043,7 +1043,7 @@ void S3ObjectStorage::copyObjectToAnotherObjectStorage( // NOLINT if (new_client) { updated = true; - client->set(std::move(new_client)); + client.set(std::move(new_client)); } } if (!updated) @@ -1077,7 +1077,7 @@ void S3ObjectStorage::copyObject( // NOLINT "(allow_native_copy=false) for object storage {}", getName()); - auto current_client = client->get(); + auto current_client = client.get(); auto settings_ptr = s3_settings.get(); auto size = S3::getObjectSize(*current_client, uri.bucket, object_from.remote_path, {}); auto scheduler = threadPoolCallbackRunnerUnsafe(getThreadPoolWriter(), ThreadName::S3_COPY_POOL); @@ -1107,7 +1107,7 @@ void S3ObjectStorage::shutdown() /// If S3 request is failed and the method below is executed S3 client immediately returns the last failed S3 request outcome. /// If S3 is healthy nothing wrong will be happened and S3 requests will be processed in a regular way without errors. /// This should significantly speed up shutdown process if S3 is unhealthy. - const_cast(*client->get()).DisableRequestProcessing(); + const_cast(*client.get()).DisableRequestProcessing(); /// Parity with the main client above, not a stronger guarantee. `DisableRequestProcessing` cannot /// prevent a request's INITIAL dispatch, and cannot interrupt an attempt already in flight: contrib/aws's @@ -1135,7 +1135,7 @@ void S3ObjectStorage::shutdown() void S3ObjectStorage::startup() { /// Need to be enabled if it was disabled during shutdown() call. - const_cast(*client->get()).EnableRequestProcessing(); + const_cast(*client.get()).EnableRequestProcessing(); std::lock_guard lock(single_attempt_client_mutex); single_attempt_clients_disabled = false; @@ -1219,7 +1219,7 @@ void S3ObjectStorage::applyNewSettings( && (current_settings->auth_settings.hasUpdates(modified_settings->auth_settings) || for_disk_s3)) { auto new_client = getClient(uri, *modified_settings, context, for_disk_s3, disk_name); - client->set(std::move(new_client)); + client.set(std::move(new_client)); } s3_settings.set(std::move(modified_settings)); } @@ -1234,17 +1234,17 @@ ObjectStorageKeyGeneratorPtr S3ObjectStorage::createKeyGenerator() const std::shared_ptr S3ObjectStorage::getS3StorageClient() { - return client->get(); + return client.get(); } std::shared_ptr S3ObjectStorage::tryGetS3StorageClient() { - return client->get(); + return client.get(); } std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64_t request_timeout_ms, uint64_t connect_timeout_cap_ms) const { - auto base = client->get(); + auto base = client.get(); std::lock_guard lock(single_attempt_client_mutex); if (single_attempt_client_base != base) { @@ -1296,7 +1296,7 @@ std::shared_ptr S3ObjectStorage::clientForRetryProfile(const O /// pays for building or locking the clone. if (request.profile == ObjectStorageRetryProfile::SingleAttempt) return getSingleAttemptClient(request.attempt_timeout_ms, request.connect_timeout_cap_ms); - return client->get(); + return client.get(); } bool S3ObjectStorage::tryRefreshCredentialsViaCallback() @@ -1308,7 +1308,7 @@ bool S3ObjectStorage::tryRefreshCredentialsViaCallback() auto new_client = credentials_refresh_callback(); if (!new_client) return false; - client->set(std::move(new_client)); + client.set(std::move(new_client)); return true; } } diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 1da1a6b38d99..8840175abc51 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -25,6 +25,7 @@ namespace S3RequestSetting extern const S3RequestSettingsBool read_only; } + class S3ObjectStorage : public IObjectStorage { public: @@ -45,7 +46,8 @@ class S3ObjectStorage : public IObjectStorage const S3CredentialsRefreshCallback & credentials_refresh_callback_ = [] -> std::unique_ptr{ return nullptr; }) : uri(uri_) , disk_name(disk_name_) - , client(std::make_shared>(std::move(client_))) + , client_slot(std::make_shared>(std::move(client_))) + , client(*client_slot) , s3_settings(std::move(s3_settings_)) , s3_capabilities(s3_capabilities_) , key_generator(std::move(key_generator_)) @@ -246,7 +248,12 @@ class S3ObjectStorage : public IObjectStorage /// Held by `shared_ptr` so a read buffer -- which can outlive this storage -- carries the SLOT /// rather than a pointer to the storage: a refresh that arrives late then replaces a client /// nobody will read again, instead of writing into a destroyed object. - const std::shared_ptr> client; + const std::shared_ptr> client_slot; + /// Reference into the slot above. Every ordinary call site keeps using `client.get()`/`client.set()` + /// unchanged; only code that must capture the client independently of this storage's own lifetime + /// (the credential-refresh lambda handed to a read buffer that can outlive this object) captures + /// `client_slot` directly instead. + MultiVersion & client; MultiVersion s3_settings; S3Capabilities s3_capabilities; From fd1717b715d714854a672216c61fed1d504caaa9 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:31:37 +0200 Subject: [PATCH 61/81] s3: shutdown and clone comments say only what is true and needed The shutdown()/getSingleAttemptClient() comments explaining why DisableRequestProcessing does not by itself stop dispatch, and what does, had grown to ~27 lines across the header and the .cpp. Compress each to the facts, one sentence per fact, and make them accurate against contrib/aws: - AWSClient checks IsRequestProcessingEnabled() BEFORE calling ShouldRetry (AWSClient.cpp:324) and breaks out of the loop when it is false; a SingleAttemptRetryStrategy clone lands on the same behaviour through the other branch of that check. The one reissue the flag does still suppress is the SDK's own region redirect of an `AWS_GLOBAL` client after a 301/307/400/403 reply, which precedes the retry strategy. - New open-plane requests after teardown are refused at Pool::teardownBegun() (CasPool.cpp), which CasOperation::readLoop (CasRequests.h) checks before every attempt; the mount and farewell planes stay admitting through this window. - The clone-site and header-field comments point back to shutdown()'s comment instead of repeating it. The `using Aws::S3::S3Client::GetHttpClient;` in S3::Client re-exposes a privately-inherited member; its only external caller is gtest_cas_s3_single_attempt_client.cpp, so it is marked test-only. Production code has no need of it since Client's own methods already reach GetHttpClient through the private inheritance. No behaviour change -- comment text only. diff --stat vs altinity/antalya-26.6 (continuing from the client-member commit): S3ObjectStorage.cpp: before 355+/23-, after 345+/23- S3ObjectStorage.h: before 73+/7-, after 66+/7- Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ObjectStorages/S3/S3ObjectStorage.cpp | 36 ++++++++----------- .../ObjectStorages/S3/S3ObjectStorage.h | 13 ++----- src/IO/S3/Client.h | 7 ++-- 3 files changed, 21 insertions(+), 35 deletions(-) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 97ba900a0966..7c0debc711db 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -1109,23 +1109,17 @@ void S3ObjectStorage::shutdown() /// This should significantly speed up shutdown process if S3 is unhealthy. const_cast(*client.get()).DisableRequestProcessing(); - /// Parity with the main client above, not a stronger guarantee. `DisableRequestProcessing` cannot - /// prevent a request's INITIAL dispatch, and cannot interrupt an attempt already in flight: contrib/aws's - /// AWSClient checks it only after an attempt has already failed and returned, right before deciding - /// whether to retry (AWSClient.cpp, between `ShouldRetry` and the backoff sleep). Every cached clone - /// here runs `SingleAttemptRetryStrategy` (max_retries=0), whose `ShouldRetry` already always says no - /// -- so the flag is still consulted on a clone's failed attempt, it just changes nothing observable - /// there, since no retry was ever going to happen regardless of the flag's value. - /// - /// What actually stops a NEW request on the OPEN plane -- GC, FSCK, the probe; the plane the write-once - /// bulk-delete verb this storage serves runs on -- from being dispatched at all is admission, refused - /// earlier and on a different plane: `DiskObjectStorage::shutdown()` calls `metadata_storage->shutdown()` - /// (which arms `Pool::beginTeardown()`, tripping the open-plane fence `CasPool.cpp` wires to - /// `teardownBegun()`) BEFORE it calls this object storage's `shutdown()`, and `CasOperation::readLoop` - /// (CasRequests.h) checks that fence before every attempt, including the first -- so an open-plane - /// request issued after the disk's shutdown began throws at admission and never reaches a clone at - /// all. The mount and farewell planes are NOT covered by this: they intentionally stay admitting - /// through this same window, since teardown's own drain and farewell I/O run on them. + /// The SDK checks this flag only after an attempt has failed, right before deciding whether to + /// retry -- it neither blocks a request's initial dispatch nor interrupts one already in flight. + /// Every cached clone below runs `SingleAttemptRetryStrategy` (max_retries=0), so its retry strategy + /// never asks for a reissue; the one reissue the SDK makes on its own regardless of the strategy -- + /// an `AWS_GLOBAL` client re-signing for the region a 301/307/400/403 reply names -- is what the + /// disabled flag stops on a clone, since that check runs before the region redirect is considered. + /// What actually blocks a NEW request on the CAS engine's open plane (GC, FSCK, the probe) after + /// shutdown began is admission, refused at `Pool::teardownBegun()` (`CasPool.cpp`), which + /// `CasOperation::readLoop` (`CasRequests.h`) checks before every attempt, including the first. + /// The mount and farewell planes stay admitting through this window, since teardown's own drain + /// and farewell I/O run on them. std::lock_guard lock(single_attempt_client_mutex); single_attempt_clients_disabled = true; for (const auto & [_, clone] : single_attempt_clients) @@ -1279,11 +1273,9 @@ std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64 const auto & clone = single_attempt_clients.emplace(cache_key, base->cloneWithConfigurationOverride(cfg)).first->second; - /// A fresh clone's own `Aws::Http::HttpClient` starts with request processing enabled regardless of - /// the main client's state; kept in parity with `shutdown()` for the same reason that flag is set - /// there in the first place (see the comment on `shutdown()`) -- this does not, by itself, stop a - /// request already dispatched on this clone, which a single-attempt clone never reaches anyway once - /// admission is refused (see `shutdown()`). + /// A fresh clone starts with request processing enabled regardless of the main client's state, so + /// one built after shutdown() must be disabled to match it (see the comment on shutdown() for what + /// that flag does and does not do). if (single_attempt_clients_disabled) const_cast(*clone).DisableRequestProcessing(); diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 8840175abc51..335ad74b716a 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -282,16 +282,9 @@ class S3ObjectStorage : public IObjectStorage /// released as soon as the next rotation is observed and the clones are dropped — which is what /// makes the identity comparison in getSingleAttemptClient sound. mutable std::shared_ptr single_attempt_client_base; - /// Set for the duration of a `shutdown()` (cleared by the matching `startup()`), kept in parity with - /// the main client's `DisableRequestProcessing`/`EnableRequestProcessing` toggle. Every clone already - /// cached at the moment `shutdown()` runs is disabled there and then, under the same lock; this flag - /// is what makes a clone built afterwards by `getSingleAttemptClient` come into being already - /// disabled too. NOTE: this flag cannot prevent a request's initial dispatch or interrupt one already - /// in flight -- the AWS SDK checks it only after an attempt has failed, right before deciding whether - /// to retry, and every clone here runs `SingleAttemptRetryStrategy` (max_retries=0), whose own answer - /// to that question is already always no. What actually prevents a NEW request on the CAS engine's - /// open plane (GC, FSCK, the probe) from reaching a clone after shutdown is admission, refused - /// earlier at that engine's own fence (see the comment on `S3ObjectStorage::shutdown()`). + /// Set for the duration of a `shutdown()` (cleared by the matching `startup()`); every clone already + /// cached when `shutdown()` runs is disabled there and then, and this flag disables any built + /// afterwards to match. See the comment on `shutdown()` for what that disabling does and does not do. mutable bool single_attempt_clients_disabled = false; }; diff --git a/src/IO/S3/Client.h b/src/IO/S3/Client.h index bf74689138fd..92d8e1cfa929 100644 --- a/src/IO/S3/Client.h +++ b/src/IO/S3/Client.h @@ -236,9 +236,10 @@ class Client : private Aws::S3::S3Client using Aws::S3::S3Client::EnableRequestProcessing; using Aws::S3::S3Client::DisableRequestProcessing; - /// Lets a caller (a shutdown-state test, in particular) observe whether Enable/DisableRequestProcessing - /// last took effect on this client's own `Aws::Http::HttpClient`, without exposing the rest of the - /// privately-inherited `Aws::S3::S3Client` surface. + /// Test-only: lets a gtest observe whether Enable/DisableRequestProcessing last took effect on this + /// client's own `Aws::Http::HttpClient`, without exposing the rest of the privately-inherited + /// `Aws::S3::S3Client` surface. Production code reaches `GetHttpClient` directly (private + /// inheritance already permits that from this class's own methods) and has no need of this `using`. using Aws::S3::S3Client::GetHttpClient; void BuildHttpRequest(const Aws::AmazonWebServiceRequest& request, From cd284f2da451bc32e9be89976c785f51a2d47ea8 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:31:37 +0200 Subject: [PATCH 62/81] s3: move fork-added client/attempt-seed tests out of gtest_aws_s3_client.cpp gtest_aws_s3_client.cpp is an upstream test file; every fork-added test in it made rebasing conflict. The fork's additions (attempt-seed local-retry test, its TestPocoHTTPSequenceServer helper, and the SingleAttempt/network-error logging tests with their NetworkFailingClient/ScopedS3ClientErrorLogCapture helpers) are self-contained and move into the new fork-owned gtest_cas_aws_s3_client.cpp, under the suite name CASIOTestAwsS3Client so it does not share a suite name with the upstream file's suite. Running the relocated tests in the full battery exposed a pre-existing fragility in every local-HTTP-server helper of these files (TestPocoHTTPServer, TestPocoHTTPStsServer, TestPocoHTTPSequenceServer, ScriptedResponseServer): each constructed its Poco::Net::HTTPServer on Poco::ThreadPool::defaultPool(), one pool shared, unsynchronized, across every live TCPServerDispatcher in the test binary. TCPServerDispatcher's "can we start a thread" check is per-dispatcher against that shared capacity, so once enough other servers saturate the pool the dispatcher's startWithPriority throws and the just-accepted connection is closed without a response (seen via strace as a client-side "Connection reset by peer"). Each helper now owns a private Poco::ThreadPool, binds to and reports 127.0.0.1 explicitly instead of the wildcard bind address, and has a destructor that calls HTTPServer::stopAll(true) then joinAll() so an idle keep-alive worker does not cost PooledThread::release's 10 s join cap per test. This is the one deviation from gtest_aws_s3_client.cpp being byte-identical to altinity/antalya-26.6 (+26/-1). Verified: `unit_tests_dbms --gtest_filter='*S3*:*ReadBuffer*:*WriteBuffer*'` (263 tests, 45 suites) 3/3 green at 13.2-13.5 s, matching the 13.0 s measured before any thread-pool change; `CAS*:*S3*:*ObjectStorage*:*Teardown*` (2699 tests) green, 0 failures; the same set green under ASan. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/IO/S3/tests/TestPocoHTTPServer.h | 38 +- src/IO/S3/tests/gtest_aws_s3_client.cpp | 404 +---------------- src/IO/S3/tests/gtest_cas_aws_s3_client.cpp | 475 ++++++++++++++++++++ 3 files changed, 525 insertions(+), 392 deletions(-) create mode 100644 src/IO/S3/tests/gtest_cas_aws_s3_client.cpp diff --git a/src/IO/S3/tests/TestPocoHTTPServer.h b/src/IO/S3/tests/TestPocoHTTPServer.h index f81091b75ddc..33fde7cb85b5 100644 --- a/src/IO/S3/tests/TestPocoHTTPServer.h +++ b/src/IO/S3/tests/TestPocoHTTPServer.h @@ -16,6 +16,7 @@ #include #include #include +#include #include class MockRequestHandler : public Poco::Net::HTTPRequestHandler @@ -59,6 +60,11 @@ class TestPocoHTTPServer std::unique_ptr server_socket; Poco::SharedPtr handler_factory; Poco::AutoPtr server_params; + /// A dedicated pool, not `Poco::ThreadPool::defaultPool()` (the `HTTPServer` default): that pool + /// is shared with every other local-server test in this binary, and `TCPServerDispatcher::enqueue` + /// (base/poco/Net/src/TCPServerDispatcher.cpp) has an acknowledged-in-comment saturation-check race + /// when it's shared, which can accept a connection and then close it with no response. + Poco::ThreadPool thread_pool; std::unique_ptr server; // Stores the last request header handled. It's obviously not thread-safe to share the same // reference across request handlers, but it's good enough for this the purposes of this test. @@ -69,14 +75,25 @@ class TestPocoHTTPServer server_socket(std::make_unique(0)), handler_factory(new HTTPRequestHandlerFactory(last_request_header)), server_params(new Poco::Net::HTTPServerParams()), - server(std::make_unique(handler_factory, *server_socket, server_params)) + thread_pool("TestPocoHTTPServer"), + server(std::make_unique(handler_factory, thread_pool, *server_socket, server_params)) { server->start(); } + /// `stopAll(true)` aborts any active connection immediately, so its worker thread isn't still + /// blocked reading for a next request when `thread_pool`'s destructor tries to join it. + ~TestPocoHTTPServer() + { + server->stopAll(true); + thread_pool.joinAll(); + } + + /// `server_socket->address()` is the wildcard bind address (`0.0.0.0:PORT`), which is not a usable + /// connection target. Build the URL from an explicit loopback address plus the bound port instead. std::string getUrl() { - return "http://" + server_socket->address().toString(); + return "http://127.0.0.1:" + std::to_string(server_socket->address().port()); } const Poco::Net::MessageHeader & getLastRequestHeader() const @@ -157,6 +174,9 @@ class TestPocoHTTPStsServer std::unique_ptr server_socket; Poco::SharedPtr handler_factory; Poco::AutoPtr server_params; + /// See the identical member in `TestPocoHTTPServer` above: a private pool avoids + /// `TCPServerDispatcher`'s shared-pool saturation bug (base/poco/Net/src/TCPServerDispatcher.cpp). + Poco::ThreadPool thread_pool; std::unique_ptr server; // Stores the last request header handled. It's obviously not thread-safe to share the same // reference across request handlers, but it's good enough for this the purposes of this test. @@ -167,14 +187,24 @@ class TestPocoHTTPStsServer server_socket(std::make_unique(0)), handler_factory(new StsHTTPRequestHandlerFactory(last_request_info, std::move(role_access_key), std::move(role_secret_key))), server_params(new Poco::Net::HTTPServerParams()), - server(std::make_unique(handler_factory, *server_socket, server_params)) + thread_pool("TestPocoHTTPStsServer"), + server(std::make_unique(handler_factory, thread_pool, *server_socket, server_params)) { server->start(); } + /// See `TestPocoHTTPServer`'s destructor above. + ~TestPocoHTTPStsServer() + { + server->stopAll(true); + thread_pool.joinAll(); + } + + /// `server_socket->address()` is the wildcard bind address (`0.0.0.0:PORT`), which is not a usable + /// connection target. Build the URL from an explicit loopback address plus the bound port instead. std::string getUrl() { - return "http://" + server_socket->address().toString(); + return "http://127.0.0.1:" + std::to_string(server_socket->address().port()); } void resetLastRequest() diff --git a/src/IO/S3/tests/gtest_aws_s3_client.cpp b/src/IO/S3/tests/gtest_aws_s3_client.cpp index 2b477dae1c91..bdd879c4c19f 100644 --- a/src/IO/S3/tests/gtest_aws_s3_client.cpp +++ b/src/IO/S3/tests/gtest_aws_s3_client.cpp @@ -17,10 +17,8 @@ #include -#include -#include #include -#include +#include #include #include @@ -32,7 +30,6 @@ #include #include -#include #include #include #include @@ -59,7 +56,6 @@ namespace DB::S3RequestSetting namespace ProfileEvents { extern const Event S3SingleAttemptRetryConsultations; - extern const Event S3WriteRequestsErrors; } /* @@ -148,214 +144,6 @@ static void doWriteRequest(std::shared_ptr client, const D using RequestFn = std::function, const DB::S3::URI &)>; -/// Parses the `attempt=N` value `S3::setClickhouseAttemptNumber` writes into the `clickhouse-request` -/// header, straight off the wire header a real HTTP server received -- mirrors -/// `S3::getAttemptFromInfo`/`getOrEmpty` (both `static` in `Requests.cpp`, not exported), 1 when the -/// header is missing. -static size_t attemptFromHeader(const Poco::Net::MessageHeader & header) -{ - const std::string & value = header.get("clickhouse-request", ""); - static const std::string key = "attempt="; - auto pos = value.find(key); - if (pos == std::string::npos) - return 1; - try - { - return static_cast(std::stol(value.substr(pos + key.size()))); - } - catch (const std::exception &) - { - return 1; - } -} - -static std::shared_ptr makeTestClient(const DB::S3::URI & uri) -{ - DB::RemoteHostFilter remote_host_filter; - DB::S3::PocoHTTPClientConfiguration client_configuration = DB::S3::ClientFactory::instance().createClientConfiguration( - "us-east-1", - remote_host_filter, - /*s3_max_redirects=*/100, - DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, - /*s3_slow_all_threads_after_network_error=*/false, - /*s3_slow_all_threads_after_retryable_error=*/false, - /*enable_s3_requests_logging=*/false, - /*for_disk_s3=*/false, - /*opt_disk_name=*/{}, - /*request_throttler=*/{}, - uri.uri.getScheme()); - client_configuration.endpointOverride = uri.endpoint; - /// `ClientFactory::create` installs the SDK's actual retry strategy itself from - /// `client_configuration.retry_strategy`/`s3_slow_all_threads_after_retryable_error` (any - /// `retryStrategy` set here is overwritten) -- with `s3_slow_all_threads_after_retryable_error` - /// true it forces `max_retries = 1` regardless of the `RetryStrategy{.max_retries = 0}` passed - /// above, so the SDK itself retries a retryable error once before `ReadBufferFromS3`'s own - /// local-retry loop ever sees a failure, and both physical requests carry the same seeded header. - /// `false` here keeps the SDK to exactly one physical attempt, matching the CAS single-attempt - /// client's own setup. - - DB::S3::ClientSettings client_settings{ - .use_virtual_addressing = uri.is_virtual_hosted_style, - .disable_checksum = false, - .gcs_issue_compose_request = false, - .is_s3express_bucket = false, - }; - - return DB::S3::ClientFactory::instance().create( - client_configuration, - client_settings, - "ACCESS_KEY_ID", - "SECRET_ACCESS_KEY", - /*server_side_encryption_customer_key_base64=*/"", - DB::S3::ServerSideEncryptionKMSConfig(), - DB::HTTPHeaderEntries(), - DB::S3::CredentialsConfiguration{ - .use_environment_credentials = false, - .use_insecure_imds_request = false, - }); -} - -/// Fails the first `fail_first_n` requests with `fail_status` (empty body), then serves `body` with a -/// 200 to every request after. Records every request's header (not just the last) so a caller can -/// check the sequence a local retry produced. -class SequenceRecordingRequestHandler : public Poco::Net::HTTPRequestHandler -{ - std::vector & all_request_headers; - size_t & requests_seen; - size_t fail_first_n; - Poco::Net::HTTPResponse::HTTPStatus fail_status; - std::string body; - -public: - SequenceRecordingRequestHandler( - std::vector & all_request_headers_, - size_t & requests_seen_, - size_t fail_first_n_, - Poco::Net::HTTPResponse::HTTPStatus fail_status_, - std::string body_) - : all_request_headers(all_request_headers_) - , requests_seen(requests_seen_) - , fail_first_n(fail_first_n_) - , fail_status(fail_status_) - , body(std::move(body_)) - { - } - - void handleRequest(Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) override - { - all_request_headers.push_back(request); - ++requests_seen; - - if (requests_seen <= fail_first_n) - { - response.setStatus(fail_status); - response.send(); - return; - } - - response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); - response.setContentLength(static_cast(body.size())); - auto & out = response.send(); - out << body; - out.flush(); - } -}; - -class SequenceRecordingRequestHandlerFactory : public Poco::Net::HTTPRequestHandlerFactory -{ - std::vector & all_request_headers; - size_t & requests_seen; - size_t fail_first_n; - Poco::Net::HTTPResponse::HTTPStatus fail_status; - std::string body; - - Poco::Net::HTTPRequestHandler * createRequestHandler(const Poco::Net::HTTPServerRequest &) override - { - return new SequenceRecordingRequestHandler(all_request_headers, requests_seen, fail_first_n, fail_status, body); - } - -public: - SequenceRecordingRequestHandlerFactory( - std::vector & all_request_headers_, - size_t & requests_seen_, - size_t fail_first_n_, - Poco::Net::HTTPResponse::HTTPStatus fail_status_, - std::string body_) - : all_request_headers(all_request_headers_) - , requests_seen(requests_seen_) - , fail_first_n(fail_first_n_) - , fail_status(fail_status_) - , body(std::move(body_)) - { - } - - ~SequenceRecordingRequestHandlerFactory() override = default; -}; - -/// Like `TestPocoHTTPServer`, but for driving a real local retry: the first `fail_first_n` requests -/// get `fail_status`, every one after gets `body` with a 200, and every request's header is kept (not -/// just the last). Its only user is the seed test right below -- localized here rather than in the -/// shared `TestPocoHTTPServer.h` header. -class TestPocoHTTPSequenceServer -{ - std::unique_ptr server_socket; - Poco::SharedPtr handler_factory; - Poco::AutoPtr server_params; - std::unique_ptr server; - std::vector all_request_headers; - size_t requests_seen = 0; - -public: - TestPocoHTTPSequenceServer(size_t fail_first_n, Poco::Net::HTTPResponse::HTTPStatus fail_status, std::string body = {}): - server_socket(std::make_unique(0)), - handler_factory(new SequenceRecordingRequestHandlerFactory(all_request_headers, requests_seen, fail_first_n, fail_status, std::move(body))), - server_params(new Poco::Net::HTTPServerParams()), - server(std::make_unique(handler_factory, *server_socket, server_params)) - { - server->start(); - } - - std::string getUrl() - { - return "http://" + server_socket->address().toString(); - } - - const std::vector & getAllRequestHeaders() const - { - return all_request_headers; - } -}; - -/// An unset seed sends `[1, 2]` across a local retry, a seed of 2 sends `[2, 3]` -- a real HTTP round -/// trip through `TestPocoHTTPSequenceServer` is the only way to drive the retry through -/// `ReadBufferFromS3`'s actual success path (the SDK's response stream wraps a real -/// `Poco::Net::HTTPBasicStreamBuf`, which `ReadBufferFromIStream` requires). -TEST(IOTestAwsS3Client, ReadBufferFromS3AttemptSeedCarriesAcrossLocalRetry) -{ - for (const auto [seed, first, second] : {std::tuple{0, 1, 2}, {2, 2, 3}}) - { - TestPocoHTTPSequenceServer http(/*fail_first_n=*/1, Poco::Net::HTTPResponse::HTTP_INTERNAL_SERVER_ERROR, "seeded-body"); - DB::S3::URI uri(http.getUrl() + "/seeded-bucket/seeded-key"); - auto client = makeTestClient(uri); - ASSERT_TRUE(client); - - DB::ReadSettings read_settings; - read_settings.object_storage_attempt_number = seed; - DB::S3::S3RequestSettings request_settings; - request_settings[DB::S3RequestSetting::max_single_read_retries] = 2; - DB::ReadBufferFromS3 read_buffer(client, uri.bucket, uri.key, /*version_id=*/{}, request_settings, read_settings); - - String content; - DB::readStringUntilEOF(content, read_buffer); - EXPECT_EQ(content, "seeded-body"); - - const auto & headers = http.getAllRequestHeaders(); - ASSERT_EQ(headers.size(), 2u); - EXPECT_EQ(attemptFromHeader(headers[0]), first); - EXPECT_EQ(attemptFromHeader(headers[1]), second); - } -} - static void testServerSideEncryption( RequestFn do_request, bool disable_checksum, @@ -471,180 +259,6 @@ TEST(IOTestAwsS3Client, SingleAttemptRetryStrategyRefusesAndCounts) EXPECT_EQ(global_counters[ProfileEvents::S3SingleAttemptRetryConsultations].load() - before, 2u); } -namespace -{ - -/// Captures what the `S3Client` logger (`Client::log`) writes at ERROR and above. A message logged -/// below Error (e.g. Debug) never reaches the channel at this threshold, so an empty capture proves -/// the site logged below Error rather than merely that this particular text was absent. -class ScopedS3ClientErrorLogCapture -{ -public: - ScopedS3ClientErrorLogCapture() - : logger(getLogger("S3Client")) - , channel(new Poco::StreamChannel(stream)) - , old_channel(logger->getChannel(), /*shared=*/true) - , old_level(logger->getLevel()) - { - logger->setChannel(channel.get()); - logger->setLevel("error"); - } - - ~ScopedS3ClientErrorLogCapture() - { - logger->setChannel(old_channel); - logger->setLevel(old_level); - } - - std::string captured() const { return stream.str(); } - -private: - LoggerPtr logger; - std::ostringstream stream; - Poco::AutoPtr channel; - /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. - Poco::AutoPtr old_channel; - int old_level; -}; - -/// A `Client` whose `PutObject` always fails as though the connection dropped while the response body -/// was being read -- the scenario `Client::doRequestWithRetryNetworkErrors`'s `net_exception_handler` -/// exists for (the comment on that function: "network error happens when XML document is being read -/// from the response body"). Throwing here, through the same virtual `Aws::S3::S3Client::PutObject` -/// slot `Client::PutObject`'s retry loop calls, reaches `net_exception_handler` exactly as a genuine -/// mid-body network failure would, without adding a test seam to production code -- the protected -/// `Client` constructor is already exposed "for testing" (see `RecordingClient` above). -class NetworkFailingClient : public DB::S3::Client -{ -public: - NetworkFailingClient( - size_t max_redirects_, - DB::S3::ServerSideEncryptionKMSConfig sse_kms_config_, - const std::shared_ptr & credentials_provider_, - const DB::S3::PocoHTTPClientConfiguration & client_configuration_, - Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy sign_payloads_, - const DB::S3::ClientSettings & client_settings_) - : DB::S3::Client(max_redirects_, std::move(sse_kms_config_), credentials_provider_, client_configuration_, sign_payloads_, client_settings_) - { - } - - Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest &) const override - { - ++attempts; - throw Poco::TimeoutException("mock timeout reading the response body"); - } - - mutable size_t attempts = 0; -}; - -std::shared_ptr makeNetworkFailingClient(std::shared_ptr retry_strategy) -{ - DB::RemoteHostFilter remote_host_filter; - DB::S3::PocoHTTPClientConfiguration client_configuration = DB::S3::ClientFactory::instance().createClientConfiguration( - /*force_region=*/"us-east-1", - remote_host_filter, - /*s3_max_redirects=*/100, - DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, - /*s3_slow_all_threads_after_network_error=*/false, - /*s3_slow_all_threads_after_retryable_error=*/false, - /*enable_s3_requests_logging=*/false, - /*for_disk_s3=*/false, - /*opt_disk_name=*/{}, - /*request_throttler=*/{}); - /// `PutObject` never reaches the wire (it is overridden below), so the endpoint is irrelevant -- - /// only the installed retry strategy, which is what `usesSingleAttemptRetryStrategy` inspects. - client_configuration.retryStrategy = std::move(retry_strategy); - - DB::S3::ClientSettings client_settings{ - .use_virtual_addressing = true, - .disable_checksum = false, - .gcs_issue_compose_request = false, - .is_s3express_bucket = false, - }; - - Aws::Auth::AWSCredentials credentials("ACCESS_KEY_ID", "SECRET_ACCESS_KEY"); - auto credentials_provider = DB::S3::getCredentialsProvider( - client_configuration, - credentials, - DB::S3::CredentialsConfiguration{.use_environment_credentials = false, .use_insecure_imds_request = false}); - - return std::make_shared( - /*max_redirects_=*/100, - DB::S3::ServerSideEncryptionKMSConfig{}, - credentials_provider, - client_configuration, - Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, - client_settings); -} - -} - -/// The client's own retry strategy still says "do not retry" (`SingleAttemptRetryStrategy::ShouldRetry` -/// always false, see `SingleAttemptRetryStrategyRefusesAndCounts` above); `usesSingleAttemptRetryStrategy` -/// is a separate, purely descriptive check of which strategy is installed, tested directly here. -TEST(IOTestAwsS3Client, UsesSingleAttemptRetryStrategyIdentifiesTheInstalledStrategy) -{ - auto single_attempt_client = makeNetworkFailingClient(std::make_shared()); - EXPECT_TRUE(single_attempt_client->usesSingleAttemptRetryStrategy()); - - DB::S3::PocoHTTPClientConfiguration::RetryStrategy zero_retries{.max_retries = 0}; - auto ordinary_client = makeNetworkFailingClient(std::make_shared(zero_retries)); - EXPECT_FALSE(ordinary_client->usesSingleAttemptRetryStrategy()); -} - -/// A client carrying the `SingleAttemptRetryStrategy` (the CAS conditional-write client, see -/// `S3ObjectStorage::getSingleAttemptClient`) is owned by an outer retry loop that resolves the outcome -/// and reissues; its one failed attempt is not terminal, so the network-error log site must not reach -/// Error. -TEST(IOTestAwsS3Client, NetworkErrorLogsDebugForSingleAttemptStrategy) -{ - using ProfileEvents::global_counters; - const auto errors_before = global_counters[ProfileEvents::S3WriteRequestsErrors].load(); - - auto client = makeNetworkFailingClient(std::make_shared()); - DB::S3::PutObjectRequest request; - - /// Call through the `DB::S3::Client&` interface, exactly as production code (which only ever - /// holds a `Client`, never `NetworkFailingClient`) does: `NetworkFailingClient::PutObject` hides - /// `Client::PutObject(PutObjectRequest&)` -- the retry-loop wrapper under test -- from lookup on - /// the derived type, so calling through the base is what makes this test exercise that wrapper - /// rather than the override directly. - const DB::S3::Client & base_client = *client; - ScopedS3ClientErrorLogCapture log_capture; - const auto outcome = base_client.PutObject(request); - - EXPECT_FALSE(outcome.IsSuccess()); - EXPECT_EQ(outcome.GetError().GetErrorType(), Aws::S3::S3Errors::NETWORK_CONNECTION); - EXPECT_EQ(client->attempts, 1u); - EXPECT_EQ(global_counters[ProfileEvents::S3WriteRequestsErrors].load() - errors_before, 1u); - EXPECT_TRUE(log_capture.captured().empty()); -} - -/// `max_retries = 0` on the ORDINARY strategy is a supported user configuration (`s3_retry_attempts`) -/// with no outer retry loop: its one failed attempt IS the final answer, so it must keep logging at -/// Error -- this is exactly the case a signal keyed on `max_retries == 0` alone would misclassify. -TEST(IOTestAwsS3Client, NetworkErrorLogsErrorForOrdinaryZeroRetryStrategy) -{ - using ProfileEvents::global_counters; - const auto errors_before = global_counters[ProfileEvents::S3WriteRequestsErrors].load(); - - DB::S3::PocoHTTPClientConfiguration::RetryStrategy zero_retries{.max_retries = 0}; - auto client = makeNetworkFailingClient(std::make_shared(zero_retries)); - DB::S3::PutObjectRequest request; - - /// See the comment in `NetworkErrorLogsDebugForSingleAttemptStrategy`: calling through the base - /// is what reaches `Client::PutObject`'s retry-loop wrapper rather than the override directly. - const DB::S3::Client & base_client = *client; - ScopedS3ClientErrorLogCapture log_capture; - const auto outcome = base_client.PutObject(request); - - EXPECT_FALSE(outcome.IsSuccess()); - EXPECT_EQ(outcome.GetError().GetErrorType(), Aws::S3::S3Errors::NETWORK_CONNECTION); - EXPECT_EQ(client->attempts, 1u); - EXPECT_EQ(global_counters[ProfileEvents::S3WriteRequestsErrors].load() - errors_before, 1u); - EXPECT_NE(log_capture.captured().find("Network error on S3 request, attempt 1 of 1"), std::string::npos); -} - struct ConditionalPutWireObservation { bool negotiated_expect_continue = false; @@ -1442,11 +1056,20 @@ class ScriptedResponseServer , server_socket(std::make_unique(0)) , handler_factory(new Factory(*this)) , server_params(new Poco::Net::HTTPServerParams()) - , server(std::make_unique(handler_factory, *server_socket, server_params)) + , thread_pool("ScriptedResponseServer") + , server(std::make_unique(handler_factory, thread_pool, *server_socket, server_params)) { server->start(); } + /// `stopAll(true)` aborts any active connection immediately, so its worker thread isn't still + /// blocked reading for a next request when `thread_pool`'s destructor tries to join it. + ~ScriptedResponseServer() + { + server->stopAll(true); + thread_pool.joinAll(); + } + /// `server_socket->address()` is the wildcard bind address (`0.0.0.0:PORT`), which is not a usable /// connection target and could silently conflate distinct servers under the same host string. Build /// the URL from an explicit loopback address plus the bound port instead. @@ -1501,6 +1124,11 @@ class ScriptedResponseServer std::unique_ptr server_socket; Poco::SharedPtr handler_factory; Poco::AutoPtr server_params; + /// A dedicated pool, not `Poco::ThreadPool::defaultPool()` (the `HTTPServer` default): that pool + /// is shared with every other local-server test in this binary, and `TCPServerDispatcher::enqueue` + /// (base/poco/Net/src/TCPServerDispatcher.cpp) has an acknowledged-in-comment saturation-check race + /// when it's shared, which can accept a connection and then close it with no response. + Poco::ThreadPool thread_pool; std::unique_ptr server; }; diff --git a/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp b/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp new file mode 100644 index 000000000000..22e7951fbeec --- /dev/null +++ b/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp @@ -0,0 +1,475 @@ +#include + +#include +#include "config.h" + + +#if USE_AWS_S3 + +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB::S3RequestSetting +{ + extern const S3RequestSettingsUInt64 max_single_read_retries; +} + +namespace ProfileEvents +{ + extern const Event S3WriteRequestsErrors; +} + +/// Parses the `attempt=N` value `S3::setClickhouseAttemptNumber` writes into the `clickhouse-request` +/// header, straight off the wire header a real HTTP server received -- mirrors +/// `S3::getAttemptFromInfo`/`getOrEmpty` (both `static` in `Requests.cpp`, not exported), 1 when the +/// header is missing. +static size_t attemptFromHeader(const Poco::Net::MessageHeader & header) +{ + const std::string & value = header.get("clickhouse-request", ""); + static const std::string key = "attempt="; + auto pos = value.find(key); + if (pos == std::string::npos) + return 1; + try + { + return static_cast(std::stol(value.substr(pos + key.size()))); + } + catch (const std::exception &) + { + return 1; + } +} + +static std::shared_ptr makeTestClient(const DB::S3::URI & uri) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration client_configuration = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", + remote_host_filter, + /*s3_max_redirects=*/100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /*s3_slow_all_threads_after_network_error=*/false, + /*s3_slow_all_threads_after_retryable_error=*/false, + /*enable_s3_requests_logging=*/false, + /*for_disk_s3=*/false, + /*opt_disk_name=*/{}, + /*request_throttler=*/{}, + uri.uri.getScheme()); + client_configuration.endpointOverride = uri.endpoint; + /// `ClientFactory::create` installs the SDK's actual retry strategy itself from + /// `client_configuration.retry_strategy`/`s3_slow_all_threads_after_retryable_error` (any + /// `retryStrategy` set here is overwritten) -- with `s3_slow_all_threads_after_retryable_error` + /// true it forces `max_retries = 1` regardless of the `RetryStrategy{.max_retries = 0}` passed + /// above, so the SDK itself retries a retryable error once before `ReadBufferFromS3`'s own + /// local-retry loop ever sees a failure, and both physical requests carry the same seeded header. + /// `false` here keeps the SDK to exactly one physical attempt, matching the CAS single-attempt + /// client's own setup. + + DB::S3::ClientSettings client_settings{ + .use_virtual_addressing = uri.is_virtual_hosted_style, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }; + + return DB::S3::ClientFactory::instance().create( + client_configuration, + client_settings, + "ACCESS_KEY_ID", + "SECRET_ACCESS_KEY", + /*server_side_encryption_customer_key_base64=*/"", + DB::S3::ServerSideEncryptionKMSConfig(), + DB::HTTPHeaderEntries(), + DB::S3::CredentialsConfiguration{ + .use_environment_credentials = false, + .use_insecure_imds_request = false, + }); +} + +/// Anonymous namespace: these three classes have no counterpart in gtest_aws_s3_client.cpp today, but +/// giving them internal linkage costs nothing and avoids ever silently colliding with a same-named +/// class that file adds later (see the equivalent note in gtest_cas_readbuffer_s3.cpp for what such a +/// collision actually does at link time). +namespace +{ + +/// Fails the first `fail_first_n` requests with `fail_status` (empty body), then serves `body` with a +/// 200 to every request after. Records every request's header (not just the last) so a caller can +/// check the sequence a local retry produced. +class SequenceRecordingRequestHandler : public Poco::Net::HTTPRequestHandler +{ + std::vector & all_request_headers; + size_t & requests_seen; + size_t fail_first_n; + Poco::Net::HTTPResponse::HTTPStatus fail_status; + std::string body; + +public: + SequenceRecordingRequestHandler( + std::vector & all_request_headers_, + size_t & requests_seen_, + size_t fail_first_n_, + Poco::Net::HTTPResponse::HTTPStatus fail_status_, + std::string body_) + : all_request_headers(all_request_headers_) + , requests_seen(requests_seen_) + , fail_first_n(fail_first_n_) + , fail_status(fail_status_) + , body(std::move(body_)) + { + } + + void handleRequest(Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) override + { + all_request_headers.push_back(request); + ++requests_seen; + + if (requests_seen <= fail_first_n) + { + response.setStatus(fail_status); + response.send(); + return; + } + + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.setContentLength(static_cast(body.size())); + auto & out = response.send(); + out << body; + out.flush(); + } +}; + +class SequenceRecordingRequestHandlerFactory : public Poco::Net::HTTPRequestHandlerFactory +{ + std::vector & all_request_headers; + size_t & requests_seen; + size_t fail_first_n; + Poco::Net::HTTPResponse::HTTPStatus fail_status; + std::string body; + + Poco::Net::HTTPRequestHandler * createRequestHandler(const Poco::Net::HTTPServerRequest &) override + { + return new SequenceRecordingRequestHandler(all_request_headers, requests_seen, fail_first_n, fail_status, body); + } + +public: + SequenceRecordingRequestHandlerFactory( + std::vector & all_request_headers_, + size_t & requests_seen_, + size_t fail_first_n_, + Poco::Net::HTTPResponse::HTTPStatus fail_status_, + std::string body_) + : all_request_headers(all_request_headers_) + , requests_seen(requests_seen_) + , fail_first_n(fail_first_n_) + , fail_status(fail_status_) + , body(std::move(body_)) + { + } + + ~SequenceRecordingRequestHandlerFactory() override = default; +}; + +/// Like `TestPocoHTTPServer`, but for driving a real local retry: the first `fail_first_n` requests +/// get `fail_status`, every one after gets `body` with a 200, and every request's header is kept (not +/// just the last). Its only user is the seed test right below -- localized here rather than in the +/// shared `TestPocoHTTPServer.h` header. +class TestPocoHTTPSequenceServer +{ + std::unique_ptr server_socket; + Poco::SharedPtr handler_factory; + Poco::AutoPtr server_params; + /// A dedicated pool, not `Poco::ThreadPool::defaultPool()` (the `HTTPServer` default): that pool + /// is shared with every other local-server test in this binary, and `TCPServerDispatcher::enqueue` + /// (base/poco/Net/src/TCPServerDispatcher.cpp) has an acknowledged-in-comment saturation-check race + /// when it's shared, which can accept a connection and then close it with no response. + Poco::ThreadPool thread_pool; + std::unique_ptr server; + std::vector all_request_headers; + size_t requests_seen = 0; + +public: + TestPocoHTTPSequenceServer(size_t fail_first_n, Poco::Net::HTTPResponse::HTTPStatus fail_status, std::string body = {}): + /// Bind to the loopback address explicitly, not `ServerSocket(0)`'s wildcard `0.0.0.0`: the + /// latter is not a valid connection target, even though the kernel happens to tolerate a + /// connect() to it as loopback on Linux. + server_socket(std::make_unique(Poco::Net::SocketAddress("127.0.0.1", 0))), + handler_factory(new SequenceRecordingRequestHandlerFactory(all_request_headers, requests_seen, fail_first_n, fail_status, std::move(body))), + server_params(new Poco::Net::HTTPServerParams()), + thread_pool("TestPocoHTTPSequenceServer"), + server(std::make_unique(handler_factory, thread_pool, *server_socket, server_params)) + { + server->start(); + } + + /// `stopAll(true)` aborts any active connection immediately, so its worker thread isn't still + /// blocked reading for a next request when `thread_pool`'s destructor tries to join it. + ~TestPocoHTTPSequenceServer() + { + server->stopAll(true); + thread_pool.joinAll(); + } + + std::string getUrl() + { + return "http://" + server_socket->address().toString(); + } + + const std::vector & getAllRequestHeaders() const + { + return all_request_headers; + } +}; + +} + +/// An unset seed sends `[1, 2]` across a local retry, a seed of 2 sends `[2, 3]` -- a real HTTP round +/// trip through `TestPocoHTTPSequenceServer` is the only way to drive the retry through +/// `ReadBufferFromS3`'s actual success path (the SDK's response stream wraps a real +/// `Poco::Net::HTTPBasicStreamBuf`, which `ReadBufferFromIStream` requires). +TEST(CASIOTestAwsS3Client, ReadBufferFromS3AttemptSeedCarriesAcrossLocalRetry) +{ + for (const auto [seed, first, second] : {std::tuple{0, 1, 2}, {2, 2, 3}}) + { + TestPocoHTTPSequenceServer http(/*fail_first_n=*/1, Poco::Net::HTTPResponse::HTTP_INTERNAL_SERVER_ERROR, "seeded-body"); + DB::S3::URI uri(http.getUrl() + "/seeded-bucket/seeded-key"); + auto client = makeTestClient(uri); + ASSERT_TRUE(client); + + DB::ReadSettings read_settings; + read_settings.object_storage_attempt_number = seed; + DB::S3::S3RequestSettings request_settings; + request_settings[DB::S3RequestSetting::max_single_read_retries] = 2; + DB::ReadBufferFromS3 read_buffer(client, uri.bucket, uri.key, /*version_id=*/{}, request_settings, read_settings); + + String content; + DB::readStringUntilEOF(content, read_buffer); + EXPECT_EQ(content, "seeded-body"); + + const auto & headers = http.getAllRequestHeaders(); + ASSERT_EQ(headers.size(), 2u); + EXPECT_EQ(attemptFromHeader(headers[0]), first); + EXPECT_EQ(attemptFromHeader(headers[1]), second); + } +} + +namespace +{ + +/// Captures what the `S3Client` logger (`Client::log`) writes at ERROR and above. A message logged +/// below Error (e.g. Debug) never reaches the channel at this threshold, so an empty capture proves +/// the site logged below Error rather than merely that this particular text was absent. +class ScopedS3ClientErrorLogCapture +{ +public: + ScopedS3ClientErrorLogCapture() + : logger(getLogger("S3Client")) + , channel(new Poco::StreamChannel(stream)) + , old_channel(logger->getChannel(), /*shared=*/true) + , old_level(logger->getLevel()) + { + logger->setChannel(channel.get()); + logger->setLevel("error"); + } + + ~ScopedS3ClientErrorLogCapture() + { + logger->setChannel(old_channel); + logger->setLevel(old_level); + } + + std::string captured() const { return stream.str(); } + +private: + LoggerPtr logger; + std::ostringstream stream; + Poco::AutoPtr channel; + /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. + Poco::AutoPtr old_channel; + int old_level; +}; + +/// A `Client` whose `PutObject` always fails as though the connection dropped while the response body +/// was being read -- the scenario `Client::doRequestWithRetryNetworkErrors`'s `net_exception_handler` +/// exists for (the comment on that function: "network error happens when XML document is being read +/// from the response body"). Throwing here, through the same virtual `Aws::S3::S3Client::PutObject` +/// slot `Client::PutObject`'s retry loop calls, reaches `net_exception_handler` exactly as a genuine +/// mid-body network failure would, without adding a test seam to production code -- the protected +/// `Client` constructor is already exposed "for testing" (see `RecordingClient` in `gtest_aws_s3_client.cpp`). +class NetworkFailingClient : public DB::S3::Client +{ +public: + NetworkFailingClient( + size_t max_redirects_, + DB::S3::ServerSideEncryptionKMSConfig sse_kms_config_, + const std::shared_ptr & credentials_provider_, + const DB::S3::PocoHTTPClientConfiguration & client_configuration_, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy sign_payloads_, + const DB::S3::ClientSettings & client_settings_) + : DB::S3::Client(max_redirects_, std::move(sse_kms_config_), credentials_provider_, client_configuration_, sign_payloads_, client_settings_) + { + } + + Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest &) const override + { + ++attempts; + throw Poco::TimeoutException("mock timeout reading the response body"); + } + + mutable size_t attempts = 0; +}; + +std::shared_ptr makeNetworkFailingClient(std::shared_ptr retry_strategy) +{ + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration client_configuration = DB::S3::ClientFactory::instance().createClientConfiguration( + /*force_region=*/"us-east-1", + remote_host_filter, + /*s3_max_redirects=*/100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /*s3_slow_all_threads_after_network_error=*/false, + /*s3_slow_all_threads_after_retryable_error=*/false, + /*enable_s3_requests_logging=*/false, + /*for_disk_s3=*/false, + /*opt_disk_name=*/{}, + /*request_throttler=*/{}); + /// `PutObject` never reaches the wire (it is overridden below), so the endpoint is irrelevant -- + /// only the installed retry strategy, which is what `usesSingleAttemptRetryStrategy` inspects. + client_configuration.retryStrategy = std::move(retry_strategy); + + DB::S3::ClientSettings client_settings{ + .use_virtual_addressing = true, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }; + + Aws::Auth::AWSCredentials credentials("ACCESS_KEY_ID", "SECRET_ACCESS_KEY"); + auto credentials_provider = DB::S3::getCredentialsProvider( + client_configuration, + credentials, + DB::S3::CredentialsConfiguration{.use_environment_credentials = false, .use_insecure_imds_request = false}); + + return std::make_shared( + /*max_redirects_=*/100, + DB::S3::ServerSideEncryptionKMSConfig{}, + credentials_provider, + client_configuration, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + client_settings); +} + +} + +/// The client's own retry strategy still says "do not retry" (`SingleAttemptRetryStrategy::ShouldRetry` +/// always false, tested directly against `DoesNotRetryPreconditionFailed`/`SingleAttemptRetryStrategyRefusesAndCounts` +/// in `gtest_aws_s3_client.cpp`); `usesSingleAttemptRetryStrategy` is a separate, purely descriptive +/// check of which strategy is installed, tested directly here. +TEST(CASIOTestAwsS3Client, UsesSingleAttemptRetryStrategyIdentifiesTheInstalledStrategy) +{ + auto single_attempt_client = makeNetworkFailingClient(std::make_shared()); + EXPECT_TRUE(single_attempt_client->usesSingleAttemptRetryStrategy()); + + DB::S3::PocoHTTPClientConfiguration::RetryStrategy zero_retries{.max_retries = 0}; + auto ordinary_client = makeNetworkFailingClient(std::make_shared(zero_retries)); + EXPECT_FALSE(ordinary_client->usesSingleAttemptRetryStrategy()); +} + +/// A client carrying the `SingleAttemptRetryStrategy` (the CAS conditional-write client, see +/// `S3ObjectStorage::getSingleAttemptClient`) is owned by an outer retry loop that resolves the outcome +/// and reissues; its one failed attempt is not terminal, so the network-error log site must not reach +/// Error. +TEST(CASIOTestAwsS3Client, NetworkErrorLogsDebugForSingleAttemptStrategy) +{ + using ProfileEvents::global_counters; + const auto errors_before = global_counters[ProfileEvents::S3WriteRequestsErrors].load(); + + auto client = makeNetworkFailingClient(std::make_shared()); + DB::S3::PutObjectRequest request; + + /// Call through the `DB::S3::Client&` interface, exactly as production code (which only ever + /// holds a `Client`, never `NetworkFailingClient`) does: `NetworkFailingClient::PutObject` hides + /// `Client::PutObject(PutObjectRequest&)` -- the retry-loop wrapper under test -- from lookup on + /// the derived type, so calling through the base is what makes this test exercise that wrapper + /// rather than the override directly. + const DB::S3::Client & base_client = *client; + ScopedS3ClientErrorLogCapture log_capture; + const auto outcome = base_client.PutObject(request); + + EXPECT_FALSE(outcome.IsSuccess()); + EXPECT_EQ(outcome.GetError().GetErrorType(), Aws::S3::S3Errors::NETWORK_CONNECTION); + EXPECT_EQ(client->attempts, 1u); + EXPECT_EQ(global_counters[ProfileEvents::S3WriteRequestsErrors].load() - errors_before, 1u); + EXPECT_TRUE(log_capture.captured().empty()); +} + +/// `max_retries = 0` on the ORDINARY strategy is a supported user configuration (`s3_retry_attempts`) +/// with no outer retry loop: its one failed attempt IS the final answer, so it must keep logging at +/// Error -- this is exactly the case a signal keyed on `max_retries == 0` alone would misclassify. +TEST(CASIOTestAwsS3Client, NetworkErrorLogsErrorForOrdinaryZeroRetryStrategy) +{ + using ProfileEvents::global_counters; + const auto errors_before = global_counters[ProfileEvents::S3WriteRequestsErrors].load(); + + DB::S3::PocoHTTPClientConfiguration::RetryStrategy zero_retries{.max_retries = 0}; + auto client = makeNetworkFailingClient(std::make_shared(zero_retries)); + DB::S3::PutObjectRequest request; + + /// See the comment in `NetworkErrorLogsDebugForSingleAttemptStrategy`: calling through the base + /// is what reaches `Client::PutObject`'s retry-loop wrapper rather than the override directly. + const DB::S3::Client & base_client = *client; + ScopedS3ClientErrorLogCapture log_capture; + const auto outcome = base_client.PutObject(request); + + EXPECT_FALSE(outcome.IsSuccess()); + EXPECT_EQ(outcome.GetError().GetErrorType(), Aws::S3::S3Errors::NETWORK_CONNECTION); + EXPECT_EQ(client->attempts, 1u); + EXPECT_EQ(global_counters[ProfileEvents::S3WriteRequestsErrors].load() - errors_before, 1u); + EXPECT_NE(log_capture.captured().find("Network error on S3 request, attempt 1 of 1"), std::string::npos); +} + +#endif From 9b099880aa1a1bdfa5b4caf3cb26abac05822112 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:31:37 +0200 Subject: [PATCH 63/81] s3: move fork-added identity-check tests out of gtest_readbuffer_s3.cpp gtest_readbuffer_s3.cpp is an upstream test file; every fork-added test in it made rebasing conflict. The fork's additions (the responseIdentityChanged coverage plus BreakingHTTPBasicStreamBuf/rangeStart/makeGetObjectOutcome helpers) do not touch any upstream code and are not used by any upstream test, so they move into the new fork-owned gtest_cas_readbuffer_s3.cpp. The moved tests still need ClientFake/readAndAssert/CountedSession/ StringHTTPBasicStreamBuf and the fixture. The new file carries its own trimmed copies inside an anonymous namespace (internal linkage), and the fixture is renamed CASReadBufferFromS3Test: a same-named external-linkage copy would be a One Definition Rule violation -- the trimmed ClientFake is a different class than the upstream file's, and the fixture references file-local statics (cache_base_path, caches_dir, TEST_LOG_LEVEL) that are distinct entities per translation unit. An early draft with external linkage made the linker keep one ClientFake vtable for both files and sent the upstream ListObjectsV2-based tests through the trimmed override set, where they failed with a real 403 against a real endpoint. `git diff --stat` for gtest_readbuffer_s3.cpp against altinity/antalya-26.6 goes from 412 insertions to 0 (byte-identical to base). Test count for `unit_tests_dbms --gtest_filter='*S3*:*ReadBuffer*:*WriteBuffer*'` is unchanged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/IO/tests/gtest_cas_readbuffer_s3.cpp | 546 +++++++++++++++++++++++ src/IO/tests/gtest_readbuffer_s3.cpp | 412 ----------------- 2 files changed, 546 insertions(+), 412 deletions(-) create mode 100644 src/IO/tests/gtest_cas_readbuffer_s3.cpp diff --git a/src/IO/tests/gtest_cas_readbuffer_s3.cpp b/src/IO/tests/gtest_cas_readbuffer_s3.cpp new file mode 100644 index 000000000000..0852e6460e08 --- /dev/null +++ b/src/IO/tests/gtest_cas_readbuffer_s3.cpp @@ -0,0 +1,546 @@ +#include + +#include +#include + +#include +#include "config.h" + +#if USE_AWS_S3 + +#include +#include +#include +#include +#include +#include + +namespace DB::ErrorCodes +{ + extern const int S3_ERROR; +} + +static constexpr auto TEST_LOG_LEVEL = "debug"; +static fs::path caches_dir = fs::current_path() / "readbuffer_s3"; +static std::string cache_base_path = caches_dir / "cache1" / ""; + +/// Everything below, including the fixture, has internal linkage: `gtest_readbuffer_s3.cpp` defines +/// its own, different `ClientFake`/`CountedSession`/etc. under the same names, and this fixture +/// references file-local `static`s, so external linkage here would be an ODR violation. The suite is +/// renamed to `CASReadBufferFromS3Test` (test names unchanged) so it doesn't share a suite name with +/// that file's fixture, which gtest's own registration-time check would otherwise reject. +namespace +{ + +/// A copy of `ReadBufferFromS3Test` from `gtest_readbuffer_s3.cpp`, renamed per the note above. +class CASReadBufferFromS3Test : public ::testing::Test +{ +public: + static void setupLogs(const std::string & level) + { + Poco::AutoPtr channel(new Poco::ConsoleChannel(std::cerr)); + Poco::Logger::root().setChannel(channel); + Poco::Logger::root().setLevel(level); + } + + void SetUp() override + { + if (const char * test_log_level = std::getenv("TEST_LOG_LEVEL")) // NOLINT(concurrency-mt-unsafe) + setupLogs(test_log_level); + else + setupLogs(TEST_LOG_LEVEL); + + if (fs::exists(cache_base_path)) + fs::remove_all(cache_base_path); + fs::create_directories(cache_base_path); + } + + void TearDown() override + { + if (fs::exists(cache_base_path)) + fs::remove_all(cache_base_path); + } +}; + +/// A copy of `CountedSession` from `gtest_readbuffer_s3.cpp`, an opaque session marker for +/// `SessionAwareIOStream`. That file's version counts live instances for its session-lifetime tests; +/// no test in this file reads such a count, and an internal-linkage member nothing calls or reads +/// trips `-Wunused-member-function`/`-Wunneeded-member-function`, so this copy carries no state at all. +class CountedSession +{ +}; + +using CountedSessionPtr = std::shared_ptr; + +class StringHTTPBasicStreamBuf : public Poco::Net::HTTPBasicStreamBuf +{ +public: + explicit StringHTTPBasicStreamBuf(std::string body) : BasicBufferedStreamBuf(body.size(), IOS::in), bodyStream(std::stringstream(body)) + { + } + +private: + std::stringstream bodyStream; + + int readFromDevice(char_type * buf, std::streamsize n) override + { + bodyStream.read(buf, n); + return static_cast(bodyStream.gcount()); + } +}; + +/// A response body stream that throws once `bytes_before_failure` bytes have been handed out (0 means +/// the very first read fails), simulating a GET whose headers arrived successfully but whose body read +/// broke before delivering that many bytes to the consumer. +class BreakingHTTPBasicStreamBuf : public Poco::Net::HTTPBasicStreamBuf +{ +public: + BreakingHTTPBasicStreamBuf(std::string body, size_t bytes_before_failure_) + : BasicBufferedStreamBuf(body.size(), IOS::in), bodyStream(std::stringstream(std::move(body))), bytes_before_failure(bytes_before_failure_) + { + } + +private: + std::stringstream bodyStream; + size_t bytes_before_failure; + + int readFromDevice(char_type * buf, std::streamsize n) override + { + if (bytes_before_failure == 0) + throw DB::Exception(DB::ErrorCodes::S3_ERROR, "Simulated S3 body read failure"); + + bodyStream.read(buf, std::min(n, static_cast(bytes_before_failure))); + const auto got = bodyStream.gcount(); + bytes_before_failure -= static_cast(got); + return static_cast(got); + } +}; + +/// The byte offset the request's Range header asks for, or 0 when no Range was set. sendRequest() +/// always emits "bytes=-" or "bytes=-", so parsing out lets a mock GetObject +/// serve the bytes a reissued request should actually receive. +static size_t rangeStart(const Aws::S3::Model::GetObjectRequest & request) +{ + if (!request.RangeHasBeenSet()) + return 0; + const std::string & range = request.GetRange(); + const size_t begin_pos = range.find('=') + 1; + const size_t dash_pos = range.find('-', begin_pos); + return std::stoull(range.substr(begin_pos, dash_pos - begin_pos)); +} + +static Aws::S3::Model::GetObjectOutcome makeGetObjectOutcome(std::streambuf * sb, const std::string & etag) +{ + Aws::Http::HeaderValueCollection headers; + headers["etag"] = etag; + auto response_stream = Aws::Utils::Stream::ResponseStream( + Aws::New>("test response stream", std::make_shared(), sb)); + Aws::AmazonWebServiceResult aws_result(std::move(response_stream), std::move(headers)); + DB::S3::Model::GetObjectResult result(std::move(aws_result)); + return Aws::S3::Model::GetObjectOutcome(std::move(result)); +} + +using GetObjectFn = std::function; + +/// A trimmed copy of `ClientFake` from `gtest_readbuffer_s3.cpp`: only the `GetObject` override this +/// file's tests need. It deliberately does NOT match that file's `ClientFake` (which also overrides +/// `ListObjectsV2`) -- see the anonymous-namespace comment above for why that's required, not optional. +struct ClientFake : DB::S3::Client +{ + explicit ClientFake() + : DB::S3::Client( + 1, + DB::S3::ServerSideEncryptionKMSConfig(), + std::make_shared("test_access_key", "test_secret"), + DB::S3::ClientFactory::instance().createClientConfiguration( + "test_region", + DB::RemoteHostFilter(), + 1, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + true, + true, + true, + false, + {}, + /* request_throttler = */ {}, + "http"), + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + DB::S3::ClientSettings()) + { + } + + std::optional getObjectImpl; + + Aws::S3::Model::GetObjectOutcome GetObject([[maybe_unused]] const Aws::S3::Model::GetObjectRequest & request) const override + { + chassert(getObjectImpl); + return (*getObjectImpl)(request); + } +}; + +static void readAndAssert(DB::ReadBuffer & buf, const char * str) +{ + size_t n = strlen(str); + std::vector tmp(n); + buf.readStrict(tmp.data(), n); + ASSERT_EQ(strncmp(tmp.data(), str, n), 0); +} + +} + +TEST_F(CASReadBufferFromS3Test, IdentityNotFlaggedWhenFailedAttemptDeliveredNoBytes) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 20; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto failing_buf = std::make_shared(body, /* bytes_before_failure */ 0); + auto full_buf = std::make_shared(body); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + EXPECT_EQ(rangeStart(request), 0); + if (call == 1) + return makeGetObjectOutcome(failing_buf.get(), "A"); + return makeGetObjectOutcome(full_buf.get(), "B"); + }; + + /// First attempt's headers carried ETag "A", but its body read fails before any byte reaches the + /// consumer; the reissue delivers the whole object under ETag "B". No bytes of "A" were ever + /// consumed, so this must not be flagged as a coherence problem. + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, IdentityFlaggedWhenBytesDeliveredBeforeFailure) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + /// The first response (ETag "A") delivers 3 bytes before its stream breaks; the reissue, resuming + /// from offset 3, answers with ETag "B". Bytes from two different incarnations reached the + /// consumer, so this must be flagged. + readAndAssert(subject, body.c_str()); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, IdentityNotFlaggedWhenReissuedEtagMatches) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "A"); + }; + + /// Same as above, but the reissue answers with the same ETag "A": both attempts belong to the same + /// incarnation, so this must not be flagged. + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, ThreeResponsesABytesThenAEmptyFailThenBBytesIsFlagged) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto delivers_then_fails = std::make_shared(body, /* bytes_before_failure */ 3); + auto fails_empty = std::make_shared(body, /* bytes_before_failure */ 0); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(delivers_then_fails.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + if (call == 2) + return makeGetObjectOutcome(fails_empty.get(), "A"); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + /// A delivers 3 bytes, then breaks. The reissue (same ETag "A") fails before delivering anything. + /// The next reissue answers with ETag "B" and delivers the rest: A-bytes and B-bytes were mixed, so + /// this must be flagged, even though an empty failed attempt for "A" sat in between. + readAndAssert(subject, body.c_str()); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, ThreeResponsesABytesThenBEmptyFailThenABytesIsNotFlagged) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto delivers_then_fails = std::make_shared(body, /* bytes_before_failure */ 3); + auto fails_empty = std::make_shared(body, /* bytes_before_failure */ 0); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(delivers_then_fails.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + if (call == 2) + return makeGetObjectOutcome(fails_empty.get(), "B"); + return makeGetObjectOutcome(rest_buf.get(), "A"); + }; + + /// A delivers 3 bytes, then breaks. The reissue under ETag "B" fails before delivering anything, so + /// it never contributes to the read. The next reissue answers with ETag "A" (matching the only + /// response that ever delivered bytes) and delivers the rest: the read is coherent and must not be + /// flagged, even though a differently-ETagged empty failed attempt sat in between. + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); + ASSERT_EQ(subject.getObjectMetadataFromTheLastRequest().etag, "A"); +} + +TEST_F(CASReadBufferFromS3Test, SeekReissueAcceptsNewEtagWithoutFlag) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + read_settings.remote_fs_settings.min_bytes_for_seek = 0; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto first_buf = std::make_shared(body); + auto after_seek_buf = std::make_shared(body.substr(8)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(first_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 8); + return makeGetObjectOutcome(after_seek_buf.get(), "B"); + }; + + readAndAssert(subject, "123"); + /// A seek far enough ahead to force a reissue (not an in-buffer rewind, not a small forward skip): + /// the caller explicitly repositioned to a different range, so the new response's ETag "B" must not + /// be compared against "A". + subject.seek(8, SEEK_SET); + readAndAssert(subject, "9"); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, SetReadUntilPositionReissueAcceptsNewEtagWithoutFlag) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 2; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456"; + auto first_buf = std::make_shared(body); + auto after_reposition_buf = std::make_shared(body.substr(2, 3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(first_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 2); + return makeGetObjectOutcome(after_reposition_buf.get(), "B"); + }; + + readAndAssert(subject, "12"); + /// impl is still open (no read-until-position was set yet, so nothing released it). Narrowing the + /// read-until bound now tears impl down to reissue for the new bound: an explicit reposition, so + /// the new response's ETag "B" must not be compared against "A". + subject.setReadUntilPosition(5); + readAndAssert(subject, "345"); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, SetReadUntilEndReissueAcceptsNewEtagWithoutFlag) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + subject.setReadUntilPosition(3); + + const std::string body = "123456789"; + auto first_buf = std::make_shared(body.substr(0, 3)); + auto after_reposition_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(first_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(after_reposition_buf.get(), "B"); + }; + + readAndAssert(subject, "123"); + /// Reading exactly up to the bound releases the result (does not reset impl). Removing the bound + /// now tears impl down to reissue for the rest of the object: an explicit reposition, so the new + /// response's ETag "B" must not be compared against "A". + subject.setReadUntilEnd(); + readAndAssert(subject, "456789"); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, InBufferSeekPreservesBaselineAndLaterMixedRetryIsFlagged) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 3; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + readAndAssert(subject, "123"); + /// Rewind within the bytes already buffered: this hits the in-buffer fast path in seek(), which + /// never touches impl, so it must not forget the identity baseline. + subject.seek(1, SEEK_SET); + readAndAssert(subject, "23"); + /// Reading past the buffer now reissues on the SAME impl (a retry after a stream break, not an + /// explicit reposition); the baseline from "A" must have survived the harmless seek above, so the + /// mismatched ETag "B" here must still be flagged. + readAndAssert(subject, "456789"); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, ExternalBufferFlagsMixedIncarnations) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + auto subject = DB::ReadBufferFromS3( + client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings, /* use_external_buffer */ true); + + const std::string body = "123456789"; + auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); + auto rest_buf = std::make_shared(body.substr(3)); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + if (call == 1) + { + EXPECT_EQ(rangeStart(request), 0); + return makeGetObjectOutcome(breaking_buf.get(), "A"); + } + EXPECT_EQ(rangeStart(request), 3); + return makeGetObjectOutcome(rest_buf.get(), "B"); + }; + + std::vector external_memory(3); + + /// Drive the external-buffer path the way a prefetching/threadpool reader does: supply the memory + /// with set() and pull one chunk with next(), rather than relying on the buffer's own allocation. + subject.set(external_memory.data(), external_memory.size()); + ASSERT_TRUE(subject.next()); + ASSERT_EQ(std::string(subject.buffer().begin(), subject.buffer().end()), "123"); + ASSERT_FALSE(subject.responseIdentityChanged()); + + /// This next() call breaks the "A" stream and reissues; the reissue answers with ETag "B" and + /// delivers bytes via the external buffer. Bytes were consumed on this path too, so it must flag. + subject.set(external_memory.data(), external_memory.size()); + ASSERT_TRUE(subject.next()); + ASSERT_EQ(std::string(subject.buffer().begin(), subject.buffer().end()), "456"); + ASSERT_TRUE(subject.responseIdentityChanged()); +} + +TEST_F(CASReadBufferFromS3Test, PartialInternalFillNeverExposedDoesNotCountAsDelivery) +{ + const auto client = std::make_shared(); + DB::ReadSettings read_settings; + read_settings.remote_fs_settings.buffer_size = 5; + auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); + + const std::string body = "123456789"; + /// internal_buffer is 5 bytes but only 2 bytes are ever produced before the stream throws, so + /// ReadBufferFromIStream's fill loop calls readFromDevice a second time (asking for more) and gets + /// the exception before it ever assigns `working_buffer` - those 2 bytes are read off the wire but + /// never exposed to the consumer. + auto partial_then_fails = std::make_shared(body, /* bytes_before_failure */ 2); + auto full_buf = std::make_shared(body); + + client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome + { + ++call; + EXPECT_EQ(rangeStart(request), 0); + if (call == 1) + return makeGetObjectOutcome(partial_then_fails.get(), "A"); + return makeGetObjectOutcome(full_buf.get(), "B"); + }; + + readAndAssert(subject, body.c_str()); + ASSERT_FALSE(subject.responseIdentityChanged()); +} + +#endif diff --git a/src/IO/tests/gtest_readbuffer_s3.cpp b/src/IO/tests/gtest_readbuffer_s3.cpp index d2d2947e55bc..4bf502a366bf 100644 --- a/src/IO/tests/gtest_readbuffer_s3.cpp +++ b/src/IO/tests/gtest_readbuffer_s3.cpp @@ -21,15 +21,9 @@ #include #include #include -#include #include #include -namespace DB::ErrorCodes -{ - extern const int S3_ERROR; -} - static constexpr auto TEST_LOG_LEVEL = "debug"; static fs::path caches_dir = fs::current_path() / "readbuffer_s3"; static std::string cache_base_path = caches_dir / "cache1" / ""; @@ -117,57 +111,6 @@ class StringHTTPBasicStreamBuf : public Poco::Net::HTTPBasicStreamBuf } }; -/// A response body stream that throws once `bytes_before_failure` bytes have been handed out (0 means -/// the very first read fails), simulating a GET whose headers arrived successfully but whose body read -/// broke before delivering that many bytes to the consumer. -class BreakingHTTPBasicStreamBuf : public Poco::Net::HTTPBasicStreamBuf -{ -public: - BreakingHTTPBasicStreamBuf(std::string body, size_t bytes_before_failure_) - : BasicBufferedStreamBuf(body.size(), IOS::in), bodyStream(std::stringstream(std::move(body))), bytes_before_failure(bytes_before_failure_) - { - } - -private: - std::stringstream bodyStream; - size_t bytes_before_failure; - - int readFromDevice(char_type * buf, std::streamsize n) override - { - if (bytes_before_failure == 0) - throw DB::Exception(DB::ErrorCodes::S3_ERROR, "Simulated S3 body read failure"); - - bodyStream.read(buf, std::min(n, static_cast(bytes_before_failure))); - const auto got = bodyStream.gcount(); - bytes_before_failure -= static_cast(got); - return static_cast(got); - } -}; - -/// The byte offset the request's Range header asks for, or 0 when no Range was set. sendRequest() -/// always emits "bytes=-" or "bytes=-", so parsing out lets a mock GetObject -/// serve the bytes a reissued request should actually receive. -static size_t rangeStart(const Aws::S3::Model::GetObjectRequest & request) -{ - if (!request.RangeHasBeenSet()) - return 0; - const std::string & range = request.GetRange(); - const size_t begin_pos = range.find('=') + 1; - const size_t dash_pos = range.find('-', begin_pos); - return std::stoull(range.substr(begin_pos, dash_pos - begin_pos)); -} - -static Aws::S3::Model::GetObjectOutcome makeGetObjectOutcome(std::streambuf * sb, const std::string & etag) -{ - Aws::Http::HeaderValueCollection headers; - headers["etag"] = etag; - auto response_stream = Aws::Utils::Stream::ResponseStream( - Aws::New>("test response stream", std::make_shared(), sb)); - Aws::AmazonWebServiceResult aws_result(std::move(response_stream), std::move(headers)); - DB::S3::Model::GetObjectResult result(std::move(aws_result)); - return Aws::S3::Model::GetObjectOutcome(std::move(result)); -} - using GetObjectFn = std::function; struct ClientFake : DB::S3::Client @@ -312,361 +255,6 @@ TEST_F(ReadBufferFromS3Test, ReleaseSessionWhenReadUntilPosition) ASSERT_FALSE(subject.nextImpl()); } -TEST_F(ReadBufferFromS3Test, IdentityNotFlaggedWhenFailedAttemptDeliveredNoBytes) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 20; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto failing_buf = std::make_shared(body, /* bytes_before_failure */ 0); - auto full_buf = std::make_shared(body); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - EXPECT_EQ(rangeStart(request), 0); - if (call == 1) - return makeGetObjectOutcome(failing_buf.get(), "A"); - return makeGetObjectOutcome(full_buf.get(), "B"); - }; - - /// First attempt's headers carried ETag "A", but its body read fails before any byte reaches the - /// consumer; the reissue delivers the whole object under ETag "B". No bytes of "A" were ever - /// consumed, so this must not be flagged as a coherence problem. - readAndAssert(subject, body.c_str()); - ASSERT_FALSE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, IdentityFlaggedWhenBytesDeliveredBeforeFailure) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); - auto rest_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(breaking_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - return makeGetObjectOutcome(rest_buf.get(), "B"); - }; - - /// The first response (ETag "A") delivers 3 bytes before its stream breaks; the reissue, resuming - /// from offset 3, answers with ETag "B". Bytes from two different incarnations reached the - /// consumer, so this must be flagged. - readAndAssert(subject, body.c_str()); - ASSERT_TRUE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, IdentityNotFlaggedWhenReissuedEtagMatches) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); - auto rest_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(breaking_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - return makeGetObjectOutcome(rest_buf.get(), "A"); - }; - - /// Same as above, but the reissue answers with the same ETag "A": both attempts belong to the same - /// incarnation, so this must not be flagged. - readAndAssert(subject, body.c_str()); - ASSERT_FALSE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, ThreeResponsesABytesThenAEmptyFailThenBBytesIsFlagged) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto delivers_then_fails = std::make_shared(body, /* bytes_before_failure */ 3); - auto fails_empty = std::make_shared(body, /* bytes_before_failure */ 0); - auto rest_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(delivers_then_fails.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - if (call == 2) - return makeGetObjectOutcome(fails_empty.get(), "A"); - return makeGetObjectOutcome(rest_buf.get(), "B"); - }; - - /// A delivers 3 bytes, then breaks. The reissue (same ETag "A") fails before delivering anything. - /// The next reissue answers with ETag "B" and delivers the rest: A-bytes and B-bytes were mixed, so - /// this must be flagged, even though an empty failed attempt for "A" sat in between. - readAndAssert(subject, body.c_str()); - ASSERT_TRUE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, ThreeResponsesABytesThenBEmptyFailThenABytesIsNotFlagged) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto delivers_then_fails = std::make_shared(body, /* bytes_before_failure */ 3); - auto fails_empty = std::make_shared(body, /* bytes_before_failure */ 0); - auto rest_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(delivers_then_fails.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - if (call == 2) - return makeGetObjectOutcome(fails_empty.get(), "B"); - return makeGetObjectOutcome(rest_buf.get(), "A"); - }; - - /// A delivers 3 bytes, then breaks. The reissue under ETag "B" fails before delivering anything, so - /// it never contributes to the read. The next reissue answers with ETag "A" (matching the only - /// response that ever delivered bytes) and delivers the rest: the read is coherent and must not be - /// flagged, even though a differently-ETagged empty failed attempt sat in between. - readAndAssert(subject, body.c_str()); - ASSERT_FALSE(subject.responseIdentityChanged()); - ASSERT_EQ(subject.getObjectMetadataFromTheLastRequest().etag, "A"); -} - -TEST_F(ReadBufferFromS3Test, SeekReissueAcceptsNewEtagWithoutFlag) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - read_settings.remote_fs_settings.min_bytes_for_seek = 0; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto first_buf = std::make_shared(body); - auto after_seek_buf = std::make_shared(body.substr(8)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(first_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 8); - return makeGetObjectOutcome(after_seek_buf.get(), "B"); - }; - - readAndAssert(subject, "123"); - /// A seek far enough ahead to force a reissue (not an in-buffer rewind, not a small forward skip): - /// the caller explicitly repositioned to a different range, so the new response's ETag "B" must not - /// be compared against "A". - subject.seek(8, SEEK_SET); - readAndAssert(subject, "9"); - ASSERT_FALSE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, SetReadUntilPositionReissueAcceptsNewEtagWithoutFlag) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 2; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456"; - auto first_buf = std::make_shared(body); - auto after_reposition_buf = std::make_shared(body.substr(2, 3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(first_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 2); - return makeGetObjectOutcome(after_reposition_buf.get(), "B"); - }; - - readAndAssert(subject, "12"); - /// impl is still open (no read-until-position was set yet, so nothing released it). Narrowing the - /// read-until bound now tears impl down to reissue for the new bound: an explicit reposition, so - /// the new response's ETag "B" must not be compared against "A". - subject.setReadUntilPosition(5); - readAndAssert(subject, "345"); - ASSERT_FALSE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, SetReadUntilEndReissueAcceptsNewEtagWithoutFlag) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - subject.setReadUntilPosition(3); - - const std::string body = "123456789"; - auto first_buf = std::make_shared(body.substr(0, 3)); - auto after_reposition_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(first_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - return makeGetObjectOutcome(after_reposition_buf.get(), "B"); - }; - - readAndAssert(subject, "123"); - /// Reading exactly up to the bound releases the result (does not reset impl). Removing the bound - /// now tears impl down to reissue for the rest of the object: an explicit reposition, so the new - /// response's ETag "B" must not be compared against "A". - subject.setReadUntilEnd(); - readAndAssert(subject, "456789"); - ASSERT_FALSE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, InBufferSeekPreservesBaselineAndLaterMixedRetryIsFlagged) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 3; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); - auto rest_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(breaking_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - return makeGetObjectOutcome(rest_buf.get(), "B"); - }; - - readAndAssert(subject, "123"); - /// Rewind within the bytes already buffered: this hits the in-buffer fast path in seek(), which - /// never touches impl, so it must not forget the identity baseline. - subject.seek(1, SEEK_SET); - readAndAssert(subject, "23"); - /// Reading past the buffer now reissues on the SAME impl (a retry after a stream break, not an - /// explicit reposition); the baseline from "A" must have survived the harmless seek above, so the - /// mismatched ETag "B" here must still be flagged. - readAndAssert(subject, "456789"); - ASSERT_TRUE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, ExternalBufferFlagsMixedIncarnations) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - auto subject = DB::ReadBufferFromS3( - client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings, /* use_external_buffer */ true); - - const std::string body = "123456789"; - auto breaking_buf = std::make_shared(body, /* bytes_before_failure */ 3); - auto rest_buf = std::make_shared(body.substr(3)); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - if (call == 1) - { - EXPECT_EQ(rangeStart(request), 0); - return makeGetObjectOutcome(breaking_buf.get(), "A"); - } - EXPECT_EQ(rangeStart(request), 3); - return makeGetObjectOutcome(rest_buf.get(), "B"); - }; - - std::vector external_memory(3); - - /// Drive the external-buffer path the way a prefetching/threadpool reader does: supply the memory - /// with set() and pull one chunk with next(), rather than relying on the buffer's own allocation. - subject.set(external_memory.data(), external_memory.size()); - ASSERT_TRUE(subject.next()); - ASSERT_EQ(std::string(subject.buffer().begin(), subject.buffer().end()), "123"); - ASSERT_FALSE(subject.responseIdentityChanged()); - - /// This next() call breaks the "A" stream and reissues; the reissue answers with ETag "B" and - /// delivers bytes via the external buffer. Bytes were consumed on this path too, so it must flag. - subject.set(external_memory.data(), external_memory.size()); - ASSERT_TRUE(subject.next()); - ASSERT_EQ(std::string(subject.buffer().begin(), subject.buffer().end()), "456"); - ASSERT_TRUE(subject.responseIdentityChanged()); -} - -TEST_F(ReadBufferFromS3Test, PartialInternalFillNeverExposedDoesNotCountAsDelivery) -{ - const auto client = std::make_shared(); - DB::ReadSettings read_settings; - read_settings.remote_fs_settings.buffer_size = 5; - auto subject = DB::ReadBufferFromS3(client, "test_bucket", "test_key", "test_version_id", DB::S3::S3RequestSettings(), read_settings); - - const std::string body = "123456789"; - /// internal_buffer is 5 bytes but only 2 bytes are ever produced before the stream throws, so - /// ReadBufferFromIStream's fill loop calls readFromDevice a second time (asking for more) and gets - /// the exception before it ever assigns `working_buffer` - those 2 bytes are read off the wire but - /// never exposed to the consumer. - auto partial_then_fails = std::make_shared(body, /* bytes_before_failure */ 2); - auto full_buf = std::make_shared(body); - - client->getObjectImpl = [&, call = 0](const Aws::S3::Model::GetObjectRequest & request) mutable -> Aws::S3::Model::GetObjectOutcome - { - ++call; - EXPECT_EQ(rangeStart(request), 0); - if (call == 1) - return makeGetObjectOutcome(partial_then_fails.get(), "A"); - return makeGetObjectOutcome(full_buf.get(), "B"); - }; - - readAndAssert(subject, body.c_str()); - ASSERT_FALSE(subject.responseIdentityChanged()); -} - TEST_F(ReadBufferFromS3Test, IterateUsesStartAfter) { std::unique_ptr client = std::make_unique(); From e4bf7a874a000edec8261d1e75c59c68ae911df3 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 11:18:33 +0200 Subject: [PATCH 64/81] s3: move fork-added write tests out of gtest_writebuffer_s3.cpp gtest_writebuffer_s3.cpp is an upstream test file; every fork-added test in it made rebasing conflict. Unlike the read-side files, the fork's changes here are threaded through MockS3::Client's PutObject/HeadObject/DeleteObject bodies (an attempts_seen recorder) and add two overrides (ListObjectsV2, DeleteObjects) plus an injection struct, all inside the same `namespace MockS3` block the upstream tests use. The upstream file goes back to byte-identical with altinity/antalya-26.6. The fork's tests move into the new fork-owned gtest_cas_writebuffer_s3.cpp, which carries its own private copy of the whole `namespace MockS3` block (with the fork's instrumentation folded in), the writeAsOneBlock/ writeAsPieces helpers and the fixtures, renamed CASWBS3Test/CASSyncAsync with their own INSTANTIATE_TEST_SUITE_P. Everything copied lives in an anonymous namespace: both files link into unit_tests_dbms, and two different `MockS3::Client` definitions with external linkage would be a One Definition Rule violation, while gtest rejects two fixture types under one suite name. Six copied members the fork's tests never call are marked `[[maybe_unused]]`, since internal-linkage members trip -Wunused-member-function under -Weverything. A duplicated mock in a fork-owned file is cheaper than a shared header that turns every upstream change to the mock into a rebase conflict: the upstream file's diff against base goes from 329 insertions to 0. Test names and bodies are unchanged; the fork-only ScopedWriteBufferS3ErrorLogCapture helper stays local to the new file. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/IO/tests/gtest_cas_writebuffer_s3.cpp | 1165 +++++++++++++++++++++ src/IO/tests/gtest_writebuffer_s3.cpp | 329 ------ 2 files changed, 1165 insertions(+), 329 deletions(-) create mode 100644 src/IO/tests/gtest_cas_writebuffer_s3.cpp diff --git a/src/IO/tests/gtest_cas_writebuffer_s3.cpp b/src/IO/tests/gtest_cas_writebuffer_s3.cpp new file mode 100644 index 000000000000..37ac44df9b1e --- /dev/null +++ b/src/IO/tests/gtest_cas_writebuffer_s3.cpp @@ -0,0 +1,1165 @@ +#include + +#include "config.h" + +#if USE_AWS_S3 + +#include + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include + +#include +#include +#include + + +namespace DB +{ +namespace Setting +{ + extern const SettingsBool s3_check_objects_after_upload; + extern const SettingsUInt64 s3_max_inflight_parts_for_one_file; + extern const SettingsUInt64 s3_max_single_part_upload_size; + extern const SettingsUInt64 s3_max_upload_part_size; + extern const SettingsUInt64 s3_min_upload_part_size; + extern const SettingsUInt64 s3_strict_upload_part_size; + extern const SettingsUInt64 s3_upload_part_size_multiply_factor; + extern const SettingsUInt64 s3_upload_part_size_multiply_parts_count_threshold; +} + +namespace S3RequestSetting +{ + extern const S3RequestSettingsBool allow_native_copy; +} + +namespace ErrorCodes +{ + extern const int LOGICAL_ERROR; + extern const int S3_ERROR; + extern const int NOT_IMPLEMENTED; +} + +} + +using namespace DB; + +namespace +{ + +/// A private copy of `gtest_writebuffer_s3.cpp`'s mock S3 client, extended with the attempt-seed +/// recorder and `PreconditionFailed` injection this file's tests need -- duplicated so an upstream +/// change to the mock never produces a conflict here on rebase. Anonymous namespace: both `.cpp` +/// files link into `unit_tests_dbms`, and without internal linkage the duplicate types below would +/// violate the One Definition Rule. A few members this file's own tests never call are kept +/// for parity with the upstream mock and marked `[[maybe_unused]]`, since the anonymous namespace +/// (unlike the external linkage of a shared header) makes an unused member function an error. +namespace MockS3 +{ + +class Sequencer +{ +public: + size_t next() { return counter++; } + std::string next_id() + { + std::stringstream ss; + ss << "id-" << next(); + return ss.str(); + } + +private: + size_t counter = 0; +}; + +class BucketMemStore +{ +public: + using Key = std::string; + using Data = std::string; + using ETag = std::string; + using MPU_ID = std::string; + using MPUPartsInProgress = std::map; + using MPUParts = std::vector; + + + std::map objects; + std::map multiPartUploads; + std::vector> CompletedPartUploads; + + Sequencer sequencer; + + std::string CreateMPU() + { + auto id = sequencer.next_id(); + multiPartUploads.emplace(id, MPUPartsInProgress{}); + return id; + } + + std::string UploadPart(const std::string & upload_id, const std::string & part) + { + auto etag = sequencer.next_id(); + auto & parts = multiPartUploads.at(upload_id); + parts.emplace(etag, part); + return etag; + } + + void PutObject(const std::string & key, const std::string & data) + { + objects[key] = data; + } + + void CompleteMPU(const std::string & key, const std::string & upload_id, const std::vector & etags) + { + MPUParts completedParts; + completedParts.reserve(etags.size()); + + auto & parts = multiPartUploads.at(upload_id); + for (const auto & tag: etags) { + completedParts.push_back(parts.at(tag)); + } + + std::stringstream file_data; + for (const auto & part_data: completedParts) { + file_data << part_data; + } + + CompletedPartUploads.emplace_back(upload_id, std::move(completedParts)); + objects[key] = file_data.str(); + multiPartUploads.erase(upload_id); + } + + void AbortMPU(const std::string & upload_id) + { + multiPartUploads.erase(upload_id); + } + + + const std::vector> & GetCompletedPartUploads() const + { + return CompletedPartUploads; + } + + [[maybe_unused]] static std::vector GetPartSizes(const MPUParts & parts) + { + std::vector result; + result.reserve(parts.size()); + for (const auto & part_data : parts) + result.push_back(part_data.size()); + + return result; + } + +}; + +class S3MemStrore +{ +public: + void CreateBucket(const std::string & bucket) + { + chassert(!buckets.contains(bucket)); + buckets.emplace(bucket, BucketMemStore{}); + } + + BucketMemStore& GetBucketStore(const std::string & bucket) { + return buckets.at(bucket); + } + +private: + std::map buckets; +}; + +struct EventCounts +{ + size_t headObject = 0; + size_t getObject = 0; + size_t putObject = 0; + size_t multiUploadCreate = 0; + size_t multiUploadComplete = 0; + size_t multiUploadAbort = 0; + size_t uploadParts = 0; + size_t writtenSize = 0; + size_t copyObject = 0; + size_t deleteObject = 0; + size_t getBucketVersioning = 0; + + [[maybe_unused]] size_t totalRequestsCount() const + { + return headObject + getObject + putObject + multiUploadCreate + multiUploadComplete + uploadParts; + } +}; + +struct Client; + +struct InjectionModel +{ + virtual ~InjectionModel() = default; + +#define DeclareInjectCall(ObjectTypePart) \ + virtual std::optional call(const Aws::S3::Model::ObjectTypePart##Request & /*request*/) \ + { \ + return std::nullopt; \ + } + DeclareInjectCall(PutObject) + DeclareInjectCall(HeadObject) + DeclareInjectCall(CreateMultipartUpload) + DeclareInjectCall(CompleteMultipartUpload) + DeclareInjectCall(AbortMultipartUpload) + DeclareInjectCall(UploadPart) + DeclareInjectCall(CopyObject) + DeclareInjectCall(DeleteObject) + DeclareInjectCall(GetBucketVersioning) +#undef DeclareInjectCall +}; + +/// `DB::S3::getClickhouseAttemptNumber(const Aws::AmazonWebServiceRequest &)` reads `GetHeaders()`, +/// which for a plain S3 request never includes `SetAdditionalCustomHeaderValue`'s custom headers -- +/// only `AWSClient::BuildHttpRequest` merges those into the wire-level `Aws::Http::HttpRequest` that +/// `PocoHTTPClient` actually inspects (the overload production code reads). This mock overrides the +/// `S3Client` virtuals directly, below that merge, so it reads the custom header collection itself. +/// `nullopt` means the `clickhouse-request` header is absent -- distinct from an explicit `attempt=1`, +/// since a seed of 0 leaves every verb but the read path unseeded (no header at all; see +/// `S3::seededAttemptNumber`'s callers). +std::optional attemptNumberFromCustomHeaders(const Aws::AmazonWebServiceRequest & request) +{ + const auto & headers = request.GetAdditionalCustomHeaders(); + auto it = headers.find("clickhouse-request"); + if (it == headers.end()) + return std::nullopt; + static const std::string key = "attempt="; + auto pos = it->second.find(key); + if (pos == std::string::npos) + return std::nullopt; + try + { + return static_cast(std::stol(it->second.substr(pos + key.size()))); + } + catch (const std::exception &) + { + return std::nullopt; + } +} + +struct Client : DB::S3::Client +{ + explicit Client(std::shared_ptr mock_s3_store) + : DB::S3::Client( + 100, + DB::S3::ServerSideEncryptionKMSConfig(), + std::make_shared("", ""), + GetClientConfiguration(), + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + DB::S3::ClientSettings{ + .use_virtual_addressing = true, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }) + , store(mock_s3_store) + {} + + static std::shared_ptr CreateClient(String bucket = "mock-s3-bucket") + { + auto s3store = std::make_shared(); + s3store->CreateBucket(bucket); + return std::make_shared(s3store); + } + + static DB::S3::PocoHTTPClientConfiguration GetClientConfiguration() + { + DB::RemoteHostFilter remote_host_filter; + return DB::S3::ClientFactory::instance().createClientConfiguration( + "some-region", + remote_host_filter, + /* s3_max_redirects = */ 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + /* s3_slow_all_threads_after_network_error = */ true, + /* s3_slow_all_threads_after_retryable_error = */ true, + /* enable_s3_requests_logging = */ true, + /* for_disk_s3 = */ false, + /* opt_disk_name = */ {}, + /* request_throttler = */ {}); + } + + void setInjectionModel(std::shared_ptr injections_) + { + injections = injections_; + } + + /// `clickhouse-request` attempt of every verb, in order -- test-only recorder for the attempt-seed tests. + mutable std::vector> attempts_seen; + + Aws::S3::Model::ListObjectsV2Outcome ListObjectsV2(const Aws::S3::Model::ListObjectsV2Request & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + auto & bStore = store->GetBucketStore(request.GetBucket()); + Aws::S3::Model::ListObjectsV2Result result; + result.SetPrefix(request.GetPrefix()); + int emitted = 0; + std::string last; + const std::string after = request.ContinuationTokenHasBeenSet() ? request.GetContinuationToken() + : request.StartAfterHasBeenSet() ? request.GetStartAfter() : ""; + for (const auto & [key, data] : bStore.objects) + { + if (!key.starts_with(request.GetPrefix()) || key <= after) + continue; + if (emitted == request.GetMaxKeys()) + { + result.SetIsTruncated(true); + result.SetNextContinuationToken(last); + break; + } + Aws::S3::Model::Object object; + object.SetKey(key); + object.SetSize(static_cast(data.size())); + result.AddContents(std::move(object)); + last = key; + ++emitted; + } + return Aws::S3::Model::ListObjectsV2Outcome(std::move(result)); + } + + Aws::S3::Model::DeleteObjectsOutcome DeleteObjects(const Aws::S3::Model::DeleteObjectsRequest & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + + auto & bStore = store->GetBucketStore(request.GetBucket()); + for (const auto & identifier : request.GetDelete().GetObjects()) + bStore.objects.erase(identifier.GetKey()); + + Aws::S3::Model::DeleteObjectsResult result; + return Aws::S3::Model::DeleteObjectsOutcome(std::move(result)); + } + + Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + ++counters.putObject; + + if (const auto * wrapper = dynamic_cast(&request)) + last_put_object_native_conditional = wrapper->isNativeConditional(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + { + return *opt_val; + } + } + + auto & bStore = store->GetBucketStore(request.GetBucket()); + std::stringstream data; + data << request.GetBody()->rdbuf(); + bStore.PutObject(request.GetKey(), data.str()); + counters.writtenSize += data.str().length(); + + Aws::S3::Model::PutObjectOutcome outcome; + Aws::S3::Model::PutObjectResult result(outcome.GetResultWithOwnership()); + result.SetETag("etag-singlepart-" + request.GetKey()); + return result; + } + + Aws::S3::Model::GetObjectOutcome GetObject(const Aws::S3::Model::GetObjectRequest & request) const override + { + ++counters.getObject; + + auto & bStore = store->GetBucketStore(request.GetBucket()); + const String data = bStore.objects[request.GetKey()]; + + size_t begin = 0; + size_t end = data.size() - 1; + + const String & range = request.GetRange(); + const String prefix = "bytes="; + if (range.starts_with(prefix)) + { + int ret = sscanf(range.c_str(), "bytes=%zu-%zu", &begin, &end); /// NOLINT + chassert(ret == 2); + } + + auto factory = request.GetResponseStreamFactory(); + Aws::Utils::Stream::ResponseStream responseStream(factory); + responseStream.GetUnderlyingStream() << std::stringstream(data.substr(begin, end - begin + 1)).rdbuf(); + + Aws::AmazonWebServiceResult awsStream(std::move(responseStream), Aws::Http::HeaderValueCollection()); + Aws::S3::Model::GetObjectResult getObjectResult(std::move(awsStream)); + return Aws::S3::Model::GetObjectOutcome(std::move(getObjectResult)); + } + + Aws::S3::Model::HeadObjectOutcome HeadObject(const Aws::S3::Model::HeadObjectRequest & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + ++counters.headObject; + + /// The request's DYNAMIC type is still the production `DB::S3::HeadObjectRequest` wrapper -- + /// this override only sees it through the SDK base-class reference. Mirrors the dynamic_cast + /// `Client::BuildHttpRequest` itself does, so a test can observe the mark this mock never + /// forwards through an HTTP layer. + if (const auto * wrapper = dynamic_cast(&request)) + last_head_object_native_conditional = wrapper->isNativeConditional(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + { + return std::move(*opt_val); + } + } + + auto & bStore = store->GetBucketStore(request.GetBucket()); + auto obj = bStore.objects[request.GetKey()]; + Aws::S3::Model::HeadObjectOutcome outcome; + Aws::S3::Model::HeadObjectResult result(outcome.GetResultWithOwnership()); + result.SetContentLength(obj.length()); + return result; + } + + Aws::S3::Model::CreateMultipartUploadOutcome CreateMultipartUpload(const Aws::S3::Model::CreateMultipartUploadRequest & request) const override + { + ++counters.multiUploadCreate; + + if (const auto * wrapper = dynamic_cast(&request)) + last_create_multipart_native_conditional = wrapper->isNativeConditional(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + { + return std::move(*opt_val); + } + } + + auto & bStore = store->GetBucketStore(request.GetBucket()); + auto mpu_id = bStore.CreateMPU(); + + Aws::S3::Model::CreateMultipartUploadResult result; + result.SetUploadId(mpu_id.c_str()); + return Aws::S3::Model::CreateMultipartUploadOutcome(result); + } + + Aws::S3::Model::UploadPartOutcome UploadPart(const Aws::S3::Model::UploadPartRequest & request) const override + { + ++counters.uploadParts; + + if (const auto * wrapper = dynamic_cast(&request)) + last_upload_part_native_conditional = wrapper->isNativeConditional(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + { + return std::move(*opt_val); + } + } + + std::stringstream data; + data << request.GetBody()->rdbuf(); + counters.writtenSize += data.str().length(); + + auto & bStore = store->GetBucketStore(request.GetBucket()); + auto etag = bStore.UploadPart(request.GetUploadId(), data.str()); + + Aws::S3::Model::UploadPartResult result; + result.SetETag(etag); + return Aws::S3::Model::UploadPartOutcome(result); + } + + Aws::S3::Model::CompleteMultipartUploadOutcome CompleteMultipartUpload(const Aws::S3::Model::CompleteMultipartUploadRequest & request) const override + { + ++counters.multiUploadComplete; + + if (const auto * wrapper = dynamic_cast(&request)) + last_complete_multipart_native_conditional = wrapper->isNativeConditional(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + { + return std::move(*opt_val); + } + } + + auto & bStore = store->GetBucketStore(request.GetBucket()); + + std::vector etags; + for (const auto & x: request.GetMultipartUpload().GetParts()) { + etags.push_back(x.GetETag()); + } + bStore.CompleteMPU(request.GetKey(), request.GetUploadId(), etags); + + Aws::S3::Model::CompleteMultipartUploadResult result; + result.SetETag("etag-multipart-" + request.GetKey()); + return Aws::S3::Model::CompleteMultipartUploadOutcome(result); + } + + Aws::S3::Model::AbortMultipartUploadOutcome AbortMultipartUpload(const Aws::S3::Model::AbortMultipartUploadRequest & request) const override + { + ++counters.multiUploadAbort; + + if (injections) + { + if (auto opt_val = injections->call(request)) + { + return std::move(*opt_val); + } + } + + auto & bStore = store->GetBucketStore(request.GetBucket()); + bStore.AbortMPU(request.GetUploadId()); + + Aws::S3::Model::AbortMultipartUploadResult result; + return Aws::S3::Model::AbortMultipartUploadOutcome(result); + } + + Aws::S3::Model::CopyObjectOutcome CopyObject(const Aws::S3::Model::CopyObjectRequest & request) const override + { + ++counters.copyObject; + + if (const auto * wrapper = dynamic_cast(&request)) + last_copy_object_native_conditional = wrapper->isNativeConditional(); + + last_copy_object_if_match = request.IfMatchHasBeenSet(); + last_copy_object_if_none_match = request.IfNoneMatchHasBeenSet(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + return std::move(*opt_val); + } + + /// CopySource is "/"; parse it back apart to look the source object up + /// (both source and destination live in the same S3MemStrore in these tests). + const std::string & copy_source = request.GetCopySource(); + const size_t sep = copy_source.find('/'); + chassert(sep != std::string::npos); + const std::string src_bucket_name = copy_source.substr(0, sep); + const std::string src_key = copy_source.substr(sep + 1); + + auto & src_store = store->GetBucketStore(src_bucket_name); + const std::string data = src_store.objects.at(src_key); + + auto & dst_store = store->GetBucketStore(request.GetBucket()); + dst_store.PutObject(request.GetKey(), data); + + Aws::S3::Model::CopyObjectResult result; + Aws::S3::Model::CopyObjectResultDetails details; + details.SetETag("etag-copy-" + request.GetKey()); + result.SetCopyObjectResultDetails(details); + return Aws::S3::Model::CopyObjectOutcome(result); + } + + Aws::S3::Model::DeleteObjectOutcome DeleteObject(const Aws::S3::Model::DeleteObjectRequest & request) const override + { + attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); + ++counters.deleteObject; + + if (const auto * wrapper = dynamic_cast(&request)) + last_delete_object_native_conditional = wrapper->isNativeConditional(); + + if (injections) + { + if (auto opt_val = injections->call(request)) + return std::move(*opt_val); + } + + auto & bStore = store->GetBucketStore(request.GetBucket()); + bStore.objects.erase(request.GetKey()); + + Aws::S3::Model::DeleteObjectResult result; + return Aws::S3::Model::DeleteObjectOutcome(result); + } + + Aws::S3::Model::GetBucketVersioningOutcome GetBucketVersioning(const Aws::S3::Model::GetBucketVersioningRequest & request) const override + { + ++counters.getBucketVersioning; + + if (injections) + { + if (auto opt_val = injections->call(request)) + return std::move(*opt_val); + } + + Aws::S3::Model::GetBucketVersioningResult result; + result.SetStatus(Aws::S3::Model::BucketVersioningStatus::Enabled); + return Aws::S3::Model::GetBucketVersioningOutcome(result); + } + + std::shared_ptr store; + mutable EventCounts counters; + mutable std::shared_ptr injections; + mutable bool last_head_object_native_conditional = false; + mutable bool last_delete_object_native_conditional = false; + mutable bool last_put_object_native_conditional = false; + mutable bool last_create_multipart_native_conditional = false; + mutable bool last_upload_part_native_conditional = false; + mutable bool last_complete_multipart_native_conditional = false; + mutable bool last_copy_object_native_conditional = false; + mutable bool last_copy_object_if_match = false; + mutable bool last_copy_object_if_none_match = false; + void resetCounters() const { counters = {}; } +}; + +struct PutObjectFailIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::PutObjectRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::VALIDATION, "FailInjection", "PutObjectFailIngection", false); + } +}; + +/// A conditional-write 412, matched by `S3::isPreconditionFailedError` on the canonical `` name. +struct PutObjectPreconditionFailedIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::PutObjectRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::UNKNOWN, "PreconditionFailed", "precondition failed", false); + } +}; + +struct HeadObjectFailIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::HeadObjectRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::VALIDATION, "FailInjection", "HeadObjectFailIngection", false); + } +}; + +struct CreateMPUFailIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::CreateMultipartUploadRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::VALIDATION, "FailInjection", "CreateMPUFailIngection", false); + } +}; + +struct CompleteMPUFailIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::CompleteMultipartUploadRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::VALIDATION, "FailInjection", "CompleteMPUFailIngection", false); + } +}; + +struct UploadPartFailIngection: InjectionModel +{ + std::optional call(const Aws::S3::Model::UploadPartRequest & /*request*/) override + { + return Aws::Client::AWSError(Aws::Client::CoreErrors::VALIDATION, "FailInjection", "UploadPartFailIngection", false); + } +}; + +/// Injects an arbitrary AWSError on DeleteObject -- used to drive the conditional-remove +/// (`removeObjectIfTokenMatches`) outcome mapping: a 412-shaped error (exception name "PreconditionFailed", +/// matched by `S3::isPreconditionFailedError`) must map to `ConditionalRemoveOutcome::TokenMismatch`, and a +/// 404-shaped error (a `NO_SUCH_KEY`/`RESOURCE_NOT_FOUND`/`NO_SUCH_BUCKET` error type, matched by +/// `S3::isNotFoundError`) must map to `ConditionalRemoveOutcome::NotFound`. +struct DeleteObjectErrorInjection: InjectionModel +{ + [[maybe_unused]] explicit DeleteObjectErrorInjection(Aws::Client::AWSError error_) : error(std::move(error_)) {} + + std::optional call(const Aws::S3::Model::DeleteObjectRequest & /*request*/) override + { + return error; + } + + Aws::Client::AWSError error; +}; + +/// Injects an arbitrary `CopyObject` error to exercise ordinary-copy fallback and native-only +/// fail-close behavior. +struct CopyObjectErrorInjection: InjectionModel +{ + [[maybe_unused]] explicit CopyObjectErrorInjection(Aws::Client::AWSError error_) : error(std::move(error_)) {} + + std::optional call(const Aws::S3::Model::CopyObjectRequest & /*request*/) override + { + return error; + } + + Aws::Client::AWSError error; +}; + +struct BaseSyncPolicy +{ + virtual ~BaseSyncPolicy() = default; + virtual DB::ThreadPoolCallbackRunnerUnsafe getScheduler() { return {}; } + virtual void execute(size_t) {} + virtual void setAutoExecute(bool) {} + + virtual size_t size() const { return 0; } + virtual bool empty() const { return size() == 0; } +}; + +struct SimpleAsyncTasks : BaseSyncPolicy +{ + bool auto_execute = false; + std::deque> queue; + + DB::ThreadPoolCallbackRunnerUnsafe getScheduler() override + { + return [this] (std::function && operation, size_t /*priority*/) + { + if (auto_execute) + { + auto task = std::packaged_task(std::move(operation)); + task(); + return task.get_future(); + } + + queue.emplace_back(std::move(operation)); + return queue.back().get_future(); + }; + } + + void execute(size_t limit) override + { + if (limit == 0) + limit = queue.size(); + + while (!queue.empty() && limit) + { + auto & request = queue.front(); + request(); + + queue.pop_front(); + --limit; + } + } + + void setAutoExecute(bool value) override + { + auto_execute = value; + if (auto_execute) + execute(0); + } + + size_t size() const override { return queue.size(); } +}; + +} + +static void writeAsOneBlock(WriteBuffer& buf, size_t size) +{ + std::vector data(size, 'a'); + buf.write(data.data(), data.size()); +} + +static void writeAsPieces(WriteBuffer& buf, size_t size) +{ + size_t ceil = 15ull*1024*1024*1024; + size_t piece = 1; + size_t written = 0; + while (written < size) { + size_t len = std::min({piece, size-written, ceil}); + writeAsOneBlock(buf, len); + written += len; + piece *= 2; + } +} + +class CASWBS3Test : public ::testing::Test +{ +public: + const String bucket = "CASWBS3Test-bucket"; + + Settings & getSettings() + { + return settings; + } + + MockS3::BaseSyncPolicy & getAsyncPolicy() + { + return *async_policy; + } + + std::unique_ptr getWriteBuffer(String file_name = "file", const WriteSettings & write_settings = {}) + { + S3::S3RequestSettings request_settings; + request_settings.updateFromSettings(settings, /* if_changed */true, /* validate_settings */false); + + client->resetCounters(); + + getAsyncPolicy().setAutoExecute(false); + + return std::make_unique( + client, + bucket, + file_name, + DBMS_DEFAULT_BUFFER_SIZE, + request_settings, + nullptr, + std::nullopt, + getAsyncPolicy().getScheduler(), + write_settings); + } + + void setInjectionModel(std::shared_ptr injections_) + { + client->setInjectionModel(injections_); + } + + [[maybe_unused]] void runSimpleScenario(MockS3::EventCounts expected_counters, size_t size) + { + auto scenario = [&] (std::function writeMethod) { + auto buffer = getWriteBuffer("file"); + writeMethod(*buffer, size); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + + expected_counters.writtenSize = size; + assertCountersEQ(expected_counters); + + auto & bStore = client->store->GetBucketStore(bucket); + auto & data = bStore.objects["file"]; + ASSERT_EQ(size, data.size()); + for (char c : data) + ASSERT_EQ('a', c); + }; + + scenario(writeAsOneBlock); + scenario(writeAsPieces); + } + + void assertCountersEQ(const MockS3::EventCounts & canonical) { + const auto & actual = client->counters; + ASSERT_EQ(canonical.headObject, actual.headObject); + ASSERT_EQ(canonical.getObject, actual.getObject); + ASSERT_EQ(canonical.putObject, actual.putObject); + ASSERT_EQ(canonical.multiUploadCreate, actual.multiUploadCreate); + ASSERT_EQ(canonical.multiUploadComplete, actual.multiUploadComplete); + ASSERT_EQ(canonical.multiUploadAbort, actual.multiUploadAbort); + ASSERT_EQ(canonical.uploadParts, actual.uploadParts); + ASSERT_EQ(canonical.writtenSize, actual.writtenSize); + } + + [[maybe_unused]] auto getCompletedPartUploads () + { + return client->store->GetBucketStore(bucket).GetCompletedPartUploads(); + } + +protected: + Settings settings; + + std::shared_ptr client; + std::unique_ptr async_policy; + + void SetUp() override + { + client = MockS3::Client::CreateClient(bucket); + async_policy = std::make_unique(); + } + + void TearDown() override + { + client.reset(); + async_policy.reset(); + } +}; + +class CASSyncAsync : public CASWBS3Test, public ::testing::WithParamInterface +{ +protected: + bool test_with_pool = false; + + void SetUp() override + { + test_with_pool = GetParam(); + client = MockS3::Client::CreateClient(bucket); + if (test_with_pool) + { + /// Do not block the main thread awaiting the others task. + /// This test use the only one thread at all + getSettings()[Setting::s3_max_inflight_parts_for_one_file] = 0; + async_policy = std::make_unique(); + } + else + { + async_policy = std::make_unique(); + } + } +}; + +/// Captures what `WriteBufferFromS3` logs at `threshold` and above (default: Error). A message +/// logged below the threshold never reaches the channel, so an empty capture proves the site logged +/// below it rather than merely that this particular text was absent. +class ScopedWriteBufferS3ErrorLogCapture +{ +public: + explicit ScopedWriteBufferS3ErrorLogCapture(const std::string & threshold = "error") + : logger(getLogger("WriteBufferFromS3")) + , channel(new Poco::StreamChannel(stream)) + , old_channel(logger->getChannel(), /*shared=*/true) + , old_level(logger->getLevel()) + { + logger->setChannel(channel.get()); + logger->setLevel(threshold); + } + + ~ScopedWriteBufferS3ErrorLogCapture() + { + logger->setChannel(old_channel); + logger->setLevel(old_level); + } + + std::string captured() const { return stream.str(); } + +private: + LoggerPtr logger; + std::ostringstream stream; + Poco::AutoPtr channel; + /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. + Poco::AutoPtr old_channel; + int old_level; +}; + +} + +INSTANTIATE_TEST_SUITE_P(CASWBS3 + , CASSyncAsync + , ::testing::Values(true, false) + , [] (const ::testing::TestParamInfo& info_param) { + std::string name = info_param.param ? "async" : "sync"; + return name; + }); + +/// A non-412 `PutObject` failure on the ordinary (Default) retry profile is a genuine error: the +/// client's one attempt IS the final answer, so the site logs it at Error. +TEST_P(CASSyncAsync, PutObjectErrorLogsErrorForDefaultProfile) +{ + setInjectionModel(std::make_shared()); + + ScopedWriteBufferS3ErrorLogCapture log_capture; + EXPECT_THROW({ + auto buffer = getWriteBuffer("put_object_error_default_profile"); + buffer->write('A'); + buffer->next(); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + }, DB::S3Exception); + + EXPECT_THAT(log_capture.captured(), testing::HasSubstr("S3Exception name FailInjection")); + EXPECT_THAT(log_capture.captured(), testing::HasSubstr("PutObjectFailIngection")); +} + +/// The same failure on the SingleAttempt profile (the CAS conditional-write client) is owned by an +/// outer retry loop that resolves the outcome and reissues; the one failed attempt is not terminal, +/// so nothing here reaches Error. +TEST_P(CASSyncAsync, PutObjectErrorLogsDebugForSingleAttemptProfile) +{ + setInjectionModel(std::make_shared()); + + WriteSettings write_settings; + write_settings.object_storage_retry_profile = ObjectStorageRetryProfile::SingleAttempt; + + ScopedWriteBufferS3ErrorLogCapture log_capture; + EXPECT_THROW({ + auto buffer = getWriteBuffer("put_object_error_single_attempt_profile", write_settings); + buffer->write('A'); + buffer->next(); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + }, DB::S3Exception); + + EXPECT_TRUE(log_capture.captured().empty()); +} + +/// A conditional write losing its precondition (412) is the caller's expected answer, handled one +/// frame up -- it says nothing to the operator, so it must stay below Information, independent of the +/// retry profile. The capture threshold is Information so that an Info-level line from the site would +/// be caught; the cancel path logs its own Info lines, so the assertion is on the site's text, not on +/// an empty capture. +TEST_P(CASSyncAsync, PreconditionFailedNeverLogsAtError) +{ + setInjectionModel(std::make_shared()); + + ScopedWriteBufferS3ErrorLogCapture log_capture("information"); + EXPECT_THROW({ + auto buffer = getWriteBuffer("put_object_precondition_failed"); + buffer->write('A'); + buffer->next(); + + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + }, DB::S3Exception); + + EXPECT_THAT(log_capture.captured(), testing::Not(testing::HasSubstr("S3Exception name"))); +} + +TEST_F(CASWBS3Test, S3RequestAttemptSeedPutHeadDeleteCarryTheSeed) +{ + WriteSettings write_settings; + write_settings.object_storage_attempt_number = 3; + client->attempts_seen.clear(); + { + auto buffer = getWriteBuffer("seeded_put", write_settings); + buffer->write('A'); + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + } + ASSERT_FALSE(client->attempts_seen.empty()); + EXPECT_EQ(client->attempts_seen.front(), 3u); + /// Seed 0 adds no header at all (the spec's rule for every verb but the read path). + client->attempts_seen.clear(); + { + auto buffer = getWriteBuffer("unseeded_put"); + buffer->write('A'); + getAsyncPolicy().setAutoExecute(true); + buffer->finalize(); + } + ASSERT_EQ(client->attempts_seen.size(), 1u); + EXPECT_FALSE(client->attempts_seen.front().has_value()); + + /// The native HEAD's seed: `S3ObjectStorage::tryGetObjectMetadataWithNativeToken`'s profile-aware + /// overload now forwards `request.attempt_number`, like every other verb here; this exercises the + /// seed-carrying layer directly -- `S3::getObjectInfoIfExists`, the same call + /// `tryGetObjectMetadataImpl` makes. + client->attempts_seen.clear(); + S3::getObjectInfoIfExists(*client, bucket, "seeded_head", /*version_id=*/{}, /*with_metadata=*/false, + /*with_tags=*/false, ObjectStorageRequestMode::Default, /*attempt_seed=*/4); + ASSERT_EQ(client->attempts_seen.size(), 1u); + EXPECT_EQ(client->attempts_seen.front(), 4u); + client->attempts_seen.clear(); + S3::getObjectInfoIfExists(*client, bucket, "unseeded_head"); + ASSERT_EQ(client->attempts_seen.size(), 1u); + EXPECT_FALSE(client->attempts_seen.front().has_value()); + + /// Conditional (single) and bulk DELETE: reachable now through `S3ObjectStorage`'s + /// `ObjectStorageControlRequest`-carrying overloads, which is what actually drives + /// `removeObjectIfTokenMatchesImpl`/`removeObjectsIfExistImpl` with a real nonzero seed, through the + /// object storage's own API rather than a lower-level free function. + (void)getContext(); // BlobStorageLogWriter::create falls back to the global context + auto delete_store = std::make_shared(); + delete_store->CreateBucket(bucket); + auto owned_delete_client = std::make_unique(delete_store); + MockS3::Client * delete_client = owned_delete_client.get(); + S3::URI delete_uri; + delete_uri.bucket = bucket; + auto delete_object_storage = std::make_shared( + std::move(owned_delete_client), + std::make_unique(), + delete_uri, + S3Capabilities{}, + ObjectStorageKeyGeneratorPtr{}, + "seed-delete-disk"); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectIfTokenMatches(StoredObject("unseeded-delete-key"), "etag-1"); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_FALSE(delete_client->attempts_seen.front().has_value()); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectIfTokenMatches( + StoredObject("seeded-delete-key"), "etag-1", ObjectStorageControlRequest{.attempt_number = 3}); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_EQ(delete_client->attempts_seen.front(), 3u); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectsIfExistUnderProfile({StoredObject("unseeded-bulk-key")}, ObjectStorageControlRequest{}); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_FALSE(delete_client->attempts_seen.front().has_value()); + + delete_client->attempts_seen.clear(); + delete_object_storage->removeObjectsIfExistUnderProfile( + {StoredObject("seeded-bulk-key")}, ObjectStorageControlRequest{.attempt_number = 3}); + ASSERT_EQ(delete_client->attempts_seen.size(), 1u); + EXPECT_EQ(delete_client->attempts_seen.front(), 3u); +} + +TEST_F(CASWBS3Test, S3RequestAttemptSeedListPagesCarryTheSeed) +{ + /// Drives the seed through the public `iterate` overload a real caller (the CAS backend's LIST + /// primitive) uses, rather than the anonymous-namespace `S3IteratorAsync` directly -- that class is + /// an implementation detail of `S3ObjectStorage.cpp` and not reachable from a test in this file. + auto list_store = std::make_shared(); + list_store->CreateBucket(bucket); + auto owned_list_client = std::make_unique(list_store); + MockS3::Client * list_client = owned_list_client.get(); + S3::URI list_uri; + list_uri.bucket = bucket; + auto list_object_storage = std::make_shared( + std::move(owned_list_client), + std::make_unique(), + list_uri, + S3Capabilities{}, + ObjectStorageKeyGeneratorPtr{}, + "seed-list-disk"); + + auto & bucket_store = list_store->GetBucketStore(bucket); + for (int i = 0; i < 5; ++i) + bucket_store.PutObject(fmt::format("p/{}", i), "x"); + + /// Profile is left at Default (not SingleAttempt): that would route through + /// `clientForRetryProfile`'s single-attempt clone, whose `cloneWithConfigurationOverride` the mock + /// client does not override, and the test would stop exercising the mock entirely. + list_client->attempts_seen.clear(); + auto iterator = list_object_storage->iterate( + "p/", /*max_keys=*/2, /*with_tags=*/false, std::optional("p/0"), + ObjectStorageControlRequest{.attempt_number = 2}); + size_t seen = 0; + for (; iterator->isValid(); iterator->next()) + ++seen; + EXPECT_EQ(seen, 4u); + ASSERT_EQ(list_client->attempts_seen.size(), 2u); /// the initial page and one rebuilt page + EXPECT_EQ(list_client->attempts_seen[0], 2u); + EXPECT_EQ(list_client->attempts_seen[1], 2u); + + /// Seed 0 adds no header on either page. + list_client->attempts_seen.clear(); + auto unseeded_iterator = list_object_storage->iterate( + "p/", /*max_keys=*/2, /*with_tags=*/false, std::optional("p/0"), ObjectStorageControlRequest{}); + seen = 0; + for (; unseeded_iterator->isValid(); unseeded_iterator->next()) + ++seen; + EXPECT_EQ(seen, 4u); + ASSERT_EQ(list_client->attempts_seen.size(), 2u); + EXPECT_FALSE(list_client->attempts_seen[0].has_value()); + EXPECT_FALSE(list_client->attempts_seen[1].has_value()); +} + +#endif diff --git a/src/IO/tests/gtest_writebuffer_s3.cpp b/src/IO/tests/gtest_writebuffer_s3.cpp index ce3fc108b840..138f96ed7285 100644 --- a/src/IO/tests/gtest_writebuffer_s3.cpp +++ b/src/IO/tests/gtest_writebuffer_s3.cpp @@ -20,8 +20,6 @@ #include #include #include -#include -#include #include #include #include @@ -33,8 +31,6 @@ #include #include #include -#include -#include #include #include @@ -42,18 +38,12 @@ #include #include #include -#include #include #include -#include #include #include -#include - -#include -#include #include @@ -243,34 +233,6 @@ struct InjectionModel #undef DeclareInjectCall }; -/// `DB::S3::getClickhouseAttemptNumber(const Aws::AmazonWebServiceRequest &)` reads `GetHeaders()`, -/// which for a plain S3 request never includes `SetAdditionalCustomHeaderValue`'s custom headers -- -/// only `AWSClient::BuildHttpRequest` merges those into the wire-level `Aws::Http::HttpRequest` that -/// `PocoHTTPClient` actually inspects (the overload production code reads). This mock overrides the -/// `S3Client` virtuals directly, below that merge, so it reads the custom header collection itself. -/// `nullopt` means the `clickhouse-request` header is absent -- distinct from an explicit `attempt=1`, -/// since a seed of 0 leaves every verb but the read path unseeded (no header at all; see -/// `S3::seededAttemptNumber`'s callers). -static std::optional attemptNumberFromCustomHeaders(const Aws::AmazonWebServiceRequest & request) -{ - const auto & headers = request.GetAdditionalCustomHeaders(); - auto it = headers.find("clickhouse-request"); - if (it == headers.end()) - return std::nullopt; - static const std::string key = "attempt="; - auto pos = it->second.find(key); - if (pos == std::string::npos) - return std::nullopt; - try - { - return static_cast(std::stol(it->second.substr(pos + key.size()))); - } - catch (const std::exception &) - { - return std::nullopt; - } -} - struct Client : DB::S3::Client { explicit Client(std::shared_ptr mock_s3_store) @@ -317,54 +279,8 @@ struct Client : DB::S3::Client injections = injections_; } - /// `clickhouse-request` attempt of every verb, in order -- test-only recorder for the attempt-seed tests. - mutable std::vector> attempts_seen; - - Aws::S3::Model::ListObjectsV2Outcome ListObjectsV2(const Aws::S3::Model::ListObjectsV2Request & request) const override - { - attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); - auto & bStore = store->GetBucketStore(request.GetBucket()); - Aws::S3::Model::ListObjectsV2Result result; - result.SetPrefix(request.GetPrefix()); - int emitted = 0; - std::string last; - const std::string after = request.ContinuationTokenHasBeenSet() ? request.GetContinuationToken() - : request.StartAfterHasBeenSet() ? request.GetStartAfter() : ""; - for (const auto & [key, data] : bStore.objects) - { - if (!key.starts_with(request.GetPrefix()) || key <= after) - continue; - if (emitted == request.GetMaxKeys()) - { - result.SetIsTruncated(true); - result.SetNextContinuationToken(last); - break; - } - Aws::S3::Model::Object object; - object.SetKey(key); - object.SetSize(static_cast(data.size())); - result.AddContents(std::move(object)); - last = key; - ++emitted; - } - return Aws::S3::Model::ListObjectsV2Outcome(std::move(result)); - } - - Aws::S3::Model::DeleteObjectsOutcome DeleteObjects(const Aws::S3::Model::DeleteObjectsRequest & request) const override - { - attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); - - auto & bStore = store->GetBucketStore(request.GetBucket()); - for (const auto & identifier : request.GetDelete().GetObjects()) - bStore.objects.erase(identifier.GetKey()); - - Aws::S3::Model::DeleteObjectsResult result; - return Aws::S3::Model::DeleteObjectsOutcome(std::move(result)); - } - Aws::S3::Model::PutObjectOutcome PutObject(const Aws::S3::Model::PutObjectRequest & request) const override { - attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); ++counters.putObject; if (const auto * wrapper = dynamic_cast(&request)) @@ -419,7 +335,6 @@ struct Client : DB::S3::Client Aws::S3::Model::HeadObjectOutcome HeadObject(const Aws::S3::Model::HeadObjectRequest & request) const override { - attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); ++counters.headObject; /// The request's DYNAMIC type is still the production `DB::S3::HeadObjectRequest` wrapper -- @@ -581,7 +496,6 @@ struct Client : DB::S3::Client Aws::S3::Model::DeleteObjectOutcome DeleteObject(const Aws::S3::Model::DeleteObjectRequest & request) const override { - attempts_seen.push_back(attemptNumberFromCustomHeaders(request)); ++counters.deleteObject; if (const auto * wrapper = dynamic_cast(&request)) @@ -638,15 +552,6 @@ struct PutObjectFailIngection: InjectionModel } }; -/// A conditional-write 412, matched by `S3::isPreconditionFailedError` on the canonical `` name. -struct PutObjectPreconditionFailedIngection: InjectionModel -{ - std::optional call(const Aws::S3::Model::PutObjectRequest & /*request*/) override - { - return Aws::Client::AWSError(Aws::Client::CoreErrors::UNKNOWN, "PreconditionFailed", "precondition failed", false); - } -}; - struct HeadObjectFailIngection: InjectionModel { std::optional call(const Aws::S3::Model::HeadObjectRequest & /*request*/) override @@ -771,44 +676,6 @@ struct SimpleAsyncTasks : BaseSyncPolicy using namespace DB; -namespace -{ - -/// Captures what `WriteBufferFromS3` logs at `threshold` and above (default: Error). A message -/// logged below the threshold never reaches the channel, so an empty capture proves the site logged -/// below it rather than merely that this particular text was absent. -class ScopedWriteBufferS3ErrorLogCapture -{ -public: - explicit ScopedWriteBufferS3ErrorLogCapture(const std::string & threshold = "error") - : logger(getLogger("WriteBufferFromS3")) - , channel(new Poco::StreamChannel(stream)) - , old_channel(logger->getChannel(), /*shared=*/true) - , old_level(logger->getLevel()) - { - logger->setChannel(channel.get()); - logger->setLevel(threshold); - } - - ~ScopedWriteBufferS3ErrorLogCapture() - { - logger->setChannel(old_channel); - logger->setLevel(old_level); - } - - std::string captured() const { return stream.str(); } - -private: - LoggerPtr logger; - std::ostringstream stream; - Poco::AutoPtr channel; - /// `shared=true` is load-bearing: `AutoPtr(ptr)` would steal a reference the fixture never owned. - Poco::AutoPtr old_channel; - int old_level; -}; - -} - static void writeAsOneBlock(WriteBuffer& buf, size_t size) { std::vector data(size, 'a'); @@ -1039,71 +906,6 @@ TEST_P(SyncAsync, ExceptionOnPut) { } -/// A non-412 `PutObject` failure on the ordinary (Default) retry profile is a genuine error: the -/// client's one attempt IS the final answer, so the site logs it at Error. -TEST_P(SyncAsync, PutObjectErrorLogsErrorForDefaultProfile) -{ - setInjectionModel(std::make_shared()); - - ScopedWriteBufferS3ErrorLogCapture log_capture; - EXPECT_THROW({ - auto buffer = getWriteBuffer("put_object_error_default_profile"); - buffer->write('A'); - buffer->next(); - - getAsyncPolicy().setAutoExecute(true); - buffer->finalize(); - }, DB::S3Exception); - - EXPECT_THAT(log_capture.captured(), testing::HasSubstr("S3Exception name FailInjection")); - EXPECT_THAT(log_capture.captured(), testing::HasSubstr("PutObjectFailIngection")); -} - -/// The same failure on the SingleAttempt profile (the CAS conditional-write client) is owned by an -/// outer retry loop that resolves the outcome and reissues; the one failed attempt is not terminal, -/// so nothing here reaches Error. -TEST_P(SyncAsync, PutObjectErrorLogsDebugForSingleAttemptProfile) -{ - setInjectionModel(std::make_shared()); - - WriteSettings write_settings; - write_settings.object_storage_retry_profile = ObjectStorageRetryProfile::SingleAttempt; - - ScopedWriteBufferS3ErrorLogCapture log_capture; - EXPECT_THROW({ - auto buffer = getWriteBuffer("put_object_error_single_attempt_profile", write_settings); - buffer->write('A'); - buffer->next(); - - getAsyncPolicy().setAutoExecute(true); - buffer->finalize(); - }, DB::S3Exception); - - EXPECT_TRUE(log_capture.captured().empty()); -} - -/// A conditional write losing its precondition (412) is the caller's expected answer, handled one -/// frame up -- it says nothing to the operator, so it must stay below Information, independent of the -/// retry profile. The capture threshold is Information so that an Info-level line from the site would -/// be caught; the cancel path logs its own Info lines, so the assertion is on the site's text, not on -/// an empty capture. -TEST_P(SyncAsync, PreconditionFailedNeverLogsAtError) -{ - setInjectionModel(std::make_shared()); - - ScopedWriteBufferS3ErrorLogCapture log_capture("information"); - EXPECT_THROW({ - auto buffer = getWriteBuffer("put_object_precondition_failed"); - buffer->write('A'); - buffer->next(); - - getAsyncPolicy().setAutoExecute(true); - buffer->finalize(); - }, DB::S3Exception); - - EXPECT_THAT(log_capture.captured(), testing::Not(testing::HasSubstr("S3Exception name"))); -} - TEST_P(SyncAsync, ExceptionOnCreateMPU) { setInjectionModel(std::make_shared()); @@ -1333,137 +1135,6 @@ TEST_F(WBS3Test, ResultObjectETagIsCaptured) { } } -TEST_F(WBS3Test, S3RequestAttemptSeedPutHeadDeleteCarryTheSeed) -{ - WriteSettings write_settings; - write_settings.object_storage_attempt_number = 3; - client->attempts_seen.clear(); - { - auto buffer = getWriteBuffer("seeded_put", write_settings); - buffer->write('A'); - getAsyncPolicy().setAutoExecute(true); - buffer->finalize(); - } - ASSERT_FALSE(client->attempts_seen.empty()); - EXPECT_EQ(client->attempts_seen.front(), 3u); - /// Seed 0 adds no header at all (the spec's rule for every verb but the read path). - client->attempts_seen.clear(); - { - auto buffer = getWriteBuffer("unseeded_put"); - buffer->write('A'); - getAsyncPolicy().setAutoExecute(true); - buffer->finalize(); - } - ASSERT_EQ(client->attempts_seen.size(), 1u); - EXPECT_FALSE(client->attempts_seen.front().has_value()); - - /// The native HEAD's seed: `S3ObjectStorage::tryGetObjectMetadataWithNativeToken`'s profile-aware - /// overload now forwards `request.attempt_number`, like every other verb here; this exercises the - /// seed-carrying layer directly -- `S3::getObjectInfoIfExists`, the same call - /// `tryGetObjectMetadataImpl` makes. - client->attempts_seen.clear(); - S3::getObjectInfoIfExists(*client, bucket, "seeded_head", /*version_id=*/{}, /*with_metadata=*/false, - /*with_tags=*/false, ObjectStorageRequestMode::Default, /*attempt_seed=*/4); - ASSERT_EQ(client->attempts_seen.size(), 1u); - EXPECT_EQ(client->attempts_seen.front(), 4u); - client->attempts_seen.clear(); - S3::getObjectInfoIfExists(*client, bucket, "unseeded_head"); - ASSERT_EQ(client->attempts_seen.size(), 1u); - EXPECT_FALSE(client->attempts_seen.front().has_value()); - - /// Conditional (single) and bulk DELETE: reachable now through `S3ObjectStorage`'s - /// `ObjectStorageControlRequest`-carrying overloads, which is what actually drives - /// `removeObjectIfTokenMatchesImpl`/`removeObjectsIfExistImpl` with a real nonzero seed, through the - /// object storage's own API rather than a lower-level free function. - (void)getContext(); // BlobStorageLogWriter::create falls back to the global context - auto delete_store = std::make_shared(); - delete_store->CreateBucket(bucket); - auto owned_delete_client = std::make_unique(delete_store); - MockS3::Client * delete_client = owned_delete_client.get(); - S3::URI delete_uri; - delete_uri.bucket = bucket; - auto delete_object_storage = std::make_shared( - std::move(owned_delete_client), - std::make_unique(), - delete_uri, - S3Capabilities{}, - ObjectStorageKeyGeneratorPtr{}, - "seed-delete-disk"); - - delete_client->attempts_seen.clear(); - delete_object_storage->removeObjectIfTokenMatches(StoredObject("unseeded-delete-key"), "etag-1"); - ASSERT_EQ(delete_client->attempts_seen.size(), 1u); - EXPECT_FALSE(delete_client->attempts_seen.front().has_value()); - - delete_client->attempts_seen.clear(); - delete_object_storage->removeObjectIfTokenMatches( - StoredObject("seeded-delete-key"), "etag-1", ObjectStorageControlRequest{.attempt_number = 3}); - ASSERT_EQ(delete_client->attempts_seen.size(), 1u); - EXPECT_EQ(delete_client->attempts_seen.front(), 3u); - - delete_client->attempts_seen.clear(); - delete_object_storage->removeObjectsIfExistUnderProfile({StoredObject("unseeded-bulk-key")}, ObjectStorageControlRequest{}); - ASSERT_EQ(delete_client->attempts_seen.size(), 1u); - EXPECT_FALSE(delete_client->attempts_seen.front().has_value()); - - delete_client->attempts_seen.clear(); - delete_object_storage->removeObjectsIfExistUnderProfile( - {StoredObject("seeded-bulk-key")}, ObjectStorageControlRequest{.attempt_number = 3}); - ASSERT_EQ(delete_client->attempts_seen.size(), 1u); - EXPECT_EQ(delete_client->attempts_seen.front(), 3u); -} - -TEST_F(WBS3Test, S3RequestAttemptSeedListPagesCarryTheSeed) -{ - /// Drives the seed through the public `iterate` overload a real caller (the CAS backend's LIST - /// primitive) uses, rather than the anonymous-namespace `S3IteratorAsync` directly -- that class is - /// an implementation detail of `S3ObjectStorage.cpp` and not reachable from a test in this file. - auto list_store = std::make_shared(); - list_store->CreateBucket(bucket); - auto owned_list_client = std::make_unique(list_store); - MockS3::Client * list_client = owned_list_client.get(); - S3::URI list_uri; - list_uri.bucket = bucket; - auto list_object_storage = std::make_shared( - std::move(owned_list_client), - std::make_unique(), - list_uri, - S3Capabilities{}, - ObjectStorageKeyGeneratorPtr{}, - "seed-list-disk"); - - auto & bucket_store = list_store->GetBucketStore(bucket); - for (int i = 0; i < 5; ++i) - bucket_store.PutObject(fmt::format("p/{}", i), "x"); - - /// Profile is left at Default (not SingleAttempt): that would route through - /// `clientForRetryProfile`'s single-attempt clone, whose `cloneWithConfigurationOverride` the mock - /// client does not override, and the test would stop exercising the mock entirely. - list_client->attempts_seen.clear(); - auto iterator = list_object_storage->iterate( - "p/", /*max_keys=*/2, /*with_tags=*/false, std::optional("p/0"), - ObjectStorageControlRequest{.attempt_number = 2}); - size_t seen = 0; - for (; iterator->isValid(); iterator->next()) - ++seen; - EXPECT_EQ(seen, 4u); - ASSERT_EQ(list_client->attempts_seen.size(), 2u); /// the initial page and one rebuilt page - EXPECT_EQ(list_client->attempts_seen[0], 2u); - EXPECT_EQ(list_client->attempts_seen[1], 2u); - - /// Seed 0 adds no header on either page. - list_client->attempts_seen.clear(); - auto unseeded_iterator = list_object_storage->iterate( - "p/", /*max_keys=*/2, /*with_tags=*/false, std::optional("p/0"), ObjectStorageControlRequest{}); - seen = 0; - for (; unseeded_iterator->isValid(); unseeded_iterator->next()) - ++seen; - EXPECT_EQ(seen, 4u); - ASSERT_EQ(list_client->attempts_seen.size(), 2u); - EXPECT_FALSE(list_client->attempts_seen[0].has_value()); - EXPECT_FALSE(list_client->attempts_seen[1].has_value()); -} - TEST_P(SyncAsync, EmptyFile) { getSettings()[Setting::s3_check_objects_after_upload] = true; From 173a4391e4a21959a8de6135ed22c41d43eb743a Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 08:27:50 +0200 Subject: [PATCH 65/81] s3: the delivery-time identity bookkeeping comment says the one thing that matters The comment above the response-identity move in ReadBufferFromS3::nextImpl had grown to explain a fixed historical bug (the field used to be copied, not moved) instead of the invariant a future edit actually needs to preserve. Say only that: the identity-baseline update must stay before `next_result` is set, since a throw after that point would exit the retry loop with `impl` left null while the code past the loop still dereferences it. Comment-only change. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/IO/ReadBufferFromS3.cpp | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/src/IO/ReadBufferFromS3.cpp b/src/IO/ReadBufferFromS3.cpp index e1f4bd15837f..368ceeccc358 100644 --- a/src/IO/ReadBufferFromS3.cpp +++ b/src/IO/ReadBufferFromS3.cpp @@ -217,15 +217,9 @@ bool ReadBufferFromS3::nextImpl() /// reaches this point (fails before delivering anything) never touches the baseline, /// so any number of empty failed attempts in between are transparent to the check. /// - /// Nothing here may throw: `last_delivering_response_etag = pending_response_etag` - /// used to be a copy, which can allocate and throw for a non-SSO ETag; if that throw - /// happened after `next_result` was already set to true, the catch block below resets - /// `impl` (since `processException` retries), but the loop's `!next_result` condition - /// is already false, so it exits with `impl` null while the code past the loop still - /// dereferences it. A `std::string` move is noexcept, so this block cannot throw; as a - /// second line of defense, `next_result` itself is set only once this block is done, so - /// even a future throwing addition here would leave the loop's retry invariant intact - /// instead of exiting with a dangling `impl`. + /// Must run before `next_result = delivered_more_data` below exits the loop via + /// `break`: a throw after that point would leave the loop exiting with `impl` null + /// (reset by the catch handler) while the code past the loop still dereferences it. if (last_delivering_response_etag && *last_delivering_response_etag != pending_response_etag) response_identity_changed = true; last_delivering_response_etag = std::move(pending_response_etag); From 090bf01261a390168ebe31ae66022f7da5871c68 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 08:28:02 +0200 Subject: [PATCH 66/81] io: ObjectStorageControlRequest and the attempt-seed settings describe themselves without naming CAS ObjectStorageControlRequest, ReadSettings::object_storage_attempt_number, and WriteSettings::object_storage_attempt_number/s3_max_unexpected_write_error_retries_override/ s3_check_objects_after_upload_override are generic per-request knobs any caller can set, not CAS-specific fields; their comments named CAS as if it were the only caller. Reword each to describe what the field carries and why, for any caller. Also drops a dangling internal-RFC citation ("RFC cas-s3-timeout-retry-control") that named a document outside the branch. Comment-only change. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/IO/ObjectStorageRequestProfile.h | 6 +++--- src/IO/ReadSettings.h | 3 ++- src/IO/WriteSettings.h | 17 +++++++++-------- 3 files changed, 14 insertions(+), 12 deletions(-) diff --git a/src/IO/ObjectStorageRequestProfile.h b/src/IO/ObjectStorageRequestProfile.h index 185c5c625b3a..2b9c7816fdb5 100644 --- a/src/IO/ObjectStorageRequestProfile.h +++ b/src/IO/ObjectStorageRequestProfile.h @@ -17,9 +17,9 @@ enum class ObjectStorageRetryProfile : uint8_t SingleAttempt, }; -/// What a CAS control request carries into the object storage: the retry profile, the per-attempt -/// budget and connect cap the storage's single-attempt client must honour, and the caller's own -/// attempt number (0 = unset) so the HTTP client sees a reissue as attempt ≥ 2. +/// A per-request override of retry behavior for an object storage call: which retry profile to use, +/// the per-attempt budget and connect cap the storage's single-attempt client must honour, and the +/// caller's own attempt number (0 = unset) so the HTTP client sees a reissue as attempt ≥ 2. struct ObjectStorageControlRequest { ObjectStorageRetryProfile profile = ObjectStorageRetryProfile::Default; diff --git a/src/IO/ReadSettings.h b/src/IO/ReadSettings.h index c810c7311402..069c8b117c74 100644 --- a/src/IO/ReadSettings.h +++ b/src/IO/ReadSettings.h @@ -174,7 +174,8 @@ struct ReadSettings uint64_t object_storage_connect_timeout_cap_ms = 0; /// The caller's own attempt number for the request built from these settings, 1-based; 0 leaves the - /// buffer's own numbering. A CAS reissue passes its count so the HTTP client sees attempt ≥ 2. + /// buffer's own numbering. A caller reissuing this read passes its count so the HTTP client sees + /// attempt ≥ 2. size_t object_storage_attempt_number = 0; ReadSettings adjustBufferSize(size_t file_size) const; diff --git a/src/IO/WriteSettings.h b/src/IO/WriteSettings.h index 2e1a8b5c8daf..01507d7e370c 100644 --- a/src/IO/WriteSettings.h +++ b/src/IO/WriteSettings.h @@ -36,10 +36,10 @@ struct WriteSettings bool s3_allow_parallel_part_upload = true; /// Overrides S3RequestSetting::check_objects_after_upload for this write (nullopt = no - /// override). Writers of CAS-MUTABLE keys (content-addressed shard manifests) set `false`: - /// such a key is legitimately replaced by a concurrent conditional PUT between this upload and - /// the check's HEAD, so the size comparison false-positives ("it's a bug in S3") under normal - /// contention. Integrity for those keys is the conditional PUT outcome + token, not a recheck. + /// override). A writer whose key can legitimately be replaced by a concurrent conditional PUT + /// between this upload and the check's HEAD sets `false`: otherwise the size comparison + /// false-positives ("it's a bug in S3") under normal contention. Integrity for such a key comes + /// from the conditional PUT outcome and token, not a recheck. std::optional s3_check_objects_after_upload_override; bool azure_allow_parallel_part_upload = true; @@ -62,9 +62,9 @@ struct WriteSettings /// Overrides S3RequestSetting::max_unexpected_write_error_retries (default 4) for this write. /// WriteBufferFromS3::makeSinglepartUpload/completeMultipartUpload run their OWN retry loop above /// the S3 client that reissues the identical request (WITH its If-None-Match/If-Match condition) - /// on a NO_SUCH_KEY response — a second retry-affecting layer a client-level override - /// (a client-level profile override) does not reach. A CAS conditional write sets this to 1 for - /// exactly one attempt at this layer too (RFC cas-s3-timeout-retry-control). 0 = no override. + /// on a NO_SUCH_KEY response — a second retry-affecting layer a client-level profile override does + /// not reach. A conditional write that must not retry at that layer either sets this to 1 for + /// exactly one attempt. 0 = no override. size_t s3_max_unexpected_write_error_retries_override = 0; /// Selects the retry profile the object storage should execute this write under; see @@ -80,7 +80,8 @@ struct WriteSettings uint64_t object_storage_connect_timeout_cap_ms = 0; /// The caller's own attempt number for the request built from these settings, 1-based; 0 leaves the - /// buffer's own numbering. A CAS reissue passes its count so the HTTP client sees attempt ≥ 2. + /// buffer's own numbering. A caller reissuing this write passes its count so the HTTP client sees + /// attempt ≥ 2. size_t object_storage_attempt_number = 0; /// Selects the transport requirement for an object storage copy; see `ObjectStorageCopyMode`. From 92e993b2cbf74b8fb5fa617e6f08396f831c5fd8 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 08:32:10 +0200 Subject: [PATCH 67/81] io: collapse the two identical DEBUG log branches in makeSinglepartUpload PreconditionFailed and a SingleAttempt write's failed attempt were logged with the exact same LOG_DEBUG call, each behind its own explanatory comment, duplicating the format string and every argument. Combine the two conditions into one branch with one call, keeping both reasons in a single comment. No behaviour change: same log level, same message, same arguments, for the same two conditions. diff --stat vs altinity/antalya-26.6 for this file: before 11+/2-, after 10+/4- (the comment combines two explanations into one paragraph); net line count in the file is 3 lines shorter, confirmed by `git diff --stat` of this commit (6 insertions, 9 deletions). Verified by building WriteBufferFromS3.cpp.o directly. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/IO/WriteBufferFromS3.cpp | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/src/IO/WriteBufferFromS3.cpp b/src/IO/WriteBufferFromS3.cpp index 4be4973b0cb9..fe827006312c 100644 --- a/src/IO/WriteBufferFromS3.cpp +++ b/src/IO/WriteBufferFromS3.cpp @@ -812,15 +812,12 @@ void WriteBufferFromS3::makeSinglepartUpload(WriteBufferFromS3::PartData && data } else { - /// PreconditionFailed is an expected response for conditional writes (e.g. If-None-Match: *), - /// not a genuine error — the caller handles it (see `S3::isPreconditionFailedError`), and it - /// says nothing to the operator. - if (S3::isPreconditionFailedError(outcome.GetError())) - LOG_DEBUG(log, "S3Exception name {}, Message: {}, bucket {}, key {}, object size {}", - outcome.GetError().GetExceptionName(), outcome.GetError().GetMessage(), bucket, key, content_length); - /// A SingleAttempt write is owned by an outer retry loop that resolves the outcome and - /// reissues; its one failed attempt is not terminal, so it is not an error. - else if (write_settings.object_storage_retry_profile == ObjectStorageRetryProfile::SingleAttempt) + /// Neither says anything to the operator: PreconditionFailed is an expected response for + /// conditional writes (e.g. If-None-Match: *), handled by the caller (see + /// `S3::isPreconditionFailedError`); a SingleAttempt write is owned by an outer retry loop + /// that resolves the outcome and reissues, so its one failed attempt is not terminal either. + if (S3::isPreconditionFailedError(outcome.GetError()) + || write_settings.object_storage_retry_profile == ObjectStorageRetryProfile::SingleAttempt) LOG_DEBUG(log, "S3Exception name {}, Message: {}, bucket {}, key {}, object size {}", outcome.GetError().GetExceptionName(), outcome.GetError().GetMessage(), bucket, key, content_length); else From 2c3c71f77e27ba4ac16f3a9b2b5fd9800796ddf4 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 10:45:47 +0200 Subject: [PATCH 68/81] s3: deleteFileFromS3 takes an optional attempt seed; the single-object CAS delete reuses it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The one-object branch of `removeObjectsIfExistImpl` re-implemented `deleteFileFromS3` (request, profile event, blob-storage-log event, 404 tolerance, error text) only because the helper could not stamp the `clickhouse-request` attempt header the CAS request engine requires. Give `deleteFileFromS3` an optional trailing `attempt_seed` (default 0, which keeps every existing caller unchanged and sends no header) and call it from that branch. Observable differences for that branch: the profile event `S3DeleteObjects` is now incremented alongside `DiskS3DeleteObjects`, and the success line `Object with path {} was removed from S3` is logged, exactly as the ordinary `removeObjectImpl` path already does. The batch path and the conditional (`If-Match`) delete are unchanged; the latter cannot reuse the helper because it needs the precondition, the native-conditional mode and the three-way outcome. Footprint against upstream: `S3ObjectStorage.cpp` −29 lines, `deleteFileFromS3.{h,cpp}` +4 lines. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ObjectStorages/S3/S3ObjectStorage.cpp | 43 ++++--------------- src/IO/S3/deleteFileFromS3.cpp | 5 ++- src/IO/S3/deleteFileFromS3.h | 3 +- 3 files changed, 14 insertions(+), 37 deletions(-) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 7c0debc711db..9b0000dc9ad0 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -674,44 +674,17 @@ void S3ObjectStorage::removeObjectsIfExistImpl( if (objects.empty()) return; - /// A batch of exactly one object is always a plain `DeleteObject`, never `DeleteObjects` -- mirroring - /// `deleteFilesFromS3`'s own `keys.size() == 1` rule (IO/S3/deleteFileFromS3.cpp), which skips the - /// batch request for the same single key for the same reason: there is no need for it. This is what - /// makes the CAS-side per-key fallback actually delete anything on a backend with no `DeleteObjects` - /// at all (GCS): that backend rejects the VERB outright, not by key count, so a "batch" of one - /// object sent as `DeleteObjects` would fail there identically to a bigger one. A single physical - /// request is never a loop, so this does not reintroduce what the capability check below exists to - /// rule out. + /// A batch of exactly one object is a plain `DeleteObject`, never `DeleteObjects` -- the same rule + /// `deleteFilesFromS3` applies to a single key. This is what makes the CAS-side per-key fallback + /// work on a backend with no `DeleteObjects` at all (GCS): that backend rejects the verb itself, not + /// a key count, so a "batch" of one object sent as `DeleteObjects` would fail there too. if (objects.size() == 1) { const StoredObject & object = objects.front(); - S3::DeleteObjectRequest request; - request.SetBucket(uri.bucket); - request.SetKey(object.remote_path); - if (attempt_seed != 0) - S3::setClickhouseAttemptNumber(request, attempt_seed); - - ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); - Stopwatch watch; - auto outcome = used_client->DeleteObject(request); - auto elapsed = watch.elapsedMicroseconds(); - - if (auto blob_storage_log = BlobStorageLogWriter::create(disk_name)) - blob_storage_log->addEvent(BlobStorageLogElement::EventType::Delete, - uri.bucket, object.remote_path, - object.local_path, object.bytes_size, elapsed, - outcome.IsSuccess() ? 0 : static_cast(outcome.GetError().GetErrorType()), - outcome.IsSuccess() ? "" : outcome.GetError().GetMessage()); - - if (outcome.IsSuccess()) - return; - - const auto & err = outcome.GetError(); - if (S3::isNotFoundError(err.GetErrorType())) - return; - - throw S3Exception(err.GetErrorType(), "{} (Code: {}) while removing object with path {} from S3", - err.GetMessage(), static_cast(err.GetErrorType()), object.remote_path); + deleteFileFromS3(used_client, uri.bucket, object.remote_path, /*if_exists=*/ true, + BlobStorageLogWriter::create(disk_name), object.local_path, object.bytes_size, + ProfileEvents::DiskS3DeleteObjects, attempt_seed); + return; } /// GCS has no `DeleteObjects`: a capability the config declared false, or that an earlier batch diff --git a/src/IO/S3/deleteFileFromS3.cpp b/src/IO/S3/deleteFileFromS3.cpp index 380747f33156..0111e00534a8 100644 --- a/src/IO/S3/deleteFileFromS3.cpp +++ b/src/IO/S3/deleteFileFromS3.cpp @@ -28,11 +28,14 @@ void deleteFileFromS3( BlobStorageLogWriterPtr blob_storage_log, const String & local_path_for_blob_storage_log, size_t file_size_for_blob_storage_log, - std::optional profile_event) + std::optional profile_event, + size_t attempt_seed) { S3::DeleteObjectRequest request; request.SetBucket(bucket); request.SetKey(key); + if (attempt_seed != 0) + S3::setClickhouseAttemptNumber(request, attempt_seed); ProfileEvents::increment(ProfileEvents::S3DeleteObjects); if (profile_event && *profile_event != ProfileEvents::S3DeleteObjects) diff --git a/src/IO/S3/deleteFileFromS3.h b/src/IO/S3/deleteFileFromS3.h index fad49982827e..69e0f409c768 100644 --- a/src/IO/S3/deleteFileFromS3.h +++ b/src/IO/S3/deleteFileFromS3.h @@ -32,7 +32,8 @@ void deleteFileFromS3( BlobStorageLogWriterPtr blob_storage_log = nullptr, const String & local_path_for_blob_storage_log = {}, size_t file_size_for_blob_storage_log = 0, - std::optional profile_event = std::nullopt); + std::optional profile_event = std::nullopt, + size_t attempt_seed = 0); /// Deletes multiple files from S3 using batch requests when it's possible. void deleteFilesFromS3( From b03efbf724445983b427cc98879c1b1df65f77a1 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 00:04:34 +0200 Subject: [PATCH 69/81] tests: give every inline content-addressed disk a per-run unique identity `SYSTEM CAS FORGET` (commit 761cc8b8488) decommissions a content-addressed pool for the lifetime of the server: a later `CREATE TABLE` naming the same `cas_server_root_id` / `name` / `path` fails with `INVALID_STATE` ("content-addressed pool decommissioned by SYSTEM CAS FORGET"). CI's `tests/clickhouse-test --repeat-newly-modified-tests` repeats the highest-numbered tests several times against ONE server, so every repeated run of a CAS test hit this and failed the CREATE (or, in `.sh` tests where the failed CREATE isn't fatal, failed later with `UNKNOWN_TABLE`) -- this is what took down every first-stage stateless lane in CI run 7 of PR #2300. Every stateless test that creates an inline `cas` disk now derives all three identity fields -- `cas_server_root_id`, `name` and `path` -- from the per-run database name (`$CLICKHOUSE_DATABASE` in `.sh` tests), so a repeat of the same test in the same server never reuses a decommissioned pool's identity. A few tests already derived part of the identity from `$CLICKHOUSE_TEST_UNIQUE_NAME`/`$RANDOM` (04290, 04295, 05008, 05020, 05023, 05025); those only needed their remaining static field(s) fixed. `05024_cas_freeze_two_roots.sh` and `05025_cas_attach_partition_cross_disk.sh` each mount several named disks (some intentionally sharing one pool path across two `cas_server_root_id`s) and needed each disk's three fields threaded through consistently. `05003_cas_freeze.sh`'s `FREEZE ... WITH NAME` snapshot name also embedded the same static id, so it is now suffixed with `$CLICKHOUSE_DATABASE` too, with the `.reference` normalizing the database name out of the printed `backup_name` column. Thirteen of these tests were `.sql` files: clickhouse-test does no textual substitution on `.sql` query files, and `disk(...)` settings do not accept query parameters (`disk(... name = {p:String})` fails to parse), so there is no way to inject a per-run value into a `.sql` test's `disk(...)` literal. Each is converted to an equivalent `.sh` test of the same name, feeding the same statements through `$CLICKHOUSE_CLIENT --multiquery` so the per-run `$CLICKHOUSE_DATABASE` variable can be substituted; none of them use `-- { serverError }` / `-- { clientError }` hints, comments and output stay otherwise unchanged, and every `.reference` file stays byte-identical. Verified by running all 31 touched tests twice against one local server (`tests/clickhouse-test --test-runs 2`), matching the repeat mechanism that exposed the bug: 62/62 passed. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../{04278_cas_disk.sql => 04278_cas_disk.sh} | 29 ++++++---- .../{04279_cas_gc.sql => 04279_cas_gc.sh} | 35 +++++++----- ...e_state.sql => 04282_cas_mutable_state.sh} | 37 ++++++++----- ...d.sql => 04283_cas_replicated_rejected.sh} | 43 +++++++++------ .../04284_cas_backup_pointer_holding.sh | 10 ++-- ...5_cas_deduplication_window_inline_disk.sh} | 29 ++++++---- .../04286_cas_remote_data_paths.sh | 53 ++++++++++++++++++ .../04286_cas_remote_data_paths.sql | 46 ---------------- ... => 04287_cas_detach_partition_listing.sh} | 27 +++++---- ...288_cas_detached_part_modification_time.sh | 52 ++++++++++++++++++ ...88_cas_detached_part_modification_time.sql | 45 --------------- .../04289_cas_multi_detach_drop.sh | 55 +++++++++++++++++++ .../04289_cas_multi_detach_drop.sql | 48 ---------------- .../0_stateless/04290_cas_no_leftovers.sh | 2 +- .../0_stateless/04292_cas_mutations.sh | 8 +-- .../04293_cas_lightweight_delete.sh | 8 +-- .../0_stateless/04294_cas_patch_parts.sh | 8 +-- .../04295_cas_mutation_no_leftovers.sh | 2 +- ...ql => 04299_cas_projection_inline_disk.sh} | 37 ++++++++----- ...sql => 04300_cas_projection_multiblock.sh} | 33 ++++++----- ...ition.sql => 05002_cas_fetch_partition.sh} | 43 +++++++++------ .../0_stateless/05003_cas_freeze.reference | 2 +- tests/queries/0_stateless/05003_cas_freeze.sh | 17 +++--- .../0_stateless/05004_cas_transactions.sh | 8 +-- .../0_stateless/05005_cas_backup_restore.sh | 10 ++-- .../0_stateless/05007_cas_gc_introspection.sh | 24 ++++---- .../05008_cas_gc_snapshot_prune.sh | 2 +- ...s_event_log.sql => 05009_cas_event_log.sh} | 39 +++++++------ .../0_stateless/05010_cas_mounts_gc_health.sh | 14 ++--- .../05015_cas_reject_fake_transaction.sh | 6 +- tests/queries/0_stateless/05020_cas_fsck.sh | 2 +- .../05023_cas_dropns_leaked_namespace.sh | 2 +- .../0_stateless/05024_cas_freeze_two_roots.sh | 18 +++--- .../05025_cas_attach_partition_cross_disk.sh | 27 ++++----- .../05026_cas_manifest_path_newline.sh | 2 +- 35 files changed, 459 insertions(+), 364 deletions(-) rename tests/queries/0_stateless/{04278_cas_disk.sql => 04278_cas_disk.sh} (66%) mode change 100644 => 100755 rename tests/queries/0_stateless/{04279_cas_gc.sql => 04279_cas_gc.sh} (69%) mode change 100644 => 100755 rename tests/queries/0_stateless/{04282_cas_mutable_state.sql => 04282_cas_mutable_state.sh} (64%) mode change 100644 => 100755 rename tests/queries/0_stateless/{04283_cas_replicated_rejected.sql => 04283_cas_replicated_rejected.sh} (50%) mode change 100644 => 100755 rename tests/queries/0_stateless/{04285_cas_deduplication_window_inline_disk.sql => 04285_cas_deduplication_window_inline_disk.sh} (60%) mode change 100644 => 100755 create mode 100755 tests/queries/0_stateless/04286_cas_remote_data_paths.sh delete mode 100644 tests/queries/0_stateless/04286_cas_remote_data_paths.sql rename tests/queries/0_stateless/{04287_cas_detach_partition_listing.sql => 04287_cas_detach_partition_listing.sh} (54%) mode change 100644 => 100755 create mode 100755 tests/queries/0_stateless/04288_cas_detached_part_modification_time.sh delete mode 100644 tests/queries/0_stateless/04288_cas_detached_part_modification_time.sql create mode 100755 tests/queries/0_stateless/04289_cas_multi_detach_drop.sh delete mode 100644 tests/queries/0_stateless/04289_cas_multi_detach_drop.sql rename tests/queries/0_stateless/{04299_cas_projection_inline_disk.sql => 04299_cas_projection_inline_disk.sh} (83%) mode change 100644 => 100755 rename tests/queries/0_stateless/{04300_cas_projection_multiblock.sql => 04300_cas_projection_multiblock.sh} (65%) mode change 100644 => 100755 rename tests/queries/0_stateless/{05002_cas_fetch_partition.sql => 05002_cas_fetch_partition.sh} (55%) mode change 100644 => 100755 rename tests/queries/0_stateless/{05009_cas_event_log.sql => 05009_cas_event_log.sh} (52%) mode change 100644 => 100755 diff --git a/tests/queries/0_stateless/04278_cas_disk.sql b/tests/queries/0_stateless/04278_cas_disk.sh old mode 100644 new mode 100755 similarity index 66% rename from tests/queries/0_stateless/04278_cas_disk.sql rename to tests/queries/0_stateless/04278_cas_disk.sh index df68616fcaab..99d1e9995764 --- a/tests/queries/0_stateless/04278_cas_disk.sql +++ b/tests/queries/0_stateless/04278_cas_disk.sh @@ -1,12 +1,18 @@ --- Tags: no-fasttest --- ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. +#!/usr/bin/env bash +# Tags: no-fasttest +# ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. --- Natural black-box oracle: a table on a `cas` disk must behave --- identically to a normal MergeTree table for the same data. We compare the two --- directly so the test is deterministic regardless of environment, and we also --- exercise INSERT (content-addressed write), SELECT (ref->part_id->footer->blob --- resolution), blob-level dedup of identical inserts, a merge, and DROP (removal). +# Natural black-box oracle: a table on a `cas` disk must behave +# identically to a normal MergeTree table for the same data. We compare the two +# directly so the test is deterministic regardless of environment, and we also +# exercise INSERT (content-addressed write), SELECT (ref->part_id->footer->blob +# resolution), blob-level dedup of identical inserts, a merge, and DROP (removal). +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery < 0 on a plain MergeTree keeps an on-disk deduplication log --- (deduplication_logs/deduplication_log_N.txt) at the table root. On a cas disk that log --- works the same way it does on a plain s3 disk: the disk cannot host append writes, so the log --- rewrites a fresh rotated log object per record, stored verbatim in the table's files/ namespace. This --- test uses an INLINE cas disk, so it exercises the CA path on any test config. +# A non_replicated_deduplication_window > 0 on a plain MergeTree keeps an on-disk deduplication log +# (deduplication_logs/deduplication_log_N.txt) at the table root. On a cas disk that log +# works the same way it does on a plain s3 disk: the disk cannot host append writes, so the log +# rewrites a fresh rotated log object per record, stored verbatim in the table's files/ namespace. This +# test uses an INLINE cas disk, so it exercises the CA path on any test config. +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery <= 0 +FROM system.remote_data_paths +WHERE disk_name = '${CLICKHOUSE_DATABASE}_04286_cas_rdp' +SETTINGS traverse_shadow_remote_data_paths = 1; + +DROP TABLE t_cas_rdp; +SELECT 'dropped_ok'; + +-- FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would +-- stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. +SET send_logs_level = 'fatal'; +SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_04286_cas_rdp'; +EOF diff --git a/tests/queries/0_stateless/04286_cas_remote_data_paths.sql b/tests/queries/0_stateless/04286_cas_remote_data_paths.sql deleted file mode 100644 index d15a17aad4e4..000000000000 --- a/tests/queries/0_stateless/04286_cas_remote_data_paths.sql +++ /dev/null @@ -1,46 +0,0 @@ --- Tags: no-fasttest, no-cas-storage --- no-fasttest: cas is an object-storage metadata type; keep it off the minimal --- fasttest image. --- no-cas-storage: the coverage lives in the INLINE content-addressed disk created --- below, so the test is meaningful on every ordinary lane. On lanes where the DEFAULT MergeTree --- storage is itself content-addressed, `system.remote_data_paths` (whose `disk_name` filter is --- applied only after the traversal) also walks the huge shared default pool holding the whole --- run's data, and on the S3 (RustFS) variant that walk does not fit the 600s test timeout. - --- B38: querying system.remote_data_paths traverses the disk and probes existsFile on pool sub-dirs --- (e.g. the "store" directory). Such a path resolves to a directory object key; existsFile must treat --- a directory as not-a-file and not let the raw filesystem "Is a directory" error escape. So a --- system.remote_data_paths query over a cas table must succeed (return the parts' --- remote paths) instead of throwing. - -DROP TABLE IF EXISTS t_cas_rdp; - -CREATE TABLE t_cas_rdp (a UInt64, b UInt64) -ENGINE = MergeTree ORDER BY a -SETTINGS disk = disk( - type = object_storage, - object_storage_type = local, - metadata_type = cas, - cas_server_root_id = '04286', - name = '04286_cas_rdp', - path = '04286_cas_rdp_pool/'); - -INSERT INTO t_cas_rdp SELECT number, number * 2 FROM numbers(100); - --- The traversal (with shadow paths) must be QUERYABLE: it used to throw `Is a directory` (Code 1001) --- when it probed the CA pool sub-dir (e.g. "store") via existsFile. After the B38 fix it returns a --- result without raising. We assert the query succeeds (count() is a non-negative number) rather than --- a specific row count: the CA disk's object-storage directory model determines how many rows the --- traversal yields, which is orthogonal to the not-throwing contract this test pins. -SELECT count() >= 0 -FROM system.remote_data_paths -WHERE disk_name = '04286_cas_rdp' -SETTINGS traverse_shadow_remote_data_paths = 1; - -DROP TABLE t_cas_rdp; -SELECT 'dropped_ok'; - --- FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would --- stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. -SET send_logs_level = 'fatal'; -SYSTEM CAS FORGET '04286_cas_rdp'; diff --git a/tests/queries/0_stateless/04287_cas_detach_partition_listing.sql b/tests/queries/0_stateless/04287_cas_detach_partition_listing.sh old mode 100644 new mode 100755 similarity index 54% rename from tests/queries/0_stateless/04287_cas_detach_partition_listing.sql rename to tests/queries/0_stateless/04287_cas_detach_partition_listing.sh index a9117ef0c7a5..7d89dc790fac --- a/tests/queries/0_stateless/04287_cas_detach_partition_listing.sql +++ b/tests/queries/0_stateless/04287_cas_detach_partition_listing.sh @@ -1,11 +1,17 @@ --- Tags: no-fasttest --- ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. +#!/usr/bin/env bash +# Tags: no-fasttest +# ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. --- B36: after DETACH PARTITION on a cas disk, system.detached_parts must list the --- detached part DIRECTORY name (e.g. all_1_2_1), not a sidecar / mutable file (metadata_version.txt). --- The detached namespace is a container of detached part directories; the CA disk listing of the --- "detached" path must yield the part directory names, not the files inside them. +# B36: after DETACH PARTITION on a cas disk, system.detached_parts must list the +# detached part DIRECTORY name (e.g. all_1_2_1), not a sidecar / mutable file (metadata_version.txt). +# The detached namespace is a container of detached part directories; the CA disk listing of the +# "detached" path must yield the part directory names, not the files inside them. +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery </; system.detached_parts reads the modification time by calling +# IDisk::getLastModified on the detached part DIRECTORY (/detached/). Before the +# fix, parsePartFilePath reported part_name="detached" + a non-empty file equal to the detached part +# directory name, so getLastModified fell through to the part-file manifest lookup and threw +# "ContentAddressed: file not in manifest". getLastModified now recognises the detached +# part directory and reports the "detached" ref manifest object's mtime. + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery </; system.detached_parts reads the modification time by calling --- IDisk::getLastModified on the detached part DIRECTORY (
/detached/). Before the --- fix, parsePartFilePath reported part_name="detached" + a non-empty file equal to the detached part --- directory name, so getLastModified fell through to the part-file manifest lookup and threw --- "ContentAddressed: file not in manifest". getLastModified now recognises the detached --- part directory and reports the "detached" ref manifest object's mtime. - -DROP TABLE IF EXISTS t_cas_detach_mtime; - -CREATE TABLE t_cas_detach_mtime (a UInt64, b UInt64) -ENGINE = MergeTree ORDER BY a -SETTINGS disk = disk( - type = object_storage, - object_storage_type = local, - metadata_type = cas, - cas_server_root_id = '04288', - name = '04288_cas_detach_mtime', - path = '04288_cas_detach_mtime_pool/'); - -INSERT INTO t_cas_detach_mtime SELECT number, number * 2 FROM numbers(50); - -SELECT 'count_before', count() FROM t_cas_detach_mtime; - -ALTER TABLE t_cas_detach_mtime DETACH PARTITION tuple(); - -SELECT 'count_after', count() FROM t_cas_detach_mtime; - --- The modification_time read must succeed (be non-NULL) instead of throwing FILE_DOESNT_EXIST. -SELECT 'detached_mtime_readable', name, modification_time IS NOT NULL -FROM system.detached_parts -WHERE database = currentDatabase() AND table = 't_cas_detach_mtime' -ORDER BY name; - -DROP TABLE t_cas_detach_mtime; -SELECT 'dropped_ok'; - --- FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would --- stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. -SET send_logs_level = 'fatal'; -SYSTEM CAS FORGET '04288_cas_detach_mtime'; diff --git a/tests/queries/0_stateless/04289_cas_multi_detach_drop.sh b/tests/queries/0_stateless/04289_cas_multi_detach_drop.sh new file mode 100755 index 000000000000..65d3e77038ad --- /dev/null +++ b/tests/queries/0_stateless/04289_cas_multi_detach_drop.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. + +# B46/B47: multiple partitions detached on a cas disk must COEXIST under the one +# shared "detached" ref, and DROP DETACHED PARTITION ALL must remove them. +# B46: each DETACH PARTITION clones one part into detached// via a fresh CA commit; +# the commit used to REWRITE the shared "detached" ref, so each detach overwrote the previous +# one and only the last detached part was listed. commit now MERGES into the existing detached +# ref's manifest + sidecar, so all detached parts coexist. +# B47: DROP DETACHED PARTITION first renames the detached part to "deleting_" +# (PartsTemporaryRename) then removes it; CA moveDirectory ignored a detached->detached rename +# (the rename was a no-op, so removeRecursive on the renamed dir found nothing). moveDirectory +# now re-keys the detached part dir within the shared detached ref, and removeRecursive handles a +# detached part directory by removing only that part's keys from the shared ref. + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery </ via a fresh CA commit; --- the commit used to REWRITE the shared "detached" ref, so each detach overwrote the previous --- one and only the last detached part was listed. commit now MERGES into the existing detached --- ref's manifest + sidecar, so all detached parts coexist. --- B47: DROP DETACHED PARTITION first renames the detached part to "deleting_" --- (PartsTemporaryRename) then removes it; CA moveDirectory ignored a detached->detached rename --- (the rename was a no-op, so removeRecursive on the renamed dir found nothing). moveDirectory --- now re-keys the detached part dir within the shared detached ref, and removeRecursive handles a --- detached part directory by removing only that part's keys from the shared ref. - -DROP TABLE IF EXISTS t_cas_multi_detach; - -CREATE TABLE t_cas_multi_detach (p UInt64, v UInt64) -ENGINE = MergeTree PARTITION BY p ORDER BY v -SETTINGS disk = disk( - type = object_storage, - object_storage_type = local, - metadata_type = cas, - cas_server_root_id = '04289', - name = '04289_cas_multi_detach', - path = '04289_cas_multi_detach_pool/'); - -INSERT INTO t_cas_multi_detach VALUES (1, 1), (2, 2), (3, 3); -SELECT 'active_parts', count() FROM system.parts WHERE database = currentDatabase() AND table = 't_cas_multi_detach' AND active; - -ALTER TABLE t_cas_multi_detach DETACH PARTITION ALL; - --- All three partitions must be listed as detached parts (not just the last one detached). -SELECT 'detached_after', count() FROM system.detached_parts WHERE database = currentDatabase() AND table = 't_cas_multi_detach'; -SELECT 'detached_names', name FROM system.detached_parts WHERE database = currentDatabase() AND table = 't_cas_multi_detach' ORDER BY name; - -ALTER TABLE t_cas_multi_detach DROP DETACHED PARTITION ALL SETTINGS allow_drop_detached = 1; - --- DROP DETACHED PARTITION ALL must remove every detached part. -SELECT 'detached_after_drop', count() FROM system.detached_parts WHERE database = currentDatabase() AND table = 't_cas_multi_detach'; - -DROP TABLE t_cas_multi_detach; -SELECT 'dropped_ok'; - --- FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would --- stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. -SET send_logs_level = 'fatal'; -SYSTEM CAS FORGET '04289_cas_multi_detach'; diff --git a/tests/queries/0_stateless/04290_cas_no_leftovers.sh b/tests/queries/0_stateless/04290_cas_no_leftovers.sh index 9b05005d7bb9..c1abf1be3603 100755 --- a/tests/queries/0_stateless/04290_cas_no_leftovers.sh +++ b/tests/queries/0_stateless/04290_cas_no_leftovers.sh @@ -46,7 +46,7 @@ DISK_DEF="disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '04290', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_04290', name = '${DISK_NAME}', path = '${POOL_DIR}/', cas_gc_enabled = 1, diff --git a/tests/queries/0_stateless/04292_cas_mutations.sh b/tests/queries/0_stateless/04292_cas_mutations.sh index 1c2c504356c0..cea79a47d89b 100755 --- a/tests/queries/0_stateless/04292_cas_mutations.sh +++ b/tests/queries/0_stateless/04292_cas_mutations.sh @@ -22,9 +22,9 @@ DISK_CA="disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '04292', - name = '04292_cas_mut', - path = '04292_cas_mut_pool/')" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_04292', + name = '${CLICKHOUSE_DATABASE}_04292_cas_mut', + path = '${CLICKHOUSE_DATABASE}_04292_cas_mut_pool/')" $CLICKHOUSE_CLIENT --query "DROP TABLE IF EXISTS t_ca SYNC" $CLICKHOUSE_CLIENT --query "DROP TABLE IF EXISTS t_plain SYNC" @@ -116,5 +116,5 @@ $CLICKHOUSE_CLIENT --query "DROP TABLE t_plain SYNC" # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. $CLICKHOUSE_CLIENT --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '04292_cas_mut'" || { + --query "SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_04292_cas_mut'" || { echo "FORGET failed"; exit 1; } diff --git a/tests/queries/0_stateless/04293_cas_lightweight_delete.sh b/tests/queries/0_stateless/04293_cas_lightweight_delete.sh index d57d6a57b4b6..4fc9942a37e4 100755 --- a/tests/queries/0_stateless/04293_cas_lightweight_delete.sh +++ b/tests/queries/0_stateless/04293_cas_lightweight_delete.sh @@ -22,9 +22,9 @@ DISK_CA="disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '04293', - name = '04293_cas_lwd', - path = '04293_cas_lwd_pool/')" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_04293', + name = '${CLICKHOUSE_DATABASE}_04293_cas_lwd', + path = '${CLICKHOUSE_DATABASE}_04293_cas_lwd_pool/')" $CLICKHOUSE_CLIENT --query "DROP TABLE IF EXISTS t_ca SYNC" $CLICKHOUSE_CLIENT --query "DROP TABLE IF EXISTS t_plain SYNC" @@ -103,5 +103,5 @@ $CLICKHOUSE_CLIENT --query "DROP TABLE t_plain SYNC" # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. $CLICKHOUSE_CLIENT --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '04293_cas_lwd'" || { + --query "SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_04293_cas_lwd'" || { echo "FORGET failed"; exit 1; } diff --git a/tests/queries/0_stateless/04294_cas_patch_parts.sh b/tests/queries/0_stateless/04294_cas_patch_parts.sh index 95e219d3582a..0b4123b5cbfa 100755 --- a/tests/queries/0_stateless/04294_cas_patch_parts.sh +++ b/tests/queries/0_stateless/04294_cas_patch_parts.sh @@ -22,9 +22,9 @@ DISK_CA="disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '04294', - name = '04294_cas_patch', - path = '04294_cas_patch_pool/')" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_04294', + name = '${CLICKHOUSE_DATABASE}_04294_cas_patch', + path = '${CLICKHOUSE_DATABASE}_04294_cas_patch_pool/')" $CLICKHOUSE_CLIENT --query "DROP TABLE IF EXISTS t_ca SYNC" $CLICKHOUSE_CLIENT --query "DROP TABLE IF EXISTS t_plain SYNC" @@ -90,5 +90,5 @@ $CLICKHOUSE_CLIENT --query "DROP TABLE t_plain SYNC" # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. $CLICKHOUSE_CLIENT --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '04294_cas_patch'" || { + --query "SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_04294_cas_patch'" || { echo "FORGET failed"; exit 1; } diff --git a/tests/queries/0_stateless/04295_cas_mutation_no_leftovers.sh b/tests/queries/0_stateless/04295_cas_mutation_no_leftovers.sh index ee8655a88f86..70767fd6c7b0 100755 --- a/tests/queries/0_stateless/04295_cas_mutation_no_leftovers.sh +++ b/tests/queries/0_stateless/04295_cas_mutation_no_leftovers.sh @@ -35,7 +35,7 @@ DISK_DEF="disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '04295', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_04295', name = '${DISK_NAME}', path = '${POOL_DIR}/', cas_gc_enabled = 1, diff --git a/tests/queries/0_stateless/04299_cas_projection_inline_disk.sql b/tests/queries/0_stateless/04299_cas_projection_inline_disk.sh old mode 100644 new mode 100755 similarity index 83% rename from tests/queries/0_stateless/04299_cas_projection_inline_disk.sql rename to tests/queries/0_stateless/04299_cas_projection_inline_disk.sh index 8b792e664312..ef8f68c4726b --- a/tests/queries/0_stateless/04299_cas_projection_inline_disk.sql +++ b/tests/queries/0_stateless/04299_cas_projection_inline_disk.sh @@ -1,10 +1,16 @@ --- Tags: no-fasttest --- ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. +#!/usr/bin/env bash +# Tags: no-fasttest +# ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. --- Projections on a cas disk: the projection's files are stored as nested keys --- (.proj/) in the parent part's manifest. Verify INSERT writes a projection, a --- projection-optimized SELECT returns correct results, and a merge (OPTIMIZE FINAL) rebuilds it. +# Projections on a cas disk: the projection's files are stored as nested keys +# (.proj/) in the parent part's manifest. Verify INSERT writes a projection, a +# projection-optimized SELECT returns correct results, and a merge (OPTIMIZE FINAL) rebuilds it. +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery <1 temp projection part to merge (selected_parts.size() > 1); with a single --- temp part it just renames it. The temp-part flush threshold is min_insert_block_size_rows, and the --- background merge/mutation runs in the server's background context (NOT the client query settings), so --- the threshold is the server default (DEFAULT_INSERT_BLOCK_SIZE = 1048449). We therefore make the --- projection emit MORE rows than that: a high-cardinality GROUP BY key (1.3M distinct groups) forces >=2 --- temp projection parts for BOTH an OPTIMIZE merge and an ALTER ... MATERIALIZE PROJECTION rebuild. +#!/usr/bin/env bash +# Tags: no-fasttest +# ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. +# A projection built across MULTIPLE temp projection blocks (spill-and-merge) must read its own staged +# temp blocks back on a content-addressed disk (B59). MergeProjectionPartsTask only EXERCISES the +# read-back path when it has >1 temp projection part to merge (selected_parts.size() > 1); with a single +# temp part it just renames it. The temp-part flush threshold is min_insert_block_size_rows, and the +# background merge/mutation runs in the server's background context (NOT the client query settings), so +# the threshold is the server default (DEFAULT_INSERT_BLOCK_SIZE = 1048449). We therefore make the +# projection emit MORE rows than that: a high-cardinality GROUP BY key (1.3M distinct groups) forces >=2 +# temp projection parts for BOTH an OPTIMIZE merge and an ALTER ... MATERIALIZE PROJECTION rebuild. + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery <' on a cas disk: the gate is lifted --- and a to_detached fetch takes the byte-fetch path (the downloaded files content-address into the --- detached/ namespace; relink-into-detached is deferred). The fetched part must land usably in the CA --- detached/ namespace: system.detached_parts lists it, ATTACH publishes an active part out of it, and a --- SELECT reads back the exact source data. Both tables share one inline CA pool (a single server fetches --- from its own zk path, as 03350 does), so this also exercises the cross-table detached landing. +# ALTER TABLE ... FETCH PARTITION ... FROM '' on a cas disk: the gate is lifted +# and a to_detached fetch takes the byte-fetch path (the downloaded files content-address into the +# detached/ namespace; relink-into-detached is deferred). The fetched part must land usably in the CA +# detached/ namespace: system.detached_parts lists it, ATTACH publishes an active part out of it, and a +# SELECT reads back the exact source data. Both tables share one inline CA pool (a single server fetches +# from its own zk path, as 03350 does), so this also exercises the cross-table detached landing. +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery < two parts; deterministic rows. ${CLICKHOUSE_CLIENT} --query "INSERT INTO t_cas_br VALUES (1, 'a'), (2, 'b'), (1, 'c');" @@ -51,8 +51,8 @@ ${CLICKHOUSE_CLIENT} --query "SELECT 'done';" # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. -# RESTORE re-created t_cas_br_restored on the same named pool ('05005_cas_backup_restore'), so one FORGET +# RESTORE re-created t_cas_br_restored on the same named pool ('${CLICKHOUSE_DATABASE}_05005_cas_backup_restore'), so one FORGET # covers it. ${CLICKHOUSE_CLIENT} --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '05005_cas_backup_restore'" || { + --query "SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_05005_cas_backup_restore'" || { echo "FORGET failed"; exit 1; } diff --git a/tests/queries/0_stateless/05007_cas_gc_introspection.sh b/tests/queries/0_stateless/05007_cas_gc_introspection.sh index 51f5495836d1..01b660f352d7 100755 --- a/tests/queries/0_stateless/05007_cas_gc_introspection.sh +++ b/tests/queries/0_stateless/05007_cas_gc_introspection.sh @@ -30,9 +30,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05007', - name = '05007_cas_gc_introspection', - path = '05007_cas_gc_introspection_pool/', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05007', + name = '${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection', + path = '${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection_pool/', cas_gc_enabled = 1, cas_gc_interval_sec = 1), old_parts_lifetime = 1; @@ -46,9 +46,9 @@ TRUNCATE TABLE t_cas_gc_introspection; # Run several synchronous rounds: the first rounds mark the retired candidates, later rounds delete # them once the durable watermark floor advances past the builds (the background renewer does this). -${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '05007_cas_gc_introspection'" > /dev/null -${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '05007_cas_gc_introspection'" > /dev/null -${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '05007_cas_gc_introspection'" > /dev/null +${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection'" > /dev/null +${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection'" > /dev/null +${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection'" > /dev/null ${CLICKHOUSE_CLIENT} --multiline -q """ SYSTEM FLUSH LOGS cas_gc_log; @@ -59,14 +59,14 @@ SELECT countIf(event_type = 'Finish') > 0, countIf(trigger = 'Manual') > 0 FROM system.cas_gc_log -WHERE disk_name LIKE '%05007_cas_gc_introspection%'; +WHERE disk_name LIKE '%${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection%'; -- A synchronous Manual Finish captured a non-empty per-round ProfileEvents delta (the round touches -- the object storage, so Cas*/Disk*/S3* counters are non-zero). The query thread is always attached, -- so capture is active for the Manual path. SELECT any(length(ProfileEvents)) > 0 FROM system.cas_gc_log -WHERE disk_name LIKE '%05007_cas_gc_introspection%' +WHERE disk_name LIKE '%${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection%' AND event_type = 'Finish' AND trigger = 'Manual'; @@ -77,7 +77,7 @@ SELECT countDistinct(phase) >= 10, countIf(phase = 'fold_ref_group') > 0, countIf(phase = 'round_commit') > 0 FROM system.cas_gc_log -WHERE disk_name LIKE '%05007_cas_gc_introspection%' +WHERE disk_name LIKE '%${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection%' AND event_type = 'Phase'; -- The correlator: every row of the most recent round of this disk -- its Start, each Phase, and its @@ -88,7 +88,7 @@ SELECT countIf(event_type = 'Start') = 1, FROM system.cas_gc_log WHERE round_id = ( SELECT round_id FROM system.cas_gc_log - WHERE disk_name LIKE '%05007_cas_gc_introspection%' AND event_type = 'Finish' + WHERE disk_name LIKE '%${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection%' AND event_type = 'Finish' ORDER BY event_time_microseconds DESC LIMIT 1); -- A Phase row's \`ProfileEvents\` is that phase's own delta, not the whole round's: the phase that @@ -98,7 +98,7 @@ WHERE round_id = ( -- own delta is empty and this assertion would answer 0 there no matter how healthy the round was. SELECT max(length(ProfileEvents)) > 0 FROM system.cas_gc_log -WHERE disk_name LIKE '%05007_cas_gc_introspection%' +WHERE disk_name LIKE '%${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection%' AND event_type = 'Phase' AND phase = 'defer_decision'; -- The error path: a non-CA disk (the always-present local \`default\`) is rejected. @@ -111,5 +111,5 @@ SELECT 'ok'; # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. ${CLICKHOUSE_CLIENT} --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '05007_cas_gc_introspection'" || { + --query "SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_05007_cas_gc_introspection'" || { echo "FORGET failed"; exit 1; } diff --git a/tests/queries/0_stateless/05008_cas_gc_snapshot_prune.sh b/tests/queries/0_stateless/05008_cas_gc_snapshot_prune.sh index 66f11f10f329..af7f58052eaa 100755 --- a/tests/queries/0_stateless/05008_cas_gc_snapshot_prune.sh +++ b/tests/queries/0_stateless/05008_cas_gc_snapshot_prune.sh @@ -20,7 +20,7 @@ CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # shellcheck source=../shell_config.sh . "$CUR_DIR"/../shell_config.sh -DISK="05008_cas_gc_snapshot_prune" +DISK="${CLICKHOUSE_DATABASE}_05008_cas_gc_snapshot_prune" # CA-over-LOCAL object storage emits a one-time about emulated conditional operations on # mount; the .sh harness fails on ANY client stderr, so send only error+ logs to the client (real diff --git a/tests/queries/0_stateless/05009_cas_event_log.sql b/tests/queries/0_stateless/05009_cas_event_log.sh old mode 100644 new mode 100755 similarity index 52% rename from tests/queries/0_stateless/05009_cas_event_log.sql rename to tests/queries/0_stateless/05009_cas_event_log.sh index 285b9166c2fe..dc6866f24945 --- a/tests/queries/0_stateless/05009_cas_event_log.sql +++ b/tests/queries/0_stateless/05009_cas_event_log.sh @@ -1,13 +1,19 @@ --- Tags: no-fasttest --- ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. - --- Default-ON contract for `system.cas_log`: the per-event content-addressed audit log is --- enabled by default. `programs/server/config.xml` ships a `` section because the --- CAS disk feature is experimental and this audit log is its primary forensic instrument (it costs --- nothing when no CAS disk is configured — events are emitted only by content-addressed disks). After we --- exercise a content-addressed disk end-to-end (INSERT, OPTIMIZE), the table exists and carries this --- disk's write-path events. - +#!/usr/bin/env bash +# Tags: no-fasttest +# ^ cas is an object-storage metadata type; keep it off the minimal fasttest image. + +# Default-ON contract for `system.cas_log`: the per-event content-addressed audit log is +# enabled by default. `programs/server/config.xml` ships a `` section because the +# CAS disk feature is experimental and this audit log is its primary forensic instrument (it costs +# nothing when no CAS disk is configured — events are emitted only by content-addressed disks). After we +# exercise a content-addressed disk end-to-end (INSERT, OPTIMIZE), the table exists and carries this +# disk's write-path events. + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} --multiquery < 0 FROM system.cas_log -WHERE disk_name = '05009_cas_event_log' AND event_type = 'blob_put'; +WHERE disk_name = '${CLICKHOUSE_DATABASE}_05009_cas_event_log' AND event_type = 'blob_put'; DROP TABLE t_cas_event_log; -- FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would -- stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. SET send_logs_level = 'fatal'; -SYSTEM CAS FORGET '05009_cas_event_log'; +SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_05009_cas_event_log'; SELECT 'ok'; +EOF diff --git a/tests/queries/0_stateless/05010_cas_mounts_gc_health.sh b/tests/queries/0_stateless/05010_cas_mounts_gc_health.sh index 3bb813e71d8c..cbfef7af8310 100755 --- a/tests/queries/0_stateless/05010_cas_mounts_gc_health.sh +++ b/tests/queries/0_stateless/05010_cas_mounts_gc_health.sh @@ -25,9 +25,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05010', - name = '05010_cas_mounts_gc_health', - path = '05010_cas_mounts_gc_health_pool/', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05010', + name = '${CLICKHOUSE_DATABASE}_05010_cas_mounts_gc_health', + path = '${CLICKHOUSE_DATABASE}_05010_cas_mounts_gc_health_pool/', cas_gc_enabled = 1, cas_gc_interval_sec = 1), old_parts_lifetime = 1; @@ -36,16 +36,16 @@ INSERT INTO t_cas_mounts_gc_health SELECT number, toString(number) FROM numbers( TRUNCATE TABLE t_cas_mounts_gc_health; """ -${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '05010_cas_mounts_gc_health'" > /dev/null +${CLICKHOUSE_CLIENT} -q "SYSTEM CAS GC RUN '${CLICKHOUSE_DATABASE}_05010_cas_mounts_gc_health'" > /dev/null ${CLICKHOUSE_CLIENT} --multiline -q """ SELECT is_leader, wedged_namespace_count FROM system.cas_mounts -WHERE disk LIKE '%05010_cas_mounts_gc_health%'; +WHERE disk LIKE '%${CLICKHOUSE_DATABASE}_05010_cas_mounts_gc_health%'; SELECT pending_reclaim >= 0, last_success_age_seconds < 60 FROM system.cas_mounts -WHERE disk LIKE '%05010_cas_mounts_gc_health%'; +WHERE disk LIKE '%${CLICKHOUSE_DATABASE}_05010_cas_mounts_gc_health%'; DROP TABLE t_cas_mounts_gc_health; SELECT 'ok'; @@ -54,5 +54,5 @@ SELECT 'ok'; # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET call only. ${CLICKHOUSE_CLIENT} --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '05010_cas_mounts_gc_health'" || { + --query "SYSTEM CAS FORGET '${CLICKHOUSE_DATABASE}_05010_cas_mounts_gc_health'" || { echo "FORGET failed"; exit 1; } diff --git a/tests/queries/0_stateless/05015_cas_reject_fake_transaction.sh b/tests/queries/0_stateless/05015_cas_reject_fake_transaction.sh index 7496a8519800..b6756518f71a 100755 --- a/tests/queries/0_stateless/05015_cas_reject_fake_transaction.sh +++ b/tests/queries/0_stateless/05015_cas_reject_fake_transaction.sh @@ -19,9 +19,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05015', - name = '05015_cas_reject_fake_transaction', - path = '05015_cas_reject_fake_transaction_pool/', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05015', + name = '${CLICKHOUSE_DATABASE}_05015_cas_reject_fake_transaction', + path = '${CLICKHOUSE_DATABASE}_05015_cas_reject_fake_transaction_pool/', use_fake_transaction = 1); " 2>&1 | grep -cm1 "use_fake_transaction. cannot be enabled for metadata type" diff --git a/tests/queries/0_stateless/05020_cas_fsck.sh b/tests/queries/0_stateless/05020_cas_fsck.sh index a10ca4e33762..ee3462a24531 100755 --- a/tests/queries/0_stateless/05020_cas_fsck.sh +++ b/tests/queries/0_stateless/05020_cas_fsck.sh @@ -24,7 +24,7 @@ DISK_CA="disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05020', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05020', name = '${DISK_NAME}', path = '${POOL_DIR}/')" diff --git a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh index 1cc521a210ca..f775f84bbbf0 100755 --- a/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh +++ b/tests/queries/0_stateless/05023_cas_dropns_leaked_namespace.sh @@ -22,7 +22,7 @@ CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) POOL_DIR="${CLICKHOUSE_USER_FILES_UNIQUE}_05023_${RANDOM}" DISK_NAME="ca_05023_${CLICKHOUSE_TEST_UNIQUE_NAME}_${RANDOM}" -SERVER_ROOT_ID="dropns05023" +SERVER_ROOT_ID="${CLICKHOUSE_DATABASE}_dropns05023" rm -rf "${POOL_DIR:?}" mkdir -p "${POOL_DIR}" diff --git a/tests/queries/0_stateless/05024_cas_freeze_two_roots.sh b/tests/queries/0_stateless/05024_cas_freeze_two_roots.sh index 263f718a5579..e83c42fdabda 100755 --- a/tests/queries/0_stateless/05024_cas_freeze_two_roots.sh +++ b/tests/queries/0_stateless/05024_cas_freeze_two_roots.sh @@ -24,7 +24,9 @@ CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) TABLE_UUID=$(${CLICKHOUSE_CLIENT} --query "SELECT generateUUIDv4()") SHARED_BACKUP='shared_05024' B_BACKUP='own_b_05024' -DISK_B='05024_cas_freeze_b' +DISK_A="${CLICKHOUSE_DATABASE}_05024_cas_freeze_a" +DISK_B="${CLICKHOUSE_DATABASE}_05024_cas_freeze_b" +POOL_PATH="${CLICKHOUSE_DATABASE}_05024_cas_freeze_pool/" UNFREEZE_STRUCTURE='command_type String, partition_id String, part_name String, backup_name String, backup_path String, part_backup_path String' # `ALTER ... UNFREEZE` returns rows only under `alter_partition_verbose_result=1`; the default is off. @@ -46,7 +48,7 @@ create_on_root() { metadata_type = cas, cas_server_root_id = '$2', name = '$3', - path = '05024_cas_freeze_pool/', + path = '${POOL_PATH}', cas_gc_enabled = 1, cas_gc_interval_sec = 100000);" } @@ -77,14 +79,14 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05024_root_b', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05024_root_b', name = '${DISK_B}', - path = '05024_cas_freeze_pool/', + path = '${POOL_PATH}', cas_gc_enabled = 1, cas_gc_interval_sec = 100000);" # Root A freezes, then releases the UUID. Its freeze must outlive both the table and a collection round. -create_on_root t_cas_freeze_a 05024_root_a 05024_cas_freeze_a +create_on_root t_cas_freeze_a "${CLICKHOUSE_DATABASE}_05024_root_a" "${DISK_A}" ${CLICKHOUSE_CLIENT} --query "INSERT INTO t_cas_freeze_a VALUES (1, 'a');" ${CLICKHOUSE_CLIENT} --query "ALTER TABLE t_cas_freeze_a FREEZE PARTITION 1 WITH NAME '${SHARED_BACKUP}';" ${CLICKHOUSE_CLIENT} --query "DROP TABLE t_cas_freeze_a;" @@ -93,7 +95,7 @@ drain_gc # Root B reaches the SAME table path by reusing the UUID. Its own backup uses a distinct name: making # both roots publish the same ref would mix two independent CAS writer lifecycles before `UNFREEZE` # gets a chance to exercise the destructive lookup under test. -create_on_root t_cas_freeze_b 05024_root_b "${DISK_B}" +create_on_root t_cas_freeze_b "${CLICKHOUSE_DATABASE}_05024_root_b" "${DISK_B}" ${CLICKHOUSE_CLIENT} --query "INSERT INTO t_cas_freeze_b VALUES (1, 'b');" ${CLICKHOUSE_CLIENT} --query "ALTER TABLE t_cas_freeze_b FREEZE PARTITION 1 WITH NAME '${B_BACKUP}';" @@ -115,7 +117,7 @@ ${CLICKHOUSE_CLIENT} --query "DROP TABLE t_cas_freeze_b;" # (3) A's freeze must still be there. Recreate A's table on root A with the same UUID -- the freeze is # addressed by path, so the recreated table reaches its predecessor's snapshot -- and release it. # Pre-fix this prints nothing, because B's foreign unfreeze above already dropped the shared namespace. -create_on_root t_cas_freeze_a 05024_root_a 05024_cas_freeze_a +create_on_root t_cas_freeze_a "${CLICKHOUSE_DATABASE}_05024_root_a" "${DISK_A}" echo 'unfreeze_a' unfreeze_and_print t_cas_freeze_a "${SHARED_BACKUP}" @@ -127,6 +129,6 @@ ${CLICKHOUSE_CLIENT} --query "SELECT 'dropped_ok';" # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET calls. # Two independent disks were created (root A's and root B's); each needs its own FORGET. ${CLICKHOUSE_CLIENT} --allow_repeated_settings --send_logs_level=fatal \ - --query "SYSTEM CAS FORGET '05024_cas_freeze_a'" + --query "SYSTEM CAS FORGET '${DISK_A}'" ${CLICKHOUSE_CLIENT} --allow_repeated_settings --send_logs_level=fatal \ --query "SYSTEM CAS FORGET '${DISK_B}'" diff --git a/tests/queries/0_stateless/05025_cas_attach_partition_cross_disk.sh b/tests/queries/0_stateless/05025_cas_attach_partition_cross_disk.sh index 5b83c4d42d51..34f4ff411591 100755 --- a/tests/queries/0_stateless/05025_cas_attach_partition_cross_disk.sh +++ b/tests/queries/0_stateless/05025_cas_attach_partition_cross_disk.sh @@ -47,9 +47,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05025_dst', - name = '05025_cas_dst', - path = '05025_cas_dst_pool/');" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05025_dst', + name = '${CLICKHOUSE_DATABASE}_05025_cas_dst', + path = '${CLICKHOUSE_DATABASE}_05025_cas_dst_pool/');" ${CLICKHOUSE_CLIENT} --query "INSERT INTO src_plain SELECT number % 2, toString(number) FROM numbers(64);" @@ -73,9 +73,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05025_shared_a', - name = '05025_cas_shared_a', - path = '05025_cas_shared_pool/');" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05025_shared_a', + name = '${CLICKHOUSE_DATABASE}_05025_cas_shared_a', + path = '${CLICKHOUSE_DATABASE}_05025_cas_shared_pool/');" ${CLICKHOUSE_CLIENT} --query " CREATE TABLE dst_cas_same_pool (k UInt32, v String) @@ -84,9 +84,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05025_shared_b', - name = '05025_cas_shared_b', - path = '05025_cas_shared_pool/');" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05025_shared_b', + name = '${CLICKHOUSE_DATABASE}_05025_cas_shared_b', + path = '${CLICKHOUSE_DATABASE}_05025_cas_shared_pool/');" ${CLICKHOUSE_CLIENT} --query "INSERT INTO src_cas SELECT number % 2, toString(number) FROM numbers(64);" @@ -119,9 +119,9 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05025_dst_repl', - name = '05025_cas_dst_repl', - path = '05025_cas_dst_repl_pool/');" + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05025_dst_repl', + name = '${CLICKHOUSE_DATABASE}_05025_cas_dst_repl', + path = '${CLICKHOUSE_DATABASE}_05025_cas_dst_repl_pool/');" ${CLICKHOUSE_CLIENT} --query "INSERT INTO src_plain_repl SELECT number % 2, toString(number) FROM numbers(64);" ${CLICKHOUSE_CLIENT} --query "ALTER TABLE dst_cas_repl ATTACH PARTITION 1 FROM src_plain_repl;" @@ -144,7 +144,8 @@ ${CLICKHOUSE_CLIENT} --query "SELECT 'dropped_ok';" # FORGET logs an operator WARNING; the harness runs the client at --send_logs_level=warning, which would # stream that expected warning to stderr and be flagged as a failure. Suppress it for the FORGET calls. # Four independent content-addressed disks were created across the three legs; each needs its own FORGET. -for disk in 05025_cas_dst 05025_cas_shared_a 05025_cas_shared_b 05025_cas_dst_repl; do +for disk in "${CLICKHOUSE_DATABASE}_05025_cas_dst" "${CLICKHOUSE_DATABASE}_05025_cas_shared_a" \ + "${CLICKHOUSE_DATABASE}_05025_cas_shared_b" "${CLICKHOUSE_DATABASE}_05025_cas_dst_repl"; do ${CLICKHOUSE_CLIENT} --allow_repeated_settings --send_logs_level=fatal \ --query "SYSTEM CAS FORGET '${disk}'" || { echo "FORGET failed for ${disk}"; exit 1; } diff --git a/tests/queries/0_stateless/05026_cas_manifest_path_newline.sh b/tests/queries/0_stateless/05026_cas_manifest_path_newline.sh index 099ad92a9c01..809c14cc24c5 100755 --- a/tests/queries/0_stateless/05026_cas_manifest_path_newline.sh +++ b/tests/queries/0_stateless/05026_cas_manifest_path_newline.sh @@ -24,7 +24,7 @@ SETTINGS disk = disk( type = object_storage, object_storage_type = local, metadata_type = cas, - cas_server_root_id = '05026', + cas_server_root_id = '${CLICKHOUSE_DATABASE}_05026', name = '${DISK_NAME}', path = '${POOL_DIR}/');" From 16fd0d65e57964a00a13da4e5e7d5db2e82143ab Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 00:39:55 +0200 Subject: [PATCH 70/81] cas: unify the decommission drain clock with the mount lease's boot clock CI run 7 of PR #2300 turned CASDecommission.FailedDrainKeepsSlotThenResumes and CASDecommission.ManifestDebrisFailureKeepsSlotThenResumes red under ASan/UBSan: the farewell write on Pool teardown gave up at the lease deadline after zero attempts, so the mount slot was never retired and the resume half of each test refused with "pool member is alive or contended". decommissionPoolMember's drain_now_fn/drain_sleep_fn seam replaces the clock the drain's own request engine paces its retries on, but the opened Pool's mount-lease renewer binds its farewell deadline to a separate clock, PoolConfig::boot_ms_fn, which was left on the real boot clock (CLOCK_BOOTTIME) regardless. On a freshly booted CI VM (real boot time below FakeClock's starting instant) the two clocks disagreed enough that the farewell's own Retry::untilLeaseSafe bound looked already-expired before the first attempt; a long-lived dev box's real uptime dwarfing the fake clock is why this passed locally. Fold drain_now_fn into config.boot_ms_fn whenever the caller left the latter unset, so a test (or any other caller) faking only the request clock does not end up comparing it against an unrelated one. Verified both CasRequests.cpp's bootClockMs and CasServerRoot.cpp's defaultBootMs are CLOCK_BOOTTIME, so production itself already uses one clock for both purposes -- the mismatch was confined to this test seam. Add a regression test that pins the fake clock far beyond any real host's boot time, so the mismatch (and the fix) reproduce deterministically on every machine rather than only a freshly booted one. Unifying the clocks also exposed a second-order timing issue in FailedDrainKeepsSlotThenResumes: its FakeClock fast-forwards through the entire 90 s Retry::standard() window while draining one failing object, and drain_now_fn is now also the admin session's boot clock, so the default 30 s mount lease TTL was no longer enough for the farewell to fit by the time the drain's own retry exhaustion had "elapsed". A real decommission's background renewer keeps the lease fresh over that much real time; nothing in the test advances real time to let it. Widen that one test's admin PoolConfig::mount_lease_ttl_ms to give the farewell comfortable headroom past its own retry-exhaustion budget. Gates: release and ASan unit_tests_dbms --gtest_filter='CASDecommission*' (37/37, including 8x --gtest_repeat --gtest_shuffle), release+ASan --gtest_filter='CASDecommission*:CASEnvelopeWiring*:CASGCBoundedWalk*' (49/49), and the full ASan gate --gtest_filter='CAS*:*S3*:*ObjectStorage*:*Teardown*' (2703/2703, one pre-existing unrelated skip). All green. CI report: https://github.com/ClickHouse/ClickHouse/pull/2300 Follow-up folded in: `FakeClock` is thread-safe (the renewal thread now reads it through `boot_ms_fn` while the drain thread advances it), and `PoolConfig::retry_sleep_fn` installs the test sleep together with the test clock so the bootstrap requests of `openForDecommission` never pace real sleeps against a frozen clock; a pacing regression test pins that. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ContentAddressed/Pool/CasPool.cpp | 13 ++- .../ContentAddressed/Pool/CasPool.h | 9 ++ .../Tools/CasDecommission.cpp | 24 +++- .../ContentAddressed/Tools/CasDecommission.h | 7 ++ src/Disks/tests/cas_test_helpers.h | 22 +++- src/Disks/tests/gtest_cas_decommission.cpp | 105 +++++++++++++++++- 6 files changed, 165 insertions(+), 15 deletions(-) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 041e8d1d9086..8ea206618492 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -181,9 +181,9 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) [this](uint64_t g, uint64_t needed) { return mount_runtime.admit(g, needed); }, [this](uint64_t g) { mount_runtime.checkFenceOrThrow(g); }}, config.boot_ms_fn, - mountPlaneSleepFn(), + config.retry_sleep_fn ? config.retry_sleep_fn : mountPlaneSleepFn(), &hot_keys) - , farewell_requests(pool_backend, Fence::open(), config.boot_ms_fn, {}, &hot_keys) + , farewell_requests(pool_backend, Fence::open(), config.boot_ms_fn, config.retry_sleep_fn, &hot_keys) /// The open plane's fence is the pool's teardown flag: generation 0 forever, exactly like /// `Fence::open`, but `admit` refuses once `beginTeardown` ran. A GC round, an FSCK or a probe in /// flight is then refused at its next request instead of running to completion under a disk that @@ -199,7 +199,7 @@ Pool::Pool(BackendPtr backend_, PoolConfig config_, PoolMeta meta_) [this](uint64_t, uint64_t) { return teardownBegun() ? Fence::Admit::LostOrRearmed : Fence::Admit::Ok; }, [](uint64_t) {}}, config.boot_ms_fn, - openPlaneSleepFn(), + config.retry_sleep_fn ? config.retry_sleep_fn : openPlaneSleepFn(), &hot_keys) /// Seed the monotone admitted-algo cache from the pool state `createOrValidate` already /// established (fresh create, steady-state member, or a just-completed admission union) -- @@ -928,8 +928,11 @@ PoolPtr Pool::openForDecommission(BackendPtr backend, PoolConfig config, const S /// observation wait -- see `mountWritable`). Owner anchor absent + mount absent = nothing to /// decommission. /// The open plane: this factory impersonates the victim to take its mount, so there is no lease of - /// ours to be gated on until the claim below establishes one. - CasRequests bootstrap_requests(backend, Fence::open(), config.boot_ms_fn); + /// ours to be gated on until the claim below establishes one. `config.retry_sleep_fn` must travel + /// with `config.boot_ms_fn`: a retry loop bound to a frozen test clock that only a fake sleep + /// advances would otherwise retry forever against this engine's default REAL sleep, which never + /// calls it -- the deadline it measures against would never appear to elapse. + CasRequests bootstrap_requests(backend, Fence::open(), config.boot_ms_fn, config.retry_sleep_fn); CasOperation owner_op = bootstrap_requests.admit(); std::optional victim_uuid = readOwnerUuid(owner_op, layout, victim_srid); if (!victim_uuid) diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 71a10fc803a2..cbde11caa87c 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -250,6 +250,15 @@ struct PoolConfig /// boot clock (`Pool::bootMs`); injected by tests to drive the fence deadline deterministically. std::function boot_ms_fn = {}; + /// The inter-attempt sleep for the mount, farewell and GC request planes (`mount_requests`, + /// `farewell_requests`, `gc_requests`), installed at their CONSTRUCTION -- before this `Pool` has + /// claimed or read anything. Empty = each plane's own production default (an interruptible real + /// sleep for the mount and GC planes, `CasRequests`'s own real sleep for the farewell plane). A test + /// that also freezes `boot_ms_fn` must supply a matching sleep here: a retry loop bound to a clock + /// that only moves when this function is called would otherwise retry forever against a REAL sleep + /// that never calls it, because the deadline it measures against never appears to elapse. + std::function retry_sleep_fn = {}; + /// Test hook for open/remount waits: `Pool::waitSleep` -- the mount-claim observation loop's poll -- /// routes through this function when set instead of a real `std::this_thread::sleep_for`, so a test /// observes every wait without actually blocking. Empty (the production default) sleeps for real. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp index eaee5cf485f4..eef2ba04d9bf 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.cpp @@ -149,11 +149,33 @@ DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const CasRefCatalog::Snapshot catalog_cut = CasRefCatalog::read(preflight_op, catalog_layout); catalog_cut.life_index.throwIfAmbiguous("CAS decommission"); + /// `drain_now_fn`/`drain_sleep_fn` below replace the clock and sleep the STANDALONE `requests` + /// engine (opened further down) paces its own retries on, but `config.boot_ms_fn`/ + /// `config.retry_sleep_fn` -- the seams `Pool::openForDecommission` itself constructs its + /// mount/farewell/GC planes with -- are distinct. Left unset, those planes retry on the real boot + /// clock and a real sleep, and a caller that fakes only the standalone engine's clock ends up + /// comparing it against an unrelated one at the mount lease's farewell bound (or, worse, against a + /// clock that only the standalone engine's sleep advances: a retry loop on `mount_requests`/ + /// `gc_requests` bound to that frozen clock while actually sleeping for real would never see its + /// own deadline elapse). Fold BOTH into `config` here, together, unless the caller already asked + /// for a specific clock or sleep of its own -- installing only one of the two is exactly the + /// half-fix that leaves the other seam retrying forever. + if (drain_now_fn && drain_sleep_fn) + { + if (!config.boot_ms_fn) + config.boot_ms_fn = drain_now_fn; + if (!config.retry_sleep_fn) + config.retry_sleep_fn = drain_sleep_fn; + } + config.event_sink = sink; PoolPtr admin = Pool::openForDecommission(std::move(backend), std::move(config), victim_srid); if (drain_now_fn && drain_sleep_fn) { - /// `sweepNamespace` below issues its deletes on `admin`'s own GC plane. + /// Re-affirms the same values `config` above already installed on `mount_requests`/ + /// `farewell_requests`/`gc_requests` at construction, and additionally wires `ref_ledger`'s own + /// retry sleep, which has no construction-time seam of its own. `sweepNamespace` below issues + /// its deletes on `admin`'s own GC plane. admin->setCasRequestNowFnForTest(drain_now_fn); admin->setCasRetrySleepForTest(drain_sleep_fn); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h index 69c6875d27c7..6832fd9cfe73 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Tools/CasDecommission.h @@ -54,6 +54,13 @@ struct DecommissionReport /// backend, not one of `Pool`'s planes, so `Pool::setCasRetrySleepForTest` cannot reach it. A test /// driving a latched per-object fault to `Retry::standard()`'s own give-up needs this seam, or it pays /// the real 90-second deadline. +/// +/// `drain_now_fn`/`drain_sleep_fn`, when BOTH set, also become the opened `Pool`'s own boot clock and +/// retry sleep (`PoolConfig::boot_ms_fn`/`retry_sleep_fn`) whenever the caller left those fields unset: +/// the mount lease's farewell deadline is bound to the boot clock, and `Pool::openForDecommission`'s own +/// mount/farewell/GC planes are constructed with it too, so a caller that fakes only the standalone +/// engine's clock must not end up comparing it against the real one, or -- worse -- against a plane that +/// shares the frozen clock but still sleeps for real between retries. DecommissionReport decommissionPoolMember(BackendPtr backend, PoolConfig config, const String & victim_srid, const CasEventSink & sink = {}, const std::function & request_gc_round = {}, diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 24cf357ed110..541479c57434 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -190,22 +190,32 @@ inline DB::ContentAddressedSettings makeSettingsForTest(const std::string & serv /// A clock that only ever moves when something sleeps on it, plus the record of every sleep it /// served. Injected into `CasRequests` so a policy's whole 90-second deadline is exercised in a test /// that takes no wall-clock time, and so the schedule itself -- how many pauses, how long -- becomes -/// an assertion rather than a wait. Single-threaded by construction: two threads sharing one would -/// race on both fields, so a concurrency test uses the real clock instead. +/// an assertion rather than a wait. `now` is atomic and `sleeps` is mutex-guarded because a +/// decommission session's background mount-lease renewer reads this same clock (via +/// `PoolConfig::boot_ms_fn`) from its own thread while the caller's thread drives it forward through +/// `sleepFn` -- relaxed ordering is enough since nothing here needs a happens-before relationship +/// beyond the value eventually becoming visible; direct field reads from a single thread after the +/// clock stops moving (the common case in this file's other users) are unaffected. struct FakeClock { - uint64_t now = 1'000'000; + std::atomic now{1'000'000}; std::vector sleeps; - std::function nowFn() { return [this] { return now; }; } + std::function nowFn() { return [this] { return now.load(std::memory_order_relaxed); }; } std::function sleepFn() { return [this](uint64_t ms) { - sleeps.push_back(ms); - now += ms; + { + std::lock_guard lock(sleeps_mutex); + sleeps.push_back(ms); + } + now.fetch_add(ms, std::memory_order_relaxed); }; } + +private: + std::mutex sleeps_mutex; }; /// Run `fn`, expect a DB::Exception with EXACTLY `expected_code` (CORRUPTED_DATA-vs-NOT_IMPLEMENTED diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index f81fc7b8e591..3548af8a7753 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -82,6 +82,32 @@ class FailingDeleteBackend : public InMemoryBackend bool latched = false; }; +/// Fails a read of one designated key a fixed number of times with a transient `Poco::TimeoutException` +/// -- the class the request engine classifies as a transport failure and reissues -- before delegating +/// to the base `InMemoryBackend`. Drives the owner-object read `Pool::openForDecommission` itself issues +/// on the OPEN plane (`claimOwnerOrThrow` -> `readOwnerObject`, CasServerRoot.cpp) through a bounded +/// number of paced retries DURING opening, before `decommissionPoolMember` gets a chance to install +/// anything on the already-open `Pool`. +class FlakyReadBackend : public InMemoryBackend +{ +public: + void failReadNTimes(const String & key, int times) { flaky_key = key; remaining = times; } + + std::optional read(const String & key, DB::Cas::TransportAccess & access) override + { + if (key == flaky_key && remaining > 0) + { + --remaining; + throw Poco::TimeoutException("injected transient read failure for " + key); + } + return InMemoryBackend::read(key, access); + } + +private: + String flaky_key; + int remaining = 0; +}; + /// Replaces the durable catalog immediately after returning the first armed catalog read. This /// distinguishes the immutable cut validated before decommission impersonation from a later mount /// safety observation without assuming those two decisions share one GET. @@ -1309,11 +1335,17 @@ TEST(CASDecommission, FailedDrainKeepsSlotThenResumes) auto failing = std::make_shared(inner, "p/roots/victim/"); /// The engine reissues an unresolved delete until its own retry window closes, measured on this /// clock, so the fault (armed across every reissue) reaches a genuine give-up with no real time - /// passing. + /// passing -- `clock` fast-forwards through the whole 90 s `Retry::standard()` window in one call. + /// `drain_now_fn` is also this session's boot clock (decommissionPoolMember unifies the two), so the + /// admin's own mount lease needs a TTL well past that fast-forward or the farewell it attempts on + /// the way out would refuse against a deadline the retry exhaustion already ran past -- a real + /// decommission's background renewer would have kept the deadline current over 90 real seconds, but + /// nothing here advances real time to let it. DB::Cas::tests::FakeClock clock; const auto first = decommissionPoolMember( - failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + failing, + PoolConfig{.pool_prefix = "p", .server_root_id = "a1", .mount_lease_ttl_ms = std::chrono::milliseconds(300'000)}, + "victim", /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) @@ -1384,6 +1416,73 @@ TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) << "the slot is still the resume anchor -- nothing was retired against unreclaimed debris"; } +/// CI run 7 (PR #2300): `drain_now_fn` fakes the drain's own request clock, but the mount lease's +/// farewell deadline is bound to `PoolConfig::boot_ms_fn`, a distinct clock that -- before this test's +/// fix -- stayed on the real boot clock regardless. On a freshly booted CI host (real boot time below +/// `FakeClock`'s starting instant) the two clocks disagreed enough that the farewell's own bound looked +/// already-expired, so `admin.reset()` released nothing and the slot was never retired -- passing locally +/// only because a long-lived dev box's real uptime dwarfs the fake clock. Pin the fake clock far beyond +/// ANY real host's boot time (rather than relying on the host actually being fresh) so the mismatch -- +/// and the fix -- are exercised deterministically on every machine. +TEST(CASDecommission, DrainClockUnifiesWithTheFarewellBootClockRegardlessOfHostUptime) +{ + auto backend = std::make_shared(); + { auto victim = openVictim(backend); } /// identity only -- no namespace, so retirement runs straight + /// to the farewell instead of stopping on an unrelated warning + + DB::Cas::tests::FakeClock clock; + clock.now = 1'000'000'000'000'000ULL; /// dwarfs any real CLOCK_BOOTTIME on any host + const auto report = decommissionPoolMember( + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", + /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.slot_removed); + OperationForTest raw_op(*backend); + EXPECT_FALSE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) + << "one clock for both the request engine and the mount lease -- the farewell must commit and the " + "slot must retire in a single call, with no leftover mount to resume against"; +} + +/// Review round 9r2: folding `drain_now_fn` into `config.boot_ms_fn` alone left `Pool::openForDecommission`'s +/// own mount/farewell/GC planes -- constructed and used DURING opening, before `decommissionPoolMember` +/// gets a chance to call `setCasRequestNowFnForTest`/`setCasRetrySleepForTest` on the already-open `Pool` +/// -- retrying on a clock that only a fake SLEEP advances, while those planes still slept for REAL between +/// attempts. A transient failure during opening (the owner-object read on the open plane) would then never +/// see its own `Retry::standard()` deadline elapse, because the bound is measured against a clock frozen +/// at the value it had when the retry loop started: nothing calls the fake clock's `sleepFn` from a path +/// that still sleeps for real. `PoolConfig::retry_sleep_fn` closes this by installing the matching fake +/// sleep on those same planes AT CONSTRUCTION, together with `boot_ms_fn`. +/// +/// Failing-first without risking an actual hang: this fixture flakes the owner read 5 times, so a correct +/// fix paces exactly 5 retries on the fake clock and returns in well under a second of real time; the +/// pre-fix code either never returns (the bound never elapses) or, if it did return, would show an empty +/// `clock.sleeps` (nothing ever called the fake sleep) and real wall time consumed by 5 real backoffs. +TEST(CASDecommission, OpeningRetriesPaceOnTheSameFakeClockAndSleepAsTheDrain) +{ + auto backend = std::make_shared(); + { auto victim = openVictim(backend); } /// identity only + + const Layout layout("p"); + backend->failReadNTimes(layout.ownerKey("victim"), /*times=*/5); + + DB::Cas::tests::FakeClock clock; + clock.now = 1'000'000'000'000'000ULL; + + const auto started = std::chrono::steady_clock::now(); + const auto report = decommissionPoolMember( + backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", + /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + const auto wall_elapsed = std::chrono::steady_clock::now() - started; + + EXPECT_TRUE(report.warnings.empty()); + EXPECT_TRUE(report.slot_removed); + EXPECT_FALSE(clock.sleeps.empty()) + << "the opening retries must have been paced on the injected clock, not a real sleep"; + EXPECT_LT(wall_elapsed, std::chrono::seconds(5)) + << "paced on the fake clock, five retries during opening should cost no real wall time at all"; +} + /// Task 5 (Task-1 carry-forward, escalated by review): preserve recovery from the legacy partial /// hand-cleanup shape where owner and epoch are absent but the mount lease remains. Triage #9 changed /// new retirements to delete `mountKey`/`epochKey` and tombstone `ownerKey`, so the current tail no From 6d34b1d4bd3aa66aacdcb7304e6743575b5c263d Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 00:39:55 +0200 Subject: [PATCH 71/81] cas: assert the S3 single-attempt connect cap by difference under sanitizers CI run 7 of PR #2300 turned three CASEnvelopeWiring connect-cap timing assertions red under MSan: EXPECT_LT(capped_elapsed.count(), 1000) failed with 1470-2494 ms (gtest_cas_s3_single_attempt_client.cpp:702, :725, :786), against a base connect timeout of 2000 ms and a single-attempt connect cap of 100 ms, with the uncapped attempt asserted >= 1500 ms. A sanitizer build adds a roughly constant addend to both the capped and uncapped connect attempts, so an absolute upper bound on the capped side alone is not sanitizer-safe. Assert the DIFFERENCE instead: the cap must remove at least half of the connect budget it is capping (default_elapsed - capped_elapsed >= (base_connect_timeout_ms - cap) / 2), keeping the existing EXPECT_GE(default_elapsed, 1500) lower bound on the uncapped side. Applied to all three failing sites; the third (FreezeConnectTimeoutCapReachesTheBackendOverProductionDispatch) reuses the uncapped backend's own measurement at :773 as its baseline instead of a second literal 1500 ms floor. Gates: release and ASan unit_tests_dbms --gtest_filter='CASEnvelopeWiring*' (4/4, including 5x --gtest_repeat), release+ASan --gtest_filter='CASDecommission*:CASEnvelopeWiring*:CASGCBoundedWalk*' (49/49), and the full ASan gate --gtest_filter='CAS*:*S3*:*ObjectStorage*:*Teardown*' (2703/2703, one pre-existing unrelated skip). All green. No MSan build was available locally; the fenced values match the MSan CI log exactly. CI report: https://github.com/ClickHouse/ClickHouse/pull/2300 The absolute `< 1000 ms` bound stays on non-sanitizer builds (the difference bound alone is weaker there), `capped < uncapped` holds unconditionally, and the frozen cap is also read back from the backend so sanitizer builds keep a non-timing discriminator. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../gtest_cas_s3_single_attempt_client.cpp | 43 ++++++++++++++++++- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp index 94d2ca0288a9..a0b4e0f8e3be 100644 --- a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp +++ b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -699,7 +700,18 @@ TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTim .attempt_timeout_ms = single_attempt_timeout_ms, .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); }); + EXPECT_LT(capped_elapsed.count(), default_elapsed.count()) + << "the cap must remove SOME of the connect budget, unconditionally"; + /// A sanitizer build adds a roughly constant addend to both measurements, so the PRIMARY fence + /// is the DIFFERENCE the cap made, not an absolute bound: at least half of the connect budget it + /// removed. + EXPECT_GE(default_elapsed.count() - capped_elapsed.count(), + (base_connect_timeout_ms - static_cast(single_attempt_connect_cap_ms)) / 2); +#if !defined(DEBUG_OR_SANITIZER_BUILD) + /// Release builds keep the original tighter absolute bound too: sanitizer instrumentation + /// overhead is the only reason it was loosened to a difference above. EXPECT_LT(capped_elapsed.count(), 1000); +#endif } /// Conditional DELETE: removeObjectIfTokenMatches's ObjectStorageControlRequest-taking overload. @@ -722,7 +734,18 @@ TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTim .attempt_timeout_ms = single_attempt_timeout_ms, .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); }); + EXPECT_LT(capped_elapsed.count(), default_elapsed.count()) + << "the cap must remove SOME of the connect budget, unconditionally"; + /// A sanitizer build adds a roughly constant addend to both measurements, so the PRIMARY fence + /// is the DIFFERENCE the cap made, not an absolute bound: at least half of the connect budget it + /// removed. + EXPECT_GE(default_elapsed.count() - capped_elapsed.count(), + (base_connect_timeout_ms - static_cast(single_attempt_connect_cap_ms)) / 2); +#if !defined(DEBUG_OR_SANITIZER_BUILD) + /// Release builds keep the original tighter absolute bound too: sanitizer instrumentation + /// overhead is the only reason it was loosened to a difference above. EXPECT_LT(capped_elapsed.count(), 1000); +#endif } } @@ -767,11 +790,12 @@ TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapReachesTheBackendOverProductionDi auto uncapped_backend = std::make_shared( storage, DB::Cas::ObjectStorageBackend::Mode::Native, /*single_attempt_control_plane_=*/false, /*attempt_timeout_ms_=*/0, /*connect_timeout_cap_ms_=*/0); + std::chrono::milliseconds uncapped_elapsed{}; { DB::Cas::CasRequests requests(DB::Cas::BackendPtr(uncapped_backend), DB::Cas::Fence::open()); auto op = requests.admit(); - const auto elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); - EXPECT_GE(elapsed.count(), 1500); + uncapped_elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); + EXPECT_GE(uncapped_elapsed.count(), 1500); } /// The derived cap, handed to the backend exactly as `openPoolView` constructs it (:812-822) for a @@ -779,11 +803,26 @@ TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapReachesTheBackendOverProductionDi auto capped_backend = std::make_shared( storage, DB::Cas::ObjectStorageBackend::Mode::Native, /*single_attempt_control_plane_=*/true, cas_attempt_timeout_ms, *cap); + /// Deterministic, non-timing corroboration alongside the timing assertions below: cheap because + /// `ObjectStorageBackend` already exposes its own budget, though it only proves the constructor + /// argument the line above passed was stored -- not that it reached the S3 client's actual connect + /// timeout, which only the timing assertions below can show. + EXPECT_EQ(capped_backend->connectTimeoutCapMs(), *cap); { DB::Cas::CasRequests requests(DB::Cas::BackendPtr(capped_backend), DB::Cas::Fence::open()); auto op = requests.admit(); const auto elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); + EXPECT_LT(elapsed.count(), uncapped_elapsed.count()) + << "the cap must remove SOME of the connect budget, unconditionally"; + /// A sanitizer build adds a roughly constant addend to both measurements, so the PRIMARY fence + /// is the DIFFERENCE the cap made, not an absolute bound: at least half of the connect budget it + /// removed. + EXPECT_GE(uncapped_elapsed.count() - elapsed.count(), (base_connect_timeout_ms - static_cast(*cap)) / 2); +#if !defined(DEBUG_OR_SANITIZER_BUILD) + /// Release builds keep the original tighter absolute bound too: sanitizer instrumentation + /// overhead is the only reason it was loosened to a difference above. EXPECT_LT(elapsed.count(), 1000); +#endif } } From d876cda683f06972f54470910c0366a93002671d Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Tue, 8 Sep 2026 23:32:35 +0200 Subject: [PATCH 72/81] cas: guard ChasingWriterBackend's hook state with a mutex under TSan CI run 7 of PR #2300 turned CASGCBoundedWalk.ARoundFoldsThroughItsRoundStartTailAndLeavesTheStragglers red under TSan: a data race on ChasingWriterBackend::appending/published/ layout (gtest_cas_gc_bounded_walk.cpp:167), because read() mutates them without synchronization while the GC fold read-ahead (Gc/CasGcReadAhead.h, a ThreadPool in front of one operation) can land several hinted reads on different worker threads at once. The class comment's premise -- "a synchronous hook rather than a thread" -- no longer holds for the reads themselves; it was true only for the single append the hook performs, not for the concurrent reads that can trigger it. Guard the hook's mutable state (layout, ns, published, limit, appending) with a mutex, held across the check-and-publish so the re-entrancy guard keeps meaning across threads, not just within one call. publishAt issues backend calls of its own, so the lock is released before it runs and re-acquired only to record the result -- holding it across publishAt would either self-deadlock on a re-entrant call or serialize every read-ahead worker behind the one doing the append. arm/disarm/ publishedThrough take the same lock. Updated the class comment to state the current concurrency model instead of the no-longer-true synchronous premise. No build_tsan exists in this worktree, so the fix was verified under ASan and release only, with repeats; TSan was not run locally. Gates: release and ASan (5x --gtest_repeat) unit_tests_dbms --gtest_filter='CASGCBoundedWalk*' (8/8), release+ASan --gtest_filter='CASDecommission*:CASEnvelopeWiring*:CASGCBoundedWalk*' (49/49), and the full ASan gate --gtest_filter='CAS*:*S3*:*ObjectStorage*:*Teardown*' (2703/2703, one pre-existing unrelated skip). All green. CI report: https://github.com/ClickHouse/ClickHouse/pull/2300 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_gc_bounded_walk.cpp | 67 +++++++++++++------ 1 file changed, 48 insertions(+), 19 deletions(-) diff --git a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp index 076487c7d88b..f97235ffd5aa 100644 --- a/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp +++ b/src/Disks/tests/gtest_cas_gc_bounded_walk.cpp @@ -1,5 +1,7 @@ #include +#include + #include #include #include @@ -129,13 +131,14 @@ std::map runRoundCapturingIntake(Gc & gc, UniversePolicy policy /// A store whose writer keeps pace with the walker EXACTLY: every time the fold reads the newest record /// by exact key, one more record lands above it. /// -/// This is a mid-round appender expressed as a synchronous hook rather than as a thread, and the -/// determinism is the point. The property under test is "the round stops at the tail it froze, however -/// much arrives afterwards", and a thread can only make appends arrive at times the scheduler chooses -- -/// including, on an unlucky run, entirely after the walk has gone past. The hook reproduces the WORST -/// case (writer rate == walker rate, the rate at which the unbounded walk provably never terminates) on -/// every run, and `max_appends` bounds it so that the UNPATCHED walk still finishes and can be measured -/// rather than hanging the suite. +/// This is a mid-round appender expressed as a hook rather than as a background thread, and the +/// determinism of WHEN it fires is the point: a real thread can only make appends arrive at times the +/// scheduler chooses, including, on an unlucky run, entirely after the walk has gone past. The hook +/// reproduces the WORST case (writer rate == walker rate, the rate at which the unbounded walk provably +/// never terminates) on every run, and `max_appends` bounds it so that the UNPATCHED walk still finishes +/// and can be measured rather than hanging the suite. The GC fold read-ahead can still land `read` calls +/// for several hinted keys on different worker threads at once, so the hook's own state is mutex-guarded +/// rather than assumed single-threaded. class ChasingWriterBackend : public CountingBackend { public: @@ -143,6 +146,7 @@ class ChasingWriterBackend : public CountingBackend /// `max_appends` further records. void arm(const Layout * layout_, const RootNamespace & ns_, uint64_t published_through, uint64_t max_appends) { + std::lock_guard lock(hook_mutex); layout = layout_; ns = ns_; published = published_through; @@ -150,29 +154,54 @@ class ChasingWriterBackend : public CountingBackend } /// Stop appending; the tail stands still from here on. - void disarm() { layout = nullptr; } + void disarm() + { + std::lock_guard lock(hook_mutex); + layout = nullptr; + } - uint64_t publishedThrough() const { return published; } + uint64_t publishedThrough() const + { + std::lock_guard lock(hook_mutex); + return published; + } std::optional read(const String & key, DB::Cas::TransportAccess & access) override { auto result = CountingBackend::read(key, access); - if (!layout || appending || published >= limit) - return result; - if (key != layout->refLogKey(fixture::fixtureLife(ns), RefTxnId{1, published})) - return result; - - /// The walk just consumed the tail; the writer answers with the next record. Guarded against - /// re-entry because publishing issues backend calls of its own. - appending = true; - const uint64_t next = published + 1; - publishAt(*this, *layout, ns, RefTxnId{1, next}, "ref_" + std::to_string(next), next, DB::UInt128(next)); + + const Layout * layout_snapshot = nullptr; + RootNamespace ns_snapshot; + uint64_t next = 0; + { + std::lock_guard lock(hook_mutex); + if (!layout || appending || published >= limit) + return result; + if (key != layout->refLogKey(fixture::fixtureLife(ns), RefTxnId{1, published})) + return result; + + /// The walk just consumed the tail; the writer answers with the next record. Guarded + /// against re-entry (by another read-ahead worker, not just the same thread) because + /// publishing issues backend calls of its own. + appending = true; + layout_snapshot = layout; + ns_snapshot = ns; + next = published + 1; + } + + /// `publishAt` below must run with the mutex released: it issues backend calls of its own, and + /// holding the lock across them would either self-deadlock on a re-entrant call or serialize + /// every read-ahead worker behind this one append. + publishAt(*this, *layout_snapshot, ns_snapshot, RefTxnId{1, next}, "ref_" + std::to_string(next), next, DB::UInt128(next)); + + std::lock_guard lock(hook_mutex); published = next; appending = false; return result; } private: + mutable std::mutex hook_mutex; const Layout * layout = nullptr; RootNamespace ns{}; uint64_t published = 0; From 9dfa69c4b2de8c9bc94537b0437b1ba2781ac959 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 11:54:40 +0200 Subject: [PATCH 73/81] cas: every gtest hook handed to a Pool owns its state instead of referencing test-frame locals MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `Pool` can outlive the test body that opened it: a detached task (`Pool::tryDispatchDetached`, `DetachedTaskLease::Completion`) or a background publish holding `shared_from_this()` drops the last `shared_ptr` later, and `~Pool` then runs the mount-lease farewell, which calls `PoolConfig::boot_ms_fn` / `wait_sleep_fn` / `retry_sleep_fn`, the event sink and the other config hooks. A hook that captured a test-frame local by reference read a dead stack slot; ASan caught one site (fixed in a726e933419) and MSan the next (`gtest_cas_ref_snapshot_publish_ordering.cpp`, reported from `~Pool` on a detached thread and attributed to a later test). Sweep of the whole class across the CAS gtests: 92 `_fn = [&` sites, 17 `event_sink`/`*_hook_for_test` sites and 47 setter-installed hooks (`setCasRetrySleepForTest`, `setCasRequestNowFnForTest`, `setEventSink`, `setWaitSleepForTest`, ...). Every hook that reaches a real `Pool` now owns its state: `shared_ptr>` for clocks and counters, a heap-owned `FakeClock`, and two small helpers in `cas_test_helpers.h` (`SharedWaitLog`, `SharedEventLog`: heap-owned, mutex-guarded vectors read through `snapshot()`, since a background renewer or farewell can push from a thread the test never joins). Sites whose receiver provably cannot outlive the frame (the synchronous `claimMountAwaitingExpiry`, a bare `CasRequests` local, `RuntimeUnderTest` whose destructor joins its workers) keep their by-reference captures with a comment saying why. Two real lifetime hazards found on the way are fixed too: a detached publisher's counters in `gtest_cas_detached_work.cpp` and a remount-callback barrier in `gtest_cas_pool.cpp` that an early assertion failure destroyed before the Pool joined the worker. Test-only. Each touched suite ran 5× under ASan; full ASan and release CAS gates green. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Disks/tests/cas_test_helpers.h | 60 ++ src/Disks/tests/gtest_cas_decommission.cpp | 88 ++- src/Disks/tests/gtest_cas_detached_work.cpp | 71 ++- .../tests/gtest_cas_event_dispatcher.cpp | 37 +- src/Disks/tests/gtest_cas_event_log.cpp | 220 ++++--- src/Disks/tests/gtest_cas_gc_ack_floor.cpp | 60 +- src/Disks/tests/gtest_cas_gc_log.cpp | 18 +- src/Disks/tests/gtest_cas_gc_rebuild.cpp | 20 +- src/Disks/tests/gtest_cas_mount.cpp | 52 +- src/Disks/tests/gtest_cas_observability.cpp | 85 ++- .../tests/gtest_cas_part_folder_access.cpp | 28 +- src/Disks/tests/gtest_cas_part_write.cpp | 63 +- src/Disks/tests/gtest_cas_pool.cpp | 574 ++++++++++++------ .../tests/gtest_cas_ref_recovery_cas_walk.cpp | 64 +- ...test_cas_ref_snapshot_publish_ordering.cpp | 39 +- src/Disks/tests/gtest_cas_ref_writer.cpp | 75 ++- .../tests/gtest_cas_retirement_sweep.cpp | 43 +- src/Disks/tests/gtest_cas_writer_duties.cpp | 52 +- 18 files changed, 1162 insertions(+), 487 deletions(-) diff --git a/src/Disks/tests/cas_test_helpers.h b/src/Disks/tests/cas_test_helpers.h index 541479c57434..ae87f129fce3 100644 --- a/src/Disks/tests/cas_test_helpers.h +++ b/src/Disks/tests/cas_test_helpers.h @@ -158,6 +158,66 @@ class ManualBarrier bool released = false; }; +/// Heap-owned wait/sleep log for a `wait_sleep_fn`-shaped test hook: a hook that pushed into a +/// stack-local vector would read (or write) a dead frame if a background completion outlives the test +/// -- a `Pool`'s own detached publish can hold `shared_from_this()` past the test function's return. +/// Mutex-guarded because that background call can race a foreground read. Construct via +/// `std::make_shared` and capture the shared_ptr by value into the hook, never the bare object by +/// reference. +class SharedWaitLog +{ +public: + void push(uint64_t ms) + { + std::lock_guard lock(mutex); + values.push_back(ms); + } + size_t size() const + { + std::lock_guard lock(mutex); + return values.size(); + } + bool empty() const + { + std::lock_guard lock(mutex); + return values.empty(); + } + std::vector snapshot() const + { + std::lock_guard lock(mutex); + return values; + } + +private: + mutable std::mutex mutex; + std::vector values; +}; + +/// Heap-owned event log for an `event_sink`-shaped test hook: a hook that pushed into a stack-local +/// vector would read (or write) a dead frame if a background completion outlives the test -- a `Pool`'s +/// own detached publish can hold `shared_from_this()` past the test function's return, and its farewell +/// or a background renewer can emit events from a thread the test itself never joins. Mutex-guarded +/// because that background call can race a foreground read. Construct via `std::make_shared` and capture +/// the shared_ptr by value into the sink, never the bare object by reference. +class SharedEventLog +{ +public: + void push(CasEvent event) + { + std::lock_guard lock(mutex); + values.push_back(std::move(event)); + } + std::vector snapshot() const + { + std::lock_guard lock(mutex); + return values; + } + +private: + mutable std::mutex mutex; + std::vector values; +}; + /// Bring up the server-wide blob upload pool (stage-1 §1) if it is not already up, so any test that /// drives a `ContentAddressedTransaction` commit -- whose `uploadPendingBlobs` fans out on this pool -- /// finds it initialized. ROBUST (init-if-not-initialized, NOT `call_once`): the raw-lifecycle suite in diff --git a/src/Disks/tests/gtest_cas_decommission.cpp b/src/Disks/tests/gtest_cas_decommission.cpp index 3548af8a7753..d590eaed7a59 100644 --- a/src/Disks/tests/gtest_cas_decommission.cpp +++ b/src/Disks/tests/gtest_cas_decommission.cpp @@ -951,10 +951,23 @@ TEST(CASDecommission, PerObjectFailureWarnsAndContinuesDrain) /// The engine reissues an unresolved delete until its own retry window closes, measured on this /// clock, so the latched fault reaches a genuine give-up with no real time passing. - DB::Cas::tests::FakeClock clock; + /// Heap-owned, not a plain stack local: `decommissionPoolMember` installs this clock into the + /// Pool's `boot_ms_fn` background mount-lease renewer, which can still be running on a detached + /// thread after this function returns, so a by-reference capture of a local would dangle. Wrapped + /// (rather than passing `clock->nowFn()`/`clock->sleepFn()` directly) so the closures stored in + /// `PoolConfig` hold the shared_ptr itself, not just the raw `FakeClock*` those methods capture. + auto clock = std::make_shared(); const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + /*sink=*/{}, /*request_gc_round=*/{}, + [clock] + { + return clock->nowFn()(); + }, + [clock](uint64_t ms) + { + clock->sleepFn()(ms); + }); EXPECT_EQ(report.staging_objects_removed, 1u) << "the OTHER staging object must still be deleted despite the injected failure on its sibling"; @@ -1018,10 +1031,23 @@ TEST(CASDecommission, ManifestDebrisDeleteFailureWarnsAndContinues) /// The engine reissues an unresolved delete until its own retry window closes, measured on this /// clock, so the latched fault reaches a genuine give-up with no real time passing. - DB::Cas::tests::FakeClock clock; + /// Heap-owned, not a plain stack local: `decommissionPoolMember` installs this clock into the + /// Pool's `boot_ms_fn` background mount-lease renewer, which can still be running on a detached + /// thread after this function returns, so a by-reference capture of a local would dangle. Wrapped + /// (rather than passing `clock->nowFn()`/`clock->sleepFn()` directly) so the closures stored in + /// `PoolConfig` hold the shared_ptr itself, not just the raw `FakeClock*` those methods capture. + auto clock = std::make_shared(); const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "admin"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + /*sink=*/{}, /*request_gc_round=*/{}, + [clock] + { + return clock->nowFn()(); + }, + [clock](uint64_t ms) + { + clock->sleepFn()(ms); + }); EXPECT_EQ(report.namespaces_removed, 1u) << "victim/db/t1's namespace erasure (Task 2) is untouched by either injected failure"; @@ -1341,11 +1367,19 @@ TEST(CASDecommission, FailedDrainKeepsSlotThenResumes) /// the way out would refuse against a deadline the retry exhaustion already ran past -- a real /// decommission's background renewer would have kept the deadline current over 90 real seconds, but /// nothing here advances real time to let it. - DB::Cas::tests::FakeClock clock; + auto clock = std::make_shared(); const auto first = decommissionPoolMember( failing, PoolConfig{.pool_prefix = "p", .server_root_id = "a1", .mount_lease_ttl_ms = std::chrono::milliseconds(300'000)}, - "victim", /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + "victim", /*sink=*/{}, /*request_gc_round=*/{}, + [clock] + { + return clock->nowFn()(); + }, + [clock](uint64_t ms) + { + clock->sleepFn()(ms); + }); EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); EXPECT_TRUE((*raw_op).head("p/gc/server-roots/victim/mount", Retry::once()).has_value()) @@ -1385,10 +1419,18 @@ TEST(CASDecommission, ManifestDebrisFailureKeepsSlotThenResumes) /// The engine reissues an unresolved delete until its own retry window closes, measured on this /// clock, so the latched fault reaches a genuine give-up with no real time passing. - DB::Cas::tests::FakeClock clock; + auto clock = std::make_shared(); const auto first = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + /*sink=*/{}, /*request_gc_round=*/{}, + [clock] + { + return clock->nowFn()(); + }, + [clock](uint64_t ms) + { + clock->sleepFn()(ms); + }); EXPECT_FALSE(first.warnings.empty()); EXPECT_FALSE(first.slot_removed); EXPECT_EQ(first.manifest_debris_removed, 0u); @@ -1430,11 +1472,19 @@ TEST(CASDecommission, DrainClockUnifiesWithTheFarewellBootClockRegardlessOfHostU { auto victim = openVictim(backend); } /// identity only -- no namespace, so retirement runs straight /// to the farewell instead of stopping on an unrelated warning - DB::Cas::tests::FakeClock clock; - clock.now = 1'000'000'000'000'000ULL; /// dwarfs any real CLOCK_BOOTTIME on any host + auto clock = std::make_shared(); + clock->now = 1'000'000'000'000'000ULL; /// dwarfs any real CLOCK_BOOTTIME on any host const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + /*sink=*/{}, /*request_gc_round=*/{}, + [clock] + { + return clock->nowFn()(); + }, + [clock](uint64_t ms) + { + clock->sleepFn()(ms); + }); EXPECT_TRUE(report.warnings.empty()); EXPECT_TRUE(report.slot_removed); @@ -1466,18 +1516,26 @@ TEST(CASDecommission, OpeningRetriesPaceOnTheSameFakeClockAndSleepAsTheDrain) const Layout layout("p"); backend->failReadNTimes(layout.ownerKey("victim"), /*times=*/5); - DB::Cas::tests::FakeClock clock; - clock.now = 1'000'000'000'000'000ULL; + auto clock = std::make_shared(); + clock->now = 1'000'000'000'000'000ULL; const auto started = std::chrono::steady_clock::now(); const auto report = decommissionPoolMember( backend, PoolConfig{.pool_prefix = "p", .server_root_id = "a1"}, "victim", - /*sink=*/{}, /*request_gc_round=*/{}, clock.nowFn(), clock.sleepFn()); + /*sink=*/{}, /*request_gc_round=*/{}, + [clock] + { + return clock->nowFn()(); + }, + [clock](uint64_t ms) + { + clock->sleepFn()(ms); + }); const auto wall_elapsed = std::chrono::steady_clock::now() - started; EXPECT_TRUE(report.warnings.empty()); EXPECT_TRUE(report.slot_removed); - EXPECT_FALSE(clock.sleeps.empty()) + EXPECT_FALSE(clock->sleeps.empty()) << "the opening retries must have been paced on the injected clock, not a real sleep"; EXPECT_LT(wall_elapsed, std::chrono::seconds(5)) << "paced on the fake clock, five retries during opening should cost no real wall time at all"; diff --git a/src/Disks/tests/gtest_cas_detached_work.cpp b/src/Disks/tests/gtest_cas_detached_work.cpp index e08a51ea1c00..2b569de9eb82 100644 --- a/src/Disks/tests/gtest_cas_detached_work.cpp +++ b/src/Disks/tests/gtest_cas_detached_work.cpp @@ -632,38 +632,42 @@ TEST(CASDetachedWork, FailedPublisherDispatchKeepsMutationAndClearsReservation) TEST(CASDetachedWork, SettlementSurvivesAThrowingErrorHandler) { auto backend = std::make_shared(); - std::atomic handler_ran{false}; + /// Held in shared, heap-owned atomics, not plain locals: an `ASSERT_*` below can return early and + /// skip the `stopAndDrainDetachedWork` cleanup at the bottom, and even a successful drain there only + /// guarantees no NEW detached task starts -- an already-dispatched one can still be running and can + /// still invoke these hooks against a frame that has already unwound. + auto handler_ran = std::make_shared>(false); PoolConfig config; /// No real backoff wait: the injected throw leaves the tail over-threshold, and a REAL backoff /// sleep here would still be paid at teardown drain even though the test's own assertions never /// wait on it directly. config.snapshot_publish_backoff_initial_ms = 0; config.snapshot_publish_backoff_max_ms = 0; - config.publish_error_hook_for_test = [&handler_ran] + config.publish_error_hook_for_test = [handler_ran] { - handler_ran.store(true); + handler_ran->store(true); throw std::runtime_error("injected: the error handler itself throws"); }; auto store = openPublishingPool(backend, config); const RootNamespace ns{"srv1/handler_throws"}; - std::atomic capture_hook_ran{false}; - std::atomic capture_hook_armed{true}; - store->setSnapshotAfterCaptureHookForTest([&capture_hook_ran, &capture_hook_armed] + auto capture_hook_ran = std::make_shared>(false); + auto capture_hook_armed = std::make_shared>(true); + store->setSnapshotAfterCaptureHookForTest([capture_hook_ran, capture_hook_armed] { - capture_hook_ran.store(true); - if (capture_hook_armed.exchange(false)) + capture_hook_ran->store(true); + if (capture_hook_armed->exchange(false)) throw std::runtime_error("injected: the dispatched attempt itself throws"); }); ASSERT_NO_THROW(publishRef(store, ns, "ref_1", 1)); store->waitForSnapshotPublishSettleForTest(ns); - ASSERT_TRUE(capture_hook_ran.load()) << "the dispatched attempt never reached the injected throw"; - EXPECT_TRUE(handler_ran.load()) << "the injected throw must have reached the (throwing) error handler"; + ASSERT_TRUE(capture_hook_ran->load()) << "the dispatched attempt never reached the injected throw"; + EXPECT_TRUE(handler_ran->load()) << "the injected throw must have reached the (throwing) error handler"; EXPECT_EQ(store->pendingSnapshotPublishesForTest(ns), 0); - /// Same lifetime rule as below: the detached publisher reads the hooks' captured locals, so it is - /// stopped and drained before they go out of scope. + /// Same lifetime rule as below: the detached publisher reads the hooks' captured state, so it is + /// stopped and drained before this function returns. ASSERT_TRUE(store->stopAndDrainDetachedWork(/*deadline_ms=*/10000)); store->setSnapshotAfterCaptureHookForTest(nullptr); } @@ -678,22 +682,33 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) { auto backend = std::make_shared(); constexpr uint64_t step_ms = 100; - std::atomic fake_boot{1000}; - std::atomic error_hook_calls{0}; + /// Held in shared, heap-owned atomics, not plain locals: `stopAndDrainDetachedWork` at the bottom + /// permanently closes admission of NEW detached tasks (via `beginTeardown`), but an `ASSERT_*` above + /// it can return early and skip that call entirely, and even a call that runs only guarantees no new + /// task starts -- an already-dispatched redispatch can still be running (or the drain can simply time + /// out) and can still invoke these hooks against a frame that has already unwound. + auto fake_boot = std::make_shared>(1000); + auto error_hook_calls = std::make_shared>(0); PoolConfig config; - config.boot_ms_fn = [&fake_boot] { return fake_boot.load(); }; + config.boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }; /// Initial == max, so every step of the schedule is the same virtual `step_ms` and the test can /// advance the clock by a constant. config.snapshot_publish_backoff_initial_ms = step_ms; config.snapshot_publish_backoff_max_ms = step_ms; - config.publish_error_hook_for_test = [&error_hook_calls] { error_hook_calls.fetch_add(1); }; + config.publish_error_hook_for_test = [error_hook_calls] + { + error_hook_calls->fetch_add(1); + }; auto store = openPublishingPool(backend, config); const RootNamespace ns{"srv1/throwing_publisher_pacing"}; - std::atomic attempts{0}; - store->setSnapshotAfterCaptureHookForTest([&attempts] + auto attempts = std::make_shared>(0); + store->setSnapshotAfterCaptureHookForTest([attempts] { - attempts.fetch_add(1); + attempts->fetch_add(1); throw std::runtime_error("injected: every publish attempt throws before its write"); }); @@ -702,10 +717,10 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) /// The virtual clock does not move here, so the armed deadline is still in the future for the whole /// window and exactly ONE attempt may have run. const auto observe_until = std::chrono::steady_clock::now() + std::chrono::milliseconds(500); - while (std::chrono::steady_clock::now() < observe_until && attempts.load() <= 1) + while (std::chrono::steady_clock::now() < observe_until && attempts->load() <= 1) std::this_thread::yield(); - EXPECT_EQ(attempts.load(), 1u) << "the throwing attempt redispatched without arming the publish backoff"; - EXPECT_GE(error_hook_calls.load(), 1u) << "the injected throw never reached the error handler"; + EXPECT_EQ(attempts->load(), 1u) << "the throwing attempt redispatched without arming the publish backoff"; + EXPECT_GE(error_hook_calls->load(), 1u) << "the injected throw never reached the error handler"; /// One step of the schedule per iteration: the tail is still over threshold, so each mutation /// re-evaluates admission, and AT MOST one attempt may pass per elapsed backoff interval. At most, @@ -716,7 +731,7 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) /// is what catches it; progress is asserted once after the loop. for (uint64_t step = 1; step <= 3; ++step) { - fake_boot.fetch_add(step_ms); + fake_boot->fetch_add(step_ms); ASSERT_NO_THROW(publishRef(store, ns, "ref_" + std::to_string(step + 1), step + 1)); /// Bounded poll rather than `waitForSnapshotPublishSettleForTest`: that call waits on a condvar /// predicate with no deadline, and on an unpaced-redispatch regression the reservation count @@ -730,14 +745,14 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) << "pending_snapshot_publishes stayed nonzero"; std::this_thread::yield(); } - EXPECT_LE(attempts.load(), 1 + step) << "more than one publish attempt ran within one backoff step"; + EXPECT_LE(attempts->load(), 1 + step) << "more than one publish attempt ran within one backoff step"; } /// Progress, on the injected clock so it is deterministic rather than a race with a worker: an /// elapsed backoff must eventually admit a further attempt, or the pacing gate would be a wedge. - for (uint64_t extra = 0; attempts.load() < 2 && extra < 20; ++extra) + for (uint64_t extra = 0; attempts->load() < 2 && extra < 20; ++extra) { - fake_boot.fetch_add(step_ms); + fake_boot->fetch_add(step_ms); ASSERT_NO_THROW(publishRef(store, ns, "ref_progress_" + std::to_string(extra), 100 + extra)); const auto settle = std::chrono::steady_clock::now() + std::chrono::seconds(10); while (store->pendingSnapshotPublishesForTest(ns) != 0) @@ -746,10 +761,10 @@ TEST(CASDetachedWork, ThrowingPublishAttemptIsPacedByTheBackoff) std::this_thread::yield(); } } - EXPECT_GE(attempts.load(), 2u) + EXPECT_GE(attempts->load(), 2u) << "no elapsed backoff ever admitted a further publish attempt: the gate is a wedge, not a pace"; - EXPECT_EQ(error_hook_calls.load(), attempts.load()); + EXPECT_EQ(error_hook_calls->load(), attempts->load()); /// The publisher is detached work: a redispatch admitted by the last elapsed backoff can still be /// running when this body returns, and it reads `fake_boot` through `boot_ms_fn`. Stop and drain it /// while the locals it reads are alive. diff --git a/src/Disks/tests/gtest_cas_event_dispatcher.cpp b/src/Disks/tests/gtest_cas_event_dispatcher.cpp index 325c70de59f4..424194bfa22d 100644 --- a/src/Disks/tests/gtest_cas_event_dispatcher.cpp +++ b/src/Disks/tests/gtest_cas_event_dispatcher.cpp @@ -125,34 +125,40 @@ TEST(CASEventDispatcher, ReentrantSinkDoesNotDeadlock) TEST(CASEventDispatcher, LedgerEmissionOutsideLocks) { auto b = std::make_shared(); - std::vector seen; /// declared before the Pool so it outlives any late background emit - std::mutex seen_mutex; + /// Heap-owned, not plain locals: `seen`'s own declaration-before-the-Pool comment protects only + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); const RootNamespace ns{"srv1/tbl"}; const String ref = "all_0_0_0"; publishOneBlobPart(s, ns.string(), ref, "the-resolvable-payload"); - std::atomic reentered{false}; - s->setEventSink([&](CasEvent e) + auto reentered = std::make_shared>(false); + /// `s` is captured as a raw pointer (`s.get()`), not by reference and not by `shared_ptr`: a + /// `shared_ptr` capture here would make the Pool's own `event_sink` hold a permanent reference to + /// its owning Pool, a cycle that leaks it; a by-reference capture of the local `s` would dangle once + /// this frame returns. Validity is the same invariant every self-referencing hook in the production + /// code relies on (e.g. `CasPool.cpp`'s `[s = store.get()]`): the hook can only run while some other + /// `shared_ptr` keeps the Pool alive. + Pool * const s_ptr = s.get(); + s->setEventSink([seen, reentered, s_ptr, ns, ref](CasEvent e) { - { - std::lock_guard g(seen_mutex); - seen.push_back(e); - } + seen->push(e); /// Re-enter a ledger read that takes `state_mutex`, exactly once (`Deferred` => this read /// itself emits nothing, so there is no unbounded emit recursion). Under the pre-fix code the /// outer `resolveRef` still holds `state_mutex` here, so this call self-deadlocks. - if (e.type == CasEventType::RefResolve && !reentered.exchange(true)) - (void)s->resolveRef(ns, ref, false, ResolveAudit::Deferred); + if (e.type == CasEventType::RefResolve && !reentered->exchange(true)) + (void)s_ptr->resolveRef(ns, ref, false, ResolveAudit::Deferred); }); - std::promise resolve_done; - auto resolve_future = resolve_done.get_future(); + auto resolve_done = std::make_shared>(); + auto resolve_future = resolve_done->get_future(); std::thread resolver([&] { (void)s->resolveRef(ns, ref); /// ResolveAudit::Emit (default) -> emits RefResolve -> drives the sink - resolve_done.set_value(); + resolve_done->set_value(); }); /// A second thread emits upload-task-style events concurrently with the resolve, so the dispatcher's @@ -178,11 +184,10 @@ TEST(CASEventDispatcher, LedgerEmissionOutsideLocks) resolver.join(); uploader.join(); - EXPECT_TRUE(reentered.load()) << "the reentrant ledger read must have run"; - std::lock_guard g(seen_mutex); + EXPECT_TRUE(reentered->load()) << "the reentrant ledger read must have run"; size_t resolves = 0; size_t uploads = 0; - for (const auto & e : seen) + for (const auto & e : seen->snapshot()) { if (e.type == CasEventType::RefResolve) ++resolves; diff --git a/src/Disks/tests/gtest_cas_event_log.cpp b/src/Disks/tests/gtest_cas_event_log.cpp index fea227a73bc8..e55f775ef4f8 100644 --- a/src/Disks/tests/gtest_cas_event_log.cpp +++ b/src/Disks/tests/gtest_cas_event_log.cpp @@ -130,9 +130,12 @@ CasRequestBudget renewalEventBudget() }; } +/// `boot_ms` is a shared, heap-owned atomic, not a plain reference parameter: some callers mutate it +/// after the Pool exists, and the Pool can outlive this function's own call (a background publish +/// holds `shared_from_this()`), so a by-reference capture of a caller-local would dangle. PoolPtr openRenewalEventPool( const std::shared_ptr & backend, - uint64_t & boot_ms, + const std::shared_ptr> & boot_ms, CasRequestBudget budget = renewalEventBudget(), String prefix = "renewal-events", String server_root_id = "test") @@ -145,7 +148,10 @@ PoolPtr openRenewalEventPool( .server_root_id = std::move(server_root_id), .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .cas_request_budget = budget, - .boot_ms_fn = [&] { return boot_ms; }, + .boot_ms_fn = [boot_ms] + { + return boot_ms->load(); + }, }); } @@ -194,47 +200,63 @@ TEST(CASEvent, ConstructAndCopyAndName) TEST(CASEvent, PoolEmitsToSink) { auto b = std::make_shared(); - std::vector seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); - s->setEventSink([&](const CasEvent & e){ seen.push_back(e); }); + s->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); CasEvent e; e.type = CasEventType::BlobPut; e.object_hash = "h"; s->emitEvent(std::move(e)); - ASSERT_EQ(seen.size(), 1u); - EXPECT_EQ(seen[0].type, CasEventType::BlobPut); + ASSERT_EQ(seen->snapshot().size(), 1u); + EXPECT_EQ(seen->snapshot()[0].type, CasEventType::BlobPut); /// null sink => no-op (no crash, no row); a fresh event, not the one already moved above. s->setEventSink(nullptr); CasEvent e2; e2.type = CasEventType::BlobPut; s->emitEvent(std::move(e2)); - EXPECT_EQ(seen.size(), 1u); + EXPECT_EQ(seen->snapshot().size(), 1u); } TEST(CASEvent, FirstAttemptRenewalIsSilent) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; + auto boot_ms = std::make_shared>(100); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto store = openRenewalEventPool(backend, boot_ms); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); + store->setEventSink([events](CasEvent event) + { + events->push(std::move(event)); + }); EXPECT_NO_THROW(store->renewWatermarkOnce()); - EXPECT_TRUE(watermarkRenewEvents(events).empty()); + EXPECT_TRUE(watermarkRenewEvents(events->snapshot()).empty()); } TEST(CASEvent, WatermarkRenewEventsAreBoundedAndComplete) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; + auto boot_ms = std::make_shared>(100); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto store = openRenewalEventPool(backend, boot_ms); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); + store->setEventSink([events](CasEvent event) + { + events->push(std::move(event)); + }); backend->throw_before_next_write = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); - const std::vector renewals = watermarkRenewEvents(events); + const std::vector renewals = watermarkRenewEvents(events->snapshot()); /// ONE event per logical renewal, whatever the physical attempts cost: the engine owns its own /// reissues, and the terminal event carries their count rather than announcing each one. ASSERT_EQ(renewals.size(), 1u); @@ -270,22 +292,30 @@ TEST(CASEvent, WatermarkRenewEventsAreBoundedAndComplete) TEST(CASEvent, AnAmbiguityPastTheLeaseBoundNeverStartsTheResolvingRead) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; + auto boot_ms = std::make_shared>(100); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto store = openRenewalEventPool( backend, boot_ms, renewalEventBudget(), "renewal-inflight-ambiguity"); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); + store->setEventSink([events](CasEvent event) + { + events->push(std::move(event)); + }); /// The lease was anchored at 100 with a 1000 ms TTL, so the fence expires at 1100 and holds a 20 ms /// safety margin. At 1081 only 19 ms remain, and admission refuses the resolve read. - backend->before_throw = [&] { boot_ms = 1'081; }; + backend->before_throw = [boot_ms] + { + boot_ms->store(1'081); + }; backend->throw_before_next_write = true; backend->armResolveProbe(); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); EXPECT_FALSE(backend->resolveStarted()) << "an attempt that consumed the lease must not start the resolving read"; - const std::vector renewals = watermarkRenewEvents(events); + const std::vector renewals = watermarkRenewEvents(events->snapshot()); ASSERT_EQ(renewals.size(), 1u); EXPECT_EQ(renewals[0].outcome, "failed"); EXPECT_EQ(renewals[0].detail.at("attempts_sent"), "1"); @@ -385,7 +415,7 @@ TEST(CASEvent, DeepReentrancyPreservesDeterministicPhysicalAttemptTruth) TEST(CASEvent, WatermarkRenewSinkFailureCannotChangeOutcome) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; + auto boot_ms = std::make_shared>(100); auto store = openRenewalEventPool(backend, boot_ms); const String mount_key = store->layout().mountKey("test"); const uint64_t seq_before = decodeMountLease(backend->readForTest(mount_key)->bytes).seq; @@ -421,14 +451,19 @@ TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; + auto boot_ms = std::make_shared>(100); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish + /// holds `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto store = openRenewalEventPool(backend, boot_ms, renewalEventBudget(), "renewal-deterministic-details"); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); + store->setEventSink([events](CasEvent event) + { + events->push(std::move(event)); + }); backend->throw_nonretryable_next_write = true; EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); - const std::optional failed = one_failed_event(events); + const std::optional failed = one_failed_event(events->snapshot()); ASSERT_TRUE(failed.has_value()) << "the store's refusal must reach the event log"; /// A deterministic failure reaches the renewer as the exception the engine refuses to reissue, /// and an exception carries no attempt count -- so the classification is all this ending states. @@ -437,16 +472,21 @@ TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; + auto boot_ms = std::make_shared>(100); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish + /// holds `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto store = openRenewalEventPool(backend, boot_ms, renewalEventBudget(), "renewal-deadline-details"); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); + store->setEventSink([events](CasEvent event) + { + events->push(std::move(event)); + }); /// The lease was anchored at 100 with a 1000 ms TTL and holds a 20 ms safety margin, so 1090 /// leaves 10 ms of it and admission refuses the renewal before its first attempt. - boot_ms = 1090; + boot_ms->store(1090); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); - const std::optional failed = one_failed_event(events); + const std::optional failed = one_failed_event(events->snapshot()); ASSERT_TRUE(failed.has_value()) << "the refused admission must reach the event log"; EXPECT_EQ(failed->detail.at("attempts_sent"), "0"); EXPECT_EQ(failed->detail.at("classification"), "external_lease_deadline"); @@ -456,30 +496,40 @@ TEST(CASEvent, TerminalRenewalDetailsPreservePhysicalTruthAndClassification) TEST(CASEvent, ReentrantRenewalSinkPreservesOuterObservationIdentity) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; - std::vector events; + auto boot_ms = std::make_shared>(100); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); PoolPtr store = openRenewalEventPool(backend, boot_ms, renewalEventBudget(), "renewal-reentrant-sink"); - bool reentered = false; - store->setEventSink([&](CasEvent event) + auto reentered = std::make_shared>(false); + /// `store` is captured as a raw pointer (`store.get()`), not by reference and not by `shared_ptr`: a + /// `shared_ptr` capture here would make the Pool's own `event_sink` hold a permanent reference to + /// its owning Pool, a cycle that leaks it; a by-reference capture of the local `store` would dangle + /// once this frame returns. Validity is the same invariant every self-referencing hook in the + /// production code relies on (e.g. `CasPool.cpp`'s `[s = store.get()]`): the hook can only run while + /// some other `shared_ptr` keeps the Pool alive. + Pool * const store_ptr = store.get(); + store->setEventSink([events, reentered, store_ptr](CasEvent event) { if (event.type != CasEventType::WatermarkRenew) return; - events.push_back(event); - if (event.outcome == "recovered" && !std::exchange(reentered, true)) - store->renewWatermarkOnce(); + events->push(event); + if (event.outcome == "recovered" && !reentered->exchange(true)) + store_ptr->renewWatermarkOnce(); }); backend->throw_before_next_write = true; EXPECT_NO_THROW(store->renewWatermarkOnce()); - ASSERT_TRUE(reentered); + ASSERT_TRUE(reentered->load()); /// The nested renewal commits on its first attempt, which is silent, so the outer recovery is the /// only event -- and it still names the outer renewal's own seq while the durable lease has already /// moved past it. An observation the nested call reused would report seq 3 here. - ASSERT_EQ(events.size(), 1u); - EXPECT_EQ(events[0].outcome, "recovered"); - EXPECT_EQ(events[0].detail.at("attempts_sent"), "2"); - EXPECT_EQ(events[0].detail.at("seq"), "2"); + const std::vector observed_events = events->snapshot(); + ASSERT_EQ(observed_events.size(), 1u); + EXPECT_EQ(observed_events[0].outcome, "recovered"); + EXPECT_EQ(observed_events[0].detail.at("attempts_sent"), "2"); + EXPECT_EQ(observed_events[0].detail.at("seq"), "2"); EXPECT_EQ(decodeMountLease(backend->readForTest(store->layout().mountKey("test"))->bytes).seq, 3u) << "the nested first-attempt success must run without replacing the outer observation"; } @@ -487,20 +537,24 @@ TEST(CASEvent, ReentrantRenewalSinkPreservesOuterObservationIdentity) TEST(CASEvent, PreCompletionConflictReentrancyPreservesOuterTerminalObservation) { auto inner_backend = std::make_shared(); - uint64_t inner_boot_ms = 100; + auto inner_boot_ms = std::make_shared>(100); auto inner = openRenewalEventPool( inner_backend, inner_boot_ms, renewalEventBudget(), "renewal-reentrant-inner", "inner"); auto outer_backend = std::make_shared(); - uint64_t outer_boot_ms = 100; + auto outer_boot_ms = std::make_shared>(100); auto outer = openRenewalEventPool( outer_backend, outer_boot_ms, renewalEventBudget(), "renewal-reentrant-outer", "outer"); - std::vector outer_events; - bool reentered = false; - outer->setEventSink([&](CasEvent event) + /// Heap-owned, not plain locals: `outer`'s `event_sink` mutates them, and the Pool can outlive this + /// stack frame (a background publish holds `shared_from_this()`), so a by-reference capture of a + /// local would dangle. `inner` (a DIFFERENT Pool from `outer`) is captured by value -- a `shared_ptr` + /// copy here is not a self-reference cycle, unlike capturing `outer` into its own sink would be. + auto outer_events = std::make_shared(); + auto reentered = std::make_shared>(false); + outer->setEventSink([outer_events, reentered, inner](CasEvent event) { - outer_events.push_back(event); - if (event.type == CasEventType::MountConflict && !std::exchange(reentered, true)) + outer_events->push(event); + if (event.type == CasEventType::MountConflict && !reentered->exchange(true)) inner->renewWatermarkOnce(); }); @@ -510,8 +564,8 @@ TEST(CASEvent, PreCompletionConflictReentrancyPreservesOuterTerminalObservation) outer_backend->vanish_on_next_write = true; EXPECT_THROW(outer->renewWatermarkOnce(), DB::Exception); - ASSERT_TRUE(reentered); - const std::vector renewals = watermarkRenewEvents(outer_events); + ASSERT_TRUE(reentered->load()); + const std::vector renewals = watermarkRenewEvents(outer_events->snapshot()); ASSERT_EQ(renewals.size(), 1u); EXPECT_EQ(renewals[0].outcome, "failed"); EXPECT_EQ(renewals[0].detail.at("server_root_id"), "outer"); @@ -525,21 +579,33 @@ TEST(CASEvent, PreCompletionConflictReentrancyPreservesOuterTerminalObservation) TEST(CASEvent, EmitEventMovesSourceIntoSink) { auto b = std::make_shared(); - String captured_reason; - std::map captured_detail; + /// Heap-owned, mutex-guarded, not plain locals: the Pool can outlive this stack frame (a background + /// publish holds `shared_from_this()`), so a by-reference capture of a local would dangle, and a + /// background emit could race the foreground read below. + struct Captured + { + std::mutex mutex; + String reason; + std::map detail; + }; + auto captured = std::make_shared(); auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); - s->setEventSink([&](CasEvent ev) + s->setEventSink([captured](CasEvent ev) { - captured_reason = std::move(ev.reason); - captured_detail = std::move(ev.detail); + std::lock_guard lock(captured->mutex); + captured->reason = std::move(ev.reason); + captured->detail = std::move(ev.detail); }); CasEvent e; e.type = CasEventType::BlobPut; e.reason = "sentinel-reason"; e.detail["k"] = "v"; s->emitEvent(std::move(e)); - EXPECT_EQ(captured_reason, "sentinel-reason"); - EXPECT_EQ(captured_detail.at("k"), "v"); + { + std::lock_guard lock(captured->mutex); + EXPECT_EQ(captured->reason, "sentinel-reason"); + EXPECT_EQ(captured->detail.at("k"), "v"); + } /// the source event must be MOVED-FROM after emit, not merely aliased/copied through -- reading /// `e` here is the whole point of the test, not an oversight. EXPECT_TRUE(e.reason.empty()); // NOLINT(bugprone-use-after-move, hicpp-invalid-access-moved) @@ -614,18 +680,17 @@ bool hasType(const std::vector & events, CasEventType t) TEST(CASEvent, LifecycleReconstructionFromRows) { auto b = std::make_shared(); - /// Declared BEFORE the Pool so they OUTLIVE it: the Pool's background retired-view syncer can emit - /// (e.g. a view-advance event) right up to the Pool's destructor, and a sink capturing locals that - /// die first is a use-after-scope (found by ASan 2026-07-09; the production sink captures the Context - /// shared_ptr by value and is immune). - std::vector events; - std::mutex events_mutex; + /// Heap-owned, not a plain local: the Pool's background retired-view syncer can emit (e.g. a + /// view-advance event) right up to the Pool's destructor, and a background publish can hold an + /// extra `shared_from_this()` past this frame's return regardless of declaration order relative to + /// the Pool (found by ASan 2026-07-09; the production sink captures the Context shared_ptr by value + /// and is immune) -- a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto s = Pool::open(b, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); - s->setEventSink([&](const CasEvent & e) + s->setEventSink([events](const CasEvent & e) { - std::lock_guard lock(events_mutex); - events.push_back(e); + events->push(e); }); const RootNamespace ns{"srv1/tbl"}; @@ -651,27 +716,28 @@ TEST(CASEvent, LifecycleReconstructionFromRows) } /// (a) the expected taxonomy was emitted across the lifecycle (manifest model: no standalone trees). - EXPECT_TRUE(hasType(events, CasEventType::BlobPut)); - EXPECT_TRUE(hasType(events, CasEventType::RootAdd)) + const std::vector observed_events = events->snapshot(); + EXPECT_TRUE(hasType(observed_events, CasEventType::BlobPut)); + EXPECT_TRUE(hasType(observed_events, CasEventType::RootAdd)) << "a fold must have recorded the manifest owner's blob edge (+1)"; - EXPECT_TRUE(hasType(events, CasEventType::RefDrop)); - EXPECT_TRUE(hasType(events, CasEventType::IndegZero)); - EXPECT_TRUE(hasType(events, CasEventType::GcRetireObserve) - || hasType(events, CasEventType::GcRetireDecision) - || hasType(events, CasEventType::GcRecheckVerdict)) + EXPECT_TRUE(hasType(observed_events, CasEventType::RefDrop)); + EXPECT_TRUE(hasType(observed_events, CasEventType::IndegZero)); + EXPECT_TRUE(hasType(observed_events, CasEventType::GcRetireObserve) + || hasType(observed_events, CasEventType::GcRetireDecision) + || hasType(observed_events, CasEventType::GcRecheckVerdict)) << "a GC retire/recheck transition must be recorded"; - EXPECT_TRUE(hasType(events, CasEventType::BlobDelete) || hasType(events, CasEventType::ManifestDelete)) + EXPECT_TRUE(hasType(observed_events, CasEventType::BlobDelete) || hasType(observed_events, CasEventType::ManifestDelete)) << "the single content-delete site must emit a delete row"; /// (b) completeness mandate: every emitted event has a non-empty reason (the human WHY). - for (const auto & e : events) + for (const auto & e : observed_events) EXPECT_FALSE(e.reason.empty()) << "event " << toString(e.type) << " (" << e.object_hash << ") has an empty reason"; /// (c) lifecycle reconstruction: filtering by the deleted blob's object_hash yields, in time /// order, at least its in-degree-zero -> retire-observe -> delete chain — its whole story. std::vector chain; - for (const auto & e : events) + for (const auto & e : observed_events) if (e.object_hash == blob_hash) chain.push_back(e.type); diff --git a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp index 10b8ae92b3f1..ea63268a8fe3 100644 --- a/src/Disks/tests/gtest_cas_gc_ack_floor.cpp +++ b/src/Disks/tests/gtest_cas_gc_ack_floor.cpp @@ -898,9 +898,10 @@ namespace /// two-round scenario described below. Parameterized only by `config` so the same scenario can be run /// against the default `PoolConfig` (`ExpiredMountFencedOutAndExcluded`) and against /// `unsafe_remount_no_delay = true` (`CASGcFenceOut.ThresholdUnchangedByUnsafeKnob`), proving the knob -/// changes nothing about the fence-out threshold or its round count. `backend` and `events` are declared -/// BEFORE the Pool inside this same function so they outlive the background syncer's emits (ASan -/// 2026-07-09) -- the Pool must never outlive the function that opened it. +/// changes nothing about the fence-out threshold or its round count. `events` is heap-owned (not a +/// plain local) because the Pool CAN outlive the function that opened it: a background publish can hold +/// an extra `shared_from_this()` past this function's return, so a stack-local sink target -- even one +/// declared before the Pool (the fix for the 2026-07-09 ASan finding) -- is not enough. /// /// A dead mount is fenced out by the round's heartbeat step: gc_fenced is set on its body (a /// token-guarded rewrite that bumps seq). The fence is pure liveness (re-arms the write fence so a @@ -919,7 +920,10 @@ namespace void runExpiredMountFenceOutScenario(const PoolConfig & config) { auto backend = std::make_shared(); - std::vector events; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto events = std::make_shared(); auto store = Pool::open(backend, config); const Layout & layout = store->layout(); @@ -947,7 +951,10 @@ void runExpiredMountFenceOutScenario(const PoolConfig & config) Gc gc(store, kGc, [&] { return gc_now; }, [&] { return gc_mono; }); // Capture the emitted events so we can assert the round emits exactly one GcFenceOut row for srid2. - store->setEventSink([&](const CasEvent & e) { events.push_back(e); }); + store->setEventSink([events](const CasEvent & e) + { + events->push(e); + }); const RootNamespace ns{"00/aa@cas@"}; const ManifestRef r = ref("srv-a:1", 1, 0xAA); @@ -984,7 +991,7 @@ void runExpiredMountFenceOutScenario(const PoolConfig & config) // Exactly one GcFenceOut audit row was emitted, naming srid2 in its detail. size_t fence_out_rows = 0; - for (const CasEvent & e : events) + for (const CasEvent & e : events->snapshot()) if (e.type == CasEventType::GcFenceOut) { ++fence_out_rows; @@ -1040,9 +1047,15 @@ TEST(CASGcFenceOut, ThresholdUnchangedByUnsafeKnob) TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) { auto backend = std::make_shared(); - uint64_t fake_boot = 0; + /// Held in a shared atomic, not a plain local: this test mutates the clock below, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto fake_boot = std::make_shared>(0); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test", - .boot_ms_fn = [&] { return fake_boot; }}); + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }}); const Layout & layout = store->layout(); // A stale mount, exactly as `ExpiredMountFencedOutAndExcluded`: one claim, never renewed again. @@ -1050,7 +1063,11 @@ TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) CasRequests renewer_requests = openRequestsForTest(backend); MountLeaseRenewer srid2_renewer(renewer_requests, renewer_requests, layout, srid2, DB::UInt128(0x2222), /*writer_epoch=*/1, - std::chrono::milliseconds(100), [] { return 1000u; }, [&] { return fake_boot; }); + std::chrono::milliseconds(100), [] { return 1000u; }, + [fake_boot] + { + return fake_boot->load(); + }); srid2_renewer.start(); ASSERT_FALSE(decodeMountLease(readObj(*backend, layout.mountKey(srid2))->bytes).gc_fenced); @@ -1065,7 +1082,7 @@ TEST(CASGCAckFloor, DefaultMonoClockTracksPoolsInjectedBootClockNotWallClock) EXPECT_EQ(rep1.fence_outs, 0u); store->renewWatermarkOnce(); - fake_boot = threshold_ms; // advance the FAKE clock only; this test runs in well under a millisecond + fake_boot->store(threshold_ms); // advance the FAKE clock only; this test runs in well under a millisecond const RoundReport rep2 = gc.runRegularRound(); EXPECT_EQ(rep2.fence_outs, 1u) @@ -1283,13 +1300,20 @@ TEST(CASGCCondemnMarker, SwallowedMarkerWriteCarriesEntryInsteadOfDeleting) /// millisecond, because full-jitter backoff may draw zero and a clock that never moves never closes /// the window. Scoped to the GC plane, which is where `writeCondemnedMeta` runs, so the mount /// plane's lease-bound policies keep their real clock. - std::atomic engine_now_ms{0}; - std::atomic engine_sleeps{0}; - store->openRequests().setNowFnForTest([&] { return engine_now_ms.load(); }); - store->openRequests().setSleepFnForTest([&](uint64_t pause_ms) + /// Held in shared, heap-owned atomics, not plain locals: `store->openRequests()` is the Pool's own + /// persistent engine, and the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local -- even an already-atomic one -- + /// would dangle once the frame returns. + auto engine_now_ms = std::make_shared>(0); + auto engine_sleeps = std::make_shared>(0); + store->openRequests().setNowFnForTest([engine_now_ms] + { + return engine_now_ms->load(); + }); + store->openRequests().setSleepFnForTest([engine_now_ms, engine_sleeps](uint64_t pause_ms) { - engine_sleeps.fetch_add(1); - engine_now_ms.fetch_add(pause_ms + 1); + engine_sleeps->fetch_add(1); + engine_now_ms->fetch_add(pause_ms + 1); }); const RootNamespace ns{"00/aa@cas@"}; @@ -1310,9 +1334,9 @@ TEST(CASGCCondemnMarker, SwallowedMarkerWriteCarriesEntryInsteadOfDeleting) /// seconds, so it cannot happen before the clock has passed `Retry::standard()`'s window minus /// that draw. Both assertions pin the REISSUING, which is what the ambiguous kind buys; neither /// can tell the injected clock from the real one -- that seam bounds the reissuing in real time. - EXPECT_GT(engine_sleeps.load(), 1u) + EXPECT_GT(engine_sleeps->load(), 1u) << "an ambiguous marker write must be resolved and reissued, not surfaced on its first attempt"; - EXPECT_GT(engine_now_ms.load(), 85'000u) + EXPECT_GT(engine_now_ms->load(), 85'000u) << "the marker write must have spent its whole retry window before reporting failure"; ASSERT_TRUE(currentEntryFor(*backend, store->layout(), blob).has_value()) << "precondition: the retired entry must have been committed despite the lost marker"; diff --git a/src/Disks/tests/gtest_cas_gc_log.cpp b/src/Disks/tests/gtest_cas_gc_log.cpp index b0fd72cb5869..d70cc9aa78d8 100644 --- a/src/Disks/tests/gtest_cas_gc_log.cpp +++ b/src/Disks/tests/gtest_cas_gc_log.cpp @@ -334,11 +334,13 @@ TEST(CASGCLog, TransientThrowIsClassifiedAborted) { auto backend = std::make_shared(); /// A PERSISTENT transient fault is reissued for the whole retry window, so the window has to run - /// on a clock this test advances -- otherwise one read spends ninety real seconds. Declared BEFORE - /// the store: the store's teardown still calls the now-function, so the clock must outlive it. - std::atomic engine_now_ms{0}; + /// on a clock this test advances -- otherwise one read spends ninety real seconds. Heap-owned, not + /// a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local -- even an already-atomic one -- + /// would dangle once the frame returns. + auto engine_now_ms = std::make_shared>(0); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); - store->setCasRequestNowFnForTest([&] { return engine_now_ms.fetch_add(10'000) + 10'000; }); + store->setCasRequestNowFnForTest([engine_now_ms] { return engine_now_ms->fetch_add(10'000) + 10'000; }); store->setCasRetrySleepForTest([](uint64_t) {}); std::vector rows; @@ -477,10 +479,12 @@ TEST(CASGCScheduler, TransientRoundFailureKeepsLeadershipAndHeartbeat) { auto backend = std::make_shared(); /// See `TransientThrowIsClassifiedAborted`: the transient mode is persistent while it is armed, so - /// the retry window runs on a clock this test advances, declared before the store it outlives. - std::atomic engine_now_ms{0}; + /// the retry window runs on a clock this test advances. Heap-owned, not a plain local: the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local -- even an already-atomic one -- would dangle once the frame returns. + auto engine_now_ms = std::make_shared>(0); auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); - store->setCasRequestNowFnForTest([&] { return engine_now_ms.fetch_add(10'000) + 10'000; }); + store->setCasRequestNowFnForTest([engine_now_ms] { return engine_now_ms->fetch_add(10'000) + 10'000; }); store->setCasRetrySleepForTest([](uint64_t) {}); std::mutex rows_mutex; diff --git a/src/Disks/tests/gtest_cas_gc_rebuild.cpp b/src/Disks/tests/gtest_cas_gc_rebuild.cpp index 81d1505dc4cd..cbfee9d0d5e5 100644 --- a/src/Disks/tests/gtest_cas_gc_rebuild.cpp +++ b/src/Disks/tests/gtest_cas_gc_rebuild.cpp @@ -651,7 +651,10 @@ TEST(CASGCRebuild, LeaseConflictRefuses) TEST(CASGCClampSuppression, LandedEdgeBehindClampNeverDeleted) { auto backend = std::make_shared(); - std::vector seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto store = openPoolForTest(backend); const RootNamespace ns{"00/aa@cas@"}; @@ -679,7 +682,11 @@ TEST(CASGCClampSuppression, LandedEdgeBehindClampNeverDeleted) /// Rounds with acks current: X reaches folded in-degree 0 and is condemned, but every pass is /// CLAMPED (the bodiless precommit persists), so nothing may graduate or delete. /// Observability (2026-07-03): every clamp emits a gc_fold_clamp event with the reason. - store->setEventSink([&](const CasEvent & e){ if (e.type == CasEventType::GcFoldClamp) seen.push_back(e); }); + store->setEventSink([seen](const CasEvent & e) + { + if (e.type == CasEventType::GcFoldClamp) + seen->push(e); + }); const String blob_key = store->layout().blobKey(BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(DB::UInt128(1))}); for (int i = 0; i < 6; ++i) { @@ -689,13 +696,14 @@ TEST(CASGCClampSuppression, LandedEdgeBehindClampNeverDeleted) << "round " << i << ": X was deleted while its landed +1 sat unfolded behind the clamp"; } - ASSERT_FALSE(seen.empty()) << "each clamped pass must emit a gc_fold_clamp event"; - EXPECT_NE(seen.front().reason.find("fold barrier"), String::npos); + const std::vector observed_events = seen->snapshot(); + ASSERT_FALSE(observed_events.empty()) << "each clamped pass must emit a gc_fold_clamp event"; + EXPECT_NE(observed_events.front().reason.find("fold barrier"), String::npos); /// Snapshot+log ref model: the clamp is per-table (one ref-log stream per namespace, no ref shards), /// so the event names the clamped `log` and the `resolved_through` cursor rather than a shard number. - EXPECT_TRUE(seen.front().detail.contains("log")) + EXPECT_TRUE(observed_events.front().detail.contains("log")) << "clamp event must name the clamped log id"; - EXPECT_TRUE(seen.front().detail.contains("resolved_through")) + EXPECT_TRUE(observed_events.front().detail.contains("resolved_through")) << "clamp event must name the cursor it resolved through"; store->setEventSink(nullptr); diff --git a/src/Disks/tests/gtest_cas_mount.cpp b/src/Disks/tests/gtest_cas_mount.cpp index 31865606e788..a9472994237f 100644 --- a/src/Disks/tests/gtest_cas_mount.cpp +++ b/src/Disks/tests/gtest_cas_mount.cpp @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -309,7 +310,11 @@ CasRequestBudget renewalLogBudget() /// attempts. No replacement test is needed: there is no per-attempt log call left to race. TEST(CASMountAudit, RenewalDefaultLogsAreBounded) { - const auto open_store = [](const std::shared_ptr & backend, uint64_t & boot_ms, const String & prefix) + /// `boot_ms` is a shared, heap-owned atomic, not a plain reference parameter: the last block below + /// mutates it after the Pool exists, and the Pool can outlive this lambda's own call (a background + /// publish holds `shared_from_this()`), so a by-reference capture of a caller-local would dangle. + const auto open_store = [](const std::shared_ptr & backend, + const std::shared_ptr> & boot_ms, const String & prefix) { /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the /// budget field alone; pair the two so the fence math below matches what admits. @@ -319,13 +324,16 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .cas_request_budget = renewalLogBudget(), - .boot_ms_fn = [&] { return boot_ms; }, + .boot_ms_fn = [boot_ms] + { + return boot_ms->load(); + }, }); }; { auto backend = std::make_shared(); - uint64_t boot_ms = 100; + auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-silent"); ScopedRenewalLogCapture capture("information"); EXPECT_NO_THROW(store->renewWatermarkOnce()); @@ -334,7 +342,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; + auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-recovered"); ScopedRenewalLogCapture capture("information"); backend->throw_before_next_overwrite = true; @@ -347,7 +355,7 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; + auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-debug"); ScopedRenewalLogCapture capture("debug"); backend->throw_before_next_overwrite = true; @@ -357,13 +365,13 @@ TEST(CASMountAudit, RenewalDefaultLogsAreBounded) { auto backend = std::make_shared(); - uint64_t boot_ms = 100; + auto boot_ms = std::make_shared>(100); auto store = open_store(backend, boot_ms, "renewal-log-fenced"); ScopedRenewalLogCapture capture("information"); /// The lease was claimed at boot 100 with the 1000 ms TTL above, so it expires at 1100. The /// fence admits only while the remaining time strictly clears the safety margin plus whatever /// the attempt reserves, so exactly `margin` remaining (with the reservation on top) refuses. - boot_ms = 1100 - renewalLogBudget().lease_safety_margin_ms; + boot_ms->store(1100 - renewalLogBudget().lease_safety_margin_ms); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); const String output = capture.captured(); EXPECT_EQ(countRenewalLogText(output, "CAS mount renewal"), 1u) << output; @@ -1385,7 +1393,10 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) /// fake `boot_ms_fn` + `wait_sleep_fn` (mirroring /// `CASMountOpenWaits.UncleanOpenPaysOnlyTheObservationWindow`) so the observation window resolves /// instantly instead of blocking this test on real time. - uint64_t a2_fake_boot = 0; + /// Held in a shared atomic, not a plain local: `wait_sleep_fn` below mutates it, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto a2_fake_boot = std::make_shared>(0); PoolPtr a2; EXPECT_NO_THROW( a2 = Pool::open(b, PoolConfig{ @@ -1393,8 +1404,14 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) .mount_lease_ttl_ms = std::chrono::milliseconds(300), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = tiny_budget, - .boot_ms_fn = [&a2_fake_boot] { return a2_fake_boot; }, - .wait_sleep_fn = [&a2_fake_boot](uint64_t ms) { a2_fake_boot += ms; }})); + .boot_ms_fn = [a2_fake_boot] + { + return a2_fake_boot->load(); + }, + .wait_sleep_fn = [a2_fake_boot](uint64_t ms) + { + *a2_fake_boot += ms; + }})); ASSERT_NE(a2, nullptr); EXPECT_GT(a2->writerEpoch(), e1); @@ -1412,14 +1429,23 @@ TEST(CASMountStartup, StaleSelfMountReclaimedAfterWait) .cas_request_budget = tiny_budget}); const String overlap_mount_key = first->layout().mountKey("r"); - uint64_t overlap_fake_boot = 0; + /// Held in a shared atomic, not a plain local: `wait_sleep_fn` below mutates it, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto overlap_fake_boot = std::make_shared>(0); auto replacement = Pool::open(overlap_backend, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "r", .mount_lease_ttl_ms = std::chrono::milliseconds(300), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = tiny_budget, - .boot_ms_fn = [&overlap_fake_boot] { return overlap_fake_boot; }, - .wait_sleep_fn = [&overlap_fake_boot](uint64_t ms) { overlap_fake_boot += ms; }}); + .boot_ms_fn = [overlap_fake_boot] + { + return overlap_fake_boot->load(); + }, + .wait_sleep_fn = [overlap_fake_boot](uint64_t ms) + { + *overlap_fake_boot += ms; + }}); ASSERT_NE(replacement, nullptr); Ops overlap_ops(overlap_backend); diff --git a/src/Disks/tests/gtest_cas_observability.cpp b/src/Disks/tests/gtest_cas_observability.cpp index 1a114b2144c0..2521aae58294 100644 --- a/src/Disks/tests/gtest_cas_observability.cpp +++ b/src/Disks/tests/gtest_cas_observability.cpp @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -150,13 +151,19 @@ TEST(CASObservability, RenewalCountersHaveExactPhysicalAndLogicalDeltas) { auto backend = std::make_shared(); backend->setAttemptTimeoutMs(renewalCounterBudget().attempt_timeout_ms); - uint64_t boot_ms = 100; + /// Captured by value: `boot_ms` is never mutated in this test, and the Pool can outlive this + /// lambda's own stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + const uint64_t boot_ms = 100; auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "renewal-counter-" + std::to_string(attempts) + "-" + std::to_string(resolved), .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .cas_request_budget = renewalCounterBudget(), - .boot_ms_fn = [&] { return boot_ms; }, + .boot_ms_fn = [] + { + return boot_ms; + }, }); backend->fault = fault; const RenewalCounterSnapshot before = renewalCounters(); @@ -174,20 +181,26 @@ TEST(CASObservability, ExternalLeaseDeadlineCountsOnceWithoutReconstructingAttem { auto backend = std::make_shared(); backend->setAttemptTimeoutMs(renewalCounterBudget().attempt_timeout_ms); - uint64_t boot_ms = 100; + /// Held in a shared atomic, not a plain local: this test mutates it below, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto boot_ms = std::make_shared>(100); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "renewal-deadline-counter", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .cas_request_budget = renewalCounterBudget(), - .boot_ms_fn = [&] { return boot_ms; }, + .boot_ms_fn = [boot_ms] + { + return boot_ms->load(); + }, }); /// The fence deadline is 1100 and the safety margin 20, so admission refuses once fewer than /// twenty milliseconds of lease remain. At 1090 only ten milliseconds remain, short of the margin /// however much a single attempt reserves, so nothing can be started and the logical renewal ends /// without reconstructing a sent attempt. - boot_ms = 1090; + boot_ms->store(1090); const RenewalCounterSnapshot before = renewalCounters(); EXPECT_THROW(store->renewWatermarkOnce(), DB::Exception); const RenewalCounterSnapshot after = renewalCounters(); @@ -204,9 +217,15 @@ TEST(CASObservability, ExternalLeaseDeadlineCountsOnceWithoutReconstructingAttem TEST(CASObservability, StageManifestEmitsManifestPut) { std::shared_ptr b; - std::vector seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto s = openPool(b); - s->setEventSink([&](const CasEvent & e){ seen.push_back(e); }); + s->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); const RootNamespace ns{"srv/tbl@cas@"}; auto build = s->beginPartWrite(PartWriteInfo{.intended_ref = ns.string() + "/all_0_0_0", .intended_namespace = ns}); @@ -217,12 +236,13 @@ TEST(CASObservability, StageManifestEmitsManifestPut) const ManifestId id = build->stageManifest({e}); s->setEventSink(nullptr); - EXPECT_EQ(std::count_if(seen.begin(), seen.end(), + const std::vector observed = seen->snapshot(); + EXPECT_EQ(std::count_if(observed.begin(), observed.end(), [](const CasEvent & x){ return x.type == CasEventType::ManifestPut; }), 1); - const auto it = std::find_if(seen.begin(), seen.end(), + const auto it = std::find_if(observed.begin(), observed.end(), [](const CasEvent & x){ return x.type == CasEventType::ManifestPut; }); - ASSERT_NE(it, seen.end()); + ASSERT_NE(it, observed.end()); EXPECT_EQ(it->object_kind, CasEventObjectKind::Manifest); EXPECT_EQ(it->object_hash, manifestRefDebugString(id.ref)); EXPECT_FALSE(it->token.empty()); @@ -235,7 +255,10 @@ TEST(CASObservability, StageManifestEmitsManifestPut) TEST(CASObservability, AbandonEmitsPrecommitRemoved) { std::shared_ptr b; - std::vector seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"srv/tbl@cas@"}; @@ -247,16 +270,20 @@ TEST(CASObservability, AbandonEmitsPrecommitRemoved) const ManifestId id = build->stageManifest({e}); build->precommitAdd(ns, "all_0_0_0", id); - s->setEventSink([&](const CasEvent & x){ seen.push_back(x); }); + s->setEventSink([seen](const CasEvent & x) + { + seen->push(x); + }); build->abandon(); s->setEventSink(nullptr); - EXPECT_EQ(std::count_if(seen.begin(), seen.end(), + const std::vector observed = seen->snapshot(); + EXPECT_EQ(std::count_if(observed.begin(), observed.end(), [](const CasEvent & x){ return x.type == CasEventType::PrecommitRemoved; }), 1); - const auto it = std::find_if(seen.begin(), seen.end(), + const auto it = std::find_if(observed.begin(), observed.end(), [](const CasEvent & x){ return x.type == CasEventType::PrecommitRemoved; }); - ASSERT_NE(it, seen.end()); + ASSERT_NE(it, observed.end()); EXPECT_EQ(it->namespace_, ns.string()); EXPECT_EQ(it->ref_name, "all_0_0_0"); EXPECT_EQ(it->object_kind, CasEventObjectKind::Root); @@ -268,7 +295,10 @@ TEST(CASObservability, AbandonEmitsPrecommitRemoved) TEST(CASObservability, AbandonWithoutPrecommitEmitsNoPrecommitRemoved) { std::shared_ptr b; - std::vector seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"srv/tbl@cas@"}; @@ -279,11 +309,15 @@ TEST(CASObservability, AbandonWithoutPrecommitEmitsNoPrecommitRemoved) e.inline_bytes = "AAA"; build->stageManifest({e}); /// staged, never precommitted - s->setEventSink([&](const CasEvent & x){ seen.push_back(x); }); + s->setEventSink([seen](const CasEvent & x) + { + seen->push(x); + }); build->abandon(); s->setEventSink(nullptr); - EXPECT_EQ(std::count_if(seen.begin(), seen.end(), + const std::vector observed = seen->snapshot(); + EXPECT_EQ(std::count_if(observed.begin(), observed.end(), [](const CasEvent & x){ return x.type == CasEventType::PrecommitRemoved; }), 0); } @@ -300,7 +334,10 @@ TEST(CASObservability, AbandonWithoutPrecommitEmitsNoPrecommitRemoved) TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) { std::shared_ptr b; - std::vector seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"test/tbl"}; const String P = "republish-payload-audit"; @@ -341,7 +378,10 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) const auto condemned_before = global_counters[ProfileEvents::CASGCRetiredCondemned].load(); const auto replaced_before = global_counters[ProfileEvents::CASGCRetireReplaced].load(); - s->setEventSink([&](const CasEvent & e){ seen.push_back(e); }); + s->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); const RoundReport rep = gc.runRegularRound(); s->setEventSink(nullptr); ASSERT_TRUE(rep.acquired_lease); @@ -354,12 +394,13 @@ TEST(CASObservability, ResurrectSupersedeEmitsOnlyRetireReplacedWithOldToken) const String hash_hex = DB::Cas::blobIdOf(DB::Cas::BlobRef{DB::Cas::BlobHashAlgo::CityHash128, DB::Cas::BlobDigest::fromU128(u128Of(P))}); const auto is_this_blob = [&](const CasEvent & e){ return e.object_hash == hash_hex; }; - EXPECT_EQ(std::count_if(seen.begin(), seen.end(), + const std::vector observed = seen->snapshot(); + EXPECT_EQ(std::count_if(observed.begin(), observed.end(), [&](const CasEvent & e){ return is_this_blob(e) && e.type == CasEventType::BlobRetire; }), 0) << "supersede must not also emit blob_retire (that is the fresh-condemn hook's event)"; std::vector replaced_events; - std::copy_if(seen.begin(), seen.end(), std::back_inserter(replaced_events), + std::copy_if(observed.begin(), observed.end(), std::back_inserter(replaced_events), [&](const CasEvent & e){ return is_this_blob(e) && e.type == CasEventType::BlobRetireReplaced; }); ASSERT_EQ(replaced_events.size(), 1u) << "exactly one blob_retire_replaced for the supersede"; /// The event's token text is dialect-qualified ("emulated:", matching `Etag::render` diff --git a/src/Disks/tests/gtest_cas_part_folder_access.cpp b/src/Disks/tests/gtest_cas_part_folder_access.cpp index 6462d0ce4878..b3215cff2216 100644 --- a/src/Disks/tests/gtest_cas_part_folder_access.cpp +++ b/src/Disks/tests/gtest_cas_part_folder_access.cpp @@ -857,13 +857,19 @@ TEST(CASPartFolderAccess, GetViewEmitsRefResolveOnlyOnRealResolveWork) publishPart(store, ns, "part_1", {inlineEntry("checksums.txt", "cs")}); const Cas::PartRefKey key{ns, "part_1"}; - std::vector seen; - store->setEventSink([&](const Cas::CasEvent & e) { seen.push_back(e); }); + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto seen = std::make_shared(); + store->setEventSink([seen](const Cas::CasEvent & e) + { + seen->push(e); + }); Cas::CachedPartFolderAccess access(store, cacheOn()); /// retention on const auto refResolveCount = [&] { - return std::count_if(seen.begin(), seen.end(), + const std::vector observed = seen->snapshot(); + return std::count_if(observed.begin(), observed.end(), [](const Cas::CasEvent & e) { return e.type == Cas::CasEventType::RefResolve; }); }; @@ -1114,8 +1120,15 @@ TEST(CASPartFolderAccess, APostCommitFailureLeavesTheHandleTerminal) auto prepared = access.prepareEntries(key, {inlineEntry("f", "one")}, Cas::ProvenanceOp::Insert); - std::vector seen; - store->setEventSink([&](const Cas::CasEvent & e) { seen.push_back(e); }); + /// Heap-owned, not a plain local: `setEventSink(nullptr)` below only stops FUTURE sink installs + /// from using this closure -- it does not guarantee an already-in-flight background call is not + /// still executing the old one -- and the Pool can outlive this stack frame regardless (a + /// background publish holds `shared_from_this()`). + auto seen = std::make_shared(); + store->setEventSink([seen](const Cas::CasEvent & e) + { + seen->push(e); + }); /// `MEMORY_LIMIT_EXCEEDED` -- what a tracked allocation failure actually raises -- and deliberately /// not `LOGICAL_ERROR`, which aborts at construction in debug/sanitizer builds. @@ -1137,12 +1150,13 @@ TEST(CASPartFolderAccess, APostCommitFailureLeavesTheHandleTerminal) /// abandoned an ALREADY PROMOTED build -- which succeeds, because a promoted build no longer owes a /// precommit removal -- and so ended up terminal too, by accident. What the abandon leaves behind is /// the audit trail of a publish that is reported as thrown away while its ref is committed. - const auto build_aborts = std::count_if(seen.begin(), seen.end(), + const std::vector observed = seen->snapshot(); + const auto build_aborts = std::count_if(observed.begin(), observed.end(), [](const Cas::CasEvent & e) { return e.type == Cas::CasEventType::BuildAbort; }); EXPECT_EQ(build_aborts, 0) << "a build whose promote is DURABLE was abandoned by the failed-promote catch: the handle had " "not yet recorded the commit when the post-commit work threw"; - EXPECT_EQ(std::count_if(seen.begin(), seen.end(), + EXPECT_EQ(std::count_if(observed.begin(), observed.end(), [](const Cas::CasEvent & e) { return e.type == Cas::CasEventType::BuildPublish; }), 1); } diff --git a/src/Disks/tests/gtest_cas_part_write.cpp b/src/Disks/tests/gtest_cas_part_write.cpp index 68fb07080b04..7259f13360ca 100644 --- a/src/Disks/tests/gtest_cas_part_write.cpp +++ b/src/Disks/tests/gtest_cas_part_write.cpp @@ -1938,8 +1938,10 @@ TEST(CASPartWriteTxn, PromoteSucceedsWhenPrecommitIsLiveOwner) TEST(CASPartWriteTxnRepoint, PromoteRepointsCommittedRef) { auto b = std::make_shared(); - /// The sink target must outlive the Pool: `~Pool` emits terminate events into the sink. - std::vector events; + /// Heap-owned, not a plain local: `~Pool` emits terminate events into the sink, and a background + /// publish can hold an extra `shared_from_this()` past this frame's return regardless of + /// declaration order relative to the Pool, so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"srv1/tbl"}; @@ -1967,7 +1969,10 @@ TEST(CASPartWriteTxnRepoint, PromoteRepointsCommittedRef) /// The failed no-flag attempt threw BEFORE appendRefOps returned, so build2's precommit is still the /// live owner (no removal was appended) -- the SAME build/manifest can be retried with the flag. - s->setEventSink([&](const CasEvent & e) { events.push_back(e); }); + s->setEventSink([events](const CasEvent & e) + { + events->push(e); + }); EXPECT_NO_THROW(build2->promote(ns, "part_1", build2->buildId(), m2_id, /*allow_repoint=*/true)); auto resolved = s->resolveRef(ns, "part_1"); ASSERT_TRUE(resolved); @@ -1976,7 +1981,7 @@ TEST(CASPartWriteTxnRepoint, PromoteRepointsCommittedRef) /// Every effective repoint is loud (spec §4): exactly one RefRepoint event, naming the ref and the /// old manifest it replaced. size_t repoint_events = 0; - for (const CasEvent & e : events) + for (const CasEvent & e : events->snapshot()) if (e.type == CasEventType::RefRepoint) { ++repoint_events; @@ -2490,12 +2495,17 @@ TEST(CASPartWriteTxnStageManifestRetry, AmbiguousTimeoutsThenCommitSucceedsWithi TEST(CASPartWriteTxnStageManifestRetry, AmbiguousLandedWriteResolvesToCommittedWithoutReissue) { auto b = std::make_shared(); - /// The sink target must outlive the Pool: `~Pool` emits terminate events into the sink. - std::vector events; + /// Heap-owned, not a plain local: `~Pool` emits terminate events into the sink, and a background + /// publish can hold an extra `shared_from_this()` past this frame's return regardless of + /// declaration order relative to the Pool, so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto s = openPool(b); const RootNamespace ns{"srv/tbl"}; - s->setEventSink([&](const CasEvent & e) { events.push_back(e); }); + s->setEventSink([events](const CasEvent & e) + { + events->push(e); + }); auto build = startBuildFor(s, ns, "part_landed"); b->fault_count = 1; @@ -2509,9 +2519,10 @@ TEST(CASPartWriteTxnStageManifestRetry, AmbiguousLandedWriteResolvesToCommittedW ASSERT_TRUE((*op).read(key, Retry::once()).has_value()); } - const auto ev = std::find_if(events.begin(), events.end(), + const std::vector observed_events = events->snapshot(); + const auto ev = std::find_if(observed_events.begin(), observed_events.end(), [](const CasEvent & e) { return e.type == CasEventType::ManifestPut; }); - ASSERT_NE(ev, events.end()) << "the stage must still emit its ManifestPut audit event"; + ASSERT_NE(ev, observed_events.end()) << "the stage must still emit its ManifestPut audit event"; CasRequests probe(b, Fence::open()); CasOperation probe_op = probe.admit(); const auto landed = probe_op.head(key, Retry::standard()); @@ -2703,13 +2714,18 @@ TEST(CASPartWrite, AmbiguousTimeoutsThenCommitRestreamsFromSource) TEST(CASPartWrite, AmbiguousLandedWriteAdoptsOccupantWithoutReupload) { auto b = std::make_shared(); - /// The sink target must outlive the Pool: `~Pool` emits terminate events into the sink. - std::vector events; + /// Heap-owned, not a plain local: `~Pool` emits terminate events into the sink, and a background + /// publish can hold an extra `shared_from_this()` past this frame's return regardless of + /// declaration order relative to the Pool, so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto s = openBlobFaultPool(b); const RootNamespace ns{"srv/tbl"}; const String payload = "blob-payload-B"; - s->setEventSink([&](const CasEvent & e) { events.push_back(e); }); + s->setEventSink([events](const CasEvent & e) + { + events->push(e); + }); auto build = startBuildFor(s, ns, "part_blob_landed"); const ManifestId id = build->stageManifest({blobManifestEntry("a.bin", payload)}); @@ -2726,16 +2742,17 @@ TEST(CASPartWrite, AmbiguousLandedWriteAdoptsOccupantWithoutReupload) EXPECT_EQ(payload_streams, 1); const String key = s->layout().blobKey(idOf(payload)); - const auto adopt = std::find_if(events.begin(), events.end(), + const std::vector observed_events = events->snapshot(); + const auto adopt = std::find_if(observed_events.begin(), observed_events.end(), [](const CasEvent & e) { return e.type == CasEventType::BlobReuseAdopt; }); - ASSERT_NE(adopt, events.end()) << "the landed occupant must be ADOPTED (the standard dedup leg)"; + ASSERT_NE(adopt, observed_events.end()) << "the landed occupant must be ADOPTED (the standard dedup leg)"; CasRequests probe(b, Fence::open()); CasOperation probe_op = probe.admit(); const auto landed = probe_op.head(key, Retry::standard()); ASSERT_TRUE(landed.has_value()); EXPECT_EQ(adopt->token, landed->etag.render()) << "the adopted token must be the landed incarnation, rendered"; - EXPECT_EQ(std::count_if(events.begin(), events.end(), + EXPECT_EQ(std::count_if(observed_events.begin(), observed_events.end(), [](const CasEvent & e) { return e.type == CasEventType::BlobPut; }), 0) << "no fresh-upload event: the body was never re-uploaded"; } @@ -2909,8 +2926,10 @@ TEST(CASPartWrite, AmbiguousNonLandingPublicationStopsAtOuterBound) TEST(CASPartWrite, AmbiguousCopyLandedAdoptsDestinationWithoutRecopy) { auto b = std::make_shared(); - /// The sink target must outlive the Pool: `~Pool` emits terminate events into the sink. - std::vector events; + /// Heap-owned, not a plain local: `~Pool` emits terminate events into the sink, and a background + /// publish can hold an extra `shared_from_this()` past this frame's return regardless of + /// declaration order relative to the Pool, so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto s = openBlobFaultPool(b); const RootNamespace ns{"srv/tbl"}; const String payload = "staged-payload-A"; @@ -2922,7 +2941,10 @@ TEST(CASPartWrite, AmbiguousCopyLandedAdoptsDestinationWithoutRecopy) ASSERT_TRUE(std::holds_alternative((*seed_op).create(staging_key, staging_bytes, Retry::once()))); } - s->setEventSink([&](const CasEvent & e) { events.push_back(e); }); + s->setEventSink([events](const CasEvent & e) + { + events->push(e); + }); auto build = startBuildFor(s, ns, "part_copy_landed"); const ManifestId id = build->stageManifest({blobManifestEntry("a.bin", payload)}); @@ -2944,9 +2966,10 @@ TEST(CASPartWrite, AmbiguousCopyLandedAdoptsDestinationWithoutRecopy) const auto got = (*op).read(key, Retry::once()); ASSERT_TRUE(got.has_value()); EXPECT_EQ(got->bytes, staging_bytes) << "the destination is the staging object's verbatim copy"; - EXPECT_NE(std::find_if(events.begin(), events.end(), + const std::vector observed_events = events->snapshot(); + EXPECT_NE(std::find_if(observed_events.begin(), observed_events.end(), [](const CasEvent & e) { return e.type == CasEventType::BlobReuseAdopt; }), - events.end()) << "the landed destination must be ADOPTED"; + observed_events.end()) << "the landed destination must be ADOPTED"; } /// A server-side copy publication is ambiguous-and-absent: the first copy attempt times out with diff --git a/src/Disks/tests/gtest_cas_pool.cpp b/src/Disks/tests/gtest_cas_pool.cpp index 115912061d70..452511f5f761 100644 --- a/src/Disks/tests/gtest_cas_pool.cpp +++ b/src/Disks/tests/gtest_cas_pool.cpp @@ -54,6 +54,7 @@ using namespace DB::Cas; using DB::Cas::tests::blobEntryFor; using DB::Cas::tests::expectThrowsCode; using DB::Cas::tests::idOf; +using DB::Cas::tests::SharedWaitLog; using DB::Cas::tests::u128Of; namespace @@ -1529,13 +1530,22 @@ TEST(CASPoolMountFence, OpenRecoversFromFenceInAdoptWindowWithFreshEpoch) /// -> `MountPriorState::Fenced` (a fenced prior is reclaimed on the first attempt, with no /// observation polling -- see `CASMountOpenWaits.FencedPriorReclaimsWithoutAnyWait`). The injected /// `boot_ms_fn`/`wait_sleep_fn` below keep this test off the real clock regardless. - uint64_t fake_boot = 0; + /// Held in a shared atomic, not a plain local: `wait_sleep_fn` below mutates it, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto fake_boot = std::make_shared>(0); DB::Cas::PoolPtr store; ASSERT_NO_THROW( store = DB::Cas::Pool::open(fencing, DB::Cas::PoolConfig{.pool_prefix = "p", .server_root_id = "test", - .boot_ms_fn = [&fake_boot] { return fake_boot; }, - .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }})) + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot](uint64_t ms) + { + *fake_boot += ms; + }})) << "open must recover from a fence in the adopt window, not wedge (exit-49 S13 bug)"; ASSERT_TRUE(store); @@ -1557,25 +1567,31 @@ TEST(CASPoolMountFence, OpenRecoversFromFenceInAdoptWindowWithFreshEpoch) TEST(CASPool, WriteFenceUsesInjectedBootClock) { auto backend = std::make_shared(); - uint64_t fake_boot = 1'000'000; /// arbitrary boottime origin (ms) + /// Held in a shared atomic, not a plain local: this test mutates the clock below, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); /// arbitrary boottime origin (ms) auto store = DB::Cas::Pool::open(backend, DB::Cas::PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30000), - .boot_ms_fn = [&] { return fake_boot; }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, }); /// Freshly armed at open (deadline = fake_boot + ttl): well within the ttl, mutations are allowed. EXPECT_TRUE(store->mayMutate()); /// Advance the boot clock just short of the deadline — still armed. - fake_boot += 29999; + *fake_boot += 29999; EXPECT_TRUE(store->mayMutate()); /// Cross the deadline (ttl elapsed with no renew — a resumed sleeper's view). The fence must expire. /// (The "a gated mutate then fails closed with ABORTED" leg used `mutateShardForTest` -- the held /// Phase-E shard lane -- and moves there; here we pin the boot-clock fence flip itself.) - fake_boot += 2; /// now fake_boot = origin + 30001 > origin + 30000 + *fake_boot += 2; /// now fake_boot = origin + 30001 > origin + 30000 EXPECT_FALSE(store->mayMutate()); } @@ -1768,12 +1784,17 @@ struct SequencedBootClock /// report, not re-asserted here: this test body only encodes the FIXED expectation.) TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) { - SequencedBootClock clock; + /// Heap-owned, not a plain stack local: the Pool can outlive this stack frame (a background + /// publish holds `shared_from_this()`), so a by-reference capture of a local would dangle. + auto clock = std::make_shared(); auto backend = std::make_shared(); auto store = DB::Cas::Pool::open(backend, DB::Cas::PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30'000), - .boot_ms_fn = [&] { return clock(); }, + .boot_ms_fn = [clock] + { + return (*clock)(); + }, }); ASSERT_TRUE(store); @@ -1784,8 +1805,8 @@ TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) /// an unrelated number of `bootMsNow()` calls (all served from `.steady = 0` -- irrelevant, since /// nothing probes the resulting arm before this point). Reset the counter so the FIRST call from /// here on is the remount attempt's own call #1. - clock.queue = {10000, 11000}; - clock.next = 0; + clock->queue = {10000, 11000}; + clock->next = 0; ASSERT_TRUE(store->tryRemountOnce()); @@ -1793,7 +1814,7 @@ TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) /// (10000), so the fence has JUST expired here -- `mayMutate` must be false. (The pre-fix code /// would still read `mayMutate` as true here, armed from 11000 + 30000 -- see the TDD run in the /// report.) - clock.steady = 40000; + clock->steady = 40000; EXPECT_FALSE(store->mayMutate()) << "the remount arm must anchor at the claim attempt's pre-I/O instant, not a later " "response-time reading taken after renewerStart/quiesceRefTablesForRemount"; @@ -1810,14 +1831,17 @@ TEST(CASPoolRemount, RemountArmAnchorsAtClaimAttemptNotResponseTime) TEST(CASMountRemount, SupersededIncarnationDoesNotReclaimALiveSuccessor) { auto backend = std::make_shared(); - uint64_t boot_a = 0; - uint64_t boot_b = 0; + /// Held in shared atomics, not plain locals: `wait_sleep_fn`/`setWaitSleepForTest` below mutate + /// them, and each Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference or by-raw-pointer capture of a local would dangle. + auto boot_a = std::make_shared>(0); + auto boot_b = std::make_shared>(0); /// Mirrors `UncleanOpenPaysOnlyTheObservationWindow`'s tiny budget: the 1s lease TTL below is far /// under the default `cas_request_budget`, so it must be scaled down to fit the required-timeout /// inequality (attempt_timeout + safety_margin < lease TTL). const CasRequestBudget tiny_budget{ .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; - auto config_for = [&](uint64_t * boot, bool unsafe) + auto config_for = [&](const std::shared_ptr> & boot, bool unsafe) { return PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", @@ -1825,16 +1849,22 @@ TEST(CASMountRemount, SupersededIncarnationDoesNotReclaimALiveSuccessor) .mount_renew_period = std::chrono::milliseconds(200), .unsafe_remount_no_delay = unsafe, .cas_request_budget = tiny_budget, - .boot_ms_fn = [boot] { return *boot; }, - .wait_sleep_fn = [boot](uint64_t ms) { *boot += ms; }, + .boot_ms_fn = [boot] + { + return boot->load(); + }, + .wait_sleep_fn = [boot](uint64_t ms) + { + *boot += ms; + }, }; }; - PoolPtr pool_a = Pool::open(backend, config_for(&boot_a, /*unsafe=*/false)); + PoolPtr pool_a = Pool::open(backend, config_for(boot_a, /*unsafe=*/false)); ASSERT_TRUE(pool_a); /// B carries the SAME (server_root_id, server_id) as A -- a copied uuid file -- and opens over A's /// still-live slot under the operator's unsafe knob, reclaiming it at once (no observation). - PoolPtr pool_b = Pool::open(backend, config_for(&boot_b, /*unsafe=*/true)); + PoolPtr pool_b = Pool::open(backend, config_for(boot_b, /*unsafe=*/true)); ASSERT_TRUE(pool_b); EXPECT_NE(pool_a->liveWriterEpoch(), pool_b->liveWriterEpoch()) << "the unsafe reclaim must have minted B a fresh epoch over A's slot"; @@ -1866,12 +1896,13 @@ TEST(CASMountRemount, SupersededIncarnationDoesNotReclaimALiveSuccessor) /// `remount_mutex`), and the wait fires between `claimMountAwaitingExpiry`'s polls -- with no /// backend request of A's own in flight -- so B's call is the only one touching the shared /// in-memory backend at that instant. - size_t polls = 0; - pool_a->setWaitSleepForTest([&](uint64_t ms) + /// Heap-owned, not a plain local: same lifetime rule as `boot_a`/`boot_b` above. + auto polls = std::make_shared>(0); + pool_a->setWaitSleepForTest([boot_a, boot_b, polls, pool_b](uint64_t ms) { - boot_a += ms; - ++polls; - boot_b += ms; + *boot_a += ms; + ++(*polls); + *boot_b += ms; EXPECT_NO_THROW(pool_b->renewWatermarkOnce()); }); EXPECT_FALSE(pool_a->tryRemountOnce()) @@ -1882,7 +1913,7 @@ TEST(CASMountRemount, SupersededIncarnationDoesNotReclaimALiveSuccessor) /// `sleep_ms_fn` call per iteration that does not itself exceed the bound, and none on the /// terminal iteration that does. A widened or removed restart bound would make this hang instead /// of failing, so pin the exact count rather than only asserting it ran. - EXPECT_EQ(polls, DB::Cas::kMaxObservationRestarts + 1) + EXPECT_EQ(polls->load(), DB::Cas::kMaxObservationRestarts + 1) << "the observation must give up after exactly kMaxObservationRestarts restarts, not wait " "indefinitely for a live twin to go quiet"; @@ -1898,7 +1929,10 @@ TEST(CASMountRemount, SupersededIncarnationDoesNotReclaimALiveSuccessor) TEST(CASMountRemount, CutoffFencesWithoutRenewals) { auto backend = std::make_shared(); - uint64_t boot = 0; + /// Held in a shared atomic, not a plain local: this test mutates the clock below, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto boot = std::make_shared>(0); const CasRequestBudget tiny_budget{ .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; PoolPtr store = Pool::open(backend, PoolConfig{ @@ -1906,15 +1940,21 @@ TEST(CASMountRemount, CutoffFencesWithoutRenewals) .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .mount_renew_period = std::chrono::milliseconds(200), .cas_request_budget = tiny_budget, - .boot_ms_fn = [&] { return boot; }, - .wait_sleep_fn = [&](uint64_t ms) { boot += ms; }, + .boot_ms_fn = [boot] + { + return boot->load(); + }, + .wait_sleep_fn = [boot](uint64_t ms) + { + *boot += ms; + }, }); ASSERT_TRUE(store); EXPECT_TRUE(store->mayMutate()) << "freshly armed at open, well within the ttl"; /// No renewals at all -- advance the boot clock past the armed deadline (open's claim anchor plus /// the lease ttl) on this incarnation's own clock alone. - boot += 1001; + *boot += 1001; EXPECT_FALSE(store->mayMutate()) << "crossing the armed deadline must fence closed on the boot clock alone, with no renewal " "conflict needed to trip it"; @@ -2367,14 +2407,26 @@ TEST(CASPoolShutdown, UnresolvedWedgeSkipsFarewell) /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); - uint64_t fake_boot = 1'000'000; + /// Held in a shared atomic, not a plain local: `wait_sleep_fn` and the retry-sleep hook below + /// mutate it, and the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); auto store = DB::Cas::Pool::open(backend, DB::Cas::PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .cas_request_budget = budget, - .boot_ms_fn = [&fake_boot] { return fake_boot; }, - .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }}); + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot](uint64_t ms) + { + *fake_boot += ms; + }}); /// The engine's own inter-attempt sleep advances the same clock its deadlines are read from, so the /// retry bound is reached in test time rather than in ninety real seconds. - store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); + store->setCasRetrySleepForTest([fake_boot](uint64_t ms) + { + *fake_boot += ms; + }); /// By value: `layout` is used after `store.reset()` below, a reference would dangle. const Layout layout = store->layout(); const RootNamespace ns{"srv/wedge_shutdown"}; @@ -2441,8 +2493,11 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) const CasRequestBudget tiny_budget{ .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; - uint64_t fake_boot = 0; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(0); + auto waits = std::make_shared(); PoolPtr store; ASSERT_NO_THROW( store = Pool::open(b, PoolConfig{ @@ -2450,8 +2505,15 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = tiny_budget, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, })); ASSERT_TRUE(store); @@ -2461,15 +2523,16 @@ TEST(CASMountOpenWaits, UncleanOpenPaysOnlyTheObservationWindow) /// loop only re-checks the threshold between polls, so the observed wait rounds UP to the next /// whole poll: ceil(575 / 50) * 50 = 600 ms, i.e. exactly 12 polls of 50 ms each -- because this /// predecessor's death was never certified, only observed. + const std::vector observed_waits = waits->snapshot(); uint64_t total = 0; - for (uint64_t w : waits) + for (uint64_t w : observed_waits) total += w; EXPECT_EQ(total, 600u) << "the observation window must be paid in full, poll-rounded to the " "configured threshold -- neither less (a shortened wait) nor more " "(a reintroduced grace period)"; /// And every one of those polls is exactly one poll interval -- no wait beyond the observation /// poll (the straggler it used to wait out is fenced by the recovery seal instead). - for (uint64_t w : waits) + for (uint64_t w : observed_waits) EXPECT_EQ(w, 50u) << "an unclean reclaim must not block on any wait beyond the observation poll -- the " "straggler it used to wait out is fenced by the recovery seal instead"; @@ -2488,32 +2551,46 @@ TEST(CASMountOpenWaits, UnsafeNoDelayOpensWithoutTheObservationWindow) /// A real predecessor at epoch 7 durably minted this first; seed it here too, or the successor's /// own `allocateWriterEpoch` trips the Phase C guard (epoch absent, mount present -> fail closed). createObj(*b, l.epochKey("test"), encodeServerEpoch(ServerEpoch{.next_writer_epoch = 8})); - std::vector events; - uint64_t fake_boot = 0; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto events = std::make_shared(); + auto fake_boot = std::make_shared>(0); + auto waits = std::make_shared(); PoolPtr store; /// Same server_id (uuid) as the seeded predecessor and a different epoch -- exactly the shape /// `unsafe_remount_no_delay` is for. Unlike the neighbour test, no wait is expected: the bare /// `claimMount` reclaims at once under the operator's authorization. ASSERT_NO_THROW(store = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", - .event_sink = [&](CasEvent e) { events.push_back(std::move(e)); }, + .event_sink = [events](CasEvent e) + { + events->push(std::move(e)); + }, .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), .unsafe_remount_no_delay = true, .cas_request_budget = CasRequestBudget{.attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, })); ASSERT_TRUE(store); - EXPECT_TRUE(waits.empty()) << "no observation window under the unsafe setting"; + EXPECT_TRUE(waits->snapshot().empty()) << "no observation window under the unsafe setting"; /// `Pool` has no test accessor for the adopted `MountPriorState`, so the `UncleanUnsafe` /// classification is asserted through the mount audit event instead: `claimMount`'s unsafe-reclaim /// branch (`CasServerRoot.cpp`) emits exactly one `MountClaim`/"reclaim" event whose reason names /// the setting, and `CASMountClaim.UnsafeAuthorizationIsTokenExact` already pins the classification /// itself at the `claimMount` level. - const auto reclaim_event = std::ranges::find_if(events, + const std::vector observed_events = events->snapshot(); + const auto reclaim_event = std::ranges::find_if(observed_events, [](const CasEvent & e) { return e.reason.find("cas_unsafe_remount_no_delay") != String::npos; }); - ASSERT_NE(reclaim_event, events.end()); + ASSERT_NE(reclaim_event, observed_events.end()); EXPECT_EQ(reclaim_event->type, CasEventType::MountClaim); EXPECT_EQ(reclaim_event->outcome, "reclaim"); EXPECT_EQ(decodeMountLease((*DB::Cas::tests::OperationForTest(b)).read(l.mountKey("test"), Retry::standard())->bytes).writer_epoch, 8u); @@ -2528,16 +2605,21 @@ TEST(CASMountOpenWaits, CleanOpenSkipsAllWaits) .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test"}); predecessor.reset(); - std::vector waits; + /// Heap-owned, not a plain local: the hook below mutates it, and the Pool can outlive this stack + /// frame (a background publish holds `shared_from_this()`), so a by-reference capture would dangle. + auto waits = std::make_shared(); PoolPtr successor; ASSERT_NO_THROW( successor = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", - .wait_sleep_fn = [&](uint64_t ms) { waits.push_back(ms); }, + .wait_sleep_fn = [waits](uint64_t ms) + { + waits->push(ms); + }, })); ASSERT_TRUE(successor); - EXPECT_TRUE(waits.empty()) + EXPECT_TRUE(waits->snapshot().empty()) << "a clean farewell (Task 5) needs no observation window"; } @@ -2576,15 +2658,20 @@ TEST(CASMountOpenWaits, CleanTeardownUnderDefaultBudgetLeavesAFarewell) "shipped default budget (2 * 7000 ms) -- otherwise a clean teardown never hands the mount " "slot back"; - std::vector waits; + /// Heap-owned, not a plain local: the hook below mutates it, and the Pool can outlive this stack + /// frame (a background publish holds `shared_from_this()`), so a by-reference capture would dangle. + auto waits = std::make_shared(); PoolPtr successor; ASSERT_NO_THROW( successor = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", - .wait_sleep_fn = [&](uint64_t ms) { waits.push_back(ms); }, + .wait_sleep_fn = [waits](uint64_t ms) + { + waits->push(ms); + }, })); ASSERT_TRUE(successor); - EXPECT_TRUE(waits.empty()) + EXPECT_TRUE(waits->snapshot().empty()) << "a clean farewell needs no observation window on reopen, even at the shipped default budget"; } @@ -2607,21 +2694,26 @@ TEST(CASMountOpenWaits, FencedPriorReclaimsWithoutAnyWait) const CasRequestBudget tiny_budget{ .attempt_timeout_ms = 50, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = std::nullopt}; - std::vector waits; + /// Heap-owned, not a plain local: the hook below mutates it, and the Pool can outlive this stack + /// frame (a background publish holds `shared_from_this()`), so a by-reference capture would dangle. + auto waits = std::make_shared(); PoolPtr store; ASSERT_NO_THROW( store = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(500), .cas_request_budget = tiny_budget, - .wait_sleep_fn = [&](uint64_t ms) { waits.push_back(ms); }, + .wait_sleep_fn = [waits](uint64_t ms) + { + waits->push(ms); + }, })); ASSERT_TRUE(store); /// A GC-fenced prior is a terminal, already-threshold-gated certificate of death -- reclaimed on the /// FIRST attempt, with no observation polling. It is also an UNCLEAN prior, which used to mean it /// paid the materialization grace; nothing is owed now, so this open blocks on nothing at all. - EXPECT_TRUE(waits.empty()) + EXPECT_TRUE(waits->snapshot().empty()) << "a certified-dead predecessor needs neither the observation window nor any grace period"; } @@ -2642,14 +2734,23 @@ TEST(CASMountOpenWaits, PublicationHorizonUsesTheEnvelope) auto b = std::make_shared(); Layout l{"p"}; DB::Cas::tests::seedPoolMetaForRestart(*b); - uint64_t fake_boot = 0; + /// Held in a shared atomic, not a plain local: the hooks below mutate it, and the Pool can + /// outlive this lambda's own stack frame (a background publish holds `shared_from_this()`), so + /// a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(0); PoolPtr store; store = Pool::open(b, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .cas_request_budget = CasRequestBudget{.attempt_timeout_ms = 100, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = 100}, - .boot_ms_fn = [&] { const uint64_t now = fake_boot; fake_boot += per_call_ms; return now; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; }, + .boot_ms_fn = [fake_boot, per_call_ms] + { + return fake_boot->fetch_add(per_call_ms); + }, + .wait_sleep_fn = [fake_boot](uint64_t ms) + { + *fake_boot += ms; + }, }); if (!store) return 0; @@ -2684,23 +2785,38 @@ TEST(CASPoolRemount, RemountRenewerRedoUsesTheEnvelope) const auto remountConditionalMountWrites = [](uint64_t quiesce_ms) -> uint64_t { auto backend = std::make_shared(); - uint64_t fake_boot = 1'000'000; - DB::Cas::tests::ManualBarrier committed; + /// Held in a shared atomic, not a plain local: the hooks below mutate it, and the Pool can + /// outlive this lambda's own stack frame (a background publish holds `shared_from_this()`), so + /// a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); + /// Heap-owned, not a plain local: declaration order relative to `store` only protects against an + /// ordinary same-thread unwind, not a detached background completion that holds an extra + /// `shared_from_this()` and can still be running on another thread after this call returns. + auto committed = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "remount-renewer-redo-envelope", .server_root_id = "test", .background_watermark = true, - .event_sink = [&committed](const CasEvent & event) + .event_sink = [committed](const CasEvent & event) { if (event.type == CasEventType::MountRemount && event.outcome == "ok") - committed.arriveAndWait(); + committed->arriveAndWait(); }, .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = CasRequestBudget{.attempt_timeout_ms = 100, .lease_safety_margin_ms = 50, .connect_timeout_cap_ms = 100}, - .boot_ms_fn = [&fake_boot] { return fake_boot; }, - .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }, - .remount_quiesce_hook_for_test = [&fake_boot, quiesce_ms] { fake_boot += quiesce_ms; }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot](uint64_t ms) + { + *fake_boot += ms; + }, + .remount_quiesce_hook_for_test = [fake_boot, quiesce_ms] + { + *fake_boot += quiesce_ms; + }, }); const String mount_key = store->layout().mountKey("test"); @@ -2708,9 +2824,9 @@ TEST(CASPoolRemount, RemountRenewerRedoUsesTheEnvelope) const uint64_t before = backend->putOverwriteCount(mount_key); EXPECT_TRUE(store->scheduleRemountForTest()) << "the remount must be latched with quiesce_ms=" << quiesce_ms; - committed.waitUntilArrived(); + committed->waitUntilArrived(); const uint64_t writes = backend->putOverwriteCount(mount_key) - before; - committed.release(); + committed->release(); return writes; }; @@ -2799,18 +2915,27 @@ TEST(CASPool, StartupArmRedoesLeaseWriteWhenTheClaimConsumesTtl) /// before the mount claim); seed that durable epoch object here too, or `Pool::open`'s own /// `allocateWriterEpoch` trips the Phase C guard (epoch absent, mount present -> fail closed). createObj(*backend, layout.epochKey(srid), DB::Cas::encodeServerEpoch(DB::Cas::ServerEpoch{.next_writer_epoch = 8})); - uint64_t fake_boot_ms = 10'000; + /// Held in a shared atomic, not a plain local: `on_second_mount_write` below mutates it, and the + /// Pool can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot_ms = std::make_shared>(10'000); DB::Cas::PoolConfig cfg; cfg.pool_prefix = "pool"; cfg.server_id = uuid; cfg.server_root_id = srid; cfg.background_watermark = true; cfg.mount_lease_ttl_ms = std::chrono::milliseconds(30'000); - cfg.boot_ms_fn = [&] { return fake_boot_ms; }; + cfg.boot_ms_fn = [fake_boot_ms] + { + return fake_boot_ms->load(); + }; /// The renewer's adopt write stalls for 15 s of boot clock. That consumes the publication horizon /// (one 10 s cadence plus one 5 s attempt) while leaving one physical attempt admissible inside /// the old lease's safety window, so the synchronous redo can safely re-anchor. - backend->on_second_mount_write = [&] { fake_boot_ms += 15'000; }; + backend->on_second_mount_write = [fake_boot_ms] + { + *fake_boot_ms += 15'000; + }; auto store = DB::Cas::Pool::open(backend, cfg); ASSERT_NE(store, nullptr); @@ -2836,28 +2961,41 @@ TEST(CASPool, StartupArmRedoesLeaseWriteWhenTheClaimConsumesTtl) TEST(CASRemountWaits, DrainedRemountPaysNoWait) { auto backend = std::make_shared(); - uint64_t fake_boot = 1'000'000; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); + auto waits = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30000), - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, }); ASSERT_TRUE(store); - EXPECT_TRUE(waits.empty()) << "a fresh mount (no predecessor) pays no wait at open"; - store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); + EXPECT_TRUE(waits->snapshot().empty()) << "a fresh mount (no predecessor) pays no wait at open"; + store->setCasRetrySleepForTest([fake_boot](uint64_t ms) + { + *fake_boot += ms; + }); /// Trip the fence: advance the local boot clock past the deadline (as in `WriteFenceUsesInjectedBootClock` /// above) and mark the durable lease `gc_fenced` (the certificate `claimMountAwaitingExpiry` reclaims /// on its FIRST attempt, no observation polling -- avoids a real sleep in this test). - fake_boot += 30001; + *fake_boot += 30001; fenceOutMount(*backend, store->layout().mountKey("test")); /// No in-flight ref-log PUT at all -- the easy direction. ASSERT_TRUE(store->tryRemountOnce()); - EXPECT_TRUE(waits.empty()) + EXPECT_TRUE(waits->snapshot().empty()) << "a drained self-remount must pay no wait"; } @@ -2871,23 +3009,36 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); - uint64_t fake_boot = 1'000'000; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); + auto waits = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30000), .cas_request_budget = budget, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, }); ASSERT_TRUE(store); - EXPECT_TRUE(waits.empty()) << "a fresh mount (no predecessor) pays no wait at open"; + EXPECT_TRUE(waits->snapshot().empty()) << "a fresh mount (no predecessor) pays no wait at open"; /// `dropRef` below drives the fault through `ensureRefTableRecovered`'s own recovery-retry loop, /// which sleeps via `recovery_retry_sleep_fn` (a REAL 200ms-slice sleep by default) while measuring /// elapsed time against `boot_ms_now_fn` -- the frozen `fake_boot` this fixture already injects. /// Without also virtualizing the sleep, that elapsed check never advances and the loop spins for /// real until the harness times the test out. - store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); + store->setCasRetrySleepForTest([fake_boot](uint64_t ms) + { + *fake_boot += ms; + }); const Layout & layout = store->layout(); const RootNamespace ns{"srv/remount_wedge"}; @@ -2904,7 +3055,7 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) ASSERT_TRUE(store->refLaneWedgedForTest(ns)); /// Trip the fence exactly as in `DrainedRemountSkipsGrace` above. - fake_boot += 30001; + *fake_boot += 30001; fenceOutMount(*backend, store->layout().mountKey("test")); /// THE HARD DIRECTION, and the one the retired wait existed for: a ref lane that still holds an @@ -2913,7 +3064,7 @@ TEST(CASRemountWaits, UnresolvedWedgeRemountPaysNoWaitEither) /// recovery writes into its slot. ASSERT_TRUE(store->tryRemountOnce()); - EXPECT_TRUE(waits.empty()) + EXPECT_TRUE(waits->snapshot().empty()) << "an unresolved ref-lane wedge must not make the remount block: the straggler it describes is " "fenced by the recovery seal, not waited out"; } @@ -2940,16 +3091,28 @@ TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPrede /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); - uint64_t fake_boot = 1'000'000; + /// Held in a shared atomic, not a plain local: the hooks below mutate it, and the Pool can outlive + /// this stack frame (a background publish holds `shared_from_this()`), so a by-reference capture + /// of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30000), .cas_request_budget = budget, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot](uint64_t ms) + { + *fake_boot += ms; + }, }); ASSERT_TRUE(store); - store->setCasRetrySleepForTest([&fake_boot](uint64_t ms) { fake_boot += ms; }); + store->setCasRetrySleepForTest([fake_boot](uint64_t ms) + { + *fake_boot += ms; + }); const Layout & layout = store->layout(); const RootNamespace ns1{"srv/table_a"}; @@ -2974,14 +3137,14 @@ TEST(CASRemountWaits, ALateTouchedTableClosesEveryDeadEpochInBandHoweverItsPrede ASSERT_TRUE(store->refLaneWedgedForTest(ns1)); /// Self-remount #1: UNCLEAN (the wedge above). Epoch 1 -> 2. - fake_boot += 30001; + *fake_boot += 30001; fenceOutMount(*backend, store->layout().mountKey("test")); ASSERT_TRUE(store->tryRemountOnce()); ASSERT_EQ(store->liveWriterEpoch(), 2u); /// Self-remount #2: CLEAN (no wedge left behind -- `quiesceRefTablesForRemount` already cleared the /// cache). Epoch 2 -> 3. - fake_boot += 30001; + *fake_boot += 30001; fenceOutMount(*backend, store->layout().mountKey("test")); ASSERT_TRUE(store->tryRemountOnce()); ASSERT_EQ(store->liveWriterEpoch(), 3u); @@ -3052,8 +3215,15 @@ TEST(CASPool, ReadManifestAbsentBodyEmitsReadMissingWithOneGetAndNoHead) const ManifestId id{.root_namespace = ns, .ref = manifestRefFor("absent-body-event")}; const String key = layout.manifestKey(id); - std::vector events; - s->setEventSink([&](CasEvent e) { events.push_back(std::move(e)); }); + /// Heap-owned, not a plain local: `setEventSink(nullptr)` below only stops FUTURE sink installs from + /// using this closure -- it does not guarantee an already-in-flight background call is not still + /// executing the old one -- and the Pool can outlive this stack frame regardless (a background + /// publish holds `shared_from_this()`). + auto events = std::make_shared(); + s->setEventSink([events](CasEvent e) + { + events->push(std::move(e)); + }); b->resetCounts(); expectThrowsCode(DB::ErrorCodes::FILE_DOESNT_EXIST, [&] { s->readManifest(id); }); s->setEventSink(nullptr); @@ -3061,7 +3231,7 @@ TEST(CASPool, ReadManifestAbsentBodyEmitsReadMissingWithOneGetAndNoHead) EXPECT_EQ(b->getCount(key), 1u); EXPECT_EQ(b->headCount(key), 0u); size_t read_missing = 0; - for (const auto & e : events) + for (const auto & e : events->snapshot()) { if (e.type != CasEventType::ReadMissing) continue; @@ -3884,56 +4054,70 @@ TEST(CASPoolRemount, ImmediatePostRemountRenewalFailureIsNotDropped) TEST(CASPoolRemount, StaleRemountAnchorPerformsParkedRedo) { auto backend = std::make_shared(); - uint64_t fake_boot = 100; - DB::Cas::tests::ManualBarrier committed; + /// Held in a shared atomic, not a plain local: `remount_quiesce_hook_for_test` below mutates it, + /// and the Pool can outlive this stack frame (a background publish holds `shared_from_this()`), so + /// a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(100); + /// Heap-owned, not a plain local: declaration order relative to the Pool below only protects + /// against an ordinary same-thread unwind, not a detached background completion that holds an + /// extra `shared_from_this()` and can still be running on another thread after this frame returns. + auto committed = std::make_shared(); PoolConfig config{ .pool_prefix = "stale-remount-anchor", .server_root_id = "test", .background_watermark = true, - .event_sink = [&](const CasEvent & event) + .event_sink = [committed](const CasEvent & event) { if (event.type == CasEventType::MountRemount && event.outcome == "ok") - committed.arriveAndWait(); + committed->arriveAndWait(); }, .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = runtimeRenewBudget(), - .boot_ms_fn = [&] { return fake_boot; }, - .remount_quiesce_hook_for_test = [&] { fake_boot += 900; }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .remount_quiesce_hook_for_test = [fake_boot] + { + *fake_boot += 900; + }, }; auto store = Pool::open(backend, config); const String key = store->layout().mountKey("test"); fenceOutMount(*backend, key); const uint64_t writes_before = backend->putOverwriteCount(key); ASSERT_TRUE(store->scheduleRemountForTest()); - committed.waitUntilArrived(); + committed->waitUntilArrived(); EXPECT_GE(backend->putOverwriteCount(key), writes_before + 3) << "claim, renewer start, and the stale-anchor parked redo must all write"; - committed.release(); + committed->release(); } TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) { auto backend = std::make_shared(); - uint64_t fake_boot = 100; - std::promise result_observed; - std::future result_future = result_observed.get_future(); - std::atomic result_published{false}; - std::mutex events_mutex; - std::vector events; + /// Held in a shared atomic, not a plain local: `remount_quiesce_hook_for_test` below mutates it, + /// and the Pool can outlive this stack frame (a background publish holds `shared_from_this()`), so + /// a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(100); + /// Heap-owned, not plain locals: `event_sink` below mutates them, and the Pool can outlive this + /// stack frame (a background publish holds `shared_from_this()`), so a by-reference capture of a + /// local -- including a non-copyable `std::promise` -- would dangle. + auto result_observed = std::make_shared>(); + std::future result_future = result_observed->get_future(); + auto result_published = std::make_shared>(false); + auto events = std::make_shared(); PoolConfig config{ .pool_prefix = "parked-redo-recovered-observability", .server_root_id = "test", .background_watermark = true, - .event_sink = [&](CasEvent event) + .event_sink = [result_observed, result_published, events](CasEvent event) { const bool final_remount = event.type == CasEventType::MountRemount && event.outcome == "ok"; - { - std::lock_guard lock(events_mutex); - events.push_back(std::move(event)); - } - if (final_remount && !result_published.exchange(true)) - result_observed.set_value(); + events->push(std::move(event)); + if (final_remount && !result_published->exchange(true)) + result_observed->set_value(); }, .mount_lease_ttl_ms = std::chrono::milliseconds(1000), /// 500 with a 700 ms quiescence, so the redo's window (period + attempt timeout = 510) does not @@ -3941,10 +4125,16 @@ TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) /// the engine's jittered backoff draws from its first-reissue range of at most 200 ms. .mount_renew_period = std::chrono::milliseconds(500), .cas_request_budget = runtimeRenewBudget(), - .boot_ms_fn = [&] { return fake_boot; }, - .remount_quiesce_hook_for_test = [&] + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + /// `backend` is captured BY VALUE (a copy of the shared_ptr, not the stack slot holding it): + /// the Pool can outlive this frame, so a by-reference capture of the local `shared_ptr` itself + /// would dangle even though the pointee it owns is heap-allocated. + .remount_quiesce_hook_for_test = [fake_boot, backend] { - fake_boot += 700; + *fake_boot += 700; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; }, }; @@ -3955,11 +4145,7 @@ TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) ASSERT_TRUE(store->scheduleRemountForTest()); ASSERT_EQ(result_future.wait_for(std::chrono::seconds(20)), std::future_status::ready); - std::vector observed; - { - std::lock_guard lock(events_mutex); - observed = events; - } + const std::vector observed = events->snapshot(); const auto recovered = std::find_if(observed.begin(), observed.end(), [](const CasEvent & event) { return event.type == CasEventType::WatermarkRenew && event.outcome == "recovered"; @@ -3986,37 +4172,47 @@ TEST(CASPoolRemount, ParkedRedoRecoveryObservabilityPrecedesRemountResult) TEST(CASPoolRemount, ParkedRedoFailureObservabilityPrecedesRemountResult) { auto backend = std::make_shared(); - uint64_t fake_boot = 100; - std::promise result_observed; - std::future result_future = result_observed.get_future(); - std::atomic result_published{false}; - std::mutex events_mutex; - std::vector events; + /// Held in a shared atomic, not a plain local: the hooks below mutate it, and the Pool can outlive + /// this stack frame (a background publish holds `shared_from_this()`), so a by-reference capture + /// of a local would dangle. + auto fake_boot = std::make_shared>(100); + /// Heap-owned, not plain locals: `event_sink` below mutates them, and the Pool can outlive this + /// stack frame (a background publish holds `shared_from_this()`), so a by-reference capture of a + /// local -- including a non-copyable `std::promise` -- would dangle. + auto result_observed = std::make_shared>(); + std::future result_future = result_observed->get_future(); + auto result_published = std::make_shared>(false); + auto events = std::make_shared(); PoolConfig config{ .pool_prefix = "parked-redo-failed-observability", .server_root_id = "test", .background_watermark = true, - .event_sink = [&](CasEvent event) + .event_sink = [result_observed, result_published, events](CasEvent event) { const bool final_remount = event.type == CasEventType::MountRemount && event.outcome == "failed"; - { - std::lock_guard lock(events_mutex); - events.push_back(std::move(event)); - } - if (final_remount && !result_published.exchange(true)) - result_observed.set_value(); + events->push(std::move(event)); + if (final_remount && !result_published->exchange(true)) + result_observed->set_value(); }, .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = runtimeRenewBudget(), - .boot_ms_fn = [&] { return fake_boot; }, - .remount_quiesce_hook_for_test = [&] + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + /// `backend` is captured BY VALUE (a copy of the shared_ptr): the Pool can outlive this frame, + /// so a by-reference capture of the local `shared_ptr` itself would dangle. + .remount_quiesce_hook_for_test = [fake_boot, backend] { - fake_boot += 900; + *fake_boot += 900; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; /// The attempt is admitted 80 ms before its lease-safe bound; spending 90 inside it puts the /// resolve read past that bound, so the ambiguity is refused instead of reissued. - backend->before_throw = [&] { fake_boot += 90; }; + backend->before_throw = [fake_boot] + { + *fake_boot += 90; + }; }, }; auto store = Pool::open(backend, config); @@ -4026,11 +4222,7 @@ TEST(CASPoolRemount, ParkedRedoFailureObservabilityPrecedesRemountResult) ASSERT_TRUE(store->scheduleRemountForTest()); ASSERT_EQ(result_future.wait_for(std::chrono::seconds(20)), std::future_status::ready); - std::vector observed; - { - std::lock_guard lock(events_mutex); - observed = events; - } + const std::vector observed = events->snapshot(); const auto failed_renew = std::find_if(observed.begin(), observed.end(), [](const CasEvent & event) { return event.type == CasEventType::WatermarkRenew && event.outcome == "failed"; @@ -4058,21 +4250,26 @@ TEST(CASPoolRemount, ThrowingEventSinkAfterCommitLeavesRuntimeLive) auto backend = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "throwing-remount-event", .server_root_id = "test", .background_watermark = true}); - DB::Cas::tests::ManualBarrier committed; - store->setEventSink([&](const CasEvent & event) + /// Heap-owned, not a plain local declared after `store`: if `waitUntilArrived` below throws on its + /// own internal timeout, unwinding would destroy a stack-local barrier before `store`'s destructor + /// joins the remount worker, and that worker can still be inside `arriveAndWait` on the dangling + /// reference. A `shared_ptr` capture keeps the barrier alive for as long as the worker needs it, + /// independent of declaration order. + auto committed = std::make_shared(); + store->setEventSink([committed](const CasEvent & event) { if (event.type == CasEventType::MountRemount && event.outcome == "ok") { - committed.arriveAndWait(); + committed->arriveAndWait(); throw DB::Exception(DB::ErrorCodes::NETWORK_ERROR, "injected remount event sink failure"); } }); fenceOutMount(*backend, store->layout().mountKey("test")); ASSERT_TRUE(store->scheduleRemountForTest()); - committed.waitUntilArrived(); + committed->waitUntilArrived(); EXPECT_EQ(store->lifecycle(), PoolLifecycle::Live); EXPECT_TRUE(store->mayMutate()); - committed.release(); + committed->release(); EXPECT_NO_THROW(store.reset()); } @@ -4215,7 +4412,10 @@ TEST(CASPool, DecommissionCadenceValidationPrecedesAuthorityWrites) TEST(CASPool, DisabledBackgroundDoesNotReserveRenewalCadence) { auto backend = std::make_shared(); - uint64_t fake_boot = 100; + /// Captured by value: `fake_boot` is never mutated in this test, and the Pool can outlive this + /// stack frame (a background publish holds `shared_from_this()`), so a by-reference capture would + /// dangle. + const uint64_t fake_boot = 100; PoolConfig config{ .pool_prefix = "disabled-renew-cadence", .server_root_id = "test", @@ -4223,7 +4423,7 @@ TEST(CASPool, DisabledBackgroundDoesNotReserveRenewalCadence) .mount_lease_ttl_ms = std::chrono::milliseconds(100), .mount_renew_period = std::chrono::hours(24), .cas_request_budget = runtimeRenewBudget(), - .boot_ms_fn = [&] { return fake_boot; }, + .boot_ms_fn = [] { return fake_boot; }, }; auto store = Pool::open(backend, config); const String key = store->layout().mountKey("test"); @@ -4270,25 +4470,34 @@ TEST(CASPool, DeterministicWorkerFailureFencesWithoutWaitingForCadence) TEST(CASPool, RenewWatermarkOnceRefreshesFenceAndDepositsOneFailure) { auto backend = std::make_shared(); - uint64_t fake_boot = 100; + /// Held in a shared atomic, not a plain local: this test mutates it directly below, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(100); PoolConfig config{ .pool_prefix = "direct-renew", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(1000), .cas_request_budget = runtimeRenewBudget(), - .boot_ms_fn = [&] { return fake_boot; }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, }; auto store = Pool::open(backend, config); - fake_boot = 500; + fake_boot->store(500); EXPECT_NO_THROW(store->renewWatermarkOnce()); - fake_boot = 1200; + fake_boot->store(1200); EXPECT_TRUE(store->mayMutate()) << "direct success must refresh the local fence from attempt start"; backend->fault = RuntimeRenewBackend::Fault::ThrowBefore; /// The renewal that succeeded at 500 anchored the lease for its 1000 ms TTL, so it expires at 1500. /// Expire it from inside the attempt: the fault alone no longer ends a renewal, because the engine /// settles the ambiguity by reading and reissues, and the reissue commits. - backend->before_throw = [&] { fake_boot = 1500; }; + backend->before_throw = [fake_boot] + { + fake_boot->store(1500); + }; const uint64_t schedules_before = store->scheduleRemountCallCountForTest(); expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->renewWatermarkOnce(); }); EXPECT_FALSE(store->mayMutate()); @@ -4298,12 +4507,17 @@ TEST(CASPool, RenewWatermarkOnceRefreshesFenceAndDepositsOneFailure) TEST(CASPoolRemount, WholeChainResultsAreNumberedAndStepLabelled) { auto backend = std::make_shared(); - std::vector events; + /// Heap-owned, not a plain local: the Pool can outlive this stack frame (a background publish holds + /// `shared_from_this()`), so a by-reference capture of a local would dangle. + auto events = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "remount-observability", .server_root_id = "test", }); - store->setEventSink([&](CasEvent event) { events.push_back(std::move(event)); }); + store->setEventSink([events](CasEvent event) + { + events->push(std::move(event)); + }); ScopedRemountLogCapture logs; store->tripMountLost(); @@ -4320,8 +4534,9 @@ TEST(CASPoolRemount, WholeChainResultsAreNumberedAndStepLabelled) EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRemountSucceeded].load(), succeeded_before + 1); EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRemountFailed].load(), failed_before + 1); + const std::vector observed_events = events->snapshot(); std::vector remounts; - std::copy_if(events.begin(), events.end(), std::back_inserter(remounts), [](const CasEvent & event) + std::copy_if(observed_events.begin(), observed_events.end(), std::back_inserter(remounts), [](const CasEvent & event) { return event.type == CasEventType::MountRemount; }); @@ -4357,20 +4572,35 @@ TEST(CASPoolRemount, TheRenewerRedoRenewsOnTheOpenPlane) const auto remountConditionalMountWrites = [](uint64_t quiesce_ms) -> uint64_t { auto backend = std::make_shared(); - uint64_t fake_boot = 1'000'000; - DB::Cas::tests::ManualBarrier committed; + /// Held in a shared atomic, not a plain local: the hooks below mutate it, and the Pool can + /// outlive this lambda's own stack frame (a background publish holds `shared_from_this()`), so + /// a by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); + /// Heap-owned, not a plain local: declaration order relative to `store` only protects against an + /// ordinary same-thread unwind, not a detached background completion that holds an extra + /// `shared_from_this()` and can still be running on another thread after this call returns. + auto committed = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "remount-renewer-redo", .server_root_id = "test", .background_watermark = true, - .event_sink = [&committed](const CasEvent & event) + .event_sink = [committed](const CasEvent & event) { if (event.type == CasEventType::MountRemount && event.outcome == "ok") - committed.arriveAndWait(); + committed->arriveAndWait(); + }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot](uint64_t ms) + { + *fake_boot += ms; + }, + .remount_quiesce_hook_for_test = [fake_boot, quiesce_ms] + { + *fake_boot += quiesce_ms; }, - .boot_ms_fn = [&fake_boot] { return fake_boot; }, - .wait_sleep_fn = [&fake_boot](uint64_t ms) { fake_boot += ms; }, - .remount_quiesce_hook_for_test = [&fake_boot, quiesce_ms] { fake_boot += quiesce_ms; }, }); const String mount_key = store->layout().mountKey("test"); @@ -4378,9 +4608,9 @@ TEST(CASPoolRemount, TheRenewerRedoRenewsOnTheOpenPlane) const uint64_t before = backend->putOverwriteCount(mount_key); EXPECT_TRUE(store->scheduleRemountForTest()) << "the remount must be latched with quiesce_ms=" << quiesce_ms; - committed.waitUntilArrived(); + committed->waitUntilArrived(); const uint64_t writes = backend->putOverwriteCount(mount_key) - before; - committed.release(); + committed->release(); return writes; }; diff --git a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp index c3488fb44902..7919711c1753 100644 --- a/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp +++ b/src/Disks/tests/gtest_cas_ref_recovery_cas_walk.cpp @@ -1052,14 +1052,23 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEveryOccupiedObjectBeforeAdvancingP : makeOrdinaryTxn(ns, test_case.occupant, "late", /*birth=*/false); backend->late_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(occupant)); - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: the retry-sleep hook below mutates it, and the + /// Pool can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config = walkTestConfig(); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] + { + return fake_now->load(); + }; config.cas_request_budget.recovery_retry_budget_ms = 1; config.cas_request_budget.recovery_retry_initial_backoff_ms = 1; config.cas_request_budget.recovery_retry_max_backoff_ms = 1; auto store = openWalkPool(backend, config); - store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); + store->setCasRetrySleepForTest([fake_now](uint64_t ms) + { + *fake_now += ms; + }); backend->ambiguous_cas_substr = layout.refCkptKey(life); backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; @@ -1128,15 +1137,24 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesEachCreatedSealBeforeCreatingTheNex seedCkpt(*backend, layout, ns, lifeEpochCkpt(1, initial_frontier)); const NamespaceLifeId life = catalogLife(backend, layout, ns); - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: the retry-sleep hook below mutates it, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config = walkTestConfig(); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] + { + return fake_now->load(); + }; config.cas_request_budget.recovery_retry_budget_ms = 1; config.cas_request_budget.recovery_retry_initial_backoff_ms = 1; config.cas_request_budget.recovery_retry_max_backoff_ms = 1; auto store = openWalkPool(backend, config); ASSERT_EQ(store->liveWriterEpoch(), 3u); - store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); + store->setCasRetrySleepForTest([fake_now](uint64_t ms) + { + *fake_now += ms; + }); backend->ambiguous_cas_substr = layout.refCkptKey(life); backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; @@ -1181,14 +1199,23 @@ TEST(CASRefRecoveryCasWalk, RecoveryPublishesAnAdoptedStragglerBeforeCreatingIts backend->late_bytes = sealObject(FormatId::RefLog, encodeRefLogTxn(makeOrdinaryTxn(ns, straggler, "late", /*birth=*/false))); - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: the retry-sleep hook below mutates it, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config = walkTestConfig(); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] + { + return fake_now->load(); + }; config.cas_request_budget.recovery_retry_budget_ms = 1; config.cas_request_budget.recovery_retry_initial_backoff_ms = 1; config.cas_request_budget.recovery_retry_max_backoff_ms = 1; auto store = openWalkPool(backend, config); - store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); + store->setCasRetrySleepForTest([fake_now](uint64_t ms) + { + *fake_now += ms; + }); backend->ambiguous_cas_substr = layout.refCkptKey(life); backend->ambiguous_cas_count = kFaultsBeyondTheRetryWindow; @@ -1952,16 +1979,25 @@ TEST(CASRefRecoveryCasWalk, UnresolvedSealSlotFailsClosedWithoutInstalling) /// envelope is spent in a handful of iterations instead of spinning against a frozen clock. Not /// cosmetic: with a frozen clock this test burns ~700k retries and the same number of log lines, /// which is how a real regression in this arm would become invisible in the noise. - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: the retry-sleep hook below mutates it, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config = walkTestConfig(); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] + { + return fake_now->load(); + }; auto store = openWalkPool(backend, config); ASSERT_TRUE(store); - store->setCasRetrySleepForTest([&fake_now](uint64_t ms) { fake_now += ms; }); + store->setCasRetrySleepForTest([fake_now](uint64_t ms) + { + *fake_now += ms; + }); backend->ambiguous_put_substr = "/_log/"; - const uint64_t fake_now_before = fake_now; + const uint64_t fake_now_before = fake_now->load(); EXPECT_ANY_THROW(store->listRefs(ns)); EXPECT_FALSE(store->refTableRecoveredForTest(ns)) << "a table whose dead epoch may or may not be closed must never be exposed as recovered"; @@ -1969,7 +2005,7 @@ TEST(CASRefRecoveryCasWalk, UnresolvedSealSlotFailsClosedWithoutInstalling) /// injected clock before giving up; a fault settled by a single, unretried attempt would not /// exercise the transient-retry path this test's own name and docstring claim to drive. EXPECT_GT(backend->ambiguous_put_attempts.load(), 1u); - EXPECT_GT(fake_now, fake_now_before); + EXPECT_GT(fake_now->load(), fake_now_before); } /// --------------------------------------------------------------------------------------------- diff --git a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp index 918fa7391780..dbb53e10fe0a 100644 --- a/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp +++ b/src/Disks/tests/gtest_cas_ref_snapshot_publish_ordering.cpp @@ -15,6 +15,7 @@ #include #include +#include #include #include #include @@ -365,14 +366,20 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) budget.attempt_timeout_ms = 100; budget.lease_safety_margin_ms = 100; - uint64_t fake_now = 1'000'000; + /// Held in a shared atomic, not a plain local: this test mutates the clock below, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto fake_now = std::make_shared>(1'000'000); PoolConfig config; config.snapshot_log_count_threshold = 0; /// any nonempty tail is over-threshold config.snapshot_log_bytes_threshold = 1ULL << 40; config.snapshot_publish_backoff_initial_ms = 1000; config.snapshot_publish_backoff_max_ms = 4000; config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] + { + return fake_now->load(); + }; config.cas_request_budget = budget; /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. @@ -411,13 +418,13 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) EXPECT_EQ(dispatchCount(), d1) << "a read within the initial backoff window must not re-dispatch"; /// Cross the 1000ms deadline: exactly one retry dispatches (and fails again, doubling to 2000ms). - fake_now += 1000; + *fake_now += 1000; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d1 + 1) << "past the first deadline, exactly one retry dispatches"; /// Short of the DOUBLED (2000ms) deadline: still refused. - fake_now += 1000; + *fake_now += 1000; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d1 + 1) @@ -425,7 +432,7 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) /// Cross the doubled deadline: one more retry dispatches (and fails again -- the third and last armed /// failure -- doubling to the 4000ms cap). - fake_now += 1000; + *fake_now += 1000; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d1 + 2) << "past the doubled deadline, exactly one more retry dispatches"; @@ -434,7 +441,7 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) /// 2000ms, or that read `initial` where it means `max`, would still pass -- the only check so far /// is AT the +4000 crossing below. 2000ms past the doubled deadline is still short of the capped /// 4000ms backoff, so no third retry may dispatch yet. - fake_now += 2000; + *fake_now += 2000; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d1 + 2) @@ -444,7 +451,7 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) /// `resetPublishBackoff` clears the cooldown -- proved by the NEXT trigger dispatching with no wait /// at all. backend->armWriteFailure("_snap/", 0); - fake_now += 2000; + *fake_now += 2000; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d1 + 3) << "past the second (capped) deadline, the retry dispatches and succeeds"; @@ -467,11 +474,11 @@ TEST(CASRefSnapshotPublishOrdering, PublishBackoffDecisionsAreCharacterized) ASSERT_EQ(publishRef(store, ns, "ref_4", 4), (RefTxnId{store->writerEpoch(), 4})); store->waitForSnapshotPublishSettleForTest(ns); const uint64_t d2 = dispatchCount(); - fake_now += 500; + *fake_now += 500; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d2) << "short of 1000ms since the reset, no retry may dispatch yet"; - fake_now += 500; + *fake_now += 500; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatchCount(), d2 + 1) @@ -493,14 +500,20 @@ TEST(CASRefSnapshotPublishOrdering, NotReadyRefusalBacksOffAndResetsAfterDurable budget.attempt_timeout_ms = 100; budget.lease_safety_margin_ms = 100; - uint64_t fake_now = 2'000'000; + /// Held in a shared atomic, not a plain local: this test mutates the clock below, and the Pool can + /// outlive this stack frame (a background publish holds `shared_from_this()`), so a by-reference + /// capture of a local would dangle. + auto fake_now = std::make_shared>(2'000'000); PoolConfig config; config.snapshot_log_count_threshold = 0; config.snapshot_log_bytes_threshold = 1ULL << 40; config.snapshot_publish_backoff_initial_ms = 200; config.snapshot_publish_backoff_max_ms = 30'000; config.mount_lease_ttl_ms = std::chrono::milliseconds(10'000'000); - config.boot_ms_fn = [&fake_now] { return fake_now; }; + config.boot_ms_fn = [fake_now] + { + return fake_now->load(); + }; config.cas_request_budget = budget; /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. @@ -584,14 +597,14 @@ TEST(CASRefSnapshotPublishOrdering, NotReadyRefusalBacksOffAndResetsAfterDurable 400, 800, 1600, 3200, 6400, 12'800, 25'600, 30'000, 30'000}; for (const uint64_t next_delay_ms : next_delays) { - fake_now += delay_ms - 1; + *fake_now += delay_ms - 1; store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); EXPECT_EQ(dispatch_count(), production_dispatches + 1 + admitted_retries) << "no retry may dispatch one millisecond before the current deadline"; EXPECT_EQ(backoff_count(), production_backoffs + 1 + admitted_retries); - ++fake_now; + ++(*fake_now); store->resolveRef(ns, "ref_1"); store->waitForSnapshotPublishSettleForTest(ns); ++admitted_retries; diff --git a/src/Disks/tests/gtest_cas_ref_writer.cpp b/src/Disks/tests/gtest_cas_ref_writer.cpp index 5835f1867010..159b12fb99e9 100644 --- a/src/Disks/tests/gtest_cas_ref_writer.cpp +++ b/src/Disks/tests/gtest_cas_ref_writer.cpp @@ -138,27 +138,12 @@ class VirtualRetryClock /// background syncer happens to be on, and the test reads the accumulated events afterward from the /// main test thread with no other ordering between the two -- a bare `std::vector` there is a real data /// race (the class this file's four `setEventSink` call sites all had, hidden because a debug/ASan build -/// doesn't reliably catch an unsynchronized push_back/iterator-read pair on a small vector). `add` takes -/// the lock only around the push; `snapshot` copies out under the lock and returns, so a caller iterating -/// the result never holds the mutex across anything that could call back into the pool (which an -/// event-sink callback legitimately can, on other seams in this file). -class SynchronizedEventLog -{ -public: - void add(const CasEvent & e) - { - std::lock_guard lock(mutex); - events.push_back(e); - } - std::vector snapshot() const - { - std::lock_guard lock(mutex); - return events; - } -private: - mutable std::mutex mutex; - std::vector events; -}; +/// doesn't reliably catch an unsynchronized push_back/iterator-read pair on a small vector), and even a +/// mutex-guarded one declared as a plain local is not enough on its own: a background publish can hold +/// an extra `shared_from_this()` past this frame's return, so the log itself must be heap-owned too. +/// `DB::Cas::tests::SharedEventLog` is exactly this shape (push under lock, snapshot copies out under +/// lock so a caller iterating the result never holds the mutex across a callback into the pool). +using DB::Cas::tests::SharedEventLog; template PoolPtr openPool(const std::shared_ptr & backend, CasRequestBudget budget = {}) @@ -2030,7 +2015,10 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) const CasRequestBudget budget = wedgeTestBudget(); auto backend = std::make_shared(); - SynchronizedEventLog seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto store = openPool(backend, budget); auto clock = VirtualRetryClock::installOn(store); const Layout & layout = store->layout(); @@ -2038,7 +2026,10 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) publishEmptyPart(store, ns, "x"); publishEmptyPart(store, ns, "y"); - store->setEventSink([&](const CasEvent & e) { seen.add(e); }); + store->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); /// Wedge the lane: every attempt of the log create is unresolved and nothing lands. backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; @@ -2070,7 +2061,7 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) EXPECT_EQ(store->scheduleRemountCallCountForTest(), 1u) << "reportImpossibleInterference must have called scheduleRemount exactly once"; - const std::vector observed = seen.snapshot(); + const std::vector observed = seen->snapshot(); const auto has_event = std::any_of(observed.begin(), observed.end(), [](const CasEvent & e) { return e.type == CasEventType::ForeignInterference; }); EXPECT_TRUE(has_event) << "a ForeignInterference CasEvent must be audited"; @@ -2082,13 +2073,19 @@ TEST(CASAnomalyPolicy, ForeignBytesAtWedgeKeyTripFenceAndRemount) TEST(CASAnomalyPolicy, NonReadyAtNewIdAllocationFaultsAndFailsClosed) { auto backend = std::make_shared(); - SynchronizedEventLog seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto store = openPool(backend); const Layout & layout = store->layout(); const RootNamespace ns{"srv1/wedge_contract"}; publishEmptyPart(store, ns, "x"); - store->setEventSink([&](const CasEvent & e) { seen.add(e); }); + store->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); store->setRefPreCarveHookForTest([&] { @@ -2137,7 +2134,7 @@ TEST(CASAnomalyPolicy, NonReadyAtNewIdAllocationFaultsAndFailsClosed) EXPECT_EQ(store->scheduleRemountCallCountForTest(), 1u) << "reportImpossibleInterference must have called scheduleRemount exactly once"; - const std::vector observed = seen.snapshot(); + const std::vector observed = seen->snapshot(); const auto has_event = std::any_of(observed.begin(), observed.end(), [](const CasEvent & e) { return e.type == CasEventType::ForeignInterference; }); EXPECT_TRUE(has_event) << "a ForeignInterference CasEvent must be audited"; @@ -3567,12 +3564,18 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) ++(*mount_wait_calls); *fake_now += ms; }; - SynchronizedEventLog seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto successor = openPoolWithConfig(backend, config); EXPECT_GT(mount_wait_calls->load(), 0u) << "the unclean predecessor must exercise the injected mount-observation wait"; - successor->setEventSink([&](const CasEvent & e) { seen.add(e); }); + successor->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); /// The boot clock this test drives the sweep backoff on is its own; the request engine gets a /// separate advancing clock, or an armed fault reissues for ever instead of ending its call. auto clock = VirtualRetryClock::installOn(successor); @@ -3628,7 +3631,7 @@ TEST(CASRefWriterStalePrecommitSweep, FailedSweepRearmsAndRetriesUntilClean) /// Audit (INTROSPECTION-1): exactly ONE `precommit_reclaim` event per reclaimed stale binding -- /// this is what makes the S13 card's "abandoned precommits reclaimed" counter falsifiable. std::vector reclaimed_refs; - for (const CasEvent & e : seen.snapshot()) + for (const CasEvent & e : seen->snapshot()) if (e.type == CasEventType::PrecommitReclaim) reclaimed_refs.push_back(e.ref_name); std::sort(reclaimed_refs.begin(), reclaimed_refs.end()); @@ -3650,9 +3653,15 @@ TEST(CASRefWriterStalePrecommitSweep, VerifiedCleanSweepClearsFlagWithoutEvents) publishEmptyPart(predecessor, ns, "committed_x"); /// committed work only; nothing dangles } - SynchronizedEventLog seen; /// declared BEFORE the Pool so it outlives the background syncer's emits (ASan 2026-07-09) + /// Heap-owned, not a plain local: declaring it before the Pool (ASan 2026-07-09) only protects + /// against an ordinary same-thread unwind, not a detached background completion holding an extra + /// `shared_from_this()` that can still be running on another thread after this frame returns. + auto seen = std::make_shared(); auto successor = openPool(backend); - successor->setEventSink([&](const CasEvent & e) { seen.add(e); }); + successor->setEventSink([seen](const CasEvent & e) + { + seen->push(e); + }); const uint64_t deferred_before = ProfileEvents::global_counters[ProfileEvents::CASRefSweepDeferred].load(); const uint64_t reclaimed_before = global_counters[ProfileEvents::CASRefStalePrecommitsReclaimed].load(); @@ -3661,7 +3670,7 @@ TEST(CASRefWriterStalePrecommitSweep, VerifiedCleanSweepClearsFlagWithoutEvents) << "a clean first pass IS the verified-clean sweep: the flag clears without any removal"; EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRefSweepDeferred].load(), deferred_before); EXPECT_EQ(global_counters[ProfileEvents::CASRefStalePrecommitsReclaimed].load(), reclaimed_before); - const std::vector observed = seen.snapshot(); + const std::vector observed = seen->snapshot(); EXPECT_EQ(std::count_if(observed.begin(), observed.end(), [](const CasEvent & e) { return e.type == CasEventType::PrecommitReclaim; }), 0); } diff --git a/src/Disks/tests/gtest_cas_retirement_sweep.cpp b/src/Disks/tests/gtest_cas_retirement_sweep.cpp index 57ce4d5c4d76..95d85c907740 100644 --- a/src/Disks/tests/gtest_cas_retirement_sweep.cpp +++ b/src/Disks/tests/gtest_cas_retirement_sweep.cpp @@ -12,6 +12,7 @@ #include #include +#include #include #include #include @@ -47,6 +48,7 @@ namespace DB::ErrorCodes using namespace DB::Cas; using DB::Cas::tests::idOf; +using DB::Cas::tests::SharedWaitLog; using DB::Cas::tests::u128Of; namespace @@ -352,18 +354,28 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS /// What the request engine reserves per attempt is the BACKEND's attempt timeout, not the budget /// field alone; pair the two so the mount lease's admission arithmetic sees what the budget claims. backend->setAttemptTimeoutMs(budget.attempt_timeout_ms); - uint64_t fake_boot = 1'000'000; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(1'000'000); + auto waits = std::make_shared(); /// The append's own retry clock and its sleep log, installed further down; declared here, before /// the store, because the store's teardown still calls the now-function they back. - uint64_t fake_retry = 0; - std::vector retry_sleeps; + auto fake_retry = std::make_shared>(0); + auto retry_sleeps = std::make_shared(); auto store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_root_id = "test", .mount_lease_ttl_ms = std::chrono::milliseconds(30000), .cas_request_budget = budget, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, }); ASSERT_TRUE(store); const Layout & layout = store->layout(); @@ -387,11 +399,14 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS /// give-up is the append's own retry window -- paced on ITS OWN virtual clock, separate from /// `fake_boot` (the mount fence's), so the standard policy's full window is available to reissue /// against rather than being cut short by the 30s lease `fake_boot` also measures. - store->setCasRequestNowFnForTest([&fake_retry] { return fake_retry; }); - store->setCasRetrySleepForTest([&fake_retry, &retry_sleeps](uint64_t ms) + store->setCasRequestNowFnForTest([fake_retry] + { + return fake_retry->load(); + }); + store->setCasRetrySleepForTest([fake_retry, retry_sleeps](uint64_t ms) { - fake_retry += ms + 1; - retry_sleeps.push_back(ms); + *fake_retry += ms + 1; + retry_sleeps->push(ms); }); backend->fault_key_substr = layout.namespaceStreamPrefix(DB::Cas::tests::fixture::fixtureLife(ns)) + "_log/"; DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::NETWORK_ERROR, [&] { store->dropRef(ns, "x"); }); @@ -399,8 +414,8 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS EXPECT_GT(backend->fault_hits, 1) << "the append must have reissued more than once against the persistent fault before giving up " "-- a single attempt would not distinguish this from a non-retrying policy"; - EXPECT_GT(retry_sleeps.size(), 1u) << "more than one paced retry must have occurred before the give-up"; - EXPECT_GT(fake_retry, 0u) << "the retry clock must have advanced past the policy's own deadline"; + EXPECT_GT(retry_sleeps->size(), 1u) << "more than one paced retry must have occurred before the give-up"; + EXPECT_GT(fake_retry->load(), 0u) << "the retry clock must have advanced past the policy's own deadline"; /// The id the straggler would occupy: one past the greatest record that is actually durable in the /// dying epoch. That is also, by construction, where the recovery seal goes. @@ -411,11 +426,11 @@ TEST(CASRetirementSweep, AStragglerFromTheDyingEpochLosesItsCreateToTheRecoveryS << "the slot must be empty before recovery -- otherwise this test proves nothing about who won"; /// Fence and remount. No wait: this is the case that used to cost 30 seconds. - fake_boot += 30001; + *fake_boot += 30001; fenceOutMount(*backend, layout.mountKey("test")); ASSERT_TRUE(store->tryRemountOnce()); ASSERT_EQ(store->liveWriterEpoch(), 2u); - EXPECT_TRUE(waits.empty()) + EXPECT_TRUE(waits->empty()) << "the remount blocked on an operator-configured wait; the grace is supposed to be gone"; /// Touch the namespace so it re-recovers under the new epoch: the walk closes epoch 1 in band. The diff --git a/src/Disks/tests/gtest_cas_writer_duties.cpp b/src/Disks/tests/gtest_cas_writer_duties.cpp index 576a8b2728dc..6706df7e0109 100644 --- a/src/Disks/tests/gtest_cas_writer_duties.cpp +++ b/src/Disks/tests/gtest_cas_writer_duties.cpp @@ -9,6 +9,7 @@ #include +#include #include #include #include @@ -20,6 +21,7 @@ extern const int NETWORK_ERROR; } using namespace DB::Cas; +using DB::Cas::tests::SharedWaitLog; namespace { @@ -403,8 +405,11 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem EXPECT_NE(decodeMountLease(mount->bytes).min_active_build_sequence, std::numeric_limits::max()) << "a live writer-cleanup duty forbids the clean-release certificate"; - uint64_t fake_boot = 0; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(0); + auto waits = std::make_shared(); auto successor_store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), @@ -413,11 +418,18 @@ TEST(CASWriterDuties, PendingDutySkipsCleanFarewellAndSuccessorSweepsTheCrashRem .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = budget, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, }); ASSERT_GT(successor_store->writerEpoch(), predecessor_epoch); - ASSERT_FALSE(waits.empty()) << "the predecessor supplied no clean-death certificate"; + ASSERT_FALSE(waits->empty()) << "the predecessor supplied no clean-death certificate"; ManifestId successor_id; auto successor = stageEmptyManifest(successor_store, ns, "successor", successor_id); @@ -465,7 +477,10 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) /// lease, and on a loaded machine the give-up below stops being a retry give-up and becomes a /// no-budget refusal before the first attempt. Freeze the boot clock, exactly as the successor /// pool further down already does, so the only bound on that give-up is the one it asserts. - uint64_t predecessor_boot = 0; + /// Captured by value: `predecessor_boot` is never mutated in this test, and the Pool can outlive + /// this stack frame (a background publish holds `shared_from_this()`), so a by-reference capture + /// would dangle. + const uint64_t predecessor_boot = 0; auto predecessor = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), @@ -477,7 +492,10 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = budget, - .boot_ms_fn = [&] { return predecessor_boot; }, + .boot_ms_fn = [] + { + return predecessor_boot; + }, }); auto clock = DB::Cas::tests::VirtualRetryClock::installOn(predecessor); @@ -512,8 +530,11 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) rejected.reset(); predecessor.reset(); - uint64_t fake_boot = 0; - std::vector waits; + /// Held in shared, heap-owned state, not plain locals: the hooks below mutate them, and the Pool + /// can outlive this stack frame (a background publish holds `shared_from_this()`), so a + /// by-reference capture of a local would dangle. + auto fake_boot = std::make_shared>(0); + auto waits = std::make_shared(); auto successor_store = Pool::open(backend, PoolConfig{ .pool_prefix = "p", .server_id = UInt128(1), @@ -525,11 +546,18 @@ TEST(CASWriterDuties, RejectedAttemptBodyIsEventuallyNominatedAndSwept) .mount_lease_ttl_ms = std::chrono::milliseconds(500), .mount_renew_period = std::chrono::milliseconds(100), .cas_request_budget = budget, - .boot_ms_fn = [&] { return fake_boot; }, - .wait_sleep_fn = [&](uint64_t ms) { fake_boot += ms; waits.push_back(ms); }, + .boot_ms_fn = [fake_boot] + { + return fake_boot->load(); + }, + .wait_sleep_fn = [fake_boot, waits](uint64_t ms) + { + *fake_boot += ms; + waits->push(ms); + }, }); ASSERT_GT(successor_store->writerEpoch(), predecessor_epoch); - ASSERT_FALSE(waits.empty()) << "the predecessor supplied no clean-death certificate"; + ASSERT_FALSE(waits->empty()) << "the predecessor supplied no clean-death certificate"; /// An ordinary successor mutation both drains the inherited duty as a no-op (the rejected grant /// was never durable) and forces the predecessor's dead epoch to close with an arithmetic seal -- From 96fe21d294bbf860fb589df8fd8b9b53962bb3e6 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 11:54:40 +0200 Subject: [PATCH 74/81] cas: the connect-cap gtests prove the cap without a clock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `CASEnvelopeWiring` stalled a real TCP connect and measured elapsed time (`EXPECT_LT(capped_elapsed, 1000)`), which failed under MSan at 1.5–2.5 s and would fail on any slow host. Timing bounds, ratios or sanitizer-gated asserts only move the threshold. The tests now dispatch through the production path against an ordinary mock server and assert two clock-free facts: the single-attempt client cache holds exactly the (attempt timeout, frozen cap) key the request must have used (`hasSingleAttemptClientForTest`, a test-only const accessor that inspects the cache and never creates a clone), and that clone's `getClientConfiguration().connectTimeoutMs` equals the cap while the Default client keeps the base value. A wrong dispatch fails immediately. That `PocoHTTPClient` applies `connectTimeoutMs` to the socket is upstream behaviour and is not re-proved here. The connect-stall helper and its `tcp_abort_on_overflow` skip logic are gone; the shared `DelayedResponseServer` fixture disables HTTP keep-alive so a pooled connection cannot outlive the ephemeral-port server that served it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../ObjectStorages/S3/S3ObjectStorage.cpp | 6 + .../ObjectStorages/S3/S3ObjectStorage.h | 5 + .../gtest_cas_s3_single_attempt_client.cpp | 346 +++++++----------- 3 files changed, 135 insertions(+), 222 deletions(-) diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 9b0000dc9ad0..6eb155dd4c44 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -1255,6 +1255,12 @@ std::shared_ptr S3ObjectStorage::getSingleAttemptClient(uint64 return clone; } +bool S3ObjectStorage::hasSingleAttemptClientForTest(uint64_t request_timeout_ms, uint64_t connect_timeout_cap_ms) const +{ + std::lock_guard lock(single_attempt_client_mutex); + return single_attempt_clients.contains(std::make_pair(request_timeout_ms, connect_timeout_cap_ms)); +} + std::shared_ptr S3ObjectStorage::clientForRetryProfile(const ObjectStorageControlRequest & request) const { /// getSingleAttemptClient is only invoked when actually selected, so an ordinary request never diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 335ad74b716a..5f5903f2c6aa 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -212,6 +212,11 @@ class S3ObjectStorage : public IObjectStorage /// pair, so two callers asking for the same request timeout but different caps get distinct clones. std::shared_ptr getSingleAttemptClient(uint64_t request_timeout_ms, uint64_t connect_timeout_cap_ms = 0) const; + /// True iff a clone for exactly this (request timeout, connect cap) pair is already cached -- + /// never builds one. Lets a test prove dispatch used a SPECIFIC key (and no other) without ever + /// creating a clone itself and without measuring anything. + bool hasSingleAttemptClientForTest(uint64_t request_timeout_ms, uint64_t connect_timeout_cap_ms) const; + private: void removeObjectImpl(const StoredObject & object, bool if_exists); void removeObjectsImpl(const StoredObjects & objects, bool if_exists); diff --git a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp index a0b4e0f8e3be..30ee3192bc54 100644 --- a/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp +++ b/src/Disks/tests/gtest_cas_s3_single_attempt_client.cpp @@ -20,14 +20,12 @@ #include #include -#include #include #include #include #include #include #include -#include #include @@ -40,8 +38,6 @@ #include #include #include -#include -#include #include #include @@ -195,114 +191,6 @@ class DelayedResponseServer void resetRequestsSeen() { requests_seen = 0; } }; -/// A TCP listener whose accept queue is permanently full of connections it never accept()s: `backlog = -/// 1` requests a one-entry queue, but Linux's actual capacity for a given backlog is not exactly that -/// number (historically `backlog + 1`, and kernel-version-dependent besides), so a single pre-filled -/// connection is not reliably enough to make the very next connect attempt stall. Instead, this keeps -/// connecting -- each attempt bounded by a short timeout -- until an attempt itself times out: that IS -/// the proof the queue is now genuinely full, whatever slack the nominal backlog actually bought, and -/// every established connection up to that point is kept (never accepted) to hold the queue full for -/// the rest of this object's life. Once full, every FURTHER inbound SYN finds no room: Linux's default -/// `tcp_abort_on_overflow = 0` then just drops that SYN instead of answering it (no RST, no SYN-ACK), -/// so a connecting client's kernel silently retransmits in the background while the client's OWN -/// socket-connect timeout -- not the kernel's SYN retry timer -- is what actually bounds how long a -/// caller waits. Nothing here ever completes a handshake with a real peer, so a call against this -/// listener can only ever fail on CONNECT, never on request/response -- unlike `DelayedResponseServer` -/// above, which answers every request and so can only discriminate the request/response phase. -class ConnectStallServer -{ - Poco::Net::ServerSocket listener; - std::vector prefill_connections; - -public: - ConnectStallServer() : listener(Poco::Net::SocketAddress("127.0.0.1", 0), /*backlog=*/1) - { - /// The loop bound is only a safety net (the queue always fills well before it on Linux): without - /// one, an environment where the queue somehow never fills would hang the constructor forever. - for (size_t i = 0; i < 64; ++i) - { - Poco::Net::StreamSocket prefill; - try - { - prefill.connect(listener.address(), Poco::Timespan(200 * 1000)); - } - catch (const Poco::TimeoutException &) - { - return; - } - prefill_connections.push_back(prefill); - } - throw Poco::RuntimeException("ConnectStallServer: accept queue never filled"); - } - - std::string getUrl() const { return "http://" + listener.address().toString(); } -}; - -/// `ConnectStallServer` relies on Linux dropping the overflow SYN silently, which only happens while -/// `net.ipv4.tcp_abort_on_overflow` stays at its default 0; a host with it set to 1 resets the -/// connection instead, so the queue-full state this fixture depends on never actually stalls a connect. -/// An unreadable file is treated the same as "1": this is a fixture precondition, not the behaviour -/// under test, so silently assuming the default would risk fencing that at the fixture layer. -bool tcpAbortOnOverflowPreventsStallServer() -{ - std::ifstream sysctl_file("/proc/sys/net/ipv4/tcp_abort_on_overflow"); - char value = '\0'; - if (!(sysctl_file >> value)) - return true; - return value != '0'; -} - -/// A genuine `S3ObjectStorage` for the CONNECT-phase discriminator: `connect_timeout_ms` governs only -/// the base client's TCP connect deadline, while `requestTimeoutMs` is set far wider so a call against -/// `ConnectStallServer` can only ever fail on connect, never on request/response (the peer there never -/// completes a handshake at all, so no request is ever sent). Adaptive timeouts are disabled for the -/// same reason `makeDispatchStorageForTest` disables them: the adaptive strategy would shrink the first -/// attempt's own connect deadline below whatever this function configures. -std::shared_ptr makeConnectStallStorageForTest(const std::string & endpoint, long connect_timeout_ms) -{ - DB::RemoteHostFilter remote_host_filter; - DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( - "us-east-1", - remote_host_filter, - /* s3_max_redirects = */ 100, - DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, - /* s3_slow_all_threads_after_network_error = */ false, - /* s3_slow_all_threads_after_retryable_error = */ false, - /* enable_s3_requests_logging = */ false, - /* for_disk_s3 = */ true, - /* opt_disk_name = */ {}, - /* request_throttler = */ {}); - cfg.endpointOverride = endpoint; - cfg.connectTimeoutMs = connect_timeout_ms; - cfg.requestTimeoutMs = 30000; - cfg.s3_use_adaptive_timeouts = false; - auto client = DB::S3::ClientFactory::instance().create( - cfg, clientSettingsForTest(), "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); - return std::make_shared( - std::move(client), std::make_unique(), - DB::S3::URI(endpoint + "/test-bucket/"), DB::S3Capabilities{}, - DB::ObjectStorageKeyGeneratorPtr{}, "disk"); -} - -/// Runs `attempt`, expecting a connection-class failure -- a stalled connect is classified by -/// `PocoHTTPClient` as `Aws::Client::CoreErrors::NETWORK_CONNECTION` from the `Poco::TimeoutException` -/// its connect poll raises, never as a request/response error -- and returns how long it took. -template -std::chrono::milliseconds expectConnectFailureAndMeasure(F && attempt) -{ - const auto start = std::chrono::steady_clock::now(); - try - { - attempt(); - ADD_FAILURE() << "expected a connection failure, the call unexpectedly succeeded"; - } - catch (const DB::S3Exception & e) - { - EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::NETWORK_CONNECTION); - } - return std::chrono::duration_cast(std::chrono::steady_clock::now() - start); -} - /// A genuine `S3ObjectStorage` pointed at `endpoint`. `base_request_timeout_ms` is the base client's /// request AND connect timeout -- comfortably above the server's simulated delay, so a `Default` call /// succeeds. No SDK-level retry (`RetryStrategy{.max_retries = 0}`, @@ -330,6 +218,14 @@ std::shared_ptr makeDispatchStorageForTest(const std::strin /// first (short) deadline is the only one this client ever gets, which would time out well under /// `server_delay` regardless of `requestTimeoutMs`. Off, so `requestTimeoutMs` governs uniformly. cfg.s3_use_adaptive_timeouts = false; + /// Each `{ }` block below creates and destroys its OWN ephemeral-port server; the default 30 s + /// keep-alive would let the client pool a persistent connection that can outlive it. If a LATER + /// block's server happens to be assigned that same now-free port (routine under many back-to-back + /// server creations within one process), the pooled connection is reused against an unrelated dead + /// peer and the request fails with "Connection reset by peer" -- reproduced empirically by running + /// this file's dispatch tests together under `--gtest_repeat`. Disabling keep-alive forces a fresh + /// connection per request, which is what a short-lived test server should get anyway. + cfg.http_keep_alive_timeout = 0; auto client = DB::S3::ClientFactory::instance().create( cfg, clientSettingsForTest(), "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); return std::make_shared( @@ -633,19 +529,19 @@ TEST(CASEnvelopeWiring, ProductionDispatchSelectsTheFrozenSingleAttemptClientPer /// The test above proves production dispatch selects a short-REQUEST-timeout clone, but every server /// there answers every request -- it can never tell whether the frozen `connect_timeout_cap_ms` reaches /// the CONNECTION phase at all, only whether SOME clone with a short deadline was picked. This test -/// closes that gap with `ConnectStallServer`, which never completes a handshake with anyone: a call -/// against it can only fail on connect. Under `Default`, the base client's own 2000 ms connect timeout -/// governs; under `SingleAttempt` with a 100 ms `connect_timeout_cap_ms` and a much wider 5000 ms -/// `attempt_timeout_ms` (so the request/response budget, which this discriminator never reaches, is not -/// what is being measured), a dropped or ignored cap would fall back to the base client's 2000 ms -/// connect timeout -- making the SingleAttempt call take just as long as Default. The discrimination is -/// therefore specifically on the CAP, not merely on whether a SingleAttempt clone was selected at all. +/// closes that gap WITHOUT any wall-clock measurement or stalled connect: every call below goes through +/// production dispatch against an ordinary, immediately-answering server, so it can only prove two +/// clock-free facts. First, that dispatch built (or reused) the single-attempt clone under EXACTLY the +/// (attempt timeout, connect cap) key the request carried -- `hasSingleAttemptClientForTest` only +/// inspects `S3ObjectStorage`'s clone cache, it never creates an entry, so a wrong key or a missing clone +/// fails the assertion immediately rather than timing out. Second, that the clone found under that key +/// actually carries the cap as its `connectTimeoutMs`, while the Default profile's own client keeps the +/// disk's (wider) base connect timeout untouched. Whether Poco's HTTP client actually enforces +/// `connectTimeoutMs` at the socket level is `PocoHTTPClient`/`Poco::Net::HTTPClientSession` behaviour +/// upstream of this class, and is not re-proved here; `S3SingleAttemptClient.ConnectTimeoutIsCappedAndFrozen` +/// above pins the MIN/cache-key arithmetic `getSingleAttemptClient` applies in isolation. TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTime) { - if (tcpAbortOnOverflowPreventsStallServer()) - GTEST_SKIP() << "net.ipv4.tcp_abort_on_overflow is not 0 (or unreadable): ConnectStallServer " - "cannot reliably stall a connect on this host"; - (void)contextForTest(); // getThreadPoolWriter/BlobStorageLogWriter::create fall back to the global context constexpr long base_connect_timeout_ms = 2000; @@ -654,8 +550,14 @@ TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTim /// PUT: writeObject; the profile and cap ride on WriteSettings, not an ObjectStorageControlRequest. { - ConnectStallServer server; - auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); + DelayedResponseServer server(std::chrono::milliseconds(0), [](Poco::Net::HTTPServerResponse & response) + { + response.set("ETag", "\"put-etag\""); + response.setContentLength(0); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.send(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_connect_timeout_ms); auto put = [&](DB::ObjectStorageRetryProfile profile, uint64_t attempt_timeout_ms, uint64_t connect_cap_ms) { @@ -669,83 +571,84 @@ TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTim buffer->finalize(); }; - const auto default_elapsed = expectConnectFailureAndMeasure( - [&] { put(DB::ObjectStorageRetryProfile::Default, 0, 0); }); - EXPECT_GE(default_elapsed.count(), 1500); - - const auto capped_elapsed = expectConnectFailureAndMeasure([&] - { - put(DB::ObjectStorageRetryProfile::SingleAttempt, single_attempt_timeout_ms, single_attempt_connect_cap_ms); - }); - EXPECT_LT(capped_elapsed.count(), 1000); + EXPECT_NO_THROW(put(DB::ObjectStorageRetryProfile::Default, 0, 0)); + EXPECT_EQ(storage->getS3StorageClient()->getClientConfiguration().connectTimeoutMs, base_connect_timeout_ms) + << "the Default profile must dispatch on the disk's own client, unchanged"; + + EXPECT_NO_THROW(put(DB::ObjectStorageRetryProfile::SingleAttempt, single_attempt_timeout_ms, single_attempt_connect_cap_ms)); + ASSERT_TRUE(storage->hasSingleAttemptClientForTest(single_attempt_timeout_ms, single_attempt_connect_cap_ms)) + << "dispatch must have built the single-attempt clone under exactly this (attempt timeout, cap) key"; + EXPECT_FALSE(storage->hasSingleAttemptClientForTest(single_attempt_timeout_ms, 0)) + << "dispatch must not fall back to an uncapped clone for this attempt timeout"; + EXPECT_EQ( + storage->getSingleAttemptClient(single_attempt_timeout_ms, single_attempt_connect_cap_ms) + ->getClientConfiguration().connectTimeoutMs, + static_cast(single_attempt_connect_cap_ms)); } /// HEAD: tryGetObjectMetadataWithNativeToken's ObjectStorageControlRequest-taking overload. { - ConnectStallServer server; - auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); - - const auto default_elapsed = expectConnectFailureAndMeasure([&] + DelayedResponseServer server(std::chrono::milliseconds(0), [](Poco::Net::HTTPServerResponse & response) { - storage->tryGetObjectMetadataWithNativeToken("head-key", /*with_tags=*/false, DB::ObjectStorageControlRequest{}); + response.set("ETag", "\"head-etag\""); + response.setContentLength(5); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.send(); }); - EXPECT_GE(default_elapsed.count(), 1500); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_connect_timeout_ms); - const auto capped_elapsed = expectConnectFailureAndMeasure([&] - { - storage->tryGetObjectMetadataWithNativeToken( - "head-key", /*with_tags=*/false, - DB::ObjectStorageControlRequest{ - .profile = DB::ObjectStorageRetryProfile::SingleAttempt, - .attempt_timeout_ms = single_attempt_timeout_ms, - .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); - }); - EXPECT_LT(capped_elapsed.count(), default_elapsed.count()) - << "the cap must remove SOME of the connect budget, unconditionally"; - /// A sanitizer build adds a roughly constant addend to both measurements, so the PRIMARY fence - /// is the DIFFERENCE the cap made, not an absolute bound: at least half of the connect budget it - /// removed. - EXPECT_GE(default_elapsed.count() - capped_elapsed.count(), - (base_connect_timeout_ms - static_cast(single_attempt_connect_cap_ms)) / 2); -#if !defined(DEBUG_OR_SANITIZER_BUILD) - /// Release builds keep the original tighter absolute bound too: sanitizer instrumentation - /// overhead is the only reason it was loosened to a difference above. - EXPECT_LT(capped_elapsed.count(), 1000); -#endif + EXPECT_TRUE(storage->tryGetObjectMetadataWithNativeToken( + "head-key", /*with_tags=*/false, DB::ObjectStorageControlRequest{}).has_value()); + EXPECT_EQ(storage->getS3StorageClient()->getClientConfiguration().connectTimeoutMs, base_connect_timeout_ms) + << "the Default profile must dispatch on the disk's own client, unchanged"; + + EXPECT_TRUE(storage->tryGetObjectMetadataWithNativeToken( + "head-key", /*with_tags=*/false, + DB::ObjectStorageControlRequest{ + .profile = DB::ObjectStorageRetryProfile::SingleAttempt, + .attempt_timeout_ms = single_attempt_timeout_ms, + .connect_timeout_cap_ms = single_attempt_connect_cap_ms}).has_value()); + ASSERT_TRUE(storage->hasSingleAttemptClientForTest(single_attempt_timeout_ms, single_attempt_connect_cap_ms)) + << "dispatch must have built the single-attempt clone under exactly this (attempt timeout, cap) key"; + EXPECT_FALSE(storage->hasSingleAttemptClientForTest(single_attempt_timeout_ms, 0)) + << "dispatch must not fall back to an uncapped clone for this attempt timeout"; + EXPECT_EQ( + storage->getSingleAttemptClient(single_attempt_timeout_ms, single_attempt_connect_cap_ms) + ->getClientConfiguration().connectTimeoutMs, + static_cast(single_attempt_connect_cap_ms)); } /// Conditional DELETE: removeObjectIfTokenMatches's ObjectStorageControlRequest-taking overload. { - ConnectStallServer server; - auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); - - const auto default_elapsed = expectConnectFailureAndMeasure([&] + DelayedResponseServer server(std::chrono::milliseconds(0), [](Poco::Net::HTTPServerResponse & response) { - storage->removeObjectIfTokenMatches(DB::StoredObject("delete-key"), "\"etag\"", DB::ObjectStorageControlRequest{}); + response.setStatus(Poco::Net::HTTPResponse::HTTP_NO_CONTENT); + response.setContentLength(0); + response.send(); }); - EXPECT_GE(default_elapsed.count(), 1500); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_connect_timeout_ms); - const auto capped_elapsed = expectConnectFailureAndMeasure([&] - { - storage->removeObjectIfTokenMatches( - DB::StoredObject("delete-key"), "\"etag\"", - DB::ObjectStorageControlRequest{ - .profile = DB::ObjectStorageRetryProfile::SingleAttempt, - .attempt_timeout_ms = single_attempt_timeout_ms, - .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); - }); - EXPECT_LT(capped_elapsed.count(), default_elapsed.count()) - << "the cap must remove SOME of the connect budget, unconditionally"; - /// A sanitizer build adds a roughly constant addend to both measurements, so the PRIMARY fence - /// is the DIFFERENCE the cap made, not an absolute bound: at least half of the connect budget it - /// removed. - EXPECT_GE(default_elapsed.count() - capped_elapsed.count(), - (base_connect_timeout_ms - static_cast(single_attempt_connect_cap_ms)) / 2); -#if !defined(DEBUG_OR_SANITIZER_BUILD) - /// Release builds keep the original tighter absolute bound too: sanitizer instrumentation - /// overhead is the only reason it was loosened to a difference above. - EXPECT_LT(capped_elapsed.count(), 1000); -#endif + const auto default_result = storage->removeObjectIfTokenMatches( + DB::StoredObject("delete-key"), "\"etag\"", DB::ObjectStorageControlRequest{}); + EXPECT_EQ(default_result.outcome, DB::ConditionalRemoveOutcome::Removed); + EXPECT_EQ(storage->getS3StorageClient()->getClientConfiguration().connectTimeoutMs, base_connect_timeout_ms) + << "the Default profile must dispatch on the disk's own client, unchanged"; + + const auto capped_result = storage->removeObjectIfTokenMatches( + DB::StoredObject("delete-key"), "\"etag\"", + DB::ObjectStorageControlRequest{ + .profile = DB::ObjectStorageRetryProfile::SingleAttempt, + .attempt_timeout_ms = single_attempt_timeout_ms, + .connect_timeout_cap_ms = single_attempt_connect_cap_ms}); + EXPECT_EQ(capped_result.outcome, DB::ConditionalRemoveOutcome::Removed); + ASSERT_TRUE(storage->hasSingleAttemptClientForTest(single_attempt_timeout_ms, single_attempt_connect_cap_ms)) + << "dispatch must have built the single-attempt clone under exactly this (attempt timeout, cap) key"; + EXPECT_FALSE(storage->hasSingleAttemptClientForTest(single_attempt_timeout_ms, 0)) + << "dispatch must not fall back to an uncapped clone for this attempt timeout"; + EXPECT_EQ( + storage->getSingleAttemptClient(single_attempt_timeout_ms, single_attempt_connect_cap_ms) + ->getClientConfiguration().connectTimeoutMs, + static_cast(single_attempt_connect_cap_ms)); } } @@ -757,29 +660,32 @@ TEST(CASEnvelopeWiring, ProductionDispatchAppliesTheFrozenConnectCapAtConnectTim /// constructor (the backend handoff at ~812-822) exactly as a writable Native mount does. This test /// drives that whole chain end to end -- real client -> freezeConnectTimeoutCapMs -> ObjectStorageBackend /// -> CasRequests/CasOperation -> the SAME production S3ObjectStorage dispatch the tests above cover -- -/// with no recording subclass anywhere in it. `Pool::open` itself is not driven here: it needs a live -/// store (PoolMeta creation/validation) that a stalled-connect endpoint cannot provide, so the backend -/// composition above is the reachable end of the chain from a unit test. +/// with no recording subclass anywhere in it, and, like the test above, with no wall-clock measurement: +/// both backends' HEAD goes through an ordinary, immediately-answering server. /// -/// A read-only backend (`single_attempt_control_plane_ = false`, matching `openPoolView`'s own choice -/// for a read-only mount) keeps the storage's DEFAULT client for its read-class requests -- the base -/// 2000 ms connect timeout -- as the uncapped control. The SAME derived cap and attempt timeout, handed -/// to a WRITABLE Native backend exactly as `openPoolView` constructs one, must then fail an order of -/// magnitude faster: a dropped or corrupted handoff anywhere in the chain would silently fall back to -/// the uncapped control's timing instead. +/// A read-only backend (`single_attempt_control_plane_ = false`, matching `openPoolView`'s own choice for +/// a read-only mount) dispatches its read-class requests under the Default profile -- proven here by the +/// storage never having built ANY single-attempt clone afterward, i.e. it used the disk's own client +/// untouched. The SAME derived cap and attempt timeout, handed to a WRITABLE Native backend exactly as +/// `openPoolView` constructs one, must then dispatch under EXACTLY that (attempt timeout, cap) key, and +/// the clone found under that key must carry the cap as its `connectTimeoutMs`: a dropped or corrupted +/// handoff anywhere in the chain would either leave no clone under that key or leave one with the wrong +/// timeout, and either way the assertion below fails immediately rather than by timing out. TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapReachesTheBackendOverProductionDispatch) { - if (tcpAbortOnOverflowPreventsStallServer()) - GTEST_SKIP() << "net.ipv4.tcp_abort_on_overflow is not 0 (or unreadable): ConnectStallServer " - "cannot reliably stall a connect on this host"; - (void)contextForTest(); constexpr long base_connect_timeout_ms = 2000; constexpr uint64_t cas_attempt_timeout_ms = 100; - ConnectStallServer server; - auto storage = makeConnectStallStorageForTest(server.getUrl(), base_connect_timeout_ms); + DelayedResponseServer server(std::chrono::milliseconds(0), [](Poco::Net::HTTPServerResponse & response) + { + response.set("ETag", "\"head-etag\""); + response.setContentLength(5); + response.setStatus(Poco::Net::HTTPResponse::HTTP_OK); + response.send(); + }); + auto storage = makeDispatchStorageForTest(server.getUrl(), base_connect_timeout_ms); /// The exact derivation `ContentAddressedMetadataStorage::openPoolView` uses: min(base connect /// timeout, attempt timeout) = 100 here, never the wide 2000 ms base timeout. @@ -790,12 +696,12 @@ TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapReachesTheBackendOverProductionDi auto uncapped_backend = std::make_shared( storage, DB::Cas::ObjectStorageBackend::Mode::Native, /*single_attempt_control_plane_=*/false, /*attempt_timeout_ms_=*/0, /*connect_timeout_cap_ms_=*/0); - std::chrono::milliseconds uncapped_elapsed{}; { DB::Cas::CasRequests requests(DB::Cas::BackendPtr(uncapped_backend), DB::Cas::Fence::open()); auto op = requests.admit(); - uncapped_elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); - EXPECT_GE(uncapped_elapsed.count(), 1500); + EXPECT_TRUE(op.head("k", DB::Cas::Retry::once()).has_value()); + EXPECT_FALSE(storage->hasSingleAttemptClientForTest(0, 0)) + << "a read-only (Default-profile) backend must never build a single-attempt clone"; } /// The derived cap, handed to the backend exactly as `openPoolView` constructs it (:812-822) for a @@ -803,26 +709,22 @@ TEST(CASEnvelopeWiring, FreezeConnectTimeoutCapReachesTheBackendOverProductionDi auto capped_backend = std::make_shared( storage, DB::Cas::ObjectStorageBackend::Mode::Native, /*single_attempt_control_plane_=*/true, cas_attempt_timeout_ms, *cap); - /// Deterministic, non-timing corroboration alongside the timing assertions below: cheap because - /// `ObjectStorageBackend` already exposes its own budget, though it only proves the constructor - /// argument the line above passed was stored -- not that it reached the S3 client's actual connect - /// timeout, which only the timing assertions below can show. + /// Cheap, deterministic corroboration alongside the dispatch-level assertions below: it proves the + /// constructor argument was stored, not that it reached the S3 client's actual connect timeout, which + /// only `hasSingleAttemptClientForTest`/`getSingleAttemptClient` below can show. EXPECT_EQ(capped_backend->connectTimeoutCapMs(), *cap); { DB::Cas::CasRequests requests(DB::Cas::BackendPtr(capped_backend), DB::Cas::Fence::open()); auto op = requests.admit(); - const auto elapsed = expectConnectFailureAndMeasure([&] { (void)op.head("k", DB::Cas::Retry::once()); }); - EXPECT_LT(elapsed.count(), uncapped_elapsed.count()) - << "the cap must remove SOME of the connect budget, unconditionally"; - /// A sanitizer build adds a roughly constant addend to both measurements, so the PRIMARY fence - /// is the DIFFERENCE the cap made, not an absolute bound: at least half of the connect budget it - /// removed. - EXPECT_GE(uncapped_elapsed.count() - elapsed.count(), (base_connect_timeout_ms - static_cast(*cap)) / 2); -#if !defined(DEBUG_OR_SANITIZER_BUILD) - /// Release builds keep the original tighter absolute bound too: sanitizer instrumentation - /// overhead is the only reason it was loosened to a difference above. - EXPECT_LT(elapsed.count(), 1000); -#endif + EXPECT_TRUE(op.head("k", DB::Cas::Retry::once()).has_value()); + ASSERT_TRUE(storage->hasSingleAttemptClientForTest(cas_attempt_timeout_ms, *cap)) + << "the WRITABLE backend must dispatch its read-class requests under exactly the frozen " + "(attempt timeout, cap) key"; + EXPECT_EQ( + storage->getSingleAttemptClient(cas_attempt_timeout_ms, *cap)->getClientConfiguration().connectTimeoutMs, + static_cast(*cap)) + << "socket-level enforcement of connectTimeoutMs is PocoHTTPClient behaviour upstream of this " + "class, not re-proved here"; } } From e5690c3c58c805109cd2ae9bc72e426df0282b3e Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 11:54:40 +0200 Subject: [PATCH 75/81] cas: test_cas_mount_renewal_retry stops depending on real time Two MSan failures in CI run 8 had the same root: the module ran with a 1 s lease TTL and a 50 ms attempt budget, so a single lost-response resolve read (1059 ms under MSan) fenced the renewal instead of resolving it, and the hard-restart test asserted the token-stability observation log line right after `start_clickhouse` returned while the CAS disk's `Pool::open` was still probing the mount object. One fixed budget for every build (`mount_lease_ttl_ms` 10000, renew period 2000, attempt timeout 500, connect cap 500, safety margin 500; both `validateCasRequestBudget` inequalities hold with wide margin), every expectation derived from those constants (the observation string is `ttl + ttl/20 + poll`), the hard-restart test waits for `system.cas_mounts` to report `live` before counting the line, and every `_wait_until` probe runs under one shared deadline that is recomputed before each query, HTTP control call and RustFS request (connect/read timeouts on the S3 client), rejecting late results. Assertions keep their meaning; the module runs in ~52 s locally. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../configs/storage_conf.xml | 21 +- .../test_cas_mount_renewal_retry/test.py | 181 ++++++++++++++---- 2 files changed, 155 insertions(+), 47 deletions(-) diff --git a/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml b/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml index 11ab2a4c08b6..2277a75b9e68 100644 --- a/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml +++ b/tests/integration/test_cas_mount_renewal_retry/configs/storage_conf.xml @@ -21,15 +21,18 @@ clickhouse clickhouse false - - 1000 - 200 - 50 - 50 + + 10000 + 2000 + 500 + 500 diff --git a/tests/integration/test_cas_mount_renewal_retry/test.py b/tests/integration/test_cas_mount_renewal_retry/test.py index 667a7f332896..42f09de3a090 100644 --- a/tests/integration/test_cas_mount_renewal_retry/test.py +++ b/tests/integration/test_cas_mount_renewal_retry/test.py @@ -6,6 +6,9 @@ import urllib.request import pytest +import urllib3 +from minio import Minio +from urllib3.util import Timeout as _Urllib3Timeout from helpers.cluster import ClickHouseCluster @@ -28,8 +31,27 @@ "CASRemountFailed", ) - -def _control(base_url, path, patch=None): +# The lease timing `configs/storage_conf.xml` compiles into `disk_cas_renewal`. One fixed budget for +# every build (no sanitizer-conditional scaling): wide enough that a single physical attempt, however +# slow the host, cannot plausibly cross the lease TTL (see the config file's own comment for the +# validateCasRequestBudget arithmetic), while still short enough that the hard-restart tests below +# observe the token-stability wait within a bounded test timeout. Mirrored here (rather than read back +# from the server) so every expectation string/number in this module derives from one place; keep both +# sides in sync with configs/storage_conf.xml. +MOUNT_LEASE_TTL_MS = 10000 +MOUNT_RENEW_PERIOD_MS = 2000 +ATTEMPT_TIMEOUT_MS = 500 +LEASE_SAFETY_MARGIN_MS = 500 + + +def _control(base_url, path, patch=None, timeout=10): + # `timeout` bounds each individual blocking socket operation (connect, then each read), not the + # wall-clock time to a fully-read response: a peer that trickles bytes in slowly enough to keep + # resetting the read timeout, without ever exceeding it, could still keep this call running past + # the caller's deadline. Accepted: the s3proxy control server this talks to answers in one small, + # immediate response with nothing in the path that could trickle, and every `_wait_until` probe in + # this module calls it at most once, so the worst case this leaves open is bounded and small -- not + # worth a cancellation thread for a well-behaved local test double. if patch is None: request = urllib.request.Request("{}{}".format(base_url, path)) else: @@ -39,26 +61,51 @@ def _control(base_url, path, patch=None): headers={"Content-Type": "application/json"}, method="POST", ) - with urllib.request.urlopen(request, timeout=10) as response: + with urllib.request.urlopen(request, timeout=timeout) as response: return json.loads(response.read().decode()) +class _Deadline: + """One absolute deadline, shared across a whole `_wait_until` call (every retry) AND across every + query one of its probes issues. `remaining()` must be re-read before EACH query in a probe that + issues more than one: reusing a single `remaining()` value across sequential queries would let + EACH one spend the full remaining budget, so an N-query probe could run up to Nx as long as the + caller intended before anything ever times out. + """ + + def __init__(self, timeout): + self._deadline = time.monotonic() + timeout + + def remaining(self): + return max(0.0, self._deadline - time.monotonic()) + + def expired(self): + return time.monotonic() >= self._deadline + + def _wait_until(probe, timeout=40, interval=0.2): - deadline = time.monotonic() + timeout + # `probe` is called with a shared `_Deadline` on every attempt, so it (and every query it issues) + # can bound its own work at `deadline.remaining()` instead of inheriting a client's much larger + # default -- otherwise one slow or stuck call could silently burn the whole budget of a caller that + # is itself waiting on a much tighter deadline. A truthy result that only comes back after this + # deadline has already passed (the probe ran long enough to blow through its own remaining budget) + # is rejected here too, rather than accepted as an on-time success. + deadline = _Deadline(timeout) last = None - while time.monotonic() < deadline: - last = probe() - if last: + while not deadline.expired(): + last = probe(deadline) + if last and not deadline.expired(): return last - time.sleep(interval) + time.sleep(min(interval, deadline.remaining())) raise AssertionError("condition did not become true within {}s; last={!r}".format(timeout, last)) -def _profile_events(node): +def _profile_events(node, timeout=None): rows = node.query( "SELECT event, value FROM system.events WHERE event IN ({}) FORMAT TSV".format( ", ".join("'{}'".format(event) for event in RENEWAL_EVENTS) - ) + ), + timeout=timeout, ) values = {event: 0 for event in RENEWAL_EVENTS} for row in rows.splitlines(): @@ -71,13 +118,14 @@ def _event_delta(before, after): return {event: after[event] - before[event] for event in RENEWAL_EVENTS} -def _mount_snapshot(node): +def _mount_snapshot(node, timeout=None): row = node.query( "SELECT renewal_sequence, state, lifecycle, gc_fenced " "FROM system.cas_mounts " "WHERE disk = '{}' AND server_root_id = '{}' LIMIT 1 FORMAT TSV".format( DISK, SERVER_ROOT_ID - ) + ), + timeout=timeout, ).strip() assert row, "the local CAS mount row must be visible" sequence, state, lifecycle, gc_fenced = row.split("\t") @@ -89,14 +137,53 @@ def _mount_snapshot(node): } -def _read_mount_object(): - response = cluster.rustfs_client.get_object(cluster.rustfs_bucket, MOUNT_OBJECT_KEY) +def _rustfs_client(timeout): + # A fresh, deadline-scoped client rather than the shared `cluster.rustfs_client`: that one's + # `http_client` (see wait_rustfs_to_start in helpers/cluster.py) has no configured timeout, so a + # stalled RustFS response through it could block a `_wait_until` probe past its own deadline + # without ever timing out on its own. Cheap to construct; only used for this module's polling reads. + # + # `timeout` bounds each individual blocking socket operation (connect, then each read) through this + # client, not the wall-clock time to a fully-read response: a peer trickling bytes slowly enough to + # keep resetting the read timeout, without ever exceeding it, could still run past the caller's + # deadline. Accepted: RustFS is a local, well-behaved test double (never observed to trickle), and + # every operation _read_mount_object issues through a client built here recomputes ITS OWN fresh + # timeout first, so the number of such operations per probe is fixed and small -- not worth a + # cancellation thread for a local test double. + return Minio( + "{}:{}".format(cluster.rustfs_ip, cluster.rustfs_port), + access_key=cluster.rustfs_access_key, + secret_key=cluster.rustfs_secret_key, + secure=False, + http_client=urllib3.PoolManager( + cert_reqs="CERT_NONE", + timeout=_Urllib3Timeout(connect=timeout, read=timeout), + ), + ) + + +def _read_mount_object(deadline=None): + # Three sequential RustFS operations (client construction for the GET, the GET/body-read, then + # client construction for the HEAD): `deadline.remaining()` is read again before EACH one rather + # than reused from the first, so a slow GET cannot silently gift the HEAD the same full budget + # again. Standalone callers (outside any `_wait_until` probe) get a fresh 20s deadline of their own. + if deadline is None: + deadline = _Deadline(20) + + remaining = deadline.remaining() + if remaining <= 0: + raise AssertionError("_read_mount_object: deadline already expired before the GET") + response = _rustfs_client(remaining).get_object(cluster.rustfs_bucket, MOUNT_OBJECT_KEY) try: body = response.read() finally: response.close() response.release_conn() - stat = cluster.rustfs_client.stat_object(cluster.rustfs_bucket, MOUNT_OBJECT_KEY) + + remaining = deadline.remaining() + if remaining <= 0: + raise AssertionError("_read_mount_object: deadline already expired before the HEAD") + stat = _rustfs_client(remaining).stat_object(cluster.rustfs_bucket, MOUNT_OBJECT_KEY) return body, stat.etag.strip('"') @@ -127,8 +214,10 @@ def _log_count_since_last_restart(node, pattern): return int(node.exec_in_container(["bash", "-c", script]).strip()) -def _renewal_log_rows(node, since): - node.query("SYSTEM FLUSH LOGS") +def _renewal_log_rows(node, since, deadline=None): + # Two sequential queries: `deadline.remaining()` is read again before the second one rather than + # reused from the first, so this whole call cannot spend twice the caller's remaining budget. + node.query("SYSTEM FLUSH LOGS", timeout=deadline.remaining() if deadline else None) rows = node.query( "SELECT outcome, detail['seq'], detail['write_attempt_id'], " "detail['attempts_sent'], detail['classification'] " @@ -138,7 +227,8 @@ def _renewal_log_rows(node, since): "AND event_time_microseconds >= toDateTime64('{}', 6) " "ORDER BY event_time_microseconds FORMAT TSV".format( DISK, SERVER_ROOT_ID, since - ) + ), + timeout=deadline.remaining() if deadline else None, ) return [tuple(row.split("\t")) for row in rows.splitlines() if row] @@ -171,7 +261,7 @@ def start_cluster(): cluster.base_cmd + ["port", "s3proxy", "8474"], text=True ).strip() control_url = "http://{}".format(binding) - _wait_until(lambda: _control(control_url, "/healthz"), timeout=30) + _wait_until(lambda deadline: _control(control_url, "/healthz", timeout=deadline.remaining()), timeout=30) _control(control_url, "/config", {"reset": True}) node = cluster.instances["node"] @@ -237,14 +327,16 @@ def test_transient_mount_renewal_retries_without_remount(start_cluster): }, ) - def recovered_snapshot(): + def recovered_snapshot(deadline): # Read the counter before the mount row: `CASMountRenewalRecovered` is incremented as soon as # the renewal decides its outcome, strictly before the mount row's `renewal_sequence` (and the # matching cas_log row) is updated to the new sequence. With the shortened renewal period a # background (fault-free) renewal can land between the two reads; reading counters first makes # the subsequent mount read very unlikely to still observe the pre-recovery sequence. - counters = _profile_events(node) - mount = _mount_snapshot(node) + # `deadline.remaining()` is read again for the second query rather than reused from the first, + # so this probe cannot spend twice its caller's remaining budget. + counters = _profile_events(node, timeout=deadline.remaining()) + mount = _mount_snapshot(node, timeout=deadline.remaining()) if ( mount["sequence"] > mount_before["sequence"] and counters["CASMountRenewalRecovered"] @@ -265,12 +357,12 @@ def recovered_snapshot(): # renewal period, background renewals can advance `system.cas_mounts` past the exact sequence this # recovery landed on before either of these two reads gets to it. rows = _wait_until( - lambda: ( + lambda deadline: ( found if any(row[0] == "recovered" for row in found) else None ) - if (found := _renewal_log_rows(node, since)) + if (found := _renewal_log_rows(node, since, deadline=deadline)) else None, timeout=20, ) @@ -335,24 +427,26 @@ def test_landed_response_lost_adopts_exact_mount_write(start_cluster): # background renewal that started immediately after this one resolved (see the mount_before/after # sequence race this replaced). Wait for the proxy's own record first, then poll the object for # its exact upstream_etag, so body_after is unambiguously the write this test is about. - def dropped_record(): - found_stats = _control(control_url, "/stats") + def dropped_record(deadline): + found_stats = _control(control_url, "/stats", timeout=deadline.remaining()) found_records = found_stats["drop_after_forward"] return (found_stats, found_records[0]) if len(found_records) == 1 else None stats, record = _wait_until(dropped_record) target_etag = record["upstream_etag"].strip('"') - def matching_object(): - body, token = _read_mount_object() + def matching_object(deadline): + body, token = _read_mount_object(deadline) return (body, token) if token == target_etag else None body_after, token_after = _wait_until(matching_object) mount_body = _decode_mount(body_after) - def resolved_snapshot(): - counters = _profile_events(node) - mount = _mount_snapshot(node) + def resolved_snapshot(deadline): + # `deadline.remaining()` is read again for the second query rather than reused from the + # first, so this probe cannot spend twice its caller's remaining budget. + counters = _profile_events(node, timeout=deadline.remaining()) + mount = _mount_snapshot(node, timeout=deadline.remaining()) if ( mount["sequence"] > mount_before["sequence"] and counters["CASMountRenewalResolved"] @@ -363,9 +457,9 @@ def resolved_snapshot(): return mount, counters return None - # Polling at the renewal's own 200 ms period (the default `interval`) can alias with it under a - # sanitizer build's slowdown -- a resolved renewal can land and be superseded by the next one - # between two polls. Poll faster than the cadence it observes, with a longer timeout to match. + # Polling at the renewal's own period (`MOUNT_RENEW_PERIOD_MS`) can alias with it on a slow host -- + # a resolved renewal can land and be superseded by the next one between two polls. Poll faster than + # the cadence it observes, with a longer timeout to match. mount_after, counters_after = _wait_until(resolved_snapshot, timeout=120, interval=0.05) _control(control_url, "/config", {"rate": 0.0}) delta = _event_delta(counters_before, counters_after) @@ -373,12 +467,12 @@ def resolved_snapshot(): # (see _renewal_log_rows): background renewals can advance `system.cas_mounts` past the exact # sequence this recovery landed on before either of these reads gets to it. rows = _wait_until( - lambda: ( + lambda deadline: ( found if any(row[0] == "recovered" and row[4] == "committed_by_read" for row in found) else None ) - if (found := _renewal_log_rows(node, since)) + if (found := _renewal_log_rows(node, since, deadline=deadline)) else None, timeout=20, ) @@ -425,8 +519,12 @@ def test_hard_restart_observes_then_the_unsafe_knob_skips_the_observation(start_ def log_count_since_last_restart(pattern): return _log_count_since_last_restart(node, pattern) - # 1150 = mountObservationThresholdMs(ttl_ms=1000, poll=max(1, period/2)=100): ttl + ttl/20 + poll. - observation = "waiting ~1150 ms (token-stability observation)" + # mountObservationThresholdMs(ttl_ms, poll=max(1, period_ms / 2)) = ttl_ms + ttl_ms / 20 + poll: + # derived from the module's own lease constants (see their definition above) rather than + # hard-coded, so this stays correct if that fixed budget ever changes. + poll_ms = max(1, MOUNT_RENEW_PERIOD_MS // 2) + threshold_ms = MOUNT_LEASE_TTL_MS + MOUNT_LEASE_TTL_MS // 20 + poll_ms + observation = "waiting ~{} ms (token-stability observation)".format(threshold_ms) epoch_before = int( node.query( "SELECT writer_epoch FROM system.cas_mounts WHERE disk = '{}' LIMIT 1".format(DISK) @@ -437,6 +535,11 @@ def log_count_since_last_restart(pattern): # token-stability observation wait once before it can safely reclaim it. node.stop_clickhouse(kill=True) node.start_clickhouse() + # `node.start_clickhouse()` only waits for the server to accept queries, not for the CAS disk's + # `Pool::open` (which performs the observation wait itself) to finish; reading the log or the mount + # row before that completes races the very thing being measured. Wait for the mount to report + # "live" first, then the log line and row are both settled. + _wait_until(lambda deadline: _mount_snapshot(node, timeout=deadline.remaining())["state"] == "live", timeout=120) assert log_count_since_last_restart(observation) == 1 assert _mount_snapshot(node)["state"] == "live" # This restart already reclaims the slot and advances the epoch on its own (via the observation @@ -460,6 +563,8 @@ def log_count_since_last_restart(pattern): ) try: node.start_clickhouse() + # Same race as the safe restart above: wait for the mount to settle before reading the log. + _wait_until(lambda deadline: _mount_snapshot(node, timeout=deadline.remaining())["state"] == "live", timeout=120) assert log_count_since_last_restart(observation) == 0 assert _mount_snapshot(node)["state"] == "live" epoch_after_knob_restart = int( From b21a9cc96fb13e8eab28426103e078734f5f619f Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 11:54:40 +0200 Subject: [PATCH 76/81] cas: the streaming-publish gtests in gtest_cas_backend.cpp cannot hang The two publish-visibility tests waited 2 s on a real `std::async` task (the tightest bare future wait in the suite), and a failed assertion before `release_source.set_value` unwound into the future's blocking destructor while the publisher waited forever. Every wait on both sides now has the file's 20 s bound, a scope guard releases the publisher on every exit path, an expired barrier is an explicit test failure instead of a silent release, and the publisher lambda's total runtime is bounded so a stuck publisher costs at most the sum of its internal bounds before the destructor returns. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_backend.cpp | 99 +++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 7 deletions(-) diff --git a/src/Disks/tests/gtest_cas_backend.cpp b/src/Disks/tests/gtest_cas_backend.cpp index 5d7604ff7e02..04052060d7eb 100644 --- a/src/Disks/tests/gtest_cas_backend.cpp +++ b/src/Disks/tests/gtest_cas_backend.cpp @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -323,15 +324,28 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB std::promise source_opened; std::promise release_source; const std::shared_future release = release_source.get_future().share(); + /// Set when `open_payload`'s own internal wait below times out instead of observing the release. + /// The timeout alone silently lets the publisher proceed either way -- this flag is what lets the + /// test body downstream (after calling `releaseOnce`) assert that the release actually reached the + /// publisher, so a regression that leaves it unreleased for the full bound FAILS instead of quietly + /// passing because the timeout eventually let it through anyway. + std::atomic release_wait_expired{false}; const BlobPublishRequest request{ .destination_key = "blob", .publication = StreamingBlobPublication{ .payload_size = 7, .fresh_envelope = "fresh-envelope", - .open_payload = [&source_opened, release] + .open_payload = [&source_opened, release, &release_wait_expired] { source_opened.set_value(); - release.wait(); + /// Bounded, not `.wait()`: even if the release guards below somehow never fire, this + /// lambda -- and therefore `publish()`, and therefore the `std::async` task wrapping it + /// -- must still return within a bounded time, so `publication`'s blocking destructor (a + /// `std::async` future's destructor blocks until its task finishes) can never hang the + /// whole process. This is the ONLY wait `open_payload` performs, so it also bounds + /// `publish()`'s total time to this 20s plus whatever negligible in-memory work follows. + if (release.wait_for(20s) != std::future_status::ready) + release_wait_expired = true; return std::make_unique(String("payload")); }}}; @@ -340,11 +354,38 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB /// racing on `op`. CasOperation publish_op = requests.admit(); auto publication = std::async(std::launch::async, [&] { publish_op.publish(request, Retry::once()); }); - source_opened.get_future().wait(); + + /// `publication` and (below) `observation` are both futures returned by `std::async`, so EACH one's + /// destructor blocks until its own task finishes. A failing ASSERT_*/EXPECT_* can unwind this + /// function while the publisher is still parked on `release`; this releases it promptly on that + /// exit rather than relying solely on the 20s bound above. Idempotent (the ordinary release near the + /// end sets `released` first) and declared right after `publication` so it protects every exit from + /// here on -- including the one right below, before `observation` exists. + bool released = false; + const auto releaseOnce = [&] + { + if (!released) + { + released = true; + release_source.set_value(); + } + }; + SCOPE_EXIT({ releaseOnce(); }); + + ASSERT_EQ(source_opened.get_future().wait_for(20s), std::future_status::ready) + << "publish() never reached open_payload"; CasOperation read_op = requests.admit(); auto observation = std::async(std::launch::async, [&] { return read_op.read("blob", Retry::once()); }); - const auto observation_status = observation.wait_for(2s); + /// A SECOND copy of the same guard, declared AFTER `observation` so it tears down BEFORE + /// `observation`'s own blocking destructor on any unwind past this point. Without it, a genuine + /// visibility-lock regression (exactly what this test exists to catch) would leave `read_op.read` + /// blocked on the same lock `publish_op.publish` holds while parked on `release`, and the FIRST + /// guard above -- which, being declared earlier, tears down only AFTER `observation`'s destructor -- + /// would then release the publisher too late to ever unblock that read. + SCOPE_EXIT({ releaseOnce(); }); + + const auto observation_status = observation.wait_for(20s); EXPECT_EQ(observation_status, std::future_status::ready) << "publication must not hold the visibility lock while draining its source"; if (observation_status == std::future_status::ready) @@ -354,7 +395,15 @@ TEST(CASInMemory, PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteB EXPECT_EQ(visible->bytes, "old-complete-body"); } - release_source.set_value(); + releaseOnce(); + /// `open_payload` above is the ONLY wait reachable from `publish_op.publish`, and it is itself + /// bounded to 20s: a stuck publisher therefore costs at most that 20s bound (plus negligible + /// in-memory work) before `publish()` returns and `publication`'s `std::async` destructor can + /// complete -- never an unbounded hang. `EXPECT_FALSE` below turns a timeout that silently released + /// the publisher into a visible test failure instead of a pass for the wrong reason. + ASSERT_EQ(publication.wait_for(20s), std::future_status::ready) << "publish() never completed after release"; + EXPECT_FALSE(release_wait_expired.load()) + << "open_payload's internal wait timed out instead of observing the release"; EXPECT_NO_THROW(publication.get()); const auto after = op.read("blob", Retry::once()); ASSERT_TRUE(after.has_value()); @@ -765,6 +814,12 @@ struct PublicationWriteBarrier std::promise opened; std::promise release; std::shared_future release_future = release.get_future().share(); + /// Set when the wait on `release_future` below times out instead of observing the release. The + /// timeout alone silently lets the write proceed either way -- this flag is what lets the test body + /// assert (after calling the release itself) that it actually reached the write, so a regression + /// that leaves it unreleased for the full bound FAILS instead of quietly passing because the + /// timeout eventually let it through anyway. + std::atomic release_wait_expired{false}; }; class PublicationRecordingLocalObjectStorage final : public DB::LocalObjectStorage @@ -792,7 +847,14 @@ class PublicationRecordingLocalObjectStorage final : public DB::LocalObjectStora if (write_barrier) { write_barrier->opened.set_value(); - write_barrier->release_future.wait(); + /// Bounded, not `.wait()`: even if a caller's release guard somehow never fires, this call + /// -- and therefore the `std::async` task wrapping the publish that reaches it -- must still + /// return within a bounded time, so that future's blocking destructor (a `std::async` + /// future's destructor blocks until its task finishes) can never hang the whole process. + /// This is the ONLY wait this write performs, so it also bounds the whole call's total time + /// to this 20s plus whatever negligible local-filesystem work follows. + if (write_barrier->release_future.wait_for(std::chrono::seconds(20)) != std::future_status::ready) + write_barrier->release_wait_expired = true; } return out; } @@ -938,12 +1000,35 @@ TEST(CASObjectStorageBackend, PublishBlobEmulatedKeepsDestinationCompleteUntilAt op.publish(streamingPublication(key, "fresh-envelope", "payload", 7), Retry::once()); }); - const auto opened_status = opened.wait_for(2s); + /// Same hazard as `PublishBlobKeepsThePreviousIncarnationVisibleUntilTheCompleteBodyIsReady`: an + /// exception unwinding out of this function (a failing ASSERT_*, or `readStorageObject` throwing) + /// while the write is still parked on `barrier->release_future.wait()` would deadlock `publication`'s + /// blocking `std::async` destructor against `barrier`'s own (later) teardown. Declared after + /// `publication` so it tears down FIRST, this guard releases the write unconditionally. + bool released = false; + SCOPE_EXIT({ + if (!released) + { + released = true; + barrier->release.set_value(); + } + }); + + const auto opened_status = opened.wait_for(20s); EXPECT_EQ(opened_status, std::future_status::ready); if (opened_status == std::future_status::ready) EXPECT_EQ(readStorageObject(storage, physical_key), "old-complete-body"); + released = true; barrier->release.set_value(); + /// The write inside `writeObject` is the ONLY wait reachable from `op.publish`, and it is itself + /// bounded to 20s: a stuck write therefore costs at most that 20s bound (plus negligible + /// local-filesystem work) before `publish()` returns and `publication`'s `std::async` destructor + /// can complete -- never an unbounded hang. `EXPECT_FALSE` below turns a timeout that silently + /// released the write into a visible test failure instead of a pass for the wrong reason. + ASSERT_EQ(publication.wait_for(20s), std::future_status::ready) << "publish() never completed after release"; + EXPECT_FALSE(barrier->release_wait_expired.load()) + << "the write's internal wait timed out instead of observing the release"; EXPECT_NO_THROW(publication.get()); EXPECT_EQ(storage->metadata_calls, 0u); EXPECT_EQ(readStorageObject(storage, physical_key), "fresh-envelopepayload"); From 8be3f301ecfc12f2d930616f35f675915fed2401 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 10:44:02 +0200 Subject: [PATCH 77/81] tests: pin index_granularity in 04299_cas_projection_inline_disk The test asserts `force_optimize_projection` on the normal projection `p_by_b` of two tables. Under a randomized MergeTree `index_granularity` (703 in the failing CI runs) a merge re-granulates the base part continuously but rebuilds the projection at the source parts' granule boundaries, so the projection ends up with more marks than the table and the planner correctly refuses it with `PROJECTION_NOT_USED`. Not a CAS bug: the same statements fail on a plain `MergeTree` on upstream 26.6.2. Pin `index_granularity = 8192, index_granularity_bytes = 10485760` on both tables, the same guard `04300_cas_projection_multiblock` already carries. Verified with clickhouse-local: `index_granularity = 703` reproduces the 584, the pinned tables use the projection after two inserts and an `OPTIMIZE FINAL`, and `SYSTEM CAS FORGET` still succeeds. Closes: https://github.com/Altinity/ClickHouse/issues/2325 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov --- .../0_stateless/04299_cas_projection_inline_disk.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/tests/queries/0_stateless/04299_cas_projection_inline_disk.sh b/tests/queries/0_stateless/04299_cas_projection_inline_disk.sh index ef8f68c4726b..fd2e24661fde 100755 --- a/tests/queries/0_stateless/04299_cas_projection_inline_disk.sh +++ b/tests/queries/0_stateless/04299_cas_projection_inline_disk.sh @@ -21,7 +21,12 @@ SETTINGS disk = disk( metadata_type = cas, cas_server_root_id = '${CLICKHOUSE_DATABASE}_04299', name = '${CLICKHOUSE_DATABASE}_04299_cas_projection', - path = '${CLICKHOUSE_DATABASE}_04299_cas_projection_pool/'); + path = '${CLICKHOUSE_DATABASE}_04299_cas_projection_pool/'), + -- The normal projection p_by_b is asserted with force_optimize_projection below; a merge + -- re-granulates the base part but rebuilds the projection at the source parts' boundaries, so a + -- randomized index_granularity (703 in CI) can leave the projection with more marks than the + -- table and the planner rightly refuses it. Pin the defaults, as 04300 does. + index_granularity = 8192, index_granularity_bytes = 10485760; INSERT INTO t_proj_cas SELECT number, number % 10 FROM numbers(1000); INSERT INTO t_proj_cas SELECT number, number % 10 FROM numbers(1000, 1000); @@ -58,7 +63,9 @@ DROP TABLE t_proj_cas; DROP TABLE IF EXISTS t_proj_cas_alter; CREATE TABLE t_proj_cas_alter (a UInt64, b UInt64, PROJECTION p_by_b (SELECT a, b ORDER BY b)) -ENGINE = MergeTree ORDER BY a; +ENGINE = MergeTree ORDER BY a +-- Same pin as t_proj_cas: the after_merge_reload_uses_projection check below forces p_by_b. +SETTINGS index_granularity = 8192, index_granularity_bytes = 10485760; INSERT INTO t_proj_cas_alter SELECT number, number % 10 FROM numbers(1000); INSERT INTO t_proj_cas_alter SELECT number, number % 10 FROM numbers(1000, 1000); From 711efad88b6bfafba7207b458e179c799f480beb Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 13:14:07 +0200 Subject: [PATCH 78/81] poco: make HTTPServerConnection's stop flag atomic `HTTPServerConnection::run` polls `_stopped` in its loop condition outside `_mutex`, while `onServerStopped` (reached from `HTTPServer::stopAll(true)`) writes it from the stopping thread. A plain `bool` is a data race; TSan reported it from the test servers that abort their connections on teardown (`base/poco/Net/src/HTTPServerConnection.cpp:154` vs `:61`). `std::atomic` keeps the exact semantics with sequentially consistent loads and stores. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov (cherry picked from commit 5340794a38066ce0d9fce854aaea6c84dd3bc0d4) --- base/poco/Net/include/Poco/Net/HTTPServerConnection.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/base/poco/Net/include/Poco/Net/HTTPServerConnection.h b/base/poco/Net/include/Poco/Net/HTTPServerConnection.h index f0faa1b8c719..8b15924cf07a 100644 --- a/base/poco/Net/include/Poco/Net/HTTPServerConnection.h +++ b/base/poco/Net/include/Poco/Net/HTTPServerConnection.h @@ -19,6 +19,7 @@ #include "Poco/Mutex.h" +#include #include "Poco/Net/HTTPRequestHandlerFactory.h" #include "Poco/Net/HTTPResponse.h" #include "Poco/Net/HTTPServerParams.h" @@ -56,7 +57,9 @@ namespace Net private: HTTPServerParams::Ptr _pParams; HTTPRequestHandlerFactory::Ptr _pFactory; - bool _stopped; + /// Written by onServerStopped() from the thread that calls HTTPServer::stopAll(true) while run() + /// polls it outside _mutex; a plain bool is a data race (TSan), an atomic keeps the same semantics. + std::atomic _stopped; Poco::FastMutex _mutex; }; From 4e57fd75e1ec465d6c25ade4a15b402b9e37cf92 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 14:14:21 +0200 Subject: [PATCH 79/81] cas: test S3 clients against ephemeral mock servers use one connection per request Every test in these files starts its own HTTP server on an ephemeral port and destroys it with the test. With keep-alive on, the process-wide connection pool can hand a later test a pooled connection to a port whose server is already gone, and the request fails with `Connection reset by peer`; reproduced with `--gtest_repeat=3` on `S3BulkDeleteFallback` (three tests failed in iteration 3 only). `http_keep_alive_timeout = 0`, the same setting `gtest_cas_s3_single_attempt_client.cpp` already uses for the same reason. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Only the bulk-delete fallback suite needs it: `makeNetworkFailingClient` in `gtest_cas_aws_s3_client.cpp` creates no mock server, so that file is left as is. Signed-off-by: Mikhail Filimonov --- src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp | 5 +++++ src/IO/S3/tests/gtest_cas_aws_s3_client.cpp | 3 +++ 2 files changed, 8 insertions(+) diff --git a/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp b/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp index e3c7590fa136..c0e4ee5f5306 100644 --- a/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp +++ b/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp @@ -201,6 +201,11 @@ std::shared_ptr makeStorageForTest(const std::string & endp cfg.connectTimeoutMs = 10000; cfg.requestTimeoutMs = 10000; cfg.s3_use_adaptive_timeouts = false; + /// Every test here starts its own server on an ephemeral port; with keep-alive on, the process-wide + /// HTTP connection pool can hand a later test a connection to a port whose server is already gone + /// (`Connection reset by peer` under `--gtest_repeat`). One connection per request is what a + /// short-lived test server should get. + cfg.http_keep_alive_timeout = 0; auto client = DB::S3::ClientFactory::instance().create( cfg, DB::S3::ClientSettings{ diff --git a/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp b/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp index 22e7951fbeec..689729e34f9e 100644 --- a/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp +++ b/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp @@ -99,6 +99,9 @@ static std::shared_ptr makeTestClient(const DB::S3::URI & uri) /*opt_disk_name=*/{}, /*request_throttler=*/{}, uri.uri.getScheme()); + /// Fresh connection per request: this file's servers live on ephemeral ports and die with the test, + /// and a pooled keep-alive connection can outlive its server (`Connection reset by peer` under `--gtest_repeat`). + client_configuration.http_keep_alive_timeout = 0; client_configuration.endpointOverride = uri.endpoint; /// `ClientFactory::create` installs the SDK's actual retry strategy itself from /// `client_configuration.retry_strategy`/`s3_slow_all_threads_after_retryable_error` (any From a8d9dd1f1f681223281e2dd25f81eef2db2e1355 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 20:13:06 +0200 Subject: [PATCH 80/81] cas: stop the test mock servers without Poco's abort path `stopAll(true)` reaches `HTTPServerConnection::onServerStopped(abortCurrent = true)`, which shuts the connection's socket down without taking the connection mutex so that it can interrupt a handler holding it. A worker that is leaving `run` at the same moment closes that socket from its own thread, and TSan reports the race on `SocketImpl::_sockfd` (CI run 9 of PR #2300, Unit tests (tsan), `IOTestAwsS3Client.NativeConditionalModeIsRederivedOnEverySdkAttempt`). The abort path was only there to unblock workers waiting for the next request on a pooled keep-alive connection before `thread_pool.joinAll`. Close those connections from the client side instead: drop the process-wide `HTTPConnectionPools` cache, which the AWS SDK client uses through `makeHTTPSession`, so every worker sees end of stream and closes its own socket on its own thread; then `stop` the server (accept thread joined, dispatcher stopped, no `serverStopped` notification) and join the pool. Applied to all four fork mock servers: `TestPocoHTTPServer`, `TestPocoHTTPStsServer`, `TestPocoHTTPSequenceServer`, `ScriptedResponseServer`. Production is unaffected: `DB::HTTPServer::stopAll` never used the abort path. Signed-off-by: Mikhail Filimonov Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx --- src/IO/S3/tests/TestPocoHTTPServer.h | 12 ++++++++---- src/IO/S3/tests/gtest_aws_s3_client.cpp | 9 ++++++--- src/IO/S3/tests/gtest_cas_aws_s3_client.cpp | 9 ++++++--- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/src/IO/S3/tests/TestPocoHTTPServer.h b/src/IO/S3/tests/TestPocoHTTPServer.h index 33fde7cb85b5..62a51e063ca5 100644 --- a/src/IO/S3/tests/TestPocoHTTPServer.h +++ b/src/IO/S3/tests/TestPocoHTTPServer.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -81,11 +82,13 @@ class TestPocoHTTPServer server->start(); } - /// `stopAll(true)` aborts any active connection immediately, so its worker thread isn't still - /// blocked reading for a next request when `thread_pool`'s destructor tries to join it. + /// Closing the cached client sockets wakes the server workers without Poco's abort notification, + /// whose unlocked socket shutdown races the worker's own close. Precondition: callers have released + /// their sessions, otherwise `joinAll` waits for the server's request timeout. ~TestPocoHTTPServer() { - server->stopAll(true); + DB::HTTPConnectionPools::instance().dropCache(); + server->stop(); thread_pool.joinAll(); } @@ -196,7 +199,8 @@ class TestPocoHTTPStsServer /// See `TestPocoHTTPServer`'s destructor above. ~TestPocoHTTPStsServer() { - server->stopAll(true); + DB::HTTPConnectionPools::instance().dropCache(); + server->stop(); thread_pool.joinAll(); } diff --git a/src/IO/S3/tests/gtest_aws_s3_client.cpp b/src/IO/S3/tests/gtest_aws_s3_client.cpp index bdd879c4c19f..e5db854878ba 100644 --- a/src/IO/S3/tests/gtest_aws_s3_client.cpp +++ b/src/IO/S3/tests/gtest_aws_s3_client.cpp @@ -1,3 +1,4 @@ +#include #include #include @@ -1062,11 +1063,13 @@ class ScriptedResponseServer server->start(); } - /// `stopAll(true)` aborts any active connection immediately, so its worker thread isn't still - /// blocked reading for a next request when `thread_pool`'s destructor tries to join it. + /// Closing the cached client sockets wakes the server workers without Poco's abort notification, + /// whose unlocked socket shutdown races the worker's own close. Precondition: callers have released + /// their sessions, otherwise `joinAll` waits for the server's request timeout. ~ScriptedResponseServer() { - server->stopAll(true); + DB::HTTPConnectionPools::instance().dropCache(); + server->stop(); thread_pool.joinAll(); } diff --git a/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp b/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp index 689729e34f9e..8ca6c6c79b8c 100644 --- a/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp +++ b/src/IO/S3/tests/gtest_cas_aws_s3_client.cpp @@ -1,3 +1,4 @@ +#include #include #include @@ -249,11 +250,13 @@ class TestPocoHTTPSequenceServer server->start(); } - /// `stopAll(true)` aborts any active connection immediately, so its worker thread isn't still - /// blocked reading for a next request when `thread_pool`'s destructor tries to join it. + /// Closing the cached client sockets wakes the server workers without Poco's abort notification, + /// whose unlocked socket shutdown races the worker's own close. Precondition: callers have released + /// their sessions, otherwise `joinAll` waits for the server's request timeout. ~TestPocoHTTPSequenceServer() { - server->stopAll(true); + DB::HTTPConnectionPools::instance().dropCache(); + server->stop(); thread_pool.joinAll(); } From 47246154f62a5e931f1ea3dd6a608f43f8f04da1 Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 9 Sep 2026 20:08:12 +0200 Subject: [PATCH 81/81] Revert "poco: make HTTPServerConnection's stop flag atomic" This reverts commit 5340794a380, a fork patch to upstream Poco: the only callers of `HTTPServer::stopAll(true)` in this tree were the test mock servers, and they no longer use Poco's abort notification (see the previous commit), so `onServerStopped` never runs concurrently with `run` here and the fork patch to `base/poco` is not needed. Signed-off-by: Mikhail Filimonov Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx (cherry picked from commit ca187904e613d3c0755bdf3ac38dc008a127cf4b) --- base/poco/Net/include/Poco/Net/HTTPServerConnection.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/base/poco/Net/include/Poco/Net/HTTPServerConnection.h b/base/poco/Net/include/Poco/Net/HTTPServerConnection.h index 8b15924cf07a..f0faa1b8c719 100644 --- a/base/poco/Net/include/Poco/Net/HTTPServerConnection.h +++ b/base/poco/Net/include/Poco/Net/HTTPServerConnection.h @@ -19,7 +19,6 @@ #include "Poco/Mutex.h" -#include #include "Poco/Net/HTTPRequestHandlerFactory.h" #include "Poco/Net/HTTPResponse.h" #include "Poco/Net/HTTPServerParams.h" @@ -57,9 +56,7 @@ namespace Net private: HTTPServerParams::Ptr _pParams; HTTPRequestHandlerFactory::Ptr _pFactory; - /// Written by onServerStopped() from the thread that calls HTTPServer::stopAll(true) while run() - /// polls it outside _mutex; a plain bool is a data race (TSan), an atomic keeps the same semantics. - std::atomic _stopped; + bool _stopped; Poco::FastMutex _mutex; };