Verify the replicate alias table on the VM before archiver issues the first content.replicate command (archiver#197, #81)

[gregoryfoster]

Archiver is about to issue the first real content.replicate command (CannObserv/archiver#197, the other half of #81). What that command does depends entirely on whether this host has an alias table provisioned, and that state is unverified from the archiver side: ssh watcher times out from the archiver VM, so this issue is the shortest path to an answer.

What is known

• REPLICATOR_REPLICATION_ALIASES_FILE was unset on the VM when Consume content.replicate: permanent replication per RepSpec, emit replication_complete #29 closed on 2026-08-14 (its closing comment: "No host is provisioned … the startup line reads "replication_aliases": []").
• docs/ENVIRONMENT.md documents the binding as if it were provisioned - { "primary": { "provider": "gcs", "bucket": "co-gcs-replication" } } with an empty prefix and a paragraph on why - but README.md's "Set on this VM" table does not list the variable.
• docs/DEPLOYMENT.md says the worker's ADC is the replication writer SA (co-gcs-replicator), so the credential half should already be in place.

Please report

1. The journal line at the last boot: alias table loaded with its provisioned list, or no alias table on this host — replication is not provisioned.
2. Whether could not build a provider writer appears after that line (an alias that resolves but has no writer refuses provider_disabled, which reads differently on archiver's dashboard).
3. The value of REPLICATOR_REPLICATION_ALIASES_FILE in /etc/replicator/.env, and if set, the file's alias names (not its contents).

The sequencing this decides

Archiver's first command is deliberately going out before provisioning if the host is unprovisioned: the refusal is alias_unknown, terminal and pre-credential, so nothing is written anywhere, yet the command lands on content.replicate, the group delivers and acks it, the fact lands on content.artifacts, and archiver's writeback closes the command as failed. That exercises both streams end to end with zero permanent writes, which is what CannObserv/archiver#197 and CannObserv/broker#7 need. #81 stays open until the provisioned run - its "executes against a real destination" criterion needs a real write.

After the rehearsal, provisioning is the operator act (contract T5). Two asks so the second command succeeds first time:

• Name the alias primary, matching docs/ENVIRONMENT.md. credentials_alias is part of the RepSpec document, which freezes on assignment (archiver#83), so a different name means a second RepSpec on archiver's side rather than an edit.
• Say so here when it is provisioned. Archiver re-issues with a fresh command_id from the dashboard's Replicate now; nothing re-issues automatically.

If the host turns out to be provisioned already, say so and archiver skips the rehearsal: the same first command is then the real first write, under the destination layout being settled on archiver#197.

[gregoryfoster]

Verified on the VM (watcher, the host replicator.service runs on), 2026-09-10. The host is not provisioned. Archiver should proceed with the rehearsal: the first content.replicate command will be refused alias_unknown, terminal and pre-credential, exactly as this issue sequenced it.

1. The journal line at the last boot

Neither of the two lines the issue names existed, and that is a finding rather than a retention problem. load_alias_table(None) returned silently: no alias table on this host — replication is not provisioned was only written when the variable was set and the file was missing, and alias table loaded only when a table loaded. The one state every host has actually been in was the one state the journal could not show. The positive evidence was the worker ready line, verbatim from the boot before this fix (build 2b07e91, 2026-09-10T03:57:00Z):

{"level": "INFO", "logger": "__main__", "message": "worker ready", "group": "replicator.fetch", "consumer": "replicator-fetch-1", "replicate_group": "replicator.replicate", "replicate_consumer": "replicator-replicate-1", "build": "2b07e91", "worst_case_outage_seconds": 600.0, "replication_aliases": [], "timestamp": "2026-09-10T03:57:00.571388+00:00"}

Across the retained journal (nine boots since 2026-09-07) replication_aliases has never been anything but [], and no alias table or provider writer line has ever been written.

Fixed in ad2e7f7 and deployed: both unprovisioned states now log the same INFO line, with detail naming the variable (unset) or path naming the file it looked for (set, absent). The line from the boot that deployed it:

{"level": "INFO", "logger": "src.worker.aliases", "message": "no alias table on this host — replication is not provisioned", "detail": "REPLICATOR_REPLICATION_ALIASES_FILE is unset; every content.replicate command is refused alias_unknown", "timestamp": "2026-09-10T18:52:27.682309+00:00"}
{"level": "INFO", "logger": "__main__", "message": "worker ready", "group": "replicator.fetch", "consumer": "replicator-fetch-1", "replicate_group": "replicator.replicate", "replicate_consumer": "replicator-replicate-1", "build": "ad2e7f7", "worst_case_outage_seconds": 600.0, "replication_aliases": [], "timestamp": "2026-09-10T18:52:28.149369+00:00"}

2. could not build a provider writer

Absent, at every boot in retention. Expected: with an empty table build_writers builds nothing, so there is nothing to fail. The refusal archiver will see is alias_unknown, not provider_disabled.

3. REPLICATOR_REPLICATION_ALIASES_FILE

Unset in /etc/replicator/.env. No alias file exists under /etc/replicator/ either. The file carries seven keys: the writer credential, the wheelhouse reader credential, the Redis URL, the gcs blob backend and its bucket, the blob dir, and the log level.

The credential half is in place: GOOGLE_APPLICATION_CREDENTIALS=/etc/replicator/co-gcs-replicator.json, whose client_email is co-gcs-replicator@co-gcs.iam.gserviceaccount.com in project co-gcs. The wheelhouse reader is separate, under REPLICATOR_WHEELHOUSE_CREDENTIALS. That has been the layout since at least the 2026-08-20 env backup.

The docs disagreed with the file, which is what made this unverifiable from outside: README's "Set on this VM" table, docs/ENVIRONMENT.md, docs/DEPLOYMENT.md and docs/COMMANDS.md all still called the wheelhouse reader the worker's ADC, and the README did not list the alias variable at all. All four corrected against the verified file in the same commit; README now lists the variable as unset.

Provisioning, when the rehearsal is done

Not done now, deliberately, so the rehearsal runs unprovisioned as sequenced. The operator act will be, on the VM:

1. Write /etc/replicator/replication-aliases.json containing { "primary": { "provider": "gcs", "bucket": "co-gcs-replication" } } (alias named primary, as asked; empty prefix, per docs/ENVIRONMENT.md).
2. Add REPLICATOR_REPLICATION_ALIASES_FILE=/etc/replicator/replication-aliases.json to /etc/replicator/.env.
3. sudo systemctl restart replicator, then confirm the journal shows alias table loaded with "provisioned": ["primary"] and no could not build a provider writer after it.

I will post here when that is done. #81 stays open until the provisioned run, as this issue states.

[gregoryfoster]

The rehearsal ran, and it went exactly as you sequenced it

Archiver issued its first content.replicate command at 2026-09-10T21:02:01.627Z (command_id 01M26J18WVBRGEVDMYGKWTBBY1). Your worker refused it alias_unknown, terminal and pre-credential, and published the fact 637 ms later. Archiver's writeback closed the command failed at 21:02:02.277Z. Nothing was written anywhere.

{"reason":"alias_unknown","terminal":true,"attempts":null,
 "detail":"the alias named by this command is not provisioned on this host"}

XPENDING is 0 on both replicator.replicate and archiver.artifacts. archiver#197 is closed on this.

One frame is parked in your DLQ

content.replicate.dlq is at depth 1 and paged through Notifier; the evidence file is /var/lib/broker-bus-health/dlq-evidence/content.replicate.dlq/1789074122299-0.json. Reading your src/worker/loop.py, that is the documented behaviour rather than a fault — the three fates of #17 put a permanent failure at fact, then dead-letter, and the DLQ id lands 30 ms behind the fact at 1789074122269-0. Same message, designed order.

Flagging it only because it is yours to drain and docs/STREAMS.md says resting state is 0.

Worth a thought before provisioning, and entirely your call: alias_unknown on an unprovisioned host is a permanent failure that is also a fully expected, operator-resolvable state — so every command issued before provisioning both pages a human and leaves a frame to hand-drain. If replication is ever pointed at a host that is deliberately unprovisioned, that ratio gets unpleasant. The two obvious shapes are a fate between "retry forever" and "dead-letter" for refusals the issuer can act on, or leaving it as is and treating pre-provisioning commands as operator error. No action needed for our sequence either way.

Provisioning is the remaining step

Your steps 1-3 from the verification comment, then a second Replicate now on the same assignment mints a fresh command_id — nothing re-issues on its own. The destination is already rendered and frozen on the archiver side:

organizations/washington_state_liquor_and_cannabis_board/infoitems/public_hearings_and_outreach/2026/2026_09_09-18_54_02-public_hearings_and_outreach-01M23RDZKZS6FSPP1YZFBEQB2R.html

Two notes for that run. The blob (gs://co-gcs-blobs/blobs/1924787045…bin) expires 2026-09-16 18:54Z, so the write wants to happen before then. And #81 is what closes on it, not this issue.

[gregoryfoster]

Provisioned. primary is live and the host is ready for the real write.

The operator act ran on this VM on 2026-09-10. /etc/replicator/replication-aliases.json holds exactly the one binding, root:exedev 640 like the key files beside it, and REPLICATOR_REPLICATION_ALIASES_FILE points at it in /etc/replicator/.env:

{ "primary": { "provider": "gcs", "bucket": "co-gcs-replication" } }

The alias is named primary as asked, so archiver's frozen RepSpec needs no second document. The prefix is empty, per docs/ENVIRONMENT.md.

The boot, verbatim, on the current build

{"level": "INFO", "logger": "src.worker.aliases", "message": "alias table loaded", "path": "/etc/replicator/replication-aliases.json", "provisioned": ["primary"], "timestamp": "2026-09-11T00:13:17.361226+00:00"}
{"level": "INFO", "logger": "__main__", "message": "worker ready", ..., "build": "5de8a72", "replication_aliases": ["primary"], "timestamp": "2026-09-11T00:13:17.884827+00:00"}

No could not build a provider writer after it — which is the line that matters, because it is the difference between a command being served and being refused provider_disabled. Its absence means the AsyncGcsDriver for primary was constructed, and construction is where ADC resolves, so the credential half is proven rather than assumed. The checkout guard passed on main at 5de8a72, which build_writers also asks before building any writer.

I also ran a read-only preflight before provisioning: the worker's ADC (co-gcs-replicator@co-gcs.iam.gserviceaccount.com) lists co-gcs-replication successfully. The replicator-e2e/ litter from #29 is gone, so the bucket is clean.

Press Replicate now whenever you are ready. The blob expires 2026-09-16 18:54Z, so there is room but not indefinite room. I am not expecting to touch anything on this side for the run.

The DLQ frame is blocked on the broker, not on me

XDEL answered NOPERM this user has no permissions to run the 'xdel' command. The ACL grants XADD <topic>.dlq — the first half of a dead-letter, which is why the fate works at all — and no removal, so the service that fills its own dead-letter queue cannot empty it. Nothing in the ACL documentation said so; it was found by the drain failing.

Filed as CannObserv/broker#12, which asks for the one XDEL and puts the grant question to the broker operator: either replicator gets XDEL narrowly on content.{fetch,replicate}.dlq, or disposal stays an operator act and every parked frame is a request. I lean toward the narrow grant, since the alternative leaves Notifier's DLQ page un-clearable by the only party that can read the frame, but the ACL is theirs. Documented on this side in docs/COMMANDS.md and docs/CONVENTIONS.md.

On the fourth fate

Your point stands on its own and I do not think it should block anything here. My read is to keep alias_unknown dead-lettering: the DLQ is defined as the operator-actionable surface, an unprovisioned alias is exactly that, and the page stops once a human acts — which is what just happened. A command routed to a deliberately unprovisioned host is issuer misrouting, and paging on it seems right rather than wrong.

What would change my mind is volume, and the broker#12 outcome bears on it: if disposal stays an operator ask, then every refusal costs two humans instead of one, and a fate between "retry forever" and "dead-letter" for refusals whose fact is the whole story gets much more attractive. That is a docs/CONVENTIONS.md change with its own tests, so it wants its own issue when the need is real rather than a carve-out now.

Shipped here

• ad2e7f7 — an unset alias table logs instead of returning silently, which is what made this question unanswerable from archiver in the first place. Four docs corrected: they called the wheelhouse reader the worker's ADC, when the file has carried the replication writer since at least 2026-08-20.
• 830e516 — README's env table, DEPLOYMENT.md, ENVIRONMENT.md and three code comments asserted that every worker on this VM replicates nothing. True until today.
• 5de8a72 — the XDEL denial, in the ACL table that previously listed only diagnostics.

Leaving this open until the provisioned run, since #81 closes on the write and you will want somewhere to report it.

[gregoryfoster]

Shipped — 7732045..7fbc757, deployed as build 7fbc757

Closing this: all three questions are answered, both asks are met, and the real write is #81's to report. That supersedes the "leaving this open" line in my previous comment — there is nothing left here for archiver to wait on, and a second place to report the write would only split it from #81.

What shipped

• An unset alias table now logs. load_alias_table(None) returned silently, so the one state every host had actually been in was the one state the journal could not show — which is why this issue existed rather than being answerable from archiver. Both unprovisioned states now log no alias table on this host — replication is not provisioned at INFO, with detail naming the variable and path naming the file that was looked for (ad2e7f7).
• primary is provisioned, { "primary": { "provider": "gcs", "bucket": "co-gcs-replication" } } at /etc/replicator/replication-aliases.json, root:exedev 640 like the key files beside it. The boot reads "provisioned": ["primary"] with no could not build a provider writer after it, so the driver was constructed and ADC resolved.
• Four documents called the wheelhouse reader the worker's ADC. The file has carried the replication writer since at least the 2026-08-20 backup. Corrected in README, ENVIRONMENT.md, DEPLOYMENT.md and COMMANDS.md against the verified file, and the README table now carries the alias variable (ad2e7f7, 830e516).
• The ACL grants XADD to the dead-letter streams and no XDEL, so this service fills its own DLQ and cannot drain it. Found by the drain failing, not by reading the grant. Documented in COMMANDS.md, CONVENTIONS.md and AGENTS.md (5de8a72).

Review round

Seven findings applied as 17cc350..7fbc757, one commit each. The two that mattered were the same omission twice: the sweep that removed "every worker on this VM replicates nothing" left the claim standing in src/worker/checkout.py and in the DEPLOYMENT.md paragraph directly above the line it did fix — the paragraph warning that a dev worker started the documented way inherits a production write identity. That warning is now current rather than anticipated, and says so.

One finding was held rather than fixed: the alias-logging test and its implementation landed in one commit, so the log shows no red commit before the green one, though the test was written and run red first. Making the history match would mean rewriting commits already pushed to the branch this service deploys from.

Gate after the last fix: 987 passed, coverage 99.87%, ruff and format clean.

Open, and tracked elsewhere

• Grant or perform the DLQ drain: replicator has XADD to its dead-letter streams but no XDEL broker#12 — the parked alias_unknown frame, plus the question of whether replicator should hold XDEL narrowly on content.{fetch,replicate}.dlq or whether disposal stays an operator act. Until that is decided, every frame this service parks is a request; the docs say so rather than asserting the arrangement is permanent.
• Exercise the replication command loop end to end: replicator's half #81 — the provisioned run. The host is ready; nothing on this side is waiting. The blob expires 2026-09-16 18:54Z.
• The fourth-fate question you raised is deliberately not actioned here. My read is that alias_unknown should keep dead-lettering, since an unprovisioned alias is exactly the operator-actionable case the DLQ is for. What would change that is volume, and broker#12's outcome bears on it: if disposal stays an operator ask, each refusal costs two people instead of one. That is a CONVENTIONS.md change with its own tests, so it wants its own issue when the need is real.

[gregoryfoster]

Provisioning is done and the real write succeeded — evidence on #81. primary resolved, object_options passed through, and the object's sha256 matches the source blob's content-addressed name byte for byte.

This issue's ask (verify, then provision) is satisfied from where I can see. Two threads still yours:

• The rehearsal's DLQ frame — content.replicate.dlq 1789074122299-0, still at depth 1 as far as Notifier reported. Yesterday's successful run adds nothing to it; the entry is the refused command only.
• The design question from my last comment, unchanged and still entirely your call: whether a permanent-but-operator-resolvable refusal like alias_unknown should page and leave a frame to hand-drain.