OpenID4VCI: make server-to-server credential-offer delivery reliable (retry, DLQ, base-URL setup)

[reinkrul]

Related: #2825, #2318

Problem Statement

Private credentials are delivered from a credential issuer's node to a credential receiver's node two ways: primarily via an OpenID4VCI server-to-server credential offer (a synchronous HTTP push at issuance time, pre-authorized_code grant — vcr/issuer/openid.go), falling back to publishing the encrypted transaction on the Nuts gRPC network for the receiver to fetch (vcr/issuer/network_publisher.go, network/transport/v2/protocol.go) whenever the OpenID4VCI push fails or isn't supported by the receiver.

Both paths have reliability gaps that, together, can silently drop a credential with no trace above Trace-level logging:

1. No retry for the OpenID4VCI push. Issue() (vcr/issuer/issuer.go:137-203) makes exactly one synchronous HTTP attempt (issueUsingOpenID4VCI, line 180) and falls straight through to the gRPC/DAG fallback on any failure. There's no queue, no backoff, no persistence of the failed attempt — unlike the private-payload-fetch mechanism on the receiving side, which already has a go-retry-based notifier with a payload_fetch_dlq diagnostic (network/dag/notifier.go, network/transport/v2/protocol.go:332-345).

2. The issuer's own base-URL identifier resolution can get permanently stuck empty. tlsIdentifierResolver (vcr/openid4vci/identifiers.go) caches whatever it resolves — including an empty string — the first time it runs, and never invalidates it before process restart: the cache read (identifiers.go:102-106) only checks for a non-nil pointer, not a non-empty value, and the cache write (identifiers.go:116-125) stores on err == nil regardless of whether the identifier is non-empty. One bad resolution (missing node-http-services-baseurl service at the time, or a transient failure of the TLS-certificate self-discovery fallback) permanently breaks every subsequent OpenID4VCI offer from that issuer DID for the life of the process. Fixing (1) alone wouldn't help — every retry would hit the same poisoned cache.

   Observed on the sender side (issuer), for a credential_offer sent with credential_issuer:"":

   level=warning msg="Couldn't publish credential over OpenID4VCI, fallback to publish over Nuts network" credentialID="did:nuts:ISSUER_DID_REDACTED#REDACTED-CREDENTIAL-UUID" error="unable to offer the credential over OpenID4VCI to (wallet: https://receiver.example.com/n2n/identity/did:nuts:RECEIVER_DID_REDACTED): unable to offer credential (client-metadata-url=https://receiver.example.com/n2n/identity/did:nuts:RECEIVER_DID_REDACTED/openid4vci/credential_offer): offer credential error: unexpected http response code (...): 500" module=VCR
   

   Observed on the receiver side, rejecting that same offer:

   time="2026-09-03T08:16:47Z" level=error msg="HandleCredentialOffer failed" error="server_error - unable to create issuer client: empty Credential Issuer Identifier" module=VCR/OpenID4VCI operation=HandleCredentialOffer operationID=HandleCredentialOffer requestURI="/n2n/identity/did:nuts:RECEIVER_DID_REDACTED/openid4vci/credential_offer?credential_offer=..." user="<nil>"
   

3. The gRPC/DAG fallback silently drops credentials for some key-storage backends. Decrypting a private transaction's PAL header needs the raw keyAgreement private key in-process for ECIES (network/dag/pal.go, crypto/crypto.go:27-44). Backends that never expose raw key material (e.g. a remote HSM/KMS backend with non-exportable keys) return a generic, unclassified error that doesn't match the one sentinel (crypto.ErrPrivateKeyNotFound) which triggers a loud failure. The receiver falls through to "this transaction is not for me," marks the fetch job successfully finished, and never retries — no error above Trace level, no DLQ entry, ever.

4. node-http-services-baseurl registration isn't a guaranteed, verifiable setup step. It's currently a side effect of the GoldenHammer auto-heal module (golden_hammer/module.go), which only covers a DID that is owned by the node, carries its own NutsComm service entry, and — for care-organization/subject DIDs — whose referenced vendor DID already has the base-URL service. GoldenHammer is also earmarked for removal once OpenID4VCI is the sole issuance mechanism (Remove Golden Hammer module  #2318), so it isn't a stable long-term source of this service.

Trigger: a production incident where a credential receiver never received a private credential — traced through points 1-3 above, all contributing.

Solution

Invest in the OpenID4VCI server-to-server path rather than hardening the gRPC/DAG fallback for every key-storage backend (see Further Notes for the alternative and why it's set aside). This also aligns with the broader move away from the gRPC network (#2825).

• Give the OpenID4VCI credential-offer push retry/DLQ, reusing avast/retry-go/v4 (already a dependency, go.mod:12) and the existing dag.Notifier pattern — without removing the existing gRPC/DAG fallback. The fallback stays the reliability backstop; the retry queue is what makes OpenID4VCI itself eventually reliable enough to reduce reliance on that backstop.
• Fix the base-URL identifier cache so one bad resolution can't permanently poison delivery.
• Make node-http-services-baseurl registration a formal, documented, checkable part of node setup, independent of GoldenHammer.
• Audit the live network for DID documents (vendor and care organization) missing the base-URL service, to size the current gap before relying on OpenID4VCI delivery being reliable.

User Stories

User Stories

1. As a credential issuer, I want a failed OpenID4VCI credential offer to retry automatically, so that a transient failure on the receiver's side doesn't permanently drop the credential.
2. As a node operator, I want failed/pending credential offers visible in diagnostics, so that I can tell delivery is stuck instead of assuming it succeeded.
3. As a node operator, I want to manually requeue a stuck credential offer once I've fixed the underlying issue, so that I don't have to re-issue the credential or restart the node.
4. As a node operator, I want setup documentation to tell me explicitly to register node-http-services-baseurl, so that I don't depend on an auto-heal module scheduled for removal.

Implementation Decisions

Retry + DLQ for the OpenID4VCI offer

Model this on the existing "private" notifier (network/transport/v2/protocol.go:153-169): a persistent, named dag.Notifier-style queue for pending offers, retried via retry.Do with an increasing retry interval, capped at a maximum total retry duration of 24 hours — the same backoff shape already used for the private-payload-fetch notifier (network/dag/notifier.go:359-395, retry.MaxDelay(24*time.Hour)). After 24 hours of failed retries, the offer moves to the DLQ.

The gRPC/DAG fallback fires once the OpenID4VCI retry queue gives up (i.e. after that 24-hour window is exhausted), not on the very first failure. Publishing to the DAG is irreversible and network-wide replicated (network/network.go:662-750, network/transport/v2/protocol.go:228-318) with no dedup — two publishes of the same VC create two permanent, separate transactions, and there's no delete/cancel API (network/interface.go:33-63). A full day of retries makes it very likely that, by the time the fallback fires, the failure is genuinely persistent rather than transient — at which point paying the permanent DAG cost is justified.

The retry queue only retries the OpenID4VCI HTTP push (OfferCredential) — it never calls networkPublisher.PublishCredential again. Only the fallback (fired once, after the 24-hour retry window is exhausted) creates a DAG transaction; a later successful OpenID4VCI retry is a pure HTTP delivery and creates no DAG transaction at all. So the only "dual delivery" scenario is: the fallback already published to the DAG, and a delayed OpenID4VCI retry later also succeeds — redundant but harmless, since vcr.StoreCredential (vcr/store.go:42-65) treats a repeat of an identical VC as a silent no-op on the receiver's side.

Confirmed via e2e-tests/openid4vci/offer-retry: node A's transaction_count diagnostic is asserted unchanged across a full outage-and-recover cycle, proving delivery happened purely over OpenID4VCI, not the DAG.

Merge-order note: #4471 branches directly off master (verified it doesn't need #4470 to compile/test) and therefore doesn't yet special-case openid4vci.ErrIdentifierNotConfigured in issueUsingOpenID4VCI — that sentinel doesn't exist on master until #4470 merges. #4470's branch already has this special case (errors.Is(err, openid4vci.ErrIdentifierNotConfigured) → (false, nil), same bucket as an unsupported wallet → immediate DAG fallback, no pointless 24h retry of a local misconfiguration). Verified it composes cleanly: #4471 doesn't otherwise touch issueUsingOpenID4VCI, so rebasing #4471 onto master after #4470 merges picks this up with no manual changes needed. Just don't merge #4471 before #4470 without that rebase, or local misconfiguration will be needlessly retried for up to 24h before falling back.

Fix the base-URL identifier cache

tlsIdentifierResolver.Resolve (vcr/openid4vci/identifiers.go:102-125) must not cache an empty identifier forever. Don't remove the negative caching outright, though — this resolver can be called on every OpenID4VCI request, so re-running resolution (including the DID document lookup) on every single call for a DID that never gets fixed isn't free; the original negative caching was likely deliberate. Reuse the existing lastAttempt/tlsAttemptInterval throttle (already there to rate-limit the expensive TLS-certificate-derived resolution) to also bound how long an empty result is treated as cached, instead of adding a separate cache window. A successful (non-empty) identifier stays cached indefinitely, unchanged.

Separately, close the actual sending-side gap: nothing currently stops an OpenID4VCI credential offer from being sent with an empty credential_issuer in the first place — which is what produces the receiver-side "empty Credential Issuer Identifier" rejection seen in the observed errors above. Return a distinguishable error (openid4vci.ErrIdentifierNotConfigured) when resolution succeeds but yields no identifier, and have issueUsingOpenID4VCI treat it like an unsupported wallet — fallback to the network, no broken offer ever sent. Unlike an unsupported wallet (the other party's problem), this is the local operator's own misconfiguration, so it also logs a Warn pointing them at what to search the documentation for.

PR: #4470

Diagnostics + admin requeue endpoint

Expose the new queue's failed entries in /status/diagnostics, alongside protocol_v2.payload_fetch_dlq. Add an admin endpoint to requeue a specific failed offer by credential ID. No existing endpoint does this today — reprocess (docs/_static/network/v1.yaml:165-183, network/network.go:850-895) walks the DAG by transaction content-type and has no concept of a specific failed job or a DLQ; it's not a fit to extend.

Formalize node-http-services-baseurl setup

Document it as a required manual step in node setup docs (docs/pages/deployment/), with an example. Keep GoldenHammer as a safety net for now (#2318 tracks its eventual removal, dependent on this and #3972 landing).

Modules to build/modify

1. vcr/issuer/: new persistent retry queue for OpenID4VCI offers, wired from Issue()'s failure path, with a 24-hour retry window before the DAG fallback fires.
2. vcr/openid4vci/identifiers.go: fix the caching bug in tlsIdentifierResolver.
3. vcr/api/vcr/v2/: diagnostics entry + admin requeue endpoint.
4. docs/pages/deployment/: formal node-http-services-baseurl setup instructions and example.
5. Ops: one-off audit of the live network's DID documents for missing node-http-services-baseurl.

Testing Decisions

Tests should exercise the retry queue's external behavior — a failed offer eventually succeeds after the receiver becomes reachable, retries continue with increasing interval until the 24-hour window is exhausted before the DAG fallback fires, a permanently-failing offer surfaces in diagnostics, a requeued offer is retried — not its internal storage shape. Model the notifier tests on network/dag/notifier_test.go's existing coverage of the "private" queue.

Modules to test:

• vcr/issuer/: retry-until-success, retry-window-exhausted-then-fallback, retry-exhaustion-to-DLQ, manual requeue.
• vcr/openid4vci/identifiers.go: cache is not poisoned by an empty resolution; a later successful resolution overrides a cached empty one.
• vcr/store.go: receiving the same credential via both OpenID4VCI and the DAG payload-fetch path is idempotent — a repeat delivery of an identical VC is a no-op, not a duplicate or error. Already true today (StoreCredential, vcr/store.go:42-56, dedups by id before writing) — add a regression test covering the specific dual-channel scenario (fallback publishes, then a delayed OpenID4VCI retry also succeeds) so this reliability work doesn't silently break it.
• Prior art: network/dag/notifier_test.go, network/transport/v2/protocol_test.go (existing "private" notifier tests).

Impact Assessment

Backwards compatibility: additive, but with one real behavior change worth calling out explicitly: since the DAG fallback is now gated behind the 24-hour retry window rather than firing synchronously on first failure, POST /internal/vcr/v2/issuer/vc can no longer block for up to 24h waiting for it. A 200 still means the credential was created and stored, and that the single synchronous OpenID4VCI attempt was made — but if that attempt fails, the response returns immediately after the retry job is durably enqueued, before delivery (via either path) is confirmed. This is a change from today's guarantee that a 200 means the credential was already published somewhere by response time. Worth a release-notes callout.
Versioning: minor.
Configuration/deployment: new required setup step (node-http-services-baseurl); existing deployments need the audit (see Implementation Plan) before this can be relied on network-wide.
Security: none new — reuses existing retry/notifier infrastructure and endpoint auth model.

Out of Scope

• Hardening the gRPC/DAG-network path to work with key-storage backends that don't expose raw key material (e.g. adding an alternate keyAgreement key type usable by remote-decrypt-capable KMS/HSM backends) — investigated as an alternative, feasible but multi-layer (crypto backend interfaces, PAL encrypt/decrypt algorithm branching, DID key-type plumbing). Not pursued now since it invests further in the gRPC network we're moving away from (gRPC-less Nuts network without big-bang migration to did:web #2825). Tracked separately if revisited.
• Removing GoldenHammer (Remove Golden Hammer module  #2318) — depends on this issue and Make OpenID4VCI implementation ready for general use #3972 landing first.

Further Notes

The receiver-side misclassification behind point 3 — any decrypt failure, not just non-exportable-key backends, currently reads as "confirmed not a participant" and is silently, permanently dropped, with no error above Trace level and no DLQ entry — is a correctness/observability gap worth a standalone follow-up regardless of which delivery path we invest in here.

Implementation Plan

Each fix is its own PR, applied to master first and then backported in the same shape to the v6 line (released as v6.3) and the v5 line (released as v5.5). Backport PRs for a given fix aren't opened until that fix's master PR has been reviewed (per established convention). Verify exact branch names via git ls-remote at backport time rather than assuming (existing convention is capitalized, e.g. V6.2).

One-off, not per-branch:

• Audit the live network for DID documents missing node-http-services-baseurl

master

• Fix base-URL identifier cache so it can't be poisoned by an empty result (PR: fix(vcr): bound negative OpenID4VCI identifier caching, guard against empty offers #4470, draft)
• Add increasing-interval retry (max 24h total) + persistent DLQ for the OpenID4VCI credential-offer push; gate the DAG fallback behind that window (PR: feat(vcr): retry + DLQ for the OpenID4VCI credential-offer push #4471, draft)
• Expose the offer-delivery DLQ in diagnostics + admin requeue endpoint (PR: —, depends on retry+DLQ above)
• Formalize node-http-services-baseurl as a documented setup step (PR: docs(deployment): document node-http-services-baseurl as a required setup step #4475, merged)

v6.3 (backport)

• Fix base-URL identifier cache so it can't be poisoned by an empty result (PR: —)
• Add increasing-interval retry (max 24h total) + persistent DLQ for the OpenID4VCI credential-offer push; gate the DAG fallback behind that window (PR: —)
• Expose the offer-delivery DLQ in diagnostics + admin requeue endpoint (PR: —, depends on retry+DLQ above)
• Formalize node-http-services-baseurl as a documented setup step (already in V6.2, will carry over when v6.3 is cut - see below)

v5.5 (backport)

• Fix base-URL identifier cache so it can't be poisoned by an empty result (PR: —)
• Add increasing-interval retry (max 24h total) + persistent DLQ for the OpenID4VCI credential-offer push; gate the DAG fallback behind that window (PR: —)
• Expose the offer-delivery DLQ in diagnostics + admin requeue endpoint (PR: —, depends on retry+DLQ above)
• Formalize node-http-services-baseurl as a documented setup step (already in V5.4, will carry over when v5.5 is cut - see below)

v5.4 / V6.2 (direct backport, docs-only)

• Formalize node-http-services-baseurl as a documented setup step (v5.4: docs(deployment): document node-http-services-baseurl as a required setup step (backport of #4475 to V5.4) #4480, v6.2: docs(deployment): document node-http-services-baseurl as a required setup step (backport of #4475 to V6.2) #4481) - backported straight to the current release branches instead of waiting for the v5.5/v6.3 cut, since it's docs-only with no behavior change.

[stevenvegt]

lets discus with the current 5.4 users if they want this and if it is a possibility for them to move to 6.3 for this. I suspect that a 5.x backport is more then a simple code copy.

I also wonder if 24 hours isn't too long. What do we find "acceptable" downtime of a node? Assuming a short update window of perhaps an outage of an few hours. After that, there is no harm in GRPC publishing. I think the action also depends on the failure reason: host not found is something different then a 404 on the offer endpoint or an TLS error vs a timeout. Some of those are likely to recover in a short time vs a wrong configuration which probably doesn't.

From a use perspective, I wonder if the following is a valid user story:

As a credential issuer, I want to know when the initial issuance failed and I cannot continue my work, how long I should wait before retrying or even start calling someone.

[reinkrul]

lets discus with the current 5.4 users if they want this and if it is a possibility for them to move to 6.3 for this. I suspect that a 5.x backport is more then a simple code copy.

We discussed this; plan:

• Backport cache bugfix and retry queue to v5.4 and release as patch
• Do the same on v6.2
• Release the retry-queue-diagnostics and endpoint to put the credential from the DLQ back to the queue as minor on v6.3

[reinkrul]

I also wonder if 24 hours isn't too long. What do we find "acceptable" downtime of a node? Assuming a short update window of perhaps an outage of an few hours. After that, there is no harm in GRPC publishing. I think the action also depends on the failure reason: host not found is something different then a 404 on the offer endpoint or an TLS error vs a timeout. Some of those are likely to recover in a short time vs a wrong configuration which probably doesn't.

I don't think there's harm in immediately publishing over gRPC if the initial attempt fails. At worst, they receive the credential twice, but it being idempotent, the second receival will be ignored. Checking underlying errors is complex and error prone, I'd like to avoid it if possible.

From a use perspective, I wonder if the following is a valid user story:

As a credential issuer, I want to know when the initial issuance failed and I cannot continue my work, how long I should wait before retrying or even start calling someone.

There's no retrying for the user: the system handles that. What we could do, however, is add a field to the API response whether the wallet successfully received the credential, or that it's delayed for technical reasons. The EHR developer can then shape the UX according to their taste.