docs: add k3s boot disk replacement runbook

[xnoto]

Summary

Documents the confirmation-gated recovery path for the already-authored k3s boot-disk capacity reconciliation. The runbook distinguishes the replace-only boot volume/domain path from the persistent vdb K3s-data volume, records the expected loss of the in-cluster arc-tf apply runner, and requires a fresh plan before any re-drive.

Fixes #
N/A — operational recovery documentation for the recurring root-filesystem capacity alert.

Type of change

• Documentation
• Infrastructure (OpenTofu root or module)

Validation

• Required pull-request checks pass — OpenTofu test and plan passed.
• Generated or centrally distributed files were regenerated by their owning automation, not hand-edited — N/A; README.md is generated and unchanged.

Review evidence before PR creation:

• Live libvirt inspection verified k3s:vda is 5 GiB and vdb is 400 GiB.
• main.tf already requests a 20 GiB k3s boot disk; the prior capacity PR was #23.
• The current module/provider replacement behavior and shared-workflow plan/apply gates were reviewed.
• QA and release-readiness review completed; the runbook was amended to cover expected runner self-interruption and the fresh-plan re-drive gate.
• Owner waiver: the owner explicitly waived the remaining procedural High holds from adversarial and infrastructure-security reviewers, which could not consume the remote-only diff. This waiver does not authorize merge, production approval, apply, or any live mutation.

Impact and rollout

Blocker — do not merge: the authoritative PR plan creates only the two cloud-init disks; it proposes no boot-volume, domain, or vdb action. This conflicts with live libvirt evidence that k3s:vda remains 5 GiB while desired configuration/state is 20 GiB. The current provider/state path does not surface the required boot-disk reconciliation, so merging this documentation PR cannot fix the alert.

The next corrective design must safely reconcile the real 5 GiB libvirt volume with the OpenTofu state before a replacement plan can be trusted. It must preserve the 400 GiB vdb volume and account for self-interruption of the in-cluster apply runner.

Delivery stages: documentation authored and validated; no desired-state reconciliation, artifact publication, GitOps selection, Argo reconciliation, infrastructure apply, or functional verification has occurred.

Safety and secrets

• Contains no plaintext secrets, decrypted SOPS values, state files, kubeconfigs, tokens, or private endpoints
• No local OpenTofu init/plan/apply/destroy/import/state operations were run or claimed — plans come from pull-request checks
• Breaking or irreversible effects are described above with rollback notes

AI-assisted change; review the full runbook and the credentialed PR plan before considering any merge or production approval.

[github-actions]

OpenTofu Test

OpenTofu test passed.

View the workflow run.

View run output

Terraform validate.......................................................Passed
Terraform validate with tflint...........................................Passed
Checkov..................................................................Passed
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Detect hardcoded secrets.................................................Passed
check for case conflicts.................................................Passed
check for merge conflicts................................................Passed
check for broken symlinks............................(no files to check)Skipped
check vcs permalinks.....................................................Passed
detect destroyed symlinks................................................Passed
detect private key.......................................................Passed
fix end of files.........................................................Passed
mixed line ending........................................................Passed
trim trailing whitespace.................................................Passed
don't commit to branch..................................................Skipped
check for added large files..............................................Passed

[github-actions]

OpenTofu Plan

OpenTofu plan passed.

View the workflow run.

View run output

OpenTofu will perform the following actions:

  # module.k3s.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: k3s
            local-hostname: k3s
        EOT
      + name           = "k3s_commoninit"
      + network_config = <<-EOT
            version: 2
            ethernets:
              enp1s0:
                dhcp4: true
              enp2s0:
                dhcp4: false
                addresses:
                  - 192.168.102.2/24
        EOT
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

  # module.runner.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: runner
            local-hostname: runner
        EOT
      + name           = "runner_commoninit"
      + network_config = (sensitive value)
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

Plan: 2 to add, 0 to change, 0 to destroy.
OpenTofu will perform the following actions:

  # module.k3s.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: k3s
            local-hostname: k3s
        EOT
      + name           = "k3s_commoninit"
      + network_config = <<-EOT
            version: 2
            ethernets:
              enp1s0:
                dhcp4: true
              enp2s0:
                dhcp4: false
                addresses:
                  - 192.168.102.2/24
        EOT
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

  # module.runner.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: runner
            local-hostname: runner
        EOT
      + name           = "runner_commoninit"
      + network_config = (sensitive value)
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

Plan: 2 to add, 0 to change, 0 to destroy.