One-time cloud setup needed before infra/terraform-test and infra/terraform-staging can be applied

[amar-python]

Context

infra/terraform-test/ and infra/terraform-staging/ were added, mirroring infra/terraform-prod/'s architecture tier (managed Postgres Flexible Server, private VNet, private endpoints, Container Apps Job). They're code only — no Azure resources exist yet, and terraform init/apply won't work until the one-time setup below is done. This issue is the reminder to do that setup before actioning (applying) either stack.

Follow docs/PROD_DEPLOY.md's "One-time setup" section as the template, substituting test/staging for prod throughout.

Checklist — Test environment

• Create Terraform state storage (storage account + tfstate container) for test
• Uncomment the backend "azurerm" block in infra/terraform-test/main.tf and fill in the real storage_account_name
• Create a Test-only service principal with OIDC (separate from Dev/Staging/Prod's SPs)
• Configure a GitHub test Environment + secrets (mirroring the production Environment setup in docs/PROD_DEPLOY.md)
• terraform init / plan — review the plan and cost impact before apply

Checklist — Staging environment

• Create Terraform state storage (storage account + tfstate container) for staging
• Uncomment the backend "azurerm" block in infra/terraform-staging/main.tf and fill in the real storage_account_name
• Create a Staging-only service principal with OIDC
• Configure a GitHub staging Environment + secrets
• terraform init / plan — review the plan and cost impact before apply

Cost warning

Each stack runs the same architecture tier as Prod — roughly AUD 285-385/mo per environment at default sizing (see the cost estimate in each main.tf header). Applying both Test and Staging on top of the existing Dev + Prod stacks meaningfully increases total Azure spend. Review terraform plan and the SKU/HA/backup defaults in each variables.tf before running apply.

Do not apply casually

terraform apply in either stack provisions real, billed Azure resources (Flexible Server, VNet, private endpoints, Key Vault, ACR Premium, Log Analytics). Don't run it until the setup above is complete and the plan has been reviewed.

[amar-python]

Blocked on #32 — az CLI on the dev machine currently can't authenticate (SSL certificate verification failure), so the state-storage creation step below hasn't been done yet. Nothing in this checklist has been actioned; infra/terraform-test and infra/terraform-staging remain code-only with the backend block commented out.

[amar-python]

Still blocked on #32 — the SSL certificate issue turned out to be Norton Antivirus's SSL/TLS scanning feature generating a non-compliant root CA (missing critical Basic Constraints), not a simple missing-from-bundle problem. See #32 for the full diagnosis and what's needed to actually unblock this (a Norton settings change, or running from an unaffected network/Cloud Shell). Nothing in this checklist has been actioned — infra/terraform-test and infra/terraform-staging remain code-only, as they should until this is resolved and reviewed.

[amar-python]

Update: found a second, independent local blocker — #34 (Terraform's own provider-plugin IPC handshake fails locally, unrelated to #32/Azure/Norton — reproduces even with no network calls, and even against the pre-existing terraform-prod stack). So local terraform apply isn't viable right now for two separate reasons, not just #32. Practical path forward is running from Azure Cloud Shell instead of this machine — offered to drive that via browser automation but holding for your go-ahead given it provisions real, billed resources (~AUD 570-770/mo combined for test+staging at default sizing).

[amar-python]

Execution plan — Azure Cloud Shell walkthrough

Decided approach: since local az/terraform are blocked on this machine (#32, #34 — both traced to Norton's local TLS interception, declined to fix by changing Norton settings), this runs from Azure Cloud Shell instead, driven manually by the repo owner (not automated). Script pushes directly to main (no PR) — a deliberate choice, not an oversight.

1. Open Azure Cloud Shell

Go to shell.azure.com (or portal.azure.com → the >_ icon). First run: choose Bash, not PowerShell. If prompted to create a Cloud Shell storage account, accept the default — that's Cloud Shell's own file storage, unrelated to the Terraform state storage the script below creates.

2. Paste the script

Click into the terminal, paste with Ctrl+Shift+V (or right-click → Paste; Cmd+V on Mac), then Enter:

#!/usr/bin/env bash
set -euo pipefail

REGION="australiaeast"

# ── 0. Clone the repo ─────────────────────────────────────────────────────────
if [ ! -d PostgreDataMigrationAppWithCSVLoader ]; then
  git clone https://github.com/amar-python/PostgreDataMigrationAppWithCSVLoader.git
fi
cd PostgreDataMigrationAppWithCSVLoader

# ── 1. Shared state-storage resource group ────────────────────────────────────
az group create --name rg-tfstate --location "$REGION"

# ── 2. Per-environment state storage account + container ──────────────────────
setup_state_storage() {
  local env="$1"
  local suffix
  suffix=$(openssl rand -hex 3)
  local account="sttfstate${env}${suffix}"

  az storage account create \
      --name "$account" \
      --resource-group rg-tfstate \
      --location "$REGION" \
      --sku Standard_LRS \
      --encryption-services blob \
      --min-tls-version TLS1_2

  az storage container create \
      --account-name "$account" \
      --name tfstate \
      --auth-mode login

  echo "$account"
}

TEST_ACCOUNT=$(setup_state_storage test)
STAGING_ACCOUNT=$(setup_state_storage staging)

echo "Test storage account:    $TEST_ACCOUNT"
echo "Staging storage account: $STAGING_ACCOUNT"

# ── 3. Wire the backend "azurerm" block in each main.tf ────────────────────────
python3 - "$TEST_ACCOUNT" "$STAGING_ACCOUNT" <<'PYEOF'
import re, sys

test_account, staging_account = sys.argv[1], sys.argv[2]

def wire_backend(path: str, account: str, key: str) -> None:
    text = open(path, encoding="utf-8").read()
    pattern = re.compile(
        r"  #\n"
        r'  # backend "azurerm" \{\n'
        r'  #   resource_group_name  = "rg-tfstate"\n'
        r"  #   storage_account_name = \"[^\"]+\"  # from the one-time setup script\n"
        r'  #   container_name       = "tfstate"\n'
        r'  #   key                  = "[^"]+"\n'
        r"  # \}\n",
    )
    replacement = (
        '  backend "azurerm" {\n'
        '    resource_group_name  = "rg-tfstate"\n'
        f'    storage_account_name = "{account}"\n'
        '    container_name       = "tfstate"\n'
        f'    key                  = "{key}"\n'
        "  }\n"
    )
    new_text, n = pattern.subn(replacement, text)
    if n != 1:
        sys.exit(f"Expected exactly 1 match in {path}, found {n} — check the file wasn't already edited.")
    open(path, "w", encoding="utf-8").write(new_text)
    print(f"Wired backend in {path} -> {account}")

wire_backend("infra/terraform-test/main.tf", test_account, "te-test.tfstate")
wire_backend("infra/terraform-staging/main.tf", staging_account, "te-staging.tfstate")
PYEOF

# ── 4. Commit and push the wired backend blocks to main ────────────────────────
git add infra/terraform-test/main.tf infra/terraform-staging/main.tf
git commit -m "Wire remote state backend for test/staging (issue #31)"
git push

# ── 5. init -> plan -> apply, per environment ───────────────────────────────────
run_environment() {
  local env_dir="$1"
  echo
  echo "════════════════════════════════════════════════════════════════"
  echo " $env_dir"
  echo "════════════════════════════════════════════════════════════════"
  ( cd "$env_dir" && terraform init )
  ( cd "$env_dir" && terraform plan )
  echo
  echo ">>> Review the plan above carefully before continuing."
  echo ">>> terraform apply will now show the plan again and prompt you to"
  echo ">>> type 'yes' — it will NOT create anything until you do."
  echo
  ( cd "$env_dir" && terraform apply )
}

run_environment infra/terraform-test
run_environment infra/terraform-staging

echo
echo "Done. Outputs:"
( cd infra/terraform-test && terraform output )
( cd infra/terraform-staging && terraform output )

3. What happens, and where to act

1. Repo clones — a few seconds. If the repo is private and Cloud Shell isn't already authenticated to GitHub, git clone/git push prompt for a device-login flow or a PAT.
2. Two storage accounts get created (az storage account create × 2) — ~1 minute each, no interaction.
3. Backend blocks wired + pushed to main — automatic, unless git auth is needed (see above).
4. For terraform-test, then terraform-staging, in turn:
   • terraform init — downloads providers, ~1 minute.
   • terraform plan — prints everything about to be created. Read it.
   • terraform apply — prints the plan again, then stops and prompts:

     Do you want to perform these actions?
       Enter a value:
     

     Must type yes to actually provision. Anything else cancels — no -auto-approve anywhere in this script, on purpose.
   • Provisioning itself: ~10-15+ minutes per environment (Flexible Server + private endpoints are the slow part).
5. Repeats for staging.
6. Prints outputs — resource group names, DB hostnames, etc. — at the end.

4. If something goes wrong

It's safe to close the Cloud Shell tab and resume later — nothing here is destroyed by closing the window (state lives in the storage accounts from step 2). Re-open Cloud Shell, cd PostgreDataMigrationAppWithCSVLoader, re-run whichever step was left off.

Cost reminder

Full apply for both environments provisions real, billed Azure resources — realistically ~AUD 570-780/mo combined for test+staging at the default (Prod-tier) sizing set in each terraform.tfvars.example. Review the terraform plan output for each environment before typing yes.