az CLI fails with SSL certificate verification error on this dev machine — blocks all Azure resource creation

[amar-python]

Problem

Every az CLI call from this dev machine that needs a fresh login token fails with:

ERROR: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /<tenant-id>/v2.0/.well-known/openid-configuration (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1032)')))

Confirmed with plain read-only calls (az group show, az account show for cached/basic info works, but calls needing a fresh token fail). Reproduces identically regardless of sandboxing — this is the local machine's network/certificate configuration, not tool-specific.

Likely cause

The network path from this machine to login.microsoftonline.com appears to go through a proxy or endpoint-security product that intercepts HTTPS with its own certificate. Python's certifi bundle (used by the az CLI) doesn't include that proxy's root CA, so TLS verification fails before authentication can even happen.

Confirmed: no REQUESTS_CA_BUNDLE or similar override is currently set; certifi is using its stock bundle (certifi.where() → the default cacert.pem).

Impact

Blocks any az CLI action that needs a live token — including the one-time Azure setup in #31 (creating Terraform remote state storage for test/staging, and eventually terraform init/apply for those stacks, since the azurerm provider goes through the same auth path).

Fix options

1. If this is a corporate/antivirus SSL-inspecting proxy: get the proxy's root CA certificate from IT and either import it into the Windows trusted root store, or point REQUESTS_CA_BUNDLE at a combined bundle that includes it.
2. Sidestep the local machine entirely: run the az commands from Azure Cloud Shell (portal.azure.com) or another machine that isn't behind the intercepting proxy.

Not attempted

Disabling TLS certificate verification (e.g. --debug flags or connection-verification overrides) — that defeats the purpose of the check and isn't a call to make unilaterally on an authenticated cloud CLI session without knowing whether the intercepting proxy is actually trusted.

[amar-python]

Root cause confirmed

$ echo | openssl s_client -connect login.microsoftonline.com:443 -servername login.microsoftonline.com | openssl x509 -noout -issuer -subject
issuer=OU=generated by Norton Antivirus for SSL/TLS scanning, O=Norton Web/Mail Shield, CN=Norton Web/Mail Shield Root
subject=C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=stamp2.login.microsoftonline.com

This is Norton Antivirus's own SSL/TLS scanning feature — a legitimate, already-installed security product on this machine, not a malicious MITM. Norton intercepts HTTPS to scan traffic and re-signs it with its own certificate, injecting its root CA (Norton Web/Mail Shield Root) into the Windows trusted root store. Confirmed present there via certutil -store Root. Windows-native apps (browsers using the OS cert store) trust this fine; Python's certifi bundle (used by the az CLI) is a separate, vendored list that has no knowledge of it.

What I tried

1. pip-system-certs (makes Python's ssl module defer to the OS trust store instead of certifi's bundled list) — the general, correct fix for exactly this class of problem, and it doesn't add any new trust, only aligns Python with certificates Windows (and by extension, this user) already trusts. Installed successfully into the system Python, but the az CLI ships its own isolated Python interpreter (C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe, invoked with -I, which ignores site-packages/env-based Python config by design) — so the system-Python fix doesn't reach it. Installing directly into az CLI's own site-packages requires admin rights to write under Program Files, which this session doesn't have.

2. Exported the Windows Root store to a PEM bundle (93 certs, includes Norton's), combined it with certifi's bundle, and pointed REQUESTS_CA_BUNDLE at it for the az invocation — this env var isn't blocked by -I isolation (it's read by the requests library, not Python's interpreter-level isolation). This got further: the "unable to get local issuer certificate" error went away, replaced by a new one:

   SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
   Basic Constraints of CA cert not marked critical
   

   Norton's self-generated interception root CA doesn't set the Basic Constraints X.509 extension as critical, which is technically non-compliant with strict CA certificate requirements — OpenSSL rejects it as an intermediate/leaf-signing CA even once it's in the trusted bundle. This is a defect in how Norton generated its own certificate, not something fixable by adjusting what's trusted.

Where this leaves things

Getting past this now requires either:

• Disabling or excluding Azure domains from Norton's SSL/TLS scanning (a Norton settings change, not something I can or should do from here), or
• Relaxing certificate validation strictness in some way — which I'm not doing unilaterally; that's a real security-relevant tradeoff for you to decide, not me.

Recommend: open Norton's settings and either disable "SSL scanning" / "Secure VPN traffic inspection" entirely, or check whether it supports excluding specific domains/apps (az, *.microsoftonline.com, *.azure.com) from inspection. Alternatively, run the Azure setup from a machine/network without Norton's SSL scanning enabled (e.g. Azure Cloud Shell in the browser, unaffected since it doesn't run a local az process).

[amar-python]

Note: even if this is resolved, local terraform commands are separately blocked by #34 (an unrelated provider-plugin IPC certificate issue, not an Azure/network problem). Fixing this one alone won't unblock #31 — see #34 too.

[amar-python]

Follow-up: checked the remaining local-workaround options — none pan out

Investigated the four items raised as possible remaining paths. Summary: no new local fix. Cloud Shell remains the answer.

1. WSL2

wsl --status shows the WSL platform itself is already enabled on this machine (Default Version: 2), but wsl -l -v reports zero installed distributions — there's no Linux userland to actually test Norton's interception against. Testing this option requires wsl --install <Distro> first, which is a real system change (installs a distro + downloads an image), so I didn't do it unilaterally.

This is the one item worth revisiting: if you're willing to install a distro (e.g. Ubuntu), it's genuinely worth then testing az/terraform from inside it — WSL2's NAT'd virtual network adapter may or may not sit behind Norton's interception layer, and that's not knowable without a distro present to test from. Flagging as your call, not doing it myself.

2. Docker Desktop

Not installed — no docker command, no Docker service, no install directory under Program Files. Same reasoning as WSL2: untestable without installing it first, and I'm not installing it without checking with you.

3. Newer az CLI version

Installed here: 2.87.0. Latest available: 2.89.1. But this is not a promising path:

• OS/system trust store support for az CLI is an open, backlog-status feature request (Azure/azure-cli#19305, #28050) — no PR, no milestone, not shipped in any released version.
• If anything, the trend is the opposite direction: Azure/azure-cli#32083, filed against 2.77.0, shows recent az releases tightening X.509 validation strictness (a different but analogous "CERTIFICATE_VERIFY_FAILED... Missing Authority Key Identifier" error), not relaxing it. The only workaround mentioned there is AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 — a verification bypass, which is off the table here for the same reason --insecure is.

Upgrading az CLI is not expected to resolve this and isn't worth doing on that basis.

4. Norton fix for the "Basic Constraints not critical" defect

No documented fix, KB article, or acknowledgment found in Norton's support site or community forums for this specific defect in their SSL/TLS scanning root CA generation. Separately confirmed (read-only registry check, no settings touched) that this machine's Norton 360 is already on a current, recently auto-updated build (26.7.11086.2615), alongside Norton Utilities Ultimate 26.6.18812.9304 and Norton AntiTrack 4.8.7182.14364 — so this isn't a stale-software gap; the defect is present in Norton's current release with no public sign they've fixed their own cert generation.

Conclusion

No new viable local fix. Confirming explicitly: Norton's settings were not touched, and no TLS/certificate verification was weakened or bypassed during this investigation — only read-only checks (wsl --status, wsl -l -v, Docker presence checks, registry reads, az version, public issue/doc research). Cloud Shell remains the recommended path for #31 until/unless a distro gets installed under WSL2 to test option 1, which is the only item left with any unexplored upside.

[amar-python]

Status update — where this stands

Tried so far (see comments above for full detail):

1. pip-system-certs on system Python — doesn't reach az CLI's isolated bundled Python (-I flag ignores site-packages; installing into its actual site-packages under Program Files needs admin rights not available in this session).
2. Combined Windows Root store + certifi bundle via REQUESTS_CA_BUNDLE — got past "unable to get local issuer certificate," but hit a deeper wall: Norton's self-generated root CA doesn't mark its Basic Constraints extension critical, which OpenSSL correctly rejects regardless of trust. Not fixable by adding more trust.
3. Investigated WSL2 (enabled, no distro installed — untested), Docker (not installed), newer az CLI version (2.89.1 vs installed 2.87.0 — OS-trust-store support confirmed still unshipped upstream, wouldn't help), and a Norton-side fix for the cert defect (no known fix, Norton already on current build). None panned out.

Recommended fix, decided on together: a Norton exclusion (SSL/TLS scanning or Firewall Program Control) for the specific processes hitting this:

• C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe (what az actually runs)
• Domains, if Norton supports domain-level exclusions instead: login.microsoftonline.com, management.azure.com, *.azure.com

Where we're stuck: this requires changing Norton's configuration, which I'm not able to do myself (modifying system/security settings is outside what I can perform, even with explicit direction — it needs to happen in Norton's own UI, by the machine's owner). Owner needs to add the exclusion manually; exact menu path varies by Norton 360 version.

Next step once the exclusion is added: re-run the verification checks (openssl s_client handshake against login.microsoftonline.com, az account show, az group show) to confirm it actually resolves the issue before touching #31.

[amar-python]

Steps to resolve — for the machine owner

I can't make this change myself (modifying security software settings isn't something I can do, even on request) — here's exactly what to do.

1. Open Norton 360's settings

Open Norton from the system tray or Start menu → Settings (gear icon).

2. Find the SSL/TLS scanning or program-exclusion setting

The exact menu path varies by Norton 360 version — look for one of these (try Norton's in-app settings search box for "SSL" or "exclusion" if you don't see them right away):

• Firewall → Program Control — lets you set a per-program rule; look for an option to exclude a specific program from traffic inspection/scanning
• Antivirus → Scans and Risks → Exclusions / Items to Exclude from Scans — lets you add a specific .exe to a trusted/excluded list

Per-program exclusion is preferable to any global "disable SSL scanning" toggle — it's narrower and doesn't drop protection for everything else.

3. Add this process to the exclusion

C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe

This is what az actually runs under the hood (az.cmd at C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd just wraps it).

4. If Norton only offers domain-based exclusions instead of per-program

Add these instead:

login.microsoftonline.com
management.azure.com
*.azure.com

5. Tell me once it's done

I'll re-run the exact checks used to diagnose this (openssl s_client handshake, az account show, az group show) to confirm it's actually fixed before we touch #31.

[amar-python]

Correction — "Firewall → Program Control" was wrong

Verified against Norton's own official documentation (Configuring Smart Firewall settings): Program Control governs network access permissions only (Allow/Block/Ask per app) — it has nothing to do with SSL/TLS scanning or certificate interception. Different feature. Sorry for sending you looking in the wrong place.

What's actually responsible

Norton's "Email Protection" feature. Confirmed via the certificate file path: Norton's own troubleshooting docs reference wscert.der at C:/ProgramData/Norton/AntiVirus/ as the Email Protection certificate — this matches exactly the wscert.pem path the #34 investigation found in this machine's NODE_EXTRA_CA_CERTS env var. Despite the name, it's evidently intercepting general HTTPS (port 443) traffic, not just email ports.

No documented per-program exclusion exists

Checked Norton's official support docs and community forums — found no setting to exclude a specific program or domain from this scanning. A Norton community moderator states directly: "they don't offer you a way to STOP Norton using its own certificate unfortunately." (source)

Norton's own official troubleshooting article for this exact certificate error (Troubleshooting email client warnings about invalid server certificates) offers only:

1. Import Norton's certificate into the app's trust store — a dead end here, already confirmed: Norton's cert is structurally rejected (missing critical Basic Constraints extension) regardless of whether it's trusted.
2. Reinstall Norton — unlikely to fix a certificate-generation defect.

The one real toggle found — broader than we wanted

Settings → Features tab → Antivirus → Email Protection tab (alternatively: Settings → Antivirus → Scans and Risks tab → "Email antivirus scan" toggle) turns Email Protection scanning off entirely — not scoped to just python.exe/terraform.exe. This is a bigger tradeoff than the narrow per-app exclusion we were hoping for: it affects whatever protection Email Protection provides more broadly, not just the two processes blocking this work.

Decision needed

• Try the broader Email Protection toggle if you're comfortable with that wider tradeoff — I can re-verify immediately after.
• Contact Norton support directly (chat/phone) to ask if there's an undocumented exclusion path — docs and forums didn't surface one, but support may know something not publicly documented.
• Skip this entirely and use Cloud Shell for One-time cloud setup needed before infra/terraform-test and infra/terraform-staging can be applied #31 — the zero-compromise path that was already the fallback recommendation.

Sources:

• Configuring Smart Firewall settings
• Where are SSL/TLS scanning settings in Norton 360 — Norton Community
• Troubleshooting email client warnings about invalid server certificates
• How to Disable or Turn Off Email Scanning in Norton 360

[amar-python]

Decision: declined the Email Protection toggle — Norton stays untouched, full stop. No further local fix is being pursued for this issue. Path forward for the underlying work (#31) is Azure Cloud Shell, which sidesteps this machine entirely.

[amar-python]

Final summary — all actions taken

Problem: az CLI fails on this dev machine with SSLError: certificate verify failed: unable to get local issuer certificate.

Root cause confirmed: Norton 360's "Email Protection" feature intercepts HTTPS (including login.microsoftonline.com) and re-signs it with its own generated root CA. That CA doesn't mark its Basic Constraints extension critical — a structural defect that OpenSSL correctly rejects regardless of whether it's trusted.

Actions taken, in order:

1. Installed pip-system-certs on system Python to defer to the Windows trust store — didn't reach az CLI's isolated bundled Python (-I flag ignores site-packages); fixing that needs admin rights to write under Program Files, not available here.
2. Exported the Windows Root store (93 certs, includes Norton's), combined with certifi's bundle, set REQUESTS_CA_BUNDLE — resolved the "unknown issuer" error but exposed the deeper "Basic Constraints not critical" defect, which is unfixable via trust configuration.
3. Investigated WSL2 (platform enabled, no distro installed — left untested rather than installing one unprompted), Docker Desktop (not installed, left alone), a newer az CLI version (2.89.1 vs installed 2.87.0 — researched upstream, OS-trust-store support confirmed still unshipped, wouldn't help), and whether Norton has a known fix for the cert defect (none found; Norton already on a current build).
4. Identified the actual responsible feature as Norton Email Protection (corrected initial wrong guidance pointing at Firewall → Program Control, which only governs network access, not SSL scanning).
5. Searched Norton's official docs and community forums for a per-program or per-domain exclusion — none exists. Norton's own troubleshooting article for this exact error offers only certificate import (dead end, per Bump actions/checkout from 4 to 7 #2 above) or reinstalling Norton.
6. Found one real, working toggle: Settings → Features → Antivirus → Email Protection tab, which disables Email Protection scanning entirely — broader than a scoped exclusion.

Decision: declined to change any Norton settings, including the broader toggle. Norton's configuration remains untouched throughout this entire investigation — no changes were made to it at any point.

Outcome: no local fix exists within the given constraints (don't touch Norton, don't weaken TLS verification). Closing as not planned — the path forward for the underlying work (test/staging cloud environment setup) is Azure Cloud Shell, which runs entirely outside this machine and isn't subject to this interception.