Azure ExpressRoute + VPN Gateway Outage: What Happened Across Multiple Regions and How to Build Resilience

Detection: 20:30–20:58 UTC

Customer impact began at 20:30. ExpressRoute connectivity monitoring detected degradation at 20:43, followed by broader control-plane health failures at 20:58.

Operational lesson: component and end-to-end connectivity monitoring must complement control-plane health signals.

Stabilization: 21:45–22:36 UTC

Microsoft paused further gateway operating-system updates at 21:45. Traffic shifting and added resources provided incomplete relief. By 22:36, affected ExpressRoute gateways had restored connectivity using locally stored last-known-good configurations.

Operational lesson: preserve evidence and avoid destructive gateway recreation while a provider mitigation is in progress.

Recovery: 23:55–02:15 UTC

Microsoft associated increased configuration-processing time with a previously deployed change at 23:55. The rollback began at 01:15 and completed across affected regions at 02:15.

Operational lesson: end the incident only after routing and application transactions remain stable through an agreed observation window.

1. Rule Out Your Own Side in the First Five Minutes

Check edge-interface state, BGP and IKE/IPsec sessions, recent firewall or routing changes, and connectivity-provider status. Compare more than one Azure region or path where available.

Decision: if a local fault or change explains the symptoms, follow the owned-device rollback or provider escalation path while continuing to watch Azure health.

2. Confirm Scope in Azure Service Health

Open Service Health, filter Service Issues by the affected subscriptions, services and regions, and inspect the Impacted Resources tab. Capture the tracking ID and subscribe the incident team to updates.

Decision: align your internal severity with confirmed business impact, not only the provider's incident label.

3. Fail Over to an Independent Path

Use a secondary circuit, another regional gateway, a separate Virtual WAN hub or a backup VPN only after confirming that its gateway, provider, region and routes do not share the active failure.

Validation: test route convergence, security policy, DNS, application transactions and return-path symmetry before declaring recovery.

4. Avoid Risky Gateway Changes

Do not delete or recreate a gateway during a suspected platform incident. Re-creation can be slow and can invalidate allowed public IP addresses or other dependencies.

Precaution: a gateway reset is disruptive and will not fix an Azure-side service fault. Use it only through an approved change after the platform is healthy.

5. Open a Support Case

Provide subscription, circuit, gateway and connection resource IDs; affected regions; the earliest UTC symptom; BGP/tunnel evidence; and the business impact. Ask Microsoft to associate the case with the relevant tracking ID.

Evidence: preserve portal exports, metrics, edge logs, screenshots, change records and provider case numbers.

6. Communicate and Monitor

Publish a fixed update cadence, state what is confirmed versus suspected, list the next decision point, and track primary and alternate paths throughout recovery.

Exit criteria: require stable routing and successful application tests for an agreed observation window before ending the incident.

az network vnet-gateway show — Inspect Gateway State

Use: confirm the gateway provisioning state, SKU and BGP setting before interpreting tunnel or routing symptoms.

az network vnet-gateway show \
  --resource-group MyRG \
  --name MyGateway \
  --query "{state:provisioningState,sku:sku.name,bgp:enableBgp}" \
  --output table

Interpretation: preserve the output, but do not assume a successful provisioning state proves that the data path is healthy.

az network vpn-connection show — Check VPN Connection Status

Use: view the connection state and transferred byte counters for a site-to-site connection.

az network vpn-connection show \
  --resource-group MyRG \
  --name MyVpnConnection \
  --query "{status:connectionStatus,ingress:ingressBytesTransferred,egress:egressBytesTransferred}" \
  --output table

Interpretation: correlate NotConnected and stalled counters with IKE, tunnel and on-premises firewall logs.

az network vnet-gateway list-bgp-peer-status — Verify BGP Peers

Use: retrieve the current state of BGP peers on the virtual network gateway.

az network vnet-gateway list-bgp-peer-status \
  --resource-group MyRG \
  --name MyGateway \
  --output table

Interpretation: compare peer state, connected duration and route counts with the on-premises router. A disconnected IPsec tunnel prevents BGP over that VPN path.

az network express-route show — Inspect Circuit Provisioning

Use: compare Azure and service-provider provisioning states for the ExpressRoute circuit.

az network express-route show \
  --resource-group MyRG \
  --name MyCircuit \
  --query "{circuit:circuitProvisioningState,provider:serviceProviderProvisioningState}" \
  --output table

Interpretation: provisioning state is one input; also inspect ARP availability, BGP availability, traffic and provider evidence.

az network express-route list-route-tables — Inspect Learned Routes

Use: view private-peering routes through the primary ExpressRoute path.

az network express-route list-route-tables \
  --resource-group MyRG \
  --name MyCircuit \
  --peering-name AzurePrivatePeering \
  --path primary \
  --output table

Interpretation: compare primary and secondary paths and confirm that the prefixes needed by the application exist in both directions.

az network vnet-gateway reset — Disruptive Recovery Action

Use only after review: reset a VPN gateway when Azure is healthy but tunnels still do not re-establish and your change authority approves the interruption.

az network vnet-gateway reset \
  --resource-group MyRG \
  --name MyGateway

Warning: this action interrupts connectivity. Capture state first, notify stakeholders, define rollback and never use reset as a reflex during an active platform incident.