Skip Navigation

Azure status history

This page contains Post Incident Reviews (PIRs) of previous service issues, each retained for 5 years. From November 20, 2019, this included PIRs for all issues about which we communicated publicly. From June 1, 2022, this includes PIRs for broad issues as described in our documentation.

Product:

Region:

Date:

July 2026

7/23

Preliminary Post Incident Review (PIR) - Issues connecting to resources in West US (Tracking ID ZJV6-SGG)

This is our Preliminary PIR to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details.

What happened?

Between 14:44 UTC and 19:41 UTC on 23 July 2026, customers may have experienced connectivity failures, increased latency, or difficulty accessing Azure and other Microsoft cloud services hosted within the West US region. Impact was limited to network traffic entering or exiting the West US region; traffic that remained entirely within the region was not affected.

Impacted services included, but may have not been limited to:

App Service, Application Gateway, Application Insights, Azure AD B2C, Azure AI Search, Azure AI Speech, Azure API Management, Azure Bastion, Azure Bot Service, Azure Cosmos DB, Azure Data Explorer, Azure Database for PostgreSQL, Azure Databricks, Azure Firewall, Azure Kubernetes Service (AKS), Azure Monitor, Azure Virtual Desktop, Azure VMware Solution, ExpressRoute Circuits, ExpressRoute Gateways, Log Analytics, Microsoft Graph, Microsoft Sentinel, Partner Center, Power BI Embedded, Virtual WAN, VPN Gateway

What do we know so far?

On July 23, 2026, a routine device maintenance required isolating specific network paths. Our maintenance process converts these requests into system-readable requests and verifies that at least one of the two redundant paths remains healthy. Before maintenance begins, these requests undergo safety checks to confirm the work will be impact-less. In this case, a bug in the request conversion system incorrectly marked additional devices as a part of the maintenance event and caused a set of IP routes to be removed from more devices than intended. The routes were removed between our datacenter and wide-area network, impacting traffic entering or exiting the region. 

At 14:44 UTC, customers started experiencing impact and the engineering team was engaged immediately. The issue initially presented as large-scale route churn in our Wide-Area Network (WAN). It was later found that route removal was from a datacenter in the West US region. Once confirmed, engineers began roll back at 17:45 UTC. By 18:26 UTC, the network was restored and healthy, and all impacted services had fully recovered by 19:41 UTC.

How did we respond?

  • 14:44 UTC – Device maintenance began. Earliest time when customers may have begun to experience impact. Multiple Azure services began to detect and correlate service degradation.
  • 14:45 UTC - Networking, service teams, and incident responders begin reviewing traffic anomalies, routing behavior, packet loss signals, and recent changes.
  • 15:00 - 16:00 UTC - We validated impact reports and telemetry to determine impact scope and blast radius. Began ruling out components, including addressing the large-scale route churn occurring in our WAN.
  • 16:00 - 17:45 UTC - Following continued degradation, we continued narrowing down our investigation. Identifying abnormal network routing behavior due to indications of improper advertising of some routes. Then identified the recent fiber maintenance activity, which we correlated to the identified routing behavior.
  • 17:45 UTC – We initiated the rollback of the maintenance change request.
  • 18:26 UTC – We completed the rollback and began monitoring recovering telemetry and services.
  • 19:41 UTC - All services recovered. Networking confirmed services can failback to West US as needed.

What happens next?

  • We will be preforming a full analysis focusing on safety checks, automated maintenance request change process, and more as we progress through our post mitigation internal retrospective.
  • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability may vary between customers and resources – for guidance on implementing monitoring to understand granular impact:
  • For mission-critical workloads, customers should consider a multi-region deployment strategy to maintain availability during events that affect a single region: and
  • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review:
  • Consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:
  • This is our Preliminary PIR to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details.

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

May 2026

5/29

Post Incident Review (PIR) – Azure OpenAI Service – Latency and intermittent failures in multiple regions (Tracking ID LYXT-C1Z)

Watch our 'Azure Incident Retrospective' video about this incident:

What happened?

Between 09:39 UTC and 17:05 UTC on 29 May 2026, customers using the Azure OpenAI Service may have experienced increased latency, intermittent request failures, timeouts, and/or HTTP 5XX errors when submitting inference requests. The impact was more pronounced in regions in Europe and Australia East that were directly processing the highest levels of affected traffic.

The incident was triggered by an upstream change that altered how certain capacity-related failures were surfaced, which led to a rapid and unexpected increase in internal retry traffic. This retry amplification overwhelmed a shared inference load balancer component, which distributes the inference traffic, degrading service reliability across multiple regions.

What went wrong and why?

Azure OpenAI relies on an internal routing and load balancing layer to direct inference requests across regions and model capacity. This layer is designed to distribute traffic efficiently and support resiliency during transient failures. During this incident, that layer became overloaded and unstable after it was subjected to a large volume of retry traffic from an upstream internal workload.

The triggering condition was a change in an upstream API layer used by a Microsoft first-party workload, Microsoft 365, which relies on Azure OpenAI for some request processing scenarios, including Copilot-related experiences. This change altered how capacity-related failures were surfaced to higher layers in the request path. The rollout first occurred in Australia East, and later expanded to Sweden Central.

Before this change, some lower-level capacity or resource exhaustion conditions could still result in retry activity, as designed - but after repeated failures, they would ultimately return in a form that did not trigger broad retry behavior across multiple upstream layers. After the change, those same conditions began surfacing as transient server failures instead. Because those responses were now interpreted as retriable, multiple layers in the calling stack retried immediately and repeatedly, without sufficient backoff or jitter. In some cases, a single failed request resulted in up to 48 additional retry attempts. This created a cascading retry amplification effect that dramatically increased internal traffic volume, even though underlying customer demand had not materially changed.

The first wave of impact occurred in Australia East, where the change was initially deployed. As retry traffic increased, it began exhausting resources in the Azure OpenAI inference routing layer and triggered failures in the affected backend path. During that first wave, early crash signatures pointed to an internal feature associated with error handling, which appeared to be a plausible explanation for the instability. We disabled that feature as a potential mitigation, and around the same time traffic in Australia East naturally declined as part of the normal regional usage pattern. Because service health improved after those two things happened in close succession, it initially appeared that the mitigation had addressed the problem. In reality, the feature we disabled was not the underlying cause, and the retry amplification issue remained in place.

When the upstream rollout later expanded to Sweden Central, the same failure pattern reappeared, but at much greater scale. As a major European processing hub, Sweden Central serves significantly higher traffic volume - so the retry amplification effect was stronger and the resulting overload was more severe. This second wave made it clear that the issue was not limited to the internal feature we had first suspected. Instead, it exposed the true underlying problem: a large internal retry storm overwhelming the shared routing layer.

As the retry storm intensified, it exhausted resources in the Azure OpenAI inference routing layer. The affected component scaled out, but the magnitude of the retry amplification exceeded the protection available at the time. In a small number of regions, recovery was further constrained because scale-out options were limited by available compute quota for replacement virtual machine SKUs. The result was resource exhaustion, out-of-memory crashes on multiple instances, and broad degradation across the shared routing path.

This incident also exposed a gap in how internal workloads were protected, relative to traffic from external sources. The affected upstream workload operated in a model where traffic was not subject to the same rate-limiting and overload controls used for many external scenarios. In addition, some of this traffic reached the internal routing layer more directly than standard external traffic paths. That meant the system did not sufficiently suppress or isolate the amplified retry volume before it reached shared infrastructure.

While the initial impact in Australia East led us toward a credible but incomplete explanation, the apparent recovery after disabling the suspected feature obscured the real cause. Once the rollout reached Sweden Central and the issue reappeared, albeit at greater scale, we were able to definitively correlate the degradation to retry amplification, correctly diagnose the trigger condition as the change in the upstream dependency API layer, then execute the targeted mitigations that restored stability.

How did we respond?

  • 09:20 UTC on 29 May 2026 – Initial customer impact began as the first wave of retry amplification, associated with the upstream rollout in Australia East, caused increased failures and latency across multiple regions.
  • 09:39 UTC on 29 May 2026 – Automated monitoring detected a drop in request success rates and initiated incident investigation.
  • 10:10 UTC on 29 May 2026 – We identified widespread service instability and correlated failures across regions and models.
  • 12:17 UTC on 29 May 2026 – Based on early crash diagnostics, we disabled a component that was suspected of contributing to the instability.
  • 12:40 UTC on 29 May 2026 – Service health began improving as traffic conditions in Australia East eased, which initially made the earlier mitigation appear effective.
  • 14:20 UTC on 29 May 2026 – The upstream rollout expanded to Sweden Central, where significantly higher traffic volume increased the impact of the retry amplification.
  • 14:40 UTC on 29 May 2026 – The issue reoccurred after the upstream rollout reached Sweden Central, where higher traffic volume magnified the retry amplification and overloaded the system at greater scale.
  • 16:20 UTC on 29 May 2026 – We identified that the earlier mitigation had addressed a secondary symptom rather than the underlying cause, and refocused the investigation on upstream retry behavior.
  • 16:27 UTC on 29 May 2026 – We confirmed that the amplified traffic was originating from an internal first-party workload and began targeted mitigation planning.
  • 16:30 UTC on 29 May 2026 – We identified the source of the amplified retry traffic, isolated the offending internal workload onto dedicated infrastructure, and coordinated with the upstream service to roll back the triggering change and reduce traffic volume.
  • 16:25 UTC on 29 May 2026 – The upstream service began rollback of the triggering change and took additional actions to reduce retry-driven traffic volume.
  • 17:05 UTC on 29 May 2026 – Following the completion of the isolation and rollback, and all backlogged requests were cleared, customer impact was confirmed as mitigated.

How are we making incidents like this less likely or less impactful?

  • We are reducing shared infrastructure blast radius by moving large first-party generative AI workloads onto dedicated routing infrastructure, rather than allowing them to share the same inference load balancing stamp with broader multi-tenant traffic. (Estimated completion: June 2026)
  • We are improving our diagnostic guidance, runbooks, and alerting so that engineers can more quickly distinguish between secondary crash symptoms and the true underlying source of overload during fast-moving incidents. (Estimated completion: June 2026)
  • We are partnering with upstream service teams to improve retry policy design, dependency contracts, and telemetry so retry amplification conditions can be detected earlier and prevented from propagating across shared infrastructure. (Estimated completion: July 2026)
  • We are implementing stronger overload prevention and workload throttling controls so that excessive retry traffic from any single internal workload cannot overwhelm shared components. (Estimated completion: July 2026)
  • Finally, we are continuing work to identify and eliminate single points of failure and improve resiliency in the inference routing layer. (Perpetual/Ongoing)

How can customers make incidents like this less impactful?

  • For mission-critical AI workloads, customers should consider a multi-region resiliency strategy so that service degradation affecting a specific processing path or region is less likely to impact the full application experience. Customers using regional deployments may wish to evaluate whether additional geographic diversity or fallback patterns would improve resilience for their scenarios. See:
  • Customers should also review application retry behavior to ensure that client-side retries use appropriate backoff and jitter, and that applications degrade gracefully when dependent services are under stress. Well-controlled retry behavior helps avoid compounding platform-level recovery events. See:
  • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact:
  • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

5/29

Post Incident Review (PIR) – Multiple services – Power/cooling issues in West US 2 (Tracking ID GHRP-84G)

Watch our 'Azure Incident Retrospective' video about this incident:

What happened?

Between 04:24 UTC on 29 May and 02:30 UTC on 30 May 2026, customers with resources in the West US 2 region may have experienced failures when attempting to access or manage some Azure services in the region. Impact included connectivity failures, resource unavailability, and service management failures.

The issue was triggered by a severe thunderstorm that caused utility power disturbances affecting multiple datacenter facilities serving the region. These disturbances led to some cooling systems entering a protective mode to preserve equipment integrity. This caused elevated temperatures which triggered proactive shutdowns of infrastructure affecting compute, networking, and storage systems across two different availability zones in the region (physical AZ-01 and AZ-03). After cooling systems stabilized, storage services - and the networking on which they depend - required extensive manual recovery actions, which contributed to the total duration of customer impact.

The following services were among those affected: Application Insights, Azure App Service, Azure Backup, Azure Cache for Redis, Azure Container Registry, Azure Cosmos DB, Azure Databricks, Azure Data Explorer, Azure Data Factory, Azure Functions, Azure IoT Hub, Azure Kubernetes Service, Azure Monitor, Azure NetApp Files, Azure Resource Manager, Azure Site Recovery, Azure SQL Database, Azure Storage, Azure Synapse Analytics, Azure Virtual Machines, Log Analytics, Service Bus, and Virtual Machine Scale Sets.

What went wrong and why?

A severe thunderstorm produced lightning strikes, resulting in utility power disturbances across multiple datacenter facilities serving the West US 2 region. These disturbances affected a wide enough geographic area that a subset of datacenters within two Availability Zones experienced impact. While this event was not a complete utility power interruption, these datacenters experienced multiple voltage sag/swell events, meaning utility power remained present but was unstable. The region consists of datacenters built at different times with varying infrastructure designs, including differences in cooling control systems and power protection mechanisms. Some datacenters transferred to generator power, while others did not transfer, as the utility power source remained available even as voltage fluctuated.

Independently of whether each datacenter transferred to backup power, a subset of components within the mechanical cooling systems - across multiple datacenters - detected abnormal electrical conditions and, by design, entered a 'lockout' protective state. This reduced cooling capability and prevented parts of the cooling system from automatically returning to normal operation after the power disturbance. As temperatures rose beyond safe operating thresholds, a subset of cloud infrastructure automatically shut down to prevent physical damage and preserve data integrity.

Impacted cooling systems required manual intervention and troubleshooting, including multiple restart attempts before stable cooling capacity could be reestablished. Additionally, we had to identify which devices required manual recovery and which were expected to self-recover. Impacted Storage scale units required additional validation. These factors contributed to the extended recovery time for services dependent on storage, including telemetry processing systems.

How did we respond?

Recovery efforts began immediately, prioritizing cooling restoration, then network infrastructure, followed by compute and storage systems. Cooling was restored within approximately two hours of impact start. The majority of compute resources recovered within eight hours. Storage validation, which must be performed sequentially, extended over approximately 14 hours. Finally, telemetry services (Application Insights and Log Analytics) required an additional six hours beyond storage recovery to process accumulated backlogs. The timeline below details the progression of recovery:

  • 04:14 UTC on 29 May 2026 – Utility power sag/swells were observed across multiple datacenters. These caused a subset of the mechanical cooling systems to enter a 'lockout' protective state.
  • 04:24 UTC on 29 May 2026 – Customer impact began; our monitoring produced thermal alerts as temperatures rose due to reduced cooling functionality.
  • 04:31 UTC on 29 May 2026 – Manual cooling diagnosis and restoration efforts began.
  • 04:40 UTC on 29 May 2026 – As temperatures rose, some cloud infrastructure shut down to prevent damage.
  • 04:50 UTC on 29 May 2026 – Networking and Storage engineers began assessing infrastructure health and necessary restoration efforts.
  • 05:55 UTC on 29 May 2026 – All cooling was restored in impacted datacenters, and temperatures had stabilized. Assessments continued on remaining infrastructure issues.
  • 06:15 UTC on 29 May 2026 – Approximately 50% of impacted underlying Virtual Machines (VMs) had recovered.
  • 12:00 UTC on 29 May 2026 – Approximately 95% of impacted underlying Virtual Machines (VMs) had recovered, after premium Storage had been largely recovered. Compute and Storage teams continued to identify and mitigate remaining unhealthy nodes.
  • 18:46 UTC on 29 May 2026 – Storage was confirmed as the remaining recovery bottleneck; validation of Storage-specific issues was ongoing.
  • 20:18 UTC on 29 May 2026 – All affected services confirmed as recovered, except for Application Insights and Log Analytics which were still processing backlogs.
  • 02:30 UTC on 30 May 2026 – Customer impact for Log Analytics and Application Insights confirmed as mitigated.

How are we making incidents like this less likely or less impactful?

  • First and foremost, we are reviewing and enhancing our operational processes for cooling system recovery based on learnings from this incident, to ensure that all employees have clear and documented actions to support faster recovery during high-stress scenarios. (Estimated completion: June 2026)
  • Our Log Analytics team is improving the resiliency of service startup processes so that transient initialization failures self-recover automatically, reducing the need for manual intervention during large-scale restarts. (Estimated completion: June 2026)
  • Our Networking team is addressing platform limitations that constrained the speed at which storage nodes could be returned to service following the thermal event. This includes reducing dependencies on manual coordination steps, and enabling faster parallel recovery of storage infrastructure. (Estimated completion: July 2026)
  • Our Networking team is also improving the tooling and processes used to identify which infrastructure devices have entered protective states and require manual recovery. This includes developing clearer prioritization of foundational devices (such as management and networking components) earlier in the recovery sequence, and improving awareness across teams when thermal events affect shared infrastructure. (Estimated completion: August 2026)
  • Our Log Analytics team is improving the architectural resiliency of our telemetry processing services by enabling multi-cluster flexibility and support for a broader pool of compute resource types, reducing dependence on any single cluster or resource type's availability and improving the ability to recover during large-scale scenarios. (Estimated completion: August 2026)
  • Finally, in the longer term we are performing a joint, holistic engineering review to evaluate the fail-state behavior of the cooling system protection logic. The current system prioritizes equipment protection and data security over operational continuity, which resulted in a full lockout rather than staged or degraded operation. This includes evaluating proactive row-level shutdown strategies during rapid temperature increases, and expanding the decision-making window available to operators before protective systems engage. (Estimated completion: October 2026)

How can customers make incidents like this less impactful?

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey: