微軟公布 Microsoft 365 大當機原因:潛伏四個月的程式缺陷加上兩個意外,八成伺服器停擺
微軟(Microsoft)執行副總裁 Charles Lamanna 在 X 發文說明 11 月 12 日 Microsoft 365 全球中斷的原因,指出是三個因素同時發生:一個 2026 年 7 月引入、潛伏數月的程式缺陷,加上流量增加與伺服器容量減少,最低點時逾 80% 伺服器停止回應。
微軟負責 Copilot、代理人與平台業務的執行副總裁 Charles Lamanna 於美國時間 11 月 16 日在 X 發文,公布上週 Microsoft 365 全球大當機的初步原因,並表示「對所有客戶受到的影響深感抱歉」。這次中斷從 11 月 12 日美國上班尖峰時段開始,Outlook 郵件、行事曆與會議邀請都受波及,橫跨兩個工作天、歷時約 24 小時才全部恢復。
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.
First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments.
The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact.
Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.
Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts.
Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.
Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.
When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.
Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.
Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway.
Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.
After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region).
The final Post-Incident Review (PIR) will provide more details later this week.
— Charles Lamanna (@clamanna) November 16, 2026
Lamanna 指出,第一個因素是一個「潛在缺陷」:2026 年 7 月為 Microsoft 365 網路閘道伺服器開發的修正程式,會讓伺服器在特定負載條件下過載而失去回應。這項變更在全球運作三個多月都沒有出問題;微軟的測試沒抓到它,是因為實驗室環境沒有重現後來觸發錯誤的流量與硬體組合,而且這個錯誤在較新、效能較強的機器上更嚴重,測試卻多半集中在較弱的硬體。他也強調,這個缺陷與修正都不涉及一般 Windows 部署。
第二個因素是流量轉移。微軟為 Microsoft 365 維持兩套獨立的網路閘道系統,但在中斷發生前 15 小時,Teams 的流量從備用閘道切換到同時支援 Exchange Online 郵件服務的主閘道。第三個因素是容量不足:前一天另一個無關的問題,已讓該系統將近 14% 的伺服器容量離線。三者疊加後,請求不斷堆積,伺服器開始停止回應;系統自動把流量導向仍在運作的機器,結果連那些機器也被壓垮,「最低點時,超過 80% 的伺服器沒有回應」。另一個問題則讓防護機制無法把故障侷限在個別網段,最終擴散到全球。
復原之所以拖長,是因為伺服器一恢復就立刻被排隊中的流量淹沒而再度當機,微軟原本的自動化復原流程並不是為這種大範圍失效設計的;又因為錯誤源自數個月前的變更,平常的回滾、分批推送與功能旗標都無法快速撤銷。工程師最後套用修正程式,分批重啟伺服器再逐步恢復服務。Teams、OneDrive、SharePoint 與 Office 應用程式能改走其他基礎設施而較早恢復,但 Exchange Online 無法切換。Lamanna 先前在 11 月 13 日已表示,這「不是網路攻擊,也不是 AI 工具造成的問題」,最終完整報告將於本週稍晚發布。
影響範圍相當廣。據《衛報》(The Guardian)報導,中斷期間有監獄無法釋放受刑人、醫院改回紙本紀錄、學校與家長失聯、警察局收不到報案。Fox Business 報導,微軟的 Copilot AI 助理也受影響,印度航空公司 IndiGo 與 Akasa Air 回報客服電子郵件系統出問題,OpenAI 與 Anthropic 也回報與 Microsoft 365 整合的功能異常。Mashable 指出,受影響的企業可以向微軟申請停機的服務抵免。
對於微軟的解釋,外界的關注點已經轉向單一供應商的系統性風險。美國喬治亞州共和黨眾議員 Rich McCormick 在 X 表示,這次中斷暴露企業、醫院、學校與政府機關所依賴的數位系統有其弱點,「想像一下,如果這是惡意人士所為,後果會是什麼」。他藉此推動自己提出的《常識雲端安全法》(Common-Sense Cloud Security Act),主張美國要改善評估雲端風險、偵測威脅與在系統失效時控制損害的方式:「問題不只是如何防止下一次失效,而是失效發生時如何限縮衝擊範圍。」
資安公司 NCC Group 網路情報與應變副總裁 Matt Hull 則對《衛報》表示,這是「不只需要網路韌性,也需要營運韌性」的典型案例,醫院、學校與其他基本服務受影響的報導,說明不論原因為何,一次系統中斷的衝擊可以有多大。換言之,即使根因與駭客無關,企業對單一平台的依賴程度,仍是這次事件留下的核心問題。
Microsoft reveals hidden bug behind global Outlook and Microsoft 365 outage
Microsoft outage cause revealed: It was a perfect storm of problems
Microsoft identifies months-old bug as a root cause of global outage
(首圖來源:shutterstock)