跳到内容
现在

科技 · Microsoft 365 服务故障持续报道 · 第 5 篇,共 5 篇

微软将全球故障追溯到自身代码中7月出现的一个漏洞,但尚未解释为何该漏洞一直未被发现。

微软的初步报告称,该公司7月发布的一个缺陷、一次常规 Teams 流量转移,以及原本已减少的容量,共同导致超过80%的网关服务器瘫痪。报告说明了哪里出了问题,却没有说明为何无人发现该缺陷,也没有解决由谁买单的问题。

文字 阅读需 8 分钟 · 392 个来源

原语言视频

译自英文原文。

自上次报道以来的变化

4 天后

当时

微软将延迟邮件保护期限延长至48小时,服务故障尚未完全恢复

我们当时说尚无定论的问题

  1. 故障的起因、受影响用户和组织的审计总数、总损失以及连接器全面恢复服务的时间仍未查明。

    尚无定论

    微软的初步说法现已给出故障原因:7月出现的网关缺陷,在容量已经减少时被 Teams 流量转移触发。这一说法尚未经过独立核实。受影响用户和组织的审计统计、总损失,以及 OpenAI 和 Anthropic 的 Microsoft 365 连接器恢复服务的时间仍未公布。⁠3⁠4

  2. 随着流量回归,已恢复的容量能否继续保持响应——以及微软承诺发布的事故报告是否会解释多次恢复尝试失败的原因。

    已有答案

现在

微软的初步报告称,该公司7月发布的一个缺陷、一次常规 Teams 流量转移,以及原本已减少的容量,共同导致超过80%的网关服务器瘫痪。报告说明了哪里出了问题,却没有说明为何无人发现该缺陷,也没有解决由谁买单的问题。

此后已有答案

  1. 随着流量回归,已恢复的容量能否继续保持响应——以及微软承诺发布的事故报告是否会解释多次恢复尝试失败的原因。

    拉曼纳称,先前的恢复尝试之所以失败,是因为等待服务器恢复的流量再次将它们压垮。微软通过回滚代码并分批恢复服务器解决了问题;公司称,自11月14日宣布恢复以来,服务一直保持稳定。⁠4⁠8⁠27

新的进展

  • 微软所称客户受到实际影响的结束时间。
  • 微软表示,所有仍受影响的客户和地区服务均已恢复,并继续监控延迟邮件。
  • 初步报告发布;拉曼纳在 X 上致歉。

微软现在已经回答了上周故障期间究竟出了什么问题。但这留下了一个更难回答的问题:微软自己发布的缺陷为何在生产环境中运行数月而无人发现,随后又在不到两小时内导致全球大部分地区的电子邮件服务瘫痪。

11月13日,我们报道了微软恢复服务的尝试屡屡受挫,公司延长了电子邮件超时窗口以保护排队中的邮件,全面恢复尚未得到确认,故障原因也尚不明确。此后,微软表示,所有剩余客户和地区的服务均已恢复;11月16日星期一,公司发布了初步事故报告。报告将故障追溯至7月一次更新中的缺陷,受影响的是位于 Exchange Online、Outlook、Teams、Copilot 和其他 Microsoft 365 服务前端的网络网关服务器。⁠3⁠27

23小时50分钟微软所称客户受到实际影响的时段 38
80%+故障最严重时无响应的网关服务器 38
约14%故障发生前已离线的网关容量 32
740万+Downdetector 全球问题报告数,并非受影响人数 1230

三个故障因素同时出现

微软负责 Copilot、智能体和平台业务的执行副总裁查尔斯·拉曼纳在 X 上发表长文,解释了此次故障。他称这篇文章是微软 365 管理中心所提交报告的删节版。他表示,缺陷来自一项专为网关服务器编写的修复程序,在某些负载条件下会使服务器不堪重负并失去响应。对报告的报道将该缺陷描述为锁竞争:内部进程中的任务在相互等待时陷入阻塞,最终耗尽服务器的处理器和内存。iTnews 报道称,该漏洞在网关服务器群中隐藏了三个多月。⁠373⁠4⁠372⁠8

仅凭缺陷本身并不会造成故障。根据报告,还需要另外两个条件。故障发生前约15小时,一次计划中的 Teams 故障切换至 Exchange Online 网络基础设施,将额外流量推向共用网关。拉曼纳称,两个网络网关中有一个无法为 Teams 工作,导致主网关超载。与此同时,前一天发生的另一起无关问题已使部分网关容量离线。周四美国流量达到峰值时,请求不断堆积,直到服务器停止响应。⁠3⁠2⁠4

各媒体对这部分容量的规模说法不一。BleepingComputer 在报道初步报告时称,接近网关容量的14%。《卫报》援引拉曼纳的话称,超过服务器容量的14%。这两个数字相近,而报告本身讨论的是网关。⁠3⁠2

随后,微软自身的安全系统反而让情况恶化。自动重定向将请求从故障服务器转移到健康服务器上,但后者也有相同缺陷,继而同样失效。原本用于将故障限制在单个区域内的屏障未能发挥作用。根据 n-tv 对报告的报道,故障在不到两小时内蔓延至几乎整个系统;到欧洲时间周四晚间,相关机器中超过80%已无响应。GeekWire 报道称,微软将这种情况称为“惊群效应”。⁠5⁠371

关于80%的数字也有两种报道说法。BleepingComputer 称,超过80%的网络网关服务器受到影响;《国家报》则报道称,拉曼纳表示,微软在全球管理的服务器中超过80%受到影响。根据媒体报道,报告本身的措辞指的是网关服务器。如果微软运行的所有服务器中有五分之四故障,那将是规模完全不同、严重得多的事件。⁠3⁠6

对于给所有客户造成的影响,我们深表歉意,也认识到这对他们的日常业务和运营造成了影响。

查尔斯·拉曼纳 373

同样的情况也解释了为什么周四和周五的修复一再失败。拉曼纳表示,每次服务器恢复运行,等待处理的流量都会再次将它们压垮。最终奏效的修复方式是回滚代码,使受影响的代码恢复到上一个正常工作的状态。工程师将服务器移出服务,重启后分批恢复,而不是一次全部恢复。微软表示,此后已增加容量、改进监控并设置防护措施,但目前报道所述的初步报告并未详细列出这些改动。微软称,此次故障并非网络攻击、恶意活动或 AI 代理故障。⁠4⁠8⁠3⁠33

这一事件的说法在一周内发生了变化。周五,微软只表示问题是在此前一次网络容量升级期间引入的,服务总体正在恢复,但不知道何时能够完全恢复。周一的说法具体得多,但仍属初步报告,且仅来自微软。没有外部调查人员核实过这一说法。⁠1⁠15

从7月的修复到周一的解释

  1. 2026年7月

    网关服务器修复程序发布,其中带有一个潜伏缺陷。

    35
  2. 11月12日星期四,约01:20 UTC

    一次计划中的 Teams 故障切换将额外流量转移到共用的 Exchange Online 网关。

    3
  3. 11月12日星期四,上午11:20 ET

    微软所称客户受到实际影响的开始时间。

    3
  4. 11月12日星期四,17:54 GMT

    微软首次承认,美洲地区用户无法访问 Exchange Online 邮箱。

    1
  5. 11月13日星期五,上午11:10 ET

    微软所称客户受到实际影响的结束时间。

    3
  6. 11月14日星期六

    微软表示,所有仍受影响的客户和地区服务均已恢复,并继续监控延迟邮件。

    2710
  7. 11月16日星期一

    初步报告发布;拉曼纳在 X 上致歉。

    34

官方公布的影响时段也解释了为什么各方公布的持续时间互相矛盾。CBC 称超过24小时,MyBroadband 称20小时,《国家报》称超过18小时,Frandroid 称接近38小时。Frandroid 的计算从 Exchange Online 事故 EX1550543 开始;该媒体称事故于周四17:17报出,直到周六宣布恢复才结束。其他媒体计算的是客户受影响的时间,或沿途截取的时间点。微软给出的实际影响时段为23小时50分钟。有两个时间点与这一时段并不完全吻合:Ookla 的网络数据显示故障约在17:25 UTC 开始,而微软首次公开承认故障的时间比其所称的开始时间晚了94分钟。即使周六宣布一切恢复后,微软仍在监控延迟邮件,其电子邮件超时窗口已延长至48小时。⁠12⁠7⁠6⁠10⁠30⁠124

谁承受了这一天的影响,谁来买单

Downdetector 记录到来自70多个国家和地区的超过740万条问题报告,峰值接近每分钟20,000条。这些是投诉,不是受影响人数统计;微软也未公布有多少人或组织失去了服务。尽管如此,损失是实实在在的。英格兰的 NHS 工作人员重新使用纸质记录。爱尔兰卫生服务执行局(HSE)让 Healthmail 下线,并通知全科医生和药剂师,须按 GMS 计划开具纸质处方。美国法院和政府办公室、加拿大医院及铁路乘客受到影响;《国家报》报道称,日本、澳大利亚和马来西亚的关键服务也不得不中断。泰国开泰银行失去电子邮件服务,将有紧急需求的客户引导至呼叫中心,但移动银行业务是否受影响尚未得到确认。汤斯维尔一家股票经纪公司警告称,如果市场大幅波动,延迟收到的交易指令可能损害客户利益。英国网络安全部长表示,999急救电话、NHS 111一线服务和交通运输持续运行。⁠12⁠15⁠26⁠25⁠6⁠11⁠31

微软公布解释的当天,英国下议院在部长伊恩·默里主持下辩论了此次故障。现有记录并未显示辩论是否促成了任何承诺。德国联邦信息安全办公室(BSI)此前已称此次故障是对经济的一大打击,也是数字依赖问题的警报信号;石勒苏益格-荷尔斯泰因州数字事务部长迪克·施罗德则呼吁增强公共部门的数字韧性。⁠387⁠390⁠5

德国 IT 用户协会 VOICE 建议企业记录停机时间,并在12月底前提交服务抵扣申请;该协会还要求微软作出切实违约后果的可用性承诺。服务抵扣是订阅费的折扣。它无法弥补损失的工时、延误的处方或错失的订单;澳大利亚广播公司(ABC)采访的专家称,微软不太可能补偿企业因生产力损失而遭受的损失。损失金额均为估算。分析师马里奥·德·奇科估计,承保损失为$300 million至$500 million,对保险公司的影响有限;《24小时太阳报》报道称,有估算认为损失至少达数亿欧元。微软股价几乎没有变动:11月11日收盘价为$521.13,11月13日为$523.88,11月16日为$517.99;没有证据表明这次下跌与故障有关。⁠388⁠385⁠12⁠386⁠378

已知

  • 微软将此次故障归因于7月出现的网关缺陷;当部分容量已离线时,Teams 故障切换流量触发了该缺陷,最终通过代码回滚修复。 348
  • 微软表示,11月14日所有剩余客户和地区的服务均已恢复。 2726

未知

  • 为何该缺陷在生产环境中数月未被发现,以及区域隔离为何失效。
  • 有多少人和组织失去服务、总损失金额,以及微软是否给予了任何服务抵扣。

后续

  • 微软计划于11月20日星期五发布最终报告,预计将列明全面的防范措施。
  • 企业是否会在12月底前提交 VOICE 建议的抵扣申请。

时间线

第 8 天 · 5 篇报道

事件时间线
  1. 微软所称客户受到实际影响的结束时间。

  2. 微软表示,所有仍受影响的客户和地区服务均已恢复,并继续监控延迟邮件。

  3. 初步报告发布;拉曼纳在 X 上致歉。

来源244

在此处引用的来源中,flavorist.com最先发布,比本报道播出早6天。 随后还有196家媒体发布。

3篇早期文章作为背景资料参阅。46家媒体未注明日期:Workhorse Group, Inc.apnews.comcloud.microsoftreddit.comAFP.comAnadolu

持续报道 · 第 5 篇,共 5 篇

按媒体查看所有来源

来自 244 家媒体的 392 个来源· 引用最多的优先

播出原貌
  1. A bug Microsoft shipped in July sat hidden for months, then took email down worldwide.
  2. At the worst point, more than eight in ten of the servers that front Outlook, Teams and the rest of Microsoft 365 stopped responding. Health services went back to paper, and users in more than seventy countries lost their mail for nearly a full day.
  3. On November thirteenth we reported that Microsoft's attempts to bring servers back kept faltering, full restoration wasn't confirmed, and the cause was unknown. Since then, Microsoft has restored service and published a preliminary report that traces it all to a July defect.
  4. What Microsoft still hasn't explained is how that defect ran in production for months without anyone catching it, and nobody has settled who pays for the lost day.
  5. The defect came from a fix Microsoft wrote in July for its networking gateway servers, the machines that sit in front of Exchange Online, Outlook, Teams and Copilot.
  6. Charles Lamanna, the executive who runs Copilot, agents and platform, said that under certain load it caused servers to become overwhelmed and unresponsive. The report describes lock contention inside an internal process, where tasks jam waiting on each other until they eat up a server's processing power and memory.
  7. For months it did nothing. Then, last Thursday, three things lined up.
  8. First, the dormant defect. Second, a planned Teams failover about fifteen hours earlier pushed extra traffic onto the shared gateway. Third, nearly fourteen percent of gateway capacity was already offline from an unrelated problem. Then peak U.S. traffic hit Thursday, and requests piled up until servers froze.
  9. Microsoft's own safety systems then spread the damage. Automatic redirection moved requests off failing servers onto healthy ones, which carried the same defect and failed in turn. The barriers meant to keep trouble inside one region didn't hold.
  10. In under two hours the failure reached almost the entire system. By Thursday evening in Europe, more than eighty percent of the gateway servers were unresponsive. Microsoft calls the pattern thundering herds.
  11. That same pattern explains why the repairs kept failing. Each time servers came back, the traffic waiting for them knocked them over again.
  12. The fix was a code rollback that returned the affected code to its last working version. Engineers pulled servers out of service, restarted them, and brought them back in batches instead of all at once.
  13. Microsoft says it has since added capacity, improved monitoring and put in new safeguards. The preliminary report doesn't spell those changes out.
  14. Microsoft says the outage wasn't a cyberattack, malicious activity or an AI agent gone wrong. Lamanna apologized in a long post on X, saying the company was deeply sorry and recognizes what this did to customers' normal business.
  15. Microsoft's final report is due Friday, and we'll be reading it for how this bug slipped through. Subscribe to this channel to follow it, and the story continues at heyday dot news.
  16. Microsoft now puts active customer impact at twenty-three hours and fifty minutes.
  17. The clock starts at eleven-twenty Thursday morning Eastern and stops at ten past eleven Friday. Microsoft first acknowledged mailbox failures about ninety minutes after that start. The all-clear for the last customers came early Saturday, with delayed mail still being watched.
  18. Microsoft also stretched its email timeout window to forty-eight hours so queued messages wouldn't be lost.
  19. Published figures for the length run from more than eighteen hours, to twenty, to more than twenty-four, to nearly thirty-eight. They're measuring different things, from the hours customers were hit to the final all-clear. Microsoft's own impact window is twenty-three hours and fifty minutes.
  20. Microsoft's explanation sharpened over the week. On Friday it blamed only a problem from an earlier network capacity upgrade, and said it couldn't say when recovery would finish.
  21. Monday's account is preliminary, and it comes from Microsoft alone. No outside investigator has checked it. The capacity already offline is put at nearly fourteen percent or more than fourteen, and the report's own wording applies the eighty percent figure to gateway servers.
  22. NHS staff in England went back to paper records.
  23. Ireland's health service took Healthmail offline and told GPs and pharmacists to write paper prescriptions. U.S. courts and government offices were disrupted, along with Canadian hospitals and rail passengers. Japan, Australia and Malaysia had to interrupt critical services.
  24. In Thailand, Krungthai Bank lost email and sent urgent customers to its call center, with no confirmed hit to mobile banking. A stockbroker in Townsville warned delayed trade instructions could hurt clients if markets moved sharply. Britain's cyber minister said 999 calls, frontline NHS 111 services and transport kept running.
  25. Users filed more than seven point four million problem reports worldwide, peaking near twenty thousand a minute, from more than seventy countries and territories. Those are complaints, and Microsoft hasn't said how many people or organizations actually lost service.
  26. Experts in Australia say Microsoft is unlikely to compensate businesses for lost productivity.
  27. On the day Microsoft published its explanation, Britain's House of Commons debated the outage, led by minister Ian Murray. Germany's cyber-security agency had already called it an alarm signal about digital dependence, and Schleswig-Holstein's digital minister Dirk Schrödter called for more resilience in the public sector.
  28. Germany's IT user association, VOICE, is telling companies to document their downtime and file service-credit claims, and it wants availability commitments from Microsoft with real consequences. A service credit is a discount on the subscription fee. It doesn't cover lost hours, delayed prescriptions or missed orders.
  29. Loss figures are still estimates. One analyst put insured losses at three hundred to five hundred million dollars, with limited impact on insurers, and an Italian estimate ran to hundreds of millions of euros.
  30. Microsoft's stock barely moved. It closed at five hundred twenty-one dollars on November eleventh, about five twenty-four on Friday, and about five eighteen on Monday. Nothing ties that dip to the outage.
  31. Microsoft's final report is due Friday, November twentieth. It has to show how the defect sat undetected in production for months, why the regional barriers failed, and what Microsoft will change. Companies in Germany are being told to file their credit claims before the end of December.

译文更新于

近期播出