Skip to content
Now

Tech · Microsoft 365 outageContinuing story · report 5 of 5

Microsoft traces its global outage to a July bug in its own code. It hasn't explained how the bug went unnoticed.

Microsoft's preliminary report says a defect it shipped in July, a routine Teams traffic shift and capacity that was already reduced together knocked out more than 80% of its gateway servers. The report says what broke. It doesn't say why nobody caught the defect, and it doesn't settle who pays.

Written 7 min read · 392 sources

What changed since our last report

4 days later

Then

Microsoft protects delayed mail for 48 hours as outage recovery remains incomplete

What we said was still open

  1. The initiating cause, audited totals of affected users and organizations, aggregate losses and full connector-restoration times remain unresolved.

    Still open

    Microsoft's preliminary account now gives a cause: a July gateway defect, triggered by a Teams traffic shift while capacity was already reduced. That account has not been independently checked. Audited counts of affected users and organizations, aggregate losses, and the restoration times for OpenAI's and Anthropic's Microsoft 365 connectors are still missing.⁠3⁠4

  2. Whether restored capacity stays responsive as traffic returns—and whether Microsoft’s promised incident report explains why repeated recovery attempts failed.

    Answered

Now

Microsoft's preliminary report says a defect it shipped in July, a routine Teams traffic shift and capacity that was already reduced together knocked out more than 80% of its gateway servers. The report says what broke. It doesn't say why nobody caught the defect, and it doesn't settle who pays.

Answered since

  1. Whether restored capacity stays responsive as traffic returns—and whether Microsoft’s promised incident report explains why repeated recovery attempts failed.

    Lamanna said earlier recovery attempts failed because traffic waiting for returning servers overwhelmed them again. Microsoft resolved this by rolling back the code and bringing servers back in groups, and it says service has held since restoration was announced on November 14.⁠4⁠8⁠27

What moved

  • Microsoft's stated end of active customer impact.
  • Microsoft says service is restored to the remaining affected customers and regions and keeps monitoring delayed mail.
  • Preliminary report published; Lamanna apologises on X.

Microsoft has now answered the question of what broke during last week's outage. That leaves a harder question: how a defect Microsoft shipped itself ran in production for months without anyone catching it, and then took down email for much of the world in under two hours.

On November 13 we reported that Microsoft's attempts to restore service kept faltering, that the company had extended its email timeout window to protect queued mail, that full restoration had not been confirmed, and that the cause was unknown. Since then Microsoft has said service is restored to all remaining customers and regions, and on Monday, November 16, it published a preliminary post-incident report. The report traces the failure to a defect in a July update to the networking gateway servers that sit in front of Exchange Online, Outlook, Teams, Copilot and other Microsoft 365 services.⁠3⁠27

23h50mMicrosoft's stated window of active customer impact 38
80%+Gateway servers unresponsive at the worst point 38
~14%Gateway capacity already offline beforehand 32
7.4m+Downdetector problem reports worldwide, not people 1230

Three faults that lined up

Charles Lamanna, Microsoft's executive vice-president for Copilot, agents and platform, set out the explanation in a long post on X, which he called an abridged version of the report filed in the Microsoft 365 admin center. He said the defect came from a fix written specifically for the gateway servers, and that under certain load conditions it caused servers to become overwhelmed and unresponsive. Coverage of the report describes the defect as lock contention: tasks inside an internal process jam while waiting on each other until they exhaust a server's processor and memory. iTnews reports that the bug sat hidden in the gateway fleet for more than three months.⁠373⁠4⁠372⁠8

The defect alone did nothing. According to the report, it took two more conditions. About 15 hours before the outage, a planned Teams failover onto the Exchange Online networking infrastructure pushed extra traffic onto the shared gateway. Lamanna described this as one of two networking gateways not working for Teams, which overloaded the primary one. Separately, an unrelated problem from the previous day had already taken a slice of gateway capacity offline. When peak U.S. traffic arrived on Thursday, requests piled up until servers stopped responding.⁠3⁠2⁠4

The outlets disagree on how large that slice was. BleepingComputer, reporting the preliminary report, puts it at nearly 14% of gateway capacity. The Guardian, quoting Lamanna, says more than 14% of server capacity. The two figures are close, and the report's own framing concerns the gateway.⁠3⁠2

Microsoft's own safety systems then made things worse. Automatic redirection moved requests off failing servers onto healthy ones that carried the same defect, and those failed in turn. The barriers meant to keep trouble inside a single region did not hold. According to n-tv's account of the report, the failure reached almost the entire system in under two hours, and by Thursday evening European time more than 80% of the relevant machines were unresponsive. GeekWire reports that Microsoft calls this dynamic "thundering herds."⁠5⁠371

The 80% figure has also been reported in two ways. BleepingComputer gives it as more than 80% of the networking gateway servers, while El País reported Lamanna as saying more than 80% of the servers Microsoft manages worldwide. The report's own wording, as covered, refers to gateway servers. A failure of four-fifths of everything Microsoft runs would be a different and much larger event.⁠3⁠6

We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.

Charles Lamanna 373

The same dynamic explains why repairs kept failing on Thursday and Friday. Lamanna said that each time servers came back, the traffic waiting for them overwhelmed them again. The fix that worked was a code rollback that returned the affected code to its last working state. Engineers pulled servers out of service, restarted them and brought them back in groups instead of all at once. Microsoft says it has since added capacity, improved monitoring and put in safeguards, but the preliminary report as covered does not list those changes in detail. Microsoft says the outage was not a cyberattack, malicious activity or an AI agent malfunction.⁠4⁠8⁠3⁠33

The story has shifted over the week. On Friday Microsoft pointed only to an issue introduced during an earlier network capacity upgrade, said services were broadly recovering, and said it did not know when recovery would be complete. Monday's account is far more specific, but it is still preliminary and comes from Microsoft alone. No outside investigator has checked it.⁠1⁠15

From a July fix to a Monday explanation

  1. July 2026

    A fix for the gateway servers ships, carrying a latent defect.

    35
  2. Thu Nov 12, ~01:20 UTC

    A planned Teams failover pushes extra traffic onto the shared Exchange Online gateway.

    3
  3. Thu Nov 12, 11:20 a.m. ET

    Microsoft's stated start of active customer impact.

    3
  4. Thu Nov 12, 17:54 GMT

    Microsoft first acknowledges that users in the Americas cannot reach Exchange Online mailboxes.

    1
  5. Fri Nov 13, 11:10 a.m. ET

    Microsoft's stated end of active customer impact.

    3
  6. Sat Nov 14

    Microsoft says service is restored to the remaining affected customers and regions and keeps monitoring delayed mail.

    2710
  7. Mon Nov 16

    Preliminary report published; Lamanna apologises on X.

    34

The official window also explains why published durations conflict. CBC said more than 24 hours, MyBroadband 20 hours, El País more than 18 hours, and Frandroid nearly 38 hours. Frandroid measured from the Exchange Online incident, EX1550543, which it says was opened Thursday at 17:17, to Saturday's restoration announcement. The others measured customer impact or snapshots taken along the way. Microsoft's own impact window is 23 hours and 50 minutes. Two timestamps do not sit neatly with it: Ookla's network data put the onset around 17:25 UTC, and Microsoft's first public acknowledgement came 94 minutes after its stated start. Even after Saturday's all-clear, Microsoft was still watching delayed mail under an email timeout window it had stretched to 48 hours.⁠12⁠7⁠6⁠10⁠30⁠124

Who absorbed the day, and who pays

Downdetector logged more than 7.4 million problem reports from more than 70 countries and territories, peaking near 20,000 a minute. Those are complaints, not a headcount, and Microsoft has not said how many people or organizations lost service. The damage was concrete all the same. NHS staff in England went back to paper records. Ireland's HSE took Healthmail offline and told GPs and pharmacists to write paper prescriptions under the GMS scheme. U.S. courts and government offices, Canadian hospitals and rail passengers were disrupted, and El País reports that Japan, Australia and Malaysia had to interrupt critical services. Thailand's Krungthai Bank lost email and sent urgent customers to its call center, with no confirmed effect on mobile banking. A Townsville stockbroker warned that delayed trade instructions could hurt clients if markets moved sharply. Britain's cyber minister said 999 calls, frontline NHS 111 services and transport kept running.⁠12⁠15⁠26⁠25⁠6⁠11⁠31

On the day Microsoft published its explanation, the House of Commons debated the outage, led by minister Ian Murray. The available record does not show what commitments, if any, came out of the debate. Germany's federal cyber-security agency, the BSI, had already called the outage a major problem for the economy and an alarm signal about digital dependence, and Schleswig-Holstein's digital minister Dirk Schrödter called for more digital resilience in the public sector.⁠387⁠390⁠5

Germany's IT user association VOICE is telling companies to document their downtime and file service-credit claims by the end of December, and it wants availability commitments from Microsoft that carry real consequences. A service credit is a discount on the subscription fee. It does not pay for lost hours, delayed prescriptions or missed orders, and experts told Australia's ABC that Microsoft is unlikely to compensate businesses for lost productivity. Loss figures are estimates. Analyst Mario De Cicco put insured losses at $300 million to $500 million, with limited impact on insurers, and Il Sole 24 Ore reported estimates of at least hundreds of millions of euros. Microsoft's stock barely moved: it closed at $521.13 on November 11, $523.88 on November 13 and $517.99 on November 16, and nothing ties that dip to the outage.⁠388⁠385⁠12⁠386⁠378

Known

  • Microsoft attributes the outage to a July gateway defect, triggered by Teams failover traffic while some capacity was already offline, and fixed by a code rollback. 348
  • Microsoft says service was restored to all remaining customers and regions on November 14. 2726

Unknown

  • How the defect went undetected in production for months, and why regional isolation failed.
  • How many people and organizations lost service, total losses, and whether Microsoft has granted any service credits.

Next

  • Microsoft's final report, which it has scheduled for Friday, November 20, should set out its full prevention measures.
  • Whether companies file the credit claims VOICE recommends before the end of December.

Timeline

Day 8 · 5 reports

Story timeline
  1. Microsoft's stated end of active customer impact.

  2. Microsoft says service is restored to the remaining affected customers and regions and keeps monitoring delayed mail.

  3. Preliminary report published; Lamanna apologises on X.

Sources244

Of the sources cited here, flavorist.com published first, 6 days before this report aired. 196 more outlets followed.

3 older pieces read for background.46 outlets with no date:Workhorse Group, Inc.apnews.comcloud.microsoftreddit.comAFP.comAnadolu

Continuing story · report 5 of 5

Every source, by outlet

392 sources from 244 outlets· Most cited first

As aired
  1. A bug Microsoft shipped in July sat hidden for months, then took email down worldwide.
  2. At the worst point, more than eight in ten of the servers that front Outlook, Teams and the rest of Microsoft 365 stopped responding. Health services went back to paper, and users in more than seventy countries lost their mail for nearly a full day.
  3. On November thirteenth we reported that Microsoft's attempts to bring servers back kept faltering, full restoration wasn't confirmed, and the cause was unknown. Since then, Microsoft has restored service and published a preliminary report that traces it all to a July defect.
  4. What Microsoft still hasn't explained is how that defect ran in production for months without anyone catching it, and nobody has settled who pays for the lost day.
  5. The defect came from a fix Microsoft wrote in July for its networking gateway servers, the machines that sit in front of Exchange Online, Outlook, Teams and Copilot.
  6. Charles Lamanna, the executive who runs Copilot, agents and platform, said that under certain load it caused servers to become overwhelmed and unresponsive. The report describes lock contention inside an internal process, where tasks jam waiting on each other until they eat up a server's processing power and memory.
  7. For months it did nothing. Then, last Thursday, three things lined up.
  8. First, the dormant defect. Second, a planned Teams failover about fifteen hours earlier pushed extra traffic onto the shared gateway. Third, nearly fourteen percent of gateway capacity was already offline from an unrelated problem. Then peak U.S. traffic hit Thursday, and requests piled up until servers froze.
  9. Microsoft's own safety systems then spread the damage. Automatic redirection moved requests off failing servers onto healthy ones, which carried the same defect and failed in turn. The barriers meant to keep trouble inside one region didn't hold.
  10. In under two hours the failure reached almost the entire system. By Thursday evening in Europe, more than eighty percent of the gateway servers were unresponsive. Microsoft calls the pattern thundering herds.
  11. That same pattern explains why the repairs kept failing. Each time servers came back, the traffic waiting for them knocked them over again.
  12. The fix was a code rollback that returned the affected code to its last working version. Engineers pulled servers out of service, restarted them, and brought them back in batches instead of all at once.
  13. Microsoft says it has since added capacity, improved monitoring and put in new safeguards. The preliminary report doesn't spell those changes out.
  14. Microsoft says the outage wasn't a cyberattack, malicious activity or an AI agent gone wrong. Lamanna apologized in a long post on X, saying the company was deeply sorry and recognizes what this did to customers' normal business.
  15. Microsoft's final report is due Friday, and we'll be reading it for how this bug slipped through. Subscribe to this channel to follow it, and the story continues at heyday dot news.
  16. Microsoft now puts active customer impact at twenty-three hours and fifty minutes.
  17. The clock starts at eleven-twenty Thursday morning Eastern and stops at ten past eleven Friday. Microsoft first acknowledged mailbox failures about ninety minutes after that start. The all-clear for the last customers came early Saturday, with delayed mail still being watched.
  18. Microsoft also stretched its email timeout window to forty-eight hours so queued messages wouldn't be lost.
  19. Published figures for the length run from more than eighteen hours, to twenty, to more than twenty-four, to nearly thirty-eight. They're measuring different things, from the hours customers were hit to the final all-clear. Microsoft's own impact window is twenty-three hours and fifty minutes.
  20. Microsoft's explanation sharpened over the week. On Friday it blamed only a problem from an earlier network capacity upgrade, and said it couldn't say when recovery would finish.
  21. Monday's account is preliminary, and it comes from Microsoft alone. No outside investigator has checked it. The capacity already offline is put at nearly fourteen percent or more than fourteen, and the report's own wording applies the eighty percent figure to gateway servers.
  22. NHS staff in England went back to paper records.
  23. Ireland's health service took Healthmail offline and told GPs and pharmacists to write paper prescriptions. U.S. courts and government offices were disrupted, along with Canadian hospitals and rail passengers. Japan, Australia and Malaysia had to interrupt critical services.
  24. In Thailand, Krungthai Bank lost email and sent urgent customers to its call center, with no confirmed hit to mobile banking. A stockbroker in Townsville warned delayed trade instructions could hurt clients if markets moved sharply. Britain's cyber minister said 999 calls, frontline NHS 111 services and transport kept running.
  25. Users filed more than seven point four million problem reports worldwide, peaking near twenty thousand a minute, from more than seventy countries and territories. Those are complaints, and Microsoft hasn't said how many people or organizations actually lost service.
  26. Experts in Australia say Microsoft is unlikely to compensate businesses for lost productivity.
  27. On the day Microsoft published its explanation, Britain's House of Commons debated the outage, led by minister Ian Murray. Germany's cyber-security agency had already called it an alarm signal about digital dependence, and Schleswig-Holstein's digital minister Dirk Schrödter called for more resilience in the public sector.
  28. Germany's IT user association, VOICE, is telling companies to document their downtime and file service-credit claims, and it wants availability commitments from Microsoft with real consequences. A service credit is a discount on the subscription fee. It doesn't cover lost hours, delayed prescriptions or missed orders.
  29. Loss figures are still estimates. One analyst put insured losses at three hundred to five hundred million dollars, with limited impact on insurers, and an Italian estimate ran to hundreds of millions of euros.
  30. Microsoft's stock barely moved. It closed at five hundred twenty-one dollars on November eleventh, about five twenty-four on Friday, and about five eighteen on Monday. Nothing ties that dip to the outage.
  31. Microsoft's final report is due Friday, November twentieth. It has to show how the defect sat undetected in production for months, why the regional barriers failed, and what Microsoft will change. Companies in Germany are being told to file their credit claims before the end of December.

Aired recently