Tech · Microsoft 365 outageContinuing story · report 5 of 5
Microsoft traces its global outage to a July bug in its own code. It hasn't explained how the bug went unnoticed.
Microsoft's preliminary report says a defect it shipped in July, a routine Teams traffic shift and capacity that was already reduced together knocked out more than 80% of its gateway servers. The report says what broke. It doesn't say why nobody caught the defect, and it doesn't settle who pays.
Written 7 min read · 392 sources
What changed since our last report
4 days later
Then
Microsoft protects delayed mail for 48 hours as outage recovery remains incompleteWhat we said was still open
The initiating cause, audited totals of affected users and organizations, aggregate losses and full connector-restoration times remain unresolved.
Still openMicrosoft's preliminary account now gives a cause: a July gateway defect, triggered by a Teams traffic shift while capacity was already reduced. That account has not been independently checked. Audited counts of affected users and organizations, aggregate losses, and the restoration times for OpenAI's and Anthropic's Microsoft 365 connectors are still missing.34
Whether restored capacity stays responsive as traffic returns—and whether Microsoft’s promised incident report explains why repeated recovery attempts failed.
Answered
Now
Microsoft's preliminary report says a defect it shipped in July, a routine Teams traffic shift and capacity that was already reduced together knocked out more than 80% of its gateway servers. The report says what broke. It doesn't say why nobody caught the defect, and it doesn't settle who pays.
Answered since
- Whether restored capacity stays responsive as traffic returns—and whether Microsoft’s promised incident report explains why repeated recovery attempts failed.
What moved
- Microsoft's stated end of active customer impact.
- Microsoft says service is restored to the remaining affected customers and regions and keeps monitoring delayed mail.
- Preliminary report published; Lamanna apologises on X.
Microsoft has now answered the question of what broke during last week's outage. That leaves a harder question: how a defect Microsoft shipped itself ran in production for months without anyone catching it, and then took down email for much of the world in under two hours.
On November 13 we reported that Microsoft's attempts to restore service kept faltering, that the company had extended its email timeout window to protect queued mail, that full restoration had not been confirmed, and that the cause was unknown. Since then Microsoft has said service is restored to all remaining customers and regions, and on Monday, November 16, it published a preliminary post-incident report. The report traces the failure to a defect in a July update to the networking gateway servers that sit in front of Exchange Online, Outlook, Teams, Copilot and other Microsoft 365 services.327
Three faults that lined up
Charles Lamanna, Microsoft's executive vice-president for Copilot, agents and platform, set out the explanation in a long post on X, which he called an abridged version of the report filed in the Microsoft 365 admin center. He said the defect came from a fix written specifically for the gateway servers, and that under certain load conditions it caused servers to become overwhelmed and unresponsive. Coverage of the report describes the defect as lock contention: tasks inside an internal process jam while waiting on each other until they exhaust a server's processor and memory. iTnews reports that the bug sat hidden in the gateway fleet for more than three months.37343728
The defect alone did nothing. According to the report, it took two more conditions. About 15 hours before the outage, a planned Teams failover onto the Exchange Online networking infrastructure pushed extra traffic onto the shared gateway. Lamanna described this as one of two networking gateways not working for Teams, which overloaded the primary one. Separately, an unrelated problem from the previous day had already taken a slice of gateway capacity offline. When peak U.S. traffic arrived on Thursday, requests piled up until servers stopped responding.324
Microsoft's own safety systems then made things worse. Automatic redirection moved requests off failing servers onto healthy ones that carried the same defect, and those failed in turn. The barriers meant to keep trouble inside a single region did not hold. According to n-tv's account of the report, the failure reached almost the entire system in under two hours, and by Thursday evening European time more than 80% of the relevant machines were unresponsive. GeekWire reports that Microsoft calls this dynamic "thundering herds."5371
The 80% figure has also been reported in two ways. BleepingComputer gives it as more than 80% of the networking gateway servers, while El País reported Lamanna as saying more than 80% of the servers Microsoft manages worldwide. The report's own wording, as covered, refers to gateway servers. A failure of four-fifths of everything Microsoft runs would be a different and much larger event.36
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The same dynamic explains why repairs kept failing on Thursday and Friday. Lamanna said that each time servers came back, the traffic waiting for them overwhelmed them again. The fix that worked was a code rollback that returned the affected code to its last working state. Engineers pulled servers out of service, restarted them and brought them back in groups instead of all at once. Microsoft says it has since added capacity, improved monitoring and put in safeguards, but the preliminary report as covered does not list those changes in detail. Microsoft says the outage was not a cyberattack, malicious activity or an AI agent malfunction.48333
The story has shifted over the week. On Friday Microsoft pointed only to an issue introduced during an earlier network capacity upgrade, said services were broadly recovering, and said it did not know when recovery would be complete. Monday's account is far more specific, but it is still preliminary and comes from Microsoft alone. No outside investigator has checked it.115
From a July fix to a Monday explanation
- July 202635
A fix for the gateway servers ships, carrying a latent defect.
- Thu Nov 12, ~01:20 UTC3
A planned Teams failover pushes extra traffic onto the shared Exchange Online gateway.
- Thu Nov 12, 11:20 a.m. ET3
Microsoft's stated start of active customer impact.
- Thu Nov 12, 17:54 GMT1
Microsoft first acknowledges that users in the Americas cannot reach Exchange Online mailboxes.
- Fri Nov 13, 11:10 a.m. ET3
Microsoft's stated end of active customer impact.
- Sat Nov 142710
Microsoft says service is restored to the remaining affected customers and regions and keeps monitoring delayed mail.
- Mon Nov 1634
Preliminary report published; Lamanna apologises on X.
The official window also explains why published durations conflict. CBC said more than 24 hours, MyBroadband 20 hours, El País more than 18 hours, and Frandroid nearly 38 hours. Frandroid measured from the Exchange Online incident, EX1550543, which it says was opened Thursday at 17:17, to Saturday's restoration announcement. The others measured customer impact or snapshots taken along the way. Microsoft's own impact window is 23 hours and 50 minutes. Two timestamps do not sit neatly with it: Ookla's network data put the onset around 17:25 UTC, and Microsoft's first public acknowledgement came 94 minutes after its stated start. Even after Saturday's all-clear, Microsoft was still watching delayed mail under an email timeout window it had stretched to 48 hours.12761030124
Who absorbed the day, and who pays
Downdetector logged more than 7.4 million problem reports from more than 70 countries and territories, peaking near 20,000 a minute. Those are complaints, not a headcount, and Microsoft has not said how many people or organizations lost service. The damage was concrete all the same. NHS staff in England went back to paper records. Ireland's HSE took Healthmail offline and told GPs and pharmacists to write paper prescriptions under the GMS scheme. U.S. courts and government offices, Canadian hospitals and rail passengers were disrupted, and El País reports that Japan, Australia and Malaysia had to interrupt critical services. Thailand's Krungthai Bank lost email and sent urgent customers to its call center, with no confirmed effect on mobile banking. A Townsville stockbroker warned that delayed trade instructions could hurt clients if markets moved sharply. Britain's cyber minister said 999 calls, frontline NHS 111 services and transport kept running.1215262561131
On the day Microsoft published its explanation, the House of Commons debated the outage, led by minister Ian Murray. The available record does not show what commitments, if any, came out of the debate. Germany's federal cyber-security agency, the BSI, had already called the outage a major problem for the economy and an alarm signal about digital dependence, and Schleswig-Holstein's digital minister Dirk Schrödter called for more digital resilience in the public sector.3873905
Germany's IT user association VOICE is telling companies to document their downtime and file service-credit claims by the end of December, and it wants availability commitments from Microsoft that carry real consequences. A service credit is a discount on the subscription fee. It does not pay for lost hours, delayed prescriptions or missed orders, and experts told Australia's ABC that Microsoft is unlikely to compensate businesses for lost productivity. Loss figures are estimates. Analyst Mario De Cicco put insured losses at $300 million to $500 million, with limited impact on insurers, and Il Sole 24 Ore reported estimates of at least hundreds of millions of euros. Microsoft's stock barely moved: it closed at $521.13 on November 11, $523.88 on November 13 and $517.99 on November 16, and nothing ties that dip to the outage.38838512386378
Known
Unknown
- How the defect went undetected in production for months, and why regional isolation failed.
- How many people and organizations lost service, total losses, and whether Microsoft has granted any service credits.
Next
- Microsoft's final report, which it has scheduled for Friday, November 20, should set out its full prevention measures.
- Whether companies file the credit claims VOICE recommends before the end of December.
Timeline
Day 8 · 5 reports
Microsoft's stated end of active customer impact.
Microsoft says service is restored to the remaining affected customers and regions and keeps monitoring delayed mail.
Preliminary report published; Lamanna apologises on X.
We aired · Update 5You are here
Microsoft traces its global outage to a July bug in its own code. It hasn't explained how the bug went unnoticed.
Sources244
3 older pieces read for background.46 outlets with no date:
Continuing story · report 5 of 5
Every source, by outlet
392 sources from 244 outlets· Most cited first
CBS Los Angelescbsnews.com13
- First reported
- Microsoft says it has "restored" service and and addressed "underlying issue" after worldwide outage
- Microsoft says it has "restored" service and and addressed "underlying issue" after worldwide outage
- Microsoft battling major worldwide outage
- Microsoft battling major worldwide outage
- Microsoft battling major worldwide outage
- Microsoft battling major worldwide outage
- Microsoft battling major worldwide outage
- Microsoft says it is seeing "continued recovery progress" as it battles major worldwide outage
- Microsoft says it is seeing "continued recovery progress" as it battles major worldwide outage
- Microsoft outage improving; Outlook, Teams, 365 service restoring Friday morning
- Microsoft says it is seeing "continued recovery progress" as it battles major worldwide outage
- LA County Sheriff's communications disrupted amid Microsoft outage
- Widespread Microsoft outage leaves users unable to send or receive Outlook emails
Al Jazeeraaljazeera.com9
- First reported
- Microsoft says affected services fully recovered following worldwide outage
- Microsoft hit by global outages: What happened and why?
- Live: Microsoft struggles to restore services as outage hits global users
- You haven’t got mail: Global Microsoft outage disrupts services
- Microsoft services begin recovering after global outage
- Microsoft says services still returning amid widespread outage
- Microsoft outage hits tens of thousands of users
- Microsoft outage hits tens of thousands of users
- Microsoft hit by global outages: What happened and why? | Technology News
CTV News9 videos9
- First reported
- Carney rejects Trump's auto demands | CTV News Ottawa at Six for November 14, 2026 · video
- ‘Pretty extraordinary’ | Microsoft services returning in parts of the world but outages persist · video
- Microsoft reports service gradually being restored · video
- Microsoft recovering from global outage · video
- Microsoft hit by worldwide outage | CTV Your Morning for Nov. 13, 2026 · video
- U.S. FAA lifts nationwide ground stop | CTV News Vancouver Island Friday November 13, 2026 · video
- Richard Neil guilty and massive Microsoft outage | CTV News Kitchener at 6 for Nov. 13, 2026 · video
- Microsoft 365 outage impacts Outlook, Teams worldwide · video
- Regina councillor denies sharing anti-Islamic conspiracy | CTV New Regina for Nov. 12, 2026 · video
Google News Australia (AU:en)theguardian.com9
- First reported
- Microsoft identifies months-old bug as a root cause of global outage
- Microsoft says services fully restored after global outage hits governments
- Microsoft outage hits hospitals, GPs, schools and police across UK - The Guardian
- Microsoft outage live: Outlook and Teams failures hit government agencies, schools and businesses - The Guardian
- Microsoft outage live: Microsoft’s global outage will cost Australia billions, expert says; US tech giants and Indian airlines also affected - The Guardian
- Microsoft ‘restarting’ as outage impacts global users, with Outlook and Teams affected | Microsoft | The Guardian
- A global Microsoft 365 outage has hit thousands – what happened and when will it be fixed? - The Guardian
- Microsoft outage live: Microsoft scrambling for new ways to fix global outage after initial attempts fail to restore Outlook services | Microsoft | The Guardian
- ‘Looks like we’ll have to talk to people’: Australia sent back to time before email with major Microsoft outage | Microsoft | The Guardian
BBC Newsbbc.co.uk1 video8
- First reported
- Microsoft 'deeply sorry' for global outage after almost 24 hours of disruption
- Watch: How a server outage took down Microsoft Teams and Outlook
- Microsoft outage: Yorkshire councils and police hit - BBC
- How Microsoft outage is affecting police, airports and hospitals in the north-west of England
- What we know about the global Microsoft outage
- What happened overnight with the Microsoft global outage
- Microsoft says attempts to fix global outage not going 'as expected' - BBC
- Chaos as Microsoft outage hits corporate and emergency services globally · video
ABC Newsabcnews.com5 videos7
- First reported
- Microsoft identifies issues leading to global Outlook outage
- Microsoft Outlook outage stretches on for hours
- Microsoft Outlook Outage: Email Slowly Returning. What You Need to Know - November 13 · video
- Microsoft apologizes for global outage that impacted millions · video
- Workers nationwide impacted by Microsoft outage · video
- Microsoft reports issues with Outlook and Teams · video
- Outlook down for thousands amid Microsoft outage across North America · video
Deutsche Welledw.com7
- First reported
- Microsoft says all services working again
- Microsoft says all services working again
- Dünya genelinde erişim yok: Microsoft "Siber saldırı kanıtı yok" dedi - DW.com
- Interrupción mundial afecta aplicación de mensajería Outlook de Microsoft - DW.com
- Microsoft works to restore Outlook and Teams after global outage
- Microsoft works to restore Outlook and Teams after global outage
- Microsoft reports mass outages in US, impacting customer accessibility
Google News Singapore (SG:en)channelnewsasia.com7
- First reported
- Singapore government’s Outlook services fully restored after global Microsoft outage - CNA
- 'Microsoft-sponsored holiday': Singapore workers scramble to cope with global outage - CNA
- Some Singapore government systems affected by Microsoft outage: GovTech - CNA
- Microsoft 365, Outlook down for thousands of users - CNA
- Microsoft outage drags on for hours, disrupting Outlook, Teams for thousands, including Singapore users - CNA
- Microsoft Outlook outage: Disruptions hit workers worldwide amid setback in recovery efforts - CNA
- Australia's spy agency, federal police hit by Microsoft outage - CNA
ABC News (Australia)abc.net.au1 video6
- First reported
- Outlook outage expected to cost Australian businesses billions of dollars
- How the Microsoft outage is affecting Australians
- Microsoft outage affects users worldwide, impacting Teams, Outlook and Copilot - ABC News
- Microsoft outage slashes productivity - ABC listen
- Microsoft unlikely to compensate for loss of productivity from outage, experts say - ABC News
- Microsoft access being restored after global disruptions | ABC NEWS · video
BBCbbc.com5
- First reported
- Microsoft says it is 'deeply sorry' for global outage after almost 24 hours of disruption
- مباشر مايكروسوفت تسعى لاستعادة خدماتها بعد عطل عالمي
- Outlook: falha na Microsoft deixa email fora do ar email e afeta Teams, Microsoft 365 e outras plataformas; o que se sabe
- Una falla en Microsoft deja fuera de servicio el correo electrónico de Outlook y otras plataformas
- Microsoft outage sees Teams, Outlook and Copilot suffer 'connectivity issues'
Google News France (FR:fr)franceinfo.fr5
- First reported
- Outlook et Teams perturbés, pas de cyberattaque d'après Microsoft... Ce que l'on sait de la panne mondiale qui touche le géant américain depuis jeudi - franceinfo
- Une panne mondiale touche la messagerie Outlook de Microsoft - franceinfo
- DIRECT. Crise sociale : la réunion entre les syndicats de fonctionnaires et le Premier ministre a débuté
- Une panne mondiale touche la messagerie Outlook de Microsoft, "aucun signe d'action malveillante ou de cyberattaque" n'a été trouvé à ce stade - franceinfo
- DIRECT. Panne mondiale de Microsoft : "Des résultats positifs en matière de stabilité du service" constatés par le géant américain – franceinfo
As aired
- A bug Microsoft shipped in July sat hidden for months, then took email down worldwide.
- At the worst point, more than eight in ten of the servers that front Outlook, Teams and the rest of Microsoft 365 stopped responding. Health services went back to paper, and users in more than seventy countries lost their mail for nearly a full day.
- On November thirteenth we reported that Microsoft's attempts to bring servers back kept faltering, full restoration wasn't confirmed, and the cause was unknown. Since then, Microsoft has restored service and published a preliminary report that traces it all to a July defect.
- What Microsoft still hasn't explained is how that defect ran in production for months without anyone catching it, and nobody has settled who pays for the lost day.
- The defect came from a fix Microsoft wrote in July for its networking gateway servers, the machines that sit in front of Exchange Online, Outlook, Teams and Copilot.
- Charles Lamanna, the executive who runs Copilot, agents and platform, said that under certain load it caused servers to become overwhelmed and unresponsive. The report describes lock contention inside an internal process, where tasks jam waiting on each other until they eat up a server's processing power and memory.
- For months it did nothing. Then, last Thursday, three things lined up.
- First, the dormant defect. Second, a planned Teams failover about fifteen hours earlier pushed extra traffic onto the shared gateway. Third, nearly fourteen percent of gateway capacity was already offline from an unrelated problem. Then peak U.S. traffic hit Thursday, and requests piled up until servers froze.
- Microsoft's own safety systems then spread the damage. Automatic redirection moved requests off failing servers onto healthy ones, which carried the same defect and failed in turn. The barriers meant to keep trouble inside one region didn't hold.
- In under two hours the failure reached almost the entire system. By Thursday evening in Europe, more than eighty percent of the gateway servers were unresponsive. Microsoft calls the pattern thundering herds.
- That same pattern explains why the repairs kept failing. Each time servers came back, the traffic waiting for them knocked them over again.
- The fix was a code rollback that returned the affected code to its last working version. Engineers pulled servers out of service, restarted them, and brought them back in batches instead of all at once.
- Microsoft says it has since added capacity, improved monitoring and put in new safeguards. The preliminary report doesn't spell those changes out.
- Microsoft says the outage wasn't a cyberattack, malicious activity or an AI agent gone wrong. Lamanna apologized in a long post on X, saying the company was deeply sorry and recognizes what this did to customers' normal business.
- Microsoft's final report is due Friday, and we'll be reading it for how this bug slipped through. Subscribe to this channel to follow it, and the story continues at heyday dot news.
- Microsoft now puts active customer impact at twenty-three hours and fifty minutes.
- The clock starts at eleven-twenty Thursday morning Eastern and stops at ten past eleven Friday. Microsoft first acknowledged mailbox failures about ninety minutes after that start. The all-clear for the last customers came early Saturday, with delayed mail still being watched.
- Microsoft also stretched its email timeout window to forty-eight hours so queued messages wouldn't be lost.
- Published figures for the length run from more than eighteen hours, to twenty, to more than twenty-four, to nearly thirty-eight. They're measuring different things, from the hours customers were hit to the final all-clear. Microsoft's own impact window is twenty-three hours and fifty minutes.
- Microsoft's explanation sharpened over the week. On Friday it blamed only a problem from an earlier network capacity upgrade, and said it couldn't say when recovery would finish.
- Monday's account is preliminary, and it comes from Microsoft alone. No outside investigator has checked it. The capacity already offline is put at nearly fourteen percent or more than fourteen, and the report's own wording applies the eighty percent figure to gateway servers.
- NHS staff in England went back to paper records.
- Ireland's health service took Healthmail offline and told GPs and pharmacists to write paper prescriptions. U.S. courts and government offices were disrupted, along with Canadian hospitals and rail passengers. Japan, Australia and Malaysia had to interrupt critical services.
- In Thailand, Krungthai Bank lost email and sent urgent customers to its call center, with no confirmed hit to mobile banking. A stockbroker in Townsville warned delayed trade instructions could hurt clients if markets moved sharply. Britain's cyber minister said 999 calls, frontline NHS 111 services and transport kept running.
- Users filed more than seven point four million problem reports worldwide, peaking near twenty thousand a minute, from more than seventy countries and territories. Those are complaints, and Microsoft hasn't said how many people or organizations actually lost service.
- Experts in Australia say Microsoft is unlikely to compensate businesses for lost productivity.
- On the day Microsoft published its explanation, Britain's House of Commons debated the outage, led by minister Ian Murray. Germany's cyber-security agency had already called it an alarm signal about digital dependence, and Schleswig-Holstein's digital minister Dirk Schrödter called for more resilience in the public sector.
- Germany's IT user association, VOICE, is telling companies to document their downtime and file service-credit claims, and it wants availability commitments from Microsoft with real consequences. A service credit is a discount on the subscription fee. It doesn't cover lost hours, delayed prescriptions or missed orders.
- Loss figures are still estimates. One analyst put insured losses at three hundred to five hundred million dollars, with limited impact on insurers, and an Italian estimate ran to hundreds of millions of euros.
- Microsoft's stock barely moved. It closed at five hundred twenty-one dollars on November eleventh, about five twenty-four on Friday, and about five eighteen on Monday. Nothing ties that dip to the outage.
- Microsoft's final report is due Friday, November twentieth. It has to show how the defect sat undetected in production for months, why the regional barriers failed, and what Microsoft will change. Companies in Germany are being told to file their credit claims before the end of December.





