2026-11-13 13:35 UTC
Paging cloud engg and SRE experts - what do you think was the root cause of this extended outage at Microsoft?
My simplistic theory is that some servers got overloaded, which got shutdown by the health check logic, traffic was diverted to other servers, which then got overloaded and the failures cascaded.
Failed servers did nor restart and join the pool on their own?
Could be some weakness in the load balancing logic or a natural traffic increase or trying to overdrive servers to save $$.
17/
Replies (1)
-
@AkaSci@fosstodon.org 2026-11-13 20:42
Based on the chart below posted by @dougmadory@infosec.exchange and other hints on the net, here is likely what happened with Microsoft servers - 1. The servers are designed to handle a certain number client conn attempts per second. 2. Normally, the rate of conn attempts is small since once established, most conns hang around for hours or days. 3. But after a server restart, for any reason, all connected clients will time out and retry, creating a surge of conn attempts. (contd) 18/n