2026-11-13 20:49 UTC
4. If the server falls behind in processing, it will queue up incoming conn requests (either at the TCP layer or in the app sw)
5. If the queue allows a large number of conns (bigger is better, right?), the response time will increase
6. Clients will time out and retry conns
7. Soon, the server is processing timed out conns only, doing some SSL layer processing, wasting CPU cycles, dropping the conn..
8. While the queue contains new conns near the tail
9. Most conns fail.
(contd)
19/n
Replies (1)
-
@AkaSci@fosstodon.org 2026-11-13 20:51
This is not a new or novel scenario. Most good designers are aware of it. There are plenty of papers and articles on the subject. The solutions are not trivial. So, what did Microsoft implement to handle client connection surges? The question above presupposes that this was the core issue in the MS outage. It could have been something else. https://learn.microsoft.com/en-us/azure/architecture/antipatterns/retry-storm/ https://read.bytesizeddesign.com/p/understanding-retry-storms-what-they https://medium.com/agoda-engineering/how-agoda-solved-retry-storms-to-boost-system-reliability-9bf0d1dfbeee https://www.linkedin.com/pulse/breaking-prod-10-retry-storm-took-everything-down-sherlocks-ai-bolcf/ 20/n