Subject: Major network failure this morning
Date: Thu, Jan 22 2026 -- 4:20 PM
Posted by: Alexis Rosen
Starting early this morning, a most extraordinary confluence of failures
at multiple upstream networks caused severe problems for Panix. One of
our carriers suffered equipment failure, followed by a fiber cut,
followed by what they think is more equipment failures. The other
suffered a massive DDoS attack which was large enough to affect all their
customers, and also a fiber cut. The end result for us was massive loss
and high latency to all parts of our network.
All of this happened while we are in the middle of a multi-week project
to move Panix from one data center to another. We were doing some network
maintenance when the failure started, and of course "the failure" was
mutiple progressive events, so it took us quite a while to figure out
what was going on (and that it wasn't something we had done!). Once we
understood the issues, we worked around it (in stages) by moving traffic
(much earlier than originally planned) to a third carrier at the new data
center.
We started getting some service back to normal in the late morning. As of
around 3:45PM Eastern, everything should be working normally, though not
necessarily using the same network paths, and possibly excepting a few
IPv6 subnets.
In the 37 years I've been doing this, I've never seen such an enormous
number of things go wrong (roughly) simultaneously. I hope it's another
37 years (at least) before I see that again.
| |
