This was painful today. Never should have happened. Great to already see the work to ensure it never will again. We make mistakes — which kills me — but proud we rarely make them twice.

I think it's great you guys also document these mistake so openly too. It helps other companies not make the same ones. Always love the transparency and generally detailed blogs you guys do. I'd love to work with you guys some day.

Really nice to see how transparent you guys are. Am definitely proud to be a customer for cloudflare.

Too many managers?

Thank you for your transparency, but it funny how disconnecting a cable can cause so much frustration - I feel bad for the technician 😅

I can certainly appreciate the in-depth transparency. Mistakes happen, it's about learning from them which will help prevent recurrences. Great work to you and your teams on how this situation was handled.

you are still awesome for incident reporting @Cloudflare

it's helpful and establishes their brand well.

Impressive transparency. Thanks for not only sharing but doing so almost immediately.

This is real leadership. Owning up to your mistakes and apologizing, documenting everything you did wrong (and you were able to document it, very hard at this level of detail), and writing about what will be different going forward. Go team #Cloudflare! 👏

We all make mistakes, and nobody is God. Hiding the mistakes is a really bad thing, yet so many companies do. It always leaks out anyway. @Cloudflare has a great responsibility to keep us online, and I have seen so many times that the communication users get is open and clear.

And of course, mistakes are to be learned from, procedures get made, and you do not repeat them, because you've gone through them. That is the right way to think and do, and I am glad all my sites and my clients' and my employer's are protected and enhanced by Cloudflare!

Remarkably candid disclosure. Also why label guns are your friends!

Whoever immediately noticed the outage probably didn't know about this maintenance. How long did it take to recognize the cause, go from "dashboard is down" to "oh the patch panel"?

We initially thought it was a power outage and the data center’s backup generators hadn’t kicked in. Was very strange to lose multiple independent network links across different providers, different fiber paths, to different cities.