See the entire conversation

14 replies and sub-replies as of Apr 16 2020

I think it's great you guys also document these mistake so openly too. It helps other companies not make the same ones. Always love the transparency and generally detailed blogs you guys do. I'd love to work with you guys some day.
Really nice to see how transparent you guys are. Am definitely proud to be a customer for cloudflare.
Thank you for your transparency, but it funny how disconnecting a cable can cause so much frustration - I feel bad for the technician 😅
I can certainly appreciate the in-depth transparency. Mistakes happen, it's about learning from them which will help prevent recurrences. Great work to you and your teams on how this situation was handled.
you are still awesome for incident reporting @Cloudflare
it's helpful and establishes their brand well.
This is real leadership. Owning up to your mistakes and apologizing, documenting everything you did wrong (and you were able to document it, very hard at this level of detail), and writing about what will be different going forward. Go team #Cloudflare! 👏
We all make mistakes, and nobody is God. Hiding the mistakes is a really bad thing, yet so many companies do. It always leaks out anyway. @Cloudflare has a great responsibility to keep us online, and I have seen so many times that the communication users get is open and clear.
And of course, mistakes are to be learned from, procedures get made, and you do not repeat them, because you've gone through them. That is the right way to think and do, and I am glad all my sites and my clients' and my employer's are protected and enhanced by Cloudflare!
Remarkably candid disclosure. Also why label guns are your friends!
Whoever immediately noticed the outage probably didn't know about this maintenance. How long did it take to recognize the cause, go from "dashboard is down" to "oh the patch panel"?
We initially thought it was a power outage and the data center’s backup generators hadn’t kicked in. Was very strange to lose multiple independent network links across different providers, different fiber paths, to different cities.