There’s a lot of buzz right now about a “massive DDoS attack” targeting the US, complete with scary-looking graphs (see Tweet below). While it makes for a good headline in these already dramatic times, it’s not accurate. The reality is far more boring. 1/X

It starts with T-Mobile. They were making some changes to their network configurations today. Unfortunately, it went badly. The result has been for around the last 6 hours a series of cascading failures for their users, impacting both their voice and data networks. 2/X

That caused a lot of T-Mobile users to complain on Twitter and other forums that they weren’t able to reach popular services. Then services like Down Detector scraps Twitter and report those services as being offline. 3/X

So now people are looking around for an explanation and they stumble across sites like the Arbor Networks attack map. It looks terrifying today! Thing is, it always looks terrifying. It’s a marketing gimmick put up to sell DDoS mitigation services so that’s not surprising. 4/X

From @Cloudflare’s vantage point, we can see a number of things that show there is no massive DDoS attack. First, traffic from WARP to supposedly impacted services is normal and has no increase in errors. 5/X

Second, there is no spike in traffic to any of the major Internet Exchanges, which you do see during actual DDoS attacks and definitely would during one allegedly this disruptive. 6/X

Finally, our team know the network operators at nearly all the other major Internet services and platforms and none of them are reporting anything anomalous. 7/X

Except T-Mobile, which is having a bad day almost certainly entirely of their own team’s making. So, please, #hugops. And don’t worry, this is one thing that does not need to get added to the list of craziness that has been 2020. 8/8

Thank you.

Thank you!

Thanks!

How much are they paying you? 😉

😂😂😂 they will make it crazy anyway! Why use logic 🤦‍♀️🤦‍♀️🤦‍♀️

thank you for taking the time to explain this so thoroughly 👍

Like look this man is the clear expert in this field and has sources as well....yet shit was fucked up way beyond tmobile. Idk I'm not in that field with any experience but I just don't buy this

Coincidences do happen like I'm not shocked fortnite event crashed today that could easily occur, its certainty odd.

Chase bank was down. Idk seems to me more of a ceo trying to save some stock prices

Psiops?

Just network ops.

Well thanks 😊

Thanks. Now let me get back to my regularly scheduled panic about Covid-19/Ebola/Murder Hornets/ Locusts/... #DDoS

Don’t forget the aliens girl!

My bad! Didn't mean no disrespect to the lil green guys 👽👽 They can surely end us same as the Murder Hornets. It's just hard keeping up with all the apocalypt-y breaking news stories...

Seems coincidental that other services experienced issues today, including one of the largest enterprise cloud telco providers. That's not T-Mobile related, but is either same cause (Tier 1 backbone), or coincidence?

*crickets*

Thanks 🥺

(No comment)

Can't wait to read the RCA from the @TMobile Ops team. Should be interesting to see where the failure was.

How hugops + social distancing.

I unfollowed a few tech folks over this debacle, and I’ll happily follow you instead 👍🏻 good thread

We'd love to believe you... But hiding an alien attack from us is S.O.P. for you guys. ... 🤖

How do you explain Verizon being spotty all day?

Well, unless your T-mobile. They can/should add this to their 2020 list. 😉

So why has Verizon been down all day?

Thank you. I've been looking for a better take on this.

How comes German T-Mobile is struggling? I highly doubt they manage their mobile network at global scale 🧐

EVERYTHING IS FINE DONT WORRY CHINA / EU and the rest of the world are attacking the United States, but this rich nerd isn't affected, so just calm down and forget about it.

Do what about Verizon & At&T having issues ad well

Thank you immensely for explaining what happened, in terminology that ordinary folk can understand. That is not a small thing. You and @hacks4pancakes talking sense 24/7/365 in spite of it all.

Thanks for this.

Thank You !

Thanks, Matty

At school we called that an RGE. Resume Generating Event.

Sadly @TMobile services are still out in SC. Seven hours plus now.

Bravo 👏🏻

#Excellent analysis, Not #Dos

I live in MI, last night our Comcast Xfinity internet service went down at same time my sons Verizon cell phone service also crashed. I still had use of my Xfinity cell service however.

and people like these are telling you everything is alright? I'm confused, who's to believe? threatmap.fortiguard.com

v. interesting; thanks. I'd just like T-Mobile to fix their damn network :/

Nice graphic analysis, so what's causing such a broad disruption? T-Mobile DDos #Verizon AT&T #Sprint

I believe it’s either a Tier 1 provider such as CenturyLink or Layer 3 having an outage but I would assume other carriers would also see issues, but on the other hand, a leaked internal T-Mobile document stated that they would be moving over 1mil Sprint customers to their network

Exactly. Cell service uses existing fiber optics and other companies to route cell traffic. It is a routing issue. It's not disputed T-Mobile may be having most of the issues. My point is that ATT and Verizon like to deny the problems even if their users are having issues.

Is that graph an Elephant or a snake?

You showed a photo from a HongKong network, bruh. HONG KONG INTERNET EXCHANGE LMAOOOO

Thank you for having understandable graphics, not like in the previous post.

What if we are dealing with an attack with advanced technologies? The graphs would be normal. ~ 10110

More... Graphs...🤷

Yet we still don’t have service

Can you please explain what these graphics represent? There isn't a title on the graphs or titles on the axies. They kind of seem BS without a concise explanation.

You guys are cloudflare, which cellular provider uses your service? I don't think anyone does so why do you think your Cloudflare graphs matter? Of course you don't see traffic, it doesn't even go through you guys...

Just thinking the same thing

But but butt... Graphs...🤷

Exactly. I've seen headlines saying this is the largest DDoS attack in US history. I doubt it's true after the 2016 attacks.

👆

Would there ever be an instance where hiding the fact a massive attack hit us would be deemed the best route to take to avoid an escalation? Kinda like how we lied about any injuries in the Iran rocket attacks so that we didn't have to go to war?

Nation states participate in covert cyberwarfare on a daily basis. Russia interfered in the U.S. election, a Chinese APT group was responsible for the Equifax hack, etc. I think the only difference here (if was a legit attack) is a lack of subtly.

Ok so what happened to all the others?

so basically a lot of "fancy" tools all are in reality pieces of shit and together they report the opposite of what's going on. wow.

It's like watching firewall logs in real time... you'll drive yourself crazy lol

Thank you for clarifying. I have seen a graph that was from yesterday and the Down detector today though it appears that the outages have passed for the most part.

Somebody’s getting fired ...

Ehhhhhhhh you’d be amazed what has been effed up without people getting fired.

^^^ exactly. Companies firing people who break things damage themselves in three ways. i. They invariably wrap themselves in "feel-good" change control which adds nothing but ass-covering. ii. It destroys any and all innovation. iii. The person you fired probably fixed it too.

Truth.....🤷‍♂️

Systems are complicated. People make mistakes. Easier to learn from them if they’re still around to tell the war stories. Firing someone for an honorable mistake is huge mistake.

I totally agree. Especially if the fault lay in processes or redundancy out of their control.

Specially because any networking system is just a nuclear reactor patched together with duct tape

Now, making the same mistake -TWICE- could be grounds for termination.

Ahh.. the armchair sysadmins are always great fun. I especially loved them after an S3 outage postmortem a while back after an operator fat fingered a command. As if any of them could run something on that scale/criticality 🤣

It’s exactly that. Find the intention. If the intention was honourable find out what went wrong and how to avoid it in future, so everybody learns. Learning is good. Throwing sacrifices into a volcano to appease the gods, less so.

My senior sysadmin in my first admin job told me: you're gonna fuck up sometimes. Fess up and get help asap to fix it, document it, learn from it, and I'll have your back if management gets angry. Sure enough, it happened, we fixed it quickly, and life moved on.

If I could only tell you. We are 5 months into recovering from one error. I think we are about wrapped up now.

🍸

Do I wish.

Hugops.

Didn’t experience it

twitter.com/densafetydept/… Um... I dunno about this - my phone has been acting haywire and now people are having trouble connecting to 911 services.

If their networks are down, you will have trouble calling.

A network configuration change? Mid-morning when traffic would be coming down from a peak? I don’t buy that.

It's days like to today that I'm glad I'm not the IT person who broke this one. 😂

Thanks a bunch!

8 hours and it’s global. Not just T-Mobile or their upgrades.

A config change? Wtf cause they couldn’t revert it back? For all telecom giants to be affected? Try again #DDoS #TMobiledown #StayWoke

Just pressing the "revert" button isn't simple, it isn't quick, and isn't painless. Sometimes a broken deploy can make reversion almost impossible.

why is facebook down?

I just had to turn my phone off. It was absurdity. SMS messages repeating like wild in both directions.

Its crazy.. Multiple companies, very wide spectrum...

Good to know, I'm sure all the websites you host and disguise on Cloudflare are safe with their Libelous Posts, #Racist Posts and Vulgarity. Shame my friends still have to suffer while you wash your hands : Cloudflare enables Criminals #BlackLivesMatter #LGBTQ

I can imagine a former disgruntled employee. "do erase startup config" and "do delete vlan.dat" followed with "reload at 00:00 15 jun". 😂

T-Mobil is a fucking tire fire, and has been since I got it. This is the kick in the ass I need to go back to AT&T

Thanks.

The intern is so getting fired

I was hoping to hear what you have to say about the DDOS

I'm sick of people being so confident in their wrong opinions, especially with events like these.

"no nothing" But you are correct.

I'm stealing this graphic

This graph is awesome, except the “no nothing,” label is making me twitch just a bit.

I know.....I was going to rt but i just can't lol!

ITS THIS! NO WAIT ITS THIS HERE. NO, WAIT ITS ACTUALLY THAT! WAIT JUST A SEC ITS NO WAIT ITS THIS

That's not how Dunning — Kruger effect works. It isn't even that pronounced. It says nothing about experience over time. In fact, the original paper says the exact opposite: the more confident a person appears, the more knowledgeable they are.

in the psychology field it is widely accepted that people who know almost nothing have high confidence according to the dunning-kruger effect. all graphs you find when you look it up show the same exact thing more or less.

>it is widely accepted No, it isn't. Can I have a source for that statement? Have you read the original 1999 paper by Dunning and Kruger even?

damn and I was gonna say respond too 😿😿

no nothing btw

It's crazy

Sick phone theme!

I smell CCP!

This is @timmaughan territory.

Phew...thank you

You are the only one that called it massive ddos attack... someone is a lil butthurt

Voice of reason finally

But no one can say what the reality is right now.

this why twitch was broken earlier?

Twitch broke because of the influx of users watching a fortnite event

Thought it was that plus the whole DDOS thing going on.

Do I smell cover-up propaganda by the police state? You mother fuckers make it painfully obvious, you know that?

Lmao what?

Do you also think vaccines cause autism, and tainted water can make frogs gay? If you do, buy some of my merch. If you're dumb enough to make that comment, you're also dumb enough to spend your money on a creator you've never heard of.

I appreciate the real information. Couldn’t reconcile the rumors with what I was actually seeing all afternoon.

Yeah.

(No comment)

good, cloudflare doh resolver has been choppy for a few days

I wish it was tomorrow because I could have said "For me in Canada... it was a Tuesday.".

But who cares about facts when you can just say 2020 sucks?

Have been outages in the UK, unrelated to this. Possibly AWS related...again hahahah

Are you sure that it isn’t as massive as it looks?

Seriously appreciate the heads up and insight as to what's going on.

Well can we actually know the cause?

nah

They appeared to have made some changes in routes without a proper blackout plan if things should go wrong. It isn't the first time this has happened. It takes me upwards of 2 hours to write up a proper blackout plan for network changes. This is sad.

Blame it on Obama!!! 😒. Figured I’d beat someone to the bullshit.

(No comment)

(No comment)

I wonder why connections are having high latency.

Who is this 12 year old? He didn't go to law school. He doesn't look old enough to have gone to HIGH school? What the hell is "Tech Crunch"? A cereal? Is anyone buying this load of bull??

...do you also think the moon landing was fake & Kennedy was assassinated by our own government? If you do, buy some of my merch. If you're dumb enough to make that comment, you're also dumb enough to spend your money on a creator you've never heard of.

lol, derail the conversation much. threatmap.fortiguard.com more proof.

The irony of T-Mobile leading the pack in 5g, AND no MSM reporting todays events. Yes, you're right, nothing to see here folks move along....🙄

There's no 5G standard approved yet, and T-Mobile isn't leading anything other than outages. Nothing to see here indeed.

I'm assuming you get your news from CNN as well. So from your "newsworthy" friends...

I assume you know nothing about 5G as well. So from your "read press releases and believe them" friends:

Speed isn't everything, though. It's still 5G. Both Verizon and T-Mobile have real 5G, with real benefits. And there's a pretty obvious reason why smaller countries have the fastest LTE speeds.

Or maybe, just maybe, considering that TMobile and Sprint are in the middle of combining their networks and rolling out major updates to both networks to make everything work cohesively, something went wrong. But please, explain to us all how this is obviously 5G's fault 🙄

T-mobile leading the what? Lmfao dude, why does every alarm artist come out the woodworks with the inanities as if on cue? ...T-mobile leads the pack on 5g 🤣

so we're supposed to believe that every single network had glitches because @TMobile made a mistake?

Yes,providers don’t exist in a vacuum

i wouldn't threatmap.fortiguard.com everything pointing at the us yes.

Fox News is scrambling

Interesting FB messenger data....

(No comment)

Ha...I knew something didn't seem right.

Looks a bit tame, actually

As someone with a just-above-layman's understanding of the Internet, may the deity of your choice bless you for breaking stuff like this down. I rely entirely on patient experts.

I welcome boring. what is going on?

Was unable to send and receive text messages today--and therefore unable to log in to work on my QuickBooks account. We have T-Mobile. It's a big deal to me. Lost a day of work.

Well I fell for it😡

I have AT&T and have had full service all day

Me too!

Not me. Mine went down for over three hours.

I’m in the East coast. Not sure if that matters

It can, not always depending on where the fault is but most ISPs have multiple local/state servers, major data centres, etc between you and the wider internet so unless the fault is higher up the ladder you can easily be fine while on the far side of the country.

🧐

these damn hackers 🤪

how did they manage get a hold of that big DDoS attack is the question

Potus said the next thing will be removing his twit acct. well q said it but same thing!

what is the red circle showing

Q

qmap.pub been giving us intel for 3 years. I try to be patient and kind at waking ppl up from the MSM.

So a configuration change at a cell phone company brought down the entire world's internet? Gee that's concerning don't you think? I don't believe you.

T-Mobile did have a major network change planned for today, and it only takes one major ISP to go down for everyone else to be impacted. Look at the trends, it's clearly not a DDOS.

(No comment)

When you realize that the vast majority of internet users are just 19 years old boomers...

is a DDoS attack of this scale even possible?

If I'm not wrong, one of highest DDoS attack had the target at Github, with the peak close to 1Tbps. That is one rease why every ISP should be part of #MANRS and do they homework.

Its not so simply to knock down a huge server like Facebook that has a lot of CDN around the world, but it cause some momentaneous slow down and depending, can make some difficult to some % of users to reach they service

Short answer yes but with lots of planning and people.

And likely super expensive to maintain

Very

see this thread

yesss. can you believe this mess?

So what actually happened 'Cause here in Ottawa Canada FB messenger stopped, 4G and internet def slowed down.

What he said. One ISP fails and it creates a domino effect.

Then what's this? threatmap.fortiguard.com

A very, very quiet day. You're only seeing tens of attacks per seconds. A large DDoS attack would be composed of millions of attacks per second.

Coming from someone with 28 followers, next.

I may have less but the websites don't show anything different. Why is the United States showing the most rn? threatmap.fortiguard.com digitalattackmap.com People are even saying their Verizon service went down, can't call 911, people are getting emergency alerts on their phone

I'm saying if someone with less than 100 followers is going to tell me that these websites don't show any status of a DDoS attack, I have to wear a cap.

You validate someone’s opinion on how many followers they have? Not, I don’t know, their experience in a field? He is right. That’s a boring day in the office.

It really is, though the impact on FB messenger of whatever the hell was going on actually going on today affected me and my partner (different countries currently) on and off for a bit.

Followers and the first 6 images I see on the right on twitter website being super political.

Properly stupid aren’t ya?

When #NVCs internet connection in Aberdeen, SD, goes down for 8-10 hours with no explanation, customers want to know why. #Maddow

Dug up fiber line, routing hardware failure, expired security certificate, leased bandwidth from an affected network operator? Why don't you ask them instead of doing whatever this is?

I asked and got “we just don’t know”.

So you want Cloudflare and Rachel Maddow to investigate a local internet outage that lasted less than 24 hours where less than 0.008% of the US was affected?

You're so full of shit. How much did China pay you to Tweet this?

How do you counter this claims, because that does not sound like a convincing argument

Counter? This too threatmap.fortiguard.com DDoS traffic doesn't require GB's of data, specifically crafted traffic can also put a server on it's knees.

Lol

Do you also think UFOs make crop circles, and psychics are legit? If you do, buy some of my merch. If you're dumb enough to make that comment, you're also dumb enough to spend your money on a creator you've never heard of.

You'd be one of those Flower pot men... Then...😂😂😂

I don’t believe ANYTHING anymore especially news via Twitter since it’s all censored to fit a narrative of them evil elites like Hillary. They are fendangling the network with more surveillance AI technology

Do you also think vampires are real, and 9-11 was an inside job? If you do, buy some of my merch. If you're dumb enough to make that comment, you're also dumb enough to spend your money on a creator you've never heard of.

Thank you very much for making clear that we are not under a DDoS attack and that a true DDoS of the scale some claimed we had would have made it impossible for you to even send what you typed.

lol. nobody believes that......

Why does there seem to be reports from more than just T-mobile service having issues?

Many mobile providers share their infrastructure, maybe has some relacion

Definitely shared infrastructure... you only see the difference on the edge because that's how Verizon or T-Mobile broadcasts it; on their frequency and technology.

(No comment)

You do have to understand that a DDoS doesn't need to use gigabytes of internet traffic. If someone knows how to attack a server to make it take time to process a lot by just using a kb of data, then so be it.

My phones lines at work were having issues. Couldn't connect. Was dead for few seconds then was busy. Most of the day. We have spectrum

Would you actually tell us if there WAS a DDoS attack or would that be bad for company stock prices so it needs to be kept hidden?

Pretty sure a major DDoS attack would be amazing for us given we’re one of the only companies that could stop it. Good lesson: when you hear hoof beats, think 🐎 not 🦓.

Major power shutdown next

Yup

Good to know. I’ll keep that in mind.

Is it right that this is called a bgp attack not a ddos attack as it’s a dns hijacking thing not an overload of packet request and there are news that it’s still under control and not considered a real attack yet!?

It's a routing issue from m t mobile he's claiming

... Unless you live in Kenya, Ethiopia, or Namibibia. Then reverse that (sorry. Old public health response to that statement. 😆)

Are there not other forms of attacks other than DDOS that could have caused the outages that people experienced on twitch and other services? What other types of disruption attacks or accidents could it be? Feels like we're pretending the other outages just didn't happen...

🙂

Ok pussy

Thank for your excellent explanation of the T-Mobile outage. I really appreciate you sharing your and your team's knowledge of the underlying issue, how people reacted, how the rumors began, where bad actors added bad info and then how it spread across social media. THANK YOU!

but he's wrong lmao threatmap.fortiguard.com a ddos doesn't need GB's of traffic, a ddos can even happen with just kb's of crafted traffic.

This is a good place to see all the outtages, Matt is right. puck.nether.net/pipermail/outa… Fist bump to the CA Geeks.

Comcast goes down every so often .... too bad it’s not permanent so we don’t have to pay these exorbitant bill amounts every month .... a total racket !!! #comcastrippoff

I've seen a lot more than just T-Mobile on the fritz today and heard from a contact at the Hawaii exchange that it did look like an attack.

Where’s the attack exactly?

In the heads of people who don't understand tech, and idiots. I understand the first group.

Looks to me like yet another poorly executed change management process. Clearly the back out was insufficient.

I don't tech, just parroting what I've heard which is attack BGP attack on Level 3 which caused downstream problems. But that BGP vulnerabilities usually only take an hour to fix, so that was maybe a separate issue.

Nope. Cc: @tompaseka

Stop parroting what you heard without evidence, that is the entire reason this stupid DDoS nonsense took off in the first place, and is the entire reason Social Media is a toxic wasteland of misinformation

(No comment)

My phone on! 👍👋👋👋

Jordan sather

It’s coronal mass ejections

Dont downplay what happened today. A town in Tennessee had to shut down their entire computer grid cause of ransomware. Alabama paid $300k to get their systems back up. Chinese group APT Gallium shutdown Australia's #1 cell provider A1 Telekom, BellTroX infiltrated high officials

And much more. DDoS attacks are used to shield the actual agenda. Look at this list of companies who were all affected just today. This never happens they are searching for our power grid and comms. DO NOT LISTEN TO ANYONE WHO DOWNPLAYS THIS.

You use a DDoS attack as a smoke screen so that you can remain persistent and undetected on compromised systems/networks, not to carry out ransomware attacks that are the opposite of covert by nature. I have absolutely no idea what correlation you are drawing there.

And A1 is an Austrian Telecom, not Australian. And that has nothing to do with today because they compromised in late 2019 and only just recently stated they had fully remediate the network.

All my homies hate cloudfare. Esp with Shoe Palace

in case u missed my thread RT @nappyisCRACKED @FreeFallsf

I do definitely think that it leaks past just T-Mobile however. The weird stuff I saw at work today on some of our customer's systems makes me think that something is up with routing somewhere. Not only are some of our SIP trunks riding out on Intelliquent are a bit higher (1/?)

...ping than normal, and one of our Centurylink PRIs has stopped calling Google Voice numbers along with most T-Mobile and AT&T numbers at times too. The PRI is a bit of a mystery but the SIP trunk latency is significant of something past T-Mobile. (2/2)

and that does NOT MEAN A DDoS attack. That could be just routing issues thanks to a misconfigration on T-Mobile's side causing a flood of traffic. (3/2)