1/ Like many, I woke up to this today. Struck by so many saying matter of time, expected (esp FB login)… Responding to major, potentially existential, incidents (quality, security, privacy, abuse) as large org is unique. Thoughts on living through and coming out the other side.

First, this isn’t about FB—I have zero info. FB remarkably secure by engr/design. This is awful. Second, w/ any crisis, there are always people somewhere (internal or external) who will say "told you this would happen". “No matter what happens someone always said it would.”

This is really about how the culture of security came about. Why do BigCo seem "slow"? Why don’t they seem to learn from past experiences of others? Key questions w/ not so obvious answers. BigCos got big by executing and exist execution mode, not crisis mode—getting stuff done.

Any crisis that arises is just another thing from customers, partners, execs, new hires, support, the internet, sales, ops, finance…that’s there to thwart execution. To be in scale/execute mode is to be under siege to prevent execution.

Startups OTOH tend to move from one existential crisis to others (scale, roadmaps, funding, hiring, firing, etc). The brain muscle for managing through sudden change is baked in to almost daily actions—default mode is managing a crisis.

For something to elevate to crisis response is (wait for it) a process. Takes many escalating issues (instances) to accumulate while responding almost like stages of grief. All along Co thinks "can weather it this time" and importantly "no, we are right" b/c of success/scale.

Four phases of response: 1 Not really a problem—"holding it wrong" 2 Will do better next time—corp apology 3 We have trained our staff—act without slowing execution 4 Here’s what we’re doing—actual crisis response (where we are as industry on breaches)

This is so difficult for most crises b/c (a) not wanting to break execution and (b) response upends everything and is very rarely one decision. 80’s Tylenol scare was "easy" b/c it was one choice (recall). Most crises rooted in systemic. Think Apollo 1.

Responding to a crisis is fundamentally about *breaking* your customer promises in order to make a new customer promise. Let that sink in because the natural response to anything is not "what can we take away" but "what can we add".

Much of the time spent crafting a response (being slow) is weighing alternatives that all suck. And because of the internet now this is being done by a chorus of people who are telling you how, it is easy, and just change this one thing.

Quick story about PC viruses (still around but vastly different). My first summer at MSFT I was "trained" (get it) in defensive coding against TSRs (gik). Thought to be enough to be secure. BTW, TSRs were useful too (gik, Borland Sidekick).

10 years later, now Windows, Internet, and ~200M PCs, woke up to a headline. Microsoft Outlook—new software just getting traction (eg still "startup")—had a new kind of virus. 10 yrs being "annoyed" we had our moment of cumulative events.

We weren’t unaware of viruses or mechanisms one used (attachments + macros). Features were Outlook’s competitive advantage. Whole industry built on these features. CRM morphed into "programming" Outlook. Sharing meant emailing executables!

Enough is enough. We decided to break what we had built. No board meeting. No committee. We stood in the Outlook hallway and made a plan first thing Monday morning. Then (here’s the thing) we used our execution capability to get THIS done. Code released in a week (on the web😺).

We broke the product in order to make it better. Removed emailing most attachments. Removed automating VBA Outlook. Everything broke. It was easy to break things. We actually (hindsight) broke too much and over-reacted. That’s common too.

This was incredibly poorly received. Customers expectation was naturally that viruses would be "fixed" without costing paying customers anything. Importantly "why didn’t you plan on this happening". Sucks.

Another crisis in mail was SPAM. Imagine a world where you wake up every day to hundreds of ads for get rich quick schemes, free stuff, no money down, etc. Email was working as it was designed. And no one in charge. Fixing followed this pattern.

Email eventually "broke" (‘please mark as safe sender’) before it worked again. How many ppl lost messages? I know a lot because Outlook got sued in federal court for "failed delivery" and *lost* because Outlook should deliver all mail it “receives” (sound familiar re: newsfeed?)

By "stage 4" of a crisis and responding so much more than product is changing. Crisis response changes successful company cultures of: • Product planning • Resource allocation • Definition of good/success • Customer/user expectations

Like any tragedy or loss, going through something like this can consume and become bitter (ask Intel old timers about Penguin flaw) OR the process can make a company, team, product stronger. It really can.

Microsoft went through a massive culture change when it came time to address security company wide (after a few viruses in other products, Windows and SQL). Output was "Trustworthy Computing". Very proud of this passage :-)

From outside, it is always easy to say something was preventable or company was slow. Easy. What takes time is for company’s to feel they have permission and capability to essentially break what was built.

I’m not defending any (in)action but I am empathizing with what it takes to respond to a crisis. Whether it is abuse, fake news, security, privacy, and more circumstances change, expectations evolve, and technologies go from great to threat.

No one is being fast/loose with security these days. FB, Google built on Microsoft learning. Security in big tech is like safety in airlines. Everything is security first. The culture is there. My feeling is the next wave of culture change is privacy. Privacy first. // END

As an addendum, some comments on this last in the thread.

#GDPR lol

Great thread Steven. And wrt privacy all I will say is god I hope so. Sam said it best:

you were on a roll today Steve.

Privacy is more than a culture, it’s part of the business model. Apple was able to focus on privacy because it was not changing the revenue stream. It’s a different issue for FB and Google. Add the separation between customer and user and I wish them well.

Yes for sure. But also "openness" was part of the software culture for a long time too and it can be changed. Yes though, fundamentally there is a shift that will either happen (or not).

It's 2018, and people still say, "Apple is fundamentally opposed to Ads" instead of "Apple tried ads and failed at it." It's like iAd never existed. Apple boasted that they connect to the "same ad exchanges," and had "great targeting."

AFAIK, it’s not that Apple is (or was) opposed to ads, it’s that they’re ostensibly committed to privacy first and tosses whatever’s left over to ads.

I'm not sure I understand what that means. I do understand that these pages say that data from iOS apps is way more valuable than web data, and by the way, here's how to re-target. bit.ly/2xR6MzU

Less scary link: bit.ly/2DKJ7Xl

This tweet is a Manifesto “Security in big tech is like safety in airlines. Everything is security first. The culture is there. My feeling is the next wave of culture change is privacy. Privacy first”

Excellent tweetstorm!

Steve - excellent thread. Otoh, MS makes money on software and systems. Apple on hardware. Google & FB on selling data to advertisers. This is hopefully FB's Ford Pinto moment. Though i'm afraid Zuckerberg is more John DeLorean and less Lee Iacocca

I'd suggest a slight correction: Google/FB sells ad-targeting

If only Twitter had an edit function...

No problem, it's kinda pedantic on my part

I did not do a great job on articulating that while MS sold software it also sold openness/hackability of the platform. Reducing that and closing it off was a massive shift in value delivered. In hindsight of course it still didn’t go far enough.

One could also argue MS sells services. I'd argue though that the balance between openness and closedness is a difficult choice to make, especially when it comes to Microsoft (for historical and political reasons). I like the current balance though

I agree with you on that. In 1990s, Microsoft was the security punchline. I'm impressed and grateful how far MS has come. The legacy of windows drivers flaws remains And I wish the Edge team learned their history - recent escapades remind me of Netscape V MS.

My WISH is that A) Microsoft stops renaming/rebranding things randomly B) EMBRACES USER SAFETY C) supports the IIHS-equivalent for software Privacy battle has been lost Security battle is ongoing Let's get ahead on SAFETY

I don’t buy the security first point but I agree with your last point. TwC in MSFT didn’t really catch on elsewhere

You are either overestimating the degree of functional security hardening in most companies, especially once you get one tier outside of Google, or do not understand aviation safety at all.

It really depends on how you would compare them and what you consider an incident and what you. At about one crash per million flights, that is not far off from breaches per million users and more if you consider sessions.

No. Really, no.

There you have it.

It’s not security first not even by a margin. I know your statement looks good but let’s not pretend the IT industry can compare with the Car or Aviation industry. By your logic we would not have a continuous stream of breaches due to lots of simple (and preventable) problems.

I feel like you live in a different universe than me.

Moving fast. Breaking things. For stable infra.

I remember you telling a story that you made up something close to the actual plan on the fly when a reporter called you.

At some level but you know me, I was always reluctant to commit to anything I didn’t understand so I did what the boss could (should) do which just commit to product changes and then watch the team create.

super you did that. thinking win8/tiles = clearly superior desktop. but msoft unable to weather customer outrage over start menu demise. would have taken some yrs till uniform experience. but sad to see that go.

Here’s the right image for this one. Weird iOS photos caching thing sorry.

Wow the mood here is taking me places