While I appreciate AWS's support for the local development model, I can't shake the feeling that it's the wrong approach to Serverless development. Can we talk about that?

It just seems like an awful lot of work to locally mock a nuanced interplay of services that will always perform differently in the AWS environment.

While true, the development lifecycle still hasn't matured enough to completely Dev in the cloud just yet, especially for those who code on commute.

Hmm. Maybe my bigger gripe is that @awscloud is releasing a lot of different things for a lot of different use cases, but not making clear which models are recommended vs. legacy?

This I agree with, especially with so many ways to do something, the flexibility afforded leads to lack of good implementation patterns.

(Possibly sensitive)

Prescriptive? No. Opinionated? Hella yes.

If you were say an AWS influencer with a gazillion followers, you fomo them into being guinea pigs. A/B testing via time zone separation. eg SF -> SAM NY -> AWSCLI Aussies -> GUI hello world aka “what does this button do?” Kiwis -> CDK (bastards have it too good there)

(I totally get that I'm weird. I'm writing this while coding in another app. From an iPad. From 30K feet. I'm not doing local dev.

You, sir, are extra ordinary. 😁

I've spent hundreds of hours building something that a few companies sell for $30 a month. Absolutely do not do what I do.

What are you using to do the code on an iPad? I'm trying embrace change, but I just haven't been able to do it with an iPad yet. Surface yes, but then it's really just Windows anyways and I have IntelliJ and the Visual Studio Code.

An ec2 instance.

Ahh that would do it.

Mosh is a game changer, and Blink supports it.

Nice, I had never seen that. Thanks. It looks very nice, I shall try.

Local dev doesn't have the potential - small but real - of accidentally costing you a chunk of cash because you made a mistake. Billing unpredictability remains a real deterrent to adoption, especially of squishy autoscaley stuff like serverless.

Ooh. That's an excellent, excellent point.

However, billing, like performance, and reliability, needs to be part of the development process. If you build it, but can't pay for it--or worse, didn't realize how much it would cost--that's not a good situation either.

Docker is the only acceptable "local emulator". I think anything else actually provides negative value.

Tend to agree, stuff that’s easy to run can be a useful toy caveat emptor. Still remember when Atlassian wrote a massive stack to emulate AWS services locally and open sourced it😳

Good heavens it still exists. Whye hast thou not smote thine nethermonster?

The ability to quickly try something out locally is extremely valuable. The round trip of make a change, push it up to Lambda, wait for the version to become available, execute, wait for logs in CloudWatch is over a minute, and becomes very frustrating.

Need to catch up on the state of play, a local executor is essential considering average latencies. The problem is more AWS docs missing unknown unknowns. Tentatively any local test should execute in Lambda right after if online.

It feels like it's reducing Serverless to "run this function" or "handle this API call" instead of what I perceive to be its real value--the event driven model.

To be clear, this is me thinking out loud. I'm not sure I'm right--and I'm certainly not accosting anyone else for their choices! I'm just hoping someone can make the opposing case. In other words "I really hope I'm not being an asshole, if I come off that way I'm sorry."

You are 100% correct. And other than meeting people where they’re at, I don’t see the reason for the emphasis. My serverless workshop emphasizes unit testing and mocking (eg. Python moto) and everything else is done in AWS.

I think a lot of this strategy is meeting people half-way. Many folks are more comfortable building software locally right now, and if that helps bridge them into working Serverlessly I’m for it. I do slightly fear a local maxima of productivity, but that’s what education is for.

Ive has this same conversation a few times. The feedback I get leads me to believe there is a misunderstanding of the Benefit of unit testing your functions. Instead trying to run an end to end test on local resources... which don’t act exactly like the real ones anyway.

I thought I was alone in this thinking! I don't understand the reason for local simulations either. We can already test the business logic with unit tests. The rest needs to be tested in a real, reactive environment.

In my experience, lowering cycle time (in the “write code”->”try feature” cycle) is one of the more effective technical ways to improve developer efficiency. How to keep that cycle short in server less land seems like a problem we’re still figuring out.

"Push to AWS & test it there" will probably never be fast enough. "Stub everything and rely on unit tests" will probably never be reliable enough. There are a few different approaches to solving this problem, and "run it against local mock services" is certainly one of them!

Yep. It's a billing issue really. Some kind of better free tier with rate limits and maybe no support, just for non-prod purposes.

Yeah. I came to this conclusion a very long time ago. There are so many things wrong with the constant mocking... Wait... You know what I mean.

We lack the tools and do not have the training tools to actually teach someone to develop cloud first instead of local first. The "local first" paradigm assumes that we have a reasonable analogue of the system available. When that assumption fails we don't have other options.

The problem is the tooling for "cloud first" doesn't exist in the same way. The tooling for "local first" didn't just spring forth all at once. And we can't assume that the same patterns for "local first" are fully transferable either.

But mocking everything is basically the only option locally. And there comes a point where you just sit there and go "this is pointless". It's like you're coding to prove something works when you can actually test it against the real thing. That's what dev environments are for!

It's primarily driven by lots of pragmatism: - not everyone has a persistent web connection - devs often have powerful machines they want to use/justify - non-local dev really emphasises lock-in compared to previous development models - it's just different, and change is hard

"Works on my laptop" doesn't work in the cloud quite as well. It works in my fully formed hodgepodge of janky code I copy-pasta'd from the interwebs just doesn't cut it.

Also a bunch of stuff about how companies budget and bill for dev environments. Same, but different, conversation about serverless costs you see elsewhere, just from a different budget with different sensibilities

I think the event driven model is the how, but not the why. The why is independent scaling (including scale to zero), which actually fits perfectly with the needs of a development environment.

Agree. If your application gets beyond a few functions and services, the local emulation ends up brittle and not very comprehensive. That said, I understand what AWS is doing. Lots of people are resistant to the full 'test in the cloud' workflow. Meet them where they are.

I'm not asking for a "their way or the highway" framing, but some best practices around "how do I do this if I'm starting from scratch" guidance would be a great idea, no?

Definitely, though I think there's still a lot of ¯\_(ツ)_/¯ for this. One thesis: there's more variability across companies in serverless architectures than in traditional three-tier webapps that makes it hard to frame best practices.

I think this is part of why we haven't seen the 'Rails for Serverless' yet. The fact that infra is such a big part of your application is new territory. Rails could assume HTTP requests and MySQL. Less common ground in serverless.

You’re assuming we “know” the best practices when we launch a new service. Often, we throw something out there and watch what customers do with it, and then take their advice ourselves. Example: we didn’t coin serverless or start the first serverless framework (JAWS).

It helps for basic primitive use cases that doesn't depend on a host of other interconnected services. I wished for local development coz pushing every single change to cloud and testing is a pain , even with cli

Interesting, and valid. I've got a few standalone functions; with Serverless Framework pushing to cloud takes ~45-60 seconds per iteration. Are you seeing faster cycle times with local development?

Yeah, local debugging with logs/traces are easier.

I thought that for the longest time! Then I had a couple of conceptual breakthroughs that fixed that for me.

Can you outline some of these conceptual breakthroughs?

Have you tried deploying directly without CF? Should be much faster during development. Serverless has that option (or had it when I was there and we put it in, not sure if still in)

I guess Azure (has tool set as well) and GCP too provides similar features for Serverless development.

I think you're right, but I think it needs to be easier & more established to spin up burner accounts on the fly . (plus gaps in CloudFormation et. al. need to be addressed, so you don't need to do things with your face all the time.)

I've had many surprised looks (even inside of Amazon!) when I suggest that every SDE should be able to create & provision an account programatically. I can't use aws.amazon.com/organizations/ for complex reasons, but it's a far better solution than shared sandbox accounts + IAM users.

Organizations has gotten way better, but try deleting an account when you're done.

+1. Docker's nice and all for local development—and some packaging—but you _really_ need account-level isolation. (I'm in furious agreement with you, but I also have a 1Gb/s ethernet pipe, so take that as you will.)

Have you actually done this? Pretty sure my request to support is still sitting there a year later. -_-

Um... with or without an escalation?

Haha. I haven't seen a ton of people talk about this, so I thought I must be the only one!

I just went through account deletion/removal from an organization and it was mind numbing. It involved a support ticket that needed to be babysat every day for a week 🙁

I don't understand this thread. I've deleted a couple handfuls of AWS organization accounts with no problems ... except the need to reset the root password ... and the need to use the web console.

You know we can do that inside Amazon, right? I assume the conversation was about external SDEs.

Yep! It’s about external SDEs.

Sweet, yeah - totally agree. :)

My personal theory is that the billing system is as it is simply because nobody calls Amazon SDEs on the carpet for resource use. I mean, yes, they do-- but it's not a CFO sweating blood who thinks "Amazon bill" equates to "bought a lot of books."

I have felt this way all along. With CloudFormation I can easily spin up test/dev environments, and with CodePipeline/CodeBuild I can continuously deploy to them for testing. What are the main arguments for local simulation? Faster testing time and offline capabilities?

At some point, we *shouldn't* be able to have reasonable local facsimiles of cloud environments ( or why have them remotely?? ) Today there exists debilitatingly little(IMO) support for running complex multi-stepped workflows via testing frameworks, and we've remediated that 1/x

by repeating a historical trope that was very useful for some high impact organizations ( namely: loosely coupled + bounded contexts ). <Deep breath> 2/x

I suspect the line in the sand that made LC+BC so incredibly effective is moving currently ( and has been for a while )... BUT IMO, it's foolhardy to expect that you can or will fully exercise the number of paths that a moderately complex cloud integrated software workflow

is going to take. In this world, limited local functionality makes A LOT of sense, because that whole LC+BC thing is a right solid idea in environments where you the coder have no control over the behavior of your dependencies

So what I'm trying to say is "local makes sense*" *As long as you accept the premise of LC+BC. Because testing in the cloud doesn't actually help you WRT being able to get comprehensive test coverage Testing in the ☁️ does of course remove the "cloud as ~imagined~ mocked" bit

Having been using Cloud9 a few months, I think that what looks like a lack of investment is a lost opportunity.

Interesting. Tell me more?

Aside from having the CLI and a little bit of "maybe we'll show you this alternate editor for templates", you could easily forget that it's AWS. In one sense that's good, it's a blank slate. In the other, there's no hand-holding ease-me-in show me the way.

I'm not a developer. I write code to get stuff done. The foray into CloudFormation has a painful ramp-up because tooling sucks -- it's hard to know how much you need to know before you won't have a 5-hour impostor syndrome session because it *doesn't make sense* until Eureka.

That's both a documentation and a tooling fail to me.

We've been struggling with this, currently using various hybrid local-cloud setups. At SF AWS loft yesterday, they demo'd Cloud 9, but not a solution for us.

Nobody tells me when there's a good session at the Loft; I can work from there as easily as home most days. Cloud9 is interesting, but has some constraints that make it not for me.

It was mostly DevOps stuff we already do, but the lunch was really good, Vietnamese. It's a nice space for co-working, hadn't been to the new location before.

If it's not a secret, what did you mean by "various hybrid local-cloud setups"? I've been doing local dev with SLS, but yeah, it was only API gateway + Lambda "emulation".

We run our web app locally, with either the API server or DB or computational engine each separately local or in AWS, all talk to S3 (no local equivalent). It works, but it's messy.

I understand. Having the similar problem here. Would be nice if one day we could have mini-local-AWS (I'm such an optimist 😁).

"Can we talk about that?" Yes, please! We're in a phase where two years ago everyone was doing local dev/testing, but two years from now everyone will dev/test in the cloud. Tooling has been a barrier (esp. managing envs), but tools like @stackeryio now take care of that ;)

Most definitely. Although I think my unwavering agreement with your take might make for a short conversation. Also,

Anyone at #awswishlist looking to bump a feature? twitter.com/mweagle/status…

I haven't looked into SaaS Chaos Monkey, but would it be viable as a SAR app with one-click deploy into your account?

Maybe? Possibly as a layer that taps/transforms API responses based on SSM config? However delivered, something like this capability would be 👌

Having a local emulation helps a lot in that business logic unit testing -- why go to all the work of a mock when you can have an emulation? We use Dynamodb and s3 on localstack extensively. Adding step functions will be very helpful also.

As others have mentioned in this thread, it's primarily about meeting developers where they are. I'm totally comfortable developing against a dev stack spun up from the same SAM template I use for production, but this is a totally new workflow for most.

As someone who has built local and remote k8s dev tools: Its the feedback loop. Hierarchy of development needs 1. Build/Test/Deploy feedback loop 2. Reproducibility (dev vs. qa vs. prod) 3. Convenience (dep management, integration w/ existing IDEs/tools)

How else would they develop serverless?

I've been in organizations that are still really new to AWS or haven't even started yet, and you often can't make accounts and put business logic up in them without approval, so local mocks are the best way to show off what serverless can do for stakeholders / execs

It's rarely a cost issue, but a compliance or procedure or other red tape issue at the start

I also don't like branching and just committing to master/trunk, so being able to run locally before committing is a good sanity

My small team has recently transitioned to doing the bulk of our serverless dev/testing in developer specific deployment stages. This has it's own issues still but it's working for us better than our local testing solutions.

We started with serverless-offline, a function or two at a time no big deal. Then when we started needing to deal with lots of interaction between functions from multiple services we started running them in docker containers but this was clunky and ate a lot of dev time.

I have always been a big fan of pushing to envs. But my dev teams have always preferred local dev. I have to admit I find that living remote and developing 90% locally then the last mile in the specific env is far more efficient than pushing to env constantly.

Though I far more often take the hit of pushing to env and saving on setting up local env. I am a glutton.

I typically test business logic with local RDS, redis and DynamoDB running locally and this is ok. What are some examples of interconnected services that is incredibly difficult locally? Keen to learn here so I know for the future.

Totally agreed. I think if we are able to pull off a fast enough deployment engine, people will completely forget about local development. Add to that some sort of auto-watch/auto-deploy to dev, then you’ll have your own test suite in the cloud.

agreed, doesn't seem like it reduces effort or cost, since you will need to test in aws anyway.