Convopage
See the entire conversation
Seeing a bunch of folks in my mentions saying “TIL Google doesn’t use Kubernetes” and ... ๐
I’m surprised there are people who think Google *does* run on k8s.
Even GKE (Google’s hosted Kubernetes offering) runs on Borg, which afaik, is far more sophisticated and battle-tested.
89 replies and sub-replies as of Apr 26 2020
I should make a t-shirt:
containers &
vms &
containers
Run a Twitter survey, you'd be surprised. I myself believed they must be using k8s in at least a few newer places even though they historically didn't.
There are some newer GCP products built using GKE.
Kubernetes is absolutely in use for various efforts, but it has not and probably will not for a long time, wholesale REPLACE Borg.
An hosted Kubernetes offering doesn't run on Kubernetes? That's still surprising :o
See this tweet for context (whole thread is nice top):
twitter.com/IanMLewis/stat…
GKE runs on top of GCE which runs on top of Borg. Containers running in VMs running in containers. Inception ftw.
What happens to eating your own dog food?
I've got no clue what that is. What is it lol?
That explains it lol
But wasn’t that the whole point? k8s was supposed to be the user friendlier version of Borg. Which in turn is still a bit more reliable because of age.
sone folks at Google realised their internal thing wouldn't be right for everyone so they created a more generalised version with similar principles (& then the project founders left Google & Google contributes less than half the code)
the marketing has done its job... add the « let’s do it like google does » pattern, and many ppl see k8s like a way to be more like google while it’s a total nonsense
If you are doing Plain Old VMs, then Kubernetes is absolutely a way to be more like Google. Borg and Kubernetes share a lot of core principles and ideas, but it's more like "evolutionary branches with a common ancestor" than "copy-cat".
sure, the point is more thinking that kubernetes is mature because it’s been battle tested at google (even if just inspired by borg). When you read k8s code and architecture under the hood, you know it’s not. The only common ground with Borg is eventually the API
We've tried never to even imply that it is the same code, but that nuance gets lost as the message gets passed along.
The common ground with Borg is principles and ideas and patterns. The APIs, the feature sets, and the operational characteristics are all very different.
I disagreed here. the marketing has been fully working towards this idea. The communication around gVisor is an example of this.
I can accept that SWE didn’t meant to, but stuff like kube proxy 2 first implementations raise real question on de development side... don’t you think?
Which questions? :-)
well if it’s not been intented for production workloads, what was the goal?
kube-proxy #2 is used in many production workloads...?
kube-proxy #1 was pre 1.0? it served to first implement the service model.
(assuming 1 = userspace, 2 = iptables)
exactly at that same time when you considered it in early stages, I know companies that were shifting from mesos to k8s, while they were running GPU jobs, which weren’t even a thing in k8s at that time. I can assure you that marketing has been full throttle for years, even early
Who said it was not intended for production?
Iptables Load Balancing ? E-W traffic routing?
iptables works REALLY well for many use-cases. It's not particularly fancy, but for a huge body of users it still works great.
It turns each node into a "sidecar proxy" in kernelspace.
do you use this between Borg and your networking overlay? I guess not. For smaller workloads where it applies, Nomad or Swarm were already credible candidates. So why k8s if not to take on higher workloads?
GVisor IS the same inside Google. That is very different from Borg/K8s
Running on ptrace !?
Borg was certainly easier to reason about because it didn't have so many layers of abstraction (app on OS image on container framework using service mesh on overlay network on kube worker on compute platform on cloud provider).
And it certainly didn't involve Go templated YAML…
I will take templated yaml over bcl...
Hahaha. You've got me there!
B/GCL had some... uh "quirks" and, like all templating, had a nasty tendency to become a ball of razor wire that was impossible to reason about.
But the idea of structured-data transformation over string interpolation was a good one.
Kubernetes happened because Red Hat and Google. OpenShift is closer to what Borg is in that it provides stuff kubernetes (which is just an orchestrator) does not.
Without knowing the subject in detail, I dare to claim that Borg is far more purpose-built solution than #kubernetes ever will be, and kubernetes have now already slipped far from the vision of the original authors.
You can just ask the original authors, they're still here and still involved with Kubernetes
Let's do that then, right here and right now! @jbeda @brendandburns @cmcluck @bgrant0607 @thockin How far has #kubernetes slipped from your original vision? Does it look like a system that you wanted it to be? Or has it also grown to unfavorable direction?
There's no "unfavorable" direction. It has fulfilled much of the original vision but it has done SO MUCH more. It's still (comparatively) immature, and it suffers tech debt and scope creep and tragedy of the commons, but none of that is unexpected or uncommon.
I think comparing Borg and Kubernetes isn't really the point, they're built with very different goals in mind.
I think in general brush-strokes Kubernetes is exactly what we imagined.
I think in the details it contains many many things we didn't imagine.
And there's been lots of evolution along the way.
Extensibility wasn't the original plan, though it was rapidly added.
I don't think any of us imagined the growth of things like the operator pattern. (credit CoreOS)
and while you're all here: is the name the terrible, terrible pun I'm convinced it is?
No, that's more happy accident than intentional.
Is there a nice comparison between the two available somewhere?
Also, the commonly used ”manage your infrastructure like Google” is non-sense when talking about Kubernetes, at least nowadays?
I personally feel like it's only about half done... Maybe that's just my area though...
A situation where you would no longer find anything to improve—how disappointing? :)
*shrugs in the general direction of rkt*
this feels like a tree falls in the forest question
if software exists with no one around to maintain it, does it make a sound?
I'm afraid disappointed users do, but it usually it is not enough to bring the dead software to life
Most (useful) software follows an S curve in which initial experiment with one use case is followed by steep growth leading to a tip to slower growth and maybe an upper bound.
So it can be 80% done, but never actually finished.
Yes. When it's last maintainer has stopped maintaining it, and it's last user has stopped using it, and it's last install has been uninstalled.
Pfft, your area is like 1/10th done. LOL
Every time we finish one half, two more show up...
but Zeno was wrong?
Zeno never tried to run a big software effort.
That's surprising because the "K" in GKE stands for Kubernetes. Is it a marketing stunt? And we interact with GKE via Kubernetes tools, kubectl, using the same web interface as minikube, etc.
Did they rebuild and adapt everything? That seems crazy. They are also announcing regular upgrades of the k8s version. The only option I can think of would be k8s running on top of Borg?
Then it means the original statement was too broad: only the core k8s engine is not used (replaced by Borg) while all the things around it are used. There is a translation layer in between.
No, you're going too far with it. There are machines. Those machines run Borg. Borg run Google Cloud VMs. GKE run on those VMs.
GKE is Kubernetes, nothing has been translated (though IMO, fair game).
Thanks, somebody helped me understand my mistake, I think I have it now: twitter.com/JnRouvignac/st…
Oh I see. I misread what was written. Google does run k8s for the GKE offering. But Google itself does not run on Kubernetes, everything runs on Borg. Even the GKE offering runs on Borg.
Oh I see. I misread what was written. Google does run k8s for the GKE offering. But Google itself does not run on Kubernetes, everything runs on Borg. Even the GKE offering runs on Borg.
To be fair, Kubernetes is probably way better documented and explained by now.
Borg has the disadvantage of being older, but it has the advantage of only having to work at Google. That simplifies things.
Google runs on legacy Infrastructure ๐คทโ๏ธ. I take any bet there are a lot of engineers who would prefer k8s very much over borg
I’d go so far and say that these days there are also smarter people working on k8s than on Borg ๐ฌ
Why does it have to be about "smarter"? The Borg team is AMAZING. I have tried for years to recruit some of them to work on Kubernetes and they're all "no thanks, I'm having plenty of good times".
You're right, it really doesn't have to. I'm just tired of this ideologically crapping on Kubernetes. First it was "nobody needs this, unless you're Google". Now it's "Not even Google uses it".
For sure, but to borrow a phrase: There are only 2 kinds of systems - the ones people complain about, and the ones nobody uses.
Kubernetes isn't perfect, and a tweet isn't enough to get into why or how to really fix that, but it Kubernetes a lot of work done every day.
But frankly I just tried to reverse the initial point of Borg and Borg people being 'ahead' but entertaining the idea that the opposite might be true instead. I don't know, I just have a gut feeling. Still, agree it's silly to discuss who is smarter
Smarter is the wrong comparator. Borg is different. It has different requirements & users, and carries 16 years of use-cases & bug-fixes & hacks. It solves problems that are similar in form, but VERY different in details.
K8s doesn't want to do what it has to do to REPLACE Borg.
"better" is veeeery subjective. I tried to read the k8s upgrade docs and felt as if I was reading a children's story book: "once upon a time there was kubelet, etc."
Compare that to HashiCorp docs and you'll see what I mean.
Disclaimer: I like the simplicity of HashitCorp ools.
Take the 'do it like Google' nonsense out of the chat for a min. Does this tool work for the needs of your org? If yes, it's a candidate. If no, don't use it. Why do we need the kind of infantile trendy nonsense to help us make decisions?
I only recently realised the awful pun underlying the naming relationship
HashiCorp Nomad was influenced by Google Borg.
Most of the times a much better alternative to Kubernetes:
hashicorp.com/blog/nomad-ann…
nomadproject.io/docs/scheduler…
@HashiCorp
HashiCorp Nomad
Today we announce Nomad, a cluster manager and scheduler designed for microservices and batch workloads. Nomad is distributed, highly available, and scales to thousands of nodes sp...
hashicorp.com
I have gotten into legit arguments with k8s fanbois about this. They tend to swear up and down that:
1. It can’t be true.
2. Even if it is true, it doesn’t matter because k8s is basically the same thing as borg.
3. I must just be a hater and therefore can’t be trusted.
borg and k8s can both schedule workloads that are run from some distributable artifact that doesn't know anything about what it's being run on. That's about where the similarities end.
Someday there will be papers talking about open source vs cloud business vs internal versions of things! There are constraints/tradeoffs wrt TCO (both infra and software maintenance), legacy couplings, limitations of monorepos, upgrade cycles dictated by internal vs external! /1
Imagine that AWS, MSFT are not immune to these pressures! #LeverageRules
We're building kind of our own cloud at my job. I mean we are a cloud, but that's a different story.
But I regularly point out that Google, MS, AMZ don't run k8s, they offer it as paid service. We're currently running Nomad, but either one is a single piece of a big puzzle.
Kubernetes is an app server