ShipTalk - SRE, DevOps, Platform Engineering, Software Delivery
ShipTalk is the podcast series on the ins, outs, ups, and downs of software delivery. This series dives into the vast ocean Software Delivery, bringing aboard industry tech leaders, seasoned engineers, and insightful customers to navigate through the currents of the ever-evolving software landscape. Each session explores the real-world challenges and victories encountered by today’s tech innovators.
Whether you’re an Engineering Manager, Software Engineer, or an enthusiast in Software delivery is your interest, you’ll gain invaluable insights, and equip yourself with the knowledge to sail through the complex waters of software delivery.
Our seasoned guests are here to share their stories, shining a light on the do's, don’ts, and the “I wish I knew” of the tech world.If you would like to be a guest on ShipTalk, send an e-mail to podcast@shiptalk.io. Be sure to check out our sponsors website - Harness.io
ShipTalk - SRE, DevOps, Platform Engineering, Software Delivery
10 years into K8s; when is a problem worth solving? Ken Ahrens - Speedscale
In this episode of ShipTalk, we are joined with Ken Ahrens, CEO and Founder of Speedscale. Kubernetes is turning 10 this summer and we dig into how the ecosystem has been evolving.
Learn more from Ken as he walks through his journey on:
* Don't forget the fundamentals
* 10 Years into K8s, what is still challenging
* Enthusiasm, a very important skill
Hi! Everybody! Welcome back to another excellent episode of ShipTalk. I'm Ravi, the host. I'm very excited to be joined by fellow Yellow Jacket and fellow greater Atlanta Resident. Ken, ken maybe a quick introduction about yourself. Ken.
Ken Ahrens: Alright, yeah, thanks a lot, Ravi good to see you today Ken Ahrens. Here I am, one of the co-founders and CEO of Speedcale. We're a technology startup sas company headquartered in the greater Atlanta area. Most, our teams here on the on the east coast. And I'm, just very excited to be on the podcast today. Thanks.
Ravi Lachhman: Yeah, thanks. Ken, always excited to be talking to a fellow Yellow Jacket. And we we were talking before this. And we love we actually live really close to each other, which is like, oh, like similar similar life past. So, yeah, very excited. So on on the issue or not issue. But just journey of life.
Ravi Lachhman: Can you have a super interesting background, like, you know, we it's kind of intertwined with companies I worked at before, but maybe you know you, you were in some really cool spaces like, really early on you get to see a lot, especially in the observability space. But maybe a couple of minutes like a minute or 2, just about your journey. And just how did you end up at speed? Scale?
Ken Ahrens: Yeah, sure. It's it's funny about half my career. Ravi has been in the in the production side of the world. Mostly around performance problems, right? And the most common question is, why is this thing so slow? Why, you know, why is it falling over? And you know, how do we get it? Back up and running? And I worked at for many years at a company called New Relic.
And actually, really, early in my career, I started a company called Wiley Technology, which was the first Java monitoring system, and I got a great appreciation for keeping things running in production.
and the other kind of half my career has been in the pre-production space.
Ken Ahrens: Literally working on the same problem. Which is, we wanna ship this code, and we wanna make sure that we don't embarrass ourselves right after we click the button right? And we don't want it to blow up when it reaches production. It's the other side of the same problem. And I worked at one Startup called it. We built a set of technology called service virtualization, which was about simulating your dependency for your code. If you could simulate your back end dependencies. It would allow you to test things a lot better and find issues earlier. And now it's speed scale. We're revisiting some of that space and helping people do traffic driven testing so that they can find their bugs by using their existing traffic from production and recreate the production conditions in non prod. If you can do that, you got a shot at finding this performance issue earlier.
Ravi Lachhman: Yeah, it's a. It's a tale as old as time. It. It goes back to like dev broad parody, like production will always have more firepower.
Ravi Lachhman: And even, you know, I it's I really like the fact that you worked at like computer services or Wiley. Like, way back. Early in my career I used to write too much Java. And so I was. I used to work for Ibm on the Web Server application server team was like where the Java gets turned into HTML, and we have a product called was profiler, and I'd be like Wiley is better, you know. Like, is this a better tool profiling right? So kudos to, you know, with you on the CA team.
Ravi Lachhman: It's always like you never have the firepower in your machine, right? Like, you know, we're constrained like I could have like a local version of walls. Or today, like I might have a d cluster, you know, running on my macbook. Pro but
Ravi Lachhman: it's no you know that that level of scale is like nowhere near move on better. And and this is that we're talking love to talk about your experience like Hey, Parity? Wise
Ravi Lachhman: Kubernetes is Kubernetes. But there's still the infrastructure card to play like
Ravi Lachhman: clearly your you know your Gke or a Eks cluster is bigger, and it has better or more dedicated. Let's say firepower or virtualization versus like, hey, you know what I have like. Spotify zoom
Ravi Lachhman: slack, which I can't close for someone. Reason on my, you know which we, which is taking up resources versus like, hey, you know we have a you know, a. That a, TT. 3 excel on aks right like, which is like very or eks, which is fairly powerful. And so, yeah, that's interesting problem. Like, have you seen that problem? You know, parity? Go away. Get get better, get worse like these are the some of the fundamentals.
Ken Ahrens: Well, I think you bring up a really good point. And I would say, containerization has been a huge improvement in this area. So if you go back to the websphere days, you'd have to get a really specialized server. You know, and you've got to get 64 CPU. Server, and you gotta get an enormous amount of memory. And this kind of thing, and figure out all the magic Java flags so it could use all the memory on the server. And so what you have on your desktop at the time didn't compare.
Ken Ahrens: when with the shift to containers, things can be made in equivalent S container sizes, so you can say, look, this container is gonna take half of CPU and it's gonna take 500 Megs of RAM, and that's what we're gonna give it. And that'll be the building block that we wanna go and design our infrastructure around.
Ken Ahrens: Kubernetes has made this dead simple to say, Give me 4 replicas of that. And if they look like they're falling behind, just increase them right. Use the pod auto scalar and give me some more replicas. And this has been a great innovation, so that it's easier to scale this thing up, cause it's much harder to go and buy a 64 CPU. Server than it is to scale up a couple of these small
Ken Ahrens: containers, and from the unit economics of those that container should run really similar in every type of infrastructure.
Ravi Lachhman: Yeah, that makes sense. Yeah. So like we, the layers have gotten more ubiquitous. And so we could say, like, Hey, like, you know, we upon running, like, you know. Let's say, docker like run like just some sort of image we bake together, and, like, you know, we'll it should respect the CPU limits and whatnot, and then we can just like extrapolate that saying, Hey, you know II made a container spec of, you know, 4 gigs of memory, and like
Ravi Lachhman: 2 V. CPU. And like, here's some like storage. And then that translates to Kubernetes like, Hey, go place this piece of work somewhere on your cluster somewhere, and that that's gotten better. Yeah, totally. And that's that's still at some point like, you know, like because there's there is parity, you you can assume things might might with an asterisk scales, or nearly right, saying, Hey, you know, add another, you know. Add, plus one, you know, when we're doing capacity playing, hey? I was able to like run a load test on my machine, and I was able to run Gatling or some other tool like successfully.
Ravi Lachhman: or like to good old like like JMeter, like date myself. Now, like, Hey, we able to run like N number of tests. We get response time similar to this, you know, we we assume back of napkin math like with with production like infrastructure. We can do do Xyz versus like, you know, as we both know, and it doesn't. A lot of times. It doesn't scale.
Ravi Lachhman: You'll run to odd bottlenecks sometimes like, Hey, like, you know, why, why is like our network through network throughput, not where it should be or like, well, because there's 2 handshakes and like, do the little things that really come up.
Ravi Lachhman: you know you that that's always that's probably been around for the dawn of time, you know. Like, if you know, if you go back to our university days we had to log into a mainfr like, you know, at some point log into a mainframe like why, you know, I guess they question that my Cs class is like I do in my Java job. I do on my, you know, windows machine. I'll log into a mainframe. But there's a reason why you do that right, you wanna access remote resources. So as as time as time goes on, right like.
Ravi Lachhman: you find interesting problem solve. But like I'm I'm cheating because I know you're a founder, and it's like, Oh, I can ask like founder, you know, for those who don't know, Ken, you know, Ken Ken's been through YCombinator.
Ken Ahrens: Yeah. So it's this is this is really good question, Ravi. And actually, when you're when you're first starting your startup the temptation is, let's go and grab the latest technology and go build a prototype and let's go and build something interesting and get get someone's feedback and opinion on it. And actually, you should spend some time hovering around the problem. And a lot of people talk about product market fit, which is how you know that. You know, people are excited about using your product, and they take it out of your hands. There's a bunch of different signals to get product market fit. But actually, before, that is problem market fit and problem market fit is people agreeing that this is an important problem and a problem worth solving. And if you were to sit back and say, I want to solve a problem related to like, let's just say like CPU related performance. And this kind of thing. People would laugh you out of the room, probably because they'd say, I can use kubernetes. I can use cloud services this whole kinda having to get a bunch of cpus to run a workload is a solved problem. I have. I have a ton of choices.
Ken Ahrens: But if you ask people things like, are you happy with the performance of your applications? Well, companies like data, dog and new relic and app dynamics and their level of popularity? I don't want to leave anyone off. By the way, Grofana and Dinah's race and we won't. We won't ever end. The podcast will run out of time. The popularity of these tools is proof
Ken Ahrens: that people care a lot about these performance problems and that they haven't gone away and so, having access to near infinite CPU appears to have not solved that. That's where we ended up starting with a with part of our problem. But you have to talk to people get feedback. How do you solve this? Do you agree that this is a real problem? And how do you solve it today? Are you happy with the solutions you use today is super uncomfortable. Talking to talking to users and finding them. And and but that discovery is is mission critical. So that you hone in on a really important problem to solve.
Ravi Lachhman: one of the things love to get your take. It's it's a lot of it's similar to like starting your own business.
Ravi Lachhman: 10-15 Years ago, devops, is just coming of age, you know, like companies were maybe like hot by assigning like a person, you know, to like devops or platform engineering, or even, you know, it's funny, like, Oh, I need someone with 20 years Kubernetes. Experience was possible because it came on 2014, you know. That's kind of a possible but you know. And and let's say the next big thing. or the currently big thing is like artificial intelligence. Use a buzz word here. So we get some more SEO with the, with the podcast but look it like same parallels as you were starting your own firm, like if we were to empathize and say, Hey, you know, there's there's one more net new piece of techn like there's there's a challenge that's out there or like, there's a new piece of technology or or look at this, some sort of people problem process, prompt technology problem like going back to solving a problem like, Hey, like, how do I know like, can I have a very specific job of my company figuring this out like I. I'm the first Devops engineer, you know, 15 years ago versus I'm now like a manager or vp of DevOps.
Ravi Lachhman: How do you know it's time it. The time is right to make the investment so like super cheap here, cause you've been through it like you lose scaling your own company, but I imagine you work for somebody else, and you have to make those same decisions.
Ken Ahrens: How do you know time is right to invest or not invest on developing these skills and learning things. You know. I gotta tell you, Ravi, you you're literally hitting one of my one of my you know, items that that I feel passionate about is so when you're building a company, you've got to hire people, you can't do it on your own, and you have to find the right folks, and you have to have this balance between what are the most important criteria for this position? Right? Yeah, let's talk about a technical role. So you could have someone on one hand who has all of the technical skills. And they're fantastic. You have someone, on the other hand, who's incredibly enthusiastic, but they have no technical skills at all right, and the ideal person has has has this good mix of both.
Ken Ahrens: Because, you cannot train people, at least when you're at a startup. For example, you need a lot of time out of people sometimes. You gotta go above and beyond. Sometimes you gotta work on the weekend, sometimes the customers asking a question, you gotta jump right in. And so that enthusiasm and willingness to jump in is critical.
Ken Ahrens: On the other hand, if you just get enthusiastic people who don't know anything about technology, it won't. It's it's not helpful. So you look for people who want to learn the new thing? Okay? And they say, Oh, I picked this up and maybe they picked something up and it didn't work.
Ken Ahrens: And then maybe they tried some technology. And I remember working with a guy he tried every single Java add on thing, and he learned he learned groovy, and he ran J. Ruby and he ran tomcat, and he ran every type of app server, you know, and that kind of thing, but he always was able to pick up any new technology very quickly, because he was in this constant state. So I wouldn't worry as much about you know. Should I learn it or not learn it? It's just figure out the anchor ones you're gonna build around. We decided Kubernetes when we were starting the company. But we don't hire just on kubernetes skills, cause we believe we can train that. But we cannot train you know, enthusiasm. We can't train people who have some of these core engineering capabilities as well.
Ravi Lachhman: Yeah, that's that's really helpful, like like that applies to anything. So let's say that we'll make something up like Foo, Foo! Foo's a new technology. And and it's it's actually very true, like when you're building something for the first time or like it's there's no playbook. Right? And so, yeah, there's no playbook. Yeah, there's no, you know, like, Ibm red book or like, there's no manual how to do this. Now, there's like like when Kubernetes space is getting more mature. So you might get a O'reilly book, or Manning book or packet book, or whatever funny animal or persons on those books, you know, like, can you buy it like kubernetes in action? Or, you know, like platform engineering applied? What are those those cool books are called these days. But that's it, like W in A in A, you know, in a crypto type of space, or like a newer paradigm like, you know you. You look for people who are willing to for the challenge ironically, like shameless, soft plug for my team, like we're doing stuff for the first time we're for harness. So, like the W. My team is charged with doing things for the first time the harms and never tried before, and that's exactly the similar profile I hire for like are you like? Are you hungry or enthusiastic like? It takes a lot of gas, you know, like there's no book how to do this.
Ravi Lachhman: Cut, re, reevaluate the the good old saying, like, you know, measure twice cut once like when you're building like you need to cut. You know, you have to cut some point, you know, like measure measure halfway. Cut a little bit measure again, like, you know. That's that's the the I would say the that that th the it says the founders dilemma. But also it's like any sort of thing you're doing for the first or second time was like the builders. Dilemma. There we go. My my manager says that. So he's like we're in the builders dilemma right now or what we're. I think you're right, Ravi, so it depends on what you're trying to optimize for. So when you get to a really big scale of company. You have a really good understanding of the methodologies and systems that work. And you have a run book
Ken Ahrens: after the Unicorn, you know who's the realistic person. But when you're building stuff and this and you're building it for the first time.
Ken Ahrens: creativity come up with a new idea. Try something different. You know. By the way, the other thing II keyed off on what you just said. Ship it right? So it's like it's time to make a decision the minute we we can measure it forever. We're gonna measure it for a little bit and then cut it and let's ship, and then and then you can come back and go. Okay, that was really terrible. Why'd we do it so fast or you go? I'm glad. I'm glad we should have hit the button earlier.
Ravi Lachhman: Yeah, you get that feedback cause, like, even like like negative feedback is still feedback. You're validating. Did it fit? Did it not fit? Are people like
Ravi Lachhman: like even bad feedback? It's very valuable, like chairs that cause someone's taking the time to like. Think through it. They might bring a different lens, you know, like, I'm so like hyper like my team is so hyper focused on delivering stuff like, Hey, we're still worried about getting bad feedback. It's like, Look, we'll get it. We're impacting people. We people say these things right? We didn't not like lack of trying.
Ken Ahrens: You know. But we'll have to, you know, since there's no book in how to do it, like we did it for the first time. Yeah. And anyway, nobody nobody has this perfect track record, right? And of course, if you follow any kind of sports, analogies and stuff, no one just sits down, and, you know, has a perfect basketball game and scores every single time you miss. Right? That's why you gotta rebound. You gotta do other stuff, too. So but if you never take shots for sure you can't win. It's it's very simple stuff. But you it's easy. Get caught up in it. This thing's gotta be perfect. And so definitely.
Ravi Lachhman: I like one of those. You know that that advice from my combinatorship, your product. And you might. It's okay. If you're a little embarrassed, and if you're not embarrassed, then you waited too long.
Ravi Lachhman: My my biggest, if I were to ask myself like was one piece of advice, I get my own self. It's like when I was when I was like much more tunier, like in like a like a sorry, my engineering career. I used to get things right all the time, like the first time, like, now II know if I get it right, the first is wrong. I'm like there's no way like this works like like, if so, but understanding like, hey like when you do something for the first time is probably wrong, right? But like how
Ravi Lachhman: the you build for the vanilla path versus like I built when I engineer something. Now I built for failure. First, I don't bill for success first versus like, oh, it's always gonna work like no ever gonna type that in like. Why, super stable like nice. So just 1 one quick, quick change of gears.
Ravi Lachhman: what's what's still a challenge like, you know, like a lot of things. You you think everything will be solved by now a decade in. But what? Why don't you still seem this like, hey? You know what people are still regardless of, like their size or scale, like you'd be one person, 100 people like
Ken Ahrens: this. This is a very good question, Robby, because clearly the what's what's so we're on Kubernetes. I don't know which version. Now, one dot, 31 to 29, something like that. Ii could be off by the way. And now, your whole podcast is messed up. But so. And and there's 3 releases here. There's a lot of releases, a lot of functionality. It's not a technical problem. By the way. so a lot of the technical surface area is fairly well solved. There's a few things that are not ideal. You go. Another storage of my local data is kind of not the way I want it to be. But, by the way, it's all pluggable. So you you don't like it. Go put another vendor system in there, I think. Actually, where Kubernetes is right now, the biggest problem is ease of use. So you know you. You have to balance these 2 kind of users, the super power users who say I need another kind of secret. That's just like the secret, but a little different for my use case, I'm just, you know, picking on some piece of infrastructure secrets or something to complain about because they get stored in plain text. So it's not a great secret. So you have to customize it, anyway.
Ken Ahrens: And then someone on the other end of the spectrum. Who's like, I've got my engine X workload. How can I quickly get it into a cluster without being overwhelmed by all the technology? And this the approaches I've seen so far haven't worked that great. So
Ken Ahrens: things like running kubernetes in your desktop actually is kinda hard. And when Apple moved to the Armx which is a you know. Been a while now it's been 3 or 4 years. Well, that through everything, through a loop, because now all the containers have to be compiled for it. So so now, running Mini cube or some other local desktop is a little bit harder it's all solved. By the way, yeah, it's it's it's done and shipping.
Ken Ahrens: And then on the on the cloud infrastructure side Google launched an interesting product called Gke autopilot which will automatically configure they're working on this ease of use problem. But it's also hard to use so in a in a weird way, by giving you less things to configure and less features. They've actually taken away part of the power usage that you might want at your fingertips.
Ken Ahrens: So I'm hopeful that someone will be able to come up with some some easier to use approaches. And and then the the whole I have a pass is built on top of Kubernetes, and I will make a pass out of Kubernetes for you, I think, has the wrong shape as well that people need to know how to use the underlying infrastructure. It's just gotta get easier. You know. So
Ken Ahrens: yeah, that. And then, Yaml, because if you have one extra space in your yaml, Robby. Your your life is sad. Yeah, it's like, I'm not good at spaces. I'm a tab person because I can't count.
Ravi Lachhman: I remember. Oh, Lordy! Like it's like you have like, it's it's space separated. And I have to use like a winter to put like little periods. So I can like count like number of like. Oh, this one has 4 on the left. This one has the worst.
Ken Ahrens: Ravi. It's it's the worst. It's so hard. And then you're like, if you're ever working with someone who's not a Kubernetes expert. And you're just trying to help them with their cluster. And you're like, let's edit this file. And they're like, I'm using. Vi, right now, we're all gonna be okay. Put an extra space, make it like the line before they're like, what's happening. That could be easier. Yeah, yeah, it's it's it's
Ravi Lachhman: it's it's funny, like, I remember the first time I saw Kubernetes was it was actually before Kelsey. Hi Towers kubernetes, the hard way that was like at Red Hat. It's like a 2,014, and someone's like, Hey, like.
Ravi Lachhman: look at this Kubernetes thing like what that is, and one of the the more principal essays on my team like he installed it
Ravi Lachhman: on like a remote rel instance, and he had, like hundreds of steps like, we need this networking Ntp protocol enabled here they need to communicate here. Here's all the you know, the permissions and like, Oh, now I can just do like Mini cube up. That's very true. I mean, you're you're not. You're not wrong, Robby. The infrastructure side is in a tremendously better place
Ken Ahrens: and and I'll actually say I've I've tested as many Kubernetes distributions as I can get my hands on. So I've run them in aws. And of course, in Google cloud, in azure and a couple that I like actually, digital ocean, very simple. Cheap to run if you're on a low budget, so you can get a small cluster. And it's actually very simple to get up and running, you fill out at one form and hit enter, and you got it. And then there's also there's also a couple of specialty kubernetes, only you know distributions as well. So I think, on the infrastructure side, where you don't have to configure it. That part is good, but on the application side it's still, I think, has a little room for improvement to get just the easy button version out there, and Rancher was the closest to it until they got you know, acquired, and and there has not been as much development on on on rancher. But I think I think that's an area for for the next couple of years. It's not around. The functionality of functionality is fantastic. By the way, it has your compute, your storage, your network all tied in, and all the little details you want. Configurations in there. Secret management's in there. Customization is in there. So it's it has a really fantastic shape, and much better than if you were trying to do the same thing using regular cloud services. It's so there's a there's a time like about. I would just like to make it a little easier for everybody. Yeah, definitely, a lot of options like.
Ravi Lachhman: software engineer forcing the platform is during then I left after that. But yeah, I'll do these architectures type of stuff. Now. It's it exposed a lot of complexity very quickly. Because everything is like going back to our favorite llamo like you, you can configure, like all the layers that you need, you can configure your storage. You can configure your networking, you can configure how that like you know what the how the application even starts you can. You can discern certain steps in there. And it's like, not everybody has all those skills like my biggest outage of all time was networking, related in my career. I block half Internet from Prod, you could never look at it. Yeah, it was. It was bad, it was. I didn't know what a CIDR was, but now I do. That was like a drink. you know, like part Sider. Nope, they have math divided by
Ken Ahrens: 32 is a 16, or that is the opposite, divided by 16 instead. 32. It doesn't matter. Ravi isn't impossible to solve problem there said. There's no way to know how to configure a CIDR correctly. So for for what it's worth. The only thing that you can do is put it in and get 2 different opinions. And then if everyone agrees and ship it. Yeah, yeah, it's like, Yeah, it's like.
Ravi Lachhman: yeah, like, I'm like it pretty sure, like anybody who told me I did it wrong. And but yeah, it was. It was all about like we were the first app going to aws, and it was like, well, the networking team who know racks and stacks like F. 5, because it's not, you know, it's not on Prem, and it's always like that. This, it's a Kubernetes makes like all these expertises, or like even like, you know post Kubernetes systems. That post case, I would say, like it exposes a lot of like expertise like on a level playing field. So like, even though you and I might not know. Storage, you know, of the Csi like container storage interface, llamo like at least as human readable.
Ravi Lachhman: Oh, yes, this is, you know they're using a you know. Ext. 4 driver. I don't know what that means. I think that's a storage thing. Ext. 4. I think. You know, I think you're right, Ravi. It does make it pretty simple. You just say this workload needs some storage, and I can make a storage, and I can see it that it's there. It's it's it's
Ken Ahrens: yeah. That has a nice shape to it. Of course. You then run into the next. The secondary problem, which is storage doesn't run in every availability zone. Right? So it's all in one place, and I go. Oh, I actually, my cluster is multi availability zone. So those are some of the things to make it easier. That's kind of what actually part of what I'm referring to. We run stateful workloads, you know, database workloads in our clusters. And that's a common problem we have. So when when they have to get cycled, these databases are allowed to be cycled if they come up in the wrong availability zone, then it can't connect to its own storage. So we had to solve that problem. And it's like, why does this problem exist.
Ken Ahrens: you know. So that's an opportunity I think the Kubernetes folks can can come up with, and that there's probably, by the way, some vendor solution I'm not aware of. And if there is, please email, it's me. And I'll take a look at your stuff. I take a look at almost everybody's stuff. Cause? Yeah, it's always so odd with something Kubernetes fails because it's like, usually a can't place to work. It's like, very, very odd, like, the failures are like
Ravi Lachhman: like it does like the cluster didn't go away, you know. Like, if once it starts. So let's be like, I can't place this work. I can't place this work like oh, but because I needed like multi, you know the infrastructure it can't find like oh, like multi zone. You know, some sort of ser like storage based level, be able to say, Oh, because of this claim, I can place it on storage that has, like multi region like the. It's hard like getting to that level of detail.
Ken Ahrens: I mean, I would see like, Oh, I have to restart 50 times. I don't know why. That's that's true. I think in the application space is very safe. That's actually my favorite thing about Kubernetes is.
Ken Ahrens: you know, I'm a human being. I write my applications. I don't know how the things gonna run over a long period of time. My my laptop turns off at the end of the day but when it's running in here, if it has a memory leak it'll grow. We'll get it will crash, and it will restart automatically. I gotta go back in later and look in data, dog, and see that I have a memory problem, or it'll let me know that it had happened. But Kubernetes has already fixed it.
Ken Ahrens: And that part I really like, because now the the ferocity of this problem has been reduced. And I go. Okay, I have a memory of League. I need to go solve it. Ii will say, though, you got me thinking, Robby, about one of my stories, the cluster, the cluster isn't gonna go away we had a I was working on with a pilot customer right when we started scale. And we were working on some technology that basically, it's kind of like a service mesh. And in the way the service meshes work, they run a thing called IP tables
Ken Ahrens: and iptables, messes with the network. And so if you mess up your iptables, rules and you run it with host networking, turn on. Then you mess up with the host themselves. And we actually, we broke somebody's cluster completely. And they were like, it's okay. This is under infrastructure's code. They hit a button and made a new one. So it was. It was eye opening to me, though, because you get so used to. How resilient it all is that yeah, don't play around with iptables on host networking. So that's like it's infra like it's not like.
Ravi Lachhman: you know. It's a good old like OP like Openstack, like I, as versus PaaS like it needs somewhere to run. It's not infinite, you know, like there is like, you know, some sort of well, I'll like, most likely, like some sort of like Linux, the infrastructure somewhere that's running it, you know. You're having windows now, but like there's there's there's a piece under it that's running right? Like people got so easy to like. Give me more. Gke, click, click, click, click, click, click, click, click. But Google is in the hard work of like, Hey, you know, they configured their IP tables. I used to drop iptables as people would yell at me and screen like, oh, I don't know what ports I need all all them like
Ken Ahrens: that. That part. I think the the industry has done a good job catching up was security was a little knock on kubernetes for a while, and th there the clusters are now being built with a lot more security in mind. So things like the networking don't don't allow open access to everything to everybody, including the control plan and stuff. So that a lot of those defaults are a lot more sane. There's a lot better tools for seeing what's going on in your environment.
Ravi Lachhman: Nice? Yeah. Great, great talk. I have one more question for you to kind of kind of close this out.
Ravi Lachhman: And this is this is a good one, because, I always ask people what they would tell themselves. I imagine you know it was young Ken. You're walking out of McCormish Pavilion the day you graduated Georgia Tech.
Ravi Lachhman: and then you ran to yourself today like, so 2, 2 of you came into contact. If you came to contact me yourself the day you graduated from Tech any advice, you know. Don't go to jail. Don't go, don't get arrested. It could be anything. It could be any piece of advice. You would just giveany piece of advice.
Ken Ahrens: you actually mentioned it earlier. By the way, Robby, when things get too easy. Right? If if things are easy and you're in the software engineering work, and it always worked when you shipped it right? So shifting into the get into the little more uncomfortable spot where you're always there's a little bit of I'm not sure if this is gonna work. That's how you know you're pushing the envelope, and then don't give up.
Ken Ahrens: So it's I know it's 2 pieces of advice, but it's kind of the same thing. So because what happens is when it's easy. You're not going to give up. Okay. But you're also going to tread water. Okay, at least for me. II wanna advance and improve, and so I put myself in tough, uncomfortable positions.
Ken Ahrens: and then then that sometimes I feel like you know, that little voice in the back of my head. Hey, man, that's too hard. This is, you know, this was a bad idea, so don't give up on it, and and and keep at it, and and I think that has worked for me pretty well, and I've taken a bunch of risks in my in my career, and and I don't. I don't regret it. So yeah, that would be. That'd be my advice.
Ravi Lachhman: Perfect. Yeah. Well, thanks, Ken. And if people want to get in touch with you at speed scale, like, what's the best way? Website, Linkedin Twitter Github?
Ken Ahrens: Yeah, absolutely. I think I probably use Linkedin the most you can find me ken Aaron's on Linkedin, or look for speed scale on Linkedin, and drop me a note. And believe it or not, I've respond. I read all these things. I respond to people all the time. Startup founders that reach out to say, I got a question. I'll hop on the phone and talk through things we've learned any kind of way I can help.
Ken Ahrens: That's another thing is always try to help other people out and don't worry about. You know. Evening the scales you know. That's that's kind of not the point. If you're just helpful to people, Karma, the universe. You're gonna get your returns and and don't don't sit there and wait for it. Just just try to be a helpful person. So, anyway.
Ken Ahrens: perfect. Well, Ken, thank you so much for coming on. And yeah, thanks for your time this afternoon. Awesome. Thank you, Ravi.