{"id":5989,"date":"2023-11-14T17:31:49","date_gmt":"2023-11-14T07:31:49","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=5989"},"modified":"2024-01-01T12:13:45","modified_gmt":"2024-01-01T02:13:45","slug":"growing-sre-5-companies-nash-seshan","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/growing-sre-5-companies-nash-seshan\/","title":{"rendered":"#15 Growing Reliability Engineering Across 5+ Companies (with Nash Seshan)"},"content":{"rendered":"\n<p><strong><span class=\"stk-highlight\" style=\"color: var(--base-3);background-color: var(--contrast-3)\">Episode 15 [SREpath Podcast]<\/span><\/strong><\/p>\n\n\n<div class=\"gb-grid-wrapper gb-grid-wrapper-de03cb60\">\n<div class=\"gb-grid-column gb-grid-column-fe1c5056\"><div class=\"gb-container gb-container-fe1c5056\">\n\n<iframe loading=\"lazy\" src=\"https:\/\/podcasters.spotify.com\/pod\/show\/srepath\/embed\/episodes\/15-Growing-Reliability-Engineering-Across-5-Companies-with-Nash-Seshan-e2bj5ev\/a-aaiupim\" height=\"142px\" width=\"600px\" frameborder=\"0\" scrolling=\"no\"><\/iframe>\n\n<\/div><\/div>\n\n<div class=\"gb-grid-column gb-grid-column-79dc71d9\"><div class=\"gb-container gb-container-79dc71d9\"><\/div><\/div>\n<\/div>\n\n\n<p>Ash Patel talks with Nash Seshan, who has supported reliability work in over 5 organizations, including Cisco, eBay, Dropbox, Lyft, Netflix, and Wayfair. He shares his learnings from reliability work at these big brands.<\/p>\n\n\n\n<p>Nash also draws from his experience as co-founder of a Y Combinator-funded startup on effective engineering leadership. He also gives his take on issues with ill-conceived automation.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"episode-transcript\">Episode Transcript<\/h2>\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Great to have you on, Nash. <\/p>\n\n\n\n<p>I am excited to learn about what you&#8217;ve been doing in this space. You have been working in the software field since 2009, and you started off at Cisco, but then it&#8217;s like logos for days. I mean, I&#8217;m looking at eBay, Netflix, Dropbox, Lyft, and that&#8217;s just to start off. How are you doing?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;I&#8217;m good, and thanks for the kind words, Ash, and thanks for having me on.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I want to hear more about what you have to say about this history you&#8217;ve packed in. It&#8217;s still a relatively short period of time for you to get all of this experience in.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah. When I graduated college back in 2009, I graduated with like a major in network engineering.<\/p>\n\n\n\n<p>It was a subject that really fascinated me. And at the time, Cisco was one of these dream companies that I wanted to work at. Little did I know that that was going to be the first job I was going to take right out of college. So I got into Cisco had a really, really fun ride working there, learned a ton of stuff.<\/p>\n\n\n\n<p>And around the time, as I was leaving Cisco, I also kind of realized that the industry was making a push towards software engineering in generally cross all verticals. So I realized that I would be left behind if I didn&#8217;t catch up or if I didn&#8217;t really teach myself how to code.<\/p>\n\n\n\n<p>Towards the end of my time at Cisco, I taught myself how to write basic scripts in Python, in bash just to kind of automate tiny bits of work I was doing. And then I felt like I needed formal training in actually learning about software engineering and some of the core concepts around it.<\/p>\n\n\n\n<p>So I got myself a master&#8217;s degree in computer science. And ever since I have kind of been in the industry, networking has always and still will always be my passion. But I decided to step into the software engineering space where I started out as a network software developer at eBay automating deployments and writing tooling for the data center network that eBay and then as the industry evolved into infrastructure engineering and SRE, I was kind of already in that space without really having the title. So as I evolved through eBay, as you mentioned in your intro, I was already in SRE before SRE was public and out there.<\/p>\n\n\n\n<p>And then eventually the SRE terminology just latched itself onto me. I&#8217;m fortunate to work at some of these logos, even though it&#8217;s a short period of time. A ton of experience and a ton of learning along the way.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;We&#8217;re definitely going to deep dive into some of that experience in a moment, but right now you&#8217;re at Wayfair, which is an e commerce Company for furniture.<\/p>\n\n\n\n<p>Is that correct?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah furniture and home goods. They&#8217;ve been around for about 20 odd years now as a company, but it&#8217;s something so surprising to me when I joined I wasn&#8217;t aware that Wayfair had been around for that long. But yeah, they&#8217;re actually, if you want to think of it, the Amazon for home goods and furniture.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;And that&#8217;s an engineering manager role that you hold.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;So I lead the database reliability engineering team here at Wayfair. And my team is responsible for all the production databases that power the Wayfair platform.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;So you had to switch from a technical leadership track.<\/p>\n\n\n\n<p>You were a technical lead at Lyft until 2021, and then you had a switch over to people leadership around that time.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah. So at Lyft, I think I was part tech leader, part people leader as well. And then around 2021, I also did my own startup. So I had the experience of building from ground up, hiring a team, leading and managing them.<\/p>\n\n\n\n<p>Ever since my startup didn&#8217;t work out for me I kind of moved on to people leadership because that&#8217;s something I feel I have a lot to give. It&#8217;s, it&#8217;s a new field for me. It&#8217;s new. It&#8217;s a new challenge for me, but at the same time, I don&#8217;t feel it to be very new because it&#8217;s something that I&#8217;ve always been doing in some capacity, either being a technical leader or a mentor or some sort.<\/p>\n\n\n\n<p>People leadership, while in terms of number of years of experience is new to me fairly new to me it doesn&#8217;t feel new at all. It feels like something I&#8217;ve been doing for a long time.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;That&#8217;s interesting. What do you think of some of the things that are very similar between these two different tracks in tech work?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;I think it really depends on which particular track within the industry that you&#8217;re in. For example, in my case I was always an SRE or always a network engineer of sorts. And I always found ways to tinker around and experiment and see how I could improve certain parts of what I was doing.<\/p>\n\n\n\n<p>Which I think is very ingrained into my work ethic and generally very ingrained into what I do or how I operate. So when I got into people leadership, it was pretty much the same. Like, how do you work with people? How do you tinker around to see what makes them tick? What doesn&#8217;t make them tick?<\/p>\n\n\n\n<p>What motivates someone versus someone else on your team? And everyone is different. The only difference that I see between the two is that people leadership is really a lesson in psychology over and above having technical expertise, but if you&#8217;re a technical leader and you&#8217;re able to like teach other people on your team or coach other people on your team and also like help them grow people leadership is very much the same thing.<\/p>\n\n\n\n<p>It&#8217;s doing that in an official capacity as opposed to being a technical leader, which may or may not be an official capacity. The only nuance there between people leadership and technical leadership is that you have this human psychology aspect that you need to understand.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I wanted to get your two cents on that because people say people leadership, technical leadership, very different things.<\/p>\n\n\n\n<p>It&#8217;s like two forks in a road, but like you said, there&#8217;s still a lot of involvement from technical leaders in coaching people, helping them become a better engineer. And people leadership just adds that psychology element of how do we get people to be the best they can be, whereas a technical leader would focus on, Hey, this is the technology that we&#8217;re wanting to focus on.<\/p>\n\n\n\n<p>Let&#8217;s see how we can help you get there. It&#8217;s a nuance, but still a similarity between both those traditionally seen as very different tracks. Completely agree. We&#8217;ve spoken about your interesting SRE journey. And I said earlier that I wanted a deep dive into some of your earlier experience. You mentioned your experience with Cisco, where you were a network consulting engineer, and you learned that actually having skills to code up infrastructure and automation was going to be useful.<\/p>\n\n\n\n<p>And you were right. How did you find that going to an organization like Netflix? People love following Netflix story. And I bet it. People must reach out to you all the time and say, Hey, what was that like?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah. So Netflix was actually one of the most gratifying experiences of my career.<\/p>\n\n\n\n<p>It was pretty short lived for its while. I was only there for about a year, but while I was there it was probably the best company culture I&#8217;ve ever worked at. I say this with a huge caveat and a huge asterisk because the particular culture I&#8217;m about to refer to isn&#8217;t for everyone. It&#8217;s the same as for example, you might be able to work through a problem really fast, or you might be able to work through a problem really slowly, but are you able to work autonomously?<\/p>\n\n\n\n<p>Do you need help? Do you keep seeking for help? Or like, do you need to be told exactly what to do? Or are you kind of a self starter? Netflix is all about self starters. And every single project I worked on at Netflix was something where it was like, I was literally given a one line problem statement.<\/p>\n\n\n\n<p>There was no context around it. And the one life problem statement was intentional because what they wanted me and other people at the company to go figure out what the constraints would be, how to solve the problem and how to limit the scope of the problem.<\/p>\n\n\n\n<p>So inherently teaches you all these skills about being an owner, literally going and figuring out who the stakeholders to this problem are what is the current impact and what do you really need to solve? Because you&#8217;d get a business direction, but you wouldn&#8217;t really have a technical solution.<\/p>\n\n\n\n<p>The idea of like determining the business problem, the stakeholders, the impact of the problem, and then eventually what is the best solution and can you phase the solution out or do you need to like build everything on day one?<\/p>\n\n\n\n<p>All these things are skills you implicitly learn when you work at Netflix. This was a huge opportunity for me and I thrive in environments where I get ownership.<\/p>\n\n\n\n<p>The more ownership you throw at me, the better I perform and the better I am as an engineer. And that was the environment that just worked perfectly for me.<\/p>\n\n\n\n<p>It was like the dream job come true. But that said, it&#8217;s not for everyone because I know that a lot of people at Netflix find the level of autonomy that&#8217;s given to you very, very intimidating. It was great for me, but it may not be great for other people.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;So in summary, you have to be a self starter to be in a place like that.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Pretty much. Pretty much. Because the managers and the leadership there: they entrust and empower you with so much to do the best job you can. That, like, you&#8217;re responsible for every decision you make. Like, you are actually part of the decision making process. You own the decisions you make.<\/p>\n\n\n\n<p>So, if they&#8217;re good, we celebrate the wins. If they&#8217;re not so good, well, it&#8217;s a learning opportunity for everyone. We all take the hit. We all take the hit.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Which is a good culture to have. I know it&#8217;s a tough culture for people who want to be told, hey, this is a job, I want to do this. But that tough culture is also a bit more of a fair culture than a place where they pretend to be here&#8217;s a job Take it, but then there&#8217;s a lot of politics that happens and that happens in a lot of places.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;That&#8217;s true I think with Netflix, I think it&#8217;s so at least my experience when I was there I&#8217;m not sure how the company&#8217;s changed and evolved as they&#8217;ve grown since I was there but During my time, this was very much the culture. They were a very small, tight little team of about 2, 000 employees worldwide at the time and they intentionally wanted to keep the size of the company small because they wanted teams to be small, tight knit, and with enough ownership on whatever they were doing to a point where, like, they would feel comfortable making those decisions within their purview.<\/p>\n\n\n\n<p>I&#8217;m pretty sure that that&#8217;s changed, as the company has also changed, but my experience was great.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;So the next place you were at after Netflix was at Dropbox and you had a title of Network Reliability Engineer. I&#8217;m deep diving into all of this because I haven&#8217;t met many people who have had so many logos. Thank you. And i&#8217;m not saying this any way that is a bad thing i&#8217;m saying it&#8217;s a good thing and this is you&#8217;ve had a variety of experience and that&#8217;s why I&#8217;m so curious what you&#8217;ve learned at each place and what each place has given you and helped you build into your current abilities. So the next place you were at was Dropbox and you had the title of Network Reliability Engineer. Right. So kind of a subset of SRE or specialized version?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;It was very much SRE for networking. This is kind of the first role I took on where the SRE title kind of came somewhat close to what I was doing.<\/p>\n\n\n\n<p>I think Dropbox wanted to have a different spin on it because the company was structured in a way where every application team or every yeah, every software team or every application team had a sister SRE team that was partnering with them. In my case, we had a network engineering team that was responsible for our data centers.<\/p>\n\n\n\n<p>And they essentially needed an SRE team, which would kind of write automation for them. Hence the birth of the network reliability engineering team, which was focused on observability, monitoring, writing, automation, writing, tooling and helping keep the network and all aspects about the network reliable enough.<\/p>\n\n\n\n<p>That experience was also very gratifying and very fruitful because I got to work at a very large scale company that was dealing in block storage and also had experience working with a hybrid cloud environment where we were partly in a public cloud, partly on premise. There were some workloads that were running in the cloud, some workloads that were running in our own private data centers.<\/p>\n\n\n\n<p>So how do you then structure for that kind of environment? Netflix was also a hybrid cloud environment, but it was a very different setup because Netflix is a content delivery network. That is a whole different paradigm of network engineering compared to like traditional data center networking.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;You probably needed that because of the complexity of the data that Dropbox in particular was dealing with.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah, because Dropbox deals with all its customers private data, right? So, I think all of Dropbox&#8217;s users entrust them with their private files. Now, these could be audio files, these could be video files, it could be media, it could be confidential documents. It could be stuff that people want to store away for archival purposes, for later use.<\/p>\n\n\n\n<p>You&#8217;re entrusting the service with a lot of your private data. One of the good things about Dropbox was that privacy and security was a top priority across the company. One of the things, and I know I&#8217;m digressing a little bit, but it&#8217;s one of the companies where I&#8217;ve seen security and privacy implemented so well that every other place I&#8217;ve been at, I&#8217;ve always compared it back to Dropbox.<\/p>\n\n\n\n<p>I&#8217;m like, that was great. And this is not quite as great. It&#8217;s worth a call out.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I definitely think that&#8217;s worth a call out because DevSecOps is now, I would say it&#8217;s considered the norm. A lot of places are still trying to figure out AppSec and now we&#8217;re talking DevSecOps. Yeah, things are getting complicated.<\/p>\n\n\n\n<p>So with your background, actually knowing how important security is and actually the practices that go along with that and integrating it with DevOps and infrastructure, that&#8217;s something that I think we need to teach people a bit more about this. I think we&#8217;re going to do an aside on this. We need to do something and teach people about this. For sure.<\/p>\n\n\n\n<p>So now we&#8217;re going on to your experience at Lyft. I love Lyft. It&#8217;s sometimes 10 percent to 15 percent cheaper in Toronto, and that helps a lot, especially if you&#8217;re going a long way.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah, it does. So at Lyft, I was within a vertical, which was kind of the R&amp; D division of Lyft. It wasn&#8217;t the traditional ride hailing service.<\/p>\n\n\n\n<p>I worked within the autonomous vehicle division of Lyft, which was called Level 5 at the time. The mission for that particular division was to bring robo taxis to the market. So we had a fleet of vehicles that we were running autonomous driving software on. It was about testing and doing a lot of the load testing and the stress testing of that code on the actual roads in Palo Alto to determine if the code is actually getting better, if the safety driver is engaging with the car as much as they need to, or if the code is able to operate pretty autonomously. That&#8217;s what the entire division was about. My job particularly on that entire team was to build infrastructure automation and observability. And kind of ensure that our pipelines and our infrastructure shipping code to the car and gathering data from the car after a trip.<\/p>\n\n\n\n<p>were reliable enough because shipping code to the car had been a big problem. When I joined the company, a lot of folks were actually shipping code on flash drives because that&#8217;s what they would build a code on, on their workstations, and they were walking down to the garage and plugging it in and essentially uploading the code.<\/p>\n\n\n\n<p>So we built some infrastructure to kind of make that process better. And then as the car was on the road you wanted to be able to, like, gather telemetry from the car. So build some intelligent solutions around that too. It was a fun project.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I have so many questions, just stemming from that, because I&#8217;m one of those nerds really interested in self driving technology and it&#8217;s kind of gone quiet recently.<\/p>\n\n\n\n<p>One thing I will ask you, and this is more of a computing related question to that. I was always thinking there&#8217;s a lot of need for edge computing capability if you&#8217;re gonna do self driving at scale, I know they&#8217;re dealing with other issues right now, but let&#8217;s just say they figured all those issues out with the current skill set that most companies have, would we be capable of doing something that utilizes it?<\/p>\n\n\n\n<p>Edge computing and all those types of scenarios to bring it out at scale?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;So that&#8217;s a great question and I think I would answer that in a slightly diplomatic way and I think the reason I&#8217;m going to do that is because my perception about self driving is the fact that you need to be able to have enough control on the actual hardware that&#8217;s on the car.<\/p>\n\n\n\n<p>And enough control on the software that you&#8217;re running on the car as well. And I feel like very few companies actually have control of the entire stack Tesla for one of them is probably the pioneer in this space because they control the car hardware, they control the software. They&#8217;re actually also working with chip makers to build custom chips that run on the actual car.<\/p>\n\n\n\n<p>When it comes to other companies, like say the Ubers, the Lyfts and the Cruz of the world, the car is leased out to them by a car manufacturer. The software is their own and the chips are probably whatever is available in the market. So when you don&#8217;t really have a lot of control on the entire stack, you can only go so far.<\/p>\n\n\n\n<p>I&#8217;m not saying that you have to be able to build your own chips or you build your own hardware, but I&#8217;m saying if you control what you need from the hardware and from the software and from the actual chips that run on the car, you&#8217;ll have more chances to kind of bring this to a reality. I feel at this point, Tesla is probably the best position in the market at this point to make that a reality.<\/p>\n\n\n\n<p>There&#8217;s videos on YouTube for folks doing the FSD trials and posting their experiences with full self driving and Tesla&#8217;s kind of proved that they&#8217;re capable of doing what is called level four autonomy, which is essentially Autonomously. You let the car do its thing right from the start to the end, but it&#8217;s on a predictable route and it&#8217;s not in unfavorable conditions like snow or hail or whatever.<\/p>\n\n\n\n<p>It&#8217;s a bright sunny day temperatures are okay your battery&#8217;s fully charged. It&#8217;s not an unforeseen circumstance. Level 5 is like, it takes, it takes into account any and all unforeseen circumstances. Will we ever get to true L5 or Level 5 autonomy? Hard to say. Maybe many years down the road. But I think level four autonomy is what we can hope for, at least to begin with.<\/p>\n\n\n\n<p>And I feel like Tesla and maybe now Polestar or some of the other companies that are doing this are probably in the best position to make that a reality for our customers.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I&#8217;m personally betting on Lucid but you know,<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah, Lucid is another one. Yeah.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;As in not to win the self driving L4 thing, but you know, I like their cars, but lucid,<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;it&#8217;s a really good car.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;So you&#8217;ve had a variety of experience in so many, not so adjacent organizations because Netflix does something very different to Dropbox, does something very different to Lyft. And then you had a YC, you co founded a YC startup.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;It was something that a friend of mine and I did. We were university roommates. He was in the biotechnology and life sciences space. And we wanted to build a platform that was going to reduce the amount of time it would take to bring newer drugs to market. We had identified an opportunity in the space where typical drug development takes anywhere between 10 to 15 years before it comes to market for consumers like you and I.<\/p>\n\n\n\n<p>And we identified along that pipeline after talking to a bunch of users and a bunch of people we knew in the industry that there was enough opportunity to automate a lot of this workflows. A lot of the things that scientists typically do with drug development is data analytics, data collection reporting of data, running a bunch of experiments, maintaining inventory.<\/p>\n\n\n\n<p>There&#8217;s a lot of these auxiliary and primary workflows, they&#8217;re just screaming for automation, just screaming for something to come and make their life better. That was the opportunity we wanted to tap into and build a platform that would start with low hanging fruit and then build on top of that.<\/p>\n\n\n\n<p>We had some success in terms of the idea and we got accepted into Y Combinator, had a few customers who were willing to pay for us. The only thing that we kind of struggled with, and I think we knew this getting into the entire idea of the company, is that a lot of the demographic of scientists who are currently doing cutting edge research in that space 55 and above.<\/p>\n\n\n\n<p>So they&#8217;ve kind of been doing research for like 20, 30 years now without the aid of any tech. It&#8217;s just been that way. All of a sudden we have ourselves coming along and kind of saying, Hey, you now have this platform that&#8217;s going to automate X percent of your work. All you need to do is plug it into your systems and give us your data.<\/p>\n\n\n\n<p>And that&#8217;s a level of trust that they were refusing to give us. Well, not, I wouldn&#8217;t say refusing to give us, but it was much harder to break through that barrier. That was the first problem. The second problem was the fact that once we started getting some customers to actually trust us and, you know, give us their data and use our platform, we noticed very quickly that their standardization of the way they did things or the standardization of data they collected was very, very different.<\/p>\n\n\n\n<p>There was literally no standardization. So some of the scientists collected data on paper napkins, some of them collected on notebooks, some of them used Google sheets, some of them used other mechanisms. Some of them took pictures of whatever they wrote.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Wait, you&#8217;re not kidding, right? They literally used, like, napkins and they took pictures of data?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah, they did. Yeah. Well, it was like they were running an experiment in the lab and they had nothing else to kind of note down data on, so they would use whatever they would get. At times it would be a paper napkin, at times it would be a picture of a paper napkin or another piece of paper. But that&#8217;s kind of what we had to work with, right?<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;As a quick aside, my university training was actually as a research scientist in biomedical science, so I&#8217;m shocked. A lot of people don&#8217;t know this, but I also have a qualification in biomedical science. I worked with PhD, like, friends who became PhDs and professors and all that, but I am shocked to hear this.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Yeah, and it&#8217;s funny, because, like, my wife is a biotechnologist herself, so, like, when we were running into these problems, I would, like chat with her and say, Hey, is this something that&#8217;s abnormal to what we&#8217;re seeing or is this kind of the norm that you&#8217;re seeing in the industry based on your experience?<\/p>\n\n\n\n<p>And she was like, yeah, I mean, we&#8217;ve done this ourselves. Like we haven&#8217;t had tooling or proper mechanisms to do it. And every scientist who comes along does it their own way. And then we also took this back and mapped it to an article we read when we were founding the company. This article was from the very renowned Nature journal, which kind of said that 70 percent of all biotech and life science research is practically not reproducible by another human being. And that just made sense because the data is not standardized. So if you publish a paper today somewhere and I&#8217;m supposed to go ahead and reproduce that research, there&#8217;s a 70 percent chance I will not be able to do it or get the same results that you did.<\/p>\n\n\n\n<p>Because my collection mechanisms or my data collection mechanisms or my analytics will be different compared to yours. That was a problem we were rushing to get a solution to. Again, but then we ran into the first barrier of like having an age demographic of scientists who are like north of 55. It was challenges on multiple fronts but it was an enriching experience.<\/p>\n\n\n\n<p>I mean, I think founding a company is no small feat and getting it to customers who want to pay for it is kind of the feather in the cap that I would take for, I mean, for life.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I have two questions related to that and people who are wondering, wait, I thought I was listening to a conversation about SRE.<\/p>\n\n\n\n<p>What&#8217;s this talk about biomed and scientists taking data? No, no, no. This actually has relevance because the question I&#8217;m going to ask you, Nash, is what parallels do you see with the industry as a whole in terms of people doing their own way in biomedical science, finding their own way to do things, and then having that and you&#8217;re trying to standardize it and the challenges you faced?<\/p>\n\n\n\n<p>Are you finding any parallels to the challenges or any, are you finding that DevOps and SRE people are actually pretty good in comparison? I personally think they would be compared to that. Yeah. But do you see any anti patterns? And that&#8217;s one of my favorite things to ask.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;I do actually. There&#8217;s quite a few anti patterns that I&#8217;ve noticed.<\/p>\n\n\n\n<p>I&#8217;ll start by answering your first question about similarities between my founder experience and what we see in the SRE world. I think the lack of standardization is a lack of runbooks and playbooks in the SRE world, which teach someone how to troubleshoot or diagnose a problem in a certain way.<\/p>\n\n\n\n<p>I think a lot of seasoned SRE folks, again, I&#8217;m not trying to like stereotype or throw a blanket statement here, but I&#8217;ve noticed that a lot of folks who are much more seasoned, they have a lot of this in their head. And when a problem arises or when an issue arises, they&#8217;re the first ones to go and troubleshoot and navigate through the problem.<\/p>\n\n\n\n<p>But they have it in their head and it&#8217;s not often documented. Which means junior engineers or folks who are trying to learn the space are going to struggle. To determine how to do it. To learn from what works or what doesn&#8217;t work. In terms of SRE, I guess standardization. That&#8217;s one of the areas that I see lack of standardization in our industry.<\/p>\n\n\n\n<p>The other thing that I also see is a lack of, well, this is getting better for sure, so I&#8217;m not going to say this in a pessimistic tone, but lack of standard monitoring practices. So when someone is, let&#8217;s say, for example running a piece of infrastructure, or like, you know, running compute on Kubernetes, or containerized system, or whatever, people monitor things in different ways.<\/p>\n\n\n\n<p>They choose to monitor what they think is important. But if we were to have some industry wide guidelines or like RFCs or whatever to kind of say, Hey, if you were to look for these few things, or if you were to monitor these few metrics, you would get these kinds of results and you should always ensure that you get these monitoring bits enabled, whatever you&#8217;re doing.<\/p>\n\n\n\n<p>I know it&#8217;s not a very strong case to push for because people can feel free to choose whatever they want to, but that is another area where standardization would certainly help. Going back to your anti patterns question, I feel like one of the biggest anti patterns in the SRE world is that folks try to automate everything, as opposed to understanding what needs automation.<\/p>\n\n\n\n<p>Trying to automate everything or trying to observe everything using monitoring is possible, theoretically, not practically. I guess the thing that is most important to understand is whatever you&#8217;re trying to automate, do you fully understand the root cause of that? Are you automating the root cause or are you automating the symptom?<\/p>\n\n\n\n<p>Sometimes the root cause is so very deep within that it might be a cascading effect, and you might be just seeing the symptom on the surface of things. So you might write some automation to fix the symptom, but are you truly understanding the root cause? I think this is something that lesser seasoned or lesser experienced SREs tend to do a lot more.<\/p>\n\n\n\n<p>&#8217;cause like, oh, we&#8217;re running into this issue. This alert is firing 10 times a week or 10 times a day. Let&#8217;s go and write some automation to fix that alert. But is that alert really the problem? Or is it something much deeper that is causing the problem? That&#8217;s one of the biggest anti patterns I&#8217;ve seen, which I hope people work to get better on.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Going back about standardization, there&#8217;s a high risk that if organizations take on a standardization drive, you&#8217;re gonna get people being too rigid. So I propose to organizations, Let your engineers experiment, and then once they have worked out a model that is effective and continues to give effective results, that&#8217;s when you say, this is how we do it.<\/p>\n\n\n\n<p>Yep. Not going in as engineering management and saying, this is how we&#8217;re going to do it moving forward.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Completely. Completely agree. I think the engineers on the team have to definitely have a good understanding of what the environment is, have to be able to try things, have to be able to, like, like you said, build a model and what works, what doesn&#8217;t work, and then kind of iterate on that, right?<\/p>\n\n\n\n<p>I think one of the key facets of SRE or in general, these days, software development of any sort, is incremental progress, right? Gone are the days where we&#8217;re all in the waterfall model, where you end up spending months and months together writing a piece of software. And then put it out there in terms of a release, and then go back and iterate on it again to fix whatever issues or bugs.<\/p>\n\n\n\n<p>People want MVPs, people want like really, really small scoped products to ship as quick as possible. Because they want to know and get instant feedback on what&#8217;s working and what&#8217;s not working. SRE is exactly the same, right? So if you identify a problem, can you fix it immediately or can you mitigate it immediately?<\/p>\n\n\n\n<p>And then work on a longer term fix that might require more time to implement. I think it&#8217;s really down to that at the end of the day.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Yeah, I recently spoke with a director of SRE and he was saying, there are teams who want an immediate fix. They want you to fix their problem yesterday.<\/p>\n\n\n\n<p>Yeah. I Want, I want the fix for what happened like now. And as an SRE, as a manager who leads SREs, you have to be ready for that kind of conversation. Yep. That&#8217;s when you say to your SREs, look, you have to be continuously looking at things. You need to be looking at the data that all these teams are working on and don&#8217;t stretch yourself out too much.<\/p>\n\n\n\n<p>But, just keep your eye, keep your finger on the pulse of what&#8217;s happening with all these teams, completely. That&#8217;s when, yeah, that&#8217;s when you&#8217;re responsive, that&#8217;s when you increase your responsiveness and that&#8217;s when teams start loving those SRE guys around the corner.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Exactly. I think one of the things I always tell my teams is, I mean, this is usually for incidents that happen off hours, like off business hours or whatever. Focus on mitigating the problem. Let&#8217;s find something to stop the bleeding. You don&#8217;t have to work on building a fix during the incident.<\/p>\n\n\n\n<p>You don&#8217;t have to stay up till 3 a. m. in the morning working on a fix. Can we stop the bleeding? If that means scaling up a machine to increase, like, throwing money at the problem. Do it until we get back to the business hours the next day. Let&#8217;s focus on the fix during business hours, which allows people to think more about what could be a longer term fix.<\/p>\n\n\n\n<p>If a good SRE is on call and an issue comes up, they would be able to stop the bleeding. But then typically I know a lot of people who are very good SREs who the problem kind of eats them up through the night. They always try to see what can they do? How can they resolve it? They&#8217;ll probably go ahead and go down a rabbit hole into trying and troubleshooting what might be the root cause.<\/p>\n\n\n\n<p>Sometimes they&#8217;ll fight it right away, sometimes they won&#8217;t, but at least they&#8217;ll come up the next morning with enough investigation done to know where we can start. And sometimes a fix is easy, sometimes it&#8217;s not. This empowers me and empowers them to kind of have conversations with upper leadership saying, And whoever is asking for a fix like yesterday, it allows me to have this conversation saying, Hey, we stopped the bleeding.<\/p>\n\n\n\n<p>There&#8217;s no impact to the business. We&#8217;ll work on a longer term fix, but this is what we&#8217;re thinking. And here&#8217;s the data to prove it. I think having data powered conversations is always useful as opposed to saying, We need a fix. Let&#8217;s do whatever it takes. Let&#8217;s drop everything we&#8217;re doing and focus on a fix.<\/p>\n\n\n\n<p>That&#8217;s not always the best idea at all times.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;There were two reasons why I wanted to talk about your startup experience. The first was what we just talked about. The second reason was I want to know your experience as a founder. Did it contribute to your ability to become a more effective engineering manager?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;For sure. I think my experience as a founder reinforced my ability to develop. I think when you&#8217;re a startup, your things change every day, right? It was, it was kind of like walking into a war zone where priorities change every day. Customer asks change every day. We were a pre product market fit startup.<\/p>\n\n\n\n<p>So we&#8217;re always on the hunt to try to achieve product market fit or find that one silver lining that would just allow us to explode in that space. And because of that, our constant struggle or our constant battle was to find that, find that needle in the haystack.<\/p>\n\n\n\n<p>It&#8217;s very similar to how SREs operate.<\/p>\n\n\n\n<p>When you&#8217;re looking for improvements in your system, you&#8217;re always looking for that one needle in the haystack or a few needles in the haystack that will allow you to make significant improvements to performance or to capacity usage or to network bandwidth or whatever it is. As an engineering leader, what it allowed me to do is it allowed me to ask a bunch of questions to my team saying, okay, do we really need this now?<\/p>\n\n\n\n<p>Is this the most minimally scoped version of what we&#8217;re trying to do? Can we cut the scope further? How many features do we need to build or how much code do we need to write? Before we can put this out there. So it allowed me to ask those kinds of questions better. It allowed me to get them to think about these things better.<\/p>\n\n\n\n<p>And then allowed me to justify business requirements directly to whatever projects we were working on as a team. People on my team always understand why we&#8217;re doing something rather than just the what, and then they figuring out the how they also know the why now. Because I&#8217;m able to tie that and be the liaison between their work and the business to be able to map that context back to them.<\/p>\n\n\n\n<p>So it, it helped me become a better leader to a point where I was always offering context on why we&#8217;re doing something. The why may not always have been something that folks resonated with or agreed with, but I think at the end of the day, you have to align with business priorities. The business comes first.<\/p>\n\n\n\n<p>So if a business decision is incorrect or someone feels it&#8217;s incorrect, that is a discussion to be had on a different day in a separate channel. But understanding why you&#8217;re doing something is always important.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;I&#8217;m sure you use that a lot in your current role at Wayfair when you&#8217;re managing a team that&#8217;s handling complex data infrastructure.<\/p>\n\n\n\n<p>And I don&#8217;t have much exposure to e commerce like I had an e commerce company 14 years ago. But&#8230; That was an offshoot of what we&#8217;re working on for a long time. I am curious about how SRE fits into an e commerce organization. So can you give me a breakdown of that?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Absolutely. I think an e commerce company on the face of it is basically a marketplace where you have sellers trying to put their products online, buyers trying to buy from them.<\/p>\n\n\n\n<p>But underneath the hood, it&#8217;s essentially a very complex piece of infrastructure that is powering this entire platform.. You have a lot of services that are running, which require heavy compute usage. You have a lot of databases that are powering different parts of the platform, which are very critical to the entire platform.<\/p>\n\n\n\n<p>You have a lot of networking that happens either on premise or in the public cloud, depending on how you&#8217;re set up. And there&#8217;s a lot of storage that&#8217;s required to store catalog information or to store a bunch of other pieces of data that are very critical to the business. When you have so much complexity in the infrastructure itself, granted that it&#8217;s not a machine learning type workload where you have tons and tons of GPUs that are operating in 100 percent of efficiency or 100 percent CPU usage or GPU usage.<\/p>\n\n\n\n<p>It&#8217;s not at that scale or that level of intensity, but at the same time, the scale is very different because Wayfair as a product or as a platform generates a lot of revenue, which means there&#8217;s a lot of orders, a lot of transactions, folks are buying stuff, folks are returning stuff. It&#8217;s a typical e commerce workflow.<\/p>\n\n\n\n<p>So when you have these pieces of infrastructure, you obviously need to keep them reliable. Any critical database that goes down or any piece of critical infrastructure that goes down impacts the business directly, which means lower revenue on that given day. Which means lower order volume on that given day.<\/p>\n\n\n\n<p>So these are critical hits to the business which need a reliability engineering organization to make sure that things are up at all times, reliably, scalably, and highly available. And that&#8217;s where my team specifically comes in because we essentially are responsible for all the production databases that power Wayfair as a platform.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;And you have a lot of revenue that you&#8217;re trying to protect here because&#8230; Exactly. 12 billion dollars, that&#8217;s staggering for an e commerce company that sells for&#8230; It&#8217;s a new world. It&#8217;s a new world.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;It is. It is a new world. I think a lot of people are shopping online. And I think that&#8217;s where the market is moving.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;With this varied experience that you&#8217;ve had in so many different categories of organizations, have you experienced any organizational challenges in bringing in effective SRE, DevOps, infrastructure engineering practices?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;That&#8217;s a great question, Ash. I think one of the things I&#8217;ve noticed is specifically at companies that are trying to establish an SRE org, not for companies that have already established or have been having an SRE org for a while now, but specifically companies that are trying to establish one, they tend to retrofit engineers with different skill sets into an SRE org because they want to bootstrap the org and they want to get people who have already been around at the company, had a lot of experience with the internals of the company and understand how the systems work, at least to some degree, to become an SRE and, you know, help kickstart.<\/p>\n\n\n\n<p>While it&#8217;s not a bad idea, the problem that I see is SRE is actually a mindset and not a skill set. You can essentially learn the skills and the tools used by SREs like Kubernetes or containers or networking or storage or compute whatever. But SRE is really a mindset where it&#8217;s about being able to tinker, being fearless to experiment, being fearless to break stuff and try and see what resiliency is in your system.<\/p>\n\n\n\n<p>I think a lot of folks who are not from that mindset or don&#8217;t have that mindset find it very hard to become really good SREs. If you&#8217;re someone who is already tinkering or like you&#8217;ve built stuff or you&#8217;re like you know broken stuff or like you know you&#8217;ve torn stuff apart when you were a kid or along the way.<\/p>\n\n\n\n<p>You&#8217;re likely going to have that mindset coming in. For me, this mindset really came to me when I did my first job at Cisco, which was in technical support. I was taught or trained to troubleshoot a problem methodically, but also figure out what might work and what might not work to fix the customer&#8217;s problem.<\/p>\n\n\n\n<p>But a lot of folks don&#8217;t have that kind of experience. A lot of folks don&#8217;t have that mentality to try stuff out. And especially when you&#8217;re on call and you&#8217;re in a high pressure environment. You need that level of composure to be able to try stuff and see what works and what doesn&#8217;t work. Retrofitting ops engineers or developers or other kinds of skillsets into an SRE org sure will bring the technical abilities out, like folks will know the technical stuff, but would they really be able to be calmer under pressure?<\/p>\n\n\n\n<p>Would they have longevity in their SRE career? I think that&#8217;s something I feel is a huge organizational challenge. I wish organizational leaders understood that more and tried to balance the team out better with seasoned SREs and folks who have internal intrinsic knowledge and then let them grow into that role over time, but bootstrapping an SRE org in a company with purely folks who have been around without a lot of tinkering experience.<\/p>\n\n\n\n<p>Thanks. I don&#8217;t know if that&#8217;s the right way to go.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;There are a lot of sysadmins who are very disgruntled right now saying Hey, my title just got changed to SRE. What is that all about? It&#8217;s only been a title change. And some of them are saying, I just want to roll up VMs. I just want to do that.<\/p>\n\n\n\n<p>And now they&#8217;re getting told No, no, no, no. You&#8217;re going to do this thing that Google wrote about in its book in 2016.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Pretty much. And it is, it is kind of a mindset change. It&#8217;s like I said, it&#8217;s not, it&#8217;s not a skill. It&#8217;s a mindset. SRE is really a mindset. It&#8217;s not a skill.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;That&#8217;s a great piece of advice to give to SRE managers who are also trying to hire good people.<\/p>\n\n\n\n<p>Is there any other piece of advice that you&#8217;d give to them for developing an effective SRE team?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;I think to managers and leaders trying to build a team, I would say empower them to feel comfortable to go and break stuff, feel empowered to go and try stuff, feel empowered to experiment, and feel empowered to make your teams feel that they can go and do stuff without their job hanging over their heads.<\/p>\n\n\n\n<p>Especially in this market, I think a lot of folks are very cautious about doing anything that might even remotely cost them their job, but I think good SRE leaders would be able to suck that up. And not let it percolate through to upper leadership at the same time give their engineers space to become better SREs over time.<\/p>\n\n\n\n<p>So that&#8217;s what I try to do. That&#8217;s what I&#8217;ve always done. That&#8217;s the only piece of advice I would give to other SRE managers out there.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;There&#8217;s another area that I&#8217;m getting a lot more interested in. I think a lot of people will get value from you giving advice in this. Individual contributors keep reaching out to me and saying, Hey, my career progression, it doesn&#8217;t make sense in this organization.<\/p>\n\n\n\n<p>Essentially, what I&#8217;m saying to them is you have to be a business of one. You have to develop your own self to be an effective individual contributor for the market, not just for the situation you&#8217;re in. And I&#8217;m guessing managers are going to hate me for saying this because I&#8217;m saying, Hey, you know, they&#8217;re&#8230;<\/p>\n\n\n\n<p>You know, they&#8217;re going to move on eventually, right? And they should be developing themselves for what the market dictates. Do you have any advice for those individual contributors who are feeling stuck in one company&#8217;s way of doing things? And I&#8217;m saying to them, expand out the knowledge. Do you have any advice for them?<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;I think I would completely agree with what you&#8217;re saying in the sense that you always have to be curious. I think curiosity is the only way that will allow you to bridge the gap between where you are as an individual and what is being asked of you in the industry. So if you&#8217;re looking for a career transition after having spent a good number of years at a company, try to see what&#8217;s out there.<\/p>\n\n\n\n<p>How has the industry involved in your particular domain, if you&#8217;re intending to stay in that domain, between when you started and now? And what is the delta between your current skill set and what is actually being asked of engineers? So for example, one of the things is that everyone&#8217;s jumping on this AI bandwagon.<\/p>\n\n\n\n<p>Every company wants people to have prompt engineers now. It&#8217;s a good thing, but at the same time, there&#8217;s going to come a time, maybe in the next few years, where prompt engineering is going to be so overloaded with folks learning about prompt engineering that there&#8217;s not going to be enough jobs for prompt engineers.<\/p>\n\n\n\n<p>So, identify areas that you can truly deliver value. How can you build on your current skill set? Or how can you learn something new and expand out wide? I think a lot of companies are looking for T shaped engineers and not I shaped engineers anymore. So, if you&#8217;re an expert at technology X or Y, how can you learn about A, B, and C and increase your breadth of expanse as well?<\/p>\n\n\n\n<p>You&#8217;re a subject matter expert in technology Y probably, but how can you learn about other things and show that you can do X, Y, and Z too? Not just Y anymore. I think that&#8217;s going to increase the market value for yourself a lot.<\/p>\n\n\n\n<p><strong>Ash Patel:<\/strong>&nbsp;Nash. I love what you just said, because you&#8217;re touching on all the topics I write about all the time. I&#8217;m loving it. Thank you so much for joining me and giving your insights about your experience, which is, I think, is so useful for people in both the management side and individual contributor side to learn about. Thank you so much.<\/p>\n\n\n\n<p><strong>Nash Seshan:<\/strong>&nbsp;Thanks for having me. Ash. This was a great conversation.<\/p>\n\n\n\n<p>I&#8217;m so glad we could do it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Episode 15 [SREpath Podcast] Ash Patel talks with Nash Seshan, who has supported reliability work in over 5 organizations, including Cisco, eBay, Dropbox, Lyft, Netflix, and Wayfair. He shares his learnings from reliability work at these big brands. Nash also draws from his experience as co-founder of a Y Combinator-funded startup on effective engineering leadership. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[53,55],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5989"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=5989"}],"version-history":[{"count":4,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5989\/revisions"}],"predecessor-version":[{"id":6053,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5989\/revisions\/6053"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=5989"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=5989"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=5989"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}