{"id":5825,"date":"2023-09-19T23:06:13","date_gmt":"2023-09-19T13:06:13","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=5825"},"modified":"2024-01-01T12:16:15","modified_gmt":"2024-01-01T02:16:15","slug":"developer-survive-you-built-it-you-run-it","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/developer-survive-you-built-it-you-run-it\/","title":{"rendered":"How developers can survive &#8220;you build it, you run it&#8221;"},"content":{"rendered":"<h2 class=\"wp-block-heading\" id=\"introduction\"><strong>Introduction<\/strong><\/h2>\n\n\n<p>As a developer, you might not have anything to do with your code once it\u2019s been committed <em>all the way to<\/em> looking after the code right up to production.<\/p>\n\n\n\n<p>The latter is called the \u201cyou build it, you run it\u201d model.<\/p>\n\n\n\n<p>It\u2019s not going away.<\/p>\n\n\n\n<p>But that depends on your organization. <\/p>\n\n\n\n<p>It\u2019s likely to increase in popularity across organizations, as:<\/p>\n\n\n\n<ul>\n<li>software operations become more complex <em>and <\/em><\/li>\n\n\n\n<li>management calls for more accountability from developers<\/li>\n<\/ul>\n\n\n\n<p>You might even get asked to respond to incidents when your code causes an outage or performance issue.<\/p>\n\n\n\n<p>(Hopefully, <a href=\"https:\/\/www.srepath.com\/site-reliability-engineering-101\/\">there\u2019s an SRE around<\/a> to join you on the incident and guide you through it!)<\/p>\n\n\n\n<p>That&#8217;s why it&#8217;s important to create reliable software from the beginning.<\/p>\n\n\n\n<p>This way, you&#8217;ll have to respond to fewer and less severe incidents.<\/p>\n\n\n\n<p>We will explore some ways you can bake reliability into your feature development work.<\/p>\n\n\n\n<p>\ud83d\udc47\ud83c\udffc Let\u2019s start off with a developer responsibility that makes obvious sense and go from there\u2026<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"code-maintenance-beyond-the-original-commit\"><strong>Code maintenance beyond the original commit<\/strong><\/h2>\n\n\n<p>It\u2019s often gone without saying, that your responsibility for code doesn\u2019t end at its first commit. From time to time, you will have to update your code to:<\/p>\n\n\n\n<ol>\n<li>improve its efficiency including performance optimizations<\/li>\n\n\n\n<li>make it work with major changes in other parts of the software e.g. changes affecting how API will be consumed and<\/li>\n\n\n\n<li>fix bugs that have been found in testing or production<\/li>\n<\/ol>\n\n\n\n<p>Not doing this effectively will increase technical debt.<\/p>\n\n\n\n<p>And technical debt is the enemy of reliability.<\/p>\n\n\n\n<p>Other than continuously improving your code, you can integrate reliability-supporting concepts into your workflow.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"get-comfortable-with-reliabilitysupporting-concepts\"><strong>Get comfortable with reliability-supporting concepts<\/strong><\/h2>\n\n\n<p>Here\u2019s a hard truth: your infrastructure is flawed.<\/p>\n\n\n\n<p>And it always will be flawed.<\/p>\n\n\n\n<p>The best you can do is prepare your apps and services to deal with such flaws when they turn into outages and performance issues.<\/p>\n\n\n\n<p>Here are a few reliability concepts you can deep-dive and contextualize to your environment:<\/p>\n\n\n\n<ul>\n<li><strong>Observability<\/strong> through monitoring, tracing, and logging<\/li>\n\n\n\n<li>Instrument code with <strong>open telemetry<\/strong><\/li>\n\n\n\n<li>Add <strong>circuit breakers<\/strong> to the code<\/li>\n\n\n\n<li>Integrate regular <strong>code reviews<\/strong> into your workflow<\/li>\n\n\n\n<li>Prepare services for <strong>load-shedding<\/strong> behavior<\/li>\n\n\n\n<li>Add <strong>feature flags<\/strong> for nuanced operational control<\/li>\n\n\n\n<li>DIY <a href=\"https:\/\/www.srepath.com\/reduce-software-outage-risk-with-passive-guardrails\/\"><strong>passive guardrails<\/strong><\/a> to protect your code-in-production<\/li>\n\n\n\n<li>Design code for <strong>graceful degradations<\/strong> and <strong>failure modes<\/strong><\/li>\n\n\n\n<li>Implement <strong>test-driven design (TDD)<\/strong> into your planning<\/li>\n\n\n\n<li>Optimize code to work with <strong>distributed database<\/strong> architectures<\/li>\n<\/ul>\n\n\n\n<p>I recently <a href=\"http:\/\/software-reliability-not-sre-pablo-bouzada\">interviewed Pablo Bouzada<\/a> of ViaPlay, who as an engineering manager espouses the reliability-focused thinking and practices above.<\/p>\n\n\n\n<p>He\u2019s not an SRE manager nor has he ever held an SRE title.<\/p>\n\n\n\n<p>His team doesn\u2019t have a single Site Reliability Engineer.<\/p>\n\n\n\n<p>Yet he has put in time and effort to integrate reliability concepts into the work of his software engineering team.<\/p>\n\n\n\n<p>They\u2019ve reaped the rewards through highly reliable software.<\/p>\n\n\n\n<p>Their legwork early in the SDLC reduces their and their operations peers\u2019 future workload in terms of incident response and large refactoring projects.<\/p>\n\n\n\n<p><a href=\"http:\/\/software-reliability-not-sre-pablo-bouzada\">Check out the episode here<\/a>.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"incident-response-for-coderelated-issues\"><strong>Incident response for code-related issues<\/strong><\/h2>\n\n\n<p>Site Reliability Engineers (SREs) make the bulk of first responders and coordinators for incidents like outages, performance degradations, etc.<\/p>\n\n\n\n<p>2023\u2019s tech recession has seen a reduction in the number of such SREs working in many organizations.<\/p>\n\n\n\n<p>It\u2019s still anecdotal with little media coverage on specific layoff numbers for this role.<\/p>\n\n\n\n<p>But I\u2019ve had enough SRE managers tell me about having even more limited resources than before.<\/p>\n\n\n\n<p>What this means is there are fewer SREs going around. <\/p>\n\n\n\n<p>And they tend to be the main or sole on-call responders to incidents.<\/p>\n\n\n\n<p>Now there\u2019s a deeper need to share responsibility for such incident responses.<\/p>\n\n\n\n<p>It\u2019s now a fine balancing act of responding to incidents with limited SRE resources and the often overworked developers who are closest to the code.<\/p>\n\n\n\n<p>To make your life easier, it\u2019s best if you have a well-integrated suite of tools and practices that can give you greater visibility into issues, as well as ideas to solve the problem.<\/p>\n\n\n\n<p>Without naming names, there are tools that can help you:<\/p>\n\n\n\n<ul>\n<li>instrument telemetry automatically to your code<\/li>\n\n\n\n<li>setup incident response war rooms as incidents occur<\/li>\n\n\n\n<li>monitor resource usage patterns by services<\/li>\n\n\n\n<li>trace across services to find redlines for latency<\/li>\n\n\n\n<li>trace <em>within<\/em> specific functions of code for the same<\/li>\n\n\n\n<li>make sense of the big data from logs, traces, and metrics<\/li>\n\n\n\n<li>use AI for troubleshooting issues with infrastructure components<\/li>\n<\/ul>\n\n\n\n<p>What I am getting at is that there is a myriad of solutions to ensure you aren\u2019t wading around in the dark when you have to respond.<\/p>\n\n\n\n<p>Each can cover a very specific aspect of your incident response needs, to give you better and better coverage of even the most complex incident.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"wrapping-up\"><strong>Wrapping up<\/strong><\/h2>\n\n\n<p>This has been a high-level overview of your responsibilities as a developer for the reliability of software.<\/p>\n\n\n\n<p>Keep in mind that you may need to support an incident response at some point in your career, and might even need to do it regularly.<\/p>\n\n\n\n<p>To minimize the pain of doing this, I suggest you design your code with reliability concepts in mind and regularly maintain it to minimize technical debt.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction As a developer, you might not have anything to do with your code once it\u2019s been committed all the way to looking after the code right up to production. The latter is called the \u201cyou build it, you run it\u201d model. It\u2019s not going away. But that depends on your organization. It\u2019s likely to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60,5],"tags":[76,12],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5825"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=5825"}],"version-history":[{"count":5,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5825\/revisions"}],"predecessor-version":[{"id":6117,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5825\/revisions\/6117"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=5825"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=5825"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=5825"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}