{"id":383,"date":"2021-12-14T12:56:00","date_gmt":"2021-12-14T02:56:00","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=383"},"modified":"2023-12-13T15:28:02","modified_gmt":"2023-12-13T05:28:02","slug":"runbooks-for-better-incident-response","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/runbooks-for-better-incident-response\/","title":{"rendered":"Runbooks for better incident response"},"content":{"rendered":"<h2 class=\"gb-headline gb-headline-6f636deb gb-headline-text\" id=\"introduction\">Introduction<\/h2>\n\n\n<p>I can confidently tell you that runbooks form a critical part of the incident response toolkit. I will also tell you that SREs are well-placed to start and oversee the development of runbooks.<\/p>\n\n\n\n<p>If you don&#8217;t have a runbook yet, let me entice you with the thought of checklist-type documentation to follow when you&#8217;re woken up to deal with a 3am production meltdown. <\/p>\n\n\n\n<p>You won&#8217;t be the only one using the runbook. Its simplicity allows you to more easily product teams into the incident response action. It gives clarity to those who may not be as experienced as you when investigating faults with their work-in-production.<\/p>\n\n\n\n<p>Runbooks are most useful when you are finding your incident response to be a case of &#8220;putting out the same fires over and over again&#8221;. It removes unnecessary thinking from incident response and helps you focus on the task at hand.  <\/p>\n\n\n\n<p>Or at least carry out the work without an overwhelmed&nbsp;\ud83e\udd2f&nbsp;feeling.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-29923964 gb-headline-text\" id=\"why-runbooks-are-useful-in-sre-incident-response\">Why runbooks are useful in SRE incident response<\/h2>\n\n\n<p>Here are 3 reasons why runbooks are superior to &#8220;I&#8217;ll figure it out as it comes&#8221; as a strategy:<\/p>\n\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-643eca4\" data-block-id=\"643eca4\"><style>.stk-643eca4 ul li,.stk-643eca4 ol li{font-size:20px !important}.stk-643eca4 li{padding-inline-start:6px !important;margin-bottom:12px !important}.stk-643eca4 ol{list-style-type:decimal !important}@media screen and (max-width:1023px){.stk-643eca4 ul li,.stk-643eca4 ol li{font-size:20px !important}}<\/style><ol><li>Automated processes don\u2019t always protect against <em>all<\/em> possible issues \u2014 so software operations needs 10s to 100s of different activities actioned by skilled humans to keep the system rolling<\/li><li>\u201c30-40% of procedures require human judgment to resolve safely, so that\u2019s still a bunch of run books that won\u2019t go away \u2013 even if large parts of deployment are push-button \/ automated processes.\u201d<\/li><li><strong>Prevents annoying experiences like this:&nbsp;<\/strong>\u201cI recently ran into a situation where I spent 6 hours understanding how something works that would have taken 20 minutes if the relevant information was stored somewhere.\u201d<\/li><\/ol><\/div>\n\n\n<h2 class=\"gb-headline gb-headline-1ad84e85 gb-headline-text\" id=\"ways-that-teams-have-set-up-their-runbooks\">Ways that teams have set up their runbooks<\/h2>\n\n\n<p class=\"has-medium-font-size\"><strong>Confluence<\/strong>&nbsp;\u2014 is not particularly designed for managing runbooks but is an open-ended tool that enables you if you have a solid enough idea of how to effectively design a runbook<\/p>\n\n\n\n<p class=\"has-medium-font-size\"><strong>Jupyter Notebooks<\/strong>&nbsp;\u2013 an open-source tool with a combo of text, image, and live code snippets so decent option if you are happy to install and maintain it<\/p>\n\n\n\n<p class=\"has-medium-font-size\"><strong>Markdown files hosted in git repo<\/strong>&nbsp;\u2014 maintenance might be an issue over time without strict guidelines within the team<\/p>\n\n\n\n<p class=\"has-medium-font-size\"><strong>Err\u2026 this \u279d&nbsp;<\/strong>\u201cSticky notes on someone\u2019s desk. We\u2019re thinking about getting a laminator to keep the coffee spills from being too serious of a problem.\u201d \ud83d\ude05<\/p>\n\n\n\n\n<h2 class=\"gb-headline gb-headline-69876e63 gb-headline-text\" id=\"factors-to-consider-when-developing-your-own-runbook\">Factors to consider when developing your own runbook<\/h2>\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-4f8fa05\" data-block-id=\"4f8fa05\"><style>.stk-4f8fa05 ul li,.stk-4f8fa05 ol li{font-size:20px !important}.stk-4f8fa05{margin-bottom:50px !important}.stk-4f8fa05 li{padding-inline-start:6px !important;margin-bottom:12px !important}.stk-4f8fa05 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZm9jdXNhYmxlPSJmYWxzZSIgZGF0YS1wcmVmaXg9ImZhciIgZGF0YS1pY29uPSJjaGVjay1zcXVhcmUiIGNsYXNzPSJzdmctaW5saW5lLS1mYSBmYS1jaGVjay1zcXVhcmUgZmEtdy0xNCIgcm9sZT0iaW1nIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0NDggNTEyIiBzdHlsZT0iZmlsbDogIzAwMCAhaW1wb3J0YW50OyBjb2xvcjogIzAwMCAhaW1wb3J0YW50O3RyYW5zZm9ybTogcm90YXRlKGRlZykgIWltcG9ydGFudDsiPjxwYXRoIGZpbGw9IiMwMDAiIGQ9Ik00MDAgMzJINDhDMjEuNDkgMzIgMCA1My40OSAwIDgwdjM1MmMwIDI2LjUxIDIxLjQ5IDQ4IDQ4IDQ4aDM1MmMyNi41MSAwIDQ4LTIxLjQ5IDQ4LTQ4VjgwYzAtMjYuNTEtMjEuNDktNDgtNDgtNDh6bTAgNDAwSDQ4VjgwaDM1MnYzNTJ6bS0zNS44NjQtMjQxLjcyNEwxOTEuNTQ3IDM2MS40OGMtNC43MDUgNC42NjctMTIuMzAzIDQuNjM3LTE2Ljk3LS4wNjhsLTkwLjc4MS05MS41MTZjLTQuNjY3LTQuNzA1LTQuNjM3LTEyLjMwMy4wNjktMTYuOTcxbDIyLjcxOS0yMi41MzZjNC43MDUtNC42NjcgMTIuMzAzLTQuNjM3IDE2Ljk3LjA2OWw1OS43OTIgNjAuMjc3IDE0MS4zNTItMTQwLjIxNmM0LjcwNS00LjY2NyAxMi4zMDMtNC42MzcgMTYuOTcuMDY4bDIyLjUzNiAyMi43MThjNC42NjcgNC43MDYgNC42MzcgMTIuMzA0LS4wNjggMTYuOTcxeiIgc3Ryb2tlPSIjMDAwIiBzdHlsZT0iZmlsbDogcmdiKDAsIDAsIDApOyBzdHJva2U6IHJnYigwLCAwLCAwKTsiLz48L3N2Zz4=') !important}.stk-4f8fa05 li::marker{font-size:2em !important}@media screen and (max-width:1023px){.stk-4f8fa05 ul li,.stk-4f8fa05 ol li{font-size:20px !important}}<\/style><ul><li>Make a standard runbook template \u2014 makes it easier to process information when in a pinch like when resolving an urgent incident<\/li><li>Have a collaborative approach to building the runbooks \u2014 don\u2019t palm the task off to technical writers \u2013 the people who design and build the systems should be the main authors or at least actively participate in the process<\/li><li>Give an explanation to the runbook user of why the component of the system was designed as it appears &#8211; ambiguity around intentions is a key reason for failing to come up with creative solutions to a tricky problem<\/li><li>Some runbooks have sub-processes \u2013 it\u2019s important to clarify what these are and how they relate to the process that they are the children of<\/li><\/ul><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Introduction I can confidently tell you that runbooks form a critical part of the incident response toolkit. I will also tell you that SREs are well-placed to start and oversee the development of runbooks. If you don&#8217;t have a runbook yet, let me entice you with the thought of checklist-type documentation to follow when you&#8217;re [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60,32],"tags":[12],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/383"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=383"}],"version-history":[{"count":10,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/383\/revisions"}],"predecessor-version":[{"id":5773,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/383\/revisions\/5773"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=383"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=383"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=383"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}