{"id":356,"date":"2022-02-09T00:19:00","date_gmt":"2022-02-08T14:19:00","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=356"},"modified":"2024-01-01T12:28:02","modified_gmt":"2024-01-01T02:28:02","slug":"rundown-of-netflixs-sre-practice","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/rundown-of-netflixs-sre-practice\/","title":{"rendered":"Rundown of Netflix&#8217;s SRE practice"},"content":{"rendered":"<h2 class=\"gb-headline gb-headline-7cdbac51 gb-headline-text\" id=\"introduction\">Introduction<\/h2>\n\n\n<p>A lot goes on in the background every time you load up your favorite Netflix movie or series.<\/p>\n\n\n\n<p>Engineers spread across Chaos Engineering, Performance Engineering and Site Reliability Engineering (SRE) are working non-stop to ensure the magic keeps happening.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-ace55941 gb-headline-text\" id=\"%25f0%259f%2593%258a-here-are-some-performance-statistics-for-netflix\">\ud83d\udcca Here are some performance statistics for Netflix<\/h2>\n\n\n<p>When it was alone on top of the streaming world in 2016&#8230;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-performance-stats-2016-1-1024x314.png\" alt=\"Netflix microservices performance statistics for 2016 \" class=\"wp-image-363\" width=\"512\" height=\"157\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-performance-stats-2016-1-1024x314.png 1024w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-performance-stats-2016-1-300x92.png 300w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-performance-stats-2016-1-768x236.png 768w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-performance-stats-2016-1.png 1039w\" sizes=\"(max-width: 512px) 100vw, 512px\" \/><\/figure>\n\n\n\n<div class=\"wp-block-stackable-spacer stk-block-spacer stk--no-padding stk-block stk-ac3d3ff\" data-block-id=\"ac3d3ff\"><style>.stk-ac3d3ff{height:24px !important}<\/style><\/div>\n\n\n\n<p>For context, the average HD video connection is 8mbps, so 30+ terabits\/second means there were <strong>3,750,000+ connections at high-definition video bitrate at any given time<\/strong>.<\/p>\n\n\n\n<p>In 2022, this number must be significantly higher, but I don&#8217;t have access to their current numbers.<\/p>\n\n\n\n<p>SREs played an essential role in making sure all of this performance ticked over smoothly.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-24d6bd82 gb-headline-text\" id=\"%25f0%259f%2594%25b1-how-sre-fits-into-the-netflix-culture\">\ud83d\udd31 How SRE fits into the Netflix culture<\/h2>\n\n<h3 class=\"gb-headline gb-headline-ae2c3782 gb-headline-text\" id=\"team-formation\">Team formation<\/h3>\n\n\n<p>The SRE team at Netflix is known as CORE (Cloud Operations Reliability Engineering). It belongs to a larger group known as&nbsp;<em>Operations Engineering<\/em>.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-ops-engg-group-layout.png\" alt=\"\" class=\"wp-image-370\" width=\"335\" height=\"218\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-ops-engg-group-layout.png 447w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/2022\/04\/netflix-ops-engg-group-layout-300x195.png 300w\" sizes=\"(max-width: 335px) 100vw, 335px\" \/><\/figure>\n\n\n\n<div class=\"wp-block-stackable-spacer stk-block-spacer stk--no-padding stk-block stk-2dfe7f7\" data-block-id=\"2dfe7f7\"><style>.stk-2dfe7f7{height:24px !important}<\/style><\/div>\n\n\n\n<p>SREs work alongside specialist roles interrelated with SRE work, such as Performance Engineers and Chaos Engineers.<\/p>\n\n\n<h3 class=\"gb-headline gb-headline-504b2434 gb-headline-text\" id=\"what-is-netflixs-sre-culture-like\">What is Netflix&#8217;s SRE culture like?<\/h3>\n\n\n<p>The culture at Netflix is <strong>freedom and responsibility<\/strong> \u2014 both are important to effective SRE work.<\/p>\n\n\n\n<p>CEO Reed Hasting&#8217;s <a href=\"https:\/\/review.firstround.com\/radical-candor-the-surprising-secret-to-being-a-good-boss\">radical candor approach<\/a> (now popularised by Kim Scott) has had an impact on this contradicting desire for freedom and responsibility.<\/p>\n\n\n\n<p>The premise of radical candor is<strong> &#8220;be critical because you care about the other person<\/strong>&#8220;. This may make it easier for SREs to call out poor production decisions.<\/p>\n\n\n\n<p>This would make it easier for SREs to tell their counterpart engineers when they are not following a more suitable path for solving a problem. Without looking like a jerk.<\/p>\n\n\n\n<p>Developers must follow the &#8220;you build it, you run it&#8221; model. <strong>SREs act as consultants for developers<\/strong> in supporting them achieve the &#8220;you run it&#8221; part of the equation.<\/p>\n\n\n\n<p>Of course, SREs will also act as the last line of defense when issues affect production.<\/p>\n\n\n\n<p>For example, a testing service goes down, which will affect the ability to push code to production. SREs may join to resolve this issue by following incident response protocols.<\/p>\n\n\n\n<p>Most of the time, Netflix&#8217;s <strong>SREs work on solving problems that don&#8217;t have a straightforward fix<\/strong>. In such instances, RTFM may not work and a willingness to experiment and seek novel solutions may help.<\/p>\n\n\n\n<p>Fixes can take minutes, hours, days, weeks, or months \u2014 there is no fixed time to solve \u2014 and can be larger projects that other teams don\u2019t have time for.<\/p>\n\n\n\n<p>There is a lot of reading source code &amp; documentation, sourcing experiment ideas, running experiments, and then measuring the outcomes<\/p>\n\n\n\n<p>It can be done in solo missions or as a temporary problem-specific team.<\/p>\n\n\n<h2 class=\"gb-headline gb-headline-6b938223 gb-headline-text\" id=\"%25f0%259f%25a7%25b0-how-netflix-sres-support-production-tooling\">\ud83e\uddf0 How Netflix SREs support production tooling<\/h2>\n\n<h3 class=\"gb-headline gb-headline-7cb34ab1 gb-headline-text\" id=\"tooling-ethos\">Tooling ethos<\/h3>\n\n\n<p>Operations engineers at Netflix have spent years developing &#8220;paved paths&#8221;. These paths are designed to help developers<strong> leverage advanced tooling without reinventing the wheel<\/strong>.<\/p>\n\n\n\n<p>Examples of what paths cover include: service discovery, application RPC calls and circuit breakers.<\/p>\n\n\n\n<p>These paved paths are not prescriptive or enforced. Developers are allowed &#8211; even <strong>empowered &#8211; to deviate if they want to create a better path<\/strong> for their service.<\/p>\n\n\n\n<p>SREs are of course there to help developers work out a better path&#8217;s design. A good idea to use their services because<strong> path deviators are still subject to attacks by the Simian army<\/strong>.<\/p>\n\n\n\n<p>Simian Army is Netflix&#8217;s suite of chaos engineering tools that test out a functional system for its resilience when it&#8217;s attacked by an internal source.<\/p>\n\n\n\n<p>It&#8217;s all useful as Netflix practices extreme DevOps \u2014&nbsp;<em>you build it, you run it<\/em>&nbsp;\u2014 engineers do the full job of developing software, deploying pipeline and running code in production.<\/p>\n\n\n\n<p>SREs codify best practices from past deployments to make sure production is optimal.<\/p>\n\n\n\n\n<h3 class=\"gb-headline gb-headline-73713571 gb-headline-text\" id=\"tooling-examples\">Tooling examples<\/h3>\n\n\n<p>Netflix is best known to SRE world for its Chaos Monkey tool in Chaos Engineering. <strong><em>But wait, there&#8217;s more!<\/em><\/strong><\/p>\n\n\n\n<p>Netflix&#8217;s SREs also work extensively with the following tools:<\/p>\n\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-adee1c0\" data-block-id=\"adee1c0\"><style>.stk-adee1c0 ul li,.stk-adee1c0 ol li{font-size:20px !important}.stk-adee1c0 li{padding-inline-start:6px !important;margin-bottom:12px !important}.stk-adee1c0 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZm9jdXNhYmxlPSJmYWxzZSIgZGF0YS1wcmVmaXg9ImZhcyIgZGF0YS1pY29uPSJ0b29scyIgY2xhc3M9InN2Zy1pbmxpbmUtLWZhIGZhLXRvb2xzIGZhLXctMTYiIHJvbGU9ImltZyIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Qm94PSIwIDAgNTEyIDUxMiIgc3R5bGU9ImZpbGw6ICNlNTA5MTQgIWltcG9ydGFudDsgY29sb3I6ICNlNTA5MTQgIWltcG9ydGFudDt0cmFuc2Zvcm06IHJvdGF0ZShkZWcpICFpbXBvcnRhbnQ7Ij48cGF0aCBmaWxsPSIjZTUwOTE0IiBkPSJNNTAxLjEgMzk1LjdMMzg0IDI3OC42Yy0yMy4xLTIzLjEtNTcuNi0yNy42LTg1LjQtMTMuOUwxOTIgMTU4LjFWOTZMNjQgMCAwIDY0bDk2IDEyOGg2Mi4xbDEwNi42IDEwNi42Yy0xMy42IDI3LjgtOS4yIDYyLjMgMTMuOSA4NS40bDExNy4xIDExNy4xYzE0LjYgMTQuNiAzOC4yIDE0LjYgNTIuNyAwbDUyLjctNTIuN2MxNC41LTE0LjYgMTQuNS0zOC4yIDAtNTIuN3pNMzMxLjcgMjI1YzI4LjMgMCA1NC45IDExIDc0LjkgMzFsMTkuNCAxOS40YzE1LjgtNi45IDMwLjgtMTYuNSA0My44LTI5LjUgMzcuMS0zNy4xIDQ5LjctODkuMyAzNy45LTEzNi43LTIuMi05LTEzLjUtMTIuMS0yMC4xLTUuNWwtNzQuNCA3NC40LTY3LjktMTEuM0wzMzQgOTguOWw3NC40LTc0LjRjNi42LTYuNiAzLjQtMTcuOS01LjctMjAuMi00Ny40LTExLjctOTkuNi45LTEzNi42IDM3LjktMjguNSAyOC41LTQxLjkgNjYuMS00MS4yIDEwMy42bDgyLjEgODIuMWM4LjEtMS45IDE2LjUtMi45IDI0LjctMi45em0tMTAzLjkgODJsLTU2LjctNTYuN0wxOC43IDQwMi44Yy0yNSAyNS0yNSA2NS41IDAgOTAuNXM2NS41IDI1IDkwLjUgMGwxMjMuNi0xMjMuNmMtNy42LTE5LjktOS45LTQxLjYtNS02Mi43ek02NCA0NzJjLTEzLjIgMC0yNC0xMC44LTI0LTI0IDAtMTMuMyAxMC43LTI0IDI0LTI0czI0IDEwLjcgMjQgMjRjMCAxMy4yLTEwLjcgMjQtMjQgMjR6IiBzdHJva2U9IiNlNTA5MTQiIHN0eWxlPSJmaWxsOiByZ2IoMjI5LCA5LCAyMCk7IHN0cm9rZTogcmdiKDIyOSwgOSwgMjApOyIvPjwvc3ZnPg==') !important}.stk-adee1c0 li::marker{color:#e50914 !important;font-size:2em !important}@media screen and (max-width:1023px){.stk-adee1c0 ul li,.stk-adee1c0 ol li{font-size:20px !important}}<\/style><ul><li>Canary tools for developers to check code and make sure there is no performance regression<\/li><li>Dashboards to review service performance like upstream error rates, alerts for supporting services<\/li><li>Distributed system tracing to trace performance across the microservices ecosystem<\/li><li>Chat rooms and pagers and ticket systems for the fun engineer-level support work<\/li><li>Actionable alerts \u2014 check the right things, go off when appropriate, quiet when not<\/li><li>Spinnaker \u2014 allows for blue-green deployment with multi-cloud setup (insanely powerful)<\/li><li>Pre-production checklist that scores each aspect of service before going into production<\/li><\/ul><\/div>\n\n\n\n<p>Below is an example of an <strong>SRE codified tool, a pre-production checklist<\/strong> \u2014&nbsp;<em>Is your service production ready?<\/em><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"425\" height=\"225\" src=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/netflix-production-readiness-checklist.png\" alt=\"\" class=\"wp-image-4935\" srcset=\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/netflix-production-readiness-checklist.png 425w, https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/netflix-production-readiness-checklist-300x159.png 300w\" sizes=\"(max-width: 425px) 100vw, 425px\" \/><figcaption class=\"wp-element-caption\"><em>Source: Jonah Horowitz, SRECon 2016<\/em><\/figcaption><\/figure>\n\n\n<h2 class=\"gb-headline gb-headline-9f1fc9de gb-headline-text\" id=\"netflix-sre-capability-highlights\">Netflix SRE capability highlights<\/h2>\n\n<h3 class=\"gb-headline gb-headline-766c7934 gb-headline-text\" id=\"%25f0%259f%2594%25a5-incident-response\">\ud83d\udd25 Incident Response<\/h3>\n\n\n<p>Netflix engineering&#8217;s <strong>#1 business metric is SPS<\/strong>&nbsp;&#8211; Starts Per Second \u2014 the number of people successfully hitting the play button.<\/p>\n\n\n\n<p>Incident response practices are designed to ensure the highest possible % for this SPS metric.<\/p>\n\n\n\n<p>Here are some of the practices that Netflix has affirmed in its incident response capability:<\/p>\n\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-49e91a1\" data-block-id=\"49e91a1\"><style>.stk-49e91a1 ul li,.stk-49e91a1 ol li{font-size:20px !important}.stk-49e91a1 li{padding-inline-start:6px !important;margin-bottom:12px !important}.stk-49e91a1 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxOTAgMTkwIiBzdHlsZT0iZmlsbDogIzAwMCAhaW1wb3J0YW50OyBjb2xvcjogIzAwMCAhaW1wb3J0YW50O3RyYW5zZm9ybTogcm90YXRlKGRlZykgIWltcG9ydGFudDsiPjxwb2x5Z29uIHBvaW50cz0iMTczLjgsMjguNCA2MC40LDE0MS44IDE1LjcsOTcuMiA1LjEsMTA3LjggNjAuNCwxNjMgMTg0LjQsMzkgMTczLjgsMjguNCIgZmlsbD0iIzAwMCIgc3Ryb2tlPSIjMDAwIiBzdHlsZT0iZmlsbDogcmdiKDAsIDAsIDApOyBzdHJva2U6IHJnYigwLCAwLCAwKTsiLz48L3N2Zz4=') !important}.stk-49e91a1 li::marker{font-size:2em !important}@media screen and (max-width:1023px){.stk-49e91a1 ul li,.stk-49e91a1 ol li{font-size:20px !important}}<\/style><ul><li>Get the right people into the room and make sure they can troubleshoot the incident<\/li><li>Document everything during the incident to help with post-mortem analysis<\/li><li>Post-mortems aren&#8217;t necessarily &#8220;blameless&#8221; \u2014 something went wrong because someone did something, but rather than punish them, make them own it as a learning process<\/li><li>Short and to-the-point checklists for handling emergencies are codified in readily accessible manuals<\/li><li>Developers can assign metrics for their services to be addressed by SRE once certain thresholds are hit<\/li><\/ul><\/div>\n\n\n<h3 class=\"gb-headline gb-headline-72a3aa3d gb-headline-text\" id=\"%25f0%259f%258f%258e%25ef%25b8%258f-support-performance-engineers\">\ud83c\udfce\ufe0f Support performance engineers<\/h3>\n\n\n<p>For Netflix operations, it&#8217;s not only about uptime but also about having the right level of performance for solid playback.<\/p>\n\n\n\n<p>There is a need for consistently good service performance rather than one-off wins \u2014 users should have <strong>acceptably low TTI and TTR<\/strong>.<\/p>\n\n\n\n<p>Here&#8217;s what these two terms mean:<\/p>\n\n\n\n<ul style=\"font-size:clamp(15.747px, 0.984rem + ((1vw - 3.2px) * 0.878), 24px);\">\n<li><strong>TTI (time-to-interactive)<\/strong> &#8211; user can interact with app contents even if not everything is fully loaded or rendered<\/li>\n\n\n\n<li class=\"has-medium-font-size\"><strong>TTR<\/strong> <strong>(time-to-render)<\/strong> &#8211; everything above the fold is rendered<\/li>\n<\/ul>\n\n\n\n<p>SREs support performance engineers with activities like:<\/p>\n\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-9b43058\" data-block-id=\"9b43058\"><style>.stk-9b43058 ul li,.stk-9b43058 ol li{font-size:20px !important}.stk-9b43058 li{padding-inline-start:6px !important;margin-bottom:12px !important}.stk-9b43058 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZm9jdXNhYmxlPSJmYWxzZSIgZGF0YS1wcmVmaXg9ImZhcyIgZGF0YS1pY29uPSJ0YWNob21ldGVyLWFsdCIgY2xhc3M9InN2Zy1pbmxpbmUtLWZhIGZhLXRhY2hvbWV0ZXItYWx0IGZhLXctMTgiIHJvbGU9ImltZyIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Qm94PSIwIDAgNTc2IDUxMiIgc3R5bGU9ImZpbGw6ICMwMDAgIWltcG9ydGFudDsgY29sb3I6ICMwMDAgIWltcG9ydGFudDt0cmFuc2Zvcm06IHJvdGF0ZShkZWcpICFpbXBvcnRhbnQ7Ij48cGF0aCBmaWxsPSIjMDAwIiBkPSJNMjg4IDMyQzEyOC45NCAzMiAwIDE2MC45NCAwIDMyMGMwIDUyLjggMTQuMjUgMTAyLjI2IDM5LjA2IDE0NC44IDUuNjEgOS42MiAxNi4zIDE1LjIgMjcuNDQgMTUuMmg0NDNjMTEuMTQgMCAyMS44My01LjU4IDI3LjQ0LTE1LjJDNTYxLjc1IDQyMi4yNiA1NzYgMzcyLjggNTc2IDMyMGMwLTE1OS4wNi0xMjguOTQtMjg4LTI4OC0yODh6bTAgNjRjMTQuNzEgMCAyNi41OCAxMC4xMyAzMC4zMiAyMy42NS0xLjExIDIuMjYtMi42NCA0LjIzLTMuNDUgNi42N2wtOS4yMiAyNy42N2MtNS4xMyAzLjQ5LTEwLjk3IDYuMDEtMTcuNjQgNi4wMS0xNy42NyAwLTMyLTE0LjMzLTMyLTMyUzI3MC4zMyA5NiAyODggOTZ6TTk2IDM4NGMtMTcuNjcgMC0zMi0xNC4zMy0zMi0zMnMxNC4zMy0zMiAzMi0zMiAzMiAxNC4zMyAzMiAzMi0xNC4zMyAzMi0zMiAzMnptNDgtMTYwYy0xNy42NyAwLTMyLTE0LjMzLTMyLTMyczE0LjMzLTMyIDMyLTMyIDMyIDE0LjMzIDMyIDMyLTE0LjMzIDMyLTMyIDMyem0yNDYuNzctNzIuNDFsLTYxLjMzIDE4NEMzNDMuMTMgMzQ3LjMzIDM1MiAzNjQuNTQgMzUyIDM4NGMwIDExLjcyLTMuMzggMjIuNTUtOC44OCAzMkgyMzIuODhjLTUuNS05LjQ1LTguODgtMjAuMjgtOC44OC0zMiAwLTMzLjk0IDI2LjUtNjEuNDMgNTkuOS02My41OWw2MS4zNC0xODQuMDFjNC4xNy0xMi41NiAxNy43My0xOS40NSAzMC4zNi0xNS4xNyAxMi41NyA0LjE5IDE5LjM1IDE3Ljc5IDE1LjE3IDMwLjM2em0xNC42NiA1Ny4ybDE1LjUyLTQ2LjU1YzMuNDctMS4yOSA3LjEzLTIuMjMgMTEuMDUtMi4yMyAxNy42NyAwIDMyIDE0LjMzIDMyIDMycy0xNC4zMyAzMi0zMiAzMmMtMTEuMzgtLjAxLTIwLjg5LTYuMjgtMjYuNTctMTUuMjJ6TTQ4MCAzODRjLTE3LjY3IDAtMzItMTQuMzMtMzItMzJzMTQuMzMtMzIgMzItMzIgMzIgMTQuMzMgMzIgMzItMTQuMzMgMzItMzIgMzJ6IiBzdHJva2U9IiMwMDAiIHN0eWxlPSJmaWxsOiByZ2IoMCwgMCwgMCk7IHN0cm9rZTogcmdiKDAsIDAsIDApOyIvPjwvc3ZnPg==') !important}.stk-9b43058 li::marker{font-size:2em !important}@media screen and (max-width:1023px){.stk-9b43058 ul li,.stk-9b43058 ol li{font-size:20px !important}}<\/style><ul><li>autoscaling for on-demand scaling \u2014 saving money vs pre-purchased on-prem computer \u2014 for encoding, precompute, failover and blue-green deployments<\/li><li>handle tricky issues involving autoscaling like under-provisioning of resources, but also sticky traffic, bursty traffic and uneven traffic distribution<\/li><li>support performance dashboards that cover load issues, errors, latency issues, saturation of resources (e.g. CPU load averages) and instance counts<\/li><\/ul><\/div>\n\n\n<h3 class=\"gb-headline gb-headline-a0549973 gb-headline-text\" id=\"%25f0%259f%2591%25be-run-chaos-engineering-at-scale\">\ud83d\udc7e Run Chaos Engineering at scale<\/h3>\n\n\n<p>Netflix is famous for its extensive use of chaos engineering to ensure all of the above metrics like SPS, TTI and TTR are going in the right direction.<\/p>\n\n\n<h4 class=\"gb-headline gb-headline-6bae109b gb-headline-text\" id=\"what-is-chaos-engineering\"><strong>What is chaos engineering?<\/strong><\/h4>\n\n\n<blockquote class=\"wp-block-quote is-style-plain is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Experimenting on a distributed system in order to build confidence in the system&#8217;s ability to withstand turbulent conditions in production       \u2014 Nora Jones, ex-Senior Chaos Engineer, Netflix<\/p>\n<\/blockquote>\n\n\n\n<p>Chaos engineering is a capability heavily based on Netflix&#8217;s work from 2008 through the early 2010s. It <strong>builds on the value of common tests<\/strong> like unit testing and integration testing.<\/p>\n\n\n\n<p>Chaos work takes it up a notch from these older methods by adding failure or latency on calls between services.<\/p>\n\n\n\n<p><strong>Why do this?<\/strong> Because it helps uncover and resolve issues typically found when services call on another like network latency, congestion and logical or scaling failure<\/p>\n\n\n\n<p>Adding Chaos to Netflix engineering has led to a culture shift from &#8220;What happens&nbsp;<em>if<\/em>&nbsp;this fails&#8221; to <strong>&#8220;What happens&nbsp;<em>when<\/em>&nbsp;this fails&#8221;.<\/strong><\/p>\n\n\n<h2 class=\"gb-headline gb-headline-603389eb gb-headline-text\" id=\"how-chaos-engineering-can-be-done\">How chaos engineering can be done<\/h2>\n\n\n<div class=\"wp-block-stackable-icon-list stk-block-icon-list stk-block stk-8214970\" data-block-id=\"8214970\"><style>.stk-8214970 ul li,.stk-8214970 ol li{font-size:20px !important}.stk-8214970 li{padding-inline-start:6px !important;margin-bottom:12px !important}.stk-8214970 ul li{list-style-image:url('data:image\/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZm9jdXNhYmxlPSJmYWxzZSIgZGF0YS1wcmVmaXg9ImZhcyIgZGF0YS1pY29uPSJwYXN0YWZhcmlhbmlzbSIgY2xhc3M9InN2Zy1pbmxpbmUtLWZhIGZhLXBhc3RhZmFyaWFuaXNtIGZhLXctMjAiIHJvbGU9ImltZyIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Qm94PSIwIDAgNjQwIDUxMiIgc3R5bGU9ImZpbGw6ICMwMDAgIWltcG9ydGFudDsgY29sb3I6ICMwMDAgIWltcG9ydGFudDt0cmFuc2Zvcm06IHJvdGF0ZShkZWcpICFpbXBvcnRhbnQ7Ij48cGF0aCBmaWxsPSIjMDAwIiBkPSJNNjI0LjU0IDM0Ny42N2MtMzIuNy0xMi41Mi01Ny4zNiA0LjI1LTc1LjM3IDE2LjQ1LTE3LjA2IDExLjUzLTIzLjI1IDE0LjQyLTMxLjQxIDExLjM2LTguMTItMy4wOS0xMC44My05LjM4LTE1Ljg5LTI5LjM4LTMuMzMtMTMuMTUtNy40NC0yOS4zMi0xNy45NS00Mi42NSAyLjI0LTIuOTEgNC40My01Ljc5IDYuMzgtOC41N0M1MDAuNDcgMzA0LjQ1IDUxMy43MSAzMTIgNTMyIDMxMmMzMy45NSAwIDUwLjg3LTI1Ljc4IDYyLjA2LTQyLjgzIDEwLjU5LTE2LjE0IDE1LTIxLjE3IDIxLjk0LTIxLjE3IDEzLjI1IDAgMjQtMTAuNzUgMjQtMjRzLTEwLjc1LTI0LTI0LTI0Yy0zMy45NSAwLTUwLjg3IDI1Ljc4LTYyLjA2IDQyLjgzLTEwLjYgMTYuMTQtMTUgMjEuMTctMjEuOTQgMjEuMTctMTcuMzEgMC0zNy40OC02MS40My05Ny4yNi0xMDEuOTFsMTcuMjUtMzQuNUM0ODUuNDMgMTI1LjUgNTEyIDk3Ljk4IDUxMiA2NGMwLTM1LjM1LTI4LjY1LTY0LTY0LTY0cy02NCAyOC42NS02NCA2NGMwIDEzLjAyIDMuOTQgMjUuMSAxMC42MiAzNS4yMWwtMTguMTUgMzYuM2MtMTYuOTgtNC42LTM1LjYtNy41MS01Ni40Ni03LjUxcy0zOS40OSAyLjkxLTU2LjQ2IDcuNTFsLTE4LjE1LTM2LjNDMjUyLjA2IDg5LjEgMjU2IDc3LjAyIDI1NiA2NGMwLTM1LjM1LTI4LjY1LTY0LTY0LTY0cy02NCAyOC42NS02NCA2NGMwIDMzLjk4IDI2LjU2IDYxLjUgNjAuMDIgNjMuNmwxNy4yNSAzNC41QzE0NS42OCAyMDIuNDQgMTI1LjE1IDI2NCAxMDggMjY0Yy02Ljk0IDAtMTEuMzQtNS4wMy0yMS45NC0yMS4xN0M3NC44OCAyMjUuNzggNTcuOTYgMjAwIDI0IDIwMGMtMTMuMjUgMC0yNCAxMC43NS0yNCAyNHMxMC43NSAyNCAyNCAyNGM2Ljk0IDAgMTEuMzQgNS4wMyAyMS45NCAyMS4xN0M1Ny4xMyAyODYuMjIgNzQuMDUgMzEyIDEwOCAzMTJjMTguMjkgMCAzMS41My03LjU1IDQxLjctMTcuMTEgMS45NSAyLjc5IDQuMTQgNS42NiA2LjM4IDguNTctMTAuNTEgMTMuMzMtMTQuNjIgMjkuNS0xNy45NSA0Mi42NS01LjA2IDIwLTcuNzcgMjYuMjgtMTUuODkgMjkuMzgtOC4xMSAzLjA2LTE0LjMzLjE3LTMxLjQxLTExLjM2LTE4LjAzLTEyLjItNDIuNzItMjguOTItNzUuMzctMTYuNDUtMTIuMzkgNC43Mi0xOC41OSAxOC41OC0xMy44NyAzMC45NyA0LjcyIDEyLjQxIDE4LjYxIDE4LjYxIDMwLjk3IDEzLjg4IDguMTYtMy4wOSAxNC4zNC0uMTkgMzEuMzkgMTEuMzYgMTMuNTUgOS4xNiAzMC44MyAyMC44NiA1Mi40MiAyMC44NCA3LjE3IDAgMTQuODMtMS4yOCAyMi45Ny00LjM5IDMyLjY2LTEyLjQ0IDM5Ljk4LTQxLjMzIDQ1LjMzLTYyLjQ0IDIuMjEtOC43MiAzLjk5LTE0LjQ5IDUuOTUtMTguODcgMTYuNjIgMTMuNjEgMzYuOTUgMjUuODggNjEuNjQgMzQuMTctOS45NiAzNy0zMi4xOCA5MC44LTYwLjI2IDkwLjgtMTMuMjUgMC0yNCAxMC43NS0yNCAyNHMxMC43NSAyNCAyNCAyNGM2Ni43NCAwIDk3LjA1LTg4LjYzIDEwNy40Mi0xMjkuMTQgNi42OS42IDEzLjQyIDEuMTQgMjAuNTggMS4xNHMxMy44OS0uNTQgMjAuNTgtMS4xNEMzNTAuOTUgNDIzLjM3IDM4MS4yNiA1MTIgNDQ4IDUxMmMxMy4yNSAwIDI0LTEwLjc1IDI0LTI0cy0xMC43NS0yNC0yNC0yNGMtMjcuOTQgMC01MC4yMS01My44MS02MC4yMi05MC44MSAyNC42OS04LjI5IDQ1LTIwLjU2IDYxLjYyLTM0LjE2IDEuOTYgNC4zOCAzLjc0IDEwLjE1IDUuOTUgMTguODcgNS4zNCAyMS4xMSAxMi42NyA1MCA0NS4zMyA2Mi40NCA4LjE0IDMuMTEgMTUuOCA0LjM5IDIyLjk3IDQuMzkgMjEuNTkgMCAzOC44Ny0xMS42OSA1Mi40Mi0yMC44NCAxNy4wNS0xMS41NSAyMy4yOC0xNC40NSAzMS4zOS0xMS4zNiAxMi4zOSA0Ljc1IDI2LjI3LTEuNDcgMzAuOTctMTMuODggNC43MS0xMi40LTEuNDktMjYuMjYtMTMuODktMzAuOTh6TTQ0OCA0OGM4LjgyIDAgMTYgNy4xOCAxNiAxNnMtNy4xOCAxNi0xNiAxNi0xNi03LjE4LTE2LTE2IDcuMTgtMTYgMTYtMTZ6bS0yNTYgMGM4LjgyIDAgMTYgNy4xOCAxNiAxNnMtNy4xOCAxNi0xNiAxNi0xNi03LjE4LTE2LTE2IDcuMTgtMTYgMTYtMTZ6IiBzdHJva2U9IiMwMDAiIHN0eWxlPSJmaWxsOiByZ2IoMCwgMCwgMCk7IHN0cm9rZTogcmdiKDAsIDAsIDApOyIvPjwvc3ZnPg==') !important}.stk-8214970 li::marker{font-size:2em !important}@media screen and (max-width:1023px){.stk-8214970 ul li,.stk-8214970 ol li{font-size:20px !important}}<\/style><ul><li>Graceful restarts and degradations using the Chaos Monkey tool<\/li><li>Targeted chaos for specific components of a system e.g. Kafka \u2014 <em>&#8220;let&#8217;s see what happens when we delete topics&#8221;<\/em><\/li><li>Cascading failure checks &#8211; see how the failure of one part of the system triggers failure in other parts of the system<\/li><li>Injecting failure into services in an automated manner with limits on # of users affected \u2014 experiment can be shorted if the SPS risks dropping below acceptable levels<\/li><\/ul><\/div>\n\n\n<h2 class=\"gb-headline gb-headline-c7954d38 gb-headline-text\" id=\"parting-remarks\">Parting remarks<\/h2>\n\n\n<p>Netflix SREs goes through all of these amazing fetes to make sure that you can easily binge-watch your fave show this weekend!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction A lot goes on in the background every time you load up your favorite Netflix movie or series. Engineers spread across Chaos Engineering, Performance Engineering and Site Reliability Engineering (SRE) are working non-stop to ensure the magic keeps happening. \ud83d\udcca Here are some performance statistics for Netflix When it was alone on top of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60,8],"tags":[49,78],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/356"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=356"}],"version-history":[{"count":20,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/356\/revisions"}],"predecessor-version":[{"id":5770,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/356\/revisions\/5770"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=356"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}