{"id":5191,"date":"2022-11-22T05:47:44","date_gmt":"2022-11-21T19:47:44","guid":{"rendered":"https:\/\/sysmit.com\/cf22\/?p=5191"},"modified":"2023-12-13T15:27:27","modified_gmt":"2023-12-13T05:27:27","slug":"analysis-sre-platform-10-tech-companies","status":"publish","type":"post","link":"https:\/\/sysmit.com\/cf22\/analysis-sre-platform-10-tech-companies\/","title":{"rendered":"Analysis of SRE and platform setup at 10+ tech companies"},"content":{"rendered":"\n<p>In this article, you will see a breakdown of <strong>the platform setup and SRE practices within 12 non-FAANG technology companies<\/strong>.<\/p>\n\n\n\n<p>This is based on the <a href=\"https:\/\/sreteams.substack.com\/\">case studies by Andrios Robert<\/a>.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p style=\"font-size:clamp(15.747px, 0.984rem + ((1vw - 3.2px) * 0.878), 24px);\">\u201cThere is a lot of content available on how Google did [Site Reliability Engineering]; let\u2019s uncover what happens with the rest of the world.\u201d <\/p>\n\n\n\n<p style=\"font-size:clamp(15.747px, 0.984rem + ((1vw - 3.2px) * 0.878), 24px);\">\u2014 Andrios Robert, founder of <a href=\"http:\/\/Runops.io\">Runops.io<\/a><\/p>\n<\/blockquote>\n\n\n\n<p>You might be thinking, \u201cWhy don\u2019t I read Andrios\u2019 writeups instead of this?\u201d. You should at some point, but chances are that you are pressed for time.<\/p>\n\n\n\n<p>I have trawled through Andrios\u2019 case studies and pulled together common threads, which I\u2019ll explain in my analysis toward the end of this article.<\/p>\n\n\n\n<p>Let\u2019s first cover a few specifics about the companies being covered:<\/p>\n\n\n\n<ul>\n<li>growth or late-stage startups and mid-sized companies \u2014 no FAANG or early-stage startups are represented below<\/li>\n\n\n\n<li>they are <em>all<\/em> Brazilian companies \u2014 founded there or having a branch there \u2014 but their experiences should still translate across to other regions<\/li>\n<\/ul>\n\n\n\n<p><em>Expanding on the last part:<\/em><\/p>\n\n\n\n<p>Yes, these companies are based in a faraway region. However, I believe their experiences can highlight <strong>what companies in similar industries would be doing elsewhere in the world<\/strong>.<\/p>\n\n\n\n<p>I doubt there is an appetite in Latin America to reinvent the wheel when serving cloud applications similar to what we see elsewhere.<\/p>\n\n\n\n<p>Let\u2019s have a business-level picture of the companies in a table format:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Company<\/th><th>Industry<\/th><th>What they do<\/th><th>Founded<\/th><th>Funding\/Revenue<\/th><\/tr><\/thead><tbody><tr><td>Hash<\/td><td>FinTech<\/td><td>Brazil\u2019s Square<\/td><td>Feb 2017<\/td><td>$58.7m, Series C<\/td><\/tr><tr><td>Dafiti<\/td><td>eCommerce<\/td><td>Fashion store for all<\/td><td>Nov 2010<\/td><td>$250m, Series C<\/td><\/tr><tr><td>Creditas<\/td><td>FinTech<\/td><td>Consumer loans<\/td><td>Apr 2012<\/td><td>$1.1b, Series F<\/td><\/tr><tr><td>TOTVS<\/td><td>ERP<\/td><td>Enterprise systems<\/td><td>Sep 1983<\/td><td>$3.8b market cap<\/td><\/tr><tr><td>Empiricus Research<\/td><td>FinTech<\/td><td>Investment tracker<\/td><td>2009<\/td><td>Unknown<\/td><\/tr><tr><td>Leroy Merlin (Brazil)<\/td><td>eCommerce<\/td><td>Home\/Hardware<\/td><td>1923<\/td><td>$84m revenue 2020<\/td><\/tr><tr><td>Dock<\/td><td>FinTech<\/td><td>Banking APIs<\/td><td>2014<\/td><td>$280m funding<\/td><\/tr><tr><td>Loggi<\/td><td>Transport<\/td><td>Delivery app<\/td><td>2013<\/td><td>$507m, Series F<\/td><\/tr><tr><td>Deliver Center<\/td><td>FoodTech<\/td><td>Delivery app management<\/td><td>2016<\/td><td>Unknown<\/td><\/tr><tr><td>Quintoandar<\/td><td>Property<\/td><td>Long-term rental marketplace<\/td><td>2012<\/td><td>$700m, Series E<\/td><\/tr><tr><td>Natura<\/td><td>eCommerce<\/td><td>Beauty shop<\/td><td>1969<\/td><td>$9.6b revenue in 2021<\/td><\/tr><tr><td>PicPay<\/td><td>FinTech<\/td><td>neobank<\/td><td>2012<\/td><td>Acquired for undisclosed sum<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\"><em>What they do <\/em><span style=\"text-decoration: underline;\">and <em>Funding\/Revenue<\/em> data obtained from Crunchbase<\/span><\/figcaption><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p>As you can see, these are <strong>not early-stage startups or small businesses<\/strong>.<\/p>\n\n\n\n<p>Now let\u2019s collate the common DevOps and SRE elements across these organizations:<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"platform-setup\">Platform setup<\/h2>\n\n<h3 class=\"wp-block-heading\" id=\"platform\">Platform<\/h3>\n\n\n<ul>\n<li>K8s (Hash, Dafiti, Creditas, TOTVS, Empiricus, Dock, Loggi, Delivery Center, Quintoandar, Natura, PicPay)<\/li>\n\n\n\n<li>AWS Lambda functions (Empiricus, Dock)<\/li>\n\n\n\n<li>Istio (Hash, Dafiti, Empiricus, Loggi)<\/li>\n\n\n\n<li>Terraform for IAC (Dafiti, Creditas, TOTVS, Natura)<\/li>\n<\/ul>\n\n\n<h3 class=\"wp-block-heading\" id=\"monitoring\">Monitoring<\/h3>\n\n\n<ul>\n<li>Prometheus + Grafana (Hash, Creditas, TOTVS, Empiricus, Quintoandar, Natura)<\/li>\n\n\n\n<li>NewRelic (Dafiti, Creditas, Delivery Center)<\/li>\n\n\n\n<li>Datadog (Empiricus, Dock)<\/li>\n\n\n\n<li>PagerDuty (Empiricus, Natura)<\/li>\n<\/ul>\n\n\n<h3 class=\"wp-block-heading\" id=\"codebase\">Codebase<\/h3>\n\n\n<ul>\n<li>Pre-microservices monolith (Leroy Merlin)<\/li>\n\n\n\n<li>Diverse multi-language codebase (Dafiti, Empiricus, Delivery Center, Quintoandar, PicPay)<\/li>\n\n\n\n<li>Single language codebase across org (Creditas, Leroy Merlin, Loggi)<\/li>\n<\/ul>\n\n\n\n\n<h2 class=\"wp-block-heading\" id=\"sre-practices\">SRE practices<\/h2>\n\n\n<ul>\n<li>Developers can autonomously deploy to production (Hash, Creditas, Empiricus, Dock, Natura, PicPay)<\/li>\n\n\n\n<li>Product teams are on-call to support their product (TOTVS, Leroy Merlin, Dock, Delivery Center, Quintoandar, PicPay)<\/li>\n\n\n\n<li>Product teams participate directly in platform\/tooling matters (Hash, Loggi, Delivery Center, Quintoandar)<\/li>\n\n\n\n<li>Feature flags enable safer rollouts (Leroy Merlin, Loggi)<\/li>\n\n\n\n<li>Emulate Google SRE principles to an extent (Hash)<\/li>\n\n\n\n<li>Deployments must pass the readiness checklist (Hash)<\/li>\n\n\n\n<li>SRE teams set alerting based on the product team\u2019s specifications (PicPay)<\/li>\n\n\n\n<li>SRE teams responsible for after-hours incidents (PicPay)<\/li>\n\n\n\n<li>Break SRE work into multiple streams e.g. observability etc (PicPay)<\/li>\n<\/ul>\n\n\n<h2 class=\"wp-block-heading\" id=\"my-brief-analysis\">My brief analysis<\/h2>\n\n<h3 class=\"wp-block-heading\" id=\"platform\">Platform<\/h3>\n\n\n<p><strong>Kubernetes &#8211;<\/strong> It shouldn\u2019t be surprising that Kubernetes (aka K8s) is the platform of choice for 11 out of 12 studied organizations. Leroy Merlin was the only exception, still migrating from legacy monolith to microservices architecture.<\/p>\n\n\n\n<p><strong>Lambda &#8211;<\/strong> A small subset of companies uses Lambda functions for executing code. Its lesser use makes sense because Lambda requires using AWS, learning a new AWS service, and can drive up cloud costs without persistent oversight.<\/p>\n\n\n\n<p><strong>Istio &#8211;<\/strong> Service mesh seems to be getting visible use in production systems now. I wasn\u2019t sure if it would make it based on how K8s practitioners viewed it in 2020.<\/p>\n\n\n\n<p><strong>Terraform &#8211;<\/strong> No surprise that Terraform is being used extensively by many of these organizations for their infrastructure provisioning capability.<\/p>\n\n\n<h3 class=\"wp-block-heading\" id=\"monitoring\">Monitoring<\/h3>\n\n\n<p>As expected, there is a fair spread of tooling used for monitoring. Nonetheless, the open-source Prometheus and Grafana combination holds the largest share, reflecting the wider CNCF community uptake.<\/p>\n\n\n\n<p>I am curious whether any of these use commercial implementations of Grafana.<\/p>\n\n\n\n<p>The market among commercial monitoring tools (NewRelic, PagerDuty, Datadog) is divided with a handful of companies taking up each offering.<\/p>\n\n\n<h3 class=\"wp-block-heading\" id=\"codebase\">Codebase<\/h3>\n\n\n<p>There was a fairly even split of organizations with single and multi-language code architectures.<\/p>\n\n\n\n<p>Organizations with a single language may have promoted their hiring brand as a single language shop or hired from within a network of engineers who all worked in that one language.<\/p>\n\n\n\n<p>There are 2 ways to look at language diversity in multi-language organizations:<\/p>\n\n\n\n<ol>\n<li>Language popularity<\/li>\n\n\n\n<li>Different service types<\/li>\n<\/ol>\n\n\n\n<p>Regarding language popularity &#8211; some organizations scaled up as different languages became popular and others faded in popularity; their engineering workforce would reflect this transition<\/p>\n\n\n\n<p>Regarding different service types &#8211; some services may function better in a certain language than others e.g. Golang for infrastructure services, Python for data scripting, C for Linux services, Javascript for frontend, etc.<\/p>\n\n\n\n<p>Feature flags seem to have a way to go before they are widely discussed and adopted. That may be because they add another layer of complexity to deploys.<\/p>\n\n\n<h3 class=\"wp-block-heading\" id=\"sre-practices\">SRE practices<\/h3>\n\n\n<p>I found a few common threads but many unique practices too. It seems like <em>how<\/em> the organization practices software reliability remains unique to its environment.<\/p>\n\n\n\n<p>I\u2019m sure a lot more common practices exist, but they are not as obvious to discuss as say, \u201cWe run EKS with Redis and Kafka monitored by NewRelic\u201d.<\/p>\n\n\n\n<p>Many organizations were open to having developers autonomously push to production but also making them responsible for any incidents that resulted.<\/p>\n\n\n\n<p>Some organizations like Quintoandar additionally incentivize their developers to sit on incident call rotations<\/p>\n\n\n\n<p>It was also interesting to note that many organizations have their developers actively participate in platform and tooling matters.<\/p>\n\n\n\n<p>Here are interesting examples of how SREs are utilized::<\/p>\n\n\n\n<ul>\n<li>as consultants to support incident capability &#8211; Delivery Center\u2019s SREs are responsible for structuring the product team\u2019s on-call schedules<\/li>\n\n\n\n<li>last-resort support, to be called upon to solve critical or tricky problems<\/li>\n\n\n\n<li>create a center of excellence for highly secure, reliable, and performant software practices like at Natura<\/li>\n<\/ul>\n\n\n\n<p>Only one organization made SREs responsible for after-hours incidents.<\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"concluding-words\">Concluding words<\/h2>\n\n\n<p>I trust that this brief analysis of Andrios\u2019 SRE team case studies has given you a new or clearer perspective on how organizations can set up their platform and SRE service culture.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this article, you will see a breakdown of the platform setup and SRE practices within 12 non-FAANG technology companies. This is based on the case studies by Andrios Robert. \u201cThere is a lot of content available on how Google did [Site Reliability Engineering]; let\u2019s uncover what happens with the rest of the world.\u201d \u2014 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60,29],"tags":[41,13,7],"_links":{"self":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5191"}],"collection":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/comments?post=5191"}],"version-history":[{"count":3,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5191\/revisions"}],"predecessor-version":[{"id":5755,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/posts\/5191\/revisions\/5755"}],"wp:attachment":[{"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/media?parent=5191"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/categories?post=5191"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysmit.com\/cf22\/wp-json\/wp\/v2\/tags?post=5191"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}