LinkedIn’s Site Reliability Engineers (SREs) ensure all that traffic gets served with minimal dropouts and performance degradation. <\/p>\n\n\n\n
SRE efforts will only continue to grow, as the company has an ambitious goal to \u201ccreate economic opportunity for every member of the global workforce\u201d. <\/p>\n\n\n\n
LinkedIn’s management aims for it to be more than an online resume and make an \u201ceconomic graph\u201d; similar to Facebook\u2019s social graph. One that maps every aspect of the global economy – companies, jobs, schools, skills, etc.<\/p>\n\n\n
\ud83d\udcca Here are some performance statistics for LinkedIn<\/h2>\n\n\n![\"\"](\"https:\/\/sysmit.com\/cf22\/wp-content\/uploads\/linkedin-sre-practice-statistics-1.png\" "\\"linkedin-sre-practice-statistics-1\\"")
<\/figure>\n\n\n\n<\/div>\n\n\nAs of 2022, LinkedIn had:<\/h3>\n\n\n\n- 850 million members<\/li>\n\n\n\n
- Between 1.25 to 1.5 billion unique visits per month <\/li>\n\n\n\n
- In May 2022, close to 1.5 billion unique global visitors had visited LinkedIn.com, up from 1.3 billion visitors in December 2021<\/li>\n\n\n\n
- 39% (or 340 million) of users as Premium members<\/li>\n<\/ul>\n\n\n
In 2017, LinkedIn had:<\/h3>\n\n\n\n- 1500+ services in production<\/li>\n\n\n\n
- 600 TB of stored data <\/li>\n\n\n\n
- ranking of 8th busiest website in the world<\/li>\n\n\n\n
- 100+ SREs supporting 1000+ SW engineers (ratio 10:1 makes sense)<\/li>\n\n\n\n
- 20,000+ production machines in operations<\/li>\n\n\n\n
- 300+ RESTful services<\/li>\n\n\n\n
- performance targets of 10ms latency for 99% of services<\/li>\n\n\n\n
- 100+ Kafka clusters handling 7 trillion messages per day<\/li>\n<\/ul>\n\n\n\n
Each Site Reliability Engineer at LinkedIn is responsible for ~500 machines, but that can go up or down depending on the needs of the system on the day.<\/p>\n\n\n\n
\n\u201cIt is impressive to be able to handle this kind of [work]. This contributes to the challenge of finding the right people. Finding SREs who can operate at this level is a very big challenge.\u201d<\/p>\n\u2014 excerpt from video talk on SRE Hiring<\/a> by Greg Leffler, SRE manager at LinkedIn from 2012-2016<\/cite><\/blockquote>\n\n\n\nLinkedIn’s SRE managers found that many SRE candidates sought work that would allow them to leave a legacy when they move on<\/strong>. <\/p>\n\n\n\nThis trait correlates with the capability to work with broad-spanning complex systems in ambiguous circumstances.<\/p>\n\n\n\n
If you can do out-of-the-box thinking and work in tough situations, you’d want that work to have a lasting impact.<\/p>\n\n\n
\ud83d\udd31 How SRE fits into LinkedIn’s engineering culture<\/h2>\n\nTeam formation<\/h3>\n\n\n\n- 300+ SREs across 4 global offices \u2014 Bay Area (San Francisco, Palo Alto), New York City, Bangalore<\/li>\n\n\n\n
- Team size depends on whether they’re embedded in a product team or part of a dedicated SRE team<\/li>\n\n\n\n
- SREs get to pick the kind of team they want to be part of <\/li>\n\n\n\n
- Hybrid roles exist i.e. SREs who focus on systems or software development<\/li>\n<\/ul>\n\n\n
Many SREs at LinkedIn as “embedded SREs”<\/strong><\/h4>\n\n\nIn the mid-2010s, many SREs were distributed across product engineering teams i.e. embedded into the product team. <\/p>\n\n\n\n
The reasoning behind this was that centralized teams eventually face issues that come with becoming a \u201cshared service\u201d i.e. not so critical to the day-to-day but only consulted if really necessary. <\/p>\n\n\n\n
The centralized model would not have been conducive to LinkedIn’s SRE needs at the time. The thing to remember is that SRE is a continuously high-involvement practice and not only for when things go wrong.<\/p>\n\n\n
LinkedIn has a large SRE base in India<\/strong><\/h4>\n\n\nLinkedIn’s engineering management focused strongly on creating SRE teams in India. This may be partly due to hiring difficulties endemic to the SRE field in North America and Europe. <\/p>\n\n\n\n
There were initial difficulties as SRE had a stigma within talent pools in India. <\/p>\n\n\n\n
Many with years of experience perceived SRE as a relabelled or glorified systems administration role. LinkedIn’s SRE hiring managers instead focused on turning junior engineers and graduates into high-potential SREs. <\/p>\n\n\n\n
They emphasized from the outset that SRE was not a typical operations role and that it would involve broad-spanning work in ambiguous settings.<\/p>\n\n\n\n
As of 2017, the Bangalore office of LinkedIn had 60 SREs in 10 teams. <\/p>\n\n\n
History of SRE at LinkedIn<\/h3>\n\nPrior to a modern SRE discipline (the early 2010s)<\/strong><\/h4>\n\n\nIn 2013, LinkedIn rebranded teams in the existing discipline of AppOps to SRE. The newly minted SREs worked alongside stratified operations teams in verticals such as systems, networks, applications, and DBA. <\/p>\n\n\n\n
This way of running software operations proved difficult as LinkedIn continued its growth trajectory. Several SREs noted issues like:<\/p>\n\n\n\n
\n- Cooperation among these teams went only by way of tickets being sent to each other <\/li>\n\n\n\n
- There were walls not only between dev and ops but within ops itself<\/li>\n\n\n\n
- Developers did not have access to production and even most non-production environments – \u201chand me the code, and I will deploy it\u201d so ops were bogged down with release operations<\/li>\n\n\n\n
- Pager fatigue was extreme with operations people training significant others to keep an eye for alerts on the Blackberry while they slept (3 alerts every 5 minutes on average)<\/li>\n\n\n\n
- Mon-Wed 6-10am outages due to capacity issues and little visibility into demand due to no observability instrumentation<\/li>\n\n\n\n
- Dealing with outages that were so frequent, there was an outage every day of the calendar year<\/li>\n\n\n\n
- MTTR was 1500 minutes, meaning issues were not getting resolved on the same day<\/li>\n<\/ul>\n\n\n\n
SREs originally started off in this environment as firefighters but evolved during a drastic shift in the software operations at LinkedIn.<\/p>\n\n\n
Transitioning to SRE from previous operational discipline<\/strong><\/h4>\n\n\nThere was a need for change in the mid-2010s considering 100 million members relied on LinkedIn at the time. <\/p>\n\n\n\n
LinkedIn could not mess up its increasingly complex software operations.<\/p>\n\n\n\n
Something had to give, as LinkedIn was suffering frequent outages and performance issues despite now having an SRE team. <\/p>\n\n\n\n
LinkedIn SREs and the wider organization had to challenge several antipatterns:<\/p>\n\n\n\nAntipattern identifier<\/strong><\/td>Antipattern description<\/strong><\/td>Shift to these propatterns\u2026<\/strong><\/td><\/tr>Firefighter \ud83d\udc68\ud83c\udffe\u200d\ud83d\ude92<\/td> \u274c react to handle incidents that happen to keep the company functioning one more day<\/td> \u2705 automate the manual work
\u2705 deliver instrumentation to get rich data on issues and alerts
\u2705 understand the stack at a deeper level<\/td><\/tr> Gatekeeper \ud83d\udc82\ud83c\udffd<\/td> \u274c control releases to protect the site from developers
\u274c \u201ctalk to me if you want to touch production\u201d
\u274c push a button to deploy someone else\u2019s work
\u274c have arbitrary schedules for releases<\/td> \u2705 develop automated gatekeepers to assure quality
\u2705 let developers own their work in production
\u2705 support developers in self-service deployment with the use of pre-launch checklists and release guidance <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe decision was then made across the organization to radically change several aspects of software operations. This included a drastic shift in software architecture and developer involvement in operations for the first time.<\/p>\n\n\n\n
To achieve this lofty goal, product development was stopped for 3 months<\/strong>. <\/p>\n\n\n\n\n\nThis signified a gutsy commitment to drive the necessary change.<\/p>\n\n\n\n
Several factors supported the transition to more effective software operations:<\/p>\n\n\n\n
\n- The transition received solid management support from all affected VPs<\/li>\n\n\n\n
- The umbrella operations organization with sysadmins, network engineers, DB admins, etc. remained, but communication made it clear that it would not be business as usual <\/li>\n\n\n\n
- Major changes to how the software was architected came with support from experienced SREs and other seasoned engineers<\/li>\n\n\n\n
- Developers were increasingly involved in discussions about their ownership of software in production <\/li>\n<\/ul>\n\n\n
How the change was actioned toward a modern SRE discipline<\/strong><\/h4>\n\n\nSREs initially advocated a few human-related practices to drive the change, including:<\/p>\n\n\n\n
\n- encouraging and supporting developers in joining the on-call roster; some development teams got more into this, while others kept a distance from on-call <\/li>\n\n\n\n
- implement DevOps, where developers are involved in operational thinking and behaviors around self-deploying code – \u201cIf it\u2019s checked-in, it\u2019s ready to go to production\u201d<\/li>\n<\/ul>\n\n\n\n
A few of the technical changes included:<\/p>\n\n\n\n
\n- Move to a service-oriented architecture (SOA) and reduce dependence on monolithic artifacts<\/li>\n\n\n\n
- Develop self-service portals for consuming metrics around services<\/li>\n\n\n\n
- Enabling graceful degradation of services rather than 500 erroring the whole request load<\/li>\n\n\n\n
- Uptake of distributed database model vs monolith databases to reduce SPOF (single point of failure) risk<\/li>\n\n\n\n
- Switch from hardware to software-based load balancing<\/li>\n\n\n\n
- Centralized branch model to simplify the commit process compared to feature branches<\/li>\n\n\n\n
- Implemented multiple testing methodologies like pre-commit, PCS, and PCL <\/li>\n\n\n\n
- Develop an auto-remediation system to replace manual network operations center (NOC) processes \u2014 resulting in a career transition program for traditional operations roles into SRE<\/li>\n\n\n\n
- helping developers run A\/B tests to ensure backward compatibility of their commit before users would see it <\/li>\n\n\n\n
- support self-service deployment with a canary push to production<\/a> to ensure stability before rollout to the entire cluster <\/li>\n<\/ul>\n\n\n\n
Here’s a quick rundown of the cadence for self-service deployments for developers<\/strong> at LinkedIn:<\/p>\n\n\n\n\n- Canary to a single production instance<\/li>\n\n\n\n
- Check the automated metrics-based validation of success<\/li>\n\n\n\n
- Promote deployment to a single production data center<\/li>\n\n\n\n
- Promote deployment to remaining production centers<\/li>\n\n\n\n
- Ramp to an increasingly growing member base <\/li>\n<\/ol>\n\n\n\n
This self-service model has supported 15,000 commits and 600+ feature ramp-ups per day<\/p>\n\n\n\n
Since the self-service model is now well entrenched in the LinkedIn culture, SREs continue their work on various other problems. They work in areas like performance, platforms, resilience, architecture, etc.<\/p>\n\n\n\n
<\/figure>\n\n\n\nAs of 2022, LinkedIn had:<\/h3>\n\n\n\n- 850 million members<\/li>\n\n\n\n
- Between 1.25 to 1.5 billion unique visits per month <\/li>\n\n\n\n
- In May 2022, close to 1.5 billion unique global visitors had visited LinkedIn.com, up from 1.3 billion visitors in December 2021<\/li>\n\n\n\n
- 39% (or 340 million) of users as Premium members<\/li>\n<\/ul>\n\n\n
In 2017, LinkedIn had:<\/h3>\n\n\n\n- 1500+ services in production<\/li>\n\n\n\n
- 600 TB of stored data <\/li>\n\n\n\n
- ranking of 8th busiest website in the world<\/li>\n\n\n\n
- 100+ SREs supporting 1000+ SW engineers (ratio 10:1 makes sense)<\/li>\n\n\n\n
- 20,000+ production machines in operations<\/li>\n\n\n\n
- 300+ RESTful services<\/li>\n\n\n\n
- performance targets of 10ms latency for 99% of services<\/li>\n\n\n\n
- 100+ Kafka clusters handling 7 trillion messages per day<\/li>\n<\/ul>\n\n\n\n
Each Site Reliability Engineer at LinkedIn is responsible for ~500 machines, but that can go up or down depending on the needs of the system on the day.<\/p>\n\n\n\n
\n\u201cIt is impressive to be able to handle this kind of [work]. This contributes to the challenge of finding the right people. Finding SREs who can operate at this level is a very big challenge.\u201d<\/p>\n\u2014 excerpt from video talk on SRE Hiring<\/a> by Greg Leffler, SRE manager at LinkedIn from 2012-2016<\/cite><\/blockquote>\n\n\n\nLinkedIn’s SRE managers found that many SRE candidates sought work that would allow them to leave a legacy when they move on<\/strong>. <\/p>\n\n\n\nThis trait correlates with the capability to work with broad-spanning complex systems in ambiguous circumstances.<\/p>\n\n\n\n
If you can do out-of-the-box thinking and work in tough situations, you’d want that work to have a lasting impact.<\/p>\n\n\n
\ud83d\udd31 How SRE fits into LinkedIn’s engineering culture<\/h2>\n\nTeam formation<\/h3>\n\n\n\n- 300+ SREs across 4 global offices \u2014 Bay Area (San Francisco, Palo Alto), New York City, Bangalore<\/li>\n\n\n\n
- Team size depends on whether they’re embedded in a product team or part of a dedicated SRE team<\/li>\n\n\n\n
- SREs get to pick the kind of team they want to be part of <\/li>\n\n\n\n
- Hybrid roles exist i.e. SREs who focus on systems or software development<\/li>\n<\/ul>\n\n\n
Many SREs at LinkedIn as “embedded SREs”<\/strong><\/h4>\n\n\nIn the mid-2010s, many SREs were distributed across product engineering teams i.e. embedded into the product team. <\/p>\n\n\n\n
The reasoning behind this was that centralized teams eventually face issues that come with becoming a \u201cshared service\u201d i.e. not so critical to the day-to-day but only consulted if really necessary. <\/p>\n\n\n\n
The centralized model would not have been conducive to LinkedIn’s SRE needs at the time. The thing to remember is that SRE is a continuously high-involvement practice and not only for when things go wrong.<\/p>\n\n\n
LinkedIn has a large SRE base in India<\/strong><\/h4>\n\n\nLinkedIn’s engineering management focused strongly on creating SRE teams in India. This may be partly due to hiring difficulties endemic to the SRE field in North America and Europe. <\/p>\n\n\n\n
There were initial difficulties as SRE had a stigma within talent pools in India. <\/p>\n\n\n\n
Many with years of experience perceived SRE as a relabelled or glorified systems administration role. LinkedIn’s SRE hiring managers instead focused on turning junior engineers and graduates into high-potential SREs. <\/p>\n\n\n\n
They emphasized from the outset that SRE was not a typical operations role and that it would involve broad-spanning work in ambiguous settings.<\/p>\n\n\n\n
As of 2017, the Bangalore office of LinkedIn had 60 SREs in 10 teams. <\/p>\n\n\n
History of SRE at LinkedIn<\/h3>\n\nPrior to a modern SRE discipline (the early 2010s)<\/strong><\/h4>\n\n\nIn 2013, LinkedIn rebranded teams in the existing discipline of AppOps to SRE. The newly minted SREs worked alongside stratified operations teams in verticals such as systems, networks, applications, and DBA. <\/p>\n\n\n\n
This way of running software operations proved difficult as LinkedIn continued its growth trajectory. Several SREs noted issues like:<\/p>\n\n\n\n
\n- Cooperation among these teams went only by way of tickets being sent to each other <\/li>\n\n\n\n
- There were walls not only between dev and ops but within ops itself<\/li>\n\n\n\n
- Developers did not have access to production and even most non-production environments – \u201chand me the code, and I will deploy it\u201d so ops were bogged down with release operations<\/li>\n\n\n\n
- Pager fatigue was extreme with operations people training significant others to keep an eye for alerts on the Blackberry while they slept (3 alerts every 5 minutes on average)<\/li>\n\n\n\n
- Mon-Wed 6-10am outages due to capacity issues and little visibility into demand due to no observability instrumentation<\/li>\n\n\n\n
- Dealing with outages that were so frequent, there was an outage every day of the calendar year<\/li>\n\n\n\n
- MTTR was 1500 minutes, meaning issues were not getting resolved on the same day<\/li>\n<\/ul>\n\n\n\n
SREs originally started off in this environment as firefighters but evolved during a drastic shift in the software operations at LinkedIn.<\/p>\n\n\n
Transitioning to SRE from previous operational discipline<\/strong><\/h4>\n\n\nThere was a need for change in the mid-2010s considering 100 million members relied on LinkedIn at the time. <\/p>\n\n\n\n
LinkedIn could not mess up its increasingly complex software operations.<\/p>\n\n\n\n
Something had to give, as LinkedIn was suffering frequent outages and performance issues despite now having an SRE team. <\/p>\n\n\n\n
LinkedIn SREs and the wider organization had to challenge several antipatterns:<\/p>\n\n\n\nAntipattern identifier<\/strong><\/td>Antipattern description<\/strong><\/td>Shift to these propatterns\u2026<\/strong><\/td><\/tr>Firefighter \ud83d\udc68\ud83c\udffe\u200d\ud83d\ude92<\/td> \u274c react to handle incidents that happen to keep the company functioning one more day<\/td> \u2705 automate the manual work
\u2705 deliver instrumentation to get rich data on issues and alerts
\u2705 understand the stack at a deeper level<\/td><\/tr> Gatekeeper \ud83d\udc82\ud83c\udffd<\/td> \u274c control releases to protect the site from developers
\u274c \u201ctalk to me if you want to touch production\u201d
\u274c push a button to deploy someone else\u2019s work
\u274c have arbitrary schedules for releases<\/td> \u2705 develop automated gatekeepers to assure quality
\u2705 let developers own their work in production
\u2705 support developers in self-service deployment with the use of pre-launch checklists and release guidance <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe decision was then made across the organization to radically change several aspects of software operations. This included a drastic shift in software architecture and developer involvement in operations for the first time.<\/p>\n\n\n\n
To achieve this lofty goal, product development was stopped for 3 months<\/strong>. <\/p>\n\n\n\n\n\nThis signified a gutsy commitment to drive the necessary change.<\/p>\n\n\n\n
Several factors supported the transition to more effective software operations:<\/p>\n\n\n\n
\n- The transition received solid management support from all affected VPs<\/li>\n\n\n\n
- The umbrella operations organization with sysadmins, network engineers, DB admins, etc. remained, but communication made it clear that it would not be business as usual <\/li>\n\n\n\n
- Major changes to how the software was architected came with support from experienced SREs and other seasoned engineers<\/li>\n\n\n\n
- Developers were increasingly involved in discussions about their ownership of software in production <\/li>\n<\/ul>\n\n\n
How the change was actioned toward a modern SRE discipline<\/strong><\/h4>\n\n\nSREs initially advocated a few human-related practices to drive the change, including:<\/p>\n\n\n\n
\n- encouraging and supporting developers in joining the on-call roster; some development teams got more into this, while others kept a distance from on-call <\/li>\n\n\n\n
- implement DevOps, where developers are involved in operational thinking and behaviors around self-deploying code – \u201cIf it\u2019s checked-in, it\u2019s ready to go to production\u201d<\/li>\n<\/ul>\n\n\n\n
A few of the technical changes included:<\/p>\n\n\n\n
\n- Move to a service-oriented architecture (SOA) and reduce dependence on monolithic artifacts<\/li>\n\n\n\n
- Develop self-service portals for consuming metrics around services<\/li>\n\n\n\n
- Enabling graceful degradation of services rather than 500 erroring the whole request load<\/li>\n\n\n\n
- Uptake of distributed database model vs monolith databases to reduce SPOF (single point of failure) risk<\/li>\n\n\n\n
- Switch from hardware to software-based load balancing<\/li>\n\n\n\n
- Centralized branch model to simplify the commit process compared to feature branches<\/li>\n\n\n\n
- Implemented multiple testing methodologies like pre-commit, PCS, and PCL <\/li>\n\n\n\n
- Develop an auto-remediation system to replace manual network operations center (NOC) processes \u2014 resulting in a career transition program for traditional operations roles into SRE<\/li>\n\n\n\n
- helping developers run A\/B tests to ensure backward compatibility of their commit before users would see it <\/li>\n\n\n\n
- support self-service deployment with a canary push to production<\/a> to ensure stability before rollout to the entire cluster <\/li>\n<\/ul>\n\n\n\n
Here’s a quick rundown of the cadence for self-service deployments for developers<\/strong> at LinkedIn:<\/p>\n\n\n\n\n- Canary to a single production instance<\/li>\n\n\n\n
- Check the automated metrics-based validation of success<\/li>\n\n\n\n
- Promote deployment to a single production data center<\/li>\n\n\n\n
- Promote deployment to remaining production centers<\/li>\n\n\n\n
- Ramp to an increasingly growing member base <\/li>\n<\/ol>\n\n\n\n
This self-service model has supported 15,000 commits and 600+ feature ramp-ups per day<\/p>\n\n\n\n
Since the self-service model is now well entrenched in the LinkedIn culture, SREs continue their work on various other problems. They work in areas like performance, platforms, resilience, architecture, etc.<\/p>\n\n\n\n
In 2017, LinkedIn had:<\/h3>\n\n\n\n- 1500+ services in production<\/li>\n\n\n\n
- 600 TB of stored data <\/li>\n\n\n\n
- ranking of 8th busiest website in the world<\/li>\n\n\n\n
- 100+ SREs supporting 1000+ SW engineers (ratio 10:1 makes sense)<\/li>\n\n\n\n
- 20,000+ production machines in operations<\/li>\n\n\n\n
- 300+ RESTful services<\/li>\n\n\n\n
- performance targets of 10ms latency for 99% of services<\/li>\n\n\n\n
- 100+ Kafka clusters handling 7 trillion messages per day<\/li>\n<\/ul>\n\n\n\n
Each Site Reliability Engineer at LinkedIn is responsible for ~500 machines, but that can go up or down depending on the needs of the system on the day.<\/p>\n\n\n\n
\n\u201cIt is impressive to be able to handle this kind of [work]. This contributes to the challenge of finding the right people. Finding SREs who can operate at this level is a very big challenge.\u201d<\/p>\n\u2014 excerpt from video talk on SRE Hiring<\/a> by Greg Leffler, SRE manager at LinkedIn from 2012-2016<\/cite><\/blockquote>\n\n\n\nLinkedIn’s SRE managers found that many SRE candidates sought work that would allow them to leave a legacy when they move on<\/strong>. <\/p>\n\n\n\nThis trait correlates with the capability to work with broad-spanning complex systems in ambiguous circumstances.<\/p>\n\n\n\n
If you can do out-of-the-box thinking and work in tough situations, you’d want that work to have a lasting impact.<\/p>\n\n\n
\ud83d\udd31 How SRE fits into LinkedIn’s engineering culture<\/h2>\n\nTeam formation<\/h3>\n\n\n\n- 300+ SREs across 4 global offices \u2014 Bay Area (San Francisco, Palo Alto), New York City, Bangalore<\/li>\n\n\n\n
- Team size depends on whether they’re embedded in a product team or part of a dedicated SRE team<\/li>\n\n\n\n
- SREs get to pick the kind of team they want to be part of <\/li>\n\n\n\n
- Hybrid roles exist i.e. SREs who focus on systems or software development<\/li>\n<\/ul>\n\n\n
Many SREs at LinkedIn as “embedded SREs”<\/strong><\/h4>\n\n\nIn the mid-2010s, many SREs were distributed across product engineering teams i.e. embedded into the product team. <\/p>\n\n\n\n
The reasoning behind this was that centralized teams eventually face issues that come with becoming a \u201cshared service\u201d i.e. not so critical to the day-to-day but only consulted if really necessary. <\/p>\n\n\n\n
The centralized model would not have been conducive to LinkedIn’s SRE needs at the time. The thing to remember is that SRE is a continuously high-involvement practice and not only for when things go wrong.<\/p>\n\n\n
LinkedIn has a large SRE base in India<\/strong><\/h4>\n\n\nLinkedIn’s engineering management focused strongly on creating SRE teams in India. This may be partly due to hiring difficulties endemic to the SRE field in North America and Europe. <\/p>\n\n\n\n
There were initial difficulties as SRE had a stigma within talent pools in India. <\/p>\n\n\n\n
Many with years of experience perceived SRE as a relabelled or glorified systems administration role. LinkedIn’s SRE hiring managers instead focused on turning junior engineers and graduates into high-potential SREs. <\/p>\n\n\n\n
They emphasized from the outset that SRE was not a typical operations role and that it would involve broad-spanning work in ambiguous settings.<\/p>\n\n\n\n
As of 2017, the Bangalore office of LinkedIn had 60 SREs in 10 teams. <\/p>\n\n\n
History of SRE at LinkedIn<\/h3>\n\nPrior to a modern SRE discipline (the early 2010s)<\/strong><\/h4>\n\n\nIn 2013, LinkedIn rebranded teams in the existing discipline of AppOps to SRE. The newly minted SREs worked alongside stratified operations teams in verticals such as systems, networks, applications, and DBA. <\/p>\n\n\n\n
This way of running software operations proved difficult as LinkedIn continued its growth trajectory. Several SREs noted issues like:<\/p>\n\n\n\n
\n- Cooperation among these teams went only by way of tickets being sent to each other <\/li>\n\n\n\n
- There were walls not only between dev and ops but within ops itself<\/li>\n\n\n\n
- Developers did not have access to production and even most non-production environments – \u201chand me the code, and I will deploy it\u201d so ops were bogged down with release operations<\/li>\n\n\n\n
- Pager fatigue was extreme with operations people training significant others to keep an eye for alerts on the Blackberry while they slept (3 alerts every 5 minutes on average)<\/li>\n\n\n\n
- Mon-Wed 6-10am outages due to capacity issues and little visibility into demand due to no observability instrumentation<\/li>\n\n\n\n
- Dealing with outages that were so frequent, there was an outage every day of the calendar year<\/li>\n\n\n\n
- MTTR was 1500 minutes, meaning issues were not getting resolved on the same day<\/li>\n<\/ul>\n\n\n\n
SREs originally started off in this environment as firefighters but evolved during a drastic shift in the software operations at LinkedIn.<\/p>\n\n\n
Transitioning to SRE from previous operational discipline<\/strong><\/h4>\n\n\nThere was a need for change in the mid-2010s considering 100 million members relied on LinkedIn at the time. <\/p>\n\n\n\n
LinkedIn could not mess up its increasingly complex software operations.<\/p>\n\n\n\n
Something had to give, as LinkedIn was suffering frequent outages and performance issues despite now having an SRE team. <\/p>\n\n\n\n
LinkedIn SREs and the wider organization had to challenge several antipatterns:<\/p>\n\n\n\nAntipattern identifier<\/strong><\/td>Antipattern description<\/strong><\/td>Shift to these propatterns\u2026<\/strong><\/td><\/tr>Firefighter \ud83d\udc68\ud83c\udffe\u200d\ud83d\ude92<\/td> \u274c react to handle incidents that happen to keep the company functioning one more day<\/td> \u2705 automate the manual work
\u2705 deliver instrumentation to get rich data on issues and alerts
\u2705 understand the stack at a deeper level<\/td><\/tr> Gatekeeper \ud83d\udc82\ud83c\udffd<\/td> \u274c control releases to protect the site from developers
\u274c \u201ctalk to me if you want to touch production\u201d
\u274c push a button to deploy someone else\u2019s work
\u274c have arbitrary schedules for releases<\/td> \u2705 develop automated gatekeepers to assure quality
\u2705 let developers own their work in production
\u2705 support developers in self-service deployment with the use of pre-launch checklists and release guidance <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe decision was then made across the organization to radically change several aspects of software operations. This included a drastic shift in software architecture and developer involvement in operations for the first time.<\/p>\n\n\n\n
To achieve this lofty goal, product development was stopped for 3 months<\/strong>. <\/p>\n\n\n\n\n\nThis signified a gutsy commitment to drive the necessary change.<\/p>\n\n\n\n
Several factors supported the transition to more effective software operations:<\/p>\n\n\n\n
\n- The transition received solid management support from all affected VPs<\/li>\n\n\n\n
- The umbrella operations organization with sysadmins, network engineers, DB admins, etc. remained, but communication made it clear that it would not be business as usual <\/li>\n\n\n\n
- Major changes to how the software was architected came with support from experienced SREs and other seasoned engineers<\/li>\n\n\n\n
- Developers were increasingly involved in discussions about their ownership of software in production <\/li>\n<\/ul>\n\n\n
How the change was actioned toward a modern SRE discipline<\/strong><\/h4>\n\n\nSREs initially advocated a few human-related practices to drive the change, including:<\/p>\n\n\n\n
\n- encouraging and supporting developers in joining the on-call roster; some development teams got more into this, while others kept a distance from on-call <\/li>\n\n\n\n
- implement DevOps, where developers are involved in operational thinking and behaviors around self-deploying code – \u201cIf it\u2019s checked-in, it\u2019s ready to go to production\u201d<\/li>\n<\/ul>\n\n\n\n
A few of the technical changes included:<\/p>\n\n\n\n
\n- Move to a service-oriented architecture (SOA) and reduce dependence on monolithic artifacts<\/li>\n\n\n\n
- Develop self-service portals for consuming metrics around services<\/li>\n\n\n\n
- Enabling graceful degradation of services rather than 500 erroring the whole request load<\/li>\n\n\n\n
- Uptake of distributed database model vs monolith databases to reduce SPOF (single point of failure) risk<\/li>\n\n\n\n
- Switch from hardware to software-based load balancing<\/li>\n\n\n\n
- Centralized branch model to simplify the commit process compared to feature branches<\/li>\n\n\n\n
- Implemented multiple testing methodologies like pre-commit, PCS, and PCL <\/li>\n\n\n\n
- Develop an auto-remediation system to replace manual network operations center (NOC) processes \u2014 resulting in a career transition program for traditional operations roles into SRE<\/li>\n\n\n\n
- helping developers run A\/B tests to ensure backward compatibility of their commit before users would see it <\/li>\n\n\n\n
- support self-service deployment with a canary push to production<\/a> to ensure stability before rollout to the entire cluster <\/li>\n<\/ul>\n\n\n\n
Here’s a quick rundown of the cadence for self-service deployments for developers<\/strong> at LinkedIn:<\/p>\n\n\n\n\n- Canary to a single production instance<\/li>\n\n\n\n
- Check the automated metrics-based validation of success<\/li>\n\n\n\n
- Promote deployment to a single production data center<\/li>\n\n\n\n
- Promote deployment to remaining production centers<\/li>\n\n\n\n
- Ramp to an increasingly growing member base <\/li>\n<\/ol>\n\n\n\n
This self-service model has supported 15,000 commits and 600+ feature ramp-ups per day<\/p>\n\n\n\n
Since the self-service model is now well entrenched in the LinkedIn culture, SREs continue their work on various other problems. They work in areas like performance, platforms, resilience, architecture, etc.<\/p>\n\n\n\n
Each Site Reliability Engineer at LinkedIn is responsible for ~500 machines, but that can go up or down depending on the needs of the system on the day.<\/p>\n\n\n\n
\n\u201cIt is impressive to be able to handle this kind of [work]. This contributes to the challenge of finding the right people. Finding SREs who can operate at this level is a very big challenge.\u201d<\/p>\n\u2014 excerpt from video talk on SRE Hiring<\/a> by Greg Leffler, SRE manager at LinkedIn from 2012-2016<\/cite><\/blockquote>\n\n\n\n
LinkedIn’s SRE managers found that many SRE candidates sought work that would allow them to leave a legacy when they move on<\/strong>. <\/p>\n\n\n\n
This trait correlates with the capability to work with broad-spanning complex systems in ambiguous circumstances.<\/p>\n\n\n\n
If you can do out-of-the-box thinking and work in tough situations, you’d want that work to have a lasting impact.<\/p>\n\n\n
\ud83d\udd31 How SRE fits into LinkedIn’s engineering culture<\/h2>\n\n
Team formation<\/h3>\n\n\n
\n
- 300+ SREs across 4 global offices \u2014 Bay Area (San Francisco, Palo Alto), New York City, Bangalore<\/li>\n\n\n\n
- Team size depends on whether they’re embedded in a product team or part of a dedicated SRE team<\/li>\n\n\n\n
- SREs get to pick the kind of team they want to be part of <\/li>\n\n\n\n
- Hybrid roles exist i.e. SREs who focus on systems or software development<\/li>\n<\/ul>\n\n\n
Many SREs at LinkedIn as “embedded SREs”<\/strong><\/h4>\n\n\n
In the mid-2010s, many SREs were distributed across product engineering teams i.e. embedded into the product team. <\/p>\n\n\n\n
The reasoning behind this was that centralized teams eventually face issues that come with becoming a \u201cshared service\u201d i.e. not so critical to the day-to-day but only consulted if really necessary. <\/p>\n\n\n\n
The centralized model would not have been conducive to LinkedIn’s SRE needs at the time. The thing to remember is that SRE is a continuously high-involvement practice and not only for when things go wrong.<\/p>\n\n\n
LinkedIn has a large SRE base in India<\/strong><\/h4>\n\n\n
LinkedIn’s engineering management focused strongly on creating SRE teams in India. This may be partly due to hiring difficulties endemic to the SRE field in North America and Europe. <\/p>\n\n\n\n
There were initial difficulties as SRE had a stigma within talent pools in India. <\/p>\n\n\n\n
Many with years of experience perceived SRE as a relabelled or glorified systems administration role. LinkedIn’s SRE hiring managers instead focused on turning junior engineers and graduates into high-potential SREs. <\/p>\n\n\n\n
They emphasized from the outset that SRE was not a typical operations role and that it would involve broad-spanning work in ambiguous settings.<\/p>\n\n\n\n
As of 2017, the Bangalore office of LinkedIn had 60 SREs in 10 teams. <\/p>\n\n\n
History of SRE at LinkedIn<\/h3>\n\n
Prior to a modern SRE discipline (the early 2010s)<\/strong><\/h4>\n\n\n
In 2013, LinkedIn rebranded teams in the existing discipline of AppOps to SRE. The newly minted SREs worked alongside stratified operations teams in verticals such as systems, networks, applications, and DBA. <\/p>\n\n\n\n
This way of running software operations proved difficult as LinkedIn continued its growth trajectory. Several SREs noted issues like:<\/p>\n\n\n\n
\n
- Cooperation among these teams went only by way of tickets being sent to each other <\/li>\n\n\n\n
- There were walls not only between dev and ops but within ops itself<\/li>\n\n\n\n
- Developers did not have access to production and even most non-production environments – \u201chand me the code, and I will deploy it\u201d so ops were bogged down with release operations<\/li>\n\n\n\n
- Pager fatigue was extreme with operations people training significant others to keep an eye for alerts on the Blackberry while they slept (3 alerts every 5 minutes on average)<\/li>\n\n\n\n
- Mon-Wed 6-10am outages due to capacity issues and little visibility into demand due to no observability instrumentation<\/li>\n\n\n\n
- Dealing with outages that were so frequent, there was an outage every day of the calendar year<\/li>\n\n\n\n
- MTTR was 1500 minutes, meaning issues were not getting resolved on the same day<\/li>\n<\/ul>\n\n\n\n
SREs originally started off in this environment as firefighters but evolved during a drastic shift in the software operations at LinkedIn.<\/p>\n\n\n
Transitioning to SRE from previous operational discipline<\/strong><\/h4>\n\n\n
There was a need for change in the mid-2010s considering 100 million members relied on LinkedIn at the time. <\/p>\n\n\n\n
LinkedIn could not mess up its increasingly complex software operations.<\/p>\n\n\n\n
Something had to give, as LinkedIn was suffering frequent outages and performance issues despite now having an SRE team. <\/p>\n\n\n\n
LinkedIn SREs and the wider organization had to challenge several antipatterns:<\/p>\n\n\n\n
Antipattern identifier<\/strong><\/td> Antipattern description<\/strong><\/td> Shift to these propatterns\u2026<\/strong><\/td><\/tr> Firefighter \ud83d\udc68\ud83c\udffe\u200d\ud83d\ude92<\/td> \u274c react to handle incidents that happen to keep the company functioning one more day<\/td> \u2705 automate the manual work
\u2705 deliver instrumentation to get rich data on issues and alerts
\u2705 understand the stack at a deeper level<\/td><\/tr>Gatekeeper \ud83d\udc82\ud83c\udffd<\/td> \u274c control releases to protect the site from developers
\u274c \u201ctalk to me if you want to touch production\u201d
\u274c push a button to deploy someone else\u2019s work
\u274c have arbitrary schedules for releases<\/td>\u2705 develop automated gatekeepers to assure quality
\u2705 let developers own their work in production
\u2705 support developers in self-service deployment with the use of pre-launch checklists and release guidance <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe decision was then made across the organization to radically change several aspects of software operations. This included a drastic shift in software architecture and developer involvement in operations for the first time.<\/p>\n\n\n\n
To achieve this lofty goal, product development was stopped for 3 months<\/strong>. <\/p>\n\n\n\n\n\n
This signified a gutsy commitment to drive the necessary change.<\/p>\n\n\n\n
Several factors supported the transition to more effective software operations:<\/p>\n\n\n\n
\n
- The transition received solid management support from all affected VPs<\/li>\n\n\n\n
- The umbrella operations organization with sysadmins, network engineers, DB admins, etc. remained, but communication made it clear that it would not be business as usual <\/li>\n\n\n\n
- Major changes to how the software was architected came with support from experienced SREs and other seasoned engineers<\/li>\n\n\n\n
- Developers were increasingly involved in discussions about their ownership of software in production <\/li>\n<\/ul>\n\n\n
How the change was actioned toward a modern SRE discipline<\/strong><\/h4>\n\n\n
SREs initially advocated a few human-related practices to drive the change, including:<\/p>\n\n\n\n
\n
- encouraging and supporting developers in joining the on-call roster; some development teams got more into this, while others kept a distance from on-call <\/li>\n\n\n\n
- implement DevOps, where developers are involved in operational thinking and behaviors around self-deploying code – \u201cIf it\u2019s checked-in, it\u2019s ready to go to production\u201d<\/li>\n<\/ul>\n\n\n\n
A few of the technical changes included:<\/p>\n\n\n\n
\n
- Move to a service-oriented architecture (SOA) and reduce dependence on monolithic artifacts<\/li>\n\n\n\n
- Develop self-service portals for consuming metrics around services<\/li>\n\n\n\n
- Enabling graceful degradation of services rather than 500 erroring the whole request load<\/li>\n\n\n\n
- Uptake of distributed database model vs monolith databases to reduce SPOF (single point of failure) risk<\/li>\n\n\n\n
- Switch from hardware to software-based load balancing<\/li>\n\n\n\n
- Centralized branch model to simplify the commit process compared to feature branches<\/li>\n\n\n\n
- Implemented multiple testing methodologies like pre-commit, PCS, and PCL <\/li>\n\n\n\n
- Develop an auto-remediation system to replace manual network operations center (NOC) processes \u2014 resulting in a career transition program for traditional operations roles into SRE<\/li>\n\n\n\n
- helping developers run A\/B tests to ensure backward compatibility of their commit before users would see it <\/li>\n\n\n\n
- support self-service deployment with a canary push to production<\/a> to ensure stability before rollout to the entire cluster <\/li>\n<\/ul>\n\n\n\n
Here’s a quick rundown of the cadence for self-service deployments for developers<\/strong> at LinkedIn:<\/p>\n\n\n\n
\n
- Canary to a single production instance<\/li>\n\n\n\n
- Check the automated metrics-based validation of success<\/li>\n\n\n\n
- Promote deployment to a single production data center<\/li>\n\n\n\n
- Promote deployment to remaining production centers<\/li>\n\n\n\n
- Ramp to an increasingly growing member base <\/li>\n<\/ol>\n\n\n\n
This self-service model has supported 15,000 commits and 600+ feature ramp-ups per day<\/p>\n\n\n\n
Since the self-service model is now well entrenched in the LinkedIn culture, SREs continue their work on various other problems. They work in areas like performance, platforms, resilience, architecture, etc.<\/p>\n\n\n\n