SRE Helper

This page exists to help you decide what to do next around adopting ideas from SRE.

Simply answer questions and expand the tree to find your own path!

Do you want to start implementing some ideas from SRE?
- No
  - Fair enough, but if you are still on the fence, maybe take a look at the free books from Google. Pop back here when you are ready!
- Yes
  - Is your highest priority (right now) to improve service quality and resilience, to reduce manual work required by your services (aka Toil), or to reduce costs?
  - Improve service quality
    - Do you have a definition of what your service is and who experiences first-hand the quality of your service?
    - No
      - Start by writing it down and getting feedback from your team and stakeholders.
    - Yes
      - Do you have metrics that describe the most important characteristics that impact the perceived quality of your service?
      - No
        
        You need to go back to NFRs, SLAs, specification documents, and then your Product Owner / a representative of the people who first-hand experience the quality of your service.
      - Yes
        
        Then these metrics will form the basis of your Service Level Indicators (SLIs).
        
        Do you have your SLIs documented including how to calculate them as percentages (where 100% means everything was good enough)?
        
        No
        
        Then you need to refine and document your SLI definitions.
        
        Yes
        
        Then you just need to ensure you have enough coverage of these SLIs, i.e. if there are ways that your service quality could become too low without those SLIs noticing, go back and create more SLIs.
        
        Do you have target objectives for what values your SLIs should be at over the course of a month (aka do you have Service Level Objectives / SLOs)?
        
        No
        
        You need to your Product Owner (or Service Owner?) to set and agree your SLO targets.
        
        Yes
        
        Do you have an agreed and implemented approach to measuring the SLIs for your service?
        
        No
        
        You need to agree how and where to measure these things, get the metrics implemented, and ensure there is a process for generating them.
        
        Yes
        
        Are your SLO scores used as part of your monthly (end of SLO measurement window) service reviews?
        
        No
        
        You need to start making sure these metrics are used as part of how quality of service is recognised and managed.
        
        Yes
        
        When an SLO is breached does this lead to work that will prevent this happening in future being made a top priority?
        
        No
        
        Your SLOs aren't having the impact they should. You need to get agreement that SLO breaches leads to automatic prioritisation.
        
        Yes
        
        Are your SLIs measured over shorter windows during the SLO window (perhaps in conjunction with tracking your "Error Budget" spend) to proactively try to hit SLOs?
        
        No
        
        You need to integrate the tracking of SLIs, Error Budget spend, and forecasted SLOs as part of your support work.
        
        Yes
        
        Looking good! You have a commonly understood definition of how to evaluate service quality (SLIs), you have service quality targets (SLOs), your SLIs are measured, service quality (i.e. SLO performance) drives your service quality. and SLI performance (possibly Error Budget spend) is measured and used proactively to try to ensure SLOs are hit. SLIs and SLOs should be creating an effective feedback loop to balance service quality with your other change priorities.
        
        Do you use SLIs, SLOs and Error Budgeting as part of delivering new features and as part of evaluating resilience?
        
        No
        
        Sounds like it's time to shift left your use of SLIs and SLOs and make them a native part of delivery as opposed to something you retrofit.
        
        Yes
        
        Just wow, please share your experiences with others! (And maybe think next about Toil reduction.)
  - Reduce Toil
    - Do you have a clear intention on how you want to reduce third party spend and/or improve the lives of colleagues?
    - No
      - Start by determining this as it will inform the extent to which you should be looking to reduce Toil, what type of Toil you want to be eliminating, how you might get Toil reduction priorities (e.g. there may be a commercial aspect to it). Using the Value stream mapping technique can help uncover and priortize toil.
    - Yes
      - If colleagues perform Toil work, ensure you have a charter that makes a commitment to them about what percentage of their workload should be made up by Toil (Google aim for 50% but that's just one example). Make sure it is clear whether this target is currently realistic and make a commitment to when it will be achieved.
      - Do you have a definition of what work should be classified as Toil? This doesn't just mean the Google book definition, it should be specific to your context with examples.
      - No
        
        Create a definition, review it with your team, test it against recent work - is it clear what was Toil and what was not? Capture examples. Be prepared for this definition to be refined over time.
      - Yes
        
        Do you have an approach to quantifying what Toil is currently getting done?
        
        No
        
        Create one! This doesn't have to be incredibly precise and comprehensive, we're looking for an indication of the most prevalent, costly, and risky (often Toil involves risky work like running privileged system commands) work.
        
        Yes
        
        Assuming you are using your approach, and now have some data about what specific Toil tasks seem to be most prevalent, costly, and risky do you have an approach to prioritising which specific Toil to go after reducing?
        
        No
        
        Create a prioritisation method. They key inputs are: cost spent performing one instance of the task, frequency task is needed, any impact to users (for example a customer/user waits for task to complete), risks for task to complete. A pseudo-scientific model (i.e. similar to a risk prioritisation matrix) should suffice.
        
        Yes
        
        Having defined Toil and prioritised some Toil to reduce, do you have the skills, authority (perhaps the Toil is a function of someone else's service) and capacity?
        
        No
        
        You need to work out how to address getting Toil reduction prioritised.
        
        Yes
        
        Excellent you have a strategy for reducing Toil, you know how to recognise it, you know how to prioritise reducing it, and you have a means for reducing it.
        
        If you have made a commitment to colleagues about how much Toil they should have to perform are you delivering on it (or are they doing too much Toil still)?
        
        No
        
        You need to find other ways to prevent colleagues doing the Toil or at least revise the commitment and expectations around Toil.
        
        Yes
        
        Just wow, you have a well-oiled approach to Toil Please share your experiences with others! (And maybe think next about service quality improvement.)
  - Reduce costs
    - From a core SRE perspective you have two levers:
    - So go back a step and pick Improve Service Quality or Reduce Toil!
    - NB. there are other cost saving levers that impact the above ideas. For example you may currently run your service with a low day rate outsourced team. Your strategy may be to reduce Toil and therefore the number of people required, or instead it might be to reduce Toil to the point that your inhouse Engineers can pick it all up cost effectively.