# Site Reliability Engineering: A Practical Introduction for Engineers

In 2003, Benjamin Treynor Sloss joined Google to lead a production team of seven engineers. He approached the job as a software engineer: instead of relying on people to repeat operational tasks, build software to do that work. That team became Google’s Site Reliability Engineering organization, applying engineering to the practical problem of keeping services running reliably.

Sloss also coined the name. As [Google’s SRE book explains](https://sre.google/sre-book/preface/), **“site” originally meant the** [**google.com**](http://google.com) **website** that the team helped keep running. The discipline now extends beyond websites to services such as storage and cloud infrastructure. **Reliability** is the focus: keeping those services dependable for their users. **Engineering** is the approach: designing and building software and systems to achieve that, rather than relying on repeated manual intervention.

The problem behind that approach is familiar: as a service grows, its operational workload can grow with it. Hiring more people to handle the same recurring tasks does not remove that work. SRE makes improving the system part of running it—automating repeatable operations, addressing recurring failures and protecting time for engineering. Sloss captures the idea in [Google’s SRE book](https://sre.google/sre-book/introduction/): “SRE is what happens when you ask a software engineer to design an operations team.”

Consider a fictional online store. A customer submits an order through its website; the website calls a checkout API, which records order state and contacts a payment provider. If the response disappears, the customer may not know whether to try again. The business needs more than a running server: it needs a trustworthy outcome and a team able to recover when things go wrong.

We will follow that service from design through launch, an incident, and the engineering work afterwards. The outcome is a small service/reliability map connecting user expectations, technical decisions, signals and ownership. You need basic familiarity with HTTP services and software releases, but no particular cloud or monitoring platform.

![Illustrative checkout service and reliability responsibilities across design, launch, observation, response and improvement.](https://cdn.hashnode.com/uploads/gql/6ad858beca6b1f0a715e21e0/37fdeb55-3463-49f1-a007-47a54ec6496c.png align="center")

*Illustrative service: customer → website → observed entry point → checkout API → order database and payment provider. The API observation includes the effects of downstream dependencies, but not failures before the entry point or the complete shopping journey.*

## What is site reliability engineering?

Site reliability engineering (SRE) applies software engineering to running reliable services. It combines production responsibility with work that changes the system itself, so reliability and operational effort do not depend on continually adding people to perform the same tasks. This is central to Google's description of SRE.[1](https://sre.google/sre-book/introduction)

For checkout, answering an urgent page is necessary operational work. Changing the payment integration so a known failure no longer requires repeated manual intervention is engineering work. Both belong in the picture; the second must not disappear beneath the first.

SRE is therefore not a new label for a ticket queue or a team that receives finished software and absorbs every production problem. The Google model explicitly protects engineering time and uses production experience to improve the service.[1](https://sre.google/sre-book/introduction) This is not a claim that operations engineers cannot write software or improve systems. The distinction is the operating model, incentives and protected capacity—not professional status.

There is overlap with DevOps: Google's introduction describes compatible principles, including collaboration across development and operations, automation and engineering practices applied to operations.[1](https://sre.google/sre-book/introduction) For this article, think of SRE as a concrete way to organize reliability work, not as a replacement name for all development or operations.

## What does this service need to promise?

Start with the customer's action and its consequences, then choose measurable objectives. For checkout, “the servers are available” is too weak: the customer needs a correct, timely response and an order state that remains understandable if communication fails.

In our hypothetical design, a valid checkout result is an accepted-order confirmation or a legitimate business decline. A payment-provider timeout is neither: the outcome remains uncertain until reconciliation establishes what happened. We also require the design to prevent duplicate processing of the same logical order, rather than asking customers or operators to resolve ambiguity by repeatedly submitting it.

An **SLI**, or service level indicator, measures a service outcome; an **SLO**, or service level objective, sets a target for that measure. An **error budget** expresses the permitted shortfall and helps prioritize reliability work against other changes. Google's workbook treats these as decision-making tools, not merely reporting metrics.[7](https://sre.google/workbook/implementing-slos)

For a first checkout objective, the team could measure the proportion of eligible API attempts receiving a valid result within an agreed time, over an agreed window. Product and engineering must choose the target together. Here, we leave its exact percentage and measurement rules to a separate design exercise rather than pretend one number suits every store.

The existing [SRE fundamentals article](https://blog.turingts.com/sre-fundamentals-designing-slis-slos-and-error-budgets) works through checkout measurement definitions, Prometheus queries and error-budget decisions in detail.[8](https://blog.turingts.com/sre-fundamentals-designing-slis-slos-and-error-budgets) Use it as optional further reading on measurement, not as a prerequisite for this lifecycle introduction.

Our boundary matters. An edge-observed API result can include a failure caused by the database or payment provider, but cannot describe every browser, DNS or shopping-flow failure. Google's workbook explicitly distinguishes an SLI specification from its implementation, with different coverage and cost trade-offs.[7](https://sre.google/workbook/implementing-slos) Nor would a good availability percentage excuse lost orders or duplicate charges; those require separate correctness checks in this design.

## How does reliability change the design?

Turn each failure concern into an explicit design decision before launch. In this example, reliable checkout requires defined dependency behaviour, durable order state, safe change controls and a named owner—not simply more replicas of the API.

Use the following map as a working agreement. Every row is a proposed choice for this fictional service, not a deployed system or a universal checklist.

| User or team need | Engineering decision | Evidence to inspect | Accountable owner in this example |
| --- | --- | --- | --- |
| Receive a usable checkout result | Define outcomes and a request deadline; bound dependency calls | Valid-result rate, response times, timeouts | Checkout team |
| Avoid duplicate order processing | Persist a logical-order identifier and state; reconcile uncertain payment outcomes | Duplicate-prevention tests and reconciliation exceptions | Checkout team with payment-integration maintainer |
| Release without exposing every customer at once | Use staged rollout and a compatible rollback path | Outcome changes by release; rollback rehearsal | Release owner in checkout team |
| Recover without improvising | Maintain a short runbook and escalation route | Response exercise and incident timeline | On-call lead |
| Keep operational work sustainable | Measure recurring reconciliation effort; engineer away repeatable cases | Human interventions and unresolved cases | Checkout engineering lead |

Take the uncertain-payment row. Suppose the provider processes a request but its response never reaches checkout. A blind retry is not our recovery design. We propose storing the operation identifier and an explicit unresolved state, then checking the outcome only through mechanisms supported by the provider's actual contract. Any retry must respect that provider's actual idempotency contract; an identifier in our database alone cannot establish its behaviour.

This is a design requirement to implement and test, not working payment code. It exposes a useful question early: what evidence lets us tell the customer what happened without creating another charge?

Google's engagement model explains why SRE involvement can start during design and implementation, rather than only when a service is handed over after launch.[6](https://sre.google/sre-book/evolving-sre-engagement-model) The concrete benefit here is the opportunity to resolve that question while the state model and API are still being designed.

![Illustrative SRE involvement across architecture and design, active development, limited availability, general availability and deprecation.](https://cdn.hashnode.com/uploads/gql/6ad858beca6b1f0a715e21e0/ef0f20cc-81a1-41d0-a775-e792e86435ba.png align="center")

*Conceptual engagement pattern based on Google’s SRE Workbook.*[*11*](https://sre.google/workbook/engagement-model/) *SRE can become involved at any stage. Curve height and phase lengths are illustrative—not measured effort or a mandatory staffing schedule. The chapter also describes the later abandoned and unsupported states.*

## What should be ready before launch?

Before exposing customers to checkout, make sure someone can detect user harm, take a safe recovery action, and explain who owns the next decision. Production readiness is a service-specific engineering review, not a document that transfers every risk to an operations team.

For this service, the developers and reliability engineer would review dependency failure tests, rollback compatibility, capacity assumptions and recovery access together. They would rehearse a payment timeout and ask whether the operator can distinguish a failed request from an uncertain payment. They would also check that restoring order data is an exercised procedure rather than merely a claim that backups exist.

Google describes production readiness reviews, training and progressive transfer of production responsibilities, with continued development-team support.[6](https://sre.google/sre-book/evolving-sre-engagement-model) That is useful guidance, not a requirement to copy Google's team structure. Our checkout team retains application ownership; an SRE may advise it or share production responsibility under an explicit agreement.

Before a sales or marketing promotion increases demand, revisit expected traffic and test the relevant limits. Google's introduction includes demand forecasting, load testing and capacity planning in reliability work.[1](https://sre.google/sre-book/introduction) A reasonable checkout question is whether database connections or payment-provider limits will constrain the service before API compute does. Adding API instances is not our default answer to an unidentified bottleneck.

## How do you know customers are being affected?

Observe the checkout outcome first, then collect enough internal evidence to explain it. Google's monitoring guidance separates symptoms from causes and identifies four useful signals: latency, traffic, errors and saturation.[3](https://sre.google/sre-book/monitoring-distributed-systems)

For checkout, that means watching how many eligible requests arrive, whether they receive valid results, how long those results take, and whether constrained resources are approaching their limits. A fast error must not make the successful-response picture look healthier. Google's guidance specifically distinguishes successful and failed latency, and notes that an HTTP success code can still carry incorrect content.[3](https://sre.google/sre-book/monitoring-distributed-systems)

Our proposed instrumentation would record outcome metrics, diagnostic events for order-state transitions, and request traces showing time spent in dependency calls. These would summarize outcomes, explain state changes and connect the work of individual requests. We would collect evidence to answer those questions, not everything a tool supports. Payment credentials and customer details do not belong in these diagnostic records.

The operational question is not “which dashboard should someone watch all day?” For this design, sustained customer-visible checkout failures should reach the on-call engineer with the affected operation, a diagnostic starting point and the runbook. A gradual capacity concern might become planned work instead. Google's monitoring chapters distinguish urgent alerts from tickets and emphasize understandable, low-noise paging; immediate action may be needed for a current or imminent problem.[1](https://sre.google/sre-book/introduction)[3](https://sre.google/sre-book/monitoring-distributed-systems)

Also check the observer. A silent stream of outcome data cannot establish that checkout is healthy. In our design, telemetry freshness and an independent synthetic check help distinguish quiet traffic from missing visibility; synthetic results remain separate from real customer outcomes.

## What happens during a checkout incident?

During an incident, reduce customer harm, coordinate changes and preserve evidence. Google's incident guidance separates command, operational work and communication, and recommends preparing and practising the response before an emergency.[4](https://sre.google/sre-book/managing-incidents)

Imagine a new checkout release coincides with increased payment timeouts. This is a fictional incident, not a report of an observed deployment. The on-call engineer sees degraded API outcomes, while dependency timing and release information suggest where to investigate. Correlation with a release does not establish the cause.

A useful first runbook is short:

1. **Confirm impact:** inspect checkout outcomes and telemetry freshness; record what is known and unknown.
2. **Stop expansion:** pause the rollout and identify its owner. Keep changes coordinated in one incident channel.
3. **Choose a safe mitigation:** consider rollback only after checking schema and payment-state compatibility. Escalate when that safety is uncertain.
4. **Protect unresolved orders:** do not replay payment requests indiscriminately. Preserve identifiers and hand ambiguous cases to the reconciliation owner.
5. **Verify recovery:** inspect customer-facing outcomes and unresolved work, then communicate the remaining impact.

The incident lead coordinates decisions; a responder performs agreed changes; someone keeps stakeholders informed and maintains a timeline. A small team may combine roles, but it should still make the responsibilities explicit. The person investigating a dependency should not also have to answer every status request.

A rollback can restore new requests without resolving payments already in an uncertain state. In our scenario, recovery therefore has two parts: restore usable checkout and account for affected orders. “The graph is green” is not sufficient closure.

## How does an incident become engineering work?

A postmortem should explain impact, contributing conditions, response and preventive actions without blaming the person who happened to deploy or respond. Google's guidance treats this as a learning process and stresses reviewing whether the action plan is appropriate.[5](https://sre.google/sre-book/postmortem-culture)

For our fictional incident, the investigation might find that a changed dependency timeout escaped testing, the rollout lacked an effective stop condition, and the runbook did not explain unresolved payments. Those are hypotheses to investigate, not conclusions supplied by the incident narrative.

Useful follow-up work could include a test where payment processing succeeds but the response is lost, a rollout check for worsening checkout outcomes, and a documented reconciliation path. Give each accepted action an owner and a way to demonstrate completion. “Be more careful” changes neither the software nor the information available to the next responder.

This also exposes **toil**. Google defines toil as production work that tends to be manual, repetitive, automatable, tactical, without enduring value, and growing with the service.[2](https://sre.google/sre-book/eliminating-toil) Suppose an operator repeatedly checks uncertain orders in two systems and copies outcomes into a spreadsheet. Resolving one case may be necessary, but doing the same work again does not improve the system.

### Why automate repetitive operational work?

Automation matters when it removes a recurring dependency on human effort. Resolving one case helps one customer; improving how that class of cases is handled can reduce future work. Google's SRE model protects engineering time precisely to avoid operational effort growing with the service.[1](https://sre.google/sre-book/introduction)

For our checkout example, more orders could mean more manual payment checks. Adding people to repeat those checks leaves the underlying work intact. Automating the repeatable evidence gathering—or changing the integration so fewer outcomes become uncertain—would address the source of the workload. These are proposed improvements, not measured savings.

The engineering response could be a reconciliation tool that gathers evidence, proposes safe transitions and leaves ambiguous cases for review. It needs tests, bounded permissions, an audit trail and a stop mechanism. Automating an unsafe decision would be a worse outcome than retaining the manual step.

Automation also takes effort to build, test and maintain. For this team, prioritize recurring work whose removal justifies that investment; do not automate a poorly understood decision simply because it is manual. Investigating a novel failure or judging an ambiguous payment can require human expertise. Google's toil definition explicitly distinguishes work requiring judgment from automatable repetition.[2](https://sre.google/sre-book/eliminating-toil)

Measure the change using recurring human interventions, time spent and unresolved cases—not merely whether a script exists. Google's toil chapter explicitly notes that repeatedly launching a script can still be toil.[2](https://sre.google/sre-book/eliminating-toil) Reducing unnecessary work, rather than accumulating automation, is the objective.

### What is a healthy balance between operations and engineering?

Protect engineering time rather than letting operational work consume the entire schedule. Google's benchmark caps aggregate operational work at 50%, reserving at least the other half for development. Treat that as a useful reference, not a universal staffing formula.[1](https://sre.google/sre-book/introduction)

The cap is a ceiling, not a quota to fill with manual work. It is also **not “50% automation and 50% manual work”**: engineering includes design, reliability features and lasting systems improvements. Google's toil chapter separately sets a goal of keeping toil below 50% of each SRE's time, with at least half spent on engineering projects.[2](https://sre.google/sre-book/eliminating-toil)

Look at the trend, not one incident-heavy week: Google's guidance assesses engineering time over several quarters or a year.[2](https://sre.google/sre-book/eliminating-toil) For checkout, record recurring checks and interruptions alongside engineering work. If operations repeatedly crowds out improvements, agree protected capacity and choose a recurring cause to remove. Keep necessary human judgment and production experience; the goal is sustainable operations, not zero manual activity.

![Illustrative comparison of repetitive manual checkout checks with engineering improvements, plus Google’s reference for protecting engineering time.](https://cdn.hashnode.com/uploads/gql/6ad858beca6b1f0a715e21e0/ed2ca78d-8cc3-49fd-95a7-3e31e2e87b41.png align="center")

*Illustrative design, not measured savings: automate repeatable work while retaining human judgment. The time-allocation reference is Google’s operational-work ceiling and protected engineering capacity—not a universal staffing rule or a split between automation and manual work.*[*1*](https://sre.google/sre-book/introduction)[*2*](https://sre.google/sre-book/eliminating-toil)

## Where should you start?

Start with one service and one important user action. Write its promise, dependencies, failure signals, recovery action and accountable owner on a single page. Then walk through a failure with someone who did not write the page and use the gaps to choose the next engineering task.

For checkout, try the lost-payment-response scenario as a tabletop exercise. Can your colleague identify the uncertain order, explain which actions are unsafe, find the escalation contact and describe the evidence required before closing the incident? No production fault injection is necessary to discover an unclear recovery contract.

This is where design, development and operations meet. SRE makes reliability a continuing engineering responsibility: define the outcome, build for failure, observe actual behaviour, recover deliberately, and improve the service so the next shift has less avoidable work.

## Frequently asked questions

### Do you need a separate SRE team?

Not to use the practices in this example. Keep service ownership explicit and dedicate engineering capacity to reliability. Google's engagement guidance itself describes consultation and documentation as alternatives to full SRE support.[6](https://sre.google/sre-book/evolving-sre-engagement-model)

### Is SRE mainly about monitoring and being on call?

No. Those provide operational visibility and response, but Google's SRE model also covers engineering, change management, capacity and efficiency.[1](https://sre.google/sre-book/introduction) For checkout, improving payment-state handling can matter as much as adding an alert.

### Should you automate every operational task?

No. Prioritize repeated work whose removal creates durable value. Some toil is unavoidable, and Google's guidance does not define all manual work as toil.[2](https://sre.google/sre-book/eliminating-toil) In our checkout example, ambiguous payment decisions stay under human review until safe behaviour is established.

## Sources

- [1](https://sre.google/sre-book/introduction) — Introduction
- [2](https://sre.google/sre-book/eliminating-toil) — Eliminating Toil
- [3](https://sre.google/sre-book/monitoring-distributed-systems) — Monitoring Distributed Systems
- [4](https://sre.google/sre-book/managing-incidents) — Managing Incidents
- [5](https://sre.google/sre-book/postmortem-culture) — Postmortem Culture: Learning from Failure
- [6](https://sre.google/sre-book/evolving-sre-engagement-model) — The Evolving SRE Engagement Model
- [7](https://sre.google/workbook/implementing-slos) — Implementing SLOs
- [8](https://blog.turingts.com/sre-fundamentals-designing-slis-slos-and-error-budgets) — SRE Fundamentals: Designing SLIs, SLOs, and Error Budgets
- [9](https://sre.google/sre-book/introduction/) — Introduction: byline and origin of SRE
- [10](https://sre.google/sre-book/preface/) — Preface: SRE name and site meaning
- [11](https://sre.google/workbook/engagement-model/) — SRE Engagement Model (The Site Reliability Workbook, chapter 18)

## FAQ

### What is Site Reliability Engineering (SRE)?

Site Reliability Engineering (SRE) applies software engineering to ensure reliable services. It combines production responsibility with system improvements, avoiding reliance on manual tasks.

### How does SRE differ from DevOps?

SRE and DevOps share principles like collaboration and automation, but SRE specifically organizes reliability work, protecting engineering time to improve systems rather than just maintaining them.

### What should be ready before launching a service?

Before launch, ensure detection of user harm, safe recovery actions, and clear ownership of decisions. Conduct reviews of dependencies, rollback compatibility, and rehearse recovery scenarios.

### Why automate repetitive operational work in SRE?

Automation eliminates recurring manual effort, preventing operational workload from growing with the service. It focuses on improving systems rather than just resolving individual cases.

### How does SRE handle incidents?

During incidents, SRE focuses on reducing harm, coordinating responses, and preserving evidence. It involves clear roles for decision-making, operational changes, and communication.

### What is the role of SLIs and SLOs in SRE?

SLIs (Service Level Indicators) measure service outcomes, while SLOs (Service Level Objectives) set targets for these measures. They guide reliability work and prioritize improvements.