# SRE Automation Tools: What to Automate and Which Tools Help > SRE automation covers four different jobs: runbook automation, self-healing infrastructure, drift detection, and resilience testing. Source: https://last9.io/blog/sre-automation-tools/ "SRE automation" gets used as one umbrella term for at least four different jobs: running a fixed sequence of steps on demand (runbook automation), a system correcting itself without a human involved (self-healing infrastructure), catching when real infrastructure no longer matches what was declared in code (drift detection), and deliberately breaking things to find weaknesses before an incident does (resilience testing). The tools built for one of these rarely do the other three well, so picking "an SRE automation tool" without naming which job you need first usually means buying the wrong thing. If you're looking for the broader reliability stack beyond automation specifically, [Last9's reliability tools guide](https://last9.io/blog/reliability-tools/) covers monitoring, alerting, and DevOps tooling as a set. If the question is about AI applied to the incident workflow rather than automation in general, [what AI-SRE means](https://last9.io/blog/what-is-ai-sre/) is a separate, more specific topic. For the stage-by-stage view of what to automate during an incident, see the [incident response automation playbook](https://last9.io/blog/incident-response-automation-playbook/). ## What counts as SRE automation? Four categories cover most of what teams mean when they say this: - **Runbook automation** turns a documented, repeatable procedure (restart a service, rotate a credential, scale a resource) into something triggered by a command or an alert instead of typed by hand each time. - **Self-healing infrastructure** goes a step further: the system itself detects a deviation from the desired state and corrects it without anyone triggering anything. - **Drift detection** catches a specific, narrower problem: infrastructure that was changed outside your normal process (a manual console edit, an emergency fix) so it no longer matches what your code says it should be. - **Resilience testing** is the odd one out because it isn't reactive at all. It automates injecting failure on purpose, in a controlled way, to find out what breaks before a real incident does. Confusing these categories is how teams end up disappointed with a tool. A drift detection tool that only alerts you to a mismatch is not broken if it doesn't also fix the mismatch. That was never its job. ## Which failure modes should you automate first? Google's SRE workbook defines the target of this kind of automation directly: toil, "the repetitive, predictable, constant stream of tasks related to maintaining a service." The [same chapter](https://sre.google/workbook/eliminating-toil/) gives six criteria for recognizing it, noting that a source of toil may not meet all six: - Manual: it requires a person to do it, not just approve it - Repetitive: it happens more than once, not a one-time task - Automatable: a script or system could do it as well as a human - Nontactical and reactive: it is interrupt-driven, a response to an alert rather than planned work - No enduring value: doing it doesn't make the underlying problem less likely to recur - Grows with the system: the more infrastructure or traffic you have, the more of this work shows up A task that scores high on most of these, restarting a specific service after a specific known failure, for example, is a strong automation candidate. A task that's manual but has enduring value, like a real architecture review, is not toil, and automating it away usually removes the value along with the effort. ## What tools handle runbook automation? [Rundeck](https://www.rundeck.com/) (now developed under PagerDuty as PagerDuty Runbook Automation) and [StackStorm](https://docs.stackstorm.com/) both turn a documented procedure into something that runs on a trigger instead of a person following a checklist. StackStorm describes itself as a platform for "taking actions in response to events." Its rules engine connects a trigger (an alert, a webhook, a schedule) to a defined workflow of actions, and its documented use cases include automated remediation and ChatOps-style execution from a chat command. The distinction worth holding onto: this category executes a known, written procedure faster and more consistently than a human typing commands during an incident. It doesn't decide what the procedure should be. Writing a bad runbook and automating it just means the mistake happens faster. ## What tools handle self-healing infrastructure? The clearest built-in example doesn't require installing anything extra if you're already running Kubernetes. A [Kubernetes controller](https://kubernetes.io/docs/concepts/architecture/controller/) runs a non-terminating control loop, the documentation uses a thermostat as the analogy, that continuously compares the current state of the system against the desired state and acts to close the gap. If a Deployment specifies three replicas and one pod crashes, the controller notices the mismatch and starts a replacement without a person paging anyone. This is genuine self-healing, but it's healing a narrow, predefined kind of deviation: pod count, not "the application is slow" or "the database is under load." StackStorm's auto-remediation workflows extend the same detect-and-correct pattern outside Kubernetes specifically, to infrastructure and application-level failures that have a known, scriptable fix. ## What tools handle infrastructure drift detection? [HCP Terraform's health assessments](https://developer.hashicorp.com/terraform/cloud-docs/workspaces/health) are a good example of drift detection done well, and a good example of where the category's limits are. The feature identifies when infrastructure changes were made outside Terraform's normal process, so the real infrastructure and the declared configuration no longer match, and separately runs continuous validation checks (a certificate is still valid, an endpoint still returns a healthy status) against custom assertions written into the configuration. What it does not do is fix the drift automatically. HashiCorp's documentation says it "proposes the necessary changes" to bring the infrastructure back in sync and sends notifications to the team. A person then decides whether to reapply the original configuration or update the code to accept the new state. That's not a limitation to work around. Auto-applying infrastructure changes without review is its own risk, and detection plus a human decision is often the safer default even where technically automating the fix would be possible. ## What tools handle resilience testing? [Gremlin](https://www.gremlin.com/chaos-engineering) automates the practice usually called chaos engineering: deliberately injecting failure (killing a process, adding network latency, exhausting a resource) into a system in a controlled, scoped way to see whether it degrades gracefully or falls over. This is automation aimed at a different moment than the other three categories. Runbook automation, self-healing, and drift detection all respond to something that already went wrong or already drifted. Resilience testing tries to surface the failure before it happens somewhere that matters, on a schedule and blast radius the team controls rather than during a real incident. ## Is this different from AI-driven SRE automation? Yes, and the distinction matters for evaluating a tool. Everything above executes a known, defined response: correcting a specific replica count, running a specific documented script, resynchronizing a specific config drift. AI-SRE tooling sits on top of this layer and adds pattern recognition and root-cause suggestion: connecting a spike in errors to the deployment that likely caused it, or summarizing what changed across a set of signals before a human opens a dashboard. It's a separate capability, not a more advanced version of runbook automation, and [a closer look at what AI-SRE means](https://last9.io/blog/what-is-ai-sre/) covers that distinction on its own. ## Name the toil before you name the tool The categories above solve different problems, and none of them substitute for the others. Before evaluating a specific product, name the actual task: a known procedure that needs to run faster (runbook automation), a narrow deviation that needs correcting on its own (self-healing), infrastructure that might have drifted from code (drift detection), or an untested assumption about how the system fails (resilience testing). Automation is only as good as the alert or signal that triggers it. [Last9's alerting](https://last9.io/alerting/) uses anomalous pattern detection to cut the false positives that static thresholds produce, and correlates config and environment changes with system health. That makes the alert that fires a runbook or a self-healing workflow less likely to be noise. ## FAQ ### What's the difference between runbook automation and self-healing infrastructure? Runbook automation executes a documented procedure when triggered, by a person, an alert, or a schedule, but the trigger and the response are both explicitly defined in advance. Self-healing infrastructure removes even the trigger step: a control loop continuously checks the system against its desired state and corrects deviations on its own, without a person or a runbook initiating the specific action each time. ### Does infrastructure drift detection fix the drift automatically? Not by default in most tools, including HCP Terraform's health assessments. Drift detection identifies that real infrastructure no longer matches the declared configuration and alerts the team, but resynchronizing it, either by reapplying the original config or updating the code to match the new state, is typically a decision a person makes rather than something the tool does unattended. ### What is toil in SRE, and why does it matter for choosing automation tools? Toil is manual, repetitive operational work that doesn't provide enduring value: doing it once doesn't reduce how often it needs doing again. Google's SRE workbook uses this definition specifically to separate work worth automating from work that only looks similar, like a genuine architecture review, which is manual and infrequent but does have lasting value and shouldn't be automated away. ### Is chaos engineering the same thing as incident response automation? No. Incident response and runbook automation react to something that has already gone wrong. Chaos engineering, the practice tools like Gremlin automate, deliberately injects failure into a system on a controlled schedule before anything has broken, specifically to find weaknesses while the team is watching and prepared, rather than during a real incident. ### Can Kubernetes' built-in self-healing replace a dedicated auto-remediation tool? For the specific failures Kubernetes controllers already watch, replica counts, node health, container restarts, no separate tool is needed. It doesn't extend to failures outside that scope, like an application-level error rate or a slow downstream dependency, which is where a broader auto-remediation tool like StackStorm's workflow engine can pick up failures Kubernetes itself has no visibility into.