What Is a SOC Runbook?

Step 4 of a log source onboarding runbook is one command, sudo filebeat test output. Step 5 is the line that command has to print before anyone is allowed to continue: talk to server... OK. That is the whole idea. A named task, an ordered set of commands, and a stated way to know each one worked.

Most SOCs say they have runbooks and mean they have a folder of alert notes. Those are playbooks. The two documents answer different questions, and teams that conflate them tend to write neither one well.

Runbook or playbook

A detection playbook answers "this alert fired, now what". It is triage logic: what the rule means, which fields to pull first, the branch where you contain and the branch where you close as benign. It is written per detection and it is full of decisions. We covered its shape in what a detection playbook is.

A runbook answers "how do we perform this task the same way every time". Onboarding a log source. Running the weekly rule health check. Isolating a host. Handing over at shift change. It is written per task and it is mostly free of decisions, because every decision left in a runbook is a place where two analysts will do the job differently.

The tell is the trigger. A playbook is triggered by an alert. A runbook is triggered by a request, a schedule, or a step inside a playbook. When a playbook says "isolate the host", the isolation procedure it calls is a runbook.

Anatomy of one that works

Six parts, in this order.

1. Trigger. The exact condition that starts the procedure, including who is allowed to start it.

2. Prerequisites and access. Every credential, group membership and tool needed, named before step 1 rather than discovered at step 6.

3. Numbered steps with the real command. Not "restart the agent" but the command line, copyable.

4. Verification per step. The output, exit code or query result that proves the step landed. Verification at the end only tells you something broke, not where.

5. Rollback. How to undo, and honestly what cannot be undone.

6. Escalation. A named owner and the threshold that hands the task over instead of improvising.

Log source onboarding is the runbook worth writing first, because it sits exactly where detection engineering meets SOC operations. Detections are written against fields that only exist if this procedure was followed.

A real runbook, start to finish

Trigger: an approved intake ticket naming the host group, the log path, and the detections that will consume the source.

Prerequisites: sudo on the host group, the output credentials in the deployment secret store, write access to the shipper config repo, and Kibana read on filebeat-*.

# 1. Record the starting state
filebeat version
sudo filebeat export config > /tmp/filebeat-before.yml
# verify: /tmp/filebeat-before.yml is non-empty

# 2. Enable the module
sudo filebeat modules enable system
sudo filebeat modules list
# verify: "system" appears under Enabled

# 3. Point the fileset at the log path
sudoedit /etc/filebeat/modules.d/system.yml   # auth.enabled: true, var.paths
sudo filebeat test config
# verify: "Config OK"

# 4. Test the output path
sudo filebeat test output
# verify: "talk to server... OK" and a version line

# 5. Apply
sudo systemctl restart filebeat
systemctl is-active filebeat
sudo journalctl -u filebeat -n 50
# verify: "active", and no ERROR lines since the restart

# 6. Emit a canary event
logger -p authpriv.notice "huntrule onboarding canary $(date +%s)"

Step 7 is the one people skip, and it is the only step that proves the source is actually usable. The agent being healthy is not the same as the data arriving with the right shape.

GET filebeat-*/_search
{
  "size": 1,
  "sort": [ { "@timestamp": "desc" } ],
  "query": {
    "bool": {
      "filter": [
        { "term":         { "event.dataset": "system.auth" } },
        { "match_phrase": { "message": "huntrule onboarding canary" } }
      ]
    }
  }
}

Verify three things in the hit, not one. The canary is present. host.name is populated and matches the inventory name rather than a short hostname. And @timestamp is within a couple of minutes of the clock on the shell that ran logger, which is how you catch a timezone or clock skew problem while it is still cheap.

Step 8 registers the host in the inventory list the silence detection reads. Step 9 enables the consuming detections and records either their first hit or an explicit zero, so that "no alerts" is a measured result rather than an assumption.

Rollback and escalation

sudo filebeat modules disable system
sudo cp /tmp/filebeat-before.yml /etc/filebeat/filebeat.yml
sudo systemctl restart filebeat

Rollback does not remove documents already indexed, and it should not. Say so in the runbook. A rollback section that quietly implies a clean slate is worse than no rollback section.

Escalation is a threshold, not a feeling. If step 7 returns nothing after 10 minutes, stop and hand the ticket to the platform owner. Do not proceed to step 8. Runbooks fail most often at the point where somebody continued past a failed verification because the next step looked independent.

The detection that watches the source

Onboarding creates something worth defending. The first rule to write is the one that fires when a shipper is deliberately stopped.

title: Log shipper service stopped or masked on a monitored Linux host
status: experimental
description: Detects systemctl being used to stop, disable or mask the log shipper on hosts that are registered as monitored sources.
logsource:
    product: linux
    category: process_creation
detection:
    selection:
        Image|endswith: '/systemctl'
        CommandLine|contains:
            - 'stop filebeat'
            - 'disable filebeat'
            - 'mask filebeat'
    condition: selection
falsepositives:
    - Agent upgrades driven by configuration management
    - The rollback section of the onboarding runbook itself
level: medium
tags:
    - attack.defense-impairment
    - attack.t1685

What it misses is the larger half. kill -9 on the process, apt remove filebeat, an edit to output.elasticsearch.hosts that points the shipper somewhere the SOC does not read, or a host that simply stopped booting. None of those touch systemctl. Absence of data is not a process event, so it needs a query rather than a rule.

GET filebeat-*/_search
{
  "size": 0,
  "query": { "range": { "@timestamp": { "gte": "now-24h" } } },
  "aggs": {
    "by_host": {
      "terms": { "field": "host.name", "size": 2000 },
      "aggs": { "last_seen": { "max": { "field": "@timestamp" } } }
    }
  }
}

That query has a blind spot severe enough to state plainly: a host that sent nothing at all in the window produces no bucket, so the host you most want to see is the one missing from the result. The output is only meaningful when diffed against the source inventory that step 8 of the runbook updates. This is the loop closing. The runbook feeds the inventory, the inventory makes the detection interpretable.

Runbooks rot when their dependencies move

Calendar review does not work. A quarterly review reads a runbook nobody has run and marks it current. Tie review to the thing the runbook depends on instead.

Give every runbook a machine-readable header naming its dependencies, then have CI compare the recorded values against reality on every pipeline run.

runbook: onboard-linux-auth-source
owner: detection-engineering
depends-on:
  - package: filebeat
    check: "filebeat version"
  - path: /etc/filebeat/modules.d/system.yml
  - index-pattern: "filebeat-*"
  - field: event.dataset
  - detection: log-shipper-service-stopped
last-executed: 2026-07-14
last-executed-by: analyst-initials

When filebeat version returns something different from the last recorded run, the runbook is flagged for review that week, not next quarter. Same when the index pattern changes or the referenced detection is retired. The signal is drift in a dependency, and drift is observable.

Untested documentation

A runbook nobody has executed top to bottom is documentation, not a procedure. It has never been proven to work. The fastest way to find out which of yours are in that state is to have someone who did not write it run it from step 1, in a lab, without asking the author a single question. Every question they need to ask is a defect in the runbook.

We wrote the onboarding runbook above against Filebeat on Linux and did not test it against Windows event forwarding or a syslog collector in front of the shipper. Both change the verification steps.

Summary

A runbook is the procedure for a recurring operational task, and a playbook is the response procedure for an alert. Keeping them separate is what lets each one be short. The parts that make a runbook usable are a trigger, stated prerequisites, real commands, verification after every step, an honest rollback, and a named escalation point. Tie review to dependency drift rather than to the calendar, because a runbook goes wrong when its tooling moves and not when a quarter ends. Then have someone else run it end to end, because until that happens it is untested documentation.

Verification commands for the log source onboarding runbook
  filebeat version
  sudo filebeat export config
  sudo filebeat modules list
  sudo filebeat test config      -> Config OK
  sudo filebeat test output      -> talk to server... OK
  systemctl is-active filebeat   -> active
  sudo journalctl -u filebeat -n 50
  logger -p authpriv.notice "canary"
  ES search on event.dataset:system.auth for the canary string
Fields to confirm on the first hit
  @timestamp, host.name, event.dataset, message

Rules for the shipper and audit tampering techniques above are in the catalog under /rules?domain=linux, and the silence query pairs with anything in /rules?severity=high that depends on a source staying alive.