What Is Alert Fatigue in SOCs?

Close every alert from a rule that fires 200 times a day and catches one intrusion a year, and you are right 99.975 percent of the time. That is the whole problem in one line. The analyst who dismisses everything from that rule has better measured accuracy than the analyst who investigates carefully, and no feedback loop in the SOC will tell either of them apart until incident response opens a case.

Alert fatigue gets described as burnout, and the remedies offered are resilience training, wellness budgets and rotation schedules. None of those change the arithmetic. Fatigue is what a detection pipeline produces when nothing in it carries a precision requirement. Bloomberg Businessweek reported in March 2014 that Target's FireEye deployment generated alerts on the 2013 intrusion and that they were not acted on. The tooling worked. The queue around it did not.

The base rate does the damage

Take a fleet of 10,000 Windows endpoints producing roughly 200 process creation events per host per day. That is 2,000,000 events a day for a rule to evaluate. Assume one intrusion per year that this rule is genuinely capable of catching, and assume it produces about 20 matching process creation events over its dwell time. Recall of 0.9 means the rule sees 18 of them.

Every number below is illustrative. The shape of the curve is not.

Three orders of magnitude separate a rule that is worthless from a rule an analyst can trust. A false positive rate of one in a million events, which sounds like an excellent rule in isolation, still yields an alert that is wrong 97.6 times out of 100. Intuition is useless here because the base rate is invisible. Nobody sees 2,000,000 events. They see 200 alerts.

What the queue does to triage

Give one analyst six productive hours against those 200 alerts and each one gets 108 seconds, assuming every other rule in the ruleset stays silent. It will not. A real triage of a rundll32 alert means pulling the parent process chain, hashing the loaded DLL, checking the destination of any outbound connection and looking at what the host did in the ten minutes before. Ten minutes, conservatively.

At 108 seconds the analyst cannot do that, so they build a cheaper decision procedure. Hostname looks like a build server, parent is explorer.exe, close. That procedure is rational. It clears the queue, it is right 99.975 percent of the time, and it is functionally identical to not reading the alerts at all. Fast dismissal and a miss produce the same artifact in the case management system: a closed ticket.

Five structural causes

Every one of these is a process gap in detection engineering, not an analyst attribute.

1. Rules ship without a volume target. Nobody wrote down what this rule is allowed to cost per day, so no number can be breached and no action can be triggered.

2. Noisy rules have no owner. The rule was written by someone who has since changed teams. The SOC inherits the output and none of the authority to change the logic.

3. There is no retirement path. Disabling a rule requires a defence against "what if it catches something", which is unfalsifiable. The ruleset only grows.

4. Severity inflates. Rules are shipped at level: high because medium gets ignored, until everything is high and the field carries no information. Sort by severity and you get the whole queue back.

5. Overlapping tools duplicate. The EDR alerts on the PsExec service install, the SIEM alerts on Windows System 7045 for the same service, and the NDR alerts on the SMB session that carried it. One action, three tickets, three triages.

Tuning a rule to a target

Here is a rule that will produce that 200 a day figure on a fleet with mixed software. Cobalt Strike's default spawnto process is rundll32.exe, launched with no arguments so the beacon can inject into it. That is T1218.011 feeding T1055.

title: Rundll32 execution with no command line arguments
status: experimental
description: Detects rundll32.exe started with no arguments, a default spawnto behaviour for injected beacons.
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\rundll32.exe'
    filter_has_args:
        CommandLine|contains: ' '
    filter_no_telemetry:
        CommandLine: null
    condition: selection and not 1 of filter_*
falsepositives:
    - Software installers and management agents that shell out to rundll32
level: high

Deployed as written, most of the volume comes from two sources. We add exclusions for them and record what each one costs.

    filter_management_agents:
        ParentImage|endswith:
            - '\msiexec.exe'
            - '\ccmexec.exe'
    filter_vdi_pool:
        Computer|startswith: 'VDI-'
    condition: selection and not 1 of filter_*

In the illustrative fleet those two blocks remove 196 of the 200 daily alerts. Four a day is a queue a person can actually work. The precision moves from the first row of the table to somewhere near the third.

What the tuned rule misses

Every exclusion is a hole an attacker can occupy, and you should write down which one.

• ParentImage: '\ccmexec.exe' means an attacker with SCCM package deployment rights executes into a blind spot. That is a realistic post-compromise capability, not a theoretical one.

• Computer|startswith: 'VDI-' excludes an entire host class. If the VDI pool is where contractors work, it is also where the risk is.

• filter_no_telemetry drops events where CommandLine is absent. Windows Security 4688 does not populate that field unless "Include command line in process creation events" is enabled under Administrative Templates, System, Audit Process Creation. On hosts without it, the rule is silently disabled rather than noisy, which is worse because it looks healthy.

Remaining false positives we would expect: some legacy printer and scanner drivers launch a bare rundll32.exe at logon, and a handful of enterprise agents do it during upgrade windows. We did not test this against Server 2016 or against a Citrix estate.

Measuring precision instead of arguing about it

You cannot budget what you do not count. This runs over the case management store, not over the log pipeline.

SELECT
  rule_id,
  COUNT(*) / 30.0 AS alerts_per_day,
  COUNT(DISTINCT host) AS distinct_hosts,
  SUM(CASE WHEN disposition = 'true_positive' THEN 1 ELSE 0 END) AS tp,
  ROUND(100.0 * SUM(CASE WHEN disposition = 'true_positive' THEN 1 ELSE 0 END)
        / NULLIF(COUNT(*), 0), 2) AS precision_pct,
  ROUND(AVG(EXTRACT(EPOCH FROM (closed_at - opened_at))), 0) AS avg_seconds_open
FROM alerts
WHERE opened_at > NOW() - INTERVAL '30 days'
GROUP BY rule_id
HAVING COUNT(*) / 30.0 > 5
ORDER BY precision_pct ASC, alerts_per_day DESC;

Sort ascending on precision and the top ten rows are your fatigue. Read avg_seconds_open next to them. Any rule averaging under 60 seconds to close is not being triaged, it is being cleared. The query has a real weakness: it trusts disposition, and in most consoles the default close reason is benign, so precision reads lower than it is. Fix the default before you trust the column.

The budget and what happens when it breaks

Attach these four fields to every deployed rule and enforce them in the pipeline, not in a meeting.

• target_alerts_per_day, set at authoring time. Our defaults: 5 per 10,000 endpoints for a high rule, 25 for medium. A rule with no target does not deploy.

• target_precision_pct, measured over the last 20 dispositioned alerts. Note what the arithmetic above already implies: against a base rate this low, no single rule reaches a high precision figure on its own, so the useful floor is relative rather than absolute. We set it at 20 percent for rules that page a human, and a rule that cannot get there becomes a hunt query rather than an alert.

• owner, a named person. Not a team alias.

• review_due, a date. Not "when we get to it".

Then run the consequences on a clock. Two consecutive weeks over the volume target opens a tuning ticket against the owner with a ten business day deadline. Missing that deadline demotes the rule to level: informational, which means it writes to the log and stops paging anyone. Ninety days at informational with no successful tuning and the rule is deleted from the repository, with the commit message carrying the reason.

Deduplication comes before all of it because it is cheap. Group on (rule_id, host, ParentImage, date) and the 200 daily alerts in the example collapse to the number of distinct noisy hosts, which is usually a dozen. That is not tuning, it is presentation, and it buys the time you need to do the tuning properly.

Summary

Alert fatigue is the predictable output of deploying detections without a precision requirement, and the arithmetic explains it completely. When the base rate of real intrusion is one per year and a rule fires hundreds of times a day, dismissing everything is the accuracy-maximising strategy for an analyst under time pressure. The remedies are numeric: a volume target per rule, a precision floor, deduplication before triage, a named owner and a hard retirement path. None of them are morale interventions. The fatigue lands on the SOC, but the failure belongs to whoever shipped a rule without a number attached to it, which is a detection engineering responsibility.

Filter the catalog by high severity rules if you want to see where your own volume budget is most likely to break first.

Telemetry required for the rule above
  Sysmon Event ID 1        process creation with CommandLine, ParentImage, Hashes
  Windows Security 4688    requires "Include command line in process creation events"
  Windows System 7045      service install, for the PsExec duplication example

Fields to store alongside every deployed rule
  target_alerts_per_day
  target_precision_pct
  owner
  last_tuned
  review_due

Techniques referenced
  T1218.011   System Binary Proxy Execution: Rundll32
  T1055       Process Injection