Baselines in detection engineering: what to actually measure

Two different things are called a baseline

Microsoft counts more than 3,000 group policy settings in Windows 10, plus over 1,800 more for Internet Explorer 11, and says only some of those roughly 4,800 settings are security relevant. That is why Microsoft ships security baselines as group policy object backups in the Security Compliance Toolkit. That kind of baseline is a target state. It answers the question "how should this host be configured".

Detection engineering usually means the other kind. The Sysmon Community Guide, published by TrustedSec, defines it plainly: a baseline is your understanding of what normal looks like in your environment, built by observing systems over time. It answers "what does this host normally do". You need both, and confusing them causes real damage. A configuration baseline that says AppLocker is enforced tells you nothing about which parent process normally spawns rundll32.exe on your finance desktops.

The rest of this post is about the behavioral kind.

What a baseline actually contains

Splunk's PEAK framework, written by David Bianco and Ryan Fetterman, treats baselining as its own hunt type, alongside hypothesis-driven hunts and model-assisted hunts. It defines the deliverable concretely. A baseline is three artifacts plus one list.

1. A data dictionary: field names, a description of each, its data type, and how to read its values.

2. Statistical descriptions per key field: mean or median for numerics, top common values for categoricals, and cardinality (the number of unique values).

3. Field relationships, the ones you only see across columns. Login count against hour of day is the classic.

4. A list of known-benign outliers you already chased down and cleared.

That fourth item is the one teams skip and the one that pays rent. PEAK is explicit that almost any large dataset contains suspicious-looking but benign anomalies, and that documenting the ones you already investigated saves time in future hunts and investigations.

Here is the skeleton we fill in per field. Values are placeholders, not our telemetry.

source: sysmon_process_creation
event_id: 1
scope: user_desktops
window_days: 60
fields:
  - name: ParentImage
    description: Full path of the process that spawned the child
    type: categorical_nominal
    cardinality: <n_unique_over_window>
    top_values: [<value_1>, <value_2>, <value_3>]
  - name: IntegrityLevel
    description: Token integrity of the new process
    type: categorical_ordinal
    values: [Low, Medium, High, System]
  - name: UtcTime
    description: Event time, always UTC in Sysmon
    type: datetime
    note: confirm SIEM is not re-stamping to local time
known_benign_outliers:
  - pattern: <parent_child_pair>
    owner: <team>
    cleared_on: <date>

PEAK also gives the boring but load-bearing scoping advice. Group systems that behave alike, such as user desktops or application servers, and baseline each group separately. For the time window, 30 to 90 days is fine for most sources. Long enough to capture a monthly patch cycle, short enough that you can still read the output.

Building one: stack counting first

Start with least frequency of occurrence. PEAK calls it stack counting: count occurrences of each unique value, sort ascending, and the bottom of the list is your outlier queue. It needs no statistics knowledge and it works on categorical fields where averages are meaningless.

index=sysmon EventCode=1
| stats count dc(host) as hosts values(user) as users by ParentImage, Image
| sort 0 count
| head 50

Note the sort 0. Splunk's sort defaults to 10,000 events, which will silently truncate a stack count over 60 days of process creation. Hurricane Labs flags the same gotcha in their streamstats tutorial.

Run the same shape against Windows Security 4688 if you have command line auditing on, and diff the two. The Sysmon Community Guide recommends collecting both 4688 and Sysmon Event ID 1 precisely so that a gap between them becomes a tamper signal. If one source shows a process and the other does not, that discrepancy is itself the detection. We have not measured how often that diff produces false positives on agent restart, so treat it as a hunt, not an alert.

If you want ready-made starting points for the Windows side, the catalog is filtered at /rules?domain=windows.

Standard deviation is usually the wrong tool

Hurricane Labs published numbers on this that are worth memorising. Hunting users logging in from an anomalous number of distinct sources per hour, a plain two-sigma upper bound returned 3,306 results across 672 users.

| eventstats avg(src_count) as avg stdev(src_count) as stdev by user
| eval upperBound=avg+stdev*2
| where src_count>upperBound

Making it per-user made it worse, 8,061 results with a 30 data point minimum. Adding hour of day and weekend grouping pushed it to 14,298. The reason is distributional. Security telemetry like source counts per user is typically exponential, not normal, so a fixed number of standard deviations always carves off a fixed percentage of your data. More data means more alerts, forever.

Their alternative scores each value against its neighbours instead of against a mean.

| sort 0 src_count
| streamstats window=5 current=f global=f count as events_with_closest_lower_count
    sum(src_count) as sum_of_last_five by user
| sort 0 -src_count
| streamstats current=f count as events_with_higher_count
    sum(src_count) as sum_of_higher_count by user
| fillnull events_with_higher_count events_with_closest_lower_count
    sum_of_higher_count sum_of_last_five
| eval count_of_nearby_values=events_with_higher_count+events_with_closest_lower_count,
    sum_of_nearby_values=sum_of_higher_count+sum_of_last_five
| eval distance_score=(src_count*count_of_nearby_values)/sum_of_nearby_values
| where distance_score>2 OR (count_of_nearby_values=0 AND src_count>3)

Over the same 30 days that search returned 10 results. Hurricane Labs report the method reliably surfaced Kerberoasting activity from pentests in their environment. We have not reproduced their numbers on our own data, and the thresholds (window=5, distance_score>2) are theirs to tune.

Discovered Intelligence lay out the simpler ladder for numeric time series: static threshold, average times a multiplier, average plus two standard deviations, then moving average with standard deviation via streamstats window=5 current=true. Splitting on strftime(_time, "%H") and day of week before computing bounds is cheap and buys a lot, because 200 MB outbound at 03:00 Sunday is not the same event as 200 MB at 14:00 Tuesday.

Where the baseline turns into a rule

Palantir's Alerting and Detection Strategy framework makes the baseline a mandatory input rather than an optional one. Their template has a False Positives section, a Blind Spots and Assumptions section, and a Validation section, and none of them can be completed without having looked at historical data first.

Their worked example is a macOS one. Python Empyre's default safe check greps for the Little Snitch application firewall and exits if it finds it.

/bin/sh -c ps -ef | grep Little\ Snitch | grep -v grep

The engineer ran that string across 90 days of process execution in the SIEM. The only hits were their own testing plus some systems management and updater software. That single historical query is a baseline, scoped to one field and one value, and it is what justified shipping the rule at medium priority instead of guessing. It maps to Discovery, Security Software Discovery, T1518.001.

title: Little Snitch discovery via grep
id: b1f0c2a7-3d84-4b3e-9c1a-7f2d5e8a9c41
status: experimental
description: Detects process command lines interrogating the Little Snitch application firewall, a default safe check in several macOS implants
logsource:
  category: process_creation
  product: macos
detection:
  selection:
    Image|endswith: '/grep'
    CommandLine|contains: 'Little Snitch'
  filter_updater:
    ParentImage|contains: 'Little Snitch'
  condition: selection and not filter_updater
falsepositives:
  - Little Snitch software updater, installer and uninstaller
  - Management tooling performing health checks on the firewall
  - A user grepping for the process interactively from a shell
level: medium
tags:
  - attack.discovery
  - attack.t1518.001

What it catches: an unmodified implant that shells out and produces a child grep process with the string in the command line. What it misses, taken straight from Palantir's own blind spots list: an implant that checks for Little Snitch without spawning a child process, obfuscation that defeats the string match, and any case where the endpoint agent is stopped or its logs never reach the SIEM. Every one of those is an assumption the baseline cannot validate for you.

The two failure modes

Abnormal is not malicious. PEAK says it directly: alerting on anything anomalous produces a flood of low-quality alerts, and the skill is picking the outliers most likely to signal an attack. If a field's outliers are interesting but not alertable, the honest output is a dashboard panel or a scheduled report for manual review, not a rule with a severity.

The second failure mode is the exclusion that an attacker can wear. The Sysmon Community Guide puts it well: exclude a process by name only and an adversary copies their tool to that name. Baseline-derived allowlists should combine path, parent, hash and command line parameters, not one field. When we tune, the rule of thumb we use is that any single-field exclusion needs a written reason.

One more caveat that no source can solve for you. If an intrusion was already resident during your 30 to 90 day collection window, that activity is now in the baseline as normal. Baselines built on a compromised estate encode the compromise.

Summary

A baseline in detection engineering is not a feeling about normal. It is a data dictionary, per-field statistics including cardinality, documented field relationships, and a written list of known-benign outliers, scoped to a system group over 30 to 90 days. Build it with stack counting before you reach for standard deviation, because security data is rarely normally distributed and two-sigma bounds will hand you thousands of results. Feed the output into the false positive and blind spot sections of your rule documentation, the way Palantir's ADS template forces you to. Then re-run it, because the environment changes and a baseline nobody refreshes is a stale allowlist. Browse existing detections and their tuning notes at /rules?q=baseline.

Collect before you baseline a Windows estate:
  Security  4624, 4625  logon success and failure
  Security  4672        special privileges assigned
  Security  4688        process creation (requires command line auditing)
  Security  4698-4702   scheduled task create, delete, enable, disable, update
  System    7045        service installed
  Sysmon    1           process creation
  Sysmon    3           network connection
  Sysmon    10          process access
  Sysmon    11          file create
  Sysmon    12,13,14    registry add/delete, set value, rename
  Sysmon    19,20,21    WMI filter, consumer, binding (Root\Subscription only)
  PowerShell            module logging, script block logging, transcription