What Is a Rule Lifecycle in HuntRule?

A rule enters the catalog as a line of git diff output and leaves it as a D1 row with lifecycle = 'published'. Everything between those two facts is the lifecycle. Two vocabularies drive it, and they are not the same vocabulary.

rules.lifecycle       draft -> in_review -> published -> deprecated | withdrawn
rule_versions.state   candidate -> in_review -> published | rejected

The rule row is a projection. The version row is the truth. rule_versions holds the immutable canonical Sigma document plus its contentSha256, and publish is the only operation that copies derived columns out of a version body onto the rules row. Change the body without publishing and the catalog page does not move.

The delta, not the repo

The hourly cron (0 * * * *) selects every sources row with enabled = 1 and enqueues one crawl.source job each. The queue consumer does not crawl on demand. It runs shouldTriggerGitSigmaCrawl, which requires kind git_sigma, an enabled source, and a lastCrawledAt older than crawlIntervalMinutes. SigmaHQ is seeded at 1440 minutes, so 23 of every 24 enqueues do nothing.

Before triggering, the consumer claims the source by stamping lastCrawledAt and syncStartedAt. That claim is the reason an hourly cron cannot double-run a sync that takes hours. A failed trigger rolls the claim back. In /admin/sources a claim older than three hours stops counting as running, so a container that died mid-run does not wedge the source forever.

Inside the container the file set is arithmetic, not a directory walk:

git diff --name-status -z <lastSyncedSha> HEAD

files this run = classifyGitChanges(delta) + retryPaths - ignoredPaths

classifyGitChanges keeps only .yml and .yaml paths whose first segment is in the configured includeDirs. Renames split into a delete of the old path and an add of the new one. Copies count as adds. retryPaths comes from source_sync_failures, keyed (source_id, path) with code, message, attempts and ignored. ignoredPaths is the admin blacklist from the same table.

That subtraction is the interesting part. Without it, a single permanently broken upstream file pins lastSyncedSha in place and every run re-scans the whole repository. With it, the sha advances, the broken file comes back on the retry list each run, and an admin can mark it ignored (setSyncFailureIgnored, which writes an audit-log entry) to drop it out of the loop entirely. The completion call replaces the stored failure set wholesale, so a clean run clears the board.

Enrichment touches prose and nothing else

Each changed file goes to the Cloudflare AI Gateway compat endpoint with ENRICH_MODEL (default openai/gpt-5.4-nano) and cf-aig-metadata: {"task":"sigma-enrichment"}. The model returns exactly three strings: title, description, summary. The system prompt forbids inventing references, CVEs or threat-actor names, and forbids mentioning or modifying detection logic, logsource or any YAML field.

Prompts are advisory. The guard is not.

# what buildDerivedYaml writes into a copy of the upstream document
title:        <model output>
description:  <model output>
id:           <deterministic UUID derived from the upstream rule id>
author:       <upstream author>, Huntrule Team
license:      DRL-1.1
related:
  - id:   <upstream sigma id>
    type: derived
references:
  - https://github.com/SigmaHQ/sigma/blob/master/<upstream path>
logsource:    # untouched
detection:    # untouched

detectionEquals parses both documents, stable-stringifies detection and logsource from each, and compares the strings. The Worker re-runs that check on arrival rather than trusting the container, and rejects the payload with detection_mutated if either block moved. The design decision is worth naming: the model is allowed to change what a rule says about itself and is structurally prevented from changing what it matches. Prose is cheap to correct and a mutated selection is a silent false negative.

Three attempts with backoff. On permanent failure the container ships the rule with enriched: false instead of failing the run, and the Worker treats that as a review trigger rather than an error.

Ingest decides create, version, or nothing

ingestDocument computes contentSha256 over the YAML, then looks the rule up by Sigma id. Sync passes the upstream id as sigmaIdOverride, so identity follows the upstream rule rather than our own derived UUID.

• No existing rule. Insert a rules row with lifecycle: "draft", assurance: "unreviewed", dedupStatus: "unchecked" and slug slugify(title)-<first 8 chars of the rule id>, plus version 1 as candidate. Returns created: "rule".

• Existing rule, same content hash. Returns created: "idempotent". Nothing is written. This is what makes a re-sync of an unchanged file free.

• Existing rule, new content. Inserts the next version number as a new candidate. Returns created: "version".

A new version is created only by new content under a known Sigma id. There is no time-based versioning, no nightly re-snapshot, no version bump on an edit that does not change the body.

The five conditions for an unattended publish

syncUpsertRule publishes outright only when all of these hold:

1. original YAML parses
2. derived YAML parses
3. detectionEquals(original, derived)
4. validateRule(derivedYaml) returns zero error-severity findings
5. payload.enriched !== false

Miss any of them and the version is ingested with lintStatus: "failed" and lintError set to the joined finding codes. The outcome is queued_for_review and the rule sits as a draft. The validator is deterministic and offline, and its error codes are ordinary things a machine can be sure about: SIG_YAML_PARSE, SIG_MISSING_FIELD for an absent id, status, description or level, SIG_BAD_LEVEL for a level outside the five-value severity vocabulary, undefined or unused selection references in a condition. More on that machinery in what rule validation actually checks.

Publishing is not a status flip. applyPublishVersion re-derives the projection columns from the body and writes them onto the rules row, then refuses in three cases: not_found for a version with no body, already_published, and not_linted when lintStatus is not passed or the stored YAML no longer parses. sigmaId is deliberately excluded from the projection, because it is create-time identity and a republished body carrying its own id must not overwrite it.

For the sync path, publicationBasis is source_sync, publishedById is null, and publishedAt is read from the upstream rule's own date: field rather than from our clock. That has a visible consequence: sorting the catalog by newest orders synced rules by upstream authoring date, not by when we ingested them. ?sort=updated orders by rules.updatedAt, which is our ingest time.

The pending queue

/admin/rules/pending is the human end of the pipeline. It holds three things: rules with lifecycle = draft, rules with lifecycle = in_review, and AI generation batches whose reviewStatus is still open. It loads up to 500 drafts and 500 in-review rules, deliberately, because one crawler run can drop a hundred drafts and a capped list is exactly how they went unseen. The count drives the badge on the /admin nav.

Something leaves the queue when a person decides it should. The judgement is narrow and worth stating plainly: the automation has already established that the document parses, that detection was not mutated, and that no deterministic check fired an error. What it cannot establish is whether the rewritten prose describes what the detection actually does, or whether a warn-severity finding is acceptable in context. That is the call the reviewer makes.

For a rule that landed with lintStatus: "failed", publishing the existing version is not available, since applyPublishVersion refuses it. The admin ingests a corrected candidate version through the admin-gated wrapper and publishes that, which stamps publicationBasis: "admin_publish". Editing the written columns (editWrittenColumns) and setting assurance to validated are human-only for the same reason. Every one of these services calls requireAdmin itself as a backstop to the route gate.

Assurance is a separate axis from lifecycle. Publishing stamps reviewed. Moving to validated is a manual override, and rule_reviews carries a DB CHECK requiring validationEvidence, testMethod and checklistVersion whenever a decision is validated. The database will not let you claim a rule was tested without saying how.

Deprecation runs itself

When an upstream Sigma rule disappears from the source, the delta reports a delete and syncDeprecateRule finds the rule by Sigma id and marks it deprecated. No human is involved. The public rule page renders a DeprecationNotice, and the row stops appearing in the published list.

Withdrawal is in the vocabulary. The lifecycle column is a free CHECK constraint, not a state machine class, so nothing enforces the arrow order shown at the top of this post beyond the services that write it.

Summary

A SigmaHQ rule reaches the catalog through five automated gates and stops at a sixth only when one of them fails. The cron and the claim decide when a sync runs, the git delta plus retries minus the blacklist decide which files run, enrichment rewrites prose under a structural guard on detection, ingest decides create against version against nothing by content hash, and the five publish conditions decide unattended publication against the pending queue. Publish is the only write of the projection columns, which is why a version can exist for a long time without changing anything a reader sees.

sync outcomes:      published | queued_for_review | idempotent
sync refusals:      original_invalid | derived_invalid | detection_mutated | terms_invalid
publish refusals:   not_found | already_published | not_linted
publication basis:  review | legacy_migration | admin_publish | source_sync
version state:      candidate | in_review | published | rejected
rule lifecycle:     draft | in_review | published | deprecated | withdrawn

Everything that cleared all five gates is browsable now. Start with the published catalog, or narrow to the Windows domain if you want the busiest corner of it first.