Fifty thousand alerts a day: triaging a security operations center with agents

The problem

A security operations center was receiving on the order of fifty thousand alerts a day across endpoint, identity, network, and cloud tooling. Analysts worked a queue that could never be cleared. The failure mode was not missing sophistication — it was alert fatigue: the true positive arrives buried among thousands of benign ones, and a tired analyst at 3am closes it in four seconds along with everything else.

The hypothesis

Most triage work is mechanical: pull the asset's context, check the user's normal behavior, look up the indicator's reputation, see whether this pattern was already dismissed last week. That is retrieval and correlation, not judgment. If an agent did the assembly and drafted a verdict with its reasoning, analysts could spend their attention on adjudication.

The build

• Enrichment agent — for every alert, gathers asset ownership, user baseline, recent authentication events, threat-intel reputation, and related alerts in a time window.
• Correlation into cases — deterministic rules group related alerts into a single case, so one incident does not arrive as forty separate tickets.
• Verdict drafting with evidence — the agent proposes benign, suspicious, or escalate, and must cite the specific evidence for each. The reasoning is stored with the case.
• Tiered autonomy — high-confidence benign patterns that matched historical dispositions auto-close with sampling review. Everything ambiguous goes to a human, and nothing escalates or takes containment action without analyst approval.

Rollout

Six weeks of shadow mode, with the agent producing verdicts that analysts never saw until after they had made their own call. That gave us a clean agreement rate against human judgment, and — more useful — a map of exactly which alert families the agent handled well and which it should never touch. Autonomy was granted per alert family, not globally.

Results

The large majority of daily volume now arrives pre-triaged with evidence attached. Mean time to triage dropped several-fold, and the analysts' working queue became small enough to actually finish. The team's own framing was the best summary: the job changed from reading alerts to making decisions.

What we'd tell you

• Shadow mode is not optional in security. You need an agreement rate before you grant any autonomy.
• Grant autonomy per alert family, with a documented threshold — never as a single global switch.
• Store the reasoning. When an incident is reviewed six months later, "the model said benign" is not an answer.
• Make containment actions human-gated. The cost asymmetry is not close.