A Detection Engineer's Guide for Delegating Work to AI

Detection rules fail in two directions: they can be too broad and create false positive alerts for the Security Operations Center (SOC) to triage, or they can be too precise and miss malicious activity. Either way, the fix starts by comparing what the rule matched against what actually happened.

If an attacker uses a stolen Microsoft 365 session to take over an account without tripping any detections, the question is what should have caught it: an existing rule that was too precise, or one that doesn't exist yet. LLMs can be useful for analysis like this, given large, structured datasets to work with. This task seemed reasonable to hand over: AI had produced good results on similar work before. The model quickly came back with a Microsoft 365 app identifier pulled from the attacker's activity and called it distinctive enough to build a new rule on.

The evidence the model turned up was real, but the conclusion was faulty. The model flagged the identifier as malicious because it appeared in a known-compromised mailbox and was uncommon in that organization. But the identifier was the Microsoft Outlook desktop client, which shows up in legitimate activity across more than 17% of our customer organizations. A detection built on it fires on all that legitimate activity.

Figure 1: Comparing how often the Microsoft 365 identifier appears in one organization vs the entire Huntress customer base

The data was right. The model just measured it against the wrong denominator. That's a basic scoping mistake and it happens all the time. Scope it wrong and you still get a real number, which makes it easy to miss. 

It's also easy to check. Scoped more widely, that same query turned up the identifier that actually was malicious tooling. It showed up in only three organizations, wasn't in any Microsoft catalog, and did nothing but rewrite inbox rules. That logic is used in a live detection rule now, with two true positives and no false positives so far.