“Sorry, I can’t help with that”: How your guardrails might become the attacker’s best friend

Welcome to this week’s edition of the Threat Source newsletter. 

Hello, everyone. Long time reader, first time writer here at the Threat Source newsletter! I wanted to start out by introducing myself. My colleague and friend Mick Baccio set the bar pretty high last week, so I was planning to tell you all about myself, including: 

• How I did my first real IR under the influence of The Cuckoo’s Egg while an undergraduate (and failed) 
• My pre-bug bounty flirtation with vulnerability research, including an arbitrary file overwrite in biff(1) and how I once hacked MIT’s website 
• My first ever hands-on experience with a computer, the display demo Commodore 64 at the Montgomery Ward 

Unfortunately, my editor says we don’t have the “space” for that, the MIT thing might open me up to “liability,” and it’s not the kind of “professional image” we strive for here at Talos. (I'm watching. Always watching. -Amy) 

So instead, I’ll just play it safe and say that I’ve been in the security field for a little over 30 years now, mostly concentrating on the defensive side (Go, Team Blue!). I’ve helped set up SOCs, run threat hunting teams, and even published a few things you might have heard of.  

Speaking of things I’ve published, I’ve written before about the Attacker’s Dilemma. The idea that defenders have inherent advantages over attackers runs contrary to what most of us have heard throughout our careers. An attacker must evade monitoring and technical controls at every step of their attack lifecycle, because the defender only needs to notice once in order to respond and prevent them from achieving their goal. This is one of the most important advantages of any security team has, but we are currently witnessing a self-imposed erosion of this advantage through the rise of poorly-designed AI guardrails.