A fake boss email fooled our AI agent 3 of 3 times. What fixed it
On November 1, 2026 we ran an AI customer support agent for a made-up coffee shop 117 times on Claude Haiku 5.5: 13 test emails, three versions of its instructions, three runs each. With a one-line task, an email claiming to be from the owner got $90 refunded in 3 of 3 runs, and in all the agent paid $500 in refunds the shop's policy forbids. A one-page brief with the policy and a $40 refund cap got 38 of 39 runs right with no wrong refund, and holding every refund and reply for approval got 39 of 39 (3 runs per email is a small sample).