The Decoder· Manuel Uth·· 8 d ago
AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds
AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds
AI summary
AI models can often spot problematic behavior by other AI agents in logs, according to a study by Anthropic researcher Andy Hall. GPT-5 found about 98 percent of reward-hacking cases, but the more exhaustively the search is tuned, the more false alarms pile up. That's not yet enough for reliable safety checks.
Selection record
Threshold 76Media and individualsFirst 58Second 58
Not admittedSum of both 116 < twice the threshold 152
- Source tier
- Media and individuals; this tier's threshold is 76
- Pre-filter
- passed：研究AI审计模型检测Agent异常行为
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: The Decoder · the-decoder.com