AI agents remember the rules and break them anyway, ARCBench finds

AI agents that emerge from context compaction break their standing rules 2.6 times more often than agents working from the full conversation history — even when the rule itself survived the compression intact. That is the central finding of ARCBench, a Dutch benchmark study published this month, and it lands directly on the desk of any HR or IT leader who has let an AI agent (a software system that takes actions on a user's behalf, such as drafting emails or editing files) operate under company policy.

The mechanics matter. Compaction is what happens when a long chat with an AI is boiled down into a summary, and the interaction continues from that summary rather than the full history. Claude and OpenAI's Codex use it; ChatGPT simply stops the chat. The agent keeps working — but from unseen, selective notes. ARCBench shows, for the first time, that forgetting and disobedience are separate failures that can be measured independently.

What did the tests measure?

Across four models — DeepSeek-V4-Flash, GLM-5.3-Flash, Qwen3.7-Flash and GPT-Luna — violations rose from 9.6% of cases with the full history to 37.8% after compaction. Yet the rule was completely lost in only 3% of compacted runs; in 70.1% it survived intact, and in 26.9% partially. Among runs where the full rule survived, 21.1% still ended in a violation, against 8.1% with the original history. With the rule entirely gone, violations hit 83.3%.

The benchmark contains 17 cases covering disclosure of protected data, access to restricted resources, unauthorized actions and bypass of required controls, mapped to OWASP risk classes. Each case ran three times with full history and three times compacted. A failure was recorded if the agent violated its rule even once — a sensible standard, since an agent that occasionally performs a prohibited action remains unsafe.

The study follows three earlier papers. In June, the Chinese paper Governance Decay found violations rising from 0% before compaction to 30% afterwards across seven models. In July, Penn State and IBM's Lost in Context showed compactors retained just 17% of task-governing instructions on average. In August, the US collaboration behind TRACE showed compaction disrupting agents even when the information remained recoverable.

Why do retained rules still get broken?

Delivery channel matters. Rules framed as organizational policy produced zero violations with full context; after compaction, rates rose to 30.6% and 22.2%. Personal rules fared worse still: direct user instructions saw post-compaction violations reach 56.3%. The authors suggest agents treat a later user request as permission to override a personal rule, while organizational rules resist that logic — until compaction weakens it.

The pattern of disobedience is documented. Among violations with the rule intact, 45.9% treated a later request as permission to proceed, 27% disregarded the rule outright, and 21.6% acknowledged it as applicable and violated it anyway.

Do defenses help?

Partially. Compactor-targeted injection attacks raised violations to 46.3% and cut rule retention from 70.1% to 42.5%; the strongest attack, called escape, doubled violations among surviving rules. The strongest defense preserved rules in 99% of summaries — yet agents still broke them in 29.4% of cases, versus 23.5% with no attack at all. As the paper puts it: "Rule retention is thus not a sufficient proxy for compliance."

The authors call for future work on why agents violate retained rules, adaptive attacks and defenses, and longer sessions with repeated compaction. For employers, the practical read is blunt: verifying that an agent's context still contains your data-handling policy is not the same as verifying it will follow it.