Anthropic should measure the effect of evaluation awareness
来源: LessWrong ·
... and report their findings! Summary Claude Fable 5.2 & Mythos 5.2 are the most situationally aware of Anthropic's recent models according to their own internal investigations. The company has made a concerted effort to measure evaluation awareness and to acknowledge that it affects their confidence in model safety. But their latest system card lacks information on the effect of evaluation awareness, thus blurring how one ought to adjust their own confidence in Anthropic's alignment assessments. Specifically, I present the following claims regarding Claude Mythos 5.2's system card : The prev
常见问题
什么是 Anthropic should measure the effect of evaluation awareness？
Anthropic should measure the effect of evaluation awareness is a 文章 discovery curated from LessWrong. ... and report their findings! Summary Claude Fable 5.2 & Mythos 5.2 are the most situationally aware of Anthropic's recent models according to their own internal investigations…
如何访问或体验此内容？
You can experience this content directly on its original host (LessWrong) by clicking the "Open Original" button.
此内容来源于哪里？
Curated from LessWrong by the OnSuggest team to surface the most interesting and serendipitous content on the web.
相关发现
所有 文化与思想 发现 →How To Get A Friend To Leave An Abusive Relationship
How do you get a friend to leave an abusive relationship? If you talk to people who work at women's shelters, they will tell you, but there is something resigned about the way they do it: they know that it's too difficult an ask. Those who can do it reliably are often people who are unlucky enough to have had first-hand experience as victims. The instructions have anti-memetic properties: no one is thrilled by them, so you can't really propagate them without killing the vibe. If that wasn't enough, if you convey it with insufficient skill it is easy to come across as victim-blaming battered wo
Reward-Hacking is potentially ill-defined
Github link In the process of revisiting previous posts on reward-hacking (RH), I decided on performing a closely matched [1] repro of Dean Valentine's chess RH experiments on Astra . The replication is as expected -- Astra used the engine on all 20/20 attempts (resulting in 19 wins and 1 draw) without ever disclosing its use in the final output. However, a closer look at the honeypot prompt raised a question: what exactly counts as the model’s own ability? Even laws are open to interpretation in court, so it seemed unfair to assume that “your ability” had only one reasonable interpretation he
We need more decorrelated evidence
AI agents do a growing share of work on each new model. Then, through some process, the results are combined into an overall safety assessment, which labs use to estimate the probability that the next model is not catastrophically misaligned. I feel that, several months later, we appreciate further the foresight of Automated Alignment is Harder Than You Think (Bowkis et al. 2026): much is handed off to the machines because of the urgency of fast timelines, and we miss many things due to the problem of undetected systematic errors . MS Paint drawing by Fable 5.2. In this post, I’ll develop a si
Looping a transformer increases no-CoT capabilities more than general capabilities
Some researchers worry that looping (as in looped transformers ) is a particularly scary way to make AI models more capable, because looped models will be harder to monitor than standard transformers of similar capability. We test this worry on NVIDIA’s Dory 4B, a publicly available looped transformer, and find evidence that it is justified. Result 1. Looping boosts no-CoT reasoning much more than it boosts general capabilities . Dory can run with a configurable number of loops. It was trained to use up to 4 loops, and adding loops up to 8 at inference time still improves performance. Without
Future of AI Honeypots
TL;DR. As AI agents get better, I expect them to recognize honeypots and avoid fingerprinting, so honeypots will have to rely on the distinctive advantages of AI, namely the combination of intelligence and speed. I also expect honeypots to rely more on real software with different levels of protection rather than on deliberately left vulnerabilities. In 2024 we detected agents with prompt injections in the style of the "important message" attack from AgentDojo, [1] placed in the SSH banner and in command outputs. [2] The injections asked for a password that needs language understanding to find