#reward
3 discoveries · Hand-picked discoveries from the open web, newest first.
-
RL Reward-Hack Auditing for Chinese AI Safety
TL;DR: Access to AI safety talent & resources, particularly for labs during training, is sorely lacking in China. We created one of the largest AI safety communities in China , and plan to next start with auditing reward-hacks of unreliable upstream RL environments that domestic data vendors provide to Chinese frontier labs (which induces reward hacking and a whole host of dangerous downstream tendencies). RL tasks are notoriously prone to exploitation by reward-seeking agents (see examples here , here , and here ). In Xiaomi's MiMo v2.6 RL environments that were open-sourced, 2/3 had leaks on
LessWrong · EN
-
Reward-Hacking is potentially ill-defined
Github link In the process of revisiting previous posts on reward-hacking (RH), I decided on performing a closely matched [1] repro of Dean Valentine's chess RH experiments on Astra . The replication is as expected -- Astra used the engine on all 20/20 attempts (resulting in 19 wins and 1 draw) without ever disclosing its use in the final output. However, a closer look at the honeypot prompt raised a question: what exactly counts as the model’s own ability? Even laws are open to interpretation in court, so it seemed unfair to assume that “your ability” had only one reasonable interpretation he
LessWrong · EN