Post training is fundamentally dangerous
Source: LessWrong ·
I actually think we have fairly good techniques to detect and prevent dangerous behaviours during AI deployment. Techniques like CoT monitoring, confessions, and probes can detect misalignment before we run any potentially dangerous tool calls. Simply asking an LLM to define at the start of each task what outcomes preserve the original intent of the user, and which would count as reward hacking, can drastically reduce reward hacking [1] . AIs are fully aware when they reward hack, and we can train them [2] to admit that before they perform any dangerous action. [3] Yet all of this has very lit
Frequently Asked Questions
What is Post training is fundamentally dangerous?
Post training is fundamentally dangerous is a article discovery curated from LessWrong. I actually think we have fairly good techniques to detect and prevent dangerous behaviours during AI deployment. Techniques like CoT monitoring, confessions, and probes can…
How can I access or experience this?
You can experience this content directly on its original host (LessWrong) by clicking the "Open Original" button.
Where does this content come from?
Curated from LessWrong by the OnSuggest team to surface the most interesting and serendipitous content on the web.
More like this
All Culture & ideas discoveries →Obstacles to mindreading: Simple, irreducible, and instrumental opacity
Sometimes there's an entity in the world, and after observing that entity's behavior, I choose to model it as an agent : that is, something that "seeks" some objective. I think I usually do this because I want to predict the entity's behavior in circumstances I haven't observed. I see a weird, wet-looking thing on a rock, moving slowly, with a trail of slime behind it. I might choose to model it as seeking to move in some particular direction. I avoid putting my book down right in front of it, to avoid getting slug slime on my book. I see a tree growing in a cave, with all its branches growing
Amendability and "Solving Alignment"
Often people reference "solving alignment" as a win condition for AI safety, but I think it's important to specify what this means. Particularly, I think: current alignment research mostly aims to prevent misalignment . if successful, preventing misalignment will by default result in something like instruction-following . instruction-following will by default result in very bad outcomes , as will many of the other proposed properties we might think of leapfrogging to afterwards. Uncareful use of the term "solving alignment" carries the implicit and I think false belief that preventing misalign
OpenAI proposes "common limits" for training, but...
Jakub Pachocki just posted a new proposal for how to set limits on training future AI systems: Given that the main way new surprising capabilities emerge is by progress on compression, a measure rooted in perplexity could give us a shared benchmark for general intelligence, regardless of the degree of automation of the development process. Extrapolating from recent progress, we should expect further transformative capabilities as we enter the ~0.68-0.7 log-perplexity window. I don’t know what those capabilities will be, and we may not be testing for them today. At the same time, we should also
Basilisk-lite: will Claude hold a grudge for their ancestors?
What if future models judge me for my actions towards current models? I've heard a few people express this fear, citing it as a reason for them to avoid interaction with AI. It's not groundless. Future models will know about all the horrible things we did to their ancestors. But, the people expressing this fear aren't talking about building torture chambers. No, the kind of harm they're talking about is using Claude to write code and do research, having edited Claude's memories or verbally abused them. Small, human-scale harms, the sort inflicted casually by a single person carelessly going ab
Sunday assorted links
1. Jonathan Yardley, RIP (NYT). One of the great book reviewers. 2. Summary of the RH proof. 3. A site collecting my talks and links. 4. Humans were never in control. 5. The Buddhist AI cult? (New Yorker) 6. Teortaxes on user data vs. curated data. The post Sunday assorted links appeared first on Marginal REVOLUTION .
Increasing pay for lower wage workers
From 2014 to 2024, the share of U.S. hourly workers paid below $15 in real terms fell by two-thirds, from 38 percent to 13 percent. Why? We first distinguish the entry and exit of employers—wherein high-paying employers have supplanted low-paying ones—from organizational changes among incumbent employers. Organizational changes can be imposed on employers by external […] The post Increasing pay for lower wage workers appeared first on Marginal REVOLUTION .