Aligned Data Can Induce Severe Misalignment via Context Confusion
Fuente: LessWrong ·
TL/DR We showed that training your aligned model on aligned data can make your model misaligned because alignment is context-dependent : An action that is appropriate in one context can be misaligned in another. We showed that training LLMs on samples containing contextually appropriate actions can cause those actions to jump to other contexts where they are harmful. We called this phenomenon context confusion and provided model organisms (datasets creating context confusion), showed its existence in a popular instruction-tuning dataset, and provided a mechanistic explanation. We speculate tha
Preguntas frecuentes
¿Qué es Aligned Data Can Induce Severe Misalignment via Context Confusion?
Aligned Data Can Induce Severe Misalignment via Context Confusion is a artículo discovery curated from LessWrong. TL/DR We showed that training your aligned model on aligned data can make your model misaligned because alignment is context-dependent : An action that is appropriate in one contex
¿Cómo puedo acceder o experimentarlo?
You can experience this content directly on its original host (LessWrong) by clicking the "Open Original" button.
¿De dónde proviene este contenido?
Curated from LessWrong by the OnSuggest team to surface the most interesting and serendipitous content on the web.
Más como esto
Todas las recomendaciones de Cultura e ideas →We urgently need to practice before we can pause
I've noticed two clusters of thought in how people expect a global AI pause to be shaped: Single-shot global pause : the global AI pause will be implemented suddenly, with little pacing precedent before it, and we will need to do a lot of research before it's implemented so that we can get it right the first time around. Incrementally-shaped global pause : the global AI pause will be implemented incrementally, with lots of pacing precedent before it, and we will improve the pause setup and get more parties to join in as we go. An incrementally-shaped global pause is both more likely and more t
Minimalism, Stuff, and Brains
Reading the comments on my post on junk I was surprised to see the extent to which people's relation to rarely used possessions varies. Some people forget rarely used possessions, and get minimal benefit from having them stored. When they need something, they don't know whether they already have it, and the effort to maintain an index [1] wouldn't be worth it. Ben writes : I end up with curtains I don't need. I decide to store them for the future in case they are helpful. Years pass. I need new curtains. I have forgotten the ones I kept, so I buy new ones and put them up. Months later I find t
Your Book Review: Orality and Literacy
Finalist #15 in the Book Review contest
How octopuses squeeze through teeny tiny holes
Robots used during search and rescue operations need to be able to wriggle through narrow cracks and crevices. Researchers have turned to a natural contortionist for inspiration — the octopus.
Busy Philipps talks about treating her brain tumor and filming a new show
On Wild Card, Philipps unpacks the experience of treating her brain tumor and filming her new show, the CBS drama Cupertino . But she also opened up about another aspect of her life.
Did Trump Learn Anything From the Midterms? We Already Have Our Answer.
A rage-driven president has picked an even angrier chief of staff.