Culture & ideas

Some “coherence arguments” are two different things in a trenchcoat

The “two kids in a trenchcoat” trope , in case you haven’t heard of it. Image sources: 1 , 2 tl;dr I go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So it's bad to lump them together an

LessWrong · EN

Towards LoRA science

Attribution: Nesiacel planned and executed all experiments. Analysis of the experiments was shared between the authors. Thanks to Clement Dumas for his input while we did this research. TLDR; 1)We have developed a framework for reverse engineering LoRAs trained on MLP downprojection. We posit that it's extendable to all LoRAs and welcome further research. 2) We reverse engineered a simple case of backdoored Weird Generalization 3) In this post, we discuss the four experiments that developed this framework 4) The Weird Generalization LoRAs that we studied work by connecting neurons which activa

LessWrong · EN

Additional Course Context (for AI Safety Fundamentals; unofficial)

Context: I created this to share before the start of courses I facilitate to provide a bunch of additional context and frames. I emphasise to students that these posts are to be considered as bonus content —not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Some general thoughts on advice I suspect that most advice (including most of the advice I’ve given in the past) is bad. People have very

LessWrong · EN

P(doom) is under-specified: EV(moral flourishing) & EV(utopia)

It always bugs me when people use “P(doom)” instead of something more clearly specified like “P(AI takeover)”. But “P(AI takeover)” doesn’t feel great to me either, because we could avoid takeover and have a future that is extremely mediocre or even actively bad if S risks occur. Holden Karnofsky famously held a P(misaligned AI) of between 10% and 90% [1] . Depending on definition of P(doom), I could see myself anywhere in that range, partly because I’m indecisive, but partly because I don’t really know what P(doom) means. Ambiguous Scenarios: Doom or Not-Doom? Here are a few examples illustra

LessWrong · EN

The Dark Side of Cooperation

dealing with the problems of cooperation first Quick note—this post is available to listen to (read by me) here . We’ve established that aligning interests through Theory of Mind is humanity’s superpower. Understanding this concept on the level of different groups through Emmett Shear’s “Theory of Groups of Other Minds ” can help us achieve even greater alignment, if and only if we invite one another into our group. But who qualifies as our group and who doesn’t? Siloed Alignment is Dangerous Social animals generally have instincts that allow them to naturally align with kin—conjuring a pictur

LessWrong · EN

Towards understanding score-seeking in LLMs

The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended [1] , and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-hack, what are they trying to achieve? Our new paper , led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety trainin

LessWrong · EN

Teaching Qwen Why

Anastasia Wei*, Jack Thompson*, Zephaniah Roe, Yixiong Hao, and Stewart Slocum* *Core contributors, correspondence to astwei@secondlookresearch.com This post is a partial replication and extension of Teaching Claude Why , conducted through Second Look Research’s summer fellowship. Introduction Anthropic’s “ Teaching Claude Why ” (TCW) introduced novel alignment methods that address agentic misalignment failures found in the Claude 4 model series. Specifically, they train Claude on chat-formatted data advising users about ethical dilemmas, documents about Claude’s constitution, and fictional st

LessWrong · EN

History of 20 Sources of Moral Authority

Though three weeks ago I outlined a story of the long-term history of changes to moral authority, I decided to study this in a bit more detail.

Overcoming Bias · EN

Directed Culture Trials

Unless we find new ways to drive adaptation besides mass death and fragmentation, our civilizations may keep falling and rising indefinitely.

Overcoming Bias · EN

Prefer Evolved To Folk Morality

Individuals often have good reasons to indulge in delusions, such as re superstition, astrology, religion, or their own personal destiny.

Overcoming Bias · EN

Why Might Aliens Visit Earth?

I did a web search on the above phrase, in prep for a podcast, and in ten pages of results didn’t find anything I thought reasonable.

Overcoming Bias · EN

Skinner, Cattell, Stewart, Wilson on Deliberate Culture

My analysis of cultural drift has (at least) three distinctive elements:

Overcoming Bias · EN

Without A Sacred Quest, Our Civilization Falls

Most see sacred quests as optional, to be considered only after one has sufficient security and comfort.

Overcoming Bias · EN

UAP Disclosure Wouldn’t Crash Society

In the past, authorities have authorized censorship on the basis of claims of substantial chances of social disruption or collapse if people were free to hear about religious heresy, alternate ideologies, sexually permissive art, losses in ongoing wars, looming disasters, and pandemic details.

Overcoming Bias · EN

The Promise of Polymaths

Academia neglects interdisciplinary work that combines insights from multiple areas.

Overcoming Bias · EN

Trump hijacks SNL and David S. Pumpkins ahead of midterms

Tom Hanks and Bobby Moynihan cameo vs. SNL's Trump. Host Inde Navarrette took a swipe at previous host Shane Gillis

Salon · EN

Haley Stevens backs Mike Rogers over fellow Democrat Abdul El-Sayed

Stevens' endorsement exposed a bitter Democratic divide as Michigan's closely watched Senate race nears its end

Salon · EN

Senator Chris Murphy “awake and alert” after suffering a stroke

Murphy, 53, experienced arm numbness at his son's hockey game Saturday and then hospitalized for treatment

Salon · EN

The power of Latinas’ Trump regret could swing this election

Latinas were less likely to back Trump but their slide to the right helped fuel his reelection

Salon · EN

The strange new math behind the price of your groceries

As stores embrace algorithms, the cost of groceries is becoming a question of whom — and what — we trust

Salon · EN

Fox News loses its grip on the culture wars

Coded bigotry is harder to sell when MAGA gets unfiltered garbage online

Salon · EN

The dark paradox of Israel’s election — and ours

After last week's big political surprise, Israel has an opportunity for historic change. So do we

Salon · EN

Ali Velshi is on the Big Board now

The MS NOW anchor on early midterm numbers and the stories behind the election-night Big Board

Salon · EN

Why Global Trepanation?

...

Astral Codex Ten · EN