#adversarial
3 discoveries · Hand-picked discoveries from the open web, newest first.
-
Combatting adversarial distillation
OpenAI’s layered approach to adversarial distillation combines model training, safeguards, detection, and enforcement to protect against capability extraction.
OpenAI Blog · EN
-
Confirming Claims of Superposition and Adversarial Examples in Vision Transformers
Figure 1: The ViT classifier with a encoder and decoder at the final layer to enforce superposition on the model This is a replication of Adversarial Attacks Leverage Interference Between Features in Superposition , completed as part of the Second Look Summer Fellowship. tl;dr In a previous blog we replicated the toy models examples from Stevenson et al. In this work, we reproduce the vision transformer experiments from the paper: On every dataset, we find that increases in superposition lead to adversarial attacks that line up more closely with class-vector geometry. Likewise, trends across a
LessWrong · EN
-
Adversarial Training Can Further Hide a Backdoor, but Does It Improve Trigger Identification?
Summary The Sleeper Agents paper demonstrated that adversarial training on red-team prompts failed to remove a planted backdoor. The authors further suggested that it taught the model to "better identify when to act unsafely." This would pose a significant danger, as a standard safety practice would be refining the very behavior it set out to remove. However, the degree of danger depends on exactly what the model is learning during adversarial training. We discuss how, due to saturation of firing rates, more benign scenarios such as the model's backdoor response being generally weakened are al
LessWrong · EN