BSO: Safety Alignment Is Density Ratio Matching
Aligning language models for both helpfulness and safety typically requires complex pipelines-separate reward and cost models, online reinforcement learning, and primal-dual updates. Recent direct preference optimization approaches simplify training but incorporate safety through ad-hoc modifications such as multi-stage procedures or heuristic margin terms, lacking a principled derivation. We show that the likelihood ratio of the optimal safe policy admits a closed-form decomposition that reduce
Record details
Published: 12 May 2026
Source: arXiv
Category: Research
Topics: Regulation · Safety & alignment
Retrieved: 14 July 2026
Related evidence
These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.
ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
arXiv · 12 May 2026
On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment
arXiv · 12 May 2026
Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
arXiv · 13 May 2026
Positive Alignment: Artificial Intelligence for Human Flourishing
arXiv · 11 May 2026
TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment
arXiv · 11 May 2026
Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization
arXiv · 11 May 2026
How to cite this record
ethics.ai (12 May 2026), “BSO: Safety Alignment Is Density Ratio Matching,” evidence record 4452, https://ethics.ai/record/4452 (originally published by arXiv).
Use and limitations
This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.