TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment
On-policy self-distillation (self-OPD) densifies reinforcement learning with verifiable rewards (RLVR) by letting a policy teach itself under privileged context. We find that when this guidance spans the full response, all-token KL spends gradients on mostly redundant positions and amplifies privileged-information leakage, causing entropy rise, shortened reasoning, and out-of-distribution degradation in long-horizon math training. We propose Token-Routed Alignment for Critical rEasoning (TRACE),
Record details
Published: 11 May 2026
Source: arXiv
Category: Research
Topics: Regulation · Safety & alignment
Retrieved: 14 July 2026
Related evidence
These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.
Positive Alignment: Artificial Intelligence for Human Flourishing
arXiv · 11 May 2026
Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization
arXiv · 11 May 2026
Guided Streaming Stochastic Interpolant Policy
arXiv · 11 May 2026
On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment
arXiv · 12 May 2026
BSO: Safety Alignment Is Density Ratio Matching
arXiv · 12 May 2026
ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
arXiv · 12 May 2026
How to cite this record
ethics.ai (11 May 2026), “TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment,” evidence record 4576, https://ethics.ai/record/4576 (originally published by arXiv).
Use and limitations
This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.