When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents
Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model. We test this proxy for pre-action policy classification in tau-bench using Qwen2.5-3B/7B classifiers. Under gold-policy conditioning, a compact structured state improves macro-F1 over raw trajectories by 0.13-0.17 after tuning. We then replace the benchmark-designated policy clause with the top-ranked clause retrieved from decision-time context. Although th
Record details
Published: 22 June 2026
Source: arXiv
Category: Research
Topics: Regulation · Agents & autonomy
Retrieved: 14 July 2026
Related evidence
These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.
Cryptographic certificates of validity for trustworthy AI
arXiv · 22 June 2026
Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning
arXiv · 22 June 2026
A Stackelberg Framework for Resource-Aware LLM Agents: Learning, Repair, and Conditional Guarantees
arXiv · 22 June 2026
Governed Shared Memory for Multi-Agent LLM Systems
arXiv · 23 June 2026
The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
arXiv · 23 June 2026
Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents
arXiv · 21 June 2026
How to cite this record
ethics.ai (22 June 2026), “When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents,” evidence record 678, https://ethics.ai/record/678 (originally published by arXiv).
Use and limitations
This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.