Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
AI assistants increasingly retrieve web content at inference time to provide fresh and grounded answers, yet it remains unclear whether these search-augmented capabilities respect website-owner restrictions expressed through robots.txt. We present a controlled empirical study of ten widely used AI assistants with advertised web-search capabilities. For each assistant, we first identify a configuration that actually produces observable web-browsing behavior and record the user-agent exposed durin
Record details
Published: 16 July 2026
Source: arXiv
Category: Research
Topics: Agents & autonomy
Retrieved: 18 July 2026
Related evidence
These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.
Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
arXiv cs.CY · 17 July 2026
Structural predictors and latent maturity regimes of robotic readiness in global health systems: evidence from machine learning-based latent clustering and class prediction
Frontiers in Robotics and AI · 16 July 2026
Hybrid task and motion planning with reactive collision handling for multi-robot disassembly of complex products: application to EV batteries
Frontiers in Robotics and AI · 16 July 2026
Exploring generative AI through core frameworks, emerging innovations, and applications
Artificial Intelligence Review · 16 July 2026
Correction: Synthetic chamber: agentic mediation in representative democracy
AI & Society · 16 July 2026
An offline approach to fNIRS-guided reinforcement learning for robot behavior
arXiv · 15 July 2026
How to cite this record
ethics.ai (16 July 2026), “Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers,” evidence record 11367, https://ethics.ai/record/11367 (originally published by arXiv).
Use and limitations
This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.