Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
arXiv:2607.14447v1 Announce Type: new Abstract: AI assistants increasingly retrieve web content at inference time to provide fresh and grounded answers, yet it remains unclear whether these search-augmented capabilities respect website-owner restrictions expressed through robots.txt. We present a controlled empirical study of ten widely used AI assistants with advertised web-search capabilities. For each assistant, we first identify a configuration that actually produces observable web-browsing
Record details
Published: 17 July 2026
Source: arXiv cs.CY
Category: Research
Topics: Agents & autonomy
Retrieved: 17 July 2026
Related evidence
These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.
Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
arXiv · 16 July 2026
Traccia: An OpenTelemetry-Based Governance Platform for AI Systems
arXiv cs.CY · 17 July 2026
Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving
arXiv · 17 July 2026
The labor-saving paradox in transition: An empirical study on the inverted U-shaped relationship between industrial robot application and overtime work in Chinese firms
Technological Forecasting and Social Change · 17 July 2026
PACE: Persona Adaptation through Conversational Elicitation in Human-Robot Interaction
arXiv cs.HC · 17 July 2026
PACE: Persona Adaptation through Conversational Elicitation in Human-Robot Interaction
arXiv cs.HC · 17 July 2026
How to cite this record
ethics.ai (17 July 2026), “Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers,” evidence record 10975, https://ethics.ai/record/10975 (originally published by arXiv cs.CY).
Use and limitations
This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.