Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
arXiv:2510.10315v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark co
Record details
Published: 13 August 2026
Source: arXiv cs.CY
Category: Research
Topics: Misinformation · Agents & autonomy · Finance, VC & PE
Retrieved: 13 August 2026
Related evidence
These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.
Governing Agentic AI in FinTech
arXiv cs.CY · 13 August 2026
Turning OSINV inside the machine: Open-source investigations and information disorder
HKS Misinformation Review · 12 August 2026
Effect of a robotic insole-type active assist device on horizontal ground reaction force and center-of-pressure stability during stepping in patients with medial knee osteoarthritis
Frontiers in Robotics and AI · 12 August 2026
Governing Agentic AI in FinTech
arXiv · 11 August 2026
ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners
arXiv cs.AI · 10 August 2026
Graphing the Everyday: A Neurosymbolic Approach to Eliciting Routines for Just-In-Time Adaptive Interventions
arXiv cs.HC · 10 August 2026
How to cite this record
ethics.ai (13 August 2026), “Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web,” evidence record 18734, https://ethics.ai/record/18734 (originally published by arXiv cs.CY).
Use and limitations
This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.