A
ai.robots.txt
ai-robots-txt/ai.robots.txt
A list of AI agents and robots to block.
★4.0kstars
Python
MIT
Updated: Today
📋 Project at a Glance
Tap to expand
What's this?Security & Governance project leveraging crawlers and crawling, built with Python, open-source
Who made it?Maintained by ai-robots-txt team, 4K⭐ on GitHub, #65 out of 413 in Security & Governance
Why does it exist?With the growing demand for crawlers and crawling in Security & Governance, ai-robots-txt created ai.robots.txt as a streamlined solution.
What can it do?Key use cases: privacy
How to install with AI?Use an AI coding assistant (Claude Code, Cursor, Copilot) to automatically set up pip dependencies and virtual env. Follow the README — the AI handles the rest.
🔗 github.com/ai-robots-txt/ai.robots.txt | 官网 https://github.com/ai-robots-txt/ai.robots.txt/releases.atom
🔗 github.com/ai-robots-txt/ai.robots.txt | 官网 https://github.com/ai-robots-txt/ai.robots.txt/releases.atom
Topics
aicrawlerscrawlingprivacy