Brightbot is a web data collection crawler by Bright Data that extracts and structures public website content at scale, providing AI-ready data for model training, RAG pi…
Official documentation ↗
Add this to your site’s /robots.txt:
/robots.txt
User-agent: Brightbot Disallow: /
← All AI bots