Zyte · AI crawler
Scrapy
Model trainingUnclearAI Data Scrapers
"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets."
Official documentation ↗
🛰️ Observed crawling piperic.com unverified
13 requests · last seen 16m ago · 1 IP(s)
Fetched: HTML (9) · other (4)
Block Scrapy in robots.txt
Add this to your site’s /robots.txt:
User-agent: Scrapy
Disallow: /
← All AI bots