Crawler to extract the newest articles in the publisher's website (via feed or parsing html) to make a carrousel with images, links and text for our native ads module in order to improve recirculation in the publisher's web. Only crawls our publisher's webpages.
Add this to your site’s /robots.txt:
User-agent: videootv Bot Disallow: /