Posted on

Forbidden by robots.txt

Pope Tech crawl results depicting an error of Forbidden by robots.txt

Forbidden by robots.txt is the most common error type. The Pope Tech crawler and scanner respect your robots.txt file. To get around this simply update the robots.txt file found on yourdomain.com/robots.txt. You can leave it to block other bots but add in a line to specifically allow the Pope Tech user agents.

User-agent: Pope Tech CrawlBot (https://pope.tech)
Disallow:

User-agent: Pope Tech ScanBot (https://pope.tech)
Disallow: