Summary: A detailed look at the Crawl settings and features. Options for how to add pages into your website to be scanned. Overview Crawler Purpose. Differences between Crawl and Scan Website Crawl – Quick Start Default Crawl behavior – Sitemap Import Traditional Page Crawl Crawler Options Crawl Limitations Troubleshooting a Crawl Additional Information Crawler Purpose…
Pope Tech release notes for October 2020. Oct 29, 2020 Errors by Group Dashboard widget updates The chart has been updated to be a horizontal bar chart with each category in its own bar underneath the one above instead of stacking in one bar. This change was made to make it more clear what the…
Forbidden by robots.txt is the most common crawl error type. The Pope Tech crawler respects your robots.txt file. To get around this simply update the robots.txt file found on yourdomain.com/robots.txt. You can leave it to block other bots but add in a line to specifically allow the Pope Tech user agents. Example robots.txt entry: User-agent:…
The defaults for two crawler settings, “Max Pages” and “Max Depth” can be set at the organization level. Once these are set, all new websites created will have this default. Setting new organization crawler defaults will not override any existing individual website’s crawler settings. You can adjust individual websites crawler settings in that website’s settings.…