Posted on

How to add websites

Summary: This guide covers how to add a new website to an account. It also includes a description of three different methods that can be used to add URIs to a website’s URI list (pages to be scanned).

Permissions

Only users with permission to “Manage Websites” are able to manage the websites and set Web pages to a website’s URI list.  (For more about assigning user permissions and roles, review the guide How to manage users roles and permissions.)

Where to add a new website

To add a new website, navigate to the “Websites” view and activate the “Create New Website” button.

Activating the “Create New Website”” button will take you to the “New Websites” view where you specify the Base URL for the website you want scanned. You can also set the crawler options for the website from this location as well.

Adding and preparing a website for scanning

There are three parts to adding a preparing a website for scanning, they are:

  • Setting the Base URL
  • Adding pages to the website
  • Managing URI lists

Setting the Base URL

Setting the Base URL includes adding a new website’s URL and setting the crawler settings that will be saved to that website.

Steps to adding a new website’s Base URL

  1. Enter the Base URL for the website you want included in your scanning account. It is recommended that you verify URL in a browser before adding it.
  2. (Optional) Add a name for the website. You can assign the website any name you want. If nothing is added, this will default to the Base URL when you save the website.
  3. If you want to assign the new website to a group, you can do that here. If no group is selected, the website will be added to accounts main group. Group assignments can be modified at a later time either at this same location or on the “Groups” page. (The guide How to manage groups provides more information changing group settings.)
  4. Modify any Crawler Options that you want applied to this Base URL. When a crawl is initiated, these are the rules the crawl will follow. The default crawler options are:
    • Max Pages =30
    • Max Depth = 4
    • No Subdomains are included in the crawl
    • No whitelist/blacklist filtering
  5. Save your Base URL and any Crawler Options you modified. It is important that if you modified any Crawler Options that you save them.

(Note: Also in Crawler Options, you can set advanced crawler settings like specifying filter settings to blacklist or whitelist directories, Web pages, or characters/symbols in the URIs.)

After the Base URL (and any Crawler Option) has been saved, you can begin adding Web pages to your URI list.

Add pages to the website

There are three ways to add pages to a website’s URI list: (1) Manually, (2) Crawling (the most common method), and (3) Uploading CSV list of URIs.

Method #1: Add URIs manually (one at a time)

You can manually add URIs to the page list.

Example use cases:

  • You blacklisted a directory from a crawl and then you want to go back and add one or two pages manually from that directory.
  • You just added a new page to your website, but you don’t want to recrawl the whole website to include it in the next scan.

(Note: Adding a page manually will append it to the any existing URI list. If you redundantly add a URI that is already in the list, it will be included twice in any scan subsequent scan of this website).

Steps to Manually Adding a page

  1. Activate the button “+ New” button.
  2. Enter a page title (optional; if this is left blank, during the scan it will be populated using the actual Web page’s title located during the scan.
  3. Enter the URI. You do not need the website base. If you wanted the page https://pope.tech/about, and the base URL was https://pope.tech, you would only enter /about.
  4. Activate the save button to import the URI
  5. The URI should now appear in the URI list below

Method #2: Crawl a website (Most commonly used method to add pages)

The most common way to URI to a website is to crawl the page. Unless otherwise specified in the Crawler Options and saved to the Base URL, the default crawler settings will be used to crawl the website (the defaults, as well as how to modify them, are previously described in this same guide).

Steps to crawling a website:

  1. Selected the Crawl Button
  2. (Optional) You may elect to begin the crawl on a location other than the BASE URL (for example a sitemap). If you leave this blank, the crawl will begin at the Base URL.
  3. Start the Crawl, by selecting the Start button.
  4. If you refresh the current view when there is an active crawl, a Crawlers dialogue box appears showing the status of the crawler for that website at the time of the refresh. To update this dialogue box, refresh the page again. When the crawl is completed, on a refresh this dialogue box will no longer display.
  5. On page refreshes, the list of the URIs will be located at the bottom of the Widget.

It should be noted that each additional crawl of the same website will append the new pages found to the URI list and will be included in your scanning account. Crawling is set to not add duplicates. If you want to increase the precision of your crawl, you can recrawl your site.

Method #3: CSV upload of the pages

Example use cases:

  • Sometimes users already have a list of URIs that they want to use instead of crawling their website.
  • CVS upload can also be used to append new URIs to the current URI list (add a list new Web pages without recrawling the site).

A CSV upload is an easy way to do either of these.

Steps to uploading a CSV URI list

  1. Select the “Upload URI List” button.
  2. Select the “Choose File” button and browse to choose your CSV file from your computer.
  3. Once you have selected your CSV file, it should appear next to the “Choose File” button.
  4. Import the URIs in the CSV file by selecting the “Save” button.
  5. The newly added URIs should now be included in list located at the bottom of the widget.

For more information on uploading CSV files including information on the file structure, you can reference our guide on CSV Upload.

Note: A CSV upload appends the new URIs to any existing URIs in your list. Unlike a crawl, uploading a duplicate URIs to the URI list will include that URI twice into your scanning list.

Manage URI lists

Once the URIs are added to a website, they can be managed from the same page. You can search and filter by URI or Page Title, you can also archive, delete, or move single pages, or selected pages.

The guide Advanced management of URI lists provides more information on managing URI lists.

Returning to a website’s settings

If you ever want to come back to these settings to review or revise them, you can do by activating the Websites button in the main menu, locating the website in the Websites list, and activating the “Actions” button for the desired website.