About the Siteliner Bot

Siteliner is a website analysis tool by Indigo Stream Technologies, the people behind Copyscape.

Since Siteliner provides its results in real time, it performs a controlled crawl of up to 250 pages of a website in a limited amount of time, while ensuring that no excess load is placed on the web server for that site. All Siteliner bot requests are identified by this user agent string:

Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)

To avoid placing excessive load on your website, Siteliner applies the following protections:

Siteliner only retrieves HTML web pages, not embedded images or videos, so it only uses a small amount of bandwidth.
A limit of 4 page requests sent at any one time to your server. This load is similar to that of a regular web browser retrieving an HTML page with embedded images, style sheets and scripts.
A limit of 250 pages retrieved from a single site during a Siteliner analysis.
Each website can only be analyzed by Siteliner users once per 30 days.
This rate limit is applied at the level of your server's IP address, rather than its domain name. This ensures that Siteliner cannot be used to generate excessive load on a single server which hosts multiple sites under different domain names.
A hidden Javascript captcha that makes it difficult for scripts to automate Siteliner runs.

Blocking Siteliner

If you wish to prevent Siteliner from analyzing your website, you may use a standard robots.txt file in the root directory of your website. Use User-agent: Siteliner to target rules to the Siteliner bot. For example, to block all Siteliner access to your server, simply place the following at the end of your robots.txt file:

User-agent: Siteliner Disallow: /

If you have any questions or concerns about the Siteliner bot, please feel free to contact us.