You may find putting some content like this on a page listing acceptable bot policies worthwhile.
Use this with the
You are encouraged to cache the results of your bot's requests, which saves the site bandwidth
and CPU time. Bots making repeated requests on the same resource (URI) in a short
amount of time will be blocked. Please do not multi-thread your bot to access multiple resources at the same time
and do not connect more than
five times per second.
Well-Formed User Agents. All bots are required to have a well-formed user agent which includes a contact e-mail address for the bot maintainer, and preferably a URL to the organization running the bot. Bots without this information have a higher chance of being blocked. An example of a well-formed user agent is:
Contact Information. If you need specific information for research or data-collection purposes,
feel free to contact us at
If we have blocked your bot and you'd like to contact us about it, please e-mail us