Graciebot |
Graciebot is the spider or robot basscrawler.org uses to crawl the Web, looking for bass tablature to index for the www.basscrawler.org search engine. Graciebot, obeys spider Netiquette rules: It checks the Robot Exclusion File and it avoids hitting each site too frequently. to stop Graciebot from spidering your website send an email to staff@basscrawler.org or use your robots.txt file in the method described below.
How to Use the Robot Exclusion File to Stop Graciebot |
Robot Exclusion File Example: The standard, easy way to control the behavior of Graciebot on your website is to use a "robots.txt" file, a simple text file which you place in the root of your webserver.
A very simple robots.txt will keep Graciebot away from your website completely:
User-agent: graciebot
Disallow: /
To keep all spiders away including Graciebot, you can use a simpler file:
User-agent: *
Disallow: /
A slightly more complicated robots.txt allows you to keep us (or others) away from selected parts of your webserver document tree, but allows us to crawl any other part of your document tree.
Use something like this:
User-agent: *
Disallow: /admin
Disallow: /chat
Disallow: /cgi-bin
Disallow: /config.html
No well-behaved spider will crawl into or below the named directory, nor will it read the explicitly named file, config.html.
| Graciebot ® -- www.basscrawler.org --- © 2005 |
