View Full Version : Robots.txt

06-01-2007, 02:15 PM
I've been browsing through my webstats and found that bots are searching for the page robots.txt in order to scan my site. I was on other sites like google/paypal, and say that they have robots.txt pages and i was wondering what mine should consist of, have any of you got this file that i could upload to my site.

06-01-2007, 05:45 PM
You dont really need it unless you dont want search engines to index some of your pages (like admin pages etc), but it doesn't really matter because you'd still need to log in to see it

06-27-2007, 11:54 PM
It should basically consist of this:

User-agent: ia_archiver
Disallow: /

Save it as robots.txt in your public_html fodler and it stops webcrawlers and robots from going through your site.