Announcement

Collapse
No announcement yet.

Robots.txt

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Robots.txt

    I've been browsing through my webstats and found that bots are searching for the page robots.txt in order to scan my site. I was on other sites like google/paypal, and say that they have robots.txt pages and i was wondering what mine should consist of, have any of you got this file that i could upload to my site.

  • #2
    Re: Robots.txt

    You dont really need it unless you dont want search engines to index some of your pages (like admin pages etc), but it doesn't really matter because you'd still need to log in to see it

    Comment


    • #3
      Re: Robots.txt

      It should basically consist of this:

      User-agent: ia_archiver
      Disallow: /


      Save it as robots.txt in your public_html fodler and it stops webcrawlers and robots from going through your site.

      Comment

      Working...
      X