Robots.txt

Discussion in 'SEO, Traffic and Revenue' started by Nick, Jun 4, 2009.

  1. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,441
    Likes Received:
    218
    What areas of your site (if any) do you 'ask' the search engine spiders to not crawl, via a robots.txt file?
     
  2. theboss

    theboss Newcomer

    Joined:
    Jun 14, 2009
    Messages:
    10
    Likes Received:
    0
    nope i want it to find all :P
     
  3. Peggy

    Peggy Regular Member

    Joined:
    May 18, 2009
    Messages:
    1,641
    Likes Received:
    58
    no you don't, really. Most members don't want their profile info listed in the search engines ;)

    You want the spidies to concentrate on content, not your registration page, or usercp page, etc.

    This is what my robots.txt file looks like -

     
    2 people like this.
  4. Demo

    Demo Regular Member

    Joined:
    Jun 9, 2009
    Messages:
    172
    Likes Received:
    6
    Location:
    Europe
    First Name:
    Demo
    Mostly vbulletin files, which can create double content or aren't content-related like ajax.php
     
  5. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston
    Yeah you want a lot blocked off. Not that many bot really pay attention to the text anymore :/

    You also never want to mention your admincp or modcp areas or files in your robots.txt. It's a big no no

     
  6. Michael Biddle

    Michael Biddle vBSEO Mentor

    Joined:
    Jun 6, 2009
    Messages:
    11
    Likes Received:
    6
    First Name:
    Michael
    Just a FYI, do not copy Kevin's above. It looks like he is using vBSEO without an extension (.html). If you use his, all of your content would be blocked.
     
    2 people like this.
  7. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston

    Yes! Thank you. hahaha Every single one of your files would be blocked ;)

    I use no extensions at all. I actually use briansol's setup.

    I probably should have mentioned that.
     
  8. doodles

    doodles Adept

    Joined:
    May 28, 2009
    Messages:
    114
    Likes Received:
    2
    First Name:
    Lisa
    Mine

     
  9. Nick

    Nick Regular Member

    Joined:
    Jul 27, 2008
    Messages:
    7,441
    Likes Received:
    218
    We don't use extensions here (unless I forgot to change some of them...), so I'm assuming it's safe to use your robots.txt?
     
  10. Bundy

    Bundy Admin Talk Staff

    Joined:
    Apr 19, 2005
    Messages:
    842
    Likes Received:
    36
    Location:
    Boston
    Yeah, it looks like your setup is pretty much the same as mine. Except you use 'Keywords' in your url's and I use all numbers for the most part. But that wont matter at all ;)
     
    2 people like this.
  11. Michael Biddle

    Michael Biddle vBSEO Mentor

    Joined:
    Jun 6, 2009
    Messages:
    11
    Likes Received:
    6
    First Name:
    Michael
    Looking at your URLs you can use his as well.
     
    2 people like this.

Share This Page