Robots.txt

Discussion in 'SEO, Traffic and Revenue' started by Nick, Jun 4, 2009.

  1. Nick

    Nick Regular Member

    7,441
    218
    458
    What areas of your site (if any) do you 'ask' the search engine spiders to not crawl, via a robots.txt file?
     
    Nick, Jun 4, 2009
    #1
  2. theboss

    theboss Newcomer

    10
    0
    35
    nope i want it to find all :P
     
    theboss, Jun 14, 2009
    #2
  3. Peggy

    Peggy Regular Member

    1,641
    58
    434
    no you don't, really. Most members don't want their profile info listed in the search engines ;)

    You want the spidies to concentrate on content, not your registration page, or usercp page, etc.

    This is what my robots.txt file looks like -

     
    Peggy, Jun 14, 2009
    #3
    2 people like this.
  4. Demo

    Demo Regular Member

    172
    6
    104
    Mostly vbulletin files, which can create double content or aren't content-related like ajax.php
     
    Demo, Jun 14, 2009
    #4
  5. Bundy

    Bundy Admin Talk Staff

    842
    36
    664
    Yeah you want a lot blocked off. Not that many bot really pay attention to the text anymore :/

    You also never want to mention your admincp or modcp areas or files in your robots.txt. It's a big no no

     
    Bundy, Jun 14, 2009
    #5
  6. Michael Biddle

    Michael Biddle vBSEO Mentor

    11
    6
    37
    Just a FYI, do not copy Kevin's above. It looks like he is using vBSEO without an extension (.html). If you use his, all of your content would be blocked.
     
    Michael Biddle, Jun 14, 2009
    #6
    2 people like this.
  7. Bundy

    Bundy Admin Talk Staff

    842
    36
    664

    Yes! Thank you. hahaha Every single one of your files would be blocked ;)

    I use no extensions at all. I actually use briansol's setup.

    I probably should have mentioned that.
     
    Bundy, Jun 14, 2009
    #7
  8. doodles

    doodles Adept

    114
    2
    104
    Mine

     
    doodles, Jun 14, 2009
    #8
  9. Nick

    Nick Regular Member

    7,441
    218
    458
    We don't use extensions here (unless I forgot to change some of them...), so I'm assuming it's safe to use your robots.txt?
     
    Nick, Jun 14, 2009
    #9
  10. Bundy

    Bundy Admin Talk Staff

    842
    36
    664
    Yeah, it looks like your setup is pretty much the same as mine. Except you use 'Keywords' in your url's and I use all numbers for the most part. But that wont matter at all ;)
     
    Bundy, Jun 14, 2009
    #10
    2 people like this.
  11. Michael Biddle

    Michael Biddle vBSEO Mentor

    11
    6
    37
    Looking at your URLs you can use his as well.
     
    Michael Biddle, Jun 14, 2009
    #11
    2 people like this.

Share This Page