What areas of your site (if any) do you 'ask' the search engine spiders to not crawl, via a robots.txt file?
no you don't, really. Most members don't want their profile info listed in the search engines You want the spidies to concentrate on content, not your registration page, or usercp page, etc. This is what my robots.txt file looks like -
Yeah you want a lot blocked off. Not that many bot really pay attention to the text anymore :/ You also never want to mention your admincp or modcp areas or files in your robots.txt. It's a big no no
Just a FYI, do not copy Kevin's above. It looks like he is using vBSEO without an extension (.html). If you use his, all of your content would be blocked.
Yes! Thank you. hahaha Every single one of your files would be blocked I use no extensions at all. I actually use briansol's setup. I probably should have mentioned that.
We don't use extensions here (unless I forgot to change some of them...), so I'm assuming it's safe to use your robots.txt?
Yeah, it looks like your setup is pretty much the same as mine. Except you use 'Keywords' in your url's and I use all numbers for the most part. But that wont matter at all
