Robots.txt Disallow /posts/ ???

I use XenForo 1.2.2 and I've noticed that many Xenforo forums disallow the /posts/ in their robots.txt files. Why not let robots crawl the /posts ?

User-agent: *
Disallow: /posts/

 

That's strange. I would think that it would be extremely useful to let robots crawl your posts.

 

There must be a good reason.

I also found that many forums disallow all pages in their robots.txt file

 

Pretty much similar to my file

User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /goto/
Disallow: /login/
Disallow: /admin.php
Disallow: /search/
Disallow: search.php
Disallow: /help/
Disallow: /members/
Disallow: /misc/
Disallow: /online/
Allow: /
User-agent: ia_archiver
Allow: /
User-agent: BoardTracker
Disallow: /
User-agent: BoardReader
Disallow: /
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /

 

Well, there is always the chance that one tweak robots.txt to suit particular needs, problem could be that at every forum settings update, the robots file can be modified on the fly.

I remember to have a long, long disallow list including search engines, crawlers and bots, but sometimes I wonder if all of those abide by the rules and refrain from crawl what is on-site.

I read some time ago that the best way to prevent bots from crawling directories or files that we don't want to get crawled is making no reference to them in robots text to avoid they know about their existence... unless they are linked to other with crawling allowance.

 

Well the idea of a robots.txt file is to stop certain bots that obey it, to not waste time looking at files that don't need indexing to speed things up for bots. I wouldn't go daft adding loads of entries, but some files and folders are obvious for blocking. Cache folders and files like install and config.php files e.t.c

It can help a little blocking those obvious files and folders from being indexed. You can also use it to limit crawl numbers, which can save you on bandwidth.

 

No robots.txt means you let all robots crawl all they can?? Is your forim well referenced in Google?

 

I have noticed that more and more web sites and blogs are using the robot.txt file recently. I n the last month I have lost count of how many times I have looked for something in Google and found a site listed with a description saying that robot.txt had blocked the search engine from crawling the site. I didn't think they would index those sites if they were restricted by robot.txt.

 

Some robots bypass the robots.txt file

 

its not very wise thing to do. Of course its none of my business but, I wouldnt do that. Some of your users may post sensitive data and - by means of not using robots.txt (=letting robots crawl your site) - these data get revealed. Its strange policy of yours.

 

Someone said that the golden rule to know if a file like robots.txt should be used or not is typing the address of a huge company followed by robots.txt, like in this particular case, and here you go,

http://www.google.com/robots.txt
http://www.nyc.com/robots.txt
http://www.microsoft.com/robots.txt
http://www.adobe.com/robots.txt

And so on...

 

I checked my competitors.... some uses robots, some others don't

 

I don't have any competitors and when I did most of them used robots.txt to block spiders from crawling their content.

 

Disallowing directories also makes your site less vulnerable (to an extent). You'd be surprised how much informtion a robot can disclose about your website. It can index vulnerable files or sensitive directories, or even show parameters that could lead to MySQL injection.