Spiders you block

If you block any spiders in robots.txt or by IP which ones are they?

I have had quite a lot of baiduspiders or harvesters beginning to get on my nerves recently and I am considering completely blocking them, also Yandex seems to be quite a nuisance as of late, is there any benefit allowing these to crawl your content?

Thanks

 

Nope. Dont block any.

How does a spider get on your nerves ??

 

Even if there isn't a benefit, I'm wondering the same thing as Paul.

 

By constantly crawling the site with at least 100-200 bots all online at once causing high server load for us when it really isnt necessary as well as ignoring robots.txt rules.

 

I have seen lately that my site has been visited by hundreds of these spiders from this ip 94.102.63.60

Anyone know how I can hide this spider on Who's online

 

I welcome all bots
if my hosting can't handle the traffic for spiders, I would get better hosting

 

Even if they ignore your robots.txt rules? Surely any robot that ignores robots.txt shouldnt be able to crawl your site and isnt behaving as it should. Our hosting is capable of handling the bots but not when theyre on for 6+ hours and are all from yandex and baidu.

@quantnet you may want to ban them in htaccess by IP if its just that one IP.

 

I still hold to my reply
all bots are welcome on my site, make your forums private if you don't want them on it
and maybe you have your robots.txt file setup wrong? it will take a few days/weeks to clear out results if you've recently edited the robot.txt file.

 

Oh, I had it in my head they downloaded it every few hours or so lol well I have decided to completely block yandex and baidu.

 

I believe I've blocked boardtracker. They basically monetize your content on their site. Almost like a digg for forums, I don't get the point other than making the owner money and at one point a couple years ago BT outranked my site for my own content, I didn't take very well to that.

I just jumped on the site it looks to of changed significantly from when it first came on my radar but I'm still not sure about it.

 

I don't block any spiders. I don't see a reason too.

 

I 2nd that. Bots should be welcomed to your site, usually the main goal is to index your content and show it to other people so hopefully it becomes another channel of content to you.

Now if they were copying your content and not linking to you that would be a different story - and you should contact them rather then just blocking outright (i've done that in the past and they just stopped crawling us)

And if your site can handle bots crawling, upgrading to another host would be a good idea. I might even suggest upgrading to forum software that is more optimized at turning out pages - if bots are hurting your performance than that means each additional person on your site is hurting performance as well - and you should try to avoid that.

 

I do not block any spiders on my site. Sure, some like Yahoo's spiders come in ridiculous amounts sometimes, but I simply limit how fast they view the site to stop all that problem, and it's far better for any practical purposes than blocking them. I don't block the Internet Archive bot unlike some people I know, since I actually like being able to see what a website looked like a few years back, and it infuriates me to see all these sites blocking off their site's history purely for the sake of a tiny bit of bandwidth.