Skrentablog

That's pretty funny about the index:-)

Rich,

This is an interesting post, but the text here implies that the Nutch crawler is "ill-behaved", which isn't the case. I'm one of the original contributors, and we wrote the crawler to observe every one of the polite practices you list. It won't even run out of the box - the user is forced to indicate a proper user-agent first.

Of course it's possible to force Nutch to do impolite things (it's open source), but the user has to be actively ill-intentioned, not simply a newbie or student.

Mike, sorry about that -- I didn't mean to imply that Nutch was a bad crawler. The suspicious webmasters do however frequently ban all open source crawlers as a matter of practice (along with things like wget), as a hurdle to keep the amateurs that downloaded some code and are running against the web off of their sites.

e.g.:
http://www.webmasterworld.com/search_engine_spiders/3407574.htm

Someone using Nutch would do well to customize the user-agent to avoid such bans.

How nice of you to supply this UNBLOCKED and indexed list of all of our site's robots.txt urls to make things easier on the site scrapers. You should not be linking to our robots.txt in this manner indexed or not.

Thw Twiceler bot is blocked from all of our sites merely for it's bad behavior.

I crawls where it should not be and basically wreaks havoc on a site when it does.

Blocking the user agent in our robots.txt has not been even effective as the bot continues to crawl how it sees fit.

If you block its IP it merely returns within seconds with a new one. We have documented a ton of IPs from this bot in many different c-classes and it always returns.

Yeah it kind of x-rays certain affiliated groups of spammy sites, doesn't it. Grouping sites by robots.txt sigs, I think I saw that mentioned in research paper somewhere. :)

Cuill is a reputable company and crawls from a known set of IPs which they list on their site. If you are getting robots violations and they are coming from other IPs the most likely bet is that someone is spoofing their user-agent to scrape your content. It's not Cuill.

Hi,

My site is listed in that list which has banned Cuill.

And yes, I did ban cuill. I must explain what happened and why I banned it.

My server started to crash on application pools. I am not sure if this is the right terminology. Basicly, The application process was hanging at recycle and causing the whole iss to han gup; so not only the particular website stopped working, but also the other sites hosted on the same server.

After hours of debugging, I noticed that cuill (which I had never heard of before) was crawling the site aggressively.

I immediately emailed Mike and asked him to stop crawling my site. and to be sure, I also banned it.

Even though it is banned, it still attempts to crawl my site.

You know the saying: once bitten twice shy...

As a webmaster I get a bit tired of constantly having to deal with the startup crawler du jour.

From law firms looking for DMCA violations to verticals search engines, to image aggregators, to company intelligence resellers... It feels to me that everybody and their brother has gotten into spidering sites.

With 10,000s of pages that have content that is only relevant to a targeted audience who is perfectly able to find us on the majors, I do not hesitate to block (and possibly ban) when I see an aggressive crawler that does not provide me or my customers with direct benefits.

I don't see the point why the robots-files of the banning sites are listed on your site. Even more so I don't see the point why they are LINKED. Helping other start up crawlers/scrapers/email collectors by showing them the way?

That'll be 10,001 sites now: I've watched the accursed Twiceler 0.9 sucking garbage content out of my spam trap in cgi-bin for the past week and have reported this problem a second time. I can only hope that a company spending that much money can get Twiceler 1.0 to actually take notice of the robots.txt file?

And garbage it is. Here's a part of my weblog, as it is for several months now. Cuill is eating more than two times more than the Google bot, the only one i really want to read my site. Since I pay for bandwidth, I guess I'll send them the bill. Let's hope the number of bans runs to a million before they see the light.

1 19711 19.05% 19711 26.15% 281056 31.19% 3 0.11% crawl-16.cuill.com
2 8747 8.45% 8735 11.59% 129899 14.42% 7 0.26% crawl-66-249-65-6.googlebot.com
3 3627 3.50% 1226 1.63% 15445 1.71% 47 1.71% ip5457af95.direct-adsl.nl
4 3553 3.43% 3295 4.37% 59330 6.58% 14 0.51% ip503d2a0a.speed.planet.nl
5 2990 2.89% 461 0.61% 6030 0.67% 14 0.51% ip5453ed4b.adsl-surfen.hetnet.nl
6 2302 2.22% 2287 3.03% 14440 1.60% 26 0.95% grootammers2.dbinet.nl

I had them crawling my site and it didn't really bother me till i heard lots of bad things about them and the fact it was always on my site day in and day out.
I did mail twicler about removing me from the crawl list, which i was promptly removed from being crawled.

I would think people who have asked to be removed and think they haven't been are actually seeing a look a like which has ill intentions.

Not only do I block them in "/robots.txt", I found their abuse and disregard for that file so obnoxious, I actually now block them in my firewall by their IP netblocks. "Cuill"'s robot repeatedly found its way into my malicious robot traps; something that "well behaved" robots should do if they properly respect the robots.txt control file.

PS: Despite how they "claim" to pronounce their name, it's spelled as if it were Quill, the old feathery writing implement. Maybe the true idea of their robot was to make us feel as if we had been stuck by a whole bunch of quills!