Bad bots, scrapers and others eating your server resources, bandwidth, etc - Solution is here!

[Nemanja]

If you are not suffering from this kind of bots, scraper, harvesters and others, you will sooner or later , trust me

OK, what are these bad bots, basically bad bots are spiders from which you have more damage than good.
They harvest your content, scrape emails, contact info, you name it and they also eat your bandwidth and server resources.

There are few ways you can block them, like with robots.txt(yeah right ), some custom made scripts and htaccess file.
Problem with robots.txt is that bad boots don't obey and follow what is in it,
only good bots obey robots.txt and we don't have problems with good bots like googlebot, bingbot, slurp, ...in the first place.
So robots.txt is just a reference to the spiders which they "might" follow.

So next logical solution would be .htaccess file, because it is very easy to implement and configure one.
If you don't have one in your root dir of your server just create one naming it ".htaccess", it is that easy

Some nasty bots where "attacking" my sites,
so I was logging all agents/spiders/scrapers, IPs into the database and ended up with nice list of bad boots and bad IPs .
Then I compiled the htaccess code you need to copy/paste into your .htaccess file.

On some sites in past 2 days my bandwidth usage dropped down by 10 times and resource usage also lowered drastically.

So here is the code you need to copy/paste, please note that I have blocked Baidu and Yandex spiders as well.

Options -Indexes

RewriteEngine on
<IfModule mod_rewrite.c>

#Block bad boots - Nemanja additions:
#Yandex &/ Baidu - you can remove this if you wish
RewriteCond %{HTTP_USER_AGENT} Yandex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Jakarta [NC,OR]
RewriteCond %{HTTP_USER_AGENT} magpie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} August [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Pipes [NC,OR]
RewriteCond %{HTTP_USER_AGENT} FairShare [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Ezooms [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Konqueror [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Twingly [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Seznam [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Feed [NC,OR]
RewriteCond %{HTTP_USER_AGENT} alexa [NC,OR]
RewriteCond %{HTTP_USER_AGENT} bender [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Aghaven [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Soso [NC,OR]
RewriteCond %{HTTP_USER_AGENT} gooblog [NC,OR]
RewriteCond %{HTTP_USER_AGENT} RSS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} butterfly [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Twee [NC,OR]
RewriteCond %{HTTP_USER_AGENT} gnip [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Fetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} js-kit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Twit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} MetaURI [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Spinn [NC,OR]
RewriteCond %{HTTP_USER_AGENT} spbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ICS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Backlink [NC,OR]
RewriteCond %{HTTP_USER_AGENT} TheFreeDictionary [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Deepnet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} discobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SPUTNIK [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Zend_Http_Client [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yeti [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WordPress [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DigExt [NC,OR]
RewriteCond %{HTTP_USER_AGENT} OffByOne [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sogo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} trendiction [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Indy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wasa [NC,OR]

deny from 178.168.31.223
deny from 209.217.244.58
deny from 121.74.248.22
deny from 199.15.234.23
deny from 188.163.107.228
deny from 41.233.16

Take care and block those nasty bots

Edit: I found that code from other people are not working and have some issues so I ended up only with mine code.

[Nemanja]

Please note that this file is updated now, I removed some of the code which was making some mess.

[Johnr]

Thanks for sharing such kind of information

[samples]

thank you great info

[Nemanja]

Just to update on this..
On some sites bandwidth dropped 20 times

[Nemanja]

Last day and a half my server was facing some high load.
I was searching for the issue without any luck, but finally I found out what was causing it.
It was same nasty bot/spider:
41.233.16.139 : host-41.233.16.139.tedata.net
so I have blocked the whole range of that class c IP in htaccess file:
"deny from 41.233.16"

I think that there are more bots like this out there, so when I found more I will post them here.
In a meanwhile be sure to block them, because they are eating your resources.

Code from above is edited also.

[gamer]

you are great, thanks ! Yet I'm not under bot attack, but I inserted the list in my htaccess for security.

[bluearrow]

This little code is very useful ! Bot can be real pests if you run your own vps or dedis. Thank Nemanja.

[stronywwwlublin]

Very nice snippet! Please add also 87.250.224.0 Yandex IP.

Title of thread reminds me Bob Marley song like "bad bots, bad bots, what you gonna do..." hahah

[Nemanja]

Hahahahah

[Phoenyx]

Very nice snippet! Please add also 87.250.224.0 Yandex IP.

Title of thread reminds me Bob Marley song like "bad bots, bad bots, what you gonna do..." hahah

Lmao, inner circle also made a version

[Nemanja]

One more thing I like to add, if you use code in your htaccess that looks like this(as from above):
"RewriteCond %{HTTP_USER_AGENT} Yandex [NC,OR]"
then every user agent with "Yandex" in his name will be blocked,
so some restrictions like:
"RewriteCond %{HTTP_USER_AGENT} Feed [NC,OR]"
can block maybe some bot or visitor with "Feed" in his user agent description.
To avoid this you can either use more specific word for each bot,
or you can use "^Feed" which will restrict user agent having name started with "Feed"

[seobunny]

Hello Nemanja... is this List up to date?

Or can you repost your newest htacces file please.

[glitterasia]

this code not working

[anthonymcauley]

Very good Nemanja. I'll load it up and block the bad babies

[chris]

@ seobunny...is the list up to date...?