Thursday, March 13, 2014
Now for those robots which are bad and do not adhere to your robots.txt's rules, you'll need to add specific htaccess code.
What the code below will do is check the bot's user agent name, and if its empty or equal to "*bot" or "bot*" or "robot" or "crawl" or "spider", it will send them to your error page.

# Block bad bots
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^\*bot$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^bot\*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^robot$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^crawl$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^spider$
RewriteRule .* - [F,L]

You can quite easily add to this list if you know of more.

posted on Thursday, March 13, 2014 6:18:02 PM (South Africa Standard Time, UTC+02:00)  #    Trackback
Related posts:
MyBB Cache Static Content
MyBB Block Robots - Part 1
myBB 1.6 Thread Tooltip Preview
myBB Updates