Thursday, 13 March 2014
Now for those robots which are bad and do not adhere to your robots.txt's rules, you'll need to add specific htaccess code.
What the code below will do is check the bot's user agent name, and if its empty or equal to "*bot" or "bot*" or "robot" or "crawl" or "spider", it will send them to your error page.

# Block bad bots
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^\*bot$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^bot\*$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^robot$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^crawl$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^spider$
RewriteRule .* - [F,L]

You can quite easily add to this list if you know of more.

posted on Thursday, 13 March 2014 18:18:02 (South Africa Standard Time, UTC+02:00)  #    Comments [0] Trackback
Related posts:
MyBB Cache Static Content
MyBB Block Robots - Part 1
myBB 1.6 Thread Tooltip Preview
myBB Updates
All comments require the approval of the site owner before being displayed.
Name
E-mail
Home page

Comment (HTML not allowed)  

Enter the code shown (prevents robots):