Thursday, 13 March 2014
If you need to block all bots except for those that matter, look at this modified robots.txt snippet.


User-agent: Googlebot
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-Google
Disallow:

User-agent: *
Disallow: /



This script allows Google and Microsoft to index your website and will help to radically reduce bandwidth consumption on your MyBB forum.

posted on Thursday, 13 March 2014 18:12:35 (South Africa Standard Time, UTC+02:00)  #    Comments [0] Trackback
Related posts:
MyBB Cache Static Content
MyBB Block (Bad) Robots - Part 2
All comments require the approval of the site owner before being displayed.
Name
E-mail
Home page

Comment (HTML not allowed)  

Enter the code shown (prevents robots):