Thursday, March 13, 2014
If you need to block all bots except for those that matter, look at this modified robots.txt snippet.

User-agent: Googlebot
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-Google

User-agent: *
Disallow: /

This script allows Google and Microsoft to index your website and will help to radically reduce bandwidth consumption on your MyBB forum.

posted on Thursday, March 13, 2014 6:12:35 PM (South Africa Standard Time, UTC+02:00)  #    Trackback
Related posts:
MyBB Cache Static Content
MyBB Block (Bad) Robots - Part 2