If you need to block all bots except for those that matter, look at this modified robots.txt snippet.


User-agent: Googlebot
User-agent: msnbot
User-agent: bingbot
User-agent: AdsBot-Google
Disallow:

User-agent: *
Disallow: /



This script allows Google and Microsoft to index your website and will help to radically reduce bandwidth consumption on your MyBB forum.