Thursday, 13 March 2014
Now for those robots which are bad and do not adhere to your robots.txt's rules, you'll need to add specific htaccess code.
posted on Thursday, 13 March 2014 18:18:02 (South Africa Standard Time, UTC+02:00)  #    Comments [0] Trackback
If you need to block all bots except for those that matter, look at this modified robots.txt snippet.
posted on Thursday, 13 March 2014 18:12:35 (South Africa Standard Time, UTC+02:00)  #    Comments [0] Trackback