Allow Search Engine Crawlers

At the moment, access is rejected for all bots. This also includes regular crawlers such as Googlebot.

• adjust robots.txt/Anubis configuration to allow regular search engine crawlers