[TASK] Improve robots.txt rules
* Deny DotBot and SemrushBot * Disallow several directories * Change Crawl-Delay to 30 seconds
Please register or sign in to comment
* Deny DotBot and SemrushBot * Disallow several directories * Change Crawl-Delay to 30 seconds