assets/etc/robots.txt · e39732349042bbe2d7df02d95f3861a931ffc11d · services / review.typo3.org / gerrit

Nov 23, 2020

Stephan Großberndt authored Nov 23, 2020

* Deny DotBot and SemrushBot
* Disallow several directories
* Change Crawl-Delay to 30 seconds

e3973234

Stephan Großberndt authored Nov 23, 2020

* Deny DotBot and SemrushBot
* Disallow several directories
* Change Crawl-Delay to 30 seconds