Skip to content
GitLab
Explore
Sign in
services
review.typo3.org
gerrit
Repository
gerrit
assets
etc
robots.txt
Find file
Blame
History
Permalink
[TASK] Improve robots.txt rules
· e3973234
Stephan Großberndt
authored
Nov 23, 2020
* Deny DotBot and SemrushBot * Disallow several directories * Change Crawl-Delay to 30 seconds
e3973234