Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
services
review.typo3.org
gerrit
Repository
e39732349042bbe2d7df02d95f3861a931ffc11d
Switch branch/tag
gerrit
assets
etc
robots.txt
Find file
Blame
History
Permalink
[TASK] Improve robots.txt rules
· e3973234
Stephan Großberndt
authored
Nov 23, 2020
* Deny DotBot and SemrushBot * Disallow several directories * Change Crawl-Delay to 30 seconds
e3973234
Stephan Großberndt authored