Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
services
review.typo3.org
gerrit
Commits
7f9d2546
Commit
7f9d2546
authored
Nov 23, 2020
by
Stephan Großberndt
Browse files
[TASK] Deny AhrefsBot
parent
e3973234
Changes
1
Hide whitespace changes
Inline
Side-by-side
assets/etc/robots.txt
View file @
7f9d2546
...
...
@@ -81,6 +81,9 @@ Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
# All allowed bots are ask to skip /changes as these return JSON code content only, impact performance and are of no value to search engines
User-agent: *
Disallow: /changes/
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment