Skip to content

GitLab

Commit 7f9d2546 authored by Stephan Großberndt's avatar Stephan Großberndt
Browse files

[TASK] Deny AhrefsBot

parent e3973234
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 0 deletions
assets/etc/robots.txt
View file @ 7f9d2546
......@@ -81,6 +81,9 @@ Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
# All allowed bots are ask to skip /changes as these return JSON code content only, impact performance and are of no value to search engines
User-agent: *
Disallow: /changes/
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment