[TASK] Clean up robots.txt

Related: T3DEMO-301
1 job for davidsteeb-main-patch-97236 in 2 minutes and 31 seconds (queued for 3 seconds)