[Answer]-Will my ALLOWED_HOSTS setting block googlebot?

1đź‘Ť

âś…

You should use robots.txt file on webserver to block bots from accessing website. http://www.robotstxt.org/robotstxt.html

Google bot isn’t using “googlebot” host, but user-agent. If bot would use “googlebot” as host, then it wouldn’t be able to access your website in case if webserver is configured with multiple websites having different hosts.

👤Agris Ameriks

Leave a comment