1đź‘Ť
âś…
You should use robots.txt file on webserver to block bots from accessing website. http://www.robotstxt.org/robotstxt.html
Google bot isn’t using “googlebot” host, but user-agent. If bot would use “googlebot” as host, then it wouldn’t be able to access your website in case if webserver is configured with multiple websites having different hosts.
👤Agris Ameriks
Source:stackexchange.com