Control search engine indexing with robots.txt
Use a
robots.txt file located in the root directory of your website to provide instructions to search engine crawlers (robots) about which parts of the site should or should not be indexed.