Robots.txt

A text file placed in the root directory of a website that tells search engine crawlers which pages or sections they can or cannot access.