< back to "What's your SEO IQ?" page

What are Robots.txt files and why are they important to search engines?

Search engine robots (sometimes called "spiders") will check in the root area of your website for a file named "robots.txt". If this file exists, it will contain information about pages or areas of your site that you would like to be off-limits to the robots -- pages that you don't want them to read, and you don't want the search engines to include in their index.

Here's a pretty good description of the robots.txt file, and some information about how to set it up.