Searching the world wide web for the latest information, I stumbled across a website that explains how to correctly write a “robots.txt” file for your website. This file tells the search engine “spiders” or “searchbots” what to do or what not to do when they come across your page.
This site is pretty interesting actually. It even has a robot.txt validator that will tell you errors, warnings, and even some things that might not be either, just bad style. I think this is actually pretty handy.