In this article I will discuss how to make document robots.txt for search engines and how to quickly get search engines to index.

File robots.txt - is a text file in the root directory of the site where special instructions are written for search engines. These regulations may prohibit indexing of certain sections or pages on this site, point to the correct "mirror" domain, to recommend to a spider to observe a certain time interval between downloading documents from the server, etc.