How to use robots.txt

In this article I will discuss how to make document robots.txt for search engines and how to quickly get search engines to index.

File robots.txt - is a text file in the root directory of the site where special instructions are written for search engines. These regulations may prohibit indexing of certain sections or pages on this site, point to the correct "mirror" domain, to recommend to a spider to observe a certain time interval between downloading documents from the server, etc.

Creating a robots.txt for uCoz

Consider the basic tags of the document:
Disallow: it prohibits the distribution or indexing of pages directory.
Allow: allowing a distribution index page or directory.
Thus, we can prevent the indexing of any directory, but allow the indexing of some pages in the same directory.

Take for example the module File Catalog. Catalog module is located at /load/

If you put:

Then the File Catalog, we will not be indexed, but if we need 2 - 3 pages of this catalog of all (for example only 10). Then write in your robots.txt file the following line:
Warning: the string with a resolution index page should be higher than the line with the prohibition of the directory index.
Example:
So we will be indexed only those pages that are listed under the tag Allow.

Now let's deal with the tag User-agent. This tag reveals information about the bot which will be available to all of the settings that below. User-agent tag is placed at the top and after the name of a bot to check.
For example:

But if you want to specify for each bot search engine or directory sites, etc. the same distribution, then put next tag:

And list the following restrictions and open the different pages and directories.

So the main work is done and we can use this document for all bots, but if you want a better indexing of your site and better display its search engines, better yet provide links to files sitemap:

This is a standard for addressing the site map of uCoz, if you have a site map, write your own.

Then, too, is not unimportant to specify the primary domain site. If you have a standard domain by uCoz you do anything inappropriately.
If you are attached to the site domain, for example. Com, you can set in your robots.txt file has this line:
Host: website_domain
Example:

Example robots.txt ready to uCoz: