6 June 2020

Search engine optimization and site indexing

The necessary information on the Internet is easiest to find using search engines. The desired query is entered in the search bar, the user receives the result – a list of pages that most closely match the search query. To get into the search results pages of the site must first be indexed by a search robot. The search system processes and enters into the database information that can be found on the site, articles, links, and illustrations are indexed.
The main ways to index a site
Forced indexing – site data is added to a special search engine form.
Natural indexing – a search robot independently searches for a site, moving from resources that link to the site. Therefore, the external link mass of the site should not be zero.
The period of indexing a site can be different, from several days to several weeks. The indexing process is cyclical and repeated at intervals of two to four weeks, depending on the particular search robot. The entire site as well as individual internal pages can be indexed. Sometimes due to technical problems, lack of access to the pages, the site may not be indexed. In this case, the search engine will do one of the following:

immediately excludes non-indexed pages from its database;
will re-index the site in a few days;
excludes pages that have not been indexed if they are not found during repeated indexing.
Indexing management and site optimization
To manage site indexing, we use a site map and a special robots.txt file and special tags.

Partial closing of the page content from Yandex search robots is performed by the tag. There is also a special robots meta tag that can allow or prohibit indexing and following links on each individual page of a site.


A sitemap is always an up-to-date list of site pages to be indexed. A special sitemap file can be created in XML format or in plain text format. A robots.txt file can be used to indicate the location of the Sitemap. To do this, add the following line to it: Sitemap: http: // site_name/sitemap.xml

Using a site map allows you to tell the search engine:

about the pages of the site that should be indexed;
how often the information on the pages of the site is updated;
how important the indexing of each page is.
Speed ​​up site indexing
There are many ways to speed up site indexing by search robots. However, to begin with, we consider the norms, the observance of which will help solve the problems of indexing the site.

Do not create too large pages (the size should not exceed 100-200 kbytes).
Links to internal pages explain the content of the page using simple headings.
It uses human-readable URLs (NC), which facilitates the indexing of the site. URLs exclude parameters and session identifiers.