Tips on creating a robots.txt file for Shop-Script

Below are provided the basic directives for the robots.txt for Shop-Script users. These directives can be used for excluding non-relevant storefront pages from the list of URLs indexed by search engines.

Disallow: /search/?query= Search results page.
Disallow: /compare/ Product comparison page.
Disallow: /tag/ Product filtering by tags.
Disallow: *&sort=<br> Disallow: */?sort= Product sorting results page.
Disallow: /cart/ Shoping cart page.
Disallow: /order/ In-cart checkout page.
Disallow: /checkout/ Multi-step checkout pages.
Disallow: /my/ Customer accounts.
Disallow: /login/ Customer login page.
Disallow: /signup/ Customer signup page.
Disallow: /forgotpassword/ Password recovery page.
Disallow: /webasyst/

Webasyst backend pages.

The above directives are suitable for online stores available at domain root, i.e. at URLs like https://yourdomain.com/. If your online store URL contains a subdirectory name; e.g., https://yourdomain.com/shop/, then you need to add that subdirectory name instead of the first slash character. For example, Disallow: /tag/ directive should be changed to Disallow: /shop/tag/, etc.

Crawl-delay directive

The load caused by search crawlers can also be reduced using Crawl-delay directive. It sets the minimum period in seconds after which a crawler is allowed to analyze another website page. An example of use:

Crawl-delay: 10

0 comments

    Add comment

    To add a comment please sign up or login