Below are provided the basic directives for the robots.txt for Shop-Script users. These directives can be used for excluding non-relevant storefront pages from the list of URLs indexed by search engines.
|Disallow: /search/?query=||Search results page.|
|Disallow: /compare/||Product comparison page.|
|Disallow: /tag/||Product filtering by tags.|
|Disallow: *&sort=<br> Disallow: */?sort=||Product sorting results page.|
|Disallow: /cart/||Shoping cart page.|
|Disallow: /order/||In-cart checkout page.|
|Disallow: /checkout/||Multi-step checkout pages.|
|Disallow: /my/||Customer accounts.|
|Disallow: /login/||Customer login page.|
|Disallow: /signup/||Customer signup page.|
|Disallow: /forgotpassword/||Password recovery page.|
Webasyst backend pages.
The above directives are suitable for online stores available at domain root, i.e. at URLs like https://yourdomain.com/. If your online store URL contains a subdirectory name; e.g., https://yourdomain.com/shop/, then you need to add that subdirectory name instead of the first slash character. For example, Disallow: /tag/ directive should be changed to Disallow: /shop/tag/, etc.
The load caused by search crawlers can also be reduced using Crawl-delay directive. It sets the minimum period in seconds after which a crawler is allowed to analyze another website page. An example of use: