What is the Robots.txt file?
The robots.txt file communicates the rules of your website for search engines. Before a search engine visits normal pages on your website, it first tries to retrieve the robots.txt file to see if there are any special instructions for crawling (retrieving) your website.
The robots.txt file is very important from a search engine optimization (SEO) point of view. It tells search engines how to crawl (fetch) your website. With the robots.txt file you can prohibit search engines from accessing certain parts of your website, prevent duplicate content problems, and tell search engines how to crawl your website more efficiently.
Installing the app:
First you need to install the app 'Robot TXT' from the App Store. You can find it via the maintenance tool of your webshop at: 'Start > App Store' This app is free of charge.
When you activate the Robot.txt app it becomes possible to customize the robot.txt of your webshop.
Please note! One error and your webshop may be incorrectly indexed, which may cause your position in the search results to deteriorate.
You can find the settings of the Robot.txt via Start > My Webshop > Settings > General > SEO.
By default there is already a piece of robot.txt filled in, this is done to guarantee the functioning of the shops.