Recommended WordPress Robots.txt
For most WordPress websites, the following robots.txt file should suffice. Simply copy and paste these lines of code into a file named robots.txt in the root of your website.
Note: Please don’t forget to change the URL in the Sitemap lines to the URL of your website
User-Agent – With an asterisk allows all bots to crawl your website
Allow – Bots are explicitly allowed to access URLs with this path
Disallow – Bots are not allowed to access URLs with this path
Sitemap – The location of a sitemap
For more information on robots.txt standards and best practices, Google’s documentation. Otherwise, if you have any question, please let me know in the comments below.