SEO - Robots and Sitemap.xml
Table of Contents
Introduction
Two of the easiest SEO improvements you can make are adding a robots.txt file and a sitemap.xml to your website. These files help search engines crawl and index your site more effectively.
robots.txt
The easiest improvement for robots is to add a file called robots.txt with the following content.
It tells any search engine robot to crawl your site but not index your
/keystone and /admin root folders. One reason you do this is so search engines
won't check those pages, which helps improve your page speed.
User-agent: *
Disallow: /keystone
Disallow: /admin
More information about robots: see Create robots.txt file or robots database.
sitemap.xml
sitemap.xml is another important file to help Google crawl your website.
I'm using the following to auto-generate my sitemap: sitemap generator.
Conclusion
Adding a robots.txt and sitemap.xml to your website are simple yet effective steps to improve your SEO. The robots.txt controls which pages search engines can access, while the sitemap.xml helps search engines discover and index your content efficiently.