SEO - Robots and Sitemap.xml

2016/1/71 min read
bookmark this
Responsive image


Easiest improvement for robots is add file robots.txt and following content. It means allow any robots engine to crawl your site, and don't index your /keystone and /admin root folder. One of the reason you do that is then these engine won't check your page so first your page speed is better.

User-agent: *
Disallow: /keystone
Disallow: /admin

More information about robots, see Create robots.txt file or robots database


sitemap.xml is another important thing to improve google crawl your website, I'm using following to auto generate sitemap. sitemap generator