SEO - Robots and Sitemap.xml
2016/1/7·1 min read
Easiest improvement for robots is add file robots.txt and following content. It means allow any robots engine to crawl your site, and don't index your /keystone and /admin root folder. One of the reason you do that is then these engine won't check your page so first your page speed is better.
User-agent: * Disallow: /keystone Disallow: /admin More information about robots, see Create robots.txt file or robots database
sitemap.xml is another important thing to improve google crawl your website, I'm using following to auto generate sitemap. sitemap generator