<h3>404 is normal</h3>

 Crawl Errors sometime show 404 because, you page is not exist anymore, or
 other website point to your page but your page is not exist, or you change the
 domain and change the folder of your url.&nbsp;


 You could look the Crawl Errors list and redirect them all as 404, but depend
 on the page I think it might could drive the traffic to the site, so I would
 redirect it to the parents page.


 For example, my blog usually start from /blog/Post/{then the page}, however I
 found out there're lots of 404 as /Home/Blog/{page url} or
 /mobile/blog/post/{page url}. So I direct them as to the parents page.
 Following is using node.js.

<pre
 class="language-javascript"
> app.get('/home/blogs/*', routes.views.blog);
 app.get('/mobile/blog/post/:post', routes.views.blog); 
</pre>
<h3>Reference</h3>
<ul>
 <li>
 <a
 href="https://support.google.com/webmasters/answer/2409439?ctx=MCE&amp;ctx=NF"
 target="_blank"
 >Not found error</a
 >
 </li>
</ul>

Fix Google Crawl Errors 404

When you see the error message "Crawl postponed because robots.txt was
 inaccessible" at your web master page. It is telling you that google can't
 crawl your site, something wrong with your robots.txt. That means, your site's
 SEO is dead, unless somebody know your site go to your site directly,
 otherwise google will not showing your site to search.


 Sometime the problem might be your site service provider block google crawl to
 access your site. I was encounter different issue.&nbsp;


 So, I had a website which had around 1000 user visit per day. It's start up
 website, one day I changed something the site become zero visit. When I look
 at the webmaster's robots.txt section, it's warning as "Crawl postponed
 because robots.txt was inaccessible". But I noticed my robots.txt is there and
 I can hit it by type the url like /robots.txt with 200 response.


 I thought it might be I need to change the text like following, allow every
 url at my site to crawl. But still I'm keep seen this warning message. So I
 removed it and still same warning.

<pre class="language-javascript"><code>User-agent: *
Allow: /</code>
</pre>

 Then I noticed that, google will crawl everything when access to url
 /robots.txt return 200 or get 400 error. My case, the application return 200
 eventhough robots.txt is not there.


 After I changed logic, need wait next time when google crawled. If you want to
 change how often google will crawled your site, you could change the crawl
 rate at your google web master. However base on following article, 2-4 days
 google will first visit your pages.


 <a
 href="http://moz.com/blog/googles-faster-crawl-rate-no-thanks"
 target="_blank"
 >googles-faster-crawl-rate</a
 >

This is a blog tell you how to change crawl rate.

 <a
 href="http://support.hostgator.com/articles/specialized-help/telling-google-how-often-to-crawl-your-website"
 target="_blank"
 >change crawl rate</a
 >

When you see the error message "Crawl postponed because robots.txt was inaccessible" at your web master page. It is telling youthat google can't crawl your site, something wrong with your robots.txt. That means, your site's SEO is dead, unless somebody knowyour site go to your site directly, otherwise google will not showing your site to search.

Crawl Postponed Because Robots.txt was Inaccessible

<ol>
 <li>
 <a href="http://seositecheckup.com/" target="_blank">seositecheckup.com</a>
 
 checked, including page statistics, site data, meta tags, page notes,
 objects, site security, and many more. The tool also helps you check your
 
 </li>
 <li>
 
 <a href="http://www.screamingfrog.co.uk" target="_blank">screamingfrog</a>
 
 </li>
 <li><a href="https://www.addthis.com" target="_blank">addthis.com</a></li>
 <li>
 <a href="http://tools.pingdom.com/" target="_blank">pingdom</a>
 
 Pingdom Tools can test the load time of a web page, DNS servers and
 settings for a domain name, and network connectivity to a server. Alerts
 can be enabled and sent via SMS and email.
 
 </li>
 <li>
 <a href="https://www.google.com/webmasters/" target="_blank">webmasters</a>
 
 quickly check crawl errors such as 404 pages and redirection problems,
 duplicate content issues, backlinks, and site indexation status including
 sitemap and crawl frequency
 
 </li>
 <li>
 <a href="http://www.seoworkers.com/tools/" target="_blank">seoworkers</a>
 
 SEO Workers is a free and very powerful SEO analysis tool that helps you
 analyze and measure the ranking potential of your web pages.
 
 </li>
 <li>
 <a href="http://gsitecrawler.com/en/download/" target="_blank"
 >gsitecrawler</a
 >
 Help you to crawler your site.
 </li>
 <li>
 <a
 href="http://www.yellowpipe.com/yis/tools/lynx/lynx_viewer.php"
 target="_blank"
 >yellowpipe</a
 >
 
 Yellowpipe Lynx Viewer Tool is an online SEO tool that allows you to see
 how search engine robots see your website. Indeed, you can see your web
 pages in Lynx, a text-mode web browser.
 
 </li>
 <li>
 <a href="http://moz.com/researchtools/ose" target="_blank">researchtools</a>
 
 Open Site Explorer is one of the most popular free SEO tools by SEOMoz. It
 offers a strong analysis of onpage signals such as HTML tags, outbound
 links and their respective authority, and many more.
 
 </li>
 <li>
 <a href="http://home.snafu.de/tilman/xenulink.html" target="_blank"
 >tilman</a
 >
 
 It detects problems on your website. For instance, Xenu checks your links
 (to see whether there are broken links), redirections issues, duplicate
 content (title, meta description, etc.), and loading time.
 
 </li>
 <li>
 <a href="http://www.linkdiagnosis.com/" target="_blank">linkdiagnosis</a>
 
 Link Diagnosis is a very popular free SEO tool that checks your backlinks
 and their anchors. The Firefox Extension can be downloaded and enables you
 to benefit from many more additional features.
 
 </li>
 <li>
 <a href="http://www.majesticseo.com/" target="_blank">majesticseo</a>
 
 Majestic SEO is a very useful free SEO tool, notably for backlinks
 analysis. Their majestic site explorer gives you a summary of all
 referring domains and external backlinks.
 
 </li>
 <li>
 <a href="http://www.seobook.com/archives/001792.shtml" target="_blank"
 >SEO external links build</a
 >
 tell you how to build external links.
 </li>
 <li>
 <a
 href="http://code.tutsplus.com/articles/50-ways-to-get-your-site-noticed--net-15620"
 target="_blank"
 >50 ways to get your site noticed</a
 >
 </li>
 <li>
 <a href="http://www.feedthebot.com/" target="_blank">feedthebot</a>
 a tool for test SEO
 </li>
 <li>
 <a href="http://moz.com/blog/meta-data-templates-123" target="_blank"
 >meta tag for SEO</a
 >
 </li>
</ol>

few very useful seo tools and tools could improve your site's SEO.

Few Very Useful Links to Help Improve the SEO

<h3>robots.txt</h3>

 Easiest improvement for robots is add file robots.txt and following content.
 It means allow any robots engine to crawl your site, and don't index your
 /keystone and /admin root folder. One of the reason you do that is then these
 engine won't check your page so first your page speed is better.

<pre class="language-javascript"><code>User-agent: *
Disallow: /keystone
Disallow: /admin

More information about robots, see Create robots.txt file or robots database
</code></pre>
<h3>sitemap.xml</h3>

 sitemap.xml is another important thing to improve google crawl your website,
 I'm using following to auto generate sitemap.&nbsp;<a
 href="https://www.xml-sitemaps.com/"
 target="_blank"
 rel="noopener"
 >sitemap generator</a
 >

Easiest improvement for robots is add file robots.txt and following content. It means allow any robots engine to crawl your site,and don't index your /keystone and /admin root folder. One of the reason you do that is then these engine won't check your page sofirst your page speed is better.

Category SEO

Fix Google Crawl Errors 404

Crawl Postponed Because Robots.txt was Inaccessible

Few Very Useful Links to Help Improve the SEO

SEO - Robots and Sitemap.xml