
## Table of Contents

1. [Introduction](#introduction)
2. [robots.txt](#robotstxt)
3. [sitemap.xml](#sitemapxml)
4. [Conclusion](#conclusion)

## Introduction

Two of the easiest SEO improvements you can make are adding a `robots.txt` file and a `sitemap.xml` to your website. These files help search engines crawl and index your site more effectively.

## robots.txt

The easiest improvement for robots is to add a file called `robots.txt` with the following content.
It tells any search engine robot to crawl your site but not index your
`/keystone` and `/admin` root folders. One reason you do this is so search engines
won't check those pages, which helps improve your page speed.

```
User-agent: *
Disallow: /keystone
Disallow: /admin
```

More information about robots: see Create robots.txt file or robots database.

## sitemap.xml

`sitemap.xml` is another important file to help Google crawl your website.
I'm using the following to auto-generate my sitemap: sitemap generator.

## Conclusion

Adding a `robots.txt` and `sitemap.xml` to your website are simple yet effective steps to improve your SEO. The `robots.txt` controls which pages search engines can access, while the `sitemap.xml` helps search engines discover and index your content efficiently.


How to improve your SEO by adding a robots.txt file and sitemap.xml to your website.

SEO - Robots and Sitemap.xml


## Table of Contents

1. [404 Is Normal](#404-is-normal)
2. [Reference](#reference)

## Introduction

Crawl errors are common when managing a website. This post explains why 404 errors appear in Google Search Console and how to fix them by redirecting broken URLs.

## 404 Is Normal

Crawl errors sometimes show 404 because your page no longer exists, another website points to your page but it doesn't exist, or you changed the domain and folder structure of your URL.

You could look at the crawl errors list and let them all return 404, but depending on the page, I think it might drive traffic to the site, so I would redirect it to the parent page.

For example, my blog usually starts from `/blog/Post/{then the page}`. However, I found out there are lots of 404 errors for `/Home/Blog/{page url}` or `/mobile/blog/post/{page url}`. So I redirect them to the parent page. The following is using Node.js.

```
app.get('/home/blogs/*', routes.views.blog);
    app.get('/mobile/blog/post/:post', routes.views.blog);
```

## Reference

- Not found error

## Conclusion

404 crawl errors are normal and can happen for various reasons. Instead of ignoring them, you can redirect old or broken URLs to the appropriate parent page to retain traffic and improve your site's SEO health.


How to fix Google crawl 404 errors by redirecting old or broken URLs to the correct pages.

Fix Google Crawl Errors 404


## Table of Contents

- [Introduction](#introduction)
- [The Problem](#the-problem)
- [Attempted Fixes](#attempted-fixes)
- [Root Cause and Solution](#root-cause-and-solution)
- [Conclusion](#conclusion)

## Introduction

When you see the error message "Crawl postponed because robots.txt was inaccessible" on your webmaster page, it is telling you that Google can't crawl your site because something is wrong with your robots.txt. That means your site's SEO is dead — unless somebody knows your site and goes to it directly, Google will not show your site in search results.

## The Problem

Sometimes the problem might be that your site's service provider is blocking Google's crawler from accessing your site. I encountered a different issue.

I had a website which had around 1,000 visitors per day. It was a startup website. One day I changed something and the site dropped to zero visits. When I looked at the webmaster's robots.txt section, it was showing the warning "Crawl postponed because robots.txt was inaccessible." But I noticed my robots.txt was there and I could access it by typing the URL like `/robots.txt` with a 200 response.

## Attempted Fixes

I thought I might need to change the text like the following, to allow every URL on my site to be crawled. But I kept seeing the same warning message. So I removed it and the warning still persisted.

```
User-agent: *
Allow: /
```

## Root Cause and Solution

Then I noticed that Google will crawl everything when accessing the URL `/robots.txt` returns a 200 status or gets a 400 error. In my case, the application was returning 200 even though robots.txt was not actually there.

After I changed the logic, I needed to wait for the next time Google crawled. If you want to change how often Google crawls your site, you can adjust the crawl rate in your Google Webmaster settings. However, based on the following article, Google will first visit your pages within 2–4 days.

googles-faster-crawl-rate

This is a blog that tells you how to change the crawl rate:

change crawl rate

## Conclusion

If you encounter the "Crawl postponed because robots.txt was inaccessible" error, make sure your `/robots.txt` endpoint is returning the correct HTTP status code. Ensure it returns a proper 200 response with the actual robots.txt content, or a 404 if the file doesn't exist. Also consider adjusting your Google Webmaster crawl rate settings to speed up re-indexing.


When you see the error message "Crawl postponed because robots.txt was inaccessible" in your webmaster page, it means Google can't crawl your site due to a problem with your robots.txt. This blog explains how to troubleshoot and fix the issue.

Crawl Postponed Because Robots.txt was Inaccessible


## Table of Contents

1. [Introduction](#introduction)
2. [SEO Tools and Resources](#seo-tools-and-resources)
3. [Conclusion](#conclusion)

## Introduction

Search Engine Optimization (SEO) is essential for improving your website's visibility. This post provides a curated list of useful SEO tools and resources.

## SEO Tools and Resources

1. [seositecheckup.com](http://seositecheckup.com/) — Checks page statistics, site data, meta tags, page notes, objects, site security, and many more.

2. [Screaming Frog](http://www.screamingfrog.co.uk) — A website crawler for SEO audits.

3. [addthis.com](https://www.addthis.com) — Social sharing tools for your website.

4. [Pingdom](http://tools.pingdom.com/) — Pingdom Tools can test the load time of a web page, DNS servers and settings for a domain name, and network connectivity to a server. Alerts can be enabled and sent via SMS and email.

5. [Google Webmasters](https://www.google.com/webmasters/) — Quickly check crawl errors such as 404 pages and redirection problems, duplicate content issues, backlinks, and site indexation status including sitemap and crawl frequency.

6. [SEO Workers](http://www.seoworkers.com/tools/) — SEO Workers is a free and very powerful SEO analysis tool that helps you analyze and measure the ranking potential of your web pages.

7. [GSiteCrawler](http://gsitecrawler.com/en/download/) — Helps you crawl your site.

8. [Xenu Link Sleuth](http://home.snafu.de/tilman/xenulink.html) — It detects problems on your website. For instance, Xenu checks your links to see whether there are broken links, redirection issues, duplicate content (title, meta description, etc.), and loading time.

9. [Majestic SEO](http://www.majesticseo.com/) — Majestic SEO is a very useful free SEO tool, notably for backlinks analysis. Their site explorer gives you a summary of all referring domains and external backlinks.

10. [SEO Book - External Link Building](http://www.seobook.com/archives/001792.shtml) — Tells you how to build external links, along with 50 ways to get your site noticed.

11. Meta tags for SEO — Ensure you have proper meta tags on your pages for better search engine visibility.

## Conclusion

Using the right SEO tools can significantly improve your website's search engine ranking. Regularly audit your site with these tools to identify and fix issues that may affect your visibility.


A collection of useful SEO tools and resources that can help improve your site's search engine optimization.

SEO

SEO - Robots and Sitemap.xml

Fix Google Crawl Errors 404

Crawl Postponed Because Robots.txt was Inaccessible

Few Very Useful Links to Help Improve the SEO