How to Get Your Website Crawled by Google Fast and Frequently

Dec 21, 2012   //   by Emma-Julie Fox   //   internet marketing, SEO Blog  //  20 Comments

SEO is all about increasing website visibility by making it rank high in search engines. Companies will always strive to make their websites appear at the first page of the SERPs for searches on related keywords. In order for that to happen, their sites have to be crawled by Google bots in the first place.

It is important to note that in order to rank, all the other pages of your website should be equally competitive in the search engines, not just your homepage. Besides, internal pages are the ones that usually make it on the SERPs (except for exact keyword searches that are directly related to a website; i.e. name of the website’s company).

If you want your website to be indexed, you have to catch the attention of the bots first.

Update Your Website Frequently

Simple updates of the content on you site can already alert the bots to pay attention to your site. Long dormant websites are usually left alone because there’s nothing new to find there anyway. Simply breaking the stagnancy signals new activity and alerts the bots that there are possibly new pages to crawl.

Updating your website alone will not guarantee that your pages will be crawled quickly, nor will it guarantee that the bots will be constantly coming back to crawl new pages in your site, but it’s a good start. To keep that ball rolling, proceed with a number of SEO activities at the same time. You can get your pages crawled faster and more frequently afterwards if you follow through the rest of this list.

Generate Content

Posting new content regularly is one of the simplest ways to make sure that search engines take notice and crawl your web pages. This is why sites like Ezine Articles, Digg, Reddit and other bookmarking and article publishing sites commonly end up on the first search engine results pages. The crawling rate in those pages is both fast and frequent because they are always monitored by Google bots for new content to index.

Not all websites are capable of posting new content regularly though, as their very nature does not permit it. A company website for instance doesn’t need constant revision of their content. After all, the mission and vision of a company will not change regularly, and the company history will still be the same. Writers will be bored out of their minds if they have to keep on revising that content.

So in order for their crawl rate to pick up, these websites can create a blog page or news roll for their website. These pages will require frequent and regular posting of new content.

Check Meta Robots Tags, Page Load Speed, 404s, Crawl Speed.

Sometimes the bigger problem is not exactly your infrequent updates and content postings, but that for some reason the bots cannot access your web pages. There are so many new, attractive features that people want to incorporate in their own websites. JavaScript, for instance, are commonly ignored by the Google bots. Websites created from Flash technology and content contained in pop-up and independent Frames also cannot be crawled by Google. Keeping things attractive yet simple is one way to be sure the pages in your website can be indexed.

Don’t forget to use robots.txt to guide the bots which pages to crawl and not to crawl. Create a sitemap and make sure the headers and meta titles of your pages are done correctly. You can even initiate a crawl yourself by submitting your URL to Google.

Also, check if the loading speed of your pages is too long. Spiders have very limited attention spans and would ignore pages that take too long to load. 404 errors are also roadblocks for getting your pages crawled by Google bots.

Gather Valuable Links

Getting inbound links into your web pages is another way to get crawlers into your site—provided that the other website you are linking with is frequently crawled by Google as well. The links become virtual bridges that bots can travel through.  If you’re just linking with another fledgling site that’s not doing so well with the crawlers, then none of you is really helping the other.

This is actually an excellent reason and motivation for SEOs to work on getting high quality links from bigger, more successful websites. Link juice isn’t the only thing they can give you, but also the chance to get your pages crawled by Google.

Similarly, if you’re going to do blog commenting and put a link back to your website, make sure that the sites you’re leaving comments on are not spammed with junk comments.

Use Sharing Buttons

Advantages Of Social Sharing

Share Your Content For Better SEO Results!

 

Making your website go social is an excellent way to invite traffic and, more importantly, gather valuable links. This doesn’t always mean having accounts in all major social media websites. Sometimes merely adding sharing buttons in your page is enough to get your content and links circulating in social media networks.

You can go to AddThis.com and ShareThis.com to get sharing buttons for free. If you are using a blogging platform like Blogger and WordPress, they already provide the widgets for social media sharing. If you go to AddThis and ShareThis though, you can choose the style of buttons to put in your website.

Use Bookmarking Sites

Submitting links of your pages to bookmarking sites will not just help bring traffic to your site, it will also help the bots find, crawl, and index them. Your links will come out as new content for these bookmarking sites. Since they are constantly generating content (which means Google bots are constantly keeping their eyes on them), your page will almost immediately be crawled. The effect on the rest of your website is similar to having valuable links on your web pages.

Bookmarking sites also increases your social activity, which is another way to garner links from other—hopefully valuable and high-ranking—sites. Here are some of the top bookmarking sites you can use today:

  • Reddit
  • Digg
  • Pinterest
  • Stumbleupon
  • Multiply
  • Tumblr
  • Buzzfeed
  • Tweetmeme
  • Delicious
  • Fark

Importance of Speed and Frequency 

Take note that we’re targeting not just a one-time crawl, but frequent visits from the Google bots. These steps will help make your website become one of the search engine’s priorities and be in its radar for constant monitoring. You want Google to index your pages as soon as they go live. These points are important because all your SEO efforts will go unnoticed if your pages are not indexed in the first place.

Similar Posts:

    None Found

Emma-Julie Fox

Emma-Julie Fox writes for Pitstop Media, a Vancouver based SEO company. Pitstop Media has been helping businesses across North America successfully increase their search visibility. If you want to invite the author to guest post on your blog please contact www.pitstopmedia.com

More Posts

"Newsletter" Our weekly newsletter features some of the best curated SEO content from around the web!



Comment Policy

  • Ray says:

    Some of my sites get indexed really fast, while others can take several days. I try to get some backlinks to various pages, both the homepage and internal pages. Nothing excessive though. I used to spend a lot of time on social bookmarking sites. Not so much lately. It’s all about time. I just don’t have as much free time on my hands as I used to.

  • Naser says:

    Thanks for the tips Emma. Can you please help me recover my blog from Google penguin update. I follow all the white hat methods but still I don’t know why Google downgraded my rankings. Thanks in advance.

  • Emma says:

    @Ray – site authority (PageRank) is one metric that dictates how often and deep a website is getting indexed. If you have a high PR site it will be indexed quite fast. If you use WordPress to build your sites you can also add more pinging services to get bots to come to your site. Additionally, you can use Google Translate on the newly created pages. Since the translation happens in real-time, Googlebot will come and hopefully index that page.

    @Matt – indeed, you can peep at your competitor’s sites, but I would not necessarily copy their options. One very nice plugin is DiggDigg social buttons.

    @Naser – recovering from Penguin is difficult and extensive and is not something one could achieve in a week. It may take hundreds of hours to asses and clean up your backlinks profile. If you have the time I would suggest you to do it yourself, otherwise it can be really expensive, with no guarantees. On the top of that, your site will not get back on where it was, because you will remove the backlinks that placed the site at the top. So, you will have to start a white hat link building campaign.

  • Praveen Soni says:

    Hello Emma,
    Hope you are great in health
    I agree with some of your points i.e. For fast crawling we have to update the content daily, but if your website is just like company so you will need a news page on which content will be updated regularly.
    Thanks for sharing such a great tips For crawling :)

    • Emma says:

      Praveen, if you are a company, then you have to have a blog to stay in touch with your audience, existing clients and to prove the leadership. If you don’t have one, then you just make your competitor’s lives easier :)

  • I am having a problem where my sitemap gets too much time to get indexed. I am having more not found pages, so how can I reduce these things to get into top searches.

    • Emma says:

      Sitemaps are used only as guidelines only. Google will not necessarily index all pages of the sitemap, but will rather used the file to discover URLs. If a URLs is worthy (content + backlinks), then it will get indexed. Also, it won’t matter if you send Googlebot to a page, using Google Webmaster Tools or Google Translate, if that page is not worthy of being indexed. Build several links from other site to those difficult pages and you should be fine.

  • Sarah says:

    I make small changes on my home pages at least few times a week and publish regular new content. That way my content is almost immediately in Google results (1-2 days after publishing).

    • Emma says:

      Hi! Sarah. Tweaking the home page is not necessary, uploading new content in the blog section is good enough. As I said earlier, if you use WordPress you can also add more pinging services to get bots to come to your site. Google Translate on the newly created pages also helps.

  • Matt says:

    Great post Emma! I’m definitely going to have to incorporate some of these principles so I can get my sites crawled a bit more frequently.

  • abhishek says:

    I think the best way is to update the website regularly and endure that website is not blocked by robots…rest google takes care of the crawling part on its own…….

  • Jamy Hoster says:

    Most important part is update on regular bases and also we have to consider the loading time of a website as there are lots of people who love those sites which have good design and information and the loading time is lower.

  • Emma says:

    @ Abhishek, good points Abhishek, but in some cases doing a little extra helps.

    @Jamy, you have a point, loading time is important.

    Thanks for your comments.

  • Woah this weblog is fantastic i love studying your articles. Stay up the good paintings! You realize, many persons are searching around for this info, you can aid them greatly.

  • Ali Fatemi says:

    This site is a very helpful, I think that. Website and search engine-optimization for better tips available here. So, i want to know that, how to Get my Website Crawled by Google Fast and Frequently.

  • Ryan Key says:

    Very good advice, I think for someone just getting started in SEO should know the best ways to get indexed really fast. I have never had really much luck with social bookmarking sites, I have use Onlywire before and really just never received the amount of traffic I was hoping for. Maybe because they have all converted to more of social networks and have taken away the do follow links?

  • akonlki231 says:

    Thanks for sharing superb informations. Your web-site is very cool. I am impressed by the details that you have on this blog. It reveals how nicely you perceive this subject. Bookmarked this website page, will come back for extra articles. You, my pal, ROCK! I found simply the information I already searched all over the place and simply could not come across. What a perfect web site.

  • akonlki231 says:

    I am a big fan of your blog.i am so excited by read of your blog’s content.really great post.Thanks for sharing superb informations. Your web-site is very cool. I am impressed by the details that you have on this blog.

    Evening gown dresses

  • dm.models says:

    This is very interesting, cool