Google spiders, their function and how they affect the web master.

GoogleBot

GoogleBot is the search spider used by Google to scour the web for new pages. 

GoogleBot has two versions - Deepbot and Freshbot.

Deepbot

Deepbot is a deep crawler that tries to folow every link on the web and download as many pages as it can for the Google index. It also examines the internal structure of a site, giving a complete picture for the index.

Freshbot

Freshbot, on the other hand, is a newer bot that crawls the web looking for fresh content. The Google freshbot was implemented to take some of the pressure off of the GoogleBot 

The freshbot recalls pages already in the index and then crawls them for new, modified, or updated pages. In this way, Google is better equipped to keep up with the ever-changing Web. This means that the more you update your web site with new, quality content, the more the Googlebot will come by to check your website out. If you'd like to see the Googlebot crawling around your website more often, you need to obtain quality inbound links. However, there is also one more step that you should take. If you haven't already done so, you should create a Google Sitemap for your site. 

Google Sitemap

Creating a Google sitemap allows you to communicate with Google, telling them about your most important pages, new pages, and updated pages. In return, Google will provide you with some valuable information as well. Google Sitemaps will tell you about pages it was unable to crawl and links it was unable to follow. This allows you to pinpoint problems and fix them so that you can gain increased exposure in the search results. 

 

More Google Spiders

MediaBot

MediaBot is used to analyze Adsense pages useragent:

Mediapartners-Google MediaBot is the Google crawler for Adsense Publishers.

Mediabot is used to determine which ads Google should display on Adsense pages. 

Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site.

To do this, simply enter the following code into your robots.txt file: User-agent: Mediapartners-Google* Disallow: This will ensure that the MediaBot is able to place relevant Adsense ads on your site. 

Keep in mind that ads can still be shown on a page if the MediaBot has not yet visited.

If that is the case, the ads chosen will be based on the overall theme of the other pages on the site. If no ads can be chosen, the dreaded public service announcements are displayed instead.

There is a strong debate over whether or not the MediaBot is giving websites with Adsense an advantage in the search engines. It has been confirmed that the Adsense Mediabot has indexed webpages for Google's main index. He states, "Pages with AdSense will not be indexed more frequently. It's literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we'd retrieve the page from the crawl cache. But there's no boost at all in rankings if you're in AdSense or Google News. You don't get any more pages crawled either.

Your website does not get any advantage by using Adsense. However, in my mind, simply getting your site updated in and of itself is an advantage.

This is very similar to Google Analytics, which also promotes a slightly higher degree of spider activity.

Those who run Google Analytics on their site can expect additional spider activity.

However, you certainly shouldn't depend on any of these tools for getting your site indexed.

The key to frequent spidering is having quality inbound links, quality content, and frequent updates. 

 

ImageBot

ImageBot is used to crawl for the Image Search useragent: GoogleBot-Image

The Imagebot prowls the Web for images to place in Google's image search. Images are ranked based upon their filename, surrounding text, alt text, and page title.

If you have a website that is primarily image based, then you would definitely want to optimize your images to receive some extra Google traffïc.

On the other hand, some web sites may not benefit from Google image search. In most cases, the traffïc from the Image search engine is very low quality and rarely converts into buyers.

Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to blockImageBot from accessing your image directory

One of the few exceptions I would make is if you have a site dedicated to downloadable images.

AdsBot

AdsBot is one of Google's newest spiders. This new crawler is being used to analyze the content of advertising landing pages, which helps determine the Quality score that Google assigns to your ads.

Google uses this Quality score in combination with the amount you are willing to bid to determine the position of your ads. Therefore, ads with a high quality score can rank higher even if other advertisers are paying more than you.

This is one of Google's many efforts to ensure that they are delivering the best results to their users. Can you still block being spidered?

Of course, but it will lower your overall Adwords quality score, which could end up lowering the positioning of your ads. If possible, it is best to give AdsBot complete access to your site.

Find out more about GoogleBot

 

Today's Google bots are becoming more advanced all the time. However, nothing beats relevant, quality, updated content. Deliver that and the search engines will eat it up. 

 

 

Why not let Spoton carry out your websiste marketing program with you!

You complete the areas you are comforatable with or have time to do and we will give you a fixed price to complete the rest. 

Give us a call on 01803 407407