What Search Engines Can See & How Google Spiders Crawl the Web

by Tyler Rice

Date Created | Nov. 13, 2017

Search engine spiders crawl through trillions of pages each day and are extremely powerful software programs. These spiders crawl the content and links on web pages and place that information into highly complex databases to be called upon for a given search query made by a user.

While a user sees a dynamic and responsive web page, the automated bots see the raw HTML of a page. You can see what that looks like by inspecting the code source of any webpage. This can be done on any web browser that you use.

In this example, we can see the raw HTML of a webpage in Google Chrome by right-clicking on any element on the page and selecting the “Inspect” option. This can also be done with the Ctrl+Shift+I shortcut. Not only does this show the raw HTML on a page, but this also shows the unique CSS and styles that go into the design of a web page.

Next, we see the same option on the Mozilla Firefox web browser. By right-clicking on a page element and selecting the “inspect element” option, it opens an inspector window at the bottom of the page.

This shows the HTML of the page on the left with the styles associated with each element on the right. It is this code that the mechanical search engine spiders crawl through and see. When evaluating this content, the spiders ignore the design of the page as it has nothing to do with the content that the user is trying to find information on.

In this example, we inspect the code on the homepage of SEM Rush. When digging into the HTML, we can see the text that a user sees when visiting the web page. This is what the search engine spiders care about when evaluating the unique content of a page.

Elements that Search Engine Spiders Evaluate

The main content of the page is not the only thing that the spiders care about.

The automated bots also take special consideration of elements such as the tags and the attribute for the image. When diving into the world of on-page optimization, we will cover the ranking benefits of these elements in detail.

Now, when it comes to processing images on the web, search engines have had little understanding of what the picture looks like.

Historically, they have relied on the information presented in the image HTML tag to inform them on what the picture is.

Lately, Google has been working towards building a platform to understand these images better. If you head to the Google Search Images platform, you can upload or drag a photo into the search bar to see results associated with that image.

When using the reverse image search, you may find results of similar images, websites that include the image, and other sizes that were uploaded. The search engine may also list results that are relevant to the photo.

For example, when dragging a photo of the popular Mona Lisa painting into the search field, the search engine displays the best guess at a query relevant to the image and shows similar images below the most valuable results.

While this service is still in development, it shows that Google is working to understand photos better than just reading the alt HTML tag that is associated with the image.

Also, while Google has the technology to understand the images better, the processing power that it would take to evaluate every image on the web is not necessary now to provide the proper user experience.

Therefore, it is important to take advantage of using the alt HTML tag on any given image to provide a description for both users who cannot see the image and search engines that cannot understand the image.

Knowing how a search engine looks at a website is one of the necessities in providing excellent search engine optimization.