HOW SEARCH ENGINES WORK

Vaughn

In my last post, I discussed Search Engine Optimization (aka SEO) and why it’s important to us as writers. In this blog, I’d like to dig a bit deeper. In order to fully understand SEO, we need to understand the thing we’re trying to optimize, the Search Engine (aka Google, Bing, Duck, Duck Go, etc.).

Crawlers. Search engines utilize crawlers. You may have heard about them as bots or spiders (going forward, I will call them spiders, which is the term I’m most familiar with). The role of the spider is to crawl pages of sites and serve up the sites that are relevant to the results pages of people who are looking for that information. They do this in seconds and crawl millions of pages at a time. They also follow links on web pages to discover other pages that are relevant and related to a particular topic.

The Search Engine Index. When the spiders discover a page, they compile and add them to an ever-growing index. The index includes all the discovered URLs (Universal Resource Locator) along with several key signals about the contents of each URL, such as:

Keywords. What essential words describe what the spider discovered within the content of the page? What topic(s) does the page cover? Content. The type of content that is being crawled (using microdata called Schema). What is included on the page? Freshness.  How recently was the page updated? User engagement.  How do people interact with the page?

What exactly is the aim of a search engine algorithm? The job of the search engine algorithm is to present high-quality and relevant pages/search results to a user’s query/question as fast as possible. The user then selects an option from the search results. Subsequent activity then feeds this information into future learnings which can affect search engine rankings going forward.

What happens when a search is performed? Whenever someone makes a search query, all relevant pages are identified in the Search Engine Index and the algorithm creates a hierarchically ranking, the Search Engine Relevant Page (SERP) Listing, of the relevant pages and serves it up to the searcher.

Note. All algorithms are not the same. They differ from one search engine to another. For example, a page that is ranked #1 by Google, may not receive the same ranking by Bing.

Search engines also use other relevant data to return results, including.

 Location. Location dependent ‘cafes near me’ or ‘movie times near me’.  Language dependent. Return results in the user’s language. Previous Search History. Will return different results for a query on something they have previously searched. Device.  You may get a different set of results based upon the device from which the query was made. PC, versus iPad, or cell phone.

Why might a page not be indexed? Thus far I have discussed why a search engine might chose to not index your site. There are several reasons for this, they may include but are not limited to:

Robots.txt file exclusions. A file that tells search engines not to visit your site. Directives on the webpage. telling search engines not to index the page (no index tag) or to search another similar page (canonical tag). Search engine algorithms judging the page to be of low quality, have thin content or contain duplicate content.  I find this to be a key point. It’s no secret that many of us face financial restrictions and try to cut corners when it comes to the marketing part of the business. However, this is an area where it is best to use a professional webpage designer. The main reason that I have posted these SEO posts is to provide a resource for writers to ask questions of a designer to get a feel for their knowledge of web design and search engines.404 Not found HTTP response code. I don’t know of anyone who has not experienced this one.

In my next, and final, post on this topic, I will talk about how Google {if you read my earlier post on SEO, you may recall that Google has an almost 90% share of search engine market (in 2024 it was 87.8%) and its closest competitor was Bing at a bit less that 7% (6.7% in 2024)} ranks pages.. Therefore, it only makes sense that we shoot for what Google wants.

Key Conclusions. Understanding SEO gives us a leg up on getting our website on Page 1 of the SERP. Always keep in mind that the vast majority of searchers don’t go past the first page. Does that ring a bell? One of the first things I was taught when I embarked on a writing career (avocation?) was: The most important chapter is chapter one. The most important page is page 1. The most important paragraph is the first paragraph of Chapter 1. The most important sentence is … by now I guess you know where this is going.

Another point to remember. It takes a while for a webpage to grow an audience, so it’s important to give a searcher a reason to stay on your page so pay close attention to content. If you currently have a page, take a close look at it and ask yourself: If I was a first-time visitor, what will make me stay on this page? What will make me return?

The answer is obvious. Quality content and continual addition of new information. I have recently put a new page online. One of the things I spoke with my developer, Melissa Current-Gerety of MCG Creative in Orono, ME, about was that I wanted the ability to personally update my site. Your site designer most likely has more clients than only you and for minor changes, such as events and appearances, the ability to update is important.

Finally. How many members of Maine Crime Writers have websites? How many of your sites have links to other members pages? Here’s a challenge for all of us. Go to three or four member sites. On their contact page send them an email listing your site address and offer them a link on you site if they put one to yours on their page. I’ll start off. my site: https://www.vaughnhardacker.com. BTW: do you have a link to this blog on your site? I just realized I don’t but will contact Melissa in the morning to add a link page to my site.