A couple months ago, David Antrobus had a post titled, My God, It’s Full of Stars. His point in this post was that an author (or reader for that matter) shouldn’t get too hung up on the numerical rating of a review, instead focusing on what the review says. He was right for many reasons, not least of which is that one person’s garbage is another’s treasure. At least that’s the only explanation I’ve found for the yard sale signs that spring up every weekend.

David’s post generated a lot of good discussion. One subject that came up among those commenting without reaching a consensus had to do with the expected distribution of reviews among the different ratings. In this post, I’m going to give my theory, which ultimately leads to the same conclusion as David’s post, that no one, authors or potential readers, should give much weight to the number of stars, instead concentrating on review content.

Most people, when presented with a rating system, picture a distribution that resembles a bell curve. That a typical five star reviewing scheme cleanly maps to the A, B, C, D, or F grades many of us received in our school days, only reinforces that thought. Other people (the easy graders?) picture a flatter distribution, with the most extreme allocating a fifth, twenty percent, to each possible rating or grade. As a starting point, either of these distributions or anywhere in between seems logical.

What picturing these distributions fails to take into account is that they apply to the entire universe of items being graded, the millions of books available. If our purchases were random, our reviews and the reviews of others would logically look like one of these distributions, but our purchases aren’t random. Instead, as readers, we use all kinds of techniques to exclude those books we are less likely to enjoy and pick those more likely to be our thing. We read the book description and, if it doesn’t appeal to us, move on. We choose the new book from an author if we liked her last one and skip the next from the author whose last book we didn’t like. We exchange recommendations with fellow readers, read reviews, and use many other ever-changing techniques to identify those books most suited to our reading requirements and tastes. Even book bloggers such as me have choices and are more likely to pick the book we’ll like than the one we won’t to read and review. Anyone who reviews all the books they read and has a rating distribution that resembles a bell curve is showing they’re either being too tough in their reviews or are exposing themselves as poor shoppers. However, it also makes the average of most customer reviews look inflated to anyone thinking of a bell curve distribution.

Amazon’s rating system exacerbates this. Here is how they define each of the star ratings.

1 star – I Hate It

2 star – I Didn’t Like It

3 star – It’s Ok

4 star – I Like It

5 star – I Love It

Among customers who are good shoppers, one and two star reviews should be rare. Unless they do a bad job of picking books to read, if they follow Amazon’s rating instructions, the majority of reviews should be four stars with their best choices receiving five stars and the books that didn’t quite make the grade as three stars. I tested my theory, randomly selecting a significant number of people rated “top reviewers” on Amazon. I threw out those who never gave negative reviews, which was almost half (yet another reason average review ratings seem high). Among those remaining, the average review rating was 4.2. The design of Amazon’s rating system gives most products a relatively high average rating. Between this, a subset of customers who only write reviews for products they like (which I’m willing to bet is higher than the other subset, who only write reviews to complain), it isn’t surprising that the average rating feels high to many people and that review ratings are of little value for evaluating a potential book purchase?