Big Data!

I was reading in a magazine an article by the excellent Simon Brett earlier today. It was an obituary for Phyllis James and mentioned that some time ago, while talking about writing, they discussed the worst questions authors are asked.

My own favourite worst is, “Where do your books sell?”

It’s a natural question. It’s the first thing people ask me who are interested in my books and sales. A friend is trying to help me with marketing. She asked me. Another friend is keen to see how to analyse my sales data to increase sales. He asked me. Many people ask me the same question.

How should I know? That kind of data is not given to the poor old authors scribbling away in their garrets. I have asked for it, many times, but apparently the vagaries of the publishing industry mean that data of that sort just isn’t available.

Let’s get down to basics here.

Many of us get really hacked off with the notes from Amazon about what we ought to be interested in. I do, anyway.

You know how it is, you sign into Amazon, and are presented with a series of items that should be just up your street, because other “Customers who viewed this also viewed” etc. Which is brilliant for me. It means I get to see all the teenage/young adult books coming around. That is, after all, what my daughter looks for on my Amazon account.

She doesn’t have her own account.

But big data is very important nowadays. Much more so than most people realise. Your supermarket knows what you eat, it knows what cleaners you use, it knows how much you value your heart and arteries. It knows how many children you have, and even how much petrol you buy. There is so much data about you sloshing around, it’s hardly surprising that a computer-driven firm like Amazon wouldn’t be making as much use of it as possible.

And there are two problems with this.

First, publishers get only a small amount of data. They can see where books are sold, generally, and they can get a feel for whether or not a special promotion has worked. But for the majority of the sales data, Amazon has a clear monopoly and doesn’t share it. Why should it? Amazon wants to publish its own books, and that means they’ll own all production and selling of books in the most efficient manner because they know what we buy, when we buy it, what we highlighted in books, what we bookmark, even which books we never finished. That means publishers have no idea where to market new books. How can they? Meanwhile, Amazon retains control of all book markets. They know what to stock, who to target for specific books, the works.

Painting of Josselin Castle, Britanny

Second, let’s just think about the concept here. Let’s suppose Amazon were to share all their data with publishers. Would that be good? Publishers could then focus on the ever-reducing number of authors who are producing exactly what the buyers want. Brilliant! At a stroke all the wonderful, less-than-bestselling books could be trashed. Instead readers would be presented with a wonderful array of near-identical genre books. The readers’ choice would reduce dramatically, with only the top-level bestsellers winning new contracts, producing the same works with all the names changed.

Okay, perhaps there would be more variety than that, but at the end of the day, where would the flair of the individual editor show itself?

The appalling thing, for me, was to learn that Amazon doesn’t use humans any more. A programmer had a brilliant idea: people who bought one product may often buy a second or third. If you have enough people in a sample, you can predict with some accuracy what the second and third products will be. It’s the same as an actuarial calculation: you may not be able to say when Joe Bloggs is going to die, but if you have a sample of 100,000 or 1,000,000 people, you can predict how many of them are likely to die in any year. You can predict based on age, based on health, based on occupation, or any number of other attributes. All you need is the data, which Amazon has now acquired.

Editors provide an essential function in publishing. They find the oddball books that can fire the imagination of a generation. Take Harry Potter. Who could have predicted that the series would grow to be so popular? I seriously doubt that Amazon would have predicted that. Similarly with Vampire series, or dystopian futures in which children are forced to fight to the death. These books would not have achieved anything if pure data sets were used.

So, from an author’s perspective, I really wish Big Data didn’t exist. Since it does, I’m sure Amazon maintaining a monopoly is not good for anyone.

The other side is, authors cannot be driven towards one type of book or another. That is good. Authors need to write with their own authorial voice, they need to be fired with enthusiasm, anger, love or loathing before they pick up a keyboard. If they are seen as mere word-jugglers, who can be instructed to “write another story like X and Y, with these factors added to the plot”, we shall all be the poorer, but that is the inevitable direction Big Data will point in.

So, as an author, I am glad that my publishers really don’t have much of a clue about where my books sell. The main thing is, they appeal to all ages. That, for me, has to be enough.

Mind you, the next few years are going to be very interesting indeed.