‘We Can Read It For You Wholesale’

One of the more irritating features of the last couple of years has been the rush to incorporate AI into everything, not because there is much evidence that it makes anything better or that there is huge public demand for this, but for Fear Of Missing Out in case it does turn out to be indispensable. One wonders how far these highly-paid and supposedly brilliant, hard-nosed business executives have actually bought into the hype and how far they simply assume that everyone else has – though “let’s make our product/service crappier because lots of people think that a fancy autocomplete widget is going to evolve into her’s Samantha” is a… courageous strategy either way.

Universities being what they are, this hasn’t happened to us yet; we’re still at the stage of educational consultancies writing reports about how There Is No Alternative to such a development so institutions should pay them large sums of money to implement the transformation. As ever, the ingrained idea that adapting to the future will involve disruption and destruction segues into the assumption that disruption and destruction will necessarily fit the institution for the future. Of course, replacing senior management teams with a robotic algorithm, or indeed a speak-your-weight machine, is unlikely to make things significantly worse, assuming that it’s even noticeable. But for the moment, the front runner in the ‘headlong rush into gAI stupidity’ stakes is… JSTOR.

I’ve moaned before about JSTOR’s ‘if you enjoyed this recent publication, you might like this 1952 article on a vaguely related subject’ widget, but was willing to believe that it was someone’s idea of being helpful. More cynical interpretations are available; isn’t this exactly the same technique that other, less ostensibly high-minded websites use to keep us on their platforms, clicking on ever more content and exposed to ever more adverts? Okay, JSTOR isn’t trying to sell us anything (yet…) – but do they have an incentive to ensure that users interact with as many publications as possible? Are we in a rabbit/duck situation, where what seems to us academics to be a hugely valuable gateway to published research is at the same time a means for publishers to maximise article views, driving up revenues via their contracts with libraries..?

Annoying as it is to have students citing lots of outdated and irrelevant publications in the belief that this is what independent research looks like, it pales in comparison with JSTOR’s latest wheeze: built-in AI, which allows you to ask questions about a given publication and get a summary of it. I mean, what the hell?!? How is this anything other than a mechanism for minimising engagement time with individual publications in order to maximise the number that can be engaged with? Quantity over quality? It takes for granted, or at best acquiesces in, the idea that reading an article is just about extracting its core content from irrelevant waffle and that the goal of research is to amass as many references as possible – seeing these as things that can be made more efficient through technology. Trying to teach research skills is going to be SO much more entertaining when the actual tools and resources are modelling bad practice. In terms of humanities research, not so much “they’re using our own satellites against us” as “the call is coming from inside the house”…

This seems very bad in principle, even if the summaries were good and reliable. As I’ve suggested before, we really need some decent research on this latter question, given how far many of our students seem to trust them. First indications aren’t great; gAI seems to ‘shorten’ rather than ‘summarise’, incapable of differentiating between sentences that are essential for articulating the argument and those that develop or support it, and continues intermittently to insert irrelevant and unreliable information. But obviously I had to try it out for myself.

The immediate temptation is to experiment with one’s own work; after all, I know exactly what I was arguing* so can easily evaluate the accuracy of the summary. It’s interesting to discover, for example, that my 2013 article on ‘Thucydides Quote Unquote’ primarily “sheds light on the enduring influence of Thucydides’ quotes and the evolving interpretations of his writings over time, highlighting the significance of these quotes in shaping perceptions of history, politics, and military strategy” – not completely wrong, except that my focus is on quotes that aren’t actually Thucydides, of which the summary offers just one example in passing, rather than making it the focus of the whole thing. You wouldn’t gather from the summary of ‘Decadence as a theory of history’ (2007) that it was a piece on historiography and the history of ideas, not ‘decadence’ as a real phenomenon in history, but apparently “The narrative of decay and decadence is a central theme in Morley’s analysis of historical processes and societal evolution.”

The obvious objection here is that I know what I meant to argue, but maybe it isn’t actually clear in practice, so the fact that the summary gets it wrong isn’t the decisive argument I think it is. Yes, it takes my 2001 JRS article completely at face value as an alternative narrative of the demography and development of Roman Italy in the Republic, ignoring the introduction and conclusion which make it clear that this is a thought experiment which concludes that this version of events isn’t plausible – but I know of at least one respected colleague who read it in the same way. Can the AI summariser be blamed if my exposition was actually a bit crap?

Better, then, to test it on something that is widely acknowledged as brilliant, insightful and above all a model of clarity of argument – exactly the sort of piece we might direct students to read both for the importance of its argument and for the way in which the ideas are developed. What does it make, for example, of Keith Hopkins’ classic piece of speculative model-building, ‘Taxes and Trade in the Roman Empire’ (JRS 1980):

In this essay on Roman economic history, the author explores the impact of taxation and trade on the Roman economy from 200 B.C. to A.D. 400. The author discusses the volume of silver coins in circulation, military expenditure, and the integration of the monetary economy in the High Empire. The text also estimates the level of taxation in the Roman Empire and its importance in the economy, highlighting the role of tax-farming and the collapse of the fiscal system in the third century A.D. The author concludes that the breakdown of central control over taxation led to the restoration of central control by the strong government of Diocletian and Constantine. The essay aims to stimulate further research and debate in the field of Roman economic history.

Uh huh. I mean, as a summary of the topics that are discussed in Hopkins’ article it’s not actually wrong, but it entirely misses every point of the exercise, above all the radical nature of Hopkins’ argument in theoretical and methodological terms. It seems fair to say that any student relying on this summary would be completely at sea as to why anyone would bother reading the article at all forty-five years later.

Okay, M.I. Finley, ‘The ancient city: from Fustel de Coulanges to Max Weber’ (1977) – THE article in which the idea of the ‘consumer city’ as an ideal type was put forward, which then dominated debates about the relationship between city and countryside in antiquity for decades:

In this academic article, M.I. Finley delves into the significance of ancient cities in the Graeco-Roman world, highlighting how even the rural population resided in communities rather than isolated farms. The growth of towns was deemed crucial for civilization, with Aristotle emphasizing health, defense, politics, and aesthetics in urban planning. Critiquing Fustel de Coulanges’ work, the author stresses the necessity for a more comprehensive understanding of urban development, considering economic and social factors that influenced ancient cities’ rise and fall. Drawing on the insights of scholars like Marx and Weber, the article explores the economic structures of ancient cities, emphasizing the separation of urban and rural areas. By advocating for a detailed analysis of available data and proposing a typology of ancient towns based on various factors, the author aims to enhance our comprehension of the role of cities in the ancient world.

‘Consumer city’? It’s there in the passing reference to typology, if you know that you should be looking for it, but you would never otherwise gather that this is the crucial analytical point. The most generous interpretation I can offer is that maybe the importance of this argument is recognisable only if you’re aware of the issues and pay attention to Finley’s footnotes and passing remarks, rather than if you just focus on the substance of what he says – but that still amounts to saying that the summary is basically useless for getting a proper understanding of the article.

This is actually quite horrifyingly addictive – and, yes, I’m already past the point where I think the tool is going to be at all useful, and instead trying to think of articles which it’s liable to get spectacularly wrong in an entertaining manner. In brief, this strongly confirms the sense that AI offers at best something like a compressed version of a piece, or rather a version in which material from the original has been subtracted on a statistical basis, i.e. focusing on repeated words and phrases, rather than on any sort of qualitative (let alone informed) basis. It tells you, more or less, what the article contains in the way of subject matter; it really isn’t very good at presenting what the author may be doing with that subject matter, and may well entirely miss the point. And it’s really not going to be much use to anyone actually trying to research the topic.

The good news is that JSTOR’s tool is still in Beta, so conceivably they might pay attention to a combination of loud objections and a great deal of jeering and mockery. But much will depend on why they thought this was a good idea in the first place – it may not be very different from trying to yell at Amazon or Google…

*Full disclosure: this is not always entirely true. I recently received my copy of Blouin & Akrigg, eds., The Routledge Handbook of Classics, Colonialism, and Postcolonial Theory, which contains a chapter by me on Thucydides, most of which I have no real recollection of writing – but it was the first thing I managed to write while struggling to emerge from Long COVID brain fog, so there is a possible reason for this…