The Costs of Using AI Editing Tools

I’m immediately predisposed to believe there may be significant costs, but this blog post’s title actually comes from a post at Jane Friedman’s blog:

The Hidden Costs of AI Copyediting Tools: An Editor’s Review

Here is what the author of this post, has to say:

To more objectively gauge the abilities of various AI tools, I took a fantasy short story and performed developmental, line editing, and copyediting on it, then compared my edits with the feedback generated by different AI-powered editing platforms. To round out the review, I also considered other criteria, such as user experience and external factors like the terms of service and how users’ work is stored. For copyediting tools, I specifically evaluated Grammarly, Hemingway Editor, ProWritingAid, and AutoCrit.

I think this editor was missing the boat if she didn’t also evaluate ChatGPT, but let’s see what she concluded. She was using a short story written by her sister, she says, and she declares this was a good quality story.

Now, what are the takeaways? The author of this post considers these tools useful for avoiding repetition, such as cutting down on the number of sentences beginning with “I.” She also noted:

However, [Hemingway] also suggested many things that were either not incorrect in the original or would be an error to introduce. For example, it recommended a hyphen in the verb “half dragging.” However, the Chicago Manual of Style recommends leaving verb forms with “half” open. It did not recognize that “keen” is a verb as well as an adjective and thus flagged it as an error. It suggested changing a period at the end of a line of dialogue to a comma. However, what followed was not a dialogue tag, so the original period was correct. 

I used to prefer hyphenating half- constructions; then I had a copyeditor who was death on hyphens for half- constructions. She re-trained my eye, and now my preference is to hyphenate with half- if and only if it is used in a two-word adjective construction, such as, here, this is a half-moon gateway, whereas that up there in the sky is a half moon and this is my half brother. I don’t know how other style guides handle this, but I presume the copyeditor who pushed me away from hyphenating half- was probably following the Chicago style.

For ProWritingAid, the author of this post notes that ProWritingAid offers an almost ridiculous amount of tools, and I have to say, I flinch when someone who says she is an editor uses “amount” in this sentence. I wonder if ProWritingAid would point out that this is an error? The main problem with ProWritingAid was apparently the flood of suggestions, many of which were irrelevant or wrong.

During my review, I found that for every one potential problem that I actually addressed or reworded, there were easily thirty that were not problems at all. For instance, in the cliché check, it flagged the word “ginger” for me (in “ginger tea”). In the diction check that searches for vague or abstract words, it flagged “all” in the phrase “all too well,” which I would not consider too vague. For the phrase “some of her concerns,” the suggested replacements for “some” were “the,” “this,” and “that,” none of which work in context. 

… the of her concerns? … … … this concerns? … … … that concerns? Too right, none of them work! That’s a dire sort of mistake! Every possible usage I can see for this would be nonsensical!

[I]n one place, it suggested that a serial comma was missing, but the sentence it highlighted wasn’t a list. It highlighted two dialogue tags as being the present tense, but they were actually past tense (and therefore matched the rest of the story). It did flag a sentence with an error but identified the error as a possible wrong verb form when the error was actually a missing word. It didn’t know the word “fletched” and did not correct “snowdrift” to be a single word.

This sounds pretty terrible to me.

Grammarly was able to suggest some solid grammatical suggestions, but the feedback was often inconsistent, sometimes suggesting a comma should appear before a conjunction that joins two independent clauses and then in other places recommending that I take it out. Commas with restrictive and nonrestrictive phrases were also inconsistently recommended or corrected. Suggested changes often included errors, the most common error being the introduction of a misplaced modifier. For more complex sentences, it often had difficulty with subject/verb agreement. … many of the issues flagged in the short story were not errors but simply words or phrasing that weren’t how Grammarly would say it, pushing me to make changes that would conform to Grammarly’s preferred simpler, skimmable writing. Changes that it suggested to me included using “daily” instead of “every day,” “silently” instead of “in silence,” …

And so on, there’s more, but this gives us an idea of what Grammarly is like as a tool. Or “tool.”

Here’s the overall conclusion:

As an editor, I find the drawbacks to using AI copyediting tools to be greater than their offered benefits at this time, particularly the risk of losing personal voice. I have edited several heavily AI-assisted texts in the past year, and though the authors did not disclose their use of AI, they didn’t have to. While the text was grammatically sleek, I never had the sense that there was someone speaking to me, someone on the other side of the words, and it was very easy to identify places where the author had accepted a change they did not understand.  

Bold is mine, because I think this is what we’re going to see: a nigh-unto-infinite flood of fake text, produced by people with inadequate writing skill who are following suggestions they don’t understand, produced by AI that does not understand anything, least of all effectiveness in writing.

An interesting post! I don’t plan ever to feed a story into any AI editorial “tool” because — and the linked post does say this — the risk that the tool is stealing your work is 100%, and no thank you.

Please Feel Free to Share:

The post The Costs of Using AI Editing Tools appeared first on Rachel Neumeier.