How not to do a Turing test

The idiot news people (sorry BBC) made a splash recently by claiming that a computer had successfully passed the Turing test. This is an idea modified from an article by the great Alan Turing, who suggested that a good test of artificial intelligence is to communicate with the computer down the wire and if you can't tell whether it's human or computer, then it passes - you have AI. (Turing's original concept is actually significantly more confusing, but this is the version usually given.)

There are several problems with this story. One is that the test as described is far too easy to pass. All that is required is that the machine is 'mistaken for a human more than 30% of the time during a series of five minute keyboard conversations.' That's a pretty low pass rate. I don't think you even get a GCSE for a 30% success. (Turing actually asked 'Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman?' - in his version the computer was rather oddly compared with two human, one who always tells the truth and one who doesn't.) Apparently the chatbot 'Eugene Goostman' convinced 33% of the judges that it was human, therefore the organisers claim a success. What we aren't told is how many judges there are - if there were only 3, we are talking about one person being convinced - hardly impressive.

Frankly this test at Reading University was a farce, and it's very sad that some academics enthused over the success the way they did (and that the Princeton web page portrayed above seems to think it was a success). The fact is, you only had to spend the requisite five minutes with Eugene and you would know perfectly well that 'he' is a program. Interestingly, the team behind Goostman have taken the chatbot down, probably due to the hilarity of those trying it out and discovering just how bad it is. (It should be here.) But even the simplest of tactics - asking the bot what a word the bot itself had used meant - showed it up as a failure. It couldn't explain the meaning of a single word it used. A 13-year-old Ukrainian speaking a second language, as Eugene is supposed to be, might not give good dictionary definitions, but could and would have a stab at this. And there are lots of other easy conversational ploys that the chatbot failed on.

If you would like a go at a very early chatbot that was also briefly claimed to have passed the Turing test, take a look at the grandmother of them all, Eliza.

I leave the final word with Dean Burnett, who claimed in a Guardian article that an actual 13-year-old  boy had passed the Turing test and was declared human. Now that, surely, is impossible.