Geekend: Replacing the Turing Test - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
IT Life
Commentary
11/21/2014
01:10 PM
David Wagner
David Wagner
Commentary
Connect Directly
Twitter
RSS
100%
0%

Geekend: Replacing the Turing Test

We need a better way to test artificial intelligence.

Artificial intelligence is here -- or very near -- and it's time we had a more sophisticated way to measure it than the outdated Turing test. A Georgia Tech professor is offering an alternative test he calls Lovelace 2.0. Is it the test we need?

First, let's discuss what is wrong with the Turing test and other major tests of artificial intelligence.

The Turing test, perhaps the most famous test of artificial intelligence, was conceived in the 1950s -- which itself is a red flag. How could its developers conceive of the revolution that was to come when computers were still using vacuum tubes? Beyond that, the Turing test relies on deception; the computer needs to trick humans. This is problematic for many reasons.

First, the notion of artificial intelligence that's specifically designed to deceive is not only creepy, but also potentially harmful. Even if you don't tell it that it is trying to deceive when it interacts with humans, the programmers have built that idea into the device's thinking.

Just as importantly, human behavior isn't always intelligent. Acting human and being intelligent are two very different things. Tricking a human into thinking you are human is a language exercise as much as it is an intelligence exercise.

[Want a difficult colleague to acknowledge a problem? Here's how: Geekend: Stubborn Deniers Demand Creative Solutions.]

Basically, it comes down to a simple fact: Tricking a human in a chat room doesn't mean your AI is suited to do anything else of value or intelligence.

Other tests have been proposed, based on answering vague language tests that often trick computers, but humans can easily pass or work around reading comprehension or writing.

What we need, according to Georgia Tech's Mark Riedl, is a test that shows a broad base of potential skills and types of intelligence. He suggests that a truly, err… intelligent artificial intelligence "develops a creative artifact from a subset of artistic genres deemed to require and the artifact meets certain creative constraints given by the human evaluator."

Interesting. To be smart, Riedl says, you must be creative and potentially "artistic" -- and that does seem to require more intelligence than a chat bot. But I'm not sure why, for instance, an artificial intelligence designed to drive a self-driving car safely isn't intelligent simply because it can't tell a joke, write a story, or paint a painting.

There are robots that can do these things -- we've seen painting robots, joke-writing software, and even an AI that creates magic tricks. It seems to me that making a magic trick takes some serious intelligence and insight into humans. A harder Turing test might create a magic trick -- rather than a chat bot -- that fools humans.

But does any of this indicate intelligence, or it is simply a sign of adapting to a specific job? Intelligence, to me, means being able to adapt to a demand you haven't been programmed to do. If your joke-writing program can switch to writing poems, for example, that's intelligence. Anything else might be really neat -- but it's specialized.

So what test would I suggest? I have a few ideas that might seem a little out there -- but here goes:

Can it take the latest Cosmo quiz?
If your computer can read, comprehend, and honestly answer a Cosmo quiz, it boasts serious reading comprehension skills, an ability to process casual language, and an ability to answer questions with shades of gray. If your AI can find out what kind of animal it is in bed, for example, it really is intelligent (and possibly a great date).

Can it get across a bridge to find the Holy Grail?
I think the ability to respond to a question with another question might be one of the best signs of intelligence. I also want to see someone ask Watson about its quest.

Can it help Bunny Watson figure out who to marry?
In the classic movie The Desk Set, Katharine Hepburn, playing Bunny Watson, believes her job as a researcher for a TV network is about to be made redundant by a computer that seems to know everything. (Incidentally, her job actually will be made redundant by the Internet decades later.) The computer in the movie can instantly answer questions of facts, but it can't evaluate. Ultimately, Spencer Tracy asks Hepburn to cancel her engagement after asking the computer if she should marry. Watch that scene at the 3:40 mark of the video below, or enjoy the whole clip. (The entire movie is available for free here).

We now have half the technology this movie envisioned in 1957. If we get the other half, we'll have a pretty smart computer.

OK, so none of these are perfect tests. One reason for that is that we've never really been able to come up with an accurate test for intelligence in people. Presumably, if we can't reliably test intelligence in people, we can't test it in a computer.

What do we really mean when we talk about an artificial intelligence? Is it simply something that can do a single task well, or is it something that can reason through any major task? Do you agree with my notion that true intelligence is doing something you've never been programmed to do -- or is that unfair? After all, I have to program myself to learn to play the violin.

What is your test of intelligence? Tell me in the comments.

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.

David has been writing on business and technology for over 10 years and was most recently Managing Editor at Enterpriseefficiency.com. Before that he was an Assistant Editor at MIT Sloan Management Review, where he covered a wide range of business topics including IT, ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
Page 1 / 2   >   >>
PedroGonzales
100%
0%
PedroGonzales,
User Rank: Ninja
11/21/2014 | 11:52:00 PM
Turing test, a worth persuit?
when I think of AI I think of the robots I have read in sci fi books. Robots that support humans on their daily tasks, such as getting help at the airport or flying complicated machinery(ex: something like the iron man computer, if that ever happens).  I think that coming up with a true intelligence test for AI isn't a worthy activity.  Just because deep blue beat the best chest player, does that mean that a computer is smart than a human? It can search to various moves ahead of a human, yes.  Is that enough to make it intelligent?.     

 

 
jastroff
100%
0%
jastroff,
User Rank: Ninja
11/22/2014 | 1:28:26 PM
Turing, Tracy and Hepburn
@dave – lots of food for thought

Anyone who knows and likes the film Desk Set has a great deal of smarts and good taste. It works on many, many levels as social and scientific testimony.

And you're right, we need a better test for AI, such as it is evolving.

What might work? A test for how a problem is approached and solved, and the ability to be a good observer. The second one might give computers some problems if they can't see the entire room/environment and interpret actions and artifacts. So, speaking and vision would be needed, and a wide corpus of "personal" knowledge and "associations" to other things would be required as well. Reporting it in a coherent and meaningful way is also critical.

Was curious about: "How could its developers conceive of the revolution that was to come when computers were still using vacuum tubes?"

Isn't that what separates the visionaries from the rest of us? Maybe Turing wasn't the best visionary, even though he was a superb scientist.

Best visionary example in that era: Dr. Vannevar Bush and the MEMEX, which in 1945,  he predicted, accurately, the need for a digital desktop for the retrieval and processing of information – which was growing in ever large amounts even then. He saw the need for relational databases and personalized information retrieval. As head of the US Office of Scientific Research and Development during WWII, Bush oversaw all science and technology for the war effort – and maybe had a better grasp of what was to come.  For further info see Wikipedia

We were privileged to have both Turing, Bush, and all the other scientists surrounding them.
Gigi3
100%
0%
Gigi3,
User Rank: Ninja
11/24/2014 | 4:25:24 AM
Re: Turing test, a worth persuit?
"Just because deep blue beat the best chest player, does that mean that a computer is smart than a human? It can search to various moves ahead of a human, yes.  Is that enough to make it intelligent?.     "

Pedro, never. Computer ca work only on the basis how the information or logics keyed to it; which's again the brain of a programmer. The only thing is it can executed very fast.
Gigi3
100%
0%
Gigi3,
User Rank: Ninja
11/24/2014 | 4:34:13 AM
Re: Turing, Tracy and Hepburn
"What might work? A test for how a problem is approached and solved, and the ability to be a good observer. The second one might give computers some problems if they can't see the entire room/environment and interpret actions and artifacts. "

Jastroff, observation is very important, where most of us are lacking. For any test or problem, keen observation is important for analyzing the issues.
David Wagner
100%
0%
David Wagner,
User Rank: Strategist
11/24/2014 | 12:57:42 PM
Re: Turing test, a worth persuit?
@Pedro- I agree with you that Deep Blue winning at chess doesn't make it smart. And even Watson winning at Jeopardy doesn't make it intelligent.

But let's face it, computers are getting smarter. We are tasking them with making more interesting and more complicated decisions on our behalf. Don't we need to be able to test computers as they get smarter so we know exactly how much we can trust them with? And what we can trust them to do?
David Wagner
100%
0%
David Wagner,
User Rank: Strategist
11/24/2014 | 1:26:51 PM
Re: Turing, Tracy and Hepburn
@jastroff- True enough about how some visionaries nail it. I stand corrected.

It is interesting you bring up "being observant" as I feel like that is a growing strength of computers. One thing that always bugged me about Data in Star Trek is that he couldn't see behind himself. I get the idea that he was artificial life, but would you have put sensors in places other than his eyes? 

The great thing about computers/robots is that they should be able to see more than we do. We can put "eyes" and "ears" all over them. 
TerryB
100%
0%
TerryB,
User Rank: Ninja
11/24/2014 | 1:27:52 PM
Futurama
Isn't Bender what a human artificial intelligience would look like? It seems so obvious. :-)

I don't know about a formal test but this would fall into the "I'll know it when I see it" category. We are light years from that now.
David Wagner
100%
0%
David Wagner,
User Rank: Strategist
11/24/2014 | 1:35:39 PM
Re: Futurama
@TerryB- Are we light years away? Google claims they will be going into production in 2018 on a self-driving car. That seems like some sort of serious intelligence to me. Not necessarily human-like but intelligent. We've got machines in assembly lines making split second sophisticated decisions. All of this adds up to something. 

We're probably decades or more from making a Data-like robot that emulates human behavior in all ways. But we're getting to something fairly soon we might need to start calling intelligent aren't we?
TerryB
50%
50%
TerryB,
User Rank: Ninja
11/24/2014 | 1:56:03 PM
Re: Futurama
The answer depends completely on definition of intelligience. I would call your two examples more like idiot savants: they know how to do one thing (or type of thing) extremely well, maybe better than us.

But intellegience is something I personally associate with creativity. That quality that led Albert Einstein to expand his basic mathematic/physics knowledge and formulate E=MC(squared). Or what allowed Led Zep to write Stairway to Heaven. When computers get to that point, then we are talking human-like.
zerox203
100%
0%
zerox203,
User Rank: Ninja
11/25/2014 | 2:43:06 AM
Re: Replacing the Turing Test
We were talking about this during the Friday podcast (which people should check out of they want to get a little more color commentary on the geekend and other posts), and I was really looking forward to this post going up to see what people had to say about it. The bit about the computer making card tricks was particularly interesting to me, because I think it most closely meets that idea of improvisation. Even if it's a little thing (slightly adjusting the perameters on established cart tricks), those are the real building blocks of establishing that machine intelligence we're looking for, in my opinion, not the ability to  do things quickly or retain a lot of information, like Watson or Deep Blue.

I see what some of you are saying about the piecemeal approach of current 'smart' robots in that they only perform specialized tasks, but I think Dave is right on the money. Even if the people working on them don't necessarily know they're doing it, these are the precursors to greater things, and eventually these disparate technologies will reach a point of convergance. In truth, it may happen faster than we realize. We may wake up one day and think 'hey, when did all this happen?'. So, again, I think Dave is right -  we're better off taking these things seriously too soon rather too late. After all, 2018 is not as far away as it sounds.
Page 1 / 2   >   >>
Slideshows
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
News
What Comes Next for the COVID-19 Computing Consortium
Joao-Pierre S. Ruth, Senior Writer,  11/24/2020
News
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
White Papers
Register for InformationWeek Newsletters
The State of Cloud Computing - Fall 2020
The State of Cloud Computing - Fall 2020
Download this report to compare how cloud usage and spending patterns have changed in 2020, and how respondents think they'll evolve over the next two years.
Video
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Slideshows
Flash Poll