Geekend: Replacing the Turing Test - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
IT Life
Commentary
11/21/2014
01:10 PM
David Wagner
David Wagner
Commentary
Connect Directly
Twitter
RSS
100%
0%

Geekend: Replacing the Turing Test

We need a better way to test artificial intelligence.

Artificial intelligence is here -- or very near -- and it's time we had a more sophisticated way to measure it than the outdated Turing test. A Georgia Tech professor is offering an alternative test he calls Lovelace 2.0. Is it the test we need?

First, let's discuss what is wrong with the Turing test and other major tests of artificial intelligence.

The Turing test, perhaps the most famous test of artificial intelligence, was conceived in the 1950s -- which itself is a red flag. How could its developers conceive of the revolution that was to come when computers were still using vacuum tubes? Beyond that, the Turing test relies on deception; the computer needs to trick humans. This is problematic for many reasons.

First, the notion of artificial intelligence that's specifically designed to deceive is not only creepy, but also potentially harmful. Even if you don't tell it that it is trying to deceive when it interacts with humans, the programmers have built that idea into the device's thinking.

Just as importantly, human behavior isn't always intelligent. Acting human and being intelligent are two very different things. Tricking a human into thinking you are human is a language exercise as much as it is an intelligence exercise.

[Want a difficult colleague to acknowledge a problem? Here's how: Geekend: Stubborn Deniers Demand Creative Solutions.]

Basically, it comes down to a simple fact: Tricking a human in a chat room doesn't mean your AI is suited to do anything else of value or intelligence.

Other tests have been proposed, based on answering vague language tests that often trick computers, but humans can easily pass or work around reading comprehension or writing.

What we need, according to Georgia Tech's Mark Riedl, is a test that shows a broad base of potential skills and types of intelligence. He suggests that a truly, err… intelligent artificial intelligence "develops a creative artifact from a subset of artistic genres deemed to require and the artifact meets certain creative constraints given by the human evaluator."

Interesting. To be smart, Riedl says, you must be creative and potentially "artistic" -- and that does seem to require more intelligence than a chat bot. But I'm not sure why, for instance, an artificial intelligence designed to drive a self-driving car safely isn't intelligent simply because it can't tell a joke, write a story, or paint a painting.

There are robots that can do these things -- we've seen painting robots, joke-writing software, and even an AI that creates magic tricks. It seems to me that making a magic trick takes some serious intelligence and insight into humans. A harder Turing test might create a magic trick -- rather than a chat bot -- that fools humans.

But does any of this indicate intelligence, or it is simply a sign of adapting to a specific job? Intelligence, to me, means being able to adapt to a demand you haven't been programmed to do. If your joke-writing program can switch to writing poems, for example, that's intelligence. Anything else might be really neat -- but it's specialized.

So what test would I suggest? I have a few ideas that might seem a little out there -- but here goes:

Can it take the latest Cosmo quiz?
If your computer can read, comprehend, and honestly answer a Cosmo quiz, it boasts serious reading comprehension skills, an ability to process casual language, and an ability to answer questions with shades of gray. If your AI can find out what kind of animal it is in bed, for example, it really is intelligent (and possibly a great date).

Can it get across a bridge to find the Holy Grail?
I think the ability to respond to a question with another question might be one of the best signs of intelligence. I also want to see someone ask Watson about its quest.

Can it help Bunny Watson figure out who to marry?
In the classic movie The Desk Set, Katharine Hepburn, playing Bunny Watson, believes her job as a researcher for a TV network is about to be made redundant by a computer that seems to know everything. (Incidentally, her job actually will be made redundant by the Internet decades later.) The computer in the movie can instantly answer questions of facts, but it can't evaluate. Ultimately, Spencer Tracy asks Hepburn to cancel her engagement after asking the computer if she should marry. Watch that scene at the 3:40 mark of the video below, or enjoy the whole clip. (The entire movie is available for free here).

We now have half the technology this movie envisioned in 1957. If we get the other half, we'll have a pretty smart computer.

OK, so none of these are perfect tests. One reason for that is that we've never really been able to come up with an accurate test for intelligence in people. Presumably, if we can't reliably test intelligence in people, we can't test it in a computer.

What do we really mean when we talk about an artificial intelligence? Is it simply something that can do a single task well, or is it something that can reason through any major task? Do you agree with my notion that true intelligence is doing something you've never been programmed to do -- or is that unfair? After all, I have to program myself to learn to play the violin.

What is your test of intelligence? Tell me in the comments.

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.

David has been writing on business and technology for over 10 years and was most recently Managing Editor at Enterpriseefficiency.com. Before that he was an Assistant Editor at MIT Sloan Management Review, where he covered a wide range of business topics including IT, ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
David Wagner
50%
50%
David Wagner,
User Rank: Strategist
11/30/2014 | 2:51:31 PM
Re: Innovative
@kstaron- I'm not so sure a "leap" of logic is necessary. I think our leaps are usually just looking at something from a new angle. Just stargiht asking the right questions. 

What it seems like you are describing to me is genius. That seems like a high bar. 

I think i'd settle for making the right decision when asked a question or asked to a task the vast majority of the time. That would beat the crap out of most humans. 

But I do think you're on to one thing-- a "leap" of logic implies being able to do a broad range of things. You can't leap if your are specialized. 
David Wagner
100%
0%
David Wagner,
User Rank: Strategist
11/30/2014 | 2:44:22 PM
Re: Replacing the Turing Test
@zerox203- Thanks. One thing that current robots remind me of is the old proverb where blind men touch an elephant and one touches the tail and sais, "Oh, a snake." And another touches the leg and says, "oh a tree." We've got robots that do a lot of single things well. One day someone is going to combine them into an elephant and we're going to be blown away.
David Wagner
50%
50%
David Wagner,
User Rank: Strategist
11/30/2014 | 2:40:32 PM
Re: Write an IW blog?
@tzubair- Writing a blog as good as mine are isn't much of a feat. But as good as the rest of the IWeek crew, that's an achievement. 

But here's the interesting part. We've got computers that can write in ways that look rather human. The hard part, I think, would be creating the computer that could gather the pertinent information and research first. i suspect Watson could be trained to fake it at this point with some specific topics. 

But you are right. I'd like to think good writing is one of the last places humans will triumph over computers.
tzubair
50%
50%
tzubair,
User Rank: Ninja
11/30/2014 | 5:08:57 AM
Re: Intelligence versus judgments
". When ethics and judgments are factored in, the decision becomes even more complex."

@impactnow: I think what really makes a difference between human decision makking and a machine's, is the fact that human decisions are rarely rational. We may think we make rational decisions but mostly our rationality is heavily influenced by emotions and the state of mind we're in. Once the machines are able to have an emotional state of their own and take decisions that can factor in the emotional aspect, they might be considered intelligent enough.
tzubair
50%
50%
tzubair,
User Rank: Ninja
11/30/2014 | 4:44:42 AM
Write an IW blog?
How about a computer that's capable of writing an IW blog that is as good as the one written by Dave Wagner and others? Would that be an easy feat to achieve? Assuming we have a computer who's capable of doing that, do we accept it to be intelligent enough?
kstaron
50%
50%
kstaron,
User Rank: Ninja
11/26/2014 | 9:01:11 AM
Innovative

Interesting. To be smart, Riedl says, you must be creative and potentially "artistic" -- and that does seem to require more intelligence than a chat bot. But I'm not sure why,

The idea of artistic, if you replaced the wording choice with innovative it might be easier to understand. Being "artistic" is not the ability to render a painting exactly (as computers can already do), but it it is to see something in a new way or to represent something in a new way. It is the essence of art but also of invention and progress. An "artistic" mind (or computer) can make leaps of logic and understand the patterns to get to a new idea from two disjointed ones.

impactnow
50%
50%
impactnow,
User Rank: Author
11/25/2014 | 12:38:02 PM
Intelligence versus judgments
Dave LOL, you're absolutely right intelligence is much deeper than programming can address today . Our definition of artificial intelligence today works great with binary decisions . However, since so many decisions are not binary nature artificial intelligence often fails . When ethics and judgments are factored in, the decision becomes even more complex. We may get there eventually but I don't think it's coming anytime soon . In the interim I loved your tests!
zerox203
100%
0%
zerox203,
User Rank: Ninja
11/25/2014 | 2:43:06 AM
Re: Replacing the Turing Test
We were talking about this during the Friday podcast (which people should check out of they want to get a little more color commentary on the geekend and other posts), and I was really looking forward to this post going up to see what people had to say about it. The bit about the computer making card tricks was particularly interesting to me, because I think it most closely meets that idea of improvisation. Even if it's a little thing (slightly adjusting the perameters on established cart tricks), those are the real building blocks of establishing that machine intelligence we're looking for, in my opinion, not the ability to  do things quickly or retain a lot of information, like Watson or Deep Blue.

I see what some of you are saying about the piecemeal approach of current 'smart' robots in that they only perform specialized tasks, but I think Dave is right on the money. Even if the people working on them don't necessarily know they're doing it, these are the precursors to greater things, and eventually these disparate technologies will reach a point of convergance. In truth, it may happen faster than we realize. We may wake up one day and think 'hey, when did all this happen?'. So, again, I think Dave is right -  we're better off taking these things seriously too soon rather too late. After all, 2018 is not as far away as it sounds.
TerryB
50%
50%
TerryB,
User Rank: Ninja
11/24/2014 | 1:56:03 PM
Re: Futurama
The answer depends completely on definition of intelligience. I would call your two examples more like idiot savants: they know how to do one thing (or type of thing) extremely well, maybe better than us.

But intellegience is something I personally associate with creativity. That quality that led Albert Einstein to expand his basic mathematic/physics knowledge and formulate E=MC(squared). Or what allowed Led Zep to write Stairway to Heaven. When computers get to that point, then we are talking human-like.
David Wagner
100%
0%
David Wagner,
User Rank: Strategist
11/24/2014 | 1:35:39 PM
Re: Futurama
@TerryB- Are we light years away? Google claims they will be going into production in 2018 on a self-driving car. That seems like some sort of serious intelligence to me. Not necessarily human-like but intelligent. We've got machines in assembly lines making split second sophisticated decisions. All of this adds up to something. 

We're probably decades or more from making a Data-like robot that emulates human behavior in all ways. But we're getting to something fairly soon we might need to start calling intelligent aren't we?
Page 1 / 2   >   >>
Slideshows
Strategies You Need to Make Digital Transformation Work
Joao-Pierre S. Ruth, Senior Writer,  11/25/2019
Commentary
Enterprise Guide to Data Privacy
Cathleen Gagne, Managing Editor, InformationWeek,  11/22/2019
News
Watch Out: 7 Digital Disruptions for IT Leaders
Jessica Davis, Senior Editor, Enterprise Apps,  11/18/2019
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll