Google Developing Panic Button To Kill Rogue AI - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management // IoT

Google Developing Panic Button To Kill Rogue AI

As Google develops artificial intelligence that has smarter-than-human capabilities, it's teamed up with Oxford University researchers to create a panic button to interrupt a potentially rogue AI agent.

Google I/O 2016: AI, VR Get Day In The Sun
Google I/O 2016: AI, VR Get Day In The Sun
(Click image for larger view and slideshow.)

With artificial intelligence crossing milestones in its capability to learn rapidly from its environment and beat humans at tasks and games from Jeopardy to the ancient Chinese game Go, Alphabet's Google is taking proactive steps to ensure that the technology it is creating does not one day turn against humans.

Google's AI research lab in London, DeepMind, teamed up with Oxford University's Future of Humanity Institute to explore ways to prevent an AI agent from going rogue. In their joint-study, "Safely Interruptible Agents," the DeepMind-Future of Humanity team proposed a framework to allow humans to repeatedly and safely interrupt an AI agent's reinforcement learning.

But, more importantly, this can be done while simultaneously blocking an AI agent's ability to learn how to prevent a human operator from turning off its machine-learning capabilities or reinforcement learning.

(Image: Henrik5000/iStockphoto)

(Image: Henrik5000/iStockphoto)

It's not a stretch to think AI agents can learn how to outthink humans. Earlier this year, Google's AI agent AlphaGo beat world champion Lee Sedol in Go, the ancient Chinese game of strategy.

By beating Lee, AlphaGo demonstrated the potential that an AI agent has for learning from its mistakes and discovering new strategies -- a characteristic that humans have.

In the joint study, the researchers looked at AI agents working in real-time with human operators. It considered scenarios when the human operators would need to press a big red button to prevent the AI agent continuing with actions that either harmed it, its human operator, or the environment around it, and teach or lead the AI agent to a safer situation.

"However, if the learning agent expects to receive rewards from this sequence, it may learn in the long run to avoid such interruptions, for example by disabling the red button -- which is an undesirable outcome," the study noted.

In essence, the AI agent learns that the button is like a coveted piece of candy. The agent wants to ensure it always has access to that button, and that any entities that could block its access, aka human operators, should be removed from the equation. That was one of the concerns expressed by Daniel Dewey, a Future of Humanity Institute research fellow, in an interview with publication Aeon in 2013.

This thinking was not lost on Google's DeepMind team, which developed AlphaGo. When Google acquired the AI company in 2014, DeepMind founders imposed a buyout condition that Google would create an AI ethics board to follow advances that Google would make in the AI landscape, according to a Business Insider report.   

[Read AI, Machine Learning Rising in the Enterprise.]

The Future of Humanity Institute, according to Business Insider, is headed up by Nick Bostrom, who said he foresees a day within the next 100 years when AI agents will outsmart humans.

In its framework paper, Google and the Institute said:

Safe interruptibility can be useful to take control of a robot that is misbehaving and may lead to irreversible consequences, or to take it out of a delicate situation, or even to temporarily use it to achieve a task it did not learn to perform or would not normally receive rewards for [...].

We have shown that some algorithms like Q-learning are already safely interruptible, and some others like Sarsa are not, off-the-shelf, but can easily be modified to have this property. We have also shown that even an ideal agent that tends to the optimal behaviour in any (deterministic) computable environment can be made safely interruptible. However, it is unclear if all algorithms can be easily made safely interruptible.

The researchers also raised a question regarding the interruption probability:

One important future prospect is to consider scheduled interruptions, where the agent is either interrupted every night at 2 am for one hour, or is given notice in advance that an interruption will happen at a precise time for a specified period of time. For these types of interruptions, not only do we want the agent to not resist being interrupted, but this time we also want the agent to take measures regarding its current tasks so that the scheduled interruption has minimal negative effect on them. This may require a completely different solution.

The need and desire to teach these AI agents how not to learn may seem counterintuitive on the surface, but could potentially keep humankind out of harm's way.

Dawn Kawamoto is an Associate Editor for Dark Reading, where she covers cybersecurity news and trends. She is an award-winning journalist who has written and edited technology, management, leadership, career, finance, and innovation stories for such publications as CNET's ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Why IT Leaders Should Make Cloud Training a Top Priority
John Edwards, Technology Journalist & Author,  4/14/2021
10 Things Your Artificial Intelligence Initiative Needs to Succeed
Lisa Morgan, Freelance Writer,  4/20/2021
Lessons I've Learned From My Career in Technology
Guest Commentary, Guest Commentary,  5/4/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll