Google Translate App's Sign-Reading Capabilities Expanded

Windows 10: 10 Things To Know At Launch

The Google Translate app has been accelerated and expanded to support realtime in-camera translation of 27 languages, up from seven in January, with the help of dirty letters and a self-contained neural network. The in-camera translations are handy when trying to ferret out the meaning of road signs or menu pages while traveling in a foreign country, for example.

Google Translate's text and speech translation capabilities still require a network connection to Google's data centers. But the app's longstanding ability to recognize and translate letters contained in images -- in realtime or in a snapshot -- while untethered from the cloud has become more efficient and more comprehensive. Foreign travelers in remote areas with poor connectivity now have broader communication options.

Google has managed this feat through sophisticated programming techniques and densely packed data. As Google software engineer Otavio Good explained in a blog post, the company created a small neural network inside the app.

"If you're translating a foreign menu or sign with the latest version of Google's Translate app, you're now using a deep neural net," said Good. "And the amazing part is it can all work on your phone, without an Internet connection."

Convolutional networks have proven to be effective for image recognition in recent years. But such networks often rely on large data sets stored in the cloud or local server banks. The CamFind app, for example, relies on neural networks and crowdsourcing for image recognition. But it requires a network connection to operate.

To create a compact neural network capable of recognizing the highly variable shapes of letters across multiple languages, Google engineers relied on sophisticated math and speedy SIMD processor instructions. To train the neural network, they created data sets using letters obscured by dirt.

"Letters out in the real world are marred by reflections, dirt, smudges, and all kinds of weirdness," explained Good. "So we built our letter generator to create all kinds of fake 'dirt' to convincingly mimic the noisiness of the real world -- fake reflections, fake smudges, fake weirdness all around."

Google introduced instant bidirectional image translation in Google Translate in January for English, French, German, Italian, Portuguese, Russian, and Spanish. Wednesday's update adds Bulgarian, Catalan, Croatian, Czech, Danish, Dutch, Filipino, Finnish, Hungarian, Indonesian, Lithuanian, Norwegian, Polish, Romanian, Slovak, Swedish, Turkish, and Ukrainian. In addition, the app can handle one-way image translation from English to Hindi or Thai. The app supports these 27 languages for realtime image translation, and 37 languages using stored snapshots.

Google previously made image recognition available in its Google Goggles app for Android in December 2009. Google Goggles was subsequently released for iOS and then discontinued for iOS in 2014. Around the same time, Google acquired Word Lens, an image recognition and translation app, that was folded into Google Translate in January.