Google Adds To BigQuery Big Data Capabilities - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Cloud // Software as a Service
News
4/20/2015
09:29 AM
Connect Directly
Twitter
RSS
E-Mail
100%
0%

Google Adds To BigQuery Big Data Capabilities

Google expands the capabilities of its BigQuery system to allow real-time data stream processing and event analysis.

8 Google Projects To Watch in 2015
8 Google Projects To Watch in 2015
(Click image for larger view and slideshow.)

Google has announced updates to Google BigQuery and Cloud Dataflow -- the search giant's two big data management systems that compete with Amazon Web Services' DynamoDB and Data Pipeline.

In a blog, Google's William Vambenepe, lead product manager for big data on Google's Cloud Platform, claimed Google has implemented a more thorough "cloud way" to managing big data than other IaaS providers. By that Vambenepe means the service is provided without the user needing to know anything about how it's deployed, scaled, or managed, making it a "NoOps" service.

In one update to BigQuery, Google has introduced row-level permissions, a finer-grained approach to granting access to data in a database, according to Vambenepe. With row-level permissions, it's possible to grant a user access to a particular type of data in a database without opening up neighboring data to inspection.

Row-level permissions make it easier to share internal data with a variety of users. Partners or other parties outside the company can be granted permission to access a BigQuery data set in the cloud, but still be restricted to specific rows, Vambenepe wrote in his April 16 blog post. 

[Want to learn more about BigQuery competitors? See MongoDB Eyes Bigger, Faster NoSQL Deployments.]

The default ingestion limit for BigQuery has been raised to 100,000 rows per-second, per-table with unlimited storage for handling large data analysis tasks. BigQuery works with large structured data sets for SQL analytics similar to a relational database system, or with loosely structured data assembled as JSON (JavaScript Object Notation) objects.

(Image: Google)

(Image: Google)

Several NoSQL systems, such as Cassandra and MongoDB, also work with JSON objects.

The Google Cloud Platform also introduced the beta version of a new service, Google Cloud Dataflow. Cloud Dataflow provides event/time-based data stream processing, available as an on-demand service. Stream processing can also be scheduled as a batch service, if the Google Cloud user choses.

A Cloud Dataflow user doesn't need to set up a cluster on which to run the stream-flow processing.

"Just write a program, submit it, and Cloud Dataflow will do the rest," Vambenepe wrote.

Stream processing and event-related processing are done on a data stream, such as a feed of stock trades from an exchange, with the system looking for trades at a particular level of pricing, or at particular time intervals. Stream processing can also be used against an application's server log, where it watches for particular software events in the application and triggers an alert when it spots one.

Google's BigQuery processing and Cloud Dataflow stream analysis are now connected to another service -- Cloud Pub/Sub -- to allow notice of event occurrence to selected IT administrators or business end-users. Vambenepe wrote that Cloud Pub/Sub "completes the platform's end-to-end support for low-latency data processing."

Open source data systems, such as Hadoop, Spark, and Flink's data stream processing capabilities may be used with BigQuery as well, Vambenepe wrote. Google will provide connectors between those systems and its BigQuery and Cloud Storage services.

"Scuba equipment helps humans operate under water," observed Vambenepe, but they're no match for the agility of creatures that belong in the water. "When it comes to big data and the cloud, be a dolphin, not a scuba diver," he concluded.

Attend Interop Las Vegas, the leading independent technology conference and expo series designed to inspire, inform, and connect the world's IT community. In 2015, look for all new programs, networking opportunities, and classes that will help you set your organization’s IT action plan. It happens April 27 to May 1. Register with Discount Code MPOIWK for $200 off Total Access & Conference Passes.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
asksqn
50%
50%
asksqn,
User Rank: Ninja
4/28/2015 | 5:54:05 PM
BigQuery is a big help
For anyone who wants to learn more about Google's BigQuery, it has a web page that is a great resource for an intro.  Just head on over to cloud dot google dot com forward slash bigquery. 
yalanand
50%
50%
yalanand,
User Rank: Ninja
4/21/2015 | 2:49:34 PM
Real time analytics
With so many Android Wear about, its time we had some really nice real tiem data analysis. This is a crucial technology if done right with the right amounts of mix, we may have another cloud computing leader in the making.
Commentary
What Becomes of CFOs During Digital Transformation?
Joao-Pierre S. Ruth, Senior Writer,  2/4/2020
News
Fighting the Coronavirus with Analytics and GIS
Jessica Davis, Senior Editor, Enterprise Apps,  2/3/2020
Slideshows
IT Careers: 10 Job Skills in High Demand This Year
Cynthia Harvey, Freelance Journalist, InformationWeek,  2/3/2020
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
IT 2020: A Look Ahead
Are you ready for the critical changes that will occur in 2020? We've compiled editor insights from the best of our network (Dark Reading, Data Center Knowledge, InformationWeek, ITPro Today and Network Computing) to deliver to you a look at the trends, technologies, and threats that are emerging in the coming year. Download it today!
Slideshows
Flash Poll