Big Data Reshapes Weather Channel Predictions - InformationWeek
Data Management // Software Platforms
11:00 AM
Connect Directly
Ransomware: Latest Developments & How to Defend Against Them
Nov 01, 2017
Ransomware is one of the fastest growing types of malware, and new breeds that escalate quickly ar ...Read More>>

Big Data Reshapes Weather Channel Predictions

The Weather Company builds a new forecasting platform using Basho's Riak NoSQL database and Amazon Web Services.

5 Big Wishes For Big Data Deployments
5 Big Wishes For Big Data Deployments
(Click image for larger view and for slideshow.)

"Weather is the original big data application," says Bryson Koehler, executive VP and CIO at the Weather Company. "When mainframes first came about, one of the first applications was a weather forecasting model."

Flash forward to today and the Weather Company ingests some 20 terabytes of data per day to spin out what Keohler bills as the world's most accurate forecasts. To stay ahead of its competition, the Weather Company is in the process of rolling out a new platform built on Basho's Riak NoSQL database and running globally in the Amazon Web Services (AWS) cloud. More than a year in the making, the new platform will bring the company's scale of analysis to a whole new level.

The Weather Company is the parent behind popular brands including the Weather Channel, WeatherFX, Weather Underground, and Intellicast. It serves hundreds of thousands of customers, including 30 airlines, emergency services, shippers, utilities, insurers, media giants, and the developers behind thousands of mobile weather apps. The demand adds up to billions of computer-based data requests per day. Performance expectations are as fast as 10-millisecond latency.

[ Want more on one of the fastest-growing options in information management? Read Why NoSQL Databases Are Gaining Fans. ]

The Weather Company's incumbent platform is more like a loose-knit collection of aging applications running across 13 data centers. Mainframes aren't on the list, but Koeler said the company uses a "one-of-everything" mix of databases, including MySQL, Microsoft SQL Server, Cassandra, MongoDB, and PostgreSQL. This generation of technology, which runs mostly on MySQL, captures 2.2 million current-weather-condition data points from around the globe four times per hour. The company's new consolidated platform, called SUN (Storage Utility Network), will capture 2.25 billion (with a "b") weather data points 15 times per hour.

"As with any large-scale, algorithmic-type modeling, the more data you have, the better the predictions will be," Koehler said, explaining the planned exponential increase in data capacity.

It's a big win for NoSQL technology and, in particular, Basho, which provides Riak Enterprise edition Multi-Datacenter Replication for ultra-high availability. NoSQL won out over relational databases primarily for its scalability, but Riak was chosen for its simplicity and ease of administration at high scale. It won out over Apache Cassandra, the runner-up choice, as well as MongoDB and Hadoop, which also got serious consideration, according to Koehler.

"When you're globally distributing massive amounts of data across Amazon nodes or Google Compute nodes, you want something that's simple to use and configure," said Koehler. "Cassandra, for example, is great at distributing data, but it's complicated and complex to run. Riak was built to handle massive data movement, replication, and data-synchronization on a cloud-based, globally distributed data platform."

The Weather Channel's new platform is also a win for Amazon, which highlighted the Weather Channel story at its Invent event in Las Vegas earlier this month. The SUN system will be deployed across four AWS availability zones: US East, US West, Europe, and Asia.

"We wanted to get away from our 13 data centers and move everything to an infrastructure-as-a-service model," Koehler said.

Amazon is the Weather Company's primary cloud provider, but the firm is also planning to add cloud capacity from Google and other providers.

"Competition in the compute space is important, so we're ensuring that we abstract ourselves from being stuck on any one platform," Koehler said.

The SUN system made its debut in August, powering WeatherFX, the company's advertising targeting engine, which matches ads with weather-driven demand patterns and forecasts. The platform was also rolled out in beta stage for the company's Forecast on-Demand platform. The Weather Underground site will move to the platform by Thanksgiving, according to Koehler, and The Weather Channel is expected to roll out between now and the first quarter of next year.

With the new platform in place, Koehler said, the Weather Company has divided the globe -- land, sea, ice caps, and all -- into more than 30,000 four-square-mile squares. It can accurately report current conditions and offer predicted weather for each square hours, days, or even weeks in advance.

IT groups need data analytics software that's visual and accessible. Vendors are getting the message. Also in the State Of Analytics issue of InformationWeek: SAP CEO envisions a younger, greener, cloudier company. (Free registration required.)

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Ninja
11/30/2013 | 8:51:28 PM
Re: It's working
@Gary_EL: Interesting -- my own observation is that the weather forecasts around Atlanta have been getting progressively worse. In particular the 5-day forecast, which used to be reasonably accurate, has been further and further from reality as time goes on. I don't think that necessarily says that the modeling is getting worse, so much as that we're seeing weather patterns that are far more unpredictable, and we simply don't have the historical data to compare against, so the models can't do such a good job.
User Rank: Ninja
11/30/2013 | 1:42:01 PM
It's working
Whatever they are doing, it is certainly working. I've noticed that in the last year or so, weather predictions have been uncannily accurate. They still do miss now and again, but  whether it's snow in Boston or hurricanes in the Midwest, people who pay close attention to the Weather Company's clients are seldom surprised.
User Rank: Strategist
11/25/2013 | 1:48:14 PM
Basho Riak gets some respect
Basho's Riak is probably one of the most underestimated systems, for the reasons Bryson Koehler suggests: simplicity, ease of administration and multiple data center replication. Not surprising to find Bryson, formerly of InterContinental Hotels, on this pioneering front.
User Rank: Author
11/25/2013 | 11:24:00 AM
Re: The Weather Company still uses one of everything
Sounds like an on-the-job big data master's degree for people who get to work on the Weather Channel's big data team. Quite the diverse tech portfolio. They must be drowning in resumes.
D. Henschen
D. Henschen,
User Rank: Author
11/25/2013 | 11:14:48 AM
The Weather Company still uses one of everything
It's not that Riak is replacing every database at The Weather Company. It still has "one of everything" in use -- including Cassandra, MySQL, MongoDB and more. It's just that Riak is the choice for the new, mega-scale SUN platform. Hadoop is also in use, but for high-scale analytics, not real-time weather data ingest and prediction.
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll