Comments
Bad Winter Weather Meets Big Data Prediction
Oldest First  |  Newest First  |  Threaded View
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/10/2014 | 11:44:06 AM
More data brings more accurate predictions
We'll hear more from Koehler on the predictive part of this implementation at the March 31-April InformationWeek Conference in Las Vegas. The SUN platform was an essential staring point because it provides so much more data than was previously available to drive accurate forecasts. More data brings greater accuracy, so stepping up from millions of data points four times per hour to billions of data points 15 times per hour is making a big difference, according to the company.
Lorna Garey
50%
50%
Lorna Garey,
User Rank: Author
3/10/2014 | 2:05:05 PM
Long-term outlook
I'm sure that closing half its data centers will make the cloud vs. in-house TCO balance sheet look pretty darn good for a few years. However, at that data volume, has Koehler done any projections out five or 10 years? 20 TB a day adds up, after all. And cloud providers thus far have not passed Moore's Law savings back to customers.
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/10/2014 | 2:13:52 PM
Re: Long-term outlook
A key metric for the new system is "time to live" (TTL -- as in the time the data has left to live -- not "live" as in, on air). As I understand it, they don't need all that detail forever. The fine-grained detail is for accurate forecasting NOW or today. Once the weather is history, far less data is needed to retain a historical record, so they can delete information they don't need. If you want to know more you can ask Bryson Koehler directly at the upcoming (March 31-April 1) InformationWeek Conference. He's one of two guests on our When Big Data Platforms Make Sense (And When They Don't) panel session.

 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
3/10/2014 | 4:47:33 PM
A treasure trove of weather data
Weather Underground is collecting 20 TBs a day but doesn't need to save it all. At the same time, what a contribution to weather history and understanding worldwide patterns if it did. Granted, Weather Underground is not a non-profit, and doing so might convert it into one. But it's the first time in history we've had that much information in hand, so much we don't know what to do with it -- other than grab short term results and throw the rest away, so to speak.
Bulat
50%
50%
Bulat,
User Rank: Apprentice
3/11/2014 | 1:35:05 PM
Don't!
Why throw away historical data if we want to predict the future weather (this is all the forecast about)?
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/13/2014 | 11:25:12 AM
Re: Don't!
I'm not certain the Weather Company is throwing detail away, but it's not likely it's keeping everything given that the 20-terabyte-a-day capture is within an operational system running on a NoSQL database. This granular data feeds near-term forecasting. I suspect they require less detail for historical trend analysis.

This is another good question I'm going to ask Bryson Koehler during our Big Data panel at the March 31-April 1 InformationWeek Conference.


Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Nov. 10, 2014
Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.