Big Data // Big Data Analytics
Commentary
6/9/2014
09:06 AM
Per Nyberg
Per Nyberg
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

3 Ways Big Data, Supercomputing Change Weather Forecasting

Early, granular, and accurate forecasts could benefit everyone from governments to retailers and healthcare providers.

6 Models Of The Modern Data Center
6 Models Of The Modern Data Center
(Click image for larger view and slideshow.)

"You can't control the weather." It's a phrase we often utter as we plan weddings, family picnics, and beach vacations. While it's true that we can't control the weather, scientists and technologists are working hard to improve our ability to predict it.

What that means for enterprise IT isn't just the ability to foresee disasters that could wipe out a data center. Granular and early forecasts can create business opportunities -- for example, alerting a brewery to supply distributors and points-of-sale greater inventory in anticipation of an unusually warm spring weekend in the Northeast. Or suppose a blizzard is due to hit Atlanta on Black Friday -- with enough notice, retailers could adjust their plans.

Beyond the burgeoning data services industry, weather has massive economic and safety implications. Weather Analytics, a company that provides climate data, estimates that weather impacts more than 33% of worldwide GDP, affecting the agriculture, tourism, fishing, recreation, and airline industries, to name just a few.

Uncertain weather conditions also impact small business owners, such as the local painter who can't complete his job on time when foul weather persists. In addition, public safety is of vital concern when officials aim to understand the impact of extreme weather events such as hurricanes, tsunamis, or wildfires. Costs associated with extreme weather across the world totaled more $125 billion in 2013, and the frequency of these events is on the rise.

Beyond the private sector, without specific data about a forthcoming event, governments may waste money casting a net that's too wide. For example, every additional mile of evacuation associated with an impending storm can result in millions of dollars in lost revenue, wages, and relocation expenses. And hospitals and emergency facilities that anticipate a severe storm that could shut down power could stock up on extra fuel for generators.

So while we're not able to control the weather, better forecasting will allow us to make more informed plans that can limit financial losses, provide new business opportunities, reduce government spending, and even save lives.

Unfortunately, improving our ability to predict the weather is challenging, both scientifically and computationally. Supercomputing has played a major role in enabling predictive models since the 1950s and remains at the cornerstone of today's weather and climate modeling. Constantly improving computational capabilities have allowed scientists and forecasters to produce results faster than ever while also investigating increasingly complex phenomena and producing specialized forecast products. From model performance to system and data management, weather prediction presents unique high-performance computing challenges.

[What percentage of your data center budget will be spent on maintenance vs. innovation in 2014? How about 2015? Tell us and enter to win a 32-GB Kindle Fire HDX.] 

Supercomputing, along with big data, can meet the future demands of weather forecasting in three key areas:

1. Managing and utilizing enormous data sets: The volume and diversity of environmental data is increasing exponentially, placing great demand on the infrastructure to transport, manage, and store this data, and requiring ever-greater computational power for simulations that use it. This creates new opportunities for specialized services, developed with researchers in public and private institutions. One example is leveraging new sources of observation, such as sensors placed on automobiles. Imagine thousands of sensors in an urban area providing real-time meteorological information. Models are also evolving to analyze this tsunami of data and augment traditional physics-based simulations.

2. Increasing model resolution: Higher-resolution models are a critical element to better estimate the long-term state of climate systems and to improve weather forecasting, particularly for severe weather events. Recent simulations of Hurricane Sandy by researchers at the National Center for Atmospheric Research and the University of Illinois using Blue Waters supercomputers have zeroed in to a 500-meter resolution -- the equivalent of a few city blocks.

3. Addressing technology hurdles: As weather modeling and analytics become more data-intensive and computationally demanding, researchers must watch for performance bottlenecks such as memory, I/O, and interconnect latencies and bandwidths. Weather simulation requires thousands of microprocessors to run in parallel, pushing hardware and software to its scalability limits. In addition, scalable operating systems, compilers, and application libraries play an essential role in achieving sustained performance. Ultimately, the underlying technology infrastructure must be tightly integrated to support simulation and analytics workflows.

Infrastructures offering simulation and data-driven analytics capabilities to support routine execution of high-resolution forecasts will combine with advanced research to promote a whole new array of specialized meteorological services for public and private sectors. The future of weather forecasting requires capabilities we couldn't even conceive of when we began predicting the weather 64 years ago. Supercomputing innovation has so far kept pace with the demands of the community, and it is poised to offer new solutions in the years to come.

Our InformationWeek Elite 100 issue -- our 26th ranking of technology innovators -- shines a spotlight on businesses that are succeeding because of their digital strategies. We take a close at look at the top five companies in this year's ranking and the eight winners of our Business Innovation awards, and offer 20 great ideas that you can use in your company. We also provide a ranked list of our Elite 100 innovators. Read our InformationWeek Elite 100 issue today.

Per Nyberg is Senior Director of Business Development at Cray, responsible for worldwide strategic planning, market development, and solutions development across Cray's vertical segments. For the last decade, he was focused on the application of supercomputers in Earth ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Lorna Garey
50%
50%
Lorna Garey,
User Rank: Author
6/10/2014 | 4:22:46 PM
Re: Advice for now
I am an editor for IW - so, my interest is purely theoretical at this point! However, given climate change, I can definitely see the value of being able to predict severe weather with greater accuracy.
Per Nyberg
IW Pick
100%
0%
Per Nyberg,
User Rank: Apprentice
6/10/2014 | 2:09:31 PM
Re: Advice for now
Hi Lorna, Good question.   I think it depends on a number of factors including the level of in-house expertise, business area, complexity of your business model and geographic location of your business.    As this is a growing industry, the range of services and service providers varies greatly by vertical industry and location.   A good place to start is always with your national meteorological and hydrological service, along with any state and local government agencies.   Many government organizations have business outreach programs or relationships with organizations in the private sector that can work with business.   For example, NOAA in the U.S. has such information on their Industrial Meteorology page (http://www.nws.noaa.gov/im/).     There are some good conferences that focus on technology and services aspects.  InterMET Asia was recently held in Singapore and the Meteorological Technology World Expo will be held in Brussels, 21-23 October.    Cray participated and presented on big data at InterMET and we will also participate at Met Expo.
Weather Analytics
50%
50%
Weather Analytics,
User Rank: Apprentice
6/10/2014 | 1:26:06 PM
Re: Advice for now
Hey Lorna,

We see lots of different applications and approaches to accessing and using weather data.  The benefit of a structured and scrubbed database is that no extended services or customized product tools are necessary.  Through some of our reporting tools and web-based map platforms, it's easy to get data across the globe and throughout 34 years of history (as well as current and forecast conditions).  Secondly, in relation to distribution modeling, pulling data through an API makes running historic analysis and live predicitive analytics much faster and easier - and can be done on an in-house platform or product.

What kind of projects are you working on?

Cheers!
Gary_EL
50%
50%
Gary_EL,
User Rank: Ninja
6/9/2014 | 1:06:08 PM
Weather predicting is getting better and better
I can remember, not all that long ago, when I never biothered to listen to weather forecasts because they were usually wrong. Not so any more. Has this process of improvement reached it's limit, or can we expect continued improvement?
Lorna Garey
50%
50%
Lorna Garey,
User Rank: Author
6/9/2014 | 11:25:32 AM
Advice for now
Per, Say a company wanted to start tapping into advanced predictive weather info now to adjust its distribution models. What's the first step? Are specialized service providers the only route for most companies, or could an advanced IT team implement in-house using APIs?
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - September 10, 2014
A high-scale relational database? NoSQL database? Hadoop? Event-processing technology? When it comes to big data, one size doesn't fit all. Here's how to decide.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.