The Weather Company: Cloud Journey Requires Cultural Change - InformationWeek
IoT
IoT
IT Leadership // CIO Insights & Innovation
News
4/28/2015
11:51 PM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
100%
0%

The Weather Company: Cloud Journey Requires Cultural Change

The Weather Company, the No. 5 company in the InformationWeek Elite 100, embraced cloud infrastructure in order to meet dramatic swings in demand for weather information.

Is it possible to transform a company's technology foundation and its culture in just three years? Ask Bryson Koehler, CIO at The Weather Company, and he'll tell you it is indeed possible.

Three years ago, The Weather Company had 13 maxed-out data centers and an aging collection of apps running on a one-of-everything infrastructure. The vision was to hit the reset button and move the entire business into the cloud. That infrastructure shift would let the company scale up data-driven weather prediction and API-based delivery of weather-related content around the globe, through a range of websites, mobile apps, and online services.

The Weather Company is best known as the parent of The Weather Channel, but it's also the company behind Weather Underground, Intellicast site-specific forecasting, and WeatherFX insights for marketers. In all, the company serves hundreds of millions of website visitors, tens of millions of TV viewers, and hundreds of thousands of commercial customers, including airlines, emergency services, shippers, utilities, insurers, media companies, and mobile app providers.

The Weather Company runs on more data than ever.

(Image: The Weather Company)

The Weather Company runs on more data than ever.

(Image: The Weather Company)

The first step in The Weather Company's transformation, launched two-and-a-half years ago, was the development of a massive Storage Utility Network (SUN) on Amazon Web Services (AWS). SUN replaced the company's on-premises weather-prediction platform. Where the old network captured 2.2 million weather-condition data points from around the globe four times per hour, the new network was designed to capture 2.2 billion data points 15 times per hour.

All that new data -- some 20 terabytes captured per day -- now supports more accurate weather forecasting around the globe. But SUN is also the company's data platform, called via APIs, that drives Web and mobile applications. Where the prior-generation platform creaked and groaned to sustain peak service loads, SUN scales with cloud elasticity to meet spikes in Web, mobile, and programmatic data calls during hurricanes and winter storms.

A Reboot For Weather.com

"In February there were 40 billion Web searches on Google, which is a huge achievement, but in that same month our data platform served 280 billion weather-related requests," says Koehler, a veteran of InterContinental Hotels Group who took over as The Weather Company's CIO in 2012. "We would never have been able to handle that level of engagement with people around the planet if we still had most of our engineering team worrying about whether a SAN switch in X data center had a successful firmware upgrade over the weekend."

The Weather Company CIO Bryson Koehler

(Image: The Weather Company)

The Weather Company CIO Bryson Koehler

(Image: The Weather Company)

With the SUN project completed, The Weather Company spent much of 2014 on a "Reboot" project that piloted a cloud-based framework and architectural approach to content management. The effort started with weather.com, the online home of the Weather Channel and regularly one of the top-20 websites in terms of traffic. Until last year, the company still served up Web and mobile content using an aging content-management system encompassing 144 servers spread across company data centers.

[ Want more innovation ideas? Read about all the Elite 100 winners. ]

"We had to run those 144 servers whether it was a boring spring afternoon or a hurricane was reaching landfall on the Gulf Coast," says Chris Hill, senior VP of data platform and services. "On most days, those servers weren't very busy, but when severe weather hit we'd all hold hands and pray."

Weather.com's new Drupal-based content-management system runs on just 10 servers hosted on AWS by the third-party service firm Acquia. Between an advanced caching strategy (with Akamai edge servers delivering copies of the site) and an architectural approach that favors client-side rendering (rather than burdensome server-side rendering), the new CMS is barely breaking a sweat, even during big weather events.

"We get 50 million page views on an average day, but since the [November 2014] relaunch we've gone over 80 million without having to touch the system," says Hill. If weather.com anticipates higher traffic levels -- like the 200-million-plus page views per day it experienced during Super Storm Sandy -- all it needs to do is call Acquia so the firm can spin up more AWS capacity.

The Reboot project has also set the stage for growth in consumer uploads and views of weather-related videos. The company served up 1.2 billion video plays across its properties in 2014, and it expects to support 2 billion plays this year.

"Reboot was the culmination of the movement to an API-driven data environment and a movement to a mobile-first strategy whereby we could use the same content and tools for desktop Web, mobile Web, and our [native] mobile applications," Koehler says.

Beware The Worthless SLA

Some will see The Weather Company's new dependence on third-party vendors, particularly on AWS, as its weak link. But Koehler contends that those who prefer the do-it-yourself approach of on-premises, company-run data center infrastructure are paying for a false sense of security in "having control."

"With the transformation to the cloud-based environment, you recognize that none of those things add value to your business," Koehler says. By not using cloud infrastructure, "you're spending all of your time working on table stakes. If you want your business to really move forward, you're going to have to rely on others for core capabilities."

Whereas 60% to 70% of The Weather Company's tech effort went into maintenance and operational support as recently as two years ago, that figure now stands at 20% to 25%, Koehler says. That has freed up the 450-person engineering team to focus on what he describes as "next-generation development."

However, for companies that do make the move to cloud infrastructure, don't count on "worthless" SLAs, Koehler warns. You have to plan and build for the inevitability of third-party providers not being able to deliver on their promises.

The Weather Company is now acutely dependent on Amazon, but in March it struck up a strategic deal with IBM to run a version of its cloud stack -- serving the business-to-business side of the house -- on that vendor's SoftLayer cloud. IBM is making Weather Company data available to cloud customers that want to inject Weather intelligence into marketing, supply chain, and Internet-of-Things-style applications. From the Weather Company's perspective, it gets alternative cloud capacity available so it's not entirely dependent on AWS.

It's not that Koehler doesn't have faith in the redundancy and resilience of multiple AWS availability zones and The Weather Company's own provisions to work around AWS failures. The point is having options and the ability to run the company's stack on other public clouds. 

"I believe that any serious cloud-based business or application needs to be built in a cloud-agnostic way," says Koehler. "That's what has enabled us to deploy our data services platform onto IBM SoftLayer so we can power our business and strategic opportunities with IBM beyond what we could do with AWS alone."

The transformation at The Weather Company has not been without its costs. Koehler admits the company has seen turnover in personnel as the company has changed its approach.

"We've seen a lot of technology change, but the hardest thing to change is your culture," he says. "IT organizations have been staffed, funded, paid, and rewarded to do things in certain ways for decades. You have to unlearn all of that behavior and evolve. To change engineers' mindsets around things that they spent the last 20 years doing and being told was the right way is hard, and it doesn't happen overnight."

That statement is obviously literally true. But taking just three years to transform in the scope and scale that The Weather Company has seems very close to an overnight success.

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
batye
50%
50%
batye,
User Rank: Ninja
5/5/2015 | 10:48:42 AM
Re: Weather in the Cloud
@SaneIT, yes, you are right, but in Canada our tech is very outdated and most of the time I see rip and replace....
SaneIT
50%
50%
SaneIT,
User Rank: Ninja
5/5/2015 | 8:05:19 AM
Re: Weather in the Cloud
Reading the article it doesn't sound like a rip and replace, it sounds like they had a solid migration.  In the end they did migrate the public facing tools away from their datacenters but I don't see much about the internal tools so I'm assuming those datacenters are still there and operating at some capacity.   That means that a good portion of their IT staff stayed on and they still have the cost of running those datacenters.  The move to AWS with their public facing sites sounds like the right move but it would still be interesting to hear more of the story. 
batye
50%
50%
batye,
User Rank: Ninja
5/5/2015 | 1:43:11 AM
Re: Weather in the Cloud
@SaneIT, it I would say would be a best example... and I think it gonna be a way of the future for all data...
batye
50%
50%
batye,
User Rank: Ninja
5/5/2015 | 1:34:13 AM
Re: The Weather Company
@zerox203 I trust you do have solid point... as technology grow just too rapidly... and no other way around...
batye
50%
50%
batye,
User Rank: Ninja
5/5/2015 | 1:33:00 AM
Re: Weather in the Cloud
@SaneIT yes, but sometimes you do have no choice as only to Rip and replace everything and start from begining...
batye
50%
50%
batye,
User Rank: Ninja
5/5/2015 | 1:31:45 AM
Re: Weather in the Cloud
@Li Tan, interesting point... I did see when the projects get scraped as Co. do run out of money... and I think same case could be here... but time will tell...
SaneIT
50%
50%
SaneIT,
User Rank: Ninja
5/4/2015 | 9:24:54 AM
Re: Weather in the Cloud
I agree, from scratch would have been very tough to do at this scale, but this isn't a new company this is a group that has had some form or web based weather reporting since the infancy of the internet.  I'm sure they had plenty of time to build up to their previous normal and that there were quite a few things glued together just to keep up.  I'd love to see how their server and support environment changed when they moved to AWS instead of trying to serve this from in house datacenters. 
Li Tan
50%
50%
Li Tan,
User Rank: Ninja
5/4/2015 | 9:18:47 AM
Re: Weather in the Cloud
By intuition I would say they should try to leverage the capability of AWS as much as possible. Building this big giant from scratch requires too much effort and it may not be a good choice for business.
zerox203
100%
0%
zerox203,
User Rank: Ninja
5/1/2015 | 2:59:41 AM
Re: The Weather Company
Doubling website traffic, collecting ten times as much data,  serving more customers, and doing it all with increased realiability, uptime, and fewer human resources is certainly no small feat - indeed, let's not forget that they maintained all their services (not to mention their television broadcasts) during this time. I'm a frequent user of their site, and I certainly couldn't have even told you a disruption was happening. An increase in the number of datapoints probably means more for a weather service organization than it does for most... to be able to keep that service consistently, that's a lot of value right there.

Nevertheless, around the halfway mark, I found myself wondering, having freed up all those resources from day-to-day operations, how did they use them (and the new platform) to innovate ? I was happy to see that, yes, they did take it as an opportunity to roll out new services and open up new partnerships. I actually took a quick glance at the SoftLayer site, because it's name I'm actually not particularly familiar with, and their partnership with The Weather Company is listed right there on their front page, offering integration to other customers. Strategic partnership indeed. I was debating which of the top 100 placer's articles I wanted to read - having finished, I'm glad I chose this one.
SaneIT
100%
0%
SaneIT,
User Rank: Ninja
4/29/2015 | 8:21:13 AM
Weather in the Cloud
"here the old network captured 2.2 million weather-condition data points from around the globe four times per hour, the new network was designed to capture 2.2 billion data points 15 times per hour."

This astonishing and a great example of why a move to the cloud can make sense especially when data collection is key.  I can't imagine what it would be like to spin up enough additional capacity to do this when they were already feeling like their datacenters were maxed out.  It was also interesting to hear that they cut from 144 client facing servers down to 10 and they're serving more clients than ever in this process.   One thing that would be nice to see though would be the cost of building this all in house vs pushing it out to AWS. 

 
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
Digital Transformation Myths & Truths
Transformation is on every IT organization's to-do list, but effectively transforming IT means a major shift in technology as well as business models and culture. In this IT Trend Report, we examine some of the misconceptions of digital transformation and look at steps you can take to succeed technically and culturally.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll