The Weather Company: Cloud Journey Requires Cultural Change

Is it possible to transform a company's technology foundation and its culture in just three years? Ask Bryson Koehler, CIO at The Weather Company, and he'll tell you it is indeed possible.

Three years ago, The Weather Company had 13 maxed-out data centers and an aging collection of apps running on a one-of-everything infrastructure. The vision was to hit the reset button and move the entire business into the cloud. That infrastructure shift would let the company scale up data-driven weather prediction and API-based delivery of weather-related content around the globe, through a range of websites, mobile apps, and online services.

The Weather Company is best known as the parent of The Weather Channel, but it's also the company behind Weather Underground, Intellicast site-specific forecasting, and WeatherFX insights for marketers. In all, the company serves hundreds of millions of website visitors, tens of millions of TV viewers, and hundreds of thousands of commercial customers, including airlines, emergency services, shippers, utilities, insurers, media companies, and mobile app providers.

The Weather Company runs on more data than ever.

The first step in The Weather Company's transformation, launched two-and-a-half years ago, was the development of a massive Storage Utility Network (SUN) on Amazon Web Services (AWS). SUN replaced the company's on-premises weather-prediction platform. Where the old network captured 2.2 million weather-condition data points from around the globe four times per hour, the new network was designed to capture 2.2 billion data points 15 times per hour.

All that new data -- some 20 terabytes captured per day -- now supports more accurate weather forecasting around the globe. But SUN is also the company's data platform, called via APIs, that drives Web and mobile applications. Where the prior-generation platform creaked and groaned to sustain peak service loads, SUN scales with cloud elasticity to meet spikes in Web, mobile, and programmatic data calls during hurricanes and winter storms.

A Reboot For Weather.com

"In February there were 40 billion Web searches on Google, which is a huge achievement, but in that same month our data platform served 280 billion weather-related requests," says Koehler, a veteran of InterContinental Hotels Group who took over as The Weather Company's CIO in 2012. "We would never have been able to handle that level of engagement with people around the planet if we still had most of our engineering team worrying about whether a SAN switch in X data center had a successful firmware upgrade over the weekend."

With the SUN project completed, The Weather Company spent much of 2014 on a "Reboot" project that piloted a cloud-based framework and architectural approach to content management. The effort started with weather.com, the online home of the Weather Channel and regularly one of the top-20 websites in terms of traffic. Until last year, the company still served up Web and mobile content using an aging content-management system encompassing 144 servers spread across company data centers.

[ Want more innovation ideas? Read about all the Elite 100 winners. ]

"We had to run those 144 servers whether it was a boring spring afternoon or a hurricane was reaching landfall on the Gulf Coast," says Chris Hill, senior VP of data platform and services. "On most days, those servers weren't very busy, but when severe weather hit we'd all hold hands and pray."

Weather.com's new Drupal-based content-management system runs on just 10 servers hosted on AWS by the third-party service firm Acquia. Between an advanced caching strategy (with Akamai edge servers delivering copies of the site) and an architectural approach that favors client-side rendering (rather than burdensome server-side rendering), the new CMS is barely breaking a sweat, even during big weather events.

"We get 50 million page views on an average day, but since the [November 2014] relaunch we've gone over 80 million without having to touch the system," says Hill. If weather.com anticipates higher traffic levels -- like the 200-million-plus page views per day it experienced during Super Storm Sandy -- all it needs to do is call Acquia so the firm can spin up more AWS capacity.

The Reboot project has also set the stage for growth in consumer uploads and views of weather-related videos. The company served up 1.2 billion video plays across its properties in 2014, and it expects to support 2 billion plays this year.

"Reboot was the culmination of the movement to an API-driven data environment and a movement to a mobile-first strategy whereby we could use the same content and tools for desktop Web, mobile Web, and our [native] mobile applications," Koehler says.

Beware The Worthless SLA

Some will see The Weather Company's new dependence on third-party vendors, particularly on AWS, as its weak link. But Koehler contends that those who prefer the do-it-yourself approach of on-premises, company-run data center infrastructure are paying for a false sense of security in "having control."

"With the transformation to the cloud-based environment, you recognize that none of those things add value to your business," Koehler says. By not using cloud infrastructure, "you're spending all of your time working on table stakes. If you want your business to really move forward, you're going to have to rely on others for core capabilities."

Whereas 60% to 70% of The Weather Company's tech effort went into maintenance and operational support as recently as two years ago, that figure now stands at 20% to 25%, Koehler says. That has freed up the 450-person engineering team to focus on what he describes as "next-generation development."

However, for companies that do make the move to cloud infrastructure, don't count on "worthless" SLAs, Koehler warns. You have to plan and build for the inevitability of third-party providers not being able to deliver on their promises.

The Weather Company is now acutely dependent on Amazon, but in March it struck up a strategic deal with IBM to run a version of its cloud stack -- serving the business-to-business side of the house -- on that vendor's SoftLayer cloud. IBM is making Weather Company data available to cloud customers that want to inject Weather intelligence into marketing, supply chain, and Internet-of-Things-style applications. From the Weather Company's perspective, it gets alternative cloud capacity available so it's not entirely dependent on AWS.

It's not that Koehler doesn't have faith in the redundancy and resilience of multiple AWS availability zones and The Weather Company's own provisions to work around AWS failures. The point is having options and the ability to run the company's stack on other public clouds. 

"I believe that any serious cloud-based business or application needs to be built in a cloud-agnostic way," says Koehler. "That's what has enabled us to deploy our data services platform onto IBM SoftLayer so we can power our business and strategic opportunities with IBM beyond what we could do with AWS alone."

The transformation at The Weather Company has not been without its costs. Koehler admits the company has seen turnover in personnel as the company has changed its approach.

"We've seen a lot of technology change, but the hardest thing to change is your culture," he says. "IT organizations have been staffed, funded, paid, and rewarded to do things in certain ways for decades. You have to unlearn all of that behavior and evolve. To change engineers' mindsets around things that they spent the last 20 years doing and being told was the right way is hard, and it doesn't happen overnight."

That statement is obviously literally true. But taking just three years to transform in the scope and scale that The Weather Company has seems very close to an overnight success.