Scaling The Data Warehouse - InformationWeek
Software // Information Management
04:45 PM

Scaling The Data Warehouse

Data warehouses aren't just exploding in size, they're also supporting more users and increasingly complex queries, all in shorter time frames. Here's how to make sure yours is ready to scale.

LGR Telecommunications has a 310-TB Oracle data warehouse that's used daily by 2,500 people at one of its telecom carrier clients. The warehouse powers an LGR service, called CDRlive, that gives its carrier customers access to call data records. It's updated round the clock, in near-real time, and is available for query 24 hours a day, 365 days a year.

"There are no batch jobs," says Hannes van Rooyen, chief architect at LGR, which supplies data warehouse software and services to the telecom industry. "Instead, as many as 13 billion records a day are added, and an equal number are dropped in an online update process that runs concurrently with user queries."

The data warehouse keeps more than a petabyte of disks spinning and has grown by a factor of 10 during the last four years. It's expected to at least double in the coming year.

InformationWeek Reports

Most companies still don't hold hundreds of terabytes of data, but they're up against the same data warehouse problems that face LGR--soaring data volume, more users, complicated queries, and fast-changing information. Throw in a growing number of vendor options and it's time for companies to re-evaluate their data warehouse strategies.

IT groups rethink the "save everything" approach to data management.
The new generation of data warehouses looks a lot like LGR's: growing at an extraordinary pace, in multiple dimensions, and supporting critical business processes that must react quickly to events around the company. Whether your company has 250 GB or 250 TB of data, you're likely facing the same questions: Do we have the right architecture? Is it on the right platform? Is the warehouse about to run out of headroom? What will it take to service new users? How do we move from batch loading to continuous update? And with technology changing so rapidly, how do we know we're on the right system?

All the answers loop back to managing scalability. Getting control of scalability might mean embracing the highly parallel processing and scale-out architectures long offered by Teradata and IBM and elements of which are now emerging in new products from Oracle and Microsoft (see story, "Microsoft And Oracle Are Scaling Out"). Or it might just require more effective management of existing data warehouse practices, including quantifying requirements, measuring alternative solutions, and acting earlier on potential problems.

1 of 4
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll