Demand for storage is soaring, but customers want vendors to deliver systems that are more scalable, easier to use, and less expensive
Northwest Airlines Inc. has 13 terabytes of data stored across 270 IBM and Sun Unix servers and 170 Windows servers from a mix of vendors. Its storage architecture comprises four systems made up of products from six vendors, and each system has its own management tool. What took $3 million to build now costs as much as six times that each year to maintain and requires the full-time attention of at least two administrators. And the airline's need for storage continues to double every year. No wonder Randy Pool's main goal for the near future is clear. "We need simplification," says Pool, a senior consultant at Northwest who oversees its storage systems.
Northwest is typical of businesses that need ever more storage space and want it simpler and less expensive. The need comes from the increasing sources of customer and supplier data, including the Internet, as well as business-continuity initiatives that call for the continued backup of most, if not all, of that electronic information. Corporate data stores have doubled every year in the last six years. The fallout: more intricate storage infrastructures that require expensive, manual maintenance. So IT managers are hunting for flexible systems that include centralized data-management for any mix of vendors' products that support business-continuity plans and that don't break the bank with maintenance costs.
The big names in storage--Compaq, EMC, Hitachi Data Systems, and IBM--are addressing those challenges head-on, because of demanding customers and upstart vendors looking to pick off niche areas of the market that the big system vendors have up to now neglected. Overall, the future looks good for storage customers, Yankee Group analyst Jamie Gruener says. Companies can add capacity as needed, buy less-expensive storage appliances rather than shell out millions for massive systems, use resource management to track storage consumption, and even charge back to internal departments for usage. An industry that started out decades ago with hard-disk storage systems attached to mainframes is now a $32 billion market with such high-tech systems as controller-based storage networks with management capabilities built in, switches that sit in the middle of networks and direct information flows, and low-cost storage appliances that attach to existing IP networks and offer capabilities such as data replication that until recently were available only on more costly controller-based systems.
"We'll see a lot of effort and aggressive behavior by vendors for storage between $30,000 and $100,000," Gruener predicts.
What's been lost amid all the technical advances is simplicity. There's no single system available to centrally manage an installation such as Northwest's, though Pool's considering a combination of two alternatives from EMC: WideSky, which lets companies manage from a single console a variety of databases, file and storage systems, Fibre Channel switches, tape libraries, and more; and Auto IS, a software tool that lets EMC Symmetrix hardware customers manage Symmetrix systems alongside those from other vendors. Together, the software could cut the time Northwest administrators spend managing the architecture and free up time for developing next-generation storage that's more responsive to business needs and costs less to maintain.
But the WideSky/Auto IS combination is a less-than-perfect solution because it locks the Minneapolis airline into EMC's management products. That's because Symmetrix, one of the airline's storage platforms, is a closed system that doesn't work well with other vendors' management software. Vendors won't offer tools that can manage all vendors' products until mid-2003 or 2004, analysts say. Until then, maintenance costs likely will remain high, even though overall storage costs are dropping because of falling hardware prices and a growing list of midrange systems (see The People Factor Adds Cost To Storage, p. 40).
Cutting costs is crucial for Northwest, which reported a net loss of $423 million last year. If the airline had one tool to control and administer its mix of storage systems, it could cut overhead and upgrades. Training costs would fall because IT staffers wouldn't need to learn how to use multiple products. "If someone comes up with one tool, we see a definite advantage," Pool says.
EMC is angling to be that company. With 25% of the market, more than twice that of any other supplier, EMC's product suite ranges from high-end systems that cost millions of dollars to storage appliances priced around $30,000. And though EMC is seen as a hard-nosed vendor that locks customers into its proprietary technology, WideSky, which has been available since last fall, suggests the vendor is softening that approach to keep up with a market that's less tolerant of inflexible systems. Last week, it unleashed a WideSky storage resource-management interface that lets customers manage a variety of databases and file systems alongside EMC storage. Partners include BMC, Computer Associates, Microsoft, and Oracle.
EMC got into storage management in 1995 and continues to increase its support of non-EMC systems and devices. In the last two years, EMC also has built a strong lineup of network-attached storage devices that includes the Clariion IP 4700 appliance and Celerra HighRoad software that integrates NAS and storage area networks, making it easier to use both Fibre Channel and IP networks.
Will the market leader become the king of cooperation? EMC's $7.09 billion in sales last year was 20% less than in 2000, and it lost $507.7 million, compared with a $1.78 billion profit in 2000. President and CEO Joe Tucci is confident that EMC's broad range of hardware and software will keep it in the lead. "I don't think that software is ever all the way there. ... We continue with several releases a year to add functionality," Tucci says. Skeptics note that EMC is taking steps to let its software work with other vendors' hardware, but the reverse isn't true, because EMC isn't expected to publish application programming interfaces anytime soon that would let competitors manage EMC systems.
Of all the leading suppliers, Compaq plays best with others. That's in part because of Compaq's 1998 acquisition of Digital Equipment Corp., which was admired for its products' ability to connect with competitors' products. For the most part, Compaq has continued this legacy. Its three main storage products underscore Compaq's efforts to work with other vendors' products. SANworks Management Appliance provides a centralized point for managing and monitoring SAN elements, including switches and storage arrays. When used with the SANworks Network View and Storage Allocation Reporter applications, the SANworks Management Appliance also lets customers monitor storage allocation for SANs from multiple vendors. SANworks Network View automatically monitors heterogeneous devices and their Fibre Channel interconnects. SANworks Data Replication Manager provides Fibre Channel controller-based data replication each time a transaction occurs at a local, regional, or global site. The software supports multiple operating systems, including HP-UX, IBM AIX, Microsoft Windows 2000/NT, Novell NetWare, Sun Solaris, Compaq OpenVMS, and Compaq Tru64 Unix. In the event of a disaster at a primary site, data is safe on the remote storage system and available for immediate connection to an alternate host. Storage administrators reconnect the remote storage system, and the business is back in operation.
Though Compaq is better known for its midrange and low-end storage systems, where it's the market leader, it has introduced products in the past few years aimed at EMC's high-end business. Last October, Compaq unveiled the Enterprise Virtual Array, which can store as much as 17 terabytes of data, move data very quickly, and is flexible enough to let companies initiate changes to storage requirements without having to shut down their systems. It uses a process called virtualization to alter system parameters automatically, so that when capacity levels reach a threshold, such as the number of hits to a Web site, the system moves data among disk drives to assign more capacity to the Web servers that support the busy site. A gap in its high-end offering is that--unlike EMC, Hitachi Data Systems, and IBM, which all offer storage systems that can process mainframe data--Compaq doesn't support mainframes at all.
More than any other vendor, Compaq sees storage hardware as a commodity, so it's betting its storage future on software such as SANworks Network View and the StorageWorks Enterprise Virtual Array. These lay the foundation as the next generation of the StorageWorks Disk Array systems, offering self-tuning, high-capacity, and high-availability virtualized storage. These systems are designed for the data center where there's a critical need for improved storage utilization and scalability.
Three software vendors say they'll soon have management software that will help companies like Northwest. CA is expected to unveil BrightStor Portal next month and says the new offering will be the start of a single view to all storage utilities on a network. It will incorporate CA's backup and recovery, storage resource-management, and SAN-management tools, and integrate management functionality from partners, including Brocade Communications, EMC, Hitachi Data Systems, and Network Appliance. CA says it will take three years to fill out the suite of products that it begins shipping next month. Veritas Software Corp. and newcomer Commvault Systems Inc. also are developing centralized management software. Rather than cobble together a new system from multiple products, Commvault's strategy is to develop a single code base, a single database, and a single policy engine.
There are also two startups that have garnered attention for their storage-management approaches. FalconStor Software Inc.'s software connects storage systems over an IP network so administrators can manage utilization and other parameters centrally (see A Startup Built On Allegiance, p. 50). TrueSAN Networks Inc. is developing software that can manage heterogeneous storage networks (see story online, informationweek.com/881/truesan.htm).
The Weather Channel needs a flexible architecture that's reliable enough to provide new services without taking the data down, VP Hamilton says.
The Weather Channel Inc. in Atlanta shows how data stores continue to grow, even as the economy slowed. Launched in 1982, the Weather Channel wants to move beyond television to reach customers via their PDAs, cell phones, pagers, and computers. It already hosts Weather.com, which averages 350 million page views and 12 million unique users per month. The Weather Channel uses Hitachi Data Systems' Lightning 9960, which can store up to 88 terabytes of data, to house its 12 terabytes. But its effort to reach more customers by offering personalized weather reports to a variety of devices could increase its data stores to as much as 18 terabytes in 16 months. That's because each report sent to a customer creates a unique data set that must be stored. And the data it sends to customers changes like, well, the weather. "We must have a flexible, scalable, reliable architecture to provide these new services and not ever take this data down," says Vicki Hamilton, VP of shared services and operations at the Weather Channel. "The shelf life of our data is short, but storage has to hold large quantities of data that's changed frequently."
Though EMC is the storage capacity leader, Hitachi Data Systems is the primary reason EMC has been forced to cut prices by almost two-thirds during the last couple of years. The wholly owned subsidiary of Hitachi Ltd. gave up the mainframe business two years ago to focus on storage. "We've come out of nowhere to be a thorn in Joe Tucci's side," says Christine Wallis, VP of global strategy and planning at Hitachi Data Systems.
The HDS Lightning 9900 series leads Hitachi Data Systems' storage lineup and sets itself apart from EMC in part because Hitachi shares APIs so other vendors' products can manage the Lightning. Though the Weather Channel won't max out on the current model's capacity for years, Hitachi Data Systems is trying to stay a step ahead of customers. The next-generation Lightning, due this summer, will provide capacity that could approach 170 terabytes in a single frame.
Hitachi Data Systems engineers also are working on a new storage architecture, one for which the network is everything. On that as-yet-unnamed system, the intelligence moves from a central controller to the network itself, Wallis says. Network-centric storage is likely the next big thing; researchers at government and commercial labs are developing similar architectures (see Delivering A Network-Utility Vision, p. 46). Though there's no time frame for the availability of Hitachi Data Systems' new architecture, Wallis indicates it could be ready early next year.
Hitachi Data Systems is making strides in enterprise storage management as well. In February, it rolled out a new version of its HiCommand management software that it says lets storage administrators manage multiple vendors' systems and also monitor and analyze resource utilization from a single console. But Yankee Group's Gruener has seen no evidence of such heterogeneous management yet and doesn't expect it until the end of next year. About 40 independent software vendors are writing to the HiCommand APIs, including BMC, CA, InterSAN, Microsoft, Sun, and Veritas.
Another factor behind storage growth is that since Sept. 11, businesses have taken a more-aggressive stance toward business continuity. Computer crashes or simple human errors still cause most system disruptions, but the attacks cast a new light on just how extensive disaster-recovery efforts need to be. The renewed focus means companies are spending more money on backup and recovery systems that can replicate data in real or near real-time to storage systems and servers housed at other locales. Not only are businesses backing up data, they're also replicating applications.
Some businesses build business-continuity systems using network-attached storage. Lockheed Martin Corp.'s Space Systems Astronautics Operations in Denver relies on network-attached storage systems (often called "filers") from Network Appliance that feature technology, known as snap mirroring, to replicate data and copy it to another site. To initiate a backup, users simply drag and drop files into a folder on their screens, and the files are automatically routed to the filers.
But the system isn't perfect, says Bruce Ferguson, IT operations manager at the division of the Lockheed Martin Space Systems Co. A filer can hold as much as 20 terabytes of data, and it takes up to a week to replicate all that data to another filer. When appliances store that many files, performance is bound to suffer. If a filer goes down before replication completes, Ferguson can deploy snap mirroring to protect the data. Ferguson says he's searching for a more time-sensitive solution.
One company already replicating data in near real time between sites is pushing its vendor, IBM, for even faster replication speeds. Pershing, a Credit Suisse Boston company in Jersey City, N.J., that provides financial firms with brokerage and technical services, uses IBM servers, storage, and software to have just a nine-second delay on the mainframe when data's transferred from one site to another. "We're hoping that with advances, we'll have two live sites, with data [existing at both sites] at the same time," says Tom Towland, Pershing's VP and manager of hardware architecture and planning. In the next 18 months, IBM claims it will add the necessary features to meet Pershing's plan, Towland says. For example, by the summer, IBM says it will increase the distance over which data can be copied from one site to another (currently only 20 kilometers) by sending data asynchronously.
IBM may trail EMC when it comes to storage sales, but the vendor has a lot of pull in disaster recovery, because of its vast research unit. Its research efforts are paying off with what IBM calls "autonomic" intelligence, a method of self-regulation that IBM likens to the human heart's ability to automatically beat faster when the body needs more oxygen. In October, IBM launched the first commercially available product, E-business Management Services, that incorporates the technology.
The new product is designed to keep critical business operations running at peak performance. The product includes a "virtual cockpit" in which senior executives can stipulate performance levels for processes such as the speed at which transactions are processed. Behind the scenes, IBM's Intelligent Device Discovery software manages infrastructure resources to ensure those processes function at the predetermined level, even if a component or the entire site fails.
For IBM, business continuity isn't just a matter of storage and remote-copy software. Nearly every customer using IBM's core storage product, the Enterprise Storage System, is enlisting the help of IBM Global Services and using IBM's prepackaged disk, tape, and software systems, as well as systems tailored for specific vertical markets.
IBM's Enterprise Storage System is priced about one-third less than EMC's high-end Symmetrix, but that's changing. Thanks in part to competition from IBM and Hitachi Data Systems, EMC is starting to drop its Symmetrix pricing. But the Enterprise Storage System still must prove it can handle very high volumes of storage data without expensive integration and configuration help from IBM Global Services. IBM continues to add new software functionality, which will come out of the newly formed storage software group headed by former Lotus executive Mike Zisman. Included in the group's work is the completion by year's end of IBM's Storage Tank, a storage area network-based file system that's intended to let administrators manage a SAN as easily as if they were viewing an E-mail message.
During the next two years, IBM will update Enterprise Storage System, the midrange FAStT storage server (which doesn't hold the same amount of data as its bigger sibling but can move data faster than 2 Gbps), and IBM's Total Storage NAS.
The four titans of storage are all revamping their product portfolios to meet customer demands and grab market share. Their development efforts center on management software that reduces complexity and covers a variety of vendors' systems and storage media, faster and more powerful systems that can accommodate growing data stores, and storage systems that are more flexible and affordable. That syncs up with customers' wish lists. But it appears clear that, at least for a while, the desire for far easier storage will remain wishful thinking.
Big Four Of Storage
Hitachi Data Systems
2001 storage sales
2001 storage sales
2001 storage sales
2001 storage sales
$1 billion to $1.50 billion (estimated)
Compaq offers open APIs and is focused on software
that will let customers choose best-of-breed components for storage
The clear market leader, EMC got IT executives
to bet their businesses on storage. It offers a wide-ranging product
lineup, and it has been cutting prices recently in response to lower-priced
HDS offers state-of-the-art flexible hardware.
It has deep pockets in Japan, as well as data-center experience, and
IBM does it all, including infrastructure outside
of storage, so it's willing to cut prices. Global Services is on call
Compaq will work with Hewlett-Packard, its merger
partner, to combine top products. HP resells Hitachi Data Systems'
storage systems, but may stop if the merger goes through. It will
leave mainframe storage opportunities to the other three vendors as
it works with HP on the networked storage approach. Thanks to midrange
storage prowess, it could gain the high ground in the purely networked
EMC's Symmetrix has lots of competition, but it's
still on top. Al- though the architecture is one of the oldest and
least flexible of the leading vendors, EMC is changing that. IT executives
should pay attention to the midrange Clariion that EMC is prepping
for the networked, commodity future of storage.
The company stands toe-to-toe with market leader
EMC in technology and capacity but publishes its APIs. It's building
network-centric storage systems in which the intelligence is distributed
across the network rather than in a central controller.
IBM is pushing hard to be the top storage vendor.
It will practically give away storage to win some accounts and counts
on follow-up services to recoup any losses. The vendor focuses on
business continuity that handles long distances.
How Enterprises Are Attacking the IT Security EnterpriseTo learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Infographic: The State of DevOps in 2017Is DevOps helping organizations reduce costs and time-to-market for software releases? What's getting in the way of DevOps adoption? Find out in this InformationWeek and Interop ITX infographic on the state of DevOps in 2017.
IT Strategies to Conquer the CloudChances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.