A Better Way To Store Data?

When John Dias met with storage vendors earlier this month, he was counting on seeing progress in long-haul data delivery--Fibre Channel switches that would move data across the state, for example. As a system analyst at Hibernia National Bank in New Orleans, Dias is charged with recommending a disaster-recovery plan to protect the bank against hurricanes that could strike the Big Easy but never reach its Shreveport, La., offices farther north.

Dias was looking for an approach that would leverage some of Hibernia's existing Fibre Channel storage area network and interconnect technologies. The company hoped that by now there would be better direction for deploying future SANs and disaster-recovery systems, Dias says. "Even after visiting the vendors, we're in the same place as before."

But things may improve as IP storage technologies evolve. IP storage already deployed at a handful of companies overcomes the long-distance limitations of Fibre Channel storage, which tops out at 20 kilometers. Most companies already have IP-based Ethernet networks in place, along with people who understand how to maintain and manage them.

So far, though, IP storage isn't mature enough for most companies. Many have invested heavily in Fibre Channel SANs--$5 billion last year, according to International Data Corp. Fibre Channel is the successor to direct-attached storage, which is slower and has even more stringent distance limitations. Until some key roadblocks for IP storage are cleared, businesses aren't likely to buy into the technology.

To start, IT managers generally prefer standards-based products. Right now, IP storage is muddled in a standards mess. Three separate efforts are under way to become the final IP storage standard: the Internet Fibre Channel Protocol (iFCP), the Fibre Channel Internet Protocol (FCIP), and the Internet Small Computer System Interface (iSCSI).

IP storage leader Nishan Systems created iFCP; about 18 months ago, the startup developed a switch, based on that spec, that lets blocks of data move over IP with acceptable performance. Nishan then opened it to the market in search of industry acceptance and a possible standard. IFCP is now a de facto standard, with iSCSI snapping at its heels.

The Internet Engineering Task Force is working on iSCSI, with support from just about every major storage vendor. The problem is that a final version of the standard is at least a year away. The iSCSI protocol appears as the SCSI interface to which all servers are accustomed, but it converts data blocks to network packets to travel over Ethernet and IP. The task force also developed FCIP, the spec least likely to survive. It's been around for a few years and lets users sneak blocks of data through the Ethernet network, but performance is slow.

Even when vendors resolve the standards issue, they'll be challenged to convince large businesses--and themselves--to scrap their huge investments in Fibre Channel SANs and management tools. Where the technology might take off, though, is in small and midsize businesses, which haven't been as eager to fork over big bucks for Fibre Channel. An IP port costs about one-sixth of a Fibre Channel port.

That lower cost is what will drive many businesses to IP storage, says John Young, an analyst at D.H. Brown Associates. He views IP storage and Fibre Channel SANs as complementary. "I'm not seeing IP replacing Fibre Channel as a direction," Young says. "And I don't see Fibre Channel being displaced at the high end."

ProHealth Care Inc. plans to use both Fibre Channel and IP storage technologies to link remote locations and set up a disaster-recovery plan, says Bill Bailey, senior network engineer at the Waukesha, Wis., company. ProHealth owns two hospitals and services two others, as well as 12 health-care clinics, a network of nursing homes and assisted living centers, and two rehabilitation centers. For its SAN, ProHealth considered Compaq, EMC, IBM, and Xiotech; it finally chose Xiotech for the long-term total cost of ownership.

The company implemented a Xiotech SAN at its headquarters. Bailey had to pick the networking product and architecture to extend the Xiotech SAN to a new data center seven miles away that would eventually serve as part of the disaster-recovery architecture. If he stuck with Fibre Channel, the high-speed pipe had to be dark fibre, the base connectivity technology for which the phone company would charge ProHealth $10,000 a month. If the health-care company installed dark fibre itself, Bailey estimates the cost at $750,000. "It would be very expensive and could still get cut off by some backhoe," Bailey says.

Instead, ProHealth will use Nishan's Storage Over IP switch with the Xiotech SANs and pay Time Warner Data Communications about $2,000 a month for the high-speed Ethernet network. "Eventually, we'll replicate our storage from the data center back to one of our hospitals," Bailey says. He expects the OC12-based network to be in place in six weeks. Before the end of summer, ProHealth should have connections from all its locations to the remote Xiotech storage at the new data center and a Nishan switch in place for high-speed disaster-recovery capabilities with one of the hospitals. Without Nishan's switch, that architecture wouldn't be possible.

Other companies are using IP storage alone. E-Office BV in Huis ter Heide, the Netherlands, turned to IBM's IP Storage 200i appliance, because Network Appliance Inc. couldn't support Microsoft Exchange. E-Office is a systems integrator that implements Exchange, Lotus Notes, and IBM WebSphere for clients, and runs Exchange in-house. The company has 24 IBM xSeries 3300s running Windows 2000 Advanced Server, including some that are clustered together, says Jorn Bijnsdorp, an IT consultant who oversees IP storage.

Bijnsdorp considered a SAN but backed off because of cost and complexity. "Our administrators need to be out in the field working with customers as much as possible," he says, meaning he doesn't want them spending too much time working on their two offices' network.

E-Office now has two 200is that store 500 Gbytes of client and internal data; it keeps them in racks or moves them out to customer sites because they can act and look like a local drive, Bijnsdorp says. "There's definitely no bottleneck on the network," he adds.

One of the key problems with IP storage is delivery notification. The TCP/IP stack that ensures delivery of packets uses up a lot of time and CPU space, and that slows down server performance for everyone. What's more, the large data volumes associated with storage never make it across, and they shut the server down in the process. Vendors must figure out some way to deal with the problem until standards are set.

IBM, for example, isolates the TCP/IP stack within the 200i so the multiple commands don't hold up the movement of data at the end points. Ethernet usually requires all those commands to confirm delivery of a file, because so many packets fall off the path during IP transmission. Nishan does the same.

Adaptec Inc. and Emulex Corp. will soon release their host bus adapter cards enabled for the latest version of iSCSI. Servers already connected to the IP network can insert these host bus adapters, eliminating performance and CPU-utilization fears about data blocks moving across Ethernet. Packet-delivery notification comes not from the computer's CPU but from the host bus adapters, which accept server information as SCSI, convert to iSCSI, off-load the TCP/IP stack to themselves for processing, and carry blocks of data over Ethernet to their storage destination.

Adaptec's host bus adapters will be able to serve vendors and value-added resellers in June for Linux and Windows, says Ram Jayam, VP and general manager for Adaptec's storage networking group. Later in the summer, versions for AIX, HP-UX, and Solaris will be available. Volume pricing will be $400 to $600 per card, he says. Once they have these host bus adapters, server vendors may start moving more quickly on IP storage.

Emulex will ship its GN9000/SI host bus adapters to partners this summer, says Gareth Taube, senior VP of IP storage networking. The gigabit speeds on the network, moving data at 80 Mbytes per second, can use 100% of a server's CPU capacity, he says. "The host bus adapter will make 10 Gigabit Ethernet processing possible." Just like Adaptec's, the Emulex host bus adapter will off-load the TCP/IP commands that slow down the CPU and suck up its capacity. "A server could never do all the processing for that fat a pipe," Taube says.

The last challenge facing IP storage is security, a huge concern for Helen Chen, a member of the technical staff at Sandia National Laboratories in Livermore, Calif. The weapons research facility is paranoid about security, she says. At the same time, researchers need to perform and access calculations, each of which can generate hundreds of terabytes of data between Livermore and Albuquerque, N.M.

The high-performance, expensive Fibre Channel SAN based on DataDirect Networks Inc. and Brocade Communications Systems Inc. switches works fine, albeit slowly, she says, and it has demonstrated connectivity with Nishan. But the lab isn't ready to move any real data yet because classified security measures aren't complete. "TCP/IP is prone to security attacks," Chen says. "While we're opening our data up to Ethernet, it's very important we make sure it's protected from Internet attacks."

Despite Sandia's high-performing SAN, some users can see the day when Fibre Channel SANs are obsolete. Hibernia can't wait for iSCSI and will set up its disaster-recovery network using Xiotech and Nishan, similar to ProHealth, by year's end.

"I don't think Fibre Channel is getting developed any further, and I see interoperability issues. It's not meant for any long-haul data delivery," Dias says. "Those things will never be overcome, and I question whether I'll continue with Fibre Channel for the long term."