October 2, 2000

http://www.informationweek.com/806/san.htm

Storage Area Networks
Everything In Its Own Place

Storage area networks are effective storage solutions, but making them work seamlessly requires planning and hard work. Interoperability suffers for throughput, and managing them isn't for the weak of heart. System integrators can expect to do well for years.

By Logan Harbaugh

-business applications require a trio of factors in great abundance: high performance, high availability, and scalability. Business partners, suppliers, and customers often demand near-continuous access to business data.

This means that data-storage systems must be designed to maximize data storage and access tasks. An increasingly popular storage optimization strategy is the use of a storage area network, which makes every storage device on a network accessible to all of a network's servers.

Storage area networks are relatively simple in concept, although they're not quite so simple to implement. A SAN is a subnetwork made up of heterogeneous storage resources and servers tied together with a high-speed connection. While a SAN is often an effective storage solution, sharing data between servers, managing storage across heterogeneous server platforms, and getting Fibre Channel devices from different vendors to work together is seldom an easy proposition.

A SAN consists of one or more storage devices, a switch or hub, adapters in the servers, plus backup and management. What makes a SAN difficult to manage? All of its components are designed for maximum throughput, often at the expense of interoperability.

To learn the state of the art, InformationWeek recently conducted tests of four SAN products at the University of Hawaii's Advanced Network Computing Lab. Our tests show that managing the finished SAN can be difficult. All the tested products work, but they offer very different levels of integration and functionality.

We tested products integrated by QLogic Corp. and Interphase Corp., and two products from Compaq. Prices as tested range from $19,543 to $143,787 for 150 Gbytes to 764 Gbytes of storage. These numbers represent a range of capabilities, from a basic SAN that would be appropriate for a small graphics department to a system that can handle the demands of a large global company.

The more-expensive Compaq products include much greater capacity, a tape library, and a storage router. The Interphase product includes a single tape drive and storage router, while the QLogic product lacks a tape drive and management software.

The Compaq products include support for Sun Microsystems' Sparc server, which the others don't have. They have capacities comparable to the others but without the Sun compatibility and Compaq's tape library are inexpensive. Compaq products also include the best management package.

Other items that make a big difference in cost are the hub or switch and the type of storage. Fibre Channel switches are two to three times more expensive than hubs for the same number of ports. They provide the same advantages as Ethernet switches--problems are limited to a single port on the switch, rather than every device connecting to the hub. In addition, overall performance is higher because each port has 1 Gbps of throughput, rather than the whole hub sharing one gigabit per second of throughput.

Some of the storage devices are redundant arrays of independent disks, while others are "just a bunch of disks, or JBODs." A RAID device has inherent redundancy: A disk in the array can fail without loss of data. The alternative doesn't have this capability, although it can be added through software on the server or storage controller. RAID is more expensive than a JBOD because it has an integrated controller.

We asked the vendors to bring two storage devices with at least 150-Gbyte capacity, one tape drive, at least one SAN switch, a management application, one server, and host bus adapters for two more servers, including a Sun workstation and Linux server. The test scenario was designed to simulate a typical heterogeneous SAN environment, with either Windows NT or Windows 2000 servers and either Sun Solaris or Linux servers.

The test's primary focus was to evaluate the SAN as a complete product--not to test throughput per se, but to look at manageability, interoperability, and ease of setup and use. We found that SANs are a viable technology today but aren't for the faint-hearted. A heterogeneous product with multiple vendors supporting multiple platforms and one that encompasses management and ease of use isn't yet available. This is an area in which systems integrators will continue to make good margins for the next few years.

Compaq's MA8000 is the best, most complete SAN product we looked at. Compaq not only provides all of the pieces asked, but it's the only company to demonstrate compatibility with non-x86 servers and to offer a complete and integrated management product. On the other hand, the Compaq products are Compaq-only, in spite of the vendor's "Open SAN" claims. Using its host bus adapters in non-Compaq servers isn't supported, although it should be possible. Everything is Compaq-branded, other than the Sun workstation used to test non-x86 compatibility, and Compaq resisted suggestions that we test with other vendor's equipment.

The smaller of Compaq's products, the StorageWorks RA4100 SAN, is geared for a handful of servers. It's cost-effective, with good management tools, but has limited expandability. Connectivity within the RA4100 is through the Compaq Fibre Channel Arbitrated Loop Switch 8, which normally contains eight ports but can be upgraded through an expansion module with three additional ports. The switch that Compaq brought to the test has all 11 ports.

The RA4100 is designed only for x86-based servers. Compaq brought a ProLiant 8500 equipped as a four-way Pentium III Xeon machine running Windows 2000 Server. We also used two Compaq servers: a ProLiant DL380 running Red Hat Linux 6.2 and a DL580 running Windows 2000 Server.

Compaq declined the Dell PowerEdge 4400 and HP NetServer LH6000 we offered to demonstrate multivendor interoperability. Each of the servers was configured with a Compaq Fibre Channel host bus adapter, which requires a 66-MHz 64-bit PCI slot. According to Compaq, its SAN supports Banyan, NetWare, OS/2, and Windows NT 4.

On the storage side, Compaq brought two StorageWorks RAID Array 4100 disk enclosures, each of which offers 12 drive bays. One is equipped with 18-Gbyte Ultra3 (Ultra160) SCSI hard drives; the other has the same-size drives, but uses Ultra2 SCSI disks. Compaq uses the same drive enclosures in the disk enclosures that it uses for internal storage in the latest ProLiant series. The disks are configured as eight logical units (LUNs), four on each disk enclosure, managed by a Compaq RA4000 SAN controller unit.

For backup, Compaq brought a StorageWorks TL891 minilibrary, which contains one 5-Mbyte per second (native) DLT drive and 35-Gbyte (native) capacity tapes. The SCSI-based library connects to the FC-AL switch using Compaq's new modular data router, which is a 1U-high Fibre Channel-to-SCSI with a single FC port and two SCSI connections.

There are different administrative tools for different parts of the SAN, such as the intuitive StorageWorks Command Console for configuring the FC-AL switch; it's a server-side product that runs on a server connected to the switch, and provides a command-line and Web-based interface to configure the switch and the LUNs. A separate utility provides local and remote management (via Ethernet) of the Fibre Channel host bus adapters.

Both of these utilities may be launched from within Compaq's Integrated Management Extended Edition (CIM-XE) management application, and they can roll alerts to CIM-XE to be handled. For a test, we configured Command Console to fire a thermal alert at 20 degrees Celsius; the alert immediately fired and percolated up to CIM-XE, which then notified us remotely of the problem.

The entire RA4100 SAN worked perfectly, but because Compaq doesn't configure it for high availability or redundancy, we were unable to test or verify any of those features.

Compaq's second product, its midsize StorageWorks MA8000 SAN, is more comprehensive--not only more scalable, but with higher-end management tools and the ability to support non-x86-based servers. The MA8000 is Compaq's midrange SAN; the company also makes a higher-end product, the MA12000.

The MA8000 is built around a Fibre Channel fabric switch. Compaq brought two eight-port switches, al-though 16-port switches are also available. The Switch/8 is Simple Network Management Protocol (SNMP)-enabled, has redundant power and cooling, and can be managed via an intuitive Web-based management suite. Compaq connects the two switches in a fully parallel arrangement with two host bus adapters in each server and dual connections to each system. There are no cross-connects between the parallel Fibre Channel networks, which provides the ultimate in redundancy for fault tolerance.

For servers, the Compaq team used the same systems as with the RA4100 SAN test: a ProLiant 8500 running Windows 2000 Server and two other ProLiant servers running Linux and Windows 2000. To demonstrate that the MA8000 SAN is designed to support non-x86 systems, the team brought and connected a Sun Microsystems Ultra5 workstation running Solaris 7, equipped with dual SBus-based FCE-1063 Fibre Channel host bus adapters from JNI Corp. According to Compaq, the MA8000 also supports AIX, HP-UX, Irix, NetWare, OpenVMS, Tru64 Unix, and Windows NT.

For storage, Compaq's MA8000 RAID controller unit drives up to six SCSI channels with a maximum of 12 drives per controller. When new firmware for the MA8000 ships, that limitation will be raised to 14 drives per controller. For this test, the company chose to bring its SW4254 disk arrays, each of which can drive as many as 14 one-inch Ultra2 SCSI devices. The MA8000 can support six SW4254 arrays, for a total of 72 disks, with the potential to upgrade to 84 disks when the firmware is revised later this year.

The Compaq team brought only a single SW4254, equipped with 14 18.2-Gbyte drives. The company also moved the same TL891 tape library and Modular Data Router, which it used for the RA4100 SAN, over to the MA8000 SAN.

The redundancy and high-availability features built into the MA8000 system are impressive. The MA8000 controller includes multiple, hot-pluggable rechargeable battery units for maintaining the RAID subsystem, as well as an environmental monitor and multiple N+1 power supplies and fans.

The system controller contains dual Fibre Channel connections, and is configured to run them both in parallel, shifting all traffic from one port to another in the event of a fault on the Fibre Channel network.

To provide automatic host bus adapter failover on the servers, a pricey software agent called SANworks Secure Path must be installed on each machine: Secure Path is priced at $4,000 per Windows NT or Windows 2000 server and $6,000 for each Sun server connected directly to the SAN.

Another impressive high-availability feature was the MA8000's ability to create a pool of global hot-spare drives that can be on any SCSI channel on the SAN. If a RAID 5 array or RAID 1 mirror set, created to service a single server, experiences a drive failure, the SAN automatically al-locates a spare drive to that array.

In our tests, we created a pool of several hot spares, and then rapidly disabled RAID 1-and RAID 5-based logical units by physically pulling the drives; we could see the global hot spares being allocated to replace them.

We caused a number of other failures on the SAN, ranging from unplugging GigaBit Interface Converters, devices used to attach network devices to fiber-based transmission systems, to turning off one of the parallel SAN switches. After a pause of less than five seconds, the MA8000 recovered and kept operating each time.

The breadth and depth of manageability of the MA8000 were also impressive. The system can be administered through Compaq Insight Manager using applications launched either from within Insight Manager or running separately, but whose alerts and reports can be monitored by Insight Manager.

Overall, control of the MA8000 controller is handled by a Windows NT/ 2000 agent, Command Console, which must run on a server or workstation directly connected to the SAN. Command Console can be administered through either a graphical or command-line interface directly on the machine hosting the agent or via a Win32 graphical application running out-of-band on the Ethernet network.

The Command Console application not only reports on the SAN's health, but is also the tool used to configure LUNs, global hot spares, and otherwise administer the storage system. A separate utility is required to manage the SAN switches.

There's no central management of the SecurePath failover utility, so it must be configured separately for each server on the SAN, although it can report via SNMP if a failover takes place or if a spare pathway goes down.

The MA8000 SAN does it all with high reliability, scalability, and even interoperability with Sun's Solaris. It's relatively expensive, but there's no doubt that it works and works well.

We're also very impressed with the level of multivendor support that Interphase demonstrates with its product--the only products bearing the company's own name are its innovative Gigabit Ethernet/Fibre Chan-nel card and its FibreView software. The flip side is that the management software isn't comprehensive and the automatic failover features aren't complete.

The first link in the SAN was Interphase's impressive 5570 SlotOptimizer host bus adapter, which provides Gigabit Ethernet and SCSI-over-Fibre Channel ports on a single 66-MHz, 64-bit PCI card. For systems with few slots to spare, full LAN and SAN connectivity would only take a single PCI slot--and a fully redundant product would take only two slots.

For our test, Interphase didn't build a redundant Fibre Channel SAN. The cards are inserted into a four-way Compaq ProLiant DL580 running Windows 2000, a dual-processor ProLiant DL380 running Linux, and a Dell PowerEdge 4400 running Windows 2000, brought by Interphase. Because the 5570 card was scheduled to begin shipping a few weeks after the test's completion, Interphase used beta drivers for our test.

The Gigabit Ethernet part of the host bus adapters are connected to an Alteon Systems' 180 switch, which provides IP for the test. The Fibre Channel part is initially connected to an eight-port Vixel 7100 series Fibre Channel fabric switch; after the testing was complete, Interphase swapped a SANBox-8 switch from Ancor Communications Inc., now part of QLogic, to demonstrate interoperability.

Storage on the SAN is provided by two Storagepath SP-8BFC storage systems from SWS Corp., each of which contains eight 18-Gbyte Fibre Channel drives. There was tape backup, too: a 108-Gbyte-per-second Mammoth 2 SCSI tape drive from Exabyte Corp. bridged to the Fibre Channel switch via a 4200 SCSI-to-FC router from Crossroads Systems Inc.

The software driving the product ranged from Interphase's own Java-based FibreView Enterprise software for managing the host bus adapters to Legato Systems Inc.'s NetWorker for administering storage to Tivoli Systems Inc.'s SANergy for providing shared access to SAN volumes and application failover. Each of the switch manufacturers had software for managing its switch: Ancor's SAN Surfer and Vixel Corp.'s SAN InSite.

The Interphase testing went off without a hitch and passed nearly all our test requirements: showing heterogeneous servers playing multiple video streams while also performing file transfers and restoring a backup to another SAN volume with the Exabyte tape drive. Overall, it's a solid performance.

As mentioned earlier, Interphase didn't perform a failover test. According to the company, that's because its current host bus adapter drivers don't yet offer this feature; auto-failover support is expected in the fall.

Of course, a SAN means more than hardware. In a real-world deployment, it must also be manageable. Each of the products that makes up the Interphase offering contains its own management software optimized for a specific task.

Interphase's FibreView Enterprise, which we spent the most time with, is a comprehensive and easy-to-use Win32-based package for centrally managing host bus adapters and the nodes of the Fibre Channel network. The IP-based software works out-of-band over the Ethernet network, not Fibre Channel.

Interphase demonstrates reasonably tight coupling between FibreView Enterprise and Vixel's SAN InSite software; SAN InSite could launch FibreView, and some alerts from FibreView could roll up into SAN InSite. Still, it would be a mistake to call the packages integrated; there are just some interprocess communication between them. Management-software connectivity between FibreView Enterprise and Ancor's SAN Surfer application is missing, although according to Interphase officials, such integration is planned for the future.

There's no integration between FibreView Enterprise and the other software used for the SAN, such as Tivoli's SANergy and Legato's NetWorker. Interphase's FibreView Enterprise also doesn't provide native fault alerts; if you're not watching the screen, you won't see the errors. For functions such as E-mail or paged alerts, you'd need to use a third-party package to poll the SAN via SNMP.

Overall, Interphase was able to demonstrate a complete product and was the only vendor to bring a diverse, multivendor product likely to be representative of what businesses deploy in the real world. We're particularly impressed with Interphase's interoperability and its dual Fibre Chan-nel/Gigabit Ethernet card, which could lead to a fully redundant SAN/Ether-net product using low-profile servers. With better integrated management and with automatic failover within or between Fibre Channel networks, Interphase's offering would be a tough product to beat.

QLogic entered a minimal configuration in our tests. Its system consisted of two servers on the SAN, a single switch, and a single storage device. Best known as a host bus adapter and chipset manufacturer for both PCI-to-SCSI and PCI-to-Fibre Channel, QLogic had some difficulty obtaining additional hardware for the tests, and was plagued by inexplicable difficulties with the Fibre Channel switch it brought for the test. The consensus was that the switch was damaged in shipping. Because of the limited hardware, QLogic wasn't able to perform many of the SAN tests that we had requested in the review plan. Even so, QLogic was able to construct and demonstrate a small working SAN product with solid failover functionality.

QLogic's SAN product uses two servers with QLA2200F/66 host bus adapters. The QLA2200F series is an optical card containing a 1-Gbyte per second Fibre Channel connection capable of full-duplex (200-Mbytes per second) data transfers. The "66" designation specifies that the card requires a 64-bit 66-MHz PCI slot. The servers were a four-Pentium III Xeon-based Compaq ProLiant DL580 running Windows NT Server and an HP LH-6000 six-way Pentium III Xeon also running Windows NT Server.

QLogic used a third server, a two-way Pentium III-based Compaq ProLiant DL380 running Windows 2000, as a management console. It connected to the other servers via Fast Ethernet and didn't have a host bus adapter.

Ancor provided the Fibre Channel fabric switch for the test. Ancor's SANBox 16HA (high availability) switch has 16 full-duplex Fibre Channel ports and can be equipped with dual non-hot-swappable power supplies, though the switch used for this test had only a single power supply.

The sole storage device is one of Sun's new StorEdge T3 Fibre Channel RAID disk arrays, which contains nine 18-Gbyte FC-AL drives; this product was the only one in our lab test to use Fibre Channel drives, rather than SCSI drives. The StorEdge was partitioned into four LUNs for the test, two for each server.

During the test, both servers read streaming video from one LUN, while copying data back and forth between the other two LUNs. The Fibre Channel drives and RAID controller contribute to Sun's relatively high price of $32,500 for the storage box alone.

QLogic chose to demonstrate a SAN with redundant paths that had two host bus adapters in each server, each connected to separate switch ports. There was only one Fibre Channel connection from the Ancor switch to the StorEdge array.

The biggest difficulty with the QLogic test appeared to be faulty switch hardware, and those problems limited the amount of time we had for testing. The test was also limited in that QLogic only brought a single storage device and no backup device, and chose to use only Windows NT servers, so we were unable to view cross-platform compatibility and a live data backup/restore operation.

Despite those limitations, QLogic demonstrated a working SAN that ran on Windows NT and was capable of handling simultaneous data transfers. We're impressed by its automatic failover of the redundant fiber links to redundant paths, as configured in its QLconfig management software.

With dual host bus adapters in each server running to the switch, the system recovers from a hardware failure (simulated by unplugging a GBIC from the switch) in less than five seconds. In another case, we used the QLview management tool to disable a server's active host bus adapter, the server hung on for about three minutes before recovering and switching to the redundant host bus adapter and path.

Our biggest complaint is the lack of management. The QLogic host bus adapters come with two tools, which can run on Linux, NetWare, and Windows: QLview for Fibre, which provides admin capabilities for host bus adapters and logs events to the NT Event Log; and QLconfig, which administers LUNs and failover paths.

Both tools work on local host bus adapters, as well as on host bus adapters in network-accessible servers using Ethernet or IP-over-Fibre Channel. Neither presents a complete product. When we failed one of the servers' redundant links, for example, the only possible notification was a blinking toolbar icon on the management console--there are no E-mail, SNMP alerts, or other notification.

When we pulled up QLconfig and checked a path view, it showed us that a failover had taken place, but not where. To learn why, we had to fire up QLview, check the appropriate server, and drill down to the host bus adapter. All we saw was that traffic wasn't flowing--not a true management product for a production SAN.

The Ancor SANbox includes an embedded Web server that downloads a Java applet called SAN Surfer to a management station; the applet runs under Windows (we used it on a notebook) and on Linux, though Solaris is promised for the future. It, too, is a configuration tool, not a management product. Although it shows the switch's current configuration, there's no alert and no active monitoring of the SANBox's SNMP traps. To tell if a port goes down, you'd have to be actively managing that switch to visually detect that a state has changed.

The Sun StorEdge array has its own management utility, called StorEdge Component Manager, which isn't integrated into the rest of the product.

Overall, we're impressed with the configurability and reliability capabilities of the QLogic/Ancor product. We can't attest to its backup, scalability, or cross-platform abil-ities, and the management is weak, so we can't give QLogic top marks, but there's no doubt that QLogic has the connectivity issue covered.

---